Skip to content

Lightning Stream Enterprise

Lightning Stream Enterprise is a tool to sync changes between a local PowerDNS Authoritative Server LMDB (Lightning Memory-Mapped Database) and an S3 bucket in near real-time. This can be used in a multi-writer setup where any instance can update any data, with a global eventually-consistent view of the data in seconds.

This Enterprise version is based on the Open-Source Lightning Stream tool, with the following improvements:

  • Supports sending and receiving deltas - This greatly improves latency, particularly for large datasets, and/or high update rates
  • Additional Prometheus metrics
  • Prometheus alert rules for detecting issues in production deployments

Currently, the only supported usage for Lightning Stream Enterprise is the sync of LMDB databases in the PowerDNS Authoritative Nameserver (PDNS Auth). We are excited about how Lightning Stream Enterprise simplifies running multiple distributed PowerDNS Authoritative servers, with full support for keeping DNSSEC keys in sync.

Although the open-source Lightning Stream usage is not limited to the PowerDNS Authoritative server, Lightning Stream Enterprise is not supported for use with other applications.

Basic Operation

Lightning Stream Enterprise is deployed next to a PowerDNS Authoritative Server that uses LMDB for its data storage:

Overview

Its operation boils down to the following:

  • Whenever it detects that the LMDB has changed, it writes a delta (and at longer intervals, a full snapshot) of the data to an S3 bucket.
  • Whenever it sees a new delta or snapshot written by a different instance in the S3 bucket, it downloads the delta or snapshot and merges the data into the local LMDB.

The merge of a key is performed based on a per-record last-modified timestamp: the most recent version of the entry wins. Deleted entries are cleared and marked as deleted, together with their deletion timestamp. This allows Lightning Stream Enterprise to provide Eventual Consistency across all nodes.

Since PowerDNS Authoritative Server uses a carefully designed data schema, this approach can be used to support multiple simultaneously active writers. In other instances, it can often be used to sync data from one writer to multiple read-only receivers. Or it can simply create a near real-time backup of a single instance.