Configuration¶

The individual configuration statements are documented in the example YAML configuration file, which is reproduced at the end of this chapter.

The configuration consists of the following general sections:

Instance name
Storage backend for snapshots
LMDBs to sync
Sync parameter tweaking
Monitoring and logging

Lightning Stream allows environment variables in the YAML configuration files, e.g.:

instance: ${LS_INSTANCE}

Instance name¶

Warning

Every instance MUST have a unique instance name.

Every instance MUST have a unique instance name. This instance name is included in the snapshot filenames and used to ignore its own snapshots.

If you accidentally configure two instances with the same name, the following will happen:

They will not see each other's changes, unless a third instance happens to include them in a snapshot.
Other instances may not see all the changes from both instances, because they will only load the most recent snapshot for this instance name.

The instance name can either be configured in the config file:

instance: unique-instance-name-here

Or it can be passed using the --instance commandline flag, but then you must be careful to always pass it.

As mentioned above, environment variables can be used in the YAML configuration, which can be useful for the instance name:

instance: ${LS_INSTANCE}

The instance name should be composed of safe characters, such as ASCII letters, numbers, dashes and dots. It MUST NOT contain underscores, slashes, spaces or other special characters. Basically, what is allowed in a hostname is safe.

Storage¶

Lightning Stream uses our Simpleblob library to support different storage backends. At the moment of writing, it supports S3 and local filesystem backends.

S3 backend¶

This is currently the only backend that makes sense for a production environment. It stores snapshots in an S3 or compatible storage. We have tested it against Amazon AWS S3 and MinIO servers.

MinIO example for testing without TLS:

storage:
  type: s3
  options:
    access_key: minioadmin
    secret_key: minioadmin
    region: us-east-1
    bucket: lightningstream
    endpoint_url: http://localhost:9000

Currently available options:

Option	Type	Summary
access_key	string	S3 access key
secret_key	string	S3 secret key
region	string	S3 region (default: "us-east-1")
bucket	string	Name of S3 bucket
create_bucket	bool	Create bucket if it does not exist
global_prefix	string	Transparently apply a global prefix to all names before storage
prefix_folders	bool	Show folders in list instead of recursively listing them
endpoint_url	string	Use a custom endpoint URL, e.g. for Minio
tls	tlsconfig.Config	TLS configuration
init_timeout	duration	Time allowed for initialisation (default: "20s")
idle_conn_timeout	duration	Maximum time a connection is allowed to remain idle before closing itself (default: "90s")
max_idle_conns	integer	Controls the maximum number of idle (keep-alive) (default: 100)
dial_timeout	duration	The maximum amount of time a dial will wait for a connect to complete (default: "10s")
dial_keep_alive	duration	Specifies the interval between keep-alive probes (default: "10s")
tls_handshake_timeout	duration	Specifies the maximum amount of time to wait for a TLS handshake (default: "10s")
client_timeout	duration	Specifies a time limit for requests made by this HTTP Client. (default: "15m")
use_update_marker	bool	Reduce LIST commands, see link below
update_marker_force_list_interval	duration	See link below for details

The use_update_marker option can reduce your AWS S3 bill in small personal deployments without compromises on update latency, as GET operations are 10 times cheaper than LIST operations, but it cannot reliably be used when you are using a bucket mirror mechanism to keep multiple buckets in sync.

You can find all the available S3 options with full descriptions in Simpleblob's S3 backend Options struct.

Filesystem backend¶

For local testing, it can be convenient to store all snapshots in a local directory instead of an S3 bucket:

storage:
  type: fs
  options:
    root_path: /tmp/snapshots

LMDBs¶

The lmdbs section configures which LMDB databases to sync. One Lightning Stream instance can sync more than one LMDB database. Snapshots are independent per LMDB.

Every database requires a name that must not change over time, as it is included in the snapshot filenames. The name should only contain lowercase letters and must not contains spaces, underscores or special characters. If you are bad at naming things and have only one database, "main" is a good safe choice.

A basic example is as follows:

lmdbs:
  main:
    path: /path/to/lmdb/dir
    options:
      create: true
      map_size: 1GB
    schema_tracks_changes: true  # native schema

The path option is the path to the LMDB directory, or file if options.no_subdir is true.

Some commonly use LMDB options:

no_subdir: the LMDB does not use a directory, but a plain file (required for PowerDNS).
create: create the LMDB if it does not exist yet.
map_size: the LMDB map size, which is the maximum size the LMDB can grow to. You can specify the units, such as KB, MB, GB etc.

Sync parameters¶

There are some top-level sync parameters that you may want to tweak for specific deployments, but these do have sensible defaults, so you probably do not need to.

The ones you are more likely to want to change are:

storage_poll_interval: how often to list the storage to check for new snapshots (default: 1s)
storage_force_snapshot_interval: how often to force a snapshot when there are no changes. This is also the interval at which we force a snapshot instead of a delta when there are frequent changes (default: 1h)
lmdb_poll_interval: how often to check the LMDB for changes (default: 1s). This check is very fast by itself, but you may want to adjust it if you want to limit the rate at which an instance can generate snapshots when there are frequent updates.
max_deltas: The maximum number of deltas per snapshot. Once this is reached, a snapshot will be generated instead (default: 1000)
max_total_change_percent: The cumulative fraction of the database entries that may have been changed in the current deltas series before a new snapshot is forced (default: 20)

More can be found with descriptions in the example configuration.

Monitoring and logging¶

Example of a few logging and monitoring options:

# HTTP server with status page, Prometheus metrics and /healthz endpoint.
# Disabled by default.
http:
  address: ":8500"    # listen on port 8500 on all interfaces

# Logging configuration
log:
   level: info        # "debug", "info", "warning", "error", "fatal"
   format: human      # "human", "logfmt", "json"
   timestamp: short   # "short", "disable", "full"

health:
   # ... see example config

Example config with comments¶

This example configuration assumes a PowerDNS Authoritative server setup with native schemas, but it explains every available option.

# This example aims to document all available options. If an option is
# commented out, its default value is shown, unless indicated otherwise.

# Every instance of LS requires a unique instance name. This instance name
# is included in the snapshot filenames and used by instances to discover
# snapshots by other instances.
# LS supports the expansion of OS environment variables in YAML configs, with
# values like ${INSTANCE}, which can simplify the management of multiple
# instances.
instance: unique-instance-name

# Check the LMDBs for newly written transactions at this interval, and write
# a new snapshot if anything has changed. The check is very cheap.
#lmdb_poll_interval: 1s

# Periodically log LMDB statistics.
# Useful when investigating issues based on logs. Defaults to 30m.
lmdb_log_stats_interval: 5m
# Include LMDB memory usage statistics from /proc/$PID/smaps for metrics.
# This can be expensive on some older kernel versions when a lot of memory
# is mapped.
#lmdb_scrape_smaps: true

# Check the storage for new snapshots at this interval
#storage_poll_interval: 1s

# When a download or upload fails, these items determine how often and at what
# interval we will retry, before existing LS with an error.
#storage_retry_interval: 5s
#storage_retry_count: 100
#storage_retry_forever: false

# Force a snapshot once in a while, even if there were no local changes, so
# that this instance will not be seen as stale, or removed by external cleaning
# actions.
# Note that external cleaning mechanisms are not recommended, it is safer to use
# the 'storage.cleanup' section.
#
# In the Enterprise version this is also the interval at which we
# force a snapshot instead of a delta when there are frequent changes.
# Here we decrease the default from four hours to one hour to avoid
# having too many deltas per snapshot. With a scan interval of one
# second and a tiny database we would end up with no more 3600 deltas
# per snapshot, which would be on the high side. In realistic larger
# deployments where a cycle takes 6 seconds we would not generate
# more than 600 deltas per snapshot with this default.
# The `deltas` configurations options provide more precision in controlling
# deltas.
#storage_force_snapshot_interval: 1h

# MemoryDownloadedSnapshots defines how many downloaded compressed snapshots
# we are allowed to keep in memory for each database (minimum: 1, default: 3).
# Setting this higher allows us to keep downloading snapshots for different
# instances, even if one download is experiencing a hiccup.
# These will transition to 'memory_decompressed_snapshots' once a slot opens
# up in there.
# Increasing this can speed up processing at the cost of memory.
#memory_downloaded_snapshots: 3

# MemoryDecompressedSnapshots defines how many decompressed snapshots
# we are allowed to keep in memory for each database (minimum: 1, default: 2).
# Keep in mind that decompressed snapshots are typically 3-10x larger than
# the downloaded compressed snapshots.
# Increasing this can speed up processing at the cost of memory.
#memory_decompressed_snapshots: 2

# Run a single merge cycle and then exit.
# Equivalent to the --only-once flag.
#only_once: false


# Enterprise delta config
deltas:
  # MaxDeltas is the maximum number of deltas per snapshot. Once this
  # is reached, a snapshot will be generated instead.
  #max_deltas: 1000

  # MaxTotalChangePercent is the cumulative fraction of the database entries
  # that may have been changed in the current deltas series before a new
  # snapshot is forced.
  # The default of 20 percent means that if a database has 1 million entries,
  # the current delta series can at most contain 200,000 change entries.
  # Every change record is counted as one, there is no special handling for
  # individual records that have been changed more than once.
  #max_total_change_percent: 20

  # CleanConcurrency sets the number of parallel delete requests when cleaning deltas
  # from the storage (default: 20).
  #clean_concurrency: 20

  # CleanBatchTimeout sets a timeout for cleaning all deltas of one snapshot.
  # This has been added out of an abundance of caution and should never be hit under normal circumstances.
  # Set this conservatively high (default: 1h).
  #clean_batch_timeout: 1h


# The 'lmdbs' section defines which LMDB database need to be synced. LS will
# start one internal syncer instance per database.
# The keys in this section ('main' and 'shard' here) are arbitrary names
# assigned to these databases. These names are used in logging and in the
# snapshot filenames, so they must match between instances and not be changed
# later.
lmdbs:
  # In PDNS Auth, this database contains all the data, except for the records.
  main:
    # Path to the LMDB database. This is typically the directory containing
    # a 'data.mdb' and 'lock.mdb' file, but PDNS Auth uses the 'no_subdir'
    # option, in which case this is a path to the data file itself.
    path: /path/to/pdns.lmdb

    # LMDB environment options
    options:
      # If set, the LMDB path refers to a file, not a directory.
      # This is required for PDNS Auth.
      no_subdir: true

      # Create the LMDB if it does not exist yet.
      create: true

      # Optional directory mask when creating a new LMDB. 0 means default.
      #dir_mask: 0775
      # Optional file mask when creating a new LMDB. 0 means default.
      #file_mask: 0664

      # The LMDB mapsize when creating the LMDB. This is the amount of memory
      # that can be used for LMDB data pages and limits the file size of an
      # LMDB. Keep in mind that an LMDB file can eventually grow to its mapsize.
      # A value of 0 means 1GB when creating a new LMDB.
      #map_size: 1GB

      # The maximum number of named DBIs within the LMDB. 0 means default.
      #max_dbs: 64

    # This indicates that the application natively supports LS headers on all
    # its database values. PDNS Auth supports this starting from version 4.8.
    # Earlier versions required this to be set to 'false'.
    # Application requirements include, but are not limited to:
    # - Every value is prefixed with an 24+ byte LS header.
    # - Deleted entries are recorded with the same timestamp and a Deleted flag.
    # When enabled, a shadow database is no longer needed to merge snapshots, and
    # conflict resolution is both more accurate and more efficient.
    # For Lightning Stream Enterprise this field is required
    schema_tracks_changes: true

    # This allows setting options per-DBI.
    # Currently, the only option supported is 'override_create_flags', which is
    # should only be used when you need both options.create=true
    # and have snapshots created by a pre-0.3.0 version of LS. Newer snapshots
    # have all the information they need to create new DBIs.
    dbi_options: {}

  # In PDNS Auth, this database contains all the records.
  # The various options available are the same as in the 'lmdb.main' section above.
  shard:
    path: /path/to/pdns.lmdb-0
    options:
      no_subdir: true
      create: true


# Sweeper settings for the LMDB sweeper that removed deleted entries after
# a while, also known as the "tomb sweeper".
#
# The key consideration for these settings is how long instance can be
# expected to be disconnected from the storage (out of sync) before
# rejoining. If the retention interval is set too low, old records that
# have been removed during the downtime can reappear, which can cause
# major issues.
#
# When picking a value, also take into account development, testing and
# migration systems that only occasionally come online.
#
sweeper:
  # Enabled controls if the sweeper is enabled.
  # It is DISABLED by default, because of the important consistency
  # considerations that depend on the kind of deployment.
  # When disabled, the deleted entries will never actually be removed.
  # Stats are only available when the sweeper is enabled.
  #enabled: false

  # RetentionDays is the number of DAYS of retention. Unlike in most
  # other places, this is specified in number of days instead of Duration
  # because of the expected length of this.
  # This is a float, so it is possible to use periods shorter than one day,
  # but this is rarely a good idea. Best to set this as high as possible.
  # Default: 370 (days, intentionally on the safe side)
  #retention_days: 370

  # Interval is the interval between sweeps of the whole database to enforce
  # RetentionDays.
  # As a guideline, on a fast server sweeping 1 million records takes
  # about 1 second.
  # Default: 6h
  #interval: 6h

  # FirstInterval is the first Interval immediately after
  # startup, to allow one soon after extended downtime.
  # Default: 10m
  #first_interval: 10m

  # LockDuration limits how long the sweeper may hold the exclusive write
  # lock at one time. This effectively controls the maximum latency spike
  # due to the sweeper for API calls that update the LMDB.
  # This is not a hard quota, the sweeper may overrun it slightly.
  # Default: 50ms
  #lock_duration: 50ms

  # ReleaseDuration determines how long the sweeper must sleep before it
  # is allowed to reacquire the exclusive write lock.
  # If this is equal to LockDuration, it means that the sweeper can hold the
  # LMDB at most half the time.
  # Do not set this too high, as every sweep cycle will record a write
  # transaction that can trigger a snapshot generation scan. It is best
  # to get it over with in a short total sweep time.
  # Default: 50ms
  #release_duration: 50ms

  # RetentionLoadCutoffDuration is the time interval close to the RetentionDays
  # limit where we will not load deletion markers from remote snapshots,
  # because they would soon be eligible for removal by the sweeper anyway.
  # Only set this if you understand the implications.
  # Default: 1% of the duration corresponding to the retention_days setting.
  #retention_load_cutoff_duration: 0


# Storage configures where LS stores its snapshots
storage:
  # For the available backend types and options, please
  # check https://github.com/PowerDNS/simpleblob

  # Example with S3 storage in Minio running on localhost.
  # For the available S3 backend options, check the Options struct in
  # https://github.com/PowerDNS/simpleblob/blob/main/backends/s3/s3.go#L43
  type: s3
  options:
    access_key: minioadmin
    secret_key: minioadmin
    region: us-east-1
    bucket: lightningstream
    endpoint_url: http://localhost:9000

  # Example with local file storage for local testing and development
  #type: fs
  #options:
  #  root_path: /path/to/snapshots

  # Periodic snapshot cleanup. This cleans old snapshots from all instances,
  # including stale ones. Multiple instances can safely try to clean the same
  # snapshots at the same time.
  # LS only really needs the latest snapshot of an instance, but we want to keep
  # older snapshots for a short interval in case a slower instance is still
  # trying to download it.
  # This is disabled by default, but highly recommended.
  cleanup:
    # Enable the cleaner
    enabled: true
    # Interval to check if snapshots need to be cleaned. Some perturbation
    # is added to this interval so that multiple instances started at exactly
    # the same time do not always try to clean the same snapshots at the same
    # time.
    interval: 5m
    # Snapshots must have been available for at least this interval before they
    # are considered for cleaning, so that slower instances have a chance to
    # download them.
    must_keep_interval: 10m
    # Remove stale instances without newer snapshots after this interval, but
    # only after we are sure this instance has downloaded and merged that
    # snapshot, and subsequently written a new snapshots that incorporates these
    # changes.
    remove_old_instances_interval: 168h   # 1 week

# HTTP server with status page, Prometheus metrics and /healthz endpoint.
# Disabled by default.
http:
  address: ":8500"    # listen on port 8500 on all interfaces

# Logging configuration
# LS uses https://github.com/sirupsen/logrus internally
log:
   level: info        # "debug", "info", "warning", "error", "fatal"
   format: human      # "human", "logfmt", "json"
   timestamp: short   # "short", "disable", "full"

# Health checkers for /healthz endpoint
# This is always enabled. This section allows tweaking the intervals.
#health:
  # Check if we can list the storage buckets
  #storage_list:
  #  interval: 5s             # Check every 5 seconds
  #  warn_duration: 1m0s      # Trigger a warning after 1 minute of failures
  #  error_duration: 5m0s     # Trigger an error after 5 minutes of failures
  #
  # Check if we successfully load snapshots
  #storage_load:
  #  interval: 5s
  #  warn_duration: 1m0s
  #  error_duration: 5m0s
  #
  # Check if we successfully store snapshots
  #storage_store:
  #  interval: 5s
  #  warn_duration: 1m0s
  #  error_duration: 5m0s
  #
  # Check if we started up and are ready to handle real traffic according to
  # some checks, like having loaded all available snapshots.
  #start:
  #  interval: 1s
  #  warn_duration: 1m0s
  #  error_duration: 5m0s
  #  # If true, a failing startup sequence will be included in the healthz
  #  # overall status. This can be used to prevent marking the node ready
  #  # before Lightning Stream has completed an initial sync.
  #  report_healthz: false
  #  # Controls if the healthz 'startup_[db name]' metadata field will be used
  #  # to report the status of the startup sequence for each db.
  #  report_metadata: true