Skip to content

Architecture

The command source converts the output of an external command into the OX Feed file layout: feed.json referencing snapshot_*.gz, delta_*.gz, and feed-codes.json. The service never serves the feed to filter clients directly — front the feed tree with a static HTTP server (Caddy or nginx) for production use.

Role in Platform Filter

flowchart LR
    A[Source command<br/>e.g. pdns-pf-admin-manage] -- stdout JSON --> S[(Command Source)]
    S -- write --> D[(Local feed tree)]
    D -- HTTP --> H[Caddy / nginx]
    H -- HTTP --> M[(OX Feed Mirror)]
    H -- HTTP --> C1[oxfeed-client A]
    M -. HTTP .-> C2[oxfeed-client B]

The command source is the producer. The mirror is optional and typically only deployed when many filter nodes share the same upstream across a WAN link.

The command contract

On every tick the service runs command_source.command through /bin/sh -c. The command must finish within command_source.timeout (it is killed otherwise) and is expected to emit a single line on stdout of the form:

OXFEED_GENERATOR_RESULT: {"v": 1, "csv_file": "/tmp/feed.csv", "codes_file": "/tmp/feed-codes.json", "version": "v123"}
JSON field Meaning
v Schema version. Must be 1.
csv_file Path to the feed CSV the command just wrote. The service removes this file after reading it.
codes_file Path to the matching feed-codes.json. The service removes this file after reading it.
version Optional opaque version tag. When set, it is passed back to the next invocation in OXFEED_LAST_VERSION so the command can wait for changes.

Environment variables passed to the command

Variable When set Purpose
OXFEED_LAST_VERSION After at least one successful run that returned a version The version tag the service accepted last. Lets the command short-circuit when nothing changed.
OXFEED_TMPDIR Always (one fresh directory per run) A scratch directory unique to this invocation. The service removes it on exit, so it is a safe place for the command to write csv_file and codes_file.

Exit codes

Code Meaning
0 Success. The service reads csv_file and codes_file.
11 Not modified. The service records this as a successful no-op cycle and waits for the next tick.
Anything else Failure. The error counter increments and the next tick retries.

Determinism and content hashing

The service hashes the contents of csv_file and codes_file and suppresses any delta where the hash matches the previous accepted content, regardless of the reported version. The command must therefore produce stable output:

  • No generation timestamps, no UUIDs, no goroutine-order artefacts.
  • A fixed sort order for rows so two runs over identical data produce byte-identical files.

The recommended pattern is to combine a content-change check inside the command (e.g. a --wait mode that blocks until the source data changes) with a command_source.interval that rate-limits how often the service polls.

Update lifecycle

For each accepted command run the service performs roughly the following steps:

  1. Hash and diff. Compute the content hash of the new CSV. If it matches the last accepted hash, nothing more happens (no delta, no snapshot, no feed.json update).
  2. Decide whether to snapshot. A snapshot is written when there is no parent version, when feed.snap_interval has elapsed since the last snapshot, or when a delta would otherwise need to outrank an existing snapshot in a way that would force clients to re-snap.
  3. Write the delta. Unless this is the very first version, a compressed delta file is written under delta_<version>.gz.
  4. Write the snapshot. When required, snapshot_<version>.gz is written and feed.snap_interval is reset.
  5. Update feed-codes.json. If the codes hash changed, the file is rewritten — this is independent of the snapshot/delta path.
  6. Write feed.json last. Until the manifest is updated the new files are invisible to downstream clients as a new version.
  7. Expire and clean. Old snapshots and deltas that fall outside feed.expiry are removed, then any file in the feed tree that no longer appears in feed.json is deleted.

Every storage call goes through the feed.retry policy, which retries transient backend errors and only surfaces a hard failure once the retry budget is exhausted.

Storage backends

The feed tree is written through simpleblob, so feed.root accepts a few shapes:

  • A bare filesystem path or file:///path — the default for production deployments.
  • A simpleblob://fs?root_path=/path URI — equivalent to the bare path form.
  • An s3://bucket?... URI for object-storage backends.
  • A memory:// URI — useful in tests, not for production.

The default in config.Default() is the in-memory backend, but the shipped example overrides it to a filesystem path. Most operators either pin feed.root to a directory or pass -o /path on the command line.

Expiry

feed.expiry.snapshots and feed.expiry.deltas each accept the same four rules:

  • must_keep_count — never drop the last N entries.
  • must_keep_interval — never drop entries newer than this age.
  • max_keep_count — drop entries beyond the N newest, subject to must-keep rules.
  • max_keep_interval — drop entries older than this age, subject to must-keep rules.

Deltas that sit on the upgrade path from the latest snapshot to the latest delta version are always kept even if expiry would otherwise remove them — clients that lag behind need them to catch up without re-snapping.

In-memory pruning

The service keeps the last full feed batch in memory so it can compute deltas. Long-running instances accumulate stringstore entries for deleted domains, which is mostly harmless but can grow large on big feeds.

feed.prune_enabled turns on periodic compaction of that in-memory state. Pruning happens on two triggers:

  • Time-based. Every feed.prune_interval (default 24h).
  • Mutation-based. When more than half the batch has been mutated since the last prune, with a small lower bound (100k entries) so small feeds do not churn.

Pruning is a CPU-heavy operation that triggers a Go GC pass when it finishes. Leave it off unless the metric oxfeed_feed_size_bytes shows unbounded growth.

Resilience and healthz

  • Per-storage retries. Backend errors retry under feed.retry; only a budget exhaustion returns the error up to systemd.
  • systemd restart on failure. A fatal error returns from Run, the process exits non-zero, and the unit's Restart=on-failure brings it back.
  • Healthz starts passive. The feed_size, feed_version, and feed_source_version checks return OK until the first successful update lands, so a freshly started instance does not page during warm-up. After the first update they reflect the actual feed state. See Operations → Healthz checks.