Skip to content

Architecture

The OX Feed Client sits at the boundary between the feed distribution layer and the filter runtime. Feed servers produce data; the client writes it into a local LMDB; the filter runtime reads from that LMDB at query time.

Role in Platform Filter

Feed servers generate and serve URL categorization data as static HTTP files. They can be zVelo downloaders, gitfeed converters, RPZ downloaders, HTTP API downloaders, or mirrors of any of these. Each runs independently and exposes a feed over HTTP.

The client connects to one or more of those feed servers, tracks which version it has locally, downloads only what has changed, and writes the data into a local LMDB database.

The filter runtime (Recursor, DNSdist, or Simulator) reads from that LMDB at query time via the oxfeed-lmdb Lua module to look up categories for a given domain or URL.

flowchart LR
    ZD[zVelo Downloader] -- HTTP feed --> C[(OX Feed Client)]
    GF[gitfeed] -- HTTP feed --> C
    MR[Mirror] -- HTTP feed --> C
    C -- writes --> DB[(LMDB)]
    DB -. reads .-> REC[Recursor / DNSdist]

The client does not push to the filter runtime. The runtime reads the LMDB directly; it picks up new data the next time it opens a read transaction.

Core concepts

Feed

A feed is a versioned, append-only stream of URL categorization data published by a feed server over HTTP. Each new version of a feed is either a full snapshot (complete dataset) or a delta (changes since a previous snapshot or delta).

The client downloads a snapshot when it first connects to a feed, or after it detects that its local version is too far behind to apply incremental deltas. After the initial load, it applies deltas so that only changed entries are transferred over the network.

Namespace

Every feed configured in the client has a namespace — a short string label that is stored in the LMDB alongside the data. The namespace determines which entries in the LMDB belong to which feed. It must be unique across all feeds in a given client config and must not change once data has been written; changing it is treated as a brand-new feed.

The filter runtime uses the namespace to locate the right entries when performing a lookup.

NSID

Internally, each namespace is assigned a small integer (the NSID) when the feed first writes to the LMDB. The NSID is stored in the database entries rather than the full namespace string to save space. The pdns-pf-oxfeed-client-query feeds command shows the NSID alongside each feed name.

Feed codes and code map

Each URL entry in the database is associated with one or more numeric codes that map to human-readable category names. The code map is a separate document (feed-codes.json) published by the feed server alongside the data files. The client downloads it separately, on its own poll interval (codes_poll_interval).

If a feed uses named codes (category overrides, domain lists), polling the code map frequently keeps the filter runtime up to date with category name changes without a full data reload.

Snapshot vs delta update strategy

On startup (or after a reset) the client downloads the latest full snapshot. For each subsequent poll cycle it requests the current feed.json index, finds the most recent delta it does not have locally, and downloads it. Multiple deltas may be applied in sequence if the client is catching up after a period of downtime.

Downgrade

If the version the client has locally disappears from the upstream feed server — for example, because the feed was rolled back to an earlier snapshot or restored from a backup — the client will not find a delta it can apply. After downgrade_after has elapsed without finding a valid continuation, the client discards its local state and downloads the current snapshot from scratch. See Configuration → feeds for the downgrade_after field.

Canary deployments

When the same feed is deployed to multiple filter nodes, the delay_updates_by field makes a specific instance hold back updates by a fixed duration. This lets one node act as a canary: if a newly published feed version causes problems, there is a window to intervene before the rest of the fleet fetches the same data. The upstream feed server must retain old snapshots for at least as long as the configured delay.

Update history

The client records each successful update in a metadata database stored alongside the main LMDB. The pdns-pf-oxfeed-client-query history command shows the per-feed update log, including the strategy used (snapshot or delta), entry counts, and timing.