Skip to content

Architecture

The HTTP API downloader sits between an upstream HTTP feed producer (such as oxfeed-vtfeed) and the rest of the OX Feed distribution chain. It is functionally analogous to the zVelo and RPZ downloaders — it takes an external dataset, normalizes it into the OX Feed file layout, and exposes the resulting feed directory for clients and mirrors to consume.

Role in Platform Filter

The upstream producer serves three HTTP endpoints — /full, /incremental, and /codes — over which it publishes a versioned stream of URL categorization data. The most common producer is oxfeed-vtfeed, which derives its data from VirusTotal, but any service that implements the same contract works.

The downloader polls those endpoints, writes the resulting snapshot and delta files into a local directory (configured by feed.root) and maintains a feed.json manifest plus a feed-codes.json code map.

The consumer is either the OX Feed client (pdns-pf-oxfeed-client) running on filter nodes, or an OX Feed mirror that fans the feed out across multiple data centers. Consumers fetch from a static HTTP server (Caddy or nginx) pointed at the downloader's feed directory; the downloader's own HTTP port is for observability, not for production feed serving.

flowchart LR
    U[Upstream HTTP API<br/>vtfeed or other producer] -- /full /incremental /codes --> D[(HTTP API Downloader)]
    D -- writes --> R[(Local feed directory)]
    R -. served by .-> S[Caddy / nginx]
    S -- HTTP --> C[oxfeed-client / oxfeed-mirror]

The downloader can optionally serve its own feed directory at /feed/ on its observability port via http.serve_feed. This is a convenience for development — the warning in the example config is explicit that production deployments should put a real HTTP server in front of the feed directory.

Upstream HTTP contract

The downloader expects three endpoints under the configured feed.http_upstream.url:

  • /full — returns the latest snapshot as a gzip-compressed body. The response includes a version identifier in its body metadata. On status 202 Accepted the downloader assumes the snapshot is being built and retries after error_interval.
  • /incremental?from_version=<v>&size=<n> — returns changes since version v, capped at n lines (configured by max_delta_size). Status 204 No Content means there are no new changes; 304 Not Modified means the same; 410 Gone means the requested version is too old and the downloader must fall back to a full snapshot.
  • /codes — returns the category code map as a JSON document. The path can be overridden with codes_url_override if the producer exposes it elsewhere.

The downloader explicitly requests Accept-Encoding: gzip for /full and /incremental and refuses responses that are not gzip-encoded. Optional HTTP Basic Auth (username / password) and custom headers are forwarded with every request.

Both successful and failed requests are recorded in the Prometheus metrics (see Operations → Metrics).

Run modes

Watch mode (the deployed default)

Started with -config <path> — the shipped systemd unit always does this. The downloader enters an infinite loop: it fetches codes in the background on codes_interval, polls for deltas on interval, and periodically writes snapshots according to snap_interval. It exits only on SIGTERM or a fatal upstream / storage error.

Run-once mode

Started without -config, or with run_once: true in the YAML. The downloader performs a single full snapshot + codes fetch, writes the resulting feed, and exits. This is intended for one-shot tooling and tests rather than production use.

Core concepts

Feed

A feed is a versioned, append-only series of URL-categorization updates served over HTTP. The downloader maintains exactly one feed in the configured feed.root directory. Multiple downloader instances (one per upstream) each write to a different feed root and run as separate systemd template instances.

Snapshot vs delta

A snapshot is the complete dataset at a given upstream version. A delta lists the entries that have changed since a known previous version. After the first successful snapshot, the downloader normally writes deltas; it falls back to a fresh snapshot when the upstream returns 410 Gone for the version it currently holds, or when the operator forces a reset by changing namespace_per_code.

Upstream version

Each /full and /incremental response carries the upstream's own version identifier in its body metadata. The downloader stores it in the feed metadata as last_source_version. On restart the downloader reads that value back and immediately requests the next delta, so a short outage does not require a full snapshot.

Feed codes

A separate JSON file (feed-codes.json) maps numeric category codes to human-readable names. The downloader polls this on its own codes_interval, independent of snapshot and delta polling, so category renames propagate quickly. The first codes fetch is awaited before the main snapshot/delta loop starts — this makes configuration errors against the upstream surface early.

Namespace per code

The feed-level namespace_per_code flag controls how entries are keyed in the underlying memdb when the feed is built. Changing it on an existing feed triggers a full reset (the feed UUID is regenerated and clients reload from scratch), so flip it only when you intend that.

Expiry

The downloader periodically removes old snapshot and delta files from the feed directory according to feed.expiry. Defaults keep the last three snapshots and at least the last 5 deltas spanning 6 h, with a hard cap of 200 deltas or 168 h. The expiry rules apply after the must-keep constraints, so an aggressive max_keep_* setting cannot delete history that a client may still need to apply.

Memdb pruning

While building each delta the downloader keeps an in-memory copy of the feed. For very large feeds the internal string store can grow faster than entries are added. Setting feed.prune_enabled: true makes the downloader prune the in-memory state when more than half the current batch has been mutated since the last prune, and on the periodic prune_interval. Pruning is CPU-intensive and is therefore opt-in; the example config flips it on for vtfeed.