Skip to content

Architecture

The RPZ downloader is one of several producers in the OX Feed pipeline. It speaks DNS *XFR to an upstream Response Policy Zone server and produces the same OX Feed file layout (feed.json referencing snapshot_*.gz, delta_*.gz, and feed-codes.json) as the other OX Feed producers. The service never serves the feed to filter clients directly — front the feed tree with a static HTTP server (Caddy or nginx) for production use.

Role in Platform Filter

flowchart LR
    U[Upstream RPZ server<br/>e.g. Spamhaus] -- AXFR / IXFR --> S[(RPZ Downloader)]
    S -- write --> D[(Local feed tree)]
    D -- HTTP --> H[Caddy / nginx]
    H -- HTTP --> M[(OX Feed Mirror)]
    H -- HTTP --> C1[oxfeed-client A]
    M -. HTTP .-> C2[oxfeed-client B]

The RPZ downloader is the producer. The mirror is optional and is typically only deployed when many filter nodes share the same upstream across a WAN link.

The RPZ contract

On every tick the service opens a fresh TCP connection to feed.rpz_downloader.address and issues either an AXFR or IXFR query for feed.rpz_downloader.zone. Only CNAME records become feed entries; other record types are either silently ignored or logged as a warning.

Record types

Record type Handling
CNAME Translated into a feed entry. See CNAME-to-feed translation.
SOA Drives serial / delta-boundary tracking. Not written to the feed.
NS, DNAME Silently ignored (no-op per draft-vixie-dnsop-dns-rpz-00 §3.6).
DNSKEY, RRSIG, NSEC, DS Silently ignored (no-op per RFC 4034).
Anything else Logged as a warning and skipped.

Trigger types

Only QNAME triggers are supported. Records whose owner name ends in .rpz-ip, .rpz-nsdname, .rpz-client-ip, or .rpz-nsip are silently dropped. The upstream zone should not produce them; if it does they are simply not represented in the feed.

CNAME-to-feed translation

Each accepted CNAME contributes one row to the feed CSV: the owner name (stripped of the configured zone suffix and any trailing dot) followed by the classification code derived from the CNAME RDATA.

CNAME RDATA Classification code Action
. 1 RPZ NXDOMAIN Action
*. 11 RPZ NODATA Action
rpz-passthru. 2 RPZ PASSTHRU Action
rpz-drop. 14 RPZ DROP Action
rpz-tcp-only. 12 RPZ TCP_ONLY Action
anything else 13 RPZ LOCAL_DATA Action (redirect target encoded in RDATA)

Wildcard owner names (*.example.com.) are normalised by stripping the leading *.; the resulting CSV row covers the bare domain without an explicit subdomain marker.

The downloader publishes a fixed feed-codes.json describing the six codes above, so downstream clients render category names without any upstream coordination.

AXFR vs IXFR selection

The first XFR after startup is always AXFR because the service has no SOA serial to ask for changes against. On every successful XFR the serial advances and is stored in the feed metadata (last_source_version), so a subsequent restart resumes IXFR from the right point.

After the first AXFR, queries are IXFR until axfr_interval has elapsed since the last successful XFR. The next query is then forced to AXFR, and the cycle repeats.

The axfr_interval clock does not override interval. With interval = 13h and axfr_interval = 24h, an AXFR fires every 26 h (2 * interval), not every 24 h. Setting axfr_interval = 0 disables forced AXFR entirely; the only AXFR will be the one at startup (and again whenever the local serial is reset, e.g. by clearing the feed root).

Update loop

For each tick the service performs roughly the following steps:

  1. Build the *XFR message. AXFR or IXFR, optionally TSIG-signed.
  2. Open a TCP connection to address, subject to dial_timeout.
  3. Send the query and read records, subject to read_timeout and write_timeout.
  4. Translate records. CNAMEs become feed-CSV rows; everything else is ignored or warned about.
  5. Hand the CSV bytes off to the feed writer, which decides whether to write a delta and / or a snapshot.
  6. Sleep for interval. On a transient (network) error, backoff is added to the sleep, and the cycle retries until max_network_failure_duration has elapsed.
  7. Any non-network error returns from the run loop; the systemd unit's Restart=on-failure brings the process back.

Update lifecycle

The output side is shared with the OX Feed Command Source: once the RPZ source hands over CSV bytes, the feed writer drives the same snapshot / delta / feed-codes.json lifecycle.

  1. First successful XFR writes snapshot_00000001.gz, the fixed feed-codes.json, and feed.json referring to them. No delta is written.
  2. Subsequent XFRs produce a delta whenever the new CSV differs from the in-memory snapshot. Identical content (same hash) writes nothing and the cycle moves on.
  3. Forced snapshots. A fresh snapshot is written every feed.snap_interval so newly-connecting clients do not have to replay long delta chains.
  4. feed-codes.json. Rewritten only when the (fixed) code map changes — in practice never during normal operation.
  5. feed.json last. Until the manifest is updated the new files are invisible to downstream clients as a new version.
  6. Expire and clean. Old snapshots and deltas that fall outside feed.expiry are removed, then any file in the feed tree that no longer appears in feed.json is deleted.

Every storage call goes through the feed.retry policy, which retries transient backend errors and only surfaces a hard failure once the retry budget is exhausted.

Storage backends

The feed tree is written through simpleblob, so feed.root accepts a few shapes:

  • A bare filesystem path or file:///path — the default for production deployments.
  • A simpleblob://fs?root_path=/path URI — equivalent to the bare path form.
  • An s3://bucket?... URI for object-storage backends.
  • A memory:// URI — useful in tests, not for production.

The shipped example uses a bare filesystem path. Operators usually either pin feed.root to a directory in the YAML or pass --oxfeed-path /path on the command line.

Expiry

feed.expiry.snapshots and feed.expiry.deltas each accept the same four rules:

  • must_keep_count — never drop the last N entries.
  • must_keep_interval — never drop entries newer than this age.
  • max_keep_count — drop entries beyond the N newest, subject to must-keep rules.
  • max_keep_interval — drop entries older than this age, subject to must-keep rules.

Deltas that sit on the upgrade path from the latest snapshot to the latest delta version are always kept even if expiry would otherwise remove them — clients that lag behind need them to catch up without re-snapping.

In-memory pruning

The service keeps the last full feed batch in memory so it can compute deltas. Long-running instances accumulate stringstore entries for deleted domains, which is mostly harmless but can grow large on big feeds.

feed.prune_enabled turns on periodic compaction of that in-memory state. Pruning happens on two triggers:

  • Time-based. Every feed.prune_interval (default 24h).
  • Mutation-based. When more than half the batch has been mutated since the last prune, with a small lower bound (100k entries) so small feeds do not churn.

Pruning is a CPU-heavy operation that triggers a Go GC pass when it finishes. Leave it off unless oxfeed_feed_size_bytes shows unbounded growth.

Resilience and healthz

The RPZ downloader registers two healthz checks of its own, plus three shared with the rest of the OX Feed producer family:

  • rpz_recent_xfr_success — warns when more than warn_no_xfr_interval has elapsed since the last successful XFR. A startup grace period resets the clock to "now" so a freshly started service does not page during warm-up.
  • rpz_network_error — warns whenever the most recent fetch attempt finished with a network error.
  • feed_size, feed_version, feed_source_version — three feed-writer checks that return OK until the first successful update lands, then reflect the real feed state. See Operations → Healthz checks.

Fatal errors return from the run loop, the process exits non-zero, and the unit's Restart=on-failure brings it back.