Skip to content

Architecture

The mirror is an HTTP cache for OX Feeds. It does not transform the data: what the upstream serves is exactly what the mirror writes to disk and exposes (via a separate HTTP server, typically Caddy or nginx) to the filter-node clients.

Role in Platform Filter

flowchart LR
    ZD[zVelo downloader] -- feed --> O[(Origin HTTP server)]
    GF[gitfeed / commandsource / etc.] -- feed --> O
    O -- poll + GET --> M[(OX Feed Mirror)]
    M -- write --> D[(Local feed tree)]
    D -- HTTP --> C1[oxfeed-client A]
    D -- HTTP --> C2[oxfeed-client B]
    D -- HTTP --> C3[oxfeed-client ...]

The mirror itself does not serve the mirrored tree on its own status port. Operators front the local feed tree with a static HTTP server (Caddy or nginx are typical) that filter nodes point at via their oxfeed-client configuration.

Per-upstream worker model

Each entry under upstreams in the config gets its own goroutine. A worker:

  1. Sleeps until the upstream's poll_interval elapses or feed.json changes (the fetcher uses ETags so a poll without changes transfers no data).
  2. Pulls the new feed.json, applies any configured expiry rules to prune entries the operator does not want stored locally.
  3. Mirrors snapshots, then deltas, then feed-codes.json, then writes feed.json last.
  4. Walks the upstream's subdirectory and removes any file not in the manifest.

Workers are independent: a slow or failing upstream does not block the others. Per-upstream status is exposed on the status page (see Operations).

File lifecycle

For each file referenced by feed.json:

  1. Skip if valid. Existing files are kept if the on-disk size matches the manifest. If the size mismatches, the file is removed and refetched.
  2. GET. The mirror requests the file from the upstream. A Content-Length that disagrees with the manifest aborts the file.
  3. SHA256 during write. The response body is streamed through a sha256 hasher and a temporary file. A mismatch removes the temp file and the run retries on the next poll.
  4. Atomic rename. On success the temp file is renamed into place, so readers never see a partial file.
  5. 404 tolerance. A 404 marks the file as "Missing" for the run and continues with the rest. The next poll retries.
  6. Write feed.json last. Until the manifest is updated the downloaded files are not visible to clients as the new version.
  7. Clean. Files not referenced by snapshots, deltas, feed.json or feed-codes.json are removed from the subdir.

Expiry rules

The mirror always follows upstream removals: if a file leaves the upstream manifest it is removed locally on the next cycle. Per-upstream expiry rules only let operators trim further; they cannot keep files longer than the upstream does.

Each of expiry.snapshots and expiry.deltas accepts the same four knobs:

  • must_keep_count — never drop the last N entries.
  • must_keep_interval — never drop entries newer than this age.
  • max_keep_count — drop entries beyond the N newest, subject to the must-keep rules.
  • max_keep_interval — drop entries older than this age, subject to the must-keep rules.

A typical pattern is to keep many deltas but only a handful of snapshots locally; see Configuration → upstreams.expiry.

Resilience

  • Transient errors retry on the poll interval. A failed run waits poll_interval (capped at one minute for the fetcher's internal error backoff) and retries against the latest known feed.json.
  • Persistent fatal errors exit the service. systemd restarts the unit on failure.
  • Disk pressure is surfaced via healthz. The free_disk check warns at 5 GiB free and errors at 1 GiB free, regardless of expiry config.
  • Stale data is surfaced via healthz. If no upstream has completed a successful run in the last 24 hours the stale_data check warns.

See Operations → HTTP endpoints for the healthz interface.