Skip to content

OX Feed Gitfeed

The PowerDNS Platform Filter OX Feed Gitfeed (pdns-pf-oxfeed-gitfeed) turns a git-tracked CSV file into a versioned OX Feed. Every commit that changes the watched file becomes one feed version: a compressed snapshot, plus a delta against the previous version. The feed tree (feed.json, snapshot_*.gz, delta_*.gz, feed-codes.json) is written to a simpleblob backend — typically a local filesystem directory served by a static HTTP server.

The service is shipped as a systemd template unit (pdns-pf-oxfeed-gitfeed@<instance>.service). One instance is configured per logical feed by copying the example config and symlinking the template unit.

What it does

  • Opens a local git checkout, clones an http/ssh remote into memory, or drives a local repo from an external command source.
  • Walks the first-parent commit log from the oldest unprocessed commit to HEAD, and for each commit that touches git.file_path writes a snapshot and (after the first) a delta.
  • Encodes a sequential number and the first 32 bits of the git hash into each feed version, so version numbers are stable across reruns and survive a feed restart.
  • Mirrors the optional git.codes_csv_file_path or git.codes_json_file_path from HEAD into feed-codes.json.
  • Detects rewritten history (force-push, rebase) and rebuilds the feed from scratch, while emitting metrics so operators can alert.
  • Optionally serves the feed tree on /feed/ for development. The shipped status server exposes /, /metrics, /healthz, and pprof.

Modes of operation

  • Local repo. git.root is a filesystem path. The service polls HEAD every git.check_interval. Useful when something else on the same host manages the repo (an admin tool, the command-source overlay, an external sync job).
  • Remote repo. git.root is an https:// or git@host:... URL. The service clones once into an in-memory filesystem, fetches every git.fetch_interval, then HEAD is polled in-process. HTTP Basic or SSH identity authentication are supported.
  • Command-source overlay. A git.command_source.command runs on a timer, produces a CSV and a JSON codes file, and gitfeed commits them to the local repo. Normal commit processing then takes over. Local-root only; CSV codes mapping is rejected (JSON required).

Use a remote repo when the feed is centrally maintained (typically on GitHub or a similar host). Use a local repo with the command-source overlay when the data lives in a database and a script generates the CSV — gitfeed gives you the change history and delta machinery without needing a separate git workflow.

Where it fits in Platform Filter

Gitfeed sits at the producer end of the OX Feed pipeline alongside the zVelo, RPZ, HTTP API, and command-source downloaders. The feed tree it writes is served to downstream clients by a static HTTP server (Caddy or nginx), optionally through an OX Feed Mirror for bandwidth fan-out, and consumed by OX Feed Client instances on the filter nodes.

See Architecture for the commit-to-version mapping and the workflow rules for editing the feed CSV.

Next steps

  • Architecture: operating modes, commit processing, history-rewrite handling, and the workflow rules.
  • Configuration: full YAML reference for a per-instance config file.
  • Operations: systemd template unit, CLI flags, HTTP endpoints, and troubleshooting.
  • Metrics: Prometheus metric reference.