Architecture¶

Gitfeed translates a git history into the OX Feed file layout: feed.json referencing snapshot_*.gz, delta_*.gz, and feed-codes.json. The service never serves the feed to filter clients directly — front the feed tree with a static HTTP server (Caddy or nginx) for production use.

Role in Platform Filter¶

flowchart LR
    R[Git repository<br/>local or remote] -- HEAD changes --> S[(Gitfeed)]
    S -- write --> D[(Local feed tree)]
    D -- HTTP --> H[Caddy / nginx]
    H -- HTTP --> M[(OX Feed Mirror)]
    H -- HTTP --> C1[oxfeed-client A]
    M -. HTTP .-> C2[oxfeed-client B]

The mirror is optional and typically only deployed when many filter nodes share the same upstream across a WAN link.

Operating modes¶

Local repo¶

flowchart LR
    G[(Local git repo)] -- check_interval --> S[(Gitfeed)]
    S -- write --> D[(Feed tree)]

git.root is a filesystem path. The service opens the repo with git.PlainOpen and polls HEAD every git.check_interval. Nothing fetches over the network in this mode — something else on the host is expected to commit to the repo (a deploy job, the command-source overlay, an admin tool).

Remote repo¶

flowchart LR
    R[(Remote git server<br/>https or ssh)] -- fetch_interval --> S[(Gitfeed)]
    S -- write --> D[(Feed tree)]

git.root is recognised as a URL when it contains :. On startup the service clones it once into an in-memory go-billy filesystem. Every git.fetch_interval it pulls from the remote; in between, HEAD is polled in-process at git.check_interval. The repo never touches disk.

retry_clone: true keeps retrying the initial clone with a 10-second sleep instead of exiting — useful when the remote is reachable only intermittently at boot.

Command-source overlay¶

flowchart LR
    X[External command<br/>e.g. db dumper] -- stdout JSON --> CS[(Command source)]
    CS -- commit CSV --> G[(Local git repo)]
    G -- HEAD changes --> S[(Gitfeed)]
    S -- write --> D[(Feed tree)]

When git.command_source.command is set, gitfeed additionally runs that command every git.command_source.interval and treats its output as new feed data. The command must follow the command contract — the same one used by the standalone OX Feed Command Source — and gitfeed commits the result to the local repo with a fixed author identity. Normal commit processing then turns the new commit into a feed version.

This mode is restricted to local roots only (the in-memory clone of a remote repo is never pushed anywhere) and requires git.codes_json_file_path (CSV codes mapping is rejected).

Linear log walk¶

Gitfeed only follows first parents. From HEAD it walks back through Commit.Parent(0) chains, ignoring branches that were merged in. A merge commit shows up once; the commits on the merged-in branch do not. This matches a "always merge into master, never rebase" workflow and avoids replaying a feed file from a topic branch that never landed on the mainline.

For each commit on that linear log, in chronological order from the oldest unprocessed commit forward:

Open the file at git.file_path. Missing-file commits are silently skipped.
Look at the same path in the parent commit. If the blob hash is identical, the commit did not change the file and is also skipped.
Hand the current and parent file to the writer, which produces a snapshot and (after the first) a delta.

Renames and moves are not followed: if the file is moved to a new path, history before the move is invisible.

Version encoding¶

Each commit that changes the file gets a monotonically increasing sequential number starting at 1. The feed version field combines that number with the first 32 bits of the git commit hash, in decimal, as <number>00<hash32> (zero-padded). Two consequences:

Versions are deterministic — two runs over the same history produce the same version numbers.
Version numbers are large but always grow, so downstream clients can detect a new version with a simple comparison.

File names are snapshot_NNNNNN_<full-hash>.gz and delta_NNNNNN_NNNNNN_<full-hash>.gz, where the leading number is the sequential number and the second number on a delta is the parent sequential number.

Commit message ends up in `feed.json`¶

For every snapshot and delta, the metadata in feed.json records:

git_commit — the full commit hash.
git_title — the first line of the commit message.
git_number — the sequential number used to detect history rewrites.
entries — entry count in this snapshot or delta.

feed.json is consumed by downstream clients and is therefore customer-visible. Keep commit message titles informative and free of internal references — see Workflow rules below.

Workflow rules¶

Operators editing the watched CSV file should observe:

TAB-separated. Domain and codes are split on a literal tab character, not commas or spaces.
No customer references in commit titles. The first line of every commit message ends up in feed.json, which is distributed to multiple customers. Treat commit titles as a public changelog.
Do not rename or move the file. Renames are not followed. Moving the file is equivalent to deleting the old history.
A single domain may carry multiple codes. Either as comma-separated codes on one line, or by listing the domain on several lines — the resulting feed contains the union.
Most specific match wins. When both a domain and one of its parents appear, the longer match wins on the client side. To keep the parent's codes on the subdomain, repeat them.
Branch merges are fine. Only the merge commit appears in the feed, so the merge-request workflow works as expected.
Avoid rewriting master history. A force-push or rebase that changes commits already published triggers a full feed reset, which forces every downstream client to re-snap. The service handles it gracefully but the network cost is real.

History-rewrite handling¶

Each run, gitfeed compares the current commit chain against the git_head stored in feed.json:

If the last processed HEAD still exists in the new history, processing resumes from the next commit (StartFromLast / incremental).
If the last HEAD is missing — the history was rewritten — the service starts from commit number 1 again, drops every existing snapshot_* / delta_* whose git_number falls in the rewritten range, and increments oxfeed_gitfeed_reset_total.
Snapshots and deltas removed during the reset increment oxfeed_gitfeed_version_remove_total.

Empty repositories¶

A freshly cloned repository with no HEAD is tolerated: the service logs a warning, writes no feed entries, and returns to the wait loop. When a first commit eventually arrives it is processed normally. This is intentional so a brand-new feed can be configured and started before any data exists.

Incremental startup¶

incremental: true skips the linear log walk on startup if the git_head in the existing feed.json is still reachable. This is faster on a large history but does not repair missing snapshot or delta files — if a file was deleted from the feed tree, only a full run (the default) will rewrite it. Use incremental mode for routine restarts; do a full run when investigating a corrupted feed tree.

Authentication¶

Remote repos support two mutually exclusive authentication methods:

HTTP Basic. git.auth_http_username and git.auth_http_password. The password can also be supplied through the GITFEED_HTTP_PASSWORD environment variable.
SSH identity. git.auth_ssh_identity_file (private key path), with optional auth_ssh_identity_passphrase for encrypted keys and auth_ssh_identity_username to override the username derived from the URL. auth_ssh_accept_any_host_key: true disables host key checking — insecure, intended for one-shot debugging only.

Without either configured, the standard system SSH agent and ~/.ssh/ configuration are used. Configuring both HTTP and SSH auth is a startup-time error.

Command-source contract¶

When git.command_source.command is configured the service runs the command through /bin/sh -c every git.command_source.interval. The command must finish in time (killed otherwise) and is expected to emit a single line on stdout of the form:

OXFEED_GENERATOR_RESULT: {"v": 1, "csv_file": "/tmp/feed.csv", "codes_file": "/tmp/feed-codes.json", "version": "v123"}

JSON field	Meaning
`v`	Schema version. Must be `1`.
`csv_file`	Path to the feed CSV the command just wrote. The service removes this file after reading it.
`codes_file`	Path to the matching `feed-codes.json`. The service removes this file after reading it.
`version`	Optional opaque version tag. Passed back to the next invocation in `OXFEED_LAST_VERSION`.

Environment passed to the command:

Variable	When set	Purpose
`OXFEED_LAST_VERSION`	After a successful run that returned a `version`	The version tag the service accepted last. Lets the command short-circuit when nothing changed.
`OXFEED_TMPDIR`	Always (one fresh directory per run)	A scratch directory unique to this invocation, removed by the service on exit.

Exit codes:

Code	Meaning
`0`	Success. The service reads `csv_file` and `codes_file`, commits them, processes the new commit.
`11`	Not modified. No commit is made; the service waits for the next tick.
Anything else	Failure. The error counter increments and the next tick retries.

The committed change is authored as OX Feed - gitfeed command source <gitfeed@invalid> with a message of command source: <path> at <version>. The command's output must be deterministic (no timestamps, fixed sort order); the service hashes the CSV and codes contents and suppresses commits when the hash matches the previous accepted run.

Storage backends¶

The feed tree is written through simpleblob, so feed.root accepts:

A bare filesystem path or file:///path — the default for production deployments.
A simpleblob://fs?root_path=/path URI — equivalent to the bare path form.
An s3://bucket?... URI for object-storage backends.
A memory:// URI — useful in tests, not for production.

The -o CLI flag overrides feed.root at startup.

Resilience¶

systemd restart on failure. A fatal error returns from main, the process exits non-zero, and the unit's Restart=on-failure brings it back.
Fetch errors are logged, not fatal. When the periodic fetch fails (DNS, TCP, auth refresh), the error is counted in oxfeed_gitfeed_fetch_error_total and the service continues. A failed fetch never triggers a feed reset on its own.
/healthz is provided by the embedded go-healthz library; gitfeed itself does not register any explicit checks today, so the endpoint reports process liveness only.