Metrics

The service exposes Prometheus metrics on /metrics at http.address. Two metric families cooperate to produce the surface operators care about:

• oxfeed_feed_* — service-level state: which version is live, how big the feed is, whether in-memory pruning is active.
• oxfeed_source_command_* — pipeline metrics from the command source layer: how often the command runs, what exit codes it returns, how long it takes, when feed and codes content changes.

The default Go runtime and process collectors from the standard Prometheus client are also exposed (go_*, process_*). They are not listed here.

Label cardinality watch-out

Most metrics are unlabelled. Two carry labels:

• oxfeed_source_command_exec_exitcode_total — labelled by exit code. Cardinality is bounded by how many distinct exit codes the source command actually returns; in practice you see 0, 11, and at worst a handful of error codes.
• oxfeed_source_command_exec_error_message_total — labelled by the raw Go error string. This is the one to watch. Transient errors with embedded addresses, timestamps, or process IDs can push the label set unbounded over time. If you ingest these metrics long-term, drop the error label at scrape time or alert on its cardinality.

Is the command running?

These metrics answer "is the source command being executed, and does it succeed?".

Metric	Type	Meaning
oxfeed_source_command_exec_total	Counter	Total invocations of the command since startup.
oxfeed_source_command_exec_success_last_time_seconds	Gauge	Unix timestamp of the last successful run (including exit 11).
oxfeed_source_command_exec_error_total	Counter	Runs that failed: any exit code other than 0 or 11, plus output-parsing failures and timeouts.
oxfeed_source_command_exec_exitcode_total	Counter	Per-exit-code counter of non-zero exits.
oxfeed_source_command_exec_error_message_total	Counter	Per-error-message counter. High cardinality — see the watch-out above.

Exit code 11 is the "not modified" signal documented under Architecture → Exit codes; it dominates the exit-code counter when the command supports --wait and the source data is quiet. Alert on the error rate, not on the raw exit code counter.

Alert on staleness with the success timestamp:

# No successful command run in the last 30 minutes.
time() - oxfeed_source_command_exec_success_last_time_seconds > 1800

Are updates landing?

Metric	Type	Meaning
oxfeed_source_command_feed_change_total	Counter	Feed CSV content changed (hash differs from the previous accepted run).
oxfeed_source_command_feed_change_last_time_seconds	Gauge	Unix timestamp of the last feed content change.
oxfeed_source_command_codes_change_total	Counter	feed-codes.json content changed.
oxfeed_source_command_codes_change_last_time_seconds	Gauge	Unix timestamp of the last codes change.

These count content changes, not command invocations. A command that runs every five seconds against unchanged data will move oxfeed_source_command_exec_success_last_time_seconds on every run but leave the change counters flat.

A long-quiet feed is normal for low-churn sources (admin overrides rarely change). To detect a truly stuck pipeline, combine this with the exec success metric: the command running successfully but the content never changing is the expected steady state.

What is the feed state?

Metric	Type	Meaning
oxfeed_feed_current_version	Gauge	Latest version written (snapshot or delta). Monotonically increasing per instance.
oxfeed_feed_current_version_entries	Gauge	Number of domain entries in the latest snapshot written. NaN before the first snapshot.
oxfeed_feed_size_bytes	Gauge	Total bytes occupied by snapshots, deltas, and feed-codes.json under feed.root.

The size gauge is the easiest signal for a runaway feed or a stuck expiry policy. If it grows monotonically over weeks the feed.expiry rules are too permissive, or if feed.prune_enabled is off the in-memory state may be growing alongside it.

How fast is the command?

Metric	Type	Meaning
oxfeed_source_command_exec_duration_seconds	Histogram	Wall-clock duration of each command run. Includes time spent inside --wait if the command supports it.
oxfeed_source_command_update_channel_blocked_seconds	Histogram	Time the source spent blocked publishing an update to the feed writer.

Long tails on exec_duration_seconds are usually the command waiting for source data to change (with --wait), not a slow source. To distinguish, also look at feed_change_total: high duration + low change rate is normal idle behaviour, high duration + high change rate means the command itself is slow.

update_channel_blocked_seconds rising signals that the feed writer is the bottleneck — usually storage backend slowness or an unusually large delta to compress and ship.

PromQL for command duration p95:

histogram_quantile(0.95,
  rate(oxfeed_source_command_exec_duration_seconds_bucket[5m]))

Is pruning recovering memory?

These metrics are only meaningful when feed.prune_enabled is true.

Metric	Type	Meaning
oxfeed_feed_prune_call_total	Counter	Times the in-memory feed state was compacted.
oxfeed_feed_prune_saved_bytes_total	Counter	Bytes freed by those compactions, cumulative.

A prune_call_total rate close to zero on a long-running instance with pruning enabled means the mutation-based trigger never fires (the feed is small or stable) and the time-based trigger is doing the work. prune_saved_bytes_total not increasing means there is no stringstore fragmentation to recover, and pruning can be turned back off.