Configuration¶

The mirror reads its configuration from a YAML file. The default path operators see on a deployed box is /etc/pdns-pf/oxfeed-mirror.yml; a commented example with every option is shipped alongside it as /etc/pdns-pf/oxfeed-mirror.example.yml.

Environment variables in YAML values are only expanded when the service is started with -expand-env. With expansion enabled you can write password: "${UPSTREAM_PASSWORD}" and supply the variable through the unit's environment file.

Top-level structure¶

Parameter	Type	Default	Description
`basedir`	`string`	—	Root directory under which each upstream is mirrored into its own subdir. Required.
`default_poll_interval`	`go: Duration`	`1m`	Fallback poll interval for upstreams that omit `poll_interval`.
`upstreams`	`[]Upstream`	—	One entry per mirrored feed. At least one is required.
`http`	`HTTP`	—	Status and metrics server.
`log`	`Log`	see section	Log format and level.
`run_once`	`boolean`	`false`	Exit after a single successful run. The `-once` CLI flag sets this too.

Deprecation policy¶

No fields have been deprecated or removed as of the current release.

Full example config¶

Click to expand the shipped example config

# Example config file for oxfeed-mirror

run_once : false
basedir: tmp/base

# Default poll intervals used when no per-feed intervals are configured.
#default_poll_interval: 5s

upstreams:
- url: http://localhost:2016/zvelo-feed/ # Base url (without feed.json)
  subdir: zvelo-feed
  # Optional HTTP Basic Auth
  #username: ""
  #password: ""
- url: http://localhost:2016/zvelo-feed2/ # Base url (without feed.json)
  subdir: zvelo-feed2
  # Optional HTTP Basic Auth
  #username: ""
  #password: ""
  # Optional Proxy
  #proxy:
  #  url: http://user:password@localhost:9040
  # You can override poll intervals per feed.
  #poll_interval: 5s

  # Conservative protection against a stuck download.
  #fetch_timeout: 4h

  # Configure snapshot and delta expiry rules to limit the updates we mirror.
  # Note that when the remote removes an update, we always follow it.
  #expiry:
  #    snapshots:
  #        must_keep_count: 3
  #        must_keep_interval: 30m
  #        max_keep_count: 3
  #        max_keep_interval: 0s
  #    deltas:
  #        must_keep_count: 5
  #        must_keep_interval: 6h
  #        max_keep_count: 200
  #        max_keep_interval: 168h

http:
  address: :8912

log:
  level: info       # options: trace, debug, info, warning, error, fatal
  format: human     # options: human, logfmt, json
  timestamp: short  # options: short, full, disable

`basedir`¶

Absolute path to the directory in which each upstream is mirrored. The mirror creates <basedir>/<subdir> per upstream and writes feed.json, feed-codes.json, and the snapshot/delta files into it.

The service creates the directory on startup if it does not exist. The process user (pdns-oxfeed) must own or be able to write to it. The free_disk healthz check watches this directory.

-basedir on the command line overrides this field.

`default_poll_interval`¶

Default polling interval used when an upstream does not set poll_interval. Minimum is 100 ms. Polling is cheap when the upstream supports ETags — no data is transferred when feed.json has not changed — so short intervals are reasonable.

`upstreams`¶

A list of feeds to mirror. Each entry produces an independent worker.

Note

url and subdir must both be unique across the list. The service refuses to start if either is duplicated.

`upstreams[]` fields¶

Parameter	Type	Default	Description
`url`	`string`	—	Base URL of the upstream feed, ending in a slash (for example `https://feeds.example/zvelo-feed/`). Required.
`subdir`	`string`	—	Subdirectory under `basedir` for this upstream. Required, unique.
`username`	`string`	—	Optional HTTP Basic Auth username.
`password`	`string`	—	Optional HTTP Basic Auth password. Masked in logs and dumped config.
`poll_interval`	`go: Duration`	`default_poll_interval`	How often to check the upstream `feed.json`. Minimum 100 ms.
`fetch_timeout`	`go: Duration`	`4h`	Per-HTTP-request timeout. See Fetch timeout sizing.
`tls`	`tlsconfig.Config`	`{}`	TLS settings for HTTPS upstreams.
`expiry`	`ExpiryConfig`	`{}`	Optional extra retention rules.
`proxy`	`proxyconfig.ProxyConfig`	—	Optional outbound HTTP/HTTPS/SOCKS5 proxy.

Fetch timeout sizing¶

fetch_timeout is a backstop against a stuck connection that keeps the TCP socket open but stops sending data. Go's TCP keepalive normally catches this faster, so the timeout almost never fires in practice. It needs to be high enough to allow a slow client to finish a large snapshot.

At the default of four hours a 1.4 GB snapshot tolerates a sustained ~100 kB/s (~800 kbit/s) average. When many clients pull the same snapshot simultaneously the per-client average drops; raise this value if you see fetches abort under load.

`upstreams[].expiry`¶

Expiry rules narrow what the mirror keeps locally. The mirror always follows upstream removals; expiry rules cannot keep files longer than the upstream does. See Architecture → expiry rules for semantics.

Each of snapshots and deltas accepts the same four fields:

Parameter	Type	Default	Description
`must_keep_count`	`int`	`0`	Never drop the last N entries, regardless of `max_keep_*`.
`must_keep_interval`	`go: Duration`	`0`	Never drop entries newer than this age.
`max_keep_count`	`int`	`0`	Drop oldest entries beyond N newest, subject to must-keep rules. `0` means no count limit.
`max_keep_interval`	`go: Duration`	`0`	Drop entries older than this age, subject to must-keep rules. `0` means no age limit.

Example: keep the last three snapshots and the last week of deltas, but cap deltas at 200 to bound disk usage.

expiry:
  snapshots:
    must_keep_count: 3
    must_keep_interval: 30m
    max_keep_count: 3
  deltas:
    must_keep_count: 5
    must_keep_interval: 6h
    max_keep_count: 200
    max_keep_interval: 168h

`upstreams[].tls`¶

Standard PowerDNS Go TLS config (see go-tlsconfig). All fields are optional; omit the section entirely for plain HTTP upstreams or default-trust HTTPS.

`upstreams[].proxy`¶

Parameter	Type	Default	Description
`url`	`string`	—	Proxy URL. Accepts `http://`, `https://`, `socks5://`, or `socks5h://`. May embed `user:password@`. Masked in logs and dumped config.

If proxy.url is empty the standard HTTP_PROXY, HTTPS_PROXY, and NO_PROXY environment variables apply as usual.

`http`¶

The HTTP server exposes the status page and Prometheus metrics on the node itself. It does not serve the mirrored feed tree to filter clients — front the feed tree with a static HTTP server such as Caddy or nginx.

Parameter	Type	Default	Description
`address`	`string`	—	Bind address in `host:port` form (for example `:8912`). Omit to disable the status/metrics server.

-http-addr on the command line overrides this field.

`log`¶

Parameter	Type	Default	Description
`level`	`string`	`info`	One of `trace`, `debug`, `info`, `warning`, `error`, `fatal`.
`format`	`string`	`human`	One of `human`, `logfmt`, `json`.
`timestamp`	`string`	`short`	One of `short`, `full`, `disable`.

-log-level, -log-format, -log-timestamp, and -debug on the command line override the matching fields.

Configuration¶

Top-level structure¶

Deprecation policy¶

Full example config¶

basedir¶

default_poll_interval¶

upstreams¶

upstreams[] fields¶

Fetch timeout sizing¶

upstreams[].expiry¶

upstreams[].tls¶

upstreams[].proxy¶

http¶

log¶