Configuration¶

The HTTP API downloader reads its configuration from a YAML file. The shipped systemd template loads /etc/pdns-pf/oxfeed-httpapi-downloader/<INSTANCE>.yml, where <INSTANCE> is the systemd instance suffix. A commented template is installed at /etc/pdns-pf/oxfeed-httpapi-downloader/example.yml.

The -expand-env CLI flag causes the service to expand $VAR and ${VAR} references from its environment before parsing the YAML. Use this to inject the upstream password from a systemd drop-in instead of storing it in the file.

Top-level structure¶

Parameter	Type	Default	Description
`run_once`	`boolean`	`false`	Exit after a single successful update cycle. The shipped systemd unit always launches with `-config`, which forces this to `false`; set it back to `true` here if you want a one-shot run from systemd.
`feed`	`Feed`	—	Where the feed is written, how it is fetched from upstream, and how old files expire. Required.
`http`	`HTTP`	`{}`	Observability HTTP server (status page, `/metrics`, `/healthz`, optional `/feed/`).
`log`	`Log`	—	Log output format and verbosity.

Deprecation policy¶

No fields have been deprecated or removed as of the current release.

Full example config¶

The two YAML files below are shipped with the source tree and the RPM. The first is the generic example installed at /etc/pdns-pf/oxfeed-httpapi-downloader/example.yml. The second is the profile used for the VirusTotal-derived feed served by pdns-pf-oxfeed-vtfeed. The per-section reference below is authoritative; these blocks are for orientation.

Click to expand the shipped generic example

# Example config file for the pdns-pf-oxfeed-httpapi-downloader service
#
# To add a systemd service instance, you need to take these steps (replace SOMENAME):
#
#   cp /etc/pdns-pf/oxfeed-httpapi-downloader/example.yml \
#      /etc/pdns-pf/oxfeed-httpapi-downloader/SOMENAME.yml
#   ln -s /lib/systemd/system/pdns-pf-oxfeed-httpapi-downloader@.service \
#         /etc/systemd/system/pdns-pf-oxfeed-httpapi-downloader@SOMENAME.service
#

# By default we run as a service when a config file is used.
# Set this to true to only do a single run.
#run_once: false

# Configures where and how the feed directory with feed.json is written
feed:
    # Path where the feed is saved to.
    # This is equivalent to `simpleblob://fs?root_path=/tmp/feed`
    # or `file:///tmp/feed`.
    root: /tmp/feed

    # Optional. Allows trying again storage operations.
    retry:
        #timeout: 5m
        #interval: 20s
        #max_jitter: 1s
        #backoff: 2s

    # Interval we want snaps to be written, 0s is on any incoming update
    #snap_interval: 6h

    # Make every code have its own namespace.
    # WARNING: Changing this later will trigger a full feed UUID reset!
    #namespace_per_code: false

    # Enable internal memdb stores pruning.
    #prune_enabled: false
    #prune_interval: 24h

    http_upstream:
        #url: base url to fetch from
        #codes_url_override: optional full url for codes
        #username: https basic auth user if needed
        #password: http basic auth password if needed
        #proxy:
        #  url: "http://proxy.example:8080"
        #tls: TLS client config if needed
        #max_delta_size: 10000
        #interval: 10s
        #codes_interval: 30s
        #error_interval: 1s
        #headers: { key: value }

    #expiry:
    #    snapshots:
    #        must_keep_count: 3
    #        must_keep_interval: 30m
    #        max_keep_count: 3
    #        max_keep_interval: 0s
    #    deltas:
    #        must_keep_count: 5
    #        must_keep_interval: 6h
    #        max_keep_count: 200
    #        max_keep_interval: 168h


# The HTTP server exposes Prometheus /metrics and a status page
http:
    #address: ":9605"
    #serve_feed: false

log:
    level: info       # options: trace, debug, info, warning, error, fatal
    format: human     # options: human, logfmt, json
    timestamp: short  # options: short, full, disable

Click to expand the shipped vtfeed profile

# Example config file for the pdns-pf-oxfeed-httpapi-downloader service with vtfeed

feed:
    root: /tmp/vtfeed
    retry: {}

    # Interval we want snaps to be written, 0s is on any incoming update
    snap_interval: 6h

    # The vtfeed can get large.
    prune_enabled: true
    prune_interval: 24h

    http_upstream:
        url: http://localhost:4716/api

    #expiry: ...

http:
    address: ":4717"
    serve_feed: true

log:
    level: info
    format: human
    timestamp: short

`feed`¶

The feed section controls where the resulting OX Feed is written, how it is fetched from upstream, and how old files are pruned.

Storage root¶

feed.root accepts either a plain path string or a structured backend definition. A bare path uses the local filesystem backend; the structured form is equivalent to writing the same path under type: fs, options.root_path: <path>. Backends are provided by simpleblob; fs, memory, and s3 are supported.

feed:
    root: /tmp/feed              # filesystem at /tmp/feed
    # or
    root:
        type: fs
        options:
            root_path: /tmp/feed
    # or
    root:
        type: s3
        options:
            access_key: "${S3_ACCESS_KEY}"
            secret_key: "${S3_SECRET_KEY}"
            region: us-east-1
            bucket: my-feed-bucket

Namespace per code¶

Setting namespace_per_code: true makes every category code live in its own internal namespace.

Warning

Changing namespace_per_code on an existing feed regenerates the feed UUID and forces every downstream client and mirror to reload from a full snapshot. Pick the value at deployment time and leave it alone afterwards.

Pruning¶

prune_enabled: true makes the downloader compact the in-memory string store while building deltas. This bounds resident memory for large feeds at the cost of extra CPU. The example config flips it on for the vtfeed profile and off for everything else. prune_interval controls the periodic prune timer; a value of 0 means "never run the periodic prune" (event-driven pruning still happens).

Fields¶

Parameter	Type	Default	Description
`root`	`string` or `simpleblob.Config`	—	Where the feed directory is written. See Storage root. Required.
`retry`	`retry.Config`	`{}`	Retry rules for storage operations.
`http_upstream`	`HTTPUpstream`	—	Upstream HTTP API to fetch from. `url` is required.
`expiry`	`ExpiryConfig`	(see below)	Rules for retaining snapshots and deltas in the feed directory.
`snap_interval`	`go: Duration`	`1h`	Minimum interval between two snapshot writes. `0s` means "write a snapshot on every upstream update".
`prune_interval`	`go: Duration`	`24h`	Time between two full memdb prunings. `0s` disables the periodic prune.
`prune_enabled`	`boolean`	`false`	Enable internal memdb pruning. See Pruning.
`namespace_per_code`	`boolean`	`false`	Place every category code in its own internal namespace. See Namespace per code.

`feed.retry`¶

Parameter	Type	Default	Description
`timeout`	`go: Duration`	—	Duration over which consecutive storage errors are tolerated before the operation returns an error. The whole `retry` block is optional; if omitted, storage operations are not retried.
`interval`	`go: Duration`	—	Time to wait between retries after a failure. Minimum 10 ms.
`max_jitter`	`go: Duration`	—	Random extra wait added to `interval` to spread retries. Minimum 2 ms when set.
`backoff`	`go: Duration`	—	Static time added to `interval` for each consecutive failure.

`feed.http_upstream`¶

The upstream HTTP API the downloader polls.

Upstream URL and overrides¶

url is the base URL under which the downloader looks up /full, /incremental, and /codes. Set codes_url_override if the producer serves the code map at an unrelated path or host. The downloader joins url and the endpoint name using URL-path semantics, so trailing slashes are forgiven.

Timeouts¶

The default short timeout (default_get_timeout, minimum 10 s) applies to /incremental and /codes. The long timeout (long_get_timeout, minimum 1 m) applies to /full, where the body can be hundreds of megabytes. Both cover connect, header read, and body read; tune the long timeout to the largest snapshot you expect plus working headroom.

Polling cadence¶

interval is how often the downloader requests an incremental update; minimum 100 ms. codes_interval is how often the codes endpoint is polled in the background; minimum 1 s. error_interval is the wait after a failed fetch before retrying; minimum 1 s.

Credentials and headers¶

username / password enable HTTP Basic Auth. The password is masked in logs but stored verbatim in the YAML file — use -expand-env with a systemd drop-in to inject it from the environment instead:

http_upstream:
    url: https://feed.internal/api
    username: ingester
    password: "${UPSTREAM_PASSWORD}"

headers is a flat string-to-string map merged into every outgoing request. Use it for API tokens or any custom header the producer expects.

Fields¶

Parameter	Type	Default	Description
`url`	`string`	—	Base URL of the upstream HTTP API. Required.
`codes_url_override`	`string`	—	Full URL for the codes endpoint, when it is not served under `url/codes`.
`username`	`string`	—	HTTP Basic Auth user.
`password`	`string`	—	HTTP Basic Auth password. Masked in logs. Prefer `${VAR}` expansion.
`proxy`	`ProxyConfig`	—	Optional HTTP proxy. Falls back to `HTTP_PROXY` / `HTTPS_PROXY` environment variables when unset.
`tls`	`tlsconfig.Config`	—	TLS settings for HTTPS upstreams. Optional.
`default_get_timeout`	`go: Duration`	`1m`	Timeout for the codes and incremental endpoints. Minimum 10 s. See Timeouts.
`long_get_timeout`	`go: Duration`	`40m`	Timeout for the full-snapshot endpoint. Minimum 1 m. See Timeouts.
`max_delta_size`	`int`	`10000`	Value passed to upstream as `size=` on `/incremental` requests. Caps the number of lines per delta.
`interval`	`go: Duration`	`10s`	Wait between two delta fetches. Minimum 100 ms.
`codes_interval`	`go: Duration`	`30s`	Wait between two codes fetches. Minimum 1 s.
`error_interval`	`go: Duration`	`1s`	Wait after an error before retrying. Minimum 1 s.
`headers`	`map[string]string`	`{}`	Custom HTTP headers included in every upstream request.

`feed.http_upstream.proxy`¶

Parameter	Type	Default	Description
`url`	`string`	—	Proxy URL, for example `http://user:password@proxy.example.com:3128`. Falls back to `HTTP_PROXY` / `HTTPS_PROXY` environment variables when not set.

`feed.http_upstream.tls`¶

TLS configuration for the upstream connection. Provided by tlsconfig.Config.

Parameter	Type	Default	Description
`ca_file`	`string`	—	Path to a PEM file containing additional CA certificates to trust.
`cert_file`	`string`	—	Path to the client certificate PEM file. Required when the upstream requires mutual TLS.
`key_file`	`string`	—	Path to the client private key PEM file. Required with `cert_file`.
`add_system_ca_pool`	`boolean`	`false`	Include the system CA pool alongside any CAs from `ca_file`.
`insecure_skip_verify`	`boolean`	`false`	Disable server certificate verification.

Warning

Do not set insecure_skip_verify: true in production. It silently accepts any certificate the upstream presents, including attacker-controlled ones in a MITM position.

`feed.expiry`¶

Retention rules applied to snapshot and delta files in the local feed directory. Each side (snapshots, deltas) takes the same four-field expiry.Config struct. Defaults are kept aggressive on snapshots (large files) and generous on deltas (so clients catching up after downtime can replay history).

The expiry algorithm applies in two phases: first the must_keep_* constraints reserve the recent updates; only the remainder is then eligible for removal under max_keep_*. This means a small max_keep_count cannot delete history that the must_keep_* rules have pinned.

Parameter	Type	Default	Description
`snapshots`	`expiry.Config`	(see below)	Retention rules for snapshot files.
`deltas`	`expiry.Config`	(see below)	Retention rules for delta files.

Each expiry.Config:

Parameter	Type	snapshots default	deltas default	Description
`must_keep_count`	`int`	`3`	`5`	Always keep at least this many of the most recent entries.
`must_keep_interval`	`go: Duration`	`30m`	`6h`	Always keep entries created within this window.
`max_keep_count`	`int`	`3`	`200`	Remove the oldest entries once the total exceeds this count, subject to the must-keep rules. `0` means no cap.
`max_keep_interval`	`go: Duration`	`0s`	`168h`	Remove entries older than this, subject to the must-keep rules. `0s` means no time-based removal.

`http`¶

The observability HTTP server. It exposes the status page, Prometheus metrics, healthz, and Go pprof. Omit address (or the entire http section) to disable the server.

Parameter	Type	Default	Description
`address`	`string`	—	Address to listen on, for example `:9605` or `127.0.0.1:9605`. Required to enable the HTTP server.
`serve_feed`	`boolean`	`false`	Also serve the feed directory at `/feed/`. Convenience for development; production deployments should front the feed directory with Caddy or nginx.

Warning

serve_feed: true exposes the entire feed directory on the observability port without any access control. Keep it off in production and put a proper HTTP server in front of the feed directory instead.

`log`¶

Controls log output format and verbosity. Same options as every other oxfeed service.

Parameter	Type	Default	Description
`level`	`string`	`info`	Log level. One of: `trace`, `debug`, `info`, `warning`, `error`, `fatal`.
`format`	`string`	`human`	Log format. One of: `human` (coloured, for terminals), `logfmt`, `json`.
`timestamp`	`string`	`short`	Timestamp format. One of: `short`, `full`, `disable`.

Configuration¶

Top-level structure¶

Deprecation policy¶

Full example config¶

feed¶

Storage root¶

Namespace per code¶

Pruning¶

Fields¶

feed.retry¶

feed.http_upstream¶

Upstream URL and overrides¶

Timeouts¶

Polling cadence¶

Credentials and headers¶

Fields¶

feed.http_upstream.proxy¶

feed.http_upstream.tls¶

feed.expiry¶

http¶

log¶

`feed`¶

`feed.retry`¶

`feed.http_upstream`¶

`feed.http_upstream.proxy`¶

`feed.http_upstream.tls`¶

`feed.expiry`¶

`http`¶

`log`¶