Operations

This page is the operator's reference for a running pdns-pf-orgmanager instance: how to control the service, how to watch it, which metrics to look at, and how to reach for the pdns-pf-orgmanager-stress companion when you need to load-test it.

Service control

The RPM ships a systemd unit named pdns-pf-orgmanager.service, running as the pdns-orgmanager user. Everyday commands:

# Status and recent logs
systemctl status pdns-pf-orgmanager

# Follow the log stream
journalctl -u pdns-pf-orgmanager -f

# Restart after a config change
systemctl reload-or-restart pdns-pf-orgmanager

The service does not watch its config file for changes; restart it after editing /etc/pdns-pf/orgmanager.yaml.

Config location

The shipped systemd unit points the service at /etc/pdns-pf/orgmanager.yaml. A heavily-commented template ships alongside it as /etc/pdns-pf/orgmanager.example.yaml.

See Configuration for the full reference.

Command-line flags

When running the service in the foreground for troubleshooting — under strace, by hand from outside the package's install path, or to verify a config change — these are all the flags it accepts:

Flag	Default	Purpose
--config, -c	/etc/pdns-pf/orgmanager.yaml	Path to the YAML config file.
--debug	false	Force log.level=debug for this process, overriding log.level from the config file. Verbose — use sparingly on production.

Observability

The service exposes its live-inspection surfaces on the single listener bound to http.address:

• Status page at /. A self-service HTML diagnostic landing page covering version, readiness, runtime memory, database and NATS connection health, the deploy cache summary, the most recent version scan, and a bounded "largest cached organisations" table. The exact layout may change between releases — check the rendered page rather than scripting against it.
• /health returns plain OK with 200. Suitable for liveness probes; it does not check downstream connectivity.
• /metrics serves the Prometheus scrape endpoint described in Metrics reference below.

Logs go to standard error. Under the default systemd unit they are captured by the journal under the pdns-pf-orgmanager syslog identifier, so both journalctl -u pdns-pf-orgmanager and journalctl -t pdns-pf-orgmanager work.

Metrics reference

Every metric is exposed at /metrics on the http.address listener. Names are prefixed with orgmanager_ (included below for grep-ability).

Build and readiness

Metric	Type	Meaning
orgmanager_version	gauge	Set to 1 with a version label carrying the running binary's version string.
orgmanager_ready_status	gauge	1 once the service has finished startup and the deploy worker pool is running, 0 otherwise.

HTTP

Metric	Type	Meaning
orgmanager_http_requests_total	counter vec	REST API requests, labelled by method, path, and status (the textual HTTP status). The path is normalised — organisation IDs are replaced with :org to keep label cardinality bounded.
orgmanager_http_request_duration_seconds	histogram vec	REST API request duration, labelled by method and path (same normalisation as above).

Deploy worker pool

Metric	Type	Meaning
orgmanager_deploy_queue_size	gauge	Number of organisations currently waiting in the pending deploy queue. A persistently non-zero queue means writes arrive faster than deploy.workers can drain.
orgmanager_deploy_active	gauge vec	Number of deploys currently executing, labelled by kind and source. Bounded by deploy.workers.
orgmanager_deploy_total	counter vec	Completed deploys, labelled by kind (normal, full, delete) and source (api_change, version_check, api_redeploy, periodic_full).
orgmanager_deploy_errors_total	counter vec	Final deploy failures after the retry budget was exhausted, labelled by kind and source. Should be very rare.
orgmanager_deploy_retries_total	counter vec	Individual deploy attempts that failed and were retried, labelled by kind and source. Climbs while a downstream is briefly unavailable; non-zero is normal during IP Mapper restarts.
orgmanager_deploy_failures_by_error_total	counter vec	Deploy failures (retries + final failures combined) broken down by kind, source, and error message. Useful for spotting whether retries are hitting the same root cause.
orgmanager_deploy_debounced_total	counter vec	Deploy requests that were coalesced into an already-pending or already-running deploy, labelled by the incoming kind and source. High values mean writes are bursty for the same organisation; the service is doing the right thing.
orgmanager_deploy_duration_seconds	histogram vec	Wall-clock execution time of one deploy, including its retries, labelled by kind and source.
orgmanager_deploy_delay_seconds	histogram vec	Time from deploy enqueue to the worker actually starting on it, labelled by kind and source. Quiet systems sit close to zero; under load this becomes the queueing delay.

NATS publish to the IP Mapper

Metric	Type	Meaning
orgmanager_nats_publish_total	counter	NATS set-session-batch publishes attempted.
orgmanager_nats_publish_errors_total	counter	Publishes that failed.
orgmanager_nats_sessions_published_total	counter vec	Session actions published, labelled by action (start, update, stop).
orgmanager_nats_request_duration_seconds	histogram	Latency of a single NATS request to the IP Mapper.

NATS cleanup

Metric	Type	Meaning
orgmanager_nats_cleanup_total	counter	clean-sessions requests sent. Driven by full redeploys and prefix removals.
orgmanager_nats_cleanup_errors_total	counter	Cleanup requests that failed.

Companion tool: pdns-pf-orgmanager-stress

The RPM also ships pdns-pf-orgmanager-stress, a load generator that bootstraps organisations and overrides through a running orgmanager's REST API and then keeps a configurable steady-state stream of updates flowing while monitoring the target's /metrics endpoint. It is the way to reproduce real-world write load when sizing deploy.workers or chasing a regression.

pdns-pf-orgmanager-stress --url <api-root> [flags]

Flag	Default	Purpose
--url	(required)	Orgmanager API root URL, e.g. http://127.0.0.1:8080.
--orgs	10	Total number of organisations to create.
--overrides-per-org	1000	Total number of overrides per organisation.
--nested-overrides	100	Nested override chains per organisation, exercising the most-specific-wins path.
--org-updates-per-second	1	Organisation update requests per second (steady-state churn).
--override-updates-per-second	200	Override item updates per second (steady-state churn).
--override-updates-per-batch	50	Override updates per single batch request.
--ipv4-allocs	255.0.0.0/8	IPv4 allocation pool used to carve per-organisation prefixes.
--ipv6-allocs	2001:db8::/32	IPv6 allocation pool used to carve per-organisation prefixes.
--ipv6	false	Use IPv6 allocations instead of IPv4.
--metrics-interval	5s	How often to scrape the target /metrics endpoint.
--http-timeout	10s	Timeout for outbound HTTP requests.
--concurrency	8	Maximum number of in-flight HTTP requests.
--duration	0	How long to run before stopping (0 runs until interrupted).
--org-name-prefix	stress-org-	Prefix used for generated organisation IDs.
--clean	false	Delete the generated stress organisations on exit.
--trace	false	Attach generated trace IDs to override mutations and forward them to the IP Mapper.
--trace-sample	0	Trace only this fraction of overrides (0..1); ignored when --trace is set.

Example — bring up 100 organisations of 1 000 overrides each, run a steady-state churn for 10 minutes, then clean up:

pdns-pf-orgmanager-stress \
  --url http://127.0.0.1:8080 \
  --orgs 100 \
  --overrides-per-org 1000 \
  --override-updates-per-second 200 \
  --duration 10m \
  --clean

While it runs, watch orgmanager_deploy_queue_size, orgmanager_deploy_delay_seconds, and the deploy-cache card on the status page on the target instance.

Troubleshooting

The status page and the metrics surface are the right starting point for almost everything — the status page exposes the version-scan error string and the deploy queue depth directly, and the metric tables above name what each counter actually measures. Two specific items are worth calling out because they look like service bugs but are not:

Service refuses to start with a sqlite: DSN

The production RPM is built without CGO and therefore does not link the SQLite driver. The startup log line names this — switch database.dsn to a PostgreSQL DSN (see Configuration → Connecting to PostgreSQL). SQLite is intended for development and tests only.

Override stuck in outside-prefixes after a prefix add

The _status of an override is recomputed when the organisation redeploys, not synchronously when a prefix is added. After a covering prefix is added, the next deploy — triggered by the prefix mutation itself, or by the next deploy.version_check_interval tick — flips the override to active. If a stuck status persists beyond that, the version-scan error string on the status page is the authoritative signal.