Skip to content

Observability

The IP-mapping pipeline is asynchronous end to end: a session update flows from its source, through the IP Mapper and Redis, into each Settings Syncer, and finally into LMDB where the filter reads it. Nothing blocks waiting for the far end, so the thing to watch is that each stage keeps up and that changes actually arrive.

Sync characteristics

  • Latency. Propagation is fast but not instant, and it has two parts. First, replication: an update travels from the IP Mapper through Redis and the Settings Syncer into LMDB. This is typically tens of milliseconds, but it can grow to tens of seconds during snapshot resyncs, large provisioning operations, or network instability. Second, the read view: the filter reads LMDB through a view it refreshes once a second, which adds up to a further second on top before a change already in LMDB becomes visible to queries.
  • Bulk load then subscribe. Each Settings Syncer first subscribes, then does a full bulk load, then applies live updates. A freshly started or reconnected syncer is therefore briefly loading a complete snapshot before it is current.
  • Failure modes. If a Settings Syncer loses its Redis connection it closes the subscription, waits for a configured delay, then repeats the subscribe-then-bulk-load cycle. A dropped connection heals by reloading, not by replaying missed messages, so a syncer always converges on the full current state rather than risking a gap.

What to monitor at each stage

Follow the data as it moves and watch the boundary metric at each hop:

Stage Watch for Where
Source → IP Mapper Sessions arriving; call errors and latency IP Mapper metrics
IP Mapper state Active session count; stolen-allocation and cleanup behavior IP Mapper metrics
IP Mapper → Redis The IP Mapper staying connected and writing IP Mapper journal / Redis server metrics
Redis → Settings Syncer Bulk load completing; subscription latency Settings Syncer metrics
Settings Syncer → LMDB Entry counts written; sync errors Settings Syncer metrics
LMDB → filter The mapping a query actually resolves to Inspection tools

IP Mapper metrics

The IP Mapper serves Prometheus metrics under the ipmapper_ prefix. The ones most relevant to IP mapping:

  • ipmapper_ready_status — 0 while loading state from Redis after a restart, 1 when ready to serve. A value stuck at 0 means the instance has not finished loading.
  • ipmapper_state_active_sessions_total — how many sessions are currently held.
  • ipmapper_state_stolen_allocations_total — allocations claimed from an existing session. Some churn here is normal during IP handover; a steadily rising rate can indicate duplicate session IDs.
  • ipmapper_session_updates_total{action} — session events by action (start, update, stop), the basic "is it receiving traffic" signal.
  • ipmapper_nats_calls_total{command} and the matching ..._error_total — call volume and error rate on the primary API.

See the full IP Mapper metrics reference for the rest, including the HA and Redis-connection metrics.

Settings Syncer metrics

The Settings Syncer serves Prometheus metrics under the settings_syncer_ prefix. The ones that tell you the read side is healthy:

  • settings_syncer_bulk_count{status="success"} — increments on each successful bulk load; settings_syncer_bulk_in_progress is 1 while one runs.
  • settings_syncer_bulk_last_count — entries written in the most recent bulk load, broken down by server, db, and region.
  • settings_syncer_incr_latency_msec — subscription latency, measured against a timestamp that pdns-pf-redis-timestamper publishes into Redis every second. Values consistently above a few hundred milliseconds mean the subscriber is falling behind.
  • settings_syncer_incr_latency_excessive_count — counts updates that arrived more than three seconds late.

See the Settings Syncer metrics reference for the complete list.

Inspection tools

Metrics tell you the pipeline is flowing; these tools tell you what a specific lookup will actually return, so you can confirm a mapping end to end:

  • pdns-pf-redis-query inspects the mapping data as it sits in Redis — the data plane between the IP Mapper and the Settings Syncers. Use it to check what the central source of truth holds. It also has subcommands that modify sessions and mappings, so use it with care on production data.
  • pdns-pf-lmdb-query inspects the local LMDB on a user-plane node — the exact data the filter reads. It can list the IPv4 and IPv6 mappings, opaque-ID and user entries, and look a specific IP, opaque ID, or user up. Use it on the box where a lookup is going wrong to see what that node actually has. Its latency and latencylog subcommands read a rolling 24-hour, per-second log of the measured sync latency straight from the node's LMDB, an easy on-box check of how fresh the data has been.

Comparing the two answers a common question directly: if pdns-pf-redis-query shows a mapping but pdns-pf-lmdb-query on a node does not, the problem is in that node's sync, not in the source data.

Simulator

The Simulator HTTP Server runs the same filtering logic against the same local LMDB that the production filters use, and exposes it through an HTTP API, which the Admin Portal's UI also uses. Point it at a subscriber IP to see which subscriber, profile, and settings a real query would resolve to, without sending live DNS traffic. It is the most direct way to verify that an IP mapping produces the filtering outcome you expect.