Skip to content

Architecture

The Carbonizer is a one-process pipeline. It scrapes Prometheus /metrics endpoints on a fixed interval, parses them, and publishes each successful scrape to an in-memory hub. Several independent consumers subscribe to that hub and turn the scrapes into Carbon, into recordings, or into debug output.

Role in Platform Filter

The producers are any service exposing Prometheus metrics — Platform Filter components, third-party exporters, the Carbonizer's own /metrics. The consumer is whatever Graphite-compatible system receives Carbon line protocol on the other end. The Carbonizer is the only translator in between; once it is running, monitoring changes are made by editing its config file instead of touching every service.

Pipeline

flowchart LR
    P1[Prometheus<br/>endpoints] -- scrape --> S[Scrapers<br/>one per target]
    S -- publish --> H((Hub))
    H -- subscribe --> CC[Carbon clients]
    CC -- TCP --> G[Graphite<br/>destinations]
    H -. subscribe .-> R[Dir recorder]
    H -. subscribe .-> D[debug_address<br/>listen_address<br/>HTTP /tap]

Each scrape target runs its own goroutine that polls on the configured scrape_interval. Initial scrapes are spread over scrape_spread (default 1s) so that all targets do not fire on the same tick. Only successful scrapes are published to the hub — fetch and parse errors are counted in metrics but never reach the consumers.

The hub is a buffered fan-out (50 messages per subscriber). If a subscriber is too slow, messages are dropped for that subscriber only; the other consumers and the scraper itself are not blocked.

Concepts

Templates and label mapping

Graphite does not natively support labels — its identifier is a dotted path. The Carbonizer bridges the two with a template, a string with $label placeholders that is evaluated for every sample. The default template is:

carbonizer.$job.$instance.$__name__.$__labels__
  • $job and $instance come from the scrape_configs entry that produced the sample (not from labels on the metric itself, unless honor_labels is set).
  • $__name__ is the Prometheus metric name (the standard __name__ synthetic label).
  • $__labels__ expands to label1.value1.label2.value2... for every label that is not ignored and is not already named in the template.
  • $=label (with an = prefix) inserts the label without escaping. Use with care — invalid characters can break the dotted path.

A scrape sample is dropped if the template references a label that does not exist on it. Order matters: the labels in $__labels__ are sorted alphabetically so the same metric always produces the same path.

The template is set globally in carbonizer.template and can be overridden per scrape_configs job. ignore_labels removes labels from $__labels__ only; labels named explicitly in the template are always used. See Configuration → scrape_configs for the per-job settings.

Label-value escaping

Carbon paths use . as a separator, so any ., /, =, or ; inside a label value is replaced with _. Single and double quotes are dropped. Other special characters are backslash-escaped or percent-encoded so that arbitrary Prometheus label values can survive the round trip. The exact rules are derived from the Graphite grammar.

The $= prefix bypasses this escaping — only use it when you are sure the source label values are already Graphite-safe.

Honor labels

When honor_labels: true is set on a job, an existing job or instance label on the scraped metric is preserved instead of being overwritten by the scrape config's values. This is meant for trusted services that aggregate metrics from multiple sources (cAdvisor, federated exporters). Leave it off for normal targets.

Recording format

When carbonizer.record.dest_dir is set, every successful scrape is also appended to a rolling recording file. Files are named by their start time and roll over every file_duration; older files are pruned once they exceed keep_duration.

Each scrape is written as a small JSON header line prefixed with #[REC], followed by the raw /metrics body and a blank line. The on-disk file is then bzip2-compressed (gzip is also accepted but offers no real benefit at this granularity). The same frame format is also produced by the HTTP /tap endpoint and the standalone pdns-carbonizer-record tool.

pdns-carbonizer-load consumes recordings in this format and either writes them to a PostgreSQL Prometheus backend or converts them to a Prometheus text metrics file with rewritten job / instance labels and the original scrape timestamps. See Operations → Companion tools.

Debug surfaces

Three independent surfaces exist for inspecting live traffic. They all subscribe to the same hub, so attaching to them does not affect the Carbon output.

  • debug_address (TCP). Dumps every successful scrape: the raw /metrics body followed by the converted Carbon lines. Very verbose.
  • listen_address (TCP). Dumps just the Carbon lines that would be sent to a destination — useful for sanity-checking templates without a real Graphite host.
  • HTTP /tap. Returns the same recording-frame format as the dir recorder. An optional ?duration=30s query parameter limits how long the connection stays open. The Carbonizer keeps streaming until the client disconnects or the duration elapses.

Per-target spot checks are available on the HTTP server at /last/scrape/<job>/<instance> (raw text) and /last/carbon/<job>/<instance> (converted Carbon). These show only the most recent scrape, not a live stream.