Metrics¶

The Carbonizer exposes Prometheus counters, gauges and two summaries at /metrics on the HTTP server configured in carbonizer.http_address. Point your scraper there (or scrape it from another Carbonizer, which is what the shipped example config does) and you are done.

This page is organised around the questions an operator actually asks ("are scrapes working?", "is Carbon getting through?"), not by subsystem. Each section lists the relevant metrics and explains what the readings mean.

Naming and conventions¶

All metrics defined by the Carbonizer itself use the carbonizer_ prefix. Build information uses pdns_platform_version_info. The default Go-runtime and process metrics (go_*, process_*) are exposed as well, because the service uses the standard promhttp.Handler().

The two *_seconds summaries (carbonizer_scrape_fetch_seconds and carbonizer_carbon_send_seconds) export quantile estimates. The counters are _total in the Prometheus sense — wrap them in rate() or increase() before plotting.

Cardinality watch-outs¶

Most label sets are bounded by static configuration:

scrape_job is the job_name from scrape_configs.
scrape_instance is one entry from static_configs.targets.
dest is one of carbonizer.destinations.

Cardinality changes only when the operator edits the config and restarts the service. There are no unbounded labels.

Are scrapes working?¶

The fastest sanity check is to look at the per-target counters by job and instance:

Metric	Type	When it ticks
`carbonizer_scrape_current_samples{scrape_job,scrape_instance}`	Gauge	Set to the number of samples in the latest successful scrape.
`carbonizer_scrape_fetch_error_total{scrape_job,scrape_instance}`	Counter	The HTTP fetch failed (DNS, connect, timeout, non-200).
`carbonizer_scrape_parse_error_total{scrape_job,scrape_instance}`	Counter	The fetch returned 200 but the body was not valid Prometheus text.
`carbonizer_scrape_fetch_seconds{scrape_job,scrape_instance}`	Summary	Wall time of successful scrapes.

In a healthy deployment current_samples is non-zero for every target and the two *_error_total counters stay flat.

A current_samples value of 0 on a target that used to be non-zero means the target is no longer returning any metrics — usually because something on the target side has changed (the exporter is restarting, the metric set was renamed). It is not the same as a fetch error; the scrape itself succeeded with an empty body.

fetch_seconds rising over time usually points at a target rather than the Carbonizer. Compare against the target's own scrape_timeout — once it crosses that, the scrape will turn into a fetch error instead.

PromQL snippets:

# Targets currently producing zero samples.
carbonizer_scrape_current_samples == 0

# Targets where fetches have been failing in the last 5 minutes.
rate(carbonizer_scrape_fetch_error_total[5m]) > 0

# 99th-percentile scrape duration per target.
carbonizer_scrape_fetch_seconds{quantile="0.99"}

Are samples surviving conversion?¶

A scrape can fetch successfully and still produce zero Carbon lines if the template references a label that the metric does not carry. That shows up here, not in the scrape counters.

Metric	Type	When it ticks
`carbonizer_carbon_convert_metric_error_total{scrape_job,scrape_instance}`	Counter	A single sample was dropped because its labels did not satisfy the template. Increments once per dropped sample.
`carbonizer_carbon_convert_error_total{scrape_job,scrape_instance}`	Counter	Increments once per scrape that had at least one conversion error — useful for "is this target affected at all" without the per-sample volume.

Per-sample errors are expected when a job mixes metrics with different label sets and the template references a label only some of them carry. If that is intentional, the conversion-error counters can be ignored; otherwise, broaden the template (use $__labels__ instead of named labels) or scope the job to one metric source.

Is Carbon getting through?¶

Once samples are converted, the per-destination metrics describe what happens on the wire to Graphite.

Metric	Type	When it ticks
`carbonizer_carbon_send_bytes_total{dest}`	Counter	Bytes successfully written to a destination.
`carbonizer_carbon_send_seconds{dest}`	Summary	Wall time per batch sent (only successful sends).
`carbonizer_carbon_send_connect_error_total{dest}`	Counter	TCP connect to the destination failed.
`carbonizer_carbon_send_write_error_total{dest}`	Counter	Connection opened, write failed (peer reset, timeout, broken pipe).

A send_bytes_total rate of zero for a destination that has scrape traffic going to it means the connect is failing — check send_connect_error_total{dest} and the Carbonizer log, which prints the underlying error at most once every 30 seconds per destination (to avoid log floods on long outages). The status page at / shows the last successful send time per destination as a quick glance.

Connect timeouts come from carbonizer.connect_timeout (default 2s). Each batch opens a new TCP connection, so a slow destination inflates send_seconds directly.

# Bytes per second per destination.
rate(carbonizer_carbon_send_bytes_total[1m])

# Destinations whose connect has been failing.
rate(carbonizer_carbon_send_connect_error_total[5m]) > 0

# Destinations that write but then fail mid-stream.
rate(carbonizer_carbon_send_write_error_total[5m]) > 0

What version is running?¶

Metric	Type	Meaning
`pdns_platform_version_info{version}`	Gauge	Always `1`, labelled with the version string baked into the binary at build time. Useful in dashboards alongside other Platform Filter components.