Relay and regions

What a relay is

mcpd relay runs the same binary as a collector that stores nothing itself. It accepts every HTTP ingest request a daemon accepts, writes the request to a spool on disk, answers the sender, and then ships the request to the daemons --upstream names. It opens no store.

Run one in each region your agents run in. A sender then talks to a collector a few milliseconds away, and the link between regions is crossed by the relay, with retries, rather than by every agent on its own.

an edge relay with two upstreams

mcpd relay \
  --ingest-key $INGEST_KEY \
  --upstream http://mcpd.eu.internal:4317 \
  --upstream http://mcpd.us.internal:4317

relay

$ MCPD_UPSTREAMS=http://mcpd.eu.internal:4317,http://mcpd.us.internal:4317 mcpd --data /data/mcpd.db config | grep ^relayrelay           2 upstream(s) in order, http://mcpd.eu.internal:4317, http://mcpd.us.internal:4317; each sender's key passed on; spool at /data/mcpd.spool/relay, 1024 MiB$ curl -s -D - -o /dev/null localhost:4317/v1/ingest -H "content-type: application/json" -d @batch.json | grep -e ^HTTP -e mcpd-relayHTTP/1.1 202 Acceptedmcpd-relay: spooled$ curl -s localhost:4317/v1/ingest -H "content-type: application/json" -d @batch.json{"accepted":0,"duplicate":0,"rejected":0}

What it answers

Any POST, on any path other than the probes, is accepted. The OTLP paths /v1/traces, /v1/logs and /v1/metrics answer 200 with an OTLP partialSuccess, which is what an OpenTelemetry exporter expects. Every other path answers 202 with the header mcpd-relay: spooled and the body {"accepted":0,"duplicate":0,"rejected":0}: the request is held and will be delivered, and the real counts belong to the upstream, which has not seen it yet. Every SDK reads any 2xx as success.

The relay checks --ingest-key itself, before it writes anything, and applies --max-body-bytes and --request-timeout-seconds as a daemon does. The request is in the spool, <spool-dir>/relay, before the answer is sent.

Delivery and failover

Four workers ship the spool. Each request goes to the first upstream in --upstream order that is not cooling off. A retryable failure cools that upstream, with backoff and jitter or for as long as its Retry-After says, and the request goes to the next upstream at once. When the cool-off ends, the primary is tried first again, so traffic returns to it on its own.

A final refusal, such as a 400, is not sent to the next upstream: the next one would refuse the same bytes. The request is moved to dead/ in the spool directory and counted.

No double counting

Delivery is at least once, and a resend is stored once. Every request carries an Idempotency-Key header: the sender's own when it sent one, otherwise relay- followed by a digest of the path, the content encoding and the body. Every ingest door on the daemon honours the header, and a key in a native batch body wins over it. A request the relay resends after a timeout, or sends to the secondary after the primary took it and the answer was lost, is recognised and reported as a duplicate.

Keys

With --upstream-key, the relay presents that key to every upstream as a Bearer token. Without it, the sender's Authorization or x-mcpd-key header is kept with the spooled request, in a file only the relay's user can read, and passed on. The second form keeps each sender's project and scope as the upstream sees them.

Back-pressure and readiness

The spool is bounded by --spool-max-bytes, 1 GiB for the relay unless you set it. When it is full, a request is refused with a 503 and Retry-After: 5. Every SDK and OpenTelemetry exporter retries that, so the sender holds the data rather than the relay dropping it.

/healthz always answers 200. /readyz answers:

200 ok while an upstream is reachable;
200 degraded, with a detail, when no upstream has answered its /readyz or taken a delivery in the last 30 seconds. Upstreams are probed every 5 seconds;
503 when the spool is full, or when no upstream is reachable and the spool is at least 80% full.

A load balancer therefore keeps sending to a relay that is buffering, and moves traffic away from one that can no longer take it.

Relay metrics

Metric	Type	Meaning
`mcpd_relay_accepted_total`	counter	Requests written to the spool
`mcpd_relay_refused_full_total`	counter	Requests refused with a 503 because the spool was full
`mcpd_relay_failovers_total`	counter	Requests delivered to an upstream other than the first
`mcpd_relay_dead_letter_total`	counter	Requests an upstream refused for good, moved to `dead/`
`mcpd_relay_spool_entries`	gauge	Requests waiting in the spool
`mcpd_relay_spool_bytes`	gauge	Bytes the spool holds
`mcpd_relay_spool_oldest_seconds`	gauge	Age of the oldest request waiting, or 0
`mcpd_relay_delivered_total`	counter	Requests an upstream accepted, labelled `upstream` and `order`
`mcpd_relay_failed_attempts_total`	counter	Attempts an upstream did not accept, labelled the same way
`mcpd_relay_upstream_up`	gauge	1 when the upstream answered in the last 30 seconds, 0 otherwise
`mcpd_uptime_seconds`	gauge	Seconds since this process started

A rising mcpd_relay_failovers_total means the primary is failing and the secondary is carrying its traffic. mcpd_relay_spool_oldest_seconds is how far behind delivery is.

Several regions

Run several daemon replicas, in one region or more, against one Postgres or ClickHouse store, with the runs in Postgres (--runs-data postgres://...) or off: an embedded run store belongs to one process. Each replica's /readyz already reflects whether it can reach the store. Put a relay in each region in front of them, with --upstream listing the nearest daemon first and a daemon in another region second.

Four settings behave differently once there is more than one replica:

Idempotency keys are held in the shared store, so a resend that lands on another replica is still recognised. Postgres records the key in the same transaction as the batch.
Per-minute quota ceilings are counted per replica. Divide the ceiling you want by the number of replicas.
Live scoring, tail sampling and monitor alerts run on one replica at a time. The replicas take turns through a lease in the shared store: the holder renews it on every pass, gives it up when it shuts down, and another replica takes over within three passes if it stops. Every replica evaluates its monitors and objectives, so each one answers the monitor API, and only the holder posts the webhook. Configure them the same way on every replica.
Aspect analysis runs on every replica that has it configured. Configure it on one replica.

Example deployments

deploy/docker/docker-compose.multi-region.yml starts Postgres, two daemons and a relay on one machine, which is the whole topology in a form you can try.

The Helm chart takes mode: relay. Three values files under deploy/helm/examples/ describe one region each: region-core.yaml, region-standby.yaml and region-edge-relay.yaml, the last for a region that runs only a relay.

Copying data on to other vendors: Forwarding.
Probes, metrics and the stores: Running it.
Every flag: Configuration.