Observability

Sable is a gateway that meters money, so an operator running it needs the ordinary things: a scrape endpoint and request traces. Both exist, both are off until you configure them, and both are aggregate-only — there is no per-account series and no request content anywhere in either stream, for the reasons in the privacy contract.

This page is for whoever runs the gateway. If you are a buyer wanting proof of what a particular request did, that is a receipt, not a metric.

Prometheus metrics

GET /metrics serves the Prometheus text exposition format at the root, beside /healthz.

Enabling it

SABLE_METRICS_TOKEN=$(openssl rand -hex 32)

Unset, the route is not mounted at all — a request to /metrics 404s exactly like any other unknown path. This is the same posture as SABLE_ADMIN_TOKEN: a route that exists and returns 401 still tells a prober the surface is there.

The token is not ceremony. The metric stream carries no user content, but it does carry request volume, token throughput and metered cost — which is to say it describes revenue. An open /metrics publishes your business volume to anyone who can reach the port.

curl -H "Authorization: Bearer $SABLE_METRICS_TOKEN" \
  https://api.buildsable.com/metrics

Scrape config

scrape_configs:
  - job_name: sable-gateway
    scheme: https
    metrics_path: /metrics
    scrape_interval: 30s
    authorization:
      type: Bearer
      credentials: "<SABLE_METRICS_TOKEN>"
    static_configs:
      - targets: ["api.buildsable.com"]

Database-backed gauges are cached for 15 seconds, so adding scrapers does not add database load — a monitoring system should never become a load generator against the thing it is monitoring.

What is exported

Counters:

Metric	Labels	Meaning
`sable_http_requests_total`	Labels`route`, `status_class`	MeaningRequests by matched route pattern and status class (`2xx`…`5xx`).
`sable_upstream_calls_total`	Labels`provider`, `outcome`	MeaningCalls to an upstream model provider, by outcome class. Failover shows up here first.
`sable_inference_requests_total`	Labels`kind`, `status`	MeaningMetered model requests by kind and terminal status.
`sable_sandbox_runs_total`	Labels`status`	MeaningSandbox executions by terminal status.
`sable_rate_limit_rejections_total`	Labels`limiter`	MeaningRequests refused, by which limiter refused them.
`sable_webhook_deliveries_total`	Labels`outcome`	MeaningWebhook delivery attempts by outcome.
`sable_auth_cache_total`	Labels`result`	MeaningVerified-key cache `hit`/`miss`. A collapsing hit rate means Argon2 is back on the request path.
`sable_prompt_tokens_total`	Labels—	MeaningPrompt tokens metered, all accounts summed.
`sable_completion_tokens_total`	Labels—	MeaningCompletion tokens metered, all accounts summed.
`sable_cost_micro_usd_total`	Labels—	MeaningMetered cost in micro-USD, all accounts summed.
`sable_db_acquires_total`	Labels—	MeaningPool acquisitions completed.
`sable_db_acquire_wait_ms_total`	Labels—	MeaningCumulative wait for a pool connection. Divide by acquisitions for the mean.
`sable_spans_recorded_total`	Labels—	MeaningSpans handed to the OTLP exporter (`0` when export is off).

Gauges:

Metric	Labels	Meaning
`sable_build_info`	Labels`version`, `region`	MeaningAlways `1`; carries the deployed version and declared region.
`sable_uptime_seconds`	Labels—	MeaningSeconds since this process started.
`sable_leader`	Labels—	Meaning`1` if this replica holds the background-task leader lock, else `0`.
`sable_db_pool_connections`	Labels—	MeaningConnections held by the pool.
`sable_db_pool_idle`	Labels—	MeaningIdle connections.
`sable_credit_holds_active`	Labels—	MeaningCredit holds currently reserving spendable balance.
`sable_webhook_outbox_depth`	Labels—	MeaningUndelivered rows in the durable outbox.
`sable_fleet_nodes`	Labels`state`	MeaningEnrolled nodes by heartbeat freshness (`online` / `stale`).
`sable_attestation_verified_backends`	Labels—	MeaningConfidential backends currently verifying. Absent when no TEE backend is configured — absence means "not configured", which a `0` could not distinguish from an outage.
`sable_attestation_total_backends`	Labels—	MeaningConfidential backends configured.
`sable_attestation_consecutive_failures`	Labels—	MeaningAttestation refresh failures since the last success.

The three alerts worth having

# The confidential tier has silently fail-closed. This has happened, more than
# once, and gone unnoticed for days each time.
- alert: SableConfidentialDown
  expr: sable_attestation_verified_backends == 0
  for: 5m

# Nobody is running the leader-gated background workers — health sampling,
# sweeps, purges and alerts have all stopped, while every replica still serves
# traffic and looks healthy.
- alert: SableNoLeader
  expr: max(sable_leader) == 0
  for: 2m

# Webhook delivery is falling behind rather than failing outright.
- alert: SableOutboxBacklog
  expr: sable_webhook_outbox_depth > 500
  for: 10m

OpenTelemetry traces

Set an OTLP/HTTP endpoint and the gateway exports spans to any OTel collector — Grafana Tempo, Datadog, Honeycomb, Langfuse:

SABLE_OTLP_ENDPOINT=https://collector.example.com/v1/traces
SABLE_OTLP_HEADERS=authorization=Bearer abc123,x-tenant=acme
SABLE_OTLP_SAMPLE_RATIO=0.1

Unset, no OpenTelemetry machinery is constructed at all and the log stack is byte-for-byte what it was before — the unconfigured path costs nothing.

SABLE_OTLP_SAMPLE_RATIO defaults to 0.1. Sampling is parent-based, so a trace your own service already sampled arrives whole instead of half-present. An unparseable or out-of-range value falls back to the default rather than refusing to boot; a bad sampling ratio is not a reason to take a gateway down.

Spans are exported in batches and flushed on graceful shutdown. That matters more than it sounds: an exporter that is never flushed drops the last seconds before every deploy, which is exactly the window you are looking at when a deploy goes wrong.

Resource attributes describe the deployment, never a user: service.name, service.version, deployment.region, and a per-process service.instance.id so replicas are distinguishable.

Span attributes follow OpenTelemetry semantic conventions where they exist — http.request.method, http.route, http.response.status_code. Note that otel.status_code is set to ERROR only for 5xx: a 4xx is the caller's error, not the server's, and marking those as failures makes a trace dashboard useless.

What Sable deliberately does not expose

This is a short list on purpose, and it is a design constraint rather than a gap to be filled later.

No content, ever. No prompt, no completion, no submitted code, no sandbox output. A span attribute and a metric label are both log lines that leave the process, so §3 governs them exactly as it governs the request log. There is no debug flag that turns this on.

No per-account series. Not account_id, not a key id, not a wallet address. Two independent reasons, either sufficient:

Cardinality. One series per customer is unbounded by definition, and a metrics endpoint that grows a series per customer is the classic way a monitoring backend takes down the service it monitors.
Disclosure. A per-account breakdown of spend and volume is a per-customer business report. Anyone who can scrape would be reading it.

Everything account-shaped is summed before it reaches the registry. If you need per-account numbers, they belong to the account holder: GET /v1/usage/events and the receipt stream, authenticated as that account.

No raw request paths. The route label is the matched pattern (/v1/receipts/:id), never the concrete path. A raw path carries resource identifiers, which would put ids into both metric labels and span names — one series per receipt, and identifiers shipped to a third-party collector. Unmatched requests are bucketed as route="unmatched" for the same reason.

No unbounded labels. Every label is drawn from a fixed vocabulary, and the registry enforces a hard ceiling on distinct label combinations per metric on top of that. Past the ceiling, new combinations collapse into __other__, so a future mistake degrades a metric instead of leaking memory.

Privacy contract — why the constraints above are constraints
Verifiable receipts — per-request proof, for buyers
Webhooks — including receipt_minted, if you want every metered call in your own audit stack