Skip to content
Vouch Docs

Observability

The API is instrumented with OpenTelemetry (FastAPI and SQLAlchemy instrumentation, OTLP/HTTP export), Traefik emits OTLP traces, and every node’s container logs are tailed. All of it lands in HyperDX, an open-source observability UI on top of ClickHouse.

The pipeline

Section titled “The pipeline”
API pods ──OTLP──┐
Traefik ──OTLP──┤──▶ otel-gateway (contrib collector, one per node) ──▶ ClickHouse ◀── HyperDX UI
node logs ──────┘        filelog + k8sattributes + clickhouse exporter
  • otel-gateway is the upstream opentelemetry-collector Helm chart in daemonset mode with the contrib image. Each pod receives OTLP on a ClusterIP Service (otel-gateway.observability.svc:4317/4318, routed to the collector on the same node) and tails its node’s container logs with Kubernetes metadata attached.
  • The clickhouse exporter writes directly into HyperDX’s ClickHouse (default.otel_logs, otel_traces, otel_metrics_*) with create_schema: true and a three-day TTL. The pipeline half of the config lives in deploy/otel/gateway-config.yaml and is validated by the real collector binary in CI (otelcol-contrib validate).
  • HyperDX’s bundled collector is disabled. It is OpAMP-supervised: its OTLP receivers open only after the HyperDX app pushes it a remote config (until then it logs No last received remote config found), so ingestion would depend on a control channel we cannot inspect. Writing the tables ourselves removes that dependency. One schema difference is handled in deploy/values/hyperdx.yaml.gotmpl: HyperDX’s own collector adds a TimestampTime column to otel_logs; the contrib exporter does not, so the Logs source is told to sort on Timestamp.

Alerting

Section titled “Alerting”

Alerting lives outside the cluster so it keeps working when the cluster does not: CloudWatch alarms on both nodes’ EC2 status checks and a Route53 HTTPS health check on api.<domain>/healthz, all notifying an SNS email topic (infra/alerting.tf). Route53 health-check metrics only exist in us-east-1, so that alarm and its topic live there; expect two subscription confirmation emails after a fresh deploy.

Inside the application

Section titled “Inside the application”

GET /admin/overview reports the background matcher’s last run, result and error, the migration revision the database is at, and the settings (secrets redacted). The vouch admin overview command and the admin UI render it.

What is not covered

Section titled “What is not covered”
  • No alert on pod crashloops, disk pressure or certificate renewal failures; the health check catches the user-visible result of all three.
  • HyperDX’s own alert rules are not configured.
  • The gateway’s ClickHouse credentials are the HyperDX chart defaults (cluster-internal); move them to Secrets Manager when the observability stack matters beyond debugging.