Configure OpenTelemetry tracing for self-hosted Qyra
Export distributed traces from your Qyra instance to any OpenTelemetry-compatible backend
🛠 This page is for engineering teams self-hosting their own Qyra instance. For instance health metrics, see Prometheus metrics.
Qyra can export distributed traces using the OpenTelemetry SDK, so you can follow a request or scheduled job across the API server, scheduler, and warehouse queries in any OpenTelemetry-compatible backend (for example Grafana Tempo, Jaeger, Honeycomb, or Datadog).
Tracing runs in one of two exclusive modes:
- Sentry mode (default): spans are created and exported through Sentry, controlled by the Sentry environment variables.
- OpenTelemetry mode: spans are created by the OpenTelemetry SDK and exported according to the standard
OTEL_*environment variables. Sentry still captures errors, but receives no spans.
Enabling OpenTelemetry tracing
By default, Qyra traces through Sentry. To switch to OpenTelemetry mode, set the following environment variable on every Qyra container (API server and scheduler):
QYRA_OTEL_TRACES_ENABLED=trueThen point the OTLP exporter at your collector:
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318Configuration options
| Variable | Description |
|---|---|
QYRA_OTEL_TRACES_ENABLED | Switches tracing to OpenTelemetry mode: spans are created by the OpenTelemetry SDK and Sentry receives errors only. When false, tracing runs through Sentry. (default=false) |
QYRA_OTEL_TRACES_SAMPLE_RATE | Trace sampling ratio from 0.0 to 1.0. Falls back to SENTRY_TRACES_SAMPLE_RATE (deprecated) when unset. (default=1) |
QYRA_OTEL_ALWAYS_SAMPLE_AI_TRACES | Set to false to stop always-sampling AI agent traces, so they follow the global sampling ratio instead. (default=true) |
QYRA_OTEL_DB_TRACES_ENABLED | Adds spans for application database (Postgres) queries to traces, including the SQL statement. Requires OpenTelemetry mode. (default=false) |
QYRA_OTEL_DB_TRACES_MAX_QUERY_LENGTH | Maximum length of the SQL statement recorded on database spans; longer statements are truncated. Must be a non-negative integer. (default=1022) |
OTEL_SDK_DISABLED | Standard OpenTelemetry kill switch. When true, OpenTelemetry mode is off even if QYRA_OTEL_TRACES_ENABLED=true. |
OTEL_TRACES_EXPORTER | Comma-separated exporters: otlp, console, zipkin, or none. Unsupported values are ignored with a startup warning; none overrides any other value. (default=otlp) |
OTEL_EXPORTER_OTLP_PROTOCOL | OTLP protocol: grpc, http/json, or http/protobuf. OTEL_EXPORTER_OTLP_TRACES_PROTOCOL takes precedence. Unsupported values fall back to the default. (default=http/protobuf) |
OTEL_EXPORTER_OTLP_ENDPOINT | Base URL of your OTLP collector. Handled by the OpenTelemetry Node SDK, along with OTEL_EXPORTER_OTLP_TRACES_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS. |
OTEL_SERVICE_NAME | Service name attached to exported spans. (default=qyra) |
OTEL_LOG_LEVEL | Enables OpenTelemetry SDK and exporter diagnostics at the given level (e.g. DEBUG). Qyra redacts credentials, tokens, and span payloads from diagnostic output. |
OTEL_TRACES_SAMPLER and OTEL_TRACES_SAMPLER_ARG are overridden by Qyra. Control sampling with QYRA_OTEL_TRACES_SAMPLE_RATE instead.
Choosing an exporter
Qyra defers exporter and protocol selection to the OpenTelemetry Node SDK's standard environment variable handling:
OTEL_TRACES_EXPORTERsupportsotlp(default),console,zipkin, andnone. You can combine exporters with a comma-separated list. If the list containsnone, no traces are exported regardless of the other values.- With the
otlpexporter,OTEL_EXPORTER_OTLP_PROTOCOL(or the traces-specificOTEL_EXPORTER_OTLP_TRACES_PROTOCOL) selectsgrpc,http/json, orhttp/protobuf(default). - Unsupported exporter or protocol values are ignored with a warning in the Qyra logs at startup.
For example, to export traces over gRPC with an authentication header:
QYRA_OTEL_TRACES_ENABLED=true
OTEL_TRACES_EXPORTER=otlp
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.example.com:4317
OTEL_EXPORTER_OTLP_HEADERS="authorization=Bearer <token>"Sampling
QYRA_OTEL_TRACES_SAMPLE_RATE sets the head-sampling ratio for trace roots, from 0.0 (nothing) to 1.0 (everything, the default). Child spans follow their root's decision, so a sampled request captures the whole waterfall.
Two behaviours to be aware of:
- AI agent traces are always sampled regardless of the ratio, so a broken agent run always has a trace to debug. Set
QYRA_OTEL_ALWAYS_SAMPLE_AI_TRACES=falseto make them follow the global ratio instead. OTEL_TRACES_SAMPLERandOTEL_TRACES_SAMPLER_ARGare overridden by Qyra's own sampler and have no effect.
Requests to health checks (/api/v1/health, livez), status polling endpoints, favicon.ico, and robots.txt are never traced.
Database query tracing
Set QYRA_OTEL_DB_TRACES_ENABLED=true to add a span for each application database (Postgres) query, so you can see where a request spends time inside Qyra's own database:
QYRA_OTEL_TRACES_ENABLED=true
QYRA_OTEL_DB_TRACES_ENABLED=true- Database spans only appear inside an existing trace, so they follow the sampling decision of their parent request or job.
- Each span records the SQL statement, truncated to
QYRA_OTEL_DB_TRACES_MAX_QUERY_LENGTHcharacters (default1022). Invalid values fall back to the default with a warning. - This traces queries to Qyra's application database only, not queries sent to your data warehouse.
Troubleshooting
At startup, Qyra logs a line confirming the tracing configuration, including the active exporters, OTLP protocol, and sampling ratio. This confirms configuration only; exporter connectivity is not validated at startup.
If traces aren't arriving in your backend, set OTEL_LOG_LEVEL=DEBUG to enable OpenTelemetry SDK and exporter diagnostics in the Qyra logs. Diagnostic output is redacted before logging: URL credentials and query strings are stripped, values containing tokens or API keys are omitted, and span payloads are not printed.
Metrics
This page covers traces only. For instance health metrics (CPU, memory, event loop, query durations), see Prometheus metrics, which can also feed an OpenTelemetry backend through the collector's Prometheus receiver.