Warehouse workers process interactive and background SQL queries against your data warehouse. When a user runs a query in Qyra, the API server publishes a job to the NATS warehouse stream, and a warehouse worker picks it up, executes the SQL, and returns the results.

flowchart LR
    API[Qyra API] -->|publish query job| NATS[NATS JetStream]
    NATS -->|deliver| Worker[Warehouse worker]
    Worker -->|execute SQL| WH[(Data warehouse)]
    WH -->|rows| Worker
    Worker -->|result| API

Example configuration

A complete Helm values configuration with NATS and a warehouse worker:

nats:
  enabled: true
  config:
    cluster:
      enabled: false
    jetstream:
      enabled: true
      fileStore:
        enabled: false
      memoryStore:
        enabled: true
        maxSize: 1Gi

warehouseNatsWorker:
  enabled: true
  replicas: 1
  concurrency: 100
  resources:
    requests:
      cpu: 250m
      memory: 1.5Gi
    limits:
      memory: 1.5Gi

The chart auto-configures NATS_ENABLED=true and NATS_URL for you. See the overview for details on JetStream configuration options.

Configuration reference

All configuration is set through your Helm values.yaml under warehouseNatsWorker:

Scaling

Resources

Auto-configured environment variables

These are set by the Helm chart:

Optional environment variables

These can be set via extraEnv or configMap if you need to override the defaults:

Troubleshooting

Queries timing out

If workers are busy, messages may expire before being processed. Scale up replicas or concurrency.

Worker OOM kills

Increase the memory request and limit. Large query result sets are held in memory during processing.

NATS connection errors

1. Confirm the NATS pod is running and healthy
2. Verify that NATS_URL is set correctly on the warehouse worker pod (should be nats://<release>-nats:4222)
3. Check that your network policies allow traffic between the worker pods and the NATS service on port 4222