Scalable Trace Processor

By default, Causely processes OpenTelemetry traces inline inside the mediator. For most environments this is sufficient. When trace volume grows or you want to scale trace processing independently from the rest of the mediator, you can enable the scalable trace processor.

What gets deployed

Enabling the scalable trace processor adds two components to your Causely installation:

• Trace controller: a stateless router that receives OTLP traces from your collectors and distributes work across workers. Exposes gRPC on port 4317.
• Trace workers: a StatefulSet of workers that analyze spans and forward results to the mediator. Workers are stateful; each one accumulates trace context for the services assigned to it.

Your OTel collectors send traces to the trace controller instead of the mediator directly. The mediator continues to handle all other telemetry sources unchanged.

When to enable it

Consider enabling the scalable trace processor when:

• Trace volume has grown to the point where the mediator's CPU is consistently elevated due to span processing
• You want to scale trace throughput by adding worker replicas without resizing the mediator

Enabling the scalable trace processor

Add the following to your causely-values.yaml:

traceController:
  enabled: true

Then upgrade your Helm release:

helm upgrade causely \
  oci://us-docker.pkg.dev/public-causely/public/causely \
  --version <version> \
  --namespace=causely \
  --values </path/to/causely-values.yaml>

Updating your trace sources

Beyla (eBPF auto-instrumentation)

If you are using Causely's built-in Beyla instrumentation, no changes are needed. When traceController.enabled: true, the Helm chart automatically reconfigures Beyla to send traces to the trace controller. The helm upgrade will trigger a rolling restart of the agent DaemonSet to pick up the new configuration.

OpenTelemetry Collector

If you have an external OTel collector sending traces to Causely, update its exporter endpoint:

exporters:
  otlp/causely:
    endpoint: trace-controller.causely:4317
    compression: none
    tls:
      insecure: true

Replace mediator.causely:4317 with trace-controller.causely:4317. All other collector settings remain unchanged. See Integrate OpenTelemetry Traces for full collector configuration examples.

Gradual migration

The mediator continues to accept traces on port 4317 even after the scalable trace processor is enabled. External collectors that have not yet been updated will continue to work via the inline path. You can migrate collectors one at a time without any gap in trace ingestion.

Sizing

The defaults are suitable for most environments. Use the replica table below as a starting point based on your service count.

Default resources

	CPU request	CPU limit	Memory request	Memory limit
Trace controller	200m	1000m	128Mi	4Gi
Trace worker	200m	1000m	256Mi	4Gi

Replica starting points

Services monitored per mediator	Worker replicas	Controller replicas
< 100	2	1
100–500	3–5	2
500–1,000	5–10	2
> 1,000	10+	3+

info

Never run fewer than 2 worker replicas in production. A single worker replica means all trace processing stops if that pod is lost; trace context is lost until the next export.

To override replicas in your values file:

traceController:
  enabled: true
  replicas: 2

traceWorker:
  replicas: 3

Scaling signals

Monitor these four signals to know when to add replicas.

Signal	Meaning	Action
causely_tc_inflight_rejected_total > 0	Controller dispatch queue is full	Check controller CPU. If high → add controller replicas. If low → add worker replicas.
causely_tc_dispatch_drops_total or causely_tc_consume_drops_total growing	Exports being dropped	Check per-worker causely_tw_analyze_inflight. If one pod is high → add workers. If all pods are high → add workers.
causely_tw_analyze_inflight > 2 sustained (per pod)	Work queuing inside a worker	Add worker replicas.
Worker RSS > 75% of memory limit	Memory pressure	Raise the worker memory limit; also check whether your service count has grown significantly.

Scale workers horizontally, not vertically. Adding worker replicas increases parallel processing capacity. Increasing pod size does not.