Skip to main content

Grafana

Telemetry Provided
  • Logs - Application and container logs for correlation with Diagnoses
  • Metrics - Performance metrics from applications and infrastructure
  • Signals - Automatic Signal detection from metrics, traces, and external monitoring systems
  • Traces - Distributed traces for service dependency discovery and communication monitoring

Causely can leverage Grafana as an observability backend with Grafana Alloy and Beyla as instrumentation.

Note: If your logs are centralized in Grafana Loki, Causely can retrieve and display them directly.
See the Loki Integration section below for configuration details and benefits.

Please use the following values for your helm install:

Install with Causely​

helm upgrade --install alloy grafana/alloy --create-namespace --namespace monitoring --values ./alloy-values.yaml

alloy-values.yaml​

controller:
  type: deployment
alloy:
  stabilityLevel: experimental
  extraPorts:
    - name: 'grpc'
      port: 4317
      targetPort: 4317
    - name: 'http'
      port: 4318
      targetPort: 4318
    - name: 'datadog'
      port: 8126
      targetPort: 8126
  configMap:
    content: |
      otelcol.exporter.otlp "causely" {
        client {
          endpoint = "mediator.causely:4317"
          tls {
            insecure = true
          }
        }
      }

      otelcol.processor.batch "default" {
        output {
          metrics = [otelcol.exporter.otlp.causely.input]
          traces  = [otelcol.exporter.otlp.causely.input]
        }
      }

      otelcol.processor.k8sattributes "default" {
        extract {
          label {
            from = "pod"
          }

          metadata = [
            "k8s.namespace.name",
            "k8s.pod.name",
            "k8s.pod.uid",
            "k8s.deployment.name",
            "k8s.node.name",
            "k8s.pod.start_time",
            "container.id",
          ]
        }

        output {
          traces = [otelcol.processor.batch.default.input]
        }
      }

      otelcol.processor.deltatocumulative "default" {
        output {
          metrics = [otelcol.processor.batch.default.input]
        }
      }

      otelcol.receiver.datadog "default" {
        endpoint = "0.0.0.0:8126"
        output {
          metrics = [otelcol.processor.deltatocumulative.default.input]
          traces  = [otelcol.processor.batch.default.input]
        }
      }

      otelcol.receiver.otlp "otlp" {
        grpc {
          endpoint = "0.0.0.0:4317"
        }
        http {
          endpoint = "0.0.0.0:4318"
        }

        output {
          metrics = [otelcol.processor.batch.default.input]
          traces  = [otelcol.processor.k8sattributes.default.input]
        }
      }

Install with Causely and Grafana Cloud​

To also send logs, metrics, and traces to Grafana Cloud, you need to use the following values for your helm install:

helm upgrade --install alloy grafana/alloy --create-namespace --namespace monitoring --values ./grafana-alloy-values.yaml

grafana-alloy-values.yaml​

controller:
  hostPID: true
alloy:
  securityContext:
    privileged: true
  stabilityLevel: experimental
  extraPorts:
    - name: 'grpc'
      port: 4317
      targetPort: 4317
    - name: 'http'
      port: 4318
      targetPort: 4318
    - name: 'datadog'
      port: 8126
      targetPort: 8126
  configMap:
    content: |
      otelcol.exporter.otlp "causely" {
        client {
          endpoint = "mediator.causely:4317"
          tls {
            insecure = true
          }
        }
      }

      otelcol.exporter.otlphttp "grafana" {
        client {
          endpoint = "https://otlp-gateway-prod-us-east-0.grafana.net/otlp"
          auth     = otelcol.auth.basic.grafana.handler
        }
      }

      otelcol.auth.basic "grafana" {
        username = "GRAFANA_CLOUD_INSTANCE_ID"
        password = "GRAFANA_CLOUD_API_KEY"
      }

      // discovery.kubernetes allows you to find scrape targets from Kubernetes resources.
      // It watches cluster state and ensures targets are continually synced with what is currently running in your cluster.
      discovery.kubernetes "pod_logs" {
        role = "pod"
      }

      // discovery.relabel rewrites the label set of the input targets by applying one or more relabeling rules.
      // If no rules are defined, then the input targets are exported as-is.
      discovery.relabel "pod_logs" {
        targets = discovery.kubernetes.pod_logs.targets

        // Label creation - "namespace" field from "__meta_kubernetes_namespace"
        rule {
          source_labels = ["__meta_kubernetes_namespace"]
          action = "replace"
          target_label = "namespace"
        }

        // Label creation - "pod" field from "__meta_kubernetes_pod_name"
        rule {
          source_labels = ["__meta_kubernetes_pod_name"]
          action = "replace"
          target_label = "pod"
        }

        // Label creation - "container" field from "__meta_kubernetes_pod_container_name"
        rule {
          source_labels = ["__meta_kubernetes_pod_container_name"]
          action = "replace"
          target_label = "container"
        }

        // Label creation -  "app" field from "__meta_kubernetes_pod_label_app_kubernetes_io_name"
        rule {
          source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_name"]
          action = "replace"
          target_label = "app"
        }

        // Label creation -  "job" field from "__meta_kubernetes_namespace" and "__meta_kubernetes_pod_container_name"
        // Concatenate values __meta_kubernetes_namespace/__meta_kubernetes_pod_container_name
        rule {
          source_labels = ["__meta_kubernetes_namespace", "__meta_kubernetes_pod_container_name"]
          action = "replace"
          target_label = "job"
          separator = "/"
          replacement = "$1"
        }

        // Label creation - "container" field from "__meta_kubernetes_pod_uid" and "__meta_kubernetes_pod_container_name"
        // Concatenate values __meta_kubernetes_pod_uid/__meta_kubernetes_pod_container_name.log
        rule {
          source_labels = ["__meta_kubernetes_pod_uid", "__meta_kubernetes_pod_container_name"]
          action = "replace"
          target_label = "__path__"
          separator = "/"
          replacement = "/var/log/pods/*$1/*.log"
        }

        // Label creation -  "container_runtime" field from "__meta_kubernetes_pod_container_id"
        rule {
          source_labels = ["__meta_kubernetes_pod_container_id"]
          action = "replace"
          target_label = "container_runtime"
          regex = "^(\\S+):\\/\\/.+$"
          replacement = "$1"
        }
      }

      // loki.source.kubernetes tails logs from Kubernetes containers using the Kubernetes API.
      loki.source.kubernetes "pod_logs" {
        targets    = discovery.relabel.pod_logs.output
        forward_to = [otelcol.receiver.loki.default.receiver]
      }

      otelcol.receiver.loki "default" {
        output {
          logs = [otelcol.processor.batch.default.input]
        }
      }

      beyla.ebpf "default" {
        attributes {
          kubernetes {
            enable = "true"
          }
          select {
            attr = "sql_client_duration"
            exclude = []
            include = ["db.query.text"]
          }
        }
        discovery {
          services {
            open_ports = "80,443,3000,8000-8999"
          }
        }
        output {
          traces = [otelcol.processor.batch.default.input]
        }
      }

      otelcol.processor.batch "default" {
        output {
          logs    = [otelcol.exporter.otlphttp.grafana.input]
          metrics = [otelcol.exporter.otlphttp.grafana.input, otelcol.exporter.otlp.causely.input]
          traces  = [otelcol.exporter.otlphttp.grafana.input, otelcol.exporter.otlp.causely.input]
        }
      }

      otelcol.processor.k8sattributes "default" {
        extract {
          label {
            from = "pod"
          }

          metadata = [
            "k8s.namespace.name",
            "k8s.pod.name",
            "k8s.pod.uid",
            "k8s.deployment.name",
            "k8s.node.name",
            "k8s.pod.start_time",
            "container.id",
          ]
        }

        output {
          traces = [otelcol.processor.batch.default.input]
        }
      }

      otelcol.processor.deltatocumulative "default" {
        output {
          metrics = [otelcol.processor.batch.default.input]
        }
      }

      otelcol.receiver.datadog "default" {
        endpoint = "0.0.0.0:8126"
        output {
          metrics = [otelcol.processor.deltatocumulative.default.input]
          traces  = [otelcol.processor.batch.default.input]
        }
      }

      otelcol.receiver.otlp "otlp" {
        grpc {
          endpoint = "0.0.0.0:4317"
        }
        http {
          endpoint = "0.0.0.0:4318"
        }

        output {
          logs    = [otelcol.processor.batch.default.input]
          metrics = [otelcol.processor.batch.default.input]
          traces  = [otelcol.processor.k8sattributes.default.input]
        }
      }

Alerts​

Causely can ingest the alerts from Grafana's unified alerting and use them as Signals for causal analysis. The Grafana scraper periodically reads the currently active alert instances from Grafana's Alertmanager-compatible API (/api/alertmanager/grafana/api/v2/alerts), maps them to the right entities, and activates the corresponding Causely Signals. When an alert clears in Grafana, Causely automatically deactivates the Signal.

This means you can:

  • Keep your existing Grafana alert rules and contact points.
  • See Grafana alerts mapped to services and dependencies inside Causely.
  • Automatically map alerts to Signals and point to the actual cause.

Authentication​

Each Grafana instance accepts one of:

  • Service account token: set token to a Grafana service account token (for example, glsa_...). The token only needs read access to alerts/alerting.
  • Basic auth: set username and password instead of token.

Auto-discovery with a Kubernetes Secret​

When auto_discovery.enabled is true, the mediator watches for Kubernetes Secrets labeled causely.ai/scraper: Grafana and creates a scraper for each one. This lets you add or rotate Grafana instances without changing the mediator configuration.

apiVersion: v1
kind: Secret
metadata:
  name: grafana-prod
  namespace: causely
  labels:
    causely.ai/scraper: Grafana
type: Opaque
stringData:
  endpoint: https://grafana.example.com
  token: glsa_xxxxxxxxxxxxxxxxxxxx
  # username: causely-readonly   # basic auth (alternative to token)
  # password: ...
  # labelFilters is an optional JSON object; only alerts matching every
  # listed label/value are ingested.
  labelFilters: '{"namespace": ["payments", "checkout"]}'

The recognized Secret keys are endpoint, token, username, password, and labelFilters.

Filtering which alerts are ingested​

Use labelFilters to restrict ingestion to alerts carrying specific labels. Each entry is a label name mapped to the set of allowed values; an alert is ingested only if it matches all of the listed labels.

Mapping alert severity​

Grafana alerts carry a severity label (commonly severity). Use alert_severity to normalize your severity values to Causely severity levels (critical, major, minor, info).

scrapers:
  grafana:
    alert_severity:
      label_name: severity
      mappings:
        - value: critical
          level: critical
        - value: warning
          level: major

Mapping alerts to entities and Signals​

By default, Causely maps error- and latency-type alerts to the corresponding Service Signals using keyword matching. Use alerts to explicitly map a Grafana alert to a Causely Signal on a specific entity. For example, the following maps the HighDBOpenConnections alert to the DBConnectionsUtilization_High symptom of the matching workload:

scrapers:
  grafana:
    alerts:
      - alert_name: 'HighDBOpenConnections'
        symptom: 'DBConnectionsUtilization_High'
        entity:
          workload: {}
        discovery:
          - kubernetes_pod:
              namespace: 'namespace'
              pod_name: 'pod'

The discovery block resolves the entity from the alert's labels: here, the namespace and pod labels on the alert identify the Kubernetes pod (and therefore its workload) the symptom is attached to.

Loki Integration​

Causely integrates with Loki to retrieve logs already centralized in your Grafana environment.
When configured, Causely automatically surfaces relevant logs in the context of active Diagnoses and service malfunctions.
This enables rapid validation of issues and faster time to understanding and resolution.

When to Use​

Use this integration when:

  • Your logs are shipped to Loki instead of remaining in the Kubernetes API.
  • You want Causely to display relevant log lines and exceptions alongside detected service degradations or Diagnoses.
  • You prefer a centralized, scalable log pipeline already managed through Grafana Cloud or self-hosted Loki.

Benefits​

  • Contextual Insight: Automatically surfaces logs correlated with active Diagnoses or degraded services.
  • Accelerated RCA: Shows container log lines, stack traces, and error spikes precisely around the time of failure.
  • Unified View: Displays Loki logs directly within Causely, alongside metrics and traces for the same service.
  • Operational Efficiency: Reduces reliance on the Kubernetes API for log collection.

Configuration​

Configure Causely:

scrapers:
  kubernetes:
    loki_endpoint: 'http://loki.monitoring:3100'
    loki_tenant: 'production'

Causely automatically uses Loki as the preferred external log source with an optional tenant id if used in a multitenant loki configuration.

Supported Log Types​

  • Container logs (stdout/stderr)

Result​

When Causely detects a service malfunction or identifies a Diagnosis, it automatically retrieves related logs from Loki.
These logs appear:

  • Under affected services, when they exhibit abnormal behavior such as elevated errors or latency.
  • Alongside Diagnoses, showing relevant exceptions or stack traces at the time of failure.

This provides clear evidence for what went wrong and why, dramatically shortening investigation and resolution times.