# Causely documentation: telemetry-sources > Full content of the "telemetry-sources" section of the Causely documentation. This file contains all documentation content in a single document following the llmstxt.org standard. ## Alertmanager ### Overview You can forward alerts from [Prometheus Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/) a to Causely by adding a webhook receiver. Once connected, Causely ingests your existing alerts, links them to the right services, and applies causal analysis to show the real underlying issues. This means you can: - Keep using your current alerting rules. - See alerts mapped to services and dependencies inside Causely. - Automatically identify which alerts are Signals and which point to the actual causes. ### Add Causely as a webhook receiver in Alertmanager If you are using the Prometheus Operator, add the following AlertmanagerConfig to forward alerts to Causely’s mediator. ```yaml kind: AlertmanagerConfig spec: route: group_by: ['alertname', 'pod', 'namespace'] # how alerts are grouped repeat_interval: 3h # how often repeated alerts are sent receiver: 'causely-mediator-webhook' # send all alerts here receivers: - name: 'causely-mediator-webhook' webhook_configs: - url: 'http://mediator.causely:9093/api/v1/alerts' # mediator endpoint send_resolved: true # send firing + resolved states ``` ### Enabling Alertmanager as a Data Source Add the following to your `values.yaml` to enable Alertmanager ingestion in the mediator: ```yaml mediator: alertmanager: enabled: true port: 9093 alert_mappings: [] # optional; see examples below ``` #### Optional: Map Alerts to Entities and Attributes Use `alert_mappings` to map Alertmanager alerts to Causely Signals. For example, the following maps the `Error Rate High` alert to the `RequestErrorRate_High` symptom of the Service Entity: ```yaml alert_mappings: - alert_name: 'Error Rate High' symptom: 'RequestErrorRate_High' entity: service: {} discovery: - kubernetes_pod: namespace: 'namespace' pod_name: 'pod' conditions: - regex_not_match: label: 'pod' regex: '^.*-exporter.*$' ``` Without an `alert_mappings` entry, Causely will automatically map error and latency-type alerts, based on keyword matching, to the corresponding Causely Service Signals. #### Example Alert Payload An example alert from Alertmanager that Causely can ingest: ```json [ { "labels": { "alertname": "HighErrorRate", "severity": "warning", "service": "causely-analysis", "namespace": "production" }, "annotations": { "summary": "Error rate above 5% for 10m" }, "startsAt": "2025-08-11T13:30:56Z", "endsAt": "0001-01-01T00:00:00Z", "generatorURL": "http://prometheus/graph?g0.expr=...", "fingerprint": "047235cd648d8d01" } ] ``` This alert would be mapped automatically to the high error rate Signal of the `causely-analysis` service. --- ## Application Instrumentation ### Overview Causely provides comprehensive observability and causal reasoning for applications written in various programming languages by leveraging industry-standard telemetry collection frameworks like [Prometheus](/telemetry-sources/prometheus) and [OpenTelemetry](/telemetry-sources/opentelemetry). This approach ensures that regardless of your application's programming language, Causely can transform high-volume observability signals into real-time, explainable insights about performance, resource utilization, and potential issues. Instead of just monitoring Signals, Causely analyzes them in real time to surface the actual Diagnosis behind application issues. By setting up the application languages integration, you will be able to do the following: - Identify causes for reliability issues originating from your application, including: - [Database Connection Pool Saturated](/reference/root-causes/applications#database-connection-pool-saturated) - [File Descriptor Exhaustion](/reference/root-causes/applications#file-descriptor-exhaustion) - [Inefficient Garbage Collection](/reference/root-causes/applications#inefficient-garbage-collection) - [Lock Contention](/reference/root-causes/applications#lock-contention) - [Invalid Client Certificate](/reference/root-causes/applications#invalid-client-certificate) - [Service Congestion](/reference/root-causes/services#congested) - [Service Malfunction](/reference/root-causes/services#malfunction) - Language-specific causal reasons, including: - [GOMAXPROCS Misconfigured](/reference/root-causes/applications#gomaxprocs-misconfigured) - [Java Heap Saturated](/reference/root-causes/applications#java-heap-saturated) - Observe the application as an entity in the Topology Graph, including it's relationships to other entities on the service map, infrastructure stack and dataflow map. ### Language Support Causely provides **automatic, zero-code instrumentation** for most applications using [eBPF-based automatic instrumentation](/telemetry-sources/ebpf). This works out-of-the-box for most applications and provides comprehensive trace coverage. For application that are already instrumented using OpenTelemetry, Causely can collect telemetry from them using the [OpenTelemetry Collector](/telemetry-sources/opentelemetry). This way Causely can provide comprehensive observability and causal reasoning for applications written in any language, including the following: - C++ - C#/.NET - Erlang/Elixir - Go - Java - JavaScript - PHP - Python - Ruby - Rust - Swift :::tip OpenTelemetry and Prometheus are open source standards and by that can be implemented by any language. This means that as long as support for your language is available, Causely can collect telemetry from it. ::: --- ## Amazon Web Services (AWS) ### Overview Causely provides comprehensive native integration with Amazon Web Services (AWS) to help you identify and resolve infrastructure and service performance issues before they impact your users. Instead of just monitoring Signals, Causely analyzes them in real time from your AWS infrastructure to surface the actual Diagnosis behind problems across your entire AWS ecosystem. By setting up the AWS integration, you will be able to: - **Identify causes for reliability issues originating from your AWS infrastructure and services**, including: - [Service Congestion](/reference/root-causes/services#congested-1) and [Service Malfunction](/reference/root-causes/services#malfunction-1) - [Application Load Balancer Misconfiguration](/reference/root-causes/services#application-load-balancer) issues - [Authentication](/reference/root-causes/services#authentication-misconfiguration) and [Network Policy](/reference/root-causes/services#network-policy-misconfiguration) misconfigurations - [Memory Pressure](/reference/root-causes/infrastructure#memory-pressure) and [CPU Congested](/reference/root-causes/infrastructure#cpu-congested) issues - [Disk Pressure](/reference/root-causes/infrastructure#disk-pressure) and [I/O congestion](/reference/root-causes/infrastructure#disk-total-iops-congested) - **Observe AWS services** as entities in the Topology Graph, including their relationships to Kubernetes services and dependencies - **Monitor performance metrics** from CloudWatch with automatic correlation to service health and user impact ### Supported AWS Services Causely integrates with the following AWS services to provide comprehensive observability and causal inference: - Amazon Elastic Block Store (EBS) - Amazon Elastic Compute Cloud (EC2) - Amazon Elastic Container Service (ECS) - Amazon ElastiCache - Amazon Managed Streaming for Apache Kafka (MSK) - Amazon OpenSearch Service - Amazon Relational Database Service (RDS) - Amazon Simple Notification Service (SNS) - Amazon Simple Queue Service (SQS) - Application Load Balancer (ALB) - AWS Lambda - Network Load Balancer (NLB) :::tip[Database-specific setup] For AWS RDS databases, see the database-specific documentation for detailed setup and configuration verification: - [PostgreSQL](/telemetry-sources/postgresql) - [MySQL](/telemetry-sources/mysql) - [OpenSearch / Elasticsearch](/telemetry-sources/elasticsearch) ::: ### Setup Guide #### Authentication Options Causely supports two authentication methods for AWS integration. Choose the method that best fits your security requirements. ##### Option 1: IAM Role IAM roles for service accounts in Amazon EKS enhance security by enabling least privilege access, isolating credentials between Pods, and improving auditability. AWS security best practices promote the use of EKS Pod identity. For more details see the [AWS documentation for EKS Pod Identity](https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html) and the [for IAM Roles for Service Accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html). ###### 1. Create an IAM role in the source account Select the Custom trust policy as trusted entity type and paste the following trust policy with your account id, your region and your OIDC provider id: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::111111111111:oidc-provider/oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:sub": "system:serviceaccount:causely:causely-mediator", "oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:aud": "sts.amazonaws.com" } } } ] } ``` ###### 2. Name the role Enter a name for the role, for example: `CauselyMediation` ###### 3. Register the OIDC provider in the target account Run the following command to register the OIDC provider in the target account: ```bash aws iam create-open-id-connect-provider \ --url "https://oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE" \ --client-id-list "sts.amazonaws.com" \ --thumbprint-list "AABBCCDDEEFFGGHHIIJJKKLLMMNNOOPPQQRRSSTT" ``` ###### 4. Create an IAM role in the target account Select the Custom trust policy as trusted entity type and paste the following trust policy with your source account id: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:sts::222222222222:assumed-role/CauselyMediation/CauselyMediatorSession" }, "Action": "sts:AssumeRole" } ] } ``` ###### 5. Name the target account role Enter a name for the role, for example: `CauselyAccess` ###### 6. Assign policies to the role **For comprehensive access (recommended):** - [ReadOnlyAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/ReadOnlyAccess.html) **For granular access, assign these specific policies:** - [AmazonEC2ReadOnlyAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonEC2ReadOnlyAccess.html) - [AmazonECS_FullAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonECS_FullAccess.html) - [AmazonElastiCache ReadOnlyAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonElastiCacheReadOnlyAccess.html) - [AmazonMSK ReadOnlyAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonMSKReadOnlyAccess.html) - [AmazonRDS ReadOnlyAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonRDSReadOnlyAccess.html) - [AmazonSNS ReadOnlyAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonSNSReadOnlyAccess.html) - [AmazonSQS ReadOnlyAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonSQSReadOnlyAccess.html) - [AWSLambda_ReadOnlyAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AWSLambda_ReadOnlyAccess.html) - [AWSResourceGroupsReadOnlyAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AWSResourceGroupsReadOnlyAccess.html) - [ElasticLoadBalancingReadOnly](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/ElasticLoadBalancingReadOnly.html) ###### 7. Create Kubernetes secret Create the following Kubernetes secret in the same namespace as the Causely mediator: ```yaml title="aws-secret.yaml" apiVersion: v1 kind: Secret type: Opaque metadata: name: aws-secret namespace: causely stringData: AWS_REGION: region-code AWS_ROLE_ARN: arn:aws:iam::222222222222:role/CauselyAccess ``` ###### 8. Update Causely mediator configuration Update the Causely mediator with the following helm values ```yaml title="causely-values.yaml" mediator: serviceAccount: annotations: eks.amazonaws.com/role-arn: arn:aws:iam::111111111111:role/CauselyMediation scrapers: aws: enabled: true accounts: - arn: arn:aws:organizations::222222222222:account/o-pyxgi8opc5/123456789012 secretName: aws-secret ``` ##### Option 2: IAM User For simpler setups or development environments, you can use IAM user credentials. ###### 1. Create an IAM user Create an IAM user with the following policies: - [ReadOnlyAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/ReadOnlyAccess.html) Or for more granular access, assign the following predefined policies to the user: - [AmazonEC2ReadOnlyAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonEC2ReadOnlyAccess.html) - [AmazonECS_FullAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonECS_FullAccess.html) - [AmazonElastiCache ReadOnlyAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonElastiCacheReadOnlyAccess.html) - [AmazonMSK ReadOnlyAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonMSKReadOnlyAccess.html) - [AmazonRDS ReadOnlyAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonRDSReadOnlyAccess.html) - [AmazonSNS ReadOnlyAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonSNSReadOnlyAccess.html) - [AmazonSQS ReadOnlyAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonSQSReadOnlyAccess.html) - [AWSLambda_ReadOnlyAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AWSLambda_ReadOnlyAccess.html) - [AWSResourceGroupsReadOnlyAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AWSResourceGroupsReadOnlyAccess.html) - [ElasticLoadBalancingReadOnly](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/ElasticLoadBalancingReadOnly.html) ###### 2. Create Kubernetes secret Create the following Kubernetes secret in the same namespace as the Causely mediator: ```yaml title="aws-secret.yaml" apiVersion: v1 kind: Secret type: Opaque metadata: name: aws-secret namespace: causely stringData: AWS_REGION: region-code AWS_ACCESS_KEY_ID: YYY AWS_SECRET_ACCESS_KEY: ZZZ ``` ###### 3. Update Causely configuration Update the Causely mediator with the following helm values ```yaml title="causely-values.yaml" scrapers: aws: enabled: true accounts: - arn: arn:aws:organizations::222222222222:account/o-pyxgi8opc5/123456789012 secretName: aws-secret ``` #### Alternative: Enable Credentials Autodiscovery Causely supports credentials autodiscovery for simplified management: ```bash kubectl --namespace causely label secret aws-secret "causely.ai/scraper=AWS" ``` ### What Data is Collected Causely collects comprehensive metadata and performance information from your AWS infrastructure. Among others, Causely collects the following data: - **Application Load Balancer (ALB) / Network Load Balancer (NLB)** - Load balancer entities with DNS names and ARNs - Request metrics including total requests, error rates (4xx, 5xx) - Performance metrics including response times and target health - Target group mappings to backend services - Connection error tracking and timeout analysis - Authentication metrics and configuration validation - Network endpoint mapping to Kubernetes services - External DNS hostname integration for custom domains - **Amazon ECS** - Cluster entities with service mappings and task definitions - Service performance including CPU and memory utilization - Task health and container metrics - Resource capacity and utilization tracking - Service-to-workload relationship mapping - Compute resource allocation and usage - **AWS Lambda** - Function entities with ARNs and configurations - Invocation metrics and error rates - Performance data including duration and memory usage - Function URL mappings and network endpoints - Compute resource tracking for serverless workloads - **Amazon RDS** - Database instance entities with connection details - Performance metrics including CPU, memory, and I/O utilization - Connection usage and capacity tracking - Database engine information and version details - Storage metrics including free space and IOPS utilization - Slow query identification and analysis - Network endpoints for database connectivity - **Amazon MSK (Managed Streaming Kafka)** - Cluster entities with broker information - Broker performance including CPU, memory, and storage usage - Network endpoints for Kafka connectivity - Storage metrics and capacity utilization - Cluster health and broker status tracking - **Amazon ElastiCache** - ElasticCache Compute, Network and Cache metrics including connection usage and cache hits - **Amazon SNS and SQS** - SNS Topics and subscribed SQS queues - Number of messages available for retrieval from the queue - **Amazon EC2** - Virtual machine entities with instance details - Resource utilization including CPU, memory, and network - Instance type information and capacity planning - Performance metrics from CloudWatch - Instance state and health monitoring - Storage relationships to EBS volumes - **Amazon EBS** - Volume entities with capacity and performance characteristics - I/O performance including IOPS and throughput metrics - Storage utilization and capacity tracking - Volume attachment relationships to EC2 instances - Performance optimization recommendations - **CloudWatch Integration** - Real-time metrics collection from all supported services - Historical data analysis for trend identification - Custom metric support for application-specific monitoring - Automatic correlation between infrastructure metrics and service health - Threshold monitoring and anomaly detection --- ## Azure ### Overview Causely provides comprehensive native integration with Microsoft Azure to help you identify and resolve infrastructure and managed service reliability issues before they impact your users. Rather than stopping at surface-level Signals, Causely analyzes them in real time from your Azure environment to infer the underlying Diagnosis behind performance degradation and risk across your cloud services. By setting up the Azure integration, you will be able to: **Identify causes for reliability issues originating from your Azure infrastructure and services**, including: - [Service Congestion](/reference/root-causes/services#congested-1) and [Service Malfunction](/reference/root-causes/services#malfunction-1) - [Application Load Balancer Misconfiguration](/reference/root-causes/services#application-load-balancer) issues - [Authentication](/reference/root-causes/services#authentication-misconfiguration) and [Network Policy](/reference/root-causes/services#network-policy-misconfiguration) misconfigurations - [Memory Pressure](/reference/root-causes/infrastructure#memory-pressure) and [CPU Congested](/reference/root-causes/infrastructure#cpu-congested) issues - [Disk Pressure](/reference/root-causes/infrastructure#disk-pressure) and [I/O congestion](/reference/root-causes/infrastructure#disk-total-iops-congested) Observe Azure resources as first-class entities in the Causely Topology Graph, including their relationships to Kubernetes workloads, cloud services, and downstream dependencies Monitor performance and health metrics from Azure-native sources with automatic correlation to service impact and causal diagnoses ### Supported Azure Services Causely integrates with the following Azure services to provide comprehensive observability and causal inference: - Azure Cache for Redis - Azure Cache for Redis Enterprise - Azure Database for MySQL Flexible Server - Azure Database for PostgreSQL Flexible Server - Azure Managed Disks - Azure Key Vault - Azure Event Hubs - Azure Load Balancer - Azure Storage (Blob, Queue, File, Table) - Azure Virtual Machines (VMs) - Azure Virtual Machine Scale Sets (VMSS) ### Step 1: Create a Service Principal (SPN) and Grant Permissions Causely authenticates to Azure with a Service Principal (client ID and client secret) and only performs **read-only** calls against the Azure Resource Manager (ARM) and Azure Monitor metrics APIs. It never modifies resources, and it does not need any data-plane access (no storage keys, Key Vault secrets, database logins, or Redis access keys). #### Required permissions The Service Principal needs the following actions on each subscription you want to monitor, grouped by the scraper that uses them: | Scraper | Azure resources | Required actions | |---------|-----------------|------------------| | All | Subscription (connectivity check) | `Microsoft.Resources/subscriptions/read` | | All | Azure Monitor platform metrics | `Microsoft.Insights/metrics/read`, `Microsoft.Insights/metricDefinitions/read` | | `azure` | Managed Disks | `Microsoft.Compute/disks/read` | | `azure` | Virtual Machine Scale Sets and their VM instances | `Microsoft.Compute/virtualMachineScaleSets/read`, `Microsoft.Compute/virtualMachineScaleSets/virtualMachines/read` | | `azure` | Load Balancers (including SNAT port usage of the AKS `kubernetes` load balancer) | `Microsoft.Network/loadBalancers/read` | | `azure` | Storage Accounts | `Microsoft.Storage/storageAccounts/read` | | `azure` | Key Vaults | `Microsoft.KeyVault/vaults/read` | | `azurePostgres` | Azure Database for PostgreSQL Flexible Server | `Microsoft.DBforPostgreSQL/flexibleServers/read` | | `azureMysql` | Azure Database for MySQL Flexible Server | `Microsoft.DBforMySQL/flexibleServers/read` | | `azureRedis` | Azure Cache for Redis and Azure Cache for Redis Enterprise | `Microsoft.Cache/redis/read`, `Microsoft.Cache/redisEnterprise/read` | | `azureEventhub` | Event Hubs namespaces and event hubs | `Microsoft.EventHub/namespaces/read`, `Microsoft.EventHub/namespaces/eventhubs/read` | Assign the role at the **subscription** scope. The scrapers list resources across the whole subscription and validate access by reading the subscription itself, so a role assigned only on individual resource groups is not sufficient. You can grant these permissions in one of two ways. #### Option 1: Built-in Reader role (simplest) The built-in [Reader](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles/general#reader) role includes all of the actions above. Create the SPN and assign the role in a single command: ```bash az ad sp create-for-rbac \ --name causely-spn \ --role Reader \ --scopes /subscriptions/00000000-0000-0000-0000-000000000000 ``` #### Option 2: Custom least-privilege role If your security policy does not allow the broad Reader role, create a custom role containing only the actions Causely uses. 1. Save the following role definition as `causely-reader-role.json`, replacing the subscription ID in `AssignableScopes` (add one entry per subscription you want to monitor): ```json { "Name": "Causely Reader", "IsCustom": true, "Description": "Read-only access to the Azure resources and metrics used by Causely.", "Actions": [ "Microsoft.Resources/subscriptions/read", "Microsoft.Insights/metrics/read", "Microsoft.Insights/metricDefinitions/read", "Microsoft.Compute/disks/read", "Microsoft.Compute/virtualMachineScaleSets/read", "Microsoft.Compute/virtualMachineScaleSets/virtualMachines/read", "Microsoft.Network/loadBalancers/read", "Microsoft.Storage/storageAccounts/read", "Microsoft.KeyVault/vaults/read", "Microsoft.DBforPostgreSQL/flexibleServers/read", "Microsoft.DBforMySQL/flexibleServers/read", "Microsoft.Cache/redis/read", "Microsoft.Cache/redisEnterprise/read", "Microsoft.EventHub/namespaces/read", "Microsoft.EventHub/namespaces/eventhubs/read" ], "NotActions": [], "DataActions": [], "NotDataActions": [], "AssignableScopes": [ "/subscriptions/00000000-0000-0000-0000-000000000000" ] } ``` You can remove the actions for any services you do not plan to scrape, but keep `Microsoft.Resources/subscriptions/read` and the `Microsoft.Insights/*` actions. 2. Create the custom role: ```bash az role definition create --role-definition @causely-reader-role.json ``` 3. Create the SPN and assign the custom role: ```bash az ad sp create-for-rbac \ --name causely-spn \ --role "Causely Reader" \ --scopes /subscriptions/00000000-0000-0000-0000-000000000000 ``` :::tip[Using the Azure portal] You can do the same in the Azure portal: register an application under **Microsoft Entra ID → App registrations**, create a client secret under **Certificates & secrets**, and then go to **Subscriptions → _your subscription_ → Access control (IAM) → Add role assignment** to assign **Reader** (or your **Causely Reader** custom role, created under **Access control (IAM) → Add → Add custom role**) to the application. ::: #### Note the SPN credentials `az ad sp create-for-rbac` prints output similar to the following: ```json { "appId": "11111111-1111-1111-1111-111111111111", "displayName": "causely-spn", "password": "...", "tenant": "22222222-2222-2222-2222-222222222222" } ``` These map to the values Causely needs in Step 2: | `az` output | Causely Secret key | |-------------|--------------------| | `tenant` | `tenantId` | | `appId` | `clientId` | | `password` | `clientSecret` | | ID of the subscription in `--scopes` | `subscriptionId` | The client secret expires (after one year by default). Rotate it with `az ad sp credential reset --id ` and update the Kubernetes Secret before it expires, otherwise scraping stops. #### Monitoring multiple subscriptions Each Kubernetes Secret holds credentials for a single subscription. To monitor several subscriptions with the same SPN, assign the role on each one (for a custom role, also list each subscription in `AssignableScopes`): ```bash az role assignment create \ --assignee \ --role Reader \ --scope /subscriptions/ ``` Then create one Kubernetes Secret per subscription in Step 2, each with its own `subscriptionId`. #### Verify the role assignment ```bash az role assignment list --assignee --all --output table ``` ### Step 2: Create a Kubernetes Secret for the SPN Credentials After creating the SPN, save its credentials in a Kubernetes Secret. Replace the placeholder values with the SPN's `tenantId`, `clientId`, `clientSecret`, and `subscriptionId`. ```bash kubectl --namespace causely create secret generic spn-credentials \ --from-literal=tenantId="..." \ --from-literal=clientId="..." \ --from-literal=clientSecret="..." \ --from-literal=subscriptionId="..." ``` ### Step 3: Update Causely Configuration Once the Secret is created, update the Causely configuration to enable scraping for the new subscription. Below is an example configuration: ```yaml scrapers: azure: # Scrape VMs & Disks enabled: true subscriptions: - secretName: spn-credentials namespace: causely azurePostgres: # Scrape Postgresql Databases enabled: true subscriptions: - secretName: spn-credentials namespace: causely azureMysql: # Scrape Mysql Databases enabled: true subscriptions: - secretName: spn-credentials namespace: causely azureRedis: # Scrape Redis enabled: true subscriptions: - secretName: spn-credentials namespace: causely azureEventhub: # Scrape EventHub enabled: true subscriptions: - secretName: spn-credentials namespace: causely ``` ### Alternative: Enable Credentials Autodiscovery Causely also supports credentials autodiscovery. This feature allows you to add new scraping targets without updating the Causely configuration. Simply label the Kubernetes Secret to enable autodiscovery for the corresponding scraper. #### Example: Labeling Secrets for Autodiscovery ```bash # Azure VMs & Disks kubectl --namespace causely label secret spn-credentials "causely.ai/scraper=Azure" # Azure Postgresql kubectl --namespace causely label secret spn-credentials "causely.ai/scraper=AzurePostgres" # Azure MySQL kubectl --namespace causely label secret spn-credentials "causely.ai/scraper=AzureMysql" # Azure Redis kubectl --namespace causely label secret spn-credentials "causely.ai/scraper=AzureRedis" # Azure EventHub kubectl --namespace causely label secret spn-credentials "causely.ai/scraper=AzureEventHub" ``` With these steps, you can seamlessly integrate Causely with the Azure API and configure it to monitor your desired resources. --- ## Backstage Integration ### Overview Causely can ingest your [Backstage](https://backstage.io) software catalog and stitch Backstage **components** and **owners** (groups and users) to Kubernetes services. This enriches the topology with team ownership, component metadata (type, lifecycle, system), and enables causal analysis and remediation with clear ownership context. The integration: - Fetches catalog entities from the Backstage Catalog API (components, groups, users) - **Stitches Backstage components to Kubernetes services** by direct lookup from component-derived name and namespace (see below) - Creates or updates **ServiceOwner** entities from Backstage groups/users and links them to services - Applies Backstage-derived labels to services and owners for filtering and display ### How Backstage components are stitched to Kubernetes services Causely links Backstage **components** to Kubernetes **Services** by **direct lookup**: it reads a **service name** and optionally a **namespace** from the component, then looks up the Kubernetes Service with that name (and in that namespace when provided). When namespace is not provided, the service is resolved by name only and must be unique cluster-wide. Ownership and component metadata are then applied to the matched service. **Defaults:** Causely reads the service name from the **`backstage.io/kubernetes-id`** annotation and the namespace from **`backstage.io/kubernetes-namespace`** on the Backstage component. **Owner:** Owner is taken from the Backstage component only, for example **`spec.owner`** or **`backstage.io/owner`** on the component, depending on your `label_map`. If no owner can be resolved from the component, the owner link is not created for that service. If your Backstage components use different fields for the service name or namespace, you can override the mapping in Helm values so Causely reads the right fields and resolves to the correct Kubernetes service. ### Alerts that carry only a Backstage component label Alerts that carry only a Backstage component label are mapped to the related Kubernetes service on a best-effort basis. This does not require the Backstage integration. Causely only reads the label on the alert. For how Causely maps alerts to entities, see [Alert Ingestion and Mapping](/configuration/alert-ingestion). ### Configuration #### Add data source from the UI You can add Backstage as a data source from the Causely UI. Go to [Integrations](https://portal.causely.app/integrations), add the Backstage integration, and select the cluster to which the configuration will be pushed. #### Basic setup (ops as code) If you prefer to manage configuration as code, enable the Backstage scraper and point it at a Kubernetes Secret that holds your Backstage API endpoint and token: ```yaml title="causely-values.yaml" scrapers: backstage: enabled: true instances: - secretName: 'backstage-credentials' namespace: 'causely' ``` Create a Kubernetes Secret with the Backstage URL and (optionally) token and TLS options, and label it so the scraper can discover it: ```bash kubectl create secret generic backstage-credentials \ --namespace causely \ --from-literal=endpoint='https://your-backstage.example.com' \ --from-literal=token='your-backstage-api-token' kubectl label secret backstage-credentials \ --namespace causely \ causely.ai/scraper=Backstage ``` The secret must have the label **`causely.ai/scraper: Backstage`** so the Backstage scraper can discover it. Secret keys: - **`endpoint`** (required): Backstage base URL, for example `https://backstage.example.com`, without a trailing slash. - **`token`** (optional): Bearer token for Backstage API authentication. - **`insecure_skip_verify`** (optional): Set to `true` to skip TLS verification, for example for dev; not recommended for production. #### Customize the mapping to your Backstage catalog (optional) If your Backstage components store the Kubernetes service name or namespace in different fields than the defaults, you can override the mapping in Helm so Causely resolves to the right Kubernetes service. Under each Backstage instance, set **`label_map`** in your values to specify which component fields to use for: - **`component_kubernetes_id`**: where to read the **name of the Kubernetes Service** from the Backstage component - **`component_kubernetes_namespace`**: where to read the **namespace of the Kubernetes Service** from the Backstage component - **`component_owner`**: configurable fields on the Backstage component used to resolve the owner (if none can be resolved, the owner link is skipped for that service) Example with the default mappings and comments: ```yaml title="causely-values.yaml" scrapers: backstage: enabled: true instances: - secretName: 'backstage-credentials' namespace: 'causely' # Override which Backstage component fields are used to resolve the K8s service (name + namespace) label_map: # Backstage field(s) used for the Kubernetes Service name component_kubernetes_id: - "annotations.backstage.io/kubernetes-id" # Backstage field(s) used for the Kubernetes Service namespace component_kubernetes_namespace: - "annotations.backstage.io/kubernetes-namespace" # Backstage field(s) used to resolve the component owner (no owner link if unresolved) component_owner: - "spec.owner" - "annotations.backstage.io/owner" ``` This way the name and namespace used for lookup match how your Backstage components are defined. ### What data is collected - **ServiceOwner entities**: Created or updated from Backstage groups and users, with display name and email when available. - **Service-owner links**: Each stitched Kubernetes service is linked to its owner when that owner can be resolved from the Backstage component. - **Labels on services**: Backstage scraper name, team (owner display name), component name/type/lifecycle/system, owner ref/kind/source, and the Kubernetes ID/namespace used for stitching. - **Labels on owners**: Owner ref, kind, display name, email, and scraper identification. Only Backstage catalog entities of kind **component** are considered for stitching to Kubernetes services; **group** and **user** entities are used to build the owner catalog. --- ## Checkly ### Overview Causely provides native integration with [Checkly](https://www.checklyhq.com/) to help you identify and resolve external monitoring and synthetic testing issues before they impact your users. Instead of just monitoring Signals, Causely analyzes them in real time from Checkly's global monitoring network to surface the actual Diagnosis behind external availability and performance problems. This integration helps you identify the following causes for reliability issues, among others: - [Service Malfunction](/reference/root-causes/services#malfunction-1) and [service Congestion](/reference/root-causes/services#congested-1) - [Faulty Error Handling in HTTP Path](/reference/root-causes/datapipeline#faulty-error-handling-in-http-path) and [Slow Execution in HTTP Path Handler](/reference/root-causes/datapipeline#slow-execution-in-http-path-handler) - [Code Change Regression: CPU Congestion](/reference/root-causes/release#code-change-regression-cpu-congestion) and [Code Change Regression: Memory Failure](/reference/root-causes/release#code-change-regression-memory-failure) - [Network endpoint issues](/reference/root-causes/infrastructure#network-endpoint) The integration supports Checkly's comprehensive monitoring platform including API checks, browser checks, and global monitoring locations. ### Setup Guide #### Step 1: Create an API Key in Checkly Create an [API key](https://app.checklyhq.com/settings/user/api-keys) for your Checkly account with the following permissions: - **Read access to checks**: Required to access check configurations and metadata - **Read access to check results**: Required to monitor check execution results and performance - **Read access to locations**: Required to access monitoring location information - **Read access to alerts**: Required to monitor alert configurations and status For more information on [Checkly API authentication](https://developers.checklyhq.com/reference/authentication) #### Step 2: Configure Monitoring Locations Ensure your Checkly checks are configured to run from multiple monitoring locations to provide comprehensive coverage: - **Public locations**: Checkly's global network of monitoring locations - **Private locations**: Your own monitoring infrastructure for internal services - **Geographic distribution**: Multiple regions for comprehensive coverage #### Step 3: Update Causely Configuration Once the API key is created, update the Causely configuration to enable scraping for Checkly checks. Below is an example configuration: ```yaml scrapers: checkly: enabled: true accounts: - account_id: 'aaaabbbb-cccc-dddd-eeee-ffffffffffff' token: 'xz_1234' ``` #### Alternative: Enable Credentials Autodiscovery Causely also supports credentials autodiscovery. This feature allows you to add new scraping targets without updating the Causely configuration. Label the Kubernetes secret to enable autodiscovery for the corresponding scraper. ```bash kubectl --namespace causely label secret checkly-credentials "causely.ai/scraper=Checkly" ``` ### What Data is Collected The Checkly scraper collects comprehensive synthetic monitoring and testing data from your Checkly checks, including: - **Check information**: ID, name, type (API checks), frequency, and locations - **Check settings**: Alert configurations, retry strategies, and timeout settings - **Request details**: HTTP method, URL, headers, body, and query parameters - **Assertions and validations**: Response validation rules and performance thresholds - **Environment variables**: Check-specific configuration and test data - **Response times**: Individual check response times and performance data - **Success/failure rates**: Track successful vs failed check executions - **Error information**: Detailed error messages and failure reasons - **Location-based results**: Performance data per monitoring location - **Recent check history**: Last hour of check results for trend analysis - **Geographic distribution**: Different monitoring locations and regions - **Location metadata**: Region names and availability zones - **Multi-location testing**: Results from various geographic points - **Public vs private locations**: Coverage from Checkly's network and your infrastructure - **Alert symptoms**: `RequestErrorRate_High` and `RequestDuration_High` for proactive issue detection - **Service mapping**: Links Checkly checks to target services - **Workload relationships**: Associates monitoring locations with workloads - **Network endpoint tracking**: Maps checks to specific service endpoints - **Access data collection**: Creates network dependency mappings for causal analysis - **HTTP response details**: Status codes, response headers, and body content - **Request timing phases**: Detailed breakdown of request/response timing - **Assertion results**: Validation rule pass/fail status and actual values - **Error categorization**: Network errors, timeout issues, and validation failures - **Performance thresholds**: Degraded and maximum response time monitoring - **Performance trend analysis**: Identifies gradual performance degradation - **Threshold-based alerting**: Proactive alerts based on configurable thresholds - **Location-specific performance**: Identifies regional performance differences - **Global availability monitoring**: Tracks service availability across regions - **Regional outage detection**: Identifies location-specific service issues - **Performance correlation**: Links geographic performance to infrastructure issues - **Current check status**: Real-time monitoring of check health - **Status transitions**: Tracks changes from healthy to degraded/failed states - **Alert state management**: Updates external alerts based on current performance - **Trend analysis**: Identifies patterns in check performance over time --- ## ClickHouse ### Overview Causely provides native integration with [ClickHouse](https://clickhouse.com/) to help you identify and resolve database issues before they impact your users. Instead of just monitoring Signals, Causely analyzes them in real time to surface the underlying Diagnosis driving database issues. By setting up the ClickHouse integration, you will be able to do the following: - Identify causes for reliability issues originating from your ClickHouse database, including: - [Table malfunction](/reference/root-causes/datapipeline#table-access-failure-in-database) and [mutation-based table locks](/reference/root-causes/datapipeline#contention-on-database-table-locks) - [Slow database queries](/reference/root-causes/applications#slow-database-queries) - [Connection exhaustion](/reference/root-causes/applications#database-connection-pool-saturated) - Observe the database as an entity in the Topology Graph, including its relationships to other entities on the service map, infrastructure stack, and dataflow map. - Get insights into the slowest queries over a rolling 12-hour window, and troubleshoot them with our [MCP Server](/agent-integration/mcp-server/) directly from your favorite agent. The integration supports both self-hosted ClickHouse instances and cloud-managed deployments. ### Setup Guide #### Step 1: Create a user Create a dedicated user in your ClickHouse instance and grant it access to the system tables that Causely requires: ```sql CREATE USER causely_user IDENTIFIED BY 'your-password'; GRANT SELECT ON system.tables TO causely_user; GRANT SELECT ON system.columns TO causely_user; GRANT SELECT ON system.databases TO causely_user; GRANT SELECT ON system.query_log TO causely_user; GRANT SELECT ON system.mutations TO causely_user; ``` **System-table grants alone are not sufficient.** The ClickHouse user must also be able to see the actual application tables in the databases Causely should discover. Without `SHOW`/`SELECT` on those databases, Causely cannot build table entities or relate them to your services. Grant database-wide visibility (choose the scope that fits your security model): ```sql GRANT SHOW DATABASES ON *.* TO causely_user; GRANT SELECT ON *.* TO causely_user; ``` For tighter scope, grant `SELECT` only on the application databases Causely should monitor: ```sql GRANT SHOW DATABASES ON *.* TO causely_user; GRANT SELECT ON .* TO causely_user; ``` Or, if you want multiple examples: ```sql GRANT SHOW DATABASES ON *.* TO causely_user; GRANT SELECT ON db1.* TO causely_user; GRANT SELECT ON db2.* TO causely_user; ``` How this maps to discovery: - **`system.databases`** controls whether Causely can discover database names. - **`system.tables`** controls whether Causely can discover tables inside those databases. If the user can see databases but not non-system tables, Causely will discover databases but create no useful entities for your application data. The integration reads from the following system tables: | Table | Purpose | | --- | --- | | `system.tables` | Table names, row counts, and sizes | | `system.columns` | Column definitions and schema information | | `system.databases` | Database discovery | | `system.query_log` | Slow query analysis (top 10 by total execution time, rolling 12-hour window) | | `system.mutations` | Active mutation detection for lock monitoring | #### Step 2: Create a Kubernetes secret for the user Create a Kubernetes secret with the ClickHouse connection details. The secret supports two protocols: - **Native** (default): binary protocol on port `9000` - **HTTP**: HTTP/HTTPS protocol on port `8123` / `8443` ##### Option 1: Single Database Configuration ```bash kubectl create secret generic \ --namespace causely clickhouse-credentials \ --from-literal=username="causely_user" \ --from-literal=password='...' \ --from-literal=host="..." \ --from-literal=port="9000" \ --from-literal=database="..." \ --from-literal=protocol="native" \ --from-literal=secure="false" ``` To connect over HTTP instead of the native protocol: ```bash kubectl create secret generic \ --namespace causely clickhouse-credentials \ --from-literal=username="causely_user" \ --from-literal=password='...' \ --from-literal=host="..." \ --from-literal=port="8123" \ --from-literal=database="..." \ --from-literal=protocol="http" \ --from-literal=secure="false" ``` ##### Option 2: Multiple Databases Configuration To monitor multiple databases within the same ClickHouse instance, specify them as a comma-separated list using the `databases` field: ```bash kubectl create secret generic \ --namespace causely clickhouse-credentials-multidb \ --from-literal=username="causely_user" \ --from-literal=password='...' \ --from-literal=host="..." \ --from-literal=port="9000" \ --from-literal=databases="database1,database2,database3" \ --from-literal=protocol="native" \ --from-literal=secure="false" ``` Alternatively, use a YAML manifest: ```yaml apiVersion: v1 kind: Secret metadata: name: clickhouse-credentials-multidb namespace: causely type: Opaque stringData: username: 'causely_user' password: '...' host: '...' port: '9000' databases: 'database1,database2,database3' protocol: 'native' secure: 'false' ``` **Note**: Use either the `database` field for a single database or the `databases` field for multiple databases. Do not use both in the same secret. ##### Option 3: Database Auto-Discovery Causely can automatically discover all databases on a ClickHouse server. Add `auto_discovery: "true"` to the secret: ```bash kubectl create secret generic \ --namespace causely clickhouse-credentials \ --from-literal=username="causely_user" \ --from-literal=password='...' \ --from-literal=host="..." \ --from-literal=port="9000" \ --from-literal=protocol="native" \ --from-literal=secure="false" \ --from-literal=auto_discovery="true" ``` When auto-discovery is enabled, Causely queries `system.databases` (excluding `system`, `INFORMATION_SCHEMA`, and `information_schema`) and starts a scraper for each discovered database. Discovery runs periodically to pick up newly created databases. For external or VM-hosted ClickHouse, **use a stable DNS hostname whenever possible**. Raw IPs may connect successfully but can break service and entity resolution in Causely. The `host` value should align with how Causely discovers infrastructure (for example, the FQDN from the Kubernetes API or your cloud provider's API). If connectivity must use an IP address, set `host` to the working IP and set `host_overwrite` to the stable DNS hostname that identifies that ClickHouse instance: ```bash --from-literal=host="10.202.225.23" --from-literal=host_overwrite="clickhouse.internal.company.net" ``` If you are connecting through a proxy, set `host` to the proxy address and `host_overwrite` to the actual ClickHouse instance hostname: ```bash --from-literal=host="my-proxy.example.com" --from-literal=host_overwrite="my-clickhouse.example.com" ``` ##### ClickHouse Cloud Configuration ClickHouse Cloud only accepts TLS connections, on different ports than a typical self-hosted instance: - **HTTPS**: port `8443` with `protocol="http"` and `secure="true"` - **Native over TLS**: port `9440` with `protocol="native"` and `secure="true"` Plaintext connections (the defaults used elsewhere on this page: port `9000`/`8123` with `secure="false"`) are not exposed by ClickHouse Cloud at all, so a secret copied from a self-hosted example will fail to connect. ```bash kubectl create secret generic \ --namespace causely clickhouse-credentials \ --from-literal=username="causely_user" \ --from-literal=password='...' \ --from-literal=host=".clickhouse.cloud" \ --from-literal=port="8443" \ --from-literal=database="..." \ --from-literal=protocol="http" \ --from-literal=secure="true" \ --from-literal=auto_discovery="true" kubectl --namespace causely label secret clickhouse-credentials "causely.ai/scraper=ClickHouse" ``` **`protocol` only accepts the literal values `native` or `http`.** `https` is *not* a valid value: if set, the scraper silently falls back to the native protocol instead of erroring, which then fails against Cloud's HTTPS-only port with a connection reset. TLS is controlled separately via `secure="true"`, not by the protocol name. ##### Secret field reference | Field | Required | Default | Description | | --- | --- | --- | --- | | `host` | Yes | | Hostname or IP used to connect to ClickHouse. Prefer a stable DNS hostname for normal setups; if you must use an IP or proxy here, set `host_overwrite` to the canonical hostname for topology and entity resolution. | | `username` | Yes | | ClickHouse user name | | `password` | Yes | | ClickHouse user password | | `database` | Yes* | | Single database to monitor | | `databases` | Yes* | | Comma-separated list of databases to monitor | | `port` | No | `9000` (native) / `8123` (HTTP) | ClickHouse port. ClickHouse Cloud uses `8443` (HTTP) or `9440` (native TLS) instead: see [ClickHouse Cloud Configuration](#clickhouse-cloud-configuration). | | `protocol` | No | `native` | Connection protocol. Only `native` or `http` are valid: `https` is not recognized and silently falls back to `native`. | | `secure` | No | `false` | Enable TLS (`true` or `false`). Must be `true` for ClickHouse Cloud. | | `host_overwrite` | No | | Override the host used for topology and entity resolution. Use when the connection `host` is an IP or proxy address, but ClickHouse should be identified by a stable DNS hostname. | | `port_overwrite` | No | | Override the port used for entity resolution | | `auto_discovery` | No | `false` | Automatically discover all databases | *Either `database` or `databases` must be set, unless `auto_discovery` is enabled. #### Step 3: Update Causely Configuration Once the secret is created, update the Causely configuration to enable scraping for the new instance: ```yaml scrapers: clickhouse: enabled: true instances: - secretName: clickhouse-credentials namespace: causely ``` #### Alternative: Enable Credentials Autodiscovery Causely also supports credentials autodiscovery, which lets you add new scraping targets without modifying the Causely configuration. Label the Kubernetes secret to enable autodiscovery: ```bash kubectl --namespace causely label secret clickhouse-credentials "causely.ai/scraper=ClickHouse" ``` ### Verify Your Configuration After completing the setup, run these queries against your ClickHouse instance to verify that the Causely user has the required access. #### Troubleshooting Use these symptoms to narrow down configuration issues: - If the UI only shows a scraper path ending in `/_discovery`, database auto-discovery is running but no per-database scrapers were created (often because no **application databases** were found, or discovery could not proceed as expected). - If the UI shows per-database scrapers with a message like **configured database has no eligible tables**, the user can see the **configured databases** (or **target databases**) but they have no eligible tables for the current scraper logic (see [Eligible table types](#eligible-table-types) below). - If logs show **failed to resolve hostname** or **service mapping** errors, set `host_overwrite` to a stable DNS hostname and avoid relying on a raw IP alone for identity. - If nothing shows up at all and mediator logs show no mention of the scraper, confirm the secret is labeled `causely.ai/scraper=ClickHouse` (required for autodiscovery to notice the secret) or that a matching `instances` entry exists in the Causely configuration. - If logs show a ping/connection failure against ClickHouse Cloud (for example, `read: connection reset by peer` on an `https://` URL), double check `protocol` is exactly `http` or `native` (not `https`) and that `port`/`secure` match one of the combinations in [ClickHouse Cloud Configuration](#clickhouse-cloud-configuration). - **A scraper does not automatically retry after a failed initialization.** If you fix the secret after a failed connection attempt, you must also restart the mediator (`kubectl rollout restart deployment/mediator -n causely`) for the fix to take effect. #### Quick Access Check ```sql SELECT (SELECT count() FROM system.tables LIMIT 1) > 0 AS tables_ok, (SELECT count() FROM system.columns LIMIT 1) > 0 AS columns_ok, (SELECT count() FROM system.databases LIMIT 1) > 0 AS databases_ok, (SELECT count() FROM system.query_log LIMIT 1) >= 0 AS query_log_ok, (SELECT count() FROM system.mutations LIMIT 1) >= 0 AS mutations_ok; ``` All columns should return `1` (true). #### Detailed Checks ##### 1. System tables access ```sql -- Each of these should return a result without error SELECT 1 FROM system.tables LIMIT 1; SELECT 1 FROM system.columns LIMIT 1; SELECT 1 FROM system.databases LIMIT 1; SELECT 1 FROM system.query_log LIMIT 1; SELECT 1 FROM system.mutations LIMIT 1; ``` If any query fails with an access denied error, grant the missing privilege to your Causely user: ```sql -- Run as admin GRANT SELECT ON system. TO causely_user; ``` ##### 2. Test slow query collection Run this query to confirm Causely can collect slow query data: ```sql SELECT normalized_query_hash, count() AS calls, sum(query_duration_ms) AS total_exec_time_ms FROM system.query_log WHERE type = 'QueryFinish' AND event_time >= now() - toIntervalHour(12) GROUP BY normalized_query_hash ORDER BY total_exec_time_ms DESC LIMIT 5; ``` This should return results without error. An empty result set is normal on a freshly configured instance, entries will appear as queries run. ##### 3. Database discovery {#database-discovery} Confirm application databases are visible to the Causely user (same filter Causely uses for auto-discovery): ```sql SELECT name FROM system.databases WHERE name NOT IN ('system', 'INFORMATION_SCHEMA', 'information_schema') ORDER BY name; ``` If this returns no application databases, auto-discovery will only create the `_discovery` scraper and no per-database scrapers. ##### 4. Table visibility {#table-visibility} Confirm the Causely user can see non-system tables: ```sql SELECT database, name, engine FROM system.tables ORDER BY database, name LIMIT 100; ``` This result set must include **non-**`system` tables. If it only returns `system.*` tables, Causely will not discover application entities. ##### 5. Eligible table types {#eligible-table-types} Replace `` with a **target database** from your configuration (or one returned by the [Database discovery](#database-discovery) query above). Causely only creates table entities for engines it treats as eligible: ```sql SELECT database, name, engine FROM system.tables WHERE database = '' AND engine NOT IN ('View', 'MaterializedView', 'Dictionary') ORDER BY name; ``` The current scraper **excludes** `View`, `MaterializedView`, and `Dictionary`. If this query returns zero rows, Causely reports that the configured database has no eligible tables and creates no ClickHouse table entities for that database. :::tip[Success] Your ClickHouse instance is correctly configured for Causely when **all** of the following are true: 1. System table checks pass (`system.tables`, `system.columns`, `system.databases`, `system.query_log`, `system.mutations`). 2. Application databases appear in `system.databases` for the Causely user (see [Database discovery](#database-discovery)). 3. Application (non-system) tables appear in `system.tables` for the Causely user (see [Table visibility](#table-visibility)). 4. At least one configured database contains at least one eligible table after excluding `View`, `MaterializedView`, and `Dictionary` (see [Eligible table types](#eligible-table-types)). ::: #### Setup Checklist - [ ] `causely_user` created in ClickHouse - [ ] `SELECT` granted on `system.tables`, `system.columns`, `system.databases`, `system.query_log`, `system.mutations` - [ ] `SHOW DATABASES` granted (for example `GRANT SHOW DATABASES ON *.*`) - [ ] `SELECT` granted on the application databases or tables Causely should discover (broad `*.*` or scoped per database) - [ ] Kubernetes secret created with correct `host`, `username`, `password`, `database`/`databases`, `protocol`, and `port` - [ ] For ClickHouse Cloud: `protocol` is exactly `http` or `native` (never `https`), `secure` is `"true"`, and `port` is `8443` or `9440` - [ ] If using an IP address or proxy for connectivity, `host_overwrite` is set to a stable DNS hostname for topology and entity resolution - [ ] Causely configuration updated, or secret labeled `causely.ai/scraper=ClickHouse` for autodiscovery - [ ] If a scraper previously failed to initialize, the mediator has been restarted after fixing the secret - [ ] Verification queries succeed without access errors - [ ] `system.tables` returns non-system tables for the Causely user - [ ] At least one configured database contains eligible tables after excluding `View`, `MaterializedView`, and `Dictionary` ### What Data is Collected The ClickHouse scraper collects comprehensive metadata and performance information from your ClickHouse databases, including: - **Database entities** with names and relationships to hosting services - **Service-to-database mappings** (which service provides which database) - **Connection details** including host, port, and protocol configuration - **Table information** for eligible tables discovered from `system.tables` (names, row counts, and sizes). The current scraper excludes `View`, `MaterializedView`, and `Dictionary` engines. - **Complete table schemas** for those eligible tables from `system.columns`, including column definitions, data types, default expressions, and comments - **Slow query analysis** using `system.query_log`: top 10 queries by total execution time over a rolling 12-hour window, including call counts, total and average execution time, and rows read - **Mutation lock monitoring** using `system.mutations`: active mutations are tracked as exclusive locks to detect contention on tables --- ## Confluent ### Overview Causely provides native integration with [Confluent Cloud](https://docs.confluent.io/cloud/current/overview.html) to help you identify and resolve streaming platform performance issues before they impact your users. Instead of just monitoring Signals, Causely analyzes them in real time from Confluent's Telemetry API to surface the actual Diagnosis behind streaming problems. This integration helps you identify the following causes for reliability issues, among others: - [Slow consumer](/reference/root-causes/applications#slow-consumer) behavior causing message processing bottlenecks - [Memory exhaustion](/reference/root-causes/applications#memory-exhaustion) from unbounded message backlogs - [Service congestion](/reference/root-causes/services#congested) affecting Kafka clusters - [Service malfunction](/reference/root-causes/services#malfunction) in streaming platform components The integration supports Confluent Cloud environments and clusters with comprehensive monitoring of Kafka topics, consumer groups, and cluster health metrics. ### Setup Guide #### Step 1: Create API Key and Secret Create an API key and secret for your Confluent Cloud account with the necessary permissions: 1. Log in to the [Confluent Cloud Console](https://confluent.cloud/) 2. Navigate to **API Keys** in the Administration section 3. Click **Create key** 4. Select **Cloud resource management** as the scope 5. Choose **My account** as the owner 6. Generate and securely store the API key and secret **Required Permissions:** - Read access to clusters, topics, and metrics - Access to Telemetry API for metrics collection #### Step 2: Create a Kubernetes Secret Create a Kubernetes secret containing your Confluent Cloud credentials: ```bash kubectl create secret generic \ --namespace causely confluent-secret \ --from-literal=CONFLUENT_CLOUD_KEY="" \ --from-literal=CONFLUENT_CLOUD_SECRET="" ``` Alternatively, you can create the secret using a YAML manifest: ```yaml apiVersion: v1 kind: Secret metadata: name: confluent-secret namespace: causely type: Opaque stringData: CONFLUENT_CLOUD_KEY: '' CONFLUENT_CLOUD_SECRET: '' ``` #### Step 3: Update Causely Configuration Enable the Confluent scraper in your Causely configuration by adding the following to your `values.yaml` file: ##### Basic Configuration ```yaml scrapers: confluent: enabled: true accounts: - secretName: confluent-secret environments: - dev - prod clusters: - dev-cluster - prod-cluster ``` #### Alternative: Enable Credentials Autodiscovery Causely also supports credentials autodiscovery. This feature allows you to add new scraping targets without updating the Causely configuration. Label the Kubernetes secret to enable autodiscovery for the corresponding scraper. ```bash kubectl --namespace causely label secret confluent-secret "causely.ai/scraper=Confluent" ``` ### What Data is Collected The Confluent scraper collects comprehensive metadata and performance information from your Confluent Cloud deployment, including: - **Cluster entities** with display names, environment labels, and Kafka bootstrap endpoints (port 9092) - **Topic entities** with cluster relationships and labels (Environment, ClusterId, ClusterName, Topic) - **Consumer lag metrics** from the `io.confluent.kafka.server/consumer_lag_offsets` telemetry API - **TopicAccess entity updates** with real-time lag attributes for existing consumer group mappings - **Environment and cluster discovery** based on configuration filters - **Network endpoint mapping** for Kafka bootstrap server connectivity --- ## Datadog Causely integrates with [Datadog](https://www.datadoghq.com/) in three ways, which you can use together: - [Discover infrastructure and application services from the Datadog API](#discover-infrastructure-and-application-services-from-the-datadog-api), without running anything in the monitored clusters. - [Receive Datadog APM traces through dual shipping](#enable-dual-shipping-for-datadog-apm). - [Use Datadog monitors as Signals](#enabling-datadog-watchdog-monitors-as-a-data-source). The first and last use the same Datadog credentials and configuration. ### Discover infrastructure and application services from the Datadog API Causely reads your Datadog account through the Datadog API to build its topology, the same way it does for [Dynatrace](./dynatrace.mdx) and [Observe](./observe.mdx). The Causely mediator doesn't need to run in, or have access to, the clusters that Datadog monitors. #### Infrastructure - **Kubernetes clusters:** every cluster that reports to Datadog is rebuilt from its Kubernetes state metrics, with the same entities as the native [Kubernetes integration](./kubernetes.mdx): - Nodes, with their capacity and conditions. Nodes on AWS and GCP are matched to the virtual machines that the [AWS](./aws.mdx) and [GCP](./gcp.mdx) integrations discover. - Pods, with their status, readiness, and eviction causes, and their containers with resource requests, limits, and status. - Deployments, StatefulSets, DaemonSets, ReplicaSets, Jobs, and CronJobs, with compute specs for their containers. - Kubernetes services, namespaces, and persistent volumes and claims. Persistent volumes are linked to the pods that mount them. - **Clusters that a Causely agent already monitors** aren't rebuilt from Datadog. Causely keeps the agent's view and adds Datadog's labels to its pods. - **Hosts:** Datadog hosts are matched to the virtual machines that the AWS and GCP integrations discover. Containers that run outside Kubernetes are linked to their hosts. #### Application services When your applications send traces to Datadog APM, Causely discovers your application services and how they communicate: - **Services:** each Datadog APM service is linked to the Kubernetes service of the pods that emitted its spans. A service without a matching Kubernetes service gets its own service in Causely. - **Operations:** HTTP routes and RPC methods, with their request and error counts. Health check endpoints are skipped by default. - **Dependencies:** calls between services become access edges, resolved to the Kubernetes services of the cluster they run in. Calls to databases add the database to the service that hosts it, and messages sent to or read from a named topic or queue, such as a Kafka topic, become topic accesses. - **Metrics:** request rate, error rate, and latency for each service. All APM environments are discovered unless you limit them with `envs`. Causely refreshes this topology every 5 minutes by default. #### Prerequisites - The Datadog Agent with the Kubernetes State Core check (`kubernetes_state_core`) on each cluster you want Causely to discover. - A Datadog API key and an application key with read access to hosts, containers, metrics, APM, and events. - Optional: Kubernetes event collection in the Datadog Agent. Causely uses the events for pod eviction causes, and to link persistent volumes to their claims. - Optional, for persistent volumes on Amazon EBS: the Datadog AWS integration. The volume tags that the EBS CSI driver sets identify which claim each volume belongs to. #### Configuration Topology discovery uses the same instances and credentials as the [Datadog monitors integration](#enabling-datadog-watchdog-monitors-as-a-data-source), and is on by default. Add the optional `site` key to the secret if your Datadog account isn't on `datadoghq.com`, for example `datadoghq.eu` or `us5.datadoghq.com`. To limit discovery to some APM environments, list them in `scrapers.datadog.apm.envs`. Set `scrapers.datadog.apm.service_topology: false` to discover only the infrastructure, for example when your traces reach Causely through [dual shipping](#enable-dual-shipping-for-datadog-apm) or OpenTelemetry, so services and calls aren't discovered twice. ### Enable Dual Shipping for Datadog APM Causely can leverage [Datadog APM instrumentation with dual shipping](https://docs.datadoghq.com/agent/configuration/dual-shipping/) to discover and monitor service dependencies. ### Option 1: Enable via Helm Deployment To enable dual shipping when deploying the Datadog Agent using **Helm**: 1. Add the following configuration to your `values.yaml`: ```yaml agents: useConfigMap: true customAgentConfig: apm_config: additional_endpoints: 'http://mediator.causely:8126': - 'datadog-receiver' ``` 2. Upgrade or install the Datadog Helm chart: ```bash helm upgrade --install datadog datadog/datadog -f ./values.yaml ``` ### Option 2: Enable via Datadog Operator If you're managing the Datadog Agent using the [Datadog Operator](https://docs.datadoghq.com/containers/datadog_operator/), modify the `DatadogAgent` custom resource as follows: ```yaml apiVersion: datadoghq.com/v2alpha1 kind: DatadogAgent metadata: name: datadog spec: override: nodeAgent: env: - name: DD_APM_ADDITIONAL_ENDPOINTS value: '{"http://mediator.causely:8126": ["datadog-receiver"]}' ``` ### Adding Host Identity Tags for EC2-Based Datadog APM Traces When applications run outside Kubernetes (such as on EC2 instances), Datadog APM traces may not include hostname or other identifying metadata by default. Without this information, Causely cannot associate incoming traces with the EC2 instances discovered from AWS. To ensure Causely can correctly match traces to EC2 hosts, add a unique identifying tag to the application's Datadog configuration: ```bash DD_TAGS="pm-name:" ``` Set this environment variable in the same location where Datadog APM configuration is applied, such as: - ECS task definitions - Docker environment variables - systemd unit files - CI/CD deployment configurations **Example (Docker):** ```bash docker run \ -e DD_AGENT_HOST="" \ -e DD_TAGS="pm-name:my-ec2-hostname" \ my-app:latest ``` Causely uses this tag to stitch Datadog APM traces to the corresponding EC2 instance in your environment model. For more information on Datadog tagging, refer to [Datadog's documentation](https://docs.datadog.com/getting_started/tagging/). ### Enabling Datadog Watchdog Monitors as a Data Source Causely can also leverage [Datadog monitors](https://docs.datadoghq.com/monitors/) for Postgres, Redis, and other integrations to be used as input to Casual Reasoning. To enable Datadog as a data source in Causely, add the following configuration to your `values.yaml`: ```yaml scrapers: datadog: enabled: true instances: - secretName: datadog-credentials # namespace: your-namespace # optional; defaults to the pod namespace # event_tag_filters: [] # optional # monitor_mapping: [] # optional ``` #### Alternative: Enable Credentials Autodiscovery Causely supports credentials autodiscovery so you can add new accounts without editing config. Label the Kubernetes secret to enable autodiscovery for the Datadog scraper: ```bash kubectl --namespace causely label secret datadog-credentials "causely.ai/scraper=Datadog" ``` For the secret, you can use the following example: ```yaml apiVersion: v1 kind: Secret metadata: name: datadog-credentials type: Opaque stringData: org: 'YourOrg' apiKey: '' appKey: '' # site: 'datadoghq.eu' # optional; used by topology discovery, defaults to datadoghq.com ``` The optional event_tag_filters is used to filter triggered Datadog events, for example: ```yaml event_tag_filters: - 'pagerduty' - 'integration:postgres' - 'integration:redisdb' ``` The optional monitor_mapping is used to map Datadog monitors to Causely entities, for example: ```yaml monitor_mapping: - entityType: ApplicationLoadBalancer createIfMissing: true attributes: service_name: label: ['hostname'] isIdentifier: true resources: - monitorId: 123267804 attribute: RequestsTotal ``` Without the monitor_mapping, Causely will try to automatically map the latency and error rate monitors to the Causely Service corresponding Signals. For example, the following event will be mapped automatically to the high error rate Signal of the `causely-analysis` service: ```json { "id": "AwAAAZiZU7KAMHtQKwAAABhBWmlaVlJXNUFBRFhWU3hKanRwb1ZyYl8AAAAkMTE5ODk5ZGMtMjgyYS00YWJmLWFjYzUtNmQ4NjkyZjBlM2FlAAEjVg", "type": "event", "attributes": { "attributes": { "aggregation_key": "2b57d1af567993ace06eebaf9dc0e669", "evt": { "uid": "AZiZVRiqAABt7uudQBprAQAA", "name": "Errors are high", "id": "8231846403567092370", "source_id": 36, "type": "log_alert" }, "monitor_id": 150830126, "monitor_notifications": [ "causely-alerts" ], "monitor": { "group_status": 5, "alert_cycle_key_txt": "8231846399607848082", "query": "logs(\"service:(causely-analysis*production) status:error -service:* -service:*compliancev2* -service:*ocr* -\\\"Request failed with status code 404\\\" -\\\"#8b6d6338\\\"\").index(\"*\").rollup(\"count\").by(\"service\").last(\"10m\") > 2000", "groups": [ "service:causely-analysis" ], "created_at": 1723077612000, "priority": 1, "type": "log alert", "transition": { "destination_state": "Warn", "transition_type": "warn", "source_state": "OK" }, "tags": [ "team:analysis-team" ], "result": { "result_id_txt": "8231846397529909389", "result_id": 8231846397529909389, "result_ts": 1754919056, "group_key": "service" }, "name": "Errors are high", "options": { "on_missing_data": "default", "thresholds": { "critical": 2000, "warning": 1000 }, "new_group_delay": 0, "enable_logs_sample": true, "include_tags": false, "groupby_simple_monitor": false, "notify_audit": false }, "modified": 1754582393000, "id": 150830516, "templated_name": "Errors are high in production", }, "priority": "normal", "title": "[P1] [Warn] Errors are high", "service": "causely-analysis", "sourcecategory": "monitor_alert", "event_object": "047235cd648d8d01abf7bc03c7c11bc8", "_dd": { "has_notification": false, "internal": "1", "version": "1" }, "timestamp": 1754919056000, "status": "warning" }, "timestamp": "2025-08-11T13:30:56Z", "tags": [ "monitor", "priority:p1", "service:causely-analysis", "source:alert", "team:analysis-team" ] } }, ``` --- ## Docker ### Overview Causely provides **native integration** with [Docker](https://www.docker.com/) that automatically discovers and monitors your container infrastructure. The integration provides visibility across your Docker environment, from host-level resources down to individual containers, helping you identify infrastructure issues before they impact your applications. ### How It Works Causely's Docker integration automatically discovers containers using the Docker API, monitors resource usage and states, creates a topology graph showing relationships between hosts, containers, workloads, and applications, and creates network endpoints for service discovery. Docker monitoring works automatically when you install Causely using the [Docker installation guide](/installation/docker). --- ## Dynatrace Causely integrates with [Dynatrace](https://www.dynatrace.com/platform/application-observability/) to ingest service dependencies and trace-based metrics from Kubernetes and ECS environments, and infrastructure and dataflow data from public cloud and on-prem environments such as VMware. This enables Causely to build a unified causality graph across mixed container platforms and underlying infrastructure, and generate a dependency and service map that accurately reflects how your system behaves under change. Causely also maps your existing Dynatrace infrastructure alerts to infrastructure Signals, alongside the application-layer alerts it already maps. By incorporating Dynatrace traces, service-level metrics, and infrastructure data, Causely strengthens reliability assurance across hybrid environments. The setup process below applies to all supported environments. ### Supported environments - Kubernetes - ECS (via Dynatrace OneAgent / Dynatrace APM instrumentation) - VMware and other on-prem or host-based environments (via Dynatrace OneAgent / Dynatrace APM instrumentation) ### Step 1: Create an API token Causely uses a read-only Dynatrace API token. The token needs the following scopes: | Scope | UI name | Used for | | -------------------------- | ---------------------------------------- | -------------------------------------------- | | `entities.read` | Read entities | Services, processes, hosts, and dependencies | | `metrics.read` | Read metrics | Request rate, error rate, and duration | | `problems.read` | Read problems | Dynatrace problems used as Signals | | `syntheticExecutions.read` | Read synthetic monitor execution results | Synthetic monitors and their results | | `syntheticLocations.read` | Read synthetic locations | Synthetic monitor locations | You can create the token in the Dynatrace UI or with the Dynatrace API. **Dynatrace UI** 1. In Dynatrace, go to **Manage** > **Access Tokens**. 2. Select **Generate new token**. 3. In **Token name**, enter a name such as `causely`. Optionally, set an **Expiration date**. 4. Leave **Template** set to **None**. 5. In **Search scopes...**, search for each scope in the table above and select its checkbox. Check that all five scopes appear under **Selected scopes**. 6. Select **Generate token**. 7. Copy the token and store it securely. Dynatrace shows the token only once. :::tip The **Generate access token** page also shows the matching cURL command for the selected scopes. Use **Copy cURL** to automate token creation, as shown in the **Dynatrace API (curl)** tab. ::: **Dynatrace API (curl)** To create the token with the [Dynatrace API tokens API](https://docs.dynatrace.com/docs/dynatrace-api/environment-api/tokens-v2/api-tokens/post-token), you need an existing token with the `apiTokens.write` scope. ```bash export DT_URL="https://.live.dynatrace.com" export DT_ADMIN_TOKEN="" curl -X POST "${DT_URL}/api/v2/apiTokens" \ -H "accept: application/json; charset=utf-8" \ -H "Content-Type: application/json; charset=utf-8" \ -H "Authorization: Api-Token ${DT_ADMIN_TOKEN}" \ -d '{ "name": "causely", "scopes": [ "entities.read", "metrics.read", "problems.read", "syntheticExecutions.read", "syntheticLocations.read" ] }' ``` The new token is in the `token` field of the response. Dynatrace returns it only once, so store it securely. ### Step 2: Create a Kubernetes Secret with the Dynatrace URL and API token After you create the API token, create a Kubernetes Secret. Set `url` to your Dynatrace environment URL, such as `https://.live.dynatrace.com`, and set `api-token` to the token from Step 1: ```bash kubectl create secret generic \ --namespace causely dynatrace-credentials \ --from-literal=url="..." \ --from-literal=api-token="..." ``` ### Step 3: Update Causely configuration Once the secret is created, update the Causely configuration to enable scraping for the dynatrace account. Enabling scraping allows Causely to ingest Dynatrace topology, telemetry, and infrastructure data for all supported environments. Below is an example configuration: ```yaml scrapers: dynatrace: enabled: true accounts: - secretName: dynatrace-credentials ``` #### Alternative: Enable Credentials Autodiscovery Causely supports credentials autodiscovery so you can add new accounts without editing config. Label the Kubernetes secret to enable autodiscovery for the Dynatrace scraper: ```bash kubectl --namespace causely label secret dynatrace-credentials "causely.ai/scraper=Dynatrace" ``` ### Unified Stitching Across Kubernetes and ECS When Dynatrace is enabled, Causely automatically discovers and models dependencies between ECS and Kubernetes services, creating a unified causal view of your environment. Causely will: - Discover and stitch dependencies across Kubernetes and ECS services, even when they span clusters or runtimes - Include service-level metrics (request rate, error rate, duration) and corresponding Signals from ECS as inputs to Causely's causal reasoning - Display ECS services alongside Kubernetes services in Causely’s environment graph, dependency views, and reliability analysis ### Infrastructure Coverage When Dynatrace is enabled, Causely ingests how data flows between your application components and the infrastructure they run on, directly from Dynatrace instrumentation. This covers public cloud and on-prem environments, such as VMware. Causely will: - Map application components to their underlying infrastructure, including VMware hosts - Map your existing Dynatrace infrastructure alerts to the infrastructure Signals Causely already tracks, alongside the application-layer alerts it already maps - Use these infrastructure Signals as inputs to Causely's causal reasoning, alongside application-layer Signals --- ## eBPF [eBPF](https://ebpf.io/) (extended Berkeley Packet Filter) is a cutting-edge Linux kernel technology that allows safe, efficient, and programmable data collection at runtime. It enables deep observability into your system and application behavior without modifying code or adding performance overhead. Causely leverages **OpenTelemetry eBPF instrumentation**, powered by [Grafana Beyla](https://grafana.com/docs/beyla/latest/), to automatically extract rich telemetry data from your services. This zero-effort integration provides actionable insights into service interactions, latencies, and system performance, with zero changes to application code. ### How It Works Causely's eBPF-based instrumentation follows a streamlined process: 1. The system inspects running applications to determine which technologies they use. 2. Once application characteristics are identified, eBPF uprobes are attached to specific user-space functions to capture application-level information, for example HTTP or gRPC calls. 3. These probes generate events related to the application's traffic, which are then converted into OpenTelemetry metrics and traces. 4. The collected telemetry is sent to the [mediator](/getting-started/architecture#mediation-layer), which will perform Signal detection, topology discovery and local processing and then send distilled insights to the Causely engine. This approach supports multiple programming languages including Go, Java, .NET, NodeJS, Python, Ruby, Rust, and more, without requiring any code modifications or language-specific agents. To learn more about how the automatic instrumentation works, see the [Grafana Beyla documentation](https://grafana.com/docs/beyla/latest/). ### Technical Implementation #### Uprobe-Based Instrumentation Causely's eBPF instrumentation uses **uprobes exclusively** to intercept specific user-space functions within your applications. This approach: - Attaches probes to application-level functions (HTTP handlers, gRPC calls, database clients) - Does **not** hook into kernel networking callbacks or the packet datapath - Does **not** act as a CNI or critical network infrastructure component - Does **not** intercept or modify network packets The instrumentation may inject trace context headers (such as W3C Trace Context) into outgoing requests to enable distributed tracing. However, it does not intercept, block, or route network traffic at the kernel level. This design choice is intentional. Kernel-level network interception (kprobes on network syscalls) can introduce per-packet latency and increase blast radius. By instrumenting at the application layer, Causely captures the telemetry needed for causal analysis without impacting network connectivity or acting as a critical path component. Beyla is built on the [OpenTelemetry eBPF Instrumentation (OBI)](https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation) project, which removed kprobe-based collection in favor of uprobe-only approaches ([PR #752](https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/pull/752)). #### Performance Characteristics - **No per-packet overhead**: Probes fire on application-level function calls, not network packets - **Targeted instrumentation**: Only specific functions are instrumented, not all syscalls - **Minimal CPU impact**: Overhead is proportional to application request rate, not network traffic volume - **Fail-safe design**: If an agent fails, applications continue to function normally, with no impact to traffic flow ### Configuration #### Java instrumentation For automatic instrumentation of Java applications the HotSpot JVM supports dynamic agent loading, attaching the Beyla/OBI Java agent at runtime to observe TLS encrypted traffic. To enable [this functionality](https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/pull/1021) modify the `causely-values.yaml` configuration: ``` agent: beyla: global_config: javaagent: enabled: true ``` #### GenAI instrumentation For automatic instrumentation of GenAI applications, with a larger payload extraction Beyla/OBI can instrument client LLM calls with GenAI semantic conventions: To enable [this functionality](https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/pull/1353) modify the `causely-values.yaml` configuration: ``` agent: beyla: global_config: ebpf: buffer_sizes: http: 8192 payload_extraction: http: genai: openai: enabled: true anthropic: enabled: true gemini: enabled: true ``` #### Application metrics (BeylaMetrics) Beyla/OBI can export [application-level HTTP and RPC metrics](https://opentelemetry.io/docs/zero-code/obi/metrics/) (request rates, status codes, and latency histograms). Causely's `BeylaMetrics` scraper reads these metrics from Prometheus (or a compatible store such as Mimir) to discover service-to-service access relationships and SLOs. ##### 1. Enable Beyla Prometheus export Expose Beyla's Prometheus metrics endpoint by setting a port in `causely-values.yaml`: ```yaml agent: beyla: prometheus_export: port: 9090 ``` The Causely agent runs with `hostNetwork: true`, so this port is bound on every node. Choose a port that does not conflict with other host-network services or NodePorts on the cluster. Ensure your Prometheus scrapes these endpoints. When `serviceMonitor.enabled` is true and a port is set, Causely also creates a PodMonitor for the `beyla-metrics` port. ##### 2. Point Causely at the scraped metrics Configure the `BeylaMetrics` scraper with the Prometheus API endpoint that holds the Beyla application metrics - similar to the [Istio Prometheus integration](/telemetry-sources/istio): ```yaml scrapers: beylametrics: prometheus: endpoint: http://prometheus.monitoring:9090 ``` Replace the endpoint with your Prometheus or Mimir query URL. Alternatively, configure the same connection from the [Integrations page](https://portal.causely.app/integrations) in the Causely UI, or by creating a Kubernetes Secret (no Helm change required). With auto-discovery enabled (the default), label the secret so the mediator picks it up: ```yaml apiVersion: v1 kind: Secret metadata: name: beyla-metrics-prometheus namespace: causely labels: causely.ai/scraper: BeylaMetrics type: Opaque stringData: endpoint: http://prometheus.monitoring:9090 # username: readonly # optional basic auth # password: ... ``` #### Disabling autoinstrumentation eBPF-based autoinstrumentation is **enabled by default** in Causely deployments. If you wish to disable it, you can modify the `causely-values.yaml` configuration: ```yaml scrapers: bpf: enabled: false ``` #### Including specific workloads eBPF-based autoinstrumentation can be selectively scoped per namespace or per deployment in a Kubernetes environment. Modify `causely-values.yaml` as needed. To whitelist instrumentation to a single Kubernetes deployment, including every pod managed by that deployment, use `discovery.instrument` with `k8s_deployment_name` (and optional port selectors). When multiple selectors appear in the same `instrument` entry, Beyla only instruments processes that match all of them: ```yaml agent: beyla: discovery: instrument: - open_ports: 80,443,2000-10000 k8s_deployment_name: "{payment,cart}" ``` #### Excluding workloads To exclude workloads from instrumentation: ```yaml agent: beyla: discovery: exclude_instrument: - k8s_namespace: {"nginx-ingress","ingress-nginx"} - k8s_deployment_name: "{cnpg-cloudnative-pg,ingress-nginx-controller}" ``` See the [Beyla service discovery documentation](https://grafana.com/docs/beyla/latest/configure/service-discovery/) for other selectors (`k8s_namespace`, `k8s_pod_labels`, and more). :::note If your application is already instrumented with OpenTelemetry, the agent will detect this and not instrument it again. ::: --- ## Elasticsearch ### Overview Causely integrates with [Elasticsearch](https://www.elastic.co/elasticsearch/) in two ways: 1. **Log Retrieval**: Causely can ingest and analyze container logs stored in Elasticsearch. This allows you to correlate error and exception patterns with service-level incidents and Diagnoses. 2. **Cluster Performance Insights**: Causely connects directly to Elasticsearch cluster APIs to detect node-level or infrastructure bottlenecks that can cause degraded application performance. Use this integration if: - Your logs are already centralized in Elasticsearch. - If you operate Elasticsearch clusters, Causely goes beyond Signal monitoring by analyzing them in real time to surface the actual Diagnosis behind Elasticsearch problems, helping you identify reliability issues such as: - [Memory Pressure](/reference/root-causes/infrastructure#memory-pressure) and [Disk Pressure](/reference/root-causes/infrastructure#disk-pressure) - [CPU Congested](/reference/root-causes/infrastructure#cpu-congested) and [Memory Congested](/reference/root-causes/infrastructure#memory-congested) - [Memory Failure](/reference/root-causes/infrastructure#memory-failure) and [Frequent Memory Failure](/reference/root-causes/infrastructure#frequent-memory-failure) - [Disk Total IOPs Congested](/reference/root-causes/infrastructure#disk-total-iops-congested) and [Inode Usage Congested](/reference/root-causes/infrastructure#inode-usage-congested) ### Use Case 1: Log Retrieval from Elasticsearch When your logs are already shipped to Elasticsearch, for example via Fluentd, Logstash, or Beats, Causely can pull those logs directly to enhance the Diagnosis. Instead of collecting logs from Kubernetes directly, Causely queries Elasticsearch indices for logs tied to specific services or containers. #### Benefits - Automatically surfaces relevant logs in the context of an active Diagnosis or service malfunction. - Displays container-level logs under affected services when abnormal behavior occurs, for example error spikes or degraded performance. - Shows log lines and exceptions alongside Diagnoses to validate issues and dramatically shorten time to understanding and resolution. #### Setup Create a Kubernetes Secret for your Elasticsearch credentials (if authentication is required): ```bash kubectl create secret generic elasticsearch-logs-credentials \ --namespace causely \ --from-literal=api_key="your-api-key" ``` Or for basic authentication: ```bash kubectl create secret generic elasticsearch-logs-credentials \ --namespace causely \ --from-literal=username="your-username" \ --from-literal=password="your-password" ``` #### How Elasticsearch Works ##### Schema Flexibility Elasticsearch is schema-less (mostly): Unlike SQL databases, you don't define a strict schema upfront. - **Dynamic mapping**: When you index a document, Elasticsearch automatically creates field mappings based on what it sees - **No enforcement**: Different documents in the same index can have completely different fields ##### Different Log Shippers/Collectors Different log shipping tools structure data differently: - **Fluentd**: Creates fields like `data.message`, `data.priority`, adds `fluentd: "true"` - **Filebeat**: Might use `message`, `log.level`, `kubernetes.namespace` - **Logstash**: Often uses `message`, `@timestamp`, custom parsed fields This is why Causely requires explicit field mappings: to know where to find pod names, messages, severity, and other metadata in your specific Elasticsearch index structure. Then update your Kubernetes scraper configuration with the Elasticsearch endpoint, indices, and field mappings: ```yaml scrapers: kubernetes: elasticsearch: endpoint: 'http://elasticsearch.logging.svc.cluster.local:9200' secret: 'elasticsearch-logs-credentials' # Optional: omit if no auth required indices: - 'logstash-app_log-*' # Example: Fluentd/Logstash format # - "filebeat-*" # Example: Filebeat format fields: timestamp: '@timestamp' message: 'data.message' # Example: nested field (Fluentd) # message: "message" # Example: flat field (Filebeat) severity: 'data.priority' # Example: explicit severity field (optional) pod: 'host' # Example: Fluentd uses "host" field # pod: "kubernetes.pod.name" # Example: Filebeat uses standard k8s metadata namespace: '' # Optional: set if available container: '' # Optional: set if available ``` The field mappings allow you to configure how Causely extracts pod names, namespaces, containers, messages, and severity from your Elasticsearch documents. This supports different log shipping formats (Fluentd, Filebeat, Logstash, etc.) with varying field structures. #### Authentication Options - **API Key**: Include `api_key` in the Kubernetes secret - **Basic Auth**: Include `username` and `password` in the Kubernetes secret - **No Auth**: Omit the `secret` field in the configuration if authentication is not required #### Supported Log Types - Container logs (stdout/stderr) ### Use Case 2: Cluster Performance Monitoring ### Setup Guide #### Step 1: Create an API Key Create an API Key for your Elasticsearch cluster with the following permissions: - **Cluster monitoring**: Required to access cluster health and node statistics - **Node stats**: Required to monitor individual node performance and resource usage - **Read access**: Required to query cluster metadata and configuration For Elastic Cloud: 1. Go to Elastic Cloud Console → Security → API Keys 2. Create a new API key with appropriate permissions 3. Copy the API key for use in the next step For self-hosted Elasticsearch: 1. Use Kibana Security → Users → Create API Key 2. Or use the Elasticsearch API: `POST /_security/api_key` 3. Ensure the API key has monitoring privileges #### Step 2: Create a Kubernetes Secret for the API Key After creating the API Key create a Kubernetes Secret: ```bash kubectl create secret generic \ --namespace causely elastic-credentials \ --from-literal=api_key="..." \ --from-literal=url='https://....eastus2.azure.elastic-cloud.com' ``` The `url` must be the endpoint URL of your Elasticsearch cluster: - **Elastic Cloud**: `https://your-cluster-id.region.elastic-cloud.com` - **AWS OpenSearch**: `https://your-domain.region.es.amazonaws.com` - **Azure Search**: `https://your-service.search.windows.net` - **Self-hosted**: `https://your-elasticsearch-host:9200` #### Step 3: Update Causely Configuration Once the Secret is created, update the Causely configuration to enable scraping for the new cluster. Below is an example configuration: ```yaml scrapers: elasticsearch: enabled: true instances: - secretName: elastic-credentials namespace: causely ``` #### Alternative: Enable Credentials Autodiscovery Causely also supports credentials autodiscovery. This feature allows you to add new scraping targets without updating the Causely configuration. Label the Kubernetes Secret to enable autodiscovery for the corresponding scraper. ```bash kubectl --namespace causely label secret elastic-credentials "causely.ai/scraper=ElasticSearch" ``` ### What Data is Collected #### From Logs - Error and warning messages from container stdout/stderr logs linked to affected services or Diagnoses. #### From Cluster Performance - **Cluster entities** with names and health status - **Service-to-cluster mappings** (which service provides the Elasticsearch cluster) - **Connection details** including endpoint URL and API authentication - **Cluster health metrics** including shard allocation status - **Cluster status** (green, yellow, red) and health indicators - **Shard allocation metrics** (active, relocating, initializing, unassigned shards) - **Task queue monitoring** (pending tasks, in-flight operations, queue wait times) - **Active shards percentage** for overall cluster health - **Node information** including names, roles, and attributes - **File descriptor usage** (`FileDescriptorUsage`, `FileDescriptorCapacity`) - **Memory utilization** (`MemoryUsage`, `MemoryCapacity`) - **CPU performance** (`CPUUsage`, `CPUCapacity`) - **Load average metrics** (1m, 5m, 15m) for trend analysis - **Disk space metrics** (`Usage`, `Capacity`) - **File system statistics** (total, free, available bytes, watermarks) - **I/O performance metrics** (read/write operations, data transfer, I/O time) - **JVM memory pool statistics** (heap used/committed/max, non-heap usage) - **Garbage collection metrics** (collection count, time per collector) - **Thread statistics** (current/peak thread counts, thread pool utilization) - **Buffer pool performance** (buffer count, usage, capacity) - **Class loading statistics** (loaded/unloaded classes) - **Operating system metrics** (memory, CPU, swap usage) - **Cgroup metrics** for containerized deployments - **Transport layer information** (addresses, ports) - **Network endpoint mapping** for service discovery - **Host and IP address tracking** for infrastructure mapping - **Service-to-node mappings** - **Node-to-VM relationships** - **VM-to-disk relationships** - **Attribute-based labeling** for custom categorization ### Result - Captures and displays logs alongside a Diagnosis to highlight the precise errors or stack traces that occurred around the time of failure. - Detects Elasticsearch cluster issues that may propagate to upstream applications. - Automatically identifies and explains the underlying Diagnosis, rather than requiring manual investigation across dashboards. --- ## GCP ### Overview Causely provides comprehensive native integration with Google Cloud Platform (GCP) to help you identify and resolve infrastructure and managed service reliability issues before they impact your users. Instead of only monitoring isolated metrics or alerts, Causely analyzes real-time signals from your GCP environment to infer the underlying causes of performance degradation and reliability risk across your cloud services. By setting up the GCP integration, you will be able to: **Identify causes for reliability issues originating from your GCP infrastructure and services**, including: - [Service Congestion](/reference/root-causes/services#congested-1) and [Service Malfunction](/reference/root-causes/services#malfunction-1) - [Application Load Balancer Misconfiguration](/reference/root-causes/services#application-load-balancer) issues - [Authentication](/reference/root-causes/services#authentication-misconfiguration) and [Network Policy](/reference/root-causes/services#network-policy-misconfiguration) misconfigurations - [Memory Pressure](/reference/root-causes/infrastructure#memory-pressure) and [CPU Congested](/reference/root-causes/infrastructure#cpu-congested) issues - [Disk Pressure](/reference/root-causes/infrastructure#disk-pressure) and [I/O congestion](/reference/root-causes/infrastructure#disk-total-iops-congested) Observe GCP resources as first-class entities in the Causely Topology Graph, including their relationships to Kubernetes workloads, managed services, and downstream dependencies Monitor performance and health metrics from GCP-native sources with automatic correlation to service impact and causal diagnoses To set up this integration, you need a Service Account with least-privilege [predefined viewer roles](https://docs.cloud.google.com/iam/docs/understanding-roles) on the GCP projects you want to monitor. ### Supported GCP Services Causely integrates with the following GCP services to provide comprehensive observability and causal inference: - Cloud Bigtable - Cloud Load Balancing - Cloud Memorystore for Redis - Cloud Memorystore for Redis (Cluster mode) - Cloud Pub/Sub - Cloud Run - Cloud Service Mesh / Traffic Director (proxyless gRPC) - Cloud SQL (MySQL and PostgreSQL) - Compute Engine - Persistent Disks ### Step 1: Create a Service Account (SA) Use the following `gcloud` command to create an SA: ```bash gcloud iam service-accounts create causely \ --description="causely" \ --display-name="causely" gcloud projects add-iam-policy-binding PROJECT_ID \ --member="serviceAccount:causely@PROJECT_ID.iam.gserviceaccount.com" \ --role="roles/compute.viewer" gcloud projects add-iam-policy-binding PROJECT_ID \ --member="serviceAccount:causely@PROJECT_ID.iam.gserviceaccount.com" \ --role="roles/compute.networkViewer" gcloud projects add-iam-policy-binding PROJECT_ID \ --member="serviceAccount:causely@PROJECT_ID.iam.gserviceaccount.com" \ --role="roles/logging.viewer" gcloud projects add-iam-policy-binding PROJECT_ID \ --member="serviceAccount:causely@PROJECT_ID.iam.gserviceaccount.com" \ --role="roles/monitoring.viewer" gcloud projects add-iam-policy-binding PROJECT_ID \ --member="serviceAccount:causely@PROJECT_ID.iam.gserviceaccount.com" \ --role="roles/bigtable.viewer" gcloud projects add-iam-policy-binding PROJECT_ID \ --member="serviceAccount:causely@PROJECT_ID.iam.gserviceaccount.com" \ --role="roles/cloudsql.viewer" gcloud projects add-iam-policy-binding PROJECT_ID \ --member="serviceAccount:causely@PROJECT_ID.iam.gserviceaccount.com" \ --role="roles/memcache.viewer" gcloud projects add-iam-policy-binding PROJECT_ID \ --member="serviceAccount:causely@PROJECT_ID.iam.gserviceaccount.com" \ --role="roles/pubsub.viewer" gcloud projects add-iam-policy-binding PROJECT_ID \ --member="serviceAccount:causely@PROJECT_ID.iam.gserviceaccount.com" \ --role="roles/redis.viewer" gcloud projects add-iam-policy-binding PROJECT_ID \ --member="serviceAccount:causely@PROJECT_ID.iam.gserviceaccount.com" \ --role="roles/run.viewer" gcloud projects add-iam-policy-binding PROJECT_ID \ --member="serviceAccount:causely@PROJECT_ID.iam.gserviceaccount.com" \ --role="roles/networkservices.viewer" ``` Role definitions: [Monitoring Viewer](https://docs.cloud.google.com/iam/docs/roles-permissions/monitoring#monitoring.viewer), [Compute Viewer](https://docs.cloud.google.com/iam/docs/roles-permissions/compute#compute.viewer), [Compute Network Viewer](https://docs.cloud.google.com/iam/docs/roles-permissions/compute#compute.networkViewer), [Logging Viewer](https://docs.cloud.google.com/iam/docs/roles-permissions/logging#logging.viewer), [Bigtable Viewer](https://docs.cloud.google.com/iam/docs/roles-permissions/bigtable#bigtable.viewer), [Cloud SQL Viewer](https://docs.cloud.google.com/iam/docs/roles-permissions/cloudsql#cloudsql.viewer), [Memorystore for Memcached Viewer](https://docs.cloud.google.com/iam/docs/roles-permissions/memcache#memcache.viewer), [Pub/Sub Viewer](https://docs.cloud.google.com/iam/docs/roles-permissions/pubsub#pubsub.viewer), [Memorystore for Redis Viewer](https://docs.cloud.google.com/iam/docs/roles-permissions/redis#redis.viewer), [Cloud Run Viewer](https://docs.cloud.google.com/iam/docs/roles-permissions/run#run.viewer), [Network Services Viewer](https://docs.cloud.google.com/iam/docs/roles-permissions/networkservices#networkservices.viewer). `roles/networkservices.viewer` is required to discover Cloud Service Mesh / Traffic Director resources (Mesh and GRPCRoutes). Backend service and Network Endpoint Group (NEG) enrichment for linking mesh services to Kubernetes backends uses the existing `roles/compute.viewer` and `roles/compute.networkViewer` roles. Grant these roles on every project that owns GRPCRoutes, backend services, or NEGs you want Causely to discover (for example, shared mesh projects and service projects). Alternatively, for simplicity, you can grant the broader `roles/viewer` role on the project instead of the granular roles above. ### Step 2: Provide GCP Credentials to the Mediator Choose one of the following authentication options. #### Option A: Service Account JSON After creating the SA, save its credentials .json in a Kubernetes Secret. ```bash kubectl create secret --namespace causely generic gcp-credentials \ --from-file=service_account.json=path_to_service_account.json \ --from-literal=project_id=PROJECT_ID ``` #### Option B: GKE Workload Identity Federation (Recommended) Use Workload Identity to bind a Kubernetes Service Account (KSA) to a Google Service Account (GSA). See the GCP guide for details: [Kubernetes Engine Workload Identity guide](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#authenticating_to). 1) Grant the KSA permission to impersonate the GSA: ```bash gcloud iam service-accounts add-iam-policy-binding @.iam.gserviceaccount.com \ --role roles/iam.workloadIdentityUser \ --member "serviceAccount:.svc.id.goog[/]" ``` Example: ```bash gcloud iam service-accounts add-iam-policy-binding monitor@playground-377422.iam.gserviceaccount.com \ --role roles/iam.workloadIdentityUser \ --member "serviceAccount:playground-377422.svc.id.goog[causely/causely-mediator]" \ --project=playground-377422 ``` 2) Annotate the mediator KSA in `values.yaml` to use the GSA: ```yaml mediator: serviceAccount: annotations: iam.gke.io/gcp-service-account: @.iam.gserviceaccount.com ``` 3) Create a Kubernetes Secret to enable the GCP integration ```bash kubectl create secret --namespace causely generic gcp-credentials ``` ### Step 3: Update Causely Configuration Once the Secret is created, update the Causely configuration to enable scraping for the new project. Below is an example configuration: ```yaml scrapers: gcp: enabled: true projects: - secretName: gcp-credentials namespace: causely ``` ### Alternative: Enable Credentials Autodiscovery Causely also supports credentials autodiscovery. This feature allows you to add new scraping targets without updating the Causely configuration. Simply label the Kubernetes Secret to enable autodiscovery for the corresponding scraper. #### Example: Labeling Secrets for Autodiscovery ```bash kubectl --namespace causely label secret gcp-credentials "causely.ai/scraper=GCP" ``` With these steps, you can seamlessly integrate Causely with the Google Cloud API and configure it to monitor your desired resources. ### Configure individual Resources You have to enable the resources you want causely to discover and monitor. --- ## Grafana Causely can leverage [Grafana](https://grafana.com/products/cloud/) as an observability backend with Grafana [Alloy](https://grafana.com/oss/alloy-opentelemetry-collector/) and [Beyla](https://grafana.com/oss/beyla-ebpf/) as instrumentation. > **Note:** If your logs are centralized in Grafana **Loki**, Causely can retrieve and display them directly. > See the [Loki Integration](#loki-integration) section below for configuration details and benefits. Please use the following [values](/telemetry-sources/grafana#alloy-valuesyaml) for your helm install: ### Install with Causely ```shell helm upgrade --install alloy grafana/alloy --create-namespace --namespace monitoring --values ./alloy-values.yaml ``` #### alloy-values.yaml ```yaml controller: type: deployment alloy: stabilityLevel: experimental extraPorts: - name: 'grpc' port: 4317 targetPort: 4317 - name: 'http' port: 4318 targetPort: 4318 - name: 'datadog' port: 8126 targetPort: 8126 configMap: content: | otelcol.exporter.otlp "causely" { client { endpoint = "mediator.causely:4317" tls { insecure = true } } } otelcol.processor.batch "default" { output { metrics = [otelcol.exporter.otlp.causely.input] traces = [otelcol.exporter.otlp.causely.input] } } otelcol.processor.k8sattributes "default" { extract { label { from = "pod" } metadata = [ "k8s.namespace.name", "k8s.pod.name", "k8s.pod.uid", "k8s.deployment.name", "k8s.node.name", "k8s.pod.start_time", "container.id", ] } output { traces = [otelcol.processor.batch.default.input] } } otelcol.processor.deltatocumulative "default" { output { metrics = [otelcol.processor.batch.default.input] } } otelcol.receiver.datadog "default" { endpoint = "0.0.0.0:8126" output { metrics = [otelcol.processor.deltatocumulative.default.input] traces = [otelcol.processor.batch.default.input] } } otelcol.receiver.otlp "otlp" { grpc { endpoint = "0.0.0.0:4317" } http { endpoint = "0.0.0.0:4318" } output { metrics = [otelcol.processor.batch.default.input] traces = [otelcol.processor.k8sattributes.default.input] } } ``` ### Install with Causely and Grafana Cloud To also send logs, metrics, and traces to Grafana Cloud, you need to use the following [values](/telemetry-sources/grafana#grafana-alloy-valuesyaml) for your helm install: ```shell helm upgrade --install alloy grafana/alloy --create-namespace --namespace monitoring --values ./grafana-alloy-values.yaml ``` #### grafana-alloy-values.yaml ```yaml controller: hostPID: true alloy: securityContext: privileged: true stabilityLevel: experimental extraPorts: - name: 'grpc' port: 4317 targetPort: 4317 - name: 'http' port: 4318 targetPort: 4318 - name: 'datadog' port: 8126 targetPort: 8126 configMap: content: | otelcol.exporter.otlp "causely" { client { endpoint = "mediator.causely:4317" tls { insecure = true } } } otelcol.exporter.otlphttp "grafana" { client { endpoint = "https://otlp-gateway-prod-us-east-0.grafana.net/otlp" auth = otelcol.auth.basic.grafana.handler } } otelcol.auth.basic "grafana" { username = "GRAFANA_CLOUD_INSTANCE_ID" password = "GRAFANA_CLOUD_API_KEY" } // discovery.kubernetes allows you to find scrape targets from Kubernetes resources. // It watches cluster state and ensures targets are continually synced with what is currently running in your cluster. discovery.kubernetes "pod_logs" { role = "pod" } // discovery.relabel rewrites the label set of the input targets by applying one or more relabeling rules. // If no rules are defined, then the input targets are exported as-is. discovery.relabel "pod_logs" { targets = discovery.kubernetes.pod_logs.targets // Label creation - "namespace" field from "__meta_kubernetes_namespace" rule { source_labels = ["__meta_kubernetes_namespace"] action = "replace" target_label = "namespace" } // Label creation - "pod" field from "__meta_kubernetes_pod_name" rule { source_labels = ["__meta_kubernetes_pod_name"] action = "replace" target_label = "pod" } // Label creation - "container" field from "__meta_kubernetes_pod_container_name" rule { source_labels = ["__meta_kubernetes_pod_container_name"] action = "replace" target_label = "container" } // Label creation - "app" field from "__meta_kubernetes_pod_label_app_kubernetes_io_name" rule { source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_name"] action = "replace" target_label = "app" } // Label creation - "job" field from "__meta_kubernetes_namespace" and "__meta_kubernetes_pod_container_name" // Concatenate values __meta_kubernetes_namespace/__meta_kubernetes_pod_container_name rule { source_labels = ["__meta_kubernetes_namespace", "__meta_kubernetes_pod_container_name"] action = "replace" target_label = "job" separator = "/" replacement = "$1" } // Label creation - "container" field from "__meta_kubernetes_pod_uid" and "__meta_kubernetes_pod_container_name" // Concatenate values __meta_kubernetes_pod_uid/__meta_kubernetes_pod_container_name.log rule { source_labels = ["__meta_kubernetes_pod_uid", "__meta_kubernetes_pod_container_name"] action = "replace" target_label = "__path__" separator = "/" replacement = "/var/log/pods/*$1/*.log" } // Label creation - "container_runtime" field from "__meta_kubernetes_pod_container_id" rule { source_labels = ["__meta_kubernetes_pod_container_id"] action = "replace" target_label = "container_runtime" regex = "^(\\S+):\\/\\/.+$" replacement = "$1" } } // loki.source.kubernetes tails logs from Kubernetes containers using the Kubernetes API. loki.source.kubernetes "pod_logs" { targets = discovery.relabel.pod_logs.output forward_to = [otelcol.receiver.loki.default.receiver] } otelcol.receiver.loki "default" { output { logs = [otelcol.processor.batch.default.input] } } beyla.ebpf "default" { attributes { kubernetes { enable = "true" } select { attr = "sql_client_duration" exclude = [] include = ["db.query.text"] } } discovery { services { open_ports = "80,443,3000,8000-8999" } } output { traces = [otelcol.processor.batch.default.input] } } otelcol.processor.batch "default" { output { logs = [otelcol.exporter.otlphttp.grafana.input] metrics = [otelcol.exporter.otlphttp.grafana.input, otelcol.exporter.otlp.causely.input] traces = [otelcol.exporter.otlphttp.grafana.input, otelcol.exporter.otlp.causely.input] } } otelcol.processor.k8sattributes "default" { extract { label { from = "pod" } metadata = [ "k8s.namespace.name", "k8s.pod.name", "k8s.pod.uid", "k8s.deployment.name", "k8s.node.name", "k8s.pod.start_time", "container.id", ] } output { traces = [otelcol.processor.batch.default.input] } } otelcol.processor.deltatocumulative "default" { output { metrics = [otelcol.processor.batch.default.input] } } otelcol.receiver.datadog "default" { endpoint = "0.0.0.0:8126" output { metrics = [otelcol.processor.deltatocumulative.default.input] traces = [otelcol.processor.batch.default.input] } } otelcol.receiver.otlp "otlp" { grpc { endpoint = "0.0.0.0:4317" } http { endpoint = "0.0.0.0:4318" } output { logs = [otelcol.processor.batch.default.input] metrics = [otelcol.processor.batch.default.input] traces = [otelcol.processor.k8sattributes.default.input] } } ``` ### Alerts Causely can ingest the alerts from Grafana's [unified alerting](https://grafana.com/docs/grafana/latest/alerting/) and use them as Signals for causal analysis. The Grafana scraper periodically reads the currently active alert instances from Grafana's Alertmanager-compatible API (`/api/alertmanager/grafana/api/v2/alerts`), maps them to the right entities, and activates the corresponding Causely Signals. When an alert clears in Grafana, Causely automatically deactivates the Signal. This means you can: - Keep your existing Grafana alert rules and contact points. - See Grafana alerts mapped to services and dependencies inside Causely. - Automatically map alerts to Signals and point to the actual cause. #### Authentication Each Grafana instance accepts one of: - **Service account token**: set `token` to a Grafana service account token (for example, `glsa_...`). The token only needs read access to alerts/alerting. - **Basic auth**: set `username` and `password` instead of `token`. #### Auto-discovery with a Kubernetes Secret When `auto_discovery.enabled` is `true`, the mediator watches for Kubernetes Secrets labeled `causely.ai/scraper: Grafana` and creates a scraper for each one. This lets you add or rotate Grafana instances without changing the mediator configuration. ```yaml apiVersion: v1 kind: Secret metadata: name: grafana-prod namespace: causely labels: causely.ai/scraper: Grafana type: Opaque stringData: endpoint: https://grafana.example.com token: glsa_xxxxxxxxxxxxxxxxxxxx # username: causely-readonly # basic auth (alternative to token) # password: ... # labelFilters is an optional JSON object; only alerts matching every # listed label/value are ingested. labelFilters: '{"namespace": ["payments", "checkout"]}' ``` The recognized Secret keys are `endpoint`, `token`, `username`, `password`, and `labelFilters`. #### Filtering which alerts are ingested Use `labelFilters` to restrict ingestion to alerts carrying specific labels. Each entry is a label name mapped to the set of allowed values; an alert is ingested only if it matches all of the listed labels. #### Mapping alert severity Grafana alerts carry a severity label (commonly `severity`). Use `alert_severity` to normalize your severity values to Causely severity levels (`critical`, `major`, `minor`, `info`). ```yaml scrapers: grafana: alert_severity: label_name: severity mappings: - value: critical level: critical - value: warning level: major ``` #### Mapping alerts to entities and Signals By default, Causely maps error- and latency-type alerts to the corresponding Service Signals using keyword matching. Use `alerts` to explicitly map a Grafana alert to a Causely Signal on a specific entity. For example, the following maps the `HighDBOpenConnections` alert to the `DBConnectionsUtilization_High` symptom of the matching workload: ```yaml scrapers: grafana: alerts: - alert_name: 'HighDBOpenConnections' symptom: 'DBConnectionsUtilization_High' entity: workload: {} discovery: - kubernetes_pod: namespace: 'namespace' pod_name: 'pod' ``` The `discovery` block resolves the entity from the alert's labels: here, the `namespace` and `pod` labels on the alert identify the Kubernetes pod (and therefore its workload) the symptom is attached to. ### Loki Integration Causely integrates with **Loki** to retrieve logs already centralized in your Grafana environment. When configured, Causely automatically surfaces relevant logs in the context of active Diagnoses and service malfunctions. This enables rapid validation of issues and faster time to understanding and resolution. #### When to Use Use this integration when: - Your logs are shipped to Loki instead of remaining in the Kubernetes API. - You want Causely to display relevant log lines and exceptions alongside detected service degradations or Diagnoses. - You prefer a centralized, scalable log pipeline already managed through Grafana Cloud or self-hosted Loki. #### Benefits - **Contextual Insight**: Automatically surfaces logs correlated with active Diagnoses or degraded services. - **Accelerated RCA**: Shows container log lines, stack traces, and error spikes precisely around the time of failure. - **Unified View**: Displays Loki logs directly within Causely, alongside metrics and traces for the same service. - **Operational Efficiency**: Reduces reliance on the Kubernetes API for log collection. #### Configuration **Configure Causely:** ```yaml scrapers: kubernetes: loki_endpoint: 'http://loki.monitoring:3100' loki_tenant: 'production' ``` Causely automatically uses Loki as the preferred external log source with an optional tenant id if used in a multitenant loki configuration. #### Supported Log Types - Container logs (stdout/stderr) #### Result When Causely detects a service malfunction or identifies a [Diagnosis](/reference/root-causes/), it automatically retrieves related logs from Loki. These logs appear: - **Under affected services**, when they exhibit abnormal behavior such as elevated errors or latency. - **Alongside Diagnoses**, showing relevant exceptions or stack traces at the time of failure. This provides clear evidence for what went wrong and why, dramatically shortening investigation and resolution times. --- ## groundcover [groundcover](https://www.groundcover.com/product/application-performance-monitoring) is an eBPF-based application performance monitoring solution that automatically collects telemetry data from your applications without requiring code instrumentation or configuration changes. ### How groundcover works groundcover uses eBPF (Extended Berkeley Packet Filter) technology to automatically capture: - **HTTP and gRPC requests** - Service-to-service communication - **Database queries** - SQL operations and connection details - **Messaging operations** - Kafka, RabbitMQ, and other messaging systems - **Service dependencies** - Automatic discovery of service relationships - **Performance metrics** - Latency, throughput, and error rates To learn more about how groundcover works, see the [groundcover documentation](https://docs.groundcover.com/). ### Integration with Causely groundcover sends collected telemetry data to Causely's mediator using the OpenTelemetry protocol (OTLP). This enables Causely to: - Analyze service dependencies and communication patterns - Identify actual causes of performance issues and outages - Correlate events across your entire infrastructure - Provide automated incident response and resolution ### Configuration To enable groundcover as a data source for Causely, add the following configuration to your groundcover installation: ```yaml title="groundcover-values.yaml" opentelemetry-collector: config: exporters: otlp/causely: endpoint: mediator.causely:4317 tls: insecure: true insecure_skip_verify: true service: pipelines: traces/causely: exporters: - otlp/causely processors: - batch/traces receivers: - otlp - datadog - faro ``` ### Installation Install groundcover with the Causely integration enabled: ```shell helm install groundcover groundcover/groundcover --values ./groundcover-values.yaml ``` Once installed, groundcover will automatically begin collecting telemetry data and forwarding it to Causely for causal reasoning. --- ## incident.io Integration ### Overview Causely can automatically ingest alerts from [incident.io](https://incident.io) and map them to Signals and entities in your system with minimal configuration required. The integration automatically processes incident.io alerts by: - Detecting Signals from alert titles and descriptions - Using Kubernetes labels (namespace, pod, service) for entity mapping ### Configuration #### Basic Setup To enable the incident.io integration, add the following to your `causely-values.yaml` file: ```yaml title="causely-values.yaml" scrapers: incident: enabled: true instances: - secretName: 'incident-io-credentials' namespace: 'causely' ``` Create a Kubernetes Secret with the incident.io API token: ```bash kubectl create secret generic incident-io-credentials \ --namespace causely \ --from-literal=token=your-incident-io-api-token ``` #### Alternative: Enable Credentials Autodiscovery Causely also supports credentials autodiscovery. This feature allows you to add new scraping targets without updating the Causely configuration. First, enable autodiscovery in your configuration: ```yaml title="causely-values.yaml" scrapers: incident: enabled: true auto_discovery: enabled: true ``` Then create a Kubernetes secret with the required label: ```bash kubectl create secret generic incident-io-credentials \ --namespace causely \ --from-literal=token=your-incident-io-api-token kubectl label secret incident-io-credentials \ --namespace causely \ "causely.ai/scraper=Incident" ``` --- ## Instana ### Overview Causely connects to [IBM Instana Observability](https://www.ibm.com/products/instana) through a locally deployed mediation layer. The mediation layer reads only the signals needed from Instana, discovers topology, and detects Signals in your environment. The mediation layer converts Instana signals into a compact stream of Signals and topology deltas, then sends only those distilled insights to Causely's causal reasoning engine (CRE) for analysis. The engine continuously infers causes, evaluates blast radius across services, endpoints, databases, and SLOs, and provides context-rich explanations with guided remediation so responders move directly from detection to resolution. ### Key capabilities: - Automatic causal inference pinpointing the Diagnosis and risk in real time. Causely’s CRE continuously ingests Instana signals to pinpoint the Diagnosis behind service degradations and failures, surfacing emerging risks before they become incidents. - Automatic impact analysis and blast radius mapping. Causely’s CRE continuously infers the impacted services, endpoints, databases, and SLOs to drive the right prioritization. - Automatic remediation. Causely provides context‑rich explanations of actual causes and can automatically remediate by applying the specific fix at runtime, configuration, or code level. - Fast, low‑lift enablement. Connect Instana via API and start diagnosing in minutes with no changes to your services. ### Setup Guide #### Step 1: Create an API token in Instana Create an API token for your Instana account with read access to applications, endpoints, metrics, and infrastructure. #### Step 2: Create a Kubernetes Secret Create a secret with your Instana account URL and API token: ```bash kubectl create secret generic \ --namespace causely instana-credentials \ --from-literal=url="https://your-instana.example.com" \ --from-literal=api-token="" # Optional (only if necessary for your environment): # --from-literal=insecure-skip-verify="true" ``` Alternatively, you can create the secret using a YAML manifest: ```yaml apiVersion: v1 kind: Secret metadata: name: instana-credentials namespace: causely type: Opaque stringData: url: 'https://your-instana.example.com' api-token: '' # insecure-skip-verify: 'true' # optional ``` #### Step 3: Update Causely configuration Enable the Instana scraper and reference your secret: ```yaml scrapers: instana: enabled: true accounts: - secretName: instana-credentials # namespace: your-namespace # optional; defaults to the mediator pod namespace ``` #### Alternative: Enable Credentials Autodiscovery Causely supports credentials autodiscovery so you can add new accounts without editing config. Label the Kubernetes secret to enable autodiscovery for the Instana scraper: ```bash kubectl --namespace causely label secret instana-credentials "causely.ai/scraper=Instana" ``` ### What Data is Collected The Instana scraper collects and maps the following into Causely: - **Service and dependency graph**: Instana application services and service-map connections, aligned to Kubernetes workloads and services - **Endpoints**: HTTP paths and RPC methods derived from Instana endpoints - **Endpoint metrics**: Requests, errors, and latency percentiles (p90, p95, p99, max) attached to endpoints - **Cross-service and cross-endpoint calls**: Observed calls with error counts and latency distributions used to ground edges in behavior - **Pod-to-service and service-to-pod metrics**: Access metrics that attribute consumption and exposure per workload - **Database insights**: Queries grouped by statement with call volumes and latency; access attributed to calling services/endpoints and mapped to database/table entities - **Kubernetes metadata**: Namespace and cluster labels added to services for context --- ## Istio #### Istio - Prometheus The Istio integration works by scraping the Istio relevant data from a Prometheus server. To enable Istio as a data source, please add the following section to your values.yaml file: ```yaml scrapers: istio: enabled: true prometheus: endpoint: http://prometheus.istio-system.svc.cluster.local:9090 ``` Please ensure that the Prometheus instance scrapes all your Istio sidecars to capture all the service-to-service communication metrics. #### Istio - w/o Prometheus The Istio integration works by scraping the Istio relevant data directly from the istio sidecars. To enable Istio as a data source, please add the following section to your values.yaml file: ```yaml scrapers: istiosidecar: enabled: true ``` --- ## Kubernetes ### Overview Causely provides **native, out-of-the-box integration** with [Kubernetes](https://kubernetes.io/) that automatically discovers and monitors your entire container infrastructure. When you install Causely using our standard [Helm](/installation/helm) or [CLI](/installation/cli) installation, Kubernetes monitoring is enabled by default with zero configuration required. The integration provides comprehensive visibility across your entire Kubernetes stack, from cluster-level resources down to individual containers, helping you identify infrastructure issues before they impact your applications. ### How It Works Causely's Kubernetes integration works automatically upon installation: 1. **Automatic Discovery**: The agent automatically discovers all Kubernetes resources in your cluster using the Kubernetes API 2. **Real-time Monitoring**: Continuously monitors resource states, events, and relationships 3. **Entity Modeling**: Creates a comprehensive topology graph showing relationships between clusters, nodes, pods, services, and applications 4. **Event Processing**: Analyzes Kubernetes events to detect issues like pod evictions, scheduling problems, and resource constraints 5. **Diagnosis**: Correlates infrastructure issues with application performance problems 6. **Auto-remediation**: Remediate issues automatically by deploying fixes or scaling resources This approach provides complete infrastructure visibility without requiring any configuration changes or additional setup beyond the standard Causely installation. ### Out-of-the-Box Setup Kubernetes monitoring is **enabled by default** when you install Causely. Simply follow our standard installation guide: 📦 Install with Helm or 💻 Install with CLI No additional configuration, secrets, or permissions are required beyond what's included in the standard installation. ### Enhanced Log Retrieval Causely surfaces relevant log lines automatically in the context of active Diagnoses and service malfunctions. For environments with centralized logging, Causely can retrieve container logs from external sources instead of the Kubernetes API. This is useful when logs are forwarded to systems like Elasticsearch or Grafana Loki. #### External Log Integrations Causely supports two primary centralized log integrations: - **[Elasticsearch](/telemetry-sources/elasticsearch#use-case-1-log-retrieval-from-elasticsearch)**: Retrieve logs directly from Elasticsearch when your containers push logs there. - **[Grafana Loki](/telemetry-sources/grafana#loki-integration)**: Integrate with Loki for scalable log aggregation and querying through Grafana. When either integration is configured, Causely prioritizes external log sources over the Kubernetes API and automatically falls back to Kubernetes logs if the external system becomes unavailable. ### What You Get #### Infrastructure Topology - **Complete service map** showing relationships between applications, services, and infrastructure - **Multi-layer visualization** from business applications down to individual containers - **Dependency tracking** across namespaces and resource types #### Workload Monitoring - **Controller analysis** for Deployments, StatefulSets, DaemonSets, Jobs, and CronJobs - **Pod lifecycle tracking** including scheduling, running, and termination states - **Container health monitoring** with resource usage and state information - **Configuration change detection** for container images, resources, environment variables, and volume mounts #### Resource Management - **Node health monitoring** with conditions like memory pressure, disk pressure, and network availability - **Persistent volume tracking** with usage and binding information - **Service discovery** with automatic endpoint creation and load balancer mapping - **Ingress routing** analysis for external traffic patterns #### Event Analysis - **Pod eviction detection** for memory pressure, disk pressure, and resource constraints - **Scheduling failure analysis** for unschedulable pods - **Image pull error tracking** for deployment issues - **Configuration change events** for version updates and resource modifications #### Application Integration - **Service-to-pod mapping** for application relationship discovery - **Network endpoint creation** for service communication analysis - **Load balancer and ingress integration** for external access patterns - **Kafka resource discovery** for message queue topology (if using Strimzi operator) #### Enhanced Log Retrieval - **Multi-source log collection** from Kubernetes API, Elasticsearch, and Loki - **Elasticsearch integration** for centralized log analysis when logs are forwarded from containers - **Loki integration** for scalable log aggregation and querying - **Priority-based retrieval** with automatic fallback to Kubernetes API - **Authentication support** for secured log aggregation systems ### Root Cause Detection The Kubernetes integration enables detection of infrastructure-related causes including: #### Node-Level Issues - [**Disk Pressure**](/reference/root-causes/infrastructure#disk-pressure) - Node disk usage triggering pod evictions - [**Memory Pressure**](/reference/root-causes/infrastructure#memory-pressure) - Node memory exhaustion causing pod evictions #### Controller and Workload Issues - [**Controller Malfunction**](/reference/root-causes/infrastructure#malfunction) - Multiple pods in NotReady state - [**Image Pull Errors**](/reference/root-causes/infrastructure#image-pull-errors) - Pods failing to start due to registry issues - [**FrequentPodEphemeralStorageEvictions**](/reference/root-causes/infrastructure#frequentpodephemeralstorageevictions) - Pods evicted due to storage limits #### Container Resource Issues - [**CPU Congested**](/reference/root-causes/infrastructure#cpu-congested) - Container CPU throttling and performance degradation - [**Memory Failure**](/reference/root-causes/infrastructure#memory-failure) - Container out-of-memory kills - [**Frequent Memory Failure**](/reference/root-causes/infrastructure#frequent-memory-failure) - Repeated memory-related crashes - [**Crash Failure**](/reference/root-causes/infrastructure#crash-failure) - Container crashes with non-zero exit codes - [**Frequent Crash Failure**](/reference/root-causes/infrastructure#frequent-crash-failure) - Repeated container crashes #### Storage and Noisy Neighbor Issues - [**Ephemeral Storage Congested**](/reference/root-causes/infrastructure#ephemeral-storage-congested) - Container storage usage causing failures - [**Ephemeral Storage Noisy Neighbor**](/reference/root-causes/infrastructure#ephemeral-storage-noisy-neighbor) - Container consuming excessive storage affecting node - [**Memory Noisy Neighbor**](/reference/root-causes/infrastructure#memory-noisy-neighbor) - Container consuming excessive memory affecting node - [**Disk Congested**](/reference/root-causes/infrastructure#congested) - Persistent volumes reaching capacity limits #### Service-Level Issues - [**Service Congested**](/reference/root-causes/services#congested) - Kubernetes services experiencing high latency - [**Service Malfunction**](/reference/root-causes/services#malfunction) - Kubernetes services with high error rates #### Release-Related Issues - [**Code Change Regression: CPU Congestion**](/reference/root-causes/release#code-change-regression-cpu-congestion) - Performance degradation after deployments - [**Code Change Regression: Memory Failure**](/reference/root-causes/release#code-change-regression-memory-failure) - Memory issues introduced by new versions ### What Data is Collected The Kubernetes integration automatically collects comprehensive metadata and state information, including: #### Cluster-Level Resources - **Cluster identity** and configuration - **Node specifications** and health conditions - **Namespace organization** and resource quotas - **Custom resource definitions** and operators #### Workload Resources - **Pod specifications** including containers, volumes, and resource requirements - **Controller configurations** for Deployments, StatefulSets, DaemonSets, Jobs, and CronJobs - **Service definitions** and endpoint mappings - **Ingress rules** and traffic routing configuration #### Runtime Information - **Container states** and restart counts - **Resource utilization** and capacity limits - **Event logs** for troubleshooting and analysis - **Container logs** from Kubernetes API, Elasticsearch, or Loki (when configured) - **Configuration changes** and version history #### Relationships and Dependencies - **Service-to-pod mappings** for application topology - **Pod-to-node assignments** for infrastructure placement - **Volume bindings** and storage relationships - **Network connectivity** patterns and endpoints :::note The Kubernetes integration respects your cluster's RBAC policies and only accesses resources that the Causely service account has permissions to read. All data collection uses read-only Kubernetes API calls. ::: --- ## MongoDB Atlas ### Overview Causely provides native integration with [MongoDB Atlas](https://www.mongodb.com/atlas) to help you identify and resolve database performance issues before they impact your users. Instead of just monitoring signals, Causely analyzes them in real time from the Atlas Monitoring API to surface the actual diagnosis behind database problems. This integration helps you identify the following causes for reliability issues, among others: - [Connection exhaustion](/reference/root-causes/services#congested) from connection pool saturation across Atlas shards - [Replication lag](/reference/root-causes/services#malfunction) causing stale reads on secondary nodes - [Service congestion](/reference/root-causes/services#congested) affecting Atlas cluster performance - [Service malfunction](/reference/root-causes/services#malfunction) in database cluster components The integration supports MongoDB Atlas clusters and automatically maps Atlas shard processes to the Causely service topology, enabling end-to-end tracing from your application to the Atlas cluster. ### Step 1: Create an Atlas API Key Create a programmatic API key for your MongoDB Atlas project with the necessary permissions: 1. Log in to the [MongoDB Atlas Console](https://cloud.mongodb.com/) 2. Navigate to your **Organization** → **Access Manager** → **API Keys** 3. Click **Create API Key** 4. Set a description and assign the **Project Read Only** role (minimum required) 5. Copy and securely store the **Public Key** and **Private Key** 6. Add your Causely mediator's IP address to the API key access list :::note You will also need the **Project ID** (also called Group ID) from your Atlas project. Navigate to your project → **Settings** to find it. ::: ### Step 2: Configure the Integration #### Add data source from the UI You can add MongoDB Atlas as a data source from the Causely UI. Go to [Integrations](https://portal.causely.app/integrations), add the MongoDB Atlas integration, enter your **Public Key**, **Private Key**, and **Project ID**, and select the cluster to which the configuration will be pushed. #### Basic setup (ops as code) If you prefer to manage configuration as code, create a Kubernetes Secret containing your Atlas API credentials and label it for autodiscovery: ```bash kubectl create secret generic atlas-credentials \ --namespace causely \ --from-literal=publicKey="" \ --from-literal=privateKey="" \ --from-literal=groupId="" kubectl label secret atlas-credentials \ --namespace causely \ causely.ai/scraper=MongoAtlas ``` Alternatively, create the secret using a YAML manifest: ```yaml apiVersion: v1 kind: Secret metadata: name: atlas-credentials namespace: causely labels: causely.ai/scraper: MongoAtlas type: Opaque stringData: publicKey: '' privateKey: '' groupId: '' ``` Secret keys: - **`publicKey`** (required): MongoDB Atlas programmatic API public key. Also accepted as `public_key`. - **`privateKey`** (required): MongoDB Atlas programmatic API private key. Also accepted as `private_key`. - **`groupId`** (required): MongoDB Atlas project ID (Group ID). Also accepted as `group_id`, `projectId`, or `project_id`. To monitor multiple Atlas projects, create one secret per project and label each with `causely.ai/scraper=MongoAtlas`. ### What Data is Collected The MongoDB Atlas scraper collects comprehensive metadata and performance information from your Atlas deployment, including: - **Cluster entities** with display names and labels (`causely.ai/service-type=MongoDB Atlas Cluster`) - **Shard process entities** (Workloads) for each mongod/mongos instance, linked to their parent cluster - **Network endpoints** for all client-facing hostnames (both the SRV parent and the individual shard seed hosts), enabling automatic topology stitching from application services to the Atlas cluster - **Connection metrics** per shard: current connections and available connections (derived from the cluster tier's connection limit) - **Replication lag** per shard (`OPLOG_REPLICATION_LAG_TIME`) - **Open cursors** per shard (`CURSORS_TOTAL_OPEN`) --- ## MySQL ### Overview Causely provides native integration with [MySQL](https://www.mysql.com/) to help you identify and resolve database performance issues before they impact your users. Instead of just monitoring Signals, Causely analyzes them in real time to surface the actual Diagnosis behind database problems. This integration helps you identify the following causes for reliability issues, among others: - [Table malfunction](/reference/root-causes/datapipeline#table-access-failure-in-database) and [excessive locks](/reference/root-causes/datapipeline#contention-on-database-table-locks) - [Slow database queries](/reference/root-causes/applications#slow-database-queries) - [Connection exhaustion](/reference/root-causes/applications#database-connection-pool-saturated) The integration supports both self-hosted MySQL instances and cloud-managed services including AWS RDS MySQL, Azure Database for MySQL, and Google Cloud SQL for MySQL. ### Setup Guide #### Step 1: Create a user Create a user for your MySQL database server with the following permissions: - `PROCESS`: Required to view running queries and process information - `REPLICATION CLIENT`: Required to monitor replication status and transaction information - `SELECT`: Required to query performance metrics, table information, and schema details - `SHOW VIEW`: Required to access view definitions and metadata #### Step 2: Enable performance monitoring features Enable performance monitoring features in your MySQL instance to collect detailed metrics and query information. This step varies depending on your MySQL deployment type, as detailed below. ##### Native MySQL For native MySQL installations, you need to enable the performance schema: 1. Edit your MySQL configuration file (typically `/etc/mysql/my.cnf` or `/etc/my.cnf`) 2. Add or update these settings: ```ini [mysqld] performance_schema=ON ``` ##### AWS RDS MySQL - [AWS RDS MySQL Documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_MySQL.html) - [Performance Insights in AWS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PerfInsights.html) - [Enable Performance Insights for Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PerfInsights.Enabling.html) Steps: 1. Go to RDS Console → Databases 2. Select your MySQL instance 3. Click "Modify" 4. Under "Performance Insights", enable "Enable Performance Insights" 5. Choose retention period (7 days or more) 6. Click "Continue" and apply changes immediately ##### Azure Database for MySQL - [Azure MySQL Documentation](https://learn.microsoft.com/en-us/azure/mysql/) - [Query Performance Insights in Azure](https://learn.microsoft.com/en-us/azure/mysql/flexible-server/tutorial-query-performance-insights) Steps: 1. Go to Azure Portal → Select your Azure Database for MySQL Flexible Server instance. 2. On the left pane, under Settings, select Server parameters. 3. For the slow_query_log parameter, select ON. 4. For the other parameters, such as long_query_time and log_slow_admin_statements, refer to the slow query logs documentation. 5. Click "Save" ##### Google Cloud SQL for MySQL - [Cloud SQL MySQL Documentation](https://cloud.google.com/sql/docs/mysql) - [Query Insights in GCP](https://cloud.google.com/sql/docs/mysql/using-query-insights) Steps: 1. Go to Cloud Console → Cloud SQL 2. Select your MySQL instance 3. Click "Edit" 4. Under "Query Insights", enable "Enable Query Insights" 5. Choose retention period (7 days or more) 6. Click "Save" #### Step 3: Create a Kubernetes secret for the user After creating the user create a Kubernetes secret. You have two options: ##### Option 1: Single Database Configuration ```bash kubectl create secret generic \ --namespace causely mysql-credentials \ --from-literal=username="..." \ --from-literal=password='...' \ --from-literal=host="..." \ --from-literal=port=3306 \ --from-literal=database="..." \ --from-literal=sslmode="disabled" ``` `host` must be a hostname or IP address, not a URL. Do not include a `mysql://` scheme. `sslmode` accepted values: - `disabled`—no TLS (typical for in-cluster MySQL) - `preferred`—try TLS, fall back to plaintext (default) - `required`—TLS without certificate verification - `verify-ca` / `verify-identity`—TLS with certificate verification ##### Option 2: Multiple Databases Configuration For monitoring multiple databases within the same MySQL instance, you can specify multiple databases using the `databases` field: ```bash kubectl create secret generic \ --namespace causely mysql-credentials-multidb \ --from-literal=username="..." \ --from-literal=password='...' \ --from-literal=host="..." \ --from-literal=port=3306 \ --from-literal=databases="database1,database2,database3" \ --from-literal=sslmode="disabled" ``` Alternatively, you can create the secret using a YAML manifest: ```yaml apiVersion: v1 kind: Secret metadata: name: mysql-credentials-multidb namespace: causely type: Opaque stringData: username: '...' password: '...' host: '...' port: '3306' databases: 'database1,database2,database3' sslmode: 'disabled' ``` **Note**: Use either the `database` field for single database configuration or the `databases` field for multiple databases. Do not use both fields in the same secret. The `host` must be the FQDN of your DB, or IP address if there's no DNS entry set up. Do not include a `mysql://` scheme. It must match the FQDN/IP Causely would discover either from the K8s Server (if it's running in your K8s Cluster) or your Cloud providers API. Examples: - _Kubernetes_: `host=my-mysql.namespace.service.cluster.local` - _AWS_: `host=myinstance.rds.amazonaws.com` If you are leveraging a proxy to connect to your database, like it's common on GCP, `host` must be the FQDN/IP of your proxy and additionally `host_overwrite` should be the IP/FQDN of the actual database service: - _GCP_: `host=localhost` (assuming proxy runs on the same host / Pod). `host_overwrite=1.2.3.4` (the IP of your GCP Cloud SQL instance as shown in the GCP Console) #### Step 4: Update Causely Configuration Once the secret is created, update the Causely configuration to enable scraping for the new database. Below is an example configuration: ```yaml scrapers: mysql: enabled: true instances: - secretName: mysql-credentials namespace: causely ``` #### Alternative: Enable Credentials Autodiscovery Causely also supports credentials autodiscovery. This feature allows you to add new scraping targets without updating the Causely configuration. Label the Kubernetes secret to enable autodiscovery for the corresponding scraper. ```bash kubectl --namespace causely label secret mysql-credentials "causely.ai/scraper=MySQL" ``` ### What Data is Collected The MySQL scraper collects comprehensive metadata and performance information from your MySQL databases, including: - **Database entities** with names and relationships to hosting services - **Service-to-database mappings** (which service provides which database) - **Connection details** including host, port, and SSL configuration - **Database and table schemas** with complete structural information - **Table information** including names, row counts, and sizes - **Complete table schemas** with column definitions, data types, constraints, and indexes - **Foreign key relationships** and table dependencies - **Parent-child table mappings** for dependency analysis - **Slow query analysis** - **Lock monitoring** --- ## Nobl9 Causely can leverage [Nobl9](https://www.nobl9.com/)'s SLO platform to observe service level objectives and reliability metrics. To enable Nobl9 as a data source, please add the following section to your `values.yaml` file: ```yaml scrapers: nobl9: enabled: true accounts: - accountName: secretName: nobl9-secret ``` ### nobl9-secret.yaml ```yaml apiVersion: v1 kind: Secret type: Opaque metadata: name: nobl9-secret namespace: causely stringData: NOBL9_CLIENT_ID: '' NOBL9_CLIENT_SECRET: '' NOBL9_ORGANIZATION: '' ``` ### SLO Labeling Since Nobl9 SLO can be defined for anything, in order for Causely to stitch the right SLO to the right Kubernetes Service, you need to label your SLO in Nobl9 with: ```yaml namespace: service: ``` --- ## Nomad ### Overview Causely provides **native integration** with [HashiCorp Nomad](https://www.nomadproject.io/) that automatically discovers and monitors your workload orchestration infrastructure. The integration provides visibility across your Nomad cluster, from nodes down to individual tasks, helping you identify infrastructure issues before they impact your applications. ### How It Works Causely's Nomad integration automatically discovers nodes, jobs, allocations, and tasks using the Nomad API, monitors job states and resource usage, creates a topology graph showing relationships between nodes, allocations, tasks, jobs, and services, integrates with Consul for service discovery, and analyzes task events to detect OOM kills and crashes. Nomad monitoring works automatically when you install Causely using the [Nomad installation guide](/installation/nomad). --- ## Observe Causely integrates with [Observe](https://www.observeinc.com/) to discover the Kubernetes clusters that send data to your Observe tenant, the application services running in them, and to use their metrics, traces, and container logs. Causely reads this data through the Observe API, so the Causely mediator doesn't need to run in, or have access to, the clusters themselves. Every cluster in your Observe tenant is discovered automatically, and new clusters are picked up without configuration changes. ### How it works Causely reads the datasets that the Observe Agent's Kubernetes Explorer package and Observe APM write: | Observe dataset | Used for | | ---------------------------------------- | ------------------------------------------------------------------------------------------- | | `Kubernetes Explorer/Kubernetes Entity` | Clusters, nodes, pods, containers, controllers, services, persistent volumes, and ingresses | | `Kubernetes Explorer/Prometheus Metrics` | Container and node CPU, memory, and ephemeral storage usage, and node connection tracking | | `Kubernetes Explorer/Kubernetes Logs` | Container logs, used as evidence for diagnoses | | `Tracing/Span` | Application services, their operations, and the calls between them | Causely refreshes the topology every 5 minutes, and metrics and application services on every sync (every 5 minutes by default). Container logs are read from Observe when Causely needs them for a diagnosis. Causely builds the same entities as its native [Kubernetes integration](./kubernetes.mdx), including controllers with their compute specs, persistent volumes linked to the pods that mount them, and Signals from pod and container status such as crashes and out-of-memory kills. #### Clusters that a Causely agent already monitors If a Causely agent already runs in a cluster that also sends data to Observe, Causely keeps the agent's view of that cluster and doesn't rebuild it from Observe. Application services and calls in that cluster are still discovered from Observe traces, and are linked to the entities the agent discovered. #### Application services from Observe APM When your applications send traces to Observe, Causely discovers your application services and how they communicate: - **Services:** each traced service is linked to the Kubernetes service of the pods that emitted its spans. A traced service without a matching Kubernetes service gets its own service in Causely. - **Operations:** HTTP routes (for example, `/api/users/{id}`) and RPC methods, with their request and error counts. Health check endpoints are skipped. - **Dependencies:** calls between services become access edges. Calls to databases add the database to the service that hosts it, and messages sent to or read from Kafka, RabbitMQ, Amazon SNS and SQS, and other messaging systems become topic accesses. - **Metrics:** request rate, error rate, and latency for each service. Causely reads spans from both OpenTelemetry and Datadog instrumentation. It summarizes them in Observe, so raw spans aren't exported. ### Prerequisites - The [Observe Agent](https://docs.observeinc.com/) with the Kubernetes Explorer package, installed on each cluster you want Causely to discover. - To discover application services: traces sent to Observe APM, so the `Tracing/Span` dataset exists in your tenant. - Your Observe customer ID. It's the number in your Observe URL, for example `123456789012` in `https://123456789012.observeinc.com`. - An Observe API token with read access to the Kubernetes Explorer datasets, and to the Tracing datasets if you use Observe APM. ### Step 1: Create a Kubernetes Secret with the Observe credentials Create a Kubernetes Secret with your Observe customer ID and API token. The `url` key is optional and defaults to `https://.observeinc.com`: ```bash kubectl create secret generic \ --namespace causely observe-credentials \ --from-literal=customer-id="..." \ --from-literal=api-token="..." ``` ### Step 2: Update Causely configuration Once the secret is created, update the Causely configuration to enable the Observe scraper: ```yaml scrapers: observe: enabled: true accounts: - secretName: observe-credentials ``` To discover only some of the clusters in your Observe tenant, list them by name: ```yaml scrapers: observe: enabled: true accounts: - secretName: observe-credentials clusters: - production-east - production-west ``` #### Application services Application services are discovered from Observe APM by default. If your traces also reach Causely another way, for example through OpenTelemetry, set `scrapers.observe.service_topology: false` so services and calls aren't discovered twice. #### Alternative: Enable Credentials Autodiscovery Causely supports credentials autodiscovery so you can add new accounts without editing config. Label the Kubernetes secret to enable autodiscovery for the Observe scraper: ```bash kubectl --namespace causely label secret observe-credentials "causely.ai/scraper=Observe" ``` ### Export Issues and Diagnoses to Observe Causely can also push [Issues](/reference/terminology#issue) and [Diagnoses](/reference/terminology#diagnosis) to Observe. For details, see [Observe export](/workflows/observe). --- ## Odigos [Odigos](https://docs.odigos.io/setup/installation) is an open source observability control plane that automatically instruments your applications to generate traces and metrics without requiring code changes. ### How Odigos works Odigos provides automatic instrumentation for your applications by: - **Auto-instrumenting applications** - Automatically adds OpenTelemetry instrumentation to your services - **Discovering service dependencies** - Maps out how your services communicate with each other - **Generating traces and metrics** - Creates distributed traces and application metrics automatically - **Supporting multiple languages** - Works with Java, Python, Node.js, Go, and other languages - **Kubernetes-native** - Designed to work seamlessly in Kubernetes environments To learn more about how Odigos works, see the [Odigos documentation](https://docs.odigos.io/). ### Integration with Causely Odigos sends collected traces and metrics to Causely's mediator service using the OpenTelemetry protocol (OTLP). This enables Causely to: - Analyze service dependencies and communication patterns - Identify actual causes of performance issues and outages - Correlate events across your entire infrastructure - Provide automated incident response and resolution ### Configuration To enable Odigos as a data source for Causely, add the following configuration to your Odigos installation: ```yaml title="odigos-destination.yaml" apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: odigos.io.dest.causely namespace: odigos-system spec: data: CAUSELY_URL: http://mediator.causely:4317 destinationName: causely signals: - TRACES - METRICS type: causely ``` For detailed configuration instructions, see the [Odigos Causely backend documentation](https://docs.odigos.io/backends/causely). ### Installation To install Odigos, please refer to the [Odigos documentation](https://docs.odigos.io/setup/installation). Once Odigos is installed and configured with the Causely destination as described above, it will automatically begin collecting telemetry data from your applications and forwarding it to Causely for causal reasoning. --- ## Integrate OpenTelemetry Traces ### What is OpenTelemetry? OpenTelemetry is an open source, vendor- and tool-neutral project that provides a comprehensive observability framework for generating, exporting and collecting telemetry data, such as traces, metrics and logs. Visit the [OpenTelemetry website](https://opentelemetry.io/) for more information. ### Integrating OpenTelemetry with Causely You can export traces from an application or an existing OpenTelemetry Collector to the [mediator](/getting-started/architecture#mediation-layer) that was installed as part of the Causely agent. The mediator listens for traces on port `4317` using the [OpenTelemetry Protocol](https://opentelemetry.io/docs/specs/otlp/) (OTLP). We recommend using OpenTelemetry with Causely because traces enable automatic discovery of service dependencies and monitoring of both synchronous and asynchronous communication signals. :::tip High trace volume For environments with high trace volume, Causely supports a horizontally scalable trace processing tier. When enabled, collectors send traces to the trace controller (`trace-controller.causely:4317`) instead of the mediator. See [Scalable Trace Processor](/installation/scalable-trace-processor) for details. ::: :::info By default, Causely will use [OpenTelemetry eBPF Instrumentation](/telemetry-sources/ebpf) for automatic instrumentation of your applications. ::: ### How it works The following diagram shows how OpenTelemetry traces and metrics are collected and forwarded to the Causely mediator. For more information on how Causely works, see the [How Causely Works](/getting-started/how-causely-works) page. ```mermaid %%{init: {'theme': 'base', 'themeVariables': { 'fontSize': '16px', 'fontFamily': 'arial' }}}%% graph LR subgraph kubernetes["Kubernetes Cluster"] OtelCollector["OpenTelemetry Collector"] App1["Application 1"] App2["Application 2"] App3["Application 3"] Mediator["Mediator
(Listening on port 4317)"] end CRE["Causal Reasoning Engine"] Backend1["Observability
Backend"] App1 -->|"traces,logs,metrics"| OtelCollector App2 -->|"traces,logs,metrics"| OtelCollector App3 -->|"traces,logs,metrics"| OtelCollector OtelCollector -->|"traces,metrics"| Mediator Mediator -->|"distilled insights"| CRE OtelCollector -..->|"full telemetry data"| Backend1 classDef application fill:#36c5f0,stroke:#333,stroke-width:1px; classDef collector fill:#4F46E5,stroke:#333,stroke-width:1px,color:#fff; classDef mediator fill:#00B67A,stroke:#333,stroke-width:1px; classDef engine fill:#0066CC,stroke:#333,stroke-width:1px,color:#fff; classDef backend fill:#4a154b,stroke:#333,stroke-width:1px,color:#fff; classDef cluster fill:#EBF3FA,stroke:#333,stroke-width:1px; class App1,App2,App3 application; class OtelCollector collector; class Mediator mediator; class CRE engine; class Backend1 backend; class kubernetes cluster; ``` :::tip Good to know No raw data is sent from the mediator to the Causely engine, only distilled insights. This way your data is secure and you can be sure that no sensitive data is sent to us. ::: ### Quick Start Guide If you don't have an OpenTelemetry Collector running in your Kubernetes cluster, you can use the following command to install the OpenTelemetry Operator and the OpenTelemetry Collector: ```shell helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts helm install opentelemetry-collector open-telemetry/opentelemetry-collector --values ./opentelemetry-values.yaml ``` For values of `opentelemetry-values.yaml` see the [OpenTelemetry Operator Configuration section](#opentelemetry-operator-configuration) below. ### OpenTelemetry Collector Configuration For an instance of the [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) running within your Kubernetes cluster, you can use the following configuration: ```yaml exporters: otlp_grpc/causely: endpoint: mediator.causely:4317 compression: none tls: insecure: true processors: batch: timeout: 1s k8s_attributes: auth_type: 'serviceAccount' passthrough: false extract: metadata: - k8s.pod.name - k8s.pod.uid - k8s.container.name - k8s.deployment.name - k8s.namespace.name - k8s.node.name - k8s.pod.start_time # Optional: Filter out internal spans filter/ignore-internal: error_mode: ignore traces: span: - 'kind.string == "Internal"' receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 prometheus/metrics: # See a more detailed example in the Configuring Prometheus Metrics Relabeling section below service: pipelines: metrics: exporters: [otlp_grpc/causely] processors: [k8s_attributes, batch] receivers: [otlp, prometheus/metrics] traces: exporters: [otlp_grpc/causely] processors: [filter/ignore-internal, k8s_attributes, batch] receivers: [otlp] ``` #### Configuration Breakdown This configuration will: - Export metrics and traces to the Causely mediator using the [OTLP gRPC Exporter](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/otlpexporter/README.md) - Filter out internal spans using the [Filter Processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/filterprocessor/README.md) - Add Kubernetes attributes to the spans using the [Kubernetes Attributes Processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/k8sattributesprocessor/README.md) - Batch the spans using the [Batch Processor](https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/batchprocessor/README.md) #### Configuring Prometheus Metrics Relabeling When using the `prometheus/metrics` receiver in the OpenTelemetry Collector, you need to configure relabeling to extract Kubernetes metadata labels. Add the following `relabel_configs` to your `prometheus/metrics` receiver configuration: ```yaml prometheus/metrics: config: scrape_configs: - job_name: 'prometheus-metrics' relabel_configs: # Filter by pod annotations - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep regex: true # Configure metrics path - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ regex: (.+) # Configure metrics port - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port] action: replace regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 target_label: __address__ # Map pod labels - action: labelmap regex: __meta_kubernetes_pod_label_(.+) # Extract metadata attributes # Pod name - source_labels: [__meta_kubernetes_pod_name] action: replace target_label: k8s.pod.name # Pod UID - source_labels: [__meta_kubernetes_pod_uid] action: replace target_label: k8s.pod.uid # Container name (set both container_name and k8s.container.name) - source_labels: [__meta_kubernetes_pod_container_name] action: replace target_label: container_name - source_labels: [__meta_kubernetes_pod_container_name] action: replace target_label: k8s.container.name # Deployment name (from controller) - source_labels: [__meta_kubernetes_pod_controller_name] action: replace target_label: k8s.deployment.name # Namespace - source_labels: [__meta_kubernetes_namespace] action: replace target_label: k8s.namespace.name # Node name - source_labels: [__meta_kubernetes_pod_node_name] action: replace target_label: k8s.node.name # Pod start time (creation timestamp) - source_labels: [__meta_kubernetes_pod_creation_timestamp] action: replace target_label: k8s.pod.start_time ``` This relabeling configuration maps Kubernetes metadata to metric labels: - `__meta_kubernetes_namespace` → `k8s.namespace.name` - `__meta_kubernetes_pod_name` → `k8s.pod.name` - `__meta_kubernetes_pod_container_name` → `k8s.container.name` - `__meta_kubernetes_pod_controller_name` → `k8s.deployment.name` **Important**: OpenTelemetry uses the `k8s.*` label format (for example `k8s.namespace.name`, `k8s.pod.name`), while Prometheus typically uses simpler label names (for example `namespace`, `pod`). When configuring Causely discovery for metrics from the OpenTelemetry Collector, ensure your discovery configuration matches the actual label names in your metrics (for example use `namespace: "k8s.namespace.name"` and `pod_name: "k8s.pod.name"`). ### OpenTelemetry Operator Configuration If you are using the [OpenTelemetry Operator for Kubernetes](https://opentelemetry.io/docs/platforms/kubernetes/operator/), you can use the following configuration: ```yaml # Valid values are "daemonset" and "deployment". # If set, agentCollector and standaloneCollector are ignored. mode: 'deployment' config: exporters: otlp_grpc/causely: endpoint: mediator.causely:4317 compression: none tls: insecure: true processors: batch: timeout: 1s k8s_attributes: auth_type: 'serviceAccount' passthrough: false extract: metadata: - k8s.pod.name - k8s.pod.uid - k8s.container.name - k8s.deployment.name - k8s.namespace.name - k8s.node.name - k8s.pod.start_time # Optional: Filter out internal spans filter/ignore-internal: error_mode: ignore traces: span: - 'kind.string == "Internal"' receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 prometheus/metrics: config: scrape_configs: - job_name: 'prometheus-metrics' relabel_configs: # See Configuring Prometheus Metrics Relabeling section above service: pipelines: metrics: exporters: [otlp_grpc/causely] processors: [k8s_attributes, batch] receivers: [otlp, prometheus/metrics] traces: exporters: [otlp_grpc/causely] processors: [filter/ignore-internal, k8s_attributes, batch] receivers: [otlp] presets: kubernetesAttributes: enabled: true podLabels: sidecar.istio.io/inject: 'disabled' ``` For the complete `relabel_configs` configuration, see the [Configuring Prometheus Metrics Relabeling](#configuring-prometheus-metrics-relabeling) section above. --- ## Connect Telemetry Sources for Causal Analysis :::important Traces Are Essential Causely requires **traces** to discover service dependencies and perform effective causal analysis. Without traces, the platform's ability to provide value is severely limited. Search for sources that provide **Traces** to ensure you have comprehensive trace coverage. ::: ### UI-based configuration You can configure telemetry sources from the UI. Navigate to [https://portal.causely.app/integrations](https://portal.causely.app/integrations) to set up the integrations you need. You can then select the Mediator to which the configuration will be pushed down. :::tip Missing a telemetry source? On that same page, click on `+ Request a new integration` to file a request to add the integration you need. ::: To ensure that Causely can reason over a wide range of Signals and causes, that explain those Signals, in your environment, it is recommended that you configure additional data sources if they are available. Telemetry sources are organized into categories based on their purpose and the types of inputs they provide to Causely's causal reasoning engine. ### Telemetry Types Different sources provide different types of telemetry to Causely's causal reasoning engine: ### Telemetry Sources by Categories #### Infrastructure Scraper Discovers infrastructure resources such as VMs, containers, clusters, databases, load balancers, queues, and disks. It imports their topology, metadata, and health indicators, forming the backbone of the causality graph Causely uses for causal mapping of infrastructure-level failure modes. **Inputs Provided to Causely**: Metrics, Infrastructure Entities, Signals, sometimes Service Discovery, sometimes Logs #### Logs Ingests log-derived signals (errors, exceptions, warnings) used for Signal activation based on defined error patterns and enhanced descriptions and remediation after a Diagnosis has been inferred. **Inputs Provided to Causely**: Logs #### Metrics & Signals Provides quantitative telemetry (latency, saturation, throughput, resource usage) and domain-specific metrics (for example, Redis, RabbitMQ). Connects Causely to critical data systems (databases, caches, search systems, message queues), pulling topology, performance attributes, and state indicators. These signals populate Causely's Attribute Dependency Graph and reveal bottlenecks and Diagnoses within data infrastructure. **Inputs Provided to Causely**: Metrics, Infrastructure Entities, Signals #### Service Communication Provides traces and metrics through standard OTEL-based data pipelines (eBPF, Collectors, dual-shipping from Datadog, Odigos, groundcover, etc.). These sources enrich Causely's service graph and enable call-level causal reasoning. **Inputs Provided to Causely**: Traces, Metrics, Signals #### SLO Providers Supplies SLO (Service Level Objective) definitions and burn rate information (for example, from Nobl9). Causely uses SLOs as high-priority Signals and impact indicators in causal reasoning. **Inputs Provided to Causely**: SLOs #### Signal Activation Allows Causely to treat external alert triggers (for example, from Datadog Monitors, Checkly, Alertmanager) as explicit signals within the causality graph. This helps Causely analyze "why this alert fired" and resolve noisy or ambiguous alerts. **Inputs Provided to Causely**: Signals ### Search and Filter Telemetry Sources To search and filter telemetry sources by name, category, or signal type, go to the [Supported Technologies](/getting-started/supported-technologies) page. --- ## PostgreSQL ### Overview Causely provides native integration with [PostgreSQL](https://www.postgresql.org/) to help you identify and resolve database issues before they impact your users. Instead of just monitoring Signals, Causely analyzes them in real time to surface the underlying Diagnosis driving database issues. By setting up the PostgreSQL integration, you will be able to do the following: - Identify causes for reliability issues originating from your PostgreSQL database, including: - [Table malfunction](/reference/root-causes/datapipeline#table-access-failure-in-database) and [excessive locks](/reference/root-causes/datapipeline#contention-on-database-table-locks) - [Slow database queries](/reference/root-causes/applications#slow-database-queries) - [Connection exhaustion](/reference/root-causes/applications#database-connection-pool-saturated) - Observe the database as an entity in the Topology Graph, including it's relationships to other entities on the service map, infrastructure stack and dataflow map. - Get insights into the slowest SQL queries, and troubleshoot them with our [MCP Server](/agent-integration/mcp-server/) directly from your favorite agent. The integration supports both self-hosted PostgreSQL instances and cloud-managed services including AWS RDS PostgreSQL, Azure Database for PostgreSQL, and Google Cloud SQL for PostgreSQL. ### Setup Guide #### Step 1: Create a user Create a user for your PostgreSQL database server with the following permissions: - `pg_read_all_stats`: Required to access statistics views and performance data - `pg_stat_statements`: Required to monitor query performance (if using pg_stat_statements extension) - `SELECT`: Required to query performance metrics, table information, and schema details - `USAGE`: Required to access schema information and metadata #### Step 2: Enable performance monitoring features Enable performance monitoring features in your PostgreSQL instance to collect detailed metrics and query information. This step varies depending on your PostgreSQL deployment type, as detailed below. ##### Native PostgreSQL For native PostgreSQL installations, you need to enable the pg_stat_statements extension: 1. Edit your PostgreSQL configuration file (typically `postgresql.conf`) 2. Add or update these settings: ```ini shared_preload_libraries = 'pg_stat_statements' pg_stat_statements.track = all ``` 3. Restart PostgreSQL 4. Connect to your database and run: ```sql CREATE EXTENSION IF NOT EXISTS pg_stat_statements; ``` ##### AWS RDS PostgreSQL - [AWS RDS PostgreSQL Documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html) - [RDS Parameter Groups](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithParamGroups.html) - [Enabling pg_stat_statements in AWS](https://pganalyze.com/docs/install/amazon_rds/01_configure_rds_instance) Steps: 1. Go to RDS Console → Parameter Groups 2. Edit your parameter group 3. Set `shared_preload_libraries` to include `pg_stat_statements` 4. Restart your instance 5. Connect and run: `CREATE EXTENSION IF NOT EXISTS pg_stat_statements;` ##### Azure Database for PostgreSQL - [Azure PostgreSQL Documentation](https://learn.microsoft.com/en-us/azure/postgresql/) - [Server Parameters in Azure](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/how-to-server-parameters-set-value) - [Enabling pg_stat_statements in Azure](https://pganalyze.com/docs/install/azure_database/01_configure_azure_instance) Steps: 1. Go to Azure Portal → PostgreSQL server 2. Select "Server parameters" 3. Set `shared_preload_libraries` to include `pg_stat_statements` 4. Restart the server 5. Connect and run: `CREATE EXTENSION IF NOT EXISTS pg_stat_statements;` ##### Google Cloud SQL for PostgreSQL - [Cloud SQL PostgreSQL Documentation](https://cloud.google.com/sql/docs/postgres) - [Database Flags](https://cloud.google.com/sql/docs/postgres/flags) - [Enabling pg_stat_statements in GCP](https://pganalyze.com/docs/install/google_cloud_sql/01_create_monitoring_user) Steps: 1. Go to Cloud Console → Cloud SQL 2. Edit your instance 3. Add database flag: `shared_preload_libraries=pg_stat_statements` 4. Restart the instance 5. Connect and run: `CREATE EXTENSION IF NOT EXISTS pg_stat_statements;` #### Step 3: Create a Kubernetes secret for the user After creating the user create a Kubernetes secret. You have two options: ##### Option 1: Single Database Configuration ```bash kubectl create secret generic \ --namespace causely postgres-credentials \ --from-literal=username="..." \ --from-literal=password='...' \ --from-literal=host="..." \ --from-literal=port=5432 \ --from-literal=database="..." \ --from-literal=sslmode="..." ``` ##### Option 2: Multiple Databases Configuration For monitoring multiple databases within the same PostgreSQL instance, you can specify multiple databases using the `databases` field: ```bash kubectl create secret generic \ --namespace causely postgres-credentials-multidb \ --from-literal=username="..." \ --from-literal=password='...' \ --from-literal=host="..." \ --from-literal=port=5432 \ --from-literal=databases="database1,database2,database3" \ --from-literal=sslmode="..." ``` Alternatively, you can create the secret using a YAML manifest: ```yaml apiVersion: v1 kind: Secret metadata: name: postgres-credentials-multidb namespace: causely type: Opaque stringData: username: '...' password: '...' host: '...' port: '5432' databases: 'database1,database2,database3' sslmode: '...' ``` **Note**: Use either the `database` field for single database configuration or the `databases` field for multiple databases. Do not use both fields in the same secret. ##### Option 3: Database auto-discovery Causely can discover all databases on a PostgreSQL server automatically with the same credentials. Add `auto_discovery: "true"` to the secret (for example in `stringData`). The `host` must be the FQDN of your DB, or IP address if there's no DNS entry set up. It must match the FQDN/IP Causely would discover either from the K8s Server (if it's running in your K8s Cluster) or your Cloud providers API. Examples: - _Kubernetes_: `host=my-postgres.namespace.service.cluster.local` - _AWS_: `host=myinstance.rds.amazonaws.com` If you are leveraging a proxy to connect to your database, like it's common on GCP, `host` must be the FQDN/IP of your proxy and additionally `host_overwrite` should be the IP/FQDN of the actual database service: - _GCP_: `host=localhost` (assuming proxy runs on the same host / Pod). `host_overwrite=1.2.3.4` (the IP of your GCP Cloud SQL instance as shown in the GCP Console) #### Step 4: Update Causely Configuration Once the secret is created, update the Causely configuration to enable scraping for the new database. Below is an example configuration: ```yaml scrapers: postgresql: enabled: true instances: - secretName: postgres-credentials namespace: causely ``` #### Alternative: Enable Credentials Autodiscovery Causely also supports credentials autodiscovery. This feature allows you to add new scraping targets without updating the Causely configuration. Label the Kubernetes secret to enable autodiscovery for the corresponding scraper. ```bash kubectl --namespace causely label secret postgres-credentials "causely.ai/scraper=Postgresql" ``` ### Verify Your Configuration After completing the setup, use these checks to verify your PostgreSQL instance is properly configured for Causely slow query monitoring. #### Quick Check Query Run this single query to check all requirements at once: ```sql SELECT CASE WHEN EXISTS (SELECT 1 FROM pg_stat_user_tables LIMIT 1) THEN 'PASS' ELSE 'FAIL: Missing pg_read_all_stats' END AS perm_check, CASE WHEN EXISTS (SELECT 1 FROM pg_extension WHERE extname = 'pg_stat_statements') THEN 'PASS' ELSE 'FAIL: Extension missing' END AS ext_check, CASE WHEN EXISTS (SELECT 1 FROM pg_stat_statements LIMIT 1) THEN 'PASS' ELSE 'FAIL: Not loaded/accessible' END AS access_check; ``` **Expected result** (all checks passing): ``` perm_check | ext_check | access_check ------------+-----------+-------------- PASS | PASS | PASS ``` | Check | What it verifies | | -------------- | ---------------------------------------------------------------- | | `perm_check` | User can access performance views (has `pg_read_all_stats` role) | | `ext_check` | The `pg_stat_statements` extension is installed | | `access_check` | The extension is loaded and queryable | If any check shows `FAIL`, follow the detailed checks below to resolve the issue. #### Detailed Checks and Fixes ##### 1. Permissions Check ```sql -- Should return a row without error (tests pg_read_all_stats permission) SELECT 1 FROM pg_stat_user_tables LIMIT 1; -- Should return a row without error (tests information_schema access) SELECT 1 FROM information_schema.columns LIMIT 1; ``` **If the query fails**, grant the required role to your Causely user: ```sql -- Run as superuser (or rds_superuser on AWS RDS) GRANT pg_read_all_stats TO ; ``` ##### 2. Extension Check ```sql -- Should return: t (true) SELECT EXISTS(SELECT 1 FROM pg_extension WHERE extname = 'pg_stat_statements'); -- Should return a row without error SELECT 1 FROM pg_stat_statements LIMIT 1; ``` **If the extension is missing**, create it: ```sql CREATE EXTENSION IF NOT EXISTS pg_stat_statements; ``` **If the query fails** even though the extension exists, the extension is not loaded. You need to configure `shared_preload_libraries` and restart your instance. See [Step 2: Enable performance monitoring features](#step-2-enable-performance-monitoring-features) for platform-specific instructions. ##### 3. Shared Preload Libraries Check ```sql -- Should show: pg_stat_statements in the value SELECT setting FROM pg_settings WHERE name = 'shared_preload_libraries'; ``` If `pg_stat_statements` is not in the output, you need to add it to your configuration and restart the instance. Refer to the platform-specific instructions in [Step 2](#step-2-enable-performance-monitoring-features). #### Test Slow Query Access Run this query to confirm Causely can collect slow query data: ```sql SELECT query, calls, total_exec_time FROM pg_stat_statements ORDER BY total_exec_time DESC LIMIT 5; ``` This query should return results without error. If the table is empty, that's normal for a freshly configured instance, queries will appear after your application runs. :::tip[Success] If the test query works, your PostgreSQL instance is correctly configured for Causely slow query monitoring. ::: #### Setup Checklist Use this checklist to verify all configuration steps are complete: - [ ] `shared_preload_libraries` includes `pg_stat_statements` (requires instance restart) - [ ] Extension created: `CREATE EXTENSION IF NOT EXISTS pg_stat_statements;` - [ ] User has `pg_read_all_stats` role: `GRANT pg_read_all_stats TO ;` - [ ] Verification query succeeds: `SELECT 1 FROM pg_stat_statements LIMIT 1;` ### What Data is Collected The PostgreSQL scraper collects comprehensive metadata and performance information from your PostgreSQL databases, including: - **Database entities** with names and relationships to hosting services - **Service-to-database mappings** (which service provides which database) - **Connection details** including host, port, and SSL configuration - **Database and table schemas** with complete structural information - **Table information** including names, row counts, and sizes - **Complete table schemas** with column definitions, data types, constraints, and indexes - **Foreign key relationships** and table dependencies - **Parent-child table mappings** for dependency analysis - **Slow query analysis** using PostgreSQL's `pg_stat_statements` - **Lock monitoring** - **Cache and I/O performance** - **Table performance metrics** --- ## Prometheus Causely leverages [Prometheus](https://github.com/prometheus-operator/kube-prometheus) to query time-series metrics from your applications and infrastructure. To enable Prometheus as a data source with the default exporters, please add the following section to your `values.yaml` file: ```yaml scrapers: prometheus: enabled: true servers: prometheus: endpoint: http://prometheus-operated.monitoring:9090 ``` ### Supported Exporters Causely will automatically pick up metrics from supported exporters. Supported Exporters: - [cAdvisor](https://github.com/google/cadvisor) - Container resource usage and performance metrics - [CockroachDB](https://www.cockroachlabs.com/docs/stable/monitor-cockroachdb-with-prometheus) - CockroachDB database metrics - [Conduktor](https://docs.conduktor.io/guide/monitor-your-kafka-clusters/prometheus/) - Consumer-group lag metrics from Conduktor Console, enriching existing Kafka topic/consumer-group entities - [Golang](https://prometheus.io/docs/guides/go-application/) - Go application metrics including mutex and garbage collection - [Java](https://prometheus.github.io/client_java/instrumentation/jvm/) - Java application metrics including JVM memory and garbage collection - [Kafka](https://github.com/danielqsj/kafka_exporter) - Kafka broker and topic metrics - [MongoDB](https://github.com/percona/mongodb_exporter) - MongoDB database metrics - [MySQL](https://github.com/prometheus/mysqld_exporter) - MySQL database metrics - [PostgreSQL](https://github.com/prometheus-community/postgres_exporter) - PostgreSQL database metrics - [Python](https://prometheus.io/docs/guides/python-client/) - Python application metrics - [RabbitMQ](https://www.rabbitmq.com/docs/prometheus) - RabbitMQ message broker metrics - [Redis](https://github.com/oliver006/redis_exporter) - Redis cache and database metrics :::info Prometheus Discovery When using Prometheus with Kubernetes service discovery, `namespace` and `pod` labels are typically added automatically by the Prometheus Helm chart through relabeling. If these labels are missing from your metrics, configure relabeling in your ServiceMonitor or PodMonitor configuration to map Kubernetes metadata (`__meta_kubernetes_namespace`, `__meta_kubernetes_pod_name`) to `namespace` and `pod` labels. If you're using the OpenTelemetry Collector with the `prometheus/metrics` receiver, see the [OpenTelemetry Prometheus metrics relabeling configuration](/telemetry-sources/opentelemetry#configuring-prometheus-metrics-relabeling) for how to configure relabeling in the collector. ::: ### How to add custom metrics To enable Prometheus as a data source with default exporters plus additional custom metrics, please add the following section to your `values.yaml` file: ```yaml scrapers: prometheus: enabled: true servers: prometheus: endpoint: http://prometheus-operated.monitoring:9090 exporters: - cadvisor - causely - go-applications - java-applications - python-applications - postgres - mongodb - cockroachdb - redis - kafka - conduktor - mysql - rabbitmq - chaosmania exporters: chaosmania: entities: - entity: workload: {} discovery: - kubernetes_pod: namespace: 'namespace' pod_name: 'pod' metrics: - attribute: TransactionCount query: 'sum by (namespace, service, pod) (rate(chaosmania_processed_transactions_duration_count[1m]))' - attribute: TransactionDuration query: 'sum by (namespace, service, pod) (rate(chaosmania_processed_transactions_duration_sum[1m]) / (rate(chaosmania_processed_transactions_duration_count[1m]) > 0 or (rate(chaosmania_processed_transactions_duration_count[1m]) + 1)))' - attribute: DBQueryDuration query: 'sum by (namespace, service, pod) (rate(postgres_queries_sum[1m]) / (rate(postgres_queries_count[1m]) > 0 or (rate(postgres_queries_count[1m]) + 1)))' ``` #### Business Application concurrent users You can also use custom metrics to map concurrent users to a Business Application, with the `ConcurrentUsersMeasured` attribute. For the mapping and an example, see [Business KPIs](/configuration/business-kpis). ### Example: Kafka Exporter You can use this helm `values.yaml` file for the kafka exporter to label the broker name `super-heroes-kafka-brokers` for the monitored topics: ```yaml kafkaServer: - super-heroes-kafka-brokers:9092 service: annotations: prometheus.io/port: '9308' prometheus.io/scrape: 'true' labels: service_name: super-heroes-kafka-brokers prometheus: serviceMonitor: enabled: true namespace: quarkus-super-heroes relabelings: - sourceLabels: [__meta_kubernetes_service_label_service_name] targetLabel: target ``` ### Basic Authentication When your Prometheus or Mimir server requires HTTP Basic Authentication, you can supply credentials so the mediator can authenticate. If the server is configured via a Kubernetes Secret (for example with auto-discovery), add `username` and `password` keys to that secret. The mediator will send HTTP Basic Auth headers on every request to the endpoint. Secret keys: - **`username`**: Basic auth username (optional; omit if the server does not use basic auth). - **`password`**: Basic auth password (optional; omit if the server does not use basic auth). If both are present, the mediator uses them for all Prometheus API and query requests to that server. This applies to both single-tenant Prometheus and multi-tenant Mimir configurations when the server is provided via a secret. ### Multi-tenant Mimir Causely supports multi-tenant [Mimir deployments](https://grafana.com/docs/mimir/latest/manage/secure/authentication-and-authorization/), allowing you to configure multiple tenants and specific exporters for each Mimir instance. To enable Mimir as a data source with multi-tenant support, add the following section to your `values.yaml` file: ```yaml scrapers: prometheus: enabled: true servers: mimir: endpoint: http://mimir-nginx.mimir:80/prometheus tenants: - tenant-1 - tenant-2 exporters: - postgres - redis ``` This configuration allows Causely to: - Connect to the Mimir instance via the specified endpoint - Query metrics from multiple tenants (`tenant-1` and `tenant-2`) --- ## Pyrra ### Overview [Pyrra](https://github.com/pyrra-dev/pyrra) turns SLO definitions into **Prometheus recording rules**: metrics whose names include `:burnrate`, carrying labels such as `slo`, `backstage_component`, or `team`. Causely reads those rules on its next Prometheus sync and creates an **SLO entity** for each one, attached to the entity that owns it: - a single **Service**, or - a **Component** (one logical piece of software, often several Kubernetes Services), for example a Backstage component, or - a **Business Application** (a whole product or system). This integration builds on the Prometheus scraper. Make sure Prometheus is enabled and reachable by the mediator before configuring Pyrra. See [Prometheus](/telemetry-sources/prometheus). ### Configuration Causely resolves the owner of each SLO from the labels on its recording rule. Two Helm settings are involved: | Setting | Helm location | When you need it | | -------------------- | --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `pyrra.associations` | `scrapers.prometheus.pyrra` | **Always.** Maps the labels on a recording rule to the owner type and to the entity that should own the SLO. | | `label_assignments` | `scrapers.kubernetes` | **Only for Component-owned SLOs.** A rule label such as `backstage_component=loki` does not tell Causely which Kubernetes Services make up Component `loki`; `label_assignments` supplies that link. | :::info Defaults are not merged When you set `associations` explicitly, list **every** pattern your rules use—Causely does not merge in defaults. Causely tries the blocks top to bottom and uses the **first** match whose rule labels are all present, so put specific Component patterns before generic Service or Business Application ones. ::: #### Service-owned SLOs The recording rule names the Kubernetes Service directly: ```yaml # Rule labels slo: loki-querier-availability service_id: loki-querier k8s_ns: observability ``` ```yaml title="causely-values.yaml" scrapers: prometheus: pyrra: associations: - entity: { service: {} } discovery: - kubernetes_service: service_name: service_id # rule label holding the Service name namespace: k8s_ns # rule label holding the namespace ``` The SLO attaches to **Service/observability/loki-querier** only. #### Business Application-owned SLOs The recording rule is scoped to a whole product or system: ```yaml # Rule labels slo: observability-ingest-availability backstage_system: grafana-observability ``` ```yaml title="causely-values.yaml" scrapers: prometheus: pyrra: associations: - entity: { business_application: {} } discovery: - business_application: name: backstage_system # rule label holding the application name ``` The SLO attaches under **Business Application/grafana-observability**. #### Component-owned SLOs The recording rule names a Component that spans several Kubernetes Services: ```yaml # Rule labels slo: loki-querier-availability backstage_component: loki backstage_system: grafana-observability backstage_domain: observability ``` Causely should create SLO `loki-querier-availability`, attach it to **Component `loki`**, and treat Services `loki-querier` and `loki-ingester` as members of that Component—not `tempo-querier` and `tempo-ingestor`, which run in the same `observability` namespace under sibling Component `tempo`. Neither the recording rule nor the cluster carries that link, so this case takes both settings. **1. Tag the Services with `label_assignments`.** Select Services by labels they **already carry** (the Loki and Tempo Helm charts stamp `app.kubernetes.io/name` on every Service they create) and assign the catalog labels your Pyrra rules use. Nothing is written back to the cluster; the labels land only on Causely Service entities, and no Component or SLO is created at this point. ```yaml title="causely-values.yaml" scrapers: kubernetes: label_assignments: - service_selector: app.kubernetes.io/name: loki # already on loki-querier, loki-ingester, … labels: causely.ai/backstage-component-name: loki causely.ai/backstage-system: grafana-observability causely.ai/backstage-domain: observability - service_selector: app.kubernetes.io/name: tempo # already on tempo-querier, tempo-ingestor, … labels: causely.ai/backstage-component-name: tempo causely.ai/backstage-system: grafana-observability causely.ai/backstage-domain: observability ``` **2. Match those labels in `pyrra.associations`.** ```yaml title="causely-values.yaml" scrapers: prometheus: pyrra: associations: - entity: { component: {} } discovery: - component: component_name: backstage_component # rule label → Component name system_name: backstage_system domain_name: backstage_domain service_labels: # rule label → Service entity label - metric_label: backstage_component entity_label: causely.ai/backstage-component-name - metric_label: backstage_system entity_label: causely.ai/backstage-system ``` Together, the two settings resolve the Loki rule like this: 1. The rule has `backstage_component=loki`, so the Component name is **loki**. 2. `service_labels` maps rule labels `backstage_component` and `backstage_system` to the entity labels set by `label_assignments`. 3. Causely finds Services `observability/loki-querier` and `observability/loki-ingester`. Tempo's Services do not match—they carry `causely.ai/backstage-component-name=tempo`. 4. Causely creates or updates **Component/loki**, links both Loki Services to it, and attaches SLO **loki-querier-availability**. `service_labels` is the bridge between the two settings: the value of a **rule label** must match the value of the **Service entity label** that `label_assignments` set. ### Full example The overlay below combines all three patterns, with `label_assignments` supporting the Component-owned one: ```yaml title="causely-values.yaml" scrapers: kubernetes: # Map existing Helm chart labels → Backstage catalog labels on Causely Service entities. label_assignments: - service_selector: app.kubernetes.io/name: loki labels: causely.ai/backstage-component-name: loki causely.ai/backstage-system: grafana-observability causely.ai/backstage-domain: observability - service_selector: app.kubernetes.io/name: tempo labels: causely.ai/backstage-component-name: tempo causely.ai/backstage-system: grafana-observability causely.ai/backstage-domain: observability prometheus: pyrra: associations: # Component-owned SLOs (backstage_component + backstage_system on recording rules) - entity: { component: {} } discovery: - component: component_name: backstage_component system_name: backstage_system domain_name: backstage_domain service_labels: - metric_label: backstage_component entity_label: causely.ai/backstage-component-name - metric_label: backstage_system entity_label: causely.ai/backstage-system # Service-owned SLOs (service_id + k8s_ns on recording rules) - entity: { service: {} } discovery: - kubernetes_service: service_name: service_id namespace: k8s_ns # Business Application-owned SLOs (backstage_system only on recording rules) - entity: { business_application: {} } discovery: - business_application: name: backstage_system ``` ### Burn alerts When Pyrra fires an **ErrorBudgetBurn** alert with `slo=loki-querier-availability`, Causely activates burn state on that SLO and elevates related Diagnoses as urgent. For how Causely uses SLO targets and burn rates in causal reasoning, see [SLO Targets and Burn Rates](/configuration/slo-configuration). --- ## Snowflake Causely provides native integration with Snowflake. ### Step 1: Create a user Create a user for your Snowflake account. ### Step 2: Create a Kubernetes Secret for the user After creating the user create a Kubernetes Secret: ```bash kubectl create secret generic --namespace causely snowflake-credentials \ --from-literal=username="..." \ --from-literal=password='...' \ --from-literal=account="xxxxxx-yyyyyy" \ --from-literal=warehouse="..." ``` ### Step 3: Update Causely Configuration Once the Secret is created, update the Causely configuration to enable scraping for the new database. Below is an example configuration: ```yaml scrapers: snowflake: enabled: true instances: - secretName: snowflake-credentials namespace: causely ``` ### Alternative: Enable Credentials Autodiscovery Causely also supports credentials autodiscovery. This feature allows you to add new scraping targets without updating the Causely configuration. Simply label the Kubernetes Secret to enable autodiscovery for the corresponding scraper. #### Example: Labeling Secrets for Autodiscovery ```bash kubectl --namespace causely label snowflake-credentials "causely.ai/scraper=Snowflake" ``` With these steps, you can seamlessly integrate Causely with Snowflake and configure it to monitor your desired resources. --- ## Splunk ### Overview Causely integrates with [Splunk](https://www.splunk.com/) for **log retrieval**: when your container logs are already centralized in Splunk, Causely can query Splunk instead of the Kubernetes API to pull those logs and correlate them with service-level incidents and Diagnoses. Use this integration if: - Your logs are already shipped to Splunk (for example via Splunk Universal Forwarder, Fluentd, or other collectors). - You want Causely to surface relevant logs in the context of an active Diagnosis or service malfunction, without collecting logs from Kubernetes directly. ### Benefits - **Automatic log context**: Surfaces relevant logs in the context of an active Diagnosis or service malfunction. - **Container-level visibility**: Displays container-level logs under affected services when abnormal behavior occurs (for example error spikes or degraded performance). - **Faster resolution**: Shows log lines and exceptions alongside Diagnoses to validate issues and shorten time to understanding and resolution. ### Setup #### Step 1: Create a Kubernetes Secret for Splunk authentication If your Splunk deployment requires authentication, create a Kubernetes Secret containing a Splunk token: ```bash kubectl create secret generic splunk-secret \ --namespace causely \ --from-literal=token="your-splunk-token" ``` Use the same secret name in the configuration below (for example `splunk-secret`). If Splunk does not require authentication, you can omit the `secret` field in the configuration. #### Step 2: Update the Kubernetes scraper configuration Enable Splunk for log retrieval by adding the `splunk` block under the Kubernetes scraper in your `causely-values.yaml` (or Helm values): ```yaml scrapers: kubernetes: splunk: endpoint: "https://splunk:8089" secret: "splunk-secret" # Optional: omit if no auth required index: "main" # Optional: Splunk index for log queries; defaults to "main" if omitted insecureSkipVerify: false # Optional: set true only for dev/test to skip TLS verification (insecure) ``` ##### Configuration reference | Field | Required | Description | |-------|----------|-------------| | `endpoint` | Yes | Splunk API endpoint URL (for example `https://splunk.example.com:8089`). | | `secret` | No | Kubernetes secret name in the Causely namespace containing the key `token` with your Splunk token. Omit if authentication is not required. | | `index` | No | Splunk index used for log queries. Defaults to `main` if omitted. | | `insecureSkipVerify` | No | Set to `true` only in dev/test environments to skip TLS certificate verification. **Insecure**; do not use in production. Defaults to `false`. | #### Step 3: Install or upgrade Causely Apply your values and install or upgrade the Causely Helm release so the mediator uses the Splunk configuration for log retrieval. ### Authentication - **Token**: Include `token` in the Kubernetes secret referenced by `secret`. This is the standard way to authenticate to Splunk (for example HEC token or API token). - **No auth**: Omit the `secret` field in the configuration if your Splunk endpoint does not require authentication. ### Supported log types - Container logs (stdout/stderr) that have been shipped to Splunk and are queryable in the configured index. ### Result - Causely retrieves and displays logs from Splunk alongside a Diagnosis, highlighting the precise errors or stack traces that occurred around the time of failure. - Logs are correlated with affected services and Diagnoses to speed up validation and resolution. --- ## Troubleshooting Telemetry Sources If entities and connections are not appearing in your Causely topology, check the [Integrations page](https://portal.causely.app/integrations) for errors. This guide helps you diagnose and resolve common issues with telemetry source integrations. ### Identifying Issues #### Check the Integrations Page 1. Navigate to the [Integrations page](https://portal.causely.app/integrations) in the Causely UI 2. Look for any integrations showing error states (red indicators or error messages) 3. Click on an integration to view detailed error information Common error indicators include: - **Initialization errors**: The scraper failed to start - **Authentication errors**: Invalid credentials or missing permissions - **Connection errors**: Unable to reach the telemetry source - **Configuration errors**: Missing or incorrect configuration values ### Common Issues and Solutions #### Authentication and Credentials ##### Invalid or Expired Credentials **Symptoms:** - Error messages mentioning "authentication failed", "unauthorized", or "invalid credentials" - Integration shows as "Initialization Failed" **Solutions:** 1. **Verify credentials are correct:** - Check that API keys, tokens, or passwords are not expired - Ensure credentials match the correct account or environment - Verify there are no extra spaces or special characters when copying credentials 2. **For Kubernetes secrets:** ```bash # Verify the secret exists and contains the correct data kubectl get secret -n causely -o yaml # Check that the secret keys match what Causely expects # (e.g., apiKey, appKey, token, etc.) ``` 3. **For cloud provider integrations (AWS, Azure, GCP):** - Verify IAM roles or service accounts have the required permissions - Check that access keys haven't been rotated - Ensure service account annotations are correctly set for IRSA (AWS) or Workload Identity (GCP) ##### Missing Secret Labels (Autodiscovery) **Symptoms:** - Integration not appearing in the Integrations page - Scraper not starting despite secret existing **Solutions:** 1. **Verify the secret has the correct label:** ```bash # Check if the secret has the scraper label kubectl get secret -n causely --show-labels # Add the label if missing (example for PostgreSQL) kubectl --namespace causely label secret "causely.ai/scraper=Postgresql" ``` 2. **Common scraper label values:** - `Postgresql` for PostgreSQL - `MySQL` for MySQL - `Datadog` for Datadog - `AWS` for AWS - `Azure` for Azure - `GCP` for GCP - `Instana` for Instana - `Dynatrace` for Dynatrace - `Confluent` for Confluent - `Snowflake` for Snowflake #### Network Connectivity ##### Cannot Reach Telemetry Source **Symptoms:** - Connection timeout errors - "Unable to connect" or "network unreachable" messages - Integration shows intermittent failures **Solutions:** 1. **Verify network connectivity from the mediator pod:** ```bash # Get the mediator pod name kubectl get pods -n causely -l app=mediator # Test connectivity to the telemetry source kubectl exec -n causely -- curl -v ``` 2. **Check firewall rules:** - Ensure the mediator can reach external APIs (for cloud providers) - Verify internal network policies allow communication - Check if the telemetry source requires allowlisting the mediator's IP 3. **For Prometheus/Istio integrations:** - Verify the Prometheus endpoint is accessible from the mediator - Check that the endpoint URL is correct (including protocol: `http://` or `https://`) - Ensure Prometheus is not behind authentication that requires additional configuration #### Configuration Errors ##### Missing Required Configuration **Symptoms:** - "Configuration error" or "missing required field" messages - Integration fails to initialize **Solutions:** 1. **Verify all required fields are set:** - Check the specific telemetry source documentation for required configuration - Ensure secrets contain all required keys - Verify values.yaml or configuration files have all necessary settings 2. **Common missing fields:** - **Database integrations**: Missing `host`, `port`, `database`, or `username` - **Cloud providers**: Missing `region` or `account ID` - **APM tools**: Missing `apiKey`, `appKey`, or `org` (for Datadog) - **Prometheus**: Missing `endpoint` URL 3. **Check configuration format:** ```bash # Validate Helm values helm template causely oci://us-docker.pkg.dev/public-causely/public/causely --values values.yaml --dry-run # Check mediator logs for configuration errors kubectl logs -n causely -l app=mediator --tail=100 ``` ##### Incorrect Endpoint URLs **Symptoms:** - Connection refused errors - 404 Not Found errors - SSL/TLS certificate errors **Solutions:** 1. **Verify endpoint URLs:** - Ensure URLs include the correct protocol (`http://` or `https://`) - Check that ports are correct (for example, `:4317` for OTLP, `:9090` for Prometheus) - Verify hostnames resolve correctly 2. **For HTTPS endpoints:** - Check if self-signed certificates require additional configuration - Verify certificate validity if using custom certificates #### Permission Issues ##### Insufficient Permissions **Symptoms:** - "Access denied" or "Forbidden" errors - Integration can connect but cannot read data - Partial data discovery (some entities missing) **Solutions:** 1. **For cloud provider integrations:** **AWS:** - Verify IAM role or user has required read permissions - Check policies include: `ReadOnlyAccess` or specific service read permissions - For EKS, ensure IRSA (IAM Roles for Service Accounts) is configured correctly **Azure:** - Verify service principal has "Reader" role or equivalent - Check that the subscription is accessible - Ensure Managed Identity is configured if using that method **GCP:** - Verify service account has required IAM roles - Check Workload Identity is configured for GKE - Ensure project-level permissions are correct 2. **For database integrations:** - Verify database user has `SELECT` permissions on system tables - Check that the user can access `pg_stat_statements` (PostgreSQL) or equivalent - Ensure the user has permissions to query metadata tables 3. **For Kubernetes:** - Verify the mediator service account has necessary RBAC permissions - Check ClusterRole and ClusterRoleBinding are correctly configured - Ensure the mediator can list and get resources in target namespaces #### Version and Compatibility Issues ##### Incompatible Versions **Symptoms:** - API errors mentioning version incompatibility - Unexpected response formats - Features not working as expected **Solutions:** 1. **Check Causely version compatibility:** - Review the telemetry source documentation for version requirements - Ensure you're using a supported version of Causely - Check changelog for breaking changes 2. **Verify telemetry source versions:** - Some integrations require minimum versions of the source system - Check if API versions have changed - Review telemetry source release notes ### Platform-Specific Troubleshooting #### Kubernetes **Common issues:** - Service account permissions - Network policies blocking communication - Resource quotas limiting scraper execution **Debug steps:** ```bash # Check mediator pod status kubectl get pods -n causely # View mediator logs kubectl logs -n causely -l app=mediator --tail=200 # Check service account kubectl get serviceaccount -n causely # Verify RBAC permissions kubectl get clusterrolebinding | grep causely ``` #### Docker **Common issues:** - Container networking issues - Volume mount permissions - Environment variable configuration **Debug steps:** ```bash # Check container status docker ps | grep causely # View container logs docker logs # Verify environment variables docker exec env | grep CAUSELY ``` #### Nomad **Common issues:** - Job allocation failures - Network configuration - Volume access issues **Debug steps:** ```bash # Check job status nomad job status causely # View allocation logs nomad alloc logs # Check job specification nomad job inspect causely ``` ### Metrics and Trace Volume #### Metrics appear missing or don't match external monitoring tools **Symptoms:** - Causely metrics graphs show gaps or periods with no data - Metric values in Causely differ from what you see in Prometheus, Datadog, or other external tools **Solutions:** Causely is designed to work from a **statistically significant sample** of telemetry, not 100% of raw data. Minor differences between Causely metrics and external tools are expected and do not affect analysis accuracy. If you observe **persistent gaps** (extended periods with no data at all), VictoriaMetrics is likely under-resourced for your environment: 1. **Check VictoriaMetrics memory usage:** ```bash kubectl top pod -n causely -l app=victoriametrics ``` 2. **Increase VictoriaMetrics memory allocation** by following the [VictoriaMetrics sizing guide](/installation/customize#sizing-victoriametrics-for-high-volume-environments). #### Traces appear to be dropped **Symptoms:** - Trace counts in Causely are lower than what your instrumentation or external tracing backend reports - You notice the Mediator pod's memory usage is near its limit **Explanation:** The Mediator's built-in OpenTelemetry Collector has a memory limiter that drops excess traces when memory pressure is reached. This is **intentional behavior**: it means the Mediator has already collected a sufficient sample of traces for analysis. Causely does not require every trace to produce a Diagnosis. Mediator memory is sized by the **number of entities under management** (services, pods, databases, and other components in your environment), not by trace or request throughput. If your cluster is growing, you may need to increase Mediator memory. See [Sizing the Mediator for large environments](/installation/customize#sizing-the-mediator-for-large-environments). If you need full-fidelity trace retention for other purposes (for example, debugging individual requests), you can route traces through an intermediate collector that fans out to both Causely and your tracing backend (for example, Grafana Tempo). If you are using Causely's built-in Beyla instrumentation, redirect Beyla's trace export to your OpenTelemetry Collector or Grafana Alloy instance by setting the following in your `causely-values.yaml`: ```yaml agent: beyla: otel_traces_export: endpoint: http://:4317 ``` Then configure your collector to export traces to **both** your tracing backend and the Causely Mediator (`mediator.causely:4317`). See [Integrating OpenTelemetry with Causely](/telemetry-sources/opentelemetry#integrating-opentelemetry-with-causely) and [Grafana integration](/telemetry-sources/grafana) for collector configuration details. ### Getting Additional Help If you've tried the solutions above and are still experiencing issues: 1. **Check mediator logs** for detailed error messages: ```bash kubectl logs -n causely -l app=mediator --tail=500 ``` 2. **Review integration-specific documentation:** - [AWS Integration](/telemetry-sources/aws) - [Azure Integration](/telemetry-sources/azure) - [GCP Integration](/telemetry-sources/gcp) - [PostgreSQL Integration](/telemetry-sources/postgresql) - [MySQL Integration](/telemetry-sources/mysql) - [Datadog Integration](/telemetry-sources/datadog) - [Instana Integration](/telemetry-sources/instana) - [Dynatrace Integration](/telemetry-sources/dynatrace) 3. **Contact Causely Support:** - Email: [support@causely.ai](mailto:support@causely.ai) - Include: - Integration name and error message - Relevant log excerpts (redact sensitive information) - Configuration details (redact credentials) - Steps you've already tried ### Prevention Best Practices 1. **Use autodiscovery labels** to ensure secrets are properly recognized 2. **Test connectivity** before configuring integrations 3. **Verify credentials** are valid and have appropriate permissions 4. **Monitor integration health** regularly in the Integrations page 5. **Keep Causely updated** to ensure compatibility with latest telemetry sources 6. **Document your configuration** to make troubleshooting easier