# Causely > Causely documentation for the causal AI engine that explains, predicts, and helps resolve software incidents. Start with the Causely GenAI primer for grounding and terminology: - [Causely GenAI Primer](https://docs.causely.ai/llms-primer.md): canonical framing, terminology, and interpretation guidance for Causely outputs. This file contains the complete Causely documentation corpus in one place for offline ingestion and evaluation. ## Quick Setup Guide for Causely export const CAUSELY_VERSION = versionHistory?.versions?.[0]?.imageVersion ?? versionHistory?.versions?.[0]?.version ?? ''; Get Causely running in your environment and connected to your agents in just a few minutes. This guide covers planning your telemetry setup, deploying the mediator, connecting data sources, verifying discovery, and giving AI agents access to Causely's causal model via MCP. After completing this guide, you are able to use Causely with your AI agents to identify root causes of issues in your environment, enabling faster problem resolution and improved system reliability. ## Plan Your Deployment Before installing Causely, it's important to understand what telemetry sources you'll need for a successful deployment. Causely requires two types of data sources to be most effective: - [Services](#services) - [Service Connections](#service-connections) :::warning Why those two are most important for your success **Service discovery** tells Causely what exist. **Service connections** shows how services interact with each other. **Both together** provide a complete picture, enabling accurate causal inference and impact assessment. Make sure you have at least one of these sources to be most effective, or use the out of the box instrumentation which is provided by eBPF-based OpenTelemetry instrumentation. ::: ### Services Service discovery helps Causely understand what services and infrastructure exist in your environment. These sources create entities like services, pods, containers, databases, and other infrastructure components. **You need at least one of these**: Kubernetes } description='Discovers clusters, nodes, pods, containers, services, and controllers' /> Docker } description='Discovers containers and allocations' /> Nomad } description='Discovers Nomad jobs, tasks, and services' /> AWS } description='Discovers EC2, ECS, Lambda, RDS, MSK, and ALB resources' /> Azure } description='Discovers Azure VMs, VMSS, disks, and storage' /> GCP } description='Discovers Compute Engine, Cloud Run, BigTable and Cloud SQL' /> For Kubernetes, Docker and Nomad, the service discovery is enabled during the installation of the Causely mediator on those platforms. For your cloud provider you can add the corresponding integration [via the UI](https://portal.causely.app/integrations). ### Service Connections Service connections reveal how services communicate with each other. These sources create connections between services, showing dependencies, request flows, and communication patterns. This is critical for building the service dependency graph and understanding how issues propagate. **You need at least one of these**: OpenTelemetry } description='Processes OTLP traces to discover service-to-service connections' /> Datadog } description="Queries Datadog's API to discover services and connections from APM traces" /> Instana } description="Queries Instana's API to discover services, connections, database access, HTTP paths, and RPC methods" /> Dynatrace } description="Queries Dynatrace's API to discover services and service connections" /> The default Causely installation includes [OpenTelemetry with eBPF](/telemetry-sources/ebpf) instrumentation automatically. ## Deploy the Causely Mediator Causely can be installed on any platform, from virtual machines and bare metal servers to container orchestration platforms like Kubernetes, Nomad, and Docker. Navigate to the [Mediators page](https://portal.causely.app/agents) in the Causely UI and click **"Add New"** to get your personalized installation command. Follow the instructions provided, which will look similar to: {`helm upgrade --install causely \\ --create-namespace oci://us-docker.pkg.dev/public-causely/public/causely \\ --version ${CAUSELY_VERSION} \\ --namespace=causely \\ --set image.tag=${CAUSELY_VERSION} \\ --set global.cluster_name=CHANGE_ME_CLUSTER_NAME \\ --set mediator.gateway.token=YOUR_ACCESS_TOKEN`} Replace `YOUR_ACCESS_TOKEN` with your access token and `CHANGE_ME_CLUSTER_NAME` with your cluster name. The Causely mediator is installed in the `causely` namespace. To verify the installation, run: ```bash kubectl get pods -n causely ``` By default, the applications in your cluster will be instrumented automatically using OpenTelemetry eBPF instrumentation. This means that Causely can use [OpenTelemetry](/telemetry-sources/opentelemetry) traces to discover service dependencies, monitor sync and async communication signals and diagnose issues in your environment. For detailed instructions on installing Causely on HashiCorp Nomad, see the [Nomad installation guide](/installation/nomad). The guide covers prerequisites, configuration, and deployment steps for running Causely on Nomad clusters. For detailed instructions on installing Causely on standalone Docker hosts, see the [Docker installation guide](/installation/docker). The guide covers prerequisites, configuration, and deployment steps for running Causely on Docker hosts outside of Kubernetes. For installation on virtual machines or bare metal servers, please [reach out to Causely support](mailto:support@causely.ai) for assistance with your specific environment and requirements. ## Connect Telemetry Sources With the mediator deployed, you can now connect your planned telemetry sources to Causely. As outlined above, begin with sources that provide you with discovery of services and service connections. To detect a wide range of [Signals](/reference/symptoms/) and causes in your environment, you can configure additional data sources. Visit the [telemetry sources](/telemetry-sources/) page to learn more about the data sources that Causely supports. ## Review Discovery You have successfully installed Causely! Navigate to https://portal.causely.app to verify your environment has been discovered. You should see entities populated in the [Topology](/in-action/topology) view: If the entities and connections you are expecting to see are not appearing, go to the [Integrations](https://portal.causely.app/integrations) page and check for any errors for the telemetry sources you have configured. Errors in the integrations page will prevent Causely from discovering entities and connections in your environment. Follow the [troubleshooting steps](/telemetry-sources/troubleshooting) to resolve the errors. ## Connect Causely to AI Agents Give your agents access to Causely's causal model via the MCP server. Any MCP-compatible agent or assistant including Claude Code, Cursor, VS Code, HolmesGPT, or your own custom agent, can query root causes, service health, dependency maps, and reliability reports directly. Claude } description="Query root causes, service health, and dependency maps directly from Claude Code or Claude Desktop via the Causely MCP server." /> Cursor } description="Investigate incidents and check service health without leaving your editor using the Causely MCP integration." /> Codex } description="Connect OpenAI's coding agent to Causely's causal model via MCP for system-aware reliability workflows." /> HolmesGPT } description="Pre-built incident investigation agent with Causely MCP configured as its causal reasoning layer." /> Visit the [agent integration](/agent-integration) page to get started. --- ## How Causely Works Causely transforms raw telemetry into a live, queryable model of your system providing the semantic and causal foundation your agents need to diagnose, evaluate impact, and act safely in production. It works alongside your existing telemetry sources, interpreting the meaning behind the metrics, logs, and traces they provide, rather than replacing them. This page explains how Causely works: the **ontology-first approach** that turns raw telemetry into higher abstractions, then how the **causal reasoning engine** uses those abstractions to infer a Diagnosis and compute the blast radius that follows from it. For deployment architecture, components, and infrastructure, see [Architecture](/getting-started/architecture). ## Issue, Diagnosis, and Signal Before going into the mechanics, it helps to know the three terms used to describe what Causely finds: an **Issue** is the persistent problem you acknowledge and resolve; a **Diagnosis** is Causely's current best explanation for it, produced by the causal reasoning engine described below; and a **Signal** is an observed anomaly a Diagnosis explains, usually a threshold violation on a metric or SLI, and sometimes an alert, log line, or event. > An Issue causes anomalies that surface as Signals. Causely infers a Diagnosis, the best explanation of those Signals at that moment. As Signals change, the Diagnosis updates. The Issue stays stable across those changes. If two Diagnoses share even one Signal, they are treated as the same Issue. The rest of this page describes the causal reasoning engine that produces a Diagnosis from Signals, using the terms root cause and symptom, the formal names for these concepts defined in [Terminology](/reference/terminology/#issue-diagnosis-and-signal). ## How the Causal Engine Works At the core of Causely is a probabilistic reasoning engine that maps symptoms to root causes using domain-specific models and dynamic system knowledge. It maintains three core data structures: the **Topology Graph**, the **Causality Graph** (with its Codebook), and the **Attribute Dependency Graph**, and uses them for analysis. The engine is composed of six interdependent components, described below. {(() => { const { colorMode } = useColorMode(); const src = colorMode === 'dark' ? '/img/how_causely_works_diagramLight1.svg' : '/img/how_causely_works_diagramDark1.svg'; return ; })()} ### 1. From telemetry to semantic understanding Instead of feeding raw telemetry straight into analysis, Causely uses an **ontology-first approach**: **mediation** distills telemetry (logs, metrics, traces) and other data locally into a structured layer of: - **[Entities](/reference/entity-types)**: services, pods, databases, queues - **Relations**: how they connect - **[Signals](/reference/symptoms/)**: observable anomalies such as "high latency here" or "errors increasing there"; detected when thresholds or patterns are met and used for reasoning even when no alert is fired That distillation turns the vast volumes of data teams commonly ingest for observability into higher-quality abstractions. This distilled data is sent to the backend, where the causal reasoning engine infers the [causes](/reference/root-causes/) explaining those Signals. :::tip Local processing and privacy All distillation runs in your environment. The **mediation layer** turns telemetry into **symptom states** and **topology**; only this **distilled data** is sent to the causal reasoning engine. Raw logs, full traces, and bulk metrics stay in your datacenter. Only semantic state and, when needed, minimal evidence leave your environment. This design keeps sensitive and high-volume telemetry local and preserves **privacy**. ::: ### 2. Ontology: causal and attribute models The engine’s **ontology** is the formal model of entities, behaviors, and failure modes used for inference. It is composed of two foundational models: the **Causal Model** and the **Attribute Dependency Model**. Together, they define what root causes and symptoms exist and how they relate, forming the semantic backbone of the causal reasoning engine. - **Causal Model**: Causely includes a built-in library of causal knowledge that captures root causes capable of degrading application performance. This model requires no configuration and enables the system to begin identifying root causes as soon as it is deployed. - Covers a broad range of entities, including applications, databases, caches, messaging systems, load balancers, DNS, compute, and storage. - Encodes how each root cause propagates through the environment and the symptoms it may produce. - Is designed to be environment-agnostic and applicable to any modern cloud native architecture. - **Attribute Dependency Model**: The attribute model extends the causal model by capturing how performance-related attributes (for example, latency, throughput, utilization) are interdependent across entities. It also encodes the operational constraints those attributes must satisfy to meet performance goals. - Represents attribute dependencies across a wide range of services and infrastructure layers. - Supports both predefined and learned functional relationships between attributes. - Defines the desired state of the system based on application goals and constraints. - Like the causal model, it is fully environment-independent and generalizable across architectures. ### 3. Topology Graph Causely continuously discovers and maintains the topology graph of services, infrastructure, and their interconnections via its integrations with [your existing telemetry sources](/telemetry-sources/). It ingests and reconciles topology from any source, including OpenTelemetry traces, cloud provider APIs and other inventories. Cloud native environments are highly dynamic, composed of applications, services, databases, caches, messaging systems, load balancers, compute, storage, and more. Through continuous discovery and ingestion from these sources, Causely determines for each entity: - **Connectivity**: which other entities it communicates with horizontally - **Layering**: which entities it is built upon or supports vertically - **Composition**: the internal components or resources that make up the entity These relationships are stitched together into a continuously updated, real-time graph that represents the full system [topology](/reference/terminology/#topology), regardless of whether data comes from OpenTelemetry, cloud APIs or other integrated sources. This graph forms the foundation for blast radius analysis, root cause attribution, and cross-service impact modeling. ### 4. Causality Mapping Causely automatically generates a [Bayesian network](/reference/terminology/#bayesian-network) that models how root causes lead to observable symptoms, based on its built-in Causal Models and the real-time Topology Graph. This causal mapping encodes probabilistic cause-effect relationships, allowing the system to infer the most likely root cause from a given set of active symptoms. It reflects both the structural dependencies in your environment and learned patterns of failure propagation. Causely represents this mapping through two core data structures: - **[Causality Graph](/reference/terminology/#causality-graph-cg)**: A directed acyclic graph (DAG) where nodes represent root causes and symptoms, and edges denote potential causal relationships. Each edge is weighted with a probability, indicating the likelihood that one node (a root cause) leads to another (a symptom). - **[Codebook](/reference/terminology/#codebook)**: A table where each column corresponds to a root cause and each row to a symptom. Each column is a vector of probabilities defining a unique signature of the root cause. A cell in the vector represents the probability that the root cause may cause the symptom. Together, these structures power Causely’s ability to perform real-time probabilistic inference and deliver explainable, high-confidence root cause insights. ### 5. Attribute Dependency Graph Causely generates this graph using its built-in Attribute Dependency Model and the live Topology Graph. The result is a directed acyclic graph (DAG) that models functional dependencies between system attributes. In this graph: - Nodes represent individual attributes, for example CPU usage of a service or queue length of a messaging system. - Edges represent dependency relationships, for example an edge from attribute A to attribute B means that B is a function of A. - Edge labels define these functions. Some may be explicitly defined in the Attribute Dependency Model, while others are learned automatically from observed behavior in your environment. - Nodes representing attributes that must satisfy a constraint are decorated with the constraint the attribute must satisfy. The Attribute Dependency Graph enables Causely to reason about how changes in one part of the system cascade across others, identify emerging bottlenecks, and validate whether the environment remains within defined performance bounds. ### 6. Analysis The analysis automatically pinpoints causes in real time based on observed symptoms, using the Codebook described above. No configuration is required, Causely can immediately identify a broad set of issues (100+ causes mapped to symptoms) ranging from application malfunctions to service congestion to infrastructure bottlenecks. In any given environment, there can be tens of thousands of different causes that may cause hundreds of thousands of symptoms. Causely prevents service degradation by detangling this mess and pinpointing the cause putting your SLOs at risk and driving remediation actions before SLOs are violated. With the ontology and causal layers in place [agents connected to Causely](/agent-integration/) can query diagnosis, blast radius, and ownership directly, answering production questions in seconds with a small fraction of the tokens querying raw telemetry would consume. ## Deployment architecture For deployment architecture, component structure, and workflow integration, see [Architecture](/getting-started/architecture). --- ## Agent Integration ## Building Reliable Agents with Causely Agents fail not because they lack data. They fail because data alone does not explain causality. An agent with access to metrics, logs, and traces still cannot reliably determine what caused an issue, how far it has spread, or what action is safe to take. That requires a causal model: a structured understanding of how services, dependencies, and failure patterns relate. Causely provides that model. Agents query Causely through the MCP server and receive structured, deterministic answers, including issues, diagnoses, blast radius, dependency maps, and remediation guidance, instead of raw signals to interpret. ## The Gap in Today's Agent Architectures Most agent-driven systems run into three core limitations: **Information gap** Agents can retrieve telemetry, but cannot consistently determine what is happening or what matters. **System gap** There is no shared understanding of how services, infrastructure, and dependencies relate to each other. **Execution gap** Agents lack a reliable way to determine which actions are safe and how to coordinate them. As a result, agents require human interpretation, and automation breaks down at scale. :::tip Benchmark: agents with and without Causely Agents using Causely cut token consumption 48%, ran 63% faster, and hit 100% diagnosis accuracy across 72 benchmark experiments. [See the benchmark →](https://causely.ai/product/benchmark) ::: ## Where Causely Fits Causely provides a system intelligence layer that continuously models how your system behaves: its services, dependencies, and failure propagation. Instead of reasoning over raw telemetry, agents interact with structured, deterministic system knowledge. Decisions are based on how the system actually behaves, not on correlation or heuristics. ## Architecture Overview ``` [Agent (for example Holmes or custom agent)] ↓ [Causely (causal model + reasoning engine)] ↓ [Observability + Infrastructure (metrics, traces, logs, alerts)] ``` - **Agent**: orchestrates workflows, queries systems, and takes action - **Causely**: builds and maintains a causal model and provides deterministic reasoning - **Observability + Infrastructure**: provides raw signals and telemetry ## What Your Agent Can Do The Causely MCP server exposes 30 tools across 5 categories. Here is what each category enables: - **Entity Resolution**: Resolve service and database names to IDs, enumerate namespaces and clusters, check current health status. Most workflows start here. - **Data Retrieval**: Retrieve time-series metrics, live logs, alert history, deployment events, configuration files, and slow query analysis for any entity. - **Health & Diagnosis**: Get active symptoms environment-wide, identify root causes with impacted services and remediation guidance, check SLOs, map service topology, and get structured health summaries for services, teams, or individual entities. - **Reporting & Postmortems**: Generate deterministic postmortem drafts and structured engineering tickets from resolved incident data. - **Reliability & Deployment**: Compare resource consumption before and after deployments for a single service or an entire fleet. ## Integration Paths Choose based on how much you want to build. | Option | Best for | What you get | |---|---|---| | [MCP Server](/agent-integration/mcp-server) | Any MCP-compatible agent or assistant | Standardized interface to all 29 Causely tools; works with Cursor, Claude Code, VS Code, and others | | [HolmesGPT](/agent-integration/holmes-gpt) | Teams already using Holmes | Pre-built agent with Causely MCP configured; no custom integration required | | [Custom Agents](/agent-integration/custom-agents) | Teams building internal tooling or automation pipelines | Full control over logic, policies, and execution; MCP or direct API | If you are starting fresh, use the MCP Server. It works with any agent that supports the Model Context Protocol and requires no custom code. ## Example Workflow **Scenario: High error rate alert** 1. The agent receives an alert 2. The agent calls `get_entities()` to resolve the alerted service name to an entity ID 3. The agent calls `get_diagnoses()` to identify the source 4. Causely returns: - Root cause service - Affected dependencies - Explanation of why this is the cause - Remediation guidance 5. The agent: - Notifies the correct team - Suggests or executes remediation ## When This Approach Is Most Valuable This architecture is most effective when: - You operate distributed systems with many interdependent services - You already have observability in place - You are building or evaluating automated incident workflows ## Summary Causely does not replace your agents or your observability stack. It provides the system intelligence layer required for agents to interpret telemetry consistently, identify true root causes, and take safe, coordinated action. --- ## API Reference Guide The Causely API enables developers to integrate Diagnosis capabilities into their applications and workflows. This comprehensive guide provides step-by-step examples for authentication, querying defects, and automating incident response using the Causely platform. ## Getting Started with Causely API ### Create API Client Credentials Before you can use the Causely API, you need to generate API client credentials (Client ID and Secret) from the Causely platform: 1. **Login to Causely:** - Go to [Causely Portal](https://portal.causely.app/) - Login with your credentials 2. **Navigate to Personal Tokens:** - Click on the Profile Icon in the top-right corner - Select "Admin Portal" - Choose "Personal Tokens" 3. **Generate Token:** - Click "Generate Token" - Provide a description and expiration date - Click "Create" 4. **Save Credentials:** - Record your Client ID and Secret Key in a safe place - You'll use these credentials in the examples below ## Next Steps Once you have your API credentials, you can proceed with: - **[Authentication](/api/authentication)** - Learn how to authenticate with the Causely GraphQL API - **[Gateway Tokens](/api/gateway-tokens)** - Create and manage gateway tokens for mediator deployment - **[GraphQL Clients](/api/graphql-clients)** - Set up reusable GraphQL client libraries - **[Query Examples](/api/queries/getting-started)** - Explore example queries and mutations ## API Support For technical questions about the Causely GraphQL API, [contact our support team](mailto:support@causely.io) for assistance with integration challenges. --- ## SLO Targets and Burn Rates Service Level Objectives (SLOs) are how Causely translates reliability signals into **urgency and action**. SLOs define what “good” looks like for reliability across services, endpoints, and queues, and are used by Causely to determine when degradations represent acceptable risk versus issues that require immediate attention. In Causely, SLOs directly influence how Diagnoses are classified and prioritized. When a Diagnosis puts an SLO at risk or violates it, Causely treats that Diagnosis as more urgent, helping teams focus on the issues most likely to impact users and the business. SLOs are applied **by default at the service level**, providing broad coverage with minimal configuration. For teams that need more granular protection, SLOs can also be defined for specific **HTTP paths**, **RPC methods**, and **queues**, allowing critical user flows or business transactions to be protected independently of overall service health. Queue SLOs are configured via the API. ## Default SLO Behavior By default, Causely applies the following SLO targets and burn rate settings to all services: - **Error rate SLO target**: 99.0% (99% of requests must be successful) - **Latency SLO target**: 95.0% (95% of requests must be under the latency threshold) - **Availability SLO target**: 99.0% (99% uptime expected) - **Burn rate threshold**: 4 (budget would be consumed in 6 hours for a 1-day SLO) - **Burn rate window**: 15 minutes (calculation window for burn rate monitoring) These defaults are designed to catch fast-burning reliability issues early, while avoiding unnecessary noise for brief or low-impact fluctuations. ## When SLOs Are Active SLOs in Causely are evaluated only when **traffic is observed** for the corresponding entity. This applies consistently to services, HTTP paths, RPC methods, and queues. SLOs measure how an entity performs when responding to real requests. If no traffic is observed, there is no performance to evaluate, and the SLO remains inactive until requests are seen. ## Customizing SLO Behavior You can customize how SLOs behave in Causely depending on the level of control you need: - **Service-level SLOs** can be customized using labels or service metadata. This is the most common approach and is described in the sections below. - **HTTP Path and RPC Method SLOs** are configured exclusively through the API. These SLOs follow the same core concepts (targets, burn rates, and windows) but apply to specific endpoints rather than entire services. See [Setting SLOs on Paths and Methods](/api/queries/setting-slos-on-paths-and-methods/) for details. - **Queue SLOs** are configured exclusively through the API. These SLOs follow the same core concepts (targets, burn rates, and windows) but apply to specific queues, such as order-processing or event ingestion queues. When a queue SLO is at risk or violated, Diagnoses affecting that queue or its upstream dependencies are automatically elevated as urgent. - **Default SLO values** can also be adjusted programmatically through the API. Documentation and examples for API-based default configuration will be added in a future update. ## Supported Labels You can configure the following SLO-related labels: | Label | Description | Default | | --------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | `causely.ai/error-rate-slo-target` | Percentage of successful requests expected, for example 99.0. This defines the percentage of requests that must not result in an error to remain within the error SLO. | `99.0` | | `causely.ai/latency-slo-target` | Percentage of requests expected to be under the latency threshold, for example 95.0. Note that the latency threshold is automatically learned by Causely, but can be manually adjusted via the [Thresholds configuration page](/configuration/thresholds/). | `95.0` | | `causely.ai/availability-slo-target` | Percentage of time the service is expected to be operational, for example 99.0. This defines the proportion of total time the service must remain available, responding successfully to requests without downtime, to remain within the availability SLO. | `99.0` | | `causely.ai/error-rate-burn-rate-threshold` | Rate of error budget burn relative to the SLO target. A default of 4 means that for a 1-day SLO, if errors continue at the current rate, the error budget would be consumed in 6 hours. | `4` | | `causely.ai/latency-burn-rate-threshold` | Rate of latency budget burn relative to the SLO target. A default of 4 means that for a 1-day SLO, latency at the current rate would consume the entire budget in 6 hours. | `4` | | `causely.ai/availability-burn-rate-threshold` | Rate of availability budget burn relative to the SLO target. A default of 4 means that for a 1-day SLO, availability at the current rate would consume the entire budget in 6 hours. | `4` | | `causely.ai/error-rate-burn-rate-window` | Burn rate calculation window (in minutes) used to indicate whether a service is rapidly consuming its error SLO budget. | `15` | | `causely.ai/latency-burn-rate-window` | Burn rate calculation window (in minutes) used to indicate whether a service is rapidly consuming its latency SLO budget. | `15` | | `causely.ai/availability-burn-rate-window` | Burn rate calculation window (in minutes) used to indicate whether a service is rapidly consuming its availability SLO budget. | `15` | ## Configuration Methods ### Using Kubernetes Labels You can apply labels directly to your Kubernetes services: ```bash # Set error rate SLO target to 99% kubectl label svc -n "causely.ai/error-rate-slo-target=99.0" # Set latency SLO target to 95% kubectl label svc -n "causely.ai/latency-slo-target=95.0" # Set availability SLO target to 99% kubectl label svc -n "causely.ai/availability-slo-target=99.0" # Set error rate burn rate threshold to 2 kubectl label svc -n "causely.ai/error-rate-burn-rate-threshold=2" # Set latency burn rate threshold to 2 kubectl label svc -n "causely.ai/latency-burn-rate-threshold=2" # Set availability burn rate threshold to 2 kubectl label svc -n "causely.ai/availability-burn-rate-threshold=2" # Set error rate burn rate window to 15 minutes kubectl label svc -n "causely.ai/error-rate-burn-rate-window=15" # Set latency burn rate window to 15 minutes kubectl label svc -n "causely.ai/latency-burn-rate-window=15" # Set availability burn rate window to 15 minutes kubectl label svc -n "causely.ai/availability-burn-rate-window=15" ``` ### Using Nomad Service Tags If you use Nomad, you can specify these as service tags: ```hcl job "example" { group "app" { service { name = "my-service" port = 8080 tags = [ "causely.ai/error-rate-slo-target=99.0", "causely.ai/latency-slo-target=95.0", "causely.ai/availability-slo-target=99.0", "causely.ai/error-rate-burn-rate-threshold=2", "causely.ai/latency-burn-rate-threshold=2", "causely.ai/availability-burn-rate-threshold=2", "causely.ai/error-rate-burn-rate-window=15", "causely.ai/latency-burn-rate-window=15", "causely.ai/availability-burn-rate-window=15" ] } } } ``` ### Using Consul Service Metadata For Consul services, you can configure these using service metadata: ```bash # Register a service with slo metadata consul services register \ -name="my-service" \ -port=8080 \ -meta="causely.ai/error-rate-slo-target=99.0" \ -meta="causely.ai/latency-slo-target=95.0" \ -meta="causely.ai/availability-slo-target=99.0" \ -meta="causely.ai/error-rate-burn-rate-threshold=2" \ -meta="causely.ai/latency-burn-rate-threshold=2" \ -meta="causely.ai/availability-burn-rate-threshold=2" \ -meta="causely.ai/error-rate-burn-rate-window=15" \ -meta="causely.ai/latency-burn-rate-window=15" \ -meta="causely.ai/availability-burn-rate-window=15" # Update existing service metadata consul services register \ -id="my-service-id" \ -name="my-service" \ -port=8080 \ -meta="causely.ai/error-rate-slo-target=99.0" \ -meta="causely.ai/latency-slo-target=95.0" \ -meta="causely.ai/availability-slo-target=99.0" \ -meta="causely.ai/error-rate-burn-rate-threshold=2" \ -meta="causely.ai/latency-burn-rate-threshold=2" \ -meta="causely.ai/availability-burn-rate-threshold=2" \ -meta="causely.ai/error-rate-burn-rate-window=15" \ -meta="causely.ai/latency-burn-rate-window=15" \ -meta="causely.ai/availability-burn-rate-window=15" ``` ## Best Practices 1. **Align with SLO policy**: Reflect organizational reliability goals. 2. **Avoid overly aggressive thresholds**: High sensitivity may create alert fatigue. 3. **Monitor and adjust**: Tune thresholds based on incident reviews and error budget consumption. 4. **Document changes**: Record rationale for each SLO configuration. ## Example Use Cases 1. **Business-critical services**: Set tighter SLO targets, for example 99.9% success, 98% low-latency. 2. **Temporary adjustments**: Raise burn rate thresholds during high-traffic events. ## Queue SLO Example Queues often represent critical business workflows, such as order processing or asynchronous event handling. For example, if you need to ensure that an order-processing queue is drained within a defined time window, you can define an SLO using **Queue Depth** (or a related metric) as the SLI via the API. When that queue’s SLO is at risk or violated, Causely elevates related Diagnoses as urgent and surfaces the most relevant causal explanations, helping teams prioritize issues that directly impact key workflows. ## Burn Rate Threshold Examples The burn rate threshold determines how aggressively your error or latency budget is being consumed, and helps you catch fast-burning issues. Here's a simple example: Suppose your service has a 1-day SLO budget, meaning it can tolerate a limited amount of errors or latency over 24 hours. - **Burn Rate Threshold = 1** The current rate of errors or latency is steady and would use up the entire budget in exactly 24 hours. No alarm yet, but you're tracking close to your SLO target. - **Burn Rate Threshold = 2** At the current rate, the service would consume its full 24-hour budget in only **12 hours**, triggering alerts about rapid budget consumption. - **Burn Rate Threshold = 4** This indicates extremely fast-burning behavior. At this pace, the full error or latency budget would be used up in just **6 hours**. In practice, burn rate thresholds allow teams to catch reliability problems earlier, before they fully consume the SLO budget. ## Burn Rate Window Examples The burn rate window helps determine how quickly your service is consuming its SLO budget by observing behavior over short time intervals. Below are simple examples to clarify: - **Short Window (5 minutes)** Useful for detecting rapid error or latency spikes. For example, if a service suddenly begins failing or slowing down at a high rate, a short burn rate window (like 5 minutes) helps you identify that it's quickly consuming its SLO budget, enabling earlier incident detection. For services where fast detection of degraded performance is critical, consider also shortening the Signal activation delay, which you can manually configure via the [Signal Delay](/configuration/symptom-delay) settings. - **Moderate Window (15 minutes)** This is the default and provides a good balance between reactivity and noise. It captures bursts of errors or latency that might not last long enough to trigger alerts in a longer window but are still significant. - **Long Window (60 minutes)** Best used to detect sustained SLO violations. For example, if a service has a consistent error rate that slowly drains the budget, the longer window provides better confidence that it’s not just a transient blip. --- ## Issues The Issues view presents the outcomes of the causal reasoning engine so you see what explains the [Signals](/reference/symptoms/) and problems surfaced in your environment. Instead of receiving an alert per Signal, you get a single Issues and Diagnosis that represents the underlying cause, dramatically reducing noise and accelerating response. For background on how Diagnoses are inferred, see [How Causely Works](/getting-started/how-causely-works). ## Issues view The Issues view shows the list of Issues and inferred Diagnoses, that explain the Signals observed in your environment. The list is filtered by urgency and state, and can be further filtered by scope, service, and workload. ### Urgency **Urgent Issues** are the ones that are currently impacting services and SLOs. They are marked red. **Non-urgent Issues** are the ones that are not currently impacting services and SLOs but indicate elevated risk. Both types are valuable: **urgent items** drive incident response; non‑urgent items drive prevention and continuous improvement. ### States - **Active**: The Issue's Signals are currently present, or cleared within the last 24 hours. - **Historical**: The Issue was active in the past and has since cleared. Useful for learning and trend analysis. - **Hidden**: The Issue exists but is suppressed from the default list (for example via filters or muted scopes). You can reveal hidden items when needed. :::info A Diagnosis for the underlying problem of an Issue is no longer active once all of its Signals pass their deactivation window, see [Signal Delay](/configuration/symptom-delay/). The Issue remains Active for a minimum of 24 hours after all Signals deactivate, and longer if a Signal recurs within that window. ::: ### Filter the list Use filters to focus on what matters most: - **Scope**: Limit Diagnoses to a specific scope (for example an environment or domain). - **Service**: Focus on a particular service. - **Workload**: Narrow down to a deployment, job, or workload of interest. We will add a screenshot of the filtering controls here. ## Inspect a single Diagnosis Opening a Issue shows a few with two tabs: - **Summary**: A clear explanation including description, current and potential impact, blast radius, recommended remediation steps, and the evidence supporting the inference. - **Causality graph**: A visual explanation of how the system arrived at the Diagnosis, showing the relationships between the observed Signals and the inferred Diagnosis. - **Timeline**: A visualization of all Diagnoses, Signals, events, and SLO breaches associated with the Issue, from first detection to resolution. ### Description A high level description of the current Diagnosis is shown in the summary tab. ### Impact and Blast Radius A list of services degraded by the Issue is shown in the summary tab, and a "blast radius" diagram is shown to help visualize the scope of the impact. ### Remediation A list of recommended remediations to fix the Issue is shown in the summary tab. ### Evidence A list of evidence supporting the inference is shown in the summary tab. This includes observed [Signals](/reference/symptoms/), exceptions, logs, and events. --- ## Troubleshooting Telemetry Sources If entities and connections are not appearing in your Causely topology, check the [Integrations page](https://portal.causely.app/integrations) for errors. This guide helps you diagnose and resolve common issues with telemetry source integrations. ## Identifying Issues ### Check the Integrations Page 1. Navigate to the [Integrations page](https://portal.causely.app/integrations) in the Causely UI 2. Look for any integrations showing error states (red indicators or error messages) 3. Click on an integration to view detailed error information Common error indicators include: - **Initialization errors**: The scraper failed to start - **Authentication errors**: Invalid credentials or missing permissions - **Connection errors**: Unable to reach the telemetry source - **Configuration errors**: Missing or incorrect configuration values ## Common Issues and Solutions ### Authentication and Credentials #### Invalid or Expired Credentials **Symptoms:** - Error messages mentioning "authentication failed", "unauthorized", or "invalid credentials" - Integration shows as "Initialization Failed" **Solutions:** 1. **Verify credentials are correct:** - Check that API keys, tokens, or passwords are not expired - Ensure credentials match the correct account or environment - Verify there are no extra spaces or special characters when copying credentials 2. **For Kubernetes secrets:** ```bash # Verify the secret exists and contains the correct data kubectl get secret -n causely -o yaml # Check that the secret keys match what Causely expects # (e.g., apiKey, appKey, token, etc.) ``` 3. **For cloud provider integrations (AWS, Azure, GCP):** - Verify IAM roles or service accounts have the required permissions - Check that access keys haven't been rotated - Ensure service account annotations are correctly set for IRSA (AWS) or Workload Identity (GCP) #### Missing Secret Labels (Autodiscovery) **Symptoms:** - Integration not appearing in the Integrations page - Scraper not starting despite secret existing **Solutions:** 1. **Verify the secret has the correct label:** ```bash # Check if the secret has the scraper label kubectl get secret -n causely --show-labels # Add the label if missing (example for PostgreSQL) kubectl --namespace causely label secret "causely.ai/scraper=Postgresql" ``` 2. **Common scraper label values:** - `Postgresql` for PostgreSQL - `MySQL` for MySQL - `Datadog` for Datadog - `AWS` for AWS - `Azure` for Azure - `GCP` for GCP - `Instana` for Instana - `Dynatrace` for Dynatrace - `Confluent` for Confluent - `Snowflake` for Snowflake ### Network Connectivity #### Cannot Reach Telemetry Source **Symptoms:** - Connection timeout errors - "Unable to connect" or "network unreachable" messages - Integration shows intermittent failures **Solutions:** 1. **Verify network connectivity from the mediator pod:** ```bash # Get the mediator pod name kubectl get pods -n causely -l app=mediator # Test connectivity to the telemetry source kubectl exec -n causely -- curl -v ``` 2. **Check firewall rules:** - Ensure the mediator can reach external APIs (for cloud providers) - Verify internal network policies allow communication - Check if the telemetry source requires allowlisting the mediator's IP 3. **For Prometheus/Istio integrations:** - Verify the Prometheus endpoint is accessible from the mediator - Check that the endpoint URL is correct (including protocol: `http://` or `https://`) - Ensure Prometheus is not behind authentication that requires additional configuration ### Configuration Errors #### Missing Required Configuration **Symptoms:** - "Configuration error" or "missing required field" messages - Integration fails to initialize **Solutions:** 1. **Verify all required fields are set:** - Check the specific telemetry source documentation for required configuration - Ensure secrets contain all required keys - Verify values.yaml or configuration files have all necessary settings 2. **Common missing fields:** - **Database integrations**: Missing `host`, `port`, `database`, or `username` - **Cloud providers**: Missing `region` or `account ID` - **APM tools**: Missing `apiKey`, `appKey`, or `org` (for Datadog) - **Prometheus**: Missing `endpoint` URL 3. **Check configuration format:** ```bash # Validate Helm values helm template causely oci://us-docker.pkg.dev/public-causely/public/causely --values values.yaml --dry-run # Check mediator logs for configuration errors kubectl logs -n causely -l app=mediator --tail=100 ``` #### Incorrect Endpoint URLs **Symptoms:** - Connection refused errors - 404 Not Found errors - SSL/TLS certificate errors **Solutions:** 1. **Verify endpoint URLs:** - Ensure URLs include the correct protocol (`http://` or `https://`) - Check that ports are correct (for example, `:4317` for OTLP, `:9090` for Prometheus) - Verify hostnames resolve correctly 2. **For HTTPS endpoints:** - Check if self-signed certificates require additional configuration - Verify certificate validity if using custom certificates ### Permission Issues #### Insufficient Permissions **Symptoms:** - "Access denied" or "Forbidden" errors - Integration can connect but cannot read data - Partial data discovery (some entities missing) **Solutions:** 1. **For cloud provider integrations:** **AWS:** - Verify IAM role or user has required read permissions - Check policies include: `ReadOnlyAccess` or specific service read permissions - For EKS, ensure IRSA (IAM Roles for Service Accounts) is configured correctly **Azure:** - Verify service principal has "Reader" role or equivalent - Check that the subscription is accessible - Ensure Managed Identity is configured if using that method **GCP:** - Verify service account has required IAM roles - Check Workload Identity is configured for GKE - Ensure project-level permissions are correct 2. **For database integrations:** - Verify database user has `SELECT` permissions on system tables - Check that the user can access `pg_stat_statements` (PostgreSQL) or equivalent - Ensure the user has permissions to query metadata tables 3. **For Kubernetes:** - Verify the mediator service account has necessary RBAC permissions - Check ClusterRole and ClusterRoleBinding are correctly configured - Ensure the mediator can list and get resources in target namespaces ### Version and Compatibility Issues #### Incompatible Versions **Symptoms:** - API errors mentioning version incompatibility - Unexpected response formats - Features not working as expected **Solutions:** 1. **Check Causely version compatibility:** - Review the telemetry source documentation for version requirements - Ensure you're using a supported version of Causely - Check changelog for breaking changes 2. **Verify telemetry source versions:** - Some integrations require minimum versions of the source system - Check if API versions have changed - Review telemetry source release notes ## Platform-Specific Troubleshooting ### Kubernetes **Common issues:** - Service account permissions - Network policies blocking communication - Resource quotas limiting scraper execution **Debug steps:** ```bash # Check mediator pod status kubectl get pods -n causely # View mediator logs kubectl logs -n causely -l app=mediator --tail=200 # Check service account kubectl get serviceaccount -n causely # Verify RBAC permissions kubectl get clusterrolebinding | grep causely ``` ### Docker **Common issues:** - Container networking issues - Volume mount permissions - Environment variable configuration **Debug steps:** ```bash # Check container status docker ps | grep causely # View container logs docker logs # Verify environment variables docker exec env | grep CAUSELY ``` ### Nomad **Common issues:** - Job allocation failures - Network configuration - Volume access issues **Debug steps:** ```bash # Check job status nomad job status causely # View allocation logs nomad alloc logs # Check job specification nomad job inspect causely ``` ## Metrics and Trace Volume ### Metrics appear missing or don't match external monitoring tools **Symptoms:** - Causely metrics graphs show gaps or periods with no data - Metric values in Causely differ from what you see in Prometheus, Datadog, or other external tools **Solutions:** Causely is designed to work from a **statistically significant sample** of telemetry, not 100% of raw data. Minor differences between Causely metrics and external tools are expected and do not affect analysis accuracy. If you observe **persistent gaps** (extended periods with no data at all), VictoriaMetrics is likely under-resourced for your environment: 1. **Check VictoriaMetrics memory usage:** ```bash kubectl top pod -n causely -l app=victoriametrics ``` 2. **Increase VictoriaMetrics memory allocation** by following the [VictoriaMetrics sizing guide](/installation/customize#sizing-victoriametrics-for-high-volume-environments). ### Traces appear to be dropped **Symptoms:** - Trace counts in Causely are lower than what your instrumentation or external tracing backend reports - You notice the Mediator pod's memory usage is near its limit **Explanation:** The Mediator's built-in OpenTelemetry Collector has a memory limiter that drops excess traces when memory pressure is reached. This is **intentional behavior**: it means the Mediator has already collected a sufficient sample of traces for analysis. Causely does not require every trace to produce a Diagnosis. Mediator memory is sized by the **number of entities under management** (services, pods, databases, and other components in your environment), not by trace or request throughput. If your cluster is growing, you may need to increase Mediator memory. See [Sizing the Mediator for large environments](/installation/customize#sizing-the-mediator-for-large-environments). If you need full-fidelity trace retention for other purposes (for example, debugging individual requests), you can route traces through an intermediate collector that fans out to both Causely and your tracing backend (for example, Grafana Tempo). If you are using Causely's built-in Beyla instrumentation, redirect Beyla's trace export to your OpenTelemetry Collector or Grafana Alloy instance by setting the following in your `causely-values.yaml`: ```yaml agent: beyla: otel_traces_export: endpoint: http://:4317 ``` Then configure your collector to export traces to **both** your tracing backend and the Causely Mediator (`mediator.causely:4317`). See [Integrating OpenTelemetry with Causely](/telemetry-sources/opentelemetry#integrating-opentelemetry-with-causely) and [Grafana integration](/telemetry-sources/grafana) for collector configuration details. ## Getting Additional Help If you've tried the solutions above and are still experiencing issues: 1. **Check mediator logs** for detailed error messages: ```bash kubectl logs -n causely -l app=mediator --tail=500 ``` 2. **Review integration-specific documentation:** - [AWS Integration](/telemetry-sources/aws) - [Azure Integration](/telemetry-sources/azure) - [GCP Integration](/telemetry-sources/gcp) - [PostgreSQL Integration](/telemetry-sources/postgresql) - [MySQL Integration](/telemetry-sources/mysql) - [Datadog Integration](/telemetry-sources/datadog) - [Instana Integration](/telemetry-sources/instana) - [Dynatrace Integration](/telemetry-sources/dynatrace) 3. **Contact Causely Support:** - Email: [support@causely.ai](mailto:support@causely.ai) - Include: - Integration name and error message - Relevant log excerpts (redact sensitive information) - Configuration details (redact credentials) - Steps you've already tried ## Prevention Best Practices 1. **Use autodiscovery labels** to ensure secrets are properly recognized 2. **Test connectivity** before configuring integrations 3. **Verify credentials** are valid and have appropriate permissions 4. **Monitor integration health** regularly in the Integrations page 5. **Keep Causely updated** to ensure compatibility with latest telemetry sources 6. **Document your configuration** to make troubleshooting easier --- ## Reference The Reference section provides comprehensive documentation about the entities Causely discovers, the Diagnoses it produces, key terminology, and security information. This documentation helps you understand what types of entities it discovers in your environment, the Diagnoses it can produce, and the terminology used throughout the documentation. ## Entity Types Causely automatically discovers over 25 different entity types from your cloud native environment through data sources like eBPF, Cloud APIs, and OpenTelemetry. These entities are used to build topologies, identify defects, and infer Diagnoses. Learn about the different types of entities that Causely automatically discovers, including applications, services, databases, compute resources, messaging systems, and data pipelines in [Entity Types](/reference/entity-types). ## Diagnoses With more than 100 types of root causes captured in its Causal Models, the causes a Diagnosis can identify, Causely can pinpoint hundreds of thousands of potential issues and their effects within your environment. These causes span applications, infrastructure, data pipelines, release management, and services. Explore the root cause types behind each Diagnosis and how they impact your systems, from application bugs to infrastructure bottlenecks to release-related issues, in [Diagnoses](/reference/root-causes). ## Signals Signals are observable anomalies in managed objects that may be caused by root causes; formally these are Symptoms, defined in [Terminology](/reference/terminology). Causely detects a wide variety of Signals across services, workloads, compute resources, databases, messaging systems, and more. Browse the complete reference of all Signals that Causely can detect, organized by category and entity type, in [Signals](/reference/symptoms). ## Terminology Understanding the key terms and concepts used throughout Causely documentation helps you get the most out of the system. The terminology covers core concepts like entities, Issues, Diagnoses, Signals, topology, causality graphs, and more. Understand the key terms and concepts used throughout Causely documentation, grouped by relatedness to help you navigate the system effectively in [Terminology](/reference/terminology). ## Security Causely is designed to protect sensitive data and ensure privacy. The system processes telemetry data locally and primarily transmits minimal, high-level information to its backend. All data is encrypted in transit and at rest. Learn about Causely's security model, data handling practices, privacy protections, and the permissions required for deployment components in [Security](/security). --- ## Custom Agents ## Building Custom Agents with Causely Most custom agents can access telemetry, but struggle to determine what is actually happening, what caused it, and what action is safe. Causely provides the system intelligence layer needed to interpret telemetry consistently and make reliable decisions. Whether you are building internal incident tooling, AI-driven workflows, or automation pipelines, Causely provides the structured system intelligence needed to interpret telemetry and act safely. ## When to Build a Custom Agent Building a custom agent is the right approach when: - You have internal workflows that do not map to existing tools - You want to integrate reliability decisions into your own systems - You are building end-to-end automation (detection → diagnosis → action) - You need full control over logic, policies, and execution ## How It Works Custom agents use Causely to move from raw telemetry to structured, system-aware decisions. Custom agents typically interact with Causely in one of two ways: ### 1. MCP Server (Recommended) Use the Causely MCP Server to provide a standardized interface for agents. Your agent: 1. Receives a signal (alert, event, user input) 2. Queries Causely via MCP 3. Receives structured outputs (issues, diagnoses, dependencies, explanation) 4. Takes action based on those outputs This is the fastest way to integrate Causely into agent-based systems. ### 2. API Integration For more control or non-MCP environments, you can interact directly with the Causely API. Your system: 1. Sends queries to Causely (for example, root cause, topology, health) 2. Receives structured, machine-consumable responses 3. Uses those responses to drive logic and actions ## Tool Ordering: Resolve Entities First :::tip **Always call `get_entities()` before querying metrics, SLOs, topology, symptoms, or slow queries.** Most structured tools require an entity ID. `get_entities()` resolves a service or database name to its ID and returns current health status, type, and labels. ::: ## Tool Selection Choose the right tool based on what your agent needs to do with the result. | Use case | Tool | |---|---| | "What's wrong with checkout?" (narrative) | `get_service_summary` | | Issue data for automated routing | `get_issues` | | Metrics for regression detection | `get_metrics` | | Entity ID resolution | `get_entities` | | SLO status with burn rate | `get_slo` | | Blast radius mapping | `get_topology` | | Post-deploy regression check | `reliability_delta` | See the [full tool reference](/agent-integration/mcp-server#full-tool-reference) for all 30 tools. ## Workflow Example: Incident Triage Workflow This example shows a custom agent performing full incident triage using structured MCP tools. **Scenario**: An alert fires. The agent needs to identify the root cause, understand impact, and route to the correct team. ```python # Step 1: Resolve the alerted service name to an entity ID entity = mcp.call("get_entities", query="checkout-service", entity_types=["Service"]) # Returns: entity ID, current health status, type, and labels # Step 2: Get issues and diagnoses for the affected service issues = mcp.call("get_issues", impacted_service_ids=[entity[0]["id"]]) # Returns: Issues with severity, diagnoses, impacted services, and remediation guidance # Step 3: Map the blast radius for the highest-severity issue topology = mcp.call( "get_topology", entity_id=issues[0]["entity_id"], mode="dependents" ) # Returns: upstream services affected by this entity's degradation # Step 4: Route to the correct team and generate a ticket ticket = mcp.call( "generate_ticket", task=f"Investigate issue: {issues[0]['name']}" ) ``` After the incident resolves: ```python # Generate postmortem documentation postmortem = mcp.call("postmortem", issue_id=issues[0]["id"]) # Returns: structured markdown with timeline, blast radius, contributing factors, action items ``` ## What Your Agent Gets from Causely When integrated, your agent can: - Identify the true root cause of issues - Understand service dependencies and failure propagation - Evaluate blast radius before taking action - Work with structured, consistent outputs - Provide explainable decisions This allows your agent to move beyond querying data to making reliable decisions. ## Design Considerations When building custom agents with Causely: - **Trust boundaries**: define what actions can be automated vs require approval - **Policy enforcement**: gate actions based on risk or impact - **Observability**: log decisions and reasoning for auditability - **Fallbacks**: handle cases where no clear diagnosis is identified ## Next Steps - [Using the MCP Server](/agent-integration/mcp-server): full tool reference and key workflows - [HolmesGPT](/agent-integration/holmes-gpt): see a reference implementation - [API](../../api): explore direct API integration --- ## HolmesGPT ## Using Causely with HolmesGPT HolmesGPT is a pre-built agent framework for incident investigation and response. When configured with Causely, Holmes gains a live causal model of your system, so it can reason about root cause, blast radius, and remediation without requiring prompt engineering or custom context. Out of the box, Holmes can access telemetry sources such as metrics and logs. However, like most agents, it operates on raw data and correlation. This makes it difficult to consistently determine what is actually happening, what caused it, and what action is safe. Causely provides the system intelligence layer that allows Holmes to move from querying telemetry to making reliable, system-aware decisions. ## How Causely Enhances Holmes When integrated with Holmes, Causely enables: - **Deterministic root cause analysis**: identify the actual source of an issue, not just correlated signals - **Dependency-aware reasoning**: understand how failures propagate across services - **Structured outputs**: return machine-consumable results instead of raw telemetry - **Explainable decisions**: provide clear reasoning that can be audited and trusted This allows Holmes to focus on orchestrating workflows while relying on Causely for system-level understanding. ## Example Workflow **Scenario: Investigating a performance issue** 1. Holmes receives an alert or user query 2. Holmes queries Causely via MCP 3. Causely returns: - Root cause service - Affected dependencies - Explanation of the issue 4. Holmes: - Summarizes the issue - Notifies the appropriate team - Suggests or executes remediation ## Setup To integrate Causely with HolmesGPT, configure Causely as a remote MCP server. ### Holmes Configuration Add the following to your Holmes configuration: ```yaml mcp_servers: causely: description: "Causal Reasoning Platform" config: url: "https://api.causely.app/mcp" mode: streamable-http headers: Authorization: "Basic {{ env.CAUSELY_MCP_CLIENT_BASIC }}" llm_instructions: "Use Causely to investigate application performance issues, analyze distributed traces, and query infrastructure metrics. Prefer this over Prometheus for APM data." ``` ### Authentication Holmes calls the MCP server **non-interactively**, so use **HTTP Basic** with your Causely MCP **OAuth client ID** and **client secret** (machine credentials), not a Bearer API key. 1. In [API tokens](https://auth.causely.app/oauth/portal/api-tokens), create or copy the **client ID** and **client secret** for MCP access. 2. Build the standard Basic user-info string: `client_id`, a single colon (`:`), then `client_secret`, with no newline or extra characters (same encoding as `Authorization: Basic` elsewhere). 3. Base64-encode that string. Use the **raw Base64 output only** in the environment variable; the Holmes snippet above adds the `Basic ` prefix in the header value. Example (macOS or Linux): ``` export CAUSELY_MCP_CLIENT_BASIC="$(printf '%s' 'YOUR_CLIENT_ID:YOUR_CLIENT_SECRET' | base64)" ``` If your stack cannot put those credentials on `Authorization`, you can send the same payload on `X-Causely-Client-Basic` instead; see [Authentication](/agent-integration/mcp-server#authentication) on the MCP Server page for the full header table and edge cases. ## When to Use This Integration This integration is most valuable when: - You are using Holmes for incident investigation or automation - You have distributed systems with complex dependencies - You want consistent, reliable root cause analysis instead of correlation - You are looking to automate decision-making, not just data retrieval ## Notes - Causely complements existing telemetry sources such as Prometheus rather than replacing them - Holmes can continue to use other data sources, but should prioritize Causely for system-level reasoning - For full details on available tools and key workflows, see the [MCP Server documentation](/agent-integration/mcp-server) ## Next Steps - [Using the MCP Server](/agent-integration/mcp-server): full tool reference, key workflows, and tool selection guide - [Custom Agents](/agent-integration/custom-agents): build your own workflows using Causely --- ## Advanced Authentication The default browser-based OAuth flow covers most setups. Use this page when you need **machine credentials** (automation, CI, non-interactive agents) or when your client **only supports stdio** transport. ## Client ID and Client Secret {#authentication} For non-interactive MCP calls, supply client credentials that the server exchanges for a Frontegg access token. The HTTP Basic username is always `client_id` and the password is always `client_secret`. **Getting credentials** Generate OAuth client credentials for your tenant at: [https://auth.causely.app/oauth/portal/api-tokens](https://auth.causely.app/oauth/portal/api-tokens) **Encoding** Concatenate `client_id`, a single colon (`:`), and `client_secret`, no newline, no extra characters, then Base64-encode the result: ```bash printf '%s:%s' "$CLIENT_ID" "$CLIENT_SECRET" | base64 -w 0 ``` Use the single line of output as the Base64 payload. **How to send it** | Method | When to use | |---|---| | `Authorization: Basic ` | Requests with no `Authorization: Bearer` token, typical for `curl` or custom HTTP clients | | `X-Causely-Client-Basic` | Same Base64 payload (with or without the `Basic ` prefix). Use when an MCP proxy reserves or rewrites `Authorization`. If both headers are present, the server prefers `X-Causely-Client-Basic`. | If the request already includes a non-empty `Authorization: Bearer` token, the server validates that JWT and ignores client credentials entirely. **Adding credentials to a client config** For native HTTP MCP clients, add a `headers` object alongside `url`. Examples for each config format: JSON `mcpServers` (Claude Code, Cursor): ```json { "mcpServers": { "causely": { "type": "http", "url": "https://api.causely.app/mcp", "headers": { "X-Causely-Client-Basic": "Basic " } } } } ``` JSON `servers` (VS Code / GitHub Copilot): ```json { "servers": { "causely": { "type": "http", "url": "https://api.causely.app/mcp", "headers": { "X-Causely-Client-Basic": "Basic " } } } } ``` TOML (Codex), use `env_http_headers` to read the value from an environment variable at runtime so secrets stay out of the file: ```toml [mcp_servers.causely] url = "https://api.causely.app/mcp" enabled = true [mcp_servers.causely.env_http_headers] "X-Causely-Client-Basic" = "CAUSELY_MCP_CLIENT_BASIC" ``` Export `CAUSELY_MCP_CLIENT_BASIC` to the Base64 string (without a `Basic ` prefix, Codex does not prepend it for you unless you include it in the variable value). :::caution Do not commit real secrets to source control. Use your tool's secret or input-variable mechanism where available, for example VS Code MCP **`inputs`** and `${input:…}` in `headers`. ::: ## Stdio Fallback (mcp-remote) {#stdio-fallback-mcp-remote} If your client cannot use streamable HTTP to `https://api.causely.app/mcp`, run [`mcp-remote`](https://www.npmjs.com/package/mcp-remote) as a local stdio bridge: your tool launches Node locally; `mcp-remote` handles OAuth and forwards MCP traffic over HTTP to Causely. Requires **Node.js** and `npx` on your PATH. **Browser OAuth (default for this path):** ```json { "mcpServers": { "causely": { "command": "npx", "args": ["mcp-remote", "https://api.causely.app/mcp/"] } } } ``` **Machine credentials:** `mcp-remote` accepts repeated `--header "Name: value"` flags and expands `${ENV_VAR}` inside header values: ```json { "mcpServers": { "causely": { "command": "npx", "args": [ "mcp-remote", "https://api.causely.app/mcp/", "--header", "X-Causely-Client-Basic: Basic ${CAUSELY_MCP_CLIENT_BASIC}" ] } } } ``` Set `CAUSELY_MCP_CLIENT_BASIC` to the Base64 string without the `Basic ` prefix, the snippet above adds the prefix in the header string. --- ## Claude Code Connect Causely to Claude Code to run incident triage, service health checks, and reliability reports without leaving the terminal. ## Prerequisites - Active [Causely account](https://portal.causely.app/) - [Claude Code](https://docs.anthropic.com/en/docs/claude-code) installed ## Configuration Run once to register Causely across all your projects: ```bash claude mcp add --scope user --transport http causely https://api.causely.app/mcp ``` This writes to `~/.claude.json` automatically. No file editing required. **Team-shared config (optional)** To register Causely for everyone who clones a specific repo, create `.mcp.json` at the repository root instead: ```json { "mcpServers": { "causely": { "type": "http", "url": "https://api.causely.app/mcp" } } } ``` Only use this if authentication is handled separately, never commit credentials. See [Advanced Authentication](/agent-integration/mcp-server/advanced-auth). Begin a new Claude Code session (claude) after any config change. Run /mcp to confirm causely appears as a connected server. ## Adding Skills Seven skills activate automatically once installed: one master router (`causely-mcp`) plus six specialists. See the [Skills page](/agent-integration/mcp-server/skills) for the full list, trigger phrases, and override options. **Install** ```bash git clone https://github.com/causely-oss/causely-client mkdir -p ~/.claude/skills cp -r causely-client/skills/* ~/.claude/skills/ ``` This installs to personal scope (`~/.claude/skills/`) so skills are available across all your projects. To commit skills to a single repo instead, replace `~/.claude/skills/` with `.claude/skills/` in each path above. **Restart** Begin a new Claude Code session (`claude`) after installing. **Verify** Try: *"What's broken right now?"* The `causely-health-reporting` skill should activate. ## Try It Now - *"List my clusters."* - *"Are there any active symptoms right now?"* - *"What services are currently degraded?"* - *"Draft a postmortem for the most recent incident."* ## Known Gotcha Project .mcp.json is committed to source control. Never add credentials directly to this file. Use --scope user when registering Causely so auth stays out of the repository. See [Advanced Authentication](/agent-integration/mcp-server/advanced-auth). --- ## Claude Desktop Ask Causely questions in natural language from the Claude desktop app, no terminal, no code. ## Prerequisites - Active [Causely account](https://portal.causely.app/) - [Claude Desktop](https://claude.ai/download) installed - [Node.js](https://nodejs.org/) on the PATH visible to the app (required for `mcp-remote`) ## Configuration There are a few ways to connect depending on how you use Claude: browser or desktop, and UI or config file. ### Claude.ai (browser) **Pro / Max** 1. Go to claude.ai → Customize → Connectors 2. Click + then Add custom connector 3. Enter a name (for example Causely) and the server URL: https://api.causely.app/mcp 4. Click Add 5. Sign in to Causely and grant access 6. Return to Claude.ai, the connector is now active **Team / Enterprise** 1. Go to Organization settings → Connectors 2. Click Add → Custom → Web 3. Enter a name (for example Causely) and the server URL: https://api.causely.app/mcp 4. Click Add 5. Team members authenticate individually at Customize → Connectors ### Claude Desktop **Desktop UI** 1. Go to Settings → Connectors 2. Click Customize 3. Click + sign and select Add custom connector 4. Enter a name (for example Causely) and the server URL: https://api.causely.app/mcp 5. Click Add 6. Sign in to Causely and grant access 7. Return to Claude desktop, the connector is now active **Developer desktop app** 1. Go to Settings → Developer 2. Edit Config 3. Edit `claude_desktop_config.json` and merge the `mcpServers` object: ```json { "mcpServers": { "causely": { "command": "npx", "args": ["mcp-remote", "https://api.causely.app/mcp"] } } } ``` Or you can edit the config file directly, **Config file location**: | Platform | Path | |---|---| | macOS | `~/Library/Application Support/Claude/claude_desktop_config.json` | | Windows | `%APPDATA%\Claude\claude_desktop_config.json` | **Restart** Quit and reopen Claude Desktop. On first connect, a browser window opens for Causely OAuth login. After you authorize, Claude Desktop stores the token and reconnects automatically on subsequent launches. ## Adding Skills Seven skills activate automatically once installed: one master router (`causely-mcp`) plus six specialists. See the [Skills page](/agent-integration/mcp-server/skills) for the full list, trigger phrases, and override options. **Install** Ensure that Skills are enabled for your Enterprise or Team plan. 1. Clone the Causely client repo ```bash git clone https://github.com/causely-oss/causely-client cd causely-client/skills ``` 2. Create a .zip file for each sub folder in the skills folder macOS ```bash for dir in causely-mcp causely-postmortem causely-k8s-investigation causely-health-reporting causely-correlated-incidents causely-change-impact causely-alert-triage; do zip -r "${dir}.zip" "$dir" \ --exclude "*/.DS_Store" \ --exclude "__MACOSX/*" \ --exclude "*/.git/*" \ --exclude "*/Thumbs.db" done ``` Windows ```powershell foreach ($dir in @( "causely-postmortem", "causely-mcp", "causely-k8s-investigation", "causely-health-reporting", "causely-correlated-incidents", "causely-change-impact", "causely-alert-triage" )) { $tmp = New-Item -ItemType Directory -Path "$env:TEMP\$dir" Copy-Item -Path $dir\* -Destination $tmp -Recurse -Exclude @('.DS_Store','Thumbs.db') Compress-Archive -Path $tmp -DestinationPath "$dir.zip" -Force Remove-Item $tmp -Recurse -Force } ``` 3. Go to Customize → Skills 4. Click + sign and select Create skill → Upload a skill 5. For each of the 7 `.zip` files: click **+** → **Create skill** → **Upload a skill**, select the file, and save. Repeat until all 7 are uploaded. Each upload creates one skill. **Restart** Quit and reopen Claude Desktop after installing. **Verify** Try: *"What's broken right now?"* The `causely-health-reporting` skill should activate. ## Try It Now - *"List my clusters."* - *"Are there any active symptoms right now?"* - *"What services are currently degraded?"* - *"Draft a postmortem for the most recent incident."* ## Known Gotcha Claude Desktop launches as a GUI app and does not inherit your shell's `PATH`. If `npx` is not found at launch, install Node.js globally, not via `nvm` or a shell-managed version manager, so the binary is visible to the app process. On macOS, the [official Node.js installer](https://nodejs.org/) or `brew install node` is the most reliable approach. --- ## OpenAI Codex Connect Causely to OpenAI Codex to run reliability checks and incident triage alongside your coding workflows. ## Prerequisites - Active [Causely account](https://portal.causely.app/) - [OpenAI Codex](https://openai.com/index/openai-codex/) CLI installed ## Configuration Add the following to `~/.codex/config.toml` or project `.codex/config.toml`: ```toml [mcp_servers.causely] url = "https://api.causely.app/mcp" enabled = true ``` **Config file location** | Scope | Path | |---|---| | User | `~/.codex/config.toml` | | Project | `.codex/config.toml` at the repository root | ## Restart Codex reads `config.toml` once at launch. After editing, start a new Codex session to pick up the change. :::tip Skills are not yet supported for Codex. You'll interact with the Causely MCP server directly; the example prompts in the [Key Workflows](/agent-integration/mcp-server#key-workflows) section still work, but tool selection is handled by Codex's agent rather than a Causely skill router. ::: ## Try It Now - *"List my clusters."* - *"Are there any active symptoms right now?"* - *"What services are currently degraded?"* ## Known Gotcha For machine credentials, use `env_http_headers` (reads from environment variables at runtime) instead of `http_headers` (static values in the file) to keep secrets out of the config: ```toml [mcp_servers.causely] url = "https://api.causely.app/mcp" enabled = true [mcp_servers.causely.env_http_headers] "X-Causely-Client-Basic" = "CAUSELY_MCP_CLIENT_BASIC" ``` Export `CAUSELY_MCP_CLIENT_BASIC` to the Base64-encoded `client_id:client_secret` string before launching Codex. See [Advanced Authentication](/agent-integration/mcp-server/advanced-auth) for encoding instructions. --- ## Cursor Connect Causely to Cursor to investigate incidents and check service health inline while you code. ## Prerequisites - Active [Causely account](https://portal.causely.app/) - [Cursor](https://www.cursor.com/) with MCP support (1.0+) ## Installation ### Option 1: Cursor plugin (recommended) Install the Causely plugin from [cursor.directory/plugins/causely](https://cursor.directory/plugins/causely) in one click. It configures the MCP server and installs all seven skills automatically, no manual steps required. After installing, open **Settings → MCP** and confirm `causely` appears with a green status indicator. ### Option 2: Manual setup **MCP server** Create or update `.cursor/mcp.json` at the root of your project: ```json { "mcpServers": { "causely": { "type": "http", "url": "https://api.causely.app/mcp" } } } ``` | Scope | Path | |---|---| | User (all projects) | `~/.cursor/mcp.json`, open via **Settings → MCP → Edit Config** | | Project | `.cursor/mcp.json` at the project root | **Add Skills** ```bash git clone https://github.com/causely-oss/causely-client mkdir -p .cursor/skills cp -r causely-client/skills/* .cursor/skills/ ``` Cursor skills are project-scoped; there is no personal skills directory. Run these commands at the root of each project where you want skills available. **Restart** After saving the config, open **Settings → MCP**, `causely` should appear with a green status indicator. If it does not, use **Cmd/Ctrl+Shift+P → MCP: Restart MCP Server** or restart Cursor entirely. **Verify** Try: *"What's broken right now?"* The `causely-health-reporting` skill should activate. ## Try It Now - *"List my clusters."* - *"Are there any active symptoms right now?"* - *"What services are currently degraded?"* - *"Draft a postmortem for the most recent incident."* ## Known Gotcha Cursor caches MCP tool descriptions at session start. If you update the server config or Causely releases new tools, the previous tool list persists until you explicitly restart the MCP server or restart Cursor. See [Advanced Authentication](/agent-integration/mcp-server/advanced-auth) if you need to add machine credentials. --- ## MCP Server Integration The Causely MCP server gives agents and AI assistants direct access to Causely's causal reasoning engine. 33 tools across 5 categories let your agent move from raw alerts to structured root cause analysis, dependency maps, and reliability reports, without writing custom integrations. ## Key Workflows These are the four workflows agents use most often. Each maps to a specific sequence of MCP tool calls. ### Incident Triage Identify what's broken and how far it has spread. **Try:** - *"What's broken in production right now?"* - *"Checkout is throwing errors and we have Alertmanager alerts firing. What's the actual root cause?"* - *"Three services are alerting at once. Which is the real problem and which are downstream noise?"* 1. `get_issues()`: list active Issues, the primary incident-level view; drill into one with `get_issue_details(issue_id=...)` 2. `get_symptoms()`: see all active Signals across the entire environment (no filters needed) 3. `get_diagnoses()`: identify all active diagnoses and impacted services 4. `get_alerts(alert_name_filters=...,)`: drill into a specific alert's cause 5. `get_topology(entity_id=..., mode="dependents")`: map which upstream services are affected Handled automatically by `causely-correlated-incidents`, or `causely-alert-triage` if you're starting from a specific alert. ### Quick Service Health Get a complete health picture for a specific service in two calls. **Try:** - *"Is checkout healthy?"* - *"Give me a full health picture for the payments service: status, open issues, SLOs."* - *"Before I page anyone, is there actually a problem with database-service or is this alert noise?"* - *"Are there are any concerning errors or warnings in the logs for the frontend service over the last hour?"* 1. `get_entities(query="service-name", entity_types=["Service"])`: resolve the service name to its entity ID 2. `get_service_summary(service="service-name")`: full snapshot: status, active signals, diagnoses, issues, SLOs, metrics, recent events, error logs 3. `get_logs(entity_id=...,)`: retrieves live log output for a running service Handled automatically by the `causely-health-reporting` skill. ### Post-Deploy Validation Check whether a deployment introduced regressions. **Try:** - *"Did the last deploy to payments cause any regressions?"* - *"We deployed cart service 30 minutes ago. How does it look compared to before?"* - *"Check all services my team owns, did anything degrade after today's deploys?"* 1. `reliability_delta(service="service-name")`: compare CPU, memory, latency, and error rate before vs after the most recent deployment 2. `fleet_reliability_delta(team="team-name")`: batch check across all services for a team, namespace, or explicit list Handled automatically by the `causely-change-impact` skill. ### Post-Incident Reporting Generate postmortem documentation and action items from a resolved incident. **Try:** - *"The payments outage is resolved. Draft a postmortem."* - *"Write up what happened to checkout this morning: timeline, issue, diagnosis, and what was affected."* - *"Payments is back up. Draft the postmortem and create a follow-up ticket for the team."* 1. `get_issue_details(issue_id=...)`: retrieve full diagnosis details, timeline, and blast radius 2. `postmortem(issue_id=...)`: generate a structured postmortem draft 3. `generate_ticket(task="...")`: create a follow-up engineering ticket for Jira, GitHub Issues, or Linear Handled automatically by the `causely-postmortem` skill. ## Skills (Recommended) Skills automate the tool-selection step shown in the workflows above. You describe your situation in natural language; the right specialist activates and runs the correct tool sequence for you. Skills are available for Claude Code, Claude Desktop, and Cursor. | Situation | Skill | Try | |---|---|---| | Incoming alert | `causely-alert-triage` | *"PagerDuty just paged for checkout-latency. What's the actual cause?"* | | Post-deploy validation | `causely-change-impact` | *"Did the last deploy to payments cause any regressions?"* | | Multi-service outage | `causely-correlated-incidents` | *"Three services are alerting at once. What's the real problem?"* | | Health summary / morning standup | `causely-health-reporting` | *"Give me a morning health report for production."* | | Kubernetes investigation | `causely-k8s-investigation` | *"The orders pod keeps OOMKilling. Why?"* | | Postmortem / ticket | `causely-postmortem` | *"Draft a postmortem for the checkout outage that resolved an hour ago."* | See the [Skills page](/agent-integration/mcp-server/skills) for install instructions, full skill detail, and override options. ## Choose Your Client Select your tool for a copy-paste config snippet, config file location, and restart instructions. | Client | Transport | Config format | |---|---|---| | [Claude Code](/agent-integration/mcp-server/claude-code) | HTTP | `.mcp.json` (`mcpServers`) | | [Claude Desktop](/agent-integration/mcp-server/claude-desktop) | stdio via `mcp-remote` | `claude_desktop_config.json` (`mcpServers`) | | [Codex](/agent-integration/mcp-server/codex) | HTTP | `config.toml` (`mcp_servers`) | | [Cursor](/agent-integration/mcp-server/cursor) | HTTP | `.cursor/mcp.json` (`mcpServers`) | | [VS Code (GitHub Copilot)](/agent-integration/mcp-server/vscode-copilot) | HTTP | `.vscode/mcp.json` (`servers`) | **Verify your connection** by asking: _”Causely: What defects are currently active?”_ ## Other MCP-compatible Clients The clients above have dedicated setup pages. The following tools also support the Causely MCP server, point them at `https://api.causely.app/mcp` using your tool's HTTP MCP config. See [Advanced Authentication](/agent-integration/mcp-server/advanced-auth) for credential options. **IDEs and Editors:** JetBrains IDEs (IntelliJ IDEA, PyCharm, WebStorm, GoLand, and others), Windsurf, Zed **CLIs:** Kiro CLI, Amp, Atlassian Rovo DEV CLI, and other MCP-compatible CLI tools **Agent Frameworks:** HolmesGPT ## Authentication {#authentication} The MCP server validates Frontegg-issued Bearer tokens. For most clients, browser-based OAuth runs automatically, no manual setup needed. For non-interactive setups (automation, CI) or clients that only support stdio, including the [stdio/mcp-remote fallback](/agent-integration/mcp-server/advanced-auth#stdio-fallback-mcp-remote), see [Advanced Authentication](/agent-integration/mcp-server/advanced-auth). ## Using the Tool Reference :::tip **The reference below is for teams building custom agents that need explicit tool control.** If you're using Claude, Cursor, Codex, or any conversational agent, you can skim it for capability awareness. In most cases, you can describe what you want and the agent picks the right tools. One thing worth knowing if you do go programmatic: most structured tools require an entity ID, so `get_entities()` is usually the right first call. ::: ## What Agents Get vs Raw Telemetry | | Raw telemetry | Causely MCP | |---|---|---| | Root cause identification | Correlation-based, requires analysis | Deterministic causal analysis | | Dependency awareness | Manual mapping required | Live topology from observed traffic | | Blast radius | Estimated | Computed from causal graph | | Structured output | Custom parsing required | Typed tool responses | | Time to insight | Minutes of analysis | Single tool call | ## Full Tool Reference 33 tools across 5 categories. All tools are available to any MCP-compatible agent or assistant. ### Entity Resolution | Tool | When to use | |---|---| | `get_entities` | **Start here.** Resolve a service or database name to its ID; list all entities in a namespace; check current health status | | `name_lookup` | Resolve any name, including service, cluster, namespace, issue name, or symptom name, to an entity ID for use in other tools | | `get_label_values` | Enumerate valid label values (team, product, cluster, namespace) before fanning out queries across environments | | `count_entities` | Count entities grouped by cluster, namespace, customer, product, or type to grasp the shape of an environment in far fewer calls, without listing every entity | ### Data Retrieval | Tool | When to use | |---|---| | `get_metrics` | Retrieve numeric metric data (p95 latency, error rate, CPU, memory, throughput): the only tool that returns time-series | | `get_logs` | Inspect live service logs, or retrieve evidence logs captured at diagnosis detection time | | `get_alerts` | Start triage from an alert name (PagerDuty, Slack, Datadog); distinguish alerts mapped to causal analysis from noise | | `get_events` | Correlate symptom onset with deployments, restarts, scaling events, or config changes | | `get_config` | Investigate configuration drift; verify deployment manifest matches expectations | | `get_slow_queries` | Identify database queries consuming the most execution time; follow up on database diagnosis | ### Health & Diagnosis | Tool | When to use | |---|---| | `get_issues` | List active Issues. An Issue groups related diagnoses for an affected entity into one incident with a designated primary diagnosis; start incident investigation at the Issue level | | `get_issue_details` | Full detail for one Issue by ID: the primary diagnosis evidence (causal chain, blast radius) plus every diagnosis grouped under it. Follow-up to `get_issues` | | `get_symptoms` | Call with no filters to see all active symptoms across the entire environment or filter for specific entity, namespace or cluster | | `get_diagnoses` | Identify active diagnoses; filter by impacted service, symptom, diagnosis ID, or a start/end date range; use start/end dates when investigating a specific past time window | | `get_diagnosis_details` | Follow-up to `get_diagnoses`: full evidence for one diagnosis given a `diagnosis_id`; includes the causal chain explaining why it was identified, blast radius, symptoms, exceptions, events, and logs | | `get_entity_health` | Structured health summary for non-Service entities (databases, pods, queues, topics, tables) | | `get_environment_health` | Structured health summary for the environment, can be scoped to specific namespaces or services | | `get_slo` | Check SLO state, error budget remaining, and burn rate | | `get_topology` | Find upstream blast radius (dependents), downstream dependencies, or full data-flow graph | | `get_integration_status` | Verify monitoring coverage; check scraper health by cluster | | `get_incident_impact` | Given a Issue ID (or an entity ID + Issue name), returns the responsible service and its business context, plus all impacted services and their business context | | `team_health` | Health summary for all services owned by a team; degraded and critical services listed first | | `get_service_summary` | Comprehensive health snapshot for a single service: status, symptoms, diagnoses, SLOs, metrics, events, logs | | `investigate_alert` | Investigate a resolved alert from `get_alerts`; maps the alert to its entity and returns the standard `get_entity_health` result alongside the original alert | | `rank_entities` | Rank services, topics, tables, or endpoints by number of dependencies or dependents, a single bulk query instead of looping `get_topology` | | `get_potential_diagnoses` | Active and model-inferred diagnosis hypotheses for a specific entity; includes inactive and causality-only potentials not returned by `get_diagnoses` | | `get_potential_observable_signals` | All observable signals on a specific entity (active, inactive, and causality potential state); use before `get_signal_potential_diagnoses` to find internal signal names | | `get_signal_potential_diagnoses` | Reverse lookup: given a symptom, event, or SLO on an entity, return the diagnosis from the causality model that could explain it | | `get_diagnosis_observable_signals` | Retrieve the theoretical causality chain for a diagnosis, including downstream symptoms, events, and SLOs it could cause according to the causal model; compare to observed signals from `get_symptoms` | ### Reporting & Postmortems | Tool | When to use | |---|---| | `postmortem` | Generate a deterministic postmortem draft for a resolved incident from Causely data | | `generate_ticket` | Create a structured engineering ticket suitable for Jira, GitHub Issues, or Linear | ### Reliability & Deployment | Tool | When to use | |---|---| | `reliability_delta` | Post-deploy regression check for a single service: compare resource consumption before/after most recent deployment | | `fleet_reliability_delta` | Batch regression check across a team, namespace, or explicit service list (up to 20 services per call) | ## Feature Demos ### Solving Slow Database Queries ### Helm Chart Example --- ## Skills Skills eliminate the need to pick the right MCP tool manually. You describe your situation; the `causely-mcp` router picks the matching specialist; the specialist runs the correct tool sequence for you. All 29 MCP tools are covered across 7 skills: one master router plus 6 specialists. ## How it works `causely-mcp` is the master router that sits above the six specialist skills. It activates on any observability or reliability question and matches your prompt against each specialist's trigger set, delegating to the closest fit. The router also pulls in `complete-investigation.md` as a shared reference so specialists can coordinate context across multi-step investigations. You never invoke a specialist directly; the router handles that automatically, though you can [override it](#overrides-and-escape-hatches) when needed. ## Client support | Client | Skills | Install | |---|---|---| | Claude Code | Yes | [Manual (copy from repo)](/agent-integration/mcp-server/claude-code) | | Claude Desktop | Yes | [Manual (copy from repo)](/agent-integration/mcp-server/claude-desktop) | | Cursor | Yes | [Plugin (one-click)](/agent-integration/mcp-server/cursor) or [Manual](/agent-integration/mcp-server/cursor#option-2-manual-setup) | | Codex | Not yet supported | | | VS Code Copilot | Not yet supported | | ## causely-alert-triage Activates on any message that names or describes an incoming alert from PagerDuty, Datadog, Prometheus/Alertmanager, Slack, or OpsGenie, or asks what a firing alert means. **Example trigger prompts** - *"PagerDuty is firing P1 for checkout-service. What's the root cause?"* - *"I have three Datadog alerts going off at once. Which one actually matters?"* - *"OpsGenie woke me up at 2 AM for payment-processor. Is this real or noise?"* - *"Alertmanager is showing HighErrorRate on api-gateway. What's actually broken?"* **Under the hood** Calls `get_alerts` to fetch the raw alert payload (supports substring search by alert name with no entity IDs needed), `investigate_alert` as a one-step follow-up to get entity health alongside the alert context, and `get_issues` to surface the confirmed diagnosis cause for mapped alerts.
## causely-change-impact Activates on any message asking whether a recent deployment, rollout, or config change caused a regression or reliability shift. **Example trigger prompts** - *"We deployed v2.3.1 of order-service 20 minutes ago. Did it break anything?"* - *"Reliability got worse right after this morning's rollout. What degraded?"* - *"Compare service health before and after today's deploy to checkout."* - *"Did our canary release introduce any downstream failures?"* **Under the hood** Calls `get_events` to locate the deployment event, `reliability_delta` to compare reliability before and after, `fleet_reliability_delta` for a fleet-wide comparison, `get_incident_impact` to retrieve the responsible service and impacted services with business context once a diagnosis is confirmed, `get_config` to inspect the deployed configuration, and `get_metrics` for specific metric trends.
## causely-correlated-incidents Activates on any message describing multiple services alerting simultaneously, or asking about blast radius, cascading failures, dependency chains, or whether separate incidents share a single cause. **Example trigger prompts** - *"Five services are alerting at the same time. Is there a single root cause?"* - *"Checkout, payments, and inventory all degraded. Is this one incident?"* - *"We have a cascade in production. Where did it start?"* - *"Which services will be affected if database-primary goes down?"* **Under the hood** Calls `get_issues` to find all active Issues and their Diagnoses, `get_topology` to map the dependency graph and blast radius, `get_alerts` to correlate firing alerts across services, and `get_incident_impact` to retrieve the responsible service and blast radius with business context for a confirmed Issue.
## causely-health-reporting Activates on any request for a health summary, SLO status update, standup report, reliability overview spanning multiple services or an entire environment, or a single-service health check ("is X healthy?"). **Example trigger prompts** - *"Give me a standup summary of service health for the platform team."* - *"Which services are closest to breaching their SLOs this week?"* - *"Summarize overnight health of all production services."* - *"What's the reliability snapshot for the checkout domain?"* **Under the hood** Calls `get_environment_health` for an environment-wide summary, `get_service_summary` for per-service detail, `get_slo` for SLO status, `team_health` for team-scoped aggregation, `get_symptoms` for a full signal scan across all entities, and `get_diagnoses` to flag any active issues.
## causely-k8s-investigation Activates on any message about Kubernetes infrastructure health: nodes, pods, namespaces, deployments, DaemonSets, or containers. These include OOMKills, pod restarts, node pressure, scheduling failures, resource exhaustion, CrashLoopBackOff, and evictions. **Example trigger prompts** - *"payment-processor pods keep OOMKilling. What's causing it?"* - *"Why does api-gateway restart every few hours?"* - *"Node pressure is high on cluster west-1. Which workloads are to blame?"* - *"My pods are crash-looping but the logs don't show an obvious error."* **Under the hood** Calls `get_service_summary` for a full service-level health check, `get_environment_health` for namespace sweeps, `get_symptoms` for a pod-level signal scan, `get_incident_impact` to retrieve the responsible service and impacted services with business context for a confirmed issue, `get_entity_health` for pod or node status, `get_events` for recent Kubernetes events (OOMKill, CrashLoopBackOff, evictions), `get_config` to inspect resource requests and limits, `get_metrics` for CPU and memory trends, `get_logs` for container log analysis, and `get_diagnoses` scoped to a namespace.
## causely-postmortem Activates on any request to write a postmortem, incident retrospective, or create a ticket documenting a completed or past outage. **Example trigger prompts** - *"Write a postmortem for yesterday's checkout outage."* - *"Draft a Jira ticket for the payment service incident last night."* - *"Create an incident retrospective for the database brownout on Friday."* - *"Generate an RCA document for the cascade failure in production on April 25."* **Under the hood** Calls `get_diagnoses` to reconstruct the incident timeline, `postmortem` to generate the structured retrospective document, and `generate_ticket` to create a Jira or Linear ticket draft.
## Overrides and escape hatches **Force a specific skill** Prefix your prompt with `use causely-` to bypass the router and invoke a specific specialist directly: ``` use causely-postmortem: write a retrospective for the payment incident on April 25 ``` **Bypass skills entirely** You can call MCP tools directly without involving any skill. See the [Full Tool Reference](/agent-integration/mcp-server#full-tool-reference) for the complete list of tools and parameters. **If the wrong skill activates** Tell the agent which skill to use instead: `use causely-`. If a skill consistently miss routes on a prompt you expect to work, email [support@causely.ai](mailto:support@causely.ai) with the prompt text and which skill activated. --- ## VS Code (GitHub Copilot) Connect Causely to GitHub Copilot in VS Code to query root causes and service health from Copilot Chat. ## Prerequisites - Active [Causely account](https://portal.causely.app/) - [VS Code](https://code.visualstudio.com/) with the [GitHub Copilot](https://marketplace.visualstudio.com/items?itemName=GitHub.copilot) extension (version 0.22+ for MCP support) ## Configuration Create or update `.vscode/mcp.json` in your workspace: ```json { "servers": { "causely": { "type": "http", "url": "https://api.causely.app/mcp" } } } ``` :::note VS Code Copilot uses `servers` as the top-level key, not `mcpServers`. Using `mcpServers` silently fails, the server simply won't appear. ::: **Config file location** | Scope | Path | |---|---| | Workspace | `.vscode/mcp.json` at the repository root | | User | Command Palette → **MCP: Open User Configuration** | ## Restart After saving the config, open the Command Palette and run **MCP: List Servers** to confirm `causely` appears. Copilot Chat picks up the server on the next conversation turn, no full restart required. :::tip Skills are not yet supported for VS Code Copilot. You'll interact with the Causely MCP server directly; the example prompts in the [Key Workflows](/agent-integration/mcp-server#key-workflows) section still work, but tool selection is handled by GitHub Copilot's agent rather than a Causely skill router. ::: ## Try It Now - *"List my clusters."* - *"Are there any active symptoms right now?"* - *"What services are currently degraded?"* ## Known Gotcha If your organization uses GitHub Copilot Business or Enterprise, an admin may need to allow MCP servers in the Copilot policy settings before user-configured servers can connect. See [Advanced Authentication](/agent-integration/mcp-server/advanced-auth) for adding machine credentials via `headers` or VS Code `inputs`. --- ## API Authentication All Causely GraphQL API requests require authentication using a Bearer token. You'll need to exchange your API client credentials for an access token using our authentication endpoint. ## How to Get API Access Token To verify that your credentials are working, you can use the following curl command. Replace `` and `` with the credentials you generated above: ```bash CAUSELY_CLIENT_ID= CAUSELY_CLIENT_SECRET= curl -X POST https://auth.causely.app/frontegg/identity/resources/auth/v2/api-token \ -H "Content-Type: application/json" \ -d "{ \"clientId\": \"${CAUSELY_CLIENT_ID}\", \"secret\": \"${CAUSELY_CLIENT_SECRET}\" }" ``` **Example Response:** ```json { "access_token": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6Ik...", "refresh_token": "25b14169-b912-4d15-8f9c-xxxxxxxx", "expires_in": 86400, "expires": "Sat, 31 May 2025 22:54:23 GMT" } ``` You can use the `access_token` for subsequent API calls. If you want to retrieve the access token and store it in an environment variable, you can use the following curl command: ```bash CAUSELY_CLIENT_ID= CAUSELY_CLIENT_SECRET= export CAUSELY_ACCESS_TOKEN=$(response=$(curl -s -w "\n%{http_code}" -X POST https://auth.causely.app/frontegg/identity/resources/auth/v2/api-token \ -H "Content-Type: application/json" \ -d "{\"clientId\": \"${CAUSELY_CLIENT_ID}\", \"secret\": \"${CAUSELY_CLIENT_SECRET}\"}"); \ http_code=$(echo "$response" | tail -n1); \ body=$(echo "$response" | sed '$d'); \ if [ "$http_code" = "200" ]; then echo "$body" | jq -r .access_token; else echo "$body" >&2; false; fi) ``` You can then use the `${CAUSELY_ACCESS_TOKEN}` environment variable for subsequent API calls. Similarly you can obtain the token using your preferred programming language. ```python # Make sure to install the requests library: pip install requests import os, requests def get_causely_access_token(client_id, client_secret, url="https://auth.causely.app/frontegg/identity/resources/auth/v2/api-token"): res = requests.post(url, headers={"Content-Type": "application/json"}, json={"clientId": client_id, "secret": client_secret}) if res.status_code != 200: raise requests.HTTPError(f"Token request failed ({res.status_code}): {res.text}") token = res.json().get("access_token") if not token: raise ValueError("No access_token in response.") return token # Example usage: if __name__ == "__main__": cid, secret = os.getenv("CAUSELY_CLIENT_ID"), os.getenv("CAUSELY_CLIENT_SECRET") if not cid or not secret: raise EnvironmentError("Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET.") try: token = get_causely_access_token(cid, secret) print("Access token:", token) except Exception as e: print(f"Error: {e}") ``` ```javascript const https = require('https'); function getCauselyAccessToken( clientId, clientSecret, url = 'https://auth.causely.app/frontegg/identity/resources/auth/v2/api-token', ) { return new Promise((resolve, reject) => { const data = JSON.stringify({ clientId, secret: clientSecret }); const req = https.request( url, { method: 'POST', headers: { 'Content-Type': 'application/json', 'Content-Length': Buffer.byteLength(data), }, }, (res) => { let body = ''; res.on('data', (chunk) => (body += chunk)); res.on('end', () => { if (res.statusCode !== 200) { return reject(new Error(`HTTP ${res.statusCode}: ${body}`)); } const json = JSON.parse(body); if (!json.access_token) return reject(new Error('No access_token in response.')); resolve(json.access_token); }); }, ); req.on('error', reject); req.write(data); req.end(); }); } // Example usage const { CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET } = process.env; if (!CAUSELY_CLIENT_ID || !CAUSELY_CLIENT_SECRET) { console.error('Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET'); process.exit(1); } getCauselyAccessToken(CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET) .then((token) => console.log('Access token:', token)) .catch((err) => console.error('Error:', err.message)); ``` ```go package main import ( "bytes" "encoding/json" "fmt" "io" "net/http" "os" ) func getToken(id, secret, url string) (string, error) { data, _ := json.Marshal(map[string]string{"clientId": id, "secret": secret}) req, _ := http.NewRequest("POST", url, bytes.NewReader(data)) req.Header.Set("Content-Type", "application/json") res, err := http.DefaultClient.Do(req) if err != nil { return "", err } defer res.Body.Close() body, _ := io.ReadAll(res.Body) if res.StatusCode != 200 { return "", fmt.Errorf("status %d: %s", res.StatusCode, body) } var resp map[string]interface{} if err := json.Unmarshal(body, &resp); err != nil { return "", err } token, ok := resp["access_token"].(string) if !ok || token == "" { return "", fmt.Errorf("no access_token in response") } return token, nil } func main() { id, secret := os.Getenv("CAUSELY_CLIENT_ID"), os.Getenv("CAUSELY_CLIENT_SECRET") if id == "" || secret == "" { fmt.Fprintln(os.Stderr, "Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET") os.Exit(1) } url := "https://auth.causely.app/frontegg/identity/resources/auth/v2/api-token" token, err := getToken(id, secret, url) if err != nil { fmt.Fprintln(os.Stderr, "Error:", err) os.Exit(1) } fmt.Println("Access token:", token) } ``` --- ## Gateway Token Management Gateway tokens authenticate the Causely mediator agent when it connects to the Causely platform. Creating tokens via the REST API enables automated deployment pipelines, for example ArgoCD or Backstage workflows that provision a mediator without manual steps in the UI. :::tip Prerequisite You need a valid Bearer access token before calling this endpoint. See [Authentication](/api/authentication) for how to obtain one. ::: ## Create a Gateway Token **`POST https://api.causely.app/api/tokens`** ```bash CAUSELY_CLIENT_ID= CAUSELY_CLIENT_SECRET= TOKEN_NAME= # Obtain a Bearer access token export CAUSELY_ACCESS_TOKEN=$(response=$(curl -s -w "\n%{http_code}" -X POST https://auth.causely.app/frontegg/identity/resources/auth/v2/api-token \ -H "Content-Type: application/json" \ -d "{\"clientId\": \"${CAUSELY_CLIENT_ID}\", \"secret\": \"${CAUSELY_CLIENT_SECRET}\"}"); \ http_code=$(echo "$response" | tail -n1); \ body=$(echo "$response" | sed '$d'); \ if [ "$http_code" = "200" ]; then echo "$body" | jq -r .access_token; else echo "$body" >&2; false; fi) # Create a gateway token curl -s -X POST https://api.causely.app/api/tokens \ -H "Authorization: Bearer ${CAUSELY_ACCESS_TOKEN}" \ -H "Content-Type: application/json" \ -d "{\"name\": \"${TOKEN_NAME}\"}" ``` ```python # Make sure to install the requests library: pip install requests import os, requests def create_gateway_token(access_token, name, url="https://api.causely.app/api/tokens"): res = requests.post( url, headers={ "Authorization": f"Bearer {access_token}", "Content-Type": "application/json", }, json={"name": name}, ) if res.status_code != 201: raise requests.HTTPError(f"Token creation failed ({res.status_code}): {res.text}") return res.json() # Example usage: if __name__ == "__main__": cid = os.getenv("CAUSELY_CLIENT_ID") secret = os.getenv("CAUSELY_CLIENT_SECRET") token_name = os.getenv("TOKEN_NAME", "my-mediator-token") if not cid or not secret: raise EnvironmentError("Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET.") access_token = get_causely_access_token(cid, secret) gateway_token = create_gateway_token(access_token, token_name) print("Gateway token value:", gateway_token["tokenValue"]) ``` ```javascript const https = require('https'); function createGatewayToken(accessToken, name, url = 'https://api.causely.app/api/tokens') { return new Promise((resolve, reject) => { const data = JSON.stringify({ name }); const parsedUrl = new URL(url); const req = https.request( { hostname: parsedUrl.hostname, path: parsedUrl.pathname, method: 'POST', headers: { Authorization: `Bearer ${accessToken}`, 'Content-Type': 'application/json', 'Content-Length': Buffer.byteLength(data), }, }, (res) => { let body = ''; res.on('data', (chunk) => (body += chunk)); res.on('end', () => { if (res.statusCode !== 201) { return reject(new Error(`HTTP ${res.statusCode}: ${body}`)); } resolve(JSON.parse(body)); }); }, ); req.on('error', reject); req.write(data); req.end(); }); } // Example usage const { CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET } = process.env; const tokenName = process.env.TOKEN_NAME || 'my-mediator-token'; if (!CAUSELY_CLIENT_ID || !CAUSELY_CLIENT_SECRET) { console.error('Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET'); process.exit(1); } (async () => { try { const accessToken = await getCauselyAccessToken(CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET); const gatewayToken = await createGatewayToken(accessToken, tokenName); console.log('Gateway token value:', gatewayToken.tokenValue); } catch (err) { console.error('Error:', err.message); process.exit(1); } })(); ``` ```go package main import ( "bytes" "encoding/json" "fmt" "io" "net/http" "os" ) type GatewayToken struct { ID string `json:"id"` Name string `json:"name"` TokenValue string `json:"tokenValue"` MediatorID string `json:"mediatorId"` ProjectID string `json:"projectId"` CreatedBy string `json:"createdBy"` CreatedByEmail string `json:"createdByEmail"` State string `json:"state"` CreatedAt string `json:"createdAt"` } func createGatewayToken(accessToken, name, url string) (*GatewayToken, error) { data, _ := json.Marshal(map[string]string{"name": name}) req, _ := http.NewRequest("POST", url, bytes.NewReader(data)) req.Header.Set("Authorization", "Bearer "+accessToken) req.Header.Set("Content-Type", "application/json") res, err := http.DefaultClient.Do(req) if err != nil { return nil, err } defer res.Body.Close() body, _ := io.ReadAll(res.Body) if res.StatusCode != 201 { return nil, fmt.Errorf("status %d: %s", res.StatusCode, body) } var token GatewayToken if err := json.Unmarshal(body, &token); err != nil { return nil, err } return &token, nil } func main() { id, secret := os.Getenv("CAUSELY_CLIENT_ID"), os.Getenv("CAUSELY_CLIENT_SECRET") tokenName := os.Getenv("TOKEN_NAME") if id == "" || secret == "" { fmt.Fprintln(os.Stderr, "Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET") os.Exit(1) } if tokenName == "" { tokenName = "my-mediator-token" } accessToken, err := getToken(id, secret, "https://auth.causely.app/frontegg/identity/resources/auth/v2/api-token") if err != nil { fmt.Fprintln(os.Stderr, "Error getting access token:", err) os.Exit(1) } gatewayToken, err := createGatewayToken(accessToken, tokenName, "https://api.causely.app/api/tokens") if err != nil { fmt.Fprintln(os.Stderr, "Error creating gateway token:", err) os.Exit(1) } fmt.Println("Gateway token value:", gatewayToken.TokenValue) } ``` **Example Response (201 Created):** ```json { "id": "c2679064-c908-4bb0-9928-a0e3ce37da80", "name": "my-mediator-token", "tokenValue": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9...", "mediatorId": "beb09338-3c0b-43ab-9b34-6a3fa496abf4", "projectId": "4877d57d-6789-4838-af52-7fe9af353c9e", "createdBy": "8df4696c-fffd-43c9-8df6-c80921471efb", "createdByEmail": "user@example.com", "state": "active", "createdAt": "2026-05-26T19:16:45Z" } ``` | Field | Description | |---|---| | `id` | Unique identifier for this token record | | `name` | The name you provided | | `tokenValue` | The token secret. Use this as `mediator.gateway.token` in Helm, `mediator.secretName` with a Kubernetes secret, or `CAUSELY_GATEWAY_TOKEN` in Docker/Nomad deployments | | `mediatorId` | ID of the mediator this token is associated with | | `projectId` | ID of the project this token belongs to | | `createdBy` | User ID of the token creator | | `createdByEmail` | Email of the token creator | | `state` | Token status (`active`) | | `createdAt` | ISO 8601 creation timestamp | :::note Using `tokenValue` in deployments The `tokenValue` from the response is the gateway token referenced throughout the [installation docs](/installation). For example: ```bash # Helm --set mediator.gateway.token="${TOKEN_VALUE}" # Docker CAUSELY_GATEWAY_TOKEN="${TOKEN_VALUE}" ``` For Kubernetes deployments, you can store the token in a secret instead of specifying it in Helm values. This is recommended for production and GitOps workflows. See [Using a Kubernetes Secret for the Access Token](/installation/customize#using-a-kubernetes-secret-for-the-access-token) for setup instructions. ::: ## List Gateway Tokens **`GET https://api.causely.app/api/tokens`** Returns all gateway tokens for the project, including both active and revoked tokens. ```bash curl -s -X GET https://api.causely.app/api/tokens \ -H "Authorization: Bearer ${CAUSELY_ACCESS_TOKEN}" ``` ```python import requests def list_gateway_tokens(access_token, url="https://api.causely.app/api/tokens"): res = requests.get( url, headers={"Authorization": f"Bearer {access_token}"}, ) if res.status_code != 200: raise requests.HTTPError(f"List tokens failed ({res.status_code}): {res.text}") return res.json() # Example usage: if __name__ == "__main__": import os cid = os.getenv("CAUSELY_CLIENT_ID") secret = os.getenv("CAUSELY_CLIENT_SECRET") if not cid or not secret: raise EnvironmentError("Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET.") access_token = get_causely_access_token(cid, secret) tokens = list_gateway_tokens(access_token) for token in tokens: print(f"{token['name']} ({token['state']}): {token['id']}") ``` ```javascript const https = require('https'); function listGatewayTokens(accessToken, url = 'https://api.causely.app/api/tokens') { return new Promise((resolve, reject) => { const parsedUrl = new URL(url); const req = https.request( { hostname: parsedUrl.hostname, path: parsedUrl.pathname, method: 'GET', headers: { Authorization: `Bearer ${accessToken}`, }, }, (res) => { let body = ''; res.on('data', (chunk) => (body += chunk)); res.on('end', () => { if (res.statusCode !== 200) { return reject(new Error(`HTTP ${res.statusCode}: ${body}`)); } resolve(JSON.parse(body)); }); }, ); req.on('error', reject); req.end(); }); } // Example usage const { CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET } = process.env; if (!CAUSELY_CLIENT_ID || !CAUSELY_CLIENT_SECRET) { console.error('Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET'); process.exit(1); } (async () => { try { const accessToken = await getCauselyAccessToken(CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET); const tokens = await listGatewayTokens(accessToken); tokens.forEach((t) => console.log(`${t.name} (${t.state}): ${t.id}`)); } catch (err) { console.error('Error:', err.message); process.exit(1); } })(); ``` ```go func listGatewayTokens(accessToken, url string) ([]GatewayToken, error) { req, _ := http.NewRequest("GET", url, nil) req.Header.Set("Authorization", "Bearer "+accessToken) res, err := http.DefaultClient.Do(req) if err != nil { return nil, err } defer res.Body.Close() body, _ := io.ReadAll(res.Body) if res.StatusCode != 200 { return nil, fmt.Errorf("status %d: %s", res.StatusCode, body) } var tokens []GatewayToken if err := json.Unmarshal(body, &tokens); err != nil { return nil, err } return tokens, nil } ``` **Example Response (200 OK):** ```json [ { "id": "afce85cb-70b4-4624-8198-d38843a7ccc0", "name": "my-mediator-token", "tokenValue": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9...", "mediatorId": "beb09338-3c0b-43ba-9b34-6a3fa496abf4", "projectId": "4877d57d-6789-4838-af52-7fe9af353c9e", "createdBy": "8df4696c-fffd-43c9-8df6-c80921471efb", "createdByEmail": "user@example.com", "state": "active", "createdAt": "2026-06-03T22:11:39Z" }, { "id": "5152a4b0-327b-46f5-8926-5716bbdd2bed", "name": "old-mediator-token", "mediatorId": "beb09338-3c0b-43ba-9b34-6a3fa496abf4", "projectId": "4877d57d-6789-4838-af52-7fe9af353c9e", "createdBy": "f67341c8-12cf-49bc-b9c7-09f44885750c", "createdByEmail": "user@example.com", "state": "revoked", "revokedAt": "2026-05-26T21:36:29Z", "revokedBy": "8df4696c-fffd-43c9-8df6-c80921471efb", "createdAt": "2026-05-26T21:24:11Z" } ] ``` The response is an array of token objects. Each object contains the same fields as the [create response](#create-a-gateway-token). Revoked tokens include additional fields: | Field | Description | |---|---| | `id` | Unique identifier for this token record | | `name` | The name assigned to the token | | `tokenValue` | The token secret. Only present on active tokens | | `mediatorId` | ID of the mediator this token is associated with | | `projectId` | ID of the project this token belongs to | | `createdBy` | User ID of the token creator | | `createdByEmail` | Email of the token creator | | `state` | Token status: `active` or `revoked` | | `createdAt` | ISO 8601 creation timestamp | | `revokedAt` | ISO 8601 timestamp of when the token was revoked (revoked tokens only) | | `revokedBy` | User ID of who performed the revocation (revoked tokens only) | ## Revoke a Gateway Token **`POST https://api.causely.app/api/tokens/{tokenId}/revoke?tokenId={tokenId}`** Revoking a token immediately deactivates it. The mediator using that token will lose connectivity to the platform. ```bash TOKEN_ID= curl -s -X POST "https://api.causely.app/api/tokens/${TOKEN_ID}/revoke?tokenId=${TOKEN_ID}" \ -H "Authorization: Bearer ${CAUSELY_ACCESS_TOKEN}" ``` ```python import requests def revoke_gateway_token(access_token, token_id, base_url="https://api.causely.app/api/tokens"): url = f"{base_url}/{token_id}/revoke" res = requests.post( url, params={"tokenId": token_id}, headers={"Authorization": f"Bearer {access_token}"}, ) if res.status_code != 200: raise requests.HTTPError(f"Token revocation failed ({res.status_code}): {res.text}") return res.json() # Example usage: if __name__ == "__main__": import os cid = os.getenv("CAUSELY_CLIENT_ID") secret = os.getenv("CAUSELY_CLIENT_SECRET") token_id = os.getenv("TOKEN_ID") if not cid or not secret or not token_id: raise EnvironmentError("Missing CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET, or TOKEN_ID.") access_token = get_causely_access_token(cid, secret) result = revoke_gateway_token(access_token, token_id) print("Token state:", result["state"]) ``` ```javascript const https = require('https'); function revokeGatewayToken(accessToken, tokenId, baseUrl = 'https://api.causely.app/api/tokens') { return new Promise((resolve, reject) => { const url = new URL(`${baseUrl}/${tokenId}/revoke`); url.searchParams.set('tokenId', tokenId); const req = https.request( { hostname: url.hostname, path: url.pathname + url.search, method: 'POST', headers: { Authorization: `Bearer ${accessToken}`, }, }, (res) => { let body = ''; res.on('data', (chunk) => (body += chunk)); res.on('end', () => { if (res.statusCode !== 200) { return reject(new Error(`HTTP ${res.statusCode}: ${body}`)); } resolve(JSON.parse(body)); }); }, ); req.on('error', reject); req.end(); }); } // Example usage const { CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET, TOKEN_ID } = process.env; if (!CAUSELY_CLIENT_ID || !CAUSELY_CLIENT_SECRET || !TOKEN_ID) { console.error('Missing CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET, or TOKEN_ID'); process.exit(1); } (async () => { try { const accessToken = await getCauselyAccessToken(CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET); const result = await revokeGatewayToken(accessToken, TOKEN_ID); console.log('Token state:', result.state); } catch (err) { console.error('Error:', err.message); process.exit(1); } })(); ``` ```go func revokeGatewayToken(accessToken, tokenID, baseURL string) (*GatewayToken, error) { url := fmt.Sprintf("%s/%s/revoke?tokenId=%s", baseURL, tokenID, tokenID) req, _ := http.NewRequest("POST", url, nil) req.Header.Set("Authorization", "Bearer "+accessToken) res, err := http.DefaultClient.Do(req) if err != nil { return nil, err } defer res.Body.Close() body, _ := io.ReadAll(res.Body) if res.StatusCode != 200 { return nil, fmt.Errorf("status %d: %s", res.StatusCode, body) } var token GatewayToken if err := json.Unmarshal(body, &token); err != nil { return nil, err } return &token, nil } ``` **Example Response (200 OK):** ```json { "id": "8323a17c-f6f5-4a7a-b04e-481790314b03", "name": "my-mediator-token", "mediatorId": "beb09338-3c0b-43ba-9b34-6a3fa496abf4", "projectId": "4877d57d-6789-4838-af52-7fe9af353c9e", "createdBy": "8df4696c-fffd-43c9-8df6-c80921471efb", "createdByEmail": "user@example.com", "state": "revoked", "revokedAt": "2026-05-26T19:42:26Z", "revokedBy": "8df4696c-fffd-43c9-8df6-c80921471efb", "createdAt": "2026-05-26T19:42:09Z" } ``` | Field | Description | |---|---| | `state` | `revoked` once the token has been invalidated | | `revokedAt` | ISO 8601 timestamp of when the token was revoked | | `revokedBy` | User ID of who performed the revocation | --- ## Setting Up GraphQL Clients To maximize efficiency and code reusability when integrating with the Causely GraphQL API, we recommend setting up proper GraphQL client libraries rather than using raw HTTP requests. This approach provides better error handling, type safety, query validation, and cleaner code that can be reused across multiple API operations. ## Install GraphQL Client Libraries Install the recommended GraphQL client libraries for your programming language to enable robust API integration with automatic query validation and enhanced developer experience: ```bash pip install gql[all] requests ``` ```bash npm install @apollo/client graphql ``` ```bash # Install gq (GraphQL CLI tool) curl -sSL https://github.com/houseabsolute/gq/releases/latest/download/gq-Linux-x86_64.tar.gz | tar xz # Or use your package manager, e.g., brew install gq ``` ```bash go get github.com/machinebox/graphql ``` ## Create Reusable GraphQL Client Functions Implement these language-specific GraphQL client functions that integrate seamlessly with the authentication methods from the previous section. These functions provide a clean, reusable interface for executing any GraphQL query against the Causely API: ```python from gql import gql, Client from gql.transport.requests import RequestsHTTPTransport def create_graphql_client(access_token): """Create a GraphQL client with authentication.""" transport = RequestsHTTPTransport( url="https://api.causely.app/query/", headers={"Authorization": f"Bearer {access_token}"} ) return Client(transport=transport, fetch_schema_from_transport=True) def post_query(client, query_string, variables=None): """Execute a GraphQL query using the client.""" query = gql(query_string) return client.execute(query, variable_values=variables) # Usage example: # client = create_graphql_client(get_causely_access_token(client_id, secret)) # result = post_query(client, query_string, variables) ``` ```javascript const { ApolloClient, InMemoryCache, createHttpLink, gql } = require('@apollo/client'); function createGraphQLClient(accessToken) { const httpLink = createHttpLink({ uri: 'https://api.causely.app/query/', headers: { Authorization: `Bearer ${accessToken}`, }, }); return new ApolloClient({ link: httpLink, cache: new InMemoryCache(), }); } async function postQuery(client, queryString, variables = {}) { const result = await client.query({ query: gql(queryString), variables, fetchPolicy: 'no-cache', }); return result.data; } // Usage example: // const client = createGraphQLClient(await getCauselyAccessToken(clientId, secret)); // const result = await postQuery(client, queryString, variables); ``` ```bash # Function to execute GraphQL queries using gq post_query() { local query_file="$1" local variables_file="$2" gq https://api.causely.app/query/ \ --header "Authorization: Bearer ${CAUSELY_ACCESS_TOKEN}" \ --query-file "$query_file" \ --variables-file "$variables_file" } # Alternative using curl with GraphQL post_query_curl() { local query="$1" local variables="$2" curl -X POST https://api.causely.app/query/ \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${CAUSELY_ACCESS_TOKEN}" \ -d "{\"query\": \"$query\", \"variables\": $variables}" } ``` ```go package main import ( "context" "github.com/machinebox/graphql" ) func createGraphQLClient(accessToken string) *graphql.Client { client := graphql.NewClient("https://api.causely.app/query/") client.WithHTTPHeader("Authorization", "Bearer "+accessToken) return client } func postQuery(client *graphql.Client, queryString string, variables map[string]interface{}) (interface{}, error) { req := graphql.NewRequest(queryString) // Add variables to the request for key, value := range variables { req.Var(key, value) } var response interface{} err := client.Run(context.Background(), req, &response) return response, err } // Usage example: // client := createGraphQLClient(token) // result, err := postQuery(client, queryString, variables) ``` --- ## Query Defects and Root Causes This query example demonstrates how to retrieve the first 20 high-severity, active defects and Diagnoses from the Causely platform using the GraphQL utilities defined above. :::note The START_TIME is set to the current time in UTC. You may need to adjust the timestamp to your needs. ::: :::tip Prerequisite This example reuses the helper utilities defined in the [Authentication](/api/authentication) and [GraphQL Clients](/api/graphql-clients) sections: fetching an access token, creating the GraphQL client, and sending a request with the `post_query` wrapper (CLI, Python, or Go). Keep those helpers in scope before running this query. ::: ```python import os from datetime import datetime, timezone if __name__ == "__main__": # Get credentials from environment cid, secret = os.getenv("CAUSELY_CLIENT_ID"), os.getenv("CAUSELY_CLIENT_SECRET") if not cid or not secret: raise EnvironmentError("Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET") # Get access token and create GraphQL client token = get_causely_access_token(cid, secret) client = create_graphql_client(token) # Prepare query variables start_time = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%S") + ".000+00:00" variables = { "defectFilter": { "state": "ACTIVE", "severities": ["Critical", "High"], "startTime": start_time, "scopesFilter": {"scopes": []}, "entityTypes": [], "defectNames": [], "entityName": "" }, "groupRecurring": True, "first": 20 } # Execute the query query = """ fragment LabelFragment on Label { key value __typename } fragment BasicEntityWithLabelsFragment on Entity { id name typeName labels { ...LabelFragment __typename } __typename } fragment DefectFragment on Defect { id name fromTime toTime remediated serviceCount entityId entityType entity { ...BasicEntityWithLabelsFragment __typename } activeCount missingCount severity __typename } fragment DefectRelatedOccurrencesFragment on Defect { relatedOccurrrences { id name fromTime toTime entity { id name typeName __typename } severity __typename } __typename } fragment BasicEntityFragment on Entity { id name typeName __typename } fragment SymptomFragmentWithoutLabels on Symptom { id name active state entityId entityType fromTime toTime isPropagated entity { ...BasicEntityFragment __typename } __typename } fragment EventFragmentWithoutLabels on Event { id name active time entity { ...BasicEntityFragment __typename } __typename } query defectConnection($defectFilter: DefectFilter, $groupRecurring: Boolean, $first: Int, $after: String, $last: Int, $before: String) { defectConnection(defectFilter: $defectFilter groupRecurring: $groupRecurring first: $first after: $after last: $last before: $before) { totalCount edges { node { ...DefectFragment ...DefectRelatedOccurrencesFragment symptoms { ...SymptomFragmentWithoutLabels __typename } events { ...EventFragmentWithoutLabels __typename } __typename } cursor __typename } pageInfo { hasNextPage hasPreviousPage startCursor endCursor totalCount __typename } __typename } } """ result = post_query(client, query, variables) print(result) ``` ```javascript const { CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET } = process.env; if (!CAUSELY_CLIENT_ID || !CAUSELY_CLIENT_SECRET) { console.error('Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET'); process.exit(1); } (async () => { try { // Get access token and create GraphQL client const token = await getCauselyAccessToken(CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET); const client = createGraphQLClient(token); // Prepare query variables const startTime = new Date().toISOString().replace(/\.\d+Z$/, '.000+00:00'); const variables = { defectFilter: { state: 'ACTIVE', severities: ['Critical', 'High'], startTime, scopesFilter: { scopes: [] }, entityTypes: [], defectNames: [], entityName: '', }, groupRecurring: true, first: 20, }; // Execute the query const query = ` fragment LabelFragment on Label { key value __typename } fragment BasicEntityWithLabelsFragment on Entity { id name typeName labels { ...LabelFragment __typename } __typename } fragment DefectFragment on Defect { id name fromTime toTime remediated serviceCount entityId entityType entity { ...BasicEntityWithLabelsFragment __typename } activeCount missingCount severity __typename } fragment DefectRelatedOccurrencesFragment on Defect { relatedOccurrrences { id name fromTime toTime entity { id name typeName __typename } severity __typename } __typename } fragment BasicEntityFragment on Entity { id name typeName __typename } fragment SymptomFragmentWithoutLabels on Symptom { id name active state entityId entityType fromTime toTime isPropagated entity { ...BasicEntityFragment __typename } __typename } fragment EventFragmentWithoutLabels on Event { id name active time entity { ...BasicEntityFragment __typename } __typename } query defectConnection($defectFilter: DefectFilter, $groupRecurring: Boolean, $first: Int, $after: String, $last: Int, $before: String) { defectConnection(defectFilter: $defectFilter groupRecurring: $groupRecurring first: $first after: $after last: $last before: $before) { totalCount edges { node { ...DefectFragment ...DefectRelatedOccurrencesFragment symptoms { ...SymptomFragmentWithoutLabels __typename } events { ...EventFragmentWithoutLabels __typename } __typename } cursor __typename } pageInfo { hasNextPage hasPreviousPage startCursor endCursor totalCount __typename } __typename } } `; const result = await postQuery(client, query, variables); console.log(JSON.stringify(result, null, 2)); } catch (err) { console.error('Error:', err.message); process.exit(1); } })(); ``` ```bash # Set up credentials and get token export CAUSELY_CLIENT_ID= export CAUSELY_CLIENT_SECRET= export CAUSELY_ACCESS_TOKEN=$(response=$(curl -s -w "\n%{http_code}" -X POST https://auth.causely.app/frontegg/identity/resources/auth/v2/api-token \ -H "Content-Type: application/json" \ -d "{\"clientId\": \"${CAUSELY_CLIENT_ID}\", \"secret\": \"${CAUSELY_CLIENT_SECRET}\"}"); \ http_code=$(echo "$response" | tail -n1); \ body=$(echo "$response" | sed '$d'); \ if [ "$http_code" = "200" ]; then echo "$body" | jq -r .access_token; else echo "$body" >&2; false; fi) # Create query and variables files cat > defect_query.graphql << 'EOF' fragment LabelFragment on Label { key value __typename } fragment BasicEntityWithLabelsFragment on Entity { id name typeName labels { ...LabelFragment __typename } __typename } fragment DefectFragment on Defect { id name fromTime toTime remediated serviceCount entityId entityType entity { ...BasicEntityWithLabelsFragment __typename } activeCount missingCount severity __typename } fragment DefectRelatedOccurrencesFragment on Defect { relatedOccurrrences { id name fromTime toTime entity { id name typeName __typename } severity __typename } __typename } fragment BasicEntityFragment on Entity { id name typeName __typename } fragment SymptomFragmentWithoutLabels on Symptom { id name active state entityId entityType fromTime toTime isPropagated entity { ...BasicEntityFragment __typename } __typename } fragment EventFragmentWithoutLabels on Event { id name active time entity { ...BasicEntityFragment __typename } __typename } query defectConnection($defectFilter: DefectFilter, $groupRecurring: Boolean, $first: Int, $after: String, $last: Int, $before: String) { defectConnection(defectFilter: $defectFilter groupRecurring: $groupRecurring first: $first after: $after last: $last before: $before) { totalCount edges { node { ...DefectFragment ...DefectRelatedOccurrencesFragment symptoms { ...SymptomFragmentWithoutLabels __typename } events { ...EventFragmentWithoutLabels __typename } __typename } cursor __typename } pageInfo { hasNextPage hasPreviousPage startCursor endCursor totalCount __typename } __typename } } EOF START_TIME="$(TZ=UTC date +%Y-%m-%dT%H:%M:%S).000+00:00" cat > variables.json << EOF { "defectFilter": { "state": "ACTIVE", "severities": ["Critical", "High"], "startTime": "${START_TIME}", "scopesFilter": {"scopes": []}, "entityTypes": [], "defectNames": [], "entityName": "" }, "groupRecurring": true, "first": 20 } EOF # Execute the query using gq (if available) or curl if command -v gq &> /dev/null; then post_query "defect_query.graphql" "variables.json" else query=$(cat defect_query.graphql | tr '\n' ' ') variables=$(cat variables.json) post_query_curl "$query" "$variables" fi # Clean up temporary files rm -f defect_query.graphql variables.json ``` ```go package main import ( "fmt" "os" "time" ) func main() { // Get credentials from environment id, secret := os.Getenv("CAUSELY_CLIENT_ID"), os.Getenv("CAUSELY_CLIENT_SECRET") if id == "" || secret == "" { fmt.Fprintln(os.Stderr, "Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET") os.Exit(1) } // Get access token and create GraphQL client token, err := getToken(id, secret, "https://auth.causely.app/frontegg/identity/resources/auth/v2/api-token") if err != nil { fmt.Fprintln(os.Stderr, "Error getting token:", err) os.Exit(1) } client := createGraphQLClient(token) // Prepare query variables startTime := time.Now().UTC().Format("2006-01-02T15:04:05") + ".000+00:00" variables := map[string]interface{}{ "defectFilter": map[string]interface{}{ "state": "ACTIVE", "severities": []string{"Critical", "High"}, "startTime": startTime, "scopesFilter": map[string]interface{}{"scopes": []string{}}, "entityTypes": []string{}, "defectNames": []string{}, "entityName": "", }, "groupRecurring": true, "first": 20, } // Execute the query query := `fragment LabelFragment on Label { key value __typename } fragment BasicEntityWithLabelsFragment on Entity { id name typeName labels { ...LabelFragment __typename } __typename } fragment DefectFragment on Defect { id name fromTime toTime remediated serviceCount entityId entityType entity { ...BasicEntityWithLabelsFragment __typename } activeCount missingCount severity __typename } fragment DefectRelatedOccurrencesFragment on Defect { relatedOccurrrences { id name fromTime toTime entity { id name typeName __typename } severity __typename } __typename } fragment BasicEntityFragment on Entity { id name typeName __typename } fragment SymptomFragmentWithoutLabels on Symptom { id name active state entityId entityType fromTime toTime isPropagated entity { ...BasicEntityFragment __typename } __typename } fragment EventFragmentWithoutLabels on Event { id name active time entity { ...BasicEntityFragment __typename } __typename } query defectConnection($defectFilter: DefectFilter, $groupRecurring: Boolean, $first: Int, $after: String, $last: Int, $before: String) { defectConnection(defectFilter: $defectFilter groupRecurring: $groupRecurring first: $first after: $after last: $last before: $before) { totalCount edges { node { ...DefectFragment ...DefectRelatedOccurrencesFragment symptoms { ...SymptomFragmentWithoutLabels __typename } events { ...EventFragmentWithoutLabels __typename } __typename } cursor __typename } pageInfo { hasNextPage hasPreviousPage startCursor endCursor totalCount __typename } __typename } }` result, err := postQuery(client, query, variables) if err != nil { fmt.Fprintln(os.Stderr, "Error running query:", err) os.Exit(1) } fmt.Println(result) } ``` First, let's define the GraphQL query and variables: ```graphql fragment LabelFragment on Label { key value __typename } fragment BasicEntityWithLabelsFragment on Entity { id name typeName labels { ...LabelFragment __typename } __typename } fragment DefectFragment on Defect { id name fromTime toTime remediated serviceCount entityId entityType entity { ...BasicEntityWithLabelsFragment __typename } activeCount missingCount severity __typename } fragment DefectRelatedOccurrencesFragment on Defect { relatedOccurrrences { id name fromTime toTime entity { id name typeName __typename } severity __typename } __typename } fragment BasicEntityFragment on Entity { id name typeName __typename } fragment SymptomFragmentWithoutLabels on Symptom { id name active state entityId entityType fromTime toTime isPropagated entity { ...BasicEntityFragment __typename } __typename } fragment EventFragmentWithoutLabels on Event { id name active time entity { ...BasicEntityFragment __typename } __typename } query defectConnection( $defectFilter: DefectFilter $groupRecurring: Boolean $first: Int $after: String $last: Int $before: String ) { defectConnection( defectFilter: $defectFilter groupRecurring: $groupRecurring first: $first after: $after last: $last before: $before ) { totalCount edges { node { ...DefectFragment ...DefectRelatedOccurrencesFragment symptoms { ...SymptomFragmentWithoutLabels __typename } events { ...EventFragmentWithoutLabels __typename } __typename } cursor __typename } pageInfo { hasNextPage hasPreviousPage startCursor endCursor totalCount __typename } __typename } } ``` --- ## Get User Scopes This query retrieves user scopes and access permissions for the authenticated user. User scopes define which resources and data the current user can access within the Causely platform. :::tip Prerequisite This example reuses the helper utilities defined in the [Authentication](/api/authentication) and [GraphQL Clients](/api/graphql-clients) sections—fetching an access token, creating the GraphQL client, and sending a request with the `post_query` wrapper (CLI, Python, or Go). Keep those helpers in scope before running this query. ::: ```python import os if __name__ == "__main__": # Get credentials from environment cid, secret = os.getenv("CAUSELY_CLIENT_ID"), os.getenv("CAUSELY_CLIENT_SECRET") if not cid or not secret: raise EnvironmentError("Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET") # Get access token and create GraphQL client token = get_causely_access_token(cid, secret) client = create_graphql_client(token) # Prepare query variables variables = { "filter": { "nameExpr": "" } } # Execute the query query = """ query getUserScopes($filter: UserScopeFilter, $first: Int, $after: String, $last: Int, $before: String) { getUserScopes( filter: $filter first: $first after: $after last: $last before: $before ) { totalCount edges { node { id name audience ownerId lastUpdate scopes { typeName typeValues __typename } __typename } cursor __typename } pageInfo { hasNextPage hasPreviousPage startCursor endCursor totalCount __typename } __typename } } """ result = post_query(client, query, variables) print(result) ``` ```javascript const { CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET } = process.env; if (!CAUSELY_CLIENT_ID || !CAUSELY_CLIENT_SECRET) { console.error('Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET'); process.exit(1); } (async () => { try { // Get access token and create GraphQL client const token = await getCauselyAccessToken(CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET); const client = createGraphQLClient(token); // Prepare query variables const variables = { filter: { nameExpr: '', }, }; // Execute the query const query = ` query getUserScopes($filter: UserScopeFilter, $first: Int, $after: String, $last: Int, $before: String) { getUserScopes( filter: $filter first: $first after: $after last: $last before: $before ) { totalCount edges { node { id name audience ownerId lastUpdate scopes { typeName typeValues __typename } __typename } cursor __typename } pageInfo { hasNextPage hasPreviousPage startCursor endCursor totalCount __typename } __typename } } `; const result = await postQuery(client, query, variables); console.log(JSON.stringify(result, null, 2)); } catch (err) { console.error('Error:', err.message); process.exit(1); } })(); ``` ```bash # Set up credentials and get token export CAUSELY_CLIENT_ID= export CAUSELY_CLIENT_SECRET= export CAUSELY_ACCESS_TOKEN=$(response=$(curl -s -w "\n%{http_code}" -X POST https://auth.causely.app/frontegg/identity/resources/auth/v2/api-token \ -H "Content-Type: application/json" \ -d "{\"clientId\": \"${CAUSELY_CLIENT_ID}\", \"secret\": \"${CAUSELY_CLIENT_SECRET}\"}"); \ http_code=$(echo "$response" | tail -n1); \ body=$(echo "$response" | sed '$d'); \ if [ "$http_code" = "200" ]; then echo "$body" | jq -r .access_token; else echo "$body" >&2; false; fi) # Create query and variables files cat > user_scopes_query.graphql << 'EOF' query getUserScopes($filter: UserScopeFilter, $first: Int, $after: String, $last: Int, $before: String) { getUserScopes( filter: $filter first: $first after: $after last: $last before: $before ) { totalCount edges { node { id name audience ownerId lastUpdate scopes { typeName typeValues __typename } __typename } cursor __typename } pageInfo { hasNextPage hasPreviousPage startCursor endCursor totalCount __typename } __typename } } EOF cat > user_scopes_variables.json << EOF { "filter": { "nameExpr": "" } } EOF # Execute the query using gq (if available) or curl if command -v gq &> /dev/null; then post_query "user_scopes_query.graphql" "user_scopes_variables.json" else query=$(cat user_scopes_query.graphql | tr '\n' ' ') variables=$(cat user_scopes_variables.json) post_query_curl "$query" "$variables" fi # Clean up temporary files rm -f user_scopes_query.graphql user_scopes_variables.json ``` ```go package main import ( "fmt" "os" ) func main() { // Get credentials from environment id, secret := os.Getenv("CAUSELY_CLIENT_ID"), os.Getenv("CAUSELY_CLIENT_SECRET") if id == "" || secret == "" { fmt.Fprintln(os.Stderr, "Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET") os.Exit(1) } // Get access token and create GraphQL client token, err := getToken(id, secret, "https://auth.causely.app/frontegg/identity/resources/auth/v2/api-token") if err != nil { fmt.Fprintln(os.Stderr, "Error getting token:", err) os.Exit(1) } client := createGraphQLClient(token) // Prepare query variables variables := map[string]interface{}{ "filter": map[string]interface{}{ "nameExpr": "", }, } // Execute the query query := `query getUserScopes($filter: UserScopeFilter, $first: Int, $after: String, $last: Int, $before: String) { getUserScopes( filter: $filter first: $first after: $after last: $last before: $before ) { totalCount edges { node { id name audience ownerId lastUpdate scopes { typeName typeValues __typename } __typename } cursor __typename } pageInfo { hasNextPage hasPreviousPage startCursor endCursor totalCount __typename } __typename } }` result, err := postQuery(client, query, variables) if err != nil { fmt.Fprintln(os.Stderr, "Error running query:", err) os.Exit(1) } fmt.Println(result) } ``` ```graphql query getUserScopes($filter: UserScopeFilter, $first: Int, $after: String, $last: Int, $before: String) { getUserScopes(filter: $filter, first: $first, after: $after, last: $last, before: $before) { totalCount edges { node { id name audience ownerId lastUpdate scopes { typeName typeValues __typename } __typename } cursor __typename } pageInfo { hasNextPage hasPreviousPage startCursor endCursor totalCount __typename } __typename } } ``` **Variables:** ```json { "filter": { "nameExpr": "" } } ``` --- ## Hiding Root Causes You may want to hide a root cause for a time period if you are not planning on addressing it immediately or if you are performing maintenance and the behavior is expected. The recommended way to hide a specific root cause is to use **Entity Configs** of type `IgnoreRC`. To avoid overwriting existing configuration for that entity, you should: 1. Query the current `IgnoreRC` config for the entity. 2. Merge your new rule with any existing entries. 3. Call the `CreateEntityConfigs` mutation with the combined config. :::tip Prerequisite Each example below reuses the helper utilities defined in the [Authentication](/api/authentication) and [GraphQL Clients](/api/graphql-clients) sections (`get_causely_access_token`, `create_graphql_client`, `post_query`, and their language equivalents). ::: ```python import os import json # Reuse helpers defined earlier in this guide: # - get_causely_access_token # - create_graphql_client # - post_query GET_CONFIGS_QUERY = """ query GetEntityConfigs($entityId: String!, $configType: String!) { entityConfigs(entityId: $entityId, configType: $configType) { entityId type data } } """ CREATE_CONFIGS_MUTATION = """ mutation CreateEntityConfigs($entityConfigs: [EntityConfigInput!]!) { createEntityConfigs(entityConfigs: $entityConfigs) { entityId type data } } """ def load_existing_ignore_rc_configs(client, entity_id, config_type="IgnoreRC"): """Fetch existing IgnoreRC configs and return them as a Python list.""" variables = {"entityId": entity_id, "configType": config_type} result = post_query(client, GET_CONFIGS_QUERY, variables) entries = [] for cfg in result.get("entityConfigs", []): data_str = cfg.get("data") or "[]" try: entries.extend(json.loads(data_str)) except json.JSONDecodeError: # If data is malformed, skip rather than failing the whole operation continue return entries def save_ignore_rc_configs(client, entity_id, configs, config_type="IgnoreRC"): """Persist the merged IgnoreRC configs back to Causely.""" variables = { "entityConfigs": [{ "entityId": entity_id, "type": config_type, # If your schema uses ENUMs, pass the enum instead "data": json.dumps(configs), # Must be a JSON string }] } return post_query(client, CREATE_CONFIGS_MUTATION, variables) if __name__ == "__main__": # Get credentials from environment cid, secret = os.getenv("CAUSELY_CLIENT_ID"), os.getenv("CAUSELY_CLIENT_SECRET") if not cid or not secret: raise EnvironmentError("Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET") # Get access token and create GraphQL client token = get_causely_access_token(cid, secret) client = create_graphql_client(token) # Target entity and root cause name you want to hide # Replace these example values with your actual entity ID and root cause name entity_id = "47c29d23-9f57-5d47-9f91-a72815edc8cf" defect_name_to_hide = "Malfunction" # 1) Load existing IgnoreRC configuration (if any) existing_configs = load_existing_ignore_rc_configs(client, entity_id) # 2) Add or update the entry for this root cause without losing existing ones new_entry = { "DefectName": defect_name_to_hide, # Set an appropriate expiry time for the ignore rule "UntilTime": "2025-11-15T10:51:00-04:00", } # Avoid adding duplicate entries for the same defect name merged_configs = [ cfg for cfg in existing_configs if cfg.get("DefectName") != defect_name_to_hide ] merged_configs.append(new_entry) # 3) Save the merged configuration back using the mutation result = save_ignore_rc_configs(client, entity_id, merged_configs) print(json.dumps(result, indent=2)) ``` ```javascript // Reuses helpers defined earlier in this guide: // - getCauselyAccessToken // - createGraphQLClient // - postQuery (for queries) const { gql } = require('@apollo/client'); const { CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET } = process.env; const GET_CONFIGS_QUERY = ` query GetEntityConfigs($entityId: String!, $configType: String!) { entityConfigs(entityId: $entityId, configType: $configType) { entityId type data } } `; const CREATE_CONFIGS_MUTATION = ` mutation CreateEntityConfigs($entityConfigs: [EntityConfigInput!]!) { createEntityConfigs(entityConfigs: $entityConfigs) { entityId type data } } `; async function loadExistingIgnoreRcConfigs(client, entityId, configType = 'IgnoreRC') { const variables = { entityId, configType }; const data = await postQuery(client, GET_CONFIGS_QUERY, variables); const entries = []; for (const cfg of data.entityConfigs || []) { const dataStr = cfg.data || '[]'; try { entries.push(...JSON.parse(dataStr)); } catch { // Skip malformed data instead of failing the whole operation continue; } } return entries; } async function saveIgnoreRcConfigs(client, entityId, configs, configType = 'IgnoreRC') { const variables = { entityConfigs: [ { entityId, type: configType, // If your schema uses ENUMs, pass the enum instead data: JSON.stringify(configs), // Must be a JSON string }, ], }; const result = await client.mutate({ mutation: gql(CREATE_CONFIGS_MUTATION), variables, }); return result.data; } (async () => { if (!CAUSELY_CLIENT_ID || !CAUSELY_CLIENT_SECRET) { console.error('Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET'); process.exit(1); } // Get access token and create GraphQL client const token = await getCauselyAccessToken(CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET); const client = createGraphQLClient(token); // Replace these example values with your actual entity ID and root cause name const entityId = '47c29d23-9f57-5d47-9f91-a72815edc8cf'; const defectNameToHide = 'Malfunction'; // 1) Load existing IgnoreRC configuration (if any) const existingConfigs = await loadExistingIgnoreRcConfigs(client, entityId); // 2) Add or update the entry for this root cause without losing existing ones const newEntry = { DefectName: defectNameToHide, // Set an appropriate expiry time for the ignore rule UntilTime: '2025-11-15T10:51:00-04:00', }; const mergedConfigs = existingConfigs.filter((cfg) => cfg.DefectName !== defectNameToHide); mergedConfigs.push(newEntry); // 3) Save the merged configuration back using the mutation const result = await saveIgnoreRcConfigs(client, entityId, mergedConfigs); console.log(JSON.stringify(result, null, 2)); })(); ``` ```bash #!/usr/bin/env bash set -euo pipefail # Prerequisites: # - CAUSELY_CLIENT_ID / CAUSELY_CLIENT_SECRET exported # - CAUSELY_ACCESS_TOKEN exported (see Authentication section) # - post_query / post_query_curl functions defined (see Bash GraphQL helpers section) # - jq installed # Replace these example values with your actual entity ID, root cause name and time ENTITY_ID="47c29d23-9f57-5d47-9f91-a72815edc8cf" DEFECT_NAME_TO_HIDE="Malfunction" UNTIL_TIME="2025-11-15T10:51:00-04:00" # 1) Define GraphQL query and mutation cat > get_entity_configs.graphql << 'EOF' query GetEntityConfigs($entityId: String!, $configType: String!) { entityConfigs(entityId: $entityId, configType: $configType) { entityId type data } } EOF cat > create_entity_configs.graphql << 'EOF' mutation CreateEntityConfigs($entityConfigs: [EntityConfigInput!]!) { createEntityConfigs(entityConfigs: $entityConfigs) { entityId type data } } EOF # 2) Query existing IgnoreRC config cat > get_entity_configs_variables.json << EOF { "entityId": "${ENTITY_ID}", "configType": "IgnoreRC" } EOF if command -v gq &> /dev/null; then RAW_RESPONSE=$(gq https://api.causely.app/query/ \ --header "Authorization: Bearer ${CAUSELY_ACCESS_TOKEN}" \ --query-file get_entity_configs.graphql \ --variables-file get_entity_configs_variables.json) else QUERY=$(tr '\n' ' ' < get_entity_configs.graphql) VARIABLES=$(cat get_entity_configs_variables.json) RAW_RESPONSE=$(post_query_curl "$QUERY" "$VARIABLES") fi # 3) Extract current config entries (data is a JSON string) CURRENT_CONFIGS=$(echo "${RAW_RESPONSE}" \ | jq -r '.data.entityConfigs[0].data // "[]"' \ | jq -c 'try fromjson catch []') # 4) Merge in the new ignore rule (avoid duplicates on DefectName) MERGED_CONFIGS=$( jq -c --arg name "${DEFECT_NAME_TO_HIDE}" --arg until "${UNTIL_TIME}" ' (map(select(.DefectName != $name))) + [ { "DefectName": $name, "UntilTime": $until } ] ' <<< "${CURRENT_CONFIGS}" ) # 5) Build variables for CreateEntityConfigs (data must be a JSON string) DATA_STRING=$(jq -Rs . <<< "${MERGED_CONFIGS}") cat > create_entity_configs_variables.json << EOF { "entityConfigs": [ { "entityId": "${ENTITY_ID}", "type": "IgnoreRC", "data": ${DATA_STRING} } ] } EOF # 6) Call the mutation if command -v gq &> /dev/null; then gq https://api.causely.app/query/ \ --header "Authorization: Bearer ${CAUSELY_ACCESS_TOKEN}" \ --query-file create_entity_configs.graphql \ --variables-file create_entity_configs_variables.json else MUTATION=$(tr '\n' ' ' < create_entity_configs.graphql) VARIABLES=$(cat create_entity_configs_variables.json) post_query_curl "$MUTATION" "$VARIABLES" fi # 7) Cleanup rm -f get_entity_configs.graphql create_entity_configs.graphql \ get_entity_configs_variables.json create_entity_configs_variables.json ``` ```go package main import ( "encoding/json" "fmt" "os" ) // Reuse helpers defined earlier: // - getToken(id, secret, url) // - createGraphQLClient(accessToken string) *graphql.Client // - postQuery(client *graphql.Client, queryString string, variables map[string]interface{}) (interface{}, error) const getConfigsQuery = ` query GetEntityConfigs($entityId: String!, $configType: String!) { entityConfigs(entityId: $entityId, configType: $configType) { entityId type data } } ` const createConfigsMutation = ` mutation CreateEntityConfigs($entityConfigs: [EntityConfigInput!]!) { createEntityConfigs(entityConfigs: $entityConfigs) { entityId type data } } ` type entityConfigsResponse struct { EntityConfigs []struct { EntityID string `json:"entityId"` Type string `json:"type"` Data string `json:"data"` } `json:"entityConfigs"` } func loadExistingIgnoreRCConfigs(gqlClient *graphql.Client, entityID string, configType string) ([]map[string]interface{}, error) { variables := map[string]interface{}{ "entityId": entityID, "configType": configType, } raw, err := postQuery(gqlClient, getConfigsQuery, variables) if err != nil { return nil, err } // postQuery returns interface{}; marshal + unmarshal into a typed struct bytes, err := json.Marshal(raw) if err != nil { return nil, err } var wrapped struct { Data entityConfigsResponse `json:"data"` } if err := json.Unmarshal(bytes, &wrapped); err != nil { return nil, err } entries := []map[string]interface{}{} for _, cfg := range wrapped.Data.EntityConfigs { dataStr := cfg.Data if dataStr == "" { continue } var arr []map[string]interface{} if err := json.Unmarshal([]byte(dataStr), &arr); err != nil { // Skip malformed data instead of failing the whole operation continue } entries = append(entries, arr...) } return entries, nil } func saveIgnoreRCConfigs(gqlClient *graphql.Client, entityID string, configs []map[string]interface{}, configType string) (interface{}, error) { dataBytes, err := json.Marshal(configs) if err != nil { return nil, err } variables := map[string]interface{}{ "entityConfigs": []map[string]interface{}{ { "entityId": entityID, "type": configType, // If your schema uses ENUMs, pass the enum instead "data": string(dataBytes), // Must be a JSON string }, }, } return postQuery(gqlClient, createConfigsMutation, variables) } func main() { id, secret := os.Getenv("CAUSELY_CLIENT_ID"), os.Getenv("CAUSELY_CLIENT_SECRET") if id == "" || secret == "" { fmt.Fprintln(os.Stderr, "Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET") os.Exit(1) } token, err := getToken(id, secret, "https://auth.causely.app/frontegg/identity/resources/auth/v2/api-token") if err != nil { fmt.Fprintln(os.Stderr, "Error getting token:", err) os.Exit(1) } client := createGraphQLClient(token) // Replace these example values with your actual entity ID, root cause name and time entityID := "47c29d23-9f57-5d47-9f91-a72815edc8cf" defectNameToHide := "Malfunction" untilTime := "2025-11-15T10:51:00-04:00" // 1) Load existing IgnoreRC configuration (if any) existing, err := loadExistingIgnoreRCConfigs(client, entityID, "IgnoreRC") if err != nil { fmt.Fprintln(os.Stderr, "Error loading configs:", err) os.Exit(1) } // 2) Add or update the entry for this root cause without losing existing ones merged := make([]map[string]interface{}, 0, len(existing)+1) for _, cfg := range existing { if name, ok := cfg["DefectName"].(string); ok && name == defectNameToHide { continue } merged = append(merged, cfg) } merged = append(merged, map[string]interface{}{ "DefectName": defectNameToHide, "UntilTime": untilTime, }) // 3) Save the merged configuration back using the mutation result, err := saveIgnoreRCConfigs(client, entityID, merged, "IgnoreRC") if err != nil { fmt.Fprintln(os.Stderr, "Error saving configs:", err) os.Exit(1) } out, _ := json.MarshalIndent(result, "", " ") fmt.Println(string(out)) } ``` ```graphql # 1) Query existing IgnoreRC config for an entity query GetEntityConfigs($entityId: String!, $configType: String!) { entityConfigs(entityId: $entityId, configType: $configType) { entityId type data } } ``` **Variables:** ```json # Replace this example values with your actual entity ID. { "entityId": "47c29d23-9f57-5d47-9f91-a72815edc8cf", "configType": "IgnoreRC" } ``` The `data` field in the response is a **JSON string** that represents an array of ignore rules, for example: ```json [{ "DefectName": "SomeOtherDefect", "UntilTime": "2025-11-10T12:00:00-04:00" }] ``` You should: 1. Parse this JSON string in your client. 2. Remove any existing entries with the same `DefectName` you want to hide. 3. Append a new entry for the root cause you want to hide. 4. Stringify the merged array and send it back via `CreateEntityConfigs`. ```graphql # 2) Save merged IgnoreRC configuration mutation CreateEntityConfigs($entityConfigs: [EntityConfigInput!]!) { createEntityConfigs(entityConfigs: $entityConfigs) { entityId type data } } ``` **Variables (example after merging):** ```json # Replace this example values with your actual entity ID, root cause name and time { "entityConfigs": [ { "entityId": "47c29d23-9f57-5d47-9f91-a72815edc8cf", "type": "IgnoreRC", "data": "[{\"DefectName\":\"SomeOtherDefect\",\"UntilTime\":\"2025-11-10T12:00:00-04:00\"},{\"DefectName\":\"Malfunction\",\"UntilTime\":\"2025-11-15T10:51:00-04:00\"}]" } ] } ``` > Note: The `data` value must be a **JSON string**, not a nested JSON object. Most clients will use `JSON.stringify` or an equivalent function to produce this string. --- ## Testing Notification Payloads This example demonstrates how to **send a test notification** to Causely using a representative payload. It follows the same pattern as the examples above: reuse your token + GraphQL client helpers and then execute one mutation with a clear `variables` object. :::tip Prerequisite This example reuses the helper utilities defined in the [Authentication](/api/authentication) and [GraphQL Clients](/api/graphql-clients) sections—fetching an access token, creating the GraphQL client, and sending a request with the `post_query` wrapper (CLI, Python, or Go). Keep those helpers in scope before running this mutation. ::: ```python import os import json if __name__ == "__main__": # Get credentials from environment cid, secret = os.getenv("CAUSELY_CLIENT_ID"), os.getenv("CAUSELY_CLIENT_SECRET") if not cid or not secret: raise EnvironmentError("Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET") # Get access token and create GraphQL client token = get_causely_access_token(cid, secret) client = create_graphql_client(token) # Prepare mutation mutation = """ mutation createNotification($notification: NotificationInput!) { createNotification(notification: $notification) } """ # Build the payload embedded_payload = { "link": "https://portal.causely.app", "name": "Causely: Test Notification 42", "type": "ProblemDetected", # Options are ProblemDetected and ProblemCleared "entity": { "id": "22307647-43e6-5f08-a5e4-a3f50088ccec", "link": "https://portal.causely.app", "name": "payments-api", "type": "Node" }, "labels": { "k8s.cluster.name": "us-prod", # Test to route based on cluster "k8s.namespace.name": "payments", # Test to route based on name space "gcp.resource.zone": "https://www.gcp.com/compute/v1/projects/example-project/zones/us-central1-a", "causely.ai/cluster": "my-important-services-cluster" }, "objectId": "2ac21477-61f3-4ae0-9fc4-0c1ea43ff727", "severity": "High", # Options are Critical, High, Medium, Low "timestamp": "2025-09-14T19:15:16.410543344Z", "description": { "summary": "Sending test notification for entity in us-prod myApp1", "details": "Webhook Test" } } variables = { "notification": { "sourceId": "2f423693-5334-4586-a366-64d458535001", "type": 1, "payload": json.dumps(embedded_payload) } } result = post_query(client, mutation, variables) print(result) ``` ```javascript // Reuses helpers defined above: getCauselyAccessToken, createGraphQLClient // Requires @apollo/client installed (as shown above) const { CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET } = process.env; const { gql } = require('@apollo/client'); if (!CAUSELY_CLIENT_ID || !CAUSELY_CLIENT_SECRET) { console.error('Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET'); process.exit(1); } (async () => { try { const token = await getCauselyAccessToken(CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET); const client = createGraphQLClient(token); // Mutation matches working script exactly const mutation = ` mutation createNotification($notification: NotificationInput!) { createNotification(notification: $notification) } `; // Embedded payload (stringified below) const embeddedPayload = { link: 'https://portal.causely.app', name: 'Causely: Test Notification 42', type: 'ProblemDetected', // ProblemDetected | ProblemCleared entity: { id: '22307647-43e6-5f08-a5e4-a3f50088ccec', link: 'https://portal.causely.app', name: 'payments-api', type: 'Node', }, labels: { 'k8s.cluster.name': 'us-prod', 'k8s.namespace.name': 'payments', 'gcp.resource.zone': 'https://www.gcp.com/compute/v1/projects/example-project/zones/us-central1-a', 'causely.ai/cluster': 'my-important-services-cluster', }, objectId: '2ac21477-61f3-4ae0-9fc4-0c1ea43ff727', severity: 'High', // Critical | High | Medium | Low timestamp: '2025-09-14T19:15:16.410543344Z', description: { summary: 'Sending test notification for entity in us-prod myApp1', details: 'Webhook Test', }, }; const variables = { notification: { sourceId: '2f423693-5334-4586-a366-64d458535001', type: 1, payload: JSON.stringify(embeddedPayload), }, }; // Use mutate for a GraphQL mutation const result = await client.mutate({ mutation: gql(mutation), variables }); console.log(JSON.stringify(result.data, null, 2)); } catch (err) { console.error('Error:', err.message); process.exit(1); } })(); ``` ```bash # Assumes CAUSELY_ACCESS_TOKEN is already exported (see Authentication section) # Creates files for the mutation, an embedded payload JSON, and variables (with payload as a JSON string), then posts them. # 1) GraphQL mutation cat > create_notification.graphql << 'EOF' mutation createNotification($notification: NotificationInput!) { createNotification(notification: $notification) } EOF # 2) Embedded payload (domain JSON) cat > embedded_payload.json << 'EOF' { "link": "https://portal.causely.app", "name": "Causely: Test Notification 42", "type": "ProblemDetected", "entity": { "id": "22307647-43e6-5f08-a5e4-a3f50088ccec", "link": "https://portal.causely.app", "name": "payments-api", "type": "Node" }, "labels": { "k8s.cluster.name": "us-prod", "k8s.namespace.name": "payments", "gcp.resource.zone": "https://www.gcp.com/compute/v1/projects/example-project/zones/us-central1-a", "causely.ai/cluster": "my-important-services-cluster" }, "objectId": "2ac21477-61f3-4ae0-9fc4-0c1ea43ff727", "severity": "High", "timestamp": "2025-09-14T19:15:16.410543344Z", "description": { "summary": "Sending test notification for entity in us-prod myApp1", "details": "Webhook Test" } } EOF # 3) Build variables.json with payload as a *string* using jq to JSON-encode the file content PAYLOAD_STR=$(jq -Rs . < embedded_payload.json) cat > variables.json << EOF { "notification": { "sourceId": "2f423693-5334-4586-a366-64d458535001", "type": 1, "payload": ${PAYLOAD_STR} } } EOF # 4) Post using gq if available; otherwise curl if command -v gq &> /dev/null; then gq https://api.causely.app/query/ \ --header "Authorization: Bearer ${CAUSELY_ACCESS_TOKEN}" \ --query-file create_notification.graphql \ --variables-file variables.json else curl -sS -X POST https://api.causely.app/query/ \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${CAUSELY_ACCESS_TOKEN}" \ -d "{\"query\":\"$(tr '\n' ' ' < create_notification.graphql)\",\"variables\":$(cat variables.json)}" fi # 5) Clean up rm -f create_notification.graphql embedded_payload.json variables.json ``` ```go package main import ( "encoding/json" "fmt" "os" ) func main() { // Reuse helpers defined above: // getToken(id, secret, url) and createGraphQLClient(accessToken) id, secret := os.Getenv("CAUSELY_CLIENT_ID"), os.Getenv("CAUSELY_CLIENT_SECRET") if id == "" || secret == "" { fmt.Fprintln(os.Stderr, "Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET") os.Exit(1) } token, err := getToken(id, secret, "https://auth.causely.app/frontegg/identity/resources/auth/v2/api-token") if err != nil { fmt.Fprintln(os.Stderr, "Error getting token:", err) os.Exit(1) } client := createGraphQLClient(token) // Mutation matches working script exactly mutation := ` mutation createNotification($notification: NotificationInput!) { createNotification(notification: $notification) } ` // Embedded payload (will be stringified) embeddedPayload := map[string]interface{}{ "link": "https://portal.causely.app", "name": "Causely: Test Notification 42", "type": "ProblemDetected", "entity": map[string]interface{}{ "id": "22307647-43e6-5f08-a5e4-a3f50088ccec", "link": "https://portal.causely.app", "name": "payments-api", "type": "Node", }, "labels": map[string]interface{}{ "k8s.cluster.name": "us-prod", "k8s.namespace.name": "payments", "gcp.resource.zone": "https://www.gcp.com/compute/v1/projects/example-project/zones/us-central1-a", "causely.ai/cluster": "my-important-services-cluster", }, "objectId": "2ac21477-61f3-4ae0-9fc4-0c1ea43ff727", "severity": "High", "timestamp": "2025-09-14T19:15:16.410543344Z", "description": map[string]interface{}{ "summary": "Sending test notification for entity in us-prod myApp1", "details": "Webhook Test", }, } payloadBytes, err := json.Marshal(embeddedPayload) if err != nil { fmt.Fprintln(os.Stderr, "Error marshaling payload:", err) os.Exit(1) } variables := map[string]interface{}{ "notification": map[string]interface{}{ "sourceId": "2f423693-5334-4586-a366-64d458535001", "type": 1, "payload": string(payloadBytes), // GraphQL expects a STRING here }, } result, err := postQuery(client, mutation, variables) if err != nil { fmt.Fprintln(os.Stderr, "Error running mutation:", err) os.Exit(1) } fmt.Println(result) } ``` When sending a notification via GraphQL, the `payload` field must be passed as a **stringified JSON** (not as a nested JSON object). The embedded JSON represents the notification details and must be stringified before being sent in the GraphQL variables. Below is a clear breakdown: **Mutation:** ```graphql mutation createNotification($notification: NotificationInput!) { createNotification(notification: $notification) } ``` **Embedded Payload (JSON)** This is the JSON you want to send as the notification payload (before stringifying): ```json { "link": "https://portal.causely.app", "name": "Causely: Test Notification 42", "type": "ProblemDetected", "entity": { "id": "22307647-43e6-5f08-a5e4-a3f50088ccec", "link": "https://portal.causely.app", "name": "payments-api", "type": "Node" }, "labels": { "k8s.cluster.name": "us-prod", "k8s.namespace.name": "payments", "gcp.resource.zone": "https://www.gcp.com/compute/v1/projects/example-project/zones/us-central1-a", "causely.ai/cluster": "my-important-services-cluster" }, "objectId": "2ac21477-61f3-4ae0-9fc4-0c1ea43ff727", "severity": "High", "timestamp": "2025-09-14T19:15:16.410543344Z", "description": { "summary": "Sending test notification for entity in us-prod myApp1", "details": "Webhook Test" } } ``` **Variables (GraphQL input)** Notice that the `payload` value is the above JSON, stringified (escaped), and can be split across multiple lines for readability: ```json { "notification": { "sourceId": "2f423693-5334-4586-a366-64d458535001", "type": 1, "payload": "{\n \"link\": \"https://portal.causely.app\",\n \"name\": \"Causely: Test Notification 42\",\n \"type\": \"ProblemDetected\",\n \"entity\": {\n \"id\": \"22307647-43e6-5f08-a5e4-a3f50088ccec\",\n \"link\": \"https://portal.causely.app\",\n \"name\": \"payments-api\",\n \"type\": \"Node\"\n },\n \"labels\": {\n \"k8s.cluster.name\": \"us-prod\",\n \"k8s.namespace.name\": \"payments\",\n \"gcp.resource.zone\": \"https://www.gcp.com/compute/v1/projects/example-project/zones/us-central1-a\",\n \"causely.ai/cluster\": \"my-important-services-cluster\"\n },\n \"objectId\": \"2ac21477-61f3-4ae0-9fc4-0c1ea43ff727\",\n \"severity\": \"High\",\n \"timestamp\": \"2025-09-14T19:15:16.410543344Z\",\n \"description\": {\n \"summary\": \"Sending test notification for entity in us-prod myApp1\",\n \"details\": \"Webhook Test\"\n }\n}" } } ``` > **Note:** > The `payload` field must be a string. Most clients will use a function like `JSON.stringify()` to convert the embedded payload JSON into a string before passing it to the GraphQL mutation. > > This lets you clearly see how the original JSON maps into the string value required by the GraphQL API. --- ## Setting SLOs on HTTP Paths and RPC Methods You can set SLOs on **HTTP Paths** or **RPC Methods** that represent key endpoints for your application (for example, revenue-critical URIs or core RPC methods). Enabling SLOs on these entities configures SLO tracking for **both Request Duration and Request Error Rate**. When a Diagnosis impacts one of these endpoints and drives its SLO **at risk** or into **violation**, Causely treats that Diagnosis as **Urgent**. The recommended way to manage these SLOs is to use **Entity Configs** of type `EntityTier`. :::tip Prerequisite Each example below reuses the helper utilities defined in the [Authentication](/api/authentication) and [GraphQL Clients](/api/graphql-clients) sections (`get_causely_access_token`, `create_graphql_client`, `post_query`, and their language equivalents). ::: ## Enable SLOs on an HTTP Path or RPC Method ```python import os import json # Reuse helpers defined earlier in this guide: # - get_causely_access_token # - create_graphql_client # - post_query CREATE_CONFIGS_MUTATION = """ mutation CreateEntityConfigs($entityConfigs: [EntityConfigInput!]!) { createEntityConfigs(entityConfigs: $entityConfigs) { entityId type data } } """ if __name__ == "__main__": cid, secret = os.getenv("CAUSELY_CLIENT_ID"), os.getenv("CAUSELY_CLIENT_SECRET") if not cid or not secret: raise EnvironmentError("Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET") token = get_causely_access_token(cid, secret) client = create_graphql_client(token) # Replace with the entityId of the HTTP Path or RPC Method you want to enable SLOs on entity_id = "7c4bba4c-1942-5fc4-87e2-d1f2c483d2c2" variables = { "entityConfigs": [{ "type": "EntityTier", "entityId": entity_id, "data": "sloEnabled", }] } result = post_query(client, CREATE_CONFIGS_MUTATION, variables) print(json.dumps(result, indent=2)) ``` ```javascript // Reuses helpers defined earlier in this guide: // - getCauselyAccessToken // - createGraphQLClient // - postQuery (for queries/mutations) const { gql } = require('@apollo/client'); const { CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET } = process.env; const CREATE_CONFIGS_MUTATION = ` mutation CreateEntityConfigs($entityConfigs: [EntityConfigInput!]!) { createEntityConfigs(entityConfigs: $entityConfigs) { entityId type data } } `; (async () => { if (!CAUSELY_CLIENT_ID || !CAUSELY_CLIENT_SECRET) { console.error('Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET'); process.exit(1); } const token = await getCauselyAccessToken(CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET); const client = createGraphQLClient(token); // Replace with the entityId of the HTTP Path or RPC Method you want to enable SLOs on const entityId = '7c4bba4c-1942-5fc4-87e2-d1f2c483d2c2'; const variables = { entityConfigs: [ { type: 'EntityTier', entityId, data: 'sloEnabled', }, ], }; const result = await client.mutate({ mutation: gql(CREATE_CONFIGS_MUTATION), variables, }); console.log(JSON.stringify(result.data, null, 2)); })(); ``` ```bash #!/usr/bin/env bash set -euo pipefail # Prerequisites: # - CAUSELY_CLIENT_ID / CAUSELY_CLIENT_SECRET exported # - CAUSELY_ACCESS_TOKEN exported (see Authentication section) # - post_query / post_query_curl functions defined (see Bash GraphQL helpers section) # - jq installed # Replace with the entityId of the HTTP Path or RPC Method you want to enable SLOs on ENTITY_ID="7c4bba4c-1942-5fc4-87e2-d1f2c483d2c2" cat > create_entity_configs.graphql << 'EOF' mutation CreateEntityConfigs($entityConfigs: [EntityConfigInput!]!) { createEntityConfigs(entityConfigs: $entityConfigs) { entityId type data } } EOF cat > create_entity_configs_variables.json << EOF { "entityConfigs": [ { "type": "EntityTier", "entityId": "${ENTITY_ID}", "data": "sloEnabled" } ] } EOF QUERY=$(tr '\n' ' ' < create_entity_configs.graphql) VARIABLES=$(cat create_entity_configs_variables.json) post_query_curl "$QUERY" "$VARIABLES" rm -f create_entity_configs.graphql create_entity_configs_variables.json ``` ```go package main import ( "encoding/json" "fmt" "os" ) // Reuse helpers defined earlier: // - getToken(id, secret, url) // - createGraphQLClient(accessToken string) *graphql.Client // - postQuery(client *graphql.Client, queryString string, variables map[string]interface{}) (interface{}, error) const createConfigsMutation = ` mutation CreateEntityConfigs($entityConfigs: [EntityConfigInput!]!) { createEntityConfigs(entityConfigs: $entityConfigs) { entityId type data } } ` func main() { id, secret := os.Getenv("CAUSELY_CLIENT_ID"), os.Getenv("CAUSELY_CLIENT_SECRET") if id == "" || secret == "" { fmt.Fprintln(os.Stderr, "Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET") os.Exit(1) } token, err := getToken(id, secret, "https://auth.causely.app/frontegg/identity/resources/auth/v2/api-token") if err != nil { fmt.Fprintln(os.Stderr, "Error getting token:", err) os.Exit(1) } client := createGraphQLClient(token) // Replace with the entityId of the HTTP Path or RPC Method you want to enable SLOs on entityID := "7c4bba4c-1942-5fc4-87e2-d1f2c483d2c2" variables := map[string]interface{}{ "entityConfigs": []map[string]interface{}{ { "type": "EntityTier", "entityId": entityID, "data": "sloEnabled", }, }, } result, err := postQuery(client, createConfigsMutation, variables) if err != nil { fmt.Fprintln(os.Stderr, "Error enabling SLO:", err) os.Exit(1) } out, _ := json.MarshalIndent(result, "", " ") fmt.Println(string(out)) } ``` ```graphql mutation CreateEntityConfigs($entityConfigs: [EntityConfigInput!]!) { createEntityConfigs(entityConfigs: $entityConfigs) { entityId type data } } ``` **Variables (example):** ```json { "entityConfigs": [ { "type": "EntityTier", "entityId": "7c4bba4c-1942-5fc4-87e2-d1f2c483d2c2", "data": "sloEnabled" } ] } ``` ## Check whether an HTTP Path or RPC Method has SLOs enabled ```python import os import json # Reuse helpers defined earlier: # - get_causely_access_token # - create_graphql_client # - post_query GET_CONFIGS_QUERY = """ query GetEntityConfigs($entityId: String!) { entityConfigs(entityId: $entityId) { type data entityId } } """ if __name__ == "__main__": cid, secret = os.getenv("CAUSELY_CLIENT_ID"), os.getenv("CAUSELY_CLIENT_SECRET") if not cid or not secret: raise EnvironmentError("Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET") token = get_causely_access_token(cid, secret) client = create_graphql_client(token) entity_id = "2f3ce84a-18b8-5d4c-820d-5b747cbb7f13" result = post_query(client, GET_CONFIGS_QUERY, {"entityId": entity_id}) print(json.dumps(result, indent=2)) ``` ```javascript const { gql } = require('@apollo/client'); const { CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET } = process.env; const GET_CONFIGS_QUERY = ` query GetEntityConfigs($entityId: String!) { entityConfigs(entityId: $entityId) { type data entityId } } `; (async () => { const token = await getCauselyAccessToken(CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET); const client = createGraphQLClient(token); const entityId = '2f3ce84a-18b8-5d4c-820d-5b747cbb7f13'; const result = await client.query({ query: gql(GET_CONFIGS_QUERY), variables: { entityId }, fetchPolicy: 'no-cache', }); console.log(JSON.stringify(result.data, null, 2)); })(); ``` ```bash #!/usr/bin/env bash set -euo pipefail ENTITY_ID="2f3ce84a-18b8-5d4c-820d-5b747cbb7f13" cat > get_entity_configs.graphql << 'EOF' query GetEntityConfigs($entityId: String!) { entityConfigs(entityId: $entityId) { type data entityId } } EOF cat > get_entity_configs_variables.json << EOF { "entityId": "${ENTITY_ID}" } EOF QUERY=$(tr '\n' ' ' < get_entity_configs.graphql) VARIABLES=$(cat get_entity_configs_variables.json) post_query_curl "$QUERY" "$VARIABLES" rm -f get_entity_configs.graphql get_entity_configs_variables.json ``` ```go package main import ( "encoding/json" "fmt" "os" ) // Reuse helpers defined earlier: // - getToken(id, secret, url) // - createGraphQLClient(accessToken string) *graphql.Client // - postQuery(client *graphql.Client, queryString string, variables map[string]interface{}) (interface{}, error) const getConfigsQuery = ` query GetEntityConfigs($entityId: String!) { entityConfigs(entityId: $entityId) { type data entityId } } ` func main() { id, secret := os.Getenv("CAUSELY_CLIENT_ID"), os.Getenv("CAUSELY_CLIENT_SECRET") if id == "" || secret == "" { fmt.Fprintln(os.Stderr, "Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET") os.Exit(1) } token, err := getToken(id, secret, "https://auth.causely.app/frontegg/identity/resources/auth/v2/api-token") if err != nil { fmt.Fprintln(os.Stderr, "Error getting token:", err) os.Exit(1) } client := createGraphQLClient(token) // Replace with the entityId of the HTTP Path or RPC Method you want to inspect entityID := "2f3ce84a-18b8-5d4c-820d-5b747cbb7f13" variables := map[string]interface{}{ "entityId": entityID, } result, err := postQuery(client, getConfigsQuery, variables) if err != nil { fmt.Fprintln(os.Stderr, "Error querying configs:", err) os.Exit(1) } out, _ := json.MarshalIndent(result, "", " ") fmt.Println(string(out)) // Look for: type == "EntityTier" and data == "sloEnabled" } ``` ```query GetEntityConfigs($entityId: String!) { entityConfigs(entityId: $entityId) { type data entityId } } ``` **Variables (example):** ```json { "entityId": "2f3ce84a-18b8-5d4c-820d-5b747cbb7f13" } ``` ## Remove SLOs from an HTTP Path or RPC Method ```python import os import json # Reuse helpers defined earlier: # - get_causely_access_token # - create_graphql_client # - post_query DELETE_CONFIG_MUTATION = """ mutation DeleteEntityConfig($entityId: String!, $configType: String!) { deleteEntityConfig(entityId: $entityId, configType: $configType) } """ if __name__ == "__main__": cid, secret = os.getenv("CAUSELY_CLIENT_ID"), os.getenv("CAUSELY_CLIENT_SECRET") if not cid or not secret: raise EnvironmentError("Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET") token = get_causely_access_token(cid, secret) client = create_graphql_client(token) entity_id = "2f3ce84a-18b8-5d4c-820d-5b747cbb7f13" result = post_query(client, DELETE_CONFIG_MUTATION, { "entityId": entity_id, "configType": "EntityTier" }) print(json.dumps(result, indent=2)) ``` ```javascript const { gql } = require('@apollo/client'); const { CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET } = process.env; const DELETE_CONFIG_MUTATION = ` mutation DeleteEntityConfig($entityId: String!, $configType: String!) { deleteEntityConfig(entityId: $entityId, configType: $configType) } `; (async () => { const token = await getCauselyAccessToken(CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET); const client = createGraphQLClient(token); const entityId = '2f3ce84a-18b8-5d4c-820d-5b747cbb7f13'; const result = await client.mutate({ mutation: gql(DELETE_CONFIG_MUTATION), variables: { entityId, configType: 'EntityTier' }, }); console.log(JSON.stringify(result.data, null, 2)); })(); ``` ```bash #!/usr/bin/env bash set -euo pipefail ENTITY_ID="2f3ce84a-18b8-5d4c-820d-5b747cbb7f13" cat > delete_entity_config.graphql << 'EOF' mutation DeleteEntityConfig($entityId: String!, $configType: String!) { deleteEntityConfig(entityId: $entityId, configType: $configType) } EOF cat > delete_entity_config_variables.json << EOF { "entityId": "${ENTITY_ID}", "configType": "EntityTier" } EOF QUERY=$(tr '\n' ' ' < delete_entity_config.graphql) VARIABLES=$(cat delete_entity_config_variables.json) post_query_curl "$QUERY" "$VARIABLES" rm -f delete_entity_config.graphql delete_entity_config_variables.json ``` ```go package main import ( "encoding/json" "fmt" "os" ) // Reuse helpers defined earlier: // - getToken(id, secret, url) // - createGraphQLClient(accessToken string) *graphql.Client // - postQuery(client *graphql.Client, queryString string, variables map[string]interface{}) (interface{}, error) const deleteConfigMutation = ` mutation DeleteEntityConfig($entityId: String!, $configType: String!) { deleteEntityConfig(entityId: $entityId, configType: $configType) } ` func main() { id, secret := os.Getenv("CAUSELY_CLIENT_ID"), os.Getenv("CAUSELY_CLIENT_SECRET") if id == "" || secret == "" { fmt.Fprintln(os.Stderr, "Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET") os.Exit(1) } token, err := getToken(id, secret, "https://auth.causely.app/frontegg/identity/resources/auth/v2/api-token") if err != nil { fmt.Fprintln(os.Stderr, "Error getting token:", err) os.Exit(1) } client := createGraphQLClient(token) // Replace with the entityId of the HTTP Path or RPC Method you want to remove SLOs from entityID := "2f3ce84a-18b8-5d4c-820d-5b747cbb7f13" variables := map[string]interface{}{ "entityId": entityID, "configType": "EntityTier", } result, err := postQuery(client, deleteConfigMutation, variables) if err != nil { fmt.Fprintln(os.Stderr, "Error deleting config:", err) os.Exit(1) } out, _ := json.MarshalIndent(result, "", " ") fmt.Println(string(out)) } ``` ```graphql mutation DeleteEntityConfig($entityId: String!, $configType: String!) { deleteEntityConfig(entityId: $entityId, configType: $configType) } ``` **Variables (example):** ```json { "entityId": "2f3ce84a-18b8-5d4c-820d-5b747cbb7f13", "configType": "EntityTier" } ``` --- ## Alert Ingestion and Mapping ## Overview Causely can ingest alerts from your existing monitoring sources and connect them to the causal reasoning engine, without replacing your current alerting setup. When an alert fires, Causely identifies which service or entity it belongs to, maps it to a known Signal, and includes it as evidence in causal analysis. The following alert sources are supported: - [Alertmanager](/telemetry-sources/alertmanager): Prometheus Alertmanager webhook receiver - [Prometheus](/telemetry-sources/prometheus): Direct Prometheus integration - [Datadog](/telemetry-sources/datadog): Datadog monitors and alerts - [Incident.io](/telemetry-sources/incident-io): Incident.io alert events - [Dynatrace](/telemetry-sources/dynatrace): Dynatrace problems and alerts ## How alerts are mapped automatically When Causely receives an alert, it attempts to resolve two things: **which entity** the alert belongs to, and **which Signal** it represents. ### Step 1: Signal resolution via keyword matching Causely examines the alert's `alertname`, title, and description, and matches them against a library of keyword rules. Matching is case-insensitive. The first matching rule wins, so more specific rules (for example, queue-related keywords) are evaluated before broader ones (for example, generic "error"). Here are a few examples of how this works. An alert named `HighErrorRate` or with a description containing "error rate exceeded" is automatically recognized as a **high error rate** condition. An alert containing "latency" or "timeout" is recognized as a **high latency** condition. Similarly, alerts mentioning "kafka lag", "consumer lag", "message wait time", "dead letter", "JVM heap", or "GC time" are each matched to their corresponding condition. Some examples of what Causely looks for in the alert name, title, or description: | Condition | Example keywords | |---|---| | High error rate | "error", "failed", "failure", "exception", "unavailable", "down" | | High latency | "latency", "duration", "timeout", "slow", "response time" | | Kafka consumer lag | "lag" and ("kafka" / "consumer" / "partition") | | Queue wait time | "message wait", "wait time" and "queue" | | Dead-letter queue | "dead letter", "deadletter", "dlx" | | JVM heap pressure | "heap" and ("java" / "jvm") | | Garbage collection | "gc", "garbage collection", "g1 young", "g1 old" | | DB connection pool | "db connection", "database connection", "postgres connection" | | Redis connection pool | "redis connection", "redis connection pool" | ### Step 2: Entity resolution via alert labels A matched Signal still needs an entity to attach to. Causely uses labels in the alert payload to look up the right entity in your topology. Each entity type requires a specific combination of labels that uniquely identifies it: see [Required label combinations by entity type](#required-label-combinations-by-entity-type) below. If the labels are missing or do not correspond to a known entity in topology, the alert remains unmapped even when the Signal is correctly identified. ## What happens after an alert is mapped ### Alert maps to a known entity and Signal The alert is fully ingested. Causely attaches it as an observed Signal to the entity and includes it in causal reasoning. You will see it reflected in the Diagnosis alongside other Signals. ### Alert maps to a known entity but no Signal match If Causely identifies which entity the alert belongs to but does not recognize the alert as a known Signal, the alert is recorded but does not contribute to causal analysis. This happens when the alert name, title, and description do not contain any matching keywords from the table above. Causely engineers are happy to work with you to add important alerts to the knowledge base so they can participate in causal reasoning. Reach out to your Causely team with the alert details. ### Alert does not map to any entity If Causely cannot resolve an entity from the alert, the alert is not attached to the topology. This is almost always caused by **missing or incorrect labels** in the alert payload. See the next section for the exact labels required for each entity type. ## Required label combinations by entity type For an alert to be attached to an entity, its payload must include the labels that uniquely identify that entity. The exact combination depends on the type of entity the alert is about. ### Kubernetes pod **Required:** `namespace` + `pod` Use this when the alert is about a specific pod (for example, high JVM heap, GC time, thread contention, connection pool exhaustion). ```json { "alerts": [ { "status": "firing", "labels": { "alertname": "HighJVMHeapUsage", "namespace": "prod", "pod": "payment-service-7d4f8b-xyz", "severity": "warning" }, "annotations": { "summary": "JVM heap utilization above 80%" }, "startsAt": "2025-02-04T12:00:00.000Z", "endsAt": "0001-01-01T00:00:00Z" } ] } ``` ### Kubernetes container **Required:** `namespace` + `pod` + `container` Use this when the alert targets a specific container within a pod (for example, a sidecar or init container). ```json { "alerts": [ { "status": "firing", "labels": { "alertname": "ContainerHighCPU", "namespace": "prod", "pod": "payment-service-7d4f8b-xyz", "container": "payment-app", "severity": "warning" }, "annotations": { "summary": "Container CPU usage above threshold" }, "startsAt": "2025-02-04T12:00:00.000Z", "endsAt": "0001-01-01T00:00:00Z" } ] } ``` ### Kubernetes service **Required:** `namespace` + `pod` Causely resolves the owning service from the pod via topology. The optional `service` label can be included as a hint but is not required for resolution. ```json { "alerts": [ { "status": "firing", "labels": { "alertname": "HighErrorRate", "namespace": "prod", "pod": "checkout-service-abc", "service": "checkout-service", "severity": "warning" }, "annotations": { "summary": "High error rate on checkout-service", "description": "Error rate increased above threshold in the last 5m" }, "startsAt": "2025-02-04T12:00:00.000Z", "endsAt": "0001-01-01T00:00:00Z" } ] } ``` ### HTTP path **Required:** `namespace` + `pod` + a path label The path label can be any of: `uri`, `url_path`, `http_path`, `path`, `http.route`. The path entity (for example, `/checkout`) must already exist in topology (for example, from ingress or service metrics). ```json { "alerts": [ { "status": "firing", "labels": { "alertname": "HighPathErrorRate", "namespace": "ingress-nginx", "pod": "ingress-nginx-controller-956c769c7-lgcxx", "uri": "/checkout", "severity": "warning" }, "annotations": { "summary": "High error rate on /checkout" }, "startsAt": "2025-02-04T12:00:00.000Z", "endsAt": "0001-01-01T00:00:00Z" } ] } ``` ### gRPC / RPC method **Required:** `rpc_service` + `rpc_method` (or `grpc_service` + `grpc_method`) Optional: `namespace`, `pod`. The RPC method entity (service + method pair) must already exist in topology (for example, from distributed traces). ```json { "alerts": [ { "status": "firing", "labels": { "alertname": "HighGRPCErrorRate", "rpc_service": "user.UserService", "rpc_method": "GetUser", "namespace": "prod", "pod": "user-service-abc", "severity": "warning" }, "annotations": { "summary": "High error rate on user.UserService/GetUser" }, "startsAt": "2025-02-04T12:00:00.000Z", "endsAt": "0001-01-01T00:00:00Z" } ] } ``` ### Kafka topic **Required:** `topic` + consumer group label The consumer group can appear under any of these label names: `consumer_group_id`, `consumer_group`, `group`, or `consumergroup`. Optionally include `namespace` and `pod` when using Micrometer-style instrumentation. **Confluent-style** (topic + consumer group ID only): ```json { "alerts": [ { "status": "firing", "labels": { "alertname": "HighKafkaConsumerLag", "topic": "orders", "consumer_group_id": "order-processor-group", "severity": "warning" }, "annotations": { "summary": "Kafka consumer lag above threshold for topic orders" }, "startsAt": "2025-02-04T12:00:00.000Z", "endsAt": "0001-01-01T00:00:00Z" } ] } ``` **Micrometer-style** (topic + group + namespace + pod): ```json { "alerts": [ { "status": "firing", "labels": { "alertname": "HighKafkaConsumerLag", "topic": "orders", "group": "order-processor-group", "namespace": "default", "pod": "order-processor-0", "severity": "warning" }, "annotations": { "summary": "Kafka consumer lag above threshold for topic orders" }, "startsAt": "2025-02-04T12:00:00.000Z", "endsAt": "0001-01-01T00:00:00Z" } ] } ``` ### Queue (RabbitMQ or similar) **Required:** a label carrying the queue name (for example, `queue`) The queue must already exist in topology (for example, discovered from RabbitMQ or OpenTelemetry). ```json { "alerts": [ { "status": "firing", "labels": { "alertname": "MessageWaitTimeHigh", "queue": "work_queue", "severity": "warning" }, "annotations": { "summary": "Message wait time above threshold on work_queue" }, "startsAt": "2025-02-04T12:00:00.000Z", "endsAt": "0001-01-01T00:00:00Z" } ] } ``` ## Summary | Entity type | Required label combination | |---|---| | Kubernetes pod | `namespace` + `pod` | | Kubernetes container | `namespace` + `pod` + `container` | | Kubernetes service | `namespace` + `pod` (service resolved from pod via topology) | | HTTP path | `namespace` + `pod` + path label (`uri`, `url_path`, `http_path`, `path`, or `http.route`) | | gRPC / RPC method | `rpc_service` + `rpc_method` (or `grpc_service` + `grpc_method`) | | Kafka topic | `topic` + consumer group (`consumer_group_id`, `group`, `consumer_group`, or `consumergroup`) | | Queue | queue name label (for example `queue`) | --- ## Credentials Autodiscovery Causely also supports credentials autodiscovery. This feature allows you to add new scraping targets without updating the Causely configuration. Label the Kubernetes Secret to enable autodiscovery for the corresponding scraper. For example, to [enable autodiscovery for the MySQL scraper](/telemetry-sources/mysql#alternative-enable-credentials-autodiscovery), you can label the Kubernetes Secret with the following command: ```bash kubectl --namespace causely label secret mysql-credentials "causely.ai/scraper=MySQL" ``` Credentials Autodiscovery is available for many of the [telemetry sources](/telemetry-sources/). --- ## Custom Causes ## Overview When Causely receives an alert that it does not automatically recognize as a known Signal, you can define a **custom cause** to tell Causely what that alert means. Once defined, every time that alert fires, Causely surfaces it as a Diagnosis with your description and recommended remediation, connects it to any related existing root causes in the causal graph, and clears it automatically when the alert resolves. Custom causes let you bring institutional knowledge about your system into Causely's causal reasoning without writing code or modifying Causely's built-in causal models. ## Prerequisites Before creating a custom cause, verify that the alert carries enough label information for Causely to identify which entity it belongs to. In the **Integrations → Alerts** view, enable the **Entity Only** filter to see only alerts that meet this requirement. Alerts that appear in this filtered view are already resolved to a specific entity in your topology. Alerts that do not appear are missing required labels and cannot be used for custom causes until their labels are corrected. For the exact label combinations required per entity type, please see [Required label combinations by entity type](/configuration/alert-ingestion#required-label-combinations-by-entity-type). ## Creating a custom cause From the **Integrations → Alerts** view, click **Define New Cause** on any alert row, or go directly to **Settings → User Defined Causes** and click **Define New Cause**. ### Choose a category Causely uses the category to understand the failure mode the cause represents. Select one: - **Error (may cause malfunction)**: the alert indicates a problem that causes errors or downtime on the affected entity. - **Latency (may cause congestion)**: the alert indicates a problem that causes increased response time or throughput degradation. The category determines how Causely weighs the cause in causal analysis and how it is presented in the UI. ### Add description and remediation You can write a description and remediation steps manually, or use the **Generate description and remediation with AI** button to have Causely draft them based on the cause name and alert context. If AI generation fails or produces an unsatisfactory result, you can trigger it again at any time from the cause detail view after saving. Click **Create** when done. The definition is active immediately with no restart required. ## Managing your causes All defined causes are listed under **Settings → User Defined Causes**, showing the cause name, matched alert name, category, creator, and creation date. Click any row to edit the display name, description, or remediation. Changes take effect on the next analysis cycle (~30 seconds). The alert name and category cannot be changed after creation. Delete and recreate the definition if you need to change those. ## How active causes appear When an alert matching a custom cause fires, Causely surfaces it as an active Diagnosis alongside built-in causes. Clicking through to a cause shows the full detail view: description, impact graph, and the remediation steps you defined (or that were AI-generated). If the same alert fires on multiple entities simultaneously, the Diagnosis appears independently on each affected entity. ## Deleting a cause Deleting a custom cause definition immediately clears any active Diagnosis of that cause across all entities. It will not reappear even if the underlying alert is still firing. --- ## Configuration # Configuration Overview Once Causely is deployed and connected to your telemetry, this section covers how to configure how it detects, prioritizes, and reasons about issues in your environment. ## Alerts & Root Causes Configure how external alerts are ingested and how Causely reasons about issues it doesn't automatically recognize: - [**Alert Ingestion and Mapping**](/configuration/alert-ingestion): How Causely maps external alerts to entities and Signals so they participate in causal analysis - [**Custom Causes**](/configuration/custom-causes): Define your own causes for alerts Causely does not automatically recognize ## Reliability Prioritization Configure which services matter most and how aggressively Causely detects and surfaces issues: - [**Service Prioritization**](/configuration/service-tiers): Assign priority tiers to services so alerts and Diagnoses reflect what matters most - [**SLO Targets and Burn Rates**](/configuration/slo-configuration): Configure SLO targets and burn rate behavior for services, HTTP paths, RPC methods, and queues - [**Thresholds**](/configuration/thresholds): Configure the thresholds used for Signal detection and SLI-based SLO evaluation - [**Signal Delay**](/configuration/symptom-delay): Configure activation and deactivation delays to avoid reacting to temporary spikes Many of these settings can be configured through the Causely UI, Kubernetes/Nomad/Consul labels, or the API. See each page for the methods it supports. ## Views - [**Scopes**](/configuration/scopes): Define custom scopes to focus on the subset of your topology that matters most to your role or current task ## Discovery - [**Credentials Autodiscovery**](/configuration/credentials-autodiscovery): Add new scraping targets without updating the Causely configuration, by labeling the corresponding Kubernetes Secret ## General Settings Accessible through the Causely UI under [**Preferences**](https://portal.causely.app/settings?tab=general): - **Screen View Mode**: Choose the layout mode for large screens. - **Auto Refresh Interval**: Choose the frequency at which the user interface views should refresh automatically. ## Deployment Configuration The Helm values file (`causely-values.yaml`) is the primary method for configuring your Causely deployment. For a complete list of available configuration options, you can view your current values using: ```bash helm get values causely --namespace causely --all ``` Read more about the [Helm values](/installation/helm) configuration. --- ## Scopes ## Overview Scopes in Causely allow you to define and manage custom subsets of your environment's topology. As Causely automatically discovers the full topology of your environment, it can present a rich but potentially overwhelming set of entities—services, infrastructure components, and identified problems. Scopes help you focus on the specific subset of data that matters most to your role, responsibilities, or current investigative tasks. Scopes can be used in conjunction with service ownership to create more focused views of your environment. While service ownership defines who is responsible for different components, scopes allow you to create custom views that may or may not align with ownership boundaries. ## What are Scopes? User Scopes define the global context for your session, determining what data is accessible across all views. You can create, save, and manage custom Scopes to focus on specific subsets of data, such as clusters or namespaces. Scopes are persistent, can be set as default, and adapt dynamically to your selections during navigation. Scopes can be created based on specific components out of the box, such as clusters, namespaces, and services. Additionally, Causely supports the use of custom labels to provide users with more granular or domain-specific scope options. See [Using Custom Labels for Scopes](#using-custom-labels-for-scopes) for details. ## Selecting a Scope In the top navigation bar, you can select the scope you want to use for your session. ## Managing Scopes To manage your scopes: 1. Click the gear icon (⚙️) in the top right corner of the Causely UI 2. Select "Settings" from the dropdown menu 3. Navigate to the "Scopes" section ### Creating a New Scope 1. In the Scopes settings page, click the "Create" button 2. Define your scope by selecting the relevant entities and components 3. Give your scope a descriptive name 4. Save your scope ### Editing a Scope 1. Find the scope you want to edit in the Scopes list 2. Click the edit icon next to the scope 3. Modify the scope's configuration as needed 4. Save your changes ### Deleting a Scope 1. Find the scope you want to delete in the Scopes list 2. Click the delete icon next to the scope 3. Confirm the deletion ## Best Practices - Create scopes that align with your team's responsibilities and organizational structure - Use descriptive names that clearly indicate the scope's purpose - Regularly review and update your scopes to ensure they remain relevant - Consider creating scopes for different investigation scenarios or team roles ## Using Custom Labels for Scopes Out of the box, Causely provides scopes for specific components, for example cloud providers, clusters, namespaces, and service types. A cluster admin can configure Causely to automatically detect entities with custom labels and make them available as scope options in the UI. The following labels are supported: - Geography - Environment - Customer - Team - Product - Project - Service For details on how to add and use custom labels to enhance scopes, refer to the [custom labels documentation](/installation/customize/#custom-labels-for-scope-configuration). --- ## Service Prioritization ## Overview The **Service Prioritization** feature allows you to assign priority tiers to services discovered within a given scope. By default, all services are assigned a Service Level Objective (SLO) and treated as **Critical**. However, not all services are equally important. Some require immediate attention, while others can be deprioritized or hidden entirely. This feature helps teams focus on what matters, ensuring that high-priority services drive alerts and incident response while low-priority or internal services remain visible but do not create unnecessary noise. ## How It Works Each service in a scope can be categorized into one of three **Service Tiers**: | Tier | Description | | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Critical: SLO Enforced** | Services essential to your business or user experience. Their SLOs are actively enforced. If their SLO is violated or at risk, related Issues are marked **Urgent**. | | **Non-Critical: SLO Not Enforced** | Services that are non-essential or low-impact. These services do not have an active SLO, and their Issues appear in the **Issues** view but are never marked urgent. | | **Hidden: Internal Noise** | Internal or auxiliary services you do not want visible in daily operations. Issues impacting these services appear only under the **Hidden** tab in the Issues view. | ## Default Behavior - Newly discovered services start as **Uncategorized**, which defaults to **Critical: SLO Enforced**. - All services in this state are monitored with an active SLO until explicitly reassigned to another tier. ## Assigning Service Tiers 1. Navigate to **Settings → Service Tiers**. 2. Search or filter for the services you want to prioritize. Note that the available services depend on the currently selected scope. 3. Select one or more services. 4. Use the **Tier Assignment** panel to set a tier: - **Critical: SLO Enforced** - **Non-Critical: SLO Not Enforced** - **Hidden: Internal Noise** You can assign tiers individually or in bulk across pages. ## Diagnosis Visibility Tier assignment affects how Diagnoses appear in **Root Causes**: - **Critical:** Appears in **Active**. Marked **Urgent** if the SLO is violated or at risk. - **Non-Critical:** Appears in **Active** but never marked urgent. - **Hidden:** Appears only in the **Hidden** tab, not in **Active** or **Historical**. ## Example Use Cases - **Critical:** Core APIs, payments, authentication, customer-facing services. - **Non-Critical:** Internal dashboards, staging workloads, low-traffic utilities. - **Hidden:** Background jobs, test services, deprecated components. --- ## Signal Delay Causely automatically detects service [Signals](/reference/symptoms/) based on various metrics. To avoid alerting on brief spikes or temporary blips, Causely uses **activation and deactivation delays**: a Signal must remain in violation of thresholds for a sustained period before it activates, and must return to normal for the same period before it clears. This document explains how these delays work and how to configure them for your services. ## How Signal Delays Work Signal delays serve two purposes: 1. **Prevent false positives**: Brief metric spikes don't trigger Signals or alerts; only sustained issues are surfaced. 2. **Provide stability**: Signals remain active until the issue is genuinely resolved, reducing flapping and noise. The Signal activation delay depends on the *type* of issue: - **Bursty issues** (sudden spikes): the 5-minute average is > a multiple of the configurable threshold - **Sustained issues** (slow creep above threshold): the 30-minute average is > the configurable threshold The default activation delay is 5 minutes. A Signal must match one of the above conditions for 5 minutes. Deactivation also uses 5 minutes. During this 5 minute period the 5-min average must be below the higher threshold and the 30-min average must be below the threshold. ## Request Error Rate Signal Causely activates a **Request Error Rate** Signal when either of these conditions is met: - **Condition 1: Bursty spike**: 5-minute average error rate > 4× the threshold (default: 1–2%) - **Condition 2: Sustained elevation**: 30-minute average error rate > threshold In either case, the condition must be true and the request rate must be > 0.3 req/sec (0.2 req/sec for HTTP Path or RPC Method) for 5 consecutive minutes. **Deactivation:** The Signal clears in either case: - **Recovery**: Neither of the above condition 1 nor 2 holds for 5 consecutive minutes. - **Silence**: The 5-min and 30-min average request rate is ≤ 0.3 req/sec (0.2 req/sec for HTTP Path or RPC Method); deactivates regardless of error rate. **Practical Effect:** A sudden 4× error rate spike that holds for 5+ minutes triggers quickly. A service that creeps above threshold gradually requires 30 minutes of sustained violation before alerting. Once fixed, it clears after a matching period of recovery. ## Request Duration (Latency) Signal Causely activates a **Request Duration** Signal when either of these conditions is met: - **Condition 1: Bursty spike**: 5-minute average latency > 1.5× threshold - **Condition 2: Sustained elevation**: 30-minute average latency > threshold In either case, the condition must be true and the request rate must be > 0.3 req/sec (0.2 req/sec for HTTP Path or RPC Method) for 5 consecutive minutes. **Deactivation:** The Signal clears if neither of the above condition 1 nor 2 holds for 5 consecutive minutes. **Practical Effect:** A sudden 1.5× latency spike that holds for 5+ minutes triggers quickly. A service that slowly creeps above its baseline requires 30 minutes of sustained violation before alerting. Once fixed, it clears after a matching period of recovery. ## Activation Delay Configuration The activation delay for Service-level Request Error Rate and Request Duration Signals is configurable. Configurations do not apply to HTTP Path or RPC Method Signals. For threshold configuration (default thresholds for triggering Signals), see [Threshold configuration](/configuration/thresholds/). ## Configuration Methods You can configure activation delay using the following methods: - **Causely UI** Best for making a targeted adjustment while looking at the metric that's driving a Signal, without touching service metadata or deployment configuration. - **Service metadata** Lets you configure activation delay declaratively using Kubernetes labels, Nomad service tags, or Consul service metadata. Well suited for version-controlled, environment-specific configuration that travels with your service definition. - **Causely API** Best for programmatic configuration, automation, and integration with internal tooling or workflows. ### Using the Causely UI To configure activation delay in the UI: 1. Navigate to the service you want to configure. 2. Select the **Metrics** tab, then find the card for the relevant Signal metric (for example, **Request Duration** or **Request Error Rate**). 3. Click the pencil icon on the metric card to open its configuration menu, then select **Set Activation Delay**. 4. Enter the activation delay in minutes and select **Save Activation Delay**. This sets an individual override for the Signal on that entity. Leaving the field empty reverts the Signal to its default (or label-configured) activation delay. ### Using Service Metadata #### Using Kubernetes Labels Apply labels to your services: ```bash # Configure error rate activation delay (example: 3 minutes) kubectl label svc -n "causely.ai/error-rate-activation-delay=3" # Configure latency activation delay (example: 3 minutes) kubectl label svc -n "causely.ai/latency-activation-delay=3" ``` #### Using Nomad Service Tags Add tags to your service definition: ```hcl job "example" { group "app" { service { name = "my-service" port = 8080 tags = [ "causely.ai/error-rate-activation-delay=3", "causely.ai/latency-activation-delay=3" ] } } } ``` #### Using Consul Service Metadata Register services with metadata: ```bash consul services register \ -name="my-service" \ -port=8080 \ -meta="causely.ai/error-rate-activation-delay=3" \ -meta="causely.ai/latency-activation-delay=3" ``` #### Delay Values - **Valid Range:** 1–60 minutes - **Configures:** Bursty activation delay only (default: 5 minutes) - **Fixed:** Sustained activation delay (always 30 minutes) - **Recommended Range:** 1–10 minutes for most use cases ### Using the Causely API Activation delay can also be configured programmatically through the Causely GraphQL API, at the individual entity, label, or global scope. This is best suited for automation, bulk configuration, or integrating activation delay changes into internal tooling. ## Best Practices 1. **Start with defaults**: 5 minutes works for most services. Only adjust if you're seeing false positives (bursty spikes that recover quickly) or need faster detection of genuine issues. 2. **Shorter delays (1–3 min) for**: - Payment or high-revenue services where quick spike detection is critical - Services where you expect sudden issues to be real problems, not transient noise 3. **Longer delays (5–10 min) for**: - Services with frequent harmless spikes, for example, traffic bursts, cache refreshes - Noisy services where bursty patterns are normal and don't indicate problems - Non-production environments 4. **Keep in mind**: The 30-minute sustained delay is fixed and cannot be adjusted. This ensures slow, creeping issues are not misclassified as false positives. 5. **Monitor after changes**: After adjusting bursty delays, observe whether spike detection improves or false positives decrease. 6. **Document your reasoning**: Keep notes on why you chose specific delays per service. This helps during onboarding and reviews. ## Examples ### Critical Service (Payment Processing) Faster detection of error spikes: **Kubernetes:** ```bash kubectl label svc -n production payment-api "causely.ai/error-rate-activation-delay=2" kubectl label svc -n production payment-api "causely.ai/latency-activation-delay=2" ``` **Effect:** Error rate spikes are detected after 2 minutes of sustained elevation (vs. default 5). Slow degradation still requires 30 minutes. ### Bursty Service (Traffic Spikes) Reduce noise from normal traffic bursts: **Kubernetes:** ```bash kubectl label svc -n production data-processor "causely.ai/error-rate-activation-delay=8" kubectl label svc -n production data-processor "causely.ai/latency-activation-delay=8" ``` **Effect:** Brief error spikes during load bursts don't trigger until 8 minutes (giving time to recover). Slow issues still activate after 30 minutes. ### Development Environment Longer delays to minimize disruptions: **Kubernetes:** ```bash kubectl label svc -n dev api-service "causely.ai/error-rate-activation-delay=10" kubectl label svc -n dev api-service "causely.ai/latency-activation-delay=10" ``` **Effect:** Dev spikes require 10 minutes to trigger (vs. 5). Sustained issues always take 30 minutes, same as production. --- ## Thresholds for Signals Causely detects service and infrastructure [Signals](/reference/symptoms/) based on a wide range of metrics, using a combination of defaults and learned behavior. However, you may want to customize these thresholds to better match your specific requirements and SLO definitions. This document explains how to configure custom thresholds for your services. ## Overview Causely uses thresholds to detect service and infrastructure Signals across a wide range of metrics, including latency, error rates, throughput, and resource utilization. These thresholds serve two related purposes: - **Signal detection**, where crossing a threshold activates a Causely issue or risk - **SLO evaluation**, where the same metrics act as Service Level Indicators (SLIs) that determine SLO health Causely provides sensible defaults for all supported thresholds. For some metrics, Causely can also **learn thresholds automatically** based on historical and real-time behavior. You can optionally configure **manual thresholds** or tune learned thresholds to better reflect business requirements, reliability targets, or known system constraints. ## Key Concepts ### Threshold Sources Each threshold in Causely has a **source**, which determines how its value is set and maintained: - **Default** A system-provided threshold value that applies when no learning or manual override is configured. Defaults are designed to be safe and broadly applicable. - **Learned** For some metrics, Causely can automatically learn a threshold based on historical and real-time behavior. Learned thresholds adapt over time as normal behavior changes. - **Manual** A user-configured threshold that explicitly overrides the default or learned value. Manual thresholds remain fixed until changed or removed. Only one source is active for a given threshold at any time (minimum learned thresholds do not change the source). ### How Thresholds Are Evaluated Thresholds are always evaluated against a **specific metric and aggregation**, even when multiple series are shown for context. For example: - **Request Duration thresholds are evaluated against P95 latency** A Signal becomes active when the P95 latency exceeds the configured threshold. - Other percentiles (such as P90 or P99) may be displayed to provide additional context but do not drive activation. The evaluation aggregation is fixed per metric and does not change when thresholds are overridden. ### Learned Threshold Minimums For metrics that support learning, you can optionally configure a **Minimum Learned Threshold**. A minimum learned threshold: - Sets a lower bound on how low a learned threshold can go - Allows Causely to continue learning above that value - Does **not** create a manual override This is useful when you want adaptive behavior while preventing learned thresholds from becoming unrealistically low due to traffic patterns or short-term anomalies. ### Thresholds and SLOs The same metrics used for Signal detection are also used as **Service Level Indicators (SLIs)** when evaluating SLOs. This means: - A threshold crossing may activate a Signal - The same metric contributes to SLO health calculations Configuring thresholds affects **both operational detection and SLO evaluation**, so changes should be made with awareness of their broader impact. ## How to Configure Thresholds Causely supports configuring thresholds through multiple mechanisms, allowing you to choose the approach that best fits your workflow and environment. Regardless of the method used, the outcome is the same: a **manual threshold**, unless you configure a minimum learned threshold. Thresholds can be configured at different levels (for example, service, workload, or infrastructure resource), depending on the metric and entity type. ### Configuration Options You can configure thresholds using the following methods: - **Causely UI** Best for inspecting learned behavior, understanding how thresholds relate to observed metrics, and making targeted adjustments. - **Service metadata** Service metadata supports configuring a subset of commonly used service-level thresholds and allows you to configure thresholds declaratively using: - Kubernetes labels - Nomad service tags - Consul service metadata This approach is well suited for version-controlled, environment-specific configuration that travels with your service definition. - **Causely API** Best for programmatic configuration, automation, and integration with internal tooling or workflows. All configuration methods support configuring **manual thresholds where applicable**, and for metrics that support learning, configuring a **minimum learned threshold** to bound adaptive learning. ### Choosing the Right Method Use the **UI** when you want to: - understand why a Signal is activating, - compare learned thresholds against real traffic, - experiment or iterate quickly. Use **service metadata** when you want to: - manage thresholds as code, - apply consistent thresholds across environments, - ensure thresholds are applied automatically during deployment. Use the **API** when you want to: - automate threshold management, - integrate threshold changes into CI/CD or internal systems, - apply changes across many entities programmatically. ### What Happens When You Configure a Threshold When you configure a threshold: - The threshold source becomes **Manual** - Automatic learning (if supported for that metric) is paused - The configured value is used consistently for: - Signal activation - SLI evaluation for SLOs If you remove a manual threshold, Causely reverts to the **default** or **learned** threshold, depending on the metric. ### Minimum Learned Thresholds For metrics that support learning, you can optionally configure a **Minimum Learned Threshold** instead of a full manual override. This allows Causely to: - continue adapting to changing behavior, - while never learning a threshold below the configured minimum. Minimum learned thresholds do **not** replace learning and do not create a manual override. ### Using the Causely UI Using the UI allows you to: - Inspect the learned threshold alongside observed metrics (for example P90 and P99 for latency) - Override thresholds to match documented SLOs or performance requirements - Set a minimum value that bounds how low a learned threshold can go while preserving adaptive learning - Immediately see how a custom threshold compares to real traffic patterns To configure thresholds in the UI: 1. Navigate to the service you want to configure. 2. Select the **Metrics** tab for the service, then select the relevant Signal metric (for example, Request Duration or Request Error Rate). 3. Click the pencil icon next to the threshold to edit the value. 4. Save the change to apply the override. UI-based configuration is best suited for teams that want quick iteration, visibility into learned behavior, and explicit control without modifying service metadata or deployment configuration. ### Using Service Metadata :::note Not all thresholds can be configured via service metadata. Metadata-based configuration currently supports a subset of service-level thresholds such as request error rate and request latency. ::: #### Using Kubernetes Labels The recommended way to configure thresholds is using Kubernetes labels. You can apply these labels to your services: ```bash # Configure error rate threshold (for example, 1% error rate) kubectl label svc -n "causely.ai/error-rate-threshold=0.01" # Configure latency threshold (for example, 500ms) kubectl label svc -n "causely.ai/latency-threshold=500.0" ``` #### Using Nomad Service Tags For Nomad services, you can configure thresholds using service tags in your job specification: ```hcl job "example" { group "app" { service { name = "my-service" port = 8080 tags = [ "causely.ai/error-rate-threshold=0.01" "causely.ai/latency-threshold=500.0" ] } } } ``` #### Using Consul Service Metadata For Consul services, you can configure thresholds using service metadata: ```bash # Register a service with threshold metadata consul services register \ -name="my-service" \ -port=8080 \ -meta="causely.ai/error-rate-threshold=0.01" \ -meta="causely.ai/latency-threshold=500.0" # Update existing service metadata consul services register \ -id="my-service-id" \ -name="my-service" \ -port=8080 \ -meta="causely.ai/error-rate-threshold=0.01" \ -meta="causely.ai/latency-activation-delay=500.0" ``` ## Supported Thresholds Causely supports configurable thresholds across a broad set of service and infrastructure entities. These thresholds are used to detect Signals and also act as Service Level Indicators (SLIs) when evaluating SLO health. Some thresholds support **automatically learned values**, while others use **system defaults** that can be manually overridden. --- ### Services, Workloads, HTTP Paths and RPC Methods | Metric | Unit | Learned | |------|------|---------| | Request Error Rate | percent | No | | Request Duration (P95) | millisecond | Yes | | Request Duration P95 (Client) | millisecond | Yes | | Request Rate | request/s | Yes | | Connections | percent | No | | Mutex Wait Time | percent | No | | Command Latency | millisecond | No | | GC Time | percent | No | | Queries Queued | count | No | | Transaction Error | percent | No | | Transaction Duration | second | No | | Transaction IDs Congested | percent | No | | Cache Size | bytes | No | | Redis Connections Utilization | percent | No | | Kafka Message Rate | message/s | No | | Server Errors | count | No | | User Errors | count | No | | File Descriptor Utilization | percent | No | | Java Heap Utilization | percent | No | | Throttled | count | No | | DB Connections Utilization | percent | No | --- ### Queues, Topics and Background Operations | Metric | Unit | Learned | |------|------|---------| | Queue Depth | count | No | | Dead Letter Count | count | No | | Queue Acks | request/s | No | | Message Wait Time | seconds | No | | Queue Size Bytes | bytes | No | | Lag | count | No | | Task Duration | millisecond | No | --- ### Database Tables | Metric | Unit | Learned | |------|------|---------| | DB Query Duration | second | Yes | | Select Query Duration (P95) | millisecond | Yes | | Table Bloat | percent | No | | Lock Exclusive Rate | percent | No | | DDL Lock Exclusive Rate | percent | No | --- ### Application Load Balancers | Metric | Unit | Learned | |------|------|---------| | Request Rate | request/s | Yes | | Request4xx Error | percent | No | | Request5xx Error | percent | No | | Request504 Error | percent | No | | ELB Auth Error | count | No | | Target Connection Error | count | No | --- ### Containers and Controllers | Metric | Unit | Learned | |------|------|---------| | CPU Utilization | percent | No | | CPU Throttled | percent | No | | Memory Utilization | percent | No | | Ephemeral Storage Utilization | percent | No | | Frequent Crash | count | No | | FrequentOOM Kill | count | No | | Frequent Pod Ephemeral Storage Evictions | count | No | --- ### Nodes and Virtual Machines | Metric | Unit | Learned | |------|------|---------| | CPU Utilization | percent | No | | Memory Utilization | percent | No | | Conntrack Table Utilization | percent | No | | SNAT Port Utilization | percent | No | | Container Ephemeral Storage Utilization | percent | No | | Memory Pressure Pod Evictions | count | No | | Disk Pressure Pod Evictions | count | No | | Disk Read IOPS Utilization | percent | No | | Disk Write IOPS Utilization | percent | No | | Disk Total IOPS Utilization | percent | No | | Disk Read Throughput Utilization | percent | No | | Disk Write Throughput Utilization | percent | No | | Disk Total Throughput Utilization | percent | No | --- ### Disks | Metric | Unit | Learned | |------|------|---------| | Utilization | percent | No | | Read IOPS Utilization | percent | No | | Write IOPS Utilization | percent | No | | Total IOPS Utilization | percent | No | | Read Throughput Utilization | percent | No | | Write Throughput Utilization | percent | No | | Total Throughput Utilization | percent | No | | Inodes Utilization | percent | No | ## Best Practices 1. **Start with Default or Learned Thresholds** Use Causely’s default or automatically learned thresholds as a baseline before introducing manual overrides. 2. **Override Only When There Is a Clear Requirement** Configure manual thresholds when you have explicit business, reliability, or compliance requirements that differ from observed behavior. 3. **Prefer Minimum Learned Thresholds Over Full Overrides** When available, use a minimum learned threshold to bound adaptive learning without disabling it entirely. 4. **Consider Both Signal Detection and SLO Impact** Threshold changes affect both Signal activation and SLI evaluation for SLOs. Validate changes in both contexts. 5. **Monitor After Changes** After updating thresholds, observe how they affect Signal frequency, noise, and SLO evaluation over time. 6. **Document Intent, Not Just Values** Record why a threshold was changed to support future reviews and adjustments. ## Example Use Cases 1. **Strict SLO Requirements** A critical service requires tighter latency bounds than normal traffic patterns allow. A manual threshold is configured to align Signal detection and SLO evaluation with the defined objective. 2. **Preventing Overly Aggressive Learned Thresholds** A service with highly variable traffic uses a minimum learned threshold to prevent latency thresholds from adapting too low during off-peak periods while preserving adaptive behavior. 3. **Infrastructure Saturation Detection** A team configures CPU or disk utilization thresholds on nodes to detect resource saturation early, independent of application-level Signals. 4. **Queue Backlog Monitoring** Queue depth and message wait time thresholds are configured to surface processing delays before they impact downstream services. 5. **Temporary Adjustments During Maintenance** Thresholds are temporarily adjusted during planned maintenance or migrations and reverted afterward. --- ## Architecture Causely is built on a split-architecture model that balances local control with a dedicated reasoning backend that can run in your cloud or Causely-managed infrastructure. This design ensures low overhead, strong data privacy, and seamless integration with existing tools. This page provides detailed information about Causely's deployment architecture and component structure. For a high-level overview of how Causely's causal reasoning engine works, see [How Causely Works](/getting-started/how-causely-works). ## System Architecture {(() => { const { colorMode } = useColorMode(); const src = colorMode === 'dark' ? '/img/how_causely_works_diagramLight1.svg' : '/img/how_causely_works_diagramDark1.svg'; return ; })()} ## Deployment Architecture Causely can be deployed across various environments, including Kubernetes clusters, standalone Docker hosts, Nomad clusters, and more. The deployment architecture consists of several components that work together to provide a real-time Diagnosis. :::info Deployment model Causely is designed for flexible deployment. The mediation layer and agents always run in your environment. The Causely backend (causal engine) can run in your cloud (BYOC) or in Causely-managed infrastructure. ::: ### Mediation Layer The mediation layer is deployed locally in your infrastructure and processes telemetry data to extract only the signals needed for reasoning. It performs: - **Signal Detection**: Converts telemetry from Prometheus, CloudWatch, Datadog, OpenTelemetry (including eBPF), and other sources into a binary stream of active/inactive [signals](/reference/symptoms/). - **Topology Discovery and Ingestion**: Leverages integrated telemetry sources to discover entities and dependencies and ingest topology from systems such as OpenTelemetry, cloud provider APIs and other sources. - **Local Processing**: Processes telemetry locally to minimize data transfer, control cost, and ensure data stays within your environment. Most raw telemetry remains local, with only distilled insights and targeted evidence sent to the Causely backend (deployed in your cloud or Causely-managed) for analysis. The mediation layer primarily sends distilled insights to the cloud. After a Diagnosis is made, a targeted subset of relevant telemetry (metrics, traces, and log-derived errors/events) may be sent as evidence to enhance its clarity. For more detail on supported telemetry sources, see [Supported Telemetry](/telemetry-sources/). The mediation layer consists of the following components: #### Mediator The Mediator is the core component that runs locally in your environment and serves as the data processing layer: - **Signal Detection**: Converts telemetry from various sources into binary signal states - **Topology Discovery**: Automatically discovers services, infrastructure, and dependencies - **Local Processing**: Processes telemetry locally, with most raw telemetry remaining in your datacenter - **OTLP Endpoint**: Listens on port 4317 for OpenTelemetry Protocol data The Mediator handles secure communication with the Causely causal reasoning backend (deployed in your cloud or Causely-managed), primarily sending distilled insights. After a Diagnosis is made, a targeted subset of relevant telemetry may be sent as evidence to enhance its clarity. The Mediator can also be optionally configured to get metrics from Prometheus or discover and monitor managed cloud services from cloud providers. :::info Mediator scale behavior The Mediator is designed to work from a **statistically significant sample** of traces and metrics. It does not need 100% of raw telemetry to perform accurate analysis. The built-in OpenTelemetry Collector limiter in the Mediator will drop excess traces when memory pressure is reached. This is **intentional behavior**, not data loss. It means the Mediator has collected sufficient data for analysis. Metrics and traces you see in Causely may therefore differ from totals in external monitoring tools. This is expected and does not affect analysis accuracy. Mediator memory scales with the **number of entities under management** (services, pods, infrastructure components), not with trace or metric throughput. A cluster with more services needs more Mediator memory; a high-traffic cluster with the same number of services does not. For sizing guidance, see [Sizing the Mediator for large environments](/installation/customize#sizing-the-mediator-for-large-environments). ::: #### Agents Agents are deployed across your infrastructure to gather node and container level metrics. The deployment method varies depending on your environment: - **Kubernetes**: Agents are deployed as a DaemonSet across all nodes in the cluster - **Docker**: Agents run as containers on standalone Docker hosts - **Nomad**: Agents are deployed as Nomad jobs across the cluster Agents leverage [eBPF](/telemetry-sources/ebpf) technology, which requires privileged access to the host system. This enables automatic instrumentation without code changes. The eBPF instrumentation uses **[uprobes](https://docs.kernel.org/trace/uprobetracer.html) exclusively** to intercept specific user-space functions within your applications. It does not hook into kernel networking callbacks or the packet datapath, and does not act as a [Container Network Interface (CNI)](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/) or network infrastructure component. The instrumentation may inject trace context headers for distributed tracing, but does not intercept, block, or route network traffic at the kernel level. Agents don't establish any outbound connections to the internet or any other service apart from the Mediator and VictoriaMetrics. The agents periodically forward the topology and manifestation data to the Mediator, which, in turn, sends it to the Causely backend for analysis (running in your cloud or Causely-managed). If an agent fails or is removed, your applications and network continue to function normally. ##### Agent Architecture {(() => { const { colorMode } = useColorMode(); const src = colorMode === 'dark' ? '/img/architecture/architecture-detailed-dark.svg' : '/img/architecture/architecture-detailed-light.svg'; return ; })()} #### Executor The Executor is an optional component responsible for executing remediation actions within your infrastructure. The Executor can be enabled as part of the deployment process. The specific permissions required depend on your deployment environment: - **Kubernetes**: The Executor's ServiceAccount is granted the `cluster-admin` role - **Docker/Nomad**: The Executor requires appropriate permissions to execute remediation actions #### VictoriaMetrics VictoriaMetrics is a timeseries database used by the agents and mediator (on `port: 8428`) to store additional timeseries data locally in your environment. If you observe gaps in metrics graphs, VictoriaMetrics may be under-resourced for your environment. See [Sizing VictoriaMetrics for high-volume environments](/installation/customize#sizing-victoriametrics-for-high-volume-environments). ### Causal Engine The Causal Engine runs in the Causely backend, which can be deployed in your cloud or in Causely-managed infrastructure. It receives the stream of signal states and performs real-time analysis using probabilistic modeling, system graphs, and causal inference. It produces Diagnoses, evaluates blast radius, validates constraints, and prioritizes remediation, all without requiring manual correlation. ### Telemetry Sources Causely supports a wide range of telemetry sources, including OpenTelemetry, Prometheus, CloudWatch, Datadog, and more. For a full list of supported telemetry sources, see [Supported Telemetry](/telemetry-sources/). Causely Agents are deployed in your infrastructure and are responsible for collecting the telemetry data from those sources. By default Causely will automatically instrument your applications to receive [OpenTelemetry](/telemetry-sources/opentelemetry) traces. This allows Causely to discover service dependencies, monitor sync and async communication signals. Additionally you can export traces to Causely from your existing OpenTelemetry Collectors. We recommend that you always send OpenTelemetry traces to Causely, as this allows Causely to provide cross-service insights. ### Agent Integration Causely exposes its causal model to AI agents via an MCP server. Any MCP-compatible agent, including Claude Code, Cursor, HolmesGPT, or a custom agent, gets deterministic access to root causes, service health, dependency maps, and blast radius. Instead of reasoning over raw telemetry, agents query structured, causal system knowledge. This reduces token consumption, eliminates guesswork, and lets agents take action with auditable evidence. For details on connecting agents to Causely, see [Agent Integration](/agent-integration/). ### Workflow Integration Causely integrates directly into your tools of choice, delivering causal insights into Slack, Alertmanager, Opsgenie, Grafana, and more. For details on how to connect Causely to your workflows, see [Supported Workflows](/workflows/). This architecture allows Causely to deliver precise, real-time insights without burdening your data pipelines or violating privacy requirements. ## Security Considerations For detailed information about security, permissions, and data handling, see the [Security](/security/) documentation. --- ## First-Time Usage Guide This short guide walks you through the core actions to take right [after setup](/getting-started/quick-setup). No deep configuration or prior knowledge needed, just follow along and start uncovering real insights from your system. ## Understanding the Summary Insight Panel Your journey begins at the [Causely Portal](https://portal.causely.app/welcome), where you'll find the **Summary Insight Panel**. This is your command center for monitoring system health and identifying critical issues. Unlike traditional monitoring systems that bombard you with alerts, Causely focuses on delivering the **underlying cause** through our innovative **Issue Cards**. Each card represents a verified issue in your system, backed by our extensive knowledge base of cloud native failure patterns. :::tip Pro Tip To maximize efficiency, [integrate Causely with your existing tools](/workflows) for seamless workflow integration. ::: ## Exploring Issue Details When you click on an Issue Card, you'll access the **Issue Details View**, your comprehensive dashboard for understanding and resolving system issues. This view is organized into three main sections: ### 1. Summary Tab The Summary tab provides essential information: - **Overview**: Current status, affected services, SLO violations, and timeline - **Diagnosis Description**: Clear explanation of the issue - **Impact Analysis**: Detailed assessment of system effects - **Evidence Collection**: - Observed [signals](/reference/symptoms/) - System exceptions - Relevant logs - Related events - **Remediation Steps**: Actionable solutions to resolve the issue ### 2. Causality Tab The Causality tab presents a visual representation of the issue's causal path, helping you understand how different system components interact. ### 3. Timeline Tab The Timeline tab provides a visualization of all Diagnoses, Signals, events, and SLO breaches associated with the Issue, from first detection to resolution. :::tip Best Practice The Issue Details View is your primary investigation tool. In many cases, the suggested remediation steps provide a complete solution to your system issues. ::: ## Connect your AI Agent to Causely Our [MCP Server](/agent-integration/mcp-server/) exposes Causely's causal model to any AI agent: Claude, Cursor, or your own tooling. Once connected, your agent can answer questions like: - "What's causing this Issue?" - "How's this affecting our services?" - "What's the timeline of an Issue?" - "What steps should we take to fix this?" [Setup the MCP Server](/agent-integration/mcp-server/) ## Next Steps for reducing MTTR Now that you've mastered the basics, we recommend: 1. [Connecting agents](/agent-integration/) for faster issue resolution and proactive remediation 2. [Adding additional telemetry sources](/telemetry-sources/) for more comprehensive analysis 3. [Setting up workflow integrations](/workflows) with your existing tools --- ## Start with Causely asdf this goes that way --- ## Supported Technologies Causely integrates with your existing infrastructure, observability stack and AI agents. This page provides a complete reference of all supported technologies across four areas: - **[Agents](/agent-integration)**: Give MCP-compatible agents like Claude Code, Cursor, and HolmesGPT deterministic causal context. - **[Installation](/installation)**: Deploy Causely's mediator in Kubernetes, Docker, or other environments - **[Telemetry Sources](/telemetry-sources)**: Connect metrics, traces, logs, and infrastructure data from platforms you already use - **[Workflows](/workflows)**: Route insights to Slack, Microsoft Teams, Grafana, and other tools your team relies on Use the search and filters below to find what you need. :::tip Missing something? [Let us know](mailto:support@causely.app)—we're always expanding our integrations. ::: --- ## Accelerate Resolution This page explains how Causely helps you accelerate incident resolution by providing precise Diagnosis insights when you need them most, reducing alert noise, and enabling faster problem resolution. Once you've [installed Causely](/getting-started/quick-setup), the system automatically generates your topology graph and infers Diagnoses that explain observed Signals. For more details on how Causely builds this understanding, see [How Causely Works](/getting-started/how-causely-works). ## Configure Workflow Notifications To receive Diagnosis insights during incidents, configure a workflow integration such as [Slack](/workflows/slack), [Microsoft Teams](/workflows/microsoft-teams), or [Alertmanager](/workflows/prometheus-alertmanager/). See the [workflows overview](/workflows) for all available integrations. ### Noise Reduction When incidents occur, Causely sends notifications for your Issues, including their Diagnoses, directly to your configured workflow. Unlike traditional alerting systems that create alert storms with one alert per Signal, Causely sends a single Issue notification that explains multiple Signals. This dramatically reduces noise and helps you focus on what actually matters. ### Example As an example, Causely sends a message to slack about a [`Slow Consumer`](/reference/root-causes/applications/#slow-consumer), which is the cause of 7 Signals across multiple services and key customer interactions. ## Use Issue View for Fast Resolution The [Issue view](/in-action/root-causes) provides detailed information about each identified Diagnosis: - **Evidence**: Metrics, logs, and traces that support the inference - **Impact**: Which services, entities, and operations are affected (blast radius) - **Causality Graph**: Visual representation of how the cause relates to Signals - **Remediation Guidance**: Actionable remediation steps tailored to the specific issue Urgent Issues are marked red and indicate problems currently impacting your services and SLOs. Use this view to quickly understand what's wrong, why it's happening, and how to fix it. ## Improve Resolution Speed To further accelerate resolution: - **Define service priorities**: Configure [service tiers](/configuration/service-tiers) to help Causely prioritize which services matter most when identifying urgent Diagnoses. - **Connect more telemetry**: Add additional [telemetry sources](/telemetry-sources) to provide more comprehensive coverage and more accurate Diagnoses. More telemetry means more accurate and complete causal insights. - **Configure thresholds**: Set up [thresholds](/configuration/thresholds) to fine-tune when Signals are considered problematic, helping Causely better identify urgent issues. - **Configure Signal activation delay**: Adjust [Signal activation delay](/configuration/symptom-delay) to balance between quick detection and reducing false positives. Shorter delays (1-3 minutes) enable faster Diagnosis for critical services, while longer delays (5-10 minutes) help filter out temporary spikes. - **Set up SLOs**: Configure [SLOs](/configuration/slo-configuration) to help Causely understand your reliability goals and better identify when Diagnoses are putting your SLOs at risk. ## Related Features - [Issues](/in-action/root-causes) - Explore all identified Issues - [MCP Server](/agent-integration/mcp-server/) - Query the system during incidents - [Topology](/in-action/topology) - Investigate affected entities - [Workflows](/workflows) - Configure notifications --- ## Optimize Performance This page explains how Causely helps you continuously improve system performance by providing actionable insights into bottlenecks, inefficiencies, and optimization opportunities. Once you've [installed Causely](/getting-started/quick-setup), the system automatically generates your topology graph and identifies performance bottlenecks, resource contention, and optimization opportunities. For more details on how Causely builds this understanding, see [How Causely Works](/getting-started/how-causely-works). ## Identify Performance Issues Causely identifies performance issues through root cause analysis that reveals: - **Resource contention and bottlenecks**: Understand where resources are constrained and causing performance degradation - **Inefficient operations and queries**: Identify operations that are consuming excessive resources or taking too long - **Scaling opportunities**: Find services or components that would benefit from scaling - **Optimization targets**: Pinpoint specific areas where performance improvements would have the most impact The [Issues view](/in-action/root-causes) shows performance-related Issues with detailed evidence, impact analysis, and remediation guidance. ## Track Performance Trends Use historical Issues and the [Topology view](/in-action/topology) to track performance trends over time and understand: - **How changes impact performance**: See how deployments, configuration changes, or infrastructure updates affect system performance - **Seasonal or cyclical patterns**: Identify recurring performance patterns that might indicate optimization opportunities - **Long-term degradation or improvement**: Track whether performance is improving or degrading over time The [Reliability Delta](/in-action/reliability-delta) feature allows you to compare performance snapshots and track improvements or regressions. ## Get Actionable Recommendations Issues include specific remediation guidance that helps you optimize performance, such as: - Scaling recommendations for services or infrastructure - Query optimizations for databases - Infrastructure improvements - Configuration adjustments You can also use your favorite AI agent through our [MCP Server](/agent-integration/mcp-server/) to query performance-related questions and get insights about optimization opportunities. ## Improve Performance Optimization To better identify and act on performance optimization opportunities: - **Connect more telemetry**: Add additional [telemetry sources](/telemetry-sources) to provide more comprehensive performance data. More telemetry means better identification of bottlenecks and optimization targets. - **Configure thresholds**: Set up [thresholds](/configuration/thresholds) to fine-tune when performance issues are considered problematic, helping Causely better identify optimization opportunities. - **Define service priorities**: Configure [service tiers](/configuration/service-tiers) to help Causely prioritize which services' performance matters most. - **Automate remediation**: Use [Automate Remediation](/in-action/automate-remediation) to automatically scale resources or take other actions when performance issues are identified. ## Related Features - [Issues](/in-action/root-causes) - Find performance-related Issues - [Automate Remediation](/in-action/automate-remediation) - Automatically scale resources - [Topology](/in-action/topology) - Analyze entity performance metrics - [MCP Server](/agent-integration/mcp-server/) - Query performance-related questions --- ## What are your goals? , title: 'Accelerate resolution', description: 'Get precise Diagnosis insights during incidents to resolve issues faster.', to: '/goals/accelerate-resolution', }, { icon: , title: 'Proactively prevent incidents', description: 'Identify emerging risks before they impact your services and SLOs.', to: '/goals/prevent-incidents', }, { icon: , title: 'Understand system behavior', description: 'Gain deep insights into why your systems behave the way they do.', to: '/goals/understand-system-behavior', }, { icon: , title: 'Scale reliability', description: 'Maintain reliability across complex, fast-changing systems at scale.', to: '/goals/scale-reliability', }, { icon: , title: 'Optimize performance', description: 'Continuously improve system performance with actionable insights.', to: '/goals/optimize-performance', }, ]} teaser='Discover how Causely helps you achieve your reliability goals and move from reactive troubleshooting to proactive reliability.' /> --- ## Proactively Prevent Incidents This page explains how Causely enables proactive incident prevention by identifying emerging risks before they impact your services and SLOs, helping you address issues before they escalate into incidents. Once you've [installed Causely](/getting-started/quick-setup), the system automatically generates your topology graph and infers both urgent Diagnoses and non-urgent Diagnoses (emerging risks). For more details on how Causely builds this understanding, see [How Causely Works](/getting-started/how-causely-works). ## Configure Workflow Notifications To receive Diagnosis insights, configure a workflow integration such as [Slack](/workflows/slack), [Microsoft Teams](/workflows/microsoft-teams), or [Alertmanager](/workflows/prometheus-alertmanager/). See the [workflows overview](/workflows) for all available integrations. Causely sends **urgent Issues** directly to your configured workflow when incidents occur, helping you [accelerate resolution](/goals/accelerate-resolution). Additionally, Causely continuously creates **non-urgent Issues** that often call out reliability issues and emerging risks before they escalate into incidents. ## Identify Emerging Risks Causely continuously monitors your environment and identifies potential issues before they escalate into incidents. Non-urgent Issues often represent emerging risks that can be addressed proactively. These are Issues that are active but not yet impacting your services and SLOs. By detecting issues early, Causely helps protect your Service Level Objectives (SLOs) and error budgets. You can address problems before they impact user experience or violate SLO commitments. ## Monitor Trends and Patterns Use the [Issue view](/in-action/root-causes) to monitor patterns and trends over time. Review historical Issues to identify: - Recurring issues that might indicate systemic problems - Gradual degradation that might not trigger immediate alerts but could lead to problems - Patterns that help you understand your system's behavior over time The [Reliability Delta](/in-action/reliability-delta) feature helps shift reliability left by allowing you to compare reliability snapshots and track improvements or regressions over time. Before deploying changes, you can create a snapshot to establish a baseline. After deployment, compare the new snapshot to identify any reliability regressions. ## Improve Prevention To better identify and prevent incidents: - **Configure workflow connections**: Set up workflow integrations such as [Slack](/workflows/slack), [Microsoft Teams](/workflows/microsoft-teams), or [incident.io](/workflows/incident-io) to receive both urgent and non-urgent Diagnosis notifications. This ensures you're notified about emerging risks as soon as they're identified. - **Configure SLOs**: Set up [SLOs](/configuration/slo-configuration) to help Causely understand your reliability goals and better identify when Diagnoses are putting your SLOs at risk, even before they become urgent. - **Set thresholds**: Configure [thresholds](/configuration/thresholds) to fine-tune when Signals are considered problematic, helping Causely identify emerging risks earlier. - **Connect more telemetry**: Add additional [telemetry sources](/telemetry-sources) to provide more comprehensive coverage and more accurate Diagnoses. More telemetry means earlier detection of potential issues. - **Define service priorities**: Configure [service tiers](/configuration/service-tiers) to help Causely prioritize which services matter most when identifying risks. ## Related Features - [Reliability Delta](/in-action/reliability-delta) - Compare reliability snapshots - [Issues](/in-action/root-causes) - Review non-urgent Diagnoses and emerging risks - [Configuration](/configuration) - Set up thresholds and SLOs --- ## Scale Reliability This page explains how Causely helps you maintain reliability across complex, fast-changing systems at scale by automatically understanding your entire environment and reducing alert noise. Once you've [installed Causely](/getting-started/quick-setup), the system automatically generates your topology graph and infers Diagnoses, even as your system grows and changes. For more details on how Causely builds this understanding, see [How Causely Works](/getting-started/how-causely-works). ## Automatic Discovery at Scale As your system evolves, Causely automatically discovers new services, entities, and relationships without manual configuration. This keeps your reliability insights current as you scale, whether you're: - Deploying new services or applications - Adding infrastructure components - Changing service dependencies - Scaling horizontally or vertically The system continuously updates its understanding of your environment, adapting to new deployments, releases, infrastructure changes, and performance patterns. ## Reduce Alert Noise At scale, alert noise becomes overwhelming. Traditional alerting systems create one alert per Signal, leading to alert storms that make it impossible to identify what actually matters. Causely reduces noise by inferring Diagnoses that explain multiple Signals. Instead of receiving hundreds of alerts, you get a single notification that explains the underlying issue. This helps you focus on what matters and maintain reliability even as your system grows. ## Improve Reliability at Scale To better maintain reliability as you scale: - **Connect more telemetry**: Add additional [telemetry sources](/telemetry-sources) to provide comprehensive coverage across your entire system. More telemetry means better discovery and more accurate Diagnoses. - **Configure scopes**: Set up [scopes](/configuration/scopes) to organize your entities and manage complexity as your system grows. - **Define service priorities**: Configure [service tiers](/configuration/service-tiers) to help Causely prioritize which services matter most, especially important at scale when you have many services. - **Set up SLOs**: Configure [SLOs](/configuration/slo-configuration) to help Causely understand your reliability goals across all your services and better identify when Diagnoses are putting your SLOs at risk. - **Use Reliability Delta**: Track reliability trends over time with [Reliability Delta](/in-action/reliability-delta) to understand how your system's reliability changes as it scales. ## Related Features - [Topology](/in-action/topology) - View all discovered entities - [Reliability Delta](/in-action/reliability-delta) - Track reliability at scale - [Configuration](/configuration) - Configure scopes and service tiers --- ## Understand System Behavior This page explains how Causely provides deep insights into why your systems behave the way they do by building a causal model of your entire environment and making it queryable. Once you've [installed Causely](/getting-started/quick-setup), the system automatically generates your topology graph and builds a comprehensive understanding of your system's behavior and relationships. For more details on how Causely builds this understanding, see [How Causely Works](/getting-started/how-causely-works). ## Explore Your System Topology The [Topology view](/in-action/topology) provides a comprehensive view of your entire system. You can explore: - **Service dependencies and relationships**: Understand how services connect and depend on each other - **Data flow across operations**: See how data moves through your system - **Infrastructure stack**: View the infrastructure components that support each entity - **Historical behavior patterns**: Analyze how your system has behaved over time This topology graph forms the foundation for understanding system behavior, blast radius analysis, causal attribution, and cross-service impact modeling. ## MCP Server Use your favorite agent with our [MCP Server](/agent-integration/mcp-server/) to query your system's behavior using natural language. Ask questions like: - "Why is service X slow?" - "What changed in the last hour?" - "Show me the dependencies for service Y" - "What's causing the latency in my API?" Causely uses its causal model and topology graph to provide intelligent answers about your system's behavior, helping you understand not just what happened, but why it happened. ## Improve Understanding To gain deeper insights into your system: - **Connect more telemetry**: Add additional [telemetry sources](/telemetry-sources) to provide more comprehensive coverage. More telemetry means a more comprehensive topology graph and better causal insights. - **Configure scopes**: Set up [scopes](/configuration/scopes) to organize your entities and focus on specific parts of your system. - **Use MCP Server regularly**: Regularly query your system to build familiarity with how it behaves and to identify patterns and relationships. ## Related Features - [Topology](/in-action/topology) - Browse system entities and relationships - [MCP Server](/agent-integration/mcp-server/) - Query system behavior - [How Causely Works](/getting-started/how-causely-works) - Learn about the causal model --- ## Automate Remediation # Automate Remediation for Resource Contention Causely allows you to automatically remediate **resource contention** issues directly from the UI, helping you restore performance faster, reduce time to resolve, and keep services within SLOs. When Causely identifies a deterministic Resource Contention Diagnosis, you can trigger automated remediation or apply a guided fix with one click. ## Supported Root Causes Causely supports automated remediation for the following resource-related root causes: ### [CPU Congested](/reference/root-causes/infrastructure/#cpu-congested) Automatically adjust CPU limits when services are experiencing CPU saturation. ### [Frequent Memory Failure](/reference/root-causes/infrastructure/#frequent-memory-failure) Resolve persistent out-of-memory issues caused by memory leaks or inefficient usage. ### [Memory Failure](/reference/root-causes/infrastructure/#memory-failure) Increase memory allocations to resolve out-of-memory issues. ### [Ephemeral Storage Noisy Neighbor](/reference/root-causes/infrastructure/#ephemeral-storage-noisy-neighbor) Isolate and manage containers that excessively consume ephemeral storage, impacting node stability. ### [Memory Noisy Neighbor](/reference/root-causes/infrastructure/#memory-noisy-neighbor) Isolate and manage containers that excessively consume memory, impacting node stability. ### [Congested Services](/reference/root-causes/services/#congested) Scale service resources to handle increased load. ## What Causely Changes Automatically When remediation is executed for a supported Resource Contention Diagnosis, Causely applies a deterministic scaling action based on the type of bottleneck identified: 1. **Vertical Scaling (+50%)**: Increases the affected container’s CPU or memory **requests and limits by 50%**. 2. **Horizontal Scaling (+1 Replica)**: Adds **one additional replica** to the deployment to immediately increase capacity. These adjustments are purposefully conservative and are only applied when Causely’s causal reasoning model confirms that scaling, rather than a correlated Signal, is the correct fix. ## Enabling Automated Remediation (Executor Required) Automated actions require the **executor** to be enabled on the mediator running in the cluster where you want remediation performed. To enable the executor, update your mediator’s `causely-values.yaml`: ```yaml executor: enabled: true ``` Once updated, redeploy or upgrade your mediator so it loads the new configuration. See [using custom values file](/installation/customize/?install-method=helm#use-a-custom-values-file) for details on applying updated values. ## Using the Remediate Now Interface In the UI, supported Diagnoses include a **Remediate now** option that provides: - An acknowledgment step showing the impacted deployment - Auditable action history tied to the entity for which value were updated If you prefer to apply the change manually, the Remediation section for the Diagnosis includes YAML examples you can use. ## Aligning Configuration with MCP Server These remediation updates are applied at runtime, and the MCP Server provides a way to commit these configuration adjustments into your codebase for long‑term consistency. If you want to standardize or persist updated sizing, you can manage configuration through the [MCP Server](/agent-integration/mcp-server/). --- ## Feature Demos This page contains links to Causely feature demos designed to help you understand what you can do and what to expect with Causely. Watch the Full Playlist export const videos = [ { id: 'GiXq71HEGwE', title: 'Causely Feature Demo: Reliability Delta', }, { id: 'nN4Iy5BuC3c', title: 'Solving Slow Database Queries with Causely and its MCP Server', }, { id: 'L-nWJr4tZ7U', title: 'Causely’s MCP Server Brings Reliability into Your IDE | Helm Chart Example', }, { id: 'sk_KmMOF1lE', title: 'Causely Feature Demo: From Root Cause to Business Impact with Causely and ClickStack by ClickHouse', }, { id: 'p07c2gy3baM', title: 'Causely Feature Demo: Accelerate Incident Response with Causely + incident.io', }, { id: 'tT0Ju5vO97w', title: 'Causely Feature Demo: Address External API Slowdowns', }, { id: 'xah1-eSqO4A', title: 'Causely Feature Demo: Solve the Root Cause of Message Queue Lag', }, { id: 'hvJDWHkxieg', title: 'Causely Feature Demo: Pinpointing the Code Change Causing Performance Issues', }, { id: 'D6Ps1VoGHvw', title: 'Causely Feature Demo: Unlock Root Cause Analysis in Grafana', }, ]; ## Latest demo ## Last videos {videos.map((v) => ( ))} --- ## Causely In Action This section shows how the Causely system surfaces what matters, explains why issues happen, and helps you move from detection to resolution. If you are new, the recommended path is: start with the [Welcome Page](/in-action/welcome-page), skim [Topology](/in-action/topology) to learn how entities are organized, then dive into [Issues](/in-action/root-causes) to investigate real problems. :::tip Causely is built to seamlessly integrate with your existing SRE tools and workflows: it replaces the tools you no longer need for reactive incident response, and plugs into your workflows to enable prediction and prevention. Therefore the UI is only one way to interact with Causely: we recommend that you use [agent](/agent-integration) or [workflow](/workflows) integrations, to have Causely available where you already work. ::: ### Welcome Page Start your day with a concise overview. The page highlights whether there is an urgent Issue that needs attention or confirms that there are no urgent items (with a link to review non‑urgent ones). It provides a set of insights from the last 24 hours: the number of [Diagnoses](/reference/root-causes/) and [Signals](/reference/symptoms/) observed (and how many are active), plus the Noise Reduction achieved. See the full page at [Welcome Page](/in-action/welcome-page). ### Topology Browse entities discovered by the engine: services, workloads, containers, HTTP paths, background operations, data stores, and more. Filter by scope, severity, and type, and use search to quickly find what you need. Inspect any entity through focused tabs including Summary, [Diagnoses](/reference/root-causes/), [Signals](/reference/symptoms/), Metrics, Service map, Infrastructure stack, Dataflow map, and Labels. Learn more at [Topology](/in-action/topology). ### Issues See the list of Issues and their inferred [Diagnoses](/reference/root-causes/) that explain observed [Signals](/reference/symptoms/) across your environment. Use urgency and state filters to focus, then open an Issue to review the Summary and the Causality graph with evidence, impact, and remediation guidance. Details at [Issues](/in-action/root-causes). ### Automate Remediation Automate remediation for resource contention issues such as CPU congestion, memory failures, and noisy-neighbor conditions. Use it to quickly restore service performance with safe scaling actions directly from the Causely UI. Learn more at [Automate Remediation](/in-action/automate-remediation). ### Reliability Delta Compare two snapshots of your environment to see whether reliability has improved or regressed. This turns subjective judgment into clear, objective reliability signals. Use it to validate release candidates, analyze load tests, and detect reliability drift over time. Learn more at [Reliability Delta](/in-action/reliability-delta). ### Feature Demos Short demonstrations that highlight key capabilities and investigative workflows. Useful for onboarding, sharing with teammates, or refreshing best practices. Explore at [Feature Demos](/in-action/feature-demos). --- ## Reliability Delta The **Reliability Delta** capability allows you to compare two snapshots of your environment to determine whether reliability has improved, regressed, or meaningfully changed. This is especially powerful when evaluating **release candidates**, validating **load test** results, or confirming the **stability** of ongoing changes in production. A **snapshot** captures the state and behavior of your system, including its services, dependencies, and underlying infrastructure, over a defined time window (up to two continuous hours). By comparing any two snapshots, Causely surfaces changes in reliability, service behavior, Diagnoses, and resource utilization, enabling you to definitively assess whether a new version or configuration is safer, riskier, or equivalent. ## When to Use Reliability Delta ### Release validation Compare a release candidate against a stable version to detect regressions in latency, error rate, throughput, or resource usage before deployment. ### Load testing and performance analysis Capture snapshots during load tests and compare them to production baselines to identify bottlenecks, capacity issues, or emerging Diagnoses. ### Continuous environment drift detection Review snapshots taken at different points in time to understand whether reliability is trending up or down, even outside of formal releases. ## How Snapshots Work A **snapshot** records: - Service behavior (latency, request rate, error rate, throughput) - Dependency and service counts - Detected Diagnoses (urgent and non-urgent) - Infrastructure behavior and resource usage (CPU, memory, network I/O) - Version, schema, and metadata you provide - Duration and time window of captured activity Snapshots can be created in two ways: 1. **Last 2 hours**: automatically captures data from the most recent 120 minutes 2. **Custom time window**: select a start date and time for a snapshot (up to 2 hours). The start time must be within the last 6 hours. Snapshots appear in the **Snapshots** panel and can be selected for comparison in the **Comparisons** panel. ## Comparing Snapshots The **Comparisons** view enables you to evaluate changes across two snapshots. You can also apply an optional scope to focus the comparison on a specific set of services or infrastructure. - **Baseline Snapshot**: the version or time period you trust - **Candidate Snapshot**: the release, change, or window you want to evaluate Causely computes the reliability delta across: ### Diagnoses - New Diagnoses introduced in the candidate window - Cleared Diagnoses no longer present - Classification by urgent vs. non-urgent ### Service Behavior For each metric, Causely shows: - Before value - After value - Absolute change - Percent change Metrics include: - Service count - Total requests - Request rate - Average request duration - Average error rate ### Resource Utilization - Entity counts - CPU utilization (avg & max) - Memory utilization (avg & max) - Network throughput (receive & transmit) ## Interpreting Results ### 🟢 Proceed with release > No concerning changes detected. The comparison shows the candidate snapshot introduces **no new urgent Diagnoses**, indicating the version is safe to proceed with. ### 🔴 Test snapshot rejected > New urgent Diagnoses detected that require investigation before deploying. This indicates the candidate snapshot **introduced one or more urgent Diagnoses**, which may reflect: - Newly degraded or failing services - Significant regressions in error rate, latency, or throughput - Resource or dependency behavior signaling instability ## UI Walkthrough ### Snapshots Panel View and manage stored snapshots. Selecting a snapshot displays its metadata, duration, and captured behavior. ### Comparisons Panel Choose any two snapshots to view detailed reliability deltas across Diagnoses, services, and resources. ### Snapshot Creation Dialog - Add a **name**, **version**, and **description** - Choose **last 2 hours** or a **custom time window** - Click **Create** to capture the snapshot ### Comparison View Displays: - **Snapshot Summary**: time window, metadata, version info - **Diagnoses Summary**: new vs. cleared causes - **Service Summary**: metric-level comparisons - **Resource Summary**: infrastructure-level deltas ## Example Workflows ### Validating a release candidate 1. Capture a snapshot of the last stable version 2. Deploy the release candidate 3. Capture a new snapshot 4. Compare the two snapshots 5. Validate that no new urgent Diagnoses or degradations are introduced ### Load test regression analysis 1. Create a snapshot of baseline load test for a stable version 2. Run the load test on a new version and capture a snapshot during it 3. Compare the snapshots to identify scalability issues or bottlenecks ### Reliability drift detection 1. Periodically capture snapshots (daily/weekly) 2. Compare snapshots to detect slow degradation or improvements 3. Use deltas to preemptively address emerging risks ## Why Reliability Delta Matters Engineers must ship quickly **without compromising reliability**. Reliability Delta provides a deterministic, data-driven method for evaluating the impact of change. It delivers: - A clear comparison of behavioral changes between versions - Root-cause-aware analysis powered by Causely’s causal model - A consistent framework for release evaluation and change management - Execution confidence even under rapid change and high-visibility releases It transforms subjective judgment into objective reliability signals. ## Feature Demo --- ## Topology The Topology view presents the entities that the causal reasoning engine has discovered across your environment, ranging from services and workloads to containers, HTTP paths, background operations, data stores, and more. It provides an interactive way to browse, filter, and drill into any entity to understand its health, relationships, and impact. For background on how entities are modeled and connected, see [How Causely Works](/getting-started/how-causely-works). ## Topology view The Topology view shows the catalog of detected entities. You can navigate by groupings (for example services or infrastructure), select an entity to open its details, and pivot to related entities as needed. ### Filter and search Use filters and search to focus on what matters most: - **Scope**: Limit the visible entities to a specific scope (for example an environment or domain). - **Severity**: Show entities with a particular severity state first, helping you triage quickly. - **Type**: Narrow the list to a type such as service, container, HTTP path, background operation, data store, or queue. - **Search**: Find entities by name, label, or identifier. Search is available across scopes and types when needed. ## Inspect a single entity Opening an entity shows a details panel organized in multiple tabs: ### Summary The Summary tab provides a concise health overview for the selected entity. It highlights SLO status and recent changes, key metrics at a glance, and the most relevant related entities (upstream, downstream, and infrastructure neighbors). Recent logs and events are surfaced inline to provide quick evidence you can drill into. ### Diagnoses This tab lists [Diagnoses](/reference/root-causes/) currently affecting this entity and those that are plausible given observed conditions. Each Diagnosis includes a title and severity. Click on a Diagnosis to see which Signals it explains for this entity. This information is available even if the Diagnosis is not currently active. This is useful to understand the potential impact of the Diagnosis and to validate the impact of the Diagnosis when it is active. ### Signals See [Signals](/reference/symptoms/) detected on the entity or propagated from dependencies. Entries show title and severity. Click on a Signal to see the Diagnoses that could lead to this Signal. This information is available even if the Signal is not currently active. This is useful to understand the potential impact of the Signal and to validate the impact of the Signal when it is active. ### Service map Visualizes the entity’s service neighborhood, including inbound dependencies and downstream consumers. Edges indicate direction and relative traffic; nodes reflect health at the selected moment. ### Infrastructure stack Shows where the entity runs within the stack, for example cluster, node, workload, container, and process. It surfaces capacity and saturation indicators (CPU, memory, disk, and network) and relevant runtime metadata to understand placement and potential resource contention. ### Dataflow map Displays upstream and downstream data dependencies such as topics, queues, databases, and batch pipelines. Use it to trace how data moves through the system, identify potential bottlenecks or back‑pressure, and assess which consumers are at risk when this entity degrades. ### Labels Lists labels and metadata attached to the entity (team, scope, environment, version, and custom tags). Labels power search and filtering, drive policy and scoping, and help organize entities. You can copy label values and use them to filter the Topology view or jump to related entities. --- ## Welcome Page The Welcome page gives you a concise, actionable overview powered by the Causely engine: what requires attention right now, how to ask targeted questions, and how your environment has behaved recently. ## Welcome page ### Urgent Issue status At the top, the page highlights whether there is an urgent Issue that needs attention. When one exists, you see a compact card showing the Issue title, the affected entity, when it was last observed, the number of linked Signals, and how many services are degraded. Selecting the card opens the full details so you can review evidence and remediation steps. If there are no urgent items, the page clearly states that everything is stable and offers a link to review non‑urgent Issues. Non‑urgent items are useful for prevention and continuous improvement. ### Causely Insights Insights summarize recent activity to help you spot trends without leaving the page: - [Diagnoses](/reference/root-causes/): The number observed over the last 24 hours, with an indication of how many are currently active versus cleared. - [Signals](/reference/symptoms/): The number observed over the last 24 hours, including how many remain active. - Noise Reduction: The percentage reduction in alert noise provided by the system compared to a baseline, illustrating how much distraction has been eliminated. Each card links to a deeper view, so you can pivot from the high‑level indicators into detailed lists and timelines. --- ## Causely Documentation --- ## Next Steps Visit the [Customize your installation](/installation/customize) page to learn more about customizing your installation. We recommend that you connect additional telemetry sources to help Causely infer Diagnoses in your environment. Visit the [Telemetry Sources](/telemetry-sources/) page to learn more about the data sources that Causely supports. To push insights into your workflows, visit the [Workflow Integrations](/workflows/) page. --- ## Prerequisites Before you begin, make sure your environment meets the following requirements: - [Kubernetes](https://kubernetes.io/releases/) 1.18+ - [Helm](https://github.com/helm/helm/releases) 3.8+ :::info **Persistent Volume Access:** Causely requires the ability to create and attach PersistentVolumeClaims (PVCs) for Mediator and VictoriaMetrics storage. - Ensure your account can create PersistentVolume, PersistentVolumeClaim, and StorageClass resources. - Check that a default StorageClass is configured (kubectl get storageclass). - If dynamic provisioning is disabled, pre-create a PersistentVolume and bind it to the PVC. If you are using OpenShift, you need to use the group id from the uid-range assigned to the project. Review the section on [OpenShift Deployment](/installation/customize#deploying-on-openshift) for more details. If you are using a custom StorageClass, you can specify the StorageClass to use for persistent volumes. Review the section on [Using a custom StorageClass instead of the default one](/installation/customize#using-a-custom-storageclass-instead-of-the-default-one) for more details. ::: --- ## Retrieve Your Access Token 1. Visit https://portal.causely.app and log in. 2. Click `Mediators` on the side navigation. 3. Click the `Add ➕` button and copy your access token to use in the next step. --- ## Review Discovery To verify the agent is successfully installed navigate to the [Integrations Agents view](https://portal.causely.app/integrations?tab=agents). Once the agent has appeared you have successfully installed Causely! Navigate to https://portal.causely.app to verify your environment has been discovered. You should see entities populated in the `Topology` view. As Diagnoses are made, they will appear in the `Root Cause` view. --- ## Argo CD Installation export const CAUSELY_VERSION = versionHistory?.versions?.[0]?.imageVersion ?? versionHistory?.versions?.[0]?.version ?? 'latest'; This installation method enables Causely on [**Kubernetes clusters**](https://kubernetes.io/) using [**Argo CD**](https://argo-cd.readthedocs.io/) for GitOps-based deployment and management. ## Prerequisites - Kubernetes cluster running - [Argo CD](https://argo-cd.readthedocs.io/en/stable/getting_started/) installed and configured - Git repository for storing configuration (if you manage Applications via Git) - Linux kernel with eBPF support (5.4+ recommended) ## 1. Install Causely using Argo CD ### Retrieve Your Access Token 1. Visit https://portal.causely.app and log in. 2. Click `Mediators` on the side navigation. 3. Click the `Add ➕` button and copy your access token to use in the next step. ### Get the Argo CD manifests Download the Argo CD manifests and copy them to your GitOps repository: ```bash # Clone the causely-deploy repository to get the manifests git clone https://github.com/causely-oss/causely-deploy.git cd causely-deploy/kubernetes/argocd/components/applications/causely # Copy the manifests to your GitOps repository cp *.yaml /path/to/your/gitops-repo/causely/ ``` Alternatively, you can copy the files directly from the [causely-deploy repository](https://github.com/causely-oss/causely-deploy/tree/main/kubernetes/argocd/components/applications/causely) into your GitOps repository. The base [`causely.yaml`](https://github.com/causely-oss/causely-deploy/blob/main/kubernetes/argocd/components/applications/causely/causely.yaml) Application deploys the public Causely Helm chart from `us-docker.pkg.dev/public-causely/public` into the `causely` namespace with automated sync, pruning, and self-heal enabled. ### Configure your environment #### Create the Kubernetes Secret for your token For production and GitOps workflows, store your gateway token in a Kubernetes Secret instead of committing it to Git. See [Using a Kubernetes Secret for the Access Token](/installation/customize#using-a-kubernetes-secret-for-the-access-token) for details. ```bash kubectl create namespace causely kubectl create secret generic causely-token \ --from-literal=gateway-token= \ -n causely ``` #### Create an Argo CD AppProject The Application references Argo CD project `causely`. Create an `AppProject` that allows the Helm chart source and the `causely` namespace (and `kube-system` if your chart installs cluster-scoped resources such as PriorityClasses): {`apiVersion: argoproj.io/v1alpha1 kind: AppProject metadata: name: causely namespace: argocd spec: description: Causely agent deployment clusterResourceWhitelist: - group: '*' kind: ClusterRole - group: '*' kind: ClusterRoleBinding - group: scheduling.k8s.io kind: PriorityClass destinations: - namespace: causely server: https://kubernetes.default.svc - namespace: kube-system server: https://kubernetes.default.svc sourceRepos: - us-docker.pkg.dev/public-causely/public `} :::note AppProject name Ensure the `AppProject` metadata name matches `spec.project` in your Application (`causely`). If you use the bundled `causely-project.yaml` from causely-deploy, rename the project or update the Application so both resources use the same project name. ::: #### Configure the Application Update `causely.yaml` for your cluster name, chart version, and token reference. The chart version should match the image tag you want to run: {`apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: causely namespace: argocd spec: project: causely destination: namespace: causely server: https://kubernetes.default.svc source: repoURL: us-docker.pkg.dev/public-causely/public chart: causely targetRevision: "${CAUSELY_VERSION}" helm: releaseName: causely parameters: - name: image.tag value: "${CAUSELY_VERSION}" - name: global.cluster_name value: - name: mediator.secretName value: causely-token ignoreDifferences: - group: apps kind: Deployment jqPathExpressions: - .spec.template.spec.containers[].image - group: apps kind: DaemonSet jqPathExpressions: - .spec.template.spec.containers[].image syncPolicy: automated: prune: true selfHeal: true syncOptions: - CreateNamespace=true - RespectIgnoreDifferences=true `} The `ignoreDifferences` entries tell Argo CD to ignore container image fields on Deployments and DaemonSets. That avoids endless OutOfSync states when image tags are updated outside the Application manifest (for example by the Causely Operator or a release pipeline). You can find the latest chart version in the [Causely Portal](https://portal.causely.app) under **Gear Icon → Integrations → Agents**, or use the version shown above (`${CAUSELY_VERSION}`). For additional Helm values (integrations, notifications, operator settings, and more), add a `values.yaml` next to the Application and reference it with `helm.valueFiles`, as in the [production overlay example](https://github.com/causely-oss/causely-deploy/tree/main/kubernetes/argocd/clusters/production/components/applications/causely) in causely-deploy. ### Deploy via GitOps Commit the manifests to the repository Argo CD watches, or apply them directly if you bootstrap Applications with `kubectl`: ```bash # Navigate to your GitOps repository cd /path/to/your/gitops-repo # Add the Causely configuration files git add causely/ git commit -m "Add Causely Argo CD configuration" git push origin main ``` Monitor the deployment: ```bash # Check Application sync status kubectl get application causely -n argocd # Check if Causely pods are running kubectl get pods -n causely ``` In the Argo CD UI, open the **causely** Application and confirm it is **Synced** and **Healthy**. ### Review Discovery To verify the agent is successfully installed navigate to the [Integrations Agents view](https://portal.causely.app/integrations?tab=agents). Once the agent has appeared you have successfully installed Causely! Navigate to https://portal.causely.app to verify your environment has been discovered. You should see entities populated in the `Topology` view. As Diagnoses are made, they will appear in the `Root Cause` view. ## Next Steps Visit the [Customize your Installation](/installation/customize) page to learn more about customizing your installation. We recommend that you connect additional telemetry sources to help Causely infer Diagnoses in your environment. Visit the [Telemetry Sources](/telemetry-sources/) page to learn more about the data sources that Causely supports. To push insights into your workflows, visit the [Workflow Integrations](/workflows/) page. --- ## CLI Installation ## Prerequisites Before you begin, make sure your environment meets the following requirements: - [Kubernetes](https://kubernetes.io/releases/) 1.18+ - [Helm](https://github.com/helm/helm/releases) 3.8+ :::info **Persistent Volume Access:** Causely requires the ability to create and attach PersistentVolumeClaims (PVCs) for Mediator and VictoriaMetrics storage. - Ensure your account can create PersistentVolume, PersistentVolumeClaim, and StorageClass resources. - Check that a default StorageClass is configured (kubectl get storageclass). - If dynamic provisioning is disabled, pre-create a PersistentVolume and bind it to the PVC. If you are using OpenShift, you need to use the group id from the uid-range assigned to the project. Review the section on [OpenShift Deployment](/installation/customize#deploying-on-openshift) for more details. If you are using a custom StorageClass, you can specify the StorageClass to use for persistent volumes. Review the section on [Using a custom StorageClass instead of the default one](/installation/customize#using-a-custom-storageclass-instead-of-the-default-one) for more details. ::: ## 1. Install the Causely CLI Run the following command to download and install the Causely CLI: ```bash bash -c "$(curl -fksSL https://install.causely.ai/install.sh)" ``` :::info The Causely CLI simply wraps Helm. If you prefer, you can alternatively run Helm commands directly. See [Helm Installation](/installation/helm) ::: ## 2. Install the Causely Agent ### Step 1: Retrieve Your Access Token 1. Visit https://portal.causely.app and log in. 2. Click `Mediators` on the side navigation. 3. Click the `Add ➕` button and copy your access token to use in the next step. ### Step 2: Install via CLI Run the following command by replacing `` with your access token. ```bash causely agent install --token ``` :::note If you have not installed the CLI in your PATH, make sure to run the command from absolute path, for example `~/bin/causely`. This installs Causely agents into the cluster defined by your current `kubectx` context. To specify a custom cluster name, see [CLI Options](#cli-options) below. ::: ### Step 3: Review Discovery To verify the agent is successfully installed navigate to the [Integrations Agents view](https://portal.causely.app/integrations?tab=agents). Once the agent has appeared you have successfully installed Causely! Navigate to https://portal.causely.app to verify your environment has been discovered. You should see entities populated in the `Topology` view. As Diagnoses are made, they will appear in the `Root Cause` view. ## CLI Options You can see a complete list of options for customizing the installation by using the `--help` option. ```bash causely agent --help ``` Here's a list of possible options: ```bash causely agent install --namespace --repository --tag --cluster-name --kube-context --values ``` ## Next Steps Visit the [Customize your installation](/installation/customize) page to learn more about customizing your installation. We recommend that you connect additional telemetry sources to help Causely infer Diagnoses in your environment. Visit the [Telemetry Sources](/telemetry-sources/) page to learn more about the data sources that Causely supports. To push insights into your workflows, visit the [Workflow Integrations](/workflows/) page. --- ## Customize your installation After you have installed the Causely agent, you can customize your installation to meet your needs. ## Use a Custom Values File To specify additional configurations - like a specific image tag, integrations, webhook notifications, etc - you can do this by creating a `causely-values.yaml` file. Below is an example of common values used for installation. ```yaml global: cluster_name: image: tag: # Locate latest version in the Causely Portal > Gear Icon > Integrations > Agents mediator: gateway: token: # Locate your token in the Causely Portal > Gear Icon > Integrations > Details ``` Now add the `--values` parameter to the command you used to install the Causely agent. ```bash helm upgrade --install causely \ --create-namespace oci://us-docker.pkg.dev/public-causely/public/causely \ --version \ --namespace=causely \ --values ``` ```bash causely agent install --token --values ``` ## Using a Kubernetes Secret for the Access Token Instead of passing your access token directly in the Helm values, you can store it in a Kubernetes secret and reference it using `mediator.secretName`. This approach is recommended for production environments and GitOps workflows. ### Step 1: Create the Kubernetes Secret Create a secret containing your Causely access token: ```bash kubectl create secret --namespace causely generic causely-token \ --from-literal=gateway-token= ``` :::note The secret must contain a key named `gateway-token` with your access token value. ::: ### Step 2: Reference the Secret in Your Values File Update your `causely-values.yaml` to reference the secret instead of the token directly: ```yaml mediator: secretName: causely-token # Name of the secret created above ``` When `mediator.secretName` is set, Causely will automatically read the token from the referenced secret instead of expecting it in `mediator.gateway.token`. ## Connect Additional Telemetry Sources You can add additional data sources to Causely which expands Causely's causality map to help infer Diagnoses. Some data sources require you to update your `causely-values.yaml` file. For more details on Telemetry Sources, see [Telemetry Sources](/telemetry-sources/). ## Push Insights into Your Workflows Causely can automatically send identified Diagnoses to your existing notification and observability tools. To explore integrations with Slack, Grafana, Opsgenie, and more, visit our [Workflow Integrations](/workflows/) page. You'll find setup instructions for webhooks (via `causely-values.yaml`) and details on our Grafana plugin, with additional native integrations coming soon. ## Custom Labels for Scope configuration You can [scope the Causely interface](/configuration/scopes/) to specific components out-of-the-box (clusters, namespaces, services). Causely can also use labels to provide users with additional scopes - this can be accomplished by adding the following to your `causely-values.yaml`. ```yaml label_semconv: scopes: geography: - 'app.kubernetes.io/geography' environment: - 'app.kubernetes.io/environment' customer: - 'app.kubernetes.io/customer' team: - 'app.kubernetes.io/team' product: - 'app.kubernetes.io/product' project: - 'app.kubernetes.io/project' service: - 'app.kubernetes.io/service' ``` When configured, Causely will automatically detect entities with these labels and make them available as scope options in the UI. ## Sizing the Mediator for large environments The Mediator defaults to a memory limit of 16Gi. Mediator memory scales with the **number of entities under management**: the more services, pods, and infrastructure components in your environment, the more memory the Mediator requires. It does not scale with trace or metric throughput. If the Mediator pod is OOMKilled or you are running a large cluster, increase the memory limits in your `causely-values.yaml`: ```yaml mediator: resources: limits: memory: "32Gi" ``` Apply the change with: ```bash helm upgrade causely \ oci://us-docker.pkg.dev/public-causely/public/causely \ --namespace=causely \ --values ``` :::tip If the Mediator is dropping traces due to memory pressure, that is expected and intentional: it means the Mediator already has a statistically sufficient sample for analysis. An OOMKill is different: it means the Mediator process itself ran out of memory and needs more headroom. Check which situation you are in by inspecting pod events: ```bash kubectl describe pod -n causely -l app=mediator ``` ::: ## Sizing VictoriaMetrics for high-volume environments VictoriaMetrics defaults to a memory limit of 8Gi, which is sufficient for most environments. In high-volume environments, such as large clusters, high service counts, or high cardinality metrics, VictoriaMetrics may hit that ceiling, causing **gaps in metrics graphs** within Causely. If you observe missing or inconsistent metrics, increase the VictoriaMetrics memory limits in your `causely-values.yaml`: ```yaml victoriametrics: resources: limits: memory: "16Gi" ``` Apply the change with: ```bash helm upgrade causely \ oci://us-docker.pkg.dev/public-causely/public/causely \ --namespace=causely \ --values ``` :::note Metrics visible in Causely may not exactly match totals in external monitoring tools even when VictoriaMetrics is healthy. The Mediator processes a statistically significant sample of telemetry by design. See [Mediator scale behavior](/getting-started/architecture#mediator) for details. ::: ## Using an external time series database By default, Causely deploys VictoriaMetrics in-cluster to store time series data. If you already operate a Prometheus-compatible store (such as Prometheus, Thanos, Mimir, Cortex, or VictoriaMetrics), you can point Causely at it instead of running the bundled VictoriaMetrics instance. The external store must support: - **Remote write** for ingesting metrics that Causely collects. - A **PromQL query API** for Causely to read metrics back. To use an external store, set `timeseries.external.enabled=true` in your `causely-values.yaml`. The bundled VictoriaMetrics is automatically skipped when external storage is enabled, so you do not need to disable it explicitly. ```yaml timeseries: external: enabled: true # Prometheus API base URL used for both PromQL queries and remote write. endpoint: "http://prometheus-operated.monitoring.svc.cluster.local:9090" ``` When only `endpoint` is set, Causely uses it for PromQL queries and derives the remote write URL as `{endpoint}/api/v1/write`. If your store exposes the query and remote write APIs at different URLs, set them explicitly: ```yaml timeseries: external: enabled: true endpoint: "http://prometheus.monitoring.svc.cluster.local:9090" # Override the PromQL query API base URL (defaults to endpoint when empty). queryEndpoint: "http://thanos-query.monitoring.svc.cluster.local:9090" # Override the remote write URL (defaults to "{endpoint}/api/v1/write" when empty). remoteWriteEndpoint: "http://thanos-receive.monitoring.svc.cluster.local:19291/api/v1/receive" ``` ### Authentication If your external store requires HTTP Basic Auth, create a Kubernetes secret with `username` and `password` keys and reference it with `timeseries.external.secretName`: ```bash kubectl create secret --namespace causely generic timeseries-credentials \ --from-literal=username= \ --from-literal=password= ``` ```yaml timeseries: external: enabled: true endpoint: "http://prometheus-operated.monitoring.svc.cluster.local:9090" secretName: timeseries-credentials ``` ::: Apply the change with: ```bash helm upgrade causely \ oci://us-docker.pkg.dev/public-causely/public/causely \ --namespace=causely \ --values ``` ## Installing in a large Kubernetes cluster (>1000 nodes) If you install Causely in a large Kubernetes cluster, you can enable centralized caching of the api server responses by adding to the `causely-values.yaml` file: ``` k8sCache: enabled: true ``` ## Connecting to a remote Kubernetes cluster If you are connecting to a remote Kubernetes API server, you need to create a secret with the kubeconfig credentials: ```bash kubectl create secret --namespace causely generic kubeconfig --from-file=kubeconfig=path_to_kubeconfig ``` and include in the `causely-values.yaml` file: ``` scrapers: kubernetes: # remote k8s cluster credentials secretName: kubeconfig agent: enabled: false ``` ## Deploying on Openshift If you are deploying on Openshift, you need to use the group id from the uid-range assigned to the project: ```bash oc new-project causely oc get ns causely -o yaml|grep uid-range openshift.io/sa.scc.uid-range: 1000630000/10000 ``` and include in the `causely-values.yaml` file: ``` global: securityContext: fsGroup: 1000630000 ``` Or you can change the security context of the project to the 'anyuid' SCC ```bash oc adm policy add-scc-to-group anyuid system:serviceaccounts:causely ``` In both cases, you need to assign the `privileged` SCC to the `causely-agent` service account used by the Causely agents: ```bash oc adm policy add-scc-to-user privileged -z causely-agent -n causely ``` ## Using a custom StorageClass instead of the default one If you are deploying into a cluster, where there is no default StorageClass defined, you can specify the StorageClass to use for persistent volumes: ``` global: storageClass: ocs-storagecluster-ceph-rbd ``` Alternatively you can annotate a default StorageClass: ``` kubectl patch storageclass ocs-storagecluster-ceph-rbd -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}' ``` --- ## Standalone Docker Host Installation This installation method extends Causely beyond Kubernetes clusters to **standalone Docker hosts**, enabling telemetry collection and root‑cause analysis for services running outside Kubernetes. ## Prerequisites - Docker Engine installed on the host - Linux kernel with eBPF support (5.4+ recommended) - Ability to run privileged containers and use `pid: host` ## 1. Install the Causely Agent on a Docker Host ### Step 1: Retrieve Your Access Token 1. Visit https://portal.causely.app and log in. 2. Click `Mediators` on the side navigation. 3. Click the `Add ➕` button and copy your access token to use in the next step. ### Step 2: Download the Docker bundle via `install.sh` ```bash curl -fsSL https://raw.githubusercontent.com/causely-oss/causely-deploy/main/docker/install.sh | bash ``` This script creates a `causely-docker/` directory, downloads `docker-compose.yaml`, config files, and a `.env` file. ### Step 3: Edit environment variables Set the following environment variables in your `.env` file. Note that `` is the token retrieved in step 1 and `` is the friendly name you'd like displayed in the Causely UI. ```env CAUSELY_GATEWAY_TOKEN= DOCKER_HOST_NAME= ``` ### Step 4: Start the stack Run the following command to start the Causely stack: ```bash docker compose up -d ``` ### Step 5: Review Discovery To verify the agent is successfully installed navigate to the [Integrations Agents view](https://portal.causely.app/integrations?tab=agents). Once the agent has appeared you have successfully installed Causely! Navigate to https://portal.causely.app to verify your environment has been discovered. You should see entities populated in the `Topology` view. As Diagnoses are made, they will appear in the `Root Cause` view. ## Next Steps Visit the [Customize your Installation](/installation/customize) page to learn more about customizing your installation. We recommend that you connect additional telemetry sources to help Causely infer Diagnoses in your environment. Visit the [Telemetry Sources](/telemetry-sources/) page to learn more about the data sources that Causely supports. To push insights into your workflows, visit the [Workflow Integrations](/workflows/) page. --- ## FluxCD Installation export const CAUSELY_VERSION = versionHistory?.versions?.[0]?.imageVersion ?? versionHistory?.versions?.[0]?.version ?? 'latest'; This installation method enables Causely on [**Kubernetes clusters**](https://kubernetes.io/) using [**FluxCD**](https://fluxcd.io/) for GitOps-based deployment and management. ## Prerequisites - Kubernetes cluster running - FluxCD installed and configured - Git repository for storing configuration - Linux kernel with eBPF support (5.4+ recommended) ## 1. Install Causely using FluxCD ### Retrieve Your Access Token 1. Visit https://portal.causely.app and log in. 2. Click `Mediators` on the side navigation. 3. Click the `Add ➕` button and copy your access token to use in the next step. ### Get the FluxCD manifests Download the FluxCD manifests and copy them to your GitOps repository: ```bash # Clone the causely-deploy repository to get the manifests git clone https://github.com/causely-oss/causely-deploy.git cd causely-deploy/kubernetes/fluxcd/causely # Copy the manifests to your GitOps repository cp *.yaml /path/to/your/gitops-repo/causely/ ``` Alternatively, you can copy the files directly from the [causely-deploy repository](https://github.com/causely-oss/causely-deploy/tree/main/kubernetes/fluxcd/causely) into your GitOps repository. ### Configure your environment #### Create the Kubernetes Secret for your token For production and GitOps workflows, store your gateway token in a Kubernetes Secret instead of committing it to Git. See [Using a Kubernetes Secret for the Access Token](/installation/customize#using-a-kubernetes-secret-for-the-access-token) for setup instructions. #### Optionally, update the configuration files Edit the file `kustomization.yaml` to match your environment. You can optionally update the version to your desired version: {`apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - namespace.yaml - release.yaml - repository.yaml postBuild: substitute: CAUSELY_VERSION: "${CAUSELY_VERSION}" `} ### Deploy via GitOps Navigate to your GitOps repository and commit the changes: ```bash # Navigate to your GitOps repository cd /path/to/your/gitops-repo # Add the Causely configuration files git add causely/ git commit -m "Add Causely FluxCD configuration" git push origin main ``` FluxCD will automatically detect the changes and deploy Causely to your cluster. You can monitor the deployment status: ```bash # Check FluxCD reconciliation status kubectl get helmreleases -n causely # Check if Causely pods are running kubectl get pods -n causely ``` ### Review Discovery To verify the agent is successfully installed navigate to the [Integrations Agents view](https://portal.causely.app/integrations?tab=agents). Once the agent has appeared you have successfully installed Causely! Navigate to https://portal.causely.app to verify your environment has been discovered. You should see entities populated in the `Topology` view. As Diagnoses are made, they will appear in the `Root Cause` view. ## Next Steps Visit the [Customize your Installation](/installation/customize) page to learn more about customizing your installation. We recommend that you connect additional telemetry sources to help Causely infer Diagnoses in your environment. Visit the [Telemetry Sources](/telemetry-sources/) page to learn more about the data sources that Causely supports. To push insights into your workflows, visit the [Workflow Integrations](/workflows/) page. --- ## Helm Installation export const CAUSELY_VERSION = versionHistory?.versions?.[0]?.imageVersion ?? versionHistory?.versions?.[0]?.version ?? ''; ## Prerequisites Before you begin, make sure your environment meets the following requirements: - [Kubernetes](https://kubernetes.io/releases/) 1.18+ - [Helm](https://github.com/helm/helm/releases) 3.8+ :::info **Persistent Volume Access:** Causely requires the ability to create and attach PersistentVolumeClaims (PVCs) for Mediator and VictoriaMetrics storage. - Ensure your account can create PersistentVolume, PersistentVolumeClaim, and StorageClass resources. - Check that a default StorageClass is configured (kubectl get storageclass). - If dynamic provisioning is disabled, pre-create a PersistentVolume and bind it to the PVC. If you are using OpenShift, you need to use the group id from the uid-range assigned to the project. Review the section on [OpenShift Deployment](/installation/customize#deploying-on-openshift) for more details. If you are using a custom StorageClass, you can specify the StorageClass to use for persistent volumes. Review the section on [Using a custom StorageClass instead of the default one](/installation/customize#using-a-custom-storageclass-instead-of-the-default-one) for more details. ::: ## 1. Install the Causely Agent ### Step 1: Retrieve Your Access Token 1. Visit https://portal.causely.app and log in. 2. Click `Mediators` on the side navigation. 3. Click the `Add ➕` button and copy your access token to use in the next step. ### Step 2: Install via Helm Run the following command by replacing `` with your access token.\ You can find the latest version in the [Causely Portal](https://portal.causely.app) > Gear Icon > Integrations > Agents. {` export CAUSELY_TOKEN= export CAUSELY_CLUSTER_NAME= export CAUSELY_VERSION=${CAUSELY_VERSION} helm upgrade --install causely \\ --create-namespace oci://us-docker.pkg.dev/public-causely/public/causely \\ --version "${CAUSELY_VERSION}" \\ --namespace=causely \\ --set image.tag="${CAUSELY_VERSION}" \\ --set global.cluster_name="\${CAUSELY_CLUSTER_NAME}" \\ --set mediator.gateway.token="\${CAUSELY_TOKEN}" `} ### Step 3: Wait for the Causely Agent to be Ready Run the following command to wait for the Causely agent to be ready: ```bash kubectl wait --for=condition=Ready pod -l app.kubernetes.io/part-of=causely -n causely --timeout=300s ``` :::info If all pods are ready, but you still cannot see any entities in the Causely UI, please check the logs of the mediator pod for errors: ```bash kubectl logs -n causely \ -l app.kubernetes.io/name=mediator \ -c mediator \ --tail=-1 | grep ERROR ``` For example the following error indicates that the credentials are invalid: ```json { "error": "failed to send update: rpc error: code = Unknown desc = unauthorized" } ``` For other errors, please reach out to us at [support@causely.app](mailto:support@causely.app) for help. ::: ### Step 4: Review Discovery To verify the agent is successfully installed navigate to the [Integrations Agents view](https://portal.causely.app/integrations?tab=agents). Once the agent has appeared you have successfully installed Causely! Navigate to https://portal.causely.app to verify your environment has been discovered. You should see entities populated in the `Topology` view. As Diagnoses are made, they will appear in the `Root Cause` view. ## Uninstall the Causely Agent Run the following command to uninstall the Causely agent: ```bash helm uninstall causely -n causely ``` After uninstalling the Causely agent, you can delete the PVCs created by the Causely agent: ```bash kubectl delete pvc -n causely -l app=victoriametrics ``` ## Next Steps Visit the [Customize your installation](/installation/customize) page to learn more about customizing your installation. We recommend that you connect additional telemetry sources to help Causely infer Diagnoses in your environment. Visit the [Telemetry Sources](/telemetry-sources/) page to learn more about the data sources that Causely supports. To push insights into your workflows, visit the [Workflow Integrations](/workflows/) page. --- ## Network Allowlisting This page outlines the URLs that need to be allowlisted in your firewall to enable communication between your Causely deployment and the Causely SaaS platform. ## Required URLs ### For Mediator Component The mediator component requires access to the following URLs to communicate with the Causely SaaS platform: | URL | Protocol | IP Endpoint & Port | Purpose | | -------------------------- | -------- | -------------------- | ---------------------------------------- | | `https://gw.causely.app` | gRPC | `35.186.230.176:443` | Gateway for mediator communication | | `https://otlp.causely.app` | gRPC | `35.186.230.176:443` | OTLP endpoint for mediator communication | ### For End Users The end user interfaces (browser, API clients, etc.) require access to the following URLs to communicate with the Causely SaaS platform: | URL | Purpose | | ---------------------------- | ------------------------------ | | `https://portal.causely.app` | Causely Portal (web interface) | | `https://auth.causely.app` | Authentication | | `https://api.causely.app` | API access | --- ## Nomad Installation This installation method enables Causely on [**HashiCorp Nomad clusters**](https://www.nomadproject.io/), providing telemetry collection and root‑cause analysis for services running on Nomad. ## Prerequisites - HashiCorp Nomad cluster running - Consul cluster for service discovery - NFS server for persistent storage - Docker driver enabled in Nomad - Linux kernel with eBPF support (5.4+ recommended) ## 1. Install Causely on Nomad ### Step 1: Retrieve Your Access Token 1. Visit https://portal.causely.app and log in. 2. Click `Mediators` on the side navigation. 3. Click the `Add ➕` button and copy your access token to use in the next step. ### Step 2: Download the Nomad bundle ```bash git clone https://github.com/causely-oss/causely-deploy.git cd causely-deploy/nomad ``` This directory contains the Nomad job specifications and configuration files needed to run Causely on Nomad. ### Step 3: Configure your environment Set your Causely gateway token as an environment variable: ```bash export CAUSELY_GATEWAY_TOKEN="your-gateway-token-here" ``` Edit the configuration files to match your environment. You'll need to update the datacenters, NFS settings, and Causely specific variables: ```hcl name="executor.hcl" datacenters = ["dc1"] # Update to match your Nomad datacenters ``` ```hcl name="mediator.hcl" # Token can be provided via environment variable $CAUSELY_GATEWAY_TOKEN # If not set, you can specify it here: # gateway_token = "your-gateway-token-here" nfs_server = "nfs.example.com" # Update to your NFS server nfs_path = "/exported/path" # Update to your NFS path ``` ```hcl name="victoriametrics.hcl" datacenters = ["dc1"] # Update to match your Nomad datacenters nfs_server = "nfs.example.com" # Update to your NFS server nfs_path = "/exported/path" # Update to your NFS path ``` ### Step 4: Deploy the jobs Deploy the Causely jobs to your Nomad cluster: ```bash # Deploy VictoriaMetrics first nomad job run -var-file=victoriametrics.hcl victoriametrics.nomad # Deploy the mediator nomad job run -var-file=mediator.hcl mediator.nomad # Deploy the executor nomad job run -var-file=executor.hcl executor.nomad ``` ### Step 5: Review Discovery To verify the agent is successfully installed navigate to the [Integrations Agents view](https://portal.causely.app/integrations?tab=agents). Once the agent has appeared you have successfully installed Causely! Navigate to https://portal.causely.app to verify your environment has been discovered. You should see entities populated in the `Topology` view. As Diagnoses are made, they will appear in the `Root Cause` view. ## Next Steps Visit the [Customize your Installation](/installation/customize) page to learn more about customizing your installation. We recommend that you connect additional telemetry sources to help Causely infer Diagnoses in your environment. Visit the [Telemetry Sources](/telemetry-sources/) page to learn more about the data sources that Causely supports. To push insights into your workflows, visit the [Workflow Integrations](/workflows/) page. --- ## Causely Operator The Causely Operator manages version updates for Causely components running in your Kubernetes cluster. It periodically checks for version updates from the Causely backend and, when a new version is available, presents an update button in the UI. You must click the button to apply the update. ## Overview The operator runs as a Kubernetes deployment and performs the following tasks: - **Version Synchronization**: Polls the Causely Gateway every 60 seconds to check for version updates - **Update Notifications**: Surfaces an update button in the UI when a new version is available, you click it to apply the update to the `mediator` Deployment and `agent` DaemonSet - **Zero Downtime**: Uses Kubernetes rolling updates to minimize service disruption ## When to Use the Operator The operator is useful in scenarios where: - You prefer centralized version control from the Causely backend - You want a guided, one-click update experience rather than managing Helm upgrades manually :::note The operator is disabled by default. Most users manage versions through Helm upgrades or GitOps workflows. ::: ## Enabling the Operator ### Using Helm Values Add the following to your `causely-values.yaml`: ```yaml operator: enabled: true ``` ### Using --set Flag You can also enable the operator directly via the `--set` flag: ```bash helm upgrade causely \ oci://us-docker.pkg.dev/public-causely/public/causely \ --namespace=causely \ --set operator.enabled=true ``` ## Verifying the Operator After enabling, verify the operator is running: ```bash kubectl get deployment operator -n causely kubectl get pods -n causely -l app=causely-operator ``` ## How It Works 1. **Polling**: The operator periodically connects to the Causely Gateway via gRPC 2. **Version Check**: It retrieves the target version configured for your repository 3. **Update Available**: If the target version differs from the current deployment, an update button appears in the Causely UI. Clicking it triggers the update for: - The `mediator` Deployment (mediator and ML containers) - The `agent` DaemonSet (agent container) 4. **Rolling Update**: Kubernetes handles the rolling update process automatically ## Troubleshooting ### Operator Pod Not Starting Check if the mediator ConfigMap exists: ```bash kubectl get configmap mediator -n causely ``` The operator shares configuration with the mediator. Ensure the mediator is properly installed first. ### "Version is empty" in Logs This indicates no target version is configured in the Causely backend. The operator will continue polling until a version is set. ### Permission Errors The operator requires permissions to update Deployments and DaemonSets. Verify the RBAC resources are created: ```bash kubectl get clusterrole causely-operator kubectl get clusterrolebinding causely-operator ``` ## Disabling the Operator To disable the operator, set `operator.enabled: false` in your values file and upgrade: ```bash helm upgrade causely \ oci://us-docker.pkg.dev/public-causely/public/causely \ --namespace=causely \ --set operator.enabled=false ``` This removes the operator Deployment but does not affect the mediator or agent versions. --- ## Installation Overview Causely agents are lightweight components that can be installed on your servers in minutes to provide a real-time Diagnosis. This page covers what gets deployed when you install Causely, the key benefits of its distributed architecture, and how to get started with the installation. Causely works alongside your existing observability tools, processing telemetry data locally while keeping your raw data secure in your datacenter. The platform automatically discovers service dependencies, builds topology maps, and infers Diagnoses without requiring you to send terabytes of telemetry data to the cloud, resulting in dramatically lower costs and stronger privacy compared to traditional observability platforms. ## Installation Options Choose the installation method that best fits your infrastructure and deployment preferences: :::note Telemetry Sources After deployment, telemetry data can be ingested from any source. Causely's mediator listens for [OpenTelemetry Protocol (OTLP)](/telemetry-sources/opentelemetry/) data on port 4317, making it easy to send data from existing OpenTelemetry Collectors or any OTLP-compatible source. ::: ## What Gets Deployed When you install Causely, the following components are deployed in your infrastructure: - [Mediator](#mediator) - [eBPF Instrumentation](#ebpf-instrumentation) - [Executor](#executor) - [VictoriaMetrics](#victoriametrics) To learn more about how causely works, see [How Causely Works](/getting-started/how-causely-works). The relevant documentation for your security considerations can be found in the [Security](/security/) section. ### Mediator The core component that runs locally in your environment and serves as the data processing layer: - **Signal Detection**: Converts telemetry from various sources into binary signal states - **Topology Discovery**: Automatically discovers services, infrastructure, and dependencies - **Local Processing**: Keeps all raw telemetry local, no raw data leaves your datacenter - **OTLP Endpoint**: Listens on port 4317 for OpenTelemetry Protocol data The mediator handles secure communication with Causely's cloud-based causal reasoning engine, sending only distilled insights, never raw data. ### eBPF Instrumentation (Default) {#ebpf-instrumentation} Causely leverages [**OpenTelemetry eBPF instrumentation**](/telemetry-sources/ebpf), powered by [Grafana Beyla](https://grafana.com/docs/beyla/latest/), to automatically collect the following telemetry data: - Service dependencies and communication patterns - Synchronous and asynchronous communication signals - Performance metrics without code changes :::note eBPF Configuration eBPF instrumentation can be disabled if you already have comprehensive telemetry coverage. However, we recommend keeping it enabled as it serves as telemetry quality assurance, ensuring you have complete visibility into service dependencies and communication patterns. If your existing [OpenTelemetry](/telemetry-sources/opentelemetry/), [Prometheus](/telemetry-sources/prometheus/), or other telemetry sources already provide excellent coverage, you can [disable eBPF](/telemetry-sources/ebpf#configuration) to reduce overhead. ::: ### Executor The Executor is an optional component and can be enabled as part of the deployment process. The Executor is responsible for executing remediation actions. ### VictoriaMetrics VictoriaMetrics is a timeseries database used by the agents and mediator to store additional timeseries data. ## Data Sources and Integration While the components are deployed via Helm or CLI, data can be sent to the Causely mediator from multiple sources. See [all available telemetry sources](/telemetry-sources/) for a complete list, but make sure you have at least one source providing traces for the platform to be most effective. Without traces, Causely's ability to discover service dependencies and perform root cause analysis is severely limited. Any source that generates OpenTelemetry traces is perfect - Causely comes with auto-instrumentation for that, but if you already have another solution like [groundcover](/telemetry-sources/groundcover), [Odigos](/telemetry-sources/odigos), [Grafana Beyla](/telemetry-sources/grafana), an existing OpenTelemetry Collector, you can use those as well. Additionally, Causely can leverage data from [datadog](/telemetry-sources/datadog) and [dynatrace](/telemetry-sources/dynatrace) to discover service dependencies. ## Key Benefits of Causely's Architecture :::info Architecture Advantages Causely's distributed architecture provides significant advantages over traditional observability approaches: ::: ### Cost of Ownership **Minimal Egress Costs** - Only distilled insights are sent to the cloud, not raw telemetry data - Dramatically reduces data transfer costs compared to sending 100% of traces/metrics/logs - Especially impactful for high-volume environments generating terabytes of telemetry **Lightweight Footprint** - Mediator doesn't retain weeks, months, or years of historical data - Focuses on real-time analysis rather than long-term storage - Minimal resource consumption in your infrastructure **Zero Instrumentation Overhead** - No need for engineers to spend time instrumenting traces in application code - Leverages eBPF for automatic, zero-code instrumentation - Starts providing value immediately upon deployment ### Security & Privacy **Your Data Stays Local** - All raw telemetry data remains in your datacenter - No sensitive application data, logs, or metrics are transmitted - Only anonymized, distilled insights are sent to Causely's cloud engine **Distributed Processing** - Mediator can be deployed at the edge of your infrastructure - No need to send data across regions or clouds - Maintains data sovereignty and compliance requirements ### Data Efficiency **Intelligent Data Processing** - Processes only the signals needed for causal reasoning - Eliminates the need to store and analyze 100% of your telemetry data - Focuses on actionable insights rather than data hoarding **Comparison to Traditional Approaches** - Traditional observability platforms require ingesting and storing all telemetry data - Causely's approach processes only what's needed for root cause analysis - Results in orders of magnitude less data transmission and storage ### Recommended Sizing Guidelines Below are the recommended resource requests and limits for the main Causely components as typically deployed. You may need or want to adjust these values based on your environment, workload, or scaling requirements. **Mediator** - CPU: minimum 100m, up to 2 - Memory: minimum 128Mi, up to 8Gi - Storage: 10Gi **Executor** - CPU: minimum 100m, up to 2 - Memory: minimum 128Mi, up to 8Gi - Storage: not required **VictoriaMetrics** - CPU: minimum 100m, up to 4 - Memory: minimum 128Mi, up to 8Gi - Storage: 50Gi ## Next Steps After installation, we recommend: 1. **Integrate Agents**: Give MCP-compatible agents like Claude Code, Cursor, and HolmesGPT deterministic causal context see [Agent Integration](/agent-integration) 2. **Connect Additional Data Sources**: Visit [Telemetry Sources](/telemetry-sources/) to learn about supported integrations 3. **Customize Your Setup**: Learn about advanced configuration options in [Customize Installation](/installation/customize) For a detailed explanation of how Causely works, see [How Causely Works](/getting-started/how-causely-works). --- ## Scalable Trace Processor By default, Causely processes OpenTelemetry traces inline inside the mediator. For most environments this is sufficient. When trace volume grows or you want to scale trace processing independently from the rest of the mediator, you can enable the scalable trace processor. ## What gets deployed Enabling the scalable trace processor adds two components to your Causely installation: - **Trace controller**: a stateless router that receives OTLP traces from your collectors and distributes work across workers. Exposes gRPC on port `4317`. - **Trace workers**: a StatefulSet of workers that analyze spans and forward results to the mediator. Workers are stateful; each one accumulates trace context for the services assigned to it. Your OTel collectors send traces to the trace controller instead of the mediator directly. The mediator continues to handle all other telemetry sources unchanged. ## When to enable it Consider enabling the scalable trace processor when: - Trace volume has grown to the point where the mediator's CPU is consistently elevated due to span processing - You want to scale trace throughput by adding worker replicas without resizing the mediator ## Enabling the scalable trace processor Add the following to your `causely-values.yaml`: ```yaml traceController: enabled: true ``` Then upgrade your Helm release: ```bash helm upgrade causely \ oci://us-docker.pkg.dev/public-causely/public/causely \ --version \ --namespace=causely \ --values ``` ## Updating your trace sources ### Beyla (eBPF auto-instrumentation) If you are using Causely's built-in Beyla instrumentation, no changes are needed. When `traceController.enabled: true`, the Helm chart automatically reconfigures Beyla to send traces to the trace controller. The `helm upgrade` will trigger a rolling restart of the agent DaemonSet to pick up the new configuration. ### OpenTelemetry Collector If you have an external OTel collector sending traces to Causely, update its exporter endpoint: ```yaml exporters: otlp/causely: endpoint: trace-controller.causely:4317 compression: none tls: insecure: true ``` Replace `mediator.causely:4317` with `trace-controller.causely:4317`. All other collector settings remain unchanged. See [Integrate OpenTelemetry Traces](/telemetry-sources/opentelemetry) for full collector configuration examples. :::info Gradual migration The mediator continues to accept traces on port `4317` even after the scalable trace processor is enabled. External collectors that have not yet been updated will continue to work via the inline path. You can migrate collectors one at a time without any gap in trace ingestion. ::: ## Sizing The defaults are suitable for most environments. Use the replica table below as a starting point based on your service count. ### Default resources | | CPU request | CPU limit | Memory request | Memory limit | |---|---|---|---|---| | Trace controller | 200m | 1000m | 128Mi | 4Gi | | Trace worker | 200m | 1000m | 256Mi | 4Gi | ### Replica starting points | Services monitored per mediator | Worker replicas | Controller replicas | |---|---|---| | < 100 | 2 | 1 | | 100–500 | 3–5 | 2 | | 500–1,000 | 5–10 | 2 | | > 1,000 | 10+ | 3+ | :::info Never run fewer than 2 worker replicas in production. A single worker replica means all trace processing stops if that pod is lost; trace context is lost until the next export. ::: To override replicas in your values file: ```yaml traceController: enabled: true replicas: 2 traceWorker: replicas: 3 ``` ## Scaling signals Monitor these four signals to know when to add replicas. | Signal | Meaning | Action | |---|---|---| | `causely_tc_inflight_rejected_total` > 0 | Controller dispatch queue is full | Check controller CPU. If high → add controller replicas. If low → add worker replicas. | | `causely_tc_dispatch_drops_total` or `causely_tc_consume_drops_total` growing | Exports being dropped | Check per-worker `causely_tw_analyze_inflight`. If one pod is high → add workers. If all pods are high → add workers. | | `causely_tw_analyze_inflight` > 2 sustained (per pod) | Work queuing inside a worker | Add worker replicas. | | Worker RSS > 75% of memory limit | Memory pressure | Raise the worker memory limit; also check whether your service count has grown significantly. | **Scale workers horizontally, not vertically.** Adding worker replicas increases parallel processing capacity. Increasing pod size does not. --- ## API Tokens Causely provides two types of API tokens depending on the scope of access needed: **Personal Tokens** scoped to an individual user, and **Tenant-wide Tokens** shared across the workspace. ## Personal Tokens Personal tokens are tied to your individual account and carry your user's permissions. **To create a personal token:** 1. Open the [Admin Portal](/manage/administration) (profile icon → **Admin Portal**) 2. Select **Personal Tokens** from the left sidebar 3. Click **Create Token** 4. Give the token a descriptive name and set its expiration 5. Copy and securely store the token value—it will not be shown again ## Tenant-wide Tokens Tenant-wide tokens are not tied to an individual user and can be used for workspace-level integrations and automation. Only **Administrators** can create and manage tenant-wide tokens. **To create a tenant-wide token:** 1. Open the [Admin Portal](/manage/administration) (profile icon → **Admin Portal**) 2. Select **API Tokens** from the left sidebar 3. Click **Create Token** 4. Give the token a descriptive name and set its expiration 5. Copy and securely store the token value—it will not be shown again ## Using Your Token Once you have a token, see [API Authentication](/api/authentication) for how to authenticate requests to the Causely API. ## Token Security - Treat tokens like passwords—store them in a secrets manager and never commit them to source control - Set expiration dates when creating tokens - Delete tokens that are no longer in use - For mediator token rotation and revocation, see [Token Management and Developer Role](/changelog/v1.0.126#token-management-and-developer-role) --- ## Audit Logs The Audit Logs page provides a chronological record of administrative actions taken in your Causely workspace, such as user invitations, role changes, SSO configuration updates, and token management events. ## Accessing Audit Logs 1. Open the [Admin Portal](/manage/administration) (profile icon → **Admin Portal**) 2. Select **Audit Logs** from the left sidebar The log lists events in reverse chronological order. Each entry shows the action performed, the user who performed it, and a timestamp. --- ## Administration The Causely Admin Portal provides self-service tools for managing your workspace—users, authentication, audit history, and API access. ## Accessing the Admin Portal To open the Admin Portal: 1. Click your **profile icon** in the top-right corner of the Causely UI 2. Select **Admin Portal** from the dropdown menu This opens the portal at `auth.causely.app`, where all administrative tasks are performed. ## What You Can Do - [User Management](/manage/administration/user-management)—Invite users and assign roles - [Single Sign-On (SSO)](/manage/administration/sso)—Configure SSO for your workspace - [Audit Logs](/manage/administration/audit-logs)—Review a log of administrative activity - [API Tokens](/manage/administration/api-tokens)—Create personal and tenant-wide API tokens --- ## Single Sign-On (SSO) Causely supports self-service SSO configuration so your team can authenticate using your existing identity provider (IDP). ## Accessing SSO Settings 1. Open the [Admin Portal](/manage/administration) (profile icon → **Admin Portal**) 2. Select **SSO** from the left sidebar ## Setting Up an SSO Connection 1. On the SSO page, click **Setup SSO connection** 2. Select your **identity provider (IDP)** (for example, Okta, Azure AD, Google Workspace) 3. Follow the on-screen instructions to configure the integration --- ## User Management The Users page lets you invite team members to your Causely workspace and control their level of access via roles. ## Accessing Users 1. Open the [Admin Portal](/manage/administration) (profile icon → **Admin Portal**) 2. Select **Users** from the left sidebar ## Inviting a User 1. On the Users page, click **Invite User** 2. Enter the user's email address 3. Select a role (see [Roles](#roles) below) 4. Click **Invite** The invited user will receive an email with a link to join your workspace. ## Roles Causely has three roles that control what a user can do in the workspace: | Role | Description | |------|-------------| | **Administrator** | Full administrative access. Can manage tokens across all mediators in the tenant, invite and manage users, and configure workspace settings. | | **Developer** | Can add new mediators and manage tokens for those mediators. Cannot manage tokens belonging to other mediators or perform broader administrative actions. | | **Read Only** | Can make most configuration changes, including service tiers, thresholds, notifications, and integrations. Cannot add mediators, manage tokens, or create public scopes. | You can change a user's role at any time from the Users page by selecting the user and updating their role. --- ## Manage Causely Install, connect, integrate, and configure your Causely deployment. :::tip Connecting AI agents to Causely? See [Agent Integration](/agent-integration/) for how to give your agents causal context. ::: ## Administration Manage users, access control, audit logs, and API tokens via the Admin Portal (profile icon → **Admin Portal**): - [Administration Overview](/manage/administration) - Access and overview of administrative tools - [User Management](/manage/administration/user-management) - Invite users and assign roles - [Single Sign-On (SSO)](/manage/administration/sso) - Configure SSO for your workspace - [Audit Logs](/manage/administration/audit-logs) - Review administrative activity - [API Tokens](/manage/administration/api-tokens) - Create personal and tenant-wide API tokens ## Installation Deploy Causely in your environment: - [Installation Overview](/installation) - Learn about installation options - [Helm](/installation/helm) - Install on Kubernetes using Helm - [CLI](/installation/cli) - Install using the command-line interface - [Docker](/installation/docker) - Deploy on standalone Docker hosts - [Nomad](/installation/nomad) - Install on HashiCorp Nomad clusters - [FluxCD](/installation/flux) - Deploy using FluxCD for GitOps - [Argo CD](/installation/argocd) - Deploy using Argo CD for GitOps - [Customize Installation](/installation/customize) - Advanced configuration options - [Network Allowlisting](/installation/network-allowlisting) - Configure network access ## Connect Telemetry Connect your observability data sources to Causely: - [Connect Telemetry Overview](/telemetry-sources) - Overview of telemetry integrations - See all available [telemetry source integrations](/telemetry-sources) including OpenTelemetry, Prometheus, Datadog, AWS, Azure, GCP, Kubernetes, and more ## Integrations Push Causely insights into your existing workflows: - [Integrations Overview](/workflows) - Overview of workflow integrations - See all available [workflow integrations](/workflows) including Slack, Jira, Grafana, Microsoft Teams, Opsgenie, and more ## Configuration Configure Causely to match your environment and requirements: - [Configuration Overview](/configuration) - Learn about configuration options - [Scopes](/configuration/scopes) - Define custom scopes for your environment - [Service Tiers](/configuration/service-tiers) - Configure service tier classifications - [SLO Configuration](/configuration/slo-configuration) - Customize service level objectives - [Thresholds](/configuration/thresholds) - Configure performance thresholds - [Signal Delay](/configuration/symptom-delay) - Configure Signal detection delays - [Credentials Autodiscovery](/configuration/credentials-autodiscovery) - Automatically discover credentials ## Accomplish Your Goals Common operational goals you can achieve with Causely, with step-by-step guidance: - [Goals Overview](/goals/) - Start here to find guidance matched to your objective - [Accelerate Resolution](/goals/accelerate-resolution) - Get precise Diagnosis insights to resolve incidents faster - [Proactively Prevent Incidents](/goals/prevent-incidents) - Identify emerging risks before they impact your services - [Understand System Behavior](/goals/understand-system-behavior) - Gain deep insights into why your systems behave as they do - [Scale Reliability](/goals/scale-reliability) - Maintain reliability across complex, fast-changing systems - [Optimize Performance](/goals/optimize-performance) - Continuously improve system performance with actionable insights ## Causely In Action Explore the Causely UI and see how topology, Diagnoses, and insights surface in practice: - [Causely In Action Overview](/in-action) - Overview of the Causely interface and daily workflows - [Welcome Page](/in-action/welcome-page) - Start your day with a concise overview of urgent issues and insights - [Topology](/in-action/topology) - Browse and inspect entities discovered by the Causely engine - [Root Causes](/in-action/root-causes) - See inferred Diagnoses and investigate with the causality graph - [Automate Remediation](/in-action/automate-remediation) - Automate remediation for resource contention issues - [Reliability Delta](/in-action/reliability-delta) - Compare environment snapshots to track reliability changes - [Feature Demos](/in-action/feature-demos) - Short demonstrations of key capabilities and workflows --- ## Add More Telemetry Step ## Add More Telemetry Sources To help Causely infer Diagnoses more effectively, connect additional telemetry sources. Visit the [Telemetry Sources](/telemetry-sources) page to learn about the data sources that Causely supports, including: - [Prometheus](/telemetry-sources/prometheus/) - [OpenTelemetry](/telemetry-sources/opentelemetry/) - [Grafana](/telemetry-sources/grafana/) - [Alertmanager](/telemetry-sources/alertmanager/) - And many more --- ## Get Installation Command Step ## Get Your Installation Command 1. Log in to the [Causely portal](https://portal.causely.app). 2. Visit the [mediators page](https://portal.causely.app/agents). 3. Click the **"Add new"** button (or **"Add ➕"** button) on the mediators page. 4. In the instructions panel, you'll see the Helm installation command with your access token pre-filled. 5. Copy the complete Helm command. It will look similar to the example below. :::info Example Command The command you copy will include your specific access token and cluster name. Here's what it typically looks like: ```bash export CAUSELY_TOKEN= export CAUSELY_CLUSTER_NAME= export CAUSELY_VERSION= helm upgrade --install causely \\ --create-namespace oci://us-docker.pkg.dev/public-causely/public/causely \\ --version "${CAUSELY_VERSION}" \\ --namespace=causely \\ --set image.tag="${CAUSELY_VERSION}" \\ --set global.cluster_name="\${CAUSELY_CLUSTER_NAME}" \\ --set mediator.gateway.token="\${CAUSELY_TOKEN}" ``` ::: :::info Screenshot Placeholder Screenshots of the mediators page, "Add new" button, and instructions panel will be added here. ::: --- ## Other Platforms Note :::tip Other Platforms Supported While this quickstart guide focuses on Kubernetes, Causely also supports: - **Container Orchestration**: [Nomad](/installation/nomad/), Docker, ECS - **GitOps**: [Argo CD](/installation/argocd/), [Flux](/installation/flux/) - **Virtual Machines**: Direct installation on VMs For detailed installation instructions for these platforms, visit the [Installation Overview](/installation) page. ::: --- ## AWS Quickstart Guide This quickstart guide will help you get Causely up and running on AWS. This setup deploys Causely in your Kubernetes cluster (including EKS) with eBPF instrumentation enabled, and connects it to your AWS account to monitor cloud resources. ## Get Your Installation Command 1. Log in to the [Causely portal](https://portal.causely.app). 2. Visit the [mediators page](https://portal.causely.app/agents). 3. Click the **"Add new"** button (or **"Add ➕"** button) on the mediators page. 4. In the instructions panel, you'll see the Helm installation command with your access token pre-filled. 5. Copy the complete Helm command. It will look similar to the example below. :::info Example Command The command you copy will include your specific access token and cluster name. Here's what it typically looks like: ```bash export CAUSELY_TOKEN= export CAUSELY_CLUSTER_NAME= export CAUSELY_VERSION= helm upgrade --install causely \\ --create-namespace oci://us-docker.pkg.dev/public-causely/public/causely \\ --version "${CAUSELY_VERSION}" \\ --namespace=causely \\ --set image.tag="${CAUSELY_VERSION}" \\ --set global.cluster_name="\${CAUSELY_CLUSTER_NAME}" \\ --set mediator.gateway.token="\${CAUSELY_TOKEN}" ``` ::: :::info Screenshot Placeholder Screenshots of the mediators page, "Add new" button, and instructions panel will be added here. ::: ## Run the Installation Command 1. Open a terminal with `kubectl` configured to access your Kubernetes cluster. 2. Paste and run the Helm command you copied from the portal. 3. Wait for the installation to complete: ```bash kubectl wait --for=condition=Ready pod -l app.kubernetes.io/part-of=causely -n causely --timeout=300s ``` :::info By default, Causely will use [eBPF-based instrumentation](/telemetry-sources/ebpf) to automatically instrument your applications running in Kubernetes. This provides zero-effort observability without code changes. ::: ## Connect Your AWS Account Now you need to configure Causely to monitor your AWS resources. Causely supports two authentication methods: IAM Role (recommended for EKS) or Access Keys. #### Option 1: IAM Role (Recommended for EKS) If you're running on Amazon EKS, we recommend using IAM roles for service accounts (IRSA) or EKS Pod Identity for secure authentication. 1. **Create an IAM role** with the necessary permissions. See the [AWS integration documentation](/telemetry-sources/aws/) for the required IAM policy and trust relationship. 2. **Annotate the mediator service account** to use the IAM role: ```bash kubectl annotate serviceaccount causely-mediator \ -n causely \ eks.amazonaws.com/role-arn=arn:aws:iam::ACCOUNT_ID:role/CAUSELY_ROLE_NAME ``` #### Option 2: Access Keys 1. **Create a Kubernetes secret** with your AWS access key credentials: ```bash kubectl create secret generic \ --namespace causely aws-credentials \ --from-literal=access-key-id="" \ --from-literal=secret-access-key="" ``` 2. **Update your Causely configuration** by creating or updating `causely-values.yaml`: ```yaml global: cluster_name: mediator: gateway: token: scrapers: aws: enabled: true accounts: - secretName: aws-credentials # namespace: causely # optional; defaults to the pod namespace ``` 3. **Upgrade your installation**: ```bash helm upgrade --install causely \\ --namespace=causely \\ --values causely-values.yaml \\ oci://us-docker.pkg.dev/public-causely/public/causely \\ --version "${CAUSELY_VERSION}" ``` :::tip Alternative: Credentials Autodiscovery You can also use credentials autodiscovery by labeling the secret: ```bash kubectl --namespace causely label secret aws-credentials "causely.ai/scraper=AWS" ``` This allows you to add new accounts without editing the configuration. See the [AWS integration documentation](/telemetry-sources/aws/) for more details. ::: For complete setup instructions, including IAM role configuration and supported AWS services, see the [AWS integration documentation](/telemetry-sources/aws/). ## What to Expect Once configured, Causely will: - **Monitor Kubernetes services**: Using eBPF instrumentation, Causely automatically discovers and monitors services running in your Kubernetes cluster. - **Monitor AWS resources**: Causely will discover and monitor AWS services including: - Application Load Balancer (ALB) and Network Load Balancer (NLB) - Amazon ECS, EC2, EBS - Amazon RDS, ElastiCache, MSK - AWS Lambda - And more - **Unified view**: Causely creates a unified topology showing relationships between your Kubernetes services and AWS resources. - **Show data in the UI**: Within a few minutes, you should start seeing entities appearing in the Causely UI at [https://portal.causely.app](https://portal.causely.app). ## Add More Telemetry Sources To help Causely infer Diagnoses more effectively, connect additional telemetry sources. Visit the [Telemetry Sources](/telemetry-sources) page to learn about the data sources that Causely supports, including: - [Prometheus](/telemetry-sources/prometheus/) - [OpenTelemetry](/telemetry-sources/opentelemetry/) - [Grafana](/telemetry-sources/grafana/) - [Alertmanager](/telemetry-sources/alertmanager/) - And many more ## Add Workflow Integrations Push insights into your existing workflows and notification tools. Visit the [Workflow Integrations](/workflows) page to set up integrations with: - [Slack](/workflows/slack/) - [Grafana](/workflows/grafana-plugin/) - [incident.io](/workflows/incident-io/) - [Microsoft Teams](/workflows/microsoft-teams/) - [Prometheus Alertmanager](/workflows/prometheus-alertmanager/) - And more ## What's Your Goal? , title: 'Accelerate resolution', description: 'Get precise Diagnosis insights during incidents to resolve issues faster.', }, { icon: , title: 'Proactively prevent incidents', description: 'Identify emerging risks before they impact your services and SLOs.', }, { icon: , title: 'Understand system behavior', description: 'Gain deep insights into why your systems behave the way they do.', }, { icon: , title: 'Scale reliability', description: 'Maintain reliability across complex, fast-changing systems at scale.', }, { icon: , title: 'Optimize performance', description: 'Continuously improve system performance with actionable insights.', }, ]} teaser='Choose a goal to explore how Causely can help you achieve it.' /> :::tip Other Platforms Supported While this quickstart guide focuses on Kubernetes, Causely also supports: - **Container Orchestration**: [Nomad](/installation/nomad/), Docker, ECS - **GitOps**: [Argo CD](/installation/argocd/), [Flux](/installation/flux/) - **Virtual Machines**: Direct installation on VMs For detailed installation instructions for these platforms, visit the [Installation Overview](/installation) page. ::: --- ## Azure Quickstart Guide This quickstart guide will help you get Causely up and running on Azure. This setup deploys Causely in your Kubernetes cluster (including AKS) with eBPF instrumentation enabled, and connects it to your Azure subscription to monitor cloud resources. ## Get Your Installation Command 1. Log in to the [Causely portal](https://portal.causely.app). 2. Visit the [mediators page](https://portal.causely.app/agents). 3. Click the **"Add new"** button (or **"Add ➕"** button) on the mediators page. 4. In the instructions panel, you'll see the Helm installation command with your access token pre-filled. 5. Copy the complete Helm command. It will look similar to the example below. :::info Example Command The command you copy will include your specific access token and cluster name. Here's what it typically looks like: ```bash export CAUSELY_TOKEN= export CAUSELY_CLUSTER_NAME= export CAUSELY_VERSION= helm upgrade --install causely \\ --create-namespace oci://us-docker.pkg.dev/public-causely/public/causely \\ --version "${CAUSELY_VERSION}" \\ --namespace=causely \\ --set image.tag="${CAUSELY_VERSION}" \\ --set global.cluster_name="\${CAUSELY_CLUSTER_NAME}" \\ --set mediator.gateway.token="\${CAUSELY_TOKEN}" ``` ::: :::info Screenshot Placeholder Screenshots of the mediators page, "Add new" button, and instructions panel will be added here. ::: ## Run the Installation Command 1. Open a terminal with `kubectl` configured to access your Kubernetes cluster. 2. Paste and run the Helm command you copied from the portal. 3. Wait for the installation to complete: ```bash kubectl wait --for=condition=Ready pod -l app.kubernetes.io/part-of=causely -n causely --timeout=300s ``` :::info By default, Causely will use [eBPF-based instrumentation](/telemetry-sources/ebpf) to automatically instrument your applications running in Kubernetes. This provides zero-effort observability without code changes. ::: ## Connect Your Azure Subscription Now you need to configure Causely to monitor your Azure resources. You'll need a Service Principal (SPN) with Reader permissions on the subscriptions you want to monitor. #### Step 5a: Create a Service Principal Use the Azure CLI to create a Service Principal: ```bash az ad sp create-for-rbac --name causely-spn --role reader --scopes /subscriptions/ ``` This will output credentials including `tenantId`, `clientId`, `clientSecret`, and `subscriptionId`. #### Step 5b: Create a Kubernetes Secret Save the Service Principal credentials in a Kubernetes Secret: ```bash kubectl --namespace causely create secret generic spn-credentials \ --from-literal=tenantId="" \ --from-literal=clientId="" \ --from-literal=clientSecret="" \ --from-literal=subscriptionId="" ``` #### Step 5c: Update Causely Configuration Update your `causely-values.yaml` to enable Azure scraping. You can enable different Azure resource types: ```yaml global: cluster_name: mediator: gateway: token: scrapers: azure: # Scrape VMs & Disks enabled: true subscriptions: - secretName: spn-credentials namespace: causely azurePostgres: # Scrape PostgreSQL Databases enabled: true subscriptions: - secretName: spn-credentials namespace: causely azureMysql: # Scrape MySQL Databases enabled: true subscriptions: - secretName: spn-credentials namespace: causely azureRedis: # Scrape Redis enabled: true subscriptions: - secretName: spn-credentials namespace: causely azureEventhub: # Scrape EventHub enabled: true subscriptions: - secretName: spn-credentials namespace: causely ``` Then upgrade your installation: ```bash helm upgrade --install causely \\ --namespace=causely \\ --values causely-values.yaml \\ oci://us-docker.pkg.dev/public-causely/public/causely \\ --version "${CAUSELY_VERSION}" ``` :::tip Alternative: Credentials Autodiscovery You can also use credentials autodiscovery by labeling the secret for each scraper: ```bash # Azure VMs & Disks kubectl --namespace causely label secret spn-credentials "causely.ai/scraper=Azure" # Azure PostgreSQL kubectl --namespace causely label secret spn-credentials "causely.ai/scraper=AzurePostgres" # Azure MySQL kubectl --namespace causely label secret spn-credentials "causely.ai/scraper=AzureMysql" # Azure Redis kubectl --namespace causely label secret spn-credentials "causely.ai/scraper=AzureRedis" # Azure EventHub kubectl --namespace causely label secret spn-credentials "causely.ai/scraper=AzureEventHub" ``` This allows you to add new accounts without editing the configuration. See the [Azure integration documentation](/telemetry-sources/azure/) for more details. ::: ## What to Expect Once configured, Causely will: - **Monitor Kubernetes services**: Using eBPF instrumentation, Causely automatically discovers and monitors services running in your Kubernetes cluster. - **Monitor Azure resources**: Causely will discover and monitor Azure resources including: - Virtual Machines and Disks - Azure PostgreSQL and MySQL databases - Azure Redis - Azure EventHub - And more - **Unified view**: Causely creates a unified topology showing relationships between your Kubernetes services and Azure resources. - **Show data in the UI**: Within a few minutes, you should start seeing entities appearing in the Causely UI at [https://portal.causely.app](https://portal.causely.app). ## Add More Telemetry Sources To help Causely infer Diagnoses more effectively, connect additional telemetry sources. Visit the [Telemetry Sources](/telemetry-sources) page to learn about the data sources that Causely supports, including: - [Prometheus](/telemetry-sources/prometheus/) - [OpenTelemetry](/telemetry-sources/opentelemetry/) - [Grafana](/telemetry-sources/grafana/) - [Alertmanager](/telemetry-sources/alertmanager/) - And many more ## Add Workflow Integrations Push insights into your existing workflows and notification tools. Visit the [Workflow Integrations](/workflows) page to set up integrations with: - [Slack](/workflows/slack/) - [Grafana](/workflows/grafana-plugin/) - [incident.io](/workflows/incident-io/) - [Microsoft Teams](/workflows/microsoft-teams/) - [Prometheus Alertmanager](/workflows/prometheus-alertmanager/) - And more ## What's Your Goal? , title: 'Accelerate resolution', description: 'Get precise Diagnosis insights during incidents to resolve issues faster.', }, { icon: , title: 'Proactively prevent incidents', description: 'Identify emerging risks before they impact your services and SLOs.', }, { icon: , title: 'Understand system behavior', description: 'Gain deep insights into why your systems behave the way they do.', }, { icon: , title: 'Scale reliability', description: 'Maintain reliability across complex, fast-changing systems at scale.', }, { icon: , title: 'Optimize performance', description: 'Continuously improve system performance with actionable insights.', }, ]} teaser='Choose a goal to explore how Causely can help you achieve it.' /> :::tip Other Platforms Supported While this quickstart guide focuses on Kubernetes, Causely also supports: - **Container Orchestration**: [Nomad](/installation/nomad/), Docker, ECS - **GitOps**: [Argo CD](/installation/argocd/), [Flux](/installation/flux/) - **Virtual Machines**: Direct installation on VMs For detailed installation instructions for these platforms, visit the [Installation Overview](/installation) page. ::: --- ## Datadog Quickstart Guide This quickstart guide will help you get Causely up and running with Datadog. This setup uses Datadog APM with dual shipping to send traces to Causely while continuing to send data to Datadog. ## Get Your Installation Command 1. Log in to the [Causely portal](https://portal.causely.app). 2. Visit the [mediators page](https://portal.causely.app/agents). 3. Click the **"Add new"** button (or **"Add ➕"** button) on the mediators page. 4. In the instructions panel, you'll see the Helm installation command with your access token pre-filled. 5. Copy the complete Helm command. It will look similar to the example below. :::info Example Command The command you copy will include your specific access token and cluster name. Here's what it typically looks like: ```bash export CAUSELY_TOKEN= export CAUSELY_CLUSTER_NAME= export CAUSELY_VERSION= helm upgrade --install causely \\ --create-namespace oci://us-docker.pkg.dev/public-causely/public/causely \\ --version "${CAUSELY_VERSION}" \\ --namespace=causely \\ --set image.tag="${CAUSELY_VERSION}" \\ --set global.cluster_name="\${CAUSELY_CLUSTER_NAME}" \\ --set mediator.gateway.token="\${CAUSELY_TOKEN}" ``` ::: :::info Screenshot Placeholder Screenshots of the mediators page, "Add new" button, and instructions panel will be added here. ::: ## Create a Values File to Disable eBPF Since you'll be using Datadog for instrumentation, create a `causely-values.yaml` file to disable eBPF instrumentation: ```yaml global: cluster_name: mediator: gateway: token: scrapers: bpf: enabled: false ``` Replace `` and `` with your actual values from the portal. ## Run the Installation Command 1. Open a terminal with `kubectl` configured to access your cluster. 2. Run the Helm command with your values file: ```bash export CAUSELY_VERSION= helm upgrade --install causely \\ --create-namespace oci://us-docker.pkg.dev/public-causely/public/causely \\ --version "${CAUSELY_VERSION}" \\ --namespace=causely \\ --set image.tag="${CAUSELY_VERSION}" \\ --set global.cluster_name="" \\ --set mediator.gateway.token="" \\ --values causely-values.yaml ``` 3. Wait for the installation to complete: ```bash kubectl wait --for=condition=Ready pod -l app.kubernetes.io/part-of=causely -n causely --timeout=300s ``` ## Configure Datadog Dual Shipping Now you need to configure your Datadog Agent to send traces to Causely using [dual shipping](https://docs.datadoghq.com/agent/configuration/dual-shipping/). #### Option 1: If using Helm to deploy Datadog Add the following to your Datadog `values.yaml`: ```yaml agents: useConfigMap: true customAgentConfig: apm_config: additional_endpoints: 'http://mediator.causely:8126': - 'datadog-receiver' ``` Then upgrade your Datadog installation: ```bash helm upgrade --install datadog datadog/datadog -f ./values.yaml ``` #### Option 2: If using the Datadog Operator Modify your `DatadogAgent` custom resource: ```yaml apiVersion: datadoghq.com/v2alpha1 kind: DatadogAgent metadata: name: datadog spec: override: nodeAgent: env: - name: DD_APM_ADDITIONAL_ENDPOINTS value: '{"http://mediator.causely:8126": ["datadog-receiver"]}' ``` For more details, see the [Datadog integration documentation](/telemetry-sources/datadog/). ## What to Expect Once configured, Causely will: - **Receive Datadog traces**: Your Datadog Agent will send traces to Causely's mediator via dual shipping. - **Discover service dependencies**: Causely will automatically discover service dependencies from your Datadog APM traces. - **Show data in the UI**: Within a few minutes, you should start seeing services and their relationships appearing in the Causely UI at [https://portal.causely.app](https://portal.causely.app). :::tip You can also enable Datadog monitors as a data source for additional insights. See the [Datadog integration documentation](/telemetry-sources/datadog/) for details. ::: ## Add More Telemetry Sources To help Causely infer Diagnoses more effectively, connect additional telemetry sources. Visit the [Telemetry Sources](/telemetry-sources) page to learn about the data sources that Causely supports, including: - [Prometheus](/telemetry-sources/prometheus/) - [OpenTelemetry](/telemetry-sources/opentelemetry/) - [Grafana](/telemetry-sources/grafana/) - [Alertmanager](/telemetry-sources/alertmanager/) - And many more ## Add Workflow Integrations Push insights into your existing workflows and notification tools. Visit the [Workflow Integrations](/workflows) page to set up integrations with: - [Slack](/workflows/slack/) - [Grafana](/workflows/grafana-plugin/) - [incident.io](/workflows/incident-io/) - [Microsoft Teams](/workflows/microsoft-teams/) - [Prometheus Alertmanager](/workflows/prometheus-alertmanager/) - And more ## What's Your Goal? , title: 'Accelerate resolution', description: 'Get precise Diagnosis insights during incidents to resolve issues faster.', }, { icon: , title: 'Proactively prevent incidents', description: 'Identify emerging risks before they impact your services and SLOs.', }, { icon: , title: 'Understand system behavior', description: 'Gain deep insights into why your systems behave the way they do.', }, { icon: , title: 'Scale reliability', description: 'Maintain reliability across complex, fast-changing systems at scale.', }, { icon: , title: 'Optimize performance', description: 'Continuously improve system performance with actionable insights.', }, ]} teaser='Choose a goal to explore how Causely can help you achieve it.' /> :::tip Other Platforms Supported While this quickstart guide focuses on Kubernetes, Causely also supports: - **Container Orchestration**: [Nomad](/installation/nomad/), Docker, ECS - **GitOps**: [Argo CD](/installation/argocd/), [Flux](/installation/flux/) - **Virtual Machines**: Direct installation on VMs For detailed installation instructions for these platforms, visit the [Installation Overview](/installation) page. ::: --- ## Dynatrace Quickstart Guide This quickstart guide will help you get Causely up and running with Dynatrace. This setup allows Causely to ingest service dependencies and traces from both Kubernetes and ECS environments, creating a unified causal model. ## Get Your Installation Command 1. Log in to the [Causely portal](https://portal.causely.app). 2. Visit the [mediators page](https://portal.causely.app/agents). 3. Click the **"Add new"** button (or **"Add ➕"** button) on the mediators page. 4. In the instructions panel, you'll see the Helm installation command with your access token pre-filled. 5. Copy the complete Helm command. It will look similar to the example below. :::info Example Command The command you copy will include your specific access token and cluster name. Here's what it typically looks like: ```bash export CAUSELY_TOKEN= export CAUSELY_CLUSTER_NAME= export CAUSELY_VERSION= helm upgrade --install causely \\ --create-namespace oci://us-docker.pkg.dev/public-causely/public/causely \\ --version "${CAUSELY_VERSION}" \\ --namespace=causely \\ --set image.tag="${CAUSELY_VERSION}" \\ --set global.cluster_name="\${CAUSELY_CLUSTER_NAME}" \\ --set mediator.gateway.token="\${CAUSELY_TOKEN}" ``` ::: :::info Screenshot Placeholder Screenshots of the mediators page, "Add new" button, and instructions panel will be added here. ::: ## Create a Values File to Disable eBPF Since you'll be using Dynatrace for instrumentation, create a `causely-values.yaml` file to disable eBPF instrumentation: ```yaml global: cluster_name: mediator: gateway: token: scrapers: bpf: enabled: false ``` Replace `` and `` with your actual values from the portal. ## Run the Installation Command 1. Open a terminal with `kubectl` configured to access your cluster. 2. Run the Helm command with your values file: ```bash export CAUSELY_VERSION= helm upgrade --install causely \\ --create-namespace oci://us-docker.pkg.dev/public-causely/public/causely \\ --version "${CAUSELY_VERSION}" \\ --namespace=causely \\ --set image.tag="${CAUSELY_VERSION}" \\ --set global.cluster_name="" \\ --set mediator.gateway.token="" \\ --values causely-values.yaml ``` 3. Wait for the installation to complete: ```bash kubectl wait --for=condition=Ready pod -l app.kubernetes.io/part-of=causely -n causely --timeout=300s ``` ## Configure Dynatrace Integration Now you need to configure Causely to scrape data from your Dynatrace account. #### Step 6a: Create a Kubernetes Secret Create a Kubernetes secret with your Dynatrace URL and API token: ```bash kubectl create secret generic \ --namespace causely dynatrace-credentials \ --from-literal=url="" \ --from-literal=api-token="" ``` #### Step 6b: Update Causely Configuration Update your `causely-values.yaml` to enable Dynatrace scraping: ```yaml global: cluster_name: mediator: gateway: token: scrapers: bpf: enabled: false dynatrace: enabled: true accounts: - secretName: dynatrace-credentials ``` Then upgrade your Causely installation: ```bash helm upgrade --install causely \\ --namespace=causely \\ --values causely-values.yaml \\ oci://us-docker.pkg.dev/public-causely/public/causely \\ --version "${CAUSELY_VERSION}" ``` :::tip Alternative: Credentials Autodiscovery You can also use credentials autodiscovery by labeling the secret: ```bash kubectl --namespace causely label secret dynatrace-credentials "causely.ai/scraper=Dynatrace" ``` This allows you to add new accounts without editing the configuration. See the [Dynatrace integration documentation](/telemetry-sources/dynatrace/) for more details. ::: ## What to Expect Once configured, Causely will: - **Ingest Dynatrace topology and telemetry**: Causely will scrape service dependencies and traces from your Dynatrace account. - **Unified view across platforms**: If you have both Kubernetes and ECS services, Causely will automatically discover and model dependencies between them, creating a unified causal view. - **Show data in the UI**: Within a few minutes, you should start seeing services from both Kubernetes and ECS appearing in the Causely UI at [https://portal.causely.app](https://portal.causely.app). For more details on how Causely unifies data across platforms, see the [Dynatrace integration documentation](/telemetry-sources/dynatrace/). ## Add More Telemetry Sources To help Causely infer Diagnoses more effectively, connect additional telemetry sources. Visit the [Telemetry Sources](/telemetry-sources) page to learn about the data sources that Causely supports, including: - [Prometheus](/telemetry-sources/prometheus/) - [OpenTelemetry](/telemetry-sources/opentelemetry/) - [Grafana](/telemetry-sources/grafana/) - [Alertmanager](/telemetry-sources/alertmanager/) - And many more ## Add Workflow Integrations Push insights into your existing workflows and notification tools. Visit the [Workflow Integrations](/workflows) page to set up integrations with: - [Slack](/workflows/slack/) - [Grafana](/workflows/grafana-plugin/) - [incident.io](/workflows/incident-io/) - [Microsoft Teams](/workflows/microsoft-teams/) - [Prometheus Alertmanager](/workflows/prometheus-alertmanager/) - And more ## What's Your Goal? , title: 'Accelerate resolution', description: 'Get precise Diagnosis insights during incidents to resolve issues faster.', }, { icon: , title: 'Proactively prevent incidents', description: 'Identify emerging risks before they impact your services and SLOs.', }, { icon: , title: 'Understand system behavior', description: 'Gain deep insights into why your systems behave the way they do.', }, { icon: , title: 'Scale reliability', description: 'Maintain reliability across complex, fast-changing systems at scale.', }, { icon: , title: 'Optimize performance', description: 'Continuously improve system performance with actionable insights.', }, ]} teaser='Choose a goal to explore how Causely can help you achieve it.' /> :::tip Other Platforms Supported While this quickstart guide focuses on Kubernetes, Causely also supports: - **Container Orchestration**: [Nomad](/installation/nomad/), Docker, ECS - **GitOps**: [Argo CD](/installation/argocd/), [Flux](/installation/flux/) - **Virtual Machines**: Direct installation on VMs For detailed installation instructions for these platforms, visit the [Installation Overview](/installation) page. ::: --- ## GCP Quickstart Guide This quickstart guide will help you get Causely up and running on GCP. This setup deploys Causely in your Kubernetes cluster (including GKE) with eBPF instrumentation enabled, and connects it to your GCP project to monitor cloud resources. ## Get Your Installation Command 1. Log in to the [Causely portal](https://portal.causely.app). 2. Visit the [mediators page](https://portal.causely.app/agents). 3. Click the **"Add new"** button (or **"Add ➕"** button) on the mediators page. 4. In the instructions panel, you'll see the Helm installation command with your access token pre-filled. 5. Copy the complete Helm command. It will look similar to the example below. :::info Example Command The command you copy will include your specific access token and cluster name. Here's what it typically looks like: ```bash export CAUSELY_TOKEN= export CAUSELY_CLUSTER_NAME= export CAUSELY_VERSION= helm upgrade --install causely \\ --create-namespace oci://us-docker.pkg.dev/public-causely/public/causely \\ --version "${CAUSELY_VERSION}" \\ --namespace=causely \\ --set image.tag="${CAUSELY_VERSION}" \\ --set global.cluster_name="\${CAUSELY_CLUSTER_NAME}" \\ --set mediator.gateway.token="\${CAUSELY_TOKEN}" ``` ::: :::info Screenshot Placeholder Screenshots of the mediators page, "Add new" button, and instructions panel will be added here. ::: ## Run the Installation Command 1. Open a terminal with `kubectl` configured to access your Kubernetes cluster. 2. Paste and run the Helm command you copied from the portal. 3. Wait for the installation to complete: ```bash kubectl wait --for=condition=Ready pod -l app.kubernetes.io/part-of=causely -n causely --timeout=300s ``` :::info By default, Causely will use [eBPF-based instrumentation](/telemetry-sources/ebpf) to automatically instrument your applications running in Kubernetes. This provides zero-effort observability without code changes. ::: ## Connect Your GCP Project Now you need to configure Causely to monitor your GCP resources. You'll need a Service Account with Viewer permissions on the projects you want to monitor. #### Step 5a: Create a Service Account Use the `gcloud` CLI to create a Service Account: ```bash gcloud iam service-accounts create causely \ --description="Causely service account" \ --display-name="Causely" gcloud projects add-iam-policy-binding PROJECT_ID \ --member="serviceAccount:causely@PROJECT_ID.iam.gserviceaccount.com" \ --role="roles/viewer" gcloud projects add-iam-policy-binding PROJECT_ID \ --member="serviceAccount:causely@PROJECT_ID.iam.gserviceaccount.com" \ --role="roles/monitoring.viewer" gcloud projects add-iam-policy-binding PROJECT_ID \ --member="serviceAccount:causely@PROJECT_ID.iam.gserviceaccount.com" \ --role="roles/bigtable.viewer" ``` #### Step 5b: Create and Download Service Account Key Create a key for the Service Account and download it: ```bash gcloud iam service-accounts keys create service_account.json \ --iam-account=causely@PROJECT_ID.iam.gserviceaccount.com ``` #### Step 5c: Create a Kubernetes Secret Save the Service Account credentials in a Kubernetes Secret: ```bash kubectl create secret --namespace causely generic gcp-credentials \ --from-file=service_account.json=service_account.json \ --from-literal=project_id=PROJECT_ID ``` #### Step 5d: Update Causely Configuration Update your `causely-values.yaml` to enable GCP scraping: ```yaml global: cluster_name: mediator: gateway: token: scrapers: gcp: enabled: true projects: - secretName: gcp-credentials namespace: causely ``` Then upgrade your installation: ```bash helm upgrade --install causely \\ --namespace=causely \\ --values causely-values.yaml \\ oci://us-docker.pkg.dev/public-causely/public/causely \\ --version "${CAUSELY_VERSION}" ``` :::tip Alternative: Credentials Autodiscovery You can also use credentials autodiscovery by labeling the secret: ```bash kubectl --namespace causely label secret gcp-credentials "causely.ai/scraper=GCP" ``` This allows you to add new projects without editing the configuration. See the [GCP integration documentation](/telemetry-sources/gcp/) for more details. ::: For complete setup instructions and information about which GCP resources are supported, see the [GCP integration documentation](/telemetry-sources/gcp/). ## What to Expect Once configured, Causely will: - **Monitor Kubernetes services**: Using eBPF instrumentation, Causely automatically discovers and monitors services running in your Kubernetes cluster. - **Monitor GCP resources**: Causely will discover and monitor GCP resources including: - Compute Engine instances - Cloud SQL databases - Cloud Storage - Bigtable - And more - **Unified view**: Causely creates a unified topology showing relationships between your Kubernetes services and GCP resources. - **Show data in the UI**: Within a few minutes, you should start seeing entities appearing in the Causely UI at [https://portal.causely.app](https://portal.causely.app). ## Add More Telemetry Sources To help Causely infer Diagnoses more effectively, connect additional telemetry sources. Visit the [Telemetry Sources](/telemetry-sources) page to learn about the data sources that Causely supports, including: - [Prometheus](/telemetry-sources/prometheus/) - [OpenTelemetry](/telemetry-sources/opentelemetry/) - [Grafana](/telemetry-sources/grafana/) - [Alertmanager](/telemetry-sources/alertmanager/) - And many more ## Add Workflow Integrations Push insights into your existing workflows and notification tools. Visit the [Workflow Integrations](/workflows) page to set up integrations with: - [Slack](/workflows/slack/) - [Grafana](/workflows/grafana-plugin/) - [incident.io](/workflows/incident-io/) - [Microsoft Teams](/workflows/microsoft-teams/) - [Prometheus Alertmanager](/workflows/prometheus-alertmanager/) - And more ## What's Your Goal? , title: 'Accelerate resolution', description: 'Get precise Diagnosis insights during incidents to resolve issues faster.', }, { icon: , title: 'Proactively prevent incidents', description: 'Identify emerging risks before they impact your services and SLOs.', }, { icon: , title: 'Understand system behavior', description: 'Gain deep insights into why your systems behave the way they do.', }, { icon: , title: 'Scale reliability', description: 'Maintain reliability across complex, fast-changing systems at scale.', }, { icon: , title: 'Optimize performance', description: 'Continuously improve system performance with actionable insights.', }, ]} teaser='Choose a goal to explore how Causely can help you achieve it.' /> :::tip Other Platforms Supported While this quickstart guide focuses on Kubernetes, Causely also supports: - **Container Orchestration**: [Nomad](/installation/nomad/), Docker, ECS - **GitOps**: [Argo CD](/installation/argocd/), [Flux](/installation/flux/) - **Virtual Machines**: Direct installation on VMs For detailed installation instructions for these platforms, visit the [Installation Overview](/installation) page. ::: --- ## Instana Quickstart Guide This quickstart guide will help you get Causely up and running with Instana. This setup allows Causely to ingest service dependencies, traces, and metrics from Instana to build a unified causal model. ## Get Your Installation Command 1. Log in to the [Causely portal](https://portal.causely.app). 2. Visit the [mediators page](https://portal.causely.app/agents). 3. Click the **"Add new"** button (or **"Add ➕"** button) on the mediators page. 4. In the instructions panel, you'll see the Helm installation command with your access token pre-filled. 5. Copy the complete Helm command. It will look similar to the example below. :::info Example Command The command you copy will include your specific access token and cluster name. Here's what it typically looks like: ```bash export CAUSELY_TOKEN= export CAUSELY_CLUSTER_NAME= export CAUSELY_VERSION= helm upgrade --install causely \\ --create-namespace oci://us-docker.pkg.dev/public-causely/public/causely \\ --version "${CAUSELY_VERSION}" \\ --namespace=causely \\ --set image.tag="${CAUSELY_VERSION}" \\ --set global.cluster_name="\${CAUSELY_CLUSTER_NAME}" \\ --set mediator.gateway.token="\${CAUSELY_TOKEN}" ``` ::: :::info Screenshot Placeholder Screenshots of the mediators page, "Add new" button, and instructions panel will be added here. ::: ## Create a Values File to Disable eBPF Since you'll be using Instana for instrumentation, create a `causely-values.yaml` file to disable eBPF instrumentation: ```yaml global: cluster_name: mediator: gateway: token: scrapers: bpf: enabled: false ``` Replace `` and `` with your actual values from the portal. ## Run the Installation Command 1. Open a terminal with `kubectl` configured to access your cluster. 2. Run the Helm command with your values file: ```bash export CAUSELY_VERSION= helm upgrade --install causely \\ --create-namespace oci://us-docker.pkg.dev/public-causely/public/causely \\ --version "${CAUSELY_VERSION}" \\ --namespace=causely \\ --set image.tag="${CAUSELY_VERSION}" \\ --set global.cluster_name="" \\ --set mediator.gateway.token="" \\ --values causely-values.yaml ``` 3. Wait for the installation to complete: ```bash kubectl wait --for=condition=Ready pod -l app.kubernetes.io/part-of=causely -n causely --timeout=300s ``` ## Configure Instana Integration Now you need to configure Causely to scrape data from your Instana account. #### Step 6a: Create an API Token in Instana 1. Log in to your Instana instance. 2. Create an API token with read access to applications, endpoints, metrics, and infrastructure. #### Step 6b: Create a Kubernetes Secret Create a Kubernetes secret with your Instana URL and API token: ```bash kubectl create secret generic \ --namespace causely instana-credentials \ --from-literal=url="https://your-instana.example.com" \ --from-literal=api-token="" ``` If your Instana instance uses a self-signed certificate, you can optionally add: ```bash --from-literal=insecure-skip-verify="true" ``` #### Step 6c: Update Causely Configuration Update your `causely-values.yaml` to enable Instana scraping: ```yaml global: cluster_name: mediator: gateway: token: scrapers: bpf: enabled: false instana: enabled: true accounts: - secretName: instana-credentials # namespace: causely # optional; defaults to the pod namespace ``` Then upgrade your Causely installation: ```bash helm upgrade --install causely \\ --namespace=causely \\ --values causely-values.yaml \\ oci://us-docker.pkg.dev/public-causely/public/causely \\ --version "${CAUSELY_VERSION}" ``` :::tip Alternative: Credentials Autodiscovery You can also use credentials autodiscovery by labeling the secret: ```bash kubectl --namespace causely label secret instana-credentials "causely.ai/scraper=Instana" ``` This allows you to add new accounts without editing the configuration. See the [Instana integration documentation](/telemetry-sources/instana/) for more details. ::: ## What to Expect Once configured, Causely will: - **Ingest Instana data**: Causely will scrape service dependencies, traces, metrics, and topology from your Instana account. - **Discover service dependencies**: Causely automatically discovers service dependencies and maps them to Kubernetes workloads. - **Collect endpoint metrics**: HTTP paths, RPC methods, and their metrics (requests, errors, latency) are collected and analyzed. - **Database insights**: Database queries and access patterns are discovered and mapped to services. - **Show data in the UI**: Within a few minutes, you should start seeing services, endpoints, and their relationships appearing in the Causely UI at [https://portal.causely.app](https://portal.causely.app). For more details on what data is collected, see the [Instana integration documentation](/telemetry-sources/instana/). ## Add More Telemetry Sources To help Causely infer Diagnoses more effectively, connect additional telemetry sources. Visit the [Telemetry Sources](/telemetry-sources) page to learn about the data sources that Causely supports, including: - [Prometheus](/telemetry-sources/prometheus/) - [OpenTelemetry](/telemetry-sources/opentelemetry/) - [Grafana](/telemetry-sources/grafana/) - [Alertmanager](/telemetry-sources/alertmanager/) - And many more ## Add Workflow Integrations Push insights into your existing workflows and notification tools. Visit the [Workflow Integrations](/workflows) page to set up integrations with: - [Slack](/workflows/slack/) - [Grafana](/workflows/grafana-plugin/) - [incident.io](/workflows/incident-io/) - [Microsoft Teams](/workflows/microsoft-teams/) - [Prometheus Alertmanager](/workflows/prometheus-alertmanager/) - And more ## What's Your Goal? , title: 'Accelerate resolution', description: 'Get precise Diagnosis insights during incidents to resolve issues faster.', }, { icon: , title: 'Proactively prevent incidents', description: 'Identify emerging risks before they impact your services and SLOs.', }, { icon: , title: 'Understand system behavior', description: 'Gain deep insights into why your systems behave the way they do.', }, { icon: , title: 'Scale reliability', description: 'Maintain reliability across complex, fast-changing systems at scale.', }, { icon: , title: 'Optimize performance', description: 'Continuously improve system performance with actionable insights.', }, ]} teaser='Choose a goal to explore how Causely can help you achieve it.' /> :::tip Other Platforms Supported While this quickstart guide focuses on Kubernetes, Causely also supports: - **Container Orchestration**: [Nomad](/installation/nomad/), Docker, ECS - **GitOps**: [Argo CD](/installation/argocd/), [Flux](/installation/flux/) - **Virtual Machines**: Direct installation on VMs For detailed installation instructions for these platforms, visit the [Installation Overview](/installation) page. ::: --- ## Kubernetes Quickstart Guide This quickstart guide will help you get Causely up and running on your Kubernetes cluster in just a few minutes. By the end of this guide, you'll have Causely installed and collecting telemetry data from your cluster. ## Get Your Installation Command 1. Log in to the [Causely portal](https://portal.causely.app). 2. Visit the [mediators page](https://portal.causely.app/agents). 3. Click the **"Add new"** button (or **"Add ➕"** button) on the mediators page. 4. In the instructions panel, you'll see the Helm installation command with your access token pre-filled. 5. Copy the complete Helm command. It will look similar to the example below. :::info Example Command The command you copy will include your specific access token and cluster name. Here's what it typically looks like: ```bash export CAUSELY_TOKEN= export CAUSELY_CLUSTER_NAME= export CAUSELY_VERSION= helm upgrade --install causely \\ --create-namespace oci://us-docker.pkg.dev/public-causely/public/causely \\ --version "${CAUSELY_VERSION}" \\ --namespace=causely \\ --set image.tag="${CAUSELY_VERSION}" \\ --set global.cluster_name="\${CAUSELY_CLUSTER_NAME}" \\ --set mediator.gateway.token="\${CAUSELY_TOKEN}" ``` ::: :::info Screenshot Placeholder Screenshots of the mediators page, "Add new" button, and instructions panel will be added here. ::: ## Run the Installation Command 1. Open a terminal with `kubectl` configured to access your cluster. 2. Paste and run the Helm command you copied from the portal. 3. Wait for the installation to complete. You can monitor the progress with: ```bash kubectl wait --for=condition=Ready pod -l app.kubernetes.io/part-of=causely -n causely --timeout=300s ``` ## What to Expect Once the installation is complete, Causely will automatically: - **Enable eBPF-based instrumentation**: Causely uses [OpenTelemetry eBPF instrumentation](/telemetry-sources/ebpf), powered by Grafana Beyla, to automatically instrument your applications without requiring code changes. This provides zero-effort observability for services running in your cluster. - **Start collecting telemetry**: The agent will begin discovering services, pods, and their dependencies in your cluster. - **Show data in the UI**: Within a few minutes, you should start seeing entities appearing in the Causely UI at [https://portal.causely.app](https://portal.causely.app). Services, pods, and their relationships will be automatically discovered and displayed in the topology graph. :::tip If you don't see entities appearing after a few minutes, check the mediator logs: ```bash kubectl logs -n causely \ -l app.kubernetes.io/name=mediator \ -c mediator \ --tail=-1 | grep ERROR ``` ::: ## Add More Telemetry Sources To help Causely infer Diagnoses more effectively, connect additional telemetry sources. Visit the [Telemetry Sources](/telemetry-sources) page to learn about the data sources that Causely supports, including: - [Prometheus](/telemetry-sources/prometheus/) - [OpenTelemetry](/telemetry-sources/opentelemetry/) - [Grafana](/telemetry-sources/grafana/) - [Alertmanager](/telemetry-sources/alertmanager/) - And many more ## Add Workflow Integrations Push insights into your existing workflows and notification tools. Visit the [Workflow Integrations](/workflows) page to set up integrations with: - [Slack](/workflows/slack/) - [Grafana](/workflows/grafana-plugin/) - [incident.io](/workflows/incident-io/) - [Microsoft Teams](/workflows/microsoft-teams/) - [Prometheus Alertmanager](/workflows/prometheus-alertmanager/) - And more ## What's Your Goal? , title: 'Accelerate resolution', description: 'Get precise Diagnosis insights during incidents to resolve issues faster.', }, { icon: , title: 'Proactively prevent incidents', description: 'Identify emerging risks before they impact your services and SLOs.', }, { icon: , title: 'Understand system behavior', description: 'Gain deep insights into why your systems behave the way they do.', }, { icon: , title: 'Scale reliability', description: 'Maintain reliability across complex, fast-changing systems at scale.', }, { icon: , title: 'Optimize performance', description: 'Continuously improve system performance with actionable insights.', }, ]} teaser='Choose a goal to explore how Causely can help you achieve it.' /> :::tip Other Platforms Supported While this quickstart guide focuses on Kubernetes, Causely also supports: - **Container Orchestration**: [Nomad](/installation/nomad/), Docker, ECS - **GitOps**: [Argo CD](/installation/argocd/), [Flux](/installation/flux/) - **Virtual Machines**: Direct installation on VMs For detailed installation instructions for these platforms, visit the [Installation Overview](/installation) page. ::: --- ## OpenTelemetry Quickstart Guide This quickstart guide will help you get Causely up and running with OpenTelemetry. This setup is ideal if you're already using OpenTelemetry instrumentation or want to use your existing OpenTelemetry Collector. ## Get Your Installation Command 1. Log in to the [Causely portal](https://portal.causely.app). 2. Visit the [mediators page](https://portal.causely.app/agents). 3. Click the **"Add new"** button (or **"Add ➕"** button) on the mediators page. 4. In the instructions panel, you'll see the Helm installation command with your access token pre-filled. 5. Copy the complete Helm command. It will look similar to the example below. :::info Example Command The command you copy will include your specific access token and cluster name. Here's what it typically looks like: ```bash export CAUSELY_TOKEN= export CAUSELY_CLUSTER_NAME= export CAUSELY_VERSION= helm upgrade --install causely \\ --create-namespace oci://us-docker.pkg.dev/public-causely/public/causely \\ --version "${CAUSELY_VERSION}" \\ --namespace=causely \\ --set image.tag="${CAUSELY_VERSION}" \\ --set global.cluster_name="\${CAUSELY_CLUSTER_NAME}" \\ --set mediator.gateway.token="\${CAUSELY_TOKEN}" ``` ::: :::info Screenshot Placeholder Screenshots of the mediators page, "Add new" button, and instructions panel will be added here. ::: ## Create a Values File to Configure OpenTelemetry Create a `causely-values.yaml` file with the following configuration to disable eBPF instrumentation and configure OpenTelemetry: ```yaml global: cluster_name: mediator: gateway: token: scrapers: bpf: enabled: false ``` Replace `` and `` with your actual values from the portal. ## Run the Installation Command 1. Open a terminal with `kubectl` configured to access your cluster. 2. Run the Helm command with your values file: ```bash export CAUSELY_VERSION= helm upgrade --install causely \\ --create-namespace oci://us-docker.pkg.dev/public-causely/public/causely \\ --version "${CAUSELY_VERSION}" \\ --namespace=causely \\ --set image.tag="${CAUSELY_VERSION}" \\ --set global.cluster_name="" \\ --set mediator.gateway.token="" \\ --values causely-values.yaml ``` 3. Wait for the installation to complete: ```bash kubectl wait --for=condition=Ready pod -l app.kubernetes.io/part-of=causely -n causely --timeout=300s ``` ## Configure Your OpenTelemetry Collector Now you need to configure your OpenTelemetry Collector to send traces and metrics to Causely's mediator. The mediator listens for OpenTelemetry Protocol (OTLP) data on port `4317`. If you don't have an OpenTelemetry Collector running, you can install one using the [OpenTelemetry Operator](https://opentelemetry.io/docs/platforms/kubernetes/operator/). For detailed configuration instructions, see the [OpenTelemetry integration guide](/telemetry-sources/opentelemetry/). The key configuration you'll need is to add an exporter that points to the Causely mediator: ```yaml exporters: otlp/causely: endpoint: mediator.causely:4317 compression: none tls: insecure: true ``` ## What to Expect Once configured, Causely will: - **Receive OpenTelemetry data**: Your OpenTelemetry Collector will send traces and metrics to the Causely mediator. - **Discover service dependencies**: Causely will automatically discover service dependencies from your OpenTelemetry traces. - **Show data in the UI**: Within a few minutes, you should start seeing services and their relationships appearing in the Causely UI at [https://portal.causely.app](https://portal.causely.app). :::tip For complete OpenTelemetry configuration examples, including Kubernetes attributes processing and filtering, see the [OpenTelemetry integration documentation](/telemetry-sources/opentelemetry/). ::: ## Add More Telemetry Sources To help Causely infer Diagnoses more effectively, connect additional telemetry sources. Visit the [Telemetry Sources](/telemetry-sources) page to learn about the data sources that Causely supports, including: - [Prometheus](/telemetry-sources/prometheus/) - [OpenTelemetry](/telemetry-sources/opentelemetry/) - [Grafana](/telemetry-sources/grafana/) - [Alertmanager](/telemetry-sources/alertmanager/) - And many more ## Add Workflow Integrations Push insights into your existing workflows and notification tools. Visit the [Workflow Integrations](/workflows) page to set up integrations with: - [Slack](/workflows/slack/) - [Grafana](/workflows/grafana-plugin/) - [incident.io](/workflows/incident-io/) - [Microsoft Teams](/workflows/microsoft-teams/) - [Prometheus Alertmanager](/workflows/prometheus-alertmanager/) - And more ## What's Your Goal? , title: 'Accelerate resolution', description: 'Get precise Diagnosis insights during incidents to resolve issues faster.', }, { icon: , title: 'Proactively prevent incidents', description: 'Identify emerging risks before they impact your services and SLOs.', }, { icon: , title: 'Understand system behavior', description: 'Gain deep insights into why your systems behave the way they do.', }, { icon: , title: 'Scale reliability', description: 'Maintain reliability across complex, fast-changing systems at scale.', }, { icon: , title: 'Optimize performance', description: 'Continuously improve system performance with actionable insights.', }, ]} teaser='Choose a goal to explore how Causely can help you achieve it.' /> :::tip Other Platforms Supported While this quickstart guide focuses on Kubernetes, Causely also supports: - **Container Orchestration**: [Nomad](/installation/nomad/), Docker, ECS - **GitOps**: [Argo CD](/installation/argocd/), [Flux](/installation/flux/) - **Virtual Machines**: Direct installation on VMs For detailed installation instructions for these platforms, visit the [Installation Overview](/installation) page. ::: --- ## Applications Software programs designed to perform specific tasks or functions for end-users or other systems. --- ### Business Application A logical grouping of applications within Causely to define a high-level application. Business Applications are constructed by default from the pod label `app.kubernetes.io/part-of` but this scoping mechanism can be changed. Causely creates Business Applications automatically upon initially discovery of the environment. --- ### Application An Application is a set of code and configurations designed to perform a specific function. These applications are typically containerized, scalable, and decoupled from the underlying infrastructure. They leverage cloud native principles such as microservices, statelessness, and automation to achieve agility and reliability. --- ### Application Load Balancer An Application Load Balancer (ALB) is a component of modern cloud infrastructure that distributes incoming traffic across multiple servers to ensure optimal resource use, reliability, and scalability. It makes intelligent routing decisions based on factors like request content, headers, or URL paths. ALBs support features like SSL termination, WebSocket connections, and advanced routing rules, making them ideal for microservices or containerized architectures. They enhance application availability and responsiveness by evenly balancing workloads and rerouting traffic in case of server failures. --- ### Operation A Background Operation in a Kubernetes microservice is a non-blocking task that runs asynchronously and independently of user requests. These operations are commonly used for tasks like data processing, maintenance jobs, or message queue consumption. They are often managed using worker pods or cron jobs and are critical for handling long-running or resource-intensive tasks without impacting the primary application workflow. --- ## Compute Servers or cloud resources to execute tasks, run applications, or perform calculations. --- ### Allocation An Allocation in Nomad represents the deployment of a task group onto a specific client node. It is responsible for running and monitoring the tasks defined in a Nomad job. Each allocation includes detailed information about the resources used, task states, and the node where the tasks are running. Allocations are key to Nomad's distributed system architecture, ensuring workloads are placed efficiently and reliably. They also include metadata that allows for tracking, troubleshooting, and scaling operations. --- ### Compute Spec A logical group created by Causely that defines a group of containers within a deployment. For example, if you have a deployment with two containers definitions: C1 and C2 then there would be two compute specs in Causely: CS1 and CS2. Since a deployment controller may create many pods each with an instance of C1 and C2, there are multiple containers within each compute spec. --- ### Container A Container is a lightweight, standalone execution environment that encapsulates an application and its dependencies. Containers provide portability and consistency across different environments, making them essential for microservices architectures. Containers are isolated from each other and the host system, ensuring security and resource efficiency. Container Orchestration platforms like Kubernetes and Nomad enables containers to be dynamically scheduled, scaled, and orchestrated as part of a broader application architecture. --- ### Controller A Controller is a control loop that monitors the state of resources within a Kubernetes cluster and works to reconcile them with the defined state. Examples include the Deployment controller, which manages replicas of an application, and the Node controller, which handles node statuses. Controllers are integral to Kubernetes’ self-healing capabilities, automatically restarting, scaling, or reconfiguring resources as needed. --- ### Disk A disk refers to a virtualized storage resource that provides persistent or temporary data storage for cloud-based virtual machines (VMs) or workloads. Disks can be provisioned in various types, such as SSDs for high performance or HDDs for cost-effective storage. They support scalability, redundancy, and features like snapshots for backups or disaster recovery. --- ### Node A Node is a physical or virtual machine that serves as a infrastructure component in a Kubernetes cluster. Nodes run pods and are managed by the Kubernetes control plane, providing the compute, memory, and storage resources needed to host pods and containers. Each node runs essential Kubernetes components such as kubelet, kube-proxy, and a container runtime. Nodes can be scaled horizontally to handle increased workloads and are monitored to ensure stability and performance. --- ### Nomad Job A Nomad Job defines a workload to be scheduled and executed by the Nomad orchestrator. Jobs consist of task groups and task definitions, specifying resource requirements, constraints, and runtime configurations for applications. They can be used to schedule various workloads, from containerized applications to batch jobs. Nomad jobs are declarative, enabling operators to define states and allowing the system to manage deployment and scaling automatically. --- ### Persistent Volume A Persistent Volume (PV) in Kubernetes is a storage resource provisioned for a cluster, independent of any pod. PVs provide durable storage that retains data beyond the lifecycle of individual pods, supporting stateful applications. They are abstracted from the underlying storage technology, allowing seamless integration with cloud-based or on-premises storage solutions. --- ### Pod A Kubernetes Pod is a deployable unit in Kubernetes, consisting of one or more containers that share resources such as networking and storage. Pods are ephemeral and designed to host tightly coupled application components. They are often used for scaling and scheduling containers in Kubernetes. Pods can be replicated and distributed across nodes to ensure high availability and load balancing. --- ### Virtual Machine A Virtual Machine (VM) is a software-based representation of a physical computer, providing an isolated environment for running containerized applications like in Nomad architecture. VMs include a virtualized CPU, memory, storage, and network interface, allowing multiple VMs to run on a single physical machine. They enable efficient utilization of hardware resources and provide strong isolation between workloads. VMs are widely used for server consolidation, development environments, and running legacy applications. --- ### Workload A Workload refers to a single running instance of an application. Each Workload operates independently and can run concurrently with others to handle varying loads—for example, a Kubernetes Pod, a Nomad Allocation, or a managed service. Causely only includes services as Workloads when the underlying infrastructure is not discovered and they are observed communicating with other services in the environment. --- ## Databases Systems for storing, organizing, and managing structured or unstructured data. --- ### Database A Database is an organized collection of data that is stored and accessed electronically. Databases can be relational, NoSQL, or graph-based, each optimized for specific types of data and query requirements. They form the backbone for storing structured or semi-structured data in most applications. Modern databases support features such as indexing, transactions, and high availability, enabling robust and performant data management for diverse use cases. --- ### Database Server The server or managed service that hosts and runs the database, it provides the processing power, memory and network capabilities to support the database operations. The Database server often runs a database management system (DBMS) such as PostgreSQL, Oracle, or SQL, which interacts with the underlying database. --- ### Database Server Instance An individual instance of a Database Server layered over the underlying infrastructure components. Each discovered Database Server has a corresponding Database Server Instance within the topology. --- ### Cache A Cache is an in-memory key-value store that provides ultra-fast data retrieval for frequently accessed data. It is commonly used to reduce database query load, improve application performance, and support features like session storage and real-time analytics. Cache platforms such as Redis supports various data structures, such as strings, hashes, lists, and sets, making it versatile for diverse caching needs. --- ### Cache Instance A Cache Instance is a single deployment of the cache software such as Redis, configured to store and manage cached data. Instances can operate independently or as part of a cluster, supporting replication and sharding for scalability and fault tolerance. Each instance maintains its own data and can be optimized for specific workloads, such as read-heavy or write-heavy operations. --- ### Table A Database Table is a structured set of data organized into rows and columns within a relational database. Each table represents an entity or relationship, with columns defining attributes and rows storing individual records. Tables can be indexed to improve query performance and may include constraints to enforce data integrity. They are the primary building blocks for representing and manipulating structured data in applications. --- ## Data Pipeline Series of processes that move and transform data between systems for analysis, storage, or further processing. --- ### HTTP Path An HTTP Path is a component of a URL that specifies the specific resource or endpoint being requested on a web server. It determines how requests are routed and processed by the application, often forming the basis for RESTful APIs. HTTP paths can include parameters and query strings to refine requests, supporting dynamic and flexible interactions with web services. They are essential for defining clear and consistent interfaces for client-server communication. --- ### RPC Method An RPC Method (Remote Procedure Call) allows a program to execute a function or procedure on a remote server as if it were local. RPCs are widely used in distributed systems to enable communication between services. They abstract the complexities of network communication, providing a simple interface for developers. Common frameworks for RPC include gRPC, which uses Protocol Buffers for serialization, and JSON-RPC. --- ## Messaging Messaging Platforms facilitate communication between distributed applications or services via queues or topics. --- ### Broker A Broker is a server that handles the storage and transmission of messages in a Kafka cluster. Brokers manage message data for topics and distribute load across the cluster. They are key components for enabling fault tolerance and high availability in Kafka’s publish-subscribe model. Each broker communicates with producers and consumers, ensuring messages are persisted and delivered efficiently. --- ### Broker Instance A Broker Instance refers to an individual server running a Kafka broker. Each broker instance is part of a Kafka cluster, responsible for managing a subset of partitions and ensuring data replication across the cluster. Broker instances work together to balance workloads, provide redundancy, and ensure data availability even if some instances go offline. They are critical for Kafka's ability to handle high-throughput and low-latency message processing. --- ### Exchange An Exchange is a messaging component responsible for receiving messages from producers and routing them to one or more queues based on routing rules and bindings. Exchanges determine how messages are distributed across the system, supporting patterns such as direct, fanout, and topic-based routing. --- ### Queue A Queue is a messaging component that stores messages until they are consumed by a subscriber. Queues buffer messages between producers and consumers, enabling asynchronous communication and decoupling of services. Queue depth and consumer lag are key indicators of whether downstream services are keeping pace with incoming message volume. --- ### Topic A Kafka Topic is a logical channel to which producers send messages and from which consumers retrieve messages. Topics can be partitioned to enable parallel processing and ensure scalability within a Kafka cluster. Each partition is replicated across multiple brokers for fault tolerance, ensuring data is durable and accessible even in the event of broker failures. --- ## Entity Types Overview Causely automatically discovers over 25 different Entity types from your cloud native environment. These entity types are discovered through data sources like eBPF, Cloud APIs, OpenTelemetry and used to build topologies, identify defects, and infer Diagnoses. --- ## Services Services are self-contained units of functionality within a system that perform specific tasks or provide specific capabilities, often accessible through defined interfaces or APIs. --- ### AI Model An AI Model refers to an AI inference endpoint hosted outside the primary orchestration platform, such as a large language model or embedding model API. These models are accessed via network interfaces, often using HTTP or gRPC. --- ### Kubernetes Service A Kubernetes Microservice is an independently deployable component of an application running in a Kubernetes environment. These microservices communicate with each other using lightweight protocols and are designed to perform specific, granular functions, promoting modularity and scalability. --- ### Network Endpoint A network endpoint refers to a specific point of interaction where services, applications, or devices connect to a network to send or receive data. It can be a URL, IP address, or a combination of both that allows communication between clients and servers, such as microservices, APIs, or databases. --- ### Service A Service refers to a service or application hosted outside the primary orchestration platform, such as a third-party API or legacy system. These services are accessed via network interfaces, often using HTTP or gRPC. --- ## Applications(Root-causes) Software programs designed to perform specific tasks or functions for end-users or other systems. :::note In the Causely user interface, these Diagnoses appear under the corresponding application entity. ::: {/* ============================================================================ AUTO-GENERATED CONTENT BELOW Generated from: data/root-causes.yaml Script: scripts/references/generate-pages.js To regenerate: yarn generate:references Note: This runs automatically during the build process. Do not manually edit below this line - changes will be overwritten. ============================================================================ */} ## .NET Unhandled Exception .NET logs show an unhandled exception, indicating the application failed unexpectedly and may have terminated the process. Unhandled exceptions in .NET typically indicate application logic errors, invalid state, or dependency failures that were not caught. Depending on hosting mode, this can fail the current request, terminate a worker loop, or crash the process entirely. ## Access Throttled The application is receiving HTTP 429 "Too Many Requests" responses, indicating that it has exceeded the rate limits set by the other service. This can cause degraded functionality, slow performance, or temporary service unavailability for end users. HTTP 429 errors are typically triggered when an API or service imposes rate limits to control the volume of incoming requests. ## Apache Worker Exhausted Apache HTTPD has reached its MaxRequestWorkers limit, meaning all worker slots are occupied and new requests cannot be processed. Apache prefork and event MPMs log "server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting" when all worker slots are busy. New connections are queued up to ListenBacklog, then dropped. This indicates either a load spike or slow request processing holding workers. ## Application Error Spike The service is emitting a high volume of ERROR or CRITICAL log lines over a rolling window, indicating an active application-side failure with no matching specific log signature. Common causes include schema or payload mismatches after a deploy, repeated business-logic exceptions, dependency failures logged as generic errors, and application regressions that produce many error-level logs without a canonical runtime signature. This signal indicates the application is actively failing rather than providing a precise diagnosis of the underlying cause. This root cause activates when generic error-severity log volume exceeds 2,000 occurrences within a one-hour window. ## Cassandra Tombstone Pressure Cassandra reads are scanning excessive tombstones, indicating partition design or TTL issues that are causing significant read latency and resource pressure. Cassandra logs a warning when a read scans more tombstones than tombstone_warn_threshold (default 1000). Tombstones are markers for deleted data that must be scanned until compaction removes them. Excessive tombstones cause read amplification, increased GC pressure, and can trigger ReadTimeoutExceptions. ## Circuit Breaker Open A circuit breaker protecting an upstream dependency has opened, causing the service to fail fast on calls to that dependency instead of waiting for timeouts. Circuit breakers such as Resilience4j and Netflix Hystrix open when an upstream dependency exceeds a configured failure rate or slow-call threshold. While open, all calls to the dependency are immediately rejected to prevent cascading failures and reduce latency. This is a protective mechanism, but it surfaces as errors for callers. ## Connection Pool Exhausted The database or service connection pool is exhausted, and new connection requests are being rejected, causing application errors for all callers. Connection pool exhaustion occurs when all connections in the pool are in use and no connection becomes available within the timeout. Sources include PostgreSQL "sorry, too many clients already", MySQL error 1040, Redis "max number of clients reached", HikariCP timeout, or pgBouncer limit. This typically indicates a connection leak, long-running transactions holding connections, or insufficient pool sizing for the load. ## Database Connection Pool Saturated The client-side database connection pool is exhausted when all available connections are in use, preventing new database queries from being executed. This can cause application requests to hang or fail, impacting user experience and potentially leading to downtime for database-dependent features. ## Database Malfunction The database is returning a high rate of errors or failing to respond to queries, causing disruptions for services and clients that depend on it. This may result in delayed or failed access to one or more tables, leading to degraded application performance, elevated latency, or complete unavailability of database-backed functionality. ## Disk Full A filesystem used by this service has run out of space (ENOSPC), causing write operations to fail. "No space left on device" errors occur when a write() syscall fails because the filesystem has no free blocks. Logs, data files, temp files, or WAL segments may all contribute to disk exhaustion. This causes immediate write failures and can cause the process to crash or enter a degraded state. ## Elasticsearch Cluster Unhealthy The Elasticsearch cluster health status has transitioned to RED, indicating one or more primary shards are unassigned and data is unavailable for those shards. Elasticsearch reports RED health when at least one primary shard is unassigned. Search and indexing requests for affected indices will fail. This is caused by node failures, insufficient nodes to satisfy the index replication factor, or shard allocation issues. ## Excessive DNS Traffic from Client The application is generating an unusually high volume of DNS requests, potentially overwhelming DNS servers, increasing latency for users, and risking service disruptions. This behavior may also incur additional costs or trigger rate-limiting from DNS providers. This typically arises when the application initiates DNS lookups more frequently than necessary due to lack of effective caching, redundant DNS resolution logic, or misconfigurations. ## File Descriptor Exhaustion The application has reached the system-imposed limit on the number of file descriptors it can open. This typically leads to errors such as "Too many open files," preventing the application from creating new connections, reading files, or accessing resources. This can severely impact functionality, particularly in high-concurrency or high-I/O scenarios. ## File Limit Exhausted The process has reached the operating system file descriptor limit (EMFILE/ENFILE), preventing it from opening new network connections, files, or sockets. Each open socket, file, or pipe consumes a file descriptor. When the per-process limit (ulimit -n / RLIMIT_NOFILE) or system-wide limit (/proc/sys/fs/file-max) is reached, new open/accept/connect calls fail with EMFILE or ENFILE. This manifests as connection refused errors in network servers or file-open failures in data pipelines. ## Go Deadlock The Go runtime has detected a full deadlock where all goroutines are permanently blocked, causing the process to panic and exit. The Go runtime prints "all goroutines are asleep - deadlock!" and exits when it determines that no goroutine can ever make progress. This is a fatal condition and the process terminates immediately. Common causes include channel send/receive with no corresponding partner, mutex lock with no unlock, or sync.WaitGroup misuse. ## Go Nil Pointer Panic The Go runtime has panicked due to an invalid memory address or nil pointer dereference, causing the handler or process to fail. This panic occurs when code dereferences a nil pointer. In Go services, an unhandled panic usually terminates the current goroutine and may crash the whole process unless recovered. Common causes include missing dependency initialization, nil interface assumptions, unchecked map or pointer fields, or races around object lifecycle. ## GOMAXPROCS Misconfigured The environment variable GOMAXPROCS, which controls the maximum number of CPU cores the Go runtime uses, has been set higher than the CPU limit of the container in which the Go application is running. This mismatch can lead to inefficient CPU usage, reduced performance, and potential throttling because the Go runtime attempts to schedule more work than the container is permitted to handle. ## HAProxy Backend Unavailable HAProxy has no available backend servers, as all backends are either down or at their connection limit, causing all incoming requests to be rejected. HAProxy logs "backend has no server available" when every server in the backend pool is in DOWN state or has reached its maxconn limit. Clients receive a 503 Service Unavailable. This can be caused by all backend servers being unhealthy, a misconfigured health check, or a connection storm saturating backends. ## Inefficient DNS Lookup The application is making an unusually high volume of DNS requests, with over 80% returning NXDomain (non-existent domain) responses. This excessive DNS activity is adding 10 to 20 ms of latency to each request, negatively impacting service performance. The issue is often caused by the service or application attempting to resolve incomplete or unqualified domain names. ## Inefficient Garbage Collection The application is experiencing high latency and reduced throughput because a significant portion of its runtime is being spent in garbage collection (GC). This leads to frequent pauses, degrading overall performance and causing delays in request handling through all the dependent services. This issue usually occurs when the Java Virtual Machine (JVM) or other garbage-collected runtime environments are under memory pressure. ## Invalid Client Certificate The application is failing to connect to a service due to invalid certificate errors, preventing secure communication over HTTPS or TLS. This can cause downtime or degraded functionality for users relying on this service. ## Java GC Pressure The JVM garbage collector is under pressure, logging allocation failures or to-space exhaustion events, indicating the heap is too small or object allocation rates are too high. GC pressure manifests as "Allocation Failure" in ParallelGC/SerialGC logs or "to-space exhausted" in G1GC logs. Both indicate that GC cannot reclaim space fast enough to keep up with allocation demand. This leads to longer GC pauses, increased latency, and potential OutOfMemoryError if unresolved. ## Java Heap Saturated The JVM is operating with limited available heap memory, resulting in degraded performance or potential application crashes. This condition typically leads to frequent or prolonged garbage collection (GC) pauses, slow response times, and, in severe cases, OutOfMemoryError. It often reflects memory leaks, improper heap sizing, or excessive object allocation. ## Java Null Pointer The JVM has thrown java.lang.NullPointerException, indicating application code dereferenced a null reference and failed unexpectedly. A NullPointerException typically means application logic assumed an object was present when it was actually null. Depending on exception handling, this can fail an individual request or crash the process. Common causes include missing dependency wiring, invalid state transitions, bad deserialization assumptions, or unguarded optional values. ## Java Out of Memory The JVM has thrown java.lang.OutOfMemoryError, indicating the heap, metaspace, or GC overhead limit is exhausted and the process cannot allocate memory. OutOfMemoryError occurs when the JVM cannot satisfy an allocation request. Common subtypes include heap space (object allocation failed), Metaspace (class metadata exhausted), GC overhead limit exceeded (GC spending over 98% of time reclaiming less than 2% of heap), or unable to create native thread. The process may continue in a degraded state or crash. ## Java Stack Overflow The JVM has thrown java.lang.StackOverflowError due to runaway recursion or an excessively deep call stack that has exhausted the thread stack space. Each thread has a fixed stack size (-Xss). When recursive calls or deep call chains exceed this limit, StackOverflowError is thrown. Unlike OutOfMemoryError, this is usually a code defect such as infinite recursion or mutual recursion without a base case, or a framework issue with excessive proxy/interceptor wrapping. ## Java Thread Pool Exhausted The JVM thread pool is saturated and rejecting new task submissions with java.util.concurrent.RejectedExecutionException. ThreadPoolExecutor throws RejectedExecutionException when both the thread pool is at max capacity and the task queue is full and the rejection policy fires (default: AbortPolicy). This causes request handlers to fail, leading to increased error rates for callers. ## Kafka ISR Shrink A Kafka broker has logged ISR (In-Sync Replica) shrinkage, indicating that one or more follower replicas have fallen behind the leader and been removed from the ISR set. Kafka logs "ISR shrunk from [X] to [Y]" when a follower fails to keep up with the leader within replica.lag.time.max.ms. While a replica is out of the ISR, the effective replication factor is reduced, increasing the risk of data loss if the leader fails. Producers with acks=all will also experience increased latency or errors. ## Kafka Partition Outage Kafka has one or more offline partitions, meaning the cluster cannot maintain a healthy leader for those partitions and affected topic data is currently unavailable. This is a hard failure condition rather than a performance degradation: produce requests may be rejected and consumers cannot fetch from affected partitions until leadership is restored. Common causes include a broker outage that simultaneously removes leaders and replicas from the in-sync replica set, replication falling far enough behind that no eligible replica can be elected as leader, controller instability during broker flaps or cluster reconfiguration, and storage corruption or severe disk failure on a broker node. ## Kafka Replication Degraded One or more Kafka partitions are under-replicated, meaning followers are not keeping pace with the partition leader. This reduces fault tolerance across the cluster: if an additional broker is lost or a leader change is forced, the partition may become unavailable. The cluster may still be serving traffic, but producers and consumers are operating with reduced resilience. Common causes include disk I/O saturation on a follower broker slowing replica fetch or log flush, network congestion between brokers, a follower broker that is restarting, stuck, or under heavy load, and replication traffic competing with high-volume client produce traffic. ## Kafka Storage Pressure Kafka broker log storage utilization is high, causing degraded performance across produce, replication, and retention paths. Kafka depends on sequential log I/O, and as data-log disks approach capacity, flush, compaction, and replica synchronization operations become less efficient. This typically creates a starvation pattern of increasing broker latency, slower replication, and delayed producer acknowledgements rather than an immediate hard failure. Common causes include topic retention policies keeping more data than the disk budget supports, write rate growth exceeding storage throughput, a log compaction backlog consuming disk bandwidth, and uneven partition placement concentrating data on a single broker. ## Lock Contention The application suffers from inefficient locking, where suboptimal lock management leads to excessive contention and prolonged mutex wait times. This inefficiency degrades performance by increasing the risk of thread starvation under heavy load. This can stem from overuse of locks, coarse-grained locking strategies, or improper lock design. ## Memory Exhaustion The Broker application has exhausted its available memory, resulting in degraded performance and potential service disruption. When memory usage reaches critical levels, the system may experience increased garbage collection (GC) activity, higher processing latency, and, in severe cases, OutOfMemoryError events that halt message processing. ## MongoDB Connections Exhausted MongoDB is at or near its maximum connection limit, and new client operations may fail when they cannot obtain a server connection. MongoDB enforces a cap on concurrent active connections, and when that limit is reached, clients cannot establish or reuse connections fast enough, causing requests to fail before any database work is attempted. Common causes include application connection pools configured with an excessively high maximum size, connection leaks where clients open connections without properly releasing them, traffic spikes that create more concurrent clients than the server can handle, and long-running queries that hold connections for extended periods. ## MongoDB Cursor Pressure MongoDB has an unusually high number of open cursors, increasing server-side state management overhead and slowing query execution for callers. Open cursors represent active or partially consumed query result streams, and when too many remain open simultaneously, MongoDB must maintain more concurrent state and expend additional resources servicing them. This typically manifests as increasing latency and resource starvation rather than immediate request failures. Common causes include clients reading through large result sets slowly, applications that do not fully consume or explicitly close cursors, excessive polling or fan-out query patterns that create many simultaneous cursors, and batch sizes configured too small, which keeps individual cursors open longer than necessary. ## MongoDB Replica Lag MongoDB replication lag is high, meaning secondaries are applying oplog entries noticeably behind the primary. Read queries directed to secondaries may return stale data, and the replica set's readiness to complete a clean failover is reduced. The database continues to operate, but replication cannot keep pace with the primary write rate, degrading both read consistency and fault tolerance. Common causes include secondary nodes with insufficient disk or CPU resources for the current oplog apply rate, network latency or bandwidth constraints between replica set members, bursty write workloads that generate oplog traffic faster than secondaries can consume, and long-running workloads on secondaries that compete with the replication apply process. ## MySQL Deadlock MySQL InnoDB has detected a deadlock between two or more transactions and rolled one back to break the cycle. MySQL InnoDB logs "Deadlock found when trying to get lock; try restarting transaction" when two transactions hold locks that the other needs. The engine automatically selects a victim and rolls it back. Frequent deadlocks indicate lock acquisition order inconsistencies in application code. ## Nginx Upstream Timeout Nginx is logging upstream connection timeouts, indicating a backend service is not responding within the configured proxy_read_timeout or proxy_connect_timeout. Nginx logs "upstream timed out (110: Connection timed out)" when a backend fails to respond within the configured timeout. This causes nginx to return a 504 Gateway Timeout to the caller. The upstream service may be overloaded, deadlocked, or experiencing a network partition. ## Nginx Worker Connections Exhausted Nginx has exhausted its worker_connections limit and cannot accept new connections, meaning incoming requests are being dropped. Nginx's worker_connections directive limits the number of simultaneous connections per worker process. When all connections are in use, nginx logs "worker_connections are not enough" and new connections are refused. Total capacity equals worker_processes multiplied by worker_connections. ## Noisy Client The application acts as a Noisy Client, generating high number of requests that burden destination services with increased load and elevated request rates. This aggressive request pattern directly impacts destination services by driving a high request rate that can overwhelm service capacity and contributing to increased load on the destination. ## Postgres Cache Hit Rate Degraded Postgres shared_buffers cache hit rate has dropped, causing queries to read from disk and significantly increasing query latency. Postgres uses shared_buffers as an in-memory page cache for table and index data, and when the working set fits in shared_buffers queries are served from memory, but when the cache hit rate drops Postgres must read data from disk or OS page cache, which is orders of magnitude slower and causes query latency to increase for all clients. Common causes include shared_buffers being too small for the working set size, new query patterns accessing large table scans that evict hot pages, database growth causing the working set to exceed available memory, sequential scans on large tables, and a cold cache after a server restart until the working set is loaded. ## Postgres Checkpoint I/O Pressure Postgres checkpoint I/O write time is high, indicating disk saturation that increases latency for all database operations. Postgres periodically performs checkpoints to flush dirty pages from shared_buffers to disk to ensure durability, and when checkpoint_write_time is high the disk is being saturated by checkpoint I/O, which competes with query I/O for reads and writes and causes elevated query latency for all clients. It may also indicate that checkpoints are too infrequent, leading to large bursts of write I/O. Common causes include a high WAL write rate generating many dirty pages between checkpoints, checkpoint_completion_target being too low and concentrating I/O into a short burst, disk I/O throughput being insufficient for the write rate, min_wal_size or max_wal_size being too small and causing frequent checkpoints, and shared storage throughput limits being hit. ## Postgres Connection Slots Exhausted Postgres active connections are approaching or have reached max_connections, and new connection attempts from services will be rejected. Postgres uses a process-per-connection model and limits total connections via max_connections, and when this limit is reached new connection attempts receive "FATAL: sorry, too many clients already," causing services that cannot connect to fail their database operations. Connections also consume shared memory, so running near the limit causes additional resource pressure. Common causes include no connection pooler being in use, connection pools being misconfigured with too high a max_size, connection leaks where connections are opened but not properly closed, sudden traffic spikes creating more service instances than the pool can handle, and long-running queries holding connections that should be idle. ## Postgres Deadlock PostgreSQL has detected a deadlock cycle between concurrent transactions and terminated one to resolve it. PostgreSQL logs "ERROR: deadlock detected" when its deadlock detector finds a cycle in the lock wait graph. One transaction is chosen as the victim and receives an error and must be retried. Frequent deadlocks degrade throughput and cause user-visible errors. ## Postgres Deadlock Storm Postgres is experiencing a high rate of deadlocks, causing transaction rollbacks and forcing retries in upstream services. A deadlock occurs when two or more transactions each hold a lock the other needs, creating a circular dependency, and Postgres detects this and aborts one transaction with "ERROR: deadlock detected." The aborted transaction must be retried by the application, and a high deadlock rate causes elevated error rates, retry traffic amplification, and query latency spikes. Common causes include application code acquiring locks in inconsistent order across transactions, bulk UPDATE or DELETE operations without consistent row ordering, missing explicit locking where the application assumes order of operations, and high-concurrency workloads on the same rows without retry logic. ## Postgres Idle-in-Transaction Accumulation Postgres has idle-in-transaction sessions accumulating, and these sessions hold locks, prevent autovacuum, and cause compounding table bloat and lock contention. An idle-in-transaction session has started a transaction with BEGIN but is not actively executing any query, and it may be waiting for application-side processing, a network call, or simply be a leaked connection. These sessions hold all locks acquired during the transaction and prevent autovacuum from cleaning dead rows, and as they accumulate they cause lock contention, connection slot exhaustion, and eventually table bloat that degrades query performance. Common causes include the application opening a transaction and making an external API call or sleeping, an ORM or framework starting a transaction at request start but not committing promptly, a connection pool returning connections that were left in a transaction, and missing idle_in_transaction_session_timeout configuration. ## Postgres Lock Contention Postgres has a high number of sessions waiting to acquire locks, indicating contention from long-running or conflicting transactions. Lock contention occurs when multiple transactions attempt to modify the same rows or tables simultaneously, and sessions that cannot acquire the lock wait in the lock queue, consuming a connection slot and blocking progress. As waiting sessions accumulate, query latency rises and eventually all connection slots may be consumed. Common causes include long-running transactions holding row or table locks, missing or overly broad UPDATE or DELETE statements without appropriate WHERE clauses, idle-in-transaction sessions holding locks without doing any work, DDL operations such as ALTER TABLE or VACUUM FULL taking exclusive locks, and high-concurrency write patterns on hot rows such as counters or status fields. ## Postgres Query Memory Spill to Disk Postgres is writing temporary files because queries exceed work_mem, causing significant I/O overhead and query slowdowns. When a sort, hash join, or aggregate operation requires more memory than work_mem allows, Postgres spills intermediate data to temporary files on disk. Disk I/O for temp files is much slower than in-memory operations, and temp files compete with other I/O on the same storage. High temp file creation rates indicate queries are consistently exceeding the configured memory budget. Common causes include work_mem being set too low for actual query complexity and result set sizes, queries using ORDER BY, GROUP BY, or JOIN without appropriate indexes, many concurrent sessions each using their full work_mem allocation, and queries with multiple sort or hash operations where each gets its own work_mem allocation. ## Postgres Replication Lag Postgres standby replication lag is high, causing replica reads to return significantly stale data. Postgres streaming replication sends WAL records from primary to standbys, and replication lag is the delay between a write being committed on the primary and the standby applying it. High lag means read queries directed to replicas return stale results, which can cause data consistency issues for applications that read their own writes via replicas. Common causes include network bandwidth saturation between primary and standby, a standby I/O bottleneck where disk cannot apply WAL as fast as it arrives, long-running queries on the standby blocking WAL apply, the primary generating WAL faster than the network can deliver it, and the standby being under-provisioned relative to the primary write rate. ## Producer Publish Rate Spike The application is publishing messages at a rate significantly higher than normal, causing queue depth to grow and producer message rate to increase. This surge in publishing activity creates backpressure and can overwhelm downstream consumers. When the application experiences a producer publish rate spike, it generates messages at an abnormally high rate that exceeds the system's normal capacity, leading to queue depth growth and propagation of congestion to downstream destinations. ## Python Unhandled Exception Python logs show an unhandled exception or fatal interpreter error, indicating the application failed unexpectedly. An unhandled Python exception usually emits a traceback and terminates the active request, worker, or process. Common causes include missing input validation, bad assumptions about returned objects, dependency failures that are not caught, or unexpected runtime state. ## RabbitMQ Resource Alarm RabbitMQ has triggered a memory or disk resource alarm, and all publishers are blocked until the watermark is cleared. RabbitMQ sets a memory alarm when used_memory exceeds vm_memory_high_watermark (default: 40% of available RAM) and a disk alarm when free disk space falls below disk_free_limit. While any alarm is active, all connections that have published are blocked, causing producer services to stall. ## Rate Limited The service logs show repeated rate-limiting errors, indicating that the service or an upstream dependency is rejecting requests because a rate limit has been reached. Rate limiting is enforced by the service itself or by an upstream dependency such as an API gateway, an ingestion pipeline, or an external API. When the limit is exceeded, new requests are rejected or throttled, causing increased error rates for callers. Common sources of this pattern include ingestion services such as Grafana Tempo when write throughput exceeds the per-tenant ingestion limit, API gateways or proxies such as the nginx ngx_http_limit_req module when request rate exceeds the configured burst, and application-level rate limiters logging rate limit exceeded or rate limit reached when quota is consumed. The root cause is typically insufficient quota allocation, a sudden traffic spike, or a misconfigured rate limit policy rather than a code defect. ## Redis Cache Miss Storm Redis cache hit rate has dropped significantly, and upstream services are falling back to the database on most requests, multiplying database load. A cache miss storm occurs when Redis cannot serve requests from memory and callers bypass it to the backing database, dramatically increasing database query volume and latency for all services sharing that database. Common causes include memory pressure and key evictions, cache invalidation or a flush clearing a large portion of the keyspace, a cold start after the cache was restarted and not yet warmed, keys expiring simultaneously in a TTL cliff, and an application bug writing cache keys with wrong names that prevents future hits. ## Redis Command Queue Saturation Redis single-threaded command pipeline is saturated, and slow O(N) commands are blocking the event loop while BLPOP or BRPOP clients accumulate, causing upstream timeouts. Redis processes commands on a single thread, and when a slow command such as KEYS, SMEMBERS, SORT, or LRANGE on a large collection runs, it blocks all other commands for its duration. Clients waiting on BLPOP or BRPOP accumulate because they cannot be served while the thread is busy, and as blocked client count rises upstream services begin timing out. This root cause requires both signals to be present simultaneously: redis_slowlog_last_id or slow command log growth, and redis_blocked_clients above threshold. Common causes include application use of KEYS * or SMEMBERS on large keysets in production, large sorted sets or lists being iterated with single-command scans, and BLPOP or BRPOP patterns with no timeout or long timeout combined with slow producers. ## Redis Connection Pool Saturated Client-side Redis connection pool exhaustion occurs when all available connections in the pool are in use, preventing new requests to Redis. This can lead to request timeouts or failures, causing application disruptions for features relying on Redis for caching, messaging, or other operations. ## Redis Memory Exhausted Redis is rejecting commands because used_memory has exceeded the configured maxmemory limit, and clients receive OOM errors and writes fail. When Redis reaches its maxmemory limit, it enforces the configured eviction policy (for example allkeys-lru or volatile-lru). If the eviction policy cannot free enough space, or if maxmemory-policy is noeviction, all write commands are rejected with "OOM command not allowed when used memory > maxmemory". ## Redis Memory Pressure Redis server memory is at or near its configured limit, causing active key evictions that degrade cache effectiveness and increase load on downstream databases. Redis evicts keys when memory usage exceeds the configured maxmemory limit, and although the eviction policy determines which keys are removed, under heavy eviction the cache hit rate drops sharply and forces callers to read from the database instead, compounding into a database load spike. Common causes include the data set growing beyond allocated memory, no TTL being set on keys and causing unbounded growth, memory fragmentation consuming effective capacity, and a sudden traffic spike filling the keyspace faster than evictions can keep up. ## Redis Server Connections Exhausted Redis has reached its maxclients limit and is rejecting new connection attempts, causing connection refused errors in upstream services. Redis enforces a maximum number of simultaneous client connections via the maxclients configuration parameter, and when this limit is reached new connection attempts are immediately rejected with "ERR max number of clients reached," causing upstream services to receive connection errors and fail their Redis-dependent operations. Common causes include maxclients being set too low for the number of services connecting to the Redis instance, connection pool misconfiguration where pools do not release idle connections, sudden traffic spikes creating many short-lived connections without pooling, and connection leaks in application code. ## Slow Consumer The application is consuming messages slower than they are produced, creating a processing bottleneck. As unprocessed messages accumulate, the system experiences increased queue lag, potential memory pressure, and downstream congestion. This often indicates that one or more instances are unable to keep up due to resource constraints, inefficient processing logic, or external dependencies. ## Slow Database Queries The application is experiencing slow database queries that lead to downstream slow consumer behavior and potential resource starvation. This condition affects instance performance, particularly when query execution times become excessively long, degrading overall system responsiveness. ## Slow Database Server Queries Warehouse congestion in Snowflake occurs when the processing capacity is overwhelmed by incoming queries, causing a significant backlog. This leads to queries being queued at high rates, indicating that the system is struggling to process them in a timely manner. The resulting resource starvation further degrades performance. ## Transaction ID Congested In databases like PostgreSQL, transaction IDs are 32-bit integers that count the number of transactions performed. High utilization occurs when the counter nears its maximum value (~2 billion transactions), requiring a wraparound to continue operation. Failure to perform routine VACUUM operations can prevent the system from marking old XIDs as reusable. ## Unauthorized Access The application is receiving numerous "Unauthorized" status codes (typically HTTP 401) when trying to access another service. This prevents the application from successfully retrieving data or performing actions, potentially causing service disruptions or degraded functionality for end users. --- ## Data Pipeline(Root-causes) Series of processes that move and transform data between systems for analysis, storage, or further processing. {/* ============================================================================ AUTO-GENERATED CONTENT BELOW Generated from: data/root-causes.yaml Script: scripts/references/generate-pages.js To regenerate: yarn generate:references Note: This runs automatically during the build process. Do not manually edit below this line - changes will be overwritten. ============================================================================ */} ## Overview - [Database Table](#database-table) - [HTTP Path](#http-path) - [RPC Method](#rpc-method) ## Database Table ### Contention on Database Table Locks The database table is experiencing an abnormally high rate of exclusive locks, preventing multiple transactions from accessing the table simultaneously. This creates a bottleneck that significantly degrades performance for all client applications depending on this table. Excessive locking typically occurs due to long-running transactions, lock contention between multiple transactions, inefficient transaction design, inappropriate isolation levels, or missing indexes leading to table scans instead of index seeks. ### Schema Change Causing Table Lock Contention The database table is experiencing an unusually high rate of Data Dictionary Lock (DDL), which blocks both read and write operations during schema modifications. This significantly impacts all client applications, causing service disruptions and performance degradation. ### Table Access Failure in Database The database table is experiencing performance degradation or errors, causing disruptions for client applications. This results in slow query response times, potential errors, and degraded service performance for systems that depend on this table. ## HTTP Path ### Faulty Error Handling in HTTP Path The HTTP path is experiencing a high rate of errors, causing disruptions for clients. This can lead to degraded performance, failed requests, or complete service unavailability, significantly affecting the user experience. ### Slow Execution in HTTP Path Handler The HTTP path is experiencing congestion, resulting in high latency for clients. This suggests that the system is unable to handle the current load efficiently, causing delays in response times. Congestion often occurs when the service receives more requests than it can handle within its capacity. ## RPC Method ### Faulty Error Handling in RPC Method The RPC Method is experiencing a high rate of errors, causing disruptions for clients. This can lead to degraded performance, failed requests, or complete service unavailability, significantly affecting the user experience. ### Slow Execution in RPC Method Handler The RPC method is experiencing congestion, resulting in high latency for clients. This suggests that the system is unable to handle the current load efficiently, causing delays in response times. Congestion often occurs when the service receives more requests than it can handle within its capacity. --- ## Infrastructure Servers or cloud resources to execute tasks, run applications, or perform calculations. {/* ============================================================================ AUTO-GENERATED CONTENT BELOW Generated from: data/root-causes.yaml Script: scripts/references/generate-pages.js To regenerate: yarn generate:references Note: This runs automatically during the build process. Do not manually edit below this line - changes will be overwritten. ============================================================================ */} ## Overview - [Compute Spec](#compute-spec) - [Container](#container) - [Controller](#controller) - [Disk](#disk) - [Network Endpoint](#network-endpoint) - [Node](#node) - [VirtualMachine](#virtualmachine) ## Compute Spec ### CPU Congested One or multiple containers in a workload are experiencing CPU congestion, leading to potential throttling. This occurs when the containers use more CPU resources than allocated, causing degraded performance, longer response times, or application crashes. CPU throttling occurs when a container exceeds its CPU quota as defined by Kubernetes or Docker. ### Crash Failure One or multiple containers of a workload has crashed with a non-zero exit code, indicating abnormal termination. This disrupts the application's functionality, leading to downtime or degraded performance depending on how the workload is designed. The non-zero exit code signifies an error during the execution of the container's process. ### Frequent Crash Failure One or multiple containers of a workload are frequently crashing with a non-zero exit code, indicating abnormal termination. This disrupts the application's functionality, leading to downtime or degraded performance depending on how the workload is designed. ### Frequent Memory Failure The application frequently runs out of memory, leading to crashes, performance degradation, or instability. This affects the application's availability and can lead to downtime or poor user experience. The issue is likely due to inefficient memory usage, such as memory leaks, excessive data loading into memory, or improper garbage collection. ### Memory Failure Containers running out of memory can lead to service crashes or degraded performance, resulting in errors for end users or failed service requests. This typically occurs when a container's allocated memory is insufficient for the workload it is handling, causing out-of-memory (OOM) errors and potential system instability. ## Container ### Ephemeral Storage Congested A container is experiencing ephemeral storage congestion when its ephemeral storage usage becomes critically high, leading to failures in operations that depend on temporary storage. This may be triggered by factors such as excessive logging, inadequate cleanup of temporary files, or unexpected bursts in data processing. ### Ephemeral Storage Noisy Neighbor A container acting as a noisy neighbor consumes excessive ephemeral storage, resulting in abnormally high storage usage and contributing to node-level disk pressure that can trigger pod evictions. This issue arises when a container consistently uses more ephemeral storage than expected. ### Memory Noisy Neighbor A container acting as a noisy neighbor consumes excessive memory, leading to abnormally high memory usage and contributing to node-level memory pressure that can trigger pod evictions. This issue occurs when a container consistently uses more memory than expected, which adversely impacts both the container and its hosting node. ## Controller ### FrequentPodEphemeralStorageEvictions A Kubernetes workload is experiencing frequent pod evictions due to ephemeral storage exhaustion. This disrupts application availability and performance, as pods are terminated when they exceed their allocated storage limits or when node-level storage is under pressure. ### Image Pull Errors Kubernetes controllers may encounter image pull errors when they cannot download container images from a registry, causing Pods to fail in starting or remain in an ImagePullBackOff state. This disrupts the deployment of applications and can affect service availability. ### Malfunction Multiple pods for a Kubernetes controller are in a "NotReady" state for an extended period, which can lead to service unavailability or degraded performance. ## Disk ### Congested The disk has reached full capacity, which prevents new data from being written and may cause applications to fail, especially those dependent on free disk space for logs, caching, or temporary files. This can also slow down or halt system operations if critical processes can no longer write to the disk. ### Inode Usage Congested The disk is experiencing inode exhaustion, meaning the file system has run out of inodes (metadata structures for file storage), which prevents new files from being created even if there is free disk space. This often causes errors in applications attempting to create files and can disrupt services reliant on file storage. ### IOPs Congested The disk is experiencing Read/Write Operations Per Second (IOPS) congestion, meaning that the total IOPS capacity is fully utilized. This causes slow performance for applications that rely on disk access, leading to delayed data processing, system lags, or even timeouts. ### Read IOPs Congested The disk is experiencing Read Operations Per Second (IOPS) congestion, meaning that the total IOPS capacity is fully utilized. This causes slow performance for applications that rely on disk access, leading to delayed data processing, system lags, or even timeouts. ### Read Throughput Congested The disk is experiencing congestion specifically in read throughput, which slows down data retrieval from the disk and can degrade the performance of applications reliant on high-speed data access. ### Write IOPs Congested The disk is experiencing Write Operations Per Second (IOPS) congestion, meaning that the total IOPS capacity is fully utilized. This causes slow performance for applications that rely on disk access, leading to delayed data processing, system lags, or even timeouts. ### Write Throughput Congested The disk is experiencing write throughput congestion, leading to slower data write speeds and affecting applications that require high-speed data recording. This issue can cause delays in data availability and reduced performance in write-intensive tasks. ## Network Endpoint ### Invalid Server Certificate The network endpoint is serving an invalid server certificate, resulting in a high rate of client request errors due to certificate validation failures. This issue propagates further, increasing the overall request error rate across the system. ## Node ### Causely Agent Down The Causely agent on a Kubernetes node is unavailable, creating an observability gap for that node. When the agent is down, telemetry and symptom collection from the affected node may be incomplete or missing, reducing Causely's ability to detect and analyze issues originating there. Common causes include the node becoming unreachable or unavailable and taking the agent down with it, the agent container crashing or failing health checks, CPU, memory, or filesystem pressure on the node preventing the agent from running reliably, and configuration or deployment problems such as invalid configuration, rollout failures, or image errors. ### Container Runtime Issue The node's container runtime is unhealthy or unstable, preventing containers from starting or running reliably. Kubernetes depends on the container runtime to manage the full container lifecycle, including creation, startup, shutdown, and health monitoring. When the runtime is impaired, pods may fail to launch, existing workloads can become unstable, and the node may drift into a degraded state. Common causes include repeated crashes or restart loops in the runtime process that interrupt container lifecycle management, invalid runtime configuration preventing the node from managing containers correctly, overlay filesystem corruption or local disk issues breaking container image operations, and CPU, memory, or I/O saturation destabilizing the runtime. ### Disk Pressure Disk pressure on a Kubernetes node indicates that the node's disk usage is high, potentially causing the eviction of pods, reduced performance, and the inability to schedule new pods. This affects application stability and the node's overall functionality. Disk pressure can arise from insufficient disk space, often caused by log accumulation, container images, temporary files, or application data. ### Filesystem Issue The node filesystem is corrupted or has been remounted read-only, preventing normal node and container operation. The kubelet and container runtime depend on writable filesystem access to maintain state, update pod data, and manage container lifecycles, and when that access is lost, workloads on the node may fail entirely. Common causes include host filesystem corruption that breaks kubelet or runtime state management, the kernel remounting the filesystem read-only after detecting storage errors, and underlying disk or cloud volume failures that surface as filesystem instability. ### Kernel Issue The node kernel is stalled or deadlocked, preventing the host from functioning correctly. Kernel-level failures are severe conditions that can block scheduling, container management, networking, and filesystem operations simultaneously, and a kernel deadlock often precedes full node unavailability. Common causes include bugs in the kernel or kernel modules that deadlock critical host operations, storage or networking drivers triggering hangs under load, and extreme resource contention exposing kernel-level instability. ### Kubelet Issue The kubelet on this node is unhealthy or restarting frequently, disrupting pod lifecycle operations and making the node unreliable. The kubelet is the primary Kubernetes agent on each node and is responsible for reporting node status, managing pod readiness, and coordinating with the container runtime. When it becomes unstable, the node may stop reporting status correctly, fail to update readiness conditions, and struggle to manage running pods. Common causes include CPU, memory, or disk pressure destabilizing the kubelet process, invalid kubelet flags or node configuration causing repeated failures, problems with the container runtime, networking stack, or host filesystem cascading into kubelet instability, and kernel or VM-level host issues making the kubelet unreliable. ### Malfunction A Kubernetes node is unavailable or unresponsive, causing workloads scheduled on it to fail. This typically means the kubelet or the underlying host is no longer operating normally, and pods on the affected node may become unavailable, fail health checks, or stop serving traffic entirely. Common causes include CPU, memory, disk, or kernel resource exhaustion destabilizing the node, the kubelet, container runtime, or other host-level services becoming unhealthy or unreachable, cloud instance failures, network isolation, or underlying VM problems making the node unavailable, and excessive pod concentration or invalid node-level configuration causing the node to become unstable. ### Memory Pressure Memory pressure on a Kubernetes node occurs when available memory falls below critical levels, potentially causing the eviction of pods and instability for applications running on the node. This reduces the node's capacity to run workloads, potentially leading to service disruptions if insufficient resources are available across the cluster. ### Network Issue The node network is unavailable or unstable, causing connectivity failures for node services and workloads. Node-level networking problems can disrupt kubelet communication, service routing, pod-to-pod traffic, and access to external systems, and in practice can make the node appear partially or fully unavailable even when the host itself is still running. Common causes include network interface instability such as resets or repeated unregister events interrupting traffic, misconfiguration in the node's CNI or routing stack isolating the node from the cluster, and underlying VM or cloud network failures cutting the node off from the rest of the infrastructure. ### PID Pressure The node is close to exhausting its process ID capacity, limiting its ability to create new processes and causing workload instability. When PID limits are approached, Kubernetes may be unable to start new processes reliably and workloads can become unstable even when CPU and memory remain available. Common causes include applications or system services creating excessive processes or threads, too many workloads concentrated on a single node, and PID limits configured too low for the workload profile running on the node. ## VirtualMachine ### Conntrack Table Congested The conntrack table on a VM is congested, causing new network connections to fail. This typically results in connectivity issues for applications, degraded performance, or downtime for services dependent on network communication. The conntrack table is responsible for tracking active network connections and has a fixed size, which can be exhausted under high connection load. ### CPU Congested A Virtual Machine (VM) experiencing CPU congestion can lead to sluggish application performance, delayed response times, or even timeout errors for users and processes. This typically indicates that the VM's CPU is overutilized, potentially due to high resource demands from applications or insufficient CPU allocation. ### Disk Read IOPs Congested The total disk read IOPS for a cloud VM are congested because the VM has reached its maximum allowable IOPS limit. This results in throttling, which can slow application performance and lead to delays or errors in read-heavy workloads. ### Disk Read Throughput Congested The total disk read throughput for a cloud VM is congested because the VM has reached its maximum allowable read bandwidth. This can lead to slower data transfer rates for read-intensive applications, causing delays in processing and reduced system performance. ### Disk Total IOPs Congested The total disk IOPS for a cloud VM are congested because the VM has reached its maximum allowable IOPS limit. This results in throttling, which can slow application performance and lead to delays or errors in read/write-heavy workloads. ### Disk Total Throughput Congested The total disk throughput for a cloud VM is congested because the VM has reached its maximum allowable bandwidth. This can lead to slower data transfer rates for read/write-intensive applications, causing delays in processing and reduced system performance. ### Disk Write IOPs Congested The total disk write IOPS for a cloud VM are congested because the VM has reached its maximum allowable IOPS limit. This results in throttling, which can slow application performance and lead to delays or errors in write-heavy workloads. ### Disk Write Throughput Congested The total disk write throughput for a cloud VM is congested because the VM has reached its maximum allowable write bandwidth. This can lead to slower data transfer rates for write-intensive applications, causing delays in processing and reduced system performance. ### Memory Congested Memory congestion in a Virtual Machine (VM) leads to slow system performance, application crashes, or even VM instability as the system struggles to allocate memory for running processes. This typically results in frequent swapping or out-of-memory (OOM) errors, impacting applications and user operations. ### SNAT Ports Congested The SNAT (Source Network Address Translation) ports on a virtual machine (VM) are congested, leading to outbound network connection failures or degraded performance for services relying on external APIs or resources. This issue primarily impacts VMs that need to establish multiple concurrent connections to the internet or external systems. --- ## Diagnosable Root Causes Modern cloud native systems evolve quickly, and complexity makes failures propagate: a small change in one part of the system can ripple across dependencies and surface as Signals that impact user experience. In Causely, a **Diagnosis** is the causal explanation for a set of Signals: the factor Causely infers is responsible for them. It is the current best explanation and updates as Signals change. A Diagnosis is not an alert or an incident; it is derived from structured dependency models, observed telemetry, and causal reasoning. ## How urgency is determined Not every Issue needs immediate attention. Causely ranks urgency automatically, using two controls you can adjust: Service Prioritization and SLOs. - **Service Prioritization** sets how much a service matters. Each service is assigned a tier. Critical services have their SLO enforced and their Diagnoses can be marked urgent. Non-Critical services stay visible but are never marked urgent. Hidden services are kept out of the active view. See [Service Tiers](/configuration/service-tiers/). - **SLOs** define what healthy looks like for a service and decide when urgency fires. When a Diagnosis puts a Critical service's SLO at risk or in violation, Causely marks that Diagnosis urgent and elevates the Issue it belongs to. See [SLO Configuration](/configuration/slo-configuration/). The two work together. Service Prioritization decides whether a service's Diagnoses can be urgent at all. SLO status decides when that urgency triggers. Assign tiers first, then tune SLOs on the services that matter. An Issue's urgency is the highest urgency of any Diagnosis over its life. Because a Diagnosis can update as Signals change, an Issue that was urgent stays urgent once any Diagnosis has crossed the threshold, even if the current Diagnosis no longer puts an SLO at risk. ## Browse Diagnoses Below you can find a list of root cause types that are captured in our Causal Models: the causes each Diagnosis can identify. With these, Causely can pinpoint hundreds of thousands of potential issues and their effects within your environment. Each Diagnosis is connected in a causal graph to the Signals it explains, with supporting evidence and impact context. Use the search and filters below to explore all Diagnoses by category, subcategory, or integration source. --- ## Release Management Diagnoses that are triggered by a code change or new release and result in a measurable reliability regression. {/* ============================================================================ AUTO-GENERATED CONTENT BELOW Generated from: data/root-causes.yaml Script: scripts/references/generate-pages.js To regenerate: yarn generate:references Note: This runs automatically during the build process. Do not manually edit below this line - changes will be overwritten. ============================================================================ */} ## Code Change Regression: CPU Congestion After a version upgrade, application containers experience high CPU usage, leading to performance degradation or unresponsiveness. This issue impacts the system's ability to handle requests effectively, potentially causing downtime or delays for end users. High CPU usage post-upgrade typically stems from changes in the application code, dependencies, or configurations. ## Code Change Regression: Database Connection Pool Saturated After a version upgrade, the client-side database connection pool is exhausted when all available connections are in use, preventing new database queries from being executed. This can cause application requests to hang or fail, impacting user experience and potentially leading to downtime for database-dependent features. ## Code Change Regression: Frequent Crash Failure One or multiple containers of a workload are frequently crashing with a non-zero exit code after a version upgrade. This disrupts the application's functionality, leading to downtime or degraded performance depending on the workload design. The issue likely stems from changes introduced in the new version. ## Code Change Regression: Frequent Memory Failure The application is running out of memory after a version upgrade, leading to crashes, degraded performance, or instability. This impacts availability and user experience, often requiring container restarts or manual intervention to restore functionality. The issue is likely tied to changes in the updated version that increase memory usage or introduce inefficiencies. ## Code Change Regression: Inefficient Garbage Collection After a version upgrade, the garbage collector is frequently running, leading to performance degradation or crashes. This issue is likely caused by changes in the application code or dependencies that increase memory usage or introduce inefficiencies. ## Code Change Regression: Java Heap Saturated After a version upgrade, the Java heap is frequently congested, leading to performance degradation or crashes. This issue is likely caused by changes in the application code or dependencies that increase memory usage or introduce inefficiencies. ## Code Change Regression: Lock Contention After a version upgrade, the application is experiencing frequent locking contention, leading to performance degradation or crashes. This issue is likely caused by changes in the application code or dependencies that increase locking or introduce inefficiencies. ## Code Change Regression: Memory Failure Memory failures after a code change can cause containers to crash or degrade performance, resulting in errors for end users or failed service requests. These issues occur when newly introduced code leads to unexpected increases in memory usage, triggering out-of-memory (OOM) errors and destabilizing the system. ## Code Change Regression: Redis Connection Pool Saturated After a version upgrade, the Redis connection pool is frequently congested, leading to performance degradation or crashes. This issue is likely caused by changes in the application code or dependencies that increase Redis usage or introduce inefficiencies. ## Code Change Regression: Slow Database Queries After a version upgrade, the application is experiencing slow database queries that lead to downstream slow consumer behavior and potential resource starvation. This condition affects instance performance, particularly when query execution times become excessively long. --- ## Services(Root-causes) Services are self-contained units of functionality within a system that perform specific tasks or provide specific capabilities, often accessible through defined interfaces or APIs. Service can be either internal or third-party external services beyond the core application and infrastructure. {/* ============================================================================ AUTO-GENERATED CONTENT BELOW Generated from: data/root-causes.yaml Script: scripts/references/generate-pages.js To regenerate: yarn generate:references Note: This runs automatically during the build process. Do not manually edit below this line - changes will be overwritten. ============================================================================ */} ## Overview - [AI Model](#ai-model) - [Application Load Balancer](#application-load-balancer) - [Messaging / Event Streaming](#messaging--event-streaming) - [Service](#service) ## AI Model ### Congested The AI model is experiencing congestion, resulting in high latency for clients. This suggests that the model is unable to handle the current inference load efficiently, causing delays in response times. Congestion often occurs when the model receives more requests than it can process within its capacity, leading to bottlenecks in inference. This may be due to insufficient resources (for example, GPU, memory, or throughput limits), inefficient batching, or a surge in inference demand. ### Malfunction The AI model is experiencing a high rate of errors, causing disruptions for clients. This can lead to degraded performance, failed inference requests, or complete model unavailability, significantly affecting the user experience. ## Application Load Balancer ### Authentication Misconfiguration Application Load Balancer (ALB) authentication misconfiguration can disrupt secure traffic routing and lead to widespread configuration issues. This misconfiguration may trigger elevated ELB authentication errors, 504 request timeouts, and target connection errors, ultimately impacting service availability and application performance. ### Idle Timeout Misconfiguration Misconfigured idle timeout settings can lead to unintended connection drops and delays, potentially triggering a high frequency of 504 gateway timeout errors. This misconfiguration may also contribute to broader configuration issues that disrupt seamless connectivity between clients and servers. ### Network Policy Misconfiguration Application Load Balancer (ALB) network policy misconfigurations can block or restrict legitimate traffic, leading to widespread configuration issues. These misconfigurations often result in elevated 504 gateway timeout errors and target connection failures, ultimately disrupting service availability. ### Unknown Configuration Failure Application Load Balancer (ALB) misconfiguration can cause widespread connectivity issues, leading to a high frequency of 504 gateway timeout errors and 5xx server errors. These issues indicate that the load balancer's settings are not properly optimized for handling traffic efficiently and reliably. ## Messaging / Event Streaming ### Congested Azure Event Hub Namespace When the Azure Event Hub namespace becomes congested, it reaches a point where its processing capacity is exceeded. This leads to consistent throttling of operations, as the system enforces limits to prevent overload. The high rate of throttling not only impacts event ingestion but also cascades into resource starvation, affecting downstream services that rely on timely event processing. Such congestion is typically caused by high message throughput, suboptimal configuration, or insufficient scaling to handle peak loads. ### High Server Errors Common causes for Azure Event Hub errors include quota exceeded (throughput or message size limits have been breached), partition or offset issues (consumers unable to connect or reading from invalid offsets), networking problems (connectivity issues due to firewall rules, DNS misconfigurations, or latency), service outage (regional Azure service disruption), and misconfigured access policies (incorrect SAS tokens, permissions, or authentication methods). ### High User Errors High user error rates in Azure Event Hub often stem from configuration issues, such as mismatched security credentials (for example Shared Access Signature (SAS) tokens), client SDK version incompatibilities, or throttling from overusing allocated resources. Insufficient permissions or quotas being exceeded can also trigger these errors. Another common cause is incorrect partition or consumer group usage, which can lead to connection limits being breached or messages being inaccessible. ## Service ### Congested The service is experiencing congestion, resulting in high latency for clients. This suggests that the system is unable to handle the current load efficiently, causing delays in response times. Congestion often occurs when the service receives more requests than it can handle within its capacity, leading to bottlenecks in processing. This may be due to insufficient resources (for example, CPU, memory, or bandwidth), unoptimized code, or a surge in traffic (for example, due to a sudden increase in demand or DDoS attack). ### Malfunction The Service is experiencing a high rate of errors, causing disruptions for clients. This can lead to degraded performance, failed requests, or complete service unavailability, significantly affecting the user experience. --- ## Applications(Symptoms) Signals related to application workloads and compute specifications. {/* ============================================================================ AUTO-GENERATED CONTENT BELOW Generated from: data/symptoms.yaml Script: scripts/references/generate-pages.js To regenerate: yarn generate:references Note: This runs automatically during the build process. Do not manually edit below this line - changes will be overwritten. ============================================================================ */} ## Compute Spec - **CPU Avg Increase Version Changed High** - Sustained increase in average CPU usage after version change - **Has DB Connections Increase High Instance** - Has instances with high database connections increase - **Has DB Query Duration Increase High Instance** - Has instances with high database query duration increase - **Has Frequent Crash High Instance** - Has instances with high crash frequency - **Has Frequent OOM Kill High Instance** - Has instances with high OOM kill frequency - **Has GC Time High Instance** - Has instances with high garbage collection time - **Has Java Heap Utilization High Instance** - Has instances with high Java heap utilization - **Has Mutex Wait Time High Instance** - Has instances with high mutex wait time - **Has Redis Connections Utilization High Instance** - Has instances with high Redis connections utilization - **Memory Avg Increase Version Changed High** - Sustained increase in average memory usage after version change ## Workload - **.Net Unhandled Exception In Log** - .NET unhandled exception detected in logs - **Apache Max Workers In Log** - Apache MaxRequestWorkers limit reached detected in logs - **Cache Size High** - High cache size - **Cassandra Tombstone In Log** - Excessive tombstone scanning detected in logs - **Circuit Breaker Open In Log** - Circuit breaker open event detected in logs - **Command Latency High** - High command latency - **Connection Pool Exhausted In Log** - Connection pool exhaustion detected in logs - **Connections High** - High number of connections - **DB Connections Utilization High** - High database connections utilization - **DB Query Duration High** - High database query duration - **Disk Full In Log** - Disk full (ENOSPC) error detected in logs - **Elasticsearch Red In Log** - Elasticsearch cluster RED status detected in logs - **File Descriptor Utilization High** - High file descriptor utilization - **File Limit In Log** - File descriptor limit reached detected in logs - **GC Time High** - High garbage collection time - **Go Goroutine Leak In Log** - Go goroutine leak detected in logs - **Go Max Procs Too High** - GOMAXPROCS value is too high - **Go Nil Pointer Panic In Log** - Go nil pointer panic detected in logs - **HAProxy Max Conn In Log** - HAProxy backend unavailable event detected in logs - **High Error Log Count** - High rate of ERROR or CRITICAL log lines - **High Kafka Data Logs Disk Utilization** - High Kafka broker data log disk utilization - **High Kafka Message Rate** - High Kafka message rate - **High Kafka Offline Partitions** - Kafka has one or more offline partitions - **High Kafka Under Replicated Partitions** - Kafka has under-replicated partitions - **High MongoDB Connection Usage** - High MongoDB active connection utilization - **High MongoDB Open Cursor Count** - High number of open MongoDB cursors - **High MongoDB Replica Lag** - MongoDB replication lag is high - **High Postgres Checkpoint Write Time** - Postgres checkpoint write time above 5 seconds, disk I/O is a bottleneck - **High Postgres Connection Utilization** - Postgres active sessions above 85% of max_connections, new connections will soon be rejected - **High Postgres Deadlock Rate** - Postgres deadlock rate above 0.1 seconds, concurrent conflicting transactions - **High Postgres Lock Wait Count** - More than 5 sessions waiting on a lock, lock contention is occurring - **High Redis Cache Miss Rate** - Redis cache miss ratio above 30%, most requests are falling through to the database - **High Redis Memory Fragmentation** - Redis memory fragmentation ratio above 1.5, significant allocator overhead - **High Redis Replication Lag** - Redis replica is more than 10 seconds behind the master - **High Redis Server Memory Usage** - Redis server memory utilization above 85%, evictions are likely or imminent - **Java GC Pause In Log** - Java GC allocation failure or to-space exhaustion detected in logs - **Java Heap Utilization High** - High Java heap utilization - **Java Null Pointer In Log** - Java NullPointerException detected in logs - **Java OOM In Log** - Java OutOfMemoryError detected in logs - **Java Stack Overflow In Log** - Java StackOverflowError detected in logs - **Java Thread Exhaustion In Log** - Java thread pool rejection detected in logs - **Kafka ISR Shrink In Log** - Kafka ISR shrinkage event detected in logs - **Low Postgres Cache Hit Rate** - Postgres cache miss rate above 10%, queries are reading from disk instead of shared_buffers - **Memory Utilization High** - High memory utilization - **Mutex Wait Time High** - High mutex wait time - **MySQL Deadlock In Log** - MySQL InnoDB deadlock detected in logs - **Nginx Upstream Timeout In Log** - Nginx upstream connection timeout detected in logs - **Nginx Worker Conn In Log** - Nginx worker connections exhausted detected in logs - **Postgres Deadlock In Log** - PostgreSQL deadlock detected in logs - **Postgres Idle-in-Transaction Sessions Accumulating** - More than 3 sessions in idle-in-transaction, they hold locks and block vacuum - **Postgres Query Memory Spill** - Postgres is creating temporary files at a rate above 0.5 seconds, queries are spilling to disk - **Python Unhandled Exception In Log** - Python unhandled exception detected in logs - **Queries Queued High** - High number of queued queries - **RabbitMQ Alarm In Log** - RabbitMQ memory or disk resource alarm detected in logs - **Rate Limit In Log** - Rate limit exceeded detected in logs - **Redis Blocked Clients Accumulating** - More than 5 clients are blocked on BLPOP/BRPOP commands - **Redis Connections Utilization High** - High Redis connections utilization - **Redis Max Memory In Log** - Redis maxmemory limit exceeded detected in logs - **Redis Rejecting Connections** - Redis is rejecting new connections at a rate above 0.1 seconds, maxclients has been reached - **Redis Slow Commands Queued** - Redis slow command log is growing, O(N) commands being run against large data structures - **Request Duration High** - High request duration/latency - **Request Error Rate High** - High request error rate - **Server Errors High** - High server error count - **Throttled High** - High throttled request count - **Transaction Duration High** - High transaction duration - **Transaction Error High** - High transaction error rate - **Transaction IDs Congested High** - High transaction ID congestion - **User Errors High** - High user error count --- ## Data Pipeline(Symptoms) Signals related to operations, access patterns, and data flow. {/* ============================================================================ AUTO-GENERATED CONTENT BELOW Generated from: data/symptoms.yaml Script: scripts/references/generate-pages.js To regenerate: yarn generate:references Note: This runs automatically during the build process. Do not manually edit below this line - changes will be overwritten. ============================================================================ */} ## Access - **Most Requests** - This access has the most requests - **Request Duration High** - High request duration/latency - **Request Error Rate High** - High request error rate - **Request Invalid Certificate Rate High** - High invalid certificate rate - **Request Throttled Rate High** - High throttled request rate - **Request Unauthorized Rate High** - High unauthorized request rate ## Background Operation - **Task Duration High** - High task duration ## DNS Access - **Request Duration High** - High request duration/latency - **Request Error Rate High** - High request error rate - **Request Invalid Certificate Rate High** - High invalid certificate rate - **Request NXDomain Rate High** - High NXDOMAIN response rate - **Request Rate High** - High request rate - **Request Throttled Rate High** - High throttled request rate - **Request Unauthorized Rate High** - High unauthorized request rate ## HTTP Path - **Request Duration High** - High request duration/latency - **Request Error Rate High** - High request error rate ## HTTP Path Access - **Request Duration High** - High request duration/latency - **Request Error Rate High** - High request error rate - **Request Throttled Rate High** - High throttled request rate - **Request Unauthorized Rate High** - High unauthorized request rate ## Queue Access - **Dead Letter Count High** - High dead letter count - **Lag High** - High consumer lag - **Message Wait Time High** - High message wait time - **Queue Depth High** - High queue depth ## RPC Method - **Request Duration High** - High request duration/latency - **Request Error Rate High** - High request error rate ## RPC Method Access - **Request Duration High** - High request duration/latency - **Request Error Rate High** - High request error rate ## Table Access - **DDL Lock Exclusive Rate High** - High DDL exclusive lock rate - **Lock Exclusive Rate High** - High exclusive lock rate - **Query Duration High** - High query duration - **Select Query Duration High** - High SELECT query duration - **Table Bloat High** - High table bloat ## Topic Access - **Lag High** - High consumer lag - **Message Wait Time High** - High message wait time - **Queue Depth High** - High queue depth --- ## Infrastructure(Symptoms) Signals related to compute resources, storage, and networking. {/* ============================================================================ AUTO-GENERATED CONTENT BELOW Generated from: data/symptoms.yaml Script: scripts/references/generate-pages.js To regenerate: yarn generate:references Note: This runs automatically during the build process. Do not manually edit below this line - changes will be overwritten. ============================================================================ */} ## Compute ### Container - **CPU Throttled High** - High CPU throttling - **CPU Utilization High** - High CPU utilization - **Ephemeral Storage Highest Usage** - Highest ephemeral storage usage - **Ephemeral Storage Utilization High** - High ephemeral storage utilization - **Frequent Crash High** - High crash frequency - **Frequent OOM Kill High** - High OOM kill frequency - **Memory Highest Usage** - Highest memory usage - **Memory Utilization High** - High memory utilization ### Controller - **Frequent Pod Ephemeral Storage Evictions High** - High frequency of pod evictions due to ephemeral storage - **Has Image Pull Error Instance** - Has instances with image pull errors ### Node - **Causely Agent Down** - Causely agent on node is unavailable - **Conn Track Table Utilization High** - High connection tracking table utilization - **Container Ephemeral Storage Utilization High** - High container ephemeral storage utilization - **Container Runtime Problem Detected** - Container runtime problem detected - **Corrupt Docker Overlay2 Detected** - Corrupt Docker overlay2 filesystem detected - **CPU Utilization High** - High CPU utilization - **Disk Pressure** - Disk pressure condition - **Disk Pressure Pod Evictions High** - High pod evictions due to disk pressure - **Disk Read IOPS Utilization High** - High disk read IOPS utilization - **Disk Read Throughput Utilization High** - High disk read throughput utilization - **Disk Total IOPS Utilization High** - High total disk IOPS utilization - **Disk Total Throughput Utilization High** - High total disk throughput utilization - **Disk Write IOPS Utilization High** - High disk write IOPS utilization - **Disk Write Throughput Utilization High** - High disk write throughput utilization - **Filesystem Corruption Problem Detected** - Filesystem corruption problem detected - **Frequent Containerd Restart Detected** - Frequent containerd restart detected - **Frequent Docker Restart Detected** - Frequent Docker restart detected - **Frequent Kubelet Restart Detected** - Frequent kubelet restart detected - **Frequent Unregister Net Device Detected** - Frequent network device unregister detected - **Kernel Deadlock Detected** - Kernel deadlock detected - **Kubelet Problem Detected** - Kubelet problem detected - **Memory Pressure** - Memory pressure condition - **Memory Pressure Pod Evictions High** - High pod evictions due to memory pressure - **Memory Utilization High** - High memory utilization - **Network Unavailable Detected** - Network unavailable detected - **Node Down** - Node is unavailable or unresponsive - **PID Pressure Detected** - PID pressure detected ### Pod - **Evicted Because Ephemeral Local Storage Usage Exceeded** - Pod evicted due to exceeding ephemeral local storage usage - **Evicted Because Node Low On Ephemeral Local Storage** - Pod evicted because node is low on ephemeral local storage - **Evicted Because Node Low On Memory** - Pod evicted because node is low on memory - **Image Pull Errors** - Image pull errors detected - **Not Ready** - Pod is not ready - **Pod Failure** - Pod is failing ### Virtual Machine - **Conn Track Table Utilization High** - High connection tracking table utilization - **Container Ephemeral Storage Utilization High** - High container ephemeral storage utilization - **CPU Utilization High** - High CPU utilization - **Disk Read IOPS Utilization High** - High disk read IOPS utilization - **Disk Read Throughput Utilization High** - High disk read throughput utilization - **Disk Total IOPS Utilization High** - High total disk IOPS utilization - **Disk Total Throughput Utilization High** - High total disk throughput utilization - **Disk Write IOPS Utilization High** - High disk write IOPS utilization - **Disk Write Throughput Utilization High** - High disk write throughput utilization - **Memory Utilization High** - High memory utilization - **SNAT Port Utilization High** - High SNAT port utilization ## Storage ### Disk - **Inodes Utilization High** - High inodes utilization - **Read IOPS Utilization High** - High read IOPS utilization - **Read Throughput Utilization High** - High read throughput utilization - **Total IOPS Utilization High** - High total IOPS utilization - **Total Throughput Utilization High** - High total disk throughput utilization - **Utilization High** - High disk utilization - **Write IOPS Utilization High** - High write IOPS utilization - **Write Throughput Utilization High** - High write throughput utilization --- ## Signals A signal is an observable condition on an entity in your system, such as elevated latency or a rising error rate. Signals are the primary inputs to Causely's causal reasoning engine. When a Signal is active, the engine uses it, together with your system topology and other active Signals, to infer a Diagnosis automatically. A signal is a named condition on an [entity](/reference/entity-types/) (service, workload, container, database, node, and others) that is either active or inactive. It becomes active when a specific metric crosses its threshold and holds for a sustained period. Each signal has three parts: the metric or attribute being monitored, the threshold at which it triggers, and a description of what an active state indicates. Signals can be: - **Local**: Observable in the entity where the Diagnosis occurs - **Propagated**: Observable in related entities due to Diagnosis propagation ## How signals activate Signals activate automatically with no configuration required, and every part of that behavior can be tuned when defaults need adjusting. **Configurable thresholds.** Every signal is driven by an underlying metric and a threshold. Thresholds are configurable and have one active source at a time: a system default, a value Causely learns from your traffic, or a manual override. See [Threshold configuration](/configuration/thresholds/). **Controllable activation delay.** A signal does not activate on the first breach. The condition must hold for a sustained period first, which suppresses transient spikes. The bursty activation delay defaults to 5 minutes and is configurable per service. See [Signal delay](/configuration/symptom-delay/). **Activation from existing alerts.** Signals can also be activated by alerts you already run. Causely maps alerts from Prometheus Alertmanager, Datadog, Dynatrace, and Grafana onto the matching signal, so existing alerts feed the causal model without changing how you alert today. See [Alert ingestion](/configuration/alert-ingestion/). ## From Signals to Diagnosis Signals are used by Causely's causal reasoning engine to infer a Diagnosis. When multiple Signals are present, Causely analyzes their relationships and patterns to identify the most likely explanation. The output is a Diagnosis with its blast radius and supporting evidence, structured for an agent, an automation workflow, or an engineer to act on directly. :::info Signal vs Diagnosis Signals describe what you observe (for example, high latency, errors, timeouts); a Diagnosis describes why it happens (for example, congestion, lock contention, queue saturation). They can sound similar, but they're different: latency is the effect, congestion is the cause, and fixing the cause is what prevents the Signal from returning. ::: --- ## Services(Symptoms) Signals related to services and load balancers. {/* ============================================================================ AUTO-GENERATED CONTENT BELOW Generated from: data/symptoms.yaml Script: scripts/references/generate-pages.js To regenerate: yarn generate:references Note: This runs automatically during the build process. Do not manually edit below this line - changes will be overwritten. ============================================================================ */} ## AI Model - **Inference Error Rate High** - High inference error rate - **Inference Latency High** - High inference latency ## SLO - **Burn Rate High** - High SLO burn rate - **Error Budget Remaining Low** - Low error budget remaining ## Service - **DB Connections Increase High** - High increase in database connections - **DB Query Duration Increase High** - High increase in database query duration - **Request Duration High** - High request duration/latency - **Request Error Rate High** - High request error rate - **Request Rate High** - High request rate - **Unavailable** - Service is unavailable (all instances are not ready) ## Service Access - **Request Error Rate High** - High request error rate --- ## Terminology This page contains the terminology used in [Causely](/). The list is not exhaustive, but rather a guide to help you understand the key concepts. Terms are not listed in alphabetical order, but rather grouped by their relatedness. ## Issue, Diagnosis, and Signal Causely's user-facing surfaces, including the UI, notifications, and agent integrations, describe reliability problems using three product terms: **Issue**, **Diagnosis**, and **Signal**. These sit on top of the formal model defined in the rest of this page; they are not renames of it. ### Issue The persistent thread that ties together every [Diagnosis](#diagnosis) of the same underlying problem. An Issue is the one thing you acknowledge, share, and resolve. It stays stable across change, even as the Diagnosis explaining it evolves. ### Diagnosis The current best explanation of the observations at a given moment. A Diagnosis is Causely's runtime inference of a [root cause](#root-cause-rc), what has traditionally been called "the root cause", and it updates as [Signals](#signal) change. ### Signal An observed anomaly a Diagnosis explains, usually a threshold violation on a metric or [SLI](#service-level-indicator-sli), and sometimes an alert, log line, or event. For how Signal maps to the formal model, see the note on [Symptom](#symptom). > An Issue causes anomalies that surface as Signals. Causely infers a Diagnosis, the best explanation of those Signals at that moment. As Signals change, the Diagnosis updates. The Issue stays stable across those changes. If two Diagnoses share even one Signal, they are treated as the same Issue. ## Managed Object An object in the managed environment. ## Managed Domain A scoped collection of [managed objects](#managed-object). ## Causely Domain Manager (CDM) A Causely instance that manages a [managed domain](#managed-domain). ## Event - An observable anomaly in a [managed object](#managed-object). - An _event_ is `TRUE` if the anomaly is observed/present/active. - Examples - HighCPUUtilization - CPU Utilization > 90% - Container OOM Killed - An external notification ## Active/Present Event An [event](#event) that is **`TRUE`**. ## Root Cause (RC) - _Root Cause_ is something that may occur in a [managed object](#managed-object). - _Root Cause_ is inferred based on [observed events](#event). - Examples - Database Congested - Application Malfunction - Container CPU Congested - etc. ## Symptom - An [event](#event) that may be caused by a [root cause](#root-cause-rc). See the [full list of symptoms](/reference/symptoms/). - Examples - Container HighCPUUtilization may be caused by Container Congested - Service HighLatency may be caused by Database Congested - Service HighErrorRate may be caused by Application Malfunction - etc. :::info Signal and the formal model In product and user-facing surfaces these observables are called **Signals**. A Signal is an observed [Event](#event), most often a threshold violation on a metric or SLI and in the model a [Symptom](#symptom), together with ingested observables such as alerts and log lines. The formal terms Event and Symptom are retained here because the causal model and Codebook are defined in terms of them. ::: ## Local Symptom of Root Cause R - A [symptom](#symptom) that is observable in the [managed object](#managed-object) in which the [root cause](#root-cause-rc) $R$ occurs. - Examples - A Container X HighCPUUtilization is a local symptom of Container X Congested - etc. ## Propagated Symptom of Root Cause R - A [symptom](#symptom) that may be caused by [root cause](#root-cause-rc) $R$ observed in a [managed object](#managed-object) related to the [managed object](#managed-object) in which root cause $R$ occurs. - Examples - Service Starvation is a propagated symptom of Database Congested propagating to the services that are accessing the database ## Service Level Indicator (SLI) - A quantitative measure of service reliability that reflects user experience. - Causely monitors three primary SLIs for services: - **Error Rate**: The percentage of successful requests (requests that do not result in errors) - **Latency**: The percentage of requests that complete within an acceptable latency threshold - **Availability**: The percentage of time a service is operational and responding successfully - SLIs are continuously measured from telemetry data (metrics, traces, logs) collected from the [managed domain](#managed-domain). - Examples - A service with 99.5% error rate SLI means 99.5% of requests are successful - A service with 95% latency SLI means 95% of requests complete within the latency threshold - A service with 99.9% availability SLI means the service is operational 99.9% of the time ## Service Level Objective (SLO) - A target value or goal for a [Service Level Indicator](#service-level-indicator-sli). - SLOs define the reliability commitments for services and help prioritize which [root causes](#root-cause-rc) require urgent attention. - When an SLI violates its SLO (or is at risk of violating it), related [root causes](#root-cause-rc) are marked as **Urgent**. - Causely supports SLOs for error rate, latency, and availability, each with configurable targets and burn rate thresholds. See [SLO Configuration](/configuration/slo-configuration) for details on customizing these values. - Examples - An error rate SLO of 99.0% means at least 99% of requests must be successful - A latency SLO of 95.0% means at least 95% of requests must complete within the latency threshold - An availability SLO of 99.0% means the service must be operational at least 99% of the time ## Logs Associated with Root Cause Causely captures logs from the containers that underlie your services to provide rich context for understanding issues. Each container (within its pod) writes logs, including errors and exceptions, to stdout. When Causely detects a service malfunction or identifies a [root cause](#root-cause-rc), it automatically collects and surfaces relevant log lines from the underlying container. These logs are shown in two places: - Under the affected service, when the service is exhibiting abnormal behavior (for example, error spikes, degraded performance). - Alongside a root cause, to highlight the precise errors or stack traces that occurred around the time of failure. This helps validate the issue and dramatically shortens time to understanding and resolution. ## Root Cause Analysis Problem (RCA) Given a set of [Symptoms](#symptom) (manifestations) identify the explanation of why they are present by using knowledge about the world. A Root Cause Analysis Problem $P$ is a 4-Tuple $P =\ < R,\ C,\ S,\ S^+ >$ where - $R = {r_1,\ ...,\ r_n}$ is a finite, non-empty set of [root causes](#root-cause-rc). - $S = {s_1,\ ...,\ s_k}$ is a finite, non-empty set of [symptoms](#symptom). - C a subset of $R\ x\ S$ is a relation with _domain(C) = R_ and _range(C) = S_ called _Causation_. - $S^+$ is a distinguished subset of S said to be _Present_ ## Effects and Causes - For any $r_i\ in\ R$ and $s_j\ in\ S\ in\ P =\ < R,\ C,\ S,\ S^+ >$, - $effects(r_i) = {s_j\ | < r_i\ ,\ s_j > in\ C}$, the set of [symptoms](#symptom) that may be caused by $r_i$ - $causes(s_j) = {r_i\ | < r_i\ ,\ s_j > in\ C}$, the set of [root causes](#root-cause-rc) that may cause $s_j$ - For any subset $R_i\ of\ R$ and subset $S_j\ of\ S$ in $P =\ < R,\ C,\ S,\ S^+ >$, - $effects(R_i) = union\ of\ effects(r_i),\ and\ r_i\ in\ R_i$ - $causes(S_j) = union\ of\ causes(s_j),\ and\ s_j\ in\ S_j$ ## Closure of Root cause R - _Closure_ of [root cause](#root-cause-rc) $R$ is $effect(R)$ - A _Closure of Root Cause $R$_ is a unique signature, a vector of probabilities, that uniquely identifies $R$. ## Spurious and Missing Symptom - For any $r_i\ in\ R$, $s_j\ in\ S$ and any subset $S\ of\ S_k\ in\ P =\ < R,\ C,\ S,\ S^+ >$, - $spurious(r_i\ ,\ S_k) = {s_j\ in\ S_k\ | < r_i\ ,\ s_j > not\ in\ C}$ the set of [symptoms](#symptom) in $S_k\ not\ caused\ by\ r_i$ - $missing(r_i\ ,\ S_k) = {s_j\ not\ in\ S_k\ | < r_i\ ,\ s_j > in\ C}$ - For any subset $R_i\ of\ R$ and subset $S_k\ of\ S\ in\ P =\ < R,\ C,\ S,\ S^+ >$, - $spurious(R_i\ ,\ S_k) = union\ of\ spurious(r_i\ ,\ S_k),\ and\ r_i\ in\ R_i$ - $missing(R_i\ ,\ S_k) = union\ of\ missing(r_i\ ,\ S_k),\ and\ r_i\ in\ R_i$ ## Bayesian Network A Bayesian network is a probabilistic graphical model that represents a set of variables and their conditional dependencies using a directed acyclic graph (DAG), allowing for reasoning under uncertainty by calculating probabilities of events based on known information and relationships between variables; essentially, it's a way to model complex causal relationships between different factors by showing how the probability of one variable changes depending on the states of other related variables. ## Causal Model - [CDM](#causely-domain-manager-cdm) is driven by _Causal Model_, "a knowledge base", of [root causes](#root-cause-rc). - _Causal Model_ captures the _cause and effect_ association between the [root causes](#root-cause-rc) and the [symptoms](#symptom). - _Causal Model_ describes how the [root causes](#root-cause-rc) will propagate across the [managed domain](#managed-domain) and what [symptoms](#symptom) may be caused/observed when each of the [root causes](#root-cause-rc) occur. - CDM is delivered with out-of-the-box built-in _Causal Model_ that captures [root causes](#root-cause-rc) that can occur in cloud native environments. - The _Causal Model_ captures potential [root causes](#root-cause-rc) in a broad range of entities - Such as, applications, databases, caches, messaging, load balancers, DNS compute, storage, etc. - The _Causal Model_ is completely independent of any specific [managed domain](#managed-domain) and is applicable to any cloud native application environment. This enables Causely to automatically pinpoint [root causes](#root-cause-rc) out-of-the-box as soon as it is deployed in an environment. ## Topology - _Topology_ is a directed graph where the nodes are entities in the [managed domain](#managed-domain) and the directed edges represents relationship between the entities. - [CDM](#causely-domain-manager-cdm) automatically discovers the entities and their relationships in the [managed domain](#managed-domain). - _Entities_ are applications, services, databases, caches, messaging, load balancers, compute, storage, etc., - Relationships are Connectivity, Layering, and Composition - _Connectivity_: for each _entity_, the _entities_ it is connected to and the _entities_ it is communicating with. - _Layering_: for each _entity_, the _entities_ it is layered over or underlying. - _Composition_: for each _entity_, the _entities_ it is composed of or part of. - [CDM](#causely-domain-manager-cdm) automatically stitches all of these relationships together to generate a dependency map, or _Topology_ Graph, of the entire _managed domain_. - The _Topology_ Graph is continually updated in real-time to reflect the current state of the _managed domain_. ## Causality Graph (CG) - _Causality Graph_ is a Directed Acyclic Graph (DAG) where the nodes are [root causes](#root-cause-rc) and [symptoms](#symptom), and the edges represent causality, i.e, $R\ \rightarrow \ S$ means R may cause S. - The edges are labeled with probability $P,\ 0 <$ _managed domain_. - The _Causality Graph_ represents all the possible $P < 1$, representing the likelihood R may cause S. - [CDM](#causely-domain-manager-cdm) automatically generates the _Causality Graph_ by applying the [Topology](#topology) Graph to the [Causal Model](#causal-model). - By applying the [Topology](#topology) of the [managed domain](#managed-domain) to the generic [Causal Model](#causal-model), [CDM](#causely-domain-manager-cdm) generates the causal knowledge that is specific to the [managed domain](#managed-domain). - The _Causality Graph_ represents all the possible [root causes](#root-cause-rc) in the [managed domain](#managed-domain), all the [symptoms](#symptom) that may be observed, and the _cause and effect_ relationships between them. - In a [managed domain](#managed-domain) of a few thousands entities, the _Causality Graph_ will incorporate the knowledge of tens of thousands of potential [root causes](#root-cause-rc) and hundreds of thousands [symptoms](#symptom) - which is well beyond human scale. - The _Causality Graph_ is automatically updated every time the [topology](#topology) changes. - The edge probabilities are learned based on the data in the [managed domain](#managed-domain). ## Codebook - A mapping of all potential [root causes](#root-cause-rc) ${r_1,\ ...,\ r_n}$ to the [symptoms](#symptom) ${s_1,\ ...,\ s_m}$ they may cause. - The Codebook is a causality table where - The columns $r_1,\ ...,\ r_n$ represent all the potential [root causes](#root-cause-rc) - The rows $s_1,\ ...,\ s_m$ represent all the potential [symptoms](#symptom) and - A cell $(r_i\ ,\ s_j)$ represents the probability [root causes](#root-cause-rc) $r_i$ may cause [symptom](#symptom) $s_j$ , that is, the likelihood [symptom](#symptom) $s_j$ will be observed/present when [root causes](#root-cause-rc) $r_i$ occurs. - Each [root causes](#root-cause-rc) in the Codebook has a unique signature, a vector of m probabilities, that uniquely identifies the [root causes](#root-cause-rc). - Using the _Codebook_, [CDM](#causely-domain-manager-cdm) quickly searches and pinpoints the [root causes](#root-cause-rc) based on the observed [symptoms](#symptom). --- ## Security ## Overview Causely integrates seamlessly with your existing observability stack. You can use your existing telemetry sources, including metrics, traces, and logs, as data sources for causal analysis. Optional lightweight automatic instrumentation is available, but Causely works equally well with bring-your-own-data approaches. Most telemetry data is processed locally in your environment. Causely is designed to protect sensitive data and ensure privacy. The system processes telemetry data locally and primarily transmits minimal, high-level information to its backend, such as service topology and observed Signals (for example high latency or CPU usage). All data is encrypted in transit and at rest. To provide context around inferred Diagnoses, Causely sends a targeted subset of telemetry, such as relevant metrics, traces, and log-derived errors/events, to the backend after a Diagnosis is made. This data is used solely to enhance the clarity and usefulness of the Diagnosis. Personally Identifiable Information (PII) is filtered out to the extent possible, and customer data is never used to train any shared or foundational models. Causely's infrastructure is strictly access-controlled. Only a small, authorized operations team has access, and customer data is never used to train any shared or foundational models. Each customer's causal graph is built and maintained independently, based on their specific service topology and behaviors, ensuring both privacy and accuracy. Our approach is rooted in abstraction: modeling cause-and-effect relationships in complex systems. The causal graph is dynamically created by combining the abstract causal model with the discovered topology uniquely for each customer and maintained automatically and continuously. It enables us to cut through complexity, deliver precise insights, and build trust through transparency and actionable outcomes. For information about Causely's deployment architecture, see the [Architecture](/getting-started/architecture) documentation. ## Mediation Layer Components The mediation layer consists of components deployed locally in your infrastructure. By default, all components operate with the minimum privileges required for their function. We will discuss the deployment details for Kubernetes, Docker, and Nomad environments in this overview. For specific setup steps, please refer to our [installation instructions](/installation). Causely can also be deployed in other environments not listed here; if you'd like more information or guidance on those deployments, please contact us directly. ### Agents Agents are deployed across your infrastructure to gather node and container level metrics. The deployment method varies depending on your environment: - **Kubernetes**: Agents are deployed as a DaemonSet across all nodes in the cluster - **Docker**: Agents run as containers on standalone Docker hosts - **Nomad**: Agents are deployed as Nomad jobs across the cluster Agents require privileged access to the host system because they leverage [eBPF](https://ebpf.io) technology. The specific permissions required depend on your deployment environment: **Kubernetes deployments:** - The container runs as privileged (`securityContext.privileged=true`) - The container runs as root (`securityContext.runAsUser: 0`) - The container readonly mounts the host filesystem into the container, granting access to the host directly (`volumeMounts.readOnly: true`) - Additional Kubernetes API permissions are required to collect specific metrics about the node and containers **Docker deployments:** - The container runs with `--privileged` flag - The container uses `pid: host` mode - The container readonly mounts the host filesystem **Nomad deployments:** - Similar privileged access requirements as Docker deployments Agents don't establish any outbound connections to the internet or any other service apart from the Mediator and VictoriaMetrics. The agents periodically forward the topology and manifestation data to the Mediator, which, in turn, sends it to the Causely backend for analysis. #### Network and System Impact While agents require privileged access to attach eBPF probes, they do **not**: - Intercept, block, or route network traffic at the kernel level - Act as a network policy enforcement point - Function as a CNI or network plugin - Hook into kernel networking callbacks or the packet datapath The instrumentation may inject trace context headers into outgoing requests to enable distributed tracing, but this occurs at the application layer and does not affect network connectivity. Agents use [uprobes](/telemetry-sources/ebpf#uprobe-based-instrumentation) to observe application behavior through user-space function instrumentation. If an agent fails or is removed, your applications and network continue to function normally with no impact to traffic flow. ### Mediator The Mediator is responsible for collecting infrastructure-level metrics and transmitting the data to the backend. The Mediator listens for OpenTelemetry signals (`port: 4317`) and updates from the agents (`port: 50051`). The Mediator has only the minimal permissions necessary to discover the local infrastructure configuration and collect metrics. The specific permissions depend on your deployment environment: - **Kubernetes**: The Mediator requires minimal Kubernetes API permissions to discover cluster configuration - **Docker/Nomad**: The Mediator requires appropriate permissions to discover and monitor the local environment The Mediator can also be optionally configured to get metrics from Prometheus or discover and monitor managed cloud services from cloud providers. ### Executor The Executor is responsible for executing remediation actions within your infrastructure. The Executor is an optional component and can be enabled as part of the deployment process. The specific permissions required depend on your deployment environment: - **Kubernetes**: The Executor's ServiceAccount is granted the `cluster-admin` role - **Docker/Nomad**: The Executor requires appropriate permissions to execute remediation actions in your environment ### VictoriaMetrics VictoriaMetrics is a timeseries database used by the agents and mediator (`port: 8428`) to store additional timeseries data locally in your environment. --- ## Alertmanager ## Overview You can forward alerts from [Prometheus Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/) a to Causely by adding a webhook receiver. Once connected, Causely ingests your existing alerts, links them to the right services, and applies causal analysis to show the real underlying issues. This means you can: - Keep using your current alerting rules. - See alerts mapped to services and dependencies inside Causely. - Automatically identify which alerts are Signals and which point to the actual causes. ## Add Causely as a webhook receiver in Alertmanager If you are using the Prometheus Operator, add the following AlertmanagerConfig to forward alerts to Causely’s mediator. ```yaml kind: AlertmanagerConfig spec: route: group_by: ['alertname', 'pod', 'namespace'] # how alerts are grouped repeat_interval: 3h # how often repeated alerts are sent receiver: 'causely-mediator-webhook' # send all alerts here receivers: - name: 'causely-mediator-webhook' webhook_configs: - url: 'http://mediator.causely:9093/api/v1/alerts' # mediator endpoint send_resolved: true # send firing + resolved states ``` ## Enabling Alertmanager as a Data Source Add the following to your `values.yaml` to enable Alertmanager ingestion in the mediator: ```yaml mediator: alertmanager: enabled: true port: 9093 alert_mappings: [] # optional; see examples below ``` ### Optional: Map Alerts to Entities and Attributes Use `alert_mappings` to map Alertmanager alerts to Causely Signals. For example, the following maps the `Error Rate High` alert to the `RequestErrorRate_High` symptom of the Service Entity: ```yaml alert_mappings: - alert_name: 'Error Rate High' symptom: 'RequestErrorRate_High' entity: service: {} discovery: - kubernetes_pod: namespace: 'namespace' pod_name: 'pod' conditions: - regex_not_match: label: 'pod' regex: '^.*-exporter.*$' ``` Without an `alert_mappings` entry, Causely will automatically map error and latency-type alerts, based on keyword matching, to the corresponding Causely Service Signals. ### Example Alert Payload An example alert from Alertmanager that Causely can ingest: ```json [ { "labels": { "alertname": "HighErrorRate", "severity": "warning", "service": "causely-analysis", "namespace": "production" }, "annotations": { "summary": "Error rate above 5% for 10m" }, "startsAt": "2025-08-11T13:30:56Z", "endsAt": "0001-01-01T00:00:00Z", "generatorURL": "http://prometheus/graph?g0.expr=...", "fingerprint": "047235cd648d8d01" } ] ``` This alert would be mapped automatically to the high error rate Signal of the `causely-analysis` service. --- ## Application Instrumentation ## Overview Causely provides comprehensive observability and causal reasoning for applications written in various programming languages by leveraging industry-standard telemetry collection frameworks like [Prometheus](/telemetry-sources/prometheus) and [OpenTelemetry](/telemetry-sources/opentelemetry). This approach ensures that regardless of your application's programming language, Causely can transform high-volume observability signals into real-time, explainable insights about performance, resource utilization, and potential issues. Instead of just monitoring Signals, Causely analyzes them in real time to surface the actual Diagnosis behind application issues. By setting up the application languages integration, you will be able to do the following: - Identify causes for reliability issues originating from your application, including: - [Database Connection Pool Saturated](/reference/root-causes/applications#database-connection-pool-saturated) - [File Descriptor Exhaustion](/reference/root-causes/applications#file-descriptor-exhaustion) - [Inefficient Garbage Collection](/reference/root-causes/applications#inefficient-garbage-collection) - [Lock Contention](/reference/root-causes/applications#lock-contention) - [Invalid Client Certificate](/reference/root-causes/applications#invalid-client-certificate) - [Service Congestion](/reference/root-causes/services#congested) - [Service Malfunction](/reference/root-causes/services#malfunction) - Language-specific causal reasons, including: - [GOMAXPROCS Misconfigured](/reference/root-causes/applications#gomaxprocs-misconfigured) - [Java Heap Saturated](/reference/root-causes/applications#java-heap-saturated) - Observe the application as an entity in the Topology Graph, including it's relationships to other entities on the service map, infrastructure stack and dataflow map. ## Language Support Causely provides **automatic, zero-code instrumentation** for most applications using [eBPF-based automatic instrumentation](/telemetry-sources/ebpf). This works out-of-the-box for most applications and provides comprehensive trace coverage. For application that are already instrumented using OpenTelemetry, Causely can collect telemetry from them using the [OpenTelemetry Collector](/telemetry-sources/opentelemetry). This way Causely can provide comprehensive observability and causal reasoning for applications written in any language, including the following: - C++ - C#/.NET - Erlang/Elixir - Go - Java - JavaScript - PHP - Python - Ruby - Rust - Swift :::tip OpenTelemetry and Prometheus are open source standards and by that can be implemented by any language. This means that as long as support for your language is available, Causely can collect telemetry from it. ::: --- ## Amazon Web Services (AWS) ## Overview Causely provides comprehensive native integration with Amazon Web Services (AWS) to help you identify and resolve infrastructure and service performance issues before they impact your users. Instead of just monitoring Signals, Causely analyzes them in real time from your AWS infrastructure to surface the actual Diagnosis behind problems across your entire AWS ecosystem. By setting up the AWS integration, you will be able to: - **Identify causes for reliability issues originating from your AWS infrastructure and services**, including: - [Service Congestion](/reference/root-causes/services#congested-1) and [Service Malfunction](/reference/root-causes/services#malfunction-1) - [Application Load Balancer Misconfiguration](/reference/root-causes/services#application-load-balancer) issues - [Authentication](/reference/root-causes/services#authentication-misconfiguration) and [Network Policy](/reference/root-causes/services#network-policy-misconfiguration) misconfigurations - [Memory Pressure](/reference/root-causes/infrastructure#memory-pressure) and [CPU Congested](/reference/root-causes/infrastructure#cpu-congested) issues - [Disk Pressure](/reference/root-causes/infrastructure#disk-pressure) and [I/O congestion](/reference/root-causes/infrastructure#disk-total-iops-congested) - **Observe AWS services** as entities in the Topology Graph, including their relationships to Kubernetes services and dependencies - **Monitor performance metrics** from CloudWatch with automatic correlation to service health and user impact ## Supported AWS Services Causely integrates with the following AWS services to provide comprehensive observability and causal inference: - Amazon Elastic Block Store (EBS) - Amazon Elastic Compute Cloud (EC2) - Amazon Elastic Container Service (ECS) - Amazon ElastiCache - Amazon Managed Streaming for Apache Kafka (MSK) - Amazon OpenSearch Service - Amazon Relational Database Service (RDS) - Amazon Simple Notification Service (SNS) - Amazon Simple Queue Service (SQS) - Application Load Balancer (ALB) - AWS Lambda - Network Load Balancer (NLB) :::tip[Database-specific setup] For AWS RDS databases, see the database-specific documentation for detailed setup and configuration verification: - [PostgreSQL](/telemetry-sources/postgresql) - [MySQL](/telemetry-sources/mysql) - [OpenSearch / Elasticsearch](/telemetry-sources/elasticsearch) ::: ## Setup Guide ### Authentication Options Causely supports two authentication methods for AWS integration. Choose the method that best fits your security requirements. #### Option 1: IAM Role IAM roles for service accounts in Amazon EKS enhance security by enabling least privilege access, isolating credentials between Pods, and improving auditability. AWS security best practices promote the use of EKS Pod identity. For more details see the [AWS documentation for EKS Pod Identity](https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html) and the [for IAM Roles for Service Accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html). ##### 1. Create an IAM role in the source account Select the Custom trust policy as trusted entity type and paste the following trust policy with your account id, your region and your OIDC provider id: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::111111111111:oidc-provider/oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:sub": "system:serviceaccount:causely:causely-mediator", "oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:aud": "sts.amazonaws.com" } } } ] } ``` ##### 2. Name the role Enter a name for the role, for example: `CauselyMediation` ##### 3. Register the OIDC provider in the target account Run the following command to register the OIDC provider in the target account: ```bash aws iam create-open-id-connect-provider \ --url "https://oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE" \ --client-id-list "sts.amazonaws.com" \ --thumbprint-list "AABBCCDDEEFFGGHHIIJJKKLLMMNNOOPPQQRRSSTT" ``` ##### 4. Create an IAM role in the target account Select the Custom trust policy as trusted entity type and paste the following trust policy with your source account id: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:sts::222222222222:assumed-role/CauselyMediation/CauselyMediatorSession" }, "Action": "sts:AssumeRole" } ] } ``` ##### 5. Name the target account role Enter a name for the role, for example: `CauselyAccess` ##### 6. Assign policies to the role **For comprehensive access (recommended):** - [ReadOnlyAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/ReadOnlyAccess.html) **For granular access, assign these specific policies:** - [AmazonEC2ReadOnlyAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonEC2ReadOnlyAccess.html) - [AmazonECS_FullAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonECS_FullAccess.html) - [AmazonElastiCache ReadOnlyAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonElastiCacheReadOnlyAccess.html) - [AmazonMSK ReadOnlyAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonMSKReadOnlyAccess.html) - [AmazonRDS ReadOnlyAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonRDSReadOnlyAccess.html) - [AmazonSNS ReadOnlyAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonSNSReadOnlyAccess.html) - [AmazonSQS ReadOnlyAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonSQSReadOnlyAccess.html) - [AWSLambda_ReadOnlyAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AWSLambda_ReadOnlyAccess.html) - [AWSResourceGroupsReadOnlyAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AWSResourceGroupsReadOnlyAccess.html) - [ElasticLoadBalancingReadOnly](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/ElasticLoadBalancingReadOnly.html) ##### 7. Create Kubernetes secret Create the following Kubernetes secret in the same namespace as the Causely mediator: ```yaml title="aws-secret.yaml" apiVersion: v1 kind: Secret type: Opaque metadata: name: aws-secret namespace: causely stringData: AWS_REGION: region-code AWS_ROLE_ARN: arn:aws:iam::222222222222:role/CauselyAccess ``` ##### 8. Update Causely mediator configuration Update the Causely mediator with the following helm values ```yaml title="causely-values.yaml" mediator: serviceAccount: annotations: eks.amazonaws.com/role-arn: arn:aws:iam::111111111111:role/CauselyMediation scrapers: aws: enabled: true accounts: - arn: arn:aws:organizations::222222222222:account/o-pyxgi8opc5/123456789012 secretName: aws-secret ``` #### Option 2: IAM User For simpler setups or development environments, you can use IAM user credentials. ##### 1. Create an IAM user Create an IAM user with the following policies: - [ReadOnlyAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/ReadOnlyAccess.html) Or for more granular access, assign the following predefined policies to the user: - [AmazonEC2ReadOnlyAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonEC2ReadOnlyAccess.html) - [AmazonECS_FullAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonECS_FullAccess.html) - [AmazonElastiCache ReadOnlyAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonElastiCacheReadOnlyAccess.html) - [AmazonMSK ReadOnlyAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonMSKReadOnlyAccess.html) - [AmazonRDS ReadOnlyAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonRDSReadOnlyAccess.html) - [AmazonSNS ReadOnlyAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonSNSReadOnlyAccess.html) - [AmazonSQS ReadOnlyAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonSQSReadOnlyAccess.html) - [AWSLambda_ReadOnlyAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AWSLambda_ReadOnlyAccess.html) - [AWSResourceGroupsReadOnlyAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AWSResourceGroupsReadOnlyAccess.html) - [ElasticLoadBalancingReadOnly](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/ElasticLoadBalancingReadOnly.html) ##### 2. Create Kubernetes secret Create the following Kubernetes secret in the same namespace as the Causely mediator: ```yaml title="aws-secret.yaml" apiVersion: v1 kind: Secret type: Opaque metadata: name: aws-secret namespace: causely stringData: AWS_REGION: region-code AWS_ACCESS_KEY_ID: YYY AWS_SECRET_ACCESS_KEY: ZZZ ``` ##### 3. Update Causely configuration Update the Causely mediator with the following helm values ```yaml title="causely-values.yaml" scrapers: aws: enabled: true accounts: - arn: arn:aws:organizations::222222222222:account/o-pyxgi8opc5/123456789012 secretName: aws-secret ``` ### Alternative: Enable Credentials Autodiscovery Causely supports credentials autodiscovery for simplified management: ```bash kubectl --namespace causely label secret aws-secret "causely.ai/scraper=AWS" ``` ## What Data is Collected Causely collects comprehensive metadata and performance information from your AWS infrastructure. Among others, Causely collects the following data: - **Application Load Balancer (ALB) / Network Load Balancer (NLB)** - Load balancer entities with DNS names and ARNs - Request metrics including total requests, error rates (4xx, 5xx) - Performance metrics including response times and target health - Target group mappings to backend services - Connection error tracking and timeout analysis - Authentication metrics and configuration validation - Network endpoint mapping to Kubernetes services - External DNS hostname integration for custom domains - **Amazon ECS** - Cluster entities with service mappings and task definitions - Service performance including CPU and memory utilization - Task health and container metrics - Resource capacity and utilization tracking - Service-to-workload relationship mapping - Compute resource allocation and usage - **AWS Lambda** - Function entities with ARNs and configurations - Invocation metrics and error rates - Performance data including duration and memory usage - Function URL mappings and network endpoints - Compute resource tracking for serverless workloads - **Amazon RDS** - Database instance entities with connection details - Performance metrics including CPU, memory, and I/O utilization - Connection usage and capacity tracking - Database engine information and version details - Storage metrics including free space and IOPS utilization - Slow query identification and analysis - Network endpoints for database connectivity - **Amazon MSK (Managed Streaming Kafka)** - Cluster entities with broker information - Broker performance including CPU, memory, and storage usage - Network endpoints for Kafka connectivity - Storage metrics and capacity utilization - Cluster health and broker status tracking - **Amazon ElastiCache** - ElasticCache Compute, Network and Cache metrics including connection usage and cache hits - **Amazon SNS and SQS** - SNS Topics and subscribed SQS queues - Number of messages available for retrieval from the queue - **Amazon EC2** - Virtual machine entities with instance details - Resource utilization including CPU, memory, and network - Instance type information and capacity planning - Performance metrics from CloudWatch - Instance state and health monitoring - Storage relationships to EBS volumes - **Amazon EBS** - Volume entities with capacity and performance characteristics - I/O performance including IOPS and throughput metrics - Storage utilization and capacity tracking - Volume attachment relationships to EC2 instances - Performance optimization recommendations - **CloudWatch Integration** - Real-time metrics collection from all supported services - Historical data analysis for trend identification - Custom metric support for application-specific monitoring - Automatic correlation between infrastructure metrics and service health - Threshold monitoring and anomaly detection --- ## Azure ## Overview Causely provides comprehensive native integration with Microsoft Azure to help you identify and resolve infrastructure and managed service reliability issues before they impact your users. Rather than stopping at surface-level Signals, Causely analyzes them in real time from your Azure environment to infer the underlying Diagnosis behind performance degradation and risk across your cloud services. By setting up the Azure integration, you will be able to: **Identify causes for reliability issues originating from your Azure infrastructure and services**, including: - [Service Congestion](/reference/root-causes/services#congested-1) and [Service Malfunction](/reference/root-causes/services#malfunction-1) - [Application Load Balancer Misconfiguration](/reference/root-causes/services#application-load-balancer) issues - [Authentication](/reference/root-causes/services#authentication-misconfiguration) and [Network Policy](/reference/root-causes/services#network-policy-misconfiguration) misconfigurations - [Memory Pressure](/reference/root-causes/infrastructure#memory-pressure) and [CPU Congested](/reference/root-causes/infrastructure#cpu-congested) issues - [Disk Pressure](/reference/root-causes/infrastructure#disk-pressure) and [I/O congestion](/reference/root-causes/infrastructure#disk-total-iops-congested) Observe Azure resources as first-class entities in the Causely Topology Graph, including their relationships to Kubernetes workloads, cloud services, and downstream dependencies Monitor performance and health metrics from Azure-native sources with automatic correlation to service impact and causal diagnoses ## Supported Azure Services Causely integrates with the following Azure services to provide comprehensive observability and causal inference: - Azure Cache for Redis - Azure Cache for Redis Enterprise - Azure Database for MySQL Flexible Server - Azure Database for PostgreSQL Flexible Server - Azure Managed Disks - Azure Key Vault - Azure Event Hubs - Azure Load Balancer - Azure Storage (Blob, Queue, File, Table) - Azure Virtual Machines (VMs) - Azure Virtual Machine Scale Sets (VMSS) ## Step 1: Create a Service Principal (SPN) Use the following `azure-cli` command to create an SPN: ```bash az ad sp create-for-rbac --name causely-spn --role reader --scopes /subscriptions/00000000-0000-0000-0000-000000000000 ``` ## Step 2: Create a Kubernetes Secret for the SPN Credentials After creating the SPN, save its credentials in a Kubernetes Secret. Replace the placeholder values with the SPN's `tenantId`, `clientId`, `clientSecret`, and `subscriptionId`. ```bash kubectl --namespace causely create secret generic spn-credentials \ --from-literal=tenantId="..." \ --from-literal=clientId="..." \ --from-literal=clientSecret="..." \ --from-literal=subscriptionId="..." ``` ## Step 3: Update Causely Configuration Once the Secret is created, update the Causely configuration to enable scraping for the new subscription. Below is an example configuration: ```yaml scrapers: azure: # Scrape VMs & Disks enabled: true subscriptions: - secretName: spn-credentials namespace: causely azurePostgres: # Scrape Postgresql Databases enabled: true subscriptions: - secretName: spn-credentials namespace: causely azureMysql: # Scrape Mysql Databases enabled: true subscriptions: - secretName: spn-credentials namespace: causely azureRedis: # Scrape Redis enabled: true subscriptions: - secretName: spn-credentials namespace: causely azureEventhub: # Scrape EventHub enabled: true subscriptions: - secretName: spn-credentials namespace: causely ``` ## Alternative: Enable Credentials Autodiscovery Causely also supports credentials autodiscovery. This feature allows you to add new scraping targets without updating the Causely configuration. Simply label the Kubernetes Secret to enable autodiscovery for the corresponding scraper. ### Example: Labeling Secrets for Autodiscovery ```bash # Azure VMs & Disks kubectl --namespace causely label secret spn-credentials "causely.ai/scraper=Azure" # Azure Postgresql kubectl --namespace causely label secret spn-credentials "causely.ai/scraper=AzurePostgres" # Azure MySQL kubectl --namespace causely label secret spn-credentials "causely.ai/scraper=AzureMysql" # Azure Redis kubectl --namespace causely label secret spn-credentials "causely.ai/scraper=AzureRedis" # Azure EventHub kubectl --namespace causely label secret spn-credentials "causely.ai/scraper=AzureEventHub" ``` With these steps, you can seamlessly integrate Causely with the Azure API and configure it to monitor your desired resources. --- ## Backstage Integration ## Overview Causely can ingest your [Backstage](https://backstage.io) software catalog and stitch Backstage **components** and **owners** (groups and users) to Kubernetes services. This enriches the topology with team ownership, component metadata (type, lifecycle, system), and enables causal analysis and remediation with clear ownership context. The integration: - Fetches catalog entities from the Backstage Catalog API (components, groups, users) - **Stitches Backstage components to Kubernetes services** by direct lookup from component-derived name and namespace (see below) - Creates or updates **ServiceOwner** entities from Backstage groups/users and links them to services - Applies Backstage-derived labels to services and owners for filtering and display ## How Backstage components are stitched to Kubernetes services Causely links Backstage **components** to Kubernetes **Services** by **direct lookup**: it reads a **service name** and optionally a **namespace** from the component, then looks up the Kubernetes Service with that name (and in that namespace when provided). When namespace is not provided, the service is resolved by name only and must be unique cluster-wide. Ownership and component metadata are then applied to the matched service. **Defaults:** Causely reads the service name from the **`backstage.io/kubernetes-id`** annotation and the namespace from **`backstage.io/kubernetes-namespace`** on the Backstage component. **Owner:** Owner is taken from the Backstage component only, for example **`spec.owner`** or **`backstage.io/owner`** on the component, depending on your `label_map`. If no owner can be resolved from the component, the owner link is not created for that service. If your Backstage components use different fields for the service name or namespace, you can override the mapping in Helm values so Causely reads the right fields and resolves to the correct Kubernetes service. ## Configuration ### Add data source from the UI You can add Backstage as a data source from the Causely UI. Go to [Integrations](https://portal.causely.app/integrations), add the Backstage integration, and select the cluster to which the configuration will be pushed. ### Basic setup (ops as code) If you prefer to manage configuration as code, enable the Backstage scraper and point it at a Kubernetes Secret that holds your Backstage API endpoint and token: ```yaml title="causely-values.yaml" scrapers: backstage: enabled: true instances: - secretName: 'backstage-credentials' namespace: 'causely' ``` Create a Kubernetes Secret with the Backstage URL and (optionally) token and TLS options, and label it so the scraper can discover it: ```bash kubectl create secret generic backstage-credentials \ --namespace causely \ --from-literal=endpoint='https://your-backstage.example.com' \ --from-literal=token='your-backstage-api-token' kubectl label secret backstage-credentials \ --namespace causely \ causely.ai/scraper=Backstage ``` The secret must have the label **`causely.ai/scraper: Backstage`** so the Backstage scraper can discover it. Secret keys: - **`endpoint`** (required): Backstage base URL, for example `https://backstage.example.com`, without a trailing slash. - **`token`** (optional): Bearer token for Backstage API authentication. - **`insecure_skip_verify`** (optional): Set to `true` to skip TLS verification, for example for dev; not recommended for production. ### Customize the mapping to your Backstage catalog (optional) If your Backstage components store the Kubernetes service name or namespace in different fields than the defaults, you can override the mapping in Helm so Causely resolves to the right Kubernetes service. Under each Backstage instance, set **`label_map`** in your values to specify which component fields to use for: - **`component_kubernetes_id`**: where to read the **name of the Kubernetes Service** from the Backstage component - **`component_kubernetes_namespace`**: where to read the **namespace of the Kubernetes Service** from the Backstage component - **`component_owner`**: configurable fields on the Backstage component used to resolve the owner (if none can be resolved, the owner link is skipped for that service) Example with the default mappings and comments: ```yaml title="causely-values.yaml" scrapers: backstage: enabled: true instances: - secretName: 'backstage-credentials' namespace: 'causely' # Override which Backstage component fields are used to resolve the K8s service (name + namespace) label_map: # Backstage field(s) used for the Kubernetes Service name component_kubernetes_id: - "annotations.backstage.io/kubernetes-id" # Backstage field(s) used for the Kubernetes Service namespace component_kubernetes_namespace: - "annotations.backstage.io/kubernetes-namespace" # Backstage field(s) used to resolve the component owner (no owner link if unresolved) component_owner: - "spec.owner" - "annotations.backstage.io/owner" ``` This way the name and namespace used for lookup match how your Backstage components are defined. ## What data is collected - **ServiceOwner entities**: Created or updated from Backstage groups and users, with display name and email when available. - **Service–owner links**: Each stitched Kubernetes service is linked to its owner when that owner can be resolved from the Backstage component. - **Labels on services**: Backstage scraper name, team (owner display name), component name/type/lifecycle/system, owner ref/kind/source, and the Kubernetes ID/namespace used for stitching. - **Labels on owners**: Owner ref, kind, display name, email, and scraper identification. Only Backstage catalog entities of kind **component** are considered for stitching to Kubernetes services; **group** and **user** entities are used to build the owner catalog. --- ## Checkly ## Overview Causely provides native integration with [Checkly](https://www.checklyhq.com/) to help you identify and resolve external monitoring and synthetic testing issues before they impact your users. Instead of just monitoring Signals, Causely analyzes them in real time from Checkly's global monitoring network to surface the actual Diagnosis behind external availability and performance problems. This integration helps you identify the following causes for reliability issues, among others: - [Service Malfunction](/reference/root-causes/services#malfunction-1) and [service Congestion](/reference/root-causes/services#congested-1) - [Faulty Error Handling in HTTP Path](/reference/root-causes/datapipeline#faulty-error-handling-in-http-path) and [Slow Execution in HTTP Path Handler](/reference/root-causes/datapipeline#slow-execution-in-http-path-handler) - [Code Change Regression: CPU Congestion](/reference/root-causes/release#code-change-regression-cpu-congestion) and [Code Change Regression: Memory Failure](/reference/root-causes/release#code-change-regression-memory-failure) - [Network endpoint issues](/reference/root-causes/infrastructure#network-endpoint) The integration supports Checkly's comprehensive monitoring platform including API checks, browser checks, and global monitoring locations. ## Setup Guide ### Step 1: Create an API Key in Checkly Create an [API key](https://app.checklyhq.com/settings/user/api-keys) for your Checkly account with the following permissions: - **Read access to checks**: Required to access check configurations and metadata - **Read access to check results**: Required to monitor check execution results and performance - **Read access to locations**: Required to access monitoring location information - **Read access to alerts**: Required to monitor alert configurations and status For more information on [Checkly API authentication](https://developers.checklyhq.com/reference/authentication) ### Step 2: Configure Monitoring Locations Ensure your Checkly checks are configured to run from multiple monitoring locations to provide comprehensive coverage: - **Public locations**: Checkly's global network of monitoring locations - **Private locations**: Your own monitoring infrastructure for internal services - **Geographic distribution**: Multiple regions for comprehensive coverage ### Step 3: Update Causely Configuration Once the API key is created, update the Causely configuration to enable scraping for Checkly checks. Below is an example configuration: ```yaml scrapers: checkly: enabled: true accounts: - account_id: 'aaaabbbb-cccc-dddd-eeee-ffffffffffff' token: 'xz_1234' ``` ### Alternative: Enable Credentials Autodiscovery Causely also supports credentials autodiscovery. This feature allows you to add new scraping targets without updating the Causely configuration. Label the Kubernetes secret to enable autodiscovery for the corresponding scraper. ```bash kubectl --namespace causely label secret checkly-credentials "causely.ai/scraper=Checkly" ``` ## What Data is Collected The Checkly scraper collects comprehensive synthetic monitoring and testing data from your Checkly checks, including: - **Check information**: ID, name, type (API checks), frequency, and locations - **Check settings**: Alert configurations, retry strategies, and timeout settings - **Request details**: HTTP method, URL, headers, body, and query parameters - **Assertions and validations**: Response validation rules and performance thresholds - **Environment variables**: Check-specific configuration and test data - **Response times**: Individual check response times and performance data - **Success/failure rates**: Track successful vs failed check executions - **Error information**: Detailed error messages and failure reasons - **Location-based results**: Performance data per monitoring location - **Recent check history**: Last hour of check results for trend analysis - **Geographic distribution**: Different monitoring locations and regions - **Location metadata**: Region names and availability zones - **Multi-location testing**: Results from various geographic points - **Public vs private locations**: Coverage from Checkly's network and your infrastructure - **Alert symptoms**: `RequestErrorRate_High` and `RequestDuration_High` for proactive issue detection - **Service mapping**: Links Checkly checks to target services - **Workload relationships**: Associates monitoring locations with workloads - **Network endpoint tracking**: Maps checks to specific service endpoints - **Access data collection**: Creates network dependency mappings for causal analysis - **HTTP response details**: Status codes, response headers, and body content - **Request timing phases**: Detailed breakdown of request/response timing - **Assertion results**: Validation rule pass/fail status and actual values - **Error categorization**: Network errors, timeout issues, and validation failures - **Performance thresholds**: Degraded and maximum response time monitoring - **Performance trend analysis**: Identifies gradual performance degradation - **Threshold-based alerting**: Proactive alerts based on configurable thresholds - **Location-specific performance**: Identifies regional performance differences - **Global availability monitoring**: Tracks service availability across regions - **Regional outage detection**: Identifies location-specific service issues - **Performance correlation**: Links geographic performance to infrastructure issues - **Current check status**: Real-time monitoring of check health - **Status transitions**: Tracks changes from healthy to degraded/failed states - **Alert state management**: Updates external alerts based on current performance - **Trend analysis**: Identifies patterns in check performance over time --- ## ClickHouse ## Overview Causely provides native integration with [ClickHouse](https://clickhouse.com/) to help you identify and resolve database issues before they impact your users. Instead of just monitoring Signals, Causely analyzes them in real time to surface the underlying Diagnosis driving database issues. By setting up the ClickHouse integration, you will be able to do the following: - Identify causes for reliability issues originating from your ClickHouse database, including: - [Table malfunction](/reference/root-causes/datapipeline#table-access-failure-in-database) and [mutation-based table locks](/reference/root-causes/datapipeline#contention-on-database-table-locks) - [Slow database queries](/reference/root-causes/applications#slow-database-queries) - [Connection exhaustion](/reference/root-causes/applications#database-connection-pool-saturated) - Observe the database as an entity in the Topology Graph, including its relationships to other entities on the service map, infrastructure stack, and dataflow map. - Get insights into the slowest queries over a rolling 12-hour window, and troubleshoot them with our [MCP Server](/agent-integration/mcp-server/) directly from your favorite agent. The integration supports both self-hosted ClickHouse instances and cloud-managed deployments. ## Setup Guide ### Step 1: Create a user Create a dedicated user in your ClickHouse instance and grant it access to the system tables that Causely requires: ```sql CREATE USER causely_user IDENTIFIED BY 'your-password'; GRANT SELECT ON system.tables TO causely_user; GRANT SELECT ON system.columns TO causely_user; GRANT SELECT ON system.databases TO causely_user; GRANT SELECT ON system.query_log TO causely_user; GRANT SELECT ON system.mutations TO causely_user; ``` **System-table grants alone are not sufficient.** The ClickHouse user must also be able to see the actual application tables in the databases Causely should discover. Without `SHOW`/`SELECT` on those databases, Causely cannot build table entities or relate them to your services. Grant database-wide visibility (choose the scope that fits your security model): ```sql GRANT SHOW DATABASES ON *.* TO causely_user; GRANT SELECT ON *.* TO causely_user; ``` For tighter scope, grant `SELECT` only on the application databases Causely should monitor: ```sql GRANT SHOW DATABASES ON *.* TO causely_user; GRANT SELECT ON .* TO causely_user; ``` Or, if you want multiple examples: ```sql GRANT SHOW DATABASES ON *.* TO causely_user; GRANT SELECT ON db1.* TO causely_user; GRANT SELECT ON db2.* TO causely_user; ``` How this maps to discovery: - **`system.databases`** controls whether Causely can discover database names. - **`system.tables`** controls whether Causely can discover tables inside those databases. If the user can see databases but not non-system tables, Causely will discover databases but create no useful entities for your application data. The integration reads from the following system tables: | Table | Purpose | | --- | --- | | `system.tables` | Table names, row counts, and sizes | | `system.columns` | Column definitions and schema information | | `system.databases` | Database discovery | | `system.query_log` | Slow query analysis (top 10 by total execution time, rolling 12-hour window) | | `system.mutations` | Active mutation detection for lock monitoring | ### Step 2: Create a Kubernetes secret for the user Create a Kubernetes secret with the ClickHouse connection details. The secret supports two protocols: - **Native** (default): binary protocol on port `9000` - **HTTP**: HTTP/HTTPS protocol on port `8123` / `8443` #### Option 1: Single Database Configuration ```bash kubectl create secret generic \ --namespace causely clickhouse-credentials \ --from-literal=username="causely_user" \ --from-literal=password='...' \ --from-literal=host="..." \ --from-literal=port="9000" \ --from-literal=database="..." \ --from-literal=protocol="native" \ --from-literal=secure="false" ``` To connect over HTTP instead of the native protocol: ```bash kubectl create secret generic \ --namespace causely clickhouse-credentials \ --from-literal=username="causely_user" \ --from-literal=password='...' \ --from-literal=host="..." \ --from-literal=port="8123" \ --from-literal=database="..." \ --from-literal=protocol="http" \ --from-literal=secure="false" ``` #### Option 2: Multiple Databases Configuration To monitor multiple databases within the same ClickHouse instance, specify them as a comma-separated list using the `databases` field: ```bash kubectl create secret generic \ --namespace causely clickhouse-credentials-multidb \ --from-literal=username="causely_user" \ --from-literal=password='...' \ --from-literal=host="..." \ --from-literal=port="9000" \ --from-literal=databases="database1,database2,database3" \ --from-literal=protocol="native" \ --from-literal=secure="false" ``` Alternatively, use a YAML manifest: ```yaml apiVersion: v1 kind: Secret metadata: name: clickhouse-credentials-multidb namespace: causely type: Opaque stringData: username: 'causely_user' password: '...' host: '...' port: '9000' databases: 'database1,database2,database3' protocol: 'native' secure: 'false' ``` **Note**: Use either the `database` field for a single database or the `databases` field for multiple databases. Do not use both in the same secret. #### Option 3: Database Auto-Discovery Causely can automatically discover all databases on a ClickHouse server. Add `auto_discovery: "true"` to the secret: ```bash kubectl create secret generic \ --namespace causely clickhouse-credentials \ --from-literal=username="causely_user" \ --from-literal=password='...' \ --from-literal=host="..." \ --from-literal=port="9000" \ --from-literal=protocol="native" \ --from-literal=secure="false" \ --from-literal=auto_discovery="true" ``` When auto-discovery is enabled, Causely queries `system.databases` (excluding `system`, `INFORMATION_SCHEMA`, and `information_schema`) and starts a scraper for each discovered database. Discovery runs periodically to pick up newly created databases. For external or VM-hosted ClickHouse, **use a stable DNS hostname whenever possible**. Raw IPs may connect successfully but can break service and entity resolution in Causely. The `host` value should align with how Causely discovers infrastructure (for example, the FQDN from the Kubernetes API or your cloud provider's API). If connectivity must use an IP address, set `host` to the working IP and set `host_overwrite` to the stable DNS hostname that identifies that ClickHouse instance: ```bash --from-literal=host="10.202.225.23" --from-literal=host_overwrite="clickhouse.internal.company.net" ``` If you are connecting through a proxy, set `host` to the proxy address and `host_overwrite` to the actual ClickHouse instance hostname: ```bash --from-literal=host="my-proxy.example.com" --from-literal=host_overwrite="my-clickhouse.example.com" ``` #### ClickHouse Cloud Configuration ClickHouse Cloud only accepts TLS connections, on different ports than a typical self-hosted instance: - **HTTPS**: port `8443` with `protocol="http"` and `secure="true"` - **Native over TLS**: port `9440` with `protocol="native"` and `secure="true"` Plaintext connections (the defaults used elsewhere on this page: port `9000`/`8123` with `secure="false"`) are not exposed by ClickHouse Cloud at all, so a secret copied from a self-hosted example will fail to connect. ```bash kubectl create secret generic \ --namespace causely clickhouse-credentials \ --from-literal=username="causely_user" \ --from-literal=password='...' \ --from-literal=host=".clickhouse.cloud" \ --from-literal=port="8443" \ --from-literal=database="..." \ --from-literal=protocol="http" \ --from-literal=secure="true" \ --from-literal=auto_discovery="true" kubectl --namespace causely label secret clickhouse-credentials "causely.ai/scraper=ClickHouse" ``` **`protocol` only accepts the literal values `native` or `http`.** `https` is *not* a valid value: if set, the scraper silently falls back to the native protocol instead of erroring, which then fails against Cloud's HTTPS-only port with a connection reset. TLS is controlled separately via `secure="true"`, not by the protocol name. #### Secret field reference | Field | Required | Default | Description | | --- | --- | --- | --- | | `host` | Yes | | Hostname or IP used to connect to ClickHouse. Prefer a stable DNS hostname for normal setups; if you must use an IP or proxy here, set `host_overwrite` to the canonical hostname for topology and entity resolution. | | `username` | Yes | | ClickHouse user name | | `password` | Yes | | ClickHouse user password | | `database` | Yes* | | Single database to monitor | | `databases` | Yes* | | Comma-separated list of databases to monitor | | `port` | No | `9000` (native) / `8123` (HTTP) | ClickHouse port. ClickHouse Cloud uses `8443` (HTTP) or `9440` (native TLS) instead: see [ClickHouse Cloud Configuration](#clickhouse-cloud-configuration). | | `protocol` | No | `native` | Connection protocol. Only `native` or `http` are valid: `https` is not recognized and silently falls back to `native`. | | `secure` | No | `false` | Enable TLS (`true` or `false`). Must be `true` for ClickHouse Cloud. | | `host_overwrite` | No | | Override the host used for topology and entity resolution. Use when the connection `host` is an IP or proxy address, but ClickHouse should be identified by a stable DNS hostname. | | `port_overwrite` | No | | Override the port used for entity resolution | | `auto_discovery` | No | `false` | Automatically discover all databases | *Either `database` or `databases` must be set, unless `auto_discovery` is enabled. ### Step 3: Update Causely Configuration Once the secret is created, update the Causely configuration to enable scraping for the new instance: ```yaml scrapers: clickhouse: enabled: true instances: - secretName: clickhouse-credentials namespace: causely ``` ### Alternative: Enable Credentials Autodiscovery Causely also supports credentials autodiscovery, which lets you add new scraping targets without modifying the Causely configuration. Label the Kubernetes secret to enable autodiscovery: ```bash kubectl --namespace causely label secret clickhouse-credentials "causely.ai/scraper=ClickHouse" ``` ## Verify Your Configuration After completing the setup, run these queries against your ClickHouse instance to verify that the Causely user has the required access. ### Troubleshooting Use these symptoms to narrow down configuration issues: - If the UI only shows a scraper path ending in `/_discovery`, database auto-discovery is running but no per-database scrapers were created (often because no **application databases** were found, or discovery could not proceed as expected). - If the UI shows per-database scrapers with a message like **configured database has no eligible tables**, the user can see the **configured databases** (or **target databases**) but they have no eligible tables for the current scraper logic (see [Eligible table types](#eligible-table-types) below). - If logs show **failed to resolve hostname** or **service mapping** errors, set `host_overwrite` to a stable DNS hostname and avoid relying on a raw IP alone for identity. - If nothing shows up at all and mediator logs show no mention of the scraper, confirm the secret is labeled `causely.ai/scraper=ClickHouse` (required for autodiscovery to notice the secret) or that a matching `instances` entry exists in the Causely configuration. - If logs show a ping/connection failure against ClickHouse Cloud (for example, `read: connection reset by peer` on an `https://` URL), double check `protocol` is exactly `http` or `native` (not `https`) and that `port`/`secure` match one of the combinations in [ClickHouse Cloud Configuration](#clickhouse-cloud-configuration). - **A scraper does not automatically retry after a failed initialization.** If you fix the secret after a failed connection attempt, you must also restart the mediator (`kubectl rollout restart deployment/mediator -n causely`) for the fix to take effect. ### Quick Access Check ```sql SELECT (SELECT count() FROM system.tables LIMIT 1) > 0 AS tables_ok, (SELECT count() FROM system.columns LIMIT 1) > 0 AS columns_ok, (SELECT count() FROM system.databases LIMIT 1) > 0 AS databases_ok, (SELECT count() FROM system.query_log LIMIT 1) >= 0 AS query_log_ok, (SELECT count() FROM system.mutations LIMIT 1) >= 0 AS mutations_ok; ``` All columns should return `1` (true). ### Detailed Checks #### 1. System tables access ```sql -- Each of these should return a result without error SELECT 1 FROM system.tables LIMIT 1; SELECT 1 FROM system.columns LIMIT 1; SELECT 1 FROM system.databases LIMIT 1; SELECT 1 FROM system.query_log LIMIT 1; SELECT 1 FROM system.mutations LIMIT 1; ``` If any query fails with an access denied error, grant the missing privilege to your Causely user: ```sql -- Run as admin GRANT SELECT ON system. TO causely_user; ``` #### 2. Test slow query collection Run this query to confirm Causely can collect slow query data: ```sql SELECT normalized_query_hash, count() AS calls, sum(query_duration_ms) AS total_exec_time_ms FROM system.query_log WHERE type = 'QueryFinish' AND event_time >= now() - toIntervalHour(12) GROUP BY normalized_query_hash ORDER BY total_exec_time_ms DESC LIMIT 5; ``` This should return results without error. An empty result set is normal on a freshly configured instance, entries will appear as queries run. #### 3. Database discovery {#database-discovery} Confirm application databases are visible to the Causely user (same filter Causely uses for auto-discovery): ```sql SELECT name FROM system.databases WHERE name NOT IN ('system', 'INFORMATION_SCHEMA', 'information_schema') ORDER BY name; ``` If this returns no application databases, auto-discovery will only create the `_discovery` scraper and no per-database scrapers. #### 4. Table visibility {#table-visibility} Confirm the Causely user can see non-system tables: ```sql SELECT database, name, engine FROM system.tables ORDER BY database, name LIMIT 100; ``` This result set must include **non-**`system` tables. If it only returns `system.*` tables, Causely will not discover application entities. #### 5. Eligible table types {#eligible-table-types} Replace `` with a **target database** from your configuration (or one returned by the [Database discovery](#database-discovery) query above). Causely only creates table entities for engines it treats as eligible: ```sql SELECT database, name, engine FROM system.tables WHERE database = '' AND engine NOT IN ('View', 'MaterializedView', 'Dictionary') ORDER BY name; ``` The current scraper **excludes** `View`, `MaterializedView`, and `Dictionary`. If this query returns zero rows, Causely reports that the configured database has no eligible tables and creates no ClickHouse table entities for that database. :::tip[Success] Your ClickHouse instance is correctly configured for Causely when **all** of the following are true: 1. System table checks pass (`system.tables`, `system.columns`, `system.databases`, `system.query_log`, `system.mutations`). 2. Application databases appear in `system.databases` for the Causely user (see [Database discovery](#database-discovery)). 3. Application (non-system) tables appear in `system.tables` for the Causely user (see [Table visibility](#table-visibility)). 4. At least one configured database contains at least one eligible table after excluding `View`, `MaterializedView`, and `Dictionary` (see [Eligible table types](#eligible-table-types)). ::: ### Setup Checklist - [ ] `causely_user` created in ClickHouse - [ ] `SELECT` granted on `system.tables`, `system.columns`, `system.databases`, `system.query_log`, `system.mutations` - [ ] `SHOW DATABASES` granted (for example `GRANT SHOW DATABASES ON *.*`) - [ ] `SELECT` granted on the application databases or tables Causely should discover (broad `*.*` or scoped per database) - [ ] Kubernetes secret created with correct `host`, `username`, `password`, `database`/`databases`, `protocol`, and `port` - [ ] For ClickHouse Cloud: `protocol` is exactly `http` or `native` (never `https`), `secure` is `"true"`, and `port` is `8443` or `9440` - [ ] If using an IP address or proxy for connectivity, `host_overwrite` is set to a stable DNS hostname for topology and entity resolution - [ ] Causely configuration updated, or secret labeled `causely.ai/scraper=ClickHouse` for autodiscovery - [ ] If a scraper previously failed to initialize, the mediator has been restarted after fixing the secret - [ ] Verification queries succeed without access errors - [ ] `system.tables` returns non-system tables for the Causely user - [ ] At least one configured database contains eligible tables after excluding `View`, `MaterializedView`, and `Dictionary` ## What Data is Collected The ClickHouse scraper collects comprehensive metadata and performance information from your ClickHouse databases, including: - **Database entities** with names and relationships to hosting services - **Service-to-database mappings** (which service provides which database) - **Connection details** including host, port, and protocol configuration - **Table information** for eligible tables discovered from `system.tables` (names, row counts, and sizes). The current scraper excludes `View`, `MaterializedView`, and `Dictionary` engines. - **Complete table schemas** for those eligible tables from `system.columns`, including column definitions, data types, default expressions, and comments - **Slow query analysis** using `system.query_log`: top 10 queries by total execution time over a rolling 12-hour window, including call counts, total and average execution time, and rows read - **Mutation lock monitoring** using `system.mutations`: active mutations are tracked as exclusive locks to detect contention on tables --- ## Confluent ## Overview Causely provides native integration with [Confluent Cloud](https://docs.confluent.io/cloud/current/overview.html) to help you identify and resolve streaming platform performance issues before they impact your users. Instead of just monitoring Signals, Causely analyzes them in real time from Confluent's Telemetry API to surface the actual Diagnosis behind streaming problems. This integration helps you identify the following causes for reliability issues, among others: - [Slow consumer](/reference/root-causes/applications#slow-consumer) behavior causing message processing bottlenecks - [Memory exhaustion](/reference/root-causes/applications#memory-exhaustion) from unbounded message backlogs - [Service congestion](/reference/root-causes/services#congested) affecting Kafka clusters - [Service malfunction](/reference/root-causes/services#malfunction) in streaming platform components The integration supports Confluent Cloud environments and clusters with comprehensive monitoring of Kafka topics, consumer groups, and cluster health metrics. ## Setup Guide ### Step 1: Create API Key and Secret Create an API key and secret for your Confluent Cloud account with the necessary permissions: 1. Log in to the [Confluent Cloud Console](https://confluent.cloud/) 2. Navigate to **API Keys** in the Administration section 3. Click **Create key** 4. Select **Cloud resource management** as the scope 5. Choose **My account** as the owner 6. Generate and securely store the API key and secret **Required Permissions:** - Read access to clusters, topics, and metrics - Access to Telemetry API for metrics collection ### Step 2: Create a Kubernetes Secret Create a Kubernetes secret containing your Confluent Cloud credentials: ```bash kubectl create secret generic \ --namespace causely confluent-secret \ --from-literal=CONFLUENT_CLOUD_KEY="" \ --from-literal=CONFLUENT_CLOUD_SECRET="" ``` Alternatively, you can create the secret using a YAML manifest: ```yaml apiVersion: v1 kind: Secret metadata: name: confluent-secret namespace: causely type: Opaque stringData: CONFLUENT_CLOUD_KEY: '' CONFLUENT_CLOUD_SECRET: '' ``` ### Step 3: Update Causely Configuration Enable the Confluent scraper in your Causely configuration by adding the following to your `values.yaml` file: #### Basic Configuration ```yaml scrapers: confluent: enabled: true accounts: - secretName: confluent-secret environments: - dev - prod clusters: - dev-cluster - prod-cluster ``` ### Alternative: Enable Credentials Autodiscovery Causely also supports credentials autodiscovery. This feature allows you to add new scraping targets without updating the Causely configuration. Label the Kubernetes secret to enable autodiscovery for the corresponding scraper. ```bash kubectl --namespace causely label secret confluent-secret "causely.ai/scraper=Confluent" ``` ## What Data is Collected The Confluent scraper collects comprehensive metadata and performance information from your Confluent Cloud deployment, including: - **Cluster entities** with display names, environment labels, and Kafka bootstrap endpoints (port 9092) - **Topic entities** with cluster relationships and labels (Environment, ClusterId, ClusterName, Topic) - **Consumer lag metrics** from the `io.confluent.kafka.server/consumer_lag_offsets` telemetry API - **TopicAccess entity updates** with real-time lag attributes for existing consumer group mappings - **Environment and cluster discovery** based on configuration filters - **Network endpoint mapping** for Kafka bootstrap server connectivity --- ## Datadog ## Enable Dual Shipping for Datadog APM Causely can leverage [Datadog APM instrumentation with dual shipping](https://docs.datadoghq.com/agent/configuration/dual-shipping/) to discover and monitor service dependencies. ## Option 1: Enable via Helm Deployment To enable dual shipping when deploying the Datadog Agent using **Helm**: 1. Add the following configuration to your `values.yaml`: ```yaml agents: useConfigMap: true customAgentConfig: apm_config: additional_endpoints: 'http://mediator.causely:8126': - 'datadog-receiver' ``` 2. Upgrade or install the Datadog Helm chart: ```bash helm upgrade --install datadog datadog/datadog -f ./values.yaml ``` ## Option 2: Enable via Datadog Operator If you're managing the Datadog Agent using the [Datadog Operator](https://docs.datadoghq.com/containers/datadog_operator/), modify the `DatadogAgent` custom resource as follows: ```yaml apiVersion: datadoghq.com/v2alpha1 kind: DatadogAgent metadata: name: datadog spec: override: nodeAgent: env: - name: DD_APM_ADDITIONAL_ENDPOINTS value: '{"http://mediator.causely:8126": ["datadog-receiver"]}' ``` ## Adding Host Identity Tags for EC2-Based Datadog APM Traces When applications run outside Kubernetes (such as on EC2 instances), Datadog APM traces may not include hostname or other identifying metadata by default. Without this information, Causely cannot associate incoming traces with the EC2 instances discovered from AWS. To ensure Causely can correctly match traces to EC2 hosts, add a unique identifying tag to the application's Datadog configuration: ```bash DD_TAGS="pm-name:" ``` Set this environment variable in the same location where Datadog APM configuration is applied, such as: - ECS task definitions - Docker environment variables - systemd unit files - CI/CD deployment configurations **Example (Docker):** ```bash docker run \ -e DD_AGENT_HOST="" \ -e DD_TAGS="pm-name:my-ec2-hostname" \ my-app:latest ``` Causely uses this tag to stitch Datadog APM traces to the corresponding EC2 instance in your environment model. For more information on Datadog tagging, refer to [Datadog's documentation](https://docs.datadog.com/getting_started/tagging/). --- ## Enabling Datadog Watchdog Monitors as a Data Source Causely can also leverage [Datadog monitors](https://docs.datadoghq.com/monitors/) for Postgres, Redis, and other integrations to be used as input to Casual Reasoning. To enable Datadog as a data source in Causely, add the following configuration to your `values.yaml`: ```yaml scrapers: datadog: enabled: true instances: - secretName: datadog-credentials # namespace: your-namespace # optional; defaults to the pod namespace # event_tag_filters: [] # optional # monitor_mapping: [] # optional ``` ### Alternative: Enable Credentials Autodiscovery Causely supports credentials autodiscovery so you can add new accounts without editing config. Label the Kubernetes secret to enable autodiscovery for the Datadog scraper: ```bash kubectl --namespace causely label secret datadog-credentials "causely.ai/scraper=Datadog" ``` For the secret, you can use the following example: ```yaml apiVersion: v1 kind: Secret metadata: name: datadog-credentials type: Opaque stringData: org: 'YourOrg' apiKey: '' appKey: '' ``` The optional event_tag_filters is used to filter triggered Datadog events, for example: ```yaml event_tag_filters: - 'pagerduty' - 'integration:postgres' - 'integration:redisdb' ``` The optional monitor_mapping is used to map Datadog monitors to Causely entities, for example: ```yaml monitor_mapping: - entityType: ApplicationLoadBalancer createIfMissing: true attributes: service_name: label: ['hostname'] isIdentifier: true resources: - monitorId: 123267804 attribute: RequestsTotal ``` Without the monitor_mapping, Causely will try to automatically map the latency and error rate monitors to the Causely Service corresponding Signals. For example, the following event will be mapped automatically to the high error rate Signal of the `causely-analysis` service: ```json { "id": "AwAAAZiZU7KAMHtQKwAAABhBWmlaVlJXNUFBRFhWU3hKanRwb1ZyYl8AAAAkMTE5ODk5ZGMtMjgyYS00YWJmLWFjYzUtNmQ4NjkyZjBlM2FlAAEjVg", "type": "event", "attributes": { "attributes": { "aggregation_key": "2b57d1af567993ace06eebaf9dc0e669", "evt": { "uid": "AZiZVRiqAABt7uudQBprAQAA", "name": "Errors are high", "id": "8231846403567092370", "source_id": 36, "type": "log_alert" }, "monitor_id": 150830126, "monitor_notifications": [ "causely-alerts" ], "monitor": { "group_status": 5, "alert_cycle_key_txt": "8231846399607848082", "query": "logs(\"service:(causely-analysis*production) status:error -service:* -service:*compliancev2* -service:*ocr* -\\\"Request failed with status code 404\\\" -\\\"#8b6d6338\\\"\").index(\"*\").rollup(\"count\").by(\"service\").last(\"10m\") > 2000", "groups": [ "service:causely-analysis" ], "created_at": 1723077612000, "priority": 1, "type": "log alert", "transition": { "destination_state": "Warn", "transition_type": "warn", "source_state": "OK" }, "tags": [ "team:analysis-team" ], "result": { "result_id_txt": "8231846397529909389", "result_id": 8231846397529909389, "result_ts": 1754919056, "group_key": "service" }, "name": "Errors are high", "options": { "on_missing_data": "default", "thresholds": { "critical": 2000, "warning": 1000 }, "new_group_delay": 0, "enable_logs_sample": true, "include_tags": false, "groupby_simple_monitor": false, "notify_audit": false }, "modified": 1754582393000, "id": 150830516, "templated_name": "Errors are high in production", }, "priority": "normal", "title": "[P1] [Warn] Errors are high", "service": "causely-analysis", "sourcecategory": "monitor_alert", "event_object": "047235cd648d8d01abf7bc03c7c11bc8", "_dd": { "has_notification": false, "internal": "1", "version": "1" }, "timestamp": 1754919056000, "status": "warning" }, "timestamp": "2025-08-11T13:30:56Z", "tags": [ "monitor", "priority:p1", "service:causely-analysis", "source:alert", "team:analysis-team" ] } }, ``` --- ## Docker ## Overview Causely provides **native integration** with [Docker](https://www.docker.com/) that automatically discovers and monitors your container infrastructure. The integration provides visibility across your Docker environment, from host-level resources down to individual containers, helping you identify infrastructure issues before they impact your applications. ## How It Works Causely's Docker integration automatically discovers containers using the Docker API, monitors resource usage and states, creates a topology graph showing relationships between hosts, containers, workloads, and applications, and creates network endpoints for service discovery. Docker monitoring works automatically when you install Causely using the [Docker installation guide](/installation/docker). --- ## Dynatrace Causely integrates with [Dynatrace](https://www.dynatrace.com/platform/application-observability/) to ingest service dependencies and trace-based metrics, from both Kubernetes and ECS environments. This enables Causely to build a unified causal model across mixed container platforms and generate a dependency and service map that accurately reflects how your system behaves under change. By incorporating Dynatrace traces and service-level metrics, Causely strengthens reliability assurance across hybrid environments. The setup process below applies to both Kubernetes and ECS-based services. ## Supported environments - Kubernetes - ECS (via Dynatrace OneAgent / Dynatrace APM instrumentation) ## Step 1: Create a user Create an API token for your Dynatrace account server. ## Step 2: Create a Kubernetes Secret with the Dynatrace URL and API token After creating the API token create a Kubernetes Secret: ```bash kubectl create secret generic \ --namespace causely dynatrace-credentials \ --from-literal=url="..." \ --from-literal=api-token="..." ``` ## Step 3: Update Causely configuration Once the secret is created, update the Causely configuration to enable scraping for the dynatrace account. Enabling scraping allows Causely to ingest Dynatrace topology and telemetry for both Kubernetes and ECS services. Below is an example configuration: ```yaml scrapers: dynatrace: enabled: true accounts: - secretName: dynatrace-credentials ``` ### Alternative: Enable Credentials Autodiscovery Causely supports credentials autodiscovery so you can add new accounts without editing config. Label the Kubernetes secret to enable autodiscovery for the Dynatrace scraper: ```bash kubectl --namespace causely label secret dynatrace-credentials "causely.ai/scraper=Dynatrace" ``` ## Unified Stitching Across Kubernetes and ECS When Dynatrace is enabled, Causely automatically discovers and models dependencies between ECS and Kubernetes services, creating a unified causal view of your environment. Causely will: - Discover and stitch dependencies across Kubernetes and ECS services, even when they span clusters or runtimes - Include service-level metrics (request rate, error rate, duration) and corresponding Signals from ECS as part of the causal model - Display ECS services alongside Kubernetes services in Causely’s environment graph, dependency views, and reliability analysis --- ## eBPF [eBPF](https://ebpf.io/) (extended Berkeley Packet Filter) is a cutting-edge Linux kernel technology that allows safe, efficient, and programmable data collection at runtime. It enables deep observability into your system and application behavior without modifying code or adding performance overhead. Causely leverages **OpenTelemetry eBPF instrumentation**, powered by [Grafana Beyla](https://grafana.com/docs/beyla/latest/), to automatically extract rich telemetry data from your services. This zero-effort integration provides actionable insights into service interactions, latencies, and system performance, with zero changes to application code. ## How It Works Causely's eBPF-based instrumentation follows a streamlined process: 1. The system inspects running applications to determine which technologies they use. 2. Once application characteristics are identified, eBPF uprobes are attached to specific user-space functions to capture application-level information, for example HTTP or gRPC calls. 3. These probes generate events related to the application's traffic, which are then converted into OpenTelemetry metrics and traces. 4. The collected telemetry is sent to the [mediator](/getting-started/architecture#mediation-layer), which will perform Signal detection, topology discovery and local processing and then send distilled insights to the Causely engine. This approach supports multiple programming languages including Go, Java, .NET, NodeJS, Python, Ruby, Rust, and more, without requiring any code modifications or language-specific agents. To learn more about how the automatic instrumentation works, see the [Grafana Beyla documentation](https://grafana.com/docs/beyla/latest/). ## Technical Implementation ### Uprobe-Based Instrumentation Causely's eBPF instrumentation uses **uprobes exclusively** to intercept specific user-space functions within your applications. This approach: - Attaches probes to application-level functions (HTTP handlers, gRPC calls, database clients) - Does **not** hook into kernel networking callbacks or the packet datapath - Does **not** act as a CNI or critical network infrastructure component - Does **not** intercept or modify network packets The instrumentation may inject trace context headers (such as W3C Trace Context) into outgoing requests to enable distributed tracing. However, it does not intercept, block, or route network traffic at the kernel level. This design choice is intentional. Kernel-level network interception (kprobes on network syscalls) can introduce per-packet latency and increase blast radius. By instrumenting at the application layer, Causely captures the telemetry needed for causal analysis without impacting network connectivity or acting as a critical path component. Beyla is built on the [OpenTelemetry eBPF Instrumentation (OBI)](https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation) project, which removed kprobe-based collection in favor of uprobe-only approaches ([PR #752](https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/pull/752)). ### Performance Characteristics - **No per-packet overhead**: Probes fire on application-level function calls, not network packets - **Targeted instrumentation**: Only specific functions are instrumented, not all syscalls - **Minimal CPU impact**: Overhead is proportional to application request rate, not network traffic volume - **Fail-safe design**: If an agent fails, applications continue to function normally, with no impact to traffic flow ## Configuration ### Java instrumentation For automatic instrumentation of Java applications the HotSpot JVM supports dynamic agent loading, attaching the Beyla/OBI Java agent at runtime to observe TLS encrypted traffic. To enable [this functionality](https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/pull/1021) modify the `causely-values.yaml` configuration: ``` agent: beyla: global_config: javaagent: enabled: true ``` ### GenAI instrumentation For automatic instrumentation of GenAI applications, with a larger payload extraction Beyla/OBI can instrument client LLM calls with GenAI semantic conventions: To enable [this functionality](https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/pull/1353) modify the `causely-values.yaml` configuration: ``` agent: beyla: global_config: ebpf: buffer_sizes: http: 8192 payload_extraction: http: genai: openai: enabled: true anthropic: enabled: true gemini: enabled: true ``` ### Application metrics (BeylaMetrics) Beyla/OBI can export [application-level HTTP and RPC metrics](https://opentelemetry.io/docs/zero-code/obi/metrics/) (request rates, status codes, and latency histograms). Causely's `BeylaMetrics` scraper reads these metrics from Prometheus (or a compatible store such as Mimir) to discover service-to-service access relationships and SLOs. #### 1. Enable Beyla Prometheus export Expose Beyla's Prometheus metrics endpoint by setting a port in `causely-values.yaml`: ```yaml agent: beyla: prometheus_export: port: 9090 ``` The Causely agent runs with `hostNetwork: true`, so this port is bound on every node. Choose a port that does not conflict with other host-network services or NodePorts on the cluster. Ensure your Prometheus scrapes these endpoints. When `serviceMonitor.enabled` is true and a port is set, Causely also creates a PodMonitor for the `beyla-metrics` port. #### 2. Point Causely at the scraped metrics Configure the `BeylaMetrics` scraper with the Prometheus API endpoint that holds the Beyla application metrics - similar to the [Istio Prometheus integration](/telemetry-sources/istio): ```yaml scrapers: beylametrics: prometheus: endpoint: http://prometheus.monitoring:9090 ``` Replace the endpoint with your Prometheus or Mimir query URL. Alternatively, configure the same connection from the [Integrations page](https://portal.causely.app/integrations) in the Causely UI, or by creating a Kubernetes Secret (no Helm change required). With auto-discovery enabled (the default), label the secret so the mediator picks it up: ```yaml apiVersion: v1 kind: Secret metadata: name: beyla-metrics-prometheus namespace: causely labels: causely.ai/scraper: BeylaMetrics type: Opaque stringData: endpoint: http://prometheus.monitoring:9090 # username: readonly # optional basic auth # password: ... ``` ### Disabling autoinstrumentation eBPF-based autoinstrumentation is **enabled by default** in Causely deployments. If you wish to disable it, you can modify the `causely-values.yaml` configuration: ```yaml scrapers: bpf: enabled: false ``` ### Including specific workloads eBPF-based autoinstrumentation can be selectively scoped per namespace or per deployment in a Kubernetes environment. Modify `causely-values.yaml` as needed. To whitelist instrumentation to a single Kubernetes deployment, including every pod managed by that deployment, use `discovery.instrument` with `k8s_deployment_name` (and optional port selectors). When multiple selectors appear in the same `instrument` entry, Beyla only instruments processes that match all of them: ```yaml agent: beyla: discovery: instrument: - open_ports: 80,443,2000-10000 k8s_deployment_name: "{payment,cart}" ``` ### Excluding workloads To exclude workloads from instrumentation: ```yaml agent: beyla: discovery: exclude_instrument: - k8s_namespace: {"nginx-ingress","ingress-nginx"} - k8s_deployment_name: "{cnpg-cloudnative-pg,ingress-nginx-controller}" ``` See the [Beyla service discovery documentation](https://grafana.com/docs/beyla/latest/configure/service-discovery/) for other selectors (`k8s_namespace`, `k8s_pod_labels`, and more). :::note If your application is already instrumented with OpenTelemetry, the agent will detect this and not instrument it again. ::: --- ## Elasticsearch ## Overview Causely integrates with [Elasticsearch](https://www.elastic.co/elasticsearch/) in two ways: 1. **Log Retrieval**: Causely can ingest and analyze container logs stored in Elasticsearch. This allows you to correlate error and exception patterns with service-level incidents and Diagnoses. 2. **Cluster Performance Insights**: Causely connects directly to Elasticsearch cluster APIs to detect node-level or infrastructure bottlenecks that can cause degraded application performance. Use this integration if: - Your logs are already centralized in Elasticsearch. - If you operate Elasticsearch clusters, Causely goes beyond Signal monitoring by analyzing them in real time to surface the actual Diagnosis behind Elasticsearch problems, helping you identify reliability issues such as: - [Memory Pressure](/reference/root-causes/infrastructure#memory-pressure) and [Disk Pressure](/reference/root-causes/infrastructure#disk-pressure) - [CPU Congested](/reference/root-causes/infrastructure#cpu-congested) and [Memory Congested](/reference/root-causes/infrastructure#memory-congested) - [Memory Failure](/reference/root-causes/infrastructure#memory-failure) and [Frequent Memory Failure](/reference/root-causes/infrastructure#frequent-memory-failure) - [Disk Total IOPs Congested](/reference/root-causes/infrastructure#disk-total-iops-congested) and [Inode Usage Congested](/reference/root-causes/infrastructure#inode-usage-congested) ## Use Case 1: Log Retrieval from Elasticsearch When your logs are already shipped to Elasticsearch, for example via Fluentd, Logstash, or Beats, Causely can pull those logs directly to enhance the Diagnosis. Instead of collecting logs from Kubernetes directly, Causely queries Elasticsearch indices for logs tied to specific services or containers. ### Benefits - Automatically surfaces relevant logs in the context of an active Diagnosis or service malfunction. - Displays container-level logs under affected services when abnormal behavior occurs, for example error spikes or degraded performance. - Shows log lines and exceptions alongside Diagnoses to validate issues and dramatically shorten time to understanding and resolution. ### Setup Create a Kubernetes Secret for your Elasticsearch credentials (if authentication is required): ```bash kubectl create secret generic elasticsearch-logs-credentials \ --namespace causely \ --from-literal=api_key="your-api-key" ``` Or for basic authentication: ```bash kubectl create secret generic elasticsearch-logs-credentials \ --namespace causely \ --from-literal=username="your-username" \ --from-literal=password="your-password" ``` ### How Elasticsearch Works #### Schema Flexibility Elasticsearch is schema-less (mostly): Unlike SQL databases, you don't define a strict schema upfront. - **Dynamic mapping**: When you index a document, Elasticsearch automatically creates field mappings based on what it sees - **No enforcement**: Different documents in the same index can have completely different fields #### Different Log Shippers/Collectors Different log shipping tools structure data differently: - **Fluentd**: Creates fields like `data.message`, `data.priority`, adds `fluentd: "true"` - **Filebeat**: Might use `message`, `log.level`, `kubernetes.namespace` - **Logstash**: Often uses `message`, `@timestamp`, custom parsed fields This is why Causely requires explicit field mappings: to know where to find pod names, messages, severity, and other metadata in your specific Elasticsearch index structure. Then update your Kubernetes scraper configuration with the Elasticsearch endpoint, indices, and field mappings: ```yaml scrapers: kubernetes: elasticsearch: endpoint: 'http://elasticsearch.logging.svc.cluster.local:9200' secret: 'elasticsearch-logs-credentials' # Optional: omit if no auth required indices: - 'logstash-app_log-*' # Example: Fluentd/Logstash format # - "filebeat-*" # Example: Filebeat format fields: timestamp: '@timestamp' message: 'data.message' # Example: nested field (Fluentd) # message: "message" # Example: flat field (Filebeat) severity: 'data.priority' # Example: explicit severity field (optional) pod: 'host' # Example: Fluentd uses "host" field # pod: "kubernetes.pod.name" # Example: Filebeat uses standard k8s metadata namespace: '' # Optional: set if available container: '' # Optional: set if available ``` The field mappings allow you to configure how Causely extracts pod names, namespaces, containers, messages, and severity from your Elasticsearch documents. This supports different log shipping formats (Fluentd, Filebeat, Logstash, etc.) with varying field structures. ### Authentication Options - **API Key**: Include `api_key` in the Kubernetes secret - **Basic Auth**: Include `username` and `password` in the Kubernetes secret - **No Auth**: Omit the `secret` field in the configuration if authentication is not required ### Supported Log Types - Container logs (stdout/stderr) ## Use Case 2: Cluster Performance Monitoring ## Setup Guide ### Step 1: Create an API Key Create an API Key for your Elasticsearch cluster with the following permissions: - **Cluster monitoring**: Required to access cluster health and node statistics - **Node stats**: Required to monitor individual node performance and resource usage - **Read access**: Required to query cluster metadata and configuration For Elastic Cloud: 1. Go to Elastic Cloud Console → Security → API Keys 2. Create a new API key with appropriate permissions 3. Copy the API key for use in the next step For self-hosted Elasticsearch: 1. Use Kibana Security → Users → Create API Key 2. Or use the Elasticsearch API: `POST /_security/api_key` 3. Ensure the API key has monitoring privileges ### Step 2: Create a Kubernetes Secret for the API Key After creating the API Key create a Kubernetes Secret: ```bash kubectl create secret generic \ --namespace causely elastic-credentials \ --from-literal=api_key="..." \ --from-literal=url='https://....eastus2.azure.elastic-cloud.com' ``` The `url` must be the endpoint URL of your Elasticsearch cluster: - **Elastic Cloud**: `https://your-cluster-id.region.elastic-cloud.com` - **AWS OpenSearch**: `https://your-domain.region.es.amazonaws.com` - **Azure Search**: `https://your-service.search.windows.net` - **Self-hosted**: `https://your-elasticsearch-host:9200` ### Step 3: Update Causely Configuration Once the Secret is created, update the Causely configuration to enable scraping for the new cluster. Below is an example configuration: ```yaml scrapers: elasticsearch: enabled: true instances: - secretName: elastic-credentials namespace: causely ``` ### Alternative: Enable Credentials Autodiscovery Causely also supports credentials autodiscovery. This feature allows you to add new scraping targets without updating the Causely configuration. Label the Kubernetes Secret to enable autodiscovery for the corresponding scraper. ```bash kubectl --namespace causely label secret elastic-credentials "causely.ai/scraper=ElasticSearch" ``` ## What Data is Collected ### From Logs - Error and warning messages from container stdout/stderr logs linked to affected services or Diagnoses. ### From Cluster Performance - **Cluster entities** with names and health status - **Service-to-cluster mappings** (which service provides the Elasticsearch cluster) - **Connection details** including endpoint URL and API authentication - **Cluster health metrics** including shard allocation status - **Cluster status** (green, yellow, red) and health indicators - **Shard allocation metrics** (active, relocating, initializing, unassigned shards) - **Task queue monitoring** (pending tasks, in-flight operations, queue wait times) - **Active shards percentage** for overall cluster health - **Node information** including names, roles, and attributes - **File descriptor usage** (`FileDescriptorUsage`, `FileDescriptorCapacity`) - **Memory utilization** (`MemoryUsage`, `MemoryCapacity`) - **CPU performance** (`CPUUsage`, `CPUCapacity`) - **Load average metrics** (1m, 5m, 15m) for trend analysis - **Disk space metrics** (`Usage`, `Capacity`) - **File system statistics** (total, free, available bytes, watermarks) - **I/O performance metrics** (read/write operations, data transfer, I/O time) - **JVM memory pool statistics** (heap used/committed/max, non-heap usage) - **Garbage collection metrics** (collection count, time per collector) - **Thread statistics** (current/peak thread counts, thread pool utilization) - **Buffer pool performance** (buffer count, usage, capacity) - **Class loading statistics** (loaded/unloaded classes) - **Operating system metrics** (memory, CPU, swap usage) - **Cgroup metrics** for containerized deployments - **Transport layer information** (addresses, ports) - **Network endpoint mapping** for service discovery - **Host and IP address tracking** for infrastructure mapping - **Service-to-node mappings** - **Node-to-VM relationships** - **VM-to-disk relationships** - **Attribute-based labeling** for custom categorization ## Result - Captures and displays logs alongside a Diagnosis to highlight the precise errors or stack traces that occurred around the time of failure. - Detects Elasticsearch cluster issues that may propagate to upstream applications. - Automatically identifies and explains the underlying Diagnosis, rather than requiring manual investigation across dashboards. --- ## GCP ## Overview Causely provides comprehensive native integration with Google Cloud Platform (GCP) to help you identify and resolve infrastructure and managed service reliability issues before they impact your users. Instead of only monitoring isolated metrics or alerts, Causely analyzes real-time signals from your GCP environment to infer the underlying causes of performance degradation and reliability risk across your cloud services. By setting up the GCP integration, you will be able to: **Identify causes for reliability issues originating from your GCP infrastructure and services**, including: - [Service Congestion](/reference/root-causes/services#congested-1) and [Service Malfunction](/reference/root-causes/services#malfunction-1) - [Application Load Balancer Misconfiguration](/reference/root-causes/services#application-load-balancer) issues - [Authentication](/reference/root-causes/services#authentication-misconfiguration) and [Network Policy](/reference/root-causes/services#network-policy-misconfiguration) misconfigurations - [Memory Pressure](/reference/root-causes/infrastructure#memory-pressure) and [CPU Congested](/reference/root-causes/infrastructure#cpu-congested) issues - [Disk Pressure](/reference/root-causes/infrastructure#disk-pressure) and [I/O congestion](/reference/root-causes/infrastructure#disk-total-iops-congested) Observe GCP resources as first-class entities in the Causely Topology Graph, including their relationships to Kubernetes workloads, managed services, and downstream dependencies Monitor performance and health metrics from GCP-native sources with automatic correlation to service impact and causal diagnoses To set up this integration, you need a Service Account with least-privilege [predefined viewer roles](https://docs.cloud.google.com/iam/docs/understanding-roles) on the GCP projects you want to monitor. ## Supported GCP Services Causely integrates with the following GCP services to provide comprehensive observability and causal inference: - Cloud Bigtable - Cloud Load Balancing - Cloud Memorystore for Redis - Cloud Memorystore for Redis (Cluster mode) - Cloud Pub/Sub - Cloud Run - Cloud Service Mesh / Traffic Director (proxyless gRPC) - Cloud SQL (MySQL and PostgreSQL) - Compute Engine - Persistent Disks ## Step 1: Create a Service Account (SA) Use the following `gcloud` command to create an SA: ```bash gcloud iam service-accounts create causely \ --description="causely" \ --display-name="causely" gcloud projects add-iam-policy-binding PROJECT_ID \ --member="serviceAccount:causely@PROJECT_ID.iam.gserviceaccount.com" \ --role="roles/compute.viewer" gcloud projects add-iam-policy-binding PROJECT_ID \ --member="serviceAccount:causely@PROJECT_ID.iam.gserviceaccount.com" \ --role="roles/compute.networkViewer" gcloud projects add-iam-policy-binding PROJECT_ID \ --member="serviceAccount:causely@PROJECT_ID.iam.gserviceaccount.com" \ --role="roles/logging.viewer" gcloud projects add-iam-policy-binding PROJECT_ID \ --member="serviceAccount:causely@PROJECT_ID.iam.gserviceaccount.com" \ --role="roles/monitoring.viewer" gcloud projects add-iam-policy-binding PROJECT_ID \ --member="serviceAccount:causely@PROJECT_ID.iam.gserviceaccount.com" \ --role="roles/bigtable.viewer" gcloud projects add-iam-policy-binding PROJECT_ID \ --member="serviceAccount:causely@PROJECT_ID.iam.gserviceaccount.com" \ --role="roles/cloudsql.viewer" gcloud projects add-iam-policy-binding PROJECT_ID \ --member="serviceAccount:causely@PROJECT_ID.iam.gserviceaccount.com" \ --role="roles/memcache.viewer" gcloud projects add-iam-policy-binding PROJECT_ID \ --member="serviceAccount:causely@PROJECT_ID.iam.gserviceaccount.com" \ --role="roles/pubsub.viewer" gcloud projects add-iam-policy-binding PROJECT_ID \ --member="serviceAccount:causely@PROJECT_ID.iam.gserviceaccount.com" \ --role="roles/redis.viewer" gcloud projects add-iam-policy-binding PROJECT_ID \ --member="serviceAccount:causely@PROJECT_ID.iam.gserviceaccount.com" \ --role="roles/run.viewer" gcloud projects add-iam-policy-binding PROJECT_ID \ --member="serviceAccount:causely@PROJECT_ID.iam.gserviceaccount.com" \ --role="roles/networkservices.viewer" ``` Role definitions: [Monitoring Viewer](https://docs.cloud.google.com/iam/docs/roles-permissions/monitoring#monitoring.viewer), [Compute Viewer](https://docs.cloud.google.com/iam/docs/roles-permissions/compute#compute.viewer), [Compute Network Viewer](https://docs.cloud.google.com/iam/docs/roles-permissions/compute#compute.networkViewer), [Logging Viewer](https://docs.cloud.google.com/iam/docs/roles-permissions/logging#logging.viewer), [Bigtable Viewer](https://docs.cloud.google.com/iam/docs/roles-permissions/bigtable#bigtable.viewer), [Cloud SQL Viewer](https://docs.cloud.google.com/iam/docs/roles-permissions/cloudsql#cloudsql.viewer), [Memorystore for Memcached Viewer](https://docs.cloud.google.com/iam/docs/roles-permissions/memcache#memcache.viewer), [Pub/Sub Viewer](https://docs.cloud.google.com/iam/docs/roles-permissions/pubsub#pubsub.viewer), [Memorystore for Redis Viewer](https://docs.cloud.google.com/iam/docs/roles-permissions/redis#redis.viewer), [Cloud Run Viewer](https://docs.cloud.google.com/iam/docs/roles-permissions/run#run.viewer), [Network Services Viewer](https://docs.cloud.google.com/iam/docs/roles-permissions/networkservices#networkservices.viewer). `roles/networkservices.viewer` is required to discover Cloud Service Mesh / Traffic Director resources (Mesh and GRPCRoutes). Backend service and Network Endpoint Group (NEG) enrichment for linking mesh services to Kubernetes backends uses the existing `roles/compute.viewer` and `roles/compute.networkViewer` roles. Grant these roles on every project that owns GRPCRoutes, backend services, or NEGs you want Causely to discover (for example, shared mesh projects and service projects). Alternatively, for simplicity, you can grant the broader `roles/viewer` role on the project instead of the granular roles above. ## Step 2: Provide GCP Credentials to the Mediator Choose one of the following authentication options. ### Option A: Service Account JSON After creating the SA, save its credentials .json in a Kubernetes Secret. ```bash kubectl create secret --namespace causely generic gcp-credentials \ --from-file=service_account.json=path_to_service_account.json \ --from-literal=project_id=PROJECT_ID ``` ### Option B: GKE Workload Identity Federation (Recommended) Use Workload Identity to bind a Kubernetes Service Account (KSA) to a Google Service Account (GSA). See the GCP guide for details: [Kubernetes Engine Workload Identity guide](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#authenticating_to). 1) Grant the KSA permission to impersonate the GSA: ```bash gcloud iam service-accounts add-iam-policy-binding @.iam.gserviceaccount.com \ --role roles/iam.workloadIdentityUser \ --member "serviceAccount:.svc.id.goog[/]" ``` Example: ```bash gcloud iam service-accounts add-iam-policy-binding monitor@playground-377422.iam.gserviceaccount.com \ --role roles/iam.workloadIdentityUser \ --member "serviceAccount:playground-377422.svc.id.goog[causely/causely-mediator]" \ --project=playground-377422 ``` 2) Annotate the mediator KSA in `values.yaml` to use the GSA: ```yaml mediator: serviceAccount: annotations: iam.gke.io/gcp-service-account: @.iam.gserviceaccount.com ``` 3) Create a Kubernetes Secret to enable the GCP integration ```bash kubectl create secret --namespace causely generic gcp-credentials ``` ## Step 3: Update Causely Configuration Once the Secret is created, update the Causely configuration to enable scraping for the new project. Below is an example configuration: ```yaml scrapers: gcp: enabled: true projects: - secretName: gcp-credentials namespace: causely ``` ## Alternative: Enable Credentials Autodiscovery Causely also supports credentials autodiscovery. This feature allows you to add new scraping targets without updating the Causely configuration. Simply label the Kubernetes Secret to enable autodiscovery for the corresponding scraper. ### Example: Labeling Secrets for Autodiscovery ```bash kubectl --namespace causely label secret gcp-credentials "causely.ai/scraper=GCP" ``` With these steps, you can seamlessly integrate Causely with the Google Cloud API and configure it to monitor your desired resources. ## Configure individual Resources You have to enable the resources you want causely to discover and monitor. --- ## Grafana Causely can leverage [Grafana](https://grafana.com/products/cloud/) as an observability backend with Grafana [Alloy](https://grafana.com/oss/alloy-opentelemetry-collector/) and [Beyla](https://grafana.com/oss/beyla-ebpf/) as instrumentation. > **Note:** If your logs are centralized in Grafana **Loki**, Causely can retrieve and display them directly. > See the [Loki Integration](#loki-integration) section below for configuration details and benefits. Please use the following [values](/telemetry-sources/grafana#alloy-valuesyaml) for your helm install: ## Install with Causely ```shell helm upgrade --install alloy grafana/alloy --create-namespace --namespace monitoring --values ./alloy-values.yaml ``` ### alloy-values.yaml ```yaml controller: type: deployment alloy: stabilityLevel: experimental extraPorts: - name: 'grpc' port: 4317 targetPort: 4317 - name: 'http' port: 4318 targetPort: 4318 - name: 'datadog' port: 8126 targetPort: 8126 configMap: content: | otelcol.exporter.otlp "causely" { client { endpoint = "mediator.causely:4317" tls { insecure = true } } } otelcol.processor.batch "default" { output { metrics = [otelcol.exporter.otlp.causely.input] traces = [otelcol.exporter.otlp.causely.input] } } otelcol.processor.k8sattributes "default" { extract { label { from = "pod" } metadata = [ "k8s.namespace.name", "k8s.pod.name", "k8s.pod.uid", "k8s.deployment.name", "k8s.node.name", "k8s.pod.start_time", "container.id", ] } output { traces = [otelcol.processor.batch.default.input] } } otelcol.processor.deltatocumulative "default" { output { metrics = [otelcol.processor.batch.default.input] } } otelcol.receiver.datadog "default" { endpoint = "0.0.0.0:8126" output { metrics = [otelcol.processor.deltatocumulative.default.input] traces = [otelcol.processor.batch.default.input] } } otelcol.receiver.otlp "otlp" { grpc { endpoint = "0.0.0.0:4317" } http { endpoint = "0.0.0.0:4318" } output { metrics = [otelcol.processor.batch.default.input] traces = [otelcol.processor.k8sattributes.default.input] } } ``` ## Install with Causely and Grafana Cloud To also send logs, metrics, and traces to Grafana Cloud, you need to use the following [values](/telemetry-sources/grafana#grafana-alloy-valuesyaml) for your helm install: ```shell helm upgrade --install alloy grafana/alloy --create-namespace --namespace monitoring --values ./grafana-alloy-values.yaml ``` ### grafana-alloy-values.yaml ```yaml controller: hostPID: true alloy: securityContext: privileged: true stabilityLevel: experimental extraPorts: - name: 'grpc' port: 4317 targetPort: 4317 - name: 'http' port: 4318 targetPort: 4318 - name: 'datadog' port: 8126 targetPort: 8126 configMap: content: | otelcol.exporter.otlp "causely" { client { endpoint = "mediator.causely:4317" tls { insecure = true } } } otelcol.exporter.otlphttp "grafana" { client { endpoint = "https://otlp-gateway-prod-us-east-0.grafana.net/otlp" auth = otelcol.auth.basic.grafana.handler } } otelcol.auth.basic "grafana" { username = "GRAFANA_CLOUD_INSTANCE_ID" password = "GRAFANA_CLOUD_API_KEY" } // discovery.kubernetes allows you to find scrape targets from Kubernetes resources. // It watches cluster state and ensures targets are continually synced with what is currently running in your cluster. discovery.kubernetes "pod_logs" { role = "pod" } // discovery.relabel rewrites the label set of the input targets by applying one or more relabeling rules. // If no rules are defined, then the input targets are exported as-is. discovery.relabel "pod_logs" { targets = discovery.kubernetes.pod_logs.targets // Label creation - "namespace" field from "__meta_kubernetes_namespace" rule { source_labels = ["__meta_kubernetes_namespace"] action = "replace" target_label = "namespace" } // Label creation - "pod" field from "__meta_kubernetes_pod_name" rule { source_labels = ["__meta_kubernetes_pod_name"] action = "replace" target_label = "pod" } // Label creation - "container" field from "__meta_kubernetes_pod_container_name" rule { source_labels = ["__meta_kubernetes_pod_container_name"] action = "replace" target_label = "container" } // Label creation - "app" field from "__meta_kubernetes_pod_label_app_kubernetes_io_name" rule { source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_name"] action = "replace" target_label = "app" } // Label creation - "job" field from "__meta_kubernetes_namespace" and "__meta_kubernetes_pod_container_name" // Concatenate values __meta_kubernetes_namespace/__meta_kubernetes_pod_container_name rule { source_labels = ["__meta_kubernetes_namespace", "__meta_kubernetes_pod_container_name"] action = "replace" target_label = "job" separator = "/" replacement = "$1" } // Label creation - "container" field from "__meta_kubernetes_pod_uid" and "__meta_kubernetes_pod_container_name" // Concatenate values __meta_kubernetes_pod_uid/__meta_kubernetes_pod_container_name.log rule { source_labels = ["__meta_kubernetes_pod_uid", "__meta_kubernetes_pod_container_name"] action = "replace" target_label = "__path__" separator = "/" replacement = "/var/log/pods/*$1/*.log" } // Label creation - "container_runtime" field from "__meta_kubernetes_pod_container_id" rule { source_labels = ["__meta_kubernetes_pod_container_id"] action = "replace" target_label = "container_runtime" regex = "^(\\S+):\\/\\/.+$" replacement = "$1" } } // loki.source.kubernetes tails logs from Kubernetes containers using the Kubernetes API. loki.source.kubernetes "pod_logs" { targets = discovery.relabel.pod_logs.output forward_to = [otelcol.receiver.loki.default.receiver] } otelcol.receiver.loki "default" { output { logs = [otelcol.processor.batch.default.input] } } beyla.ebpf "default" { attributes { kubernetes { enable = "true" } select { attr = "sql_client_duration" exclude = [] include = ["db.query.text"] } } discovery { services { open_ports = "80,443,3000,8000-8999" } } output { traces = [otelcol.processor.batch.default.input] } } otelcol.processor.batch "default" { output { logs = [otelcol.exporter.otlphttp.grafana.input] metrics = [otelcol.exporter.otlphttp.grafana.input, otelcol.exporter.otlp.causely.input] traces = [otelcol.exporter.otlphttp.grafana.input, otelcol.exporter.otlp.causely.input] } } otelcol.processor.k8sattributes "default" { extract { label { from = "pod" } metadata = [ "k8s.namespace.name", "k8s.pod.name", "k8s.pod.uid", "k8s.deployment.name", "k8s.node.name", "k8s.pod.start_time", "container.id", ] } output { traces = [otelcol.processor.batch.default.input] } } otelcol.processor.deltatocumulative "default" { output { metrics = [otelcol.processor.batch.default.input] } } otelcol.receiver.datadog "default" { endpoint = "0.0.0.0:8126" output { metrics = [otelcol.processor.deltatocumulative.default.input] traces = [otelcol.processor.batch.default.input] } } otelcol.receiver.otlp "otlp" { grpc { endpoint = "0.0.0.0:4317" } http { endpoint = "0.0.0.0:4318" } output { logs = [otelcol.processor.batch.default.input] metrics = [otelcol.processor.batch.default.input] traces = [otelcol.processor.k8sattributes.default.input] } } ``` ## Alerts Causely can ingest the alerts from Grafana's [unified alerting](https://grafana.com/docs/grafana/latest/alerting/) and use them as Signals for causal analysis. The Grafana scraper periodically reads the currently active alert instances from Grafana's Alertmanager-compatible API (`/api/alertmanager/grafana/api/v2/alerts`), maps them to the right entities, and activates the corresponding Causely Signals. When an alert clears in Grafana, Causely automatically deactivates the Signal. This means you can: - Keep your existing Grafana alert rules and contact points. - See Grafana alerts mapped to services and dependencies inside Causely. - Automatically map alerts to Signals and point to the actual cause. ### Authentication Each Grafana instance accepts one of: - **Service account token**: set `token` to a Grafana service account token (for example, `glsa_...`). The token only needs read access to alerts/alerting. - **Basic auth**: set `username` and `password` instead of `token`. ### Auto-discovery with a Kubernetes Secret When `auto_discovery.enabled` is `true`, the mediator watches for Kubernetes Secrets labeled `causely.ai/scraper: Grafana` and creates a scraper for each one. This lets you add or rotate Grafana instances without changing the mediator configuration. ```yaml apiVersion: v1 kind: Secret metadata: name: grafana-prod namespace: causely labels: causely.ai/scraper: Grafana type: Opaque stringData: endpoint: https://grafana.example.com token: glsa_xxxxxxxxxxxxxxxxxxxx # username: causely-readonly # basic auth (alternative to token) # password: ... # labelFilters is an optional JSON object; only alerts matching every # listed label/value are ingested. labelFilters: '{"namespace": ["payments", "checkout"]}' ``` The recognized Secret keys are `endpoint`, `token`, `username`, `password`, and `labelFilters`. ### Filtering which alerts are ingested Use `labelFilters` to restrict ingestion to alerts carrying specific labels. Each entry is a label name mapped to the set of allowed values; an alert is ingested only if it matches all of the listed labels. ### Mapping alert severity Grafana alerts carry a severity label (commonly `severity`). Use `alert_severity` to normalize your severity values to Causely severity levels (`critical`, `major`, `minor`, `info`). ```yaml scrapers: grafana: alert_severity: label_name: severity mappings: - value: critical level: critical - value: warning level: major ``` ### Mapping alerts to entities and Signals By default, Causely maps error- and latency-type alerts to the corresponding Service Signals using keyword matching. Use `alerts` to explicitly map a Grafana alert to a Causely Signal on a specific entity. For example, the following maps the `HighDBOpenConnections` alert to the `DBConnectionsUtilization_High` symptom of the matching workload: ```yaml scrapers: grafana: alerts: - alert_name: 'HighDBOpenConnections' symptom: 'DBConnectionsUtilization_High' entity: workload: {} discovery: - kubernetes_pod: namespace: 'namespace' pod_name: 'pod' ``` The `discovery` block resolves the entity from the alert's labels: here, the `namespace` and `pod` labels on the alert identify the Kubernetes pod (and therefore its workload) the symptom is attached to. ## Loki Integration Causely integrates with **Loki** to retrieve logs already centralized in your Grafana environment. When configured, Causely automatically surfaces relevant logs in the context of active Diagnoses and service malfunctions. This enables rapid validation of issues and faster time to understanding and resolution. ### When to Use Use this integration when: - Your logs are shipped to Loki instead of remaining in the Kubernetes API. - You want Causely to display relevant log lines and exceptions alongside detected service degradations or Diagnoses. - You prefer a centralized, scalable log pipeline already managed through Grafana Cloud or self-hosted Loki. ### Benefits - **Contextual Insight**: Automatically surfaces logs correlated with active Diagnoses or degraded services. - **Accelerated RCA**: Shows container log lines, stack traces, and error spikes precisely around the time of failure. - **Unified View**: Displays Loki logs directly within Causely, alongside metrics and traces for the same service. - **Operational Efficiency**: Reduces reliance on the Kubernetes API for log collection. ### Configuration **Configure Causely:** ```yaml scrapers: kubernetes: loki_endpoint: 'http://loki.monitoring:3100' loki_tenant: 'production' ``` Causely automatically uses Loki as the preferred external log source with an optional tenant id if used in a multitenant loki configuration. ### Supported Log Types - Container logs (stdout/stderr) ### Result When Causely detects a service malfunction or identifies a [Diagnosis](/reference/root-causes/), it automatically retrieves related logs from Loki. These logs appear: - **Under affected services**, when they exhibit abnormal behavior such as elevated errors or latency. - **Alongside Diagnoses**, showing relevant exceptions or stack traces at the time of failure. This provides clear evidence for what went wrong and why, dramatically shortening investigation and resolution times. --- ## groundcover [groundcover](https://www.groundcover.com/product/application-performance-monitoring) is an eBPF-based application performance monitoring solution that automatically collects telemetry data from your applications without requiring code instrumentation or configuration changes. ## How groundcover works groundcover uses eBPF (Extended Berkeley Packet Filter) technology to automatically capture: - **HTTP and gRPC requests** - Service-to-service communication - **Database queries** - SQL operations and connection details - **Messaging operations** - Kafka, RabbitMQ, and other messaging systems - **Service dependencies** - Automatic discovery of service relationships - **Performance metrics** - Latency, throughput, and error rates To learn more about how groundcover works, see the [groundcover documentation](https://docs.groundcover.com/). ## Integration with Causely groundcover sends collected telemetry data to Causely's mediator using the OpenTelemetry protocol (OTLP). This enables Causely to: - Analyze service dependencies and communication patterns - Identify actual causes of performance issues and outages - Correlate events across your entire infrastructure - Provide automated incident response and resolution ## Configuration To enable groundcover as a data source for Causely, add the following configuration to your groundcover installation: ```yaml title="groundcover-values.yaml" opentelemetry-collector: config: exporters: otlp/causely: endpoint: mediator.causely:4317 tls: insecure: true insecure_skip_verify: true service: pipelines: traces/causely: exporters: - otlp/causely processors: - batch/traces receivers: - otlp - datadog - faro ``` ## Installation Install groundcover with the Causely integration enabled: ```shell helm install groundcover groundcover/groundcover --values ./groundcover-values.yaml ``` Once installed, groundcover will automatically begin collecting telemetry data and forwarding it to Causely for causal reasoning. --- ## incident.io Integration ## Overview Causely can automatically ingest alerts from [incident.io](https://incident.io) and map them to Signals and entities in your system with minimal configuration required. The integration automatically processes incident.io alerts by: - Detecting Signals from alert titles and descriptions - Using Kubernetes labels (namespace, pod, service) for entity mapping ## Configuration ### Basic Setup To enable the incident.io integration, add the following to your `causely-values.yaml` file: ```yaml title="causely-values.yaml" scrapers: incident: enabled: true instances: - secretName: 'incident-io-credentials' namespace: 'causely' ``` Create a Kubernetes Secret with the incident.io API token: ```bash kubectl create secret generic incident-io-credentials \ --namespace causely \ --from-literal=token=your-incident-io-api-token ``` ### Alternative: Enable Credentials Autodiscovery Causely also supports credentials autodiscovery. This feature allows you to add new scraping targets without updating the Causely configuration. First, enable autodiscovery in your configuration: ```yaml title="causely-values.yaml" scrapers: incident: enabled: true auto_discovery: enabled: true ``` Then create a Kubernetes secret with the required label: ```bash kubectl create secret generic incident-io-credentials \ --namespace causely \ --from-literal=token=your-incident-io-api-token kubectl label secret incident-io-credentials \ --namespace causely \ "causely.ai/scraper=Incident" ``` --- ## Instana ## Overview Causely connects to [IBM Instana Observability](https://www.ibm.com/products/instana) through a locally deployed mediation layer. The mediation layer reads only the signals needed from Instana, discovers topology, and detects Signals in your environment. The mediation layer converts Instana signals into a compact stream of Signals and topology deltas, then sends only those distilled insights to Causely's causal reasoning engine (CRE) for analysis. The engine continuously infers causes, evaluates blast radius across services, endpoints, databases, and SLOs, and provides context-rich explanations with guided remediation so responders move directly from detection to resolution. ## Key capabilities: - Automatic causal inference pinpointing the Diagnosis and risk in real time. Causely’s CRE continuously ingests Instana signals to pinpoint the Diagnosis behind service degradations and failures, surfacing emerging risks before they become incidents. - Automatic impact analysis and blast radius mapping. Causely’s CRE continuously infers the impacted services, endpoints, databases, and SLOs to drive the right prioritization. - Automatic remediation. Causely provides context‑rich explanations of actual causes and can automatically remediate by applying the specific fix at runtime, configuration, or code level. - Fast, low‑lift enablement. Connect Instana via API and start diagnosing in minutes with no changes to your services. ## Setup Guide ### Step 1: Create an API token in Instana Create an API token for your Instana account with read access to applications, endpoints, metrics, and infrastructure. ### Step 2: Create a Kubernetes Secret Create a secret with your Instana account URL and API token: ```bash kubectl create secret generic \ --namespace causely instana-credentials \ --from-literal=url="https://your-instana.example.com" \ --from-literal=api-token="" # Optional (only if necessary for your environment): # --from-literal=insecure-skip-verify="true" ``` Alternatively, you can create the secret using a YAML manifest: ```yaml apiVersion: v1 kind: Secret metadata: name: instana-credentials namespace: causely type: Opaque stringData: url: 'https://your-instana.example.com' api-token: '' # insecure-skip-verify: 'true' # optional ``` ### Step 3: Update Causely configuration Enable the Instana scraper and reference your secret: ```yaml scrapers: instana: enabled: true accounts: - secretName: instana-credentials # namespace: your-namespace # optional; defaults to the mediator pod namespace ``` ### Alternative: Enable Credentials Autodiscovery Causely supports credentials autodiscovery so you can add new accounts without editing config. Label the Kubernetes secret to enable autodiscovery for the Instana scraper: ```bash kubectl --namespace causely label secret instana-credentials "causely.ai/scraper=Instana" ``` ## What Data is Collected The Instana scraper collects and maps the following into Causely: - **Service and dependency graph**: Instana application services and service-map connections, aligned to Kubernetes workloads and services - **Endpoints**: HTTP paths and RPC methods derived from Instana endpoints - **Endpoint metrics**: Requests, errors, and latency percentiles (p90, p95, p99, max) attached to endpoints - **Cross-service and cross-endpoint calls**: Observed calls with error counts and latency distributions used to ground edges in behavior - **Pod-to-service and service-to-pod metrics**: Access metrics that attribute consumption and exposure per workload - **Database insights**: Queries grouped by statement with call volumes and latency; access attributed to calling services/endpoints and mapped to database/table entities - **Kubernetes metadata**: Namespace and cluster labels added to services for context --- ## Istio ### Istio - Prometheus The Istio integration works by scraping the Istio relevant data from a Prometheus server. To enable Istio as a data source, please add the following section to your values.yaml file: ```yaml scrapers: istio: enabled: true prometheus: endpoint: http://prometheus.istio-system.svc.cluster.local:9090 ``` Please ensure that the Prometheus instance scrapes all your Istio sidecars to capture all the service-to-service communication metrics. ### Istio - w/o Prometheus The Istio integration works by scraping the Istio relevant data directly from the istio sidecars. To enable Istio as a data source, please add the following section to your values.yaml file: ```yaml scrapers: istiosidecar: enabled: true ``` --- ## Kubernetes ## Overview Causely provides **native, out-of-the-box integration** with [Kubernetes](https://kubernetes.io/) that automatically discovers and monitors your entire container infrastructure. When you install Causely using our standard [Helm](/installation/helm) or [CLI](/installation/cli) installation, Kubernetes monitoring is enabled by default with zero configuration required. The integration provides comprehensive visibility across your entire Kubernetes stack, from cluster-level resources down to individual containers, helping you identify infrastructure issues before they impact your applications. ## How It Works Causely's Kubernetes integration works automatically upon installation: 1. **Automatic Discovery**: The agent automatically discovers all Kubernetes resources in your cluster using the Kubernetes API 2. **Real-time Monitoring**: Continuously monitors resource states, events, and relationships 3. **Entity Modeling**: Creates a comprehensive topology graph showing relationships between clusters, nodes, pods, services, and applications 4. **Event Processing**: Analyzes Kubernetes events to detect issues like pod evictions, scheduling problems, and resource constraints 5. **Diagnosis**: Correlates infrastructure issues with application performance problems 6. **Auto-remediation**: Remediate issues automatically by deploying fixes or scaling resources This approach provides complete infrastructure visibility without requiring any configuration changes or additional setup beyond the standard Causely installation. ## Out-of-the-Box Setup Kubernetes monitoring is **enabled by default** when you install Causely. Simply follow our standard installation guide: 📦 Install with Helm or 💻 Install with CLI No additional configuration, secrets, or permissions are required beyond what's included in the standard installation. ## Enhanced Log Retrieval Causely surfaces relevant log lines automatically in the context of active Diagnoses and service malfunctions. For environments with centralized logging, Causely can retrieve container logs from external sources instead of the Kubernetes API. This is useful when logs are forwarded to systems like Elasticsearch or Grafana Loki. ### External Log Integrations Causely supports two primary centralized log integrations: - **[Elasticsearch](/telemetry-sources/elasticsearch#use-case-1-log-retrieval-from-elasticsearch)**: Retrieve logs directly from Elasticsearch when your containers push logs there. - **[Grafana Loki](/telemetry-sources/grafana#loki-integration)**: Integrate with Loki for scalable log aggregation and querying through Grafana. When either integration is configured, Causely prioritizes external log sources over the Kubernetes API and automatically falls back to Kubernetes logs if the external system becomes unavailable. ## What You Get ### Infrastructure Topology - **Complete service map** showing relationships between applications, services, and infrastructure - **Multi-layer visualization** from business applications down to individual containers - **Dependency tracking** across namespaces and resource types ### Workload Monitoring - **Controller analysis** for Deployments, StatefulSets, DaemonSets, Jobs, and CronJobs - **Pod lifecycle tracking** including scheduling, running, and termination states - **Container health monitoring** with resource usage and state information - **Configuration change detection** for container images, resources, environment variables, and volume mounts ### Resource Management - **Node health monitoring** with conditions like memory pressure, disk pressure, and network availability - **Persistent volume tracking** with usage and binding information - **Service discovery** with automatic endpoint creation and load balancer mapping - **Ingress routing** analysis for external traffic patterns ### Event Analysis - **Pod eviction detection** for memory pressure, disk pressure, and resource constraints - **Scheduling failure analysis** for unschedulable pods - **Image pull error tracking** for deployment issues - **Configuration change events** for version updates and resource modifications ### Application Integration - **Service-to-pod mapping** for application relationship discovery - **Network endpoint creation** for service communication analysis - **Load balancer and ingress integration** for external access patterns - **Kafka resource discovery** for message queue topology (if using Strimzi operator) ### Enhanced Log Retrieval - **Multi-source log collection** from Kubernetes API, Elasticsearch, and Loki - **Elasticsearch integration** for centralized log analysis when logs are forwarded from containers - **Loki integration** for scalable log aggregation and querying - **Priority-based retrieval** with automatic fallback to Kubernetes API - **Authentication support** for secured log aggregation systems ## Root Cause Detection The Kubernetes integration enables detection of infrastructure-related causes including: ### Node-Level Issues - [**Disk Pressure**](/reference/root-causes/infrastructure#disk-pressure) - Node disk usage triggering pod evictions - [**Memory Pressure**](/reference/root-causes/infrastructure#memory-pressure) - Node memory exhaustion causing pod evictions ### Controller and Workload Issues - [**Controller Malfunction**](/reference/root-causes/infrastructure#malfunction) - Multiple pods in NotReady state - [**Image Pull Errors**](/reference/root-causes/infrastructure#image-pull-errors) - Pods failing to start due to registry issues - [**FrequentPodEphemeralStorageEvictions**](/reference/root-causes/infrastructure#frequentpodephemeralstorageevictions) - Pods evicted due to storage limits ### Container Resource Issues - [**CPU Congested**](/reference/root-causes/infrastructure#cpu-congested) - Container CPU throttling and performance degradation - [**Memory Failure**](/reference/root-causes/infrastructure#memory-failure) - Container out-of-memory kills - [**Frequent Memory Failure**](/reference/root-causes/infrastructure#frequent-memory-failure) - Repeated memory-related crashes - [**Crash Failure**](/reference/root-causes/infrastructure#crash-failure) - Container crashes with non-zero exit codes - [**Frequent Crash Failure**](/reference/root-causes/infrastructure#frequent-crash-failure) - Repeated container crashes ### Storage and Noisy Neighbor Issues - [**Ephemeral Storage Congested**](/reference/root-causes/infrastructure#ephemeral-storage-congested) - Container storage usage causing failures - [**Ephemeral Storage Noisy Neighbor**](/reference/root-causes/infrastructure#ephemeral-storage-noisy-neighbor) - Container consuming excessive storage affecting node - [**Memory Noisy Neighbor**](/reference/root-causes/infrastructure#memory-noisy-neighbor) - Container consuming excessive memory affecting node - [**Disk Congested**](/reference/root-causes/infrastructure#congested) - Persistent volumes reaching capacity limits ### Service-Level Issues - [**Service Congested**](/reference/root-causes/services#congested) - Kubernetes services experiencing high latency - [**Service Malfunction**](/reference/root-causes/services#malfunction) - Kubernetes services with high error rates ### Release-Related Issues - [**Code Change Regression: CPU Congestion**](/reference/root-causes/release#code-change-regression-cpu-congestion) - Performance degradation after deployments - [**Code Change Regression: Memory Failure**](/reference/root-causes/release#code-change-regression-memory-failure) - Memory issues introduced by new versions ## What Data is Collected The Kubernetes integration automatically collects comprehensive metadata and state information, including: ### Cluster-Level Resources - **Cluster identity** and configuration - **Node specifications** and health conditions - **Namespace organization** and resource quotas - **Custom resource definitions** and operators ### Workload Resources - **Pod specifications** including containers, volumes, and resource requirements - **Controller configurations** for Deployments, StatefulSets, DaemonSets, Jobs, and CronJobs - **Service definitions** and endpoint mappings - **Ingress rules** and traffic routing configuration ### Runtime Information - **Container states** and restart counts - **Resource utilization** and capacity limits - **Event logs** for troubleshooting and analysis - **Container logs** from Kubernetes API, Elasticsearch, or Loki (when configured) - **Configuration changes** and version history ### Relationships and Dependencies - **Service-to-pod mappings** for application topology - **Pod-to-node assignments** for infrastructure placement - **Volume bindings** and storage relationships - **Network connectivity** patterns and endpoints :::note The Kubernetes integration respects your cluster's RBAC policies and only accesses resources that the Causely service account has permissions to read. All data collection uses read-only Kubernetes API calls. ::: --- ## MongoDB Atlas ## Overview Causely provides native integration with [MongoDB Atlas](https://www.mongodb.com/atlas) to help you identify and resolve database performance issues before they impact your users. Instead of just monitoring signals, Causely analyzes them in real time from the Atlas Monitoring API to surface the actual diagnosis behind database problems. This integration helps you identify the following causes for reliability issues, among others: - [Connection exhaustion](/reference/root-causes/services#congested) from connection pool saturation across Atlas shards - [Replication lag](/reference/root-causes/services#malfunction) causing stale reads on secondary nodes - [Service congestion](/reference/root-causes/services#congested) affecting Atlas cluster performance - [Service malfunction](/reference/root-causes/services#malfunction) in database cluster components The integration supports MongoDB Atlas clusters and automatically maps Atlas shard processes to the Causely service topology, enabling end-to-end tracing from your application to the Atlas cluster. ## Step 1: Create an Atlas API Key Create a programmatic API key for your MongoDB Atlas project with the necessary permissions: 1. Log in to the [MongoDB Atlas Console](https://cloud.mongodb.com/) 2. Navigate to your **Organization** → **Access Manager** → **API Keys** 3. Click **Create API Key** 4. Set a description and assign the **Project Read Only** role (minimum required) 5. Copy and securely store the **Public Key** and **Private Key** 6. Add your Causely mediator's IP address to the API key access list :::note You will also need the **Project ID** (also called Group ID) from your Atlas project. Navigate to your project → **Settings** to find it. ::: ## Step 2: Configure the Integration ### Add data source from the UI You can add MongoDB Atlas as a data source from the Causely UI. Go to [Integrations](https://portal.causely.app/integrations), add the MongoDB Atlas integration, enter your **Public Key**, **Private Key**, and **Project ID**, and select the cluster to which the configuration will be pushed. ### Basic setup (ops as code) If you prefer to manage configuration as code, create a Kubernetes Secret containing your Atlas API credentials and label it for autodiscovery: ```bash kubectl create secret generic atlas-credentials \ --namespace causely \ --from-literal=publicKey="" \ --from-literal=privateKey="" \ --from-literal=groupId="" kubectl label secret atlas-credentials \ --namespace causely \ causely.ai/scraper=MongoAtlas ``` Alternatively, create the secret using a YAML manifest: ```yaml apiVersion: v1 kind: Secret metadata: name: atlas-credentials namespace: causely labels: causely.ai/scraper: MongoAtlas type: Opaque stringData: publicKey: '' privateKey: '' groupId: '' ``` Secret keys: - **`publicKey`** (required): MongoDB Atlas programmatic API public key. Also accepted as `public_key`. - **`privateKey`** (required): MongoDB Atlas programmatic API private key. Also accepted as `private_key`. - **`groupId`** (required): MongoDB Atlas project ID (Group ID). Also accepted as `group_id`, `projectId`, or `project_id`. To monitor multiple Atlas projects, create one secret per project and label each with `causely.ai/scraper=MongoAtlas`. ## What Data is Collected The MongoDB Atlas scraper collects comprehensive metadata and performance information from your Atlas deployment, including: - **Cluster entities** with display names and labels (`causely.ai/service-type=MongoDB Atlas Cluster`) - **Shard process entities** (Workloads) for each mongod/mongos instance, linked to their parent cluster - **Network endpoints** for all client-facing hostnames (both the SRV parent and the individual shard seed hosts), enabling automatic topology stitching from application services to the Atlas cluster - **Connection metrics** per shard: current connections and available connections (derived from the cluster tier's connection limit) - **Replication lag** per shard (`OPLOG_REPLICATION_LAG_TIME`) - **Open cursors** per shard (`CURSORS_TOTAL_OPEN`) --- ## MySQL ## Overview Causely provides native integration with [MySQL](https://www.mysql.com/) to help you identify and resolve database performance issues before they impact your users. Instead of just monitoring Signals, Causely analyzes them in real time to surface the actual Diagnosis behind database problems. This integration helps you identify the following causes for reliability issues, among others: - [Table malfunction](/reference/root-causes/datapipeline#table-access-failure-in-database) and [excessive locks](/reference/root-causes/datapipeline#contention-on-database-table-locks) - [Slow database queries](/reference/root-causes/applications#slow-database-queries) - [Connection exhaustion](/reference/root-causes/applications#database-connection-pool-saturated) The integration supports both self-hosted MySQL instances and cloud-managed services including AWS RDS MySQL, Azure Database for MySQL, and Google Cloud SQL for MySQL. ## Setup Guide ### Step 1: Create a user Create a user for your MySQL database server with the following permissions: - `PROCESS`: Required to view running queries and process information - `REPLICATION CLIENT`: Required to monitor replication status and transaction information - `SELECT`: Required to query performance metrics, table information, and schema details - `SHOW VIEW`: Required to access view definitions and metadata ### Step 2: Enable performance monitoring features Enable performance monitoring features in your MySQL instance to collect detailed metrics and query information. This step varies depending on your MySQL deployment type, as detailed below. #### Native MySQL For native MySQL installations, you need to enable the performance schema: 1. Edit your MySQL configuration file (typically `/etc/mysql/my.cnf` or `/etc/my.cnf`) 2. Add or update these settings: ```ini [mysqld] performance_schema=ON ``` #### AWS RDS MySQL - [AWS RDS MySQL Documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_MySQL.html) - [Performance Insights in AWS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PerfInsights.html) - [Enable Performance Insights for Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PerfInsights.Enabling.html) Steps: 1. Go to RDS Console → Databases 2. Select your MySQL instance 3. Click "Modify" 4. Under "Performance Insights", enable "Enable Performance Insights" 5. Choose retention period (7 days or more) 6. Click "Continue" and apply changes immediately #### Azure Database for MySQL - [Azure MySQL Documentation](https://learn.microsoft.com/en-us/azure/mysql/) - [Query Performance Insights in Azure](https://learn.microsoft.com/en-us/azure/mysql/flexible-server/tutorial-query-performance-insights) Steps: 1. Go to Azure Portal → Select your Azure Database for MySQL Flexible Server instance. 2. On the left pane, under Settings, select Server parameters. 3. For the slow_query_log parameter, select ON. 4. For the other parameters, such as long_query_time and log_slow_admin_statements, refer to the slow query logs documentation. 5. Click "Save" #### Google Cloud SQL for MySQL - [Cloud SQL MySQL Documentation](https://cloud.google.com/sql/docs/mysql) - [Query Insights in GCP](https://cloud.google.com/sql/docs/mysql/using-query-insights) Steps: 1. Go to Cloud Console → Cloud SQL 2. Select your MySQL instance 3. Click "Edit" 4. Under "Query Insights", enable "Enable Query Insights" 5. Choose retention period (7 days or more) 6. Click "Save" ### Step 3: Create a Kubernetes secret for the user After creating the user create a Kubernetes secret. You have two options: #### Option 1: Single Database Configuration ```bash kubectl create secret generic \ --namespace causely mysql-credentials \ --from-literal=username="..." \ --from-literal=password='...' \ --from-literal=host="..." \ --from-literal=port=3306 \ --from-literal=database="..." \ --from-literal=sslmode="..." ``` #### Option 2: Multiple Databases Configuration For monitoring multiple databases within the same MySQL instance, you can specify multiple databases using the `databases` field: ```bash kubectl create secret generic \ --namespace causely mysql-credentials-multidb \ --from-literal=username="..." \ --from-literal=password='...' \ --from-literal=host="..." \ --from-literal=port=3306 \ --from-literal=databases="database1,database2,database3" \ --from-literal=sslmode="..." ``` Alternatively, you can create the secret using a YAML manifest: ```yaml apiVersion: v1 kind: Secret metadata: name: mysql-credentials-multidb namespace: causely type: Opaque stringData: username: '...' password: '...' host: '...' port: '3306' databases: 'database1,database2,database3' sslmode: '...' ``` **Note**: Use either the `database` field for single database configuration or the `databases` field for multiple databases. Do not use both fields in the same secret. The `host` must be the FQDN of your DB, or IP address if there's no DNS entry set up. It must match the FQDN/IP Causely would discover either from the K8s Server (if it's running in your K8s Cluster) or your Cloud providers API. Examples: - _Kubernetes_: `host=my-mysql.namespace.service.cluster.local` - _AWS_: `host=myinstance.rds.amazonaws.com` If you are leveraging a proxy to connect to your database, like it's common on GCP, `host` must be the FQDN/IP of your proxy and additionally `host_overwrite` should be the IP/FQDN of the actual database service: - _GCP_: `host=localhost` (assuming proxy runs on the same host / Pod). `host_overwrite=1.2.3.4` (the IP of your GCP Cloud SQL instance as shown in the GCP Console) ### Step 4: Update Causely Configuration Once the secret is created, update the Causely configuration to enable scraping for the new database. Below is an example configuration: ```yaml scrapers: mysql: enabled: true instances: - secretName: mysql-credentials namespace: causely ``` ### Alternative: Enable Credentials Autodiscovery Causely also supports credentials autodiscovery. This feature allows you to add new scraping targets without updating the Causely configuration. Label the Kubernetes secret to enable autodiscovery for the corresponding scraper. ```bash kubectl --namespace causely label secret mysql-credentials "causely.ai/scraper=MySQL" ``` ## What Data is Collected The MySQL scraper collects comprehensive metadata and performance information from your MySQL databases, including: - **Database entities** with names and relationships to hosting services - **Service-to-database mappings** (which service provides which database) - **Connection details** including host, port, and SSL configuration - **Database and table schemas** with complete structural information - **Table information** including names, row counts, and sizes - **Complete table schemas** with column definitions, data types, constraints, and indexes - **Foreign key relationships** and table dependencies - **Parent-child table mappings** for dependency analysis - **Slow query analysis** - **Lock monitoring** --- ## Nobl9 Causely can leverage [Nobl9](https://www.nobl9.com/)'s SLO platform to observe service level objectives and reliability metrics. To enable Nobl9 as a data source, please add the following section to your `values.yaml` file: ```yaml scrapers: nobl9: enabled: true accounts: - accountName: secretName: nobl9-secret ``` ## nobl9-secret.yaml ```yaml apiVersion: v1 kind: Secret type: Opaque metadata: name: nobl9-secret namespace: causely stringData: NOBL9_CLIENT_ID: '' NOBL9_CLIENT_SECRET: '' NOBL9_ORGANIZATION: '' ``` ## SLO Labeling Since Nobl9 SLO can be defined for anything, in order for Causely to stitch the right SLO to the right Kubernetes Service, you need to label your SLO in Nobl9 with: ```yaml namespace: service: ``` --- ## Nomad ## Overview Causely provides **native integration** with [HashiCorp Nomad](https://www.nomadproject.io/) that automatically discovers and monitors your workload orchestration infrastructure. The integration provides visibility across your Nomad cluster, from nodes down to individual tasks, helping you identify infrastructure issues before they impact your applications. ## How It Works Causely's Nomad integration automatically discovers nodes, jobs, allocations, and tasks using the Nomad API, monitors job states and resource usage, creates a topology graph showing relationships between nodes, allocations, tasks, jobs, and services, integrates with Consul for service discovery, and analyzes task events to detect OOM kills and crashes. Nomad monitoring works automatically when you install Causely using the [Nomad installation guide](/installation/nomad). --- ## Odigos [Odigos](https://docs.odigos.io/setup/installation) is an open source observability control plane that automatically instruments your applications to generate traces and metrics without requiring code changes. ## How Odigos works Odigos provides automatic instrumentation for your applications by: - **Auto-instrumenting applications** - Automatically adds OpenTelemetry instrumentation to your services - **Discovering service dependencies** - Maps out how your services communicate with each other - **Generating traces and metrics** - Creates distributed traces and application metrics automatically - **Supporting multiple languages** - Works with Java, Python, Node.js, Go, and other languages - **Kubernetes-native** - Designed to work seamlessly in Kubernetes environments To learn more about how Odigos works, see the [Odigos documentation](https://docs.odigos.io/). ## Integration with Causely Odigos sends collected traces and metrics to Causely's mediator service using the OpenTelemetry protocol (OTLP). This enables Causely to: - Analyze service dependencies and communication patterns - Identify actual causes of performance issues and outages - Correlate events across your entire infrastructure - Provide automated incident response and resolution ## Configuration To enable Odigos as a data source for Causely, add the following configuration to your Odigos installation: ```yaml title="odigos-destination.yaml" apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: odigos.io.dest.causely namespace: odigos-system spec: data: CAUSELY_URL: http://mediator.causely:4317 destinationName: causely signals: - TRACES - METRICS type: causely ``` For detailed configuration instructions, see the [Odigos Causely backend documentation](https://docs.odigos.io/backends/causely). ## Installation To install Odigos, please refer to the [Odigos documentation](https://docs.odigos.io/setup/installation). Once Odigos is installed and configured with the Causely destination as described above, it will automatically begin collecting telemetry data from your applications and forwarding it to Causely for causal reasoning. --- ## Integrate OpenTelemetry Traces ## What is OpenTelemetry? OpenTelemetry is an open source, vendor- and tool-neutral project that provides a comprehensive observability framework for generating, exporting and collecting telemetry data, such as traces, metrics and logs. Visit the [OpenTelemetry website](https://opentelemetry.io/) for more information. ## Integrating OpenTelemetry with Causely You can export traces from an application or an existing OpenTelemetry Collector to the [mediator](/getting-started/architecture#mediation-layer) that was installed as part of the Causely agent. The mediator listens for traces on port `4317` using the [OpenTelemetry Protocol](https://opentelemetry.io/docs/specs/otlp/) (OTLP). We recommend using OpenTelemetry with Causely because traces enable automatic discovery of service dependencies and monitoring of both synchronous and asynchronous communication signals. :::tip High trace volume For environments with high trace volume, Causely supports a horizontally scalable trace processing tier. When enabled, collectors send traces to the trace controller (`trace-controller.causely:4317`) instead of the mediator. See [Scalable Trace Processor](/installation/scalable-trace-processor) for details. ::: :::info By default, Causely will use [OpenTelemetry eBPF Instrumentation](/telemetry-sources/ebpf) for automatic instrumentation of your applications. ::: ## How it works The following diagram shows how OpenTelemetry traces and metrics are collected and forwarded to the Causely mediator. For more information on how Causely works, see the [How Causely Works](/getting-started/how-causely-works) page. ```mermaid %%{init: {'theme': 'base', 'themeVariables': { 'fontSize': '16px', 'fontFamily': 'arial' }}}%% graph LR subgraph kubernetes["Kubernetes Cluster"] OtelCollector["OpenTelemetry Collector"] App1["Application 1"] App2["Application 2"] App3["Application 3"] Mediator["Mediator
(Listening on port 4317)"] end CRE["Causal Reasoning Engine"] Backend1["Observability
Backend"] App1 -->|"traces,logs,metrics"| OtelCollector App2 -->|"traces,logs,metrics"| OtelCollector App3 -->|"traces,logs,metrics"| OtelCollector OtelCollector -->|"traces,metrics"| Mediator Mediator -->|"distilled insights"| CRE OtelCollector -..->|"full telemetry data"| Backend1 classDef application fill:#36c5f0,stroke:#333,stroke-width:1px; classDef collector fill:#4F46E5,stroke:#333,stroke-width:1px,color:#fff; classDef mediator fill:#00B67A,stroke:#333,stroke-width:1px; classDef engine fill:#0066CC,stroke:#333,stroke-width:1px,color:#fff; classDef backend fill:#4a154b,stroke:#333,stroke-width:1px,color:#fff; classDef cluster fill:#EBF3FA,stroke:#333,stroke-width:1px; class App1,App2,App3 application; class OtelCollector collector; class Mediator mediator; class CRE engine; class Backend1 backend; class kubernetes cluster; ``` :::tip Good to know No raw data is sent from the mediator to the Causely engine, only distilled insights. This way your data is secure and you can be sure that no sensitive data is sent to us. ::: ## Quick Start Guide If you don't have an OpenTelemetry Collector running in your Kubernetes cluster, you can use the following command to install the OpenTelemetry Operator and the OpenTelemetry Collector: ```shell helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts helm install opentelemetry-collector open-telemetry/opentelemetry-collector --values ./opentelemetry-values.yaml ``` For values of `opentelemetry-values.yaml` see the [OpenTelemetry Operator Configuration section](#opentelemetry-operator-configuration) below. ## OpenTelemetry Collector Configuration For an instance of the [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) running within your Kubernetes cluster, you can use the following configuration: ```yaml exporters: otlp_grpc/causely: endpoint: mediator.causely:4317 compression: none tls: insecure: true processors: batch: timeout: 1s k8s_attributes: auth_type: 'serviceAccount' passthrough: false extract: metadata: - k8s.pod.name - k8s.pod.uid - k8s.container.name - k8s.deployment.name - k8s.namespace.name - k8s.node.name - k8s.pod.start_time # Optional: Filter out internal spans filter/ignore-internal: error_mode: ignore traces: span: - 'kind.string == "Internal"' receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 prometheus/metrics: # See a more detailed example in the Configuring Prometheus Metrics Relabeling section below service: pipelines: metrics: exporters: [otlp_grpc/causely] processors: [k8s_attributes, batch] receivers: [otlp, prometheus/metrics] traces: exporters: [otlp_grpc/causely] processors: [filter/ignore-internal, k8s_attributes, batch] receivers: [otlp] ``` ### Configuration Breakdown This configuration will: - Export metrics and traces to the Causely mediator using the [OTLP gRPC Exporter](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/otlpexporter/README.md) - Filter out internal spans using the [Filter Processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/filterprocessor/README.md) - Add Kubernetes attributes to the spans using the [Kubernetes Attributes Processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/k8sattributesprocessor/README.md) - Batch the spans using the [Batch Processor](https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/batchprocessor/README.md) ### Configuring Prometheus Metrics Relabeling When using the `prometheus/metrics` receiver in the OpenTelemetry Collector, you need to configure relabeling to extract Kubernetes metadata labels. Add the following `relabel_configs` to your `prometheus/metrics` receiver configuration: ```yaml prometheus/metrics: config: scrape_configs: - job_name: 'prometheus-metrics' relabel_configs: # Filter by pod annotations - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep regex: true # Configure metrics path - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ regex: (.+) # Configure metrics port - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port] action: replace regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 target_label: __address__ # Map pod labels - action: labelmap regex: __meta_kubernetes_pod_label_(.+) # Extract metadata attributes # Pod name - source_labels: [__meta_kubernetes_pod_name] action: replace target_label: k8s.pod.name # Pod UID - source_labels: [__meta_kubernetes_pod_uid] action: replace target_label: k8s.pod.uid # Container name (set both container_name and k8s.container.name) - source_labels: [__meta_kubernetes_pod_container_name] action: replace target_label: container_name - source_labels: [__meta_kubernetes_pod_container_name] action: replace target_label: k8s.container.name # Deployment name (from controller) - source_labels: [__meta_kubernetes_pod_controller_name] action: replace target_label: k8s.deployment.name # Namespace - source_labels: [__meta_kubernetes_namespace] action: replace target_label: k8s.namespace.name # Node name - source_labels: [__meta_kubernetes_pod_node_name] action: replace target_label: k8s.node.name # Pod start time (creation timestamp) - source_labels: [__meta_kubernetes_pod_creation_timestamp] action: replace target_label: k8s.pod.start_time ``` This relabeling configuration maps Kubernetes metadata to metric labels: - `__meta_kubernetes_namespace` → `k8s.namespace.name` - `__meta_kubernetes_pod_name` → `k8s.pod.name` - `__meta_kubernetes_pod_container_name` → `k8s.container.name` - `__meta_kubernetes_pod_controller_name` → `k8s.deployment.name` **Important**: OpenTelemetry uses the `k8s.*` label format (for example `k8s.namespace.name`, `k8s.pod.name`), while Prometheus typically uses simpler label names (for example `namespace`, `pod`). When configuring Causely discovery for metrics from the OpenTelemetry Collector, ensure your discovery configuration matches the actual label names in your metrics (for example use `namespace: "k8s.namespace.name"` and `pod_name: "k8s.pod.name"`). ## OpenTelemetry Operator Configuration If you are using the [OpenTelemetry Operator for Kubernetes](https://opentelemetry.io/docs/platforms/kubernetes/operator/), you can use the following configuration: ```yaml # Valid values are "daemonset" and "deployment". # If set, agentCollector and standaloneCollector are ignored. mode: 'deployment' config: exporters: otlp_grpc/causely: endpoint: mediator.causely:4317 compression: none tls: insecure: true processors: batch: timeout: 1s k8s_attributes: auth_type: 'serviceAccount' passthrough: false extract: metadata: - k8s.pod.name - k8s.pod.uid - k8s.container.name - k8s.deployment.name - k8s.namespace.name - k8s.node.name - k8s.pod.start_time # Optional: Filter out internal spans filter/ignore-internal: error_mode: ignore traces: span: - 'kind.string == "Internal"' receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 prometheus/metrics: config: scrape_configs: - job_name: 'prometheus-metrics' relabel_configs: # See Configuring Prometheus Metrics Relabeling section above service: pipelines: metrics: exporters: [otlp_grpc/causely] processors: [k8s_attributes, batch] receivers: [otlp, prometheus/metrics] traces: exporters: [otlp_grpc/causely] processors: [filter/ignore-internal, k8s_attributes, batch] receivers: [otlp] presets: kubernetesAttributes: enabled: true podLabels: sidecar.istio.io/inject: 'disabled' ``` For the complete `relabel_configs` configuration, see the [Configuring Prometheus Metrics Relabeling](#configuring-prometheus-metrics-relabeling) section above. --- ## Connect Telemetry Sources for Causal Analysis :::important Traces Are Essential Causely requires **traces** to discover service dependencies and perform effective causal analysis. Without traces, the platform's ability to provide value is severely limited. Search for sources that provide **Traces** to ensure you have comprehensive trace coverage. ::: ## UI-based configuration You can configure telemetry sources from the UI. Navigate to [https://portal.causely.app/integrations](https://portal.causely.app/integrations) to set up the integrations you need. You can then select the Mediator to which the configuration will be pushed down. :::tip Missing a telemetry source? On that same page, click on `+ Request a new integration` to file a request to add the integration you need. ::: To ensure that Causely can reason over a wide range of Signals and causes, that explain those Signals, in your environment, it is recommended that you configure additional data sources if they are available. Telemetry sources are organized into categories based on their purpose and the types of inputs they provide to Causely's causal reasoning engine. ## Telemetry Types Different sources provide different types of telemetry to Causely's causal reasoning engine: ## Telemetry Sources by Categories ### Infrastructure Scraper Discovers infrastructure resources such as VMs, containers, clusters, databases, load balancers, queues, and disks. It imports their topology, metadata, and health indicators, forming the backbone of Causely's environment-specific model for causal mapping of infrastructure-level failure modes. **Inputs Provided to Causely**: Metrics, Infrastructure Entities, Signals, sometimes Service Discovery, sometimes Logs ### Logs Ingests log-derived signals (errors, exceptions, warnings) used for Signal activation based on defined error patterns and enhanced descriptions and remediation after a Diagnosis has been inferred. **Inputs Provided to Causely**: Logs ### Metrics & Signals Provides quantitative telemetry (latency, saturation, throughput, resource usage) and domain-specific metrics (for example, Redis, RabbitMQ). Connects Causely to critical data systems (databases, caches, search systems, message queues), pulling topology, performance attributes, and state indicators. These signals populate Causely's Attribute Dependency Graph and reveal bottlenecks and Diagnoses within data infrastructure. **Inputs Provided to Causely**: Metrics, Infrastructure Entities, Signals ### Service Communication Provides traces and metrics through standard OTEL-based data pipelines (eBPF, Collectors, dual-shipping from Datadog, Odigos, groundcover, etc.). These sources enrich Causely's service graph and enable call-level causal reasoning. **Inputs Provided to Causely**: Traces, Metrics, Signals ### SLO Providers Supplies SLO (Service Level Objective) definitions and burn rate information (for example, from Nobl9). Causely uses SLOs as high-priority Signals and impact indicators in causal reasoning. **Inputs Provided to Causely**: SLOs ### Signal Activation Allows Causely to treat external alert triggers (for example, from Datadog Monitors, Checkly, Alertmanager) as explicit signals within the causal graph. This helps Causely analyze "why this alert fired" and resolve noisy or ambiguous alerts. **Inputs Provided to Causely**: Signals ## Search and Filter Telemetry Sources To search and filter telemetry sources by name, category, or signal type, go to the [Supported Technologies](/getting-started/supported-technologies) page. --- ## PostgreSQL ## Overview Causely provides native integration with [PostgreSQL](https://www.postgresql.org/) to help you identify and resolve database issues before they impact your users. Instead of just monitoring Signals, Causely analyzes them in real time to surface the underlying Diagnosis driving database issues. By setting up the PostgreSQL integration, you will be able to do the following: - Identify causes for reliability issues originating from your PostgreSQL database, including: - [Table malfunction](/reference/root-causes/datapipeline#table-access-failure-in-database) and [excessive locks](/reference/root-causes/datapipeline#contention-on-database-table-locks) - [Slow database queries](/reference/root-causes/applications#slow-database-queries) - [Connection exhaustion](/reference/root-causes/applications#database-connection-pool-saturated) - Observe the database as an entity in the Topology Graph, including it's relationships to other entities on the service map, infrastructure stack and dataflow map. - Get insights into the slowest SQL queries, and troubleshoot them with our [MCP Server](/agent-integration/mcp-server/) directly from your favorite agent. The integration supports both self-hosted PostgreSQL instances and cloud-managed services including AWS RDS PostgreSQL, Azure Database for PostgreSQL, and Google Cloud SQL for PostgreSQL. ## Setup Guide ### Step 1: Create a user Create a user for your PostgreSQL database server with the following permissions: - `pg_read_all_stats`: Required to access statistics views and performance data - `pg_stat_statements`: Required to monitor query performance (if using pg_stat_statements extension) - `SELECT`: Required to query performance metrics, table information, and schema details - `USAGE`: Required to access schema information and metadata ### Step 2: Enable performance monitoring features Enable performance monitoring features in your PostgreSQL instance to collect detailed metrics and query information. This step varies depending on your PostgreSQL deployment type, as detailed below. #### Native PostgreSQL For native PostgreSQL installations, you need to enable the pg_stat_statements extension: 1. Edit your PostgreSQL configuration file (typically `postgresql.conf`) 2. Add or update these settings: ```ini shared_preload_libraries = 'pg_stat_statements' pg_stat_statements.track = all ``` 3. Restart PostgreSQL 4. Connect to your database and run: ```sql CREATE EXTENSION IF NOT EXISTS pg_stat_statements; ``` #### AWS RDS PostgreSQL - [AWS RDS PostgreSQL Documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html) - [RDS Parameter Groups](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithParamGroups.html) - [Enabling pg_stat_statements in AWS](https://pganalyze.com/docs/install/amazon_rds/01_configure_rds_instance) Steps: 1. Go to RDS Console → Parameter Groups 2. Edit your parameter group 3. Set `shared_preload_libraries` to include `pg_stat_statements` 4. Restart your instance 5. Connect and run: `CREATE EXTENSION IF NOT EXISTS pg_stat_statements;` #### Azure Database for PostgreSQL - [Azure PostgreSQL Documentation](https://learn.microsoft.com/en-us/azure/postgresql/) - [Server Parameters in Azure](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/how-to-server-parameters-set-value) - [Enabling pg_stat_statements in Azure](https://pganalyze.com/docs/install/azure_database/01_configure_azure_instance) Steps: 1. Go to Azure Portal → PostgreSQL server 2. Select "Server parameters" 3. Set `shared_preload_libraries` to include `pg_stat_statements` 4. Restart the server 5. Connect and run: `CREATE EXTENSION IF NOT EXISTS pg_stat_statements;` #### Google Cloud SQL for PostgreSQL - [Cloud SQL PostgreSQL Documentation](https://cloud.google.com/sql/docs/postgres) - [Database Flags](https://cloud.google.com/sql/docs/postgres/flags) - [Enabling pg_stat_statements in GCP](https://pganalyze.com/docs/install/google_cloud_sql/01_create_monitoring_user) Steps: 1. Go to Cloud Console → Cloud SQL 2. Edit your instance 3. Add database flag: `shared_preload_libraries=pg_stat_statements` 4. Restart the instance 5. Connect and run: `CREATE EXTENSION IF NOT EXISTS pg_stat_statements;` ### Step 3: Create a Kubernetes secret for the user After creating the user create a Kubernetes secret. You have two options: #### Option 1: Single Database Configuration ```bash kubectl create secret generic \ --namespace causely postgres-credentials \ --from-literal=username="..." \ --from-literal=password='...' \ --from-literal=host="..." \ --from-literal=port=5432 \ --from-literal=database="..." \ --from-literal=sslmode="..." ``` #### Option 2: Multiple Databases Configuration For monitoring multiple databases within the same PostgreSQL instance, you can specify multiple databases using the `databases` field: ```bash kubectl create secret generic \ --namespace causely postgres-credentials-multidb \ --from-literal=username="..." \ --from-literal=password='...' \ --from-literal=host="..." \ --from-literal=port=5432 \ --from-literal=databases="database1,database2,database3" \ --from-literal=sslmode="..." ``` Alternatively, you can create the secret using a YAML manifest: ```yaml apiVersion: v1 kind: Secret metadata: name: postgres-credentials-multidb namespace: causely type: Opaque stringData: username: '...' password: '...' host: '...' port: '5432' databases: 'database1,database2,database3' sslmode: '...' ``` **Note**: Use either the `database` field for single database configuration or the `databases` field for multiple databases. Do not use both fields in the same secret. #### Option 3: Database auto-discovery Causely can discover all databases on a PostgreSQL server automatically with the same credentials. Add `auto_discovery: "true"` to the secret (for example in `stringData`). The `host` must be the FQDN of your DB, or IP address if there's no DNS entry set up. It must match the FQDN/IP Causely would discover either from the K8s Server (if it's running in your K8s Cluster) or your Cloud providers API. Examples: - _Kubernetes_: `host=my-postgres.namespace.service.cluster.local` - _AWS_: `host=myinstance.rds.amazonaws.com` If you are leveraging a proxy to connect to your database, like it's common on GCP, `host` must be the FQDN/IP of your proxy and additionally `host_overwrite` should be the IP/FQDN of the actual database service: - _GCP_: `host=localhost` (assuming proxy runs on the same host / Pod). `host_overwrite=1.2.3.4` (the IP of your GCP Cloud SQL instance as shown in the GCP Console) ### Step 4: Update Causely Configuration Once the secret is created, update the Causely configuration to enable scraping for the new database. Below is an example configuration: ```yaml scrapers: postgresql: enabled: true instances: - secretName: postgres-credentials namespace: causely ``` ### Alternative: Enable Credentials Autodiscovery Causely also supports credentials autodiscovery. This feature allows you to add new scraping targets without updating the Causely configuration. Label the Kubernetes secret to enable autodiscovery for the corresponding scraper. ```bash kubectl --namespace causely label secret postgres-credentials "causely.ai/scraper=Postgresql" ``` ## Verify Your Configuration After completing the setup, use these checks to verify your PostgreSQL instance is properly configured for Causely slow query monitoring. ### Quick Check Query Run this single query to check all requirements at once: ```sql SELECT CASE WHEN EXISTS (SELECT 1 FROM pg_stat_user_tables LIMIT 1) THEN 'PASS' ELSE 'FAIL: Missing pg_read_all_stats' END AS perm_check, CASE WHEN EXISTS (SELECT 1 FROM pg_extension WHERE extname = 'pg_stat_statements') THEN 'PASS' ELSE 'FAIL: Extension missing' END AS ext_check, CASE WHEN EXISTS (SELECT 1 FROM pg_stat_statements LIMIT 1) THEN 'PASS' ELSE 'FAIL: Not loaded/accessible' END AS access_check; ``` **Expected result** (all checks passing): ``` perm_check | ext_check | access_check ------------+-----------+-------------- PASS | PASS | PASS ``` | Check | What it verifies | | -------------- | ---------------------------------------------------------------- | | `perm_check` | User can access performance views (has `pg_read_all_stats` role) | | `ext_check` | The `pg_stat_statements` extension is installed | | `access_check` | The extension is loaded and queryable | If any check shows `FAIL`, follow the detailed checks below to resolve the issue. ### Detailed Checks and Fixes #### 1. Permissions Check ```sql -- Should return a row without error (tests pg_read_all_stats permission) SELECT 1 FROM pg_stat_user_tables LIMIT 1; -- Should return a row without error (tests information_schema access) SELECT 1 FROM information_schema.columns LIMIT 1; ``` **If the query fails**, grant the required role to your Causely user: ```sql -- Run as superuser (or rds_superuser on AWS RDS) GRANT pg_read_all_stats TO ; ``` #### 2. Extension Check ```sql -- Should return: t (true) SELECT EXISTS(SELECT 1 FROM pg_extension WHERE extname = 'pg_stat_statements'); -- Should return a row without error SELECT 1 FROM pg_stat_statements LIMIT 1; ``` **If the extension is missing**, create it: ```sql CREATE EXTENSION IF NOT EXISTS pg_stat_statements; ``` **If the query fails** even though the extension exists, the extension is not loaded. You need to configure `shared_preload_libraries` and restart your instance. See [Step 2: Enable performance monitoring features](#step-2-enable-performance-monitoring-features) for platform-specific instructions. #### 3. Shared Preload Libraries Check ```sql -- Should show: pg_stat_statements in the value SELECT setting FROM pg_settings WHERE name = 'shared_preload_libraries'; ``` If `pg_stat_statements` is not in the output, you need to add it to your configuration and restart the instance. Refer to the platform-specific instructions in [Step 2](#step-2-enable-performance-monitoring-features). ### Test Slow Query Access Run this query to confirm Causely can collect slow query data: ```sql SELECT query, calls, total_exec_time FROM pg_stat_statements ORDER BY total_exec_time DESC LIMIT 5; ``` This query should return results without error. If the table is empty, that's normal for a freshly configured instance, queries will appear after your application runs. :::tip[Success] If the test query works, your PostgreSQL instance is correctly configured for Causely slow query monitoring. ::: ### Setup Checklist Use this checklist to verify all configuration steps are complete: - [ ] `shared_preload_libraries` includes `pg_stat_statements` (requires instance restart) - [ ] Extension created: `CREATE EXTENSION IF NOT EXISTS pg_stat_statements;` - [ ] User has `pg_read_all_stats` role: `GRANT pg_read_all_stats TO ;` - [ ] Verification query succeeds: `SELECT 1 FROM pg_stat_statements LIMIT 1;` ## What Data is Collected The PostgreSQL scraper collects comprehensive metadata and performance information from your PostgreSQL databases, including: - **Database entities** with names and relationships to hosting services - **Service-to-database mappings** (which service provides which database) - **Connection details** including host, port, and SSL configuration - **Database and table schemas** with complete structural information - **Table information** including names, row counts, and sizes - **Complete table schemas** with column definitions, data types, constraints, and indexes - **Foreign key relationships** and table dependencies - **Parent-child table mappings** for dependency analysis - **Slow query analysis** using PostgreSQL's `pg_stat_statements` - **Lock monitoring** - **Cache and I/O performance** - **Table performance metrics** --- ## Prometheus Causely leverages [Prometheus](https://github.com/prometheus-operator/kube-prometheus) to query time-series metrics from your applications and infrastructure. To enable Prometheus as a data source with the default exporters, please add the following section to your `values.yaml` file: ```yaml scrapers: prometheus: enabled: true servers: prometheus: endpoint: http://prometheus-operated.monitoring:9090 ``` ## Supported Exporters Causely will automatically pick up metrics from supported exporters. Supported Exporters: - [cAdvisor](https://github.com/google/cadvisor) - Container resource usage and performance metrics - [CockroachDB](https://www.cockroachlabs.com/docs/stable/monitor-cockroachdb-with-prometheus) - CockroachDB database metrics - [Golang](https://prometheus.io/docs/guides/go-application/) - Go application metrics including mutex and garbage collection - [Java](https://prometheus.github.io/client_java/instrumentation/jvm/) - Java application metrics including JVM memory and garbage collection - [Kafka](https://github.com/danielqsj/kafka_exporter) - Kafka broker and topic metrics - [MongoDB](https://github.com/percona/mongodb_exporter) - MongoDB database metrics - [MySQL](https://github.com/prometheus/mysqld_exporter) - MySQL database metrics - [PostgreSQL](https://github.com/prometheus-community/postgres_exporter) - PostgreSQL database metrics - [Python](https://prometheus.io/docs/guides/python-client/) - Python application metrics - [RabbitMQ](https://www.rabbitmq.com/docs/prometheus) - RabbitMQ message broker metrics - [Redis](https://github.com/oliver006/redis_exporter) - Redis cache and database metrics :::info Prometheus Discovery When using Prometheus with Kubernetes service discovery, `namespace` and `pod` labels are typically added automatically by the Prometheus Helm chart through relabeling. If these labels are missing from your metrics, configure relabeling in your ServiceMonitor or PodMonitor configuration to map Kubernetes metadata (`__meta_kubernetes_namespace`, `__meta_kubernetes_pod_name`) to `namespace` and `pod` labels. If you're using the OpenTelemetry Collector with the `prometheus/metrics` receiver, see the [OpenTelemetry Prometheus metrics relabeling configuration](/telemetry-sources/opentelemetry#configuring-prometheus-metrics-relabeling) for how to configure relabeling in the collector. ::: ## How to add custom metrics To enable Prometheus as a data source with default exporters plus additional custom metrics, please add the following section to your `values.yaml` file: ```yaml scrapers: prometheus: enabled: true servers: prometheus: endpoint: http://prometheus-operated.monitoring:9090 exporters: - cadvisor - causely - go-applications - java-applications - python-applications - postgres - mongodb - cockroachdb - redis - kafka - mysql - rabbitmq - chaosmania exporters: chaosmania: entities: - entity: workload: {} discovery: - kubernetes_pod: namespace: 'namespace' pod_name: 'pod' metrics: - attribute: TransactionCount query: 'sum by (namespace, service, pod) (rate(chaosmania_processed_transactions_duration_count[1m]))' - attribute: TransactionDuration query: 'sum by (namespace, service, pod) (rate(chaosmania_processed_transactions_duration_sum[1m]) / (rate(chaosmania_processed_transactions_duration_count[1m]) > 0 or (rate(chaosmania_processed_transactions_duration_count[1m]) + 1)))' - attribute: DBQueryDuration query: 'sum by (namespace, service, pod) (rate(postgres_queries_sum[1m]) / (rate(postgres_queries_count[1m]) > 0 or (rate(postgres_queries_count[1m]) + 1)))' ``` ## Example: Kafka Exporter You can use this helm `values.yaml` file for the kafka exporter to label the broker name `super-heroes-kafka-brokers` for the monitored topics: ```yaml kafkaServer: - super-heroes-kafka-brokers:9092 service: annotations: prometheus.io/port: '9308' prometheus.io/scrape: 'true' labels: service_name: super-heroes-kafka-brokers prometheus: serviceMonitor: enabled: true namespace: quarkus-super-heroes relabelings: - sourceLabels: [__meta_kubernetes_service_label_service_name] targetLabel: target ``` ## Basic Authentication When your Prometheus or Mimir server requires HTTP Basic Authentication, you can supply credentials so the mediator can authenticate. If the server is configured via a Kubernetes Secret (for example with auto-discovery), add `username` and `password` keys to that secret. The mediator will send HTTP Basic Auth headers on every request to the endpoint. Secret keys: - **`username`**: Basic auth username (optional; omit if the server does not use basic auth). - **`password`**: Basic auth password (optional; omit if the server does not use basic auth). If both are present, the mediator uses them for all Prometheus API and query requests to that server. This applies to both single-tenant Prometheus and multi-tenant Mimir configurations when the server is provided via a secret. ## Multi-tenant Mimir Causely supports multi-tenant [Mimir deployments](https://grafana.com/docs/mimir/latest/manage/secure/authentication-and-authorization/), allowing you to configure multiple tenants and specific exporters for each Mimir instance. To enable Mimir as a data source with multi-tenant support, add the following section to your `values.yaml` file: ```yaml scrapers: prometheus: enabled: true servers: mimir: endpoint: http://mimir-nginx.mimir:80/prometheus tenants: - tenant-1 - tenant-2 exporters: - postgres - redis ``` This configuration allows Causely to: - Connect to the Mimir instance via the specified endpoint - Query metrics from multiple tenants (`tenant-1` and `tenant-2`) --- ## Snowflake Causely provides native integration with Snowflake. ## Step 1: Create a user Create a user for your Snowflake account. ## Step 2: Create a Kubernetes Secret for the user After creating the user create a Kubernetes Secret: ```bash kubectl create secret generic --namespace causely snowflake-credentials \ --from-literal=username="..." \ --from-literal=password='...' \ --from-literal=account="xxxxxx-yyyyyy" \ --from-literal=warehouse="..." ``` ## Step 3: Update Causely Configuration Once the Secret is created, update the Causely configuration to enable scraping for the new database. Below is an example configuration: ```yaml scrapers: snowflake: enabled: true instances: - secretName: snowflake-credentials namespace: causely ``` ## Alternative: Enable Credentials Autodiscovery Causely also supports credentials autodiscovery. This feature allows you to add new scraping targets without updating the Causely configuration. Simply label the Kubernetes Secret to enable autodiscovery for the corresponding scraper. ### Example: Labeling Secrets for Autodiscovery ```bash kubectl --namespace causely label snowflake-credentials "causely.ai/scraper=Snowflake" ``` With these steps, you can seamlessly integrate Causely with Snowflake and configure it to monitor your desired resources. --- ## Splunk ## Overview Causely integrates with [Splunk](https://www.splunk.com/) for **log retrieval**: when your container logs are already centralized in Splunk, Causely can query Splunk instead of the Kubernetes API to pull those logs and correlate them with service-level incidents and Diagnoses. Use this integration if: - Your logs are already shipped to Splunk (for example via Splunk Universal Forwarder, Fluentd, or other collectors). - You want Causely to surface relevant logs in the context of an active Diagnosis or service malfunction, without collecting logs from Kubernetes directly. ## Benefits - **Automatic log context**: Surfaces relevant logs in the context of an active Diagnosis or service malfunction. - **Container-level visibility**: Displays container-level logs under affected services when abnormal behavior occurs (for example error spikes or degraded performance). - **Faster resolution**: Shows log lines and exceptions alongside Diagnoses to validate issues and shorten time to understanding and resolution. ## Setup ### Step 1: Create a Kubernetes Secret for Splunk authentication If your Splunk deployment requires authentication, create a Kubernetes Secret containing a Splunk token: ```bash kubectl create secret generic splunk-secret \ --namespace causely \ --from-literal=token="your-splunk-token" ``` Use the same secret name in the configuration below (for example `splunk-secret`). If Splunk does not require authentication, you can omit the `secret` field in the configuration. ### Step 2: Update the Kubernetes scraper configuration Enable Splunk for log retrieval by adding the `splunk` block under the Kubernetes scraper in your `causely-values.yaml` (or Helm values): ```yaml scrapers: kubernetes: splunk: endpoint: "https://splunk:8089" secret: "splunk-secret" # Optional: omit if no auth required index: "main" # Optional: Splunk index for log queries; defaults to "main" if omitted insecureSkipVerify: false # Optional: set true only for dev/test to skip TLS verification (insecure) ``` #### Configuration reference | Field | Required | Description | |-------|----------|-------------| | `endpoint` | Yes | Splunk API endpoint URL (for example `https://splunk.example.com:8089`). | | `secret` | No | Kubernetes secret name in the Causely namespace containing the key `token` with your Splunk token. Omit if authentication is not required. | | `index` | No | Splunk index used for log queries. Defaults to `main` if omitted. | | `insecureSkipVerify` | No | Set to `true` only in dev/test environments to skip TLS certificate verification. **Insecure**; do not use in production. Defaults to `false`. | ### Step 3: Install or upgrade Causely Apply your values and install or upgrade the Causely Helm release so the mediator uses the Splunk configuration for log retrieval. ## Authentication - **Token**: Include `token` in the Kubernetes secret referenced by `secret`. This is the standard way to authenticate to Splunk (for example HEC token or API token). - **No auth**: Omit the `secret` field in the configuration if your Splunk endpoint does not require authentication. ## Supported log types - Container logs (stdout/stderr) that have been shipped to Splunk and are queryable in the configured index. ## Result - Causely retrieves and displays logs from Splunk alongside a Diagnosis, highlighting the precise errors or stack traces that occurred around the time of failure. - Logs are correlated with affected services and Diagnoses to speed up validation and resolution. --- ## Custom Workflows To send Causely notifications to a custom destination, you route them through [CauselyBot](https://github.com/causely-oss/causelybot), an open source webhook service that receives the Causely notification payload and forwards it to the endpoints you configure. For arbitrary HTTP endpoints (n8n, Linear, custom services), use CauselyBot's `generic` hook type. :::note Do you need full control? Use CauselyBot when you want **full control** over how the webhook is processed and the exact payload sent to your target system. If you only need to forward Causely's standard payload to an HTTP endpoint as-is, use the simpler [Generic Webhook](/workflows/generic-webhook) option instead. ::: Routing and rules for Managed Notifications are configured in the Causely UI: [Set up notification routing in the UI](/workflows/ui-notifications) ## Setup ### Step 1: Enable notifications in values.yaml In your `causely-values.yaml`, enable mediator notifications: ```yaml mediator: notifications: enabled: true ``` Apply the change: ```bash helm upgrade --install causely --create-namespace \ oci://us-docker.pkg.dev/public-causely/public/causely \ --version --namespace=causely --values ./causely-values.yaml ``` ### Step 2: Deploy CauselyBot Create a `causelybot-values.yaml` with your webhook configuration. For a generic HTTP endpoint: ```yaml auth: token: "" # Required - define your token here and then use in the CauselyBot secret creation in step 3. webhooks: - name: "my-generic-webhook" # Provide a friendly name for the webhook hook_type: "generic" url: "https://" ``` See the [CauselyBot documentation](https://github.com/causely-oss/causelybot) for all supported hook types and options. Install CauselyBot via Helm: ```bash helm upgrade --install causelybot ./causelybot/helm/causelybot \ --namespace causelybot --values causelybot-values.yaml ``` ### Step 3: Create the notification Secret Create a Kubernetes Secret in the `causely` namespace that points Causely at CauselyBot (use `stringData` so values are plain text; if you use `data`, values must be base64-encoded): ```yaml apiVersion: v1 kind: Secret type: Opaque metadata: name: causelybot-notification-config namespace: causely labels: causely.ai/notif-config: CauselyBot #Required stringData: notif_config_name: causelybot-webhook notif_config_global: "false" notif_config_type: CauselyBot notif_config_url: https:///webhook # or if in-cluster: http://causelybot.causelybot.svc.cluster.local.:5000/webhook notif_config_token: "" notif_config_filters: "[]" notif_config_filters_enabled: "true" ``` :::note `notif_config_token` must match the `auth.token` you set in CauselyBot's configuration. Causely sends it as `Authorization: Bearer ` so CauselyBot can authenticate the request. ::: ## Filtering notifications (notif_config_filters) `notif_config_filters` is a JSON array that controls which notifications are forwarded to CauselyBot. When `notif_config_filters_enabled` is `"false"` or the array is empty (`"[]"`), all notifications pass through. Multiple filters are ANDed: a notification must match **all** filters to be forwarded. ### Filterable fields | Field | Description | |---|---| | `severity` | `"Low"`, `"Medium"`, `"High"`, or `"Critical"` | | `name` | Problem name (for example, `"Malfunction"`, `"CPUCongested"`) | | `entity.type` | Entity type (for example, `"KubernetesService"`, `"ApplicationInstance"`) | | `labels.k8s.cluster.name` | Kubernetes cluster name | | `labels.k8s.namespace.name` | Kubernetes namespace name | | `impactsSLO` | `true` if the notification includes impacted SLOs | Supported operators: `equals`, `not_equals`, `in`, `not_in` ### Examples **Forward only High and Critical severity notifications:** ```yaml notif_config_filters: '[{"field":"severity","operator":"in","value":["High","Critical"]}]' notif_config_filters_enabled: "true" ``` **Forward notifications from a specific cluster only:** ```yaml notif_config_filters: '[{"field":"labels.k8s.cluster.name","operator":"equals","value":"prod"}]' notif_config_filters_enabled: "true" ``` **Forward only SLO-impacting issues in a specific namespace:** ```yaml notif_config_filters: '[{"field":"impactsSLO","operator":"equals","value":true},{"field":"labels.k8s.namespace.name","operator":"equals","value":"production"}]' notif_config_filters_enabled: "true" ``` ## Example: GitHub via CauselyBot 1. Deploy [CauselyBot](https://github.com/causely-oss/causelybot) and configure a webhook with `hook_type: github`, `url: "owner/repo"`, and a GitHub token (repo and issues scope). Optionally set `assignee` in CauselyBot's config. 2. In the Secret above, set `notif_config_url` to CauselyBot's webhook URL and `notif_config_token` to CauselyBot's auth token. 3. Causely sends notifications to CauselyBot; CauselyBot creates or updates GitHub issues from the payload. :::tip You can assign an issue to [GitHub Copilot](https://github.com/features/copilot) by setting the `assignee` to `copilot-swe-agent`, which will trigger Copilot to analyze the issue and suggest a fix in a pull request. ::: ## Example: Generic webhook (n8n, Linear, etc.) To send to n8n or any other HTTP endpoint: 1. In CauselyBot's `causelybot-values.yaml`, add a webhook with `hook_type: generic` and set `url` to your destination endpoint (for example, the n8n Webhook node URL). 2. Create the Causely Secret as described in Step 3 above, pointing `notif_config_url` at CauselyBot. 3. CauselyBot forwards the raw Causely notification payload to your endpoint. 4. In n8n, read the incoming JSON fields (for example, `name`, `entity`, `severity`, `description`, `link`) from the [notification payload](/workflows/notification-payload) and use them in downstream nodes (Slack, email, Linear, etc.). ## Notification payload format Causely sends the same structured JSON to all webhook destinations. For field descriptions and an example, see [Notification Payload Format](/workflows/notification-payload). --- ## Generic Webhook A generic webhook sends Causely's standard notification payload, as JSON, to any HTTP endpoint. Use it to reach systems Causely doesn't support out of the box. All generic webhook delivery runs through the **Causely Mediator**, which sends an HTTP `POST` with the [Causely notification payload](/workflows/notification-payload) as the request body. How you define a generic webhook destination depends on your setup: - **Managed Notifications (recommended)** The endpoint URL, token, and routing rules are configured in the Causely UI. The UI stores configuration that the Mediator reads at runtime. - **Mediator-based configuration (advanced)** Define the destination programmatically with a Kubernetes secret or environment variables instead of the UI. Uses the same Mediator delivery path as managed notifications. For routing rules and filters in the UI, see [Set up notification routing in the UI](/workflows/ui-notifications). :::note When to use a generic webhook vs. Custom Workflows Use a **generic webhook** when your endpoint can accept Causely's payload as-is. If you need to **transform the payload** or **control the exact request format** sent to your target system, use [Custom Workflows](/workflows/custom-workflows) (CauselyBot) instead, which gives you full control over how the webhook is processed and formatted. ::: ## Managed Notifications (recommended) 1. Go to **Settings → Notifications**. 2. If you haven't already selected a processing mode, choose one now. Click **Create**. 3. Complete the notification configuration: | Field | Value | |-------|--------| | **Name** | A descriptive name for the notification | | **Type** | Generic | | **URL** | Your endpoint, for example `https://api.example.com/notifications` | | **Token** | Optional; see [Authentication](#authentication) below | 4. Choose the cluster or clusters that handle processing and delivery. 5. Optionally add [Notification Filters](#filters) to control when notifications are sent. 6. Click **Create**. ## Authentication The **Token** field maps directly to the HTTP `Authorization` header. Causely sends whatever you enter **verbatim**, and it does not add or assume any authentication scheme. This means you are responsible for supplying the full header value, including the scheme keyword, that your endpoint expects. | Your endpoint expects | Enter in the Token field | |-----------------------|--------------------------| | Bearer token | `Bearer ` | | Basic auth | `Basic ` | | A raw API key header value | Exactly what the endpoint expects | | No authentication | Leave blank | :::note Include the `Bearer` keyword yourself If your endpoint uses bearer authentication, enter the full value `Bearer ` in the Token field, including the word `Bearer`. Causely does **not** add the `Bearer` prefix for the generic type. ::: When the token is left blank, no `Authorization` header is sent. Requests are always sent with `Content-Type: application/json`. ## Mediator-based configuration (advanced) If you prefer not to use the UI, define the generic webhook destination directly on the Mediator with a Kubernetes secret or environment variables. ### Kubernetes secret Create a secret labelled `causely.ai/notif-config: Generic` in the Mediator namespace. Set `notif_config_url` to your endpoint and, optionally, `notif_config_token` to the full `Authorization` header value. ```yaml apiVersion: v1 kind: Secret type: Opaque metadata: name: generic-notification-config namespace: causely labels: causely.ai/notif-config: Generic stringData: notif_config_filters: "[]" notif_config_filters_enabled: "true" notif_config_global: "false" notif_config_name: generic-webhook notif_config_token: "Bearer " # optional; sent verbatim as the Authorization header notif_config_type: Generic notif_config_url: https://api.example.com/notifications ``` ### Environment variables Set these on the Causely Mediator deployment. Replace `` with your notification config name. ```bash NOTIFICATION__TYPE=Generic NOTIFICATION__URL=https://api.example.com/notifications NOTIFICATION__TOKEN=Bearer NOTIFICATION__FILTERS_ENABLED=true ``` ## Filters Generic webhook destinations support filters (severity, namespace, entity type, `impactsSLO`, and more) like other Causely notification targets. Configure them in the create/edit modal in the UI, or via `notif_config_filters` in the Kubernetes secret. `notif_config_filters` is a JSON array. When `notif_config_filters_enabled` is `"false"` or the array is empty (`"[]"`), all notifications pass through. Multiple filters are ANDed: a notification must match **all** filters to be forwarded. | Field | Description | |---|---| | `severity` | `"Low"`, `"Medium"`, `"High"`, or `"Critical"` | | `name` | Problem name (for example, `"Malfunction"`, `"CPUCongested"`) | | `entity.type` | Entity type (for example, `"KubernetesService"`, `"ApplicationInstance"`) | | `labels.k8s.cluster.name` | Kubernetes cluster name | | `labels.k8s.namespace.name` | Kubernetes namespace name | | `impactsSLO` | `true` if the notification includes impacted SLOs | Supported operators: `equals`, `not_equals`, `in`, `not_in`. For example, to forward only High and Critical severity notifications: ```yaml notif_config_filters: '[{"field":"severity","operator":"in","value":["High","Critical"]}]' notif_config_filters_enabled: "true" ``` For more on configuring filters in the UI, see [Managed Notifications](/workflows/ui-notifications). --- ### Notification Payload Format Causely sends the same structured JSON to all webhook destinations, including summaries of causal insights and service impact context. For an example payload and explanation of each field, refer to the [Notification Payload Format](/workflows/notification-payload) page. --- ## Grafana Alertmanager Causely can send causal insights directly to your existing Grafana Alertmanager setup, allowing your team to respond quickly with actionable context. This integration enables you to route high-signal alerts into your existing alerting workflows, no additional rules or duplication needed. Grafana Alertmanager can be used as a notification destination in Causely. All Grafana Alertmanager delivery runs through the **Causely Mediator**. How you define Grafana Alertmanager destinations depends on your setup: - **Managed Notifications (recommended)** Routing rules and Alertmanager credentials are configured in the Causely UI. The UI stores configuration that the Mediator reads at runtime. - **Mediator-based configuration (advanced)** Define Alertmanager destinations programmatically with environment variables or Kubernetes secrets instead of the UI. Uses the same Mediator delivery path as managed notifications. For routing rules and filters in the UI, see [Set up notification routing in the UI](/workflows/ui-notifications). ## Mediator-based configuration (advanced) If you prefer not to use the UI, define Grafana Alertmanager destinations with a Kubernetes secret in the causely namespace: ```yaml apiVersion: v1 kind: Secret type: Opaque metadata: name: grafana-notification-config namespace: causely labels: causely.ai/notif-config: AlertManager data: notif_config_name: grafana notif_config_global: false notif_config_token: "user:password" notif_config_type: AlertManager notif_config_url: https://alertmanager-prod-us-east-0.grafana.net/alertmanager/api/v2/alerts notif_config_filters: [] notif_config_filters_enabled: true ``` :::note Replace the placeholder values with your actual Alertmanager URL in the "notif_config_url" field and optional credentials in the "notif_config_token" field If you're using Grafana Cloud, refer to your instance's alerting endpoint. ::: --- ### Notification Payload Format Causely sends structured alert payloads to Alertmanager, including service-level impact and diagnostic summaries. For an example and explanation of each field, refer to the [Notification Payload Format](/workflows/notification-payload) page. --- ## Causely Grafana Plugin ## Causal Insights in Grafana Dashboards See actual causes and service impact directly in your Grafana dashboards, whether you're using Grafana Cloud or a self-hosted instance. Causely for Grafana embeds high-context diagnostics alongside your existing metrics, helping teams quickly understand what broke, why it matters, and which services are affected. No code changes or custom alerting rules required. --- ### Plugin Overview - View active causes next to dashboards and alerts - Drill into service relationships and impact - Works with Grafana Cloud or self-hosted deployments --- ### Install the Plugin You can find the plugin on the Grafana Plugin Marketplace including setup guidance and official plugin documentation. [**Install the Causely Plugin on Grafana**](https://grafana.com/grafana/plugins/esara-causely-app/?tab=installation) --- ## incident.io Integration(Workflows) Causely can automatically create and resolve incidents in [incident.io](https://incident.io) based on causal inference findings. This integration enables your team to maintain a centralized incident management workflow while leveraging Causely's deep system insights. When Causely detects critical or high-severity issues, it can automatically create structured incidents in incident.io with relevant context, severity mapping, and resolution tracking. incident.io can be used as an incident management destination in Causely. All incident.io delivery runs through the **Causely Mediator**. How you define incident.io destinations depends on your setup: - **Managed Notifications (recommended)** Routing rules and incident.io credentials are configured in the Causely UI. The UI stores configuration that the Mediator reads at runtime. - **Mediator-based configuration (advanced)** Define incident.io destinations programmatically with environment variables or Kubernetes secrets instead of the UI. Uses the same Mediator delivery path as managed notifications. For routing rules and filters in the UI, see [Set up notification routing in the UI](/workflows/ui-notifications). --- ## Mediator-based configuration (advanced) If you prefer not to use the UI, define incident.io destinations with a Kubernetes secret in the causely namespace: ```yaml apiVersion: v1 kind: Secret type: Opaque metadata: name: incidentIO-notification-config namespace: causely labels: causely.ai/notif-config: IncidentIO stringData: notif_config_filters: "[]" notif_config_filters_enabled: "true" notif_config_global: "false" notif_config_name: incidentIO notif_config_token: "your-incident-io-api-token" notif_config_type: IncidentIO notif_config_url: "" ``` :::note Replace `your-incident-io-api-token` with your actual incident.io API token. You can generate one in your incident.io settings under API tokens. ::: --- ### How It Works #### Incident Creation - **Automatic Creation**: Causely automatically creates incidents for **High** and **Critical** severity issues - **Severity Mapping**: Causely severity levels are mapped to your incident.io severity configuration - **Rich Context**: Incidents include detailed summaries, affected services, and direct links back to Causely's analysis - **Idempotency**: Uses Causely's object ID to prevent duplicate incidents for the same issue #### Incident Resolution - **Automatic Resolution**: When Causely detects that an issue is resolved, it can automatically update the incident status - **Configurable Status**: Set the target resolution status name (for example "Fixed", "Resolved") - **Channel Notifications**: Optionally notify the incident channel when incidents are resolved --- ### Configuration Options #### Required Settings - `api_token`: Your incident.io API token with incident creation permissions #### Optional Settings - `incident_type_id`: Specific incident type ID (auto-detected if not provided) - `severity_id`: Specific severity ID (defaults to "Critical" severity if not provided) - `resolved_status_name`: Status name for resolved incidents (enables auto-resolution) - `notify_channel`: Whether to notify the incident channel on resolution (default: false) #### Advanced Configuration The Helm chart automatically configures the required environment variables based on your `causely-values.yaml` settings. The following environment variables are set automatically: - `INCIDENT_IO_API_TOKEN`: Set from the secret containing your API token - `INCIDENT_IO_INCIDENT_TYPE_ID`: Set from `incident_type_id` if provided - `INCIDENT_IO_SEVERITY_ID`: Set from `severity_id` if provided - `INCIDENT_IO_RESOLVED_STATUS_NAME`: Set from `resolved_status_name` if provided - `INCIDENT_IO_NOTIFY_CHANNEL`: Set from `notify_channel` if provided --- ### Notification Payload Format Causely sends rich notification data to incident.io based on its causal inference. For an example and explanation of each field, refer to the [Notification Payload Format](/workflows/notification-payload) page. --- ## Microsoft Teams Microsoft Teams can be used as a notification destination in Causely. All Microsoft Teams delivery runs through the **Causely Mediator**. How you define Microsoft Teams destinations depends on your setup: - **Managed Notifications (recommended)** Routing rules and Microsoft Teams credentials are configured in the Causely UI. The UI stores configuration that the Mediator reads at runtime. Microsoft Teams delivery also requires a one-time Teams Workflow setup, described below. - **Mediator-based configuration (advanced)** Define Microsoft Teams destinations programmatically with environment variables or Kubernetes secrets instead of the UI. Uses the same Mediator delivery path as managed notifications. For routing rules and filters in the UI, see [Set up notification routing in the UI](/workflows/ui-notifications). ## Managed Notifications (Recommended) Most teams use Managed Notifications to route Causely insights to Microsoft Teams. In this model: - Notification routing and logic are defined in the Causely UI - Microsoft Teams is used purely as a delivery destination - Alerts are delivered as Adaptive Cards To receive notifications in Microsoft Teams, you must configure a Teams Workflow to accept webhook requests and post Adaptive Cards to a channel. ## Mediator-based configuration (advanced) If you prefer not to use the UI, define Microsoft Teams destinations with a Kubernetes secret in the causely namespace: ```yaml apiVersion: v1 kind: Secret type: Opaque metadata: name: teams-notification-config namespace: causely labels: causely.ai/notif-config: Teams stringData: notif_config_filters: "[]" notif_config_filters_enabled: "true" notif_config_global: "false" notif_config_name: teams notif_config_token: "causely" notif_config_type: Teams notif_config_url: "https://your-domain.webhook.office.com/webhookb2/..." ``` :::note Replace the placeholder values with your actual Teams Workflow webhook URL. Ensure the Microsoft Teams channel webhook is set up to accept incoming messages. ::: ## Microsoft Teams Workflow Setup Microsoft Teams requires a Teams Workflow to receive webhook requests from Causely and render notifications as Adaptive Cards. This workflow is configured once per Team or Channel. ### Step 1: Create a New Workflow 1. In Microsoft Teams, navigate to **Workflows** 2. Select **Create** 3. Choose **Create from blank** ### Step 2: Add the Webhook Trigger 1. Select the trigger **When a Teams webhook request is received** 2. For **Who can trigger the flow**, select **Anyone** This allows Causely to invoke the workflow using the generated webhook URL. ### Step 3: Process Adaptive Card Attachments Causely sends Microsoft Teams notifications as Adaptive Card attachments. 1. Add a new step and select **Apply to each** 2. For the input, select **attachments** from the webhook trigger output This ensures each Adaptive Card attachment is handled correctly by the workflow. ### Step 4: Post the Card to a Channel Inside the **Apply to each** block: 1. Add the action **Post card in a chat or channel** 2. Configure the required fields: - **Post as**: Flow bot - **Post in**: Channel - **Team**: Select your team - **Channel**: Select the destination channel - **Adaptive Card**: Select **content** from the attachments output ### Example Workflow Configuration The example below shows a completed Microsoft Teams Workflow with the webhook trigger, attachment handling, and channel post configured correctly. ### Result Once configured, each Causely notification triggers the Teams Workflow and posts a structured Adaptive Card into the selected channel, providing immediate, actionable context without alert noise. --- ### Notification Payload Format Causely sends structured alert payloads to Microsoft Teams, including summaries of causal insights and service impact context. For an example payload and explanation of each field, refer to the [Notification Payload Format](/workflows/notification-payload) page. --- ## Notification Payload Causely sends structured JSON payloads to all webhook-based integrations. Whether you're using Slack, Prometheus Alertmanager, Splunk On-Call, Microsoft Teams, or a custom system, the notification format below applies consistently. ### Diagnosis Detected The following payload is sent when Causely identifies a Diagnosis (`type: "ProblemDetected"`): ```json { "link": "https://portal.causely.app/rootCauses/b55bd69c-7417-43c7-b1d0-1c06075fa63b", "name": "ImagePullErrors", "type": "ProblemDetected", "entity": { "id": "741d88ac-0d47-5f61-b9c1-2d7739a6ebf8", "link": "https://portal.causely.app/observe/topology/741d88ac-0d47-5f61-b9c1-2d7739a6ebf8", "name": "default/pull-error-demo-2", "type": "Controller" }, "labels": { "k8s.cluster.uid": "8d92c7e7-db45-4267-a427-2fede28d1218", "k8s.cluster.name": "dev", "causely.ai/cluster": "dev", "k8s.namespace.name": "default", "k8s.controller.kind": "Deployment", "causely.ai/namespace": "default" }, "objectId": "b55bd69c-7417-43c7-b1d0-1c06075fa63b", "severity": "Low", "timestamp": "2026-04-28T19:28:31.686364684Z", "description": { "details": "Image pull errors in Kubernetes often occur due to one of the following reasons:\n\n- **Invalid image name or tag**: The specified image name or tag does not exist in the registry, or has been incorrectly referenced.\n- **Image registry authentication issues**: The Kubernetes node may lack the necessary credentials (e.g., in a private registry) or permissions to pull the image.\n- **Network connectivity issues**: Kubernetes nodes might have restricted access to the internet or specific image registries, possibly due to firewall settings, proxy issues, or DNS misconfigurations.\n- **Rate limiting by image registries**: Some registries, such as Docker Hub, impose pull rate limits, which can prevent nodes from pulling images if exceeded.\n", "summary": "Kubernetes controllers may encounter image pull errors when they cannot download container images from a registry, causing Pods to fail in starting or remain in a `ImagePullBackOff` state. This disrupts the deployment of applications and can affect service availability.", "remediationOptions": [ { "title": "Check Image Name and Tag", "description": "Verify that the image name and tag are correct and the image exists in your registry.\n" }, { "title": "Configure ImagePullSecrets for Private Registries", "description": "If using a private registry, ensure an `ImagePullSecret` is correctly set up and assigned to the namespace or Pod:\n\n```bash\nkubectl create secret docker-registry myregistrykey \\\n --docker-server= \\\n --docker-username= \\\n --docker-password= \\\n --docker-email=\n```\n\nAttach this secret to your Pod or namespace to provide credentials for image pulling.\n" } ] }, "object_type": "defect" } ``` ### Diagnosis Cleared The following payload is sent when a previously detected Diagnosis is resolved (`type: "ProblemCleared"`). It includes the same fields as the detected payload, plus `duration_ns` indicating how long the Diagnosis was active: ```json { "link": "https://portal.causely.app/rootCauses/b55bd69c-7417-43c7-b1d0-1c06075fa63b", "name": "ImagePullErrors", "type": "ProblemCleared", "entity": { "id": "741d88ac-0d47-5f61-b9c1-2d7739a6ebf8", "link": "https://portal.causely.app/observe/topology/741d88ac-0d47-5f61-b9c1-2d7739a6ebf8", "name": "default/pull-error-demo-2", "type": "Controller" }, "labels": { "k8s.cluster.uid": "8d92c7e7-db45-4267-a427-2fede28d1218", "k8s.cluster.name": "dev", "causely.ai/cluster": "dev", "k8s.namespace.name": "default", "k8s.controller.kind": "Deployment", "causely.ai/namespace": "default" }, "objectId": "b55bd69c-7417-43c7-b1d0-1c06075fa63b", "severity": "Low", "timestamp": "2026-04-28T19:28:31.686364684Z", "description": { "details": "Image pull errors in Kubernetes often occur due to one of the following reasons:\n\n- **Invalid image name or tag**: The specified image name or tag does not exist in the registry, or has been incorrectly referenced.\n- **Image registry authentication issues**: The Kubernetes node may lack the necessary credentials (e.g., in a private registry) or permissions to pull the image.\n- **Network connectivity issues**: Kubernetes nodes might have restricted access to the internet or specific image registries, possibly due to firewall settings, proxy issues, or DNS misconfigurations.\n- **Rate limiting by image registries**: Some registries, such as Docker Hub, impose pull rate limits, which can prevent nodes from pulling images if exceeded.\n", "summary": "Kubernetes controllers may encounter image pull errors when they cannot download container images from a registry, causing Pods to fail in starting or remain in a `ImagePullBackOff` state. This disrupts the deployment of applications and can affect service availability.", "remediationOptions": [ { "title": "Check Image Name and Tag", "description": "Verify that the image name and tag are correct and the image exists in your registry.\n" }, { "title": "Configure ImagePullSecrets for Private Registries", "description": "If using a private registry, ensure an `ImagePullSecret` is correctly set up and assigned to the namespace or Pod:\n\n```bash\nkubectl create secret docker-registry myregistrykey \\\n --docker-server= \\\n --docker-username= \\\n --docker-password= \\\n --docker-email=\n```\n\nAttach this secret to your Pod or namespace to provide credentials for image pulling.\n" } ] }, "duration_ns": 601639777492, "object_type": "defect" } ``` ### Payload Fields - `link`: Direct URL to the Diagnosis in the Causely portal. - `name`: The name of the Diagnosis. - `type`: The type of notification. Either `ProblemDetected` (Diagnosis identified) or `ProblemCleared` (Diagnosis resolved). - `entity`: Details of the affected entity: - `id`: Unique identifier of the entity. - `link`: Direct URL to the entity in the Causely portal topology view. - `name`: Name of the entity. - `type`: Type of the entity (for example, `Controller`). - `labels`: Metadata providing additional context such as cluster name, namespace, and controller kind. - `objectId`: Unique identifier for the Diagnosis. - `severity`: Severity level of the Diagnosis (for example, `Low`, `Medium`, `High`, or `Critical`). - `timestamp`: The timestamp when the event occurred (ISO 8601 format). - `description`: Structured details about the Diagnosis: - `summary`: A concise summary of the Diagnosis and its impact. - `details`: A detailed explanation of the Diagnosis, including possible contributing factors. - `remediationOptions`: A list of suggested remediation steps. Each item contains a `title` and a `description`. - `duration_ns`: *(ProblemCleared only)* The duration the Diagnosis was active, in nanoseconds. - `object_type`: The type of object associated with the event (for example, `defect`). --- ## Push Causal Insights Into Your Workflow Causely doesn’t just emit alerts, it delivers explainable, causal insight to the right people, in the right context, at the right time. Rather than forwarding raw metrics, logs, and traces, Causely acts as a reasoning layer between telemetry and action. It continuously models cause and effect across your system to determine what matters, why it matters, and who should care, so teams can move from noise to understanding, and from understanding to action. Causely supports multiple ways of delivering this insight, depending on how tightly you want understanding and action to be coupled. ### Native Integrations **Understand what changed and why, where engineers already work** Native integrations bring Causely’s causal explanations, including the Diagnosis, blast radius, and service impact, directly into the tools engineers use every day. This path is optimized for human understanding and collaboration. Engineers can quickly see not just that something is wrong, but what caused it, how it propagates, and what it affects, without switching context or interpreting raw telemetry. Use this approach when the primary goal is shared situational awareness, faster diagnosis, and better coordination during investigation and response. Available native integrations: ### Configuring Notifications There are two ways to configure where Causely sends notifications, and they offer the **same functionality**. Which one you use is simply a matter of preference for where you'd like to set things up: - **In the UI**: point-and-click configuration in the Causely product. - **Via a Kubernetes secret**: configuration as code, applied through the Mediator. The two stay in sync automatically: - When you configure a notification in the **UI**, the backing Kubernetes secret is **created for you automatically**. - When you create a **secret** for notifications, it appears as a **configured notification in the UI**. Pick whichever fits your workflow, both deliver the same notifications to the same destinations. ### Managed Notifications (UI-Configured) **Configure notifications directly in the Causely UI** Use the Causely UI to choose where notifications are delivered and add filters that control when they're sent, without editing configuration files. When you save a configuration, Causely creates the corresponding Kubernetes secret for you automatically. **Learn more**: [Managed notifications overview](/workflows/ui-notifications) ### Mediator-Based Notifications **Configure notifications with a Kubernetes secret** Prefer configuration as code? You can define notification destinations by creating a Kubernetes secret (or by setting values in your `values.yaml`) that the Mediator reads at runtime. This is the same delivery path used by UI-configured notifications, a notification created this way also appears as a configured notification in the UI. To configure mediator-based notifications, enable the executor and specify a webhook URL and token in the `values.yaml` file. Add the following under the notifications section of your [`causely-values.yaml`](/installation/customize#use-a-custom-values-file) file: ```shell executor: enabled: true notifications: webhook: url: "https://webhook.example.com/notify" # Replace with your webhook URL token: "your-secret-token" # Replace with your webhook token enabled: true ``` :::info For more details on webhook payload structure and field definitions, see the [Notification Payload Format](/workflows/notification-payload). ::: Make sure to replace the placeholder values with your actual webhook URL and token. Click on an integration below to view mediator-based setup instructions for receiving Causely notifications in your preferred tool. ### How to Choose - **Want engineers to quickly understand what changed and why?** → Native integrations - **Prefer to configure notifications with point-and-click?** → Managed notifications (UI-configured) - **Prefer configuration as code / GitOps?** → Mediator-based notifications (Kubernetes secret) Managed and mediator-based notifications deliver the same result, choose whichever matches how you like to work. ### Why This Matters By separating **causal understanding** from **delivery and action**, Causely ensures teams don’t just receive more alerts, they receive **the right insight, in the right place, with the right context**. This turns reliability from a reactive exercise into a system of understanding, coordination, and confident action. --- ## Prometheus Alertmanager Causely can send enriched causal insights directly to your existing Prometheus Alertmanager. This lets you keep using your current alerting pipeline while gaining deeper visibility into what's actually going wrong. By routing causal insights from Causely to Alertmanager, your team can receive high-signal notifications that include context like impacted services, severity, and SLO relevance—without needing to define new Prometheus alert rules. Prometheus Alertmanager can be used as a notification destination in Causely. How you configure this integration depends on how notifications are managed in your environment. Causely supports two configuration paths: - **Managed Notifications (recommended)** Routing rules are configured in the Causely UI based on impact, severity, and context. Prometheus Alertmanager is used purely as a delivery destination, allowing you to inject high-signal causal insights into your existing Alertmanager pipeline without duplicating Prometheus alert rules. - **Mediator-based Notifications (advanced)** Notifications are configured programmatically using the Causely Mediator and sent directly to your Prometheus Alertmanager webhook endpoint. Routing and notification rules for Managed Notifications are configured in the Causely UI: [Set up notification routing in the UI](/workflows/ui-notifications) ### Mediator-based Configuration If you would like to configure the Prometheus Alertmanager integration, you can do so by enabling the notifications in the `causely-values.yaml` file. For example, add the following in your `causely-values.yaml` file like this: ```yaml mediator: notifications: enabled: true ``` You can define a Grafana Alertmanager notification integration through the Causely UI, or by creating a secret in your Kubernetes cluster, in the causely namespace: ```yaml apiVersion: v1 kind: Secret type: Opaque metadata: name: prometheus-notification-config namespace: causely labels: causely.ai/notif-config: AlertManager data: notif_config_name: prometheus notif_config_global: false notif_config_token: "user:password" notif_config_type: AlertManager notif_config_url: https://alertmanager-prod-us-east-0.grafana.net/alertmanager/api/v2/alerts notif_config_filters: [] notif_config_filters_enabled: true ``` :::note Replace the placeholder values with your actual Alertmanager URL in the "notif_config_url" field and optional credentials in the "notif_config_token" field ::: To install the Causely agent with the Alertmanager integration enabled, run: ```bash causely agent install --cluster-name --values ./causely-values.yaml ``` --- ### Notification Payload Format Causely sends structured alert payloads to Alertmanager, including service-level impact and diagnostic summaries. For an example and explanation of each field, refer to the [Notification Payload Format](/workflows/notification-payload) page. --- ## Slack Notifications Slack can be used as a notification destination in Causely. All Slack delivery runs through the **Causely Mediator**. How you define Slack destinations depends on your setup: - **Managed Notifications (recommended)** Routing rules and Slack credentials are configured in the Causely UI. The UI stores configuration that the Mediator reads at runtime. - **Mediator-based configuration (advanced)** Define Slack destinations programmatically with environment variables or Kubernetes secrets instead of the UI. Uses the same Mediator delivery path as managed notifications. For routing rules and filters in the UI, see [Set up notification routing in the UI](/workflows/ui-notifications). ## Delivery modes Causely can deliver problem and defect notifications to Slack in two ways. Configure **one mode per notification destination**; you do not need both a webhook and a bot token for the same destination. | Mode | URL field | Token field | Delivery | |------|-----------|-------------|----------| | **Incoming webhook** | `https://hooks.slack.com/services/...` | Leave empty | One combined message per alert | | **Bot API (threaded)** | Channel ID (for example, `C01234567` or `G01234567` for private channels) | Bot token (`xoxb-...`) | Parent message + threaded replies | The bot must be a **member of the channel** (invite it with `/invite @YourBotName`). Use the channel ID (for example, `C01234567`) and not the channel name (for example, `#alerts`) in the URL field, as channel names are not supported in the configuration. The mediator selects the mode automatically from the URL and token values. --- ## Option A: Incoming webhook (simple) Best when you want a single message per alert and already have (or can create) an incoming webhook. ### 1. Create a Slack incoming webhook 1. Open [Slack API apps](https://api.slack.com/apps) or use **Slack → Tools → Workflow Builder / Apps** depending on your workspace policy. 2. Create or select an app, then enable **Incoming Webhooks**. 3. **Add New Webhook to Workspace** and pick the target channel. 4. Copy the webhook URL (`https://hooks.slack.com/services/T.../B.../...`). ### 2. Configure Causely 1. Go to **Settings → Notifications**. 2. If you haven't already selected a processing mode, choose one now. Click **Create**. 3. Complete the notification configuration: | Field | Value | |-------|--------| | **Name** | A descriptive name for the notification | | **Type** | Slack | | **URL** | The incoming webhook URL | | **Token** | Leave blank | --- Alternatively, configure the Mediator with environment variables or a Kubernetes secret instead of using the UI. ### Environment variables Set these on the Causely Mediator deployment. Replace `` with your notification config name. ```bash NOTIFICATION__TYPE=Slack NOTIFICATION__URL=https://hooks.slack.com/services/T.../B.../... NOTIFICATION__FILTERS_ENABLED=true ``` ### Kubernetes secret Create a secret labelled `causely.ai/notif-config: Slack` in the Mediator namespace: - `notif_config_type`: `Slack` - `notif_config_name`: your config name - `notif_config_url`: webhook URL - `notif_config_filters`: optional JSON filter rules ```yaml apiVersion: v1 kind: Secret type: Opaque metadata: name: slack-notification-config namespace: causely labels: causely.ai/notif-config: Slack stringData: notif_config_filters: "[]" notif_config_filters_enabled: "true" notif_config_global: "false" notif_config_name: slack notif_config_token: "" notif_config_type: Slack notif_config_url: https://hooks.slack.com/services/XXX/YYY/ZZZ ``` ### 3. Verify Post a test notification from the UI or trigger a real alert. You should see one Block Kit message with header, entity metadata, summary, evidence, remediation, SLOs, labels (when present), and a **View** button. --- ## Option B: Bot token + channel (threaded) Best when you want a compact parent message and details in a thread (summary, labels, evidence, remediation as separate replies). ### 1. Create a Slack app and bot token 1. Go to [https://api.slack.com/apps](https://api.slack.com/apps) → **Create New App** → **From scratch**. 2. Open **OAuth & Permissions** and add the bot token scope: - `chat:write`: post messages to channels the bot has joined 3. Go to **Install App** and click **Install to Workspace** (or **Reinstall to Workspace** if the app is already installed). 4. Copy the **Bot User OAuth Token** (`xoxb-...`) shown on that page. ### 2. Create or choose a channel 1. Create or pick a channel (for example, `#causely-alerts`). 2. **Invite the bot** to the channel (required): `/invite @YourBotName` 3. Copy the **channel ID** (required for the Causely URL field): - Open the channel → channel name → **About** → scroll down → copy the ID (`C...` for public, `G...` for private). ### 3. Configure Causely 1. Go to **Settings → Notifications**. 2. If you haven't already selected a processing mode, choose one now. Click **Create**. 3. Complete the notification configuration: | Field | Value | |-------|--------| | **Name** | A descriptive name for the notification | | **Type** | Slack | | **URL** | Channel ID (for example, `C0123456789`) | | **Token** | Bot User OAuth Token (`xoxb-...`) | The token field is optional for webhooks but **required** when the URL is not a `hooks.slack.com` webhook. --- Alternatively, configure the Mediator with environment variables or a Kubernetes secret instead of using the UI. ### Environment variables Set these on the Causely Mediator deployment. Replace `` with your notification config name. ```bash NOTIFICATION__TYPE=Slack NOTIFICATION__URL=C0123456789 NOTIFICATION__TOKEN=xoxb-your-bot-token NOTIFICATION__FILTERS_ENABLED=true ``` A `Bearer` prefix on the token is accepted (for example, `Bearer xoxb-...`). ### Kubernetes secret Create a secret labelled `causely.ai/notif-config: Slack` in the Mediator namespace. Set `notif_config_url` to the channel ID and `notif_config_token` to the bot token. ```yaml apiVersion: v1 kind: Secret type: Opaque metadata: name: slack-bot-notification-config namespace: causely labels: causely.ai/notif-config: Slack stringData: notif_config_filters: "[]" notif_config_filters_enabled: "true" notif_config_global: "false" notif_config_name: slack-threaded notif_config_token: xoxb-your-bot-token notif_config_type: Slack notif_config_url: C0123456789 ``` ### 4. Verify with curl ```bash curl -s https://slack.com/api/chat.postMessage \ -H "Authorization: Bearer xoxb-your-token" \ -H "Content-Type: application/json" \ -d '{"channel":"C0123456789","text":"Causely bot test"}' ``` Expect `"ok": true`. ### 5. Thread layout (bot mode) For detected problems, Causely posts: | Message | Content | |---------|---------| | **Parent** | Header, entity/severity metadata, **View** button | | **Thread 1** | Summary | | **Thread 2** | Labels | | **Thread 3** | Evidence and impacted SLOs | | **Thread 4** | Remediation | Cleared notifications are sent as a single message. Empty sections are skipped (for example, no labels thread if there are no labels). Failed thread replies are retried with backoff; remaining sections may be combined into a fallback thread message. --- ## Mediator-based configuration (advanced) Managed notifications in the UI are the recommended path. If you configure the Mediator directly instead, use the **Environment variables** or **Kubernetes secret** sections under [Option A](#option-a-incoming-webhook-simple) or [Option B](#option-b-bot-token--channel-threaded) above. --- ## Filters and object type Notification destinations support filters (severity, namespace, entity type, etc.) and object type (`defect` vs `issue`) like other Causely notification targets. Configure these in the same create/edit modal or via `notif_config_filters` / `NOTIFICATION__FILTERS`. --- ## Troubleshooting | Symptom | Likely cause | Fix | |---------|----------------|-----| | `not_in_channel` | Bot is not a channel member | `/invite @Bot` in the target channel before sending | | `channel_not_found` | Wrong channel ID or bot not in channel | Use channel ID (not `#name`); invite bot to private channels | | `invalid_auth` | Bad or revoked token | Reinstall app; copy fresh `xoxb-` token | | Webhook works but threading does not | URL still points at webhook | Use channel ID in URL, not `hooks.slack.com` | | Threaded mode not used | Missing or invalid token | Token must start with `xox` (after optional `Bearer` prefix) | | Notification marked failed but message in Slack | Partial thread delivery | Check mediator logs; retries/fallback may still leave parent visible | | UI has no token field | Older UI build | Upgrade UI or set token via env / K8s secret | --- ## Related documentation - [Managed Notifications](/workflows/ui-notifications) - Slack incoming webhooks: [https://docs.slack.dev/messaging/sending-messages-using-incoming-webhooks/](https://docs.slack.dev/messaging/sending-messages-using-incoming-webhooks/) - Slack `chat.postMessage`: [https://docs.slack.dev/reference/methods/chat.postMessage](https://docs.slack.dev/reference/methods/chat.postMessage) --- ### Notification Payload Format Causely sends structured alert payloads to Slack, including summaries of causal insights and service impact context. For an example payload and explanation of each field, refer to the [Notification Payload Format](/workflows/notification-payload) page. --- ## Splunk On-Call Causely can send high-context causal insights directly to Splunk On-Call (formerly VictorOps), so your on-call team receives the signal, not the noise. With enriched context like impacted services, problem severity, and diagnostic summaries, alerts become more actionable and reduce time-to-resolution. Splunk On-Call can be used as a notification destination in Causely. All Splunk On-Call delivery runs through the **Causely Mediator**. How you define Splunk On-Call destinations depends on your setup: - **Managed Notifications (recommended)** Routing rules and Splunk On-Call credentials are configured in the Causely UI. The UI stores configuration that the Mediator reads at runtime. - **Mediator-based configuration (advanced)** Define Splunk On-Call destinations programmatically with environment variables or Kubernetes secrets instead of the UI. Uses the same Mediator delivery path as managed notifications. For routing rules and filters in the UI, see [Set up notification routing in the UI](/workflows/ui-notifications). ## Mediator-based configuration (advanced) If you prefer not to use the UI, define Splunk On-Call destinations with a Kubernetes secret in the causely namespace: ```yaml apiVersion: v1 kind: Secret type: Opaque metadata: name: splunk-notification-config namespace: causely labels: causely.ai/notif-config: Splunk stringData: notif_config_filters: "[]" notif_config_filters_enabled: "true" notif_config_global: "false" notif_config_name: splunk notif_config_token: "causely" notif_config_type: Splunk notif_config_url: https://alert.victorops.com/integrations/generic/12356789/alert/ab12cd34-ef56-7890-ab12-34cd56ef7890 ``` :::note Replace the placeholder values with your actual integration URL and routing key from Splunk On-Call. ::: --- ### Notification Payload Format Causely sends structured alert payloads to Splunk On-Call, including rich metadata about the actual causes and their service-level impact. For an example payload and explanation of each field, refer to the [Notification Payload Format](/workflows/notification-payload) page. --- ## Managed Notifications Causely allows you to configure **managed notifications** directly in the UI, so teams can control what gets routed, where, and under which conditions, without managing configuration files or infrastructure. Managed notifications are the recommended approach for most teams. They let you route causal insights based on impact, severity, and context. ### Create a Notification Configuration 1. Navigate to Settings → Notifications. 2. Click Create to add a new notification configuration. 3. Choose global or per-cluster processing 4. Enter a name that describes the purpose of the notification (for example, prod-critical-alerts). The name is used only for identification within Causely and does not affect delivery. Each notification configuration must have a unique name. ### Choose Global or Per-Cluster Processing Select how notifications are processed across your environment. Causely supports two modes: - Global: One cluster handles notification processing for all clusters. Select the processing cluster once; all notification configurations share it. - Per-cluster: Each cluster processes its own notifications independently. You will select a processing cluster for each notification configuration. This eliminates the need to filter by cluster when scoping notifications. Notifications are processed by the Causely Mediator running in the selected cluster. ### Select a Notification Type Choose where you want to receive notifications. Supported destinations include: - Slack - Microsoft Teams - Prometheus Alertmanager - incident.io - Splunk On-Call (VictorOps) - Generic Webhook The selected type determines both delivery format and required fields in the next step. ### Configure Destination-Specific Fields Depending on the notification type, you may be asked to provide additional details, such as: - Endpoint URL (for webhooks or alert managers) - Authentication token (optional, depending on destination) These fields control how Causely delivers notifications to the selected system. For the **Generic Webhook** type, the token is sent verbatim as the HTTP `Authorization` header. See [Generic Webhook](/workflows/generic-webhook) for how to format it (for example, `Bearer `). ### For Per-Cluster Processing Choose a Processing Cluster for Select a cluster to handle notification processing. Notifications are processed by the Causely Mediator running in the selected cluster. ### Add Filters (Optional but Recommended) Filters allow you to control which causal insights trigger notifications. You can filter by: - Entity Type: for example Service, HTTP Path, or Node - Entity ID: the unique identifier of a specific entity - Entity Name: the display name of an entity. Supports regex matching (see [Match entity names with regex](#match-entity-names-with-regex) below) - Namespace - Diagnosis: for example Code Change Regression: CPU Congestion or Database Malfunction - Severity: Urgent vs. Non-urgent - Cluster: Not needed when using per-cluster processing, since each configuration is already scoped to a specific cluster. Filters help reduce noise and ensure notifications are routed only when they matter to the receiving team. #### Match entity names with regex The **Entity Name** filter supports pattern matching with two operations: - **REGEX**: notify when the entity name matches the pattern. - **NOT REGEX**: notify when the entity name does **not** match the pattern. Matching uses Go's standard regular expression engine (the [`regexp`](https://pkg.go.dev/regexp) package, which follows [RE2 syntax](https://github.com/google/re2/wiki/Syntax)). RE2 does not support backreferences or lookaround assertions. Patterns are unanchored by default, so a match anywhere in the name counts, use `^` and `$` to anchor to the start or end of the name. | Pattern | Matches | |---|---| | `checkout` | any entity name containing `checkout` | | `^checkout` | names that start with `checkout` | | `-canary$` | names that end with `-canary` | | `^checkout-service$` | exactly `checkout-service` | | `(checkout\|payment)` | names containing `checkout` or `payment` | | `(?i)prod` | `prod` matched case-insensitively (`Prod`, `PROD`, …) | For example, use the `REGEX` operation with `^prod-` to notify only on entities whose names start with `prod-`, or use `NOT REGEX` with `-canary$` to exclude every entity whose name ends in `-canary`. ### Save and Manage Notifications Once saved, the notification configuration appears in the Notifications list, where you can: - View active filters - Edit configuration details - Disable or delete notifications as needed Changes take effect immediately and do not require redeploying agents or updating configuration files. ### Test the Notification After saving, you can send a test notification to verify the configuration. - Optionally apply test-only filters to simulate specific scenarios. - Review the exact payload that will be sent to the destination. - Confirm the notification is received and formatted as expected. Testing helps validate both routing logic and downstream integrations. ### When to Use UI-Based Notifications Managed notifications are best suited for teams that want: - Centralized, product-managed routing - Explainable notification logic - Easy iteration as systems and ownership change For programmatic or infrastructure-managed setups, notifications can also be configured through the [Causely Mediator](/workflows/#mediator-based-notifications).