# Causely > Causely documentation for the causal AI engine that explains, predicts, and helps resolve software incidents. Start with the Causely GenAI primer for grounding and terminology: - [Causely GenAI Primer](https://docs.causely.ai/llms-primer.md): canonical framing, terminology, and interpretation guidance for Causely outputs. This file contains the complete Causely documentation corpus in one place for offline ingestion and evaluation. ## Quick Setup Guide for Causely export const CAUSELY_VERSION = versionHistory?.versions?.[0]?.imageVersion ?? versionHistory?.versions?.[0]?.version ?? ''; Get Causely running in your environment and connected to your agents in just a few minutes. This guide covers planning your telemetry setup, deploying the mediator, connecting data sources, verifying discovery, and giving AI agents access to Causely's causal model via MCP. After completing this guide, you are able to use Causely with your AI agents to identify root causes of issues in your environment, enabling faster problem resolution and improved system reliability. ## Plan Your Deployment Before installing Causely, it's important to understand what telemetry sources you'll need for a successful deployment. Causely requires two types of data sources to be most effective: - [Services](#services) - [Service Connections](#service-connections) :::warning Why those two are most important for your success **Service discovery** tells Causely what exist. **Service connections** shows how services interact with each other. **Both together** provide a complete picture, enabling accurate causal inference and impact assessment. Make sure you have at least one of these sources to be most effective, or use the out of the box instrumentation which is provided by eBPF-based OpenTelemetry instrumentation. ::: ### Services Service discovery helps Causely understand what services and infrastructure exist in your environment. These sources create entities like services, pods, containers, databases, and other infrastructure components. **You need at least one of these**: Kubernetes } description='Discovers clusters, nodes, pods, containers, services, and controllers' /> Docker } description='Discovers containers and allocations' /> Nomad } description='Discovers Nomad jobs, tasks, and services' /> AWS } description='Discovers EC2, ECS, Lambda, RDS, MSK, and ALB resources' /> Azure } description='Discovers Azure VMs, VMSS, disks, and storage' /> GCP } description='Discovers Compute Engine, Cloud Run, BigTable and Cloud SQL' /> For Kubernetes, Docker and Nomad, the service discovery is enabled during the installation of the Causely mediator on those platforms. For your cloud provider you can add the corresponding integration [via the UI](https://portal.causely.app/integrations). ### Service Connections Service connections reveal how services communicate with each other. These sources create connections between services, showing dependencies, request flows, and communication patterns. This is critical for building the service dependency graph and understanding how issues propagate. **You need at least one of these**: OpenTelemetry } description='Processes OTLP traces to discover service-to-service connections' /> Datadog } description="Queries Datadog's API to discover services and connections from APM traces" /> Instana } description="Queries Instana's API to discover services, connections, database access, HTTP paths, and RPC methods" /> Dynatrace } description="Queries Dynatrace's API to discover services and service connections" /> The default Causely installation includes [OpenTelemetry with eBPF](/telemetry-sources/ebpf) instrumentation automatically. ## Deploy the Causely Mediator Causely can be installed on any platform, from virtual machines and bare metal servers to container orchestration platforms like Kubernetes, Nomad, and Docker. Navigate to the [Mediators page](https://portal.causely.app/agents) in the Causely UI and click **"Add New"** to get your personalized installation command. Follow the instructions provided, which will look similar to: {`helm upgrade --install causely \\ --create-namespace oci://us-docker.pkg.dev/public-causely/public/causely \\ --version ${CAUSELY_VERSION} \\ --namespace=causely \\ --set image.tag=${CAUSELY_VERSION} \\ --set global.cluster_name=CHANGE_ME_CLUSTER_NAME \\ --set mediator.gateway.token=YOUR_ACCESS_TOKEN`} Replace `YOUR_ACCESS_TOKEN` with your access token and `CHANGE_ME_CLUSTER_NAME` with your cluster name. The Causely mediator is installed in the `causely` namespace. To verify the installation, run: ```bash kubectl get pods -n causely ``` By default, the applications in your cluster will be instrumented automatically using OpenTelemetry eBPF instrumentation. This means that Causely can use [OpenTelemetry](/telemetry-sources/opentelemetry) traces to discover service dependencies, monitor sync and async communication signals and detect root causes of issues in your environment. For detailed instructions on installing Causely on HashiCorp Nomad, see the [Nomad installation guide](/installation/nomad). The guide covers prerequisites, configuration, and deployment steps for running Causely on Nomad clusters. For detailed instructions on installing Causely on standalone Docker hosts, see the [Docker installation guide](/installation/docker). The guide covers prerequisites, configuration, and deployment steps for running Causely on Docker hosts outside of Kubernetes. For installation on virtual machines or bare metal servers, please [reach out to Causely support](mailto:support@causely.ai) for assistance with your specific environment and requirements. ## Connect Telemetry Sources With the mediator deployed, you can now connect your planned telemetry sources to Causely. As outlined above, begin with sources that provide you with discovery of services and service connections. To detect a wide range of [symptoms](/reference/symptoms/) and causes in your environment, you can configure additional data sources. Visit the [telemetry sources](/telemetry-sources/) page to learn more about the data sources that Causely supports. ## Review Discovery You have successfully installed Causely! Navigate to https://portal.causely.app to verify your environment has been discovered. You should see entities populated in the [Topology](/in-action/topology) view: If the entities and connections you are expecting to see are not appearing, go to the [Integrations](https://portal.causely.app/integrations) page and check for any errors for the telemetry sources you have configured. Errors in the integrations page will prevent Causely from discovering entities and connections in your environment. Follow the [troubleshooting steps](/telemetry-sources/troubleshooting) to resolve the errors. ## Connect Causely to AI Agents Give your agents access to Causely's causal model via the MCP server. Any MCP-compatible agent or assistant including Claude Code, Cursor, VS Code, HolmesGPT, or your own custom agent, can query root causes, service health, dependency maps, and reliability reports directly. Claude } description="Query root causes, service health, and dependency maps directly from Claude Code or Claude Desktop via the Causely MCP server." /> Cursor } description="Investigate incidents and check service health without leaving your editor using the Causely MCP integration." /> Codex } description="Connect OpenAI's coding agent to Causely's causal model via MCP for system-aware reliability workflows." /> HolmesGPT } description="Pre-built incident investigation agent with Causely MCP configured as its causal reasoning layer." /> Visit the [agent integration](/agent-integration) page to get started. --- ## How Causely Works Causely transforms raw telemetry into a live, queryable model of your system providing the semantic and causal foundation your agents need to diagnose, evaluate impact, and act safely in production. It works alongside your existing telemetry sources, interpreting the meaning behind the metrics, logs, and traces they provide, rather than replacing them. This page explains how Causely works: the **ontology-first approach** that turns raw telemetry into higher abstractions, then how the **causal reasoning engine** uses those abstractions to infer root cause and impact. For deployment architecture, components, and infrastructure, see [Architecture](/getting-started/architecture). ## How the Causal Engine Works At the core of Causely is a probabilistic reasoning engine that maps symptoms to root causes using domain-specific models and dynamic system knowledge. It maintains three core data structures—the **Topology Graph**, the **Causality Graph** (with its Codebook), and the **Attribute Dependency Graph**—and uses them for analysis. The engine is composed of six interdependent components, described below. {(() => { const { colorMode } = useColorMode(); const src = colorMode === 'dark' ? '/img/how_causely_works_diagramLight1.svg' : '/img/how_causely_works_diagramDark1.svg'; return ; })()} ### 1. From telemetry to semantic understanding Instead of feeding raw telemetry straight into analysis, Causely uses an **ontology-first approach**: **mediation** distills telemetry (logs, metrics, traces) and other data locally into a structured layer of: - **[Entities](/reference/entity-types)**: services, pods, databases, queues - **Relations**: how they connect - **[Symptoms](/reference/symptoms/)**: observable states such as "high latency here" or "errors increasing there"; detected when thresholds or patterns are met and used for reasoning even when no alert is fired That distillation turns the vast volumes of data teams commonly ingest for observability into higher-quality abstractions. This distilled data is sent to the backend, where the causal reasoning engine infers the [causes](/reference/root-causes/) explaining those symptoms. :::tip Local processing and privacy All distillation runs in your environment. The **mediation layer** turns telemetry into **symptom states** and **topology**; only this **distilled data** is sent to the causal reasoning engine. Raw logs, full traces, and bulk metrics stay in your datacenter—only semantic state and, when needed, minimal evidence leave your environment. This design keeps sensitive and high-volume telemetry local and preserves **privacy**. ::: ### 2. Ontology: causal and attribute models The engine’s **ontology** is the formal model of entities, behaviors, and failure modes used for inference. It is composed of two foundational models: the **Causal Model** and the **Attribute Dependency Model**. Together, they define what root causes and symptoms exist and how they relate, forming the semantic backbone of the causal reasoning engine. - **Causal Model**: Causely includes a built-in library of causal knowledge that captures root causes capable of degrading application performance. This model requires no configuration and enables the system to begin identifying root causes as soon as it is deployed. - Covers a broad range of entities, including applications, databases, caches, messaging systems, load balancers, DNS, compute, and storage. - Encodes how each root cause propagates through the environment and the symptoms it may produce. - Is designed to be environment-agnostic and applicable to any modern cloud native architecture. - **Attribute Dependency Model**: The attribute model extends the causal model by capturing how performance-related attributes (for example, latency, throughput, utilization) are interdependent across entities. It also encodes the operational constraints those attributes must satisfy to meet performance goals. - Represents attribute dependencies across a wide range of services and infrastructure layers. - Supports both predefined and learned functional relationships between attributes. - Defines the desired state of the system based on application goals and constraints. - Like the causal model, it is fully environment-independent and generalizable across architectures. ### 3. Topology Graph Causely continuously discovers and maintains the topology graph of services, infrastructure, and their interconnections via its integrations with [your existing telemetry sources](/telemetry-sources/). It ingests and reconciles topology from any source, including OpenTelemetry traces, cloud provider APIs and other inventories. Cloud native environments are highly dynamic, composed of applications, services, databases, caches, messaging systems, load balancers, compute, storage, and more. Through continuous discovery and ingestion from these sources, Causely determines for each entity: - **Connectivity**: which other entities it communicates with horizontally - **Layering**: which entities it is built upon or supports vertically - **Composition**: the internal components or resources that make up the entity These relationships are stitched together into a continuously updated, real-time graph that represents the full system [topology](/reference/terminology/#topology), regardless of whether data comes from OpenTelemetry, cloud APIs or other integrated sources. This graph forms the foundation for blast radius analysis, root cause attribution, and cross-service impact modeling. ### 4. Causality Mapping Causely automatically generates a [Bayesian network](/reference/terminology/#bayesian-network) that models how root causes lead to observable symptoms, based on its built-in Causal Models and the real-time Topology Graph. This causal mapping encodes probabilistic cause-effect relationships, allowing the system to infer the most likely root cause from a given set of active symptoms. It reflects both the structural dependencies in your environment and learned patterns of failure propagation. Causely represents this mapping through two core data structures: - **[Causality Graph](/reference/terminology/#causality-graph-cg)**: A directed acyclic graph (DAG) where nodes represent root causes and symptoms, and edges denote potential causal relationships. Each edge is weighted with a probability, indicating the likelihood that one node (a root cause) leads to another (a symptom). - **[Codebook](/reference/terminology/#codebook)**: A table where each column corresponds to a root cause and each row to a symptom. Each column is a vector of probabilities defining a unique signature of the root cause. A cell in the vector represents the probability that the root cause may cause the symptom. Together, these structures power Causely’s ability to perform real-time probabilistic inference and deliver explainable, high-confidence root cause insights. ### 5. Attribute Dependency Graph Causely generates this graph using its built-in Attribute Dependency Model and the live Topology Graph. The result is a directed acyclic graph (DAG) that models functional dependencies between system attributes. In this graph: - Nodes represent individual attributes, for example CPU usage of a service or queue length of a messaging system. - Edges represent dependency relationships, for example an edge from attribute A to attribute B means that B is a function of A. - Edge labels define these functions. Some may be explicitly defined in the Attribute Dependency Model, while others are learned automatically from observed behavior in your environment. - Nodes representing attributes that must satisfy a constraint are decorated with the constraint the attribute must satisfy. The Attribute Dependency Graph enables Causely to reason about how changes in one part of the system cascade across others, identify emerging bottlenecks, and validate whether the environment remains within defined performance bounds. ### 6. Analysis The analysis automatically pinpoints causes in real time based on observed symptoms, using the Codebook described above. No configuration is required, Causely can immediately identify a broad set of issues (100+ causes mapped to symptoms) ranging from application malfunctions to service congestion to infrastructure bottlenecks. In any given environment, there can be tens of thousands of different causes that may cause hundreds of thousands of symptoms. Causely prevents service degradation by detangling this mess and pinpointing the cause putting your SLOs at risk and driving remediation actions before SLOs are violated. With the ontology and causal layers in place [agents connected to Causely](/agent-integration/) can query diagnosis, blast radius, and ownership directly, answering production questions in seconds with a small fraction of the tokens querying raw telemetry would consume. ## Deployment architecture For deployment architecture, component structure, and workflow integration, see [Architecture](/getting-started/architecture). --- ## Agent Integration ## Building Reliable Agents with Causely Agents fail not because they lack data. They fail because data alone does not explain causality. An agent with access to metrics, logs, and traces still cannot reliably determine what caused an issue, how far it has spread, or what action is safe to take. That requires a causal model: a structured understanding of how services, dependencies, and failure patterns relate. Causely provides that model. Agents query Causely through the MCP server and receive structured, deterministic answers, including root causes, blast radius, dependency maps, and remediation guidance, instead of raw signals to interpret. ## The Gap in Today's Agent Architectures Most agent-driven systems run into three core limitations: **Information gap** Agents can retrieve telemetry, but cannot consistently determine what is happening or what matters. **System gap** There is no shared understanding of how services, infrastructure, and dependencies relate to each other. **Execution gap** Agents lack a reliable way to determine which actions are safe and how to coordinate them. As a result, agents require human interpretation, and automation breaks down at scale. :::tip Benchmark: agents with and without Causely Agents using Causely cut token consumption 48%, ran 63% faster, and hit 100% diagnosis accuracy across 72 benchmark experiments. [See the benchmark →](https://causely.ai/product/benchmark) ::: ## Where Causely Fits Causely provides a system intelligence layer that continuously models how your system behaves: its services, dependencies, and failure propagation. Instead of reasoning over raw telemetry, agents interact with structured, deterministic system knowledge. Decisions are based on how the system actually behaves, not on correlation or heuristics. ## Architecture Overview ``` [Agent (for example Holmes or custom agent)] ↓ [Causely (causal model + reasoning engine)] ↓ [Observability + Infrastructure (metrics, traces, logs, alerts)] ``` - **Agent**: orchestrates workflows, queries systems, and takes action - **Causely**: builds and maintains a causal model and provides deterministic reasoning - **Observability + Infrastructure**: provides raw signals and telemetry ## What Your Agent Can Do The Causely MCP server exposes 30 tools across 5 categories. Here is what each category enables: - **Entity Resolution**: Resolve service and database names to IDs, enumerate namespaces and clusters, check current health status. Most workflows start here. - **Data Retrieval**: Retrieve time-series metrics, live logs, alert history, deployment events, configuration files, and slow query analysis for any entity. - **Health & Diagnosis**: Get active symptoms environment-wide, identify root causes with impacted services and remediation guidance, check SLOs, map service topology, and get structured health summaries for services, teams, or individual entities. - **Reporting & Postmortems**: Generate deterministic postmortem drafts and structured engineering tickets from resolved incident data. - **Reliability & Deployment**: Compare resource consumption before and after deployments for a single service or an entire fleet. ## Integration Paths Choose based on how much you want to build. | Option | Best for | What you get | |---|---|---| | [MCP Server](/agent-integration/mcp-server) | Any MCP-compatible agent or assistant | Standardized interface to all 29 Causely tools; works with Cursor, Claude Code, VS Code, and others | | [HolmesGPT](/agent-integration/holmes-gpt) | Teams already using Holmes | Pre-built agent with Causely MCP configured; no custom integration required | | [Custom Agents](/agent-integration/custom-agents) | Teams building internal tooling or automation pipelines | Full control over logic, policies, and execution; MCP or direct API | If you are starting fresh, use the MCP Server. It works with any agent that supports the Model Context Protocol and requires no custom code. ## Example Workflow **Scenario: High error rate alert** 1. The agent receives an alert 2. The agent calls `get_entities()` to resolve the alerted service name to an entity ID 3. The agent calls `get_root_causes()` to identify the source 4. Causely returns: - Root cause service - Affected dependencies - Explanation of why this is the cause - Remediation guidance 5. The agent: - Notifies the correct team - Suggests or executes remediation ## When This Approach Is Most Valuable This architecture is most effective when: - You operate distributed systems with many interdependent services - You already have observability in place - You are building or evaluating automated incident workflows ## Summary Causely does not replace your agents or your observability stack. It provides the system intelligence layer required for agents to interpret telemetry consistently, identify true root causes, and take safe, coordinated action. --- ## API Reference Guide The Causely API enables developers to integrate root cause analysis capabilities into their applications and workflows. This comprehensive guide provides step-by-step examples for authentication, querying defects, and automating incident response using the Causely platform. ## Getting Started with Causely API ### Create API Client Credentials Before you can use the Causely API, you need to generate API client credentials (Client ID and Secret) from the Causely platform: 1. **Login to Causely:** - Go to [Causely Portal](https://portal.causely.app/) - Login with your credentials 2. **Navigate to Personal Tokens:** - Click on the Profile Icon in the top-right corner - Select "Admin Portal" - Choose "Personal Tokens" 3. **Generate Token:** - Click "Generate Token" - Provide a description and expiration date - Click "Create" 4. **Save Credentials:** - Record your Client ID and Secret Key in a safe place - You'll use these credentials in the examples below ## Next Steps Once you have your API credentials, you can proceed with: - **[Authentication](/api/authentication)** - Learn how to authenticate with the Causely GraphQL API - **[Gateway Tokens](/api/gateway-tokens)** - Create and manage gateway tokens for mediator deployment - **[GraphQL Clients](/api/graphql-clients)** - Set up reusable GraphQL client libraries - **[Query Examples](/api/queries/getting-started)** - Explore example queries and mutations ## API Support For technical questions about the Causely GraphQL API, [contact our support team](mailto:support@causely.io) for assistance with integration challenges. --- ## SLO Targets and Burn Rates Service Level Objectives (SLOs) are how Causely translates reliability signals into **urgency and action**. SLOs define what “good” looks like for reliability across services, endpoints, and queues, and are used by Causely to determine when degradations represent acceptable risk versus issues that require immediate attention. In Causely, SLOs directly influence how root causes are classified and prioritized. When a root cause puts an SLO at risk or violates it, Causely treats that root cause as more urgent, helping teams focus on the issues most likely to impact users and the business. SLOs are applied **by default at the service level**, providing broad coverage with minimal configuration. For teams that need more granular protection, SLOs can also be defined for specific **HTTP paths**, **RPC methods**, and **queues**, allowing critical user flows or business transactions to be protected independently of overall service health. Queue SLOs are configured via the API. ## Default SLO Behavior By default, Causely applies the following SLO targets and burn rate settings to all services: - **Error rate SLO target**: 99.0% (99% of requests must be successful) - **Latency SLO target**: 95.0% (95% of requests must be under the latency threshold) - **Availability SLO target**: 99.0% (99% uptime expected) - **Burn rate threshold**: 4 (budget would be consumed in 6 hours for a 1-day SLO) - **Burn rate window**: 15 minutes (calculation window for burn rate monitoring) These defaults are designed to catch fast-burning reliability issues early, while avoiding unnecessary noise for brief or low-impact fluctuations. ## When SLOs Are Active SLOs in Causely are evaluated only when **traffic is observed** for the corresponding entity. This applies consistently to services, HTTP paths, RPC methods, and queues. SLOs measure how an entity performs when responding to real requests. If no traffic is observed, there is no performance to evaluate, and the SLO remains inactive until requests are seen. ## Customizing SLO Behavior You can customize how SLOs behave in Causely depending on the level of control you need: - **Service-level SLOs** can be customized using labels or service metadata. This is the most common approach and is described in the sections below. - **HTTP Path and RPC Method SLOs** are configured exclusively through the API. These SLOs follow the same core concepts (targets, burn rates, and windows) but apply to specific endpoints rather than entire services. See [Setting SLOs on Paths and Methods](/api/queries/setting-slos-on-paths-and-methods/) for details. - **Queue SLOs** are configured exclusively through the API. These SLOs follow the same core concepts (targets, burn rates, and windows) but apply to specific queues, such as order-processing or event ingestion queues. When a queue SLO is at risk or violated, root causes affecting that queue or its upstream dependencies are automatically elevated as urgent. - **Default SLO values** can also be adjusted programmatically through the API. Documentation and examples for API-based default configuration will be added in a future update. ## Supported Labels You can configure the following SLO-related labels: | Label | Description | Default | | --------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | `causely.ai/error-rate-slo-target` | Percentage of successful requests expected, for example 99.0. This defines the percentage of requests that must not result in an error to remain within the error SLO. | `99.0` | | `causely.ai/latency-slo-target` | Percentage of requests expected to be under the latency threshold, for example 95.0. Note that the latency threshold is automatically learned by Causely, but can be manually adjusted via the [Thresholds configuration page](/configuration/thresholds/). | `95.0` | | `causely.ai/availability-slo-target` | Percentage of time the service is expected to be operational, for example 99.0. This defines the proportion of total time the service must remain available, responding successfully to requests without downtime, to remain within the availability SLO. | `99.0` | | `causely.ai/error-rate-burn-rate-threshold` | Rate of error budget burn relative to the SLO target. A default of 4 means that for a 1-day SLO, if errors continue at the current rate, the error budget would be consumed in 6 hours. | `4` | | `causely.ai/latency-burn-rate-threshold` | Rate of latency budget burn relative to the SLO target. A default of 4 means that for a 1-day SLO, latency at the current rate would consume the entire budget in 6 hours. | `4` | | `causely.ai/availability-burn-rate-threshold` | Rate of availability budget burn relative to the SLO target. A default of 4 means that for a 1-day SLO, availability at the current rate would consume the entire budget in 6 hours. | `4` | | `causely.ai/error-rate-burn-rate-window` | Burn rate calculation window (in minutes) used to indicate whether a service is rapidly consuming its error SLO budget. | `15` | | `causely.ai/latency-burn-rate-window` | Burn rate calculation window (in minutes) used to indicate whether a service is rapidly consuming its latency SLO budget. | `15` | | `causely.ai/availability-burn-rate-window` | Burn rate calculation window (in minutes) used to indicate whether a service is rapidly consuming its availability SLO budget. | `15` | ## Configuration Methods ### Using Kubernetes Labels You can apply labels directly to your Kubernetes services: ```bash # Set error rate SLO target to 99% kubectl label svc -n "causely.ai/error-rate-slo-target=99.0" # Set latency SLO target to 95% kubectl label svc -n "causely.ai/latency-slo-target=95.0" # Set availability SLO target to 99% kubectl label svc -n "causely.ai/availability-slo-target=99.0" # Set error rate burn rate threshold to 2 kubectl label svc -n "causely.ai/error-rate-burn-rate-threshold=2" # Set latency burn rate threshold to 2 kubectl label svc -n "causely.ai/latency-burn-rate-threshold=2" # Set availability burn rate threshold to 2 kubectl label svc -n "causely.ai/availability-burn-rate-threshold=2" # Set error rate burn rate window to 15 minutes kubectl label svc -n "causely.ai/error-rate-burn-rate-window=15" # Set latency burn rate window to 15 minutes kubectl label svc -n "causely.ai/latency-burn-rate-window=15" # Set availability burn rate window to 15 minutes kubectl label svc -n "causely.ai/availability-burn-rate-window=15" ``` ### Using Nomad Service Tags If you use Nomad, you can specify these as service tags: ```hcl job "example" { group "app" { service { name = "my-service" port = 8080 tags = [ "causely.ai/error-rate-slo-target=99.0", "causely.ai/latency-slo-target=95.0", "causely.ai/availability-slo-target=99.0", "causely.ai/error-rate-burn-rate-threshold=2", "causely.ai/latency-burn-rate-threshold=2", "causely.ai/availability-burn-rate-threshold=2", "causely.ai/error-rate-burn-rate-window=15", "causely.ai/latency-burn-rate-window=15", "causely.ai/availability-burn-rate-window=15" ] } } } ``` ### Using Consul Service Metadata For Consul services, you can configure these using service metadata: ```bash # Register a service with slo metadata consul services register \ -name="my-service" \ -port=8080 \ -meta="causely.ai/error-rate-slo-target=99.0" \ -meta="causely.ai/latency-slo-target=95.0" \ -meta="causely.ai/availability-slo-target=99.0" \ -meta="causely.ai/error-rate-burn-rate-threshold=2" \ -meta="causely.ai/latency-burn-rate-threshold=2" \ -meta="causely.ai/availability-burn-rate-threshold=2" \ -meta="causely.ai/error-rate-burn-rate-window=15" \ -meta="causely.ai/latency-burn-rate-window=15" \ -meta="causely.ai/availability-burn-rate-window=15" # Update existing service metadata consul services register \ -id="my-service-id" \ -name="my-service" \ -port=8080 \ -meta="causely.ai/error-rate-slo-target=99.0" \ -meta="causely.ai/latency-slo-target=95.0" \ -meta="causely.ai/availability-slo-target=99.0" \ -meta="causely.ai/error-rate-burn-rate-threshold=2" \ -meta="causely.ai/latency-burn-rate-threshold=2" \ -meta="causely.ai/availability-burn-rate-threshold=2" \ -meta="causely.ai/error-rate-burn-rate-window=15" \ -meta="causely.ai/latency-burn-rate-window=15" \ -meta="causely.ai/availability-burn-rate-window=15" ``` ## Best Practices 1. **Align with SLO policy**: Reflect organizational reliability goals. 2. **Avoid overly aggressive thresholds**: High sensitivity may create alert fatigue. 3. **Monitor and adjust**: Tune thresholds based on incident reviews and error budget consumption. 4. **Document changes**: Record rationale for each SLO configuration. ## Example Use Cases 1. **Business-critical services**: Set tighter SLO targets, for example 99.9% success, 98% low-latency. 2. **Temporary adjustments**: Raise burn rate thresholds during high-traffic events. ## Queue SLO Example Queues often represent critical business workflows, such as order processing or asynchronous event handling. For example, if you need to ensure that an order-processing queue is drained within a defined time window, you can define an SLO using **Queue Depth** (or a related metric) as the SLI via the API. When that queue’s SLO is at risk or violated, Causely elevates related root causes as urgent and surfaces the most relevant causal explanations, helping teams prioritize issues that directly impact key workflows. ## Burn Rate Threshold Examples The burn rate threshold determines how aggressively your error or latency budget is being consumed, and helps you catch fast-burning issues. Here's a simple example: Suppose your service has a 1-day SLO budget, meaning it can tolerate a limited amount of errors or latency over 24 hours. - **Burn Rate Threshold = 1** The current rate of errors or latency is steady and would use up the entire budget in exactly 24 hours. No alarm yet, but you're tracking close to your SLO target. - **Burn Rate Threshold = 2** At the current rate, the service would consume its full 24-hour budget in only **12 hours**, triggering alerts about rapid budget consumption. - **Burn Rate Threshold = 4** This indicates extremely fast-burning behavior. At this pace, the full error or latency budget would be used up in just **6 hours**. In practice, burn rate thresholds allow teams to catch reliability problems earlier, before they fully consume the SLO budget. ## Burn Rate Window Examples The burn rate window helps determine how quickly your service is consuming its SLO budget by observing behavior over short time intervals. Below are simple examples to clarify: - **Short Window (5 minutes)** Useful for detecting rapid error or latency spikes. For example, if a service suddenly begins failing or slowing down at a high rate, a short burn rate window (like 5 minutes) helps you identify that it's quickly consuming its SLO budget, enabling earlier incident detection. For services where fast detection of degraded performance is critical, consider also shortening the symptom activation delay, which you can manually configure via the [Symptom Delay](/configuration/symptom-delay) settings. - **Moderate Window (15 minutes)** This is the default and provides a good balance between reactivity and noise. It captures bursts of errors or latency that might not last long enough to trigger alerts in a longer window but are still significant. - **Long Window (60 minutes)** Best used to detect sustained SLO violations. For example, if a service has a consistent error rate that slowly drains the budget, the longer window provides better confidence that it’s not just a transient blip. --- ## Root Causes The Root Cause view presents the outcomes of the causal reasoning engine so you see what explains the [symptoms](/reference/symptoms/) and problems surfaced in your environment. Instead of receiving an alert per symptom, you get a single cause that represents the underlying issue, dramatically reducing noise and accelerating response. For background on how causes are inferred, see [How Causely Works](/getting-started/how-causely-works). ## Root Causes view The Root Causes view shows the list of inferred causes, that explain the symptoms observed in your environment. The list is filtered by urgency and state, and can be further filtered by scope, service, and workload. ### Urgency **Urgent causes** are the ones that are currently impacting services and SLOs. They are marked red. **Non-urgent causes** are the ones that are not currently impacting services and SLOs but indicate elevated risk. Both types are valuable: **urgent items** drive incident response; non‑urgent items drive prevention and continuous improvement. ### States - **Active**: The cause is currently inferred from the observable symptoms and conditions. - **Historical**: The cause was active in the past and has since cleared. Useful for learning and trend analysis. - **Hidden**: The cause exists but is suppressed from the default list (for example via filters or muted scopes). You can reveal hidden items when needed. ### Filter the list Use filters to focus on what matters most: - **Scope**: Limit causes to a specific scope (for example an environment or domain). - **Service**: Focus on a particular service. - **Workload**: Narrow down to a deployment, job, or workload of interest. We will add a screenshot of the filtering controls here. ## Inspect a single cause Opening a cause shows a few with two tabs: - **Summary**: A clear explanation including description, current and potential impact, blast radius, recommended remediation steps, and the evidence supporting the inference. - **Causality graph**: A visual explanation of how the system arrived at the cause, showing the relationships between the observed symptoms and the inferred cause. ### Description A high level description of the cause is shown in the summary tab. ### Impact and Blast Radius A list of services degraded by the cause is shown in the summary tab, and a "blast radius" diagram is shown to help visualize the scope of the impact. ### Remediation A list of recommended remediations to fix the cause is shown in the summary tab. ### Evidence A list of evidence supporting the inference is shown in the summary tab. This includes observed [symptoms](/reference/symptoms/), exceptions, logs, and events. --- ## Troubleshooting Telemetry Sources If entities and connections are not appearing in your Causely topology, check the [Integrations page](https://portal.causely.app/integrations) for errors. This guide helps you diagnose and resolve common issues with telemetry source integrations. ## Identifying Issues ### Check the Integrations Page 1. Navigate to the [Integrations page](https://portal.causely.app/integrations) in the Causely UI 2. Look for any integrations showing error states (red indicators or error messages) 3. Click on an integration to view detailed error information Common error indicators include: - **Initialization errors**: The scraper failed to start - **Authentication errors**: Invalid credentials or missing permissions - **Connection errors**: Unable to reach the telemetry source - **Configuration errors**: Missing or incorrect configuration values ## Common Issues and Solutions ### Authentication and Credentials #### Invalid or Expired Credentials **Symptoms:** - Error messages mentioning "authentication failed", "unauthorized", or "invalid credentials" - Integration shows as "Initialization Failed" **Solutions:** 1. **Verify credentials are correct:** - Check that API keys, tokens, or passwords are not expired - Ensure credentials match the correct account or environment - Verify there are no extra spaces or special characters when copying credentials 2. **For Kubernetes secrets:** ```bash # Verify the secret exists and contains the correct data kubectl get secret -n causely -o yaml # Check that the secret keys match what Causely expects # (e.g., apiKey, appKey, token, etc.) ``` 3. **For cloud provider integrations (AWS, Azure, GCP):** - Verify IAM roles or service accounts have the required permissions - Check that access keys haven't been rotated - Ensure service account annotations are correctly set for IRSA (AWS) or Workload Identity (GCP) #### Missing Secret Labels (Autodiscovery) **Symptoms:** - Integration not appearing in the Integrations page - Scraper not starting despite secret existing **Solutions:** 1. **Verify the secret has the correct label:** ```bash # Check if the secret has the scraper label kubectl get secret -n causely --show-labels # Add the label if missing (example for PostgreSQL) kubectl --namespace causely label secret "causely.ai/scraper=Postgresql" ``` 2. **Common scraper label values:** - `Postgresql` for PostgreSQL - `MySQL` for MySQL - `Datadog` for Datadog - `AWS` for AWS - `Azure` for Azure - `GCP` for GCP - `Instana` for Instana - `Dynatrace` for Dynatrace - `Confluent` for Confluent - `Snowflake` for Snowflake ### Network Connectivity #### Cannot Reach Telemetry Source **Symptoms:** - Connection timeout errors - "Unable to connect" or "network unreachable" messages - Integration shows intermittent failures **Solutions:** 1. **Verify network connectivity from the mediator pod:** ```bash # Get the mediator pod name kubectl get pods -n causely -l app=mediator # Test connectivity to the telemetry source kubectl exec -n causely -- curl -v ``` 2. **Check firewall rules:** - Ensure the mediator can reach external APIs (for cloud providers) - Verify internal network policies allow communication - Check if the telemetry source requires allowlisting the mediator's IP 3. **For Prometheus/Istio integrations:** - Verify the Prometheus endpoint is accessible from the mediator - Check that the endpoint URL is correct (including protocol: `http://` or `https://`) - Ensure Prometheus is not behind authentication that requires additional configuration ### Configuration Errors #### Missing Required Configuration **Symptoms:** - "Configuration error" or "missing required field" messages - Integration fails to initialize **Solutions:** 1. **Verify all required fields are set:** - Check the specific telemetry source documentation for required configuration - Ensure secrets contain all required keys - Verify values.yaml or configuration files have all necessary settings 2. **Common missing fields:** - **Database integrations**: Missing `host`, `port`, `database`, or `username` - **Cloud providers**: Missing `region` or `account ID` - **APM tools**: Missing `apiKey`, `appKey`, or `org` (for Datadog) - **Prometheus**: Missing `endpoint` URL 3. **Check configuration format:** ```bash # Validate Helm values helm template causely oci://us-docker.pkg.dev/public-causely/public/causely --values values.yaml --dry-run # Check mediator logs for configuration errors kubectl logs -n causely -l app=mediator --tail=100 ``` #### Incorrect Endpoint URLs **Symptoms:** - Connection refused errors - 404 Not Found errors - SSL/TLS certificate errors **Solutions:** 1. **Verify endpoint URLs:** - Ensure URLs include the correct protocol (`http://` or `https://`) - Check that ports are correct (for example, `:4317` for OTLP, `:9090` for Prometheus) - Verify hostnames resolve correctly 2. **For HTTPS endpoints:** - Check if self-signed certificates require additional configuration - Verify certificate validity if using custom certificates ### Permission Issues #### Insufficient Permissions **Symptoms:** - "Access denied" or "Forbidden" errors - Integration can connect but cannot read data - Partial data discovery (some entities missing) **Solutions:** 1. **For cloud provider integrations:** **AWS:** - Verify IAM role or user has required read permissions - Check policies include: `ReadOnlyAccess` or specific service read permissions - For EKS, ensure IRSA (IAM Roles for Service Accounts) is configured correctly **Azure:** - Verify service principal has "Reader" role or equivalent - Check that the subscription is accessible - Ensure Managed Identity is configured if using that method **GCP:** - Verify service account has required IAM roles - Check Workload Identity is configured for GKE - Ensure project-level permissions are correct 2. **For database integrations:** - Verify database user has `SELECT` permissions on system tables - Check that the user can access `pg_stat_statements` (PostgreSQL) or equivalent - Ensure the user has permissions to query metadata tables 3. **For Kubernetes:** - Verify the mediator service account has necessary RBAC permissions - Check ClusterRole and ClusterRoleBinding are correctly configured - Ensure the mediator can list and get resources in target namespaces ### Version and Compatibility Issues #### Incompatible Versions **Symptoms:** - API errors mentioning version incompatibility - Unexpected response formats - Features not working as expected **Solutions:** 1. **Check Causely version compatibility:** - Review the telemetry source documentation for version requirements - Ensure you're using a supported version of Causely - Check changelog for breaking changes 2. **Verify telemetry source versions:** - Some integrations require minimum versions of the source system - Check if API versions have changed - Review telemetry source release notes ## Platform-Specific Troubleshooting ### Kubernetes **Common issues:** - Service account permissions - Network policies blocking communication - Resource quotas limiting scraper execution **Debug steps:** ```bash # Check mediator pod status kubectl get pods -n causely # View mediator logs kubectl logs -n causely -l app=mediator --tail=200 # Check service account kubectl get serviceaccount -n causely # Verify RBAC permissions kubectl get clusterrolebinding | grep causely ``` ### Docker **Common issues:** - Container networking issues - Volume mount permissions - Environment variable configuration **Debug steps:** ```bash # Check container status docker ps | grep causely # View container logs docker logs # Verify environment variables docker exec env | grep CAUSELY ``` ### Nomad **Common issues:** - Job allocation failures - Network configuration - Volume access issues **Debug steps:** ```bash # Check job status nomad job status causely # View allocation logs nomad alloc logs # Check job specification nomad job inspect causely ``` ## Metrics and Trace Volume ### Metrics appear missing or don't match external monitoring tools **Symptoms:** - Causely metrics graphs show gaps or periods with no data - Metric values in Causely differ from what you see in Prometheus, Datadog, or other external tools **Solutions:** Causely is designed to work from a **statistically significant sample** of telemetry, not 100% of raw data. Minor differences between Causely metrics and external tools are expected and do not affect analysis accuracy. If you observe **persistent gaps** (extended periods with no data at all), VictoriaMetrics is likely under-resourced for your environment: 1. **Check VictoriaMetrics memory usage:** ```bash kubectl top pod -n causely -l app=victoriametrics ``` 2. **Increase VictoriaMetrics memory allocation** by following the [VictoriaMetrics sizing guide](/installation/customize#sizing-victoriametrics-for-high-volume-environments). ### Traces appear to be dropped **Symptoms:** - Trace counts in Causely are lower than what your instrumentation or external tracing backend reports - You notice the Mediator pod's memory usage is near its limit **Explanation:** The Mediator's built-in OpenTelemetry Collector has a memory limiter that drops excess traces when memory pressure is reached. This is **intentional behavior**—it means the Mediator has already collected a sufficient sample of traces for analysis. Causely does not require every trace to identify root causes. Mediator memory is sized by the **number of entities under management** (services, pods, databases, and other components in your environment), not by trace or request throughput. If your cluster is growing, you may need to increase Mediator memory—see [Sizing the Mediator for large environments](/installation/customize#sizing-the-mediator-for-large-environments). If you need full-fidelity trace retention for other purposes (for example, debugging individual requests), you can route traces through an intermediate collector that fans out to both Causely and your tracing backend (for example, Grafana Tempo). If you are using Causely's built-in Beyla instrumentation, redirect Beyla's trace export to your OpenTelemetry Collector or Grafana Alloy instance by setting the following in your `causely-values.yaml`: ```yaml agent: beyla: otel_traces_export: endpoint: http://:4317 ``` Then configure your collector to export traces to **both** your tracing backend and the Causely Mediator (`mediator.causely:4317`). See [Integrating OpenTelemetry with Causely](/telemetry-sources/opentelemetry#integrating-opentelemetry-with-causely) and [Grafana integration](/telemetry-sources/grafana) for collector configuration details. ## Getting Additional Help If you've tried the solutions above and are still experiencing issues: 1. **Check mediator logs** for detailed error messages: ```bash kubectl logs -n causely -l app=mediator --tail=500 ``` 2. **Review integration-specific documentation:** - [AWS Integration](/telemetry-sources/aws) - [Azure Integration](/telemetry-sources/azure) - [GCP Integration](/telemetry-sources/gcp) - [PostgreSQL Integration](/telemetry-sources/postgresql) - [MySQL Integration](/telemetry-sources/mysql) - [Datadog Integration](/telemetry-sources/datadog) - [Instana Integration](/telemetry-sources/instana) - [Dynatrace Integration](/telemetry-sources/dynatrace) 3. **Contact Causely Support:** - Email: [support@causely.ai](mailto:support@causely.ai) - Include: - Integration name and error message - Relevant log excerpts (redact sensitive information) - Configuration details (redact credentials) - Steps you've already tried ## Prevention Best Practices 1. **Use autodiscovery labels** to ensure secrets are properly recognized 2. **Test connectivity** before configuring integrations 3. **Verify credentials** are valid and have appropriate permissions 4. **Monitor integration health** regularly in the Integrations page 5. **Keep Causely updated** to ensure compatibility with latest telemetry sources 6. **Document your configuration** to make troubleshooting easier --- ## Learn The Learn section provides comprehensive documentation about the entities Causely discovers, the root causes it identifies, key terminology, and security information. This reference documentation helps you understand how Causely's causal reasoning engine works, what types of entities it discovers in your environment, the root causes it can identify, and the terminology used throughout the documentation. ## How Causely Works Causely uses a model-driven reasoning engine to continuously infer root causes for symptoms observed in production. The system works alongside your existing observability stack, interpreting the meaning behind metrics, logs, and traces, rather than replacing them. Learn more about the core concepts and mechanisms behind Causely's causal reasoning engine, including ontology, topology graphs, causality mapping, and analysis in [How Causely Works](/getting-started/how-causely-works). ## Entity Types Causely automatically discovers over 25 different entity types from your cloud native environment through data sources like eBPF, Cloud APIs, and OpenTelemetry. These entities are used to build topologies, identify defects, and infer root causes. Learn about the different types of entities that Causely automatically discovers, including applications, services, databases, compute resources, messaging systems, and data pipelines in [Entity Types](/reference/entity-types). ## Root Causes With more than 100 types of root causes captured in its Causal Models, Causely can pinpoint hundreds of thousands of potential issues and their effects within your environment. Root causes span applications, infrastructure, data pipelines, release management, and services. Explore the types of root causes that Causely can identify and how they impact your systems, from application bugs to infrastructure bottlenecks to release-related issues in [Root Causes](/reference/root-causes). ## Symptoms Symptoms are observable anomalies in managed objects that may be caused by root causes. Causely detects a wide variety of symptoms across services, workloads, compute resources, databases, messaging systems, and more. Browse the complete reference of all symptoms that Causely can detect, organized by category and entity type, in [Symptoms](/reference/symptoms). ## Terminology Understanding the key terms and concepts used throughout Causely documentation helps you get the most out of the system. The terminology covers core concepts like entities, symptoms, root causes, topology, causality graphs, and more. Understand the key terms and concepts used throughout Causely documentation, grouped by relatedness to help you navigate the system effectively in [Terminology](/reference/terminology). ## Security Causely is designed to protect sensitive data and ensure privacy. The system processes telemetry data locally and primarily transmits minimal, high-level information to its backend. All data is encrypted in transit and at rest. Learn about Causely's security model, data handling practices, privacy protections, and the permissions required for deployment components in [Security](/security). --- ## Custom Agents ## Building Custom Agents with Causely Most custom agents can access telemetry, but struggle to determine what is actually happening, what caused it, and what action is safe. Causely provides the system intelligence layer needed to interpret telemetry consistently and make reliable decisions. Whether you are building internal incident tooling, AI-driven workflows, or automation pipelines, Causely provides the structured system intelligence needed to interpret telemetry and act safely. ## When to Build a Custom Agent Building a custom agent is the right approach when: - You have internal workflows that do not map to existing tools - You want to integrate reliability decisions into your own systems - You are building end-to-end automation (detection → diagnosis → action) - You need full control over logic, policies, and execution ## How It Works Custom agents use Causely to move from raw telemetry to structured, system-aware decisions. Custom agents typically interact with Causely in one of two ways: ### 1. MCP Server (Recommended) Use the Causely MCP Server to provide a standardized interface for agents. Your agent: 1. Receives a signal (alert, event, user input) 2. Queries Causely via MCP 3. Receives structured outputs (root cause, dependencies, explanation) 4. Takes action based on those outputs This is the fastest way to integrate Causely into agent-based systems. ### 2. API Integration For more control or non-MCP environments, you can interact directly with the Causely API. Your system: 1. Sends queries to Causely (for example, root cause, topology, health) 2. Receives structured, machine-consumable responses 3. Uses those responses to drive logic and actions ## Tool Ordering: Resolve Entities First :::tip **Always call `get_entities()` before querying metrics, SLOs, topology, symptoms, or slow queries.** Most structured tools require an entity ID. `get_entities()` resolves a service or database name to its ID and returns current health status, type, and labels. ::: ## Tool Selection Choose the right tool based on what your agent needs to do with the result. | Use case | Tool | |---|---| | "What's wrong with checkout?" (narrative) | `get_service_summary` | | Root cause data for automated routing | `get_root_causes` | | Metrics for regression detection | `get_metrics` | | Entity ID resolution | `get_entities` | | SLO status with burn rate | `get_slo` | | Blast radius mapping | `get_topology` | | Post-deploy regression check | `reliability_delta` | See the [full tool reference](/agent-integration/mcp-server#full-tool-reference) for all 30 tools. ## Workflow Example: Incident Triage Workflow This example shows a custom agent performing full incident triage using structured MCP tools. **Scenario**: An alert fires. The agent needs to identify the root cause, understand impact, and route to the correct team. ```python # Step 1: Resolve the alerted service name to an entity ID entity = mcp.call("get_entities", query="checkout-service", entity_types=["Service"]) # Returns: entity ID, current health status, type, and labels # Step 2: Get root causes for the affected service root_causes = mcp.call("get_root_causes", impacted_service_ids=[entity[0]["id"]]) # Returns: root causes with severity, impacted services, and remediation guidance # Step 3: Map the blast radius for the highest-severity root cause topology = mcp.call( "get_topology", entity_id=root_causes[0]["entity_id"], mode="dependents" ) # Returns: upstream services affected by this entity's degradation # Step 4: Route to the correct team and generate a ticket ticket = mcp.call( "generate_ticket", task=f"Investigate root cause: {root_causes[0]['name']}" ) ``` After the incident resolves: ```python # Generate postmortem documentation postmortem = mcp.call("postmortem", root_cause_id=root_causes[0]["id"]) # Returns: structured markdown with timeline, blast radius, contributing factors, action items ``` ## What Your Agent Gets from Causely When integrated, your agent can: - Identify the true root cause of issues - Understand service dependencies and failure propagation - Evaluate blast radius before taking action - Work with structured, consistent outputs - Provide explainable decisions This allows your agent to move beyond querying data to making reliable decisions. ## Design Considerations When building custom agents with Causely: - **Trust boundaries**: define what actions can be automated vs require approval - **Policy enforcement**: gate actions based on risk or impact - **Observability**: log decisions and reasoning for auditability - **Fallbacks**: handle cases where no clear root cause is identified ## Next Steps - [Using the MCP Server](/agent-integration/mcp-server): full tool reference and key workflows - [HolmesGPT](/agent-integration/holmes-gpt): see a reference implementation - [API](../../api): explore direct API integration --- ## HolmesGPT ## Using Causely with HolmesGPT HolmesGPT is a pre-built agent framework for incident investigation and response. When configured with Causely, Holmes gains a live causal model of your system, so it can reason about root cause, blast radius, and remediation without requiring prompt engineering or custom context. Out of the box, Holmes can access telemetry sources such as metrics and logs. However, like most agents, it operates on raw data and correlation. This makes it difficult to consistently determine what is actually happening, what caused it, and what action is safe. Causely provides the system intelligence layer that allows Holmes to move from querying telemetry to making reliable, system-aware decisions. ## How Causely Enhances Holmes When integrated with Holmes, Causely enables: - **Deterministic root cause analysis**: identify the actual source of an issue, not just correlated signals - **Dependency-aware reasoning**: understand how failures propagate across services - **Structured outputs**: return machine-consumable results instead of raw telemetry - **Explainable decisions**: provide clear reasoning that can be audited and trusted This allows Holmes to focus on orchestrating workflows while relying on Causely for system-level understanding. ## Example Workflow **Scenario: Investigating a performance issue** 1. Holmes receives an alert or user query 2. Holmes queries Causely via MCP 3. Causely returns: - Root cause service - Affected dependencies - Explanation of the issue 4. Holmes: - Summarizes the issue - Notifies the appropriate team - Suggests or executes remediation ## Setup To integrate Causely with HolmesGPT, configure Causely as a remote MCP server. ### Holmes Configuration Add the following to your Holmes configuration: ```yaml mcp_servers: causely: description: "Causal Reasoning Platform" config: url: "https://api.causely.app/mcp" mode: streamable-http headers: Authorization: "Basic {{ env.CAUSELY_MCP_CLIENT_BASIC }}" llm_instructions: "Use Causely to investigate application performance issues, analyze distributed traces, and query infrastructure metrics. Prefer this over Prometheus for APM data." ``` ### Authentication Holmes calls the MCP server **non-interactively**, so use **HTTP Basic** with your Causely MCP **OAuth client ID** and **client secret** (machine credentials), not a Bearer API key. 1. In [API tokens](https://auth.causely.app/oauth/portal/api-tokens), create or copy the **client ID** and **client secret** for MCP access. 2. Build the standard Basic user-info string: `client_id`, a single colon (`:`), then `client_secret`, with no newline or extra characters (same encoding as `Authorization: Basic` elsewhere). 3. Base64-encode that string. Use the **raw Base64 output only** in the environment variable; the Holmes snippet above adds the `Basic ` prefix in the header value. Example (macOS or Linux): ``` export CAUSELY_MCP_CLIENT_BASIC="$(printf '%s' 'YOUR_CLIENT_ID:YOUR_CLIENT_SECRET' | base64)" ``` If your stack cannot put those credentials on `Authorization`, you can send the same payload on `X-Causely-Client-Basic` instead; see [Authentication](/agent-integration/mcp-server#authentication) on the MCP Server page for the full header table and edge cases. ## When to Use This Integration This integration is most valuable when: - You are using Holmes for incident investigation or automation - You have distributed systems with complex dependencies - You want consistent, reliable root cause analysis instead of correlation - You are looking to automate decision-making, not just data retrieval ## Notes - Causely complements existing telemetry sources such as Prometheus rather than replacing them - Holmes can continue to use other data sources, but should prioritize Causely for system-level reasoning - For full details on available tools and key workflows, see the [MCP Server documentation](/agent-integration/mcp-server) ## Next Steps - [Using the MCP Server](/agent-integration/mcp-server): full tool reference, key workflows, and tool selection guide - [Custom Agents](/agent-integration/custom-agents): build your own workflows using Causely --- ## Advanced Authentication The default browser-based OAuth flow covers most setups. Use this page when you need **machine credentials** (automation, CI, non-interactive agents) or when your client **only supports stdio** transport. ## Client ID and Client Secret {#authentication} For non-interactive MCP calls, supply client credentials that the server exchanges for a Frontegg access token. The HTTP Basic username is always `client_id` and the password is always `client_secret`. **Getting credentials** Generate OAuth client credentials for your tenant at: [https://auth.causely.app/oauth/portal/api-tokens](https://auth.causely.app/oauth/portal/api-tokens) **Encoding** Concatenate `client_id`, a single colon (`:`), and `client_secret`, no newline, no extra characters, then Base64-encode the result: ```bash printf '%s:%s' "$CLIENT_ID" "$CLIENT_SECRET" | base64 -w 0 ``` Use the single line of output as the Base64 payload. **How to send it** | Method | When to use | |---|---| | `Authorization: Basic ` | Requests with no `Authorization: Bearer` token, typical for `curl` or custom HTTP clients | | `X-Causely-Client-Basic` | Same Base64 payload (with or without the `Basic ` prefix). Use when an MCP proxy reserves or rewrites `Authorization`. If both headers are present, the server prefers `X-Causely-Client-Basic`. | If the request already includes a non-empty `Authorization: Bearer` token, the server validates that JWT and ignores client credentials entirely. **Adding credentials to a client config** For native HTTP MCP clients, add a `headers` object alongside `url`. Examples for each config format: JSON `mcpServers` (Claude Code, Cursor): ```json { "mcpServers": { "causely": { "type": "http", "url": "https://api.causely.app/mcp", "headers": { "X-Causely-Client-Basic": "Basic " } } } } ``` JSON `servers` (VS Code / GitHub Copilot): ```json { "servers": { "causely": { "type": "http", "url": "https://api.causely.app/mcp", "headers": { "X-Causely-Client-Basic": "Basic " } } } } ``` TOML (Codex), use `env_http_headers` to read the value from an environment variable at runtime so secrets stay out of the file: ```toml [mcp_servers.causely] url = "https://api.causely.app/mcp" enabled = true [mcp_servers.causely.env_http_headers] "X-Causely-Client-Basic" = "CAUSELY_MCP_CLIENT_BASIC" ``` Export `CAUSELY_MCP_CLIENT_BASIC` to the Base64 string (without a `Basic ` prefix, Codex does not prepend it for you unless you include it in the variable value). :::caution Do not commit real secrets to source control. Use your tool's secret or input-variable mechanism where available, for example VS Code MCP **`inputs`** and `${input:…}` in `headers`. ::: ## Stdio Fallback (mcp-remote) {#stdio-fallback-mcp-remote} If your client cannot use streamable HTTP to `https://api.causely.app/mcp`, run [`mcp-remote`](https://www.npmjs.com/package/mcp-remote) as a local stdio bridge: your tool launches Node locally; `mcp-remote` handles OAuth and forwards MCP traffic over HTTP to Causely. Requires **Node.js** and `npx` on your PATH. **Browser OAuth (default for this path):** ```json { "mcpServers": { "causely": { "command": "npx", "args": ["mcp-remote", "https://api.causely.app/mcp/"] } } } ``` **Machine credentials:** `mcp-remote` accepts repeated `--header "Name: value"` flags and expands `${ENV_VAR}` inside header values: ```json { "mcpServers": { "causely": { "command": "npx", "args": [ "mcp-remote", "https://api.causely.app/mcp/", "--header", "X-Causely-Client-Basic: Basic ${CAUSELY_MCP_CLIENT_BASIC}" ] } } } ``` Set `CAUSELY_MCP_CLIENT_BASIC` to the Base64 string without the `Basic ` prefix, the snippet above adds the prefix in the header string. --- ## Claude Code Connect Causely to Claude Code to run incident triage, service health checks, and reliability reports without leaving the terminal. ## Prerequisites - Active [Causely account](https://portal.causely.app/) - [Claude Code](https://docs.anthropic.com/en/docs/claude-code) installed ## Configuration Run once to register Causely across all your projects: ```bash claude mcp add --scope user --transport http causely https://api.causely.app/mcp ``` This writes to `~/.claude.json` automatically. No file editing required. **Team-shared config (optional)** To register Causely for everyone who clones a specific repo, create `.mcp.json` at the repository root instead: ```json { "mcpServers": { "causely": { "type": "http", "url": "https://api.causely.app/mcp" } } } ``` Only use this if authentication is handled separately, never commit credentials. See [Advanced Authentication](/agent-integration/mcp-server/advanced-auth). Begin a new Claude Code session (claude) after any config change. Run /mcp to confirm causely appears as a connected server. ## Adding Skills Seven skills activate automatically once installed: one master router (`causely-mcp`) plus six specialists. See the [Skills page](/agent-integration/mcp-server/skills) for the full list, trigger phrases, and override options. **Install** ```bash git clone https://github.com/causely-oss/causely-client mkdir -p ~/.claude/skills cp -r causely-client/skills/* ~/.claude/skills/ ``` This installs to personal scope (`~/.claude/skills/`) so skills are available across all your projects. To commit skills to a single repo instead, replace `~/.claude/skills/` with `.claude/skills/` in each path above. **Restart** Begin a new Claude Code session (`claude`) after installing. **Verify** Try: *"What's broken right now?"* The `causely-health-reporting` skill should activate. ## Try It Now - *"List my clusters."* - *"Are there any active symptoms right now?"* - *"What services are currently degraded?"* - *"Draft a postmortem for the most recent incident."* ## Known Gotcha Project .mcp.json is committed to source control. Never add credentials directly to this file. Use --scope user when registering Causely so auth stays out of the repository. See [Advanced Authentication](/agent-integration/mcp-server/advanced-auth). --- ## Claude Desktop Ask Causely questions in natural language from the Claude desktop app, no terminal, no code. ## Prerequisites - Active [Causely account](https://portal.causely.app/) - [Claude Desktop](https://claude.ai/download) installed - [Node.js](https://nodejs.org/) on the PATH visible to the app (required for `mcp-remote`) ## Configuration There are a few ways to connect depending on how you use Claude: browser or desktop, and UI or config file. ### Claude.ai (browser) **Pro / Max** 1. Go to claude.ai → Customize → Connectors 2. Click + then Add custom connector 3. Enter a name (for example Causely) and the server URL: https://api.causely.app/mcp 4. Click Add 5. Sign in to Causely and grant access 6. Return to Claude.ai, the connector is now active **Team / Enterprise** 1. Go to Organization settings → Connectors 2. Click Add → Custom → Web 3. Enter a name (for example Causely) and the server URL: https://api.causely.app/mcp 4. Click Add 5. Team members authenticate individually at Customize → Connectors ### Claude Desktop **Desktop UI** 1. Go to Settings → Connectors 2. Click Customize 3. Click + sign and select Add custom connector 4. Enter a name (for example Causely) and the server URL: https://api.causely.app/mcp 5. Click Add 6. Sign in to Causely and grant access 7. Return to Claude desktop, the connector is now active **Developer desktop app** 1. Go to Settings → Developer 2. Edit Config 3. Edit `claude_desktop_config.json` and merge the `mcpServers` object: ```json { "mcpServers": { "causely": { "command": "npx", "args": ["mcp-remote", "https://api.causely.app/mcp"] } } } ``` Or you can edit the config file directly, **Config file location**: | Platform | Path | |---|---| | macOS | `~/Library/Application Support/Claude/claude_desktop_config.json` | | Windows | `%APPDATA%\Claude\claude_desktop_config.json` | **Restart** Quit and reopen Claude Desktop. On first connect, a browser window opens for Causely OAuth login. After you authorize, Claude Desktop stores the token and reconnects automatically on subsequent launches. ## Adding Skills Seven skills activate automatically once installed: one master router (`causely-mcp`) plus six specialists. See the [Skills page](/agent-integration/mcp-server/skills) for the full list, trigger phrases, and override options. **Install** Ensure that Skills are enabled for your Enterprise or Team plan. 1. Clone the Causely client repo ```bash git clone https://github.com/causely-oss/causely-client cd causely-client/skills ``` 2. Create a .zip file for each sub folder in the skills folder macOS ```bash for dir in causely-mcp causely-postmortem causely-k8s-investigation causely-health-reporting causely-correlated-incidents causely-change-impact causely-alert-triage; do zip -r "${dir}.zip" "$dir" \ --exclude "*/.DS_Store" \ --exclude "__MACOSX/*" \ --exclude "*/.git/*" \ --exclude "*/Thumbs.db" done ``` Windows ```powershell foreach ($dir in @( "causely-postmortem", "causely-mcp", "causely-k8s-investigation", "causely-health-reporting", "causely-correlated-incidents", "causely-change-impact", "causely-alert-triage" )) { $tmp = New-Item -ItemType Directory -Path "$env:TEMP\$dir" Copy-Item -Path $dir\* -Destination $tmp -Recurse -Exclude @('.DS_Store','Thumbs.db') Compress-Archive -Path $tmp -DestinationPath "$dir.zip" -Force Remove-Item $tmp -Recurse -Force } ``` 3. Go to Customize → Skills 4. Click + sign and select Create skill → Upload a skill 5. For each of the 7 `.zip` files: click **+** → **Create skill** → **Upload a skill**, select the file, and save. Repeat until all 7 are uploaded. Each upload creates one skill. **Restart** Quit and reopen Claude Desktop after installing. **Verify** Try: *"What's broken right now?"* The `causely-health-reporting` skill should activate. ## Try It Now - *"List my clusters."* - *"Are there any active symptoms right now?"* - *"What services are currently degraded?"* - *"Draft a postmortem for the most recent incident."* ## Known Gotcha Claude Desktop launches as a GUI app and does not inherit your shell's `PATH`. If `npx` is not found at launch, install Node.js globally, not via `nvm` or a shell-managed version manager, so the binary is visible to the app process. On macOS, the [official Node.js installer](https://nodejs.org/) or `brew install node` is the most reliable approach. --- ## OpenAI Codex Connect Causely to OpenAI Codex to run reliability checks and incident triage alongside your coding workflows. ## Prerequisites - Active [Causely account](https://portal.causely.app/) - [OpenAI Codex](https://openai.com/index/openai-codex/) CLI installed ## Configuration Add the following to `~/.codex/config.toml` or project `.codex/config.toml`: ```toml [mcp_servers.causely] url = "https://api.causely.app/mcp" enabled = true ``` **Config file location** | Scope | Path | |---|---| | User | `~/.codex/config.toml` | | Project | `.codex/config.toml` at the repository root | ## Restart Codex reads `config.toml` once at launch. After editing, start a new Codex session to pick up the change. :::tip Skills are not yet supported for Codex. You'll interact with the Causely MCP server directly; the example prompts in the [Key Workflows](/agent-integration/mcp-server#key-workflows) section still work, but tool selection is handled by Codex's agent rather than a Causely skill router. ::: ## Try It Now - *"List my clusters."* - *"Are there any active symptoms right now?"* - *"What services are currently degraded?"* ## Known Gotcha For machine credentials, use `env_http_headers` (reads from environment variables at runtime) instead of `http_headers` (static values in the file) to keep secrets out of the config: ```toml [mcp_servers.causely] url = "https://api.causely.app/mcp" enabled = true [mcp_servers.causely.env_http_headers] "X-Causely-Client-Basic" = "CAUSELY_MCP_CLIENT_BASIC" ``` Export `CAUSELY_MCP_CLIENT_BASIC` to the Base64-encoded `client_id:client_secret` string before launching Codex. See [Advanced Authentication](/agent-integration/mcp-server/advanced-auth) for encoding instructions. --- ## Cursor Connect Causely to Cursor to investigate incidents and check service health inline while you code. ## Prerequisites - Active [Causely account](https://portal.causely.app/) - [Cursor](https://www.cursor.com/) with MCP support (1.0+) ## Installation ### Option 1: Cursor plugin (recommended) Install the Causely plugin from [cursor.directory/plugins/causely](https://cursor.directory/plugins/causely) in one click. It configures the MCP server and installs all seven skills automatically, no manual steps required. After installing, open **Settings → MCP** and confirm `causely` appears with a green status indicator. ### Option 2: Manual setup **MCP server** Create or update `.cursor/mcp.json` at the root of your project: ```json { "mcpServers": { "causely": { "type": "http", "url": "https://api.causely.app/mcp" } } } ``` | Scope | Path | |---|---| | User (all projects) | `~/.cursor/mcp.json`, open via **Settings → MCP → Edit Config** | | Project | `.cursor/mcp.json` at the project root | **Add Skills** ```bash git clone https://github.com/causely-oss/causely-client mkdir -p .cursor/skills cp -r causely-client/skills/* .cursor/skills/ ``` Cursor skills are project-scoped; there is no personal skills directory. Run these commands at the root of each project where you want skills available. **Restart** After saving the config, open **Settings → MCP**, `causely` should appear with a green status indicator. If it does not, use **Cmd/Ctrl+Shift+P → MCP: Restart MCP Server** or restart Cursor entirely. **Verify** Try: *"What's broken right now?"* The `causely-health-reporting` skill should activate. ## Try It Now - *"List my clusters."* - *"Are there any active symptoms right now?"* - *"What services are currently degraded?"* - *"Draft a postmortem for the most recent incident."* ## Known Gotcha Cursor caches MCP tool descriptions at session start. If you update the server config or Causely releases new tools, the previous tool list persists until you explicitly restart the MCP server or restart Cursor. See [Advanced Authentication](/agent-integration/mcp-server/advanced-auth) if you need to add machine credentials. --- ## MCP Server Integration The Causely MCP server gives agents and AI assistants direct access to Causely's causal reasoning engine. 30 tools across 5 categories let your agent move from raw alerts to structured root cause analysis, dependency maps, and reliability reports, without writing custom integrations. ## Key Workflows These are the four workflows agents use most often. Each maps to a specific sequence of MCP tool calls. ### Incident Triage Identify what's broken and how far it has spread. **Try:** - *"What's broken in production right now?"* - *"Checkout is throwing errors and we have Alertmanager alerts firing. What's the actual root cause?"* - *"Three services are alerting at once. Which is the real problem and which are downstream noise?"* 1. `get_symptoms()`: see all active symptoms across the entire environment (no filters needed) 2. `get_root_causes()`: identify all active root causes and impacted services 3. `get_alerts(alert_name_filters=...,)`: drill into a specific alert's cause 4. `get_topology(entity_id=..., mode="dependents")`: map which upstream services are affected Handled automatically by `causely-correlated-incidents`, or `causely-alert-triage` if you're starting from a specific alert. ### Quick Service Health Get a complete health picture for a specific service in two calls. **Try:** - *"Is checkout healthy?"* - *"Give me a full health picture for the payments service: status, open issues, SLOs."* - *"Before I page anyone, is there actually a problem with database-service or is this alert noise?"* - *"Are there are any concerning errors or warnings in the logs for the frontend service over the last hour?"* 1. `get_entities(query="service-name", entity_types=["Service"])`: resolve the service name to its entity ID 2. `get_service_summary(service="service-name")`: full snapshot: status, active symptoms, root causes, SLOs, metrics, recent events, error logs 3. `get_logs(entity_id=...,)`: retrieves live log output for a running service Handled automatically by the `causely-health-reporting` skill. ### Post-Deploy Validation Check whether a deployment introduced regressions. **Try:** - *"Did the last deploy to payments cause any regressions?"* - *"We deployed cart service 30 minutes ago. How does it look compared to before?"* - *"Check all services my team owns, did anything degrade after today's deploys?"* 1. `reliability_delta(service="service-name")`: compare CPU, memory, latency, and error rate before vs after the most recent deployment 2. `fleet_reliability_delta(team="team-name")`: batch check across all services for a team, namespace, or explicit list Handled automatically by the `causely-change-impact` skill. ### Post-Incident Reporting Generate postmortem documentation and action items from a resolved incident. **Try:** - *"The payments outage is resolved. Draft a postmortem."* - *"Write up what happened to checkout this morning: timeline, root cause, and what was affected."* - *"Payments is back up. Draft the postmortem and create a follow-up ticket for the team."* 1. `get_root_cause_details(root_cause_id=...)`: retrieve full root cause details, timeline, and blast radius 2. `postmortem(root_cause_id=...)`: generate a structured postmortem draft 3. `generate_ticket(task="...")`: create a follow-up engineering ticket for Jira, GitHub Issues, or Linear Handled automatically by the `causely-postmortem` skill. ## Skills (Recommended) Skills automate the tool-selection step shown in the workflows above. You describe your situation in natural language; the right specialist activates and runs the correct tool sequence for you. Skills are available for Claude Code, Claude Desktop, and Cursor. | Situation | Skill | Try | |---|---|---| | Incoming alert | `causely-alert-triage` | *"PagerDuty just paged for checkout-latency. What's the actual cause?"* | | Post-deploy validation | `causely-change-impact` | *"Did the last deploy to payments cause any regressions?"* | | Multi-service outage | `causely-correlated-incidents` | *"Three services are alerting at once. What's the real problem?"* | | Health summary / morning standup | `causely-health-reporting` | *"Give me a morning health report for production."* | | Kubernetes investigation | `causely-k8s-investigation` | *"The orders pod keeps OOMKilling. Why?"* | | Postmortem / ticket | `causely-postmortem` | *"Draft a postmortem for the checkout outage that resolved an hour ago."* | See the [Skills page](/agent-integration/mcp-server/skills) for install instructions, full skill detail, and override options. ## Choose Your Client Select your tool for a copy-paste config snippet, config file location, and restart instructions. | Client | Transport | Config format | |---|---|---| | [Claude Code](/agent-integration/mcp-server/claude-code) | HTTP | `.mcp.json` (`mcpServers`) | | [Claude Desktop](/agent-integration/mcp-server/claude-desktop) | stdio via `mcp-remote` | `claude_desktop_config.json` (`mcpServers`) | | [Codex](/agent-integration/mcp-server/codex) | HTTP | `config.toml` (`mcp_servers`) | | [Cursor](/agent-integration/mcp-server/cursor) | HTTP | `.cursor/mcp.json` (`mcpServers`) | | [VS Code (GitHub Copilot)](/agent-integration/mcp-server/vscode-copilot) | HTTP | `.vscode/mcp.json` (`servers`) | **Verify your connection** by asking: _”Causely: What defects are currently active?”_ ## Other MCP-compatible Clients The clients above have dedicated setup pages. The following tools also support the Causely MCP server, point them at `https://api.causely.app/mcp` using your tool's HTTP MCP config. See [Advanced Authentication](/agent-integration/mcp-server/advanced-auth) for credential options. **IDEs and Editors:** JetBrains IDEs (IntelliJ IDEA, PyCharm, WebStorm, GoLand, and others), Windsurf, Zed **CLIs:** Kiro CLI, Amp, Atlassian Rovo DEV CLI, and other MCP-compatible CLI tools **Agent Frameworks:** HolmesGPT ## Authentication {#authentication} The MCP server validates Frontegg-issued Bearer tokens. For most clients, browser-based OAuth runs automatically, no manual setup needed. For non-interactive setups (automation, CI) or clients that only support stdio, including the [stdio/mcp-remote fallback](/agent-integration/mcp-server/advanced-auth#stdio-fallback-mcp-remote), see [Advanced Authentication](/agent-integration/mcp-server/advanced-auth). ## Using the Tool Reference :::tip **The reference below is for teams building custom agents that need explicit tool control.** If you're using Claude, Cursor, Codex, or any conversational agent, you can skim it for capability awareness. In most cases, you can describe what you want and the agent picks the right tools. One thing worth knowing if you do go programmatic: most structured tools require an entity ID, so `get_entities()` is usually the right first call. ::: ## What Agents Get vs Raw Telemetry | | Raw telemetry | Causely MCP | |---|---|---| | Root cause identification | Correlation-based, requires analysis | Deterministic causal analysis | | Dependency awareness | Manual mapping required | Live topology from observed traffic | | Blast radius | Estimated | Computed from causal graph | | Structured output | Custom parsing required | Typed tool responses | | Time to insight | Minutes of analysis | Single tool call | ## Full Tool Reference 30 tools across 5 categories. All tools are available to any MCP-compatible agent or assistant. ### Entity Resolution | Tool | When to use | |---|---| | `get_entities` | **Start here.** Resolve a service or database name to its ID; list all entities in a namespace; check current health status | | `name_lookup` | Resolve any name, including service, cluster, namespace, root cause name, or symptom name, to an entity ID for use in other tools | | `get_label_values` | Enumerate valid label values (team, product, cluster, namespace) before fanning out queries across environments | ### Data Retrieval | Tool | When to use | |---|---| | `get_metrics` | Retrieve numeric metric data (p95 latency, error rate, CPU, memory, throughput): the only tool that returns time-series | | `get_logs` | Inspect live service logs, or retrieve evidence logs captured at root cause detection time | | `get_alerts` | Start triage from an alert name (PagerDuty, Slack, Datadog); distinguish alerts mapped to causal analysis from noise | | `get_events` | Correlate symptom onset with deployments, restarts, scaling events, or config changes | | `get_config` | Investigate configuration drift; verify deployment manifest matches expectations | | `get_slow_queries` | Identify database queries consuming the most execution time; follow up on database root causes | ### Health & Diagnosis | Tool | When to use | |---|---| | `get_symptoms` | Call with no filters to see all active symptoms across the entire environment or filter for specific entity, namespace or cluster | | `get_root_causes` | Identify active root causes; filter by impacted service, symptom, root cause ID, or a start/end date range; use start/end dates when investigating a specific past time window | | `get_root_cause_details` | Follow-up to `get_root_causes`: full evidence for one root cause given a `root_cause_id`; includes the causal chain explaining why it was identified, blast radius, symptoms, exceptions, events, and logs | | `get_entity_health` | Structured health summary for non-Service entities (databases, pods, queues, topics, tables) | | `get_environment_health` | Structured health summary for the environment, can be scoped to specific namespaces or services | | `get_slo` | Check SLO state, error budget remaining, and burn rate | | `get_topology` | Find upstream blast radius (dependents), downstream dependencies, or full data-flow graph | | `get_integration_status` | Verify monitoring coverage; check scraper health by cluster | | `get_incident_impact` | Given a root cause ID (or an entity ID + root cause name), returns the responsible service and its business context, plus all impacted services and their business context | | `team_health` | Health summary for all services owned by a team; degraded and critical services listed first | | `get_service_summary` | Comprehensive health snapshot for a single service: status, symptoms, root causes, SLOs, metrics, events, logs | | `investigate_alert` | Investigate a resolved alert from `get_alerts`; maps the alert to its entity and returns the standard `get_entity_health` result alongside the original alert | | `rank_entities` | Rank services, topics, tables, or endpoints by number of dependencies or dependents, a single bulk query instead of looping `get_topology` | | `get_potential_diagnoses` | Active and model-inferred diagnosis hypotheses for a specific entity; includes inactive and causality-only potentials not returned by `get_root_causes` | | `get_potential_observable_signals` | All observable signals on a specific entity (active, inactive, and causality potential state); use before `get_signal_potential_diagnoses` to find internal signal names | | `get_signal_potential_diagnoses` | Reverse lookup: given a symptom, event, or SLO on an entity, return the diagnosis from the causality model that could explain it | | `get_diagnosis_observable_signals` | Retrieve the theoretical causality chain for a diagnosis, including downstream symptoms, events, and SLOs it could cause according to the causal model; compare to observed signals from `get_symptoms` | ### Reporting & Postmortems | Tool | When to use | |---|---| | `postmortem` | Generate a deterministic postmortem draft for a resolved incident from Causely data | | `generate_ticket` | Create a structured engineering ticket suitable for Jira, GitHub Issues, or Linear | ### Reliability & Deployment | Tool | When to use | |---|---| | `reliability_delta` | Post-deploy regression check for a single service: compare resource consumption before/after most recent deployment | | `fleet_reliability_delta` | Batch regression check across a team, namespace, or explicit service list (up to 20 services per call) | ## Feature Demos ### Solving Slow Database Queries ### Helm Chart Example --- ## Skills Skills eliminate the need to pick the right MCP tool manually. You describe your situation; the `causely-mcp` router picks the matching specialist; the specialist runs the correct tool sequence for you. All 29 MCP tools are covered across 7 skills: one master router plus 6 specialists. ## How it works `causely-mcp` is the master router that sits above the six specialist skills. It activates on any observability or reliability question and matches your prompt against each specialist's trigger set, delegating to the closest fit. The router also pulls in `complete-investigation.md` as a shared reference so specialists can coordinate context across multi-step investigations. You never invoke a specialist directly; the router handles that automatically, though you can [override it](#overrides-and-escape-hatches) when needed. ## Client support | Client | Skills | Install | |---|---|---| | Claude Code | Yes | [Manual (copy from repo)](/agent-integration/mcp-server/claude-code) | | Claude Desktop | Yes | [Manual (copy from repo)](/agent-integration/mcp-server/claude-desktop) | | Cursor | Yes | [Plugin (one-click)](/agent-integration/mcp-server/cursor) or [Manual](/agent-integration/mcp-server/cursor#option-2-manual-setup) | | Codex | Not yet supported | | | VS Code Copilot | Not yet supported | | ## causely-alert-triage Activates on any message that names or describes an incoming alert from PagerDuty, Datadog, Prometheus/Alertmanager, Slack, or OpsGenie, or asks what a firing alert means. **Example trigger prompts** - *"PagerDuty is firing P1 for checkout-service. What's the root cause?"* - *"I have three Datadog alerts going off at once. Which one actually matters?"* - *"OpsGenie woke me up at 2 AM for payment-processor. Is this real or noise?"* - *"Alertmanager is showing HighErrorRate on api-gateway. What's actually broken?"* **Under the hood** Calls `get_alerts` to fetch the raw alert payload (supports substring search by alert name with no entity IDs needed), `investigate_alert` as a one-step follow-up to get entity health alongside the alert context, and `get_root_causes` to surface the confirmed root cause for mapped alerts.
## causely-change-impact Activates on any message asking whether a recent deployment, rollout, or config change caused a regression or reliability shift. **Example trigger prompts** - *"We deployed v2.3.1 of order-service 20 minutes ago. Did it break anything?"* - *"Reliability got worse right after this morning's rollout. What degraded?"* - *"Compare service health before and after today's deploy to checkout."* - *"Did our canary release introduce any downstream failures?"* **Under the hood** Calls `get_events` to locate the deployment event, `reliability_delta` to compare reliability before and after, `fleet_reliability_delta` for a fleet-wide comparison, `get_incident_impact` to retrieve the responsible service and impacted services with business context once a root cause is confirmed, `get_config` to inspect the deployed configuration, and `get_metrics` for specific metric trends.
## causely-correlated-incidents Activates on any message describing multiple services alerting simultaneously, or asking about blast radius, cascading failures, dependency chains, or whether separate incidents share a single cause. **Example trigger prompts** - *"Five services are alerting at the same time. Is there a single root cause?"* - *"Checkout, payments, and inventory all degraded. Is this one incident?"* - *"We have a cascade in production. Where did it start?"* - *"Which services will be affected if database-primary goes down?"* **Under the hood** Calls `get_root_causes` to find all active root causes, `get_topology` to map the dependency graph and blast radius, `get_alerts` to correlate firing alerts across services, and `get_incident_impact` to retrieve the responsible service and blast radius with business context for a confirmed root cause.
## causely-health-reporting Activates on any request for a health summary, SLO status update, standup report, reliability overview spanning multiple services or an entire environment, or a single-service health check ("is X healthy?"). **Example trigger prompts** - *"Give me a standup summary of service health for the platform team."* - *"Which services are closest to breaching their SLOs this week?"* - *"Summarize overnight health of all production services."* - *"What's the reliability snapshot for the checkout domain?"* **Under the hood** Calls `get_environment_health` for an environment-wide summary, `get_service_summary` for per-service detail, `get_slo` for SLO status, `team_health` for team-scoped aggregation, `get_symptoms` for a full signal scan across all entities, and `get_root_causes` to flag any active issues.
## causely-k8s-investigation Activates on any message about Kubernetes infrastructure health: nodes, pods, namespaces, deployments, DaemonSets, or containers. These include OOMKills, pod restarts, node pressure, scheduling failures, resource exhaustion, CrashLoopBackOff, and evictions. **Example trigger prompts** - *"payment-processor pods keep OOMKilling. What's causing it?"* - *"Why does api-gateway restart every few hours?"* - *"Node pressure is high on cluster west-1. Which workloads are to blame?"* - *"My pods are crash-looping but the logs don't show an obvious error."* **Under the hood** Calls `get_service_summary` for a full service-level health check, `get_environment_health` for namespace sweeps, `get_symptoms` for a pod-level signal scan, `get_incident_impact` to retrieve the responsible service and impacted services with business context for a confirmed root cause, `get_entity_health` for pod or node status, `get_events` for recent Kubernetes events (OOMKill, CrashLoopBackOff, evictions), `get_config` to inspect resource requests and limits, `get_metrics` for CPU and memory trends, `get_logs` for container log analysis, and `get_root_causes` scoped to a namespace.
## causely-postmortem Activates on any request to write a postmortem, incident retrospective, or create a ticket documenting a completed or past outage. **Example trigger prompts** - *"Write a postmortem for yesterday's checkout outage."* - *"Draft a Jira ticket for the payment service incident last night."* - *"Create an incident retrospective for the database brownout on Friday."* - *"Generate an RCA document for the cascade failure in production on April 25."* **Under the hood** Calls `get_root_causes` to reconstruct the incident timeline, `postmortem` to generate the structured retrospective document, and `generate_ticket` to create a Jira or Linear ticket draft.
## Overrides and escape hatches **Force a specific skill** Prefix your prompt with `use causely-` to bypass the router and invoke a specific specialist directly: ``` use causely-postmortem: write a retrospective for the payment incident on April 25 ``` **Bypass skills entirely** You can call MCP tools directly without involving any skill. See the [Full Tool Reference](/agent-integration/mcp-server#full-tool-reference) for the complete list of tools and parameters. **If the wrong skill activates** Tell the agent which skill to use instead: `use causely-`. If a skill consistently miss routes on a prompt you expect to work, email [support@causely.ai](mailto:support@causely.ai) with the prompt text and which skill activated. --- ## VS Code (GitHub Copilot) Connect Causely to GitHub Copilot in VS Code to query root causes and service health from Copilot Chat. ## Prerequisites - Active [Causely account](https://portal.causely.app/) - [VS Code](https://code.visualstudio.com/) with the [GitHub Copilot](https://marketplace.visualstudio.com/items?itemName=GitHub.copilot) extension (version 0.22+ for MCP support) ## Configuration Create or update `.vscode/mcp.json` in your workspace: ```json { "servers": { "causely": { "type": "http", "url": "https://api.causely.app/mcp" } } } ``` :::note VS Code Copilot uses `servers` as the top-level key, not `mcpServers`. Using `mcpServers` silently fails, the server simply won't appear. ::: **Config file location** | Scope | Path | |---|---| | Workspace | `.vscode/mcp.json` at the repository root | | User | Command Palette → **MCP: Open User Configuration** | ## Restart After saving the config, open the Command Palette and run **MCP: List Servers** to confirm `causely` appears. Copilot Chat picks up the server on the next conversation turn, no full restart required. :::tip Skills are not yet supported for VS Code Copilot. You'll interact with the Causely MCP server directly; the example prompts in the [Key Workflows](/agent-integration/mcp-server#key-workflows) section still work, but tool selection is handled by GitHub Copilot's agent rather than a Causely skill router. ::: ## Try It Now - *"List my clusters."* - *"Are there any active symptoms right now?"* - *"What services are currently degraded?"* ## Known Gotcha If your organization uses GitHub Copilot Business or Enterprise, an admin may need to allow MCP servers in the Copilot policy settings before user-configured servers can connect. See [Advanced Authentication](/agent-integration/mcp-server/advanced-auth) for adding machine credentials via `headers` or VS Code `inputs`. --- ## API Authentication All Causely GraphQL API requests require authentication using a Bearer token. You'll need to exchange your API client credentials for an access token using our authentication endpoint. ## How to Get API Access Token To verify that your credentials are working, you can use the following curl command. Replace `` and `` with the credentials you generated above: ```bash CAUSELY_CLIENT_ID= CAUSELY_CLIENT_SECRET= curl -X POST https://auth.causely.app/frontegg/identity/resources/auth/v2/api-token \ -H "Content-Type: application/json" \ -d "{ \"clientId\": \"${CAUSELY_CLIENT_ID}\", \"secret\": \"${CAUSELY_CLIENT_SECRET}\" }" ``` **Example Response:** ```json { "access_token": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6Ik...", "refresh_token": "25b14169-b912-4d15-8f9c-xxxxxxxx", "expires_in": 86400, "expires": "Sat, 31 May 2025 22:54:23 GMT" } ``` You can use the `access_token` for subsequent API calls. If you want to retrieve the access token and store it in an environment variable, you can use the following curl command: ```bash CAUSELY_CLIENT_ID= CAUSELY_CLIENT_SECRET= export CAUSELY_ACCESS_TOKEN=$(response=$(curl -s -w "\n%{http_code}" -X POST https://auth.causely.app/frontegg/identity/resources/auth/v2/api-token \ -H "Content-Type: application/json" \ -d "{\"clientId\": \"${CAUSELY_CLIENT_ID}\", \"secret\": \"${CAUSELY_CLIENT_SECRET}\"}"); \ http_code=$(echo "$response" | tail -n1); \ body=$(echo "$response" | sed '$d'); \ if [ "$http_code" = "200" ]; then echo "$body" | jq -r .access_token; else echo "$body" >&2; false; fi) ``` You can then use the `${CAUSELY_ACCESS_TOKEN}` environment variable for subsequent API calls. Similarly you can obtain the token using your preferred programming language. ```python # Make sure to install the requests library: pip install requests def get_causely_access_token(client_id, client_secret, url="https://auth.causely.app/frontegg/identity/resources/auth/v2/api-token"): res = requests.post(url, headers={"Content-Type": "application/json"}, json={"clientId": client_id, "secret": client_secret}) if res.status_code != 200: raise requests.HTTPError(f"Token request failed ({res.status_code}): {res.text}") token = res.json().get("access_token") if not token: raise ValueError("No access_token in response.") return token # Example usage: if __name__ == "__main__": cid, secret = os.getenv("CAUSELY_CLIENT_ID"), os.getenv("CAUSELY_CLIENT_SECRET") if not cid or not secret: raise EnvironmentError("Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET.") try: token = get_causely_access_token(cid, secret) print("Access token:", token) except Exception as e: print(f"Error: {e}") ``` ```javascript const https = require('https'); function getCauselyAccessToken( clientId, clientSecret, url = 'https://auth.causely.app/frontegg/identity/resources/auth/v2/api-token', ) { return new Promise((resolve, reject) => { const data = JSON.stringify({ clientId, secret: clientSecret }); const req = https.request( url, { method: 'POST', headers: { 'Content-Type': 'application/json', 'Content-Length': Buffer.byteLength(data), }, }, (res) => { let body = ''; res.on('data', (chunk) => (body += chunk)); res.on('end', () => { if (res.statusCode !== 200) { return reject(new Error(`HTTP ${res.statusCode}: ${body}`)); } const json = JSON.parse(body); if (!json.access_token) return reject(new Error('No access_token in response.')); resolve(json.access_token); }); }, ); req.on('error', reject); req.write(data); req.end(); }); } // Example usage const { CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET } = process.env; if (!CAUSELY_CLIENT_ID || !CAUSELY_CLIENT_SECRET) { console.error('Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET'); process.exit(1); } getCauselyAccessToken(CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET) .then((token) => console.log('Access token:', token)) .catch((err) => console.error('Error:', err.message)); ``` ```go package main "bytes" "encoding/json" "fmt" "io" "net/http" "os" ) func getToken(id, secret, url string) (string, error) { data, _ := json.Marshal(map[string]string{"clientId": id, "secret": secret}) req, _ := http.NewRequest("POST", url, bytes.NewReader(data)) req.Header.Set("Content-Type", "application/json") res, err := http.DefaultClient.Do(req) if err != nil { return "", err } defer res.Body.Close() body, _ := io.ReadAll(res.Body) if res.StatusCode != 200 { return "", fmt.Errorf("status %d: %s", res.StatusCode, body) } var resp map[string]interface{} if err := json.Unmarshal(body, &resp); err != nil { return "", err } token, ok := resp["access_token"].(string) if !ok || token == "" { return "", fmt.Errorf("no access_token in response") } return token, nil } func main() { id, secret := os.Getenv("CAUSELY_CLIENT_ID"), os.Getenv("CAUSELY_CLIENT_SECRET") if id == "" || secret == "" { fmt.Fprintln(os.Stderr, "Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET") os.Exit(1) } url := "https://auth.causely.app/frontegg/identity/resources/auth/v2/api-token" token, err := getToken(id, secret, url) if err != nil { fmt.Fprintln(os.Stderr, "Error:", err) os.Exit(1) } fmt.Println("Access token:", token) } ``` --- ## Gateway Token Management Gateway tokens authenticate the Causely mediator agent when it connects to the Causely platform. Creating tokens via the REST API enables automated deployment pipelines, for example ArgoCD or Backstage workflows that provision a mediator without manual steps in the UI. :::tip Prerequisite You need a valid Bearer access token before calling this endpoint. See [Authentication](/api/authentication) for how to obtain one. ::: ## Create a Gateway Token **`POST https://api.causely.app/api/tokens`** ```bash CAUSELY_CLIENT_ID= CAUSELY_CLIENT_SECRET= TOKEN_NAME= # Obtain a Bearer access token export CAUSELY_ACCESS_TOKEN=$(response=$(curl -s -w "\n%{http_code}" -X POST https://auth.causely.app/frontegg/identity/resources/auth/v2/api-token \ -H "Content-Type: application/json" \ -d "{\"clientId\": \"${CAUSELY_CLIENT_ID}\", \"secret\": \"${CAUSELY_CLIENT_SECRET}\"}"); \ http_code=$(echo "$response" | tail -n1); \ body=$(echo "$response" | sed '$d'); \ if [ "$http_code" = "200" ]; then echo "$body" | jq -r .access_token; else echo "$body" >&2; false; fi) # Create a gateway token curl -s -X POST https://api.causely.app/api/tokens \ -H "Authorization: Bearer ${CAUSELY_ACCESS_TOKEN}" \ -H "Content-Type: application/json" \ -d "{\"name\": \"${TOKEN_NAME}\"}" ``` ```python # Make sure to install the requests library: pip install requests def create_gateway_token(access_token, name, url="https://api.causely.app/api/tokens"): res = requests.post( url, headers={ "Authorization": f"Bearer {access_token}", "Content-Type": "application/json", }, json={"name": name}, ) if res.status_code != 201: raise requests.HTTPError(f"Token creation failed ({res.status_code}): {res.text}") return res.json() # Example usage: if __name__ == "__main__": cid = os.getenv("CAUSELY_CLIENT_ID") secret = os.getenv("CAUSELY_CLIENT_SECRET") token_name = os.getenv("TOKEN_NAME", "my-mediator-token") if not cid or not secret: raise EnvironmentError("Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET.") access_token = get_causely_access_token(cid, secret) gateway_token = create_gateway_token(access_token, token_name) print("Gateway token value:", gateway_token["tokenValue"]) ``` ```javascript const https = require('https'); function createGatewayToken(accessToken, name, url = 'https://api.causely.app/api/tokens') { return new Promise((resolve, reject) => { const data = JSON.stringify({ name }); const parsedUrl = new URL(url); const req = https.request( { hostname: parsedUrl.hostname, path: parsedUrl.pathname, method: 'POST', headers: { Authorization: `Bearer ${accessToken}`, 'Content-Type': 'application/json', 'Content-Length': Buffer.byteLength(data), }, }, (res) => { let body = ''; res.on('data', (chunk) => (body += chunk)); res.on('end', () => { if (res.statusCode !== 201) { return reject(new Error(`HTTP ${res.statusCode}: ${body}`)); } resolve(JSON.parse(body)); }); }, ); req.on('error', reject); req.write(data); req.end(); }); } // Example usage const { CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET } = process.env; const tokenName = process.env.TOKEN_NAME || 'my-mediator-token'; if (!CAUSELY_CLIENT_ID || !CAUSELY_CLIENT_SECRET) { console.error('Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET'); process.exit(1); } (async () => { try { const accessToken = await getCauselyAccessToken(CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET); const gatewayToken = await createGatewayToken(accessToken, tokenName); console.log('Gateway token value:', gatewayToken.tokenValue); } catch (err) { console.error('Error:', err.message); process.exit(1); } })(); ``` ```go package main "bytes" "encoding/json" "fmt" "io" "net/http" "os" ) type GatewayToken struct { ID string `json:"id"` Name string `json:"name"` TokenValue string `json:"tokenValue"` MediatorID string `json:"mediatorId"` ProjectID string `json:"projectId"` CreatedBy string `json:"createdBy"` CreatedByEmail string `json:"createdByEmail"` State string `json:"state"` CreatedAt string `json:"createdAt"` } func createGatewayToken(accessToken, name, url string) (*GatewayToken, error) { data, _ := json.Marshal(map[string]string{"name": name}) req, _ := http.NewRequest("POST", url, bytes.NewReader(data)) req.Header.Set("Authorization", "Bearer "+accessToken) req.Header.Set("Content-Type", "application/json") res, err := http.DefaultClient.Do(req) if err != nil { return nil, err } defer res.Body.Close() body, _ := io.ReadAll(res.Body) if res.StatusCode != 201 { return nil, fmt.Errorf("status %d: %s", res.StatusCode, body) } var token GatewayToken if err := json.Unmarshal(body, &token); err != nil { return nil, err } return &token, nil } func main() { id, secret := os.Getenv("CAUSELY_CLIENT_ID"), os.Getenv("CAUSELY_CLIENT_SECRET") tokenName := os.Getenv("TOKEN_NAME") if id == "" || secret == "" { fmt.Fprintln(os.Stderr, "Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET") os.Exit(1) } if tokenName == "" { tokenName = "my-mediator-token" } accessToken, err := getToken(id, secret, "https://auth.causely.app/frontegg/identity/resources/auth/v2/api-token") if err != nil { fmt.Fprintln(os.Stderr, "Error getting access token:", err) os.Exit(1) } gatewayToken, err := createGatewayToken(accessToken, tokenName, "https://api.causely.app/api/tokens") if err != nil { fmt.Fprintln(os.Stderr, "Error creating gateway token:", err) os.Exit(1) } fmt.Println("Gateway token value:", gatewayToken.TokenValue) } ``` **Example Response (201 Created):** ```json { "id": "c2679064-c908-4bb0-9928-a0e3ce37da80", "name": "my-mediator-token", "tokenValue": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9...", "mediatorId": "beb09338-3c0b-43ab-9b34-6a3fa496abf4", "projectId": "4877d57d-6789-4838-af52-7fe9af353c9e", "createdBy": "8df4696c-fffd-43c9-8df6-c80921471efb", "createdByEmail": "user@example.com", "state": "active", "createdAt": "2026-05-26T19:16:45Z" } ``` | Field | Description | |---|---| | `id` | Unique identifier for this token record | | `name` | The name you provided | | `tokenValue` | The token secret. Use this as `mediator.gateway.token` in Helm, `mediator.secretName` with a Kubernetes secret, or `CAUSELY_GATEWAY_TOKEN` in Docker/Nomad deployments | | `mediatorId` | ID of the mediator this token is associated with | | `projectId` | ID of the project this token belongs to | | `createdBy` | User ID of the token creator | | `createdByEmail` | Email of the token creator | | `state` | Token status (`active`) | | `createdAt` | ISO 8601 creation timestamp | :::note Using `tokenValue` in deployments The `tokenValue` from the response is the gateway token referenced throughout the [installation docs](/installation). For example: ```bash # Helm --set mediator.gateway.token="${TOKEN_VALUE}" # Docker CAUSELY_GATEWAY_TOKEN="${TOKEN_VALUE}" ``` For Kubernetes deployments, you can store the token in a secret instead of specifying it in Helm values. This is recommended for production and GitOps workflows. See [Using a Kubernetes Secret for the Access Token](/installation/customize#using-a-kubernetes-secret-for-the-access-token) for setup instructions. ::: ## List Gateway Tokens **`GET https://api.causely.app/api/tokens`** Returns all gateway tokens for the project, including both active and revoked tokens. ```bash curl -s -X GET https://api.causely.app/api/tokens \ -H "Authorization: Bearer ${CAUSELY_ACCESS_TOKEN}" ``` ```python def list_gateway_tokens(access_token, url="https://api.causely.app/api/tokens"): res = requests.get( url, headers={"Authorization": f"Bearer {access_token}"}, ) if res.status_code != 200: raise requests.HTTPError(f"List tokens failed ({res.status_code}): {res.text}") return res.json() # Example usage: if __name__ == "__main__": cid = os.getenv("CAUSELY_CLIENT_ID") secret = os.getenv("CAUSELY_CLIENT_SECRET") if not cid or not secret: raise EnvironmentError("Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET.") access_token = get_causely_access_token(cid, secret) tokens = list_gateway_tokens(access_token) for token in tokens: print(f"{token['name']} ({token['state']}): {token['id']}") ``` ```javascript const https = require('https'); function listGatewayTokens(accessToken, url = 'https://api.causely.app/api/tokens') { return new Promise((resolve, reject) => { const parsedUrl = new URL(url); const req = https.request( { hostname: parsedUrl.hostname, path: parsedUrl.pathname, method: 'GET', headers: { Authorization: `Bearer ${accessToken}`, }, }, (res) => { let body = ''; res.on('data', (chunk) => (body += chunk)); res.on('end', () => { if (res.statusCode !== 200) { return reject(new Error(`HTTP ${res.statusCode}: ${body}`)); } resolve(JSON.parse(body)); }); }, ); req.on('error', reject); req.end(); }); } // Example usage const { CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET } = process.env; if (!CAUSELY_CLIENT_ID || !CAUSELY_CLIENT_SECRET) { console.error('Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET'); process.exit(1); } (async () => { try { const accessToken = await getCauselyAccessToken(CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET); const tokens = await listGatewayTokens(accessToken); tokens.forEach((t) => console.log(`${t.name} (${t.state}): ${t.id}`)); } catch (err) { console.error('Error:', err.message); process.exit(1); } })(); ``` ```go func listGatewayTokens(accessToken, url string) ([]GatewayToken, error) { req, _ := http.NewRequest("GET", url, nil) req.Header.Set("Authorization", "Bearer "+accessToken) res, err := http.DefaultClient.Do(req) if err != nil { return nil, err } defer res.Body.Close() body, _ := io.ReadAll(res.Body) if res.StatusCode != 200 { return nil, fmt.Errorf("status %d: %s", res.StatusCode, body) } var tokens []GatewayToken if err := json.Unmarshal(body, &tokens); err != nil { return nil, err } return tokens, nil } ``` **Example Response (200 OK):** ```json [ { "id": "afce85cb-70b4-4624-8198-d38843a7ccc0", "name": "my-mediator-token", "tokenValue": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9...", "mediatorId": "beb09338-3c0b-43ba-9b34-6a3fa496abf4", "projectId": "4877d57d-6789-4838-af52-7fe9af353c9e", "createdBy": "8df4696c-fffd-43c9-8df6-c80921471efb", "createdByEmail": "user@example.com", "state": "active", "createdAt": "2026-06-03T22:11:39Z" }, { "id": "5152a4b0-327b-46f5-8926-5716bbdd2bed", "name": "old-mediator-token", "mediatorId": "beb09338-3c0b-43ba-9b34-6a3fa496abf4", "projectId": "4877d57d-6789-4838-af52-7fe9af353c9e", "createdBy": "f67341c8-12cf-49bc-b9c7-09f44885750c", "createdByEmail": "user@example.com", "state": "revoked", "revokedAt": "2026-05-26T21:36:29Z", "revokedBy": "8df4696c-fffd-43c9-8df6-c80921471efb", "createdAt": "2026-05-26T21:24:11Z" } ] ``` The response is an array of token objects. Each object contains the same fields as the [create response](#create-a-gateway-token). Revoked tokens include additional fields: | Field | Description | |---|---| | `id` | Unique identifier for this token record | | `name` | The name assigned to the token | | `tokenValue` | The token secret. Only present on active tokens | | `mediatorId` | ID of the mediator this token is associated with | | `projectId` | ID of the project this token belongs to | | `createdBy` | User ID of the token creator | | `createdByEmail` | Email of the token creator | | `state` | Token status: `active` or `revoked` | | `createdAt` | ISO 8601 creation timestamp | | `revokedAt` | ISO 8601 timestamp of when the token was revoked (revoked tokens only) | | `revokedBy` | User ID of who performed the revocation (revoked tokens only) | ## Revoke a Gateway Token **`POST https://api.causely.app/api/tokens/{tokenId}/revoke?tokenId={tokenId}`** Revoking a token immediately deactivates it. The mediator using that token will lose connectivity to the platform. ```bash TOKEN_ID= curl -s -X POST "https://api.causely.app/api/tokens/${TOKEN_ID}/revoke?tokenId=${TOKEN_ID}" \ -H "Authorization: Bearer ${CAUSELY_ACCESS_TOKEN}" ``` ```python def revoke_gateway_token(access_token, token_id, base_url="https://api.causely.app/api/tokens"): url = f"{base_url}/{token_id}/revoke" res = requests.post( url, params={"tokenId": token_id}, headers={"Authorization": f"Bearer {access_token}"}, ) if res.status_code != 200: raise requests.HTTPError(f"Token revocation failed ({res.status_code}): {res.text}") return res.json() # Example usage: if __name__ == "__main__": cid = os.getenv("CAUSELY_CLIENT_ID") secret = os.getenv("CAUSELY_CLIENT_SECRET") token_id = os.getenv("TOKEN_ID") if not cid or not secret or not token_id: raise EnvironmentError("Missing CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET, or TOKEN_ID.") access_token = get_causely_access_token(cid, secret) result = revoke_gateway_token(access_token, token_id) print("Token state:", result["state"]) ``` ```javascript const https = require('https'); function revokeGatewayToken(accessToken, tokenId, baseUrl = 'https://api.causely.app/api/tokens') { return new Promise((resolve, reject) => { const url = new URL(`${baseUrl}/${tokenId}/revoke`); url.searchParams.set('tokenId', tokenId); const req = https.request( { hostname: url.hostname, path: url.pathname + url.search, method: 'POST', headers: { Authorization: `Bearer ${accessToken}`, }, }, (res) => { let body = ''; res.on('data', (chunk) => (body += chunk)); res.on('end', () => { if (res.statusCode !== 200) { return reject(new Error(`HTTP ${res.statusCode}: ${body}`)); } resolve(JSON.parse(body)); }); }, ); req.on('error', reject); req.end(); }); } // Example usage const { CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET, TOKEN_ID } = process.env; if (!CAUSELY_CLIENT_ID || !CAUSELY_CLIENT_SECRET || !TOKEN_ID) { console.error('Missing CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET, or TOKEN_ID'); process.exit(1); } (async () => { try { const accessToken = await getCauselyAccessToken(CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET); const result = await revokeGatewayToken(accessToken, TOKEN_ID); console.log('Token state:', result.state); } catch (err) { console.error('Error:', err.message); process.exit(1); } })(); ``` ```go func revokeGatewayToken(accessToken, tokenID, baseURL string) (*GatewayToken, error) { url := fmt.Sprintf("%s/%s/revoke?tokenId=%s", baseURL, tokenID, tokenID) req, _ := http.NewRequest("POST", url, nil) req.Header.Set("Authorization", "Bearer "+accessToken) res, err := http.DefaultClient.Do(req) if err != nil { return nil, err } defer res.Body.Close() body, _ := io.ReadAll(res.Body) if res.StatusCode != 200 { return nil, fmt.Errorf("status %d: %s", res.StatusCode, body) } var token GatewayToken if err := json.Unmarshal(body, &token); err != nil { return nil, err } return &token, nil } ``` **Example Response (200 OK):** ```json { "id": "8323a17c-f6f5-4a7a-b04e-481790314b03", "name": "my-mediator-token", "mediatorId": "beb09338-3c0b-43ba-9b34-6a3fa496abf4", "projectId": "4877d57d-6789-4838-af52-7fe9af353c9e", "createdBy": "8df4696c-fffd-43c9-8df6-c80921471efb", "createdByEmail": "user@example.com", "state": "revoked", "revokedAt": "2026-05-26T19:42:26Z", "revokedBy": "8df4696c-fffd-43c9-8df6-c80921471efb", "createdAt": "2026-05-26T19:42:09Z" } ``` | Field | Description | |---|---| | `state` | `revoked` once the token has been invalidated | | `revokedAt` | ISO 8601 timestamp of when the token was revoked | | `revokedBy` | User ID of who performed the revocation | --- ## Setting Up GraphQL Clients To maximize efficiency and code reusability when integrating with the Causely GraphQL API, we recommend setting up proper GraphQL client libraries rather than using raw HTTP requests. This approach provides better error handling, type safety, query validation, and cleaner code that can be reused across multiple API operations. ## Install GraphQL Client Libraries Install the recommended GraphQL client libraries for your programming language to enable robust API integration with automatic query validation and enhanced developer experience: ```bash pip install gql[all] requests ``` ```bash npm install @apollo/client graphql ``` ```bash # Install gq (GraphQL CLI tool) curl -sSL https://github.com/houseabsolute/gq/releases/latest/download/gq-Linux-x86_64.tar.gz | tar xz # Or use your package manager, e.g., brew install gq ``` ```bash go get github.com/machinebox/graphql ``` ## Create Reusable GraphQL Client Functions Implement these language-specific GraphQL client functions that integrate seamlessly with the authentication methods from the previous section. These functions provide a clean, reusable interface for executing any GraphQL query against the Causely API: ```python from gql import gql, Client from gql.transport.requests import RequestsHTTPTransport def create_graphql_client(access_token): """Create a GraphQL client with authentication.""" transport = RequestsHTTPTransport( url="https://api.causely.app/query/", headers={"Authorization": f"Bearer {access_token}"} ) return Client(transport=transport, fetch_schema_from_transport=True) def post_query(client, query_string, variables=None): """Execute a GraphQL query using the client.""" query = gql(query_string) return client.execute(query, variable_values=variables) # Usage example: # client = create_graphql_client(get_causely_access_token(client_id, secret)) # result = post_query(client, query_string, variables) ``` ```javascript const { ApolloClient, InMemoryCache, createHttpLink, gql } = require('@apollo/client'); function createGraphQLClient(accessToken) { const httpLink = createHttpLink({ uri: 'https://api.causely.app/query/', headers: { Authorization: `Bearer ${accessToken}`, }, }); return new ApolloClient({ link: httpLink, cache: new InMemoryCache(), }); } async function postQuery(client, queryString, variables = {}) { const result = await client.query({ query: gql(queryString), variables, fetchPolicy: 'no-cache', }); return result.data; } // Usage example: // const client = createGraphQLClient(await getCauselyAccessToken(clientId, secret)); // const result = await postQuery(client, queryString, variables); ``` ```bash # Function to execute GraphQL queries using gq post_query() { local query_file="$1" local variables_file="$2" gq https://api.causely.app/query/ \ --header "Authorization: Bearer ${CAUSELY_ACCESS_TOKEN}" \ --query-file "$query_file" \ --variables-file "$variables_file" } # Alternative using curl with GraphQL post_query_curl() { local query="$1" local variables="$2" curl -X POST https://api.causely.app/query/ \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${CAUSELY_ACCESS_TOKEN}" \ -d "{\"query\": \"$query\", \"variables\": $variables}" } ``` ```go package main "context" "github.com/machinebox/graphql" ) func createGraphQLClient(accessToken string) *graphql.Client { client := graphql.NewClient("https://api.causely.app/query/") client.WithHTTPHeader("Authorization", "Bearer "+accessToken) return client } func postQuery(client *graphql.Client, queryString string, variables map[string]interface{}) (interface{}, error) { req := graphql.NewRequest(queryString) // Add variables to the request for key, value := range variables { req.Var(key, value) } var response interface{} err := client.Run(context.Background(), req, &response) return response, err } // Usage example: // client := createGraphQLClient(token) // result, err := postQuery(client, queryString, variables) ``` --- ## Query Defects and Root Causes This query example demonstrates how to retrieve the first 20 high-severity, active defects and root causes from the Causely platform using the GraphQL utilities defined above. :::note The START_TIME is set to the current time in UTC. You may need to adjust the timestamp to your needs. ::: :::tip Prerequisite This example reuses the helper utilities defined in the [Authentication](/api/authentication) and [GraphQL Clients](/api/graphql-clients) sections—fetching an access token, creating the GraphQL client, and sending a request with the `post_query` wrapper (CLI, Python, or Go). Keep those helpers in scope before running this query. ::: ```python from datetime import datetime, timezone if __name__ == "__main__": # Get credentials from environment cid, secret = os.getenv("CAUSELY_CLIENT_ID"), os.getenv("CAUSELY_CLIENT_SECRET") if not cid or not secret: raise EnvironmentError("Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET") # Get access token and create GraphQL client token = get_causely_access_token(cid, secret) client = create_graphql_client(token) # Prepare query variables start_time = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%S") + ".000+00:00" variables = { "defectFilter": { "state": "ACTIVE", "severities": ["Critical", "High"], "startTime": start_time, "scopesFilter": {"scopes": []}, "entityTypes": [], "defectNames": [], "entityName": "" }, "groupRecurring": True, "first": 20 } # Execute the query query = """ fragment LabelFragment on Label { key value __typename } fragment BasicEntityWithLabelsFragment on Entity { id name typeName labels { ...LabelFragment __typename } __typename } fragment DefectFragment on Defect { id name fromTime toTime remediated serviceCount entityId entityType entity { ...BasicEntityWithLabelsFragment __typename } activeCount missingCount severity __typename } fragment DefectRelatedOccurrencesFragment on Defect { relatedOccurrrences { id name fromTime toTime entity { id name typeName __typename } severity __typename } __typename } fragment BasicEntityFragment on Entity { id name typeName __typename } fragment SymptomFragmentWithoutLabels on Symptom { id name active state entityId entityType fromTime toTime isPropagated entity { ...BasicEntityFragment __typename } __typename } fragment EventFragmentWithoutLabels on Event { id name active time entity { ...BasicEntityFragment __typename } __typename } query defectConnection($defectFilter: DefectFilter, $groupRecurring: Boolean, $first: Int, $after: String, $last: Int, $before: String) { defectConnection(defectFilter: $defectFilter groupRecurring: $groupRecurring first: $first after: $after last: $last before: $before) { totalCount edges { node { ...DefectFragment ...DefectRelatedOccurrencesFragment symptoms { ...SymptomFragmentWithoutLabels __typename } events { ...EventFragmentWithoutLabels __typename } __typename } cursor __typename } pageInfo { hasNextPage hasPreviousPage startCursor endCursor totalCount __typename } __typename } } """ result = post_query(client, query, variables) print(result) ``` ```javascript const { CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET } = process.env; if (!CAUSELY_CLIENT_ID || !CAUSELY_CLIENT_SECRET) { console.error('Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET'); process.exit(1); } (async () => { try { // Get access token and create GraphQL client const token = await getCauselyAccessToken(CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET); const client = createGraphQLClient(token); // Prepare query variables const startTime = new Date().toISOString().replace(/\.\d+Z$/, '.000+00:00'); const variables = { defectFilter: { state: 'ACTIVE', severities: ['Critical', 'High'], startTime, scopesFilter: { scopes: [] }, entityTypes: [], defectNames: [], entityName: '', }, groupRecurring: true, first: 20, }; // Execute the query const query = ` fragment LabelFragment on Label { key value __typename } fragment BasicEntityWithLabelsFragment on Entity { id name typeName labels { ...LabelFragment __typename } __typename } fragment DefectFragment on Defect { id name fromTime toTime remediated serviceCount entityId entityType entity { ...BasicEntityWithLabelsFragment __typename } activeCount missingCount severity __typename } fragment DefectRelatedOccurrencesFragment on Defect { relatedOccurrrences { id name fromTime toTime entity { id name typeName __typename } severity __typename } __typename } fragment BasicEntityFragment on Entity { id name typeName __typename } fragment SymptomFragmentWithoutLabels on Symptom { id name active state entityId entityType fromTime toTime isPropagated entity { ...BasicEntityFragment __typename } __typename } fragment EventFragmentWithoutLabels on Event { id name active time entity { ...BasicEntityFragment __typename } __typename } query defectConnection($defectFilter: DefectFilter, $groupRecurring: Boolean, $first: Int, $after: String, $last: Int, $before: String) { defectConnection(defectFilter: $defectFilter groupRecurring: $groupRecurring first: $first after: $after last: $last before: $before) { totalCount edges { node { ...DefectFragment ...DefectRelatedOccurrencesFragment symptoms { ...SymptomFragmentWithoutLabels __typename } events { ...EventFragmentWithoutLabels __typename } __typename } cursor __typename } pageInfo { hasNextPage hasPreviousPage startCursor endCursor totalCount __typename } __typename } } `; const result = await postQuery(client, query, variables); console.log(JSON.stringify(result, null, 2)); } catch (err) { console.error('Error:', err.message); process.exit(1); } })(); ``` ```bash # Set up credentials and get token export CAUSELY_CLIENT_ID= export CAUSELY_CLIENT_SECRET= export CAUSELY_ACCESS_TOKEN=$(response=$(curl -s -w "\n%{http_code}" -X POST https://auth.causely.app/frontegg/identity/resources/auth/v2/api-token \ -H "Content-Type: application/json" \ -d "{\"clientId\": \"${CAUSELY_CLIENT_ID}\", \"secret\": \"${CAUSELY_CLIENT_SECRET}\"}"); \ http_code=$(echo "$response" | tail -n1); \ body=$(echo "$response" | sed '$d'); \ if [ "$http_code" = "200" ]; then echo "$body" | jq -r .access_token; else echo "$body" >&2; false; fi) # Create query and variables files cat > defect_query.graphql << 'EOF' fragment LabelFragment on Label { key value __typename } fragment BasicEntityWithLabelsFragment on Entity { id name typeName labels { ...LabelFragment __typename } __typename } fragment DefectFragment on Defect { id name fromTime toTime remediated serviceCount entityId entityType entity { ...BasicEntityWithLabelsFragment __typename } activeCount missingCount severity __typename } fragment DefectRelatedOccurrencesFragment on Defect { relatedOccurrrences { id name fromTime toTime entity { id name typeName __typename } severity __typename } __typename } fragment BasicEntityFragment on Entity { id name typeName __typename } fragment SymptomFragmentWithoutLabels on Symptom { id name active state entityId entityType fromTime toTime isPropagated entity { ...BasicEntityFragment __typename } __typename } fragment EventFragmentWithoutLabels on Event { id name active time entity { ...BasicEntityFragment __typename } __typename } query defectConnection($defectFilter: DefectFilter, $groupRecurring: Boolean, $first: Int, $after: String, $last: Int, $before: String) { defectConnection(defectFilter: $defectFilter groupRecurring: $groupRecurring first: $first after: $after last: $last before: $before) { totalCount edges { node { ...DefectFragment ...DefectRelatedOccurrencesFragment symptoms { ...SymptomFragmentWithoutLabels __typename } events { ...EventFragmentWithoutLabels __typename } __typename } cursor __typename } pageInfo { hasNextPage hasPreviousPage startCursor endCursor totalCount __typename } __typename } } EOF START_TIME="$(TZ=UTC date +%Y-%m-%dT%H:%M:%S).000+00:00" cat > variables.json << EOF { "defectFilter": { "state": "ACTIVE", "severities": ["Critical", "High"], "startTime": "${START_TIME}", "scopesFilter": {"scopes": []}, "entityTypes": [], "defectNames": [], "entityName": "" }, "groupRecurring": true, "first": 20 } EOF # Execute the query using gq (if available) or curl if command -v gq &> /dev/null; then post_query "defect_query.graphql" "variables.json" else query=$(cat defect_query.graphql | tr '\n' ' ') variables=$(cat variables.json) post_query_curl "$query" "$variables" fi # Clean up temporary files rm -f defect_query.graphql variables.json ``` ```go package main "fmt" "os" "time" ) func main() { // Get credentials from environment id, secret := os.Getenv("CAUSELY_CLIENT_ID"), os.Getenv("CAUSELY_CLIENT_SECRET") if id == "" || secret == "" { fmt.Fprintln(os.Stderr, "Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET") os.Exit(1) } // Get access token and create GraphQL client token, err := getToken(id, secret, "https://auth.causely.app/frontegg/identity/resources/auth/v2/api-token") if err != nil { fmt.Fprintln(os.Stderr, "Error getting token:", err) os.Exit(1) } client := createGraphQLClient(token) // Prepare query variables startTime := time.Now().UTC().Format("2006-01-02T15:04:05") + ".000+00:00" variables := map[string]interface{}{ "defectFilter": map[string]interface{}{ "state": "ACTIVE", "severities": []string{"Critical", "High"}, "startTime": startTime, "scopesFilter": map[string]interface{}{"scopes": []string{}}, "entityTypes": []string{}, "defectNames": []string{}, "entityName": "", }, "groupRecurring": true, "first": 20, } // Execute the query query := `fragment LabelFragment on Label { key value __typename } fragment BasicEntityWithLabelsFragment on Entity { id name typeName labels { ...LabelFragment __typename } __typename } fragment DefectFragment on Defect { id name fromTime toTime remediated serviceCount entityId entityType entity { ...BasicEntityWithLabelsFragment __typename } activeCount missingCount severity __typename } fragment DefectRelatedOccurrencesFragment on Defect { relatedOccurrrences { id name fromTime toTime entity { id name typeName __typename } severity __typename } __typename } fragment BasicEntityFragment on Entity { id name typeName __typename } fragment SymptomFragmentWithoutLabels on Symptom { id name active state entityId entityType fromTime toTime isPropagated entity { ...BasicEntityFragment __typename } __typename } fragment EventFragmentWithoutLabels on Event { id name active time entity { ...BasicEntityFragment __typename } __typename } query defectConnection($defectFilter: DefectFilter, $groupRecurring: Boolean, $first: Int, $after: String, $last: Int, $before: String) { defectConnection(defectFilter: $defectFilter groupRecurring: $groupRecurring first: $first after: $after last: $last before: $before) { totalCount edges { node { ...DefectFragment ...DefectRelatedOccurrencesFragment symptoms { ...SymptomFragmentWithoutLabels __typename } events { ...EventFragmentWithoutLabels __typename } __typename } cursor __typename } pageInfo { hasNextPage hasPreviousPage startCursor endCursor totalCount __typename } __typename } }` result, err := postQuery(client, query, variables) if err != nil { fmt.Fprintln(os.Stderr, "Error running query:", err) os.Exit(1) } fmt.Println(result) } ``` First, let's define the GraphQL query and variables: ```graphql fragment LabelFragment on Label { key value __typename } fragment BasicEntityWithLabelsFragment on Entity { id name typeName labels { ...LabelFragment __typename } __typename } fragment DefectFragment on Defect { id name fromTime toTime remediated serviceCount entityId entityType entity { ...BasicEntityWithLabelsFragment __typename } activeCount missingCount severity __typename } fragment DefectRelatedOccurrencesFragment on Defect { relatedOccurrrences { id name fromTime toTime entity { id name typeName __typename } severity __typename } __typename } fragment BasicEntityFragment on Entity { id name typeName __typename } fragment SymptomFragmentWithoutLabels on Symptom { id name active state entityId entityType fromTime toTime isPropagated entity { ...BasicEntityFragment __typename } __typename } fragment EventFragmentWithoutLabels on Event { id name active time entity { ...BasicEntityFragment __typename } __typename } query defectConnection( $defectFilter: DefectFilter $groupRecurring: Boolean $first: Int $after: String $last: Int $before: String ) { defectConnection( defectFilter: $defectFilter groupRecurring: $groupRecurring first: $first after: $after last: $last before: $before ) { totalCount edges { node { ...DefectFragment ...DefectRelatedOccurrencesFragment symptoms { ...SymptomFragmentWithoutLabels __typename } events { ...EventFragmentWithoutLabels __typename } __typename } cursor __typename } pageInfo { hasNextPage hasPreviousPage startCursor endCursor totalCount __typename } __typename } } ``` --- ## Get User Scopes This query retrieves user scopes and access permissions for the authenticated user. User scopes define which resources and data the current user can access within the Causely platform. :::tip Prerequisite This example reuses the helper utilities defined in the [Authentication](/api/authentication) and [GraphQL Clients](/api/graphql-clients) sections—fetching an access token, creating the GraphQL client, and sending a request with the `post_query` wrapper (CLI, Python, or Go). Keep those helpers in scope before running this query. ::: ```python if __name__ == "__main__": # Get credentials from environment cid, secret = os.getenv("CAUSELY_CLIENT_ID"), os.getenv("CAUSELY_CLIENT_SECRET") if not cid or not secret: raise EnvironmentError("Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET") # Get access token and create GraphQL client token = get_causely_access_token(cid, secret) client = create_graphql_client(token) # Prepare query variables variables = { "filter": { "nameExpr": "" } } # Execute the query query = """ query getUserScopes($filter: UserScopeFilter, $first: Int, $after: String, $last: Int, $before: String) { getUserScopes( filter: $filter first: $first after: $after last: $last before: $before ) { totalCount edges { node { id name audience ownerId lastUpdate scopes { typeName typeValues __typename } __typename } cursor __typename } pageInfo { hasNextPage hasPreviousPage startCursor endCursor totalCount __typename } __typename } } """ result = post_query(client, query, variables) print(result) ``` ```javascript const { CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET } = process.env; if (!CAUSELY_CLIENT_ID || !CAUSELY_CLIENT_SECRET) { console.error('Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET'); process.exit(1); } (async () => { try { // Get access token and create GraphQL client const token = await getCauselyAccessToken(CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET); const client = createGraphQLClient(token); // Prepare query variables const variables = { filter: { nameExpr: '', }, }; // Execute the query const query = ` query getUserScopes($filter: UserScopeFilter, $first: Int, $after: String, $last: Int, $before: String) { getUserScopes( filter: $filter first: $first after: $after last: $last before: $before ) { totalCount edges { node { id name audience ownerId lastUpdate scopes { typeName typeValues __typename } __typename } cursor __typename } pageInfo { hasNextPage hasPreviousPage startCursor endCursor totalCount __typename } __typename } } `; const result = await postQuery(client, query, variables); console.log(JSON.stringify(result, null, 2)); } catch (err) { console.error('Error:', err.message); process.exit(1); } })(); ``` ```bash # Set up credentials and get token export CAUSELY_CLIENT_ID= export CAUSELY_CLIENT_SECRET= export CAUSELY_ACCESS_TOKEN=$(response=$(curl -s -w "\n%{http_code}" -X POST https://auth.causely.app/frontegg/identity/resources/auth/v2/api-token \ -H "Content-Type: application/json" \ -d "{\"clientId\": \"${CAUSELY_CLIENT_ID}\", \"secret\": \"${CAUSELY_CLIENT_SECRET}\"}"); \ http_code=$(echo "$response" | tail -n1); \ body=$(echo "$response" | sed '$d'); \ if [ "$http_code" = "200" ]; then echo "$body" | jq -r .access_token; else echo "$body" >&2; false; fi) # Create query and variables files cat > user_scopes_query.graphql << 'EOF' query getUserScopes($filter: UserScopeFilter, $first: Int, $after: String, $last: Int, $before: String) { getUserScopes( filter: $filter first: $first after: $after last: $last before: $before ) { totalCount edges { node { id name audience ownerId lastUpdate scopes { typeName typeValues __typename } __typename } cursor __typename } pageInfo { hasNextPage hasPreviousPage startCursor endCursor totalCount __typename } __typename } } EOF cat > user_scopes_variables.json << EOF { "filter": { "nameExpr": "" } } EOF # Execute the query using gq (if available) or curl if command -v gq &> /dev/null; then post_query "user_scopes_query.graphql" "user_scopes_variables.json" else query=$(cat user_scopes_query.graphql | tr '\n' ' ') variables=$(cat user_scopes_variables.json) post_query_curl "$query" "$variables" fi # Clean up temporary files rm -f user_scopes_query.graphql user_scopes_variables.json ``` ```go package main "fmt" "os" ) func main() { // Get credentials from environment id, secret := os.Getenv("CAUSELY_CLIENT_ID"), os.Getenv("CAUSELY_CLIENT_SECRET") if id == "" || secret == "" { fmt.Fprintln(os.Stderr, "Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET") os.Exit(1) } // Get access token and create GraphQL client token, err := getToken(id, secret, "https://auth.causely.app/frontegg/identity/resources/auth/v2/api-token") if err != nil { fmt.Fprintln(os.Stderr, "Error getting token:", err) os.Exit(1) } client := createGraphQLClient(token) // Prepare query variables variables := map[string]interface{}{ "filter": map[string]interface{}{ "nameExpr": "", }, } // Execute the query query := `query getUserScopes($filter: UserScopeFilter, $first: Int, $after: String, $last: Int, $before: String) { getUserScopes( filter: $filter first: $first after: $after last: $last before: $before ) { totalCount edges { node { id name audience ownerId lastUpdate scopes { typeName typeValues __typename } __typename } cursor __typename } pageInfo { hasNextPage hasPreviousPage startCursor endCursor totalCount __typename } __typename } }` result, err := postQuery(client, query, variables) if err != nil { fmt.Fprintln(os.Stderr, "Error running query:", err) os.Exit(1) } fmt.Println(result) } ``` ```graphql query getUserScopes($filter: UserScopeFilter, $first: Int, $after: String, $last: Int, $before: String) { getUserScopes(filter: $filter, first: $first, after: $after, last: $last, before: $before) { totalCount edges { node { id name audience ownerId lastUpdate scopes { typeName typeValues __typename } __typename } cursor __typename } pageInfo { hasNextPage hasPreviousPage startCursor endCursor totalCount __typename } __typename } } ``` **Variables:** ```json { "filter": { "nameExpr": "" } } ``` --- ## Hiding Root Causes You may want to hide a root cause for a time period if you are not planning on addressing it immediately or if you are performing maintenance and the behavior is expected. The recommended way to hide a specific root cause is to use **Entity Configs** of type `IgnoreRC`. To avoid overwriting existing configuration for that entity, you should: 1. Query the current `IgnoreRC` config for the entity. 2. Merge your new rule with any existing entries. 3. Call the `CreateEntityConfigs` mutation with the combined config. :::tip Prerequisite Each example below reuses the helper utilities defined in the [Authentication](/api/authentication) and [GraphQL Clients](/api/graphql-clients) sections (`get_causely_access_token`, `create_graphql_client`, `post_query`, and their language equivalents). ::: ```python # Reuse helpers defined earlier in this guide: # - get_causely_access_token # - create_graphql_client # - post_query GET_CONFIGS_QUERY = """ query GetEntityConfigs($entityId: String!, $configType: String!) { entityConfigs(entityId: $entityId, configType: $configType) { entityId type data } } """ CREATE_CONFIGS_MUTATION = """ mutation CreateEntityConfigs($entityConfigs: [EntityConfigInput!]!) { createEntityConfigs(entityConfigs: $entityConfigs) { entityId type data } } """ def load_existing_ignore_rc_configs(client, entity_id, config_type="IgnoreRC"): """Fetch existing IgnoreRC configs and return them as a Python list.""" variables = {"entityId": entity_id, "configType": config_type} result = post_query(client, GET_CONFIGS_QUERY, variables) entries = [] for cfg in result.get("entityConfigs", []): data_str = cfg.get("data") or "[]" try: entries.extend(json.loads(data_str)) except json.JSONDecodeError: # If data is malformed, skip rather than failing the whole operation continue return entries def save_ignore_rc_configs(client, entity_id, configs, config_type="IgnoreRC"): """Persist the merged IgnoreRC configs back to Causely.""" variables = { "entityConfigs": [{ "entityId": entity_id, "type": config_type, # If your schema uses ENUMs, pass the enum instead "data": json.dumps(configs), # Must be a JSON string }] } return post_query(client, CREATE_CONFIGS_MUTATION, variables) if __name__ == "__main__": # Get credentials from environment cid, secret = os.getenv("CAUSELY_CLIENT_ID"), os.getenv("CAUSELY_CLIENT_SECRET") if not cid or not secret: raise EnvironmentError("Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET") # Get access token and create GraphQL client token = get_causely_access_token(cid, secret) client = create_graphql_client(token) # Target entity and root cause name you want to hide # Replace these example values with your actual entity ID and root cause name entity_id = "47c29d23-9f57-5d47-9f91-a72815edc8cf" defect_name_to_hide = "Malfunction" # 1) Load existing IgnoreRC configuration (if any) existing_configs = load_existing_ignore_rc_configs(client, entity_id) # 2) Add or update the entry for this root cause without losing existing ones new_entry = { "DefectName": defect_name_to_hide, # Set an appropriate expiry time for the ignore rule "UntilTime": "2025-11-15T10:51:00-04:00", } # Avoid adding duplicate entries for the same defect name merged_configs = [ cfg for cfg in existing_configs if cfg.get("DefectName") != defect_name_to_hide ] merged_configs.append(new_entry) # 3) Save the merged configuration back using the mutation result = save_ignore_rc_configs(client, entity_id, merged_configs) print(json.dumps(result, indent=2)) ``` ```javascript // Reuses helpers defined earlier in this guide: // - getCauselyAccessToken // - createGraphQLClient // - postQuery (for queries) const { gql } = require('@apollo/client'); const { CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET } = process.env; const GET_CONFIGS_QUERY = ` query GetEntityConfigs($entityId: String!, $configType: String!) { entityConfigs(entityId: $entityId, configType: $configType) { entityId type data } } `; const CREATE_CONFIGS_MUTATION = ` mutation CreateEntityConfigs($entityConfigs: [EntityConfigInput!]!) { createEntityConfigs(entityConfigs: $entityConfigs) { entityId type data } } `; async function loadExistingIgnoreRcConfigs(client, entityId, configType = 'IgnoreRC') { const variables = { entityId, configType }; const data = await postQuery(client, GET_CONFIGS_QUERY, variables); const entries = []; for (const cfg of data.entityConfigs || []) { const dataStr = cfg.data || '[]'; try { entries.push(...JSON.parse(dataStr)); } catch { // Skip malformed data instead of failing the whole operation continue; } } return entries; } async function saveIgnoreRcConfigs(client, entityId, configs, configType = 'IgnoreRC') { const variables = { entityConfigs: [ { entityId, type: configType, // If your schema uses ENUMs, pass the enum instead data: JSON.stringify(configs), // Must be a JSON string }, ], }; const result = await client.mutate({ mutation: gql(CREATE_CONFIGS_MUTATION), variables, }); return result.data; } (async () => { if (!CAUSELY_CLIENT_ID || !CAUSELY_CLIENT_SECRET) { console.error('Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET'); process.exit(1); } // Get access token and create GraphQL client const token = await getCauselyAccessToken(CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET); const client = createGraphQLClient(token); // Replace these example values with your actual entity ID and root cause name const entityId = '47c29d23-9f57-5d47-9f91-a72815edc8cf'; const defectNameToHide = 'Malfunction'; // 1) Load existing IgnoreRC configuration (if any) const existingConfigs = await loadExistingIgnoreRcConfigs(client, entityId); // 2) Add or update the entry for this root cause without losing existing ones const newEntry = { DefectName: defectNameToHide, // Set an appropriate expiry time for the ignore rule UntilTime: '2025-11-15T10:51:00-04:00', }; const mergedConfigs = existingConfigs.filter((cfg) => cfg.DefectName !== defectNameToHide); mergedConfigs.push(newEntry); // 3) Save the merged configuration back using the mutation const result = await saveIgnoreRcConfigs(client, entityId, mergedConfigs); console.log(JSON.stringify(result, null, 2)); })(); ``` ```bash #!/usr/bin/env bash set -euo pipefail # Prerequisites: # - CAUSELY_CLIENT_ID / CAUSELY_CLIENT_SECRET exported # - CAUSELY_ACCESS_TOKEN exported (see Authentication section) # - post_query / post_query_curl functions defined (see Bash GraphQL helpers section) # - jq installed # Replace these example values with your actual entity ID, root cause name and time ENTITY_ID="47c29d23-9f57-5d47-9f91-a72815edc8cf" DEFECT_NAME_TO_HIDE="Malfunction" UNTIL_TIME="2025-11-15T10:51:00-04:00" # 1) Define GraphQL query and mutation cat > get_entity_configs.graphql << 'EOF' query GetEntityConfigs($entityId: String!, $configType: String!) { entityConfigs(entityId: $entityId, configType: $configType) { entityId type data } } EOF cat > create_entity_configs.graphql << 'EOF' mutation CreateEntityConfigs($entityConfigs: [EntityConfigInput!]!) { createEntityConfigs(entityConfigs: $entityConfigs) { entityId type data } } EOF # 2) Query existing IgnoreRC config cat > get_entity_configs_variables.json << EOF { "entityId": "${ENTITY_ID}", "configType": "IgnoreRC" } EOF if command -v gq &> /dev/null; then RAW_RESPONSE=$(gq https://api.causely.app/query/ \ --header "Authorization: Bearer ${CAUSELY_ACCESS_TOKEN}" \ --query-file get_entity_configs.graphql \ --variables-file get_entity_configs_variables.json) else QUERY=$(tr '\n' ' ' < get_entity_configs.graphql) VARIABLES=$(cat get_entity_configs_variables.json) RAW_RESPONSE=$(post_query_curl "$QUERY" "$VARIABLES") fi # 3) Extract current config entries (data is a JSON string) CURRENT_CONFIGS=$(echo "${RAW_RESPONSE}" \ | jq -r '.data.entityConfigs[0].data // "[]"' \ | jq -c 'try fromjson catch []') # 4) Merge in the new ignore rule (avoid duplicates on DefectName) MERGED_CONFIGS=$( jq -c --arg name "${DEFECT_NAME_TO_HIDE}" --arg until "${UNTIL_TIME}" ' (map(select(.DefectName != $name))) + [ { "DefectName": $name, "UntilTime": $until } ] ' <<< "${CURRENT_CONFIGS}" ) # 5) Build variables for CreateEntityConfigs (data must be a JSON string) DATA_STRING=$(jq -Rs . <<< "${MERGED_CONFIGS}") cat > create_entity_configs_variables.json << EOF { "entityConfigs": [ { "entityId": "${ENTITY_ID}", "type": "IgnoreRC", "data": ${DATA_STRING} } ] } EOF # 6) Call the mutation if command -v gq &> /dev/null; then gq https://api.causely.app/query/ \ --header "Authorization: Bearer ${CAUSELY_ACCESS_TOKEN}" \ --query-file create_entity_configs.graphql \ --variables-file create_entity_configs_variables.json else MUTATION=$(tr '\n' ' ' < create_entity_configs.graphql) VARIABLES=$(cat create_entity_configs_variables.json) post_query_curl "$MUTATION" "$VARIABLES" fi # 7) Cleanup rm -f get_entity_configs.graphql create_entity_configs.graphql \ get_entity_configs_variables.json create_entity_configs_variables.json ``` ```go package main "encoding/json" "fmt" "os" ) // Reuse helpers defined earlier: // - getToken(id, secret, url) // - createGraphQLClient(accessToken string) *graphql.Client // - postQuery(client *graphql.Client, queryString string, variables map[string]interface{}) (interface{}, error) const getConfigsQuery = ` query GetEntityConfigs($entityId: String!, $configType: String!) { entityConfigs(entityId: $entityId, configType: $configType) { entityId type data } } ` const createConfigsMutation = ` mutation CreateEntityConfigs($entityConfigs: [EntityConfigInput!]!) { createEntityConfigs(entityConfigs: $entityConfigs) { entityId type data } } ` type entityConfigsResponse struct { EntityConfigs []struct { EntityID string `json:"entityId"` Type string `json:"type"` Data string `json:"data"` } `json:"entityConfigs"` } func loadExistingIgnoreRCConfigs(gqlClient *graphql.Client, entityID string, configType string) ([]map[string]interface{}, error) { variables := map[string]interface{}{ "entityId": entityID, "configType": configType, } raw, err := postQuery(gqlClient, getConfigsQuery, variables) if err != nil { return nil, err } // postQuery returns interface{}; marshal + unmarshal into a typed struct bytes, err := json.Marshal(raw) if err != nil { return nil, err } var wrapped struct { Data entityConfigsResponse `json:"data"` } if err := json.Unmarshal(bytes, &wrapped); err != nil { return nil, err } entries := []map[string]interface{}{} for _, cfg := range wrapped.Data.EntityConfigs { dataStr := cfg.Data if dataStr == "" { continue } var arr []map[string]interface{} if err := json.Unmarshal([]byte(dataStr), &arr); err != nil { // Skip malformed data instead of failing the whole operation continue } entries = append(entries, arr...) } return entries, nil } func saveIgnoreRCConfigs(gqlClient *graphql.Client, entityID string, configs []map[string]interface{}, configType string) (interface{}, error) { dataBytes, err := json.Marshal(configs) if err != nil { return nil, err } variables := map[string]interface{}{ "entityConfigs": []map[string]interface{}{ { "entityId": entityID, "type": configType, // If your schema uses ENUMs, pass the enum instead "data": string(dataBytes), // Must be a JSON string }, }, } return postQuery(gqlClient, createConfigsMutation, variables) } func main() { id, secret := os.Getenv("CAUSELY_CLIENT_ID"), os.Getenv("CAUSELY_CLIENT_SECRET") if id == "" || secret == "" { fmt.Fprintln(os.Stderr, "Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET") os.Exit(1) } token, err := getToken(id, secret, "https://auth.causely.app/frontegg/identity/resources/auth/v2/api-token") if err != nil { fmt.Fprintln(os.Stderr, "Error getting token:", err) os.Exit(1) } client := createGraphQLClient(token) // Replace these example values with your actual entity ID, root cause name and time entityID := "47c29d23-9f57-5d47-9f91-a72815edc8cf" defectNameToHide := "Malfunction" untilTime := "2025-11-15T10:51:00-04:00" // 1) Load existing IgnoreRC configuration (if any) existing, err := loadExistingIgnoreRCConfigs(client, entityID, "IgnoreRC") if err != nil { fmt.Fprintln(os.Stderr, "Error loading configs:", err) os.Exit(1) } // 2) Add or update the entry for this root cause without losing existing ones merged := make([]map[string]interface{}, 0, len(existing)+1) for _, cfg := range existing { if name, ok := cfg["DefectName"].(string); ok && name == defectNameToHide { continue } merged = append(merged, cfg) } merged = append(merged, map[string]interface{}{ "DefectName": defectNameToHide, "UntilTime": untilTime, }) // 3) Save the merged configuration back using the mutation result, err := saveIgnoreRCConfigs(client, entityID, merged, "IgnoreRC") if err != nil { fmt.Fprintln(os.Stderr, "Error saving configs:", err) os.Exit(1) } out, _ := json.MarshalIndent(result, "", " ") fmt.Println(string(out)) } ``` ```graphql # 1) Query existing IgnoreRC config for an entity query GetEntityConfigs($entityId: String!, $configType: String!) { entityConfigs(entityId: $entityId, configType: $configType) { entityId type data } } ``` **Variables:** ```json # Replace this example values with your actual entity ID. { "entityId": "47c29d23-9f57-5d47-9f91-a72815edc8cf", "configType": "IgnoreRC" } ``` The `data` field in the response is a **JSON string** that represents an array of ignore rules, for example: ```json [{ "DefectName": "SomeOtherDefect", "UntilTime": "2025-11-10T12:00:00-04:00" }] ``` You should: 1. Parse this JSON string in your client. 2. Remove any existing entries with the same `DefectName` you want to hide. 3. Append a new entry for the root cause you want to hide. 4. Stringify the merged array and send it back via `CreateEntityConfigs`. ```graphql # 2) Save merged IgnoreRC configuration mutation CreateEntityConfigs($entityConfigs: [EntityConfigInput!]!) { createEntityConfigs(entityConfigs: $entityConfigs) { entityId type data } } ``` **Variables (example after merging):** ```json # Replace this example values with your actual entity ID, root cause name and time { "entityConfigs": [ { "entityId": "47c29d23-9f57-5d47-9f91-a72815edc8cf", "type": "IgnoreRC", "data": "[{\"DefectName\":\"SomeOtherDefect\",\"UntilTime\":\"2025-11-10T12:00:00-04:00\"},{\"DefectName\":\"Malfunction\",\"UntilTime\":\"2025-11-15T10:51:00-04:00\"}]" } ] } ``` > Note: The `data` value must be a **JSON string**, not a nested JSON object. Most clients will use `JSON.stringify` or an equivalent function to produce this string. --- ## Testing Notification Payloads This example demonstrates how to **send a test notification** to Causely using a representative payload. It follows the same pattern as the examples above: reuse your token + GraphQL client helpers and then execute one mutation with a clear `variables` object. :::tip Prerequisite This example reuses the helper utilities defined in the [Authentication](/api/authentication) and [GraphQL Clients](/api/graphql-clients) sections—fetching an access token, creating the GraphQL client, and sending a request with the `post_query` wrapper (CLI, Python, or Go). Keep those helpers in scope before running this mutation. ::: ```python if __name__ == "__main__": # Get credentials from environment cid, secret = os.getenv("CAUSELY_CLIENT_ID"), os.getenv("CAUSELY_CLIENT_SECRET") if not cid or not secret: raise EnvironmentError("Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET") # Get access token and create GraphQL client token = get_causely_access_token(cid, secret) client = create_graphql_client(token) # Prepare mutation mutation = """ mutation createNotification($notification: NotificationInput!) { createNotification(notification: $notification) } """ # Build the payload embedded_payload = { "link": "https://portal.causely.app", "name": "Causely: Test Notification 42", "type": "ProblemDetected", # Options are ProblemDetected and ProblemCleared "entity": { "id": "22307647-43e6-5f08-a5e4-a3f50088ccec", "link": "https://portal.causely.app", "name": "payments-api", "type": "Node" }, "labels": { "k8s.cluster.name": "us-prod", # Test to route based on cluster "k8s.namespace.name": "payments", # Test to route based on name space "gcp.resource.zone": "https://www.gcp.com/compute/v1/projects/example-project/zones/us-central1-a", "causely.ai/cluster": "my-important-services-cluster" }, "objectId": "2ac21477-61f3-4ae0-9fc4-0c1ea43ff727", "severity": "High", # Options are Critical, High, Medium, Low "timestamp": "2025-09-14T19:15:16.410543344Z", "description": { "summary": "Sending test notification for entity in us-prod myApp1", "details": "Webhook Test" } } variables = { "notification": { "sourceId": "2f423693-5334-4586-a366-64d458535001", "type": 1, "payload": json.dumps(embedded_payload) } } result = post_query(client, mutation, variables) print(result) ``` ```javascript // Reuses helpers defined above: getCauselyAccessToken, createGraphQLClient // Requires @apollo/client installed (as shown above) const { CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET } = process.env; const { gql } = require('@apollo/client'); if (!CAUSELY_CLIENT_ID || !CAUSELY_CLIENT_SECRET) { console.error('Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET'); process.exit(1); } (async () => { try { const token = await getCauselyAccessToken(CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET); const client = createGraphQLClient(token); // Mutation matches working script exactly const mutation = ` mutation createNotification($notification: NotificationInput!) { createNotification(notification: $notification) } `; // Embedded payload (stringified below) const embeddedPayload = { link: 'https://portal.causely.app', name: 'Causely: Test Notification 42', type: 'ProblemDetected', // ProblemDetected | ProblemCleared entity: { id: '22307647-43e6-5f08-a5e4-a3f50088ccec', link: 'https://portal.causely.app', name: 'payments-api', type: 'Node', }, labels: { 'k8s.cluster.name': 'us-prod', 'k8s.namespace.name': 'payments', 'gcp.resource.zone': 'https://www.gcp.com/compute/v1/projects/example-project/zones/us-central1-a', 'causely.ai/cluster': 'my-important-services-cluster', }, objectId: '2ac21477-61f3-4ae0-9fc4-0c1ea43ff727', severity: 'High', // Critical | High | Medium | Low timestamp: '2025-09-14T19:15:16.410543344Z', description: { summary: 'Sending test notification for entity in us-prod myApp1', details: 'Webhook Test', }, }; const variables = { notification: { sourceId: '2f423693-5334-4586-a366-64d458535001', type: 1, payload: JSON.stringify(embeddedPayload), }, }; // Use mutate for a GraphQL mutation const result = await client.mutate({ mutation: gql(mutation), variables }); console.log(JSON.stringify(result.data, null, 2)); } catch (err) { console.error('Error:', err.message); process.exit(1); } })(); ``` ```bash # Assumes CAUSELY_ACCESS_TOKEN is already exported (see Authentication section) # Creates files for the mutation, an embedded payload JSON, and variables (with payload as a JSON string), then posts them. # 1) GraphQL mutation cat > create_notification.graphql << 'EOF' mutation createNotification($notification: NotificationInput!) { createNotification(notification: $notification) } EOF # 2) Embedded payload (domain JSON) cat > embedded_payload.json << 'EOF' { "link": "https://portal.causely.app", "name": "Causely: Test Notification 42", "type": "ProblemDetected", "entity": { "id": "22307647-43e6-5f08-a5e4-a3f50088ccec", "link": "https://portal.causely.app", "name": "payments-api", "type": "Node" }, "labels": { "k8s.cluster.name": "us-prod", "k8s.namespace.name": "payments", "gcp.resource.zone": "https://www.gcp.com/compute/v1/projects/example-project/zones/us-central1-a", "causely.ai/cluster": "my-important-services-cluster" }, "objectId": "2ac21477-61f3-4ae0-9fc4-0c1ea43ff727", "severity": "High", "timestamp": "2025-09-14T19:15:16.410543344Z", "description": { "summary": "Sending test notification for entity in us-prod myApp1", "details": "Webhook Test" } } EOF # 3) Build variables.json with payload as a *string* using jq to JSON-encode the file content PAYLOAD_STR=$(jq -Rs . < embedded_payload.json) cat > variables.json << EOF { "notification": { "sourceId": "2f423693-5334-4586-a366-64d458535001", "type": 1, "payload": ${PAYLOAD_STR} } } EOF # 4) Post using gq if available; otherwise curl if command -v gq &> /dev/null; then gq https://api.causely.app/query/ \ --header "Authorization: Bearer ${CAUSELY_ACCESS_TOKEN}" \ --query-file create_notification.graphql \ --variables-file variables.json else curl -sS -X POST https://api.causely.app/query/ \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${CAUSELY_ACCESS_TOKEN}" \ -d "{\"query\":\"$(tr '\n' ' ' < create_notification.graphql)\",\"variables\":$(cat variables.json)}" fi # 5) Clean up rm -f create_notification.graphql embedded_payload.json variables.json ``` ```go package main "encoding/json" "fmt" "os" ) func main() { // Reuse helpers defined above: // getToken(id, secret, url) and createGraphQLClient(accessToken) id, secret := os.Getenv("CAUSELY_CLIENT_ID"), os.Getenv("CAUSELY_CLIENT_SECRET") if id == "" || secret == "" { fmt.Fprintln(os.Stderr, "Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET") os.Exit(1) } token, err := getToken(id, secret, "https://auth.causely.app/frontegg/identity/resources/auth/v2/api-token") if err != nil { fmt.Fprintln(os.Stderr, "Error getting token:", err) os.Exit(1) } client := createGraphQLClient(token) // Mutation matches working script exactly mutation := ` mutation createNotification($notification: NotificationInput!) { createNotification(notification: $notification) } ` // Embedded payload (will be stringified) embeddedPayload := map[string]interface{}{ "link": "https://portal.causely.app", "name": "Causely: Test Notification 42", "type": "ProblemDetected", "entity": map[string]interface{}{ "id": "22307647-43e6-5f08-a5e4-a3f50088ccec", "link": "https://portal.causely.app", "name": "payments-api", "type": "Node", }, "labels": map[string]interface{}{ "k8s.cluster.name": "us-prod", "k8s.namespace.name": "payments", "gcp.resource.zone": "https://www.gcp.com/compute/v1/projects/example-project/zones/us-central1-a", "causely.ai/cluster": "my-important-services-cluster", }, "objectId": "2ac21477-61f3-4ae0-9fc4-0c1ea43ff727", "severity": "High", "timestamp": "2025-09-14T19:15:16.410543344Z", "description": map[string]interface{}{ "summary": "Sending test notification for entity in us-prod myApp1", "details": "Webhook Test", }, } payloadBytes, err := json.Marshal(embeddedPayload) if err != nil { fmt.Fprintln(os.Stderr, "Error marshaling payload:", err) os.Exit(1) } variables := map[string]interface{}{ "notification": map[string]interface{}{ "sourceId": "2f423693-5334-4586-a366-64d458535001", "type": 1, "payload": string(payloadBytes), // GraphQL expects a STRING here }, } result, err := postQuery(client, mutation, variables) if err != nil { fmt.Fprintln(os.Stderr, "Error running mutation:", err) os.Exit(1) } fmt.Println(result) } ``` When sending a notification via GraphQL, the `payload` field must be passed as a **stringified JSON** (not as a nested JSON object). The embedded JSON represents the notification details and must be stringified before being sent in the GraphQL variables. Below is a clear breakdown: **Mutation:** ```graphql mutation createNotification($notification: NotificationInput!) { createNotification(notification: $notification) } ``` **Embedded Payload (JSON)** This is the JSON you want to send as the notification payload (before stringifying): ```json { "link": "https://portal.causely.app", "name": "Causely: Test Notification 42", "type": "ProblemDetected", "entity": { "id": "22307647-43e6-5f08-a5e4-a3f50088ccec", "link": "https://portal.causely.app", "name": "payments-api", "type": "Node" }, "labels": { "k8s.cluster.name": "us-prod", "k8s.namespace.name": "payments", "gcp.resource.zone": "https://www.gcp.com/compute/v1/projects/example-project/zones/us-central1-a", "causely.ai/cluster": "my-important-services-cluster" }, "objectId": "2ac21477-61f3-4ae0-9fc4-0c1ea43ff727", "severity": "High", "timestamp": "2025-09-14T19:15:16.410543344Z", "description": { "summary": "Sending test notification for entity in us-prod myApp1", "details": "Webhook Test" } } ``` **Variables (GraphQL input)** Notice that the `payload` value is the above JSON, stringified (escaped), and can be split across multiple lines for readability: ```json { "notification": { "sourceId": "2f423693-5334-4586-a366-64d458535001", "type": 1, "payload": "{\n \"link\": \"https://portal.causely.app\",\n \"name\": \"Causely: Test Notification 42\",\n \"type\": \"ProblemDetected\",\n \"entity\": {\n \"id\": \"22307647-43e6-5f08-a5e4-a3f50088ccec\",\n \"link\": \"https://portal.causely.app\",\n \"name\": \"payments-api\",\n \"type\": \"Node\"\n },\n \"labels\": {\n \"k8s.cluster.name\": \"us-prod\",\n \"k8s.namespace.name\": \"payments\",\n \"gcp.resource.zone\": \"https://www.gcp.com/compute/v1/projects/example-project/zones/us-central1-a\",\n \"causely.ai/cluster\": \"my-important-services-cluster\"\n },\n \"objectId\": \"2ac21477-61f3-4ae0-9fc4-0c1ea43ff727\",\n \"severity\": \"High\",\n \"timestamp\": \"2025-09-14T19:15:16.410543344Z\",\n \"description\": {\n \"summary\": \"Sending test notification for entity in us-prod myApp1\",\n \"details\": \"Webhook Test\"\n }\n}" } } ``` > **Note:** > The `payload` field must be a string. Most clients will use a function like `JSON.stringify()` to convert the embedded payload JSON into a string before passing it to the GraphQL mutation. > > This lets you clearly see how the original JSON maps into the string value required by the GraphQL API. --- ## Setting SLOs on HTTP Paths and RPC Methods You can set SLOs on **HTTP Paths** or **RPC Methods** that represent key endpoints for your application (for example, revenue-critical URIs or core RPC methods). Enabling SLOs on these entities configures SLO tracking for **both Request Duration and Request Error Rate**. When a root cause impacts one of these endpoints and drives its SLO **at risk** or into **violation**, Causely treats that root cause as **Urgent**. The recommended way to manage these SLOs is to use **Entity Configs** of type `EntityTier`. :::tip Prerequisite Each example below reuses the helper utilities defined in the [Authentication](/api/authentication) and [GraphQL Clients](/api/graphql-clients) sections (`get_causely_access_token`, `create_graphql_client`, `post_query`, and their language equivalents). ::: ## Enable SLOs on an HTTP Path or RPC Method ```python # Reuse helpers defined earlier in this guide: # - get_causely_access_token # - create_graphql_client # - post_query CREATE_CONFIGS_MUTATION = """ mutation CreateEntityConfigs($entityConfigs: [EntityConfigInput!]!) { createEntityConfigs(entityConfigs: $entityConfigs) { entityId type data } } """ if __name__ == "__main__": cid, secret = os.getenv("CAUSELY_CLIENT_ID"), os.getenv("CAUSELY_CLIENT_SECRET") if not cid or not secret: raise EnvironmentError("Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET") token = get_causely_access_token(cid, secret) client = create_graphql_client(token) # Replace with the entityId of the HTTP Path or RPC Method you want to enable SLOs on entity_id = "7c4bba4c-1942-5fc4-87e2-d1f2c483d2c2" variables = { "entityConfigs": [{ "type": "EntityTier", "entityId": entity_id, "data": "sloEnabled", }] } result = post_query(client, CREATE_CONFIGS_MUTATION, variables) print(json.dumps(result, indent=2)) ``` ```javascript // Reuses helpers defined earlier in this guide: // - getCauselyAccessToken // - createGraphQLClient // - postQuery (for queries/mutations) const { gql } = require('@apollo/client'); const { CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET } = process.env; const CREATE_CONFIGS_MUTATION = ` mutation CreateEntityConfigs($entityConfigs: [EntityConfigInput!]!) { createEntityConfigs(entityConfigs: $entityConfigs) { entityId type data } } `; (async () => { if (!CAUSELY_CLIENT_ID || !CAUSELY_CLIENT_SECRET) { console.error('Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET'); process.exit(1); } const token = await getCauselyAccessToken(CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET); const client = createGraphQLClient(token); // Replace with the entityId of the HTTP Path or RPC Method you want to enable SLOs on const entityId = '7c4bba4c-1942-5fc4-87e2-d1f2c483d2c2'; const variables = { entityConfigs: [ { type: 'EntityTier', entityId, data: 'sloEnabled', }, ], }; const result = await client.mutate({ mutation: gql(CREATE_CONFIGS_MUTATION), variables, }); console.log(JSON.stringify(result.data, null, 2)); })(); ``` ```bash #!/usr/bin/env bash set -euo pipefail # Prerequisites: # - CAUSELY_CLIENT_ID / CAUSELY_CLIENT_SECRET exported # - CAUSELY_ACCESS_TOKEN exported (see Authentication section) # - post_query / post_query_curl functions defined (see Bash GraphQL helpers section) # - jq installed # Replace with the entityId of the HTTP Path or RPC Method you want to enable SLOs on ENTITY_ID="7c4bba4c-1942-5fc4-87e2-d1f2c483d2c2" cat > create_entity_configs.graphql << 'EOF' mutation CreateEntityConfigs($entityConfigs: [EntityConfigInput!]!) { createEntityConfigs(entityConfigs: $entityConfigs) { entityId type data } } EOF cat > create_entity_configs_variables.json << EOF { "entityConfigs": [ { "type": "EntityTier", "entityId": "${ENTITY_ID}", "data": "sloEnabled" } ] } EOF QUERY=$(tr '\n' ' ' < create_entity_configs.graphql) VARIABLES=$(cat create_entity_configs_variables.json) post_query_curl "$QUERY" "$VARIABLES" rm -f create_entity_configs.graphql create_entity_configs_variables.json ``` ```go package main "encoding/json" "fmt" "os" ) // Reuse helpers defined earlier: // - getToken(id, secret, url) // - createGraphQLClient(accessToken string) *graphql.Client // - postQuery(client *graphql.Client, queryString string, variables map[string]interface{}) (interface{}, error) const createConfigsMutation = ` mutation CreateEntityConfigs($entityConfigs: [EntityConfigInput!]!) { createEntityConfigs(entityConfigs: $entityConfigs) { entityId type data } } ` func main() { id, secret := os.Getenv("CAUSELY_CLIENT_ID"), os.Getenv("CAUSELY_CLIENT_SECRET") if id == "" || secret == "" { fmt.Fprintln(os.Stderr, "Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET") os.Exit(1) } token, err := getToken(id, secret, "https://auth.causely.app/frontegg/identity/resources/auth/v2/api-token") if err != nil { fmt.Fprintln(os.Stderr, "Error getting token:", err) os.Exit(1) } client := createGraphQLClient(token) // Replace with the entityId of the HTTP Path or RPC Method you want to enable SLOs on entityID := "7c4bba4c-1942-5fc4-87e2-d1f2c483d2c2" variables := map[string]interface{}{ "entityConfigs": []map[string]interface{}{ { "type": "EntityTier", "entityId": entityID, "data": "sloEnabled", }, }, } result, err := postQuery(client, createConfigsMutation, variables) if err != nil { fmt.Fprintln(os.Stderr, "Error enabling SLO:", err) os.Exit(1) } out, _ := json.MarshalIndent(result, "", " ") fmt.Println(string(out)) } ``` ```graphql mutation CreateEntityConfigs($entityConfigs: [EntityConfigInput!]!) { createEntityConfigs(entityConfigs: $entityConfigs) { entityId type data } } ``` **Variables (example):** ```json { "entityConfigs": [ { "type": "EntityTier", "entityId": "7c4bba4c-1942-5fc4-87e2-d1f2c483d2c2", "data": "sloEnabled" } ] } ``` ## Check whether an HTTP Path or RPC Method has SLOs enabled ```python # Reuse helpers defined earlier: # - get_causely_access_token # - create_graphql_client # - post_query GET_CONFIGS_QUERY = """ query GetEntityConfigs($entityId: String!) { entityConfigs(entityId: $entityId) { type data entityId } } """ if __name__ == "__main__": cid, secret = os.getenv("CAUSELY_CLIENT_ID"), os.getenv("CAUSELY_CLIENT_SECRET") if not cid or not secret: raise EnvironmentError("Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET") token = get_causely_access_token(cid, secret) client = create_graphql_client(token) entity_id = "2f3ce84a-18b8-5d4c-820d-5b747cbb7f13" result = post_query(client, GET_CONFIGS_QUERY, {"entityId": entity_id}) print(json.dumps(result, indent=2)) ``` ```javascript const { gql } = require('@apollo/client'); const { CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET } = process.env; const GET_CONFIGS_QUERY = ` query GetEntityConfigs($entityId: String!) { entityConfigs(entityId: $entityId) { type data entityId } } `; (async () => { const token = await getCauselyAccessToken(CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET); const client = createGraphQLClient(token); const entityId = '2f3ce84a-18b8-5d4c-820d-5b747cbb7f13'; const result = await client.query({ query: gql(GET_CONFIGS_QUERY), variables: { entityId }, fetchPolicy: 'no-cache', }); console.log(JSON.stringify(result.data, null, 2)); })(); ``` ```bash #!/usr/bin/env bash set -euo pipefail ENTITY_ID="2f3ce84a-18b8-5d4c-820d-5b747cbb7f13" cat > get_entity_configs.graphql << 'EOF' query GetEntityConfigs($entityId: String!) { entityConfigs(entityId: $entityId) { type data entityId } } EOF cat > get_entity_configs_variables.json << EOF { "entityId": "${ENTITY_ID}" } EOF QUERY=$(tr '\n' ' ' < get_entity_configs.graphql) VARIABLES=$(cat get_entity_configs_variables.json) post_query_curl "$QUERY" "$VARIABLES" rm -f get_entity_configs.graphql get_entity_configs_variables.json ``` ```go package main "encoding/json" "fmt" "os" ) // Reuse helpers defined earlier: // - getToken(id, secret, url) // - createGraphQLClient(accessToken string) *graphql.Client // - postQuery(client *graphql.Client, queryString string, variables map[string]interface{}) (interface{}, error) const getConfigsQuery = ` query GetEntityConfigs($entityId: String!) { entityConfigs(entityId: $entityId) { type data entityId } } ` func main() { id, secret := os.Getenv("CAUSELY_CLIENT_ID"), os.Getenv("CAUSELY_CLIENT_SECRET") if id == "" || secret == "" { fmt.Fprintln(os.Stderr, "Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET") os.Exit(1) } token, err := getToken(id, secret, "https://auth.causely.app/frontegg/identity/resources/auth/v2/api-token") if err != nil { fmt.Fprintln(os.Stderr, "Error getting token:", err) os.Exit(1) } client := createGraphQLClient(token) // Replace with the entityId of the HTTP Path or RPC Method you want to inspect entityID := "2f3ce84a-18b8-5d4c-820d-5b747cbb7f13" variables := map[string]interface{}{ "entityId": entityID, } result, err := postQuery(client, getConfigsQuery, variables) if err != nil { fmt.Fprintln(os.Stderr, "Error querying configs:", err) os.Exit(1) } out, _ := json.MarshalIndent(result, "", " ") fmt.Println(string(out)) // Look for: type == "EntityTier" and data == "sloEnabled" } ``` ```query GetEntityConfigs($entityId: String!) { entityConfigs(entityId: $entityId) { type data entityId } } ``` **Variables (example):** ```json { "entityId": "2f3ce84a-18b8-5d4c-820d-5b747cbb7f13" } ``` ## Remove SLOs from an HTTP Path or RPC Method ```python # Reuse helpers defined earlier: # - get_causely_access_token # - create_graphql_client # - post_query DELETE_CONFIG_MUTATION = """ mutation DeleteEntityConfig($entityId: String!, $configType: String!) { deleteEntityConfig(entityId: $entityId, configType: $configType) } """ if __name__ == "__main__": cid, secret = os.getenv("CAUSELY_CLIENT_ID"), os.getenv("CAUSELY_CLIENT_SECRET") if not cid or not secret: raise EnvironmentError("Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET") token = get_causely_access_token(cid, secret) client = create_graphql_client(token) entity_id = "2f3ce84a-18b8-5d4c-820d-5b747cbb7f13" result = post_query(client, DELETE_CONFIG_MUTATION, { "entityId": entity_id, "configType": "EntityTier" }) print(json.dumps(result, indent=2)) ``` ```javascript const { gql } = require('@apollo/client'); const { CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET } = process.env; const DELETE_CONFIG_MUTATION = ` mutation DeleteEntityConfig($entityId: String!, $configType: String!) { deleteEntityConfig(entityId: $entityId, configType: $configType) } `; (async () => { const token = await getCauselyAccessToken(CAUSELY_CLIENT_ID, CAUSELY_CLIENT_SECRET); const client = createGraphQLClient(token); const entityId = '2f3ce84a-18b8-5d4c-820d-5b747cbb7f13'; const result = await client.mutate({ mutation: gql(DELETE_CONFIG_MUTATION), variables: { entityId, configType: 'EntityTier' }, }); console.log(JSON.stringify(result.data, null, 2)); })(); ``` ```bash #!/usr/bin/env bash set -euo pipefail ENTITY_ID="2f3ce84a-18b8-5d4c-820d-5b747cbb7f13" cat > delete_entity_config.graphql << 'EOF' mutation DeleteEntityConfig($entityId: String!, $configType: String!) { deleteEntityConfig(entityId: $entityId, configType: $configType) } EOF cat > delete_entity_config_variables.json << EOF { "entityId": "${ENTITY_ID}", "configType": "EntityTier" } EOF QUERY=$(tr '\n' ' ' < delete_entity_config.graphql) VARIABLES=$(cat delete_entity_config_variables.json) post_query_curl "$QUERY" "$VARIABLES" rm -f delete_entity_config.graphql delete_entity_config_variables.json ``` ```go package main "encoding/json" "fmt" "os" ) // Reuse helpers defined earlier: // - getToken(id, secret, url) // - createGraphQLClient(accessToken string) *graphql.Client // - postQuery(client *graphql.Client, queryString string, variables map[string]interface{}) (interface{}, error) const deleteConfigMutation = ` mutation DeleteEntityConfig($entityId: String!, $configType: String!) { deleteEntityConfig(entityId: $entityId, configType: $configType) } ` func main() { id, secret := os.Getenv("CAUSELY_CLIENT_ID"), os.Getenv("CAUSELY_CLIENT_SECRET") if id == "" || secret == "" { fmt.Fprintln(os.Stderr, "Missing CAUSELY_CLIENT_ID or CAUSELY_CLIENT_SECRET") os.Exit(1) } token, err := getToken(id, secret, "https://auth.causely.app/frontegg/identity/resources/auth/v2/api-token") if err != nil { fmt.Fprintln(os.Stderr, "Error getting token:", err) os.Exit(1) } client := createGraphQLClient(token) // Replace with the entityId of the HTTP Path or RPC Method you want to remove SLOs from entityID := "2f3ce84a-18b8-5d4c-820d-5b747cbb7f13" variables := map[string]interface{}{ "entityId": entityID, "configType": "EntityTier", } result, err := postQuery(client, deleteConfigMutation, variables) if err != nil { fmt.Fprintln(os.Stderr, "Error deleting config:", err) os.Exit(1) } out, _ := json.MarshalIndent(result, "", " ") fmt.Println(string(out)) } ``` ```graphql mutation DeleteEntityConfig($entityId: String!, $configType: String!) { deleteEntityConfig(entityId: $entityId, configType: $configType) } ``` **Variables (example):** ```json { "entityId": "2f3ce84a-18b8-5d4c-820d-5b747cbb7f13", "configType": "EntityTier" } ``` --- ## Alert Ingestion and Mapping ## Overview Causely can ingest alerts from your existing monitoring sources and connect them to the causal reasoning engine—without replacing your current alerting setup. When an alert fires, Causely identifies which service or entity it belongs to, maps it to a known symptom, and includes it as evidence in causal analysis. The following alert sources are supported: - [Alertmanager](/telemetry-sources/alertmanager): Prometheus Alertmanager webhook receiver - [Prometheus](/telemetry-sources/prometheus): Direct Prometheus integration - [Datadog](/telemetry-sources/datadog): Datadog monitors and alerts - [Incident.io](/telemetry-sources/incident-io): Incident.io alert events - [Dynatrace](/telemetry-sources/dynatrace): Dynatrace problems and alerts ## How alerts are mapped automatically When Causely receives an alert, it attempts to resolve two things: **which entity** the alert belongs to, and **which symptom** it represents. ### Step 1: Symptom resolution via keyword matching Causely examines the alert's `alertname`, title, and description, and matches them against a library of keyword rules. Matching is case-insensitive. The first matching rule wins, so more specific rules (for example, queue-related keywords) are evaluated before broader ones (for example, generic "error"). Here are a few examples of how this works. An alert named `HighErrorRate` or with a description containing "error rate exceeded" is automatically recognized as a **high error rate** condition. An alert containing "latency" or "timeout" is recognized as a **high latency** condition. Similarly, alerts mentioning "kafka lag", "consumer lag", "message wait time", "dead letter", "JVM heap", or "GC time" are each matched to their corresponding condition. Some examples of what Causely looks for in the alert name, title, or description: | Condition | Example keywords | |---|---| | High error rate | "error", "failed", "failure", "exception", "unavailable", "down" | | High latency | "latency", "duration", "timeout", "slow", "response time" | | Kafka consumer lag | "lag" and ("kafka" / "consumer" / "partition") | | Queue wait time | "message wait", "wait time" and "queue" | | Dead-letter queue | "dead letter", "deadletter", "dlx" | | JVM heap pressure | "heap" and ("java" / "jvm") | | Garbage collection | "gc", "garbage collection", "g1 young", "g1 old" | | DB connection pool | "db connection", "database connection", "postgres connection" | | Redis connection pool | "redis connection", "redis connection pool" | ### Step 2: Entity resolution via alert labels A matched symptom still needs an entity to attach to. Causely uses labels in the alert payload to look up the right entity in your topology. Each entity type requires a specific combination of labels that uniquely identifies it—see [Required label combinations by entity type](#required-label-combinations-by-entity-type) below. If the labels are missing or do not correspond to a known entity in topology, the alert remains unmapped even when the symptom is correctly identified. ## What happens after an alert is mapped ### Alert maps to a known entity and symptom The alert is fully ingested. Causely attaches it as an observed symptom to the entity and includes it in causal reasoning. You will see it reflected in the root cause analysis alongside other signals. ### Alert maps to a known entity but no symptom match If Causely identifies which entity the alert belongs to but does not recognize the alert as a known symptom, the alert is recorded but does not contribute to causal analysis. This happens when the alert name, title, and description do not contain any matching keywords from the table above. Causely engineers are happy to work with you to add important alerts to the knowledge base so they can participate in causal reasoning. Reach out to your Causely team with the alert details. ### Alert does not map to any entity If Causely cannot resolve an entity from the alert, the alert is not attached to the topology. This is almost always caused by **missing or incorrect labels** in the alert payload. See the next section for the exact labels required for each entity type. ## Required label combinations by entity type For an alert to be attached to an entity, its payload must include the labels that uniquely identify that entity. The exact combination depends on the type of entity the alert is about. ### Kubernetes pod **Required:** `namespace` + `pod` Use this when the alert is about a specific pod (for example, high JVM heap, GC time, thread contention, connection pool exhaustion). ```json { "alerts": [ { "status": "firing", "labels": { "alertname": "HighJVMHeapUsage", "namespace": "prod", "pod": "payment-service-7d4f8b-xyz", "severity": "warning" }, "annotations": { "summary": "JVM heap utilization above 80%" }, "startsAt": "2025-02-04T12:00:00.000Z", "endsAt": "0001-01-01T00:00:00Z" } ] } ``` ### Kubernetes container **Required:** `namespace` + `pod` + `container` Use this when the alert targets a specific container within a pod (for example, a sidecar or init container). ```json { "alerts": [ { "status": "firing", "labels": { "alertname": "ContainerHighCPU", "namespace": "prod", "pod": "payment-service-7d4f8b-xyz", "container": "payment-app", "severity": "warning" }, "annotations": { "summary": "Container CPU usage above threshold" }, "startsAt": "2025-02-04T12:00:00.000Z", "endsAt": "0001-01-01T00:00:00Z" } ] } ``` ### Kubernetes service **Required:** `namespace` + `pod` Causely resolves the owning service from the pod via topology. The optional `service` label can be included as a hint but is not required for resolution. ```json { "alerts": [ { "status": "firing", "labels": { "alertname": "HighErrorRate", "namespace": "prod", "pod": "checkout-service-abc", "service": "checkout-service", "severity": "warning" }, "annotations": { "summary": "High error rate on checkout-service", "description": "Error rate increased above threshold in the last 5m" }, "startsAt": "2025-02-04T12:00:00.000Z", "endsAt": "0001-01-01T00:00:00Z" } ] } ``` ### HTTP path **Required:** `namespace` + `pod` + a path label The path label can be any of: `uri`, `url_path`, `http_path`, `path`, `http.route`. The path entity (for example, `/checkout`) must already exist in topology (for example, from ingress or service metrics). ```json { "alerts": [ { "status": "firing", "labels": { "alertname": "HighPathErrorRate", "namespace": "ingress-nginx", "pod": "ingress-nginx-controller-956c769c7-lgcxx", "uri": "/checkout", "severity": "warning" }, "annotations": { "summary": "High error rate on /checkout" }, "startsAt": "2025-02-04T12:00:00.000Z", "endsAt": "0001-01-01T00:00:00Z" } ] } ``` ### gRPC / RPC method **Required:** `rpc_service` + `rpc_method` (or `grpc_service` + `grpc_method`) Optional: `namespace`, `pod`. The RPC method entity (service + method pair) must already exist in topology (for example, from distributed traces). ```json { "alerts": [ { "status": "firing", "labels": { "alertname": "HighGRPCErrorRate", "rpc_service": "user.UserService", "rpc_method": "GetUser", "namespace": "prod", "pod": "user-service-abc", "severity": "warning" }, "annotations": { "summary": "High error rate on user.UserService/GetUser" }, "startsAt": "2025-02-04T12:00:00.000Z", "endsAt": "0001-01-01T00:00:00Z" } ] } ``` ### Kafka topic **Required:** `topic` + consumer group label The consumer group can appear under any of these label names: `consumer_group_id`, `consumer_group`, `group`, or `consumergroup`. Optionally include `namespace` and `pod` when using Micrometer-style instrumentation. **Confluent-style** (topic + consumer group ID only): ```json { "alerts": [ { "status": "firing", "labels": { "alertname": "HighKafkaConsumerLag", "topic": "orders", "consumer_group_id": "order-processor-group", "severity": "warning" }, "annotations": { "summary": "Kafka consumer lag above threshold for topic orders" }, "startsAt": "2025-02-04T12:00:00.000Z", "endsAt": "0001-01-01T00:00:00Z" } ] } ``` **Micrometer-style** (topic + group + namespace + pod): ```json { "alerts": [ { "status": "firing", "labels": { "alertname": "HighKafkaConsumerLag", "topic": "orders", "group": "order-processor-group", "namespace": "default", "pod": "order-processor-0", "severity": "warning" }, "annotations": { "summary": "Kafka consumer lag above threshold for topic orders" }, "startsAt": "2025-02-04T12:00:00.000Z", "endsAt": "0001-01-01T00:00:00Z" } ] } ``` ### Queue (RabbitMQ or similar) **Required:** a label carrying the queue name (for example, `queue`) The queue must already exist in topology (for example, discovered from RabbitMQ or OpenTelemetry). ```json { "alerts": [ { "status": "firing", "labels": { "alertname": "MessageWaitTimeHigh", "queue": "work_queue", "severity": "warning" }, "annotations": { "summary": "Message wait time above threshold on work_queue" }, "startsAt": "2025-02-04T12:00:00.000Z", "endsAt": "0001-01-01T00:00:00Z" } ] } ``` ## Summary | Entity type | Required label combination | |---|---| | Kubernetes pod | `namespace` + `pod` | | Kubernetes container | `namespace` + `pod` + `container` | | Kubernetes service | `namespace` + `pod` (service resolved from pod via topology) | | HTTP path | `namespace` + `pod` + path label (`uri`, `url_path`, `http_path`, `path`, or `http.route`) | | gRPC / RPC method | `rpc_service` + `rpc_method` (or `grpc_service` + `grpc_method`) | | Kafka topic | `topic` + consumer group (`consumer_group_id`, `group`, `consumer_group`, or `consumergroup`) | | Queue | queue name label (for example `queue`) | --- ## Credentials Autodiscovery Causely also supports credentials autodiscovery. This feature allows you to add new scraping targets without updating the Causely configuration. Label the Kubernetes Secret to enable autodiscovery for the corresponding scraper. For example, to [enable autodiscovery for the MySQL scraper](/telemetry-sources/mysql#alternative-enable-credentials-autodiscovery), you can label the Kubernetes Secret with the following command: ```bash kubectl --namespace causely label secret mysql-credentials "causely.ai/scraper=MySQL" ``` Credentials Autodiscovery is available for many of the [telemetry sources](/telemetry-sources/). --- ## Custom Causes ## Overview When Causely receives an alert that it does not automatically recognize as a known symptom, you can define a **custom cause** to tell Causely what that alert means. Once defined, every time that alert fires, Causely surfaces it as a root cause with your description and recommended remediation, connects it to any related existing root causes in the causal graph, and clears it automatically when the alert resolves. Custom causes let you bring institutional knowledge about your system into Causely's causal reasoning without writing code or modifying Causely's built-in causal models. ## Prerequisites Before creating a custom cause, verify that the alert carries enough label information for Causely to identify which entity it belongs to. In the **Integrations → Alerts** view, enable the **Entity Only** filter to see only alerts that meet this requirement. Alerts that appear in this filtered view are already resolved to a specific entity in your topology. Alerts that do not appear are missing required labels and cannot be used for custom causes until their labels are corrected. For the exact label combinations required per entity type, please see [Required label combinations by entity type](/configuration/alert-ingestion#required-label-combinations-by-entity-type). ## Creating a custom cause From the **Integrations → Alerts** view, click **Define New Cause** on any alert row, or go directly to **Settings → User Defined Causes** and click **Define New Cause**. ### Choose a category Causely uses the category to understand the failure mode the cause represents. Select one: - **Error (may cause malfunction)**: the alert indicates a problem that causes errors or downtime on the affected entity. - **Latency (may cause congestion)**: the alert indicates a problem that causes increased response time or throughput degradation. The category determines how Causely weighs the cause in causal analysis and how it is presented in the UI. ### Add description and remediation You can write a description and remediation steps manually, or use the **Generate description and remediation with AI** button to have Causely draft them based on the cause name and alert context. If AI generation fails or produces an unsatisfactory result, you can trigger it again at any time from the cause detail view after saving. Click **Create** when done. The definition is active immediately with no restart required. ## Managing your causes All defined causes are listed under **Settings → User Defined Causes**, showing the cause name, matched alert name, category, creator, and creation date. Click any row to edit the display name, description, or remediation. Changes take effect on the next analysis cycle (~30 seconds). The alert name and category cannot be changed after creation. Delete and recreate the definition if you need to change those. ## How active causes appear When an alert matching a custom cause fires, Causely surfaces it as an active root cause alongside built-in causes. Clicking through to a cause shows the full detail view: description, impact graph, and the remediation steps you defined (or that were AI-generated). If the same alert fires on multiple entities simultaneously, the cause appears independently on each affected entity. ## Deleting a cause Deleting a custom cause definition immediately clears any active instance of that cause across all entities. It will not reappear even if the underlying alert is still firing. --- ## Configuration # Configuration Overview Causely can be configured through three different methods: 1. [**UI Configuration**](#ui-configuration) - For basic settings and scopes 2. [**Helm Values**](#helm-values-configuration) - For detailed deployment configuration 3. [**Kubernetes Labels**](#kubernetes-labels) - For service-specific settings ## UI Configuration The UI configuration is accessible through the Causely UI and allows you to configure: - [**Scopes**](/configuration/scopes): Define custom scopes for your environment (geography, environment, customer, team, etc.) - **General Settings**: - **Web Analytics toggle**: We use web analytics to enhance our service and user experience. By keeping this option enabled, you consent to the collection and processing of usage data. You can disable this setting at any time. - **Auto refresh interval**: Choose the frequency at which the user interface views should refresh automatically. ## Helm Values Configuration The Helm values file (`causely-values.yaml`) is the primary method for configuring your Causely deployment. For a complete list of available configuration options, you can view your current values using: ```bash helm get values causely --namespace causely --all ``` Read more about the [Helm values](/installation/helm) configuration. ## Kubernetes Labels Kubernetes labels provide a way to configure service-specific settings directly on your Kubernetes resources. The following Kubernetes labels are available for configuration: - [Credentials Autodiscovery](/configuration/credentials-autodiscovery) - [Thresholds](/configuration/thresholds) --- ## Scopes ## Overview Scopes in Causely allow you to define and manage custom subsets of your environment's topology. As Causely automatically discovers the full topology of your environment, it can present a rich but potentially overwhelming set of entities—services, infrastructure components, and identified problems. Scopes help you focus on the specific subset of data that matters most to your role, responsibilities, or current investigative tasks. Scopes can be used in conjunction with service ownership to create more focused views of your environment. While service ownership defines who is responsible for different components, scopes allow you to create custom views that may or may not align with ownership boundaries. ## What are Scopes? User Scopes define the global context for your session, determining what data is accessible across all views. You can create, save, and manage custom Scopes to focus on specific subsets of data, such as clusters or namespaces. Scopes are persistent, can be set as default, and adapt dynamically to your selections during navigation. Scopes can be created based on specific components out of the box, such as clusters, namespaces, and services. Additionally, Causely supports the use of custom labels to provide users with more granular or domain-specific scope options. See [Using Custom Labels for Scopes](#using-custom-labels-for-scopes) for details. ## Selecting a Scope In the top navigation bar, you can select the scope you want to use for your session. ## Managing Scopes To manage your scopes: 1. Click the gear icon (⚙️) in the top right corner of the Causely UI 2. Select "Settings" from the dropdown menu 3. Navigate to the "Scopes" section ### Creating a New Scope 1. In the Scopes settings page, click the "Create" button 2. Define your scope by selecting the relevant entities and components 3. Give your scope a descriptive name 4. Save your scope ### Editing a Scope 1. Find the scope you want to edit in the Scopes list 2. Click the edit icon next to the scope 3. Modify the scope's configuration as needed 4. Save your changes ### Deleting a Scope 1. Find the scope you want to delete in the Scopes list 2. Click the delete icon next to the scope 3. Confirm the deletion ## Best Practices - Create scopes that align with your team's responsibilities and organizational structure - Use descriptive names that clearly indicate the scope's purpose - Regularly review and update your scopes to ensure they remain relevant - Consider creating scopes for different investigation scenarios or team roles ## Using Custom Labels for Scopes Out of the box, Causely provides scopes for specific components, for example cloud providers, clusters, namespaces, and service types. A cluster admin can configure Causely to automatically detect entities with custom labels and make them available as scope options in the UI. The following labels are supported: - Geography - Environment - Customer - Team - Product - Project - Service For details on how to add and use custom labels to enhance scopes, refer to the [custom labels documentation](/installation/customize/#custom-labels-for-scope-configuration). --- ## Service Prioritization ## Overview The **Service Prioritization** feature allows you to assign priority tiers to services discovered within a given scope. By default, all services are assigned a Service Level Objective (SLO) and treated as **Critical**. However, not all services are equally important—some require immediate attention, while others can be deprioritized or hidden entirely. This feature helps teams focus on what matters, ensuring that high-priority services drive alerts and incident response while low-priority or internal services remain visible but do not create unnecessary noise. ## How It Works Each service in a scope can be categorized into one of three **Service Tiers**: | Tier | Description | | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Critical: SLO Enforced** | Services essential to your business or user experience. Their SLOs are actively enforced. If their SLO is violated or at risk, related root causes are marked **Urgent**. | | **Non-Critical: SLO Not Enforced** | Services that are non-essential or low-impact. These services do not have an active SLO, and their root causes appear in the **Root Causes** view but are never marked urgent. | | **Hidden: Internal Noise** | Internal or auxiliary services you do not want visible in daily operations. Issues impacting these services appear only under the **Hidden** tab in the Root Causes view. | ## Default Behavior - Newly discovered services start as **Uncategorized**, which defaults to **Critical: SLO Enforced**. - All services in this state are monitored with an active SLO until explicitly reassigned to another tier. ## Assigning Service Tiers 1. Navigate to **Settings → Service Tiers**. 2. Search or filter for the services you want to prioritize. Note that the available services depend on the currently selected scope. 3. Select one or more services. 4. Use the **Tier Assignment** panel to set a tier: - **Critical: SLO Enforced** - **Non-Critical: SLO Not Enforced** - **Hidden: Internal Noise** You can assign tiers individually or in bulk across pages. ## Root Cause Visibility Tier assignment affects how root causes appear in **Root Causes**: - **Critical:** Appears in **Active**. Marked **Urgent** if the SLO is violated or at risk. - **Non-Critical:** Appears in **Active** but never marked urgent. - **Hidden:** Appears only in the **Hidden** tab, not in **Active** or **Historical**. ## Example Use Cases - **Critical:** Core APIs, payments, authentication, customer-facing services. - **Non-Critical:** Internal dashboards, staging workloads, low-traffic utilities. - **Hidden:** Background jobs, test services, deprecated components. --- ## Symptom Delay Causely automatically detects service [symptoms](/reference/symptoms/) based on various metrics. To avoid alerting on brief spikes or temporary blips, Causely uses **activation and deactivation delays**: a symptom must remain in violation of thresholds for a sustained period before it activates, and must return to normal for the same period before it clears. This document explains how these delays work and how to configure them for your services. ## How Symptom Delays Work Symptom delays serve two purposes: 1. **Prevent false positives**: Brief metric spikes don't trigger symptoms or alerts; only sustained issues are surfaced. 2. **Provide stability**: Symptoms remain active until the issue is genuinely resolved, reducing flapping and noise. The symptom activation delay depends on the *type* of issue: - **Bursty issues** (sudden spikes): the 5-minute average is > a multiple of the configurable threshold - **Sustained issues** (slow creep above threshold): the 30-minute average is > the configurable threshold The default activation delay is 5 minutes. A symptom must match one of the above conditions for 5 minutes. Deactivation also uses 5 minutes. During this 5 minute period the 5-min average must be below the higher threshold and the 30-min average must be below the threshold. ## Request Error Rate Symptom Causely activates a **Request Error Rate** symptom when either of these conditions is met: - **Condition 1: Bursty spike**: 5-minute average error rate > 4× the threshold (default: 1–2%) - **Condition 2: Sustained elevation**: 30-minute average error rate > threshold In either case, the condition must be true and the request rate must be > 0.3 req/sec (0.2 req/sec for HTTP Path or RPC Method) for 5 consecutive minutes. **Deactivation:** The symptom clears in either case: - **Recovery**: Neither of the above condition 1 nor 2 holds for 5 consecutive minutes. - **Silence**: The 5-min and 30-min average request rate is ≤ 0.3 req/sec (0.2 req/sec for HTTP Path or RPC Method); deactivates regardless of error rate. **Practical Effect:** A sudden 4× error rate spike that holds for 5+ minutes triggers quickly. A service that creeps above threshold gradually requires 30 minutes of sustained violation before alerting. Once fixed, it clears after a matching period of recovery. ## Request Duration (Latency) Symptom Causely activates a **Request Duration** symptom when either of these conditions is met: - **Condition 1: Bursty spike**: 5-minute average latency > 1.5× threshold - **Condition 2: Sustained elevation**: 30-minute average latency > threshold In either case, the condition must be true and the request rate must be > 0.3 req/sec (0.2 req/sec for HTTP Path or RPC Method) for 5 consecutive minutes. **Deactivation:** The symptom clears if neither of the above condition 1 nor 2 holds for 5 consecutive minutes. **Practical Effect:** A sudden 1.5× latency spike that holds for 5+ minutes triggers quickly. A service that slowly creeps above its baseline requires 30 minutes of sustained violation before alerting. Once fixed, it clears after a matching period of recovery. ## Activation Delay Configuration The activation delay for Service-level Request Error Rate and Request Duration symptoms is configurable. Configurations do not apply to HTTP Path or RPC Method symptoms. For threshold configuration (default thresholds for triggering symptoms), see [Threshold configuration](/configuration/thresholds/). ## Configuration Methods ### Using Kubernetes Labels Apply labels to your services: ```bash # Configure error rate activation delay (example: 3 minutes) kubectl label svc -n "causely.ai/error-rate-activation-delay=3" # Configure latency activation delay (example: 3 minutes) kubectl label svc -n "causely.ai/latency-activation-delay=3" ``` ### Using Nomad Service Tags Add tags to your service definition: ```hcl job "example" { group "app" { service { name = "my-service" port = 8080 tags = [ "causely.ai/error-rate-activation-delay=3", "causely.ai/latency-activation-delay=3" ] } } } ``` ### Using Consul Service Metadata Register services with metadata: ```bash consul services register \ -name="my-service" \ -port=8080 \ -meta="causely.ai/error-rate-activation-delay=3" \ -meta="causely.ai/latency-activation-delay=3" ``` ### Delay Values - **Valid Range:** 1–60 minutes - **Configures:** Bursty activation delay only (default: 5 minutes) - **Fixed:** Sustained activation delay (always 30 minutes) - **Recommended Range:** 1–10 minutes for most use cases ## Best Practices 1. **Start with defaults**: 5 minutes works for most services. Only adjust if you're seeing false positives (bursty spikes that recover quickly) or need faster detection of genuine issues. 2. **Shorter delays (1–3 min) for**: - Payment or high-revenue services where quick spike detection is critical - Services where you expect sudden issues to be real problems, not transient noise 3. **Longer delays (5–10 min) for**: - Services with frequent harmless spikes, for example, traffic bursts, cache refreshes - Noisy services where bursty patterns are normal and don't indicate problems - Non-production environments 4. **Keep in mind**: The 30-minute sustained delay is fixed and cannot be adjusted. This ensures slow, creeping issues are not misclassified as false positives. 5. **Monitor after changes**: After adjusting bursty delays, observe whether spike detection improves or false positives decrease. 6. **Document your reasoning**: Keep notes on why you chose specific delays per service. This helps during onboarding and reviews. ## Examples ### Critical Service (Payment Processing) Faster detection of error spikes: **Kubernetes:** ```bash kubectl label svc -n production payment-api "causely.ai/error-rate-activation-delay=2" kubectl label svc -n production payment-api "causely.ai/latency-activation-delay=2" ``` **Effect:** Error rate spikes are detected after 2 minutes of sustained elevation (vs. default 5). Slow degradation still requires 30 minutes. ### Bursty Service (Traffic Spikes) Reduce noise from normal traffic bursts: **Kubernetes:** ```bash kubectl label svc -n production data-processor "causely.ai/error-rate-activation-delay=8" kubectl label svc -n production data-processor "causely.ai/latency-activation-delay=8" ``` **Effect:** Brief error spikes during load bursts don't trigger until 8 minutes (giving time to recover). Slow issues still activate after 30 minutes. ### Development Environment Longer delays to minimize disruptions: **Kubernetes:** ```bash kubectl label svc -n dev api-service "causely.ai/error-rate-activation-delay=10" kubectl label svc -n dev api-service "causely.ai/latency-activation-delay=10" ``` **Effect:** Dev spikes require 10 minutes to trigger (vs. 5). Sustained issues always take 30 minutes, same as production. --- ## Thresholds for Service Symptoms Causely detects service and infrastructure [symptoms](/reference/symptoms/) based on a wide range of metrics, using a combination of defaults and learned behavior. However, you may want to customize these thresholds to better match your specific requirements and SLO definitions. This document explains how to configure custom thresholds for your services. ## Overview Causely uses thresholds to detect service and infrastructure symptoms across a wide range of metrics, including latency, error rates, throughput, and resource utilization. These thresholds serve two related purposes: - **Symptom detection**, where crossing a threshold activates a Causely issue or risk - **SLO evaluation**, where the same metrics act as Service Level Indicators (SLIs) that determine SLO health Causely provides sensible defaults for all supported thresholds. For some metrics, Causely can also **learn thresholds automatically** based on historical and real-time behavior. You can optionally configure **manual thresholds** or tune learned thresholds to better reflect business requirements, reliability targets, or known system constraints. ## Key Concepts ### Threshold Sources Each threshold in Causely has a **source**, which determines how its value is set and maintained: - **Default** A system-provided threshold value that applies when no learning or manual override is configured. Defaults are designed to be safe and broadly applicable. - **Learned** For some metrics, Causely can automatically learn a threshold based on historical and real-time behavior. Learned thresholds adapt over time as normal behavior changes. - **Manual** A user-configured threshold that explicitly overrides the default or learned value. Manual thresholds remain fixed until changed or removed. Only one source is active for a given threshold at any time (minimum learned thresholds do not change the source). ### How Thresholds Are Evaluated Thresholds are always evaluated against a **specific metric and aggregation**, even when multiple series are shown for context. For example: - **Request Duration thresholds are evaluated against P95 latency** A symptom becomes active when the P95 latency exceeds the configured threshold. - Other percentiles (such as P90 or P99) may be displayed to provide additional context but do not drive activation. The evaluation aggregation is fixed per metric and does not change when thresholds are overridden. ### Learned Threshold Minimums For metrics that support learning, you can optionally configure a **Minimum Learned Threshold**. A minimum learned threshold: - Sets a lower bound on how low a learned threshold can go - Allows Causely to continue learning above that value - Does **not** create a manual override This is useful when you want adaptive behavior while preventing learned thresholds from becoming unrealistically low due to traffic patterns or short-term anomalies. ### Thresholds and SLOs The same metrics used for symptom detection are also used as **Service Level Indicators (SLIs)** when evaluating SLOs. This means: - A threshold crossing may activate a symptom - The same metric contributes to SLO health calculations Configuring thresholds affects **both operational detection and SLO evaluation**, so changes should be made with awareness of their broader impact. ## How to Configure Thresholds Causely supports configuring thresholds through multiple mechanisms, allowing you to choose the approach that best fits your workflow and environment. Regardless of the method used, the outcome is the same: a **manual threshold**, unless you configure a minimum learned threshold. Thresholds can be configured at different levels (for example, service, workload, or infrastructure resource), depending on the metric and entity type. ### Configuration Options You can configure thresholds using the following methods: - **Causely UI** Best for inspecting learned behavior, understanding how thresholds relate to observed metrics, and making targeted adjustments. - **Service metadata** Service metadata supports configuring a subset of commonly used service-level thresholds and allows you to configure thresholds declaratively using: - Kubernetes labels - Nomad service tags - Consul service metadata This approach is well suited for version-controlled, environment-specific configuration that travels with your service definition. - **Causely API** Best for programmatic configuration, automation, and integration with internal tooling or workflows. All configuration methods support configuring **manual thresholds where applicable**, and for metrics that support learning, configuring a **minimum learned threshold** to bound adaptive learning. ### Choosing the Right Method Use the **UI** when you want to: - understand why a symptom is activating, - compare learned thresholds against real traffic, - experiment or iterate quickly. Use **service metadata** when you want to: - manage thresholds as code, - apply consistent thresholds across environments, - ensure thresholds are applied automatically during deployment. Use the **API** when you want to: - automate threshold management, - integrate threshold changes into CI/CD or internal systems, - apply changes across many entities programmatically. ### What Happens When You Configure a Threshold When you configure a threshold: - The threshold source becomes **Manual** - Automatic learning (if supported for that metric) is paused - The configured value is used consistently for: - symptom activation - SLI evaluation for SLOs If you remove a manual threshold, Causely reverts to the **default** or **learned** threshold, depending on the metric. ### Minimum Learned Thresholds For metrics that support learning, you can optionally configure a **Minimum Learned Threshold** instead of a full manual override. This allows Causely to: - continue adapting to changing behavior, - while never learning a threshold below the configured minimum. Minimum learned thresholds do **not** replace learning and do not create a manual override. ### Using the Causely UI Using the UI allows you to: - Inspect the learned threshold alongside observed metrics (for example P90 and P99 for latency) - Override thresholds to match documented SLOs or performance requirements - Set a minimum value that bounds how low a learned threshold can go while preserving adaptive learning - Immediately see how a custom threshold compares to real traffic patterns To configure thresholds in the UI: 1. Navigate to the service you want to configure. 2. Select the **Metrics** tab for the service, then select the relevant symptom metric (for example, Request Duration or Request Error Rate). 3. Click the pencil icon next to the threshold to edit the value. 4. Save the change to apply the override. UI-based configuration is best suited for teams that want quick iteration, visibility into learned behavior, and explicit control without modifying service metadata or deployment configuration. ### Using Service Metadata :::note Not all thresholds can be configured via service metadata. Metadata-based configuration currently supports a subset of service-level thresholds such as request error rate and request latency. ::: #### Using Kubernetes Labels The recommended way to configure thresholds is using Kubernetes labels. You can apply these labels to your services: ```bash # Configure error rate threshold (for example, 1% error rate) kubectl label svc -n "causely.ai/error-rate-threshold=0.01" # Configure latency threshold (for example, 500ms) kubectl label svc -n "causely.ai/latency-threshold=500.0" ``` #### Using Nomad Service Tags For Nomad services, you can configure thresholds using service tags in your job specification: ```hcl job "example" { group "app" { service { name = "my-service" port = 8080 tags = [ "causely.ai/error-rate-threshold=0.01" "causely.ai/latency-threshold=500.0" ] } } } ``` #### Using Consul Service Metadata For Consul services, you can configure thresholds using service metadata: ```bash # Register a service with threshold metadata consul services register \ -name="my-service" \ -port=8080 \ -meta="causely.ai/error-rate-threshold=0.01" \ -meta="causely.ai/latency-threshold=500.0" # Update existing service metadata consul services register \ -id="my-service-id" \ -name="my-service" \ -port=8080 \ -meta="causely.ai/error-rate-threshold=0.01" \ -meta="causely.ai/latency-activation-delay=500.0" ``` ## Supported Thresholds Causely supports configurable thresholds across a broad set of service and infrastructure entities. These thresholds are used to detect symptoms and also act as Service Level Indicators (SLIs) when evaluating SLO health. Some thresholds support **automatically learned values**, while others use **system defaults** that can be manually overridden. --- ### Services, Workloads, HTTP Paths and RPC Methods | Metric | Unit | Learned | |------|------|---------| | Request Error Rate | percent | No | | Request Duration (P95) | millisecond | Yes | | Request Duration P95 (Client) | millisecond | Yes | | Request Rate | request/s | Yes | | Connections | percent | No | | Mutex Wait Time | percent | No | | Command Latency | millisecond | No | | GC Time | percent | No | | Queries Queued | count | No | | Transaction Error | percent | No | | Transaction Duration | second | No | | Transaction IDs Congested | percent | No | | Cache Size | bytes | No | | Redis Connections Utilization | percent | No | | Kafka Message Rate | message/s | No | | Server Errors | count | No | | User Errors | count | No | | File Descriptor Utilization | percent | No | | Java Heap Utilization | percent | No | | Throttled | count | No | | DB Connections Utilization | percent | No | --- ### Queues, Topics and Background Operations | Metric | Unit | Learned | |------|------|---------| | Queue Depth | count | No | | Dead Letter Count | count | No | | Queue Acks | request/s | No | | Message Wait Time | seconds | No | | Queue Size Bytes | bytes | No | | Lag | count | No | | Task Duration | millisecond | No | --- ### Database Tables | Metric | Unit | Learned | |------|------|---------| | DB Query Duration | second | Yes | | Select Query Duration (P95) | millisecond | Yes | | Table Bloat | percent | No | | Lock Exclusive Rate | percent | No | | DDL Lock Exclusive Rate | percent | No | --- ### Application Load Balancers | Metric | Unit | Learned | |------|------|---------| | Request Rate | request/s | Yes | | Request4xx Error | percent | No | | Request5xx Error | percent | No | | Request504 Error | percent | No | | ELB Auth Error | count | No | | Target Connection Error | count | No | --- ### Containers and Controllers | Metric | Unit | Learned | |------|------|---------| | CPU Utilization | percent | No | | CPU Throttled | percent | No | | Memory Utilization | percent | No | | Ephemeral Storage Utilization | percent | No | | Frequent Crash | count | No | | FrequentOOM Kill | count | No | | Frequent Pod Ephemeral Storage Evictions | count | No | --- ### Nodes and Virtual Machines | Metric | Unit | Learned | |------|------|---------| | CPU Utilization | percent | No | | Memory Utilization | percent | No | | Conntrack Table Utilization | percent | No | | SNAT Port Utilization | percent | No | | Container Ephemeral Storage Utilization | percent | No | | Memory Pressure Pod Evictions | count | No | | Disk Pressure Pod Evictions | count | No | | Disk Read IOPS Utilization | percent | No | | Disk Write IOPS Utilization | percent | No | | Disk Total IOPS Utilization | percent | No | | Disk Read Throughput Utilization | percent | No | | Disk Write Throughput Utilization | percent | No | | Disk Total Throughput Utilization | percent | No | --- ### Disks | Metric | Unit | Learned | |------|------|---------| | Utilization | percent | No | | Read IOPS Utilization | percent | No | | Write IOPS Utilization | percent | No | | Total IOPS Utilization | percent | No | | Read Throughput Utilization | percent | No | | Write Throughput Utilization | percent | No | | Total Throughput Utilization | percent | No | | Inodes Utilization | percent | No | ## Best Practices 1. **Start with Default or Learned Thresholds** Use Causely’s default or automatically learned thresholds as a baseline before introducing manual overrides. 2. **Override Only When There Is a Clear Requirement** Configure manual thresholds when you have explicit business, reliability, or compliance requirements that differ from observed behavior. 3. **Prefer Minimum Learned Thresholds Over Full Overrides** When available, use a minimum learned threshold to bound adaptive learning without disabling it entirely. 4. **Consider Both Symptom Detection and SLO Impact** Threshold changes affect both symptom activation and SLI evaluation for SLOs. Validate changes in both contexts. 5. **Monitor After Changes** After updating thresholds, observe how they affect symptom frequency, noise, and SLO evaluation over time. 6. **Document Intent, Not Just Values** Record why a threshold was changed to support future reviews and adjustments. ## Example Use Cases 1. **Strict SLO Requirements** A critical service requires tighter latency bounds than normal traffic patterns allow. A manual threshold is configured to align symptom detection and SLO evaluation with the defined objective. 2. **Preventing Overly Aggressive Learned Thresholds** A service with highly variable traffic uses a minimum learned threshold to prevent latency thresholds from adapting too low during off-peak periods while preserving adaptive behavior. 3. **Infrastructure Saturation Detection** A team configures CPU or disk utilization thresholds on nodes to detect resource saturation early, independent of application-level symptoms. 4. **Queue Backlog Monitoring** Queue depth and message wait time thresholds are configured to surface processing delays before they impact downstream services. 5. **Temporary Adjustments During Maintenance** Thresholds are temporarily adjusted during planned maintenance or migrations and reverted afterward. --- ## Architecture Causely is built on a split-architecture model that balances local control with a dedicated reasoning backend that can run in your cloud or Causely-managed infrastructure. This design ensures low overhead, strong data privacy, and seamless integration with existing tools. This page provides detailed information about Causely's deployment architecture and component structure. For a high-level overview of how Causely's causal reasoning engine works, see [How Causely Works](/getting-started/how-causely-works). ## System Architecture {(() => { const { colorMode } = useColorMode(); const src = colorMode === 'dark' ? '/img/how_causely_works_diagramLight1.svg' : '/img/how_causely_works_diagramDark1.svg'; return ; })()} ## Deployment Architecture Causely can be deployed across various environments, including Kubernetes clusters, standalone Docker hosts, Nomad clusters, and more. The deployment architecture consists of several components that work together to provide real-time root cause analysis. :::info Deployment model Causely is designed for flexible deployment. The mediation layer and agents always run in your environment. The Causely backend (causal engine) can run in your cloud (BYOC) or in Causely-managed infrastructure. ::: ### Mediation Layer The mediation layer is deployed locally in your infrastructure and processes telemetry data to extract only the signals needed for reasoning. It performs: - **Symptom Detection**: Converts telemetry from Prometheus, CloudWatch, Datadog, OpenTelemetry (including eBPF), and other sources into a binary stream of active/inactive [symptoms](/reference/symptoms/). - **Topology Discovery and Ingestion**: Leverages integrated telemetry sources to discover entities and dependencies and ingest topology from systems such as OpenTelemetry, cloud provider APIs and other sources. - **Local Processing**: Processes telemetry locally to minimize data transfer, control cost, and ensure data stays within your environment. Most raw telemetry remains local, with only distilled insights and targeted evidence sent to the Causely backend (deployed in your cloud or Causely-managed) for analysis. The mediation layer primarily sends distilled insights to the cloud. After a root cause is identified, a targeted subset of relevant telemetry (metrics, traces, and log-derived errors/events) may be sent as evidence to enhance root cause clarity. For more detail on supported telemetry sources, see [Supported Telemetry](/telemetry-sources/). The mediation layer consists of the following components: #### Mediator The Mediator is the core component that runs locally in your environment and serves as the data processing layer: - **Symptom Detection**: Converts telemetry from various sources into binary symptom states - **Topology Discovery**: Automatically discovers services, infrastructure, and dependencies - **Local Processing**: Processes telemetry locally, with most raw telemetry remaining in your datacenter - **OTLP Endpoint**: Listens on port 4317 for OpenTelemetry Protocol data The Mediator handles secure communication with the Causely causal reasoning backend (deployed in your cloud or Causely-managed), primarily sending distilled insights. After root causes are identified, a targeted subset of relevant telemetry may be sent as evidence to enhance root cause clarity. The Mediator can also be optionally configured to get metrics from Prometheus or discover and monitor managed cloud services from cloud providers. :::info Mediator scale behavior The Mediator is designed to work from a **statistically significant sample** of traces and metrics—it does not need 100% of raw telemetry to perform accurate analysis. The built-in OpenTelemetry Collector limiter in the Mediator will drop excess traces when memory pressure is reached. This is **intentional behavior**, not data loss. It means the Mediator has collected sufficient data for analysis. Metrics and traces you see in Causely may therefore differ from totals in external monitoring tools—this is expected and does not affect analysis accuracy. Mediator memory scales with the **number of entities under management** (services, pods, infrastructure components), not with trace or metric throughput. A cluster with more services needs more Mediator memory; a high-traffic cluster with the same number of services does not. For sizing guidance, see [Sizing the Mediator for large environments](/installation/customize#sizing-the-mediator-for-large-environments). ::: #### Agents Agents are deployed across your infrastructure to gather node and container level metrics. The deployment method varies depending on your environment: - **Kubernetes**: Agents are deployed as a DaemonSet across all nodes in the cluster - **Docker**: Agents run as containers on standalone Docker hosts - **Nomad**: Agents are deployed as Nomad jobs across the cluster Agents leverage [eBPF](/telemetry-sources/ebpf) technology, which requires privileged access to the host system. This enables automatic instrumentation without code changes. The eBPF instrumentation uses **[uprobes](https://docs.kernel.org/trace/uprobetracer.html) exclusively** to intercept specific user-space functions within your applications—it does not hook into kernel networking callbacks or the packet datapath, and does not act as a [Container Network Interface (CNI)](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/) or network infrastructure component. The instrumentation may inject trace context headers for distributed tracing, but does not intercept, block, or route network traffic at the kernel level. Agents don't establish any outbound connections to the internet or any other service apart from the Mediator and VictoriaMetrics. The agents periodically forward the topology and manifestation data to the Mediator, which, in turn, sends it to the Causely backend for analysis (running in your cloud or Causely-managed). If an agent fails or is removed, your applications and network continue to function normally. ##### Agent Architecture {(() => { const { colorMode } = useColorMode(); const src = colorMode === 'dark' ? '/img/architecture/architecture-detailed-dark.svg' : '/img/architecture/architecture-detailed-light.svg'; return ; })()} #### Executor The Executor is an optional component responsible for executing remediation actions within your infrastructure. The Executor can be enabled as part of the deployment process. The specific permissions required depend on your deployment environment: - **Kubernetes**: The Executor's ServiceAccount is granted the `cluster-admin` role - **Docker/Nomad**: The Executor requires appropriate permissions to execute remediation actions #### VictoriaMetrics VictoriaMetrics is a timeseries database used by the agents and mediator (on `port: 8428`) to store additional timeseries data locally in your environment. If you observe gaps in metrics graphs, VictoriaMetrics may be under-resourced for your environment—see [Sizing VictoriaMetrics for high-volume environments](/installation/customize#sizing-victoriametrics-for-high-volume-environments). ### Causal Engine The Causal Engine runs in the Causely backend, which can be deployed in your cloud or in Causely-managed infrastructure. It receives the stream of symptom states and performs real-time analysis using probabilistic modeling, system graphs, and causal inference. It infers causes, evaluates blast radius, validates constraints, and prioritizes remediation, all without requiring manual correlation. ### Telemetry Sources Causely supports a wide range of telemetry sources, including OpenTelemetry, Prometheus, CloudWatch, Datadog, and more. For a full list of supported telemetry sources, see [Supported Telemetry](/telemetry-sources/). Causely Agents are deployed in your infrastructure and are responsible for collecting the telemetry data from those sources. By default Causely will automatically instrument your applications to receive [OpenTelemetry](/telemetry-sources/opentelemetry) traces. This allows Causely to discover service dependencies, monitor sync and async communication signals. Additionally you can export traces to Causely from your existing OpenTelemetry Collectors. We recommend that you always send OpenTelemetry traces to Causely, as this allows Causely to provide cross-service insights. ### Agent Integration Causely exposes its causal model to AI agents via an MCP server. Any MCP-compatible agent, including Claude Code, Cursor, HolmesGPT, or a custom agent, gets deterministic access to root causes, service health, dependency maps, and blast radius. Instead of reasoning over raw telemetry, agents query structured, causal system knowledge. This reduces token consumption, eliminates guesswork, and lets agents take action with auditable evidence. For details on connecting agents to Causely, see [Agent Integration](/agent-integration/). ### Workflow Integration Causely integrates directly into your tools of choice, delivering causal insights into Slack, Alertmanager, Opsgenie, Grafana, and more. For details on how to connect Causely to your workflows, see [Supported Workflows](/workflows/). This architecture allows Causely to deliver precise, real-time insights without burdening your data pipelines or violating privacy requirements. ## Security Considerations For detailed information about security, permissions, and data handling, see the [Security](/security/) documentation. --- ## First-Time Usage Guide This short guide walks you through the core actions to take right [after setup](/getting-started/quick-setup). No deep configuration or prior knowledge needed, just follow along and start uncovering real insights from your system. ## Understanding the Summary Insight Panel Your journey begins at the [Causely Portal](https://portal.causely.app/welcome), where you'll find the **Summary Insight Panel**. This is your command center for monitoring system health and identifying critical issues. Unlike traditional monitoring systems that bombard you with alerts, Causely focuses on delivering **active root causes** through our innovative **Root Cause Cards**. Each card represents a verified issue in your system, backed by our extensive knowledge base of cloud native failure patterns. :::tip Pro Tip To maximize efficiency, [integrate Causely with your existing tools](/workflows) for seamless workflow integration. ::: ## Exploring Root Cause Details When you click on a Root Cause Card, you'll access the **Root Cause Details View**, your comprehensive dashboard for understanding and resolving system issues. This view is organized into two main sections: ### 1. Summary Tab The Summary tab provides essential information: - **Overview**: Current status, affected services, SLO violations, and timeline - **Root Cause Description**: Clear explanation of the issue - **Impact Analysis**: Detailed assessment of system effects - **Evidence Collection**: - Observed [symptoms](/reference/symptoms/) - System exceptions - Relevant logs - Related events - **Remediation Steps**: Actionable solutions to resolve the issue ### 2. Causality Tab The Causality tab presents a visual representation of the issue's causal path, helping you understand how different system components interact. :::tip Best Practice The Root Cause Details View is your primary investigation tool. In many cases, the suggested remediation steps provide a complete solution to your system issues. ::: ## Connect your AI Agent to Causely Our [MCP Server](/agent-integration/mcp-server/) exposes Causely's causal model to any AI agent: Claude, Cursor, or your own tooling. Once connected, your agent can answer questions like: - "What's causing this root cause?" - "How's this affecting our services?" - "What's the timeline of root cause?" - "What steps should we take to fix this?" [Setup the MCP Server](/agent-integration/mcp-server/) ## Next Steps for Enhanced Root Cause Analysis Now that you've mastered the basics, we recommend: 1. [Connecting additional telemetry sources](/telemetry-sources/) for comprehensive root cause analysis 2. [Setting up workflow integrations](/workflows) with your existing tools --- ## Start with Causely asdf this goes that way --- ## Supported Technologies Causely integrates with your existing infrastructure, observability stack and AI agents. This page provides a complete reference of all supported technologies across four areas: - **[Agents](/agent-integration)**: Give MCP-compatible agents like Claude Code, Cursor, and HolmesGPT deterministic causal context. - **[Installation](/installation)**: Deploy Causely's mediator in Kubernetes, Docker, or other environments - **[Telemetry Sources](/telemetry-sources)**: Connect metrics, traces, logs, and infrastructure data from platforms you already use - **[Workflows](/workflows)**: Route insights to Slack, Microsoft Teams, Grafana, and other tools your team relies on Use the search and filters below to find what you need. :::tip Missing something? [Let us know](mailto:support@causely.app)—we're always expanding our integrations. ::: --- ## Accelerate Resolution This page explains how Causely helps you accelerate incident resolution by providing precise root cause insights when you need them most, reducing alert noise, and enabling faster problem resolution. Once you've [installed Causely](/getting-started/quick-setup), the system automatically generates your topology graph and identifies root causes that explain observed symptoms. For more details on how Causely builds this understanding, see [How Causely Works](/getting-started/how-causely-works). ## Configure Workflow Notifications To receive root cause insights during incidents, configure a workflow integration such as [Slack](/workflows/slack), [Microsoft Teams](/workflows/microsoft-teams), or [Alertmanager](/workflows/prometheus-alertmanager/). See the [workflows overview](/workflows) for all available integrations. ### Noise Reduction When incidents occur, Causely sends notifications for your root causes directly to your configured workflow. Unlike traditional alerting systems that create alert storms with one alert per symptom, Causely sends a single root cause notification that explains multiple symptoms. This dramatically reduces noise and helps you focus on what actually matters. ### Example As an example, Causely sends a message to slack about a [`Service Malfunction`](/reference/root-causes/services#malfunction), which is the cause of 11 symptoms across multiple services and critical customer interactions. ## Use Root Cause View for Fast Resolution The [Root Causes view](/in-action/root-causes) provides detailed information about each identified root cause: - **Evidence**: Metrics, logs, and traces that support the inference - **Impact**: Which services, entities, and operations are affected (blast radius) - **Causality Graph**: Visual representation of how the cause relates to symptoms - **Remediation Guidance**: Actionable remediation steps tailored to the specific issue Urgent root causes are marked red and indicate issues currently impacting your services and SLOs. Use this view to quickly understand what's wrong, why it's happening, and how to fix it. ## Improve Resolution Speed To further accelerate resolution: - **Define service priorities**: Configure [service tiers](/configuration/service-tiers) to help Causely prioritize which services matter most when identifying urgent root causes. - **Connect more telemetry**: Add additional [telemetry sources](/telemetry-sources) to provide more comprehensive coverage and better root cause inference. More telemetry means more accurate and complete causal insights. - **Configure thresholds**: Set up [thresholds](/configuration/thresholds) to fine-tune when symptoms are considered problematic, helping Causely better identify urgent issues. - **Configure symptom activation delay**: Adjust [symptom activation delay](/configuration/symptom-delay) to balance between quick detection and reducing false positives. Shorter delays (1-3 minutes) enable faster root cause identification for critical services, while longer delays (5-10 minutes) help filter out temporary spikes. - **Set up SLOs**: Configure [SLOs](/configuration/slo-configuration) to help Causely understand your reliability goals and better identify when root causes are putting your SLOs at risk. ## Related Features - [Root Causes](/in-action/root-causes) - Explore all identified root causes - [MCP Server](/agent-integration/mcp-server/) - Query the system during incidents - [Topology](/in-action/topology) - Investigate affected entities - [Workflows](/workflows) - Configure notifications --- ## Optimize Performance This page explains how Causely helps you continuously improve system performance by providing actionable insights into bottlenecks, inefficiencies, and optimization opportunities. Once you've [installed Causely](/getting-started/quick-setup), the system automatically generates your topology graph and identifies performance bottlenecks, resource contention, and optimization opportunities. For more details on how Causely builds this understanding, see [How Causely Works](/getting-started/how-causely-works). ## Identify Performance Issues Causely identifies performance issues through root cause analysis that reveals: - **Resource contention and bottlenecks**: Understand where resources are constrained and causing performance degradation - **Inefficient operations and queries**: Identify operations that are consuming excessive resources or taking too long - **Scaling opportunities**: Find services or components that would benefit from scaling - **Optimization targets**: Pinpoint specific areas where performance improvements would have the most impact The [Root Causes view](/in-action/root-causes) shows performance-related root causes with detailed evidence, impact analysis, and remediation guidance. ## Track Performance Trends Use historical root causes and the [Topology view](/in-action/topology) to track performance trends over time and understand: - **How changes impact performance**: See how deployments, configuration changes, or infrastructure updates affect system performance - **Seasonal or cyclical patterns**: Identify recurring performance patterns that might indicate optimization opportunities - **Long-term degradation or improvement**: Track whether performance is improving or degrading over time The [Reliability Delta](/in-action/reliability-delta) feature allows you to compare performance snapshots and track improvements or regressions. ## Get Actionable Recommendations Root causes include specific remediation guidance that helps you optimize performance, such as: - Scaling recommendations for services or infrastructure - Query optimizations for databases - Infrastructure improvements - Configuration adjustments You can also use your favorite AI agent through our [MCP Server](/agent-integration/mcp-server/) to query performance-related questions and get insights about optimization opportunities. ## Improve Performance Optimization To better identify and act on performance optimization opportunities: - **Connect more telemetry**: Add additional [telemetry sources](/telemetry-sources) to provide more comprehensive performance data. More telemetry means better identification of bottlenecks and optimization targets. - **Configure thresholds**: Set up [thresholds](/configuration/thresholds) to fine-tune when performance issues are considered problematic, helping Causely better identify optimization opportunities. - **Define service priorities**: Configure [service tiers](/configuration/service-tiers) to help Causely prioritize which services' performance matters most. - **Automate remediation**: Use [Automate Remediation](/in-action/automate-remediation) to automatically scale resources or take other actions when performance issues are identified. ## Related Features - [Root Causes](/in-action/root-causes) - Find performance-related causes - [Automate Remediation](/in-action/automate-remediation) - Automatically scale resources - [Topology](/in-action/topology) - Analyze entity performance metrics - [MCP Server](/agent-integration/mcp-server/) - Query performance-related questions --- ## What are your goals? , title: 'Accelerate resolution', description: 'Get precise root cause insights during incidents to resolve issues faster.', to: '/goals/accelerate-resolution', }, { icon: , title: 'Proactively prevent incidents', description: 'Identify emerging risks before they impact your services and SLOs.', to: '/goals/prevent-incidents', }, { icon: , title: 'Understand system behavior', description: 'Gain deep insights into why your systems behave the way they do.', to: '/goals/understand-system-behavior', }, { icon: , title: 'Scale reliability', description: 'Maintain reliability across complex, fast-changing systems at scale.', to: '/goals/scale-reliability', }, { icon: , title: 'Optimize performance', description: 'Continuously improve system performance with actionable insights.', to: '/goals/optimize-performance', }, ]} teaser='Discover how Causely helps you achieve your reliability goals and move from reactive troubleshooting to proactive reliability.' /> --- ## Proactively Prevent Incidents This page explains how Causely enables proactive incident prevention by identifying emerging risks before they impact your services and SLOs, helping you address issues before they escalate into incidents. Once you've [installed Causely](/getting-started/quick-setup), the system automatically generates your topology graph and identifies both urgent root causes and non-urgent root causes (emerging risks). For more details on how Causely builds this understanding, see [How Causely Works](/getting-started/how-causely-works). ## Configure Workflow Notifications To receive root cause insights, configure a workflow integration such as [Slack](/workflows/slack), [Microsoft Teams](/workflows/microsoft-teams), or [Alertmanager](/workflows/prometheus-alertmanager/). See the [workflows overview](/workflows) for all available integrations. Causely sends **urgent root causes** directly to your configured workflow when incidents occur, helping you [accelerate resolution](/goals/accelerate-resolution). Additionally, Causely continuously creates **non-urgent root causes** that often call out reliability issues and emerging risks before they escalate into incidents. ## Identify Emerging Risks Causely continuously monitors your environment and identifies potential issues before they escalate into incidents. Non-urgent root causes often represent emerging risks that can be addressed proactively. These are root causes that are active but not yet impacting your services and SLOs. By detecting issues early, Causely helps protect your Service Level Objectives (SLOs) and error budgets. You can address problems before they impact user experience or violate SLO commitments. ## Monitor Trends and Patterns Use the [Root Causes view](/in-action/root-causes) to monitor patterns and trends over time. Review historical root causes to identify: - Recurring issues that might indicate systemic problems - Gradual degradation that might not trigger immediate alerts but could lead to problems - Patterns that help you understand your system's behavior over time The [Reliability Delta](/in-action/reliability-delta) feature helps shift reliability left by allowing you to compare reliability snapshots and track improvements or regressions over time. Before deploying changes, you can create a snapshot to establish a baseline. After deployment, compare the new snapshot to identify any reliability regressions. ## Improve Prevention To better identify and prevent incidents: - **Configure workflow connections**: Set up workflow integrations such as [Slack](/workflows/slack), [Microsoft Teams](/workflows/microsoft-teams), or [incident.io](/workflows/incident-io) to receive both urgent and non-urgent root cause notifications. This ensures you're notified about emerging risks as soon as they're identified. - **Configure SLOs**: Set up [SLOs](/configuration/slo-configuration) to help Causely understand your reliability goals and better identify when root causes are putting your SLOs at risk, even before they become urgent. - **Set thresholds**: Configure [thresholds](/configuration/thresholds) to fine-tune when symptoms are considered problematic, helping Causely identify emerging risks earlier. - **Connect more telemetry**: Add additional [telemetry sources](/telemetry-sources) to provide more comprehensive coverage and better root cause inference. More telemetry means earlier detection of potential issues. - **Define service priorities**: Configure [service tiers](/configuration/service-tiers) to help Causely prioritize which services matter most when identifying risks. ## Related Features - [Reliability Delta](/in-action/reliability-delta) - Compare reliability snapshots - [Root Causes](/in-action/root-causes) - Review non-urgent causes and emerging risks - [Configuration](/configuration) - Set up thresholds and SLOs --- ## Scale Reliability This page explains how Causely helps you maintain reliability across complex, fast-changing systems at scale by automatically understanding your entire environment and reducing alert noise. Once you've [installed Causely](/getting-started/quick-setup), the system automatically generates your topology graph and identifies root causes, even as your system grows and changes. For more details on how Causely builds this understanding, see [How Causely Works](/getting-started/how-causely-works). ## Automatic Discovery at Scale As your system evolves, Causely automatically discovers new services, entities, and relationships without manual configuration. This keeps your reliability insights current as you scale, whether you're: - Deploying new services or applications - Adding infrastructure components - Changing service dependencies - Scaling horizontally or vertically The system continuously updates its understanding of your environment, adapting to new deployments, releases, infrastructure changes, and performance patterns. ## Reduce Alert Noise At scale, alert noise becomes overwhelming. Traditional alerting systems create one alert per symptom, leading to alert storms that make it impossible to identify what actually matters. Causely reduces noise by identifying root causes that explain multiple symptoms. Instead of receiving hundreds of alerts, you get a single root cause notification that explains the underlying issue. This helps you focus on what matters and maintain reliability even as your system grows. ## Improve Reliability at Scale To better maintain reliability as you scale: - **Connect more telemetry**: Add additional [telemetry sources](/telemetry-sources) to provide comprehensive coverage across your entire system. More telemetry means better discovery and more accurate root cause inference. - **Configure scopes**: Set up [scopes](/configuration/scopes) to organize your entities and manage complexity as your system grows. - **Define service priorities**: Configure [service tiers](/configuration/service-tiers) to help Causely prioritize which services matter most, especially important at scale when you have many services. - **Set up SLOs**: Configure [SLOs](/configuration/slo-configuration) to help Causely understand your reliability goals across all your services and better identify when root causes are putting your SLOs at risk. - **Use Reliability Delta**: Track reliability trends over time with [Reliability Delta](/in-action/reliability-delta) to understand how your system's reliability changes as it scales. ## Related Features - [Topology](/in-action/topology) - View all discovered entities - [Reliability Delta](/in-action/reliability-delta) - Track reliability at scale - [Configuration](/configuration) - Configure scopes and service tiers --- ## Understand System Behavior This page explains how Causely provides deep insights into why your systems behave the way they do by building a causal model of your entire environment and making it queryable. Once you've [installed Causely](/getting-started/quick-setup), the system automatically generates your topology graph and builds a comprehensive understanding of your system's behavior and relationships. For more details on how Causely builds this understanding, see [How Causely Works](/getting-started/how-causely-works). ## Explore Your System Topology The [Topology view](/in-action/topology) provides a comprehensive view of your entire system. You can explore: - **Service dependencies and relationships**: Understand how services connect and depend on each other - **Data flow across operations**: See how data moves through your system - **Infrastructure stack**: View the infrastructure components that support each entity - **Historical behavior patterns**: Analyze how your system has behaved over time This topology graph forms the foundation for understanding system behavior, blast radius analysis, root cause attribution, and cross-service impact modeling. ## MCP Server Use your favorite agent with our [MCP Server](/agent-integration/mcp-server/) to query your system's behavior using natural language. Ask questions like: - "Why is service X slow?" - "What changed in the last hour?" - "Show me the dependencies for service Y" - "What's causing the latency in my API?" Causely uses its causal model and topology graph to provide intelligent answers about your system's behavior, helping you understand not just what happened, but why it happened. ## Improve Understanding To gain deeper insights into your system: - **Connect more telemetry**: Add additional [telemetry sources](/telemetry-sources) to provide more comprehensive coverage. More telemetry means a more comprehensive topology graph and better causal insights. - **Configure scopes**: Set up [scopes](/configuration/scopes) to organize your entities and focus on specific parts of your system. - **Use MCP Server regularly**: Regularly query your system to build familiarity with how it behaves and to identify patterns and relationships. ## Related Features - [Topology](/in-action/topology) - Browse system entities and relationships - [MCP Server](/agent-integration/mcp-server/) - Query system behavior - [How Causely Works](/getting-started/how-causely-works) - Learn about the causal model --- ## Automate Remediation # Automate Remediation for Resource Contention Causely allows you to automatically remediate **resource contention** issues directly from the UI, helping you restore performance faster, reduce time to resolve, and keep services within SLOs. When Causely identifies a deterministic Resource Contention root cause, you can trigger automated remediation or apply a guided fix with one click. ## Supported Root Causes Causely supports automated remediation for the following resource-related root causes: ### [CPU Congested](/reference/root-causes/infrastructure/#cpu-congested) Automatically adjust CPU limits when services are experiencing CPU saturation. ### [Frequent Memory Failure](/reference/root-causes/infrastructure/#frequent-memory-failure) Resolve persistent out-of-memory issues caused by memory leaks or inefficient usage. ### [Memory Failure](/reference/root-causes/infrastructure/#memory-failure) Increase memory allocations to resolve out-of-memory issues. ### [Ephemeral Storage Noisy Neighbor](/reference/root-causes/infrastructure/#ephemeral-storage-noisy-neighbor) Isolate and manage containers that excessively consume ephemeral storage, impacting node stability. ### [Memory Noisy Neighbor](/reference/root-causes/infrastructure/#memory-noisy-neighbor) Isolate and manage containers that excessively consume memory, impacting node stability. ### [Congested Services](/reference/root-causes/services/#congested) Scale service resources to handle increased load. ## What Causely Changes Automatically When remediation is executed for a supported Resource Contention root cause, Causely applies a deterministic scaling action based on the type of bottleneck identified: 1. **Vertical Scaling (+50%)**: Increases the affected container’s CPU or memory **requests and limits by 50%**. 2. **Horizontal Scaling (+1 Replica)**: Adds **one additional replica** to the deployment to immediately increase capacity. These adjustments are purposefully conservative and are only applied when Causely’s causal reasoning model confirms that scaling, rather than a correlated symptom, is the correct fix. ## Enabling Automated Remediation (Executor Required) Automated actions require the **executor** to be enabled on the mediator running in the cluster where you want remediation performed. To enable the executor, update your mediator’s `causely-values.yaml`: ```yaml executor: enabled: true ``` Once updated, redeploy or upgrade your mediator so it loads the new configuration. See [using custom values file](/installation/customize/?install-method=helm#use-a-custom-values-file) for details on applying updated values. ## Using the Remediate Now Interface In the UI, supported RCs include a **Remediate now** option that provides: - An acknowledgment step showing the impacted deployment - Auditable action history tied to the entity for which value were updated If you prefer to apply the change manually, the Remediation section for the root cause includes YAML examples you can use. ## Aligning Configuration with MCP Server These remediation updates are applied at runtime, and the MCP Server provides a way to commit these configuration adjustments into your codebase for long‑term consistency. If you want to standardize or persist updated sizing, you can manage configuration through the [MCP Server](/agent-integration/mcp-server/). --- ## Feature Demos This page contains links to Causely feature demos designed to help you understand what you can do and what to expect with Causely. Watch the Full Playlist export const videos = [ { id: 'GiXq71HEGwE', title: 'Causely Feature Demo: Reliability Delta', }, { id: 'nN4Iy5BuC3c', title: 'Solving Slow Database Queries with Causely and its MCP Server', }, { id: 'L-nWJr4tZ7U', title: 'Causely’s MCP Server Brings Reliability into Your IDE | Helm Chart Example', }, { id: 'sk_KmMOF1lE', title: 'Causely Feature Demo: From Root Cause to Business Impact with Causely and ClickStack by ClickHouse', }, { id: 'p07c2gy3baM', title: 'Causely Feature Demo: Accelerate Incident Response with Causely + incident.io', }, { id: 'tT0Ju5vO97w', title: 'Causely Feature Demo: Address External API Slowdowns', }, { id: 'xah1-eSqO4A', title: 'Causely Feature Demo: Solve the Root Cause of Message Queue Lag', }, { id: 'hvJDWHkxieg', title: 'Causely Feature Demo: Pinpointing the Code Change Causing Performance Issues', }, { id: 'D6Ps1VoGHvw', title: 'Causely Feature Demo: Unlock Root Cause Analysis in Grafana', }, ]; ## Latest demo