# Causely documentation: configuration > Full content of the "configuration" section of the Causely documentation. This file contains all documentation content in a single document following the llmstxt.org standard. ## Alert Ingestion and Mapping ### Overview Causely can ingest alerts from your existing monitoring sources and connect them to the causal reasoning engine, without replacing your current alerting setup. When an alert fires, Causely identifies which service or entity it belongs to, maps it to a known Signal, and includes it as evidence in causal analysis. The following alert sources are supported: - [Alertmanager](/telemetry-sources/alertmanager): Prometheus Alertmanager webhook receiver - [Prometheus](/telemetry-sources/prometheus): Direct Prometheus integration - [Datadog](/telemetry-sources/datadog): Datadog monitors and alerts - [Incident.io](/telemetry-sources/incident-io): Incident.io alert events - [Dynatrace](/telemetry-sources/dynatrace): Dynatrace problems and alerts ### How alerts are mapped automatically When Causely receives an alert, it attempts to resolve two things: **which entity** the alert belongs to, and **which Signal** it represents. #### Step 1: Signal resolution via keyword matching Causely examines the alert's `alertname`, title, and description, and matches them against a library of keyword rules. Matching is case-insensitive. The first matching rule wins, so more specific rules (for example, queue-related keywords) are evaluated before broader ones (for example, generic "error"). Here are a few examples of how this works. An alert named `HighErrorRate` or with a description containing "error rate exceeded" is automatically recognized as a **high error rate** condition. An alert containing "latency" or "timeout" is recognized as a **high latency** condition. Similarly, alerts mentioning "kafka lag", "consumer lag", "message wait time", "dead letter", "JVM heap", or "GC time" are each matched to their corresponding condition. Some examples of what Causely looks for in the alert name, title, or description: | Condition | Example keywords | |---|---| | High error rate | "error", "failed", "failure", "exception", "unavailable", "down" | | High latency | "latency", "duration", "timeout", "slow", "response time" | | Kafka consumer lag | "lag" and ("kafka" / "consumer" / "partition") | | Queue wait time | "message wait", "wait time" and "queue" | | Dead-letter queue | "dead letter", "deadletter", "dlx" | | JVM heap pressure | "heap" and ("java" / "jvm") | | Garbage collection | "gc", "garbage collection", "g1 young", "g1 old" | | DB connection pool | "db connection", "database connection", "postgres connection" | | Redis connection pool | "redis connection", "redis connection pool" | #### Step 2: Entity resolution via alert labels A matched Signal still needs an entity to attach to. Causely uses labels in the alert payload to look up the right entity in your topology. Each entity type requires a specific combination of labels that uniquely identifies it: see [Required label combinations by entity type](#required-label-combinations-by-entity-type) below. If the labels are missing or do not correspond to a known entity in topology, the alert remains unmapped even when the Signal is correctly identified. ### What happens after an alert is mapped #### Alert maps to a known entity and Signal The alert is fully ingested. Causely attaches it as an observed Signal to the entity and includes it in causal reasoning. You will see it reflected in the Diagnosis alongside other Signals. #### Alert maps to a known entity but no Signal match If Causely identifies which entity the alert belongs to but does not recognize the alert as a known Signal, the alert is recorded but does not contribute to causal analysis. This happens when the alert name, title, and description do not contain any matching keywords from the table above. Causely engineers are happy to work with you to add important alerts to the knowledge base so they can participate in causal reasoning. Reach out to your Causely team with the alert details. #### Alert does not map to any entity If Causely cannot resolve an entity from the alert, the alert is not attached to the topology. This is almost always caused by **missing or incorrect labels** in the alert payload. See the next section for the exact labels required for each entity type. ### Required label combinations by entity type For an alert to be attached to an entity, its payload must include the labels that uniquely identify that entity. The exact combination depends on the type of entity the alert is about. #### Kubernetes pod **Required:** `namespace` + `pod` Use this when the alert is about a specific pod (for example, high JVM heap, GC time, thread contention, connection pool exhaustion). ```json { "alerts": [ { "status": "firing", "labels": { "alertname": "HighJVMHeapUsage", "namespace": "prod", "pod": "payment-service-7d4f8b-xyz", "severity": "warning" }, "annotations": { "summary": "JVM heap utilization above 80%" }, "startsAt": "2025-02-04T12:00:00.000Z", "endsAt": "0001-01-01T00:00:00Z" } ] } ``` #### Kubernetes container **Required:** `namespace` + `pod` + `container` Use this when the alert targets a specific container within a pod (for example, a sidecar or init container). ```json { "alerts": [ { "status": "firing", "labels": { "alertname": "ContainerHighCPU", "namespace": "prod", "pod": "payment-service-7d4f8b-xyz", "container": "payment-app", "severity": "warning" }, "annotations": { "summary": "Container CPU usage above threshold" }, "startsAt": "2025-02-04T12:00:00.000Z", "endsAt": "0001-01-01T00:00:00Z" } ] } ``` #### Kubernetes service **Required:** `namespace` + `pod` Causely resolves the owning service from the pod via topology. The optional `service` label can be included as a hint but is not required for resolution. ```json { "alerts": [ { "status": "firing", "labels": { "alertname": "HighErrorRate", "namespace": "prod", "pod": "checkout-service-abc", "service": "checkout-service", "severity": "warning" }, "annotations": { "summary": "High error rate on checkout-service", "description": "Error rate increased above threshold in the last 5m" }, "startsAt": "2025-02-04T12:00:00.000Z", "endsAt": "0001-01-01T00:00:00Z" } ] } ``` #### HTTP path **Required:** `namespace` + `pod` + a path label The path label can be any of: `uri`, `url_path`, `http_path`, `path`, `http.route`. The path entity (for example, `/checkout`) must already exist in topology (for example, from ingress or service metrics). ```json { "alerts": [ { "status": "firing", "labels": { "alertname": "HighPathErrorRate", "namespace": "ingress-nginx", "pod": "ingress-nginx-controller-956c769c7-lgcxx", "uri": "/checkout", "severity": "warning" }, "annotations": { "summary": "High error rate on /checkout" }, "startsAt": "2025-02-04T12:00:00.000Z", "endsAt": "0001-01-01T00:00:00Z" } ] } ``` #### gRPC / RPC method **Required:** `rpc_service` + `rpc_method` (or `grpc_service` + `grpc_method`) Optional: `namespace`, `pod`. The RPC method entity (service + method pair) must already exist in topology (for example, from distributed traces). ```json { "alerts": [ { "status": "firing", "labels": { "alertname": "HighGRPCErrorRate", "rpc_service": "user.UserService", "rpc_method": "GetUser", "namespace": "prod", "pod": "user-service-abc", "severity": "warning" }, "annotations": { "summary": "High error rate on user.UserService/GetUser" }, "startsAt": "2025-02-04T12:00:00.000Z", "endsAt": "0001-01-01T00:00:00Z" } ] } ``` #### Kafka topic **Required:** `topic` + consumer group label The consumer group can appear under any of these label names: `consumer_group_id`, `consumer_group`, `group`, or `consumergroup`. Optionally include `namespace` and `pod` when using Micrometer-style instrumentation. **Confluent-style** (topic + consumer group ID only): ```json { "alerts": [ { "status": "firing", "labels": { "alertname": "HighKafkaConsumerLag", "topic": "orders", "consumer_group_id": "order-processor-group", "severity": "warning" }, "annotations": { "summary": "Kafka consumer lag above threshold for topic orders" }, "startsAt": "2025-02-04T12:00:00.000Z", "endsAt": "0001-01-01T00:00:00Z" } ] } ``` **Micrometer-style** (topic + group + namespace + pod): ```json { "alerts": [ { "status": "firing", "labels": { "alertname": "HighKafkaConsumerLag", "topic": "orders", "group": "order-processor-group", "namespace": "default", "pod": "order-processor-0", "severity": "warning" }, "annotations": { "summary": "Kafka consumer lag above threshold for topic orders" }, "startsAt": "2025-02-04T12:00:00.000Z", "endsAt": "0001-01-01T00:00:00Z" } ] } ``` #### Queue (RabbitMQ or similar) **Required:** a label carrying the queue name (for example, `queue`) The queue must already exist in topology (for example, discovered from RabbitMQ or OpenTelemetry). ```json { "alerts": [ { "status": "firing", "labels": { "alertname": "MessageWaitTimeHigh", "queue": "work_queue", "severity": "warning" }, "annotations": { "summary": "Message wait time above threshold on work_queue" }, "startsAt": "2025-02-04T12:00:00.000Z", "endsAt": "0001-01-01T00:00:00Z" } ] } ``` #### Business Application (Concurrent Users Drop) An alert can set the Concurrent Users Drop Signal on a Business Application. This is the same Signal that Causely detects from concurrent users metrics. For details, see [Business KPIs](/configuration/business-kpis). Causely does not match this Signal by keyword. Map the alert with an `alert_mappings` entry, as described in [Alertmanager](/telemetry-sources/alertmanager#optional-map-alerts-to-entities-and-attributes). The `symptom` is `ConcurrentUsers_Low`, and `name` is the alert label that holds the Business Application name: ```yaml alert_mappings: - alert_name: 'CCUDrop' symptom: 'ConcurrentUsers_Low' entity: business_application: {} discovery: - business_application: name: 'domain' ``` An alert like this one is then mapped to the Business Application named `game-a`: ```json { "alerts": [ { "status": "firing", "labels": { "alertname": "CCUDrop", "domain": "game-a", "severity": "warning" }, "annotations": { "summary": "Concurrent users dropped on game-a" }, "startsAt": "2025-02-04T12:00:00.000Z", "endsAt": "0001-01-01T00:00:00Z" } ] } ``` ### Summary | Entity type | Required label combination | |---|---| | Kubernetes pod | `namespace` + `pod` | | Kubernetes container | `namespace` + `pod` + `container` | | Kubernetes service | `namespace` + `pod` (service resolved from pod via topology) | | HTTP path | `namespace` + `pod` + path label (`uri`, `url_path`, `http_path`, `path`, or `http.route`) | | gRPC / RPC method | `rpc_service` + `rpc_method` (or `grpc_service` + `grpc_method`) | | Kafka topic | `topic` + consumer group (`consumer_group_id`, `group`, `consumer_group`, or `consumergroup`) | | Queue | queue name label (for example `queue`) | --- ## Business KPIs A business KPI is a measure of how users use an application. Causely detects when a KPI drops on a Business Application and traces the drop to a diagnosis on one of the application's services. Concurrent users is the only business KPI supported today. ### Step 1: Group services into a Business Application Business Applications group services by the Kubernetes pod label `app.kubernetes.io/part-of`. Causely creates them automatically when it discovers your environment. For details, see [Business Application](/reference/entity-types/applications#business-application). ### Step 2: Map concurrent users through Prometheus Causely maps concurrent users to a Business Application through [Prometheus custom metrics](/telemetry-sources/prometheus#how-to-add-custom-metrics). Use the `ConcurrentUsersMeasured` attribute: ```yaml - entity: { business_application: {} } discovery: - business_application: { name: namespace } metrics: - attribute: ConcurrentUsersMeasured query: sum by (namespace) (mongooseim_session_count) ``` Add this entry under `entities` of a custom exporter, as described in [How to add custom metrics](/telemetry-sources/prometheus#how-to-add-custom-metrics). The label named in `name` must have the same value as the Business Application's name, which comes from the `app.kubernetes.io/part-of` pod label. In this example, the query groups by `namespace`, so each `namespace` value must match a Business Application name. If no Business Application has that name, the metric is not mapped. Change the query and the `discovery` block to match your own metric and Business Application names. ### Set your own lower threshold (optional) Causely uses a lower threshold to decide when concurrent users have dropped. If you know the right value, map the `ConcurrentUsersLowerThresholdMeasured` attribute in the same way as `ConcurrentUsersMeasured`. Causely then uses your value. ### Learned threshold If you do not map a lower threshold, Causely learns one from the application's own history. It uses one day of data, plus the daily history it stores over time. If there is no mapped threshold and no history yet, the Signal stays unknown. The learned bound appears as a related attribute of the Signal. ### Map an alert instead An external alert that is mapped to Concurrent Users Drop can also set the Signal. Causely does not match this Signal by keyword, so you map the alert yourself with an `alert_mappings` entry. For the entry and an example, see [Alert Ingestion and Mapping](/configuration/alert-ingestion#business-application-concurrent-users-drop). ### What you get #### Concurrent Users Drop Signal [Concurrent Users Drop](/reference/symptoms/applications#business-application) is a Signal on the Business Application. It activates after concurrent users stay below the lower threshold for 1 minute. It clears 3 minutes after concurrent users recover. #### Diagnoses that explain a drop A problem on a member service can lock users out of the application, which lowers the Business Application's concurrent users. These diagnoses can now explain a drop: - Service: [Throttling Rejection](/reference/root-causes/services#throttling-rejection), [Rate Limited](/reference/root-causes/applications#rate-limited), [Nginx Worker Connections Exhausted](/reference/root-causes/applications#nginx-worker-connections-exhausted), [HAProxy Backend Unavailable](/reference/root-causes/applications#haproxy-backend-unavailable), [Nginx Upstream Timeout](/reference/root-causes/applications#nginx-upstream-timeout), [Authentication Rejection](/reference/root-causes/services#authentication-rejection), [Policy Rejection](/reference/root-causes/services#policy-rejection), and [Client Connection Disruption](/reference/root-causes/services#client-connection-disruption) - Application Load Balancer: [Authentication Misconfiguration](/reference/root-causes/services#authentication-misconfiguration), [Idle Timeout Misconfiguration](/reference/root-causes/services#idle-timeout-misconfiguration), [Network Policy Misconfiguration](/reference/root-causes/services#network-policy-misconfiguration), and [Unknown Configuration Failure](/reference/root-causes/services#unknown-configuration-failure) - HTTP Path: [Throttling Rejection](/reference/root-causes/datapipeline#throttling-rejection-in-http-path) and [Authentication Rejection](/reference/root-causes/datapipeline#authentication-rejection-in-http-path) - HTTP Path and RPC Method: [Faulty Error Handling in HTTP Path](/reference/root-causes/datapipeline#faulty-error-handling-in-http-path) and [Faulty Error Handling in RPC Method](/reference/root-causes/datapipeline#faulty-error-handling-in-rpc-method) #### Client Connection Disruption requires concurrent users [Client Connection Disruption](/reference/root-causes/services#client-connection-disruption) means existing client connections were dropped, for example by a load balancer or proxy change. Clients then reconnect at the same time. This predicts an Endpoint Request Burst and lost concurrent users. Causely can only identify this diagnosis when the service's Business Application reports concurrent users. If you have not mapped concurrent users, Causely does not identify it. --- ## Credentials Autodiscovery Causely also supports credentials autodiscovery. This feature allows you to add new scraping targets without updating the Causely configuration. Label the Kubernetes Secret to enable autodiscovery for the corresponding scraper. For example, to [enable autodiscovery for the MySQL scraper](/telemetry-sources/mysql#alternative-enable-credentials-autodiscovery), you can label the Kubernetes Secret with the following command: ```bash kubectl --namespace causely label secret mysql-credentials "causely.ai/scraper=MySQL" ``` Credentials Autodiscovery is available for many of the [telemetry sources](/telemetry-sources/). --- ## Custom Causes ### Overview When Causely receives an alert that it does not automatically recognize as a known Signal, you can define a **custom cause** to tell Causely what that alert means. Once defined, every time that alert fires, Causely surfaces it as a Diagnosis with your description and recommended remediation, connects it to any related existing root causes in the causality graph, and clears it automatically when the alert resolves. Custom causes let you bring institutional knowledge about your system into Causely's causal reasoning without writing code or modifying Causely's built-in causal models. ### Prerequisites Before creating a custom cause, verify that the alert carries enough label information for Causely to identify which entity it belongs to. In the **Integrations → Alerts** view, enable the **Entity Only** filter to see only alerts that meet this requirement. Alerts that appear in this filtered view are already resolved to a specific entity in your topology. Alerts that do not appear are missing required labels and cannot be used for custom causes until their labels are corrected. For the exact label combinations required per entity type, please see [Required label combinations by entity type](/configuration/alert-ingestion#required-label-combinations-by-entity-type). ### Creating a custom cause From the **Integrations → Alerts** view, click **Define New Cause** on any alert row, or go directly to **Settings → User Defined Causes** and click **Define New Cause**. #### Choose a category Causely uses the category to understand the failure mode the cause represents. Select one: - **Error (may cause malfunction)**: the alert indicates a problem that causes errors or downtime on the affected entity. - **Latency (may cause congestion)**: the alert indicates a problem that causes increased response time or throughput degradation. The category determines how Causely weighs the cause in causal analysis and how it is presented in the UI. #### Add description and remediation You can write a description and remediation steps manually, or use the **Generate description and remediation with AI** button to have Causely draft them based on the cause name and alert context. If AI generation fails or produces an unsatisfactory result, you can trigger it again at any time from the cause detail view after saving. Click **Create** when done. The definition is active immediately with no restart required. ### Managing your causes All defined causes are listed under **Settings → User Defined Causes**, showing the cause name, matched alert name, category, creator, and creation date. Click any row to edit the display name, description, or remediation. Changes take effect on the next analysis cycle (~30 seconds). The alert name and category cannot be changed after creation. Delete and recreate the definition if you need to change those. ### How active causes appear When an alert matching a custom cause fires, Causely surfaces it as an active Diagnosis alongside built-in causes. Clicking through to a cause shows the full detail view: description, impact graph, and the remediation steps you defined (or that were AI-generated). If the same alert fires on multiple entities simultaneously, the Diagnosis appears independently on each affected entity. ### Deleting a cause Deleting a custom cause definition immediately clears any active Diagnosis of that cause across all entities. It will not reappear even if the underlying alert is still firing. --- ## Configuration ## Configuration Overview Once Causely is deployed and connected to your telemetry, this section covers how to configure how it detects, prioritizes, and reasons about issues in your environment. ### Alerts & Root Causes Configure how external alerts are ingested and how Causely reasons about issues it doesn't automatically recognize: - [**Alert Ingestion and Mapping**](/configuration/alert-ingestion): How Causely maps external alerts to entities and Signals so they participate in causal analysis - [**Custom Causes**](/configuration/custom-causes): Define your own causes for alerts Causely does not automatically recognize ### Reliability Prioritization Configure which services matter most and how aggressively Causely detects and surfaces issues: - [**Service Prioritization**](/configuration/service-tiers): Assign priority tiers to services so alerts and Diagnoses reflect what matters most - [**SLO Targets and Burn Rates**](/configuration/slo-configuration): Configure SLO targets and burn rate behavior for services, HTTP paths, RPC methods, and queues - [**Business KPIs**](/configuration/business-kpis): Map concurrent users to a Business Application so Causely can detect drops and explain them - [**Thresholds**](/configuration/thresholds): Configure the thresholds used for Signal detection and SLI-based SLO evaluation - [**Signal Delay**](/configuration/symptom-delay): Configure activation and deactivation delays to avoid reacting to temporary spikes Many of these settings can be configured through the [Causely MCP server](/agent-integration/mcp-server/), the Causely UI, Kubernetes/Nomad/Consul labels or the API. See each page for the methods it supports. ### Views - [**Scopes**](/configuration/scopes): Define custom scopes to focus on the subset of your topology that matters most to your role or current task ### Discovery - [**Credentials Autodiscovery**](/configuration/credentials-autodiscovery): Add new scraping targets without updating the Causely configuration, by labeling the corresponding Kubernetes Secret ### General Settings Accessible through the Causely UI under [**Preferences**](https://portal.causely.app/settings?tab=general): - **Screen View Mode**: Choose the layout mode for large screens. - **Auto Refresh Interval**: Choose the frequency at which the user interface views should refresh automatically. ### Deployment Configuration The Helm values file (`causely-values.yaml`) is the primary method for configuring your Causely deployment. For a complete list of available configuration options, you can view your current values using: ```bash helm get values causely --namespace causely --all ``` Read more about the [Helm values](/installation/helm) configuration. --- ## Scopes ### Overview Scopes in Causely allow you to define and manage custom subsets of your environment's topology. As Causely automatically discovers the full topology of your environment, it can present a rich but potentially overwhelming set of entities—services, infrastructure components, and identified problems. Scopes help you focus on the specific subset of data that matters most to your role, responsibilities, or current investigative tasks. Scopes can be used in conjunction with service ownership to create more focused views of your environment. While service ownership defines who is responsible for different components, scopes allow you to create custom views that may or may not align with ownership boundaries. ### What are Scopes? User Scopes define the global context for your session, determining what data is accessible across all views. You can create, save, and manage custom Scopes to focus on specific subsets of data, such as clusters or namespaces. Scopes are persistent, can be set as default, and adapt dynamically to your selections during navigation. Scopes can be created based on specific components out of the box, such as clusters, namespaces, and services. Additionally, Causely supports the use of custom labels to provide users with more granular or domain-specific scope options. See [Using Custom Labels for Scopes](#using-custom-labels-for-scopes) for details. ### Selecting a Scope In the top navigation bar, you can select the scope you want to use for your session. ### Managing Scopes To manage your scopes: 1. Click the gear icon (⚙️) in the top right corner of the Causely UI 2. Select "Settings" from the dropdown menu 3. Navigate to the "Scopes" section #### Creating a New Scope 1. In the Scopes settings page, click the "Create" button 2. Define your scope by selecting the relevant entities and components 3. Give your scope a descriptive name 4. Save your scope #### Editing a Scope 1. Find the scope you want to edit in the Scopes list 2. Click the edit icon next to the scope 3. Modify the scope's configuration as needed 4. Save your changes #### Deleting a Scope 1. Find the scope you want to delete in the Scopes list 2. Click the delete icon next to the scope 3. Confirm the deletion ### Best Practices - Create scopes that align with your team's responsibilities and organizational structure - Use descriptive names that clearly indicate the scope's purpose - Regularly review and update your scopes to ensure they remain relevant - Consider creating scopes for different investigation scenarios or team roles ### Using Custom Labels for Scopes Out of the box, Causely provides scopes for specific components, for example cloud providers, clusters, namespaces, and service types. A cluster admin can configure Causely to automatically detect entities with custom labels and make them available as scope options in the UI. The following labels are supported: - Geography - Environment - Customer - Team - Product - Project - Service For details on how to add and use custom labels to enhance scopes, refer to the [custom labels documentation](/installation/customize/#custom-labels-for-scope-configuration). --- ## Service Prioritization ### Overview The **Service Prioritization** feature allows you to assign priority tiers to services discovered within a given scope. By default, all services are assigned a Service Level Objective (SLO) and treated as **Critical**. However, not all services are equally important. Some require immediate attention, while others can be deprioritized or hidden entirely. This feature helps teams focus on what matters, ensuring that high-priority services drive alerts and incident response while low-priority or internal services remain visible but do not create unnecessary noise. ### How It Works Each service in a scope can be categorized into one of three **Service Tiers**: | Tier | Description | | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Critical: SLO Enforced** | Services essential to your business or user experience. Their SLOs are actively enforced. If their SLO is violated or at risk, related Issues are marked **Urgent**. | | **Non-Critical: SLO Not Enforced** | Services that are non-essential or low-impact. These services do not have an active SLO, and their Issues appear in the **Issues** view but are never marked urgent. | | **Hidden: Internal Noise** | Internal or auxiliary services you do not want visible in daily operations. Issues impacting these services appear only under the **Hidden** tab in the Issues view. | ### Default Behavior - Newly discovered services start as **Uncategorized**, which defaults to **Critical: SLO Enforced**. - All services in this state are monitored with an active SLO until explicitly reassigned to another tier. ### Assigning Service Tiers with MCP or the API You can assign tiers through the [Causely MCP server](/agent-integration/mcp-server/) or the Causely API. Both use the following tier values: | Tier in the UI | MCP and API value | | --- | --- | | **Critical: SLO Enforced** | `sloEnabled` | | **Non-Critical: SLO Not Enforced** | `sloDisabled` | | **Hidden: Internal Noise** | `hidden` | | **Uncategorized** | `unassigned` | Tiers are not limited to services. Setting `sloEnabled` on an HTTP path, RPC method, or queue enables SLO tracking for that entity. See [SLO Targets and Burn Rates](/configuration/slo-configuration/) for how to tune those SLOs. #### Using the Causely MCP Server The following MCP tools manage tiers. They require the Developer or Administrator role. | Tool | Use | | --- | --- | | `get_service_tier` | Check the tier assigned to a service or other entity | | `set_service_tier` | Assign a tier, or enable SLO tracking on an HTTP path, RPC method, or queue | | `clear_service_tier` | Remove the tier assignment and revert to the default | For example, you can ask your agent: > Mark the internal-metrics-exporter service as hidden. :::note Clearing the tier on an HTTP path, RPC method, or queue also removes its SLO. To keep the tier record but stop SLO tracking, set the tier to `sloDisabled` instead. ::: #### Using the Causely API Tiers are stored as entity configurations of type `EntityTier`. Use the `createEntityConfigs` mutation to assign a tier and `deleteEntityConfig` to remove it. See [Setting SLOs on Paths and Methods](/api/queries/setting-slos-on-paths-and-methods/) for complete examples. ### Assigning Service Tiers in the UI 1. Navigate to **Settings → Service Tiers**. 2. Search or filter for the services you want to prioritize. Note that the available services depend on the currently selected scope. 3. Select one or more services. 4. Use the **Tier Assignment** panel to set a tier: - **Critical: SLO Enforced** - **Non-Critical: SLO Not Enforced** - **Hidden: Internal Noise** You can assign tiers individually or in bulk across pages. ### Diagnosis Visibility Tier assignment affects how Diagnoses appear in **Root Causes**: - **Critical:** Appears in **Active**. Marked **Urgent** if the SLO is violated or at risk. - **Non-Critical:** Appears in **Active** but never marked urgent. - **Hidden:** Appears only in the **Hidden** tab, not in **Active** or **Historical**. ### Example Use Cases - **Critical:** Core APIs, payments, authentication, customer-facing services. - **Non-Critical:** Internal dashboards, staging workloads, low-traffic utilities. - **Hidden:** Background jobs, test services, deprecated components. --- ## SLO Targets and Burn Rates Service Level Objectives (SLOs) are how Causely translates reliability signals into **urgency and action**. SLOs define what “good” looks like for reliability across services, endpoints, and queues, and are used by Causely to determine when degradations represent acceptable risk versus issues that require immediate attention. In Causely, SLOs directly influence how Diagnoses are classified and prioritized. When a Diagnosis puts an SLO at risk or violates it, Causely treats that Diagnosis as more urgent, helping teams focus on the issues most likely to impact users and the business. SLOs are applied **by default at the service level**, providing broad coverage with minimal configuration. For teams that need more granular protection, SLOs can also be defined for specific **HTTP paths**, **RPC methods**, and **queues**, allowing critical user flows or business transactions to be protected independently of overall service health. Queue SLOs are enabled through the MCP server or API. ### Default SLO Behavior By default, Causely applies the following SLO targets and burn rate settings to all services: - **Error rate SLO target**: 99.0% (99% of requests must be successful) - **Latency SLO target**: 95.0% (95% of requests must be under the latency threshold) - **Availability SLO target**: 99.0% (99% uptime expected) - **Burn rate threshold**: 4 (budget would be consumed in 6 hours for a 1-day SLO) - **Burn rate window**: 15 minutes (calculation window for burn rate monitoring) These defaults are designed to catch fast-burning reliability issues early, while avoiding unnecessary noise for brief or low-impact fluctuations. ### When SLOs Are Active SLOs in Causely are evaluated only when **traffic is observed** for the corresponding entity. This applies consistently to services, HTTP paths, RPC methods, and queues. SLOs measure how an entity performs when responding to real requests. If no traffic is observed, there is no performance to evaluate, and the SLO remains inactive until requests are seen. ### Customizing SLO Behavior You can customize how SLOs behave in Causely depending on the level of control you need: - **Service-level SLOs** can be customized through the Causely MCP server or API, or using labels or service metadata. The MCP server and API are described in [Configuring Targets and Burn Rates with MCP or the API](#configuring-targets-and-burn-rates-with-mcp-or-the-api); labels are described in [Supported Labels](#supported-labels). - **HTTP Path and RPC Method SLOs** are enabled through the `set_service_tier` MCP tool or the API. These SLOs follow the same core concepts (targets, burn rates, and windows) but apply to specific endpoints rather than entire services. See [Setting SLOs on Paths and Methods](/api/queries/setting-slos-on-paths-and-methods/) for details. - **Queue SLOs** are enabled through the `set_service_tier` MCP tool or the API. These SLOs follow the same core concepts (targets, burn rates, and windows) but apply to specific queues, such as order-processing or event ingestion queues. When a queue SLO is at risk or violated, Diagnoses affecting that queue or its upstream dependencies are automatically elevated as urgent. - **Default SLO values** can be changed for every SLO at once using a `GLOBAL` threshold configuration through the MCP server or API. See [Configuring Targets and Burn Rates with MCP or the API](#configuring-targets-and-burn-rates-with-mcp-or-the-api). To enable or disable SLOs for a service or other entity, see [Service Prioritization](/configuration/service-tiers/). ### Configuring Targets and Burn Rates with MCP or the API You can set the SLO target and burn rate threshold directly on an SLO through the [Causely MCP server](/agent-integration/mcp-server/) or the Causely API. Both use [threshold configurations](/configuration/thresholds/), with two SLO-specific keys: | Key | Description | Range | Default | | ----------------- | -------------------------------------------------------------------------------------- | ---------- | ---------------------------- | | `TargetThreshold` | SLO target, as a percentage. For example, `99.9` means 99.9% of requests must be good. | 0–100 | `99` (`95` for latency SLOs) | | `BurnRate` | Burn rate threshold at which the SLO is marked at risk. | Multiplier | `4` | These keys apply to the following SLO types: | SLO type | Description | | ------------------- | ---------------- | | `SuccessSLO` | Error rate SLO | | `LatencySLO` | Latency SLO | | `AvailabilitySLO` | Availability SLO | | `MessageBacklogSLO` | Queue SLO | :::note `TargetThreshold` is a percentage on a 0–100 scale (`99.9`), unlike the Request Error Rate threshold, which is a fraction (`0.01`). ::: A configuration can apply to a single SLO (`INDIVIDUAL` scope), to SLOs that match a label selector (`LABEL` scope), or to every SLO (`GLOBAL` scope). Use `GLOBAL` to change the default target or burn rate for your whole environment. When both a label (for example, `causely.ai/error-rate-slo-target`) and an MCP or API configuration set a value for the same SLO, the MCP or API configuration takes precedence. Deleting the configuration reverts the SLO to its label-configured or default value. #### Using the Causely MCP Server The following MCP tools manage SLO targets and burn rates. They require the Developer or Administrator role. | Tool | Use | | --------------------------------- | -------------------------------------------------------------------------------- | | `get_slo` | Find an SLO's ID and check its error budget and burn rate | | `get_supported_threshold_metrics` | List the keys supported for an SLO type, for example `entity_type: "LatencySLO"` | | `create_threshold_configuration` | Set `TargetThreshold` or `BurnRate` on an SLO | | `get_threshold_configurations` | List existing configurations for an SLO | | `update_threshold_configuration` | Change the values of an existing configuration | | `delete_threshold_configuration` | Remove a configuration and revert to the label-configured or default value | For example, you can ask your agent: > Set the latency SLO target for the checkout service to 99% and lower its burn rate threshold to 2. The agent looks up the service's latency SLO with `get_slo`, then calls `create_threshold_configuration`: ```json { "scope": "INDIVIDUAL", "target_entity_id": "", "thresholds": { "TargetThreshold": 99.0, "BurnRate": 2 } } ``` #### Using the Causely API Use the `createThresholdConfiguration` mutation with the same keys. See the [API overview](/api/) for authentication and client setup. ```graphql mutation CreateThresholdConfiguration($input: CreateThresholdConfigurationInput!) { createThresholdConfiguration(input: $input) { id scope targetEntityId thresholds } } ``` ```json { "input": { "scope": "INDIVIDUAL", "targetEntityId": "", "thresholds": { "TargetThreshold": 99.9, "BurnRate": 2 } } } ``` To change the values later, use `updateThresholdConfiguration(id, input)`. To remove the configuration, use `deleteThresholdConfiguration(id)`. ### Supported Labels You can configure the following SLO-related labels: | Label | Description | Default | | --------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | `causely.ai/error-rate-slo-target` | Percentage of successful requests expected, for example 99.0. This defines the percentage of requests that must not result in an error to remain within the error SLO. | `99.0` | | `causely.ai/latency-slo-target` | Percentage of requests expected to be under the latency threshold, for example 95.0. Note that the latency threshold is automatically learned by Causely, but can be manually adjusted via the [Thresholds configuration page](/configuration/thresholds/). | `95.0` | | `causely.ai/availability-slo-target` | Percentage of time the service is expected to be operational, for example 99.0. This defines the proportion of total time the service must remain available, responding successfully to requests without downtime, to remain within the availability SLO. | `99.0` | | `causely.ai/error-rate-burn-rate-threshold` | Rate of error budget burn relative to the SLO target. A default of 4 means that for a 1-day SLO, if errors continue at the current rate, the error budget would be consumed in 6 hours. | `4` | | `causely.ai/latency-burn-rate-threshold` | Rate of latency budget burn relative to the SLO target. A default of 4 means that for a 1-day SLO, latency at the current rate would consume the entire budget in 6 hours. | `4` | | `causely.ai/availability-burn-rate-threshold` | Rate of availability budget burn relative to the SLO target. A default of 4 means that for a 1-day SLO, availability at the current rate would consume the entire budget in 6 hours. | `4` | | `causely.ai/error-rate-burn-rate-window` | Burn rate calculation window (in minutes) used to indicate whether a service is rapidly consuming its error SLO budget. | `15` | | `causely.ai/latency-burn-rate-window` | Burn rate calculation window (in minutes) used to indicate whether a service is rapidly consuming its latency SLO budget. | `15` | | `causely.ai/availability-burn-rate-window` | Burn rate calculation window (in minutes) used to indicate whether a service is rapidly consuming its availability SLO budget. | `15` | ### Configuring with Labels and Service Metadata #### Using Kubernetes Labels You can apply labels directly to your Kubernetes services: ```bash # Set error rate SLO target to 99% kubectl label svc -n "causely.ai/error-rate-slo-target=99.0" # Set latency SLO target to 95% kubectl label svc -n "causely.ai/latency-slo-target=95.0" # Set availability SLO target to 99% kubectl label svc -n "causely.ai/availability-slo-target=99.0" # Set error rate burn rate threshold to 2 kubectl label svc -n "causely.ai/error-rate-burn-rate-threshold=2" # Set latency burn rate threshold to 2 kubectl label svc -n "causely.ai/latency-burn-rate-threshold=2" # Set availability burn rate threshold to 2 kubectl label svc -n "causely.ai/availability-burn-rate-threshold=2" # Set error rate burn rate window to 15 minutes kubectl label svc -n "causely.ai/error-rate-burn-rate-window=15" # Set latency burn rate window to 15 minutes kubectl label svc -n "causely.ai/latency-burn-rate-window=15" # Set availability burn rate window to 15 minutes kubectl label svc -n "causely.ai/availability-burn-rate-window=15" ``` #### Using Nomad Service Tags If you use Nomad, you can specify these as service tags: ```hcl job "example" { group "app" { service { name = "my-service" port = 8080 tags = [ "causely.ai/error-rate-slo-target=99.0", "causely.ai/latency-slo-target=95.0", "causely.ai/availability-slo-target=99.0", "causely.ai/error-rate-burn-rate-threshold=2", "causely.ai/latency-burn-rate-threshold=2", "causely.ai/availability-burn-rate-threshold=2", "causely.ai/error-rate-burn-rate-window=15", "causely.ai/latency-burn-rate-window=15", "causely.ai/availability-burn-rate-window=15" ] } } } ``` #### Using Consul Service Metadata For Consul services, you can configure these using service metadata: ```bash # Register a service with slo metadata consul services register \ -name="my-service" \ -port=8080 \ -meta="causely.ai/error-rate-slo-target=99.0" \ -meta="causely.ai/latency-slo-target=95.0" \ -meta="causely.ai/availability-slo-target=99.0" \ -meta="causely.ai/error-rate-burn-rate-threshold=2" \ -meta="causely.ai/latency-burn-rate-threshold=2" \ -meta="causely.ai/availability-burn-rate-threshold=2" \ -meta="causely.ai/error-rate-burn-rate-window=15" \ -meta="causely.ai/latency-burn-rate-window=15" \ -meta="causely.ai/availability-burn-rate-window=15" # Update existing service metadata consul services register \ -id="my-service-id" \ -name="my-service" \ -port=8080 \ -meta="causely.ai/error-rate-slo-target=99.0" \ -meta="causely.ai/latency-slo-target=95.0" \ -meta="causely.ai/availability-slo-target=99.0" \ -meta="causely.ai/error-rate-burn-rate-threshold=2" \ -meta="causely.ai/latency-burn-rate-threshold=2" \ -meta="causely.ai/availability-burn-rate-threshold=2" \ -meta="causely.ai/error-rate-burn-rate-window=15" \ -meta="causely.ai/latency-burn-rate-window=15" \ -meta="causely.ai/availability-burn-rate-window=15" ``` ### Best Practices 1. **Align with SLO policy**: Reflect organizational reliability goals. 2. **Avoid overly aggressive thresholds**: High sensitivity may create alert fatigue. 3. **Monitor and adjust**: Tune thresholds based on incident reviews and error budget consumption. 4. **Document changes**: Record rationale for each SLO configuration. ### Example Use Cases 1. **Business-critical services**: Set tighter SLO targets, for example 99.9% success, 98% low-latency. 2. **Temporary adjustments**: Raise burn rate thresholds during high-traffic events. ### Queue SLO Example Queues often represent critical business workflows, such as order processing or asynchronous event handling. For example, if you need to ensure that an order-processing queue is drained within a defined time window, you can define an SLO using **Queue Depth** (or a related metric) as the SLI via the API. When that queue’s SLO is at risk or violated, Causely elevates related Diagnoses as urgent and surfaces the most relevant causal explanations, helping teams prioritize issues that directly impact key workflows. ### Burn Rate Threshold Examples The burn rate threshold determines how aggressively your error or latency budget is being consumed, and helps you catch fast-burning issues. Here's a simple example: Suppose your service has a 1-day SLO budget, meaning it can tolerate a limited amount of errors or latency over 24 hours. - **Burn Rate Threshold = 1** The current rate of errors or latency is steady and would use up the entire budget in exactly 24 hours. No alarm yet, but you're tracking close to your SLO target. - **Burn Rate Threshold = 2** At the current rate, the service would consume its full 24-hour budget in only **12 hours**, triggering alerts about rapid budget consumption. - **Burn Rate Threshold = 4** This indicates extremely fast-burning behavior. At this pace, the full error or latency budget would be used up in just **6 hours**. In practice, burn rate thresholds allow teams to catch reliability problems earlier, before they fully consume the SLO budget. ### Burn Rate Window Examples The burn rate window helps determine how quickly your service is consuming its SLO budget by observing behavior over short time intervals. Below are simple examples to clarify: - **Short Window (5 minutes)** Useful for detecting rapid error or latency spikes. For example, if a service suddenly begins failing or slowing down at a high rate, a short burn rate window (like 5 minutes) helps you identify that it's quickly consuming its SLO budget, enabling earlier incident detection. For services where fast detection of degraded performance is critical, consider also shortening the Signal activation delay, which you can manually configure via the [Signal Delay](/configuration/symptom-delay) settings. - **Moderate Window (15 minutes)** This is the default and provides a good balance between reactivity and noise. It captures bursts of errors or latency that might not last long enough to trigger alerts in a longer window but are still significant. - **Long Window (60 minutes)** Best used to detect sustained SLO violations. For example, if a service has a consistent error rate that slowly drains the budget, the longer window provides better confidence that it’s not just a transient blip. --- ## Signal Delay Causely automatically detects service [Signals](/reference/symptoms/) based on various metrics. To avoid alerting on brief spikes or temporary blips, Causely uses **activation and deactivation delays**: a Signal must remain in violation of thresholds for a sustained period before it activates, and must return to normal for the same period before it clears. This document explains how these delays work and how to configure them for your services. ### How Signal Delays Work Signal delays serve two purposes: 1. **Prevent false positives**: Brief metric spikes don't trigger Signals or alerts; only sustained issues are surfaced. 2. **Provide stability**: Signals remain active until the issue is genuinely resolved, reducing flapping and noise. The Signal activation delay depends on the *type* of issue: - **Bursty issues** (sudden spikes): the 5-minute average is > a multiple of the configurable threshold - **Sustained issues** (slow creep above threshold): the 30-minute average is > the configurable threshold The default activation delay is 5 minutes. A Signal must match one of the above conditions for 5 minutes. Deactivation also uses 5 minutes. During this 5 minute period the 5-min average must be below the higher threshold and the 30-min average must be below the threshold. ### Request Error Rate Signal Causely activates a **Request Error Rate** Signal when either of these conditions is met: - **Condition 1: Bursty spike**: 5-minute average error rate > 4× the threshold (default: 1–2%) - **Condition 2: Sustained elevation**: 30-minute average error rate > threshold In either case, the condition must be true and the request rate must be > 0.3 req/sec (0.2 req/sec for HTTP Path or RPC Method) for 5 consecutive minutes. **Deactivation:** The Signal clears in either case: - **Recovery**: Neither of the above condition 1 nor 2 holds for 5 consecutive minutes. - **Silence**: The 5-min and 30-min average request rate is ≤ 0.3 req/sec (0.2 req/sec for HTTP Path or RPC Method); deactivates regardless of error rate. **Practical Effect:** A sudden 4× error rate spike that holds for 5+ minutes triggers quickly. A service that creeps above threshold gradually requires 30 minutes of sustained violation before alerting. Once fixed, it clears after a matching period of recovery. ### Request Duration (Latency) Signal Causely activates a **Request Duration** Signal when either of these conditions is met: - **Condition 1: Bursty spike**: 5-minute average latency > 1.5× threshold - **Condition 2: Sustained elevation**: 30-minute average latency > threshold In either case, the condition must be true and the request rate must be > 0.3 req/sec (0.2 req/sec for HTTP Path or RPC Method) for 5 consecutive minutes. **Deactivation:** The Signal clears if neither of the above condition 1 nor 2 holds for 5 consecutive minutes. **Practical Effect:** A sudden 1.5× latency spike that holds for 5+ minutes triggers quickly. A service that slowly creeps above its baseline requires 30 minutes of sustained violation before alerting. Once fixed, it clears after a matching period of recovery. ### Activation Delay Configuration The activation delay for Service-level Request Error Rate and Request Duration Signals is configurable. Configurations do not apply to HTTP Path or RPC Method Signals. For threshold configuration (default thresholds for triggering Signals), see [Threshold configuration](/configuration/thresholds/). ### Configuration Methods You can configure activation delay using the following methods: - **Causely UI** Best for making a targeted adjustment while looking at the metric that's driving a Signal, without touching service metadata or deployment configuration. - **Service metadata** Lets you configure activation delay declaratively using Kubernetes labels, Nomad service tags, or Consul service metadata. Well suited for version-controlled, environment-specific configuration that travels with your service definition. - **Causely API** Best for programmatic configuration, automation, and integration with internal tooling or workflows. :::note The Causely MCP server does not currently provide a tool for configuring activation delay. Use the UI, service metadata, or API instead. ::: #### Using the Causely UI To configure activation delay in the UI: 1. Navigate to the service you want to configure. 2. Select the **Metrics** tab, then find the card for the relevant Signal metric (for example, **Request Duration** or **Request Error Rate**). 3. Click the pencil icon on the metric card to open its configuration menu, then select **Set Activation Delay**. 4. Enter the activation delay in minutes and select **Save Activation Delay**. This sets an individual override for the Signal on that entity. Leaving the field empty reverts the Signal to its default (or label-configured) activation delay. #### Using Service Metadata ##### Using Kubernetes Labels Apply labels to your services: ```bash # Configure error rate activation delay (example: 3 minutes) kubectl label svc -n "causely.ai/error-rate-activation-delay=3" # Configure latency activation delay (example: 3 minutes) kubectl label svc -n "causely.ai/latency-activation-delay=3" ``` ##### Using Nomad Service Tags Add tags to your service definition: ```hcl job "example" { group "app" { service { name = "my-service" port = 8080 tags = [ "causely.ai/error-rate-activation-delay=3", "causely.ai/latency-activation-delay=3" ] } } } ``` ##### Using Consul Service Metadata Register services with metadata: ```bash consul services register \ -name="my-service" \ -port=8080 \ -meta="causely.ai/error-rate-activation-delay=3" \ -meta="causely.ai/latency-activation-delay=3" ``` ##### Delay Values - **Valid Range:** 1–60 minutes - **Configures:** Bursty activation delay only (default: 5 minutes) - **Fixed:** Sustained activation delay (always 30 minutes) - **Recommended Range:** 1–10 minutes for most use cases #### Using the Causely API Activation delay can also be configured programmatically through the Causely GraphQL API, at the individual entity (`INDIVIDUAL`), label (`LABEL`), or global (`GLOBAL`) scope. This is best suited for automation, bulk configuration, or integrating activation delay changes into internal tooling. See the [API overview](/api/) for authentication and client setup. Use the `createScopedConfiguration` mutation with the `SYMPTOM_DELAY` configuration type. Delays are keyed by indicator name, `RequestErrorRate` or `RequestDuration`, and values are in minutes: ```graphql mutation CreateSignalDelay($targetEntityId: String, $value: JSON!) { createScopedConfiguration( configType: SYMPTOM_DELAY scope: INDIVIDUAL targetEntityId: $targetEntityId value: $value ) { id scope targetEntityId } } ``` ```json { "targetEntityId": "", "value": { "activationDelays": { "RequestErrorRate": 3, "RequestDuration": 3 } } } ``` To change the values later, use `updateScopedConfiguration(id, value)`. To remove the configuration, use `deleteScopedConfiguration(id)`. :::note If a service also has a `causely.ai/error-rate-activation-delay` or `causely.ai/latency-activation-delay` label, the label takes precedence over the API configuration for that Signal. ::: ### Best Practices 1. **Start with defaults**: 5 minutes works for most services. Only adjust if you're seeing false positives (bursty spikes that recover quickly) or need faster detection of genuine issues. 2. **Shorter delays (1–3 min) for**: - Payment or high-revenue services where quick spike detection is critical - Services where you expect sudden issues to be real problems, not transient noise 3. **Longer delays (5–10 min) for**: - Services with frequent harmless spikes, for example, traffic bursts, cache refreshes - Noisy services where bursty patterns are normal and don't indicate problems - Non-production environments 4. **Keep in mind**: The 30-minute sustained delay is fixed and cannot be adjusted. This ensures slow, creeping issues are not misclassified as false positives. 5. **Monitor after changes**: After adjusting bursty delays, observe whether spike detection improves or false positives decrease. 6. **Document your reasoning**: Keep notes on why you chose specific delays per service. This helps during onboarding and reviews. ### Examples #### Critical Service (Payment Processing) Faster detection of error spikes: **Kubernetes:** ```bash kubectl label svc -n production payment-api "causely.ai/error-rate-activation-delay=2" kubectl label svc -n production payment-api "causely.ai/latency-activation-delay=2" ``` **Effect:** Error rate spikes are detected after 2 minutes of sustained elevation (vs. default 5). Slow degradation still requires 30 minutes. #### Bursty Service (Traffic Spikes) Reduce noise from normal traffic bursts: **Kubernetes:** ```bash kubectl label svc -n production data-processor "causely.ai/error-rate-activation-delay=8" kubectl label svc -n production data-processor "causely.ai/latency-activation-delay=8" ``` **Effect:** Brief error spikes during load bursts don't trigger until 8 minutes (giving time to recover). Slow issues still activate after 30 minutes. #### Development Environment Longer delays to minimize disruptions: **Kubernetes:** ```bash kubectl label svc -n dev api-service "causely.ai/error-rate-activation-delay=10" kubectl label svc -n dev api-service "causely.ai/latency-activation-delay=10" ``` **Effect:** Dev spikes require 10 minutes to trigger (vs. 5). Sustained issues always take 30 minutes, same as production. --- ## Thresholds for Signals Causely detects service and infrastructure [Signals](/reference/symptoms/) based on a wide range of metrics, using a combination of defaults and learned behavior. However, you may want to customize these thresholds to better match your specific requirements and SLO definitions. This document explains how to configure custom thresholds for your services. ### Overview Causely uses thresholds to detect service and infrastructure Signals across a wide range of metrics, including latency, error rates, throughput, and resource utilization. These thresholds serve two related purposes: - **Signal detection**, where crossing a threshold activates a Causely issue or risk - **SLO evaluation**, where the same metrics act as Service Level Indicators (SLIs) that determine SLO health Causely provides sensible defaults for all supported thresholds. For some metrics, Causely can also **learn thresholds automatically** based on historical and real-time behavior. You can optionally configure **manual thresholds** or tune learned thresholds to better reflect business requirements, reliability targets, or known system constraints. ### Key Concepts #### Threshold Sources Each threshold in Causely has a **source**, which determines how its value is set and maintained: - **Default** A system-provided threshold value that applies when no learning or manual override is configured. Defaults are designed to be safe and broadly applicable. - **Learned** For some metrics, Causely can automatically learn a threshold based on historical and real-time behavior. Learned thresholds adapt over time as normal behavior changes. - **Manual** A user-configured threshold that explicitly overrides the default or learned value. Manual thresholds remain fixed until changed or removed. Only one source is active for a given threshold at any time (minimum learned thresholds do not change the source). #### How Thresholds Are Evaluated Thresholds are always evaluated against a **specific metric and aggregation**, even when multiple series are shown for context. For example: - **Request Duration thresholds are evaluated against P95 latency** A Signal becomes active when the P95 latency exceeds the configured threshold. - Other percentiles (such as P90 or P99) may be displayed to provide additional context but do not drive activation. The evaluation aggregation is fixed per metric and does not change when thresholds are overridden. #### Learned Threshold Minimums For metrics that support learning, you can optionally configure a **Minimum Learned Threshold**. A minimum learned threshold: - Sets a lower bound on how low a learned threshold can go - Allows Causely to continue learning above that value - Does **not** create a manual override This is useful when you want adaptive behavior while preventing learned thresholds from becoming unrealistically low due to traffic patterns or short-term anomalies. #### Thresholds and SLOs The same metrics used for Signal detection are also used as **Service Level Indicators (SLIs)** when evaluating SLOs. This means: - A threshold crossing may activate a Signal - The same metric contributes to SLO health calculations Configuring thresholds affects **both operational detection and SLO evaluation**, so changes should be made with awareness of their broader impact. ### How to Configure Thresholds Causely supports configuring thresholds through multiple mechanisms, allowing you to choose the approach that best fits your workflow and environment. Regardless of the method used, the outcome is the same: a **manual threshold**, unless you configure a minimum learned threshold. Thresholds can be configured at different levels (for example, service, workload, or infrastructure resource), depending on the metric and entity type. #### Configuration Options You can configure thresholds using the following methods: - **Causely MCP server** Best for asking an AI agent to inspect or change thresholds conversationally, for example while investigating a noisy Signal. - **Causely UI** Best for inspecting learned behavior, understanding how thresholds relate to observed metrics, and making targeted adjustments. - **Service metadata** Service metadata supports configuring a subset of commonly used service-level thresholds and allows you to configure thresholds declaratively using: - Kubernetes labels - Nomad service tags - Consul service metadata This approach is well suited for version-controlled, environment-specific configuration that travels with your service definition. - **Causely API** Best for programmatic configuration, automation, and integration with internal tooling or workflows. All configuration methods support configuring **manual thresholds where applicable**, and for metrics that support learning, configuring a **minimum learned threshold** to bound adaptive learning. #### Choosing the Right Method Use the **MCP server** when you want to: - have an AI agent tune a threshold while it investigates a Signal, - review or clean up existing threshold configurations conversationally. Use the **UI** when you want to: - understand why a Signal is activating, - compare learned thresholds against real traffic, - experiment or iterate quickly. Use **service metadata** when you want to: - manage thresholds as code, - apply consistent thresholds across environments, - ensure thresholds are applied automatically during deployment. Use the **API** when you want to: - automate threshold management, - integrate threshold changes into CI/CD or internal systems, - apply changes across many entities programmatically. #### What Happens When You Configure a Threshold When you configure a threshold: - The threshold source becomes **Manual** - Automatic learning (if supported for that metric) is paused - The configured value is used consistently for: - Signal activation - SLI evaluation for SLOs If you remove a manual threshold, Causely reverts to the **default** or **learned** threshold, depending on the metric. #### Minimum Learned Thresholds For metrics that support learning, you can optionally configure a **Minimum Learned Threshold** instead of a full manual override. This allows Causely to: - continue adapting to changing behavior, - while never learning a threshold below the configured minimum. Minimum learned thresholds do **not** replace learning and do not create a manual override. #### Using the Causely MCP Server The [Causely MCP server](/agent-integration/mcp-server/) provides the following tools for managing thresholds. Tools that change configuration require the Developer or Administrator role. | Tool | Use | | --- | --- | | `get_supported_threshold_metrics` | List the metric keys supported for an entity type, for example `entity_type: "Service"` | | `create_threshold_configuration` | Set a manual threshold or a minimum learned threshold | | `get_threshold_configurations` | List existing configurations, optionally filtered by entity or scope | | `update_threshold_configuration` | Change the values of an existing configuration | | `delete_threshold_configuration` | Remove a configuration and revert to the default or learned threshold | A configuration can apply to a single entity (`INDIVIDUAL` scope), to entities that match a label selector (`LABEL` scope), or to every entity of the relevant type (`GLOBAL` scope). For example, you can ask your agent: > The payments service keeps firing Request Duration Signals at 300 ms, but anything under 800 ms is fine for us. Set its latency threshold to 800 ms. The agent checks the supported keys with `get_supported_threshold_metrics`, then calls `create_threshold_configuration`: ```json { "scope": "INDIVIDUAL", "target_entity_id": "", "thresholds": { "RequestDuration": 800 } } ``` The same tools also set SLO targets and burn rates. See [SLO Targets and Burn Rates](/configuration/slo-configuration/#configuring-targets-and-burn-rates-with-mcp-or-the-api). #### Using the Causely UI Using the UI allows you to: - Inspect the learned threshold alongside observed metrics (for example P90 and P99 for latency) - Override thresholds to match documented SLOs or performance requirements - Set a minimum value that bounds how low a learned threshold can go while preserving adaptive learning - Immediately see how a custom threshold compares to real traffic patterns To configure thresholds in the UI: 1. Navigate to the service you want to configure. 2. Select the **Metrics** tab for the service, then select the relevant Signal metric (for example, Request Duration or Request Error Rate). 3. Click the pencil icon next to the threshold to edit the value. 4. Save the change to apply the override. UI-based configuration is best suited for teams that want quick iteration, visibility into learned behavior, and explicit control without modifying service metadata or deployment configuration. #### Using Service Metadata :::note Not all thresholds can be configured via service metadata. Metadata-based configuration currently supports a subset of service-level thresholds such as request error rate and request latency. ::: ##### Using Kubernetes Labels The recommended way to configure thresholds is using Kubernetes labels. You can apply these labels to your services: ```bash # Configure error rate threshold (for example, 1% error rate) kubectl label svc -n "causely.ai/error-rate-threshold=0.01" # Configure latency threshold (for example, 500ms) kubectl label svc -n "causely.ai/latency-threshold=500.0" ``` ##### Using Nomad Service Tags For Nomad services, you can configure thresholds using service tags in your job specification: ```hcl job "example" { group "app" { service { name = "my-service" port = 8080 tags = [ "causely.ai/error-rate-threshold=0.01", "causely.ai/latency-threshold=500.0" ] } } } ``` ##### Using Consul Service Metadata For Consul services, you can configure thresholds using service metadata: ```bash # Register a service with threshold metadata consul services register \ -name="my-service" \ -port=8080 \ -meta="causely.ai/error-rate-threshold=0.01" \ -meta="causely.ai/latency-threshold=500.0" # Update existing service metadata consul services register \ -id="my-service-id" \ -name="my-service" \ -port=8080 \ -meta="causely.ai/error-rate-threshold=0.01" \ -meta="causely.ai/latency-threshold=500.0" ``` ### Supported Thresholds Causely supports configurable thresholds across a broad set of service and infrastructure entities. These thresholds are used to detect Signals and also act as Service Level Indicators (SLIs) when evaluating SLO health. Some thresholds support **automatically learned values**, while others use **system defaults** that can be manually overridden. #### Services, Workloads, HTTP Paths and RPC Methods | Metric | Unit | Learned | |------|------|---------| | Request Error Rate | percent | No | | Request Duration (P95) | millisecond | Yes | | Request Duration P95 (Client) | millisecond | Yes | | Request Rate | request/s | Yes | | Policy Rejected Request Rate | percent | No | | Request Throttling Rate | percent | No | | Unauthorized Request Rate | percent | No | | Connections | percent | No | | Mutex Wait Time | percent | No | | Command Latency | millisecond | No | | GC Time | percent | No | | Queries Queued | count | No | | Transaction Error | percent | No | | Transaction Duration | second | No | | Transaction IDs Congested | percent | No | | Cache Size | bytes | No | | Redis Connections Utilization | percent | No | | Kafka Message Rate | message/s | No | | Server Errors | count | No | | User Errors | count | No | | File Descriptor Utilization | percent | No | | Java Heap Utilization | percent | No | | Throttled | count | No | | DB Connections Utilization | percent | No | #### Queues, Topics and Background Operations | Metric | Unit | Learned | |------|------|---------| | Queue Depth | count | No | | Dead Letter Count | count | No | | Queue Acks | request/s | No | | Message Wait Time | seconds | No | | Queue Size Bytes | bytes | No | | Lag | count | No | | Task Duration | millisecond | No | #### Database Tables | Metric | Unit | Learned | |------|------|---------| | DB Query Duration | second | Yes | | Select Query Duration (P95) | millisecond | Yes | | Table Bloat | percent | No | | Lock Exclusive Rate | percent | No | | DDL Lock Exclusive Rate | percent | No | #### Application Load Balancers | Metric | Unit | Learned | |------|------|---------| | Request Rate | request/s | Yes | | Request4xx Error | percent | No | | Request5xx Error | percent | No | | Request504 Error | percent | No | | ELB Auth Error | count | No | | Target Connection Error | count | No | #### Containers and Controllers | Metric | Unit | Learned | |------|------|---------| | CPU Utilization | percent | No | | CPU Throttled | percent | No | | Memory Utilization | percent | No | | Ephemeral Storage Utilization | percent | No | | Frequent Crash | count | No | | FrequentOOM Kill | count | No | | Frequent Pod Ephemeral Storage Evictions | count | No | #### Nodes and Virtual Machines | Metric | Unit | Learned | |------|------|---------| | CPU Utilization | percent | No | | Memory Utilization | percent | No | | Conntrack Table Utilization | percent | No | | SNAT Port Utilization | percent | No | | Container Ephemeral Storage Utilization | percent | No | | Memory Pressure Pod Evictions | count | No | | Disk Pressure Pod Evictions | count | No | | Disk Read IOPS Utilization | percent | No | | Disk Write IOPS Utilization | percent | No | | Disk Total IOPS Utilization | percent | No | | Disk Read Throughput Utilization | percent | No | | Disk Write Throughput Utilization | percent | No | | Disk Total Throughput Utilization | percent | No | #### Disks | Metric | Unit | Learned | |------|------|---------| | Utilization | percent | No | | Read IOPS Utilization | percent | No | | Write IOPS Utilization | percent | No | | Total IOPS Utilization | percent | No | | Read Throughput Utilization | percent | No | | Write Throughput Utilization | percent | No | | Total Throughput Utilization | percent | No | | Inodes Utilization | percent | No | #### SLOs These keys apply to `SuccessSLO`, `LatencySLO`, `AvailabilitySLO`, and `MessageBacklogSLO` entities and can be set through the MCP server or API only. See [SLO Targets and Burn Rates](/configuration/slo-configuration/#configuring-targets-and-burn-rates-with-mcp-or-the-api). | Metric | Key | Unit | Learned | |------|------|------|---------| | Target Threshold | `TargetThreshold` | percent (0–100) | No | | Burn Rate | `BurnRate` | multiplier | No | ### Best Practices 1. **Start with Default or Learned Thresholds** Use Causely’s default or automatically learned thresholds as a baseline before introducing manual overrides. 2. **Override Only When There Is a Clear Requirement** Configure manual thresholds when you have explicit business, reliability, or compliance requirements that differ from observed behavior. 3. **Prefer Minimum Learned Thresholds Over Full Overrides** When available, use a minimum learned threshold to bound adaptive learning without disabling it entirely. 4. **Consider Both Signal Detection and SLO Impact** Threshold changes affect both Signal activation and SLI evaluation for SLOs. Validate changes in both contexts. 5. **Monitor After Changes** After updating thresholds, observe how they affect Signal frequency, noise, and SLO evaluation over time. 6. **Document Intent, Not Just Values** Record why a threshold was changed to support future reviews and adjustments. ### Example Use Cases 1. **Strict SLO Requirements** A critical service requires tighter latency bounds than normal traffic patterns allow. A manual threshold is configured to align Signal detection and SLO evaluation with the defined objective. 2. **Preventing Overly Aggressive Learned Thresholds** A service with highly variable traffic uses a minimum learned threshold to prevent latency thresholds from adapting too low during off-peak periods while preserving adaptive behavior. 3. **Infrastructure Saturation Detection** A team configures CPU or disk utilization thresholds on nodes to detect resource saturation early, independent of application-level Signals. 4. **Queue Backlog Monitoring** Queue depth and message wait time thresholds are configured to surface processing delays before they impact downstream services. 5. **Temporary Adjustments During Maintenance** Thresholds are temporarily adjusted during planned maintenance or migrations and reverted afterward.