# Causely documentation: reference > Full content of the "reference" section of the Causely documentation. This file contains all documentation content in a single document following the llmstxt.org standard. ## Applications Software programs designed to perform specific tasks or functions for end-users or other systems. --- #### Business Application A logical grouping of applications within Causely to define a high-level application. Business Applications are constructed by default from the pod label `app.kubernetes.io/part-of` but this scoping mechanism can be changed. Causely creates Business Applications automatically upon initially discovery of the environment. --- #### Application An Application is a set of code and configurations designed to perform a specific function. These applications are typically containerized, scalable, and decoupled from the underlying infrastructure. They leverage cloud native principles such as microservices, statelessness, and automation to achieve agility and reliability. --- #### Application Load Balancer An Application Load Balancer (ALB) is a component of modern cloud infrastructure that distributes incoming traffic across multiple servers to ensure optimal resource use, reliability, and scalability. It makes intelligent routing decisions based on factors like request content, headers, or URL paths. ALBs support features like SSL termination, WebSocket connections, and advanced routing rules, making them ideal for microservices or containerized architectures. They enhance application availability and responsiveness by evenly balancing workloads and rerouting traffic in case of server failures. --- #### Operation A Background Operation in a Kubernetes microservice is a non-blocking task that runs asynchronously and independently of user requests. These operations are commonly used for tasks like data processing, maintenance jobs, or message queue consumption. They are often managed using worker pods or cron jobs and are critical for handling long-running or resource-intensive tasks without impacting the primary application workflow. --- ## Compute Servers or cloud resources to execute tasks, run applications, or perform calculations. --- #### Allocation An Allocation in Nomad represents the deployment of a task group onto a specific client node. It is responsible for running and monitoring the tasks defined in a Nomad job. Each allocation includes detailed information about the resources used, task states, and the node where the tasks are running. Allocations are key to Nomad's distributed system architecture, ensuring workloads are placed efficiently and reliably. They also include metadata that allows for tracking, troubleshooting, and scaling operations. --- #### Cluster A Cluster is a set of nodes that run containerized workloads, managed by a shared Kubernetes control plane. It is the boundary within which the scheduler places pods onto nodes, so problems that affect many workloads at once, such as cordoned nodes or exhausted CPU and memory capacity, are diagnosed at the cluster level. --- #### Compute Spec A logical group created by Causely that defines a group of containers within a deployment. For example, if you have a deployment with two containers definitions: C1 and C2 then there would be two compute specs in Causely: CS1 and CS2. Since a deployment controller may create many pods each with an instance of C1 and C2, there are multiple containers within each compute spec. --- #### Container A Container is a lightweight, standalone execution environment that encapsulates an application and its dependencies. Containers provide portability and consistency across different environments, making them essential for microservices architectures. Containers are isolated from each other and the host system, ensuring security and resource efficiency. Container Orchestration platforms like Kubernetes and Nomad enables containers to be dynamically scheduled, scaled, and orchestrated as part of a broader application architecture. --- #### Controller A Controller is a control loop that monitors the state of resources within a Kubernetes cluster and works to reconcile them with the defined state. Examples include the Deployment controller, which manages replicas of an application, and the Node controller, which handles node statuses. Controllers are integral to Kubernetes’ self-healing capabilities, automatically restarting, scaling, or reconfiguring resources as needed. --- #### Disk A disk refers to a virtualized storage resource that provides persistent or temporary data storage for cloud-based virtual machines (VMs) or workloads. Disks can be provisioned in various types, such as SSDs for high performance or HDDs for cost-effective storage. They support scalability, redundancy, and features like snapshots for backups or disaster recovery. --- #### Node A Node is a physical or virtual machine that serves as a infrastructure component in a Kubernetes cluster. Nodes run pods and are managed by the Kubernetes control plane, providing the compute, memory, and storage resources needed to host pods and containers. Each node runs essential Kubernetes components such as kubelet, kube-proxy, and a container runtime. Nodes can be scaled horizontally to handle increased workloads and are monitored to ensure stability and performance. --- #### Nomad Job A Nomad Job defines a workload to be scheduled and executed by the Nomad orchestrator. Jobs consist of task groups and task definitions, specifying resource requirements, constraints, and runtime configurations for applications. They can be used to schedule various workloads, from containerized applications to batch jobs. Nomad jobs are declarative, enabling operators to define states and allowing the system to manage deployment and scaling automatically. --- #### Persistent Volume A Persistent Volume (PV) in Kubernetes is a storage resource provisioned for a cluster, independent of any pod. PVs provide durable storage that retains data beyond the lifecycle of individual pods, supporting stateful applications. They are abstracted from the underlying storage technology, allowing seamless integration with cloud-based or on-premises storage solutions. --- #### Pod A Kubernetes Pod is a deployable unit in Kubernetes, consisting of one or more containers that share resources such as networking and storage. Pods are ephemeral and designed to host tightly coupled application components. They are often used for scaling and scheduling containers in Kubernetes. Pods can be replicated and distributed across nodes to ensure high availability and load balancing. --- #### Virtual Machine A Virtual Machine (VM) is a software-based representation of a physical computer, providing an isolated environment for running containerized applications like in Nomad architecture. VMs include a virtualized CPU, memory, storage, and network interface, allowing multiple VMs to run on a single physical machine. They enable efficient utilization of hardware resources and provide strong isolation between workloads. VMs are widely used for server consolidation, development environments, and running legacy applications. --- #### Workload A Workload refers to a single running instance of an application. Each Workload operates independently and can run concurrently with others to handle varying loads—for example, a Kubernetes Pod, a Nomad Allocation, or a managed service. Causely only includes services as Workloads when the underlying infrastructure is not discovered and they are observed communicating with other services in the environment. --- ## Databases Systems for storing, organizing, and managing structured or unstructured data. --- #### Database A Database is an organized collection of data that is stored and accessed electronically. Databases can be relational, NoSQL, or graph-based, each optimized for specific types of data and query requirements. They form the backbone for storing structured or semi-structured data in most applications. Modern databases support features such as indexing, transactions, and high availability, enabling robust and performant data management for diverse use cases. --- #### Database Server The server or managed service that hosts and runs the database, it provides the processing power, memory and network capabilities to support the database operations. The Database server often runs a database management system (DBMS) such as PostgreSQL, Oracle, or SQL, which interacts with the underlying database. --- #### Database Server Instance An individual instance of a Database Server layered over the underlying infrastructure components. Each discovered Database Server has a corresponding Database Server Instance within the topology. --- #### Cache A Cache is an in-memory key-value store that provides ultra-fast data retrieval for frequently accessed data. It is commonly used to reduce database query load, improve application performance, and support features like session storage and real-time analytics. Cache platforms such as Redis supports various data structures, such as strings, hashes, lists, and sets, making it versatile for diverse caching needs. --- #### Cache Instance A Cache Instance is a single deployment of the cache software such as Redis, configured to store and manage cached data. Instances can operate independently or as part of a cluster, supporting replication and sharding for scalability and fault tolerance. Each instance maintains its own data and can be optimized for specific workloads, such as read-heavy or write-heavy operations. --- #### Table A Database Table is a structured set of data organized into rows and columns within a relational database. Each table represents an entity or relationship, with columns defining attributes and rows storing individual records. Tables can be indexed to improve query performance and may include constraints to enforce data integrity. They are the primary building blocks for representing and manipulating structured data in applications. --- ## Data Pipeline Series of processes that move and transform data between systems for analysis, storage, or further processing. --- #### HTTP Path An HTTP Path is a component of a URL that specifies the specific resource or endpoint being requested on a web server. It determines how requests are routed and processed by the application, often forming the basis for RESTful APIs. HTTP paths can include parameters and query strings to refine requests, supporting dynamic and flexible interactions with web services. They are essential for defining clear and consistent interfaces for client-server communication. --- #### MCP Tool An MCP Tool is a function exposed by a Model Context Protocol (MCP) server that AI agents and applications can invoke to retrieve data or perform actions, such as querying an API, reading a file, or running a workflow. Each tool has a name and an input schema, and Causely tracks tool calls, including their latency, errors, throttling, and authentication and policy rejections, so that problems in a tool or in the callers that invoke it can be diagnosed. --- #### RPC Method An RPC Method (Remote Procedure Call) allows a program to execute a function or procedure on a remote server as if it were local. RPCs are widely used in distributed systems to enable communication between services. They abstract the complexities of network communication, providing a simple interface for developers. Common frameworks for RPC include gRPC, which uses Protocol Buffers for serialization, and JSON-RPC. --- ## Messaging Messaging Platforms facilitate communication between distributed applications or services via queues or topics. --- #### Broker A Broker is a server that handles the storage and transmission of messages in a Kafka cluster. Brokers manage message data for topics and distribute load across the cluster. They are key components for enabling fault tolerance and high availability in Kafka’s publish-subscribe model. Each broker communicates with producers and consumers, ensuring messages are persisted and delivered efficiently. --- #### Broker Instance A Broker Instance refers to an individual server running a Kafka broker. Each broker instance is part of a Kafka cluster, responsible for managing a subset of partitions and ensuring data replication across the cluster. Broker instances work together to balance workloads, provide redundancy, and ensure data availability even if some instances go offline. They are critical for Kafka's ability to handle high-throughput and low-latency message processing. --- #### Exchange An Exchange is a messaging component responsible for receiving messages from producers and routing them to one or more queues based on routing rules and bindings. Exchanges determine how messages are distributed across the system, supporting patterns such as direct, fanout, and topic-based routing. --- #### Queue A Queue is a messaging component that stores messages until they are consumed by a subscriber. Queues buffer messages between producers and consumers, enabling asynchronous communication and decoupling of services. Queue depth and consumer lag are key indicators of whether downstream services are keeping pace with incoming message volume. --- #### Topic A Kafka Topic is a logical channel to which producers send messages and from which consumers retrieve messages. Topics can be partitioned to enable parallel processing and ensure scalability within a Kafka cluster. Each partition is replicated across multiple brokers for fault tolerance, ensuring data is durable and accessible even in the event of broker failures. --- ## Entity Types Overview Causely automatically discovers over 25 different Entity types from your cloud native environment. These entity types are discovered through data sources like eBPF, Cloud APIs, OpenTelemetry and used to build topologies, identify defects, and infer Diagnoses. --- ## Services Services are self-contained units of functionality within a system that perform specific tasks or provide specific capabilities, often accessible through defined interfaces or APIs. --- #### AI Model An AI Model refers to an AI inference endpoint hosted outside the primary orchestration platform, such as a large language model or embedding model API. These models are accessed via network interfaces, often using HTTP or gRPC. --- #### Kubernetes Service A Kubernetes Microservice is an independently deployable component of an application running in a Kubernetes environment. These microservices communicate with each other using lightweight protocols and are designed to perform specific, granular functions, promoting modularity and scalability. --- #### Network Endpoint A network endpoint refers to a specific point of interaction where services, applications, or devices connect to a network to send or receive data. It can be a URL, IP address, or a combination of both that allows communication between clients and servers, such as microservices, APIs, or databases. --- #### Service A Service refers to a service or application hosted outside the primary orchestration platform, such as a third-party API or legacy system. These services are accessed via network interfaces, often using HTTP or gRPC. --- ## Reference The Reference section provides comprehensive documentation about the entities Causely discovers, the Diagnoses it produces, key terminology, and security information. This documentation helps you understand what types of entities it discovers in your environment, the Diagnoses it can produce, and the terminology used throughout the documentation. ### Entity Types Causely automatically discovers over 25 different entity types from your cloud native environment through data sources like eBPF, Cloud APIs, and OpenTelemetry. These entities are used to build topologies, identify defects, and infer Diagnoses. Learn about the different types of entities that Causely automatically discovers, including applications, services, databases, compute resources, messaging systems, and data pipelines in [Entity Types](/reference/entity-types). ### Diagnoses With more than 100 types of root causes captured in its Causal Models, the causes a Diagnosis can identify, Causely can pinpoint hundreds of thousands of potential issues and their effects within your environment. These causes span applications, infrastructure, data pipelines, release management, and services. Explore the root cause types behind each Diagnosis and how they impact your systems, from application bugs to infrastructure bottlenecks to release-related issues, in [Diagnoses](/reference/root-causes). ### Signals Signals are observable anomalies in managed objects that may be caused by root causes; formally these are Symptoms, defined in [Terminology](/reference/terminology). Causely detects a wide variety of Signals across services, workloads, compute resources, databases, messaging systems, and more. Browse the complete reference of all Signals that Causely can detect, organized by category and entity type, in [Signals](/reference/symptoms). ### Terminology Understanding the key terms and concepts used throughout Causely documentation helps you get the most out of the system. The terminology covers core concepts like entities, Issues, Diagnoses, Signals, topology, causality graphs, and more. Understand the key terms and concepts used throughout Causely documentation, grouped by relatedness to help you navigate the system effectively in [Terminology](/reference/terminology). ### Security Causely is designed to protect sensitive data and ensure privacy. The system processes telemetry data locally and primarily transmits minimal, high-level information to its backend. All data is encrypted in transit and at rest. Learn about Causely's security model, data handling practices, privacy protections, and the permissions required for deployment components in [Security](/security). --- ## Applications (Root-causes) Software programs designed to perform specific tasks or functions for end-users or other systems. :::note In the Causely user interface, these Diagnoses appear under the corresponding application entity. ::: ### .NET Unhandled Exception .NET logs show an unhandled exception, indicating the application failed unexpectedly and may have terminated the process. Unhandled exceptions in .NET typically indicate application logic errors, invalid state, or dependency failures that were not caught. Depending on hosting mode, this can fail the current request, terminate a worker loop, or crash the process entirely. ### Access Rejected By Policy Requests from this service are being rejected with HTTP 451 "Unavailable For Legal Reasons" by a service it calls, while other callers of that service are not affected. The destination's policy is working as configured; something about this client's requests changed, such as a new client version calling a gated resource or path, traffic originating from a blocked region or tenant, or a changed request attribute that the policy matches on. If every caller of the destination is rejected instead, see Policy Rejection on the destination. It also applies when the destination is an MCP tool. ### Access Throttled The application is receiving HTTP 429 "Too Many Requests" responses, indicating that it has exceeded the rate limits set by the other service. This can cause degraded functionality, slow performance, or temporary service unavailability for end users. HTTP 429 errors are typically triggered when an API or service imposes rate limits to control the volume of incoming requests. This diagnosis applies when only some callers are throttled; when most or all callers of a service are throttled, see [Throttling Rejection](/reference/root-causes/services#throttling-rejection) on that service instead. It also applies to operations and MCP tools, not only services. ### Apache Worker Exhausted Apache HTTPD has reached its MaxRequestWorkers limit, meaning all worker slots are occupied and new requests cannot be processed. Apache prefork and event MPMs log "server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting" when all worker slots are busy. New connections are queued up to ListenBacklog, then dropped. This indicates either a load spike or slow request processing holding workers. ### Application Error Spike The service is emitting a high volume of ERROR or CRITICAL log lines over a rolling window, indicating an active application-side failure with no matching specific log signature. Common causes include schema or payload mismatches after a deploy, repeated business-logic exceptions, dependency failures logged as generic errors, and application regressions that produce many error-level logs without a canonical runtime signature. This signal indicates the application is actively failing rather than providing a precise diagnosis of the underlying cause. This root cause activates when generic error-severity log volume exceeds 2,000 occurrences within a one-hour window. ### Cassandra Tombstone Pressure Cassandra reads are scanning excessive tombstones, indicating partition design or TTL issues that are causing significant read latency and resource pressure. Cassandra logs a warning when a read scans more tombstones than tombstone_warn_threshold (default 1000). Tombstones are markers for deleted data that must be scanned until compaction removes them. Excessive tombstones cause read amplification, increased GC pressure, and can trigger ReadTimeoutExceptions. ### Circuit Breaker Open A circuit breaker protecting an upstream dependency has opened, causing the service to fail fast on calls to that dependency instead of waiting for timeouts. Circuit breakers such as Resilience4j and Netflix Hystrix open when an upstream dependency exceeds a configured failure rate or slow-call threshold. While open, all calls to the dependency are immediately rejected to prevent cascading failures and reduce latency. This is a protective mechanism, but it surfaces as errors for callers. ### Connection Pool Exhausted The database or service connection pool is exhausted, and new connection requests are being rejected, causing application errors for all callers. Connection pool exhaustion occurs when all connections in the pool are in use and no connection becomes available within the timeout. Sources include PostgreSQL "sorry, too many clients already", MySQL error 1040, Redis "max number of clients reached", HikariCP timeout, or pgBouncer limit. This typically indicates a connection leak, long-running transactions holding connections, or insufficient pool sizing for the load. ### Database Connection Pool Saturated The client-side database connection pool is exhausted when all available connections are in use, preventing new database queries from being executed. This can cause application requests to hang or fail, impacting user experience and potentially leading to downtime for database-dependent features. ### Database Malfunction The database is returning a high rate of errors or failing to respond to queries, causing disruptions for services and clients that depend on it. This may result in delayed or failed access to one or more tables, leading to degraded application performance, elevated latency, or complete unavailability of database-backed functionality. ### Disk Full A filesystem used by this service has run out of space (ENOSPC), causing write operations to fail. "No space left on device" errors occur when a write() syscall fails because the filesystem has no free blocks. Logs, data files, temp files, or WAL segments may all contribute to disk exhaustion. This causes immediate write failures and can cause the process to crash or enter a degraded state. ### Elasticsearch Cluster Unhealthy The Elasticsearch cluster health status has transitioned to RED, indicating one or more primary shards are unassigned and data is unavailable for those shards. Elasticsearch reports RED health when at least one primary shard is unassigned. Search and indexing requests for affected indices will fail. This is caused by node failures, insufficient nodes to satisfy the index replication factor, or shard allocation issues. ### Excessive DNS Traffic from Client The application is generating an unusually high volume of DNS requests, potentially overwhelming DNS servers, increasing latency for users, and risking service disruptions. This behavior may also incur additional costs or trigger rate-limiting from DNS providers. This typically arises when the application initiates DNS lookups more frequently than necessary due to lack of effective caching, redundant DNS resolution logic, or misconfigurations. ### File Descriptor Exhaustion The application has reached the system-imposed limit on the number of file descriptors it can open. This typically leads to errors such as "Too many open files," preventing the application from creating new connections, reading files, or accessing resources. This can severely impact functionality, particularly in high-concurrency or high-I/O scenarios. ### File Limit Exhausted The process has reached the operating system file descriptor limit (EMFILE/ENFILE), preventing it from opening new network connections, files, or sockets. Each open socket, file, or pipe consumes a file descriptor. When the per-process limit (ulimit -n / RLIMIT_NOFILE) or system-wide limit (/proc/sys/fs/file-max) is reached, new open/accept/connect calls fail with EMFILE or ENFILE. This manifests as connection refused errors in network servers or file-open failures in data pipelines. ### Go Deadlock The Go runtime has detected a full deadlock where all goroutines are permanently blocked, causing the process to panic and exit. The Go runtime prints "all goroutines are asleep - deadlock!" and exits when it determines that no goroutine can ever make progress. This is a fatal condition and the process terminates immediately. Common causes include channel send/receive with no corresponding partner, mutex lock with no unlock, or sync.WaitGroup misuse. ### Go Nil Pointer Panic The Go runtime has panicked due to an invalid memory address or nil pointer dereference, causing the handler or process to fail. This panic occurs when code dereferences a nil pointer. In Go services, an unhandled panic usually terminates the current goroutine and may crash the whole process unless recovered. Common causes include missing dependency initialization, nil interface assumptions, unchecked map or pointer fields, or races around object lifecycle. ### GOMAXPROCS Misconfigured The environment variable GOMAXPROCS, which controls the maximum number of CPU cores the Go runtime uses, has been set higher than the CPU limit of the container in which the Go application is running. This mismatch can lead to inefficient CPU usage, reduced performance, and potential throttling because the Go runtime attempts to schedule more work than the container is permitted to handle. ### HAProxy Backend Unavailable HAProxy has no available backend servers, as all backends are either down or at their connection limit, causing all incoming requests to be rejected. HAProxy logs "backend has no server available" when every server in the backend pool is in DOWN state or has reached its maxconn limit. Clients receive a 503 Service Unavailable. This can be caused by all backend servers being unhealthy, a misconfigured health check, or a connection storm saturating backends. When all incoming requests are rejected, users can be locked out, which can explain a Concurrent Users Drop on the Business Application. ### Inefficient DNS Lookup The application is making an unusually high volume of DNS requests, with over 80% returning NXDomain (non-existent domain) responses. This excessive DNS activity is adding 10 to 20 ms of latency to each request, negatively impacting service performance. The issue is often caused by the service or application attempting to resolve incomplete or unqualified domain names. ### Inefficient Garbage Collection The application is experiencing high latency and reduced throughput because a significant portion of its runtime is being spent in garbage collection (GC). This leads to frequent pauses, degrading overall performance and causing delays in request handling through all the dependent services. This issue usually occurs when the Java Virtual Machine (JVM) or other garbage-collected runtime environments are under memory pressure. ### Invalid Client Certificate The application is failing to connect to a service due to invalid certificate errors, preventing secure communication over HTTPS or TLS. This can cause downtime or degraded functionality for users relying on this service. ### Java GC Pressure The JVM garbage collector is under pressure, logging allocation failures or to-space exhaustion events, indicating the heap is too small or object allocation rates are too high. GC pressure manifests as "Allocation Failure" in ParallelGC/SerialGC logs or "to-space exhausted" in G1GC logs. Both indicate that GC cannot reclaim space fast enough to keep up with allocation demand. This leads to longer GC pauses, increased latency, and potential OutOfMemoryError if unresolved. ### Java Heap Saturated The JVM is operating with limited available heap memory, resulting in degraded performance or potential application crashes. This condition typically leads to frequent or prolonged garbage collection (GC) pauses, slow response times, and, in severe cases, OutOfMemoryError. It often reflects memory leaks, improper heap sizing, or excessive object allocation. ### Java Null Pointer The JVM has thrown java.lang.NullPointerException, indicating application code dereferenced a null reference and failed unexpectedly. A NullPointerException typically means application logic assumed an object was present when it was actually null. Depending on exception handling, this can fail an individual request or crash the process. Common causes include missing dependency wiring, invalid state transitions, bad deserialization assumptions, or unguarded optional values. ### Java Out of Memory The JVM has thrown java.lang.OutOfMemoryError, indicating the heap, metaspace, or GC overhead limit is exhausted and the process cannot allocate memory. OutOfMemoryError occurs when the JVM cannot satisfy an allocation request. Common subtypes include heap space (object allocation failed), Metaspace (class metadata exhausted), GC overhead limit exceeded (GC spending over 98% of time reclaiming less than 2% of heap), or unable to create native thread. The process may continue in a degraded state or crash. ### Java Stack Overflow The JVM has thrown java.lang.StackOverflowError due to runaway recursion or an excessively deep call stack that has exhausted the thread stack space. Each thread has a fixed stack size (-Xss). When recursive calls or deep call chains exceed this limit, StackOverflowError is thrown. Unlike OutOfMemoryError, this is usually a code defect such as infinite recursion or mutual recursion without a base case, or a framework issue with excessive proxy/interceptor wrapping. ### Java Thread Pool Exhausted The JVM thread pool is saturated and rejecting new task submissions with java.util.concurrent.RejectedExecutionException. ThreadPoolExecutor throws RejectedExecutionException when both the thread pool is at max capacity and the task queue is full and the rejection policy fires (default: AbortPolicy). This causes request handlers to fail, leading to increased error rates for callers. ### Kafka ISR Shrink A Kafka broker has logged ISR (In-Sync Replica) shrinkage, indicating that one or more follower replicas have fallen behind the leader and been removed from the ISR set. Kafka logs "ISR shrunk from [X] to [Y]" when a follower fails to keep up with the leader within replica.lag.time.max.ms. While a replica is out of the ISR, the effective replication factor is reduced, increasing the risk of data loss if the leader fails. Producers with acks=all will also experience increased latency or errors. ### Kafka Partition Outage Kafka has one or more offline partitions, meaning the cluster cannot maintain a healthy leader for those partitions and affected topic data is currently unavailable. This is a hard failure condition rather than a performance degradation: produce requests may be rejected and consumers cannot fetch from affected partitions until leadership is restored. Common causes include a broker outage that simultaneously removes leaders and replicas from the in-sync replica set, replication falling far enough behind that no eligible replica can be elected as leader, controller instability during broker flaps or cluster reconfiguration, and storage corruption or severe disk failure on a broker node. ### Kafka Replication Degraded One or more Kafka partitions are under-replicated, meaning followers are not keeping pace with the partition leader. This reduces fault tolerance across the cluster: if an additional broker is lost or a leader change is forced, the partition may become unavailable. The cluster may still be serving traffic, but producers and consumers are operating with reduced resilience. Common causes include disk I/O saturation on a follower broker slowing replica fetch or log flush, network congestion between brokers, a follower broker that is restarting, stuck, or under heavy load, and replication traffic competing with high-volume client produce traffic. ### Kafka Storage Pressure Kafka broker log storage utilization is high, causing degraded performance across produce, replication, and retention paths. Kafka depends on sequential log I/O, and as data-log disks approach capacity, flush, compaction, and replica synchronization operations become less efficient. This typically creates a starvation pattern of increasing broker latency, slower replication, and delayed producer acknowledgements rather than an immediate hard failure. Common causes include topic retention policies keeping more data than the disk budget supports, write rate growth exceeding storage throughput, a log compaction backlog consuming disk bandwidth, and uneven partition placement concentrating data on a single broker. ### Lock Contention The application suffers from inefficient locking, where suboptimal lock management leads to excessive contention and prolonged mutex wait times. This inefficiency degrades performance by increasing the risk of thread starvation under heavy load. This can stem from overuse of locks, coarse-grained locking strategies, or improper lock design. ### Memory Exhaustion The Broker application has exhausted its available memory, resulting in degraded performance and potential service disruption. When memory usage reaches critical levels, the system may experience increased garbage collection (GC) activity, higher processing latency, and, in severe cases, OutOfMemoryError events that halt message processing. ### MongoDB Connections Exhausted MongoDB is at or near its maximum connection limit, and new client operations may fail when they cannot obtain a server connection. MongoDB enforces a cap on concurrent active connections, and when that limit is reached, clients cannot establish or reuse connections fast enough, causing requests to fail before any database work is attempted. Common causes include application connection pools configured with an excessively high maximum size, connection leaks where clients open connections without properly releasing them, traffic spikes that create more concurrent clients than the server can handle, and long-running queries that hold connections for extended periods. ### MongoDB Cursor Pressure MongoDB has an unusually high number of open cursors, increasing server-side state management overhead and slowing query execution for callers. Open cursors represent active or partially consumed query result streams, and when too many remain open simultaneously, MongoDB must maintain more concurrent state and expend additional resources servicing them. This typically manifests as increasing latency and resource starvation rather than immediate request failures. Common causes include clients reading through large result sets slowly, applications that do not fully consume or explicitly close cursors, excessive polling or fan-out query patterns that create many simultaneous cursors, and batch sizes configured too small, which keeps individual cursors open longer than necessary. ### MongoDB Replica Lag MongoDB replication lag is high, meaning secondaries are applying oplog entries noticeably behind the primary. Read queries directed to secondaries may return stale data, and the replica set's readiness to complete a clean failover is reduced. The database continues to operate, but replication cannot keep pace with the primary write rate, degrading both read consistency and fault tolerance. Common causes include secondary nodes with insufficient disk or CPU resources for the current oplog apply rate, network latency or bandwidth constraints between replica set members, bursty write workloads that generate oplog traffic faster than secondaries can consume, and long-running workloads on secondaries that compete with the replication apply process. ### MySQL Deadlock MySQL InnoDB has detected a deadlock between two or more transactions and rolled one back to break the cycle. MySQL InnoDB logs "Deadlock found when trying to get lock; try restarting transaction" when two transactions hold locks that the other needs. The engine automatically selects a victim and rolls it back. Frequent deadlocks indicate lock acquisition order inconsistencies in application code. ### Nginx Upstream Timeout Nginx is logging upstream connection timeouts, indicating a backend service is not responding within the configured proxy_read_timeout or proxy_connect_timeout. Nginx logs "upstream timed out (110: Connection timed out)" when a backend fails to respond within the configured timeout. This causes nginx to return a 504 Gateway Timeout to the caller. The upstream service may be overloaded, deadlocked, or experiencing a network partition. Because requests that time out fail for every client of that path, users can be locked out, which can explain a Concurrent Users Drop on the Business Application. ### Nginx Worker Connections Exhausted Nginx has exhausted its worker_connections limit and cannot accept new connections, meaning incoming requests are being dropped. Nginx's worker_connections directive limits the number of simultaneous connections per worker process. When all connections are in use, nginx logs "worker_connections are not enough" and new connections are refused. Total capacity equals worker_processes multiplied by worker_connections. Because new connections are refused, users can be locked out, which can explain a Concurrent Users Drop on the Business Application. ### Noisy Client The application acts as a Noisy Client, generating high number of requests that burden destination services with increased load and elevated request rates. This aggressive request pattern directly impacts destination services by driving a high request rate that can overwhelm service capacity and contributing to increased load on the destination. ### Postgres Cache Hit Rate Degraded Postgres shared_buffers cache hit rate has dropped, causing queries to read from disk and significantly increasing query latency. Postgres uses shared_buffers as an in-memory page cache for table and index data, and when the working set fits in shared_buffers queries are served from memory, but when the cache hit rate drops Postgres must read data from disk or OS page cache, which is orders of magnitude slower and causes query latency to increase for all clients. Common causes include shared_buffers being too small for the working set size, new query patterns accessing large table scans that evict hot pages, database growth causing the working set to exceed available memory, sequential scans on large tables, and a cold cache after a server restart until the working set is loaded. ### Postgres Checkpoint I/O Pressure Postgres checkpoint I/O write time is high, indicating disk saturation that increases latency for all database operations. Postgres periodically performs checkpoints to flush dirty pages from shared_buffers to disk to ensure durability, and when checkpoint_write_time is high the disk is being saturated by checkpoint I/O, which competes with query I/O for reads and writes and causes elevated query latency for all clients. It may also indicate that checkpoints are too infrequent, leading to large bursts of write I/O. Common causes include a high WAL write rate generating many dirty pages between checkpoints, checkpoint_completion_target being too low and concentrating I/O into a short burst, disk I/O throughput being insufficient for the write rate, min_wal_size or max_wal_size being too small and causing frequent checkpoints, and shared storage throughput limits being hit. ### Postgres Connection Slots Exhausted Postgres active connections are approaching or have reached max_connections, and new connection attempts from services will be rejected. Postgres uses a process-per-connection model and limits total connections via max_connections, and when this limit is reached new connection attempts receive "FATAL: sorry, too many clients already," causing services that cannot connect to fail their database operations. Connections also consume shared memory, so running near the limit causes additional resource pressure. Common causes include no connection pooler being in use, connection pools being misconfigured with too high a max_size, connection leaks where connections are opened but not properly closed, sudden traffic spikes creating more service instances than the pool can handle, and long-running queries holding connections that should be idle. ### Postgres Deadlock PostgreSQL has detected a deadlock cycle between concurrent transactions and terminated one to resolve it. PostgreSQL logs "ERROR: deadlock detected" when its deadlock detector finds a cycle in the lock wait graph. One transaction is chosen as the victim and receives an error and must be retried. Frequent deadlocks degrade throughput and cause user-visible errors. ### Postgres Deadlock Storm Postgres is experiencing a high rate of deadlocks, causing transaction rollbacks and forcing retries in upstream services. A deadlock occurs when two or more transactions each hold a lock the other needs, creating a circular dependency, and Postgres detects this and aborts one transaction with "ERROR: deadlock detected." The aborted transaction must be retried by the application, and a high deadlock rate causes elevated error rates, retry traffic amplification, and query latency spikes. Common causes include application code acquiring locks in inconsistent order across transactions, bulk UPDATE or DELETE operations without consistent row ordering, missing explicit locking where the application assumes order of operations, and high-concurrency workloads on the same rows without retry logic. ### Postgres Idle-in-Transaction Accumulation Postgres has idle-in-transaction sessions accumulating, and these sessions hold locks, prevent autovacuum, and cause compounding table bloat and lock contention. An idle-in-transaction session has started a transaction with BEGIN but is not actively executing any query, and it may be waiting for application-side processing, a network call, or simply be a leaked connection. These sessions hold all locks acquired during the transaction and prevent autovacuum from cleaning dead rows, and as they accumulate they cause lock contention, connection slot exhaustion, and eventually table bloat that degrades query performance. Common causes include the application opening a transaction and making an external API call or sleeping, an ORM or framework starting a transaction at request start but not committing promptly, a connection pool returning connections that were left in a transaction, and missing idle_in_transaction_session_timeout configuration. ### Postgres Lock Contention Postgres has a high number of sessions waiting to acquire locks, indicating contention from long-running or conflicting transactions. Lock contention occurs when multiple transactions attempt to modify the same rows or tables simultaneously, and sessions that cannot acquire the lock wait in the lock queue, consuming a connection slot and blocking progress. As waiting sessions accumulate, query latency rises and eventually all connection slots may be consumed. Common causes include long-running transactions holding row or table locks, missing or overly broad UPDATE or DELETE statements without appropriate WHERE clauses, idle-in-transaction sessions holding locks without doing any work, DDL operations such as ALTER TABLE or VACUUM FULL taking exclusive locks, and high-concurrency write patterns on hot rows such as counters or status fields. ### Postgres Query Memory Spill to Disk Postgres is writing temporary files because queries exceed work_mem, causing significant I/O overhead and query slowdowns. When a sort, hash join, or aggregate operation requires more memory than work_mem allows, Postgres spills intermediate data to temporary files on disk. Disk I/O for temp files is much slower than in-memory operations, and temp files compete with other I/O on the same storage. High temp file creation rates indicate queries are consistently exceeding the configured memory budget. Common causes include work_mem being set too low for actual query complexity and result set sizes, queries using ORDER BY, GROUP BY, or JOIN without appropriate indexes, many concurrent sessions each using their full work_mem allocation, and queries with multiple sort or hash operations where each gets its own work_mem allocation. ### Postgres Replication Lag Postgres standby replication lag is high, causing replica reads to return significantly stale data. Postgres streaming replication sends WAL records from primary to standbys, and replication lag is the delay between a write being committed on the primary and the standby applying it. High lag means read queries directed to replicas return stale results, which can cause data consistency issues for applications that read their own writes via replicas. Common causes include network bandwidth saturation between primary and standby, a standby I/O bottleneck where disk cannot apply WAL as fast as it arrives, long-running queries on the standby blocking WAL apply, the primary generating WAL faster than the network can deliver it, and the standby being under-provisioned relative to the primary write rate. ### Producer Publish Rate Spike The application is publishing messages at a rate significantly higher than normal, causing queue depth to grow and producer message rate to increase. This surge in publishing activity creates backpressure and can overwhelm downstream consumers. When the application experiences a producer publish rate spike, it generates messages at an abnormally high rate that exceeds the system's normal capacity, leading to queue depth growth and propagation of congestion to downstream destinations. ### Python Unhandled Exception Python logs show an unhandled exception or fatal interpreter error, indicating the application failed unexpectedly. An unhandled Python exception usually emits a traceback and terminates the active request, worker, or process. Common causes include missing input validation, bad assumptions about returned objects, dependency failures that are not caught, or unexpected runtime state. ### RabbitMQ Resource Alarm RabbitMQ has triggered a memory or disk resource alarm, and all publishers are blocked until the watermark is cleared. RabbitMQ sets a memory alarm when used_memory exceeds vm_memory_high_watermark (default: 40% of available RAM) and a disk alarm when free disk space falls below disk_free_limit. While any alarm is active, all connections that have published are blocked, causing producer services to stall. ### Rate Limited The service logs show repeated rate-limiting errors, indicating that the service or an upstream dependency is rejecting requests because a rate limit has been reached. Rate limiting is enforced by the service itself or by an upstream dependency such as an API gateway, an ingestion pipeline, or an external API. When the limit is exceeded, new requests are rejected or throttled, causing increased error rates for callers. Common sources of this pattern include ingestion services such as Grafana Tempo when write throughput exceeds the per-tenant ingestion limit, API gateways or proxies such as the nginx ngx_http_limit_req module when request rate exceeds the configured burst, and application-level rate limiters logging rate limit exceeded or rate limit reached when quota is consumed. The root cause is typically insufficient quota allocation, a sudden traffic spike, or a misconfigured rate limit policy rather than a code defect. Rejected requests can lock users out, which can explain a Concurrent Users Drop on the Business Application. ### Redis Cache Miss Storm Redis cache hit rate has dropped significantly, and upstream services are falling back to the database on most requests, multiplying database load. A cache miss storm occurs when Redis cannot serve requests from memory and callers bypass it to the backing database, dramatically increasing database query volume and latency for all services sharing that database. Common causes include memory pressure and key evictions, cache invalidation or a flush clearing a large portion of the keyspace, a cold start after the cache was restarted and not yet warmed, keys expiring simultaneously in a TTL cliff, and an application bug writing cache keys with wrong names that prevents future hits. ### Redis Command Queue Saturation Redis single-threaded command pipeline is saturated, and slow O(N) commands are blocking the event loop while BLPOP or BRPOP clients accumulate, causing upstream timeouts. Redis processes commands on a single thread, and when a slow command such as KEYS, SMEMBERS, SORT, or LRANGE on a large collection runs, it blocks all other commands for its duration. Clients waiting on BLPOP or BRPOP accumulate because they cannot be served while the thread is busy, and as blocked client count rises upstream services begin timing out. This root cause requires both signals to be present simultaneously: redis_slowlog_last_id or slow command log growth, and redis_blocked_clients above threshold. Common causes include application use of KEYS * or SMEMBERS on large keysets in production, large sorted sets or lists being iterated with single-command scans, and BLPOP or BRPOP patterns with no timeout or long timeout combined with slow producers. ### Redis Connection Pool Saturated Client-side Redis connection pool exhaustion occurs when all available connections in the pool are in use, preventing new requests to Redis. This can lead to request timeouts or failures, causing application disruptions for features relying on Redis for caching, messaging, or other operations. ### Redis Memory Exhausted Redis is rejecting commands because used_memory has exceeded the configured maxmemory limit, and clients receive OOM errors and writes fail. When Redis reaches its maxmemory limit, it enforces the configured eviction policy (for example allkeys-lru or volatile-lru). If the eviction policy cannot free enough space, or if maxmemory-policy is noeviction, all write commands are rejected with "OOM command not allowed when used memory > maxmemory". ### Redis Memory Pressure Redis server memory is at or near its configured limit, causing active key evictions that degrade cache effectiveness and increase load on downstream databases. Redis evicts keys when memory usage exceeds the configured maxmemory limit, and although the eviction policy determines which keys are removed, under heavy eviction the cache hit rate drops sharply and forces callers to read from the database instead, compounding into a database load spike. Common causes include the data set growing beyond allocated memory, no TTL being set on keys and causing unbounded growth, memory fragmentation consuming effective capacity, and a sudden traffic spike filling the keyspace faster than evictions can keep up. ### Redis Server Connections Exhausted Redis has reached its maxclients limit and is rejecting new connection attempts, causing connection refused errors in upstream services. Redis enforces a maximum number of simultaneous client connections via the maxclients configuration parameter, and when this limit is reached new connection attempts are immediately rejected with "ERR max number of clients reached," causing upstream services to receive connection errors and fail their Redis-dependent operations. Common causes include maxclients being set too low for the number of services connecting to the Redis instance, connection pool misconfiguration where pools do not release idle connections, sudden traffic spikes creating many short-lived connections without pooling, and connection leaks in application code. ### Slow Consumer The application is consuming messages slower than they are produced, creating a processing bottleneck. As unprocessed messages accumulate, the system experiences increased queue lag, potential memory pressure, and downstream congestion. This often indicates that one or more instances are unable to keep up due to resource constraints, inefficient processing logic, or external dependencies. ### Slow Database Queries The application is experiencing slow database queries that lead to downstream slow consumer behavior and potential resource starvation. This condition affects instance performance, particularly when query execution times become excessively long, degrading overall system responsiveness. ### Slow Database Server Queries Warehouse congestion in Snowflake occurs when the processing capacity is overwhelmed by incoming queries, causing a significant backlog. This leads to queries being queued at high rates, indicating that the system is struggling to process them in a timely manner. The resulting resource starvation further degrades performance. ### Transaction ID Congested In databases like PostgreSQL, transaction IDs are 32-bit integers that count the number of transactions performed. High utilization occurs when the counter nears its maximum value (~2 billion transactions), requiring a wraparound to continue operation. Failure to perform routine VACUUM operations can prevent the system from marking old XIDs as reusable. ### Unauthorized Access The application is receiving numerous "Unauthorized" status codes (typically HTTP 401) when trying to access another service. This prevents the application from successfully retrieving data or performing actions, potentially causing service disruptions or degraded functionality for end users. This diagnosis applies when only some callers are rejected; when most or all callers of a service are rejected, see Authentication Rejection on that service instead. It also applies when the caller invokes an MCP tool and only that caller is rejected. --- ## Data Pipeline (Root-causes) Series of processes that move and transform data between systems for analysis, storage, or further processing. ### Overview - [Database Table](#database-table) - [HTTP Path](#http-path) - [MCP Tool](#mcp-tool) - [RPC Method](#rpc-method) ### Database Table #### Contention on Database Table Locks The database table is experiencing an abnormally high rate of exclusive locks, preventing multiple transactions from accessing the table simultaneously. This creates a bottleneck that significantly degrades performance for all client applications depending on this table. Excessive locking typically occurs due to long-running transactions, lock contention between multiple transactions, inefficient transaction design, inappropriate isolation levels, or missing indexes leading to table scans instead of index seeks. #### Schema Change Causing Table Lock Contention The database table is experiencing an unusually high rate of Data Dictionary Lock (DDL), which blocks both read and write operations during schema modifications. This significantly impacts all client applications, causing service disruptions and performance degradation. #### Table Access Failure in Database The database table is experiencing performance degradation or errors, causing disruptions for client applications. This results in slow query response times, potential errors, and degraded service performance for systems that depend on this table. ### HTTP Path #### Authentication Rejection in HTTP Path The HTTP path is rejecting requests as unauthenticated or forbidden across its callers. The problem is most likely in this endpoint's authentication or authorization setup, such as a new or tightened authorization policy, a changed token audience or required scope, or a route-specific auth middleware, rather than in any single client. #### Faulty Error Handling in HTTP Path The HTTP path is experiencing a high rate of errors, causing disruptions for clients. This can lead to degraded performance, failed requests, or complete service unavailability, significantly affecting the user experience. #### Policy Rejection in HTTP Path The HTTP path is rejecting requests with HTTP 451 "Unavailable For Legal Reasons" across its callers. A resource-gating, geo-blocking, legal-hold, or feature-flag rule that covers this endpoint, for example a path prefix that matches more routes than intended, is the likely cause. #### Slow Execution in HTTP Path Handler The HTTP path is experiencing congestion, resulting in high latency for clients. This suggests that the system is unable to handle the current load efficiently, causing delays in response times. Congestion often occurs when the service receives more requests than it can handle within its capacity. #### Throttling Rejection in HTTP Path The HTTP path is throttling requests with HTTP 429 "Too Many Requests" across its callers. The problem is most likely in this endpoint's rate limit or quota rather than in any single client's request rate. ### MCP Tool #### Authentication Rejection in MCP Tool The MCP tool is rejecting calls as unauthenticated or forbidden across its callers. The problem is most likely in this tool's authentication or authorization setup, such as a new or tightened authorization policy, a changed token audience or required scope, or tool-specific auth middleware on the MCP server, rather than in any single client. #### Faulty Error Handling in MCP Tool The MCP tool is returning a high rate of errors, causing failed tool calls for the clients and agents that invoke it. Typical causes are tool execution failures, arguments that do not match the tool schema, failing downstream dependencies, or expired credentials and misconfigured tool settings on the MCP server. #### Policy Rejection in MCP Tool The MCP tool is rejecting calls because of a policy across its callers. A resource-gating, geo-blocking, legal-hold, or feature-flag rule that covers this tool, for example a tool name or namespace match that is broader than intended, is the likely cause. #### Slow Execution in MCP Tool The MCP tool is experiencing congestion, resulting in high latency for the clients and agents that invoke it. This suggests that the tool or its MCP server cannot handle the current invocation load, for example because of resource constraints on the MCP server, slow downstream operations, expensive tool logic, or a surge in tool calls. #### Throttling Rejection in MCP Tool The MCP tool is throttling calls across its callers. The problem is most likely in this tool's rate limit or quota rather than in any single client's call rate. ### RPC Method #### Authentication Rejection in RPC Method The RPC method is rejecting requests as unauthenticated or forbidden across its callers. The problem is most likely in this endpoint's authentication or authorization setup, such as a new or tightened authorization policy, a changed token audience or required scope, or a route-specific auth middleware, rather than in any single client. #### Faulty Error Handling in RPC Method The RPC Method is experiencing a high rate of errors, causing disruptions for clients. This can lead to degraded performance, failed requests, or complete service unavailability, significantly affecting the user experience. #### Slow Execution in RPC Method Handler The RPC method is experiencing congestion, resulting in high latency for clients. This suggests that the system is unable to handle the current load efficiently, causing delays in response times. Congestion often occurs when the service receives more requests than it can handle within its capacity. #### Throttling Rejection in RPC Method The RPC method is throttling requests across its callers. The problem is most likely in this endpoint's rate limit or quota rather than in any single client's request rate. --- ## Infrastructure Servers or cloud resources to execute tasks, run applications, or perform calculations. ### Overview - [Cluster](#cluster) - [Compute Spec](#compute-spec) - [Container](#container) - [Controller](#controller) - [Disk](#disk) - [Network Endpoint](#network-endpoint) - [Node](#node) - [VirtualMachine](#virtualmachine) ### Cluster #### Failed To Schedule Pods across many workloads in this Kubernetes cluster stay Pending, and the scheduler's message is not a recognized cluster-wide reason. This includes messages that report several capacity reasons at once, such as cordoned nodes and insufficient CPU during a node pool upgrade. When the message matches a recognized cause, a more specific diagnosis is shown instead. #### Failed To Schedule Insufficient CPU Pods cannot be scheduled because no node in this Kubernetes cluster has enough unreserved CPU for their requests. This happens when the cluster is at capacity and cannot grow (node pool maximum, quota, or cloud capacity stockout), or when a workload requests more CPU than any node can offer. #### Failed To Schedule Insufficient Memory Pods cannot be scheduled because no node in this Kubernetes cluster has enough unreserved memory for their requests. This happens when the cluster is at capacity and cannot grow (node pool maximum, quota, or cloud capacity stockout), or when a workload requests more memory than any node can offer. #### Failed To Schedule Nodes Cordoned Pods cannot be scheduled because the nodes in this Kubernetes cluster are cordoned or being drained. Cordoned nodes keep running their existing pods but accept no new ones. Common causes are a node pool upgrade where the replacement nodes did not come up, a manual drain that was never uncordoned, or autoscaler disruption before replacement capacity was ready. ### Compute Spec #### CPU Congested One or multiple containers in a workload are experiencing CPU congestion, leading to potential throttling. This occurs when the containers use more CPU resources than allocated, causing degraded performance, longer response times, or application crashes. CPU throttling occurs when a container exceeds its CPU quota as defined by Kubernetes or Docker. #### Crash Failure One or multiple containers of a workload has crashed with a non-zero exit code, indicating abnormal termination. This disrupts the application's functionality, leading to downtime or degraded performance depending on how the workload is designed. The non-zero exit code signifies an error during the execution of the container's process. #### Frequent Crash Failure One or multiple containers of a workload are frequently crashing with a non-zero exit code, indicating abnormal termination. This disrupts the application's functionality, leading to downtime or degraded performance depending on how the workload is designed. #### Frequent Memory Failure The application frequently runs out of memory, leading to crashes, performance degradation, or instability. This affects the application's availability and can lead to downtime or poor user experience. The issue is likely due to inefficient memory usage, such as memory leaks, excessive data loading into memory, or improper garbage collection. #### Memory Failure Containers running out of memory can lead to service crashes or degraded performance, resulting in errors for end users or failed service requests. This typically occurs when a container's allocated memory is insufficient for the workload it is handling, causing out-of-memory (OOM) errors and potential system instability. ### Container #### Ephemeral Storage Congested A container is experiencing ephemeral storage congestion when its ephemeral storage usage becomes critically high, leading to failures in operations that depend on temporary storage. This may be triggered by factors such as excessive logging, inadequate cleanup of temporary files, or unexpected bursts in data processing. #### Ephemeral Storage Noisy Neighbor A container acting as a noisy neighbor consumes excessive ephemeral storage, resulting in abnormally high storage usage and contributing to node-level disk pressure that can trigger pod evictions. This issue arises when a container consistently uses more ephemeral storage than expected. #### Memory Noisy Neighbor A container acting as a noisy neighbor consumes excessive memory, leading to abnormally high memory usage and contributing to node-level memory pressure that can trigger pod evictions. This issue occurs when a container consistently uses more memory than expected, which adversely impacts both the container and its hosting node. ### Controller #### Failed To Schedule Pods of this workload stay Pending because the Kubernetes scheduler cannot place them on any node. Replicas that are stuck never start, so the workload runs below its desired capacity, and a rollout or scale-up does not complete. This diagnosis applies when the scheduler's reason is not specific enough for a more targeted diagnosis, for example when several capacity reasons are reported at once. Common causes are pod spec constraints that no node can satisfy, missing tolerations, or volume placement. #### Failed To Schedule Node Affinity Mismatch Pods of this workload cannot be scheduled because no node matches their node selector or node affinity. Typical causes are a typo or stale label in the node selector, a removed node pool, instance type or zone that the workload targets, or several required affinity terms that together exclude every node. #### Failed To Schedule Unbound Volume Claim Pods of this workload cannot be scheduled because a PersistentVolumeClaim they use is not bound to a volume. The scheduler does not consider any node until the claim is bound. Common reasons are a wrong or missing StorageClass, no matching PersistentVolume with static provisioning, or a storage provisioner failure. #### Failed To Schedule Untolerated Taint Pods of this workload cannot be scheduled because every node they could run on has a taint that the pod spec does not tolerate. Typical causes are a dedicated node pool (GPU, spot, or team) where the workload is missing the matching toleration, node pool taints that changed, or only tainted nodes being left. #### FrequentPodEphemeralStorageEvictions A Kubernetes workload is experiencing frequent pod evictions due to ephemeral storage exhaustion. This disrupts application availability and performance, as pods are terminated when they exceed their allocated storage limits or when node-level storage is under pressure. #### Image Pull Errors Kubernetes controllers may encounter image pull errors when they cannot download container images from a registry, causing Pods to fail in starting or remain in an ImagePullBackOff state. This disrupts the deployment of applications and can affect service availability. #### Malfunction Multiple pods for a Kubernetes controller are in a "NotReady" state for an extended period, which can lead to service unavailability or degraded performance. ### Disk #### Congested The disk has reached full capacity, which prevents new data from being written and may cause applications to fail, especially those dependent on free disk space for logs, caching, or temporary files. This can also slow down or halt system operations if critical processes can no longer write to the disk. #### Inode Usage Congested The disk is experiencing inode exhaustion, meaning the file system has run out of inodes (metadata structures for file storage), which prevents new files from being created even if there is free disk space. This often causes errors in applications attempting to create files and can disrupt services reliant on file storage. #### IOPs Congested The disk is experiencing Read/Write Operations Per Second (IOPS) congestion, meaning that the total IOPS capacity is fully utilized. This causes slow performance for applications that rely on disk access, leading to delayed data processing, system lags, or even timeouts. #### Read IOPs Congested The disk is experiencing Read Operations Per Second (IOPS) congestion, meaning that the total IOPS capacity is fully utilized. This causes slow performance for applications that rely on disk access, leading to delayed data processing, system lags, or even timeouts. #### Read Throughput Congested The disk is experiencing congestion specifically in read throughput, which slows down data retrieval from the disk and can degrade the performance of applications reliant on high-speed data access. #### Write IOPs Congested The disk is experiencing Write Operations Per Second (IOPS) congestion, meaning that the total IOPS capacity is fully utilized. This causes slow performance for applications that rely on disk access, leading to delayed data processing, system lags, or even timeouts. #### Write Throughput Congested The disk is experiencing write throughput congestion, leading to slower data write speeds and affecting applications that require high-speed data recording. This issue can cause delays in data availability and reduced performance in write-intensive tasks. ### Network Endpoint #### Invalid Server Certificate The network endpoint is serving an invalid server certificate, resulting in a high rate of client request errors due to certificate validation failures. This issue propagates further, increasing the overall request error rate across the system. ### Node #### Causely Agent Down The Causely agent on a Kubernetes node is unavailable, creating an observability gap for that node. When the agent is down, telemetry and symptom collection from the affected node may be incomplete or missing, reducing Causely's ability to detect and analyze issues originating there. Common causes include the node becoming unreachable or unavailable and taking the agent down with it, the agent container crashing or failing health checks, CPU, memory, or filesystem pressure on the node preventing the agent from running reliably, and configuration or deployment problems such as invalid configuration, rollout failures, or image errors. #### Container Runtime Issue The node's container runtime is unhealthy or unstable, preventing containers from starting or running reliably. Kubernetes depends on the container runtime to manage the full container lifecycle, including creation, startup, shutdown, and health monitoring. When the runtime is impaired, pods may fail to launch, existing workloads can become unstable, and the node may drift into a degraded state. Common causes include repeated crashes or restart loops in the runtime process that interrupt container lifecycle management, invalid runtime configuration preventing the node from managing containers correctly, overlay filesystem corruption or local disk issues breaking container image operations, and CPU, memory, or I/O saturation destabilizing the runtime. #### Disk Pressure Disk pressure on a Kubernetes node indicates that the node's disk usage is high, potentially causing the eviction of pods, reduced performance, and the inability to schedule new pods. This affects application stability and the node's overall functionality. Disk pressure can arise from insufficient disk space, often caused by log accumulation, container images, temporary files, or application data. #### Filesystem Issue The node filesystem is corrupted or has been remounted read-only, preventing normal node and container operation. The kubelet and container runtime depend on writable filesystem access to maintain state, update pod data, and manage container lifecycles, and when that access is lost, workloads on the node may fail entirely. Common causes include host filesystem corruption that breaks kubelet or runtime state management, the kernel remounting the filesystem read-only after detecting storage errors, and underlying disk or cloud volume failures that surface as filesystem instability. #### Kernel Issue The node kernel is stalled or deadlocked, preventing the host from functioning correctly. Kernel-level failures are severe conditions that can block scheduling, container management, networking, and filesystem operations simultaneously, and a kernel deadlock often precedes full node unavailability. Common causes include bugs in the kernel or kernel modules that deadlock critical host operations, storage or networking drivers triggering hangs under load, and extreme resource contention exposing kernel-level instability. #### Kubelet Issue The kubelet on this node is unhealthy or restarting frequently, disrupting pod lifecycle operations and making the node unreliable. The kubelet is the primary Kubernetes agent on each node and is responsible for reporting node status, managing pod readiness, and coordinating with the container runtime. When it becomes unstable, the node may stop reporting status correctly, fail to update readiness conditions, and struggle to manage running pods. Common causes include CPU, memory, or disk pressure destabilizing the kubelet process, invalid kubelet flags or node configuration causing repeated failures, problems with the container runtime, networking stack, or host filesystem cascading into kubelet instability, and kernel or VM-level host issues making the kubelet unreliable. #### Malfunction A Kubernetes node is unavailable or unresponsive, causing workloads scheduled on it to fail. This typically means the kubelet or the underlying host is no longer operating normally, and pods on the affected node may become unavailable, fail health checks, or stop serving traffic entirely. Common causes include CPU, memory, disk, or kernel resource exhaustion destabilizing the node, the kubelet, container runtime, or other host-level services becoming unhealthy or unreachable, cloud instance failures, network isolation, or underlying VM problems making the node unavailable, and excessive pod concentration or invalid node-level configuration causing the node to become unstable. #### Memory Pressure Memory pressure on a Kubernetes node occurs when available memory falls below critical levels, potentially causing the eviction of pods and instability for applications running on the node. This reduces the node's capacity to run workloads, potentially leading to service disruptions if insufficient resources are available across the cluster. #### Network Issue The node network is unavailable or unstable, causing connectivity failures for node services and workloads. Node-level networking problems can disrupt kubelet communication, service routing, pod-to-pod traffic, and access to external systems, and in practice can make the node appear partially or fully unavailable even when the host itself is still running. Common causes include network interface instability such as resets or repeated unregister events interrupting traffic, misconfiguration in the node's CNI or routing stack isolating the node from the cluster, and underlying VM or cloud network failures cutting the node off from the rest of the infrastructure. #### PID Pressure The node is close to exhausting its process ID capacity, limiting its ability to create new processes and causing workload instability. When PID limits are approached, Kubernetes may be unable to start new processes reliably and workloads can become unstable even when CPU and memory remain available. Common causes include applications or system services creating excessive processes or threads, too many workloads concentrated on a single node, and PID limits configured too low for the workload profile running on the node. ### VirtualMachine #### Conntrack Table Congested The conntrack table on a VM is congested, causing new network connections to fail. This typically results in connectivity issues for applications, degraded performance, or downtime for services dependent on network communication. The conntrack table is responsible for tracking active network connections and has a fixed size, which can be exhausted under high connection load. #### CPU Congested A Virtual Machine (VM) experiencing CPU congestion can lead to sluggish application performance, delayed response times, or even timeout errors for users and processes. This typically indicates that the VM's CPU is overutilized, potentially due to high resource demands from applications or insufficient CPU allocation. #### Disk Read IOPs Congested The total disk read IOPS for a cloud VM are congested because the VM has reached its maximum allowable IOPS limit. This results in throttling, which can slow application performance and lead to delays or errors in read-heavy workloads. #### Disk Read Throughput Congested The total disk read throughput for a cloud VM is congested because the VM has reached its maximum allowable read bandwidth. This can lead to slower data transfer rates for read-intensive applications, causing delays in processing and reduced system performance. #### Disk Total IOPs Congested The total disk IOPS for a cloud VM are congested because the VM has reached its maximum allowable IOPS limit. This results in throttling, which can slow application performance and lead to delays or errors in read/write-heavy workloads. #### Disk Total Throughput Congested The total disk throughput for a cloud VM is congested because the VM has reached its maximum allowable bandwidth. This can lead to slower data transfer rates for read/write-intensive applications, causing delays in processing and reduced system performance. #### Disk Write IOPs Congested The total disk write IOPS for a cloud VM are congested because the VM has reached its maximum allowable IOPS limit. This results in throttling, which can slow application performance and lead to delays or errors in write-heavy workloads. #### Disk Write Throughput Congested The total disk write throughput for a cloud VM is congested because the VM has reached its maximum allowable write bandwidth. This can lead to slower data transfer rates for write-intensive applications, causing delays in processing and reduced system performance. #### Memory Congested Memory congestion in a Virtual Machine (VM) leads to slow system performance, application crashes, or even VM instability as the system struggles to allocate memory for running processes. This typically results in frequent swapping or out-of-memory (OOM) errors, impacting applications and user operations. #### SNAT Ports Congested The SNAT (Source Network Address Translation) ports on a virtual machine (VM) are congested, leading to outbound network connection failures or degraded performance for services relying on external APIs or resources. This issue primarily impacts VMs that need to establish multiple concurrent connections to the internet or external systems. --- ## Diagnosable Root Causes Modern cloud native systems evolve quickly, and complexity makes failures propagate: a small change in one part of the system can ripple across dependencies and surface as Signals that impact user experience. In Causely, a **Diagnosis** is the causal explanation for a set of Signals: the factor Causely infers is responsible for them. It is the current best explanation and updates as Signals change. A Diagnosis is not an alert or an incident; it is derived from structured dependency models, observed telemetry, and causal reasoning. ### How urgency is determined Not every Issue needs immediate attention. Causely ranks urgency automatically, using two controls you can adjust: Service Prioritization and SLOs. - **Service Prioritization** sets how much a service matters. Each service is assigned a tier. Critical services have their SLO enforced and their Diagnoses can be marked urgent. Non-Critical services stay visible but are never marked urgent. Hidden services are kept out of the active view. See [Service Tiers](/configuration/service-tiers/). - **SLOs** define what healthy looks like for a service and decide when urgency fires. When a Diagnosis puts a Critical service's SLO at risk or in violation, Causely marks that Diagnosis urgent and elevates the Issue it belongs to. See [SLO Configuration](/configuration/slo-configuration/). The two work together. Service Prioritization decides whether a service's Diagnoses can be urgent at all. SLO status decides when that urgency triggers. Assign tiers first, then tune SLOs on the services that matter. An Issue's urgency is the highest urgency of any Diagnosis over its life. Because a Diagnosis can update as Signals change, an Issue that was urgent stays urgent once any Diagnosis has crossed the threshold, even if the current Diagnosis no longer puts an SLO at risk. ### Browse Diagnoses Below you can find a list of root cause types that are captured in our Causal Models: the causes each Diagnosis can identify. With these, Causely can pinpoint hundreds of thousands of potential issues and their effects within your environment. Each Diagnosis is connected in the causality graph to the Signals it explains, with supporting evidence and impact context. Use the search and filters below to explore all Diagnoses by category, subcategory, or integration source. --- ## Release Management Diagnoses that are triggered by a code change or new release and result in a measurable reliability regression. ### Code Change Regression: CPU Congestion After a version upgrade, application containers experience high CPU usage, leading to performance degradation or unresponsiveness. This issue impacts the system's ability to handle requests effectively, potentially causing downtime or delays for end users. High CPU usage post-upgrade typically stems from changes in the application code, dependencies, or configurations. ### Code Change Regression: Database Connection Pool Saturated After a version upgrade, the client-side database connection pool is exhausted when all available connections are in use, preventing new database queries from being executed. This can cause application requests to hang or fail, impacting user experience and potentially leading to downtime for database-dependent features. ### Code Change Regression: Frequent Crash Failure One or multiple containers of a workload are frequently crashing with a non-zero exit code after a version upgrade. This disrupts the application's functionality, leading to downtime or degraded performance depending on the workload design. The issue likely stems from changes introduced in the new version. ### Code Change Regression: Frequent Memory Failure The application is running out of memory after a version upgrade, leading to crashes, degraded performance, or instability. This impacts availability and user experience, often requiring container restarts or manual intervention to restore functionality. The issue is likely tied to changes in the updated version that increase memory usage or introduce inefficiencies. ### Code Change Regression: Inefficient Garbage Collection After a version upgrade, the garbage collector is frequently running, leading to performance degradation or crashes. This issue is likely caused by changes in the application code or dependencies that increase memory usage or introduce inefficiencies. ### Code Change Regression: Java Heap Saturated After a version upgrade, the Java heap is frequently congested, leading to performance degradation or crashes. This issue is likely caused by changes in the application code or dependencies that increase memory usage or introduce inefficiencies. ### Code Change Regression: Lock Contention After a version upgrade, the application is experiencing frequent locking contention, leading to performance degradation or crashes. This issue is likely caused by changes in the application code or dependencies that increase locking or introduce inefficiencies. ### Code Change Regression: Memory Failure Memory failures after a code change can cause containers to crash or degrade performance, resulting in errors for end users or failed service requests. These issues occur when newly introduced code leads to unexpected increases in memory usage, triggering out-of-memory (OOM) errors and destabilizing the system. ### Code Change Regression: Redis Connection Pool Saturated After a version upgrade, the Redis connection pool is frequently congested, leading to performance degradation or crashes. This issue is likely caused by changes in the application code or dependencies that increase Redis usage or introduce inefficiencies. ### Code Change Regression: Slow Database Queries After a version upgrade, the application is experiencing slow database queries that lead to downstream slow consumer behavior and potential resource starvation. This condition affects instance performance, particularly when query execution times become excessively long. --- ## Services (Root-causes) Services are self-contained units of functionality within a system that perform specific tasks or provide specific capabilities, often accessible through defined interfaces or APIs. Service can be either internal or third-party external services beyond the core application and infrastructure. ### Overview - [AI Model](#ai-model) - [Application Load Balancer](#application-load-balancer) - [Messaging / Event Streaming](#messaging--event-streaming) - [Service](#service) ### AI Model #### Congested The AI model is experiencing congestion, resulting in high latency for clients. This suggests that the model is unable to handle the current inference load efficiently, causing delays in response times. Congestion often occurs when the model receives more requests than it can process within its capacity, leading to bottlenecks in inference. This may be due to insufficient resources (for example, GPU, memory, or throughput limits), inefficient batching, or a surge in inference demand. #### Malfunction The AI model is experiencing a high rate of errors, causing disruptions for clients. This can lead to degraded performance, failed inference requests, or complete model unavailability, significantly affecting the user experience. ### Application Load Balancer #### Authentication Misconfiguration Application Load Balancer (ALB) authentication misconfiguration can disrupt secure traffic routing and lead to widespread configuration issues. This misconfiguration may trigger elevated ELB authentication errors, 504 request timeouts, and target connection errors, ultimately impacting service availability and application performance. #### Idle Timeout Misconfiguration Misconfigured idle timeout settings can lead to unintended connection drops and delays, potentially triggering a high frequency of 504 gateway timeout errors. This misconfiguration may also contribute to broader configuration issues that disrupt seamless connectivity between clients and servers. #### Network Policy Misconfiguration Application Load Balancer (ALB) network policy misconfigurations can block or restrict legitimate traffic, leading to widespread configuration issues. These misconfigurations often result in elevated 504 gateway timeout errors and target connection failures, ultimately disrupting service availability. #### Unknown Configuration Failure Application Load Balancer (ALB) misconfiguration can cause widespread connectivity issues, leading to a high frequency of 504 gateway timeout errors and 5xx server errors. These issues indicate that the load balancer's settings are not properly optimized for handling traffic efficiently and reliably. ### Messaging / Event Streaming #### Congested Azure Event Hub Namespace When the Azure Event Hub namespace becomes congested, it reaches a point where its processing capacity is exceeded. This leads to consistent throttling of operations, as the system enforces limits to prevent overload. The high rate of throttling not only impacts event ingestion but also cascades into resource starvation, affecting downstream services that rely on timely event processing. Such congestion is typically caused by high message throughput, suboptimal configuration, or insufficient scaling to handle peak loads. #### High Server Errors Common causes for Azure Event Hub errors include quota exceeded (throughput or message size limits have been breached), partition or offset issues (consumers unable to connect or reading from invalid offsets), networking problems (connectivity issues due to firewall rules, DNS misconfigurations, or latency), service outage (regional Azure service disruption), and misconfigured access policies (incorrect SAS tokens, permissions, or authentication methods). #### High User Errors High user error rates in Azure Event Hub often stem from configuration issues, such as mismatched security credentials (for example Shared Access Signature (SAS) tokens), client SDK version incompatibilities, or throttling from overusing allocated resources. Insufficient permissions or quotas being exceeded can also trigger these errors. Another common cause is incorrect partition or consumer group usage, which can lead to connection limits being breached or messages being inaccessible. ### Service #### Authentication Rejection The service is rejecting requests as unauthenticated or forbidden (HTTP 401/403, or gRPC UNAUTHENTICATED/PERMISSION_DENIED) across its callers. The problem is most likely in the service's own authentication or authorization setup rather than in any single client. Common causes are a changed token issuer, audience, or JWKS configuration, an expired or rotated signing key or certificate, a tightened authorization policy, or an identity provider becoming unavailable. This diagnosis also applies to services whose callers cannot be observed, such as an internet-facing gateway, when most requests are rejected. When only a single caller is rejected while others are served normally, see Unauthorized Access on that caller instead. Rejected users are locked out, which can explain a Concurrent Users Drop on the service's Business Application. #### Client Connection Disruption Existing client connections to the service were dropped, for example by a load balancer or proxy change. Clients then reconnect at the same time, which causes a reconnect storm. This predicts an Endpoint Request Burst and lost concurrent users. It can only be identified when the service's Business Application reports concurrent users. #### Congested The service is experiencing congestion, resulting in high latency for clients. This suggests that the system is unable to handle the current load efficiently, causing delays in response times. Congestion often occurs when the service receives more requests than it can handle within its capacity, leading to bottlenecks in processing. This may be due to insufficient resources (for example, CPU, memory, or bandwidth), unoptimized code, or a surge in traffic (for example, due to a sudden increase in demand or DDoS attack). #### Malfunction The Service is experiencing a high rate of errors, causing disruptions for clients. This can lead to degraded performance, failed requests, or complete service unavailability, significantly affecting the user experience. #### Policy Rejection The service is rejecting requests with HTTP 451 "Unavailable For Legal Reasons", meaning one of its own policies refuses to serve them. Clients receive errors even though the service and its dependencies are healthy. A sudden rise in 451 responses usually means a resource-gating, geo-blocking, legal-hold, or feature-flag rule was changed or rolled out and now covers more routes, tenants, or regions than intended. When only a single caller is rejected while others are served normally, see Access Rejected By Policy on that caller instead. Rejected users are locked out, which can explain a Concurrent Users Drop on the service's Business Application. #### Throttling Rejection The service is throttling requests with HTTP 429 "Too Many Requests" across its callers. The problem is most likely in the service's own rate limit or quota rather than in any single client's request rate. Common causes are a lowered rate limit or quota, a limiter that counts requests per instance and was sized for a different number of replicas, a shared limit that one feature or tenant now exhausts, or traffic that has grown past the configured limit. When only a single caller is throttled while others are served normally, see Access Throttled on that caller instead. When most or all callers are rejected, users can be locked out, which can explain a Concurrent Users Drop on the service's Business Application. --- ## Applications (Symptoms) Signals related to Business Applications, application workloads, and compute specifications. ### Business Application - **Concurrent Users Drop** - Concurrent users are below the lower threshold. Activates after 1 minute below the threshold and clears 3 minutes after concurrent users recover. ### Compute Spec - **CPU Avg Increase Version Changed High** - Sustained increase in average CPU usage after version change - **Has DB Connections Increase High Instance** - Has instances with high database connections increase - **Has DB Query Duration Increase High Instance** - Has instances with high database query duration increase - **Has Frequent Crash High Instance** - Has instances with high crash frequency - **Has Frequent OOM Kill High Instance** - Has instances with high OOM kill frequency - **Has GC Time High Instance** - Has instances with high garbage collection time - **Has Java Heap Utilization High Instance** - Has instances with high Java heap utilization - **Has Mutex Wait Time High Instance** - Has instances with high mutex wait time - **Has Redis Connections Utilization High Instance** - Has instances with high Redis connections utilization - **Memory Avg Increase Version Changed High** - Sustained increase in average memory usage after version change ### Workload - **.Net Unhandled Exception In Log** - .NET unhandled exception detected in logs - **Apache Max Workers In Log** - Apache MaxRequestWorkers limit reached detected in logs - **Cache Size High** - High cache size - **Cassandra Tombstone In Log** - Excessive tombstone scanning detected in logs - **Circuit Breaker Open In Log** - Circuit breaker open event detected in logs - **Command Latency High** - High command latency - **Connection Pool Exhausted In Log** - Connection pool exhaustion detected in logs - **Connections High** - High number of connections - **DB Connections Utilization High** - High database connections utilization - **DB Query Duration High** - High database query duration - **Disk Full In Log** - Disk full (ENOSPC) error detected in logs - **Elasticsearch Red In Log** - Elasticsearch cluster RED status detected in logs - **File Descriptor Utilization High** - High file descriptor utilization - **File Limit In Log** - File descriptor limit reached detected in logs - **GC Time High** - High garbage collection time - **Go Goroutine Leak In Log** - Go goroutine leak detected in logs - **Go Max Procs Too High** - GOMAXPROCS value is too high - **Go Nil Pointer Panic In Log** - Go nil pointer panic detected in logs - **HAProxy Max Conn In Log** - HAProxy backend unavailable event detected in logs - **High Error Log Count** - High rate of ERROR or CRITICAL log lines - **High Kafka Data Logs Disk Utilization** - High Kafka broker data log disk utilization - **High Kafka Message Rate** - High Kafka message rate - **High Kafka Offline Partitions** - Kafka has one or more offline partitions - **High Kafka Under Replicated Partitions** - Kafka has under-replicated partitions - **High MongoDB Connection Usage** - High MongoDB active connection utilization - **High MongoDB Open Cursor Count** - High number of open MongoDB cursors - **High MongoDB Replica Lag** - MongoDB replication lag is high - **High Postgres Checkpoint Write Time** - Postgres checkpoint write time above 5 seconds, disk I/O is a bottleneck - **High Postgres Connection Utilization** - Postgres active sessions above 85% of max_connections, new connections will soon be rejected - **High Postgres Deadlock Rate** - Postgres deadlock rate above 0.1 seconds, concurrent conflicting transactions - **High Postgres Lock Wait Count** - More than 5 sessions waiting on a lock, lock contention is occurring - **High Redis Cache Miss Rate** - Redis cache miss ratio above 30%, most requests are falling through to the database - **High Redis Memory Fragmentation** - Redis memory fragmentation ratio above 1.5, significant allocator overhead - **High Redis Replication Lag** - Redis replica is more than 10 seconds behind the master - **High Redis Server Memory Usage** - Redis server memory utilization above 85%, evictions are likely or imminent - **Java GC Pause In Log** - Java GC allocation failure or to-space exhaustion detected in logs - **Java Heap Utilization High** - High Java heap utilization - **Java Null Pointer In Log** - Java NullPointerException detected in logs - **Java OOM In Log** - Java OutOfMemoryError detected in logs - **Java Stack Overflow In Log** - Java StackOverflowError detected in logs - **Java Thread Exhaustion In Log** - Java thread pool rejection detected in logs - **Kafka ISR Shrink In Log** - Kafka ISR shrinkage event detected in logs - **Low Postgres Cache Hit Rate** - Postgres cache miss rate above 10%, queries are reading from disk instead of shared_buffers - **Memory Utilization High** - High memory utilization - **Mutex Wait Time High** - High mutex wait time - **MySQL Deadlock In Log** - MySQL InnoDB deadlock detected in logs - **Nginx Upstream Timeout In Log** - Nginx upstream connection timeout detected in logs - **Nginx Worker Conn In Log** - Nginx worker connections exhausted detected in logs - **Postgres Deadlock In Log** - PostgreSQL deadlock detected in logs - **Postgres Idle-in-Transaction Sessions Accumulating** - More than 3 sessions in idle-in-transaction, they hold locks and block vacuum - **Postgres Query Memory Spill** - Postgres is creating temporary files at a rate above 0.5 seconds, queries are spilling to disk - **Python Unhandled Exception In Log** - Python unhandled exception detected in logs - **Queries Queued High** - High number of queued queries - **RabbitMQ Alarm In Log** - RabbitMQ memory or disk resource alarm detected in logs - **Rate Limit In Log** - Rate limit exceeded detected in logs - **Redis Blocked Clients Accumulating** - More than 5 clients are blocked on BLPOP/BRPOP commands - **Redis Connections Utilization High** - High Redis connections utilization - **Redis Max Memory In Log** - Redis maxmemory limit exceeded detected in logs - **Redis Rejecting Connections** - Redis is rejecting new connections at a rate above 0.1 seconds, maxclients has been reached - **Redis Slow Commands Queued** - Redis slow command log is growing, O(N) commands being run against large data structures - **Request Duration High** - High request duration/latency - **Request Error Rate High** - High request error rate - **Request Policy Rejected Rate High** - High rate of requests rejected by policy (HTTP 451) - **Request Throttled Rate High** - High throttled request rate - **Request Unauthorized Rate High** - High unauthorized request rate - **Server Errors High** - High server error count - **Throttled High** - High throttled request count - **Transaction Duration High** - High transaction duration - **Transaction Error High** - High transaction error rate - **Transaction IDs Congested High** - High transaction ID congestion - **User Errors High** - High user error count --- ## Data Pipeline (Symptoms) Signals related to operations, access patterns, and data flow. ### Access - **Most Requests** - This access has the most requests - **Request Duration High** - High request duration/latency - **Request Error Rate High** - High request error rate - **Request Invalid Certificate Rate High** - High invalid certificate rate - **Request Policy Rejected Rate High** - High rate of requests rejected by policy (HTTP 451) - **Request Throttled Rate High** - High throttled request rate - **Request Unauthorized Rate High** - High unauthorized request rate ### Background Operation - **Task Duration High** - High task duration - **Throughput Drop** - This operation handles or emits fewer units of work than expected ### DNS Access - **Request Duration High** - High request duration/latency - **Request Error Rate High** - High request error rate - **Request Invalid Certificate Rate High** - High invalid certificate rate - **Request NXDomain Rate High** - High NXDOMAIN response rate - **Request Rate High** - High request rate - **Request Throttled Rate High** - High throttled request rate - **Request Unauthorized Rate High** - High unauthorized request rate ### HTTP Path - **HTTP Request Burst** - Request rate is above 3x its 30-minute average, or above average + 1 req/s for quiet endpoints. The baseline excludes the most recent 5 minutes. - **Request Duration High** - High request duration/latency - **Request Error Rate High** - High request error rate - **Request Policy Rejected Rate High** - High rate of requests rejected by policy (HTTP 451) - **Request Throttled Rate High** - High throttled request rate - **Request Unauthorized Rate High** - High unauthorized request rate - **Throughput Drop** - This operation handles or emits fewer units of work than expected ### HTTP Path Access - **Request Duration High** - High request duration/latency - **Request Error Rate High** - High request error rate - **Request Policy Rejected Rate High** - High rate of requests rejected by policy (HTTP 451) - **Request Throttled Rate High** - High throttled request rate - **Request Unauthorized Rate High** - High unauthorized request rate ### MCP Tool - **Request Duration High** - High request duration/latency - **Request Error Rate High** - High request error rate - **Request Policy Rejected Rate High** - High rate of requests rejected by policy - **Request Throttled Rate High** - High throttled request rate - **Request Unauthorized Rate High** - High unauthorized request rate - **Throughput Drop** - This operation handles or emits fewer units of work than expected ### MCP Tool Access - **Request Duration High** - High request duration/latency - **Request Error Rate High** - High request error rate - **Request Policy Rejected Rate High** - High rate of requests rejected by policy - **Request Throttled Rate High** - High throttled request rate - **Request Unauthorized Rate High** - High unauthorized request rate ### Queue - **Consumer Throughput Drop** - Queue delivers fewer messages to consumers than expected - **Producer Throughput Drop** - Queue receives fewer messages from producers than expected - **Throughput Drop** - Queue aggregate traffic is lower than expected; direction is unknown ### Queue Access - **Dead Letter Count High** - High dead letter count - **Lag High** - High consumer lag - **Message Wait Time High** - High message wait time - **Queue Depth High** - High queue depth ### RPC Method - **Request Duration High** - High request duration/latency - **Request Error Rate High** - High request error rate - **Request Throttled Rate High** - High throttled request rate - **Request Unauthorized Rate High** - High unauthorized request rate - **RPC Request Burst** - Request rate is above 3x its 30-minute average, or above average + 1 req/s for quiet endpoints. The baseline excludes the most recent 5 minutes. - **Throughput Drop** - This operation handles or emits fewer units of work than expected ### RPC Method Access - **Request Duration High** - High request duration/latency - **Request Error Rate High** - High request error rate ### Table Access - **DDL Lock Exclusive Rate High** - High DDL exclusive lock rate - **Lock Exclusive Rate High** - High exclusive lock rate - **Query Duration High** - High query duration - **Select Query Duration High** - High SELECT query duration - **Table Bloat High** - High table bloat ### Topic - **Consumer Throughput Drop** - Topic delivers fewer messages to consumers than expected - **Producer Throughput Drop** - Topic receives fewer messages from producers than expected - **Throughput Drop** - Topic aggregate traffic is lower than expected; direction is unknown ### Topic Access - **Lag High** - High consumer lag - **Message Wait Time High** - High message wait time - **Queue Depth High** - High queue depth --- ## Infrastructure (Symptoms) Signals related to compute resources, storage, and networking. ### Compute #### Container - **CPU Throttled High** - High CPU throttling - **CPU Utilization High** - High CPU utilization - **Ephemeral Storage Highest Usage** - Highest ephemeral storage usage - **Ephemeral Storage Utilization High** - High ephemeral storage utilization - **Frequent Crash High** - High crash frequency - **Frequent OOM Kill High** - High OOM kill frequency - **Memory Highest Usage** - Highest memory usage - **Memory Utilization High** - High memory utilization #### Controller - **Frequent Pod Ephemeral Storage Evictions High** - High frequency of pod evictions due to ephemeral storage - **Has Image Pull Error Instance** - Has instances with image pull errors - **Workload Unschedulable** - Pods of the workload are Pending because the scheduler cannot place them #### Node - **Causely Agent Down** - Causely agent on node is unavailable - **Conn Track Table Utilization High** - High connection tracking table utilization - **Container Ephemeral Storage Utilization High** - High container ephemeral storage utilization - **Container Runtime Problem Detected** - Container runtime problem detected - **Corrupt Docker Overlay2 Detected** - Corrupt Docker overlay2 filesystem detected - **CPU Utilization High** - High CPU utilization - **Disk Pressure** - Disk pressure condition - **Disk Pressure Pod Evictions High** - High pod evictions due to disk pressure - **Disk Read IOPS Utilization High** - High disk read IOPS utilization - **Disk Read Throughput Utilization High** - High disk read throughput utilization - **Disk Total IOPS Utilization High** - High total disk IOPS utilization - **Disk Total Throughput Utilization High** - High total disk throughput utilization - **Disk Write IOPS Utilization High** - High disk write IOPS utilization - **Disk Write Throughput Utilization High** - High disk write throughput utilization - **Filesystem Corruption Problem Detected** - Filesystem corruption problem detected - **Frequent Containerd Restart Detected** - Frequent containerd restart detected - **Frequent Docker Restart Detected** - Frequent Docker restart detected - **Frequent Kubelet Restart Detected** - Frequent kubelet restart detected - **Frequent Unregister Net Device Detected** - Frequent network device unregister detected - **Kernel Deadlock Detected** - Kernel deadlock detected - **Kubelet Problem Detected** - Kubelet problem detected - **Memory Pressure** - Memory pressure condition - **Memory Pressure Pod Evictions High** - High pod evictions due to memory pressure - **Memory Utilization High** - High memory utilization - **Network Unavailable Detected** - Network unavailable detected - **Node Down** - Node is unavailable or unresponsive - **PID Pressure Detected** - PID pressure detected #### Pod - **Evicted Because Ephemeral Local Storage Usage Exceeded** - Pod evicted due to exceeding ephemeral local storage usage - **Evicted Because Node Low On Ephemeral Local Storage** - Pod evicted because node is low on ephemeral local storage - **Evicted Because Node Low On Memory** - Pod evicted because node is low on memory - **Image Pull Errors** - Image pull errors detected - **Not Ready** - Pod is not ready - **Pod Failure** - Pod is failing - **Pod Unschedulable** - Pod is Pending because the scheduler cannot place it on any node - **Unschedulable Insufficient CPU** - Pod cannot be scheduled because no node has enough unreserved CPU for its requests - **Unschedulable Insufficient Memory** - Pod cannot be scheduled because no node has enough unreserved memory for its requests - **Unschedulable Node Affinity Mismatch** - Pod cannot be scheduled because no node matches its node selector or node affinity - **Unschedulable Nodes Cordoned** - Pod cannot be scheduled because the nodes are cordoned (unschedulable) - **Unschedulable Unbound Volume Claim** - Pod cannot be scheduled because a PersistentVolumeClaim it uses is not bound - **Unschedulable Unknown Reason** - Pod cannot be scheduled and the scheduler's reason is not recognized - **Unschedulable Untolerated Taint** - Pod cannot be scheduled because every eligible node has a taint the pod does not tolerate #### Virtual Machine - **Conn Track Table Utilization High** - High connection tracking table utilization - **Container Ephemeral Storage Utilization High** - High container ephemeral storage utilization - **CPU Utilization High** - High CPU utilization - **Disk Read IOPS Utilization High** - High disk read IOPS utilization - **Disk Read Throughput Utilization High** - High disk read throughput utilization - **Disk Total IOPS Utilization High** - High total disk IOPS utilization - **Disk Total Throughput Utilization High** - High total disk throughput utilization - **Disk Write IOPS Utilization High** - High disk write IOPS utilization - **Disk Write Throughput Utilization High** - High disk write throughput utilization - **Memory Utilization High** - High memory utilization - **SNAT Port Utilization High** - High SNAT port utilization ### Storage #### Disk - **Inodes Utilization High** - High inodes utilization - **Read IOPS Utilization High** - High read IOPS utilization - **Read Throughput Utilization High** - High read throughput utilization - **Total IOPS Utilization High** - High total IOPS utilization - **Total Throughput Utilization High** - High total disk throughput utilization - **Utilization High** - High disk utilization - **Write IOPS Utilization High** - High write IOPS utilization - **Write Throughput Utilization High** - High write throughput utilization --- ## Signals A signal is an observable condition on an entity in your system, such as elevated latency or a rising error rate. Signals are the primary inputs to Causely's causal reasoning engine. When a Signal is active, the engine uses it, together with your system topology and other active Signals, to infer a Diagnosis automatically. A signal is a named condition on an [entity](/reference/entity-types/) (service, workload, container, database, node, and others) that is either active or inactive. It becomes active when a specific metric crosses its threshold and holds for a sustained period. Each signal has three parts: the metric or attribute being monitored, the threshold at which it triggers, and a description of what an active state indicates. Signals can be: - **Local**: Observable in the entity where the Diagnosis occurs - **Propagated**: Observable in related entities due to Diagnosis propagation ### How signals activate Signals activate automatically with no configuration required, and every part of that behavior can be tuned when defaults need adjusting. **Configurable thresholds.** Every signal is driven by an underlying metric and a threshold. Thresholds are configurable and have one active source at a time: a system default, a value Causely learns from your traffic, or a manual override. See [Threshold configuration](/configuration/thresholds/). **Controllable activation delay.** A signal does not activate on the first breach. The condition must hold for a sustained period first, which suppresses transient spikes. The bursty activation delay defaults to 5 minutes and is configurable per service. See [Signal delay](/configuration/symptom-delay/). **Activation from existing alerts.** Signals can also be activated by alerts you already run. Causely maps alerts from Prometheus Alertmanager, Datadog, Dynatrace, and Grafana onto the matching signal, so existing alerts feed Causely's causal reasoning as Signals without changing how you alert today. See [Alert ingestion](/configuration/alert-ingestion/). ### From Signals to Diagnosis Signals are used by Causely's causal reasoning engine to infer a Diagnosis. When multiple Signals are present, Causely analyzes their relationships and patterns to identify the most likely explanation. The output is a Diagnosis with its blast radius and supporting evidence, structured for an agent, an automation workflow, or an engineer to act on directly. :::info Signal vs Diagnosis Signals describe what you observe (for example, high latency, errors, timeouts); a Diagnosis describes why it happens (for example, congestion, lock contention, queue saturation). They can sound similar, but they're different: latency is the effect, congestion is the cause, and fixing the cause is what prevents the Signal from returning. ::: --- ## Services (Symptoms) Signals related to services and load balancers. ### AI Model - **Inference Error Rate High** - High inference error rate - **Inference Latency High** - High inference latency ### SLO - **Burn Rate High** - High SLO burn rate - **Error Budget Remaining Low** - Low error budget remaining ### Service - **DB Connections Increase High** - High increase in database connections - **DB Query Duration Increase High** - High increase in database query duration - **Endpoint Request Burst** - Burst of requests on the service's endpoints - **Request Duration High** - High request duration/latency - **Request Error Rate High** - High request error rate - **Request Policy Rejected Rate High** - High rate of requests rejected by policy (HTTP 451) - **Request Rate High** - High request rate - **Request Throttled Rate High** - High throttled request rate - **Request Unauthorized Rate High** - High unauthorized request rate - **Throughput Drop** - Service traffic is lower than expected - **Unavailable** - Service is unavailable (all instances are not ready) ### Service Access - **Request Error Rate High** - High request error rate --- ## Terminology This page contains the terminology used in [Causely](/). The list is not exhaustive, but rather a guide to help you understand the key concepts. Terms are not listed in alphabetical order, but rather grouped by their relatedness. ### Issue, Diagnosis, and Signal Causely's user-facing surfaces, including the UI, notifications, and agent integrations, describe reliability problems using three product terms: **Issue**, **Diagnosis**, and **Signal**. These sit on top of the formal model defined in the rest of this page; they are not renames of it. #### Issue The persistent thread that ties together every [Diagnosis](#diagnosis) of the same underlying problem. An Issue is the one thing you acknowledge, share, and resolve. It stays stable across change, even as the Diagnosis explaining it evolves. #### Diagnosis The current best explanation of the observations at a given moment. A Diagnosis is Causely's runtime inference of a [root cause](#root-cause-rc), what has traditionally been called "the root cause", and it updates as [Signals](#signal) change. #### Signal An observed anomaly a Diagnosis explains, usually a threshold violation on a metric or [SLI](#service-level-indicator-sli), and sometimes an alert, log line, or event. For how Signal maps to the formal model, see the note on [Symptom](#symptom). > An Issue causes anomalies that surface as Signals. Causely infers a Diagnosis, the best explanation of those Signals at that moment. As Signals change, the Diagnosis updates. The Issue stays stable across those changes. If two Diagnoses share even one Signal, they are treated as the same Issue. ### Managed Object An object in the managed environment. ### Managed Domain A scoped collection of [managed objects](#managed-object). ### Causely Domain Manager (CDM) A Causely instance that manages a [managed domain](#managed-domain). ### Event - An observable anomaly in a [managed object](#managed-object). - An _event_ is `TRUE` if the anomaly is observed/present/active. - Examples - HighCPUUtilization - CPU Utilization > 90% - Container OOM Killed - An external notification ### Active/Present Event An [event](#event) that is **`TRUE`**. ### Root Cause (RC) - _Root Cause_ is something that may occur in a [managed object](#managed-object). - _Root Cause_ is inferred based on [observed events](#event). - Examples - Database Congested - Application Malfunction - Container CPU Congested - etc. ### Symptom - An [event](#event) that may be caused by a [root cause](#root-cause-rc). See the [full list of symptoms](/reference/symptoms/). - Examples - Container HighCPUUtilization may be caused by Container Congested - Service HighLatency may be caused by Database Congested - Service HighErrorRate may be caused by Application Malfunction - etc. :::info Signal and the formal model In product and user-facing surfaces these observables are called **Signals**. A Signal is an observed [Event](#event), most often a threshold violation on a metric or SLI and in the model a [Symptom](#symptom), together with ingested observables such as alerts and log lines. The formal terms Event and Symptom are retained here because the causal models and Codebook are defined in terms of them. ::: ### Local Symptom of Root Cause R - A [symptom](#symptom) that is observable in the [managed object](#managed-object) in which the [root cause](#root-cause-rc) $R$ occurs. - Examples - A Container X HighCPUUtilization is a local symptom of Container X Congested - etc. ### Propagated Symptom of Root Cause R - A [symptom](#symptom) that may be caused by [root cause](#root-cause-rc) $R$ observed in a [managed object](#managed-object) related to the [managed object](#managed-object) in which root cause $R$ occurs. - Examples - Service Starvation is a propagated symptom of Database Congested propagating to the services that are accessing the database ### Service Level Indicator (SLI) - A quantitative measure of service reliability that reflects user experience. - Causely monitors three primary SLIs for services: - **Error Rate**: The percentage of successful requests (requests that do not result in errors) - **Latency**: The percentage of requests that complete within an acceptable latency threshold - **Availability**: The percentage of time a service is operational and responding successfully - SLIs are continuously measured from telemetry data (metrics, traces, logs) collected from the [managed domain](#managed-domain). - Examples - A service with 99.5% error rate SLI means 99.5% of requests are successful - A service with 95% latency SLI means 95% of requests complete within the latency threshold - A service with 99.9% availability SLI means the service is operational 99.9% of the time ### Service Level Objective (SLO) - A target value or goal for a [Service Level Indicator](#service-level-indicator-sli). - SLOs define the reliability commitments for services and help prioritize which [root causes](#root-cause-rc) require urgent attention. - When an SLI violates its SLO (or is at risk of violating it), related [root causes](#root-cause-rc) are marked as **Urgent**. - Causely supports SLOs for error rate, latency, and availability, each with configurable targets and burn rate thresholds. See [SLO Configuration](/configuration/slo-configuration) for details on customizing these values. - Examples - An error rate SLO of 99.0% means at least 99% of requests must be successful - A latency SLO of 95.0% means at least 95% of requests must complete within the latency threshold - An availability SLO of 99.0% means the service must be operational at least 99% of the time ### Logs Associated with Root Cause Causely captures logs from the containers that underlie your services to provide rich context for understanding issues. Each container (within its pod) writes logs, including errors and exceptions, to stdout. When Causely detects a service malfunction or identifies a [root cause](#root-cause-rc), it automatically collects and surfaces relevant log lines from the underlying container. These logs are shown in two places: - Under the affected service, when the service is exhibiting abnormal behavior (for example, error spikes, degraded performance). - Alongside a root cause, to highlight the precise errors or stack traces that occurred around the time of failure. This helps validate the issue and dramatically shortens time to understanding and resolution. ### Root Cause Analysis Problem (RCA) Given a set of [Symptoms](#symptom) (manifestations) identify the explanation of why they are present by using knowledge about the world. A Root Cause Analysis Problem $P$ is a 4-Tuple $P =\ < R,\ C,\ S,\ S^+ >$ where - $R = {r_1,\ ...,\ r_n}$ is a finite, non-empty set of [root causes](#root-cause-rc). - $S = {s_1,\ ...,\ s_k}$ is a finite, non-empty set of [symptoms](#symptom). - C a subset of $R\ x\ S$ is a relation with _domain(C) = R_ and _range(C) = S_ called _Causation_. - $S^+$ is a distinguished subset of S said to be _Present_ ### Effects and Causes - For any $r_i\ in\ R$ and $s_j\ in\ S\ in\ P =\ < R,\ C,\ S,\ S^+ >$, - $effects(r_i) = {s_j\ | < r_i\ ,\ s_j > in\ C}$, the set of [symptoms](#symptom) that may be caused by $r_i$ - $causes(s_j) = {r_i\ | < r_i\ ,\ s_j > in\ C}$, the set of [root causes](#root-cause-rc) that may cause $s_j$ - For any subset $R_i\ of\ R$ and subset $S_j\ of\ S$ in $P =\ < R,\ C,\ S,\ S^+ >$, - $effects(R_i) = union\ of\ effects(r_i),\ and\ r_i\ in\ R_i$ - $causes(S_j) = union\ of\ causes(s_j),\ and\ s_j\ in\ S_j$ ### Closure of Root cause R - _Closure_ of [root cause](#root-cause-rc) $R$ is $effect(R)$ - A _Closure of Root Cause $R$_ is a unique signature, a vector of probabilities, that uniquely identifies $R$. ### Spurious and Missing Symptom - For any $r_i\ in\ R$, $s_j\ in\ S$ and any subset $S\ of\ S_k\ in\ P =\ < R,\ C,\ S,\ S^+ >$, - $spurious(r_i\ ,\ S_k) = {s_j\ in\ S_k\ | < r_i\ ,\ s_j > not\ in\ C}$ the set of [symptoms](#symptom) in $S_k\ not\ caused\ by\ r_i$ - $missing(r_i\ ,\ S_k) = {s_j\ not\ in\ S_k\ | < r_i\ ,\ s_j > in\ C}$ - For any subset $R_i\ of\ R$ and subset $S_k\ of\ S\ in\ P =\ < R,\ C,\ S,\ S^+ >$, - $spurious(R_i\ ,\ S_k) = union\ of\ spurious(r_i\ ,\ S_k),\ and\ r_i\ in\ R_i$ - $missing(R_i\ ,\ S_k) = union\ of\ missing(r_i\ ,\ S_k),\ and\ r_i\ in\ R_i$ ### Bayesian Network A Bayesian network is a probabilistic graphical model that represents a set of variables and their conditional dependencies using a directed acyclic graph (DAG), allowing for reasoning under uncertainty by calculating probabilities of events based on known information and relationships between variables; essentially, it's a way to model complex causal relationships between different factors by showing how the probability of one variable changes depending on the states of other related variables. ### Causal Models - [CDM](#causely-domain-manager-cdm) is driven by _Causal Models_, "a knowledge base", of [root causes](#root-cause-rc). - _Causal Models_ capture the _cause and effect_ association between the [root causes](#root-cause-rc) and the [symptoms](#symptom). - _Causal Models_ describe how the [root causes](#root-cause-rc) will propagate across the [managed domain](#managed-domain) and what [symptoms](#symptom) may be caused/observed when each of the [root causes](#root-cause-rc) occur. - CDM is delivered with out-of-the-box built-in _Causal Models_ that capture [root causes](#root-cause-rc) that can occur in cloud native environments. - The _Causal Models_ capture potential [root causes](#root-cause-rc) in a broad range of entities - Such as, applications, databases, caches, messaging, load balancers, DNS compute, storage, etc. - The _Causal Models_ are completely independent of any specific [managed domain](#managed-domain) and are applicable to any cloud native application environment. This enables Causely to automatically pinpoint [root causes](#root-cause-rc) out-of-the-box as soon as it is deployed in an environment. - Causal models are engineered knowledge of how failures propagate. Causely calibrates them to your environment; no LLM is trained on your data. - Applying the _Causal Models_ to the discovered [Topology](#topology) produces the [Causality Graph](#causality-graph-cg). ### Topology - _Topology_ is a directed graph where the nodes are entities in the [managed domain](#managed-domain) and the directed edges represents relationship between the entities. - [CDM](#causely-domain-manager-cdm) automatically discovers the entities and their relationships in the [managed domain](#managed-domain). - _Entities_ are applications, services, databases, caches, messaging, load balancers, compute, storage, etc., - Relationships are Connectivity, Layering, and Composition - _Connectivity_: for each _entity_, the _entities_ it is connected to and the _entities_ it is communicating with. - _Layering_: for each _entity_, the _entities_ it is layered over or underlying. - _Composition_: for each _entity_, the _entities_ it is composed of or part of. - [CDM](#causely-domain-manager-cdm) automatically stitches all of these relationships together to generate a dependency map, or _Topology_ Graph, of the entire _managed domain_. - The _Topology_ Graph is continually updated in real-time to reflect the current state of the _managed domain_. ### Causality Graph (CG) - The _Causality Graph_ is the [Causal Models](#causal-models) applied to the discovered [Topology](#topology), kept live by continuous updates from telemetry. Causely's product surfaces call it the live causality graph. - _Causality Graph_ is a Directed Acyclic Graph (DAG) where the nodes are [root causes](#root-cause-rc) and [symptoms](#symptom), and the edges represent causality, i.e, $R\ \rightarrow \ S$ means R may cause S. - The edges are labeled with probability $P,\ 0 <$ _managed domain_. - The _Causality Graph_ represents all the possible $P < 1$, representing the likelihood R may cause S. - [CDM](#causely-domain-manager-cdm) automatically generates the _Causality Graph_ by applying the [Causal Models](#causal-models) to the [Topology](#topology) Graph. - By applying the [Topology](#topology) of the [managed domain](#managed-domain) to the generic [Causal Models](#causal-models), [CDM](#causely-domain-manager-cdm) generates the causal knowledge that is specific to the [managed domain](#managed-domain). - The _Causality Graph_ represents all the possible [root causes](#root-cause-rc) in the [managed domain](#managed-domain), all the [symptoms](#symptom) that may be observed, and the _cause and effect_ relationships between them. - In a [managed domain](#managed-domain) of a few thousands entities, the _Causality Graph_ will incorporate the knowledge of tens of thousands of potential [root causes](#root-cause-rc) and hundreds of thousands [symptoms](#symptom) - which is well beyond human scale. - The _Causality Graph_ is automatically updated every time the [topology](#topology) changes. - The edge probabilities are learned based on the data in the [managed domain](#managed-domain). ### Codebook - A mapping of all potential [root causes](#root-cause-rc) ${r_1,\ ...,\ r_n}$ to the [symptoms](#symptom) ${s_1,\ ...,\ s_m}$ they may cause. - The Codebook is a causality table where - The columns $r_1,\ ...,\ r_n$ represent all the potential [root causes](#root-cause-rc) - The rows $s_1,\ ...,\ s_m$ represent all the potential [symptoms](#symptom) and - A cell $(r_i\ ,\ s_j)$ represents the probability [root causes](#root-cause-rc) $r_i$ may cause [symptom](#symptom) $s_j$ , that is, the likelihood [symptom](#symptom) $s_j$ will be observed/present when [root causes](#root-cause-rc) $r_i$ occurs. - Each [root causes](#root-cause-rc) in the Codebook has a unique signature, a vector of m probabilities, that uniquely identifies the [root causes](#root-cause-rc). - Using the _Codebook_, [CDM](#causely-domain-manager-cdm) quickly searches and pinpoints the [root causes](#root-cause-rc) based on the observed [symptoms](#symptom).