# Causely documentation: in-action > Full content of the "in-action" section of the Causely documentation. This file contains all documentation content in a single document following the llmstxt.org standard. ## Automate Remediation ## Automate Remediation for Resource Contention Causely allows you to automatically remediate **resource contention** issues directly from the UI, helping you restore performance faster, reduce time to resolve, and keep services within SLOs. When Causely identifies a deterministic Resource Contention Diagnosis, you can trigger automated remediation or apply a guided fix with one click. ### Supported Root Causes Causely supports automated remediation for the following resource-related root causes: #### [CPU Congested](/reference/root-causes/infrastructure/#cpu-congested) Automatically adjust CPU limits when services are experiencing CPU saturation. #### [Frequent Memory Failure](/reference/root-causes/infrastructure/#frequent-memory-failure) Resolve persistent out-of-memory issues caused by memory leaks or inefficient usage. #### [Memory Failure](/reference/root-causes/infrastructure/#memory-failure) Increase memory allocations to resolve out-of-memory issues. #### [Ephemeral Storage Noisy Neighbor](/reference/root-causes/infrastructure/#ephemeral-storage-noisy-neighbor) Isolate and manage containers that excessively consume ephemeral storage, impacting node stability. #### [Memory Noisy Neighbor](/reference/root-causes/infrastructure/#memory-noisy-neighbor) Isolate and manage containers that excessively consume memory, impacting node stability. #### [Congested Services](/reference/root-causes/services/#congested) Scale service resources to handle increased load. ### What Causely Changes Automatically When remediation is executed for a supported Resource Contention Diagnosis, Causely applies a deterministic scaling action based on the type of bottleneck identified: 1. **Vertical Scaling (+50%)**: Increases the affected container’s CPU or memory **requests and limits by 50%**. 2. **Horizontal Scaling (+1 Replica)**: Adds **one additional replica** to the deployment to immediately increase capacity. These adjustments are purposefully conservative and are only applied when Causely’s causal reasoning confirms that scaling, rather than a correlated Signal, is the correct fix. ### Enabling Automated Remediation (Executor Required) Automated actions require the **executor** to be enabled on the mediator running in the cluster where you want remediation performed. To enable the executor, update your mediator’s `causely-values.yaml`: ```yaml executor: enabled: true ``` Once updated, redeploy or upgrade your mediator so it loads the new configuration. See [using custom values file](/installation/customize/?install-method=helm#use-a-custom-values-file) for details on applying updated values. ### Using the Remediate Now Interface In the UI, supported Diagnoses include a **Remediate now** option that provides: - An acknowledgment step showing the impacted deployment - Auditable action history tied to the entity for which value were updated If you prefer to apply the change manually, the [Remediation section](/in-action/suggested-remediation/) for the Diagnosis includes YAML examples you can use. ### Aligning Configuration with MCP Server These remediation updates are applied at runtime, and the MCP Server provides a way to commit these configuration adjustments into your codebase for long‑term consistency. If you want to standardize or persist updated sizing, you can manage configuration through the [MCP Server](/agent-integration/mcp-server/). --- ## Feature Demos This page contains links to Causely feature demos designed to help you understand what you can do and what to expect with Causely. Watch the Full Playlist export const videos = [ { id: 'GiXq71HEGwE', title: 'Causely Feature Demo: Reliability Delta', }, { id: 'nN4Iy5BuC3c', title: 'Solving Slow Database Queries with Causely and its MCP Server', }, { id: 'L-nWJr4tZ7U', title: 'Causely’s MCP Server Brings Reliability into Your IDE | Helm Chart Example', }, { id: 'sk_KmMOF1lE', title: 'Causely Feature Demo: From Root Cause to Business Impact with Causely and ClickStack by ClickHouse', }, { id: 'p07c2gy3baM', title: 'Causely Feature Demo: Accelerate Incident Response with Causely + incident.io', }, { id: 'tT0Ju5vO97w', title: 'Causely Feature Demo: Address External API Slowdowns', }, { id: 'xah1-eSqO4A', title: 'Causely Feature Demo: Solve the Root Cause of Message Queue Lag', }, { id: 'hvJDWHkxieg', title: 'Causely Feature Demo: Pinpointing the Code Change Causing Performance Issues', }, { id: 'D6Ps1VoGHvw', title: 'Causely Feature Demo: Unlock Root Cause Analysis in Grafana', }, ]; ### Latest demo ### Last videos {videos.map((v) => ( ))} --- ## Causely In Action This section shows how the Causely system surfaces what matters, explains why issues happen, and helps you move from detection to resolution. If you are new, the recommended path is: start with the [Welcome Page](/in-action/welcome-page), skim [Topology](/in-action/topology) to learn how entities are organized, then dive into [Issues](/in-action/root-causes) to investigate real problems. :::tip Causely is built to seamlessly integrate with your existing SRE tools and workflows: it replaces the tools you no longer need for reactive incident response, and plugs into your workflows to enable prediction and prevention. Therefore the UI is only one way to interact with Causely: we recommend that you use [agent](/agent-integration) or [workflow](/workflows) integrations, to have Causely available where you already work. ::: #### Welcome Page Start your day with a concise overview. The page highlights whether there is an urgent Issue that needs attention or confirms that there are no urgent items (with a link to review non‑urgent ones). It provides a set of insights from the last 24 hours: the number of [Diagnoses](/reference/root-causes/) and [Signals](/reference/symptoms/) observed (and how many are active), plus the Noise Reduction achieved. See the full page at [Welcome Page](/in-action/welcome-page). #### Topology Browse entities discovered by the engine: services, workloads, containers, HTTP paths, background operations, data stores, and more. Filter by scope, severity, and type, and use search to quickly find what you need. Inspect any entity through focused tabs including Summary, [Diagnoses](/reference/root-causes/), [Signals](/reference/symptoms/), Metrics, Service map, Infrastructure stack, Dataflow map, and Labels. Learn more at [Topology](/in-action/topology). #### Issues See the list of Issues and their inferred [Diagnoses](/reference/root-causes/) that explain observed [Signals](/reference/symptoms/) across your environment. Use urgency and state filters to focus, then open an Issue to review the Summary and the Causality graph with evidence, impact, and remediation guidance. Details at [Issues](/in-action/root-causes). #### Suggested Remediation Every Diagnosis includes remediation guidance. When Causely has relevant log evidence, it generates more specific steps for the affected entity, often with ready-to-apply configuration. Learn more at [Suggested Remediation](/in-action/suggested-remediation). #### Automate Remediation Automate remediation for resource contention issues such as CPU congestion, memory failures, and noisy-neighbor conditions. Use it to quickly restore service performance with safe scaling actions directly from the Causely UI. Learn more at [Automate Remediation](/in-action/automate-remediation). #### Reliability Delta Compare two snapshots of your environment to see whether reliability has improved or regressed. This turns subjective judgment into clear, objective reliability signals. Use it to validate release candidates, analyze load tests, and detect reliability drift over time. Learn more at [Reliability Delta](/in-action/reliability-delta). #### Feature Demos Short demonstrations that highlight key capabilities and investigative workflows. Useful for onboarding, sharing with teammates, or refreshing best practices. Explore at [Feature Demos](/in-action/feature-demos). --- ## Reliability Delta The **Reliability Delta** capability allows you to compare two snapshots of your environment to determine whether reliability has improved, regressed, or meaningfully changed. This is especially powerful when evaluating **release candidates**, validating **load test** results, or confirming the **stability** of ongoing changes in production. A **snapshot** captures the state and behavior of your system, including its services, dependencies, and underlying infrastructure, over a defined time window (up to two continuous hours). By comparing any two snapshots, Causely surfaces changes in reliability, service behavior, Diagnoses, and resource utilization, enabling you to definitively assess whether a new version or configuration is safer, riskier, or equivalent. ### When to Use Reliability Delta #### Release validation Compare a release candidate against a stable version to detect regressions in latency, error rate, throughput, or resource usage before deployment. #### Load testing and performance analysis Capture snapshots during load tests and compare them to production baselines to identify bottlenecks, capacity issues, or emerging Diagnoses. #### Continuous environment drift detection Review snapshots taken at different points in time to understand whether reliability is trending up or down, even outside of formal releases. ### How Snapshots Work A **snapshot** records: - Service behavior (latency, request rate, error rate, throughput) - Dependency and service counts - Detected Diagnoses (urgent and non-urgent) - Infrastructure behavior and resource usage (CPU, memory, network I/O) - Version, schema, and metadata you provide - Duration and time window of captured activity Snapshots can be created in two ways: 1. **Last 2 hours**: automatically captures data from the most recent 120 minutes 2. **Custom time window**: select a start date and time for a snapshot (up to 2 hours). The start time must be within the last 6 hours. Snapshots appear in the **Snapshots** panel and can be selected for comparison in the **Comparisons** panel. ### Comparing Snapshots The **Comparisons** view enables you to evaluate changes across two snapshots. You can also apply an optional scope to focus the comparison on a specific set of services or infrastructure. - **Baseline Snapshot**: the version or time period you trust - **Candidate Snapshot**: the release, change, or window you want to evaluate Causely computes the reliability delta across: #### Diagnoses - New Diagnoses introduced in the candidate window - Cleared Diagnoses no longer present - Classification by urgent vs. non-urgent #### Service Behavior For each metric, Causely shows: - Before value - After value - Absolute change - Percent change Metrics include: - Service count - Total requests - Request rate - Average request duration - Average error rate #### Resource Utilization - Entity counts - CPU utilization (avg & max) - Memory utilization (avg & max) - Network throughput (receive & transmit) ### Interpreting Results #### 🟢 Proceed with release > No concerning changes detected. The comparison shows the candidate snapshot introduces **no new urgent Diagnoses**, indicating the version is safe to proceed with. #### 🔴 Test snapshot rejected > New urgent Diagnoses detected that require investigation before deploying. This indicates the candidate snapshot **introduced one or more urgent Diagnoses**, which may reflect: - Newly degraded or failing services - Significant regressions in error rate, latency, or throughput - Resource or dependency behavior signaling instability ### Using snapshots through MCP You can also create and compare snapshots through the [Causely MCP server](/agent-integration/mcp-server/#reliability--deployment), for example from a CI/CD pipeline or an agent. The MCP server has four snapshot tools: `create_snapshot`, `get_snapshots`, `compare_snapshots`, and `delete_snapshot`. - **Create:** A snapshot captures topology, active diagnoses, and resource and service metrics over a window of up to 2 hours. Causely builds it asynchronously. Its status is PENDING, then COMPLETE or FAILED. - **Compare:** Only COMPLETE snapshots can be compared. `compare_snapshots` takes the baseline snapshot first. - **Roles:** Creating and deleting snapshots requires the Developer or Administrator role. `compare_snapshots` returns: - An assessment of ACCEPTED (no regression detected) or REJECTED - Entity changes - New and cleared diagnoses - Resource metrics (CPU, memory) - Service metrics (request rate, error rate, duration) The assessment uses the same criteria as the portal. ACCEPTED matches "Proceed with release" and REJECTED matches "Test snapshot rejected". See [Interpreting Results](#interpreting-results). ### UI Walkthrough #### Snapshots Panel View and manage stored snapshots. Selecting a snapshot displays its metadata, duration, and captured behavior. #### Comparisons Panel Choose any two snapshots to view detailed reliability deltas across Diagnoses, services, and resources. #### Snapshot Creation Dialog - Add a **name**, **version**, and **description** - Choose **last 2 hours** or a **custom time window** - Click **Create** to capture the snapshot #### Comparison View Displays: - **Snapshot Summary**: time window, metadata, version info - **Diagnoses Summary**: new vs. cleared causes - **Service Summary**: metric-level comparisons - **Resource Summary**: infrastructure-level deltas ### Example Workflows #### Validating a release candidate 1. Capture a snapshot of the last stable version 2. Deploy the release candidate 3. Capture a new snapshot 4. Compare the two snapshots 5. Validate that no new urgent Diagnoses or degradations are introduced #### Load test regression analysis 1. Create a snapshot of baseline load test for a stable version 2. Run the load test on a new version and capture a snapshot during it 3. Compare the snapshots to identify scalability issues or bottlenecks #### Reliability drift detection 1. Periodically capture snapshots (daily/weekly) 2. Compare snapshots to detect slow degradation or improvements 3. Use deltas to preemptively address emerging risks ### Why Reliability Delta Matters Engineers must ship quickly **without compromising reliability**. Reliability Delta provides a deterministic, data-driven method for evaluating the impact of change. It delivers: - A clear comparison of behavioral changes between versions - Root-cause-aware analysis powered by Causely’s causal reasoning - A consistent framework for release evaluation and change management - Execution confidence even under rapid change and high-visibility releases It transforms subjective judgment into objective reliability signals. ### Feature Demo --- ## Issues The Issues view presents the outcomes of the causal reasoning engine so you see what explains the [Signals](/reference/symptoms/) and problems surfaced in your environment. Instead of receiving an alert per Signal, you get a single Issues and Diagnosis that represents the underlying cause, dramatically reducing noise and accelerating response. For background on how Diagnoses are inferred, see [How Causely Works](/getting-started/how-causely-works). ### Issues view The Issues view shows the list of Issues and inferred Diagnoses, that explain the Signals observed in your environment. The list is filtered by urgency and state, and can be further filtered by scope, service, and workload. #### Urgency **Urgent Issues** are the ones that are currently impacting services and SLOs. They are marked red. **Non-urgent Issues** are the ones that are not currently impacting services and SLOs but indicate elevated risk. Both types are valuable: **urgent items** drive incident response; non‑urgent items drive prevention and continuous improvement. #### States - **Active**: The Issue's Signals are currently present, or cleared within the last 24 hours. - **Historical**: The Issue was active in the past and has since cleared. Useful for learning and trend analysis. - **Hidden**: The Issue exists but is suppressed from the default list (for example via filters or muted scopes). You can reveal hidden items when needed. :::info A Diagnosis for the underlying problem of an Issue is no longer active once all of its Signals pass their deactivation window, see [Signal Delay](/configuration/symptom-delay/). The Issue remains Active for a minimum of 24 hours after all Signals deactivate, and longer if a Signal recurs within that window. ::: #### Filter the list Use filters to focus on what matters most: - **Scope**: Limit Diagnoses to a specific scope (for example an environment or domain). - **Service**: Focus on a particular service. - **Workload**: Narrow down to a deployment, job, or workload of interest. We will add a screenshot of the filtering controls here. ### Inspect a single Diagnosis Opening a Issue shows a few with two tabs: - **Summary**: A clear explanation including description, current and potential impact, blast radius, [recommended remediation steps](/in-action/suggested-remediation/), and the evidence supporting the inference. - **Causality graph**: A visual explanation of how the system arrived at the Diagnosis, showing the relationships between the observed Signals and the inferred Diagnosis. - **Timeline**: A visualization of all Diagnoses, Signals, events, and SLO breaches associated with the Issue, from first detection to resolution. #### Description A high level description of the current Diagnosis is shown in the summary tab. #### Causality The causality view shows the deterministic analysis that identifies the cause of the observed Signals. It traces how the Diagnosis explains each Signal across services, workloads, and SLOs, so you can see exactly how Causely arrived at its conclusion. #### Impact and Blast Radius A list of services degraded by the Issue is shown in the summary tab, and a "blast radius" diagram is shown to help visualize the scope of the impact. #### Remediation A list of recommended remediations to fix the Issue is shown in the summary tab. #### Evidence A list of evidence supporting the inference is shown in the summary tab. This includes observed [Signals](/reference/symptoms/), exceptions, logs, and events. --- ## Suggested Remediation Suggested Remediation gives you steps to fix an Issue. It appears in the **Suggested Remediation** section of the Summary tab for an [Issue](/in-action/root-causes/) in the Causely portal. ### How it works Suggested Remediation has two tiers. 1. **Diagnosis-type guidance.** Every Diagnosis includes remediation guidance based on its diagnosis type. Each type has its own remediation template. 2. **Evidence-grounded remediation.** When Causely has log evidence relevant to the Diagnosis, it generates a more specific remediation. Relevant evidence includes error messages, connection failures, and configuration values. For the second tier, Causely passes the following to an LLM: - The Diagnosis - Its supporting evidence, including metrics and log lines - The diagnosis-type guidance The LLM returns steps tailored to the affected entity. These often include ready-to-apply configuration, such as a corrected Kubernetes manifest. :::note The Diagnosis itself is deterministic. It is produced by Causely's causal reasoning engine. Only the remediation text is LLM-generated. ::: ### Requirements Diagnosis-type guidance is available for every Diagnosis. Generated remediation depends on log evidence being available to Causely. To make log evidence available, see: - [Telemetry Sources: Logs](/telemetry-sources/#logs) - [Enhanced Log Retrieval for Kubernetes](/telemetry-sources/kubernetes/#enhanced-log-retrieval), including the Elasticsearch and Grafana Loki integrations ### Example :::info This is an illustrative example of how log evidence leads to a specific remediation. The output for your Issues depends on your Diagnosis, evidence, and environment. ::: **Diagnosis:** Application Error Spike on Service `inventory-sync` **Summary:** The synchronization logic attempts to update the immutable `_id` field, causing database write exceptions. **Key evidence:** - [Log | Error] Database write exception on immutable field | Count: 2,814 - [Log | Error] Sync update failed on inventory record | Count: 537 **Suggested remediation:** Exclude `_id` from update operations Modify the synchronization logic in the `inventory-sync` service so update statements never target the immutable `_id` field. Use `_id` only as a query filter, and strip it from the `$set` payload. ```go // Exclude _id from the update; use it only as the filter update := bson.M{ "$set": bson.M{ "sku": item.SKU, "quantity": item.Quantity, // _id is intentionally not included }, } filter := bson.M{"_id": item.ID} _, err := collection.UpdateOne(ctx, filter, update) ``` Without the log evidence, guidance for this diagnosis type would be generic ("investigate application errors"). With it, the remediation names the failing operation and the exact code change. ### Review before acting Remediations are LLM-generated from runtime telemetry and grounded in evidence. Verify them before you act. Check the suggested change against the supporting evidence on the Issue. Then apply it through your normal review and change process. ### Use with agents For more specific steps, pass the remediation and its evidence to an agent that has access to your configurations, code, or runbooks. The [MCP Server](/agent-integration/mcp-server/) lets agents retrieve Diagnoses and their evidence from Causely. ### Related - [Issues](/in-action/root-causes/): where Diagnoses, evidence, and remediation appear. - [Automate Remediation](/in-action/automate-remediation/): executes fixes for supported resource-contention Diagnoses with one click. --- ## Topology The Topology view presents the entities that the causal reasoning engine has discovered across your environment, ranging from services and workloads to containers, HTTP paths, background operations, data stores, and more. It provides an interactive way to browse, filter, and drill into any entity to understand its health, relationships, and impact. For background on how entities are modeled and connected, see [How Causely Works](/getting-started/how-causely-works). ### Topology view The Topology view shows the catalog of detected entities. You can navigate by groupings (for example services or infrastructure), select an entity to open its details, and pivot to related entities as needed. #### Filter and search Use filters and search to focus on what matters most: - **Scope**: Limit the visible entities to a specific scope (for example an environment or domain). - **Severity**: Show entities with a particular severity state first, helping you triage quickly. - **Type**: Narrow the list to a type such as service, container, HTTP path, background operation, data store, or queue. - **Search**: Find entities by name, label, or identifier. Search is available across scopes and types when needed. ### Inspect a single entity Opening an entity shows a details panel organized in multiple tabs: #### Summary The Summary tab provides a concise health overview for the selected entity. It highlights SLO status and recent changes, key metrics at a glance, and the most relevant related entities (upstream, downstream, and infrastructure neighbors). Recent logs and events are surfaced inline to provide quick evidence you can drill into. #### Diagnoses This tab lists [Diagnoses](/reference/root-causes/) currently affecting this entity and those that are plausible given observed conditions. Each Diagnosis includes a title and severity. Click on a Diagnosis to see which Signals it explains for this entity. This information is available even if the Diagnosis is not currently active. This is useful to understand the potential impact of the Diagnosis and to validate the impact of the Diagnosis when it is active. #### Signals See [Signals](/reference/symptoms/) detected on the entity or propagated from dependencies. Entries show title and severity. Click on a Signal to see the Diagnoses that could lead to this Signal. This information is available even if the Signal is not currently active. This is useful to understand the potential impact of the Signal and to validate the impact of the Signal when it is active. #### Service map Visualizes the entity’s service neighborhood, including inbound dependencies and downstream consumers. Edges indicate direction and relative traffic; nodes reflect health at the selected moment. #### Infrastructure stack Shows where the entity runs within the stack, for example cluster, node, workload, container, and process. It surfaces capacity and saturation indicators (CPU, memory, disk, and network) and relevant runtime metadata to understand placement and potential resource contention. #### Dataflow map Displays upstream and downstream data dependencies such as topics, queues, databases, and batch pipelines. Use it to trace how data moves through the system, identify potential bottlenecks or back‑pressure, and assess which consumers are at risk when this entity degrades. #### Labels Lists labels and metadata attached to the entity (team, scope, environment, version, and custom tags). Labels power search and filtering, drive policy and scoping, and help organize entities. You can copy label values and use them to filter the Topology view or jump to related entities. --- ## Welcome Page The Welcome page gives you a concise, actionable overview powered by the Causely engine: what requires attention right now, how to ask targeted questions, and how your environment has behaved recently. ### Welcome page #### Urgent Issue status At the top, the page highlights whether there is an urgent Issue that needs attention. When one exists, you see a compact card showing the Issue title, the affected entity, when it was last observed, the number of linked Signals, and how many services are degraded. Selecting the card opens the full details so you can review evidence and remediation steps. If there are no urgent items, the page clearly states that everything is stable and offers a link to review non‑urgent Issues. Non‑urgent items are useful for prevention and continuous improvement. #### Causely Insights Insights summarize recent activity to help you spot trends without leaving the page: - [Diagnoses](/reference/root-causes/): The number observed over the last 24 hours, with an indication of how many are currently active versus cleared. - [Signals](/reference/symptoms/): The number observed over the last 24 hours, including how many remain active. - Noise Reduction: The percentage reduction in alert noise provided by the system compared to a baseline, illustrating how much distraction has been eliminated. Each card links to a deeper view, so you can pivot from the high‑level indicators into detailed lists and timelines.