# Causely documentation: workflows

> Full content of the "workflows" section of the Causely documentation.

This file contains all documentation content in a single document following the llmstxt.org standard.

## Trigger Background Agents

Causely pushes structured diagnoses to your agents and automation through webhooks as Issues are detected, not only through [MCP](/agent-integration/mcp-server) queries. A background agent doesn't need to poll or wait for someone to ask a question. It starts working as soon as Causely identifies the problem.

This page describes how to set up a [Generic Webhook](/workflows/generic-webhook) notification that targets the endpoint a background agent is listening on.

### How it works

1. Causely detects an Issue and determines its underlying diagnosis.
2. The Causely Mediator sends a Generic Webhook notification to your agent's endpoint. The request body is the [Causely notification payload](/workflows/notification-payload), which includes the Issue, the affected entity, its severity, and the diagnosis.
3. The agent receives the notification and starts an investigation.
4. The agent calls the [Causely MCP server](/agent-integration/mcp-server) to retrieve the causal chain, logs, topology, and other evidence, then resolves the Issue or recommends a remediation.

The webhook tells the agent that something happened and what Causely concluded. MCP gives the agent the context to verify the diagnosis and act on it. This keeps the webhook payload small and lets the agent pull only the evidence it needs.

### Prerequisites

- A background agent with an HTTP endpoint that accepts `POST` requests with a JSON body, reachable from the cluster where the Causely Mediator runs.
- The agent is connected to the Causely MCP server. See [Causely MCP Server](/agent-integration/mcp-server) for connection details, and [Advanced Authentication](/agent-integration/mcp-server/advanced-auth) for credentials that don't require a browser, which is what a non-interactive agent needs.
- Permission to manage notifications in Causely.

### Set up the notification

1. Go to **Settings → Notifications** and click **Create**.
2. Complete the configuration:

   | Field | Value |
   |-------|--------|
   | **Name** | A descriptive name, for example `background-agent` |
   | **Type** | Generic |
   | **URL** | The endpoint your agent listens on, for example `https://agent.example.com/trigger` |
   | **Token** | The full `Authorization` header value your agent expects, for example `Bearer <token>` |

3. Choose the cluster or clusters that handle processing and delivery.
4. Add [filters](#control-what-wakes-the-agent) so the agent is only triggered by the Issues you want it to handle.
5. Click **Create**.

The token is sent verbatim as the `Authorization` header, so include the scheme keyword yourself. See [Authentication](/workflows/generic-webhook#authentication) for details. To define the destination with a Kubernetes secret or environment variables instead of the UI, see [Mediator-based configuration](/workflows/generic-webhook#mediator-based-configuration-advanced).

### Control what wakes the agent

Every notification that reaches your agent can start an investigation, and every investigation consumes agent time and LLM tokens. Filter on the Causely side so the agent only receives Issues it should act on.

For example, forward only High and Critical Issues:

```yaml
notif_config_filters: '[{"field":"severity","operator":"in","value":["High","Critical"]}]'
notif_config_filters_enabled: "true"
```

You can also filter by cluster, namespace, entity type, or whether the Issue impacts an SLO. See [Filters](/workflows/generic-webhook#filters) for the full list of fields and operators.

:::tip Start in a read-only mode
When you first connect an agent, have it investigate and report without making changes. Switch to automated remediation once you have confirmed that the diagnoses and the agent's conclusions match your expectations.
:::

### What the agent receives

The request is an HTTP `POST` with `Content-Type: application/json`. The fields most useful to an agent are:

- `type`: `ProblemDetected` when the Issue is detected, or `ProblemCleared` when it resolves.
- `objectId`: The identifier of the Issue, which the agent can pass to MCP tools to retrieve its full details.
- `name` and `severity`: What Causely found and how serious it is.
- `entity`: The affected entity, with its `id`, `name`, and `type`.
- `labels`: Cluster, namespace, and workload context.
- `description`: A summary of the diagnosis.

See [Notification Payload](/workflows/notification-payload) for a complete example and field reference.

### Write the agent's handler

Your agent's handler should:

1. **Authenticate the request.** Reject calls that don't carry the expected `Authorization` header.
2. **Acknowledge quickly.** Return a `2xx` response immediately and run the investigation asynchronously.
3. **Ignore `ProblemCleared` events,** or use them to close out an investigation in progress.
4. **De-duplicate on `objectId`,** so a repeated notification for the same Issue doesn't start a second investigation.
5. **Investigate through MCP.** Use the Causely MCP tools to retrieve the Issue, its root cause, supporting evidence, and affected services before deciding what to do. Treat the diagnosis in the payload as the starting point and verify it against live evidence.

### Examples

The [causely-oss/background-agents](https://github.com/causely-oss/background-agents) repository contains runnable examples of agents that are triggered by Causely and use the MCP server to investigate:

- [causely-background-agent](https://github.com/causely-oss/background-agents/tree/main/causely-background-agent): A standalone reference agent that receives Causely's webhook push, investigates with the Causely MCP tools, and either opens a pull request with a fix or recommends a remediation. It starts in an observe mode that never makes changes.
- [open-inspect](https://github.com/causely-oss/background-agents/tree/main/open-inspect): Connects Open-Inspect to the Causely MCP server so its background investigations cite root cause and log evidence in the pull requests they open.
- [aws-devops-agent](https://github.com/causely-oss/background-agents/tree/main/aws-devops-agent): Wakes the AWS DevOps Agent with a Causely notification through a signed-webhook relay. The agent calls back into the Causely MCP server for the causal chain.
- [claude-managed-agents](https://github.com/causely-oss/background-agents/tree/main/claude-managed-agents): A Claude Managed Agent investigating a live Kubernetes cluster through remote MCP servers.

### Related pages

- [Generic Webhook](/workflows/generic-webhook)
- [Notification Payload](/workflows/notification-payload)
- [Causely MCP Server](/agent-integration/mcp-server)
- [Managed Notifications](/workflows/ui-notifications)

### Learn more

- [How Open-Inspect and Causely close the loop on autonomous remediation](https://www.causely.ai/blog/how-open-inspect-and-causely-close-the-loop-on-autonomous-remediation)
- [What should trigger an autonomous agent in production](https://www.causely.ai/blog/what-should-trigger-an-autonomous-agent-in-production)

---

## Custom Workflows

To send Causely notifications to a custom destination, you route them through [CauselyBot](https://github.com/causely-oss/causelybot), an open source webhook service that receives the Causely notification payload and forwards it to the endpoints you configure. For arbitrary HTTP endpoints (n8n, Linear, custom services), use CauselyBot's `generic` hook type.

:::note Do you need full control?
Use CauselyBot when you want **full control** over how the webhook is processed and the exact payload sent to your target system.

If you only need to forward Causely's standard payload to an HTTP endpoint as-is, use the simpler [Generic Webhook](/workflows/generic-webhook) option instead.
:::

Routing and rules for Managed Notifications are configured in the Causely UI:  
[Set up notification routing in the UI](/workflows/ui-notifications)

### Setup

#### Step 1: Enable notifications in values.yaml

In your `causely-values.yaml`, enable mediator notifications:

```yaml
mediator:
  notifications:
    enabled: true
```

Apply the change:

```bash
helm upgrade --install causely --create-namespace \
  oci://us-docker.pkg.dev/public-causely/public/causely \
  --version <version> --namespace=causely --values ./causely-values.yaml
```

#### Step 2: Deploy CauselyBot

Create a `causelybot-values.yaml` with your webhook configuration. For a generic HTTP endpoint:

```yaml
auth:
  token: "<YourCauselyBotToken>" # Required - define your token here and then use in the CauselyBot secret creation in step 3.

webhooks:
  - name: "my-generic-webhook"    # Provide a friendly name for the webhook
    hook_type: "generic"
    url: "https://<YourWebHookEndpoint>"
```

See the [CauselyBot documentation](https://github.com/causely-oss/causelybot) for all supported hook types and options.

Install CauselyBot via Helm:

```bash
helm upgrade --install causelybot ./causelybot/helm/causelybot \
  --namespace causelybot --values causelybot-values.yaml
```

#### Step 3: Create the notification Secret

Create a Kubernetes Secret in the `causely` namespace that points Causely at CauselyBot (use `stringData` so values are plain text; if you use `data`, values must be base64-encoded):

```yaml
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: causelybot-notification-config
  namespace: causely
  labels:
    causely.ai/notif-config: CauselyBot   #Required
stringData:
  notif_config_name: causelybot-webhook
  notif_config_global: "false"
  notif_config_type: CauselyBot
  notif_config_url: https://<CauselyBotURL>/webhook  # or if in-cluster: http://causelybot.causelybot.svc.cluster.local.:5000/webhook 
  notif_config_token: "<YourCauselyBotToken>"
  notif_config_filters: "[]"
  notif_config_filters_enabled: "true"
```

:::note
`notif_config_token` must match the `auth.token` you set in CauselyBot's configuration. Causely sends it as `Authorization: Bearer <token>` so CauselyBot can authenticate the request.
:::

### Filtering notifications (notif_config_filters)

`notif_config_filters` is a JSON array that controls which notifications are forwarded to CauselyBot. When `notif_config_filters_enabled` is `"false"` or the array is empty (`"[]"`), all notifications pass through.

Multiple filters are ANDed: a notification must match **all** filters to be forwarded.

#### Filterable fields

| Field | Description |
|---|---|
| `severity` | `"Low"`, `"Medium"`, `"High"`, or `"Critical"` |
| `name` | Problem name (for example, `"Malfunction"`, `"CPUCongested"`) |
| `entity.type` | Entity type (for example, `"KubernetesService"`, `"ApplicationInstance"`) |
| `labels.k8s.cluster.name` | Kubernetes cluster name |
| `labels.k8s.namespace.name` | Kubernetes namespace name |
| `impactsSLO` | `true` if the notification includes impacted SLOs |

Supported operators: `equals`, `not_equals`, `in`, `not_in`

#### Examples

**Forward only High and Critical severity notifications:**

```yaml
notif_config_filters: '[{"field":"severity","operator":"in","value":["High","Critical"]}]'
notif_config_filters_enabled: "true"
```

**Forward notifications from a specific cluster only:**

```yaml
notif_config_filters: '[{"field":"labels.k8s.cluster.name","operator":"equals","value":"prod"}]'
notif_config_filters_enabled: "true"
```

**Forward only SLO-impacting issues in a specific namespace:**

```yaml
notif_config_filters: '[{"field":"impactsSLO","operator":"equals","value":true},{"field":"labels.k8s.namespace.name","operator":"equals","value":"production"}]'
notif_config_filters_enabled: "true"
```

### Example: GitHub via CauselyBot

1. Deploy [CauselyBot](https://github.com/causely-oss/causelybot) and configure a webhook with `hook_type: github`, `url: "owner/repo"`, and a GitHub token (repo and issues scope). Optionally set `assignee` in CauselyBot's config.
2. In the Secret above, set `notif_config_url` to CauselyBot's webhook URL and `notif_config_token` to CauselyBot's auth token.
3. Causely sends notifications to CauselyBot; CauselyBot creates or updates GitHub issues from the payload.

:::tip

You can assign an issue to [GitHub Copilot](https://github.com/features/copilot) by setting the `assignee` to `copilot-swe-agent`, which will trigger Copilot to analyze the issue and suggest a fix
in a pull request.

:::

### Example: Generic webhook (n8n, Linear, etc.)

To send to n8n or any other HTTP endpoint:

1. In CauselyBot's `causelybot-values.yaml`, add a webhook with `hook_type: generic` and set `url` to your destination endpoint (for example, the n8n Webhook node URL).
2. Create the Causely Secret as described in Step 3 above, pointing `notif_config_url` at CauselyBot.
3. CauselyBot forwards the raw Causely notification payload to your endpoint.
4. In n8n, read the incoming JSON fields (for example, `name`, `entity`, `severity`, `description`, `link`) from the [notification payload](/workflows/notification-payload) and use them in downstream nodes (Slack, email, Linear, etc.).

### Notification payload format

Causely sends the same structured JSON to all webhook destinations. For field descriptions and an example, see [Notification Payload Format](/workflows/notification-payload).

---

## Generic Webhook

A generic webhook sends Causely's standard notification payload, as JSON, to any HTTP endpoint. Use it to reach systems Causely doesn't support out of the box.

All generic webhook delivery runs through the **Causely Mediator**, which sends an HTTP `POST` with the [Causely notification payload](/workflows/notification-payload) as the request body.

How you define a generic webhook destination depends on your setup:

- **Managed Notifications (recommended)**  
  The endpoint URL, token, and routing rules are configured in the Causely UI. The UI stores configuration that the Mediator reads at runtime.

- **Mediator-based configuration (advanced)**  
  Define the destination programmatically with a Kubernetes secret or environment variables instead of the UI. Uses the same Mediator delivery path as managed notifications.

For routing rules and filters in the UI, see [Set up notification routing in the UI](/workflows/ui-notifications).

:::note When to use a generic webhook vs. Custom Workflows
Use a **generic webhook** when your endpoint can accept Causely's payload as-is.

If you need to **transform the payload** or **control the exact request format** sent to your target system, use [Custom Workflows](/workflows/custom-workflows) (CauselyBot) instead, which gives you full control over how the webhook is processed and formatted.
:::

### Managed Notifications (recommended)

1. Go to **Settings → Notifications**.
2. If you haven't already selected a processing mode, choose one now. Click **Create**.
3. Complete the notification configuration:

| Field | Value |
|-------|--------|
| **Name** | A descriptive name for the notification |
| **Type** | Generic |
| **URL** | Your endpoint, for example `https://api.example.com/notifications` |
| **Token** | Optional; see [Authentication](#authentication) below |

4. Choose the cluster or clusters that handle processing and delivery.
5. Optionally add [Notification Filters](#filters) to control when notifications are sent.
6. Click **Create**.

### Authentication

The **Token** field maps directly to the HTTP `Authorization` header. Causely sends whatever you enter **verbatim**, and it does not add or assume any authentication scheme. This means you are responsible for supplying the full header value, including the scheme keyword, that your endpoint expects.

| Your endpoint expects | Enter in the Token field |
|-----------------------|--------------------------|
| Bearer token | `Bearer <token>` |
| Basic auth | `Basic <base64 of user:password>` |
| A raw API key header value | Exactly what the endpoint expects |
| No authentication | Leave blank |

:::note Include the `Bearer` keyword yourself
If your endpoint uses bearer authentication, enter the full value `Bearer <token>` in the Token field, including the word `Bearer`. Causely does **not** add the `Bearer` prefix for the generic type.
:::

When the token is left blank, no `Authorization` header is sent. Requests are always sent with `Content-Type: application/json`.

### Mediator-based configuration (advanced)

If you prefer not to use the UI, define the generic webhook destination directly on the Mediator with a Kubernetes secret or environment variables.

#### Kubernetes secret

Create a secret labelled `causely.ai/notif-config: Generic` in the Mediator namespace. Set `notif_config_url` to your endpoint and, optionally, `notif_config_token` to the full `Authorization` header value.

```yaml
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: generic-notification-config
  namespace: causely
  labels:
    causely.ai/notif-config: Generic
stringData:
  notif_config_filters: "[]"
  notif_config_filters_enabled: "true"
  notif_config_global: "false"
  notif_config_name: generic-webhook
  notif_config_token: "Bearer <token>"   # optional; sent verbatim as the Authorization header
  notif_config_type: Generic
  notif_config_url: https://api.example.com/notifications
```

#### Environment variables

Set these on the Causely Mediator deployment. Replace `<NAME>` with your notification config name.

```bash
NOTIFICATION_<NAME>_TYPE=Generic
NOTIFICATION_<NAME>_URL=https://api.example.com/notifications
NOTIFICATION_<NAME>_TOKEN=Bearer <token>
NOTIFICATION_<NAME>_FILTERS_ENABLED=true
```

### Filters

Generic webhook destinations support filters (severity, namespace, entity type, `impactsSLO`, and more) like other Causely notification targets. Configure them in the create/edit modal in the UI, or via `notif_config_filters` in the Kubernetes secret.

`notif_config_filters` is a JSON array. When `notif_config_filters_enabled` is `"false"` or the array is empty (`"[]"`), all notifications pass through. Multiple filters are ANDed: a notification must match **all** filters to be forwarded.

| Field | Description |
|---|---|
| `severity` | `"Low"`, `"Medium"`, `"High"`, or `"Critical"` |
| `name` | Problem name (for example, `"Malfunction"`, `"CPUCongested"`) |
| `entity.type` | Entity type (for example, `"KubernetesService"`, `"ApplicationInstance"`) |
| `labels.k8s.cluster.name` | Kubernetes cluster name |
| `labels.k8s.namespace.name` | Kubernetes namespace name |
| `impactsSLO` | `true` if the notification includes impacted SLOs |

Supported operators: `equals`, `not_equals`, `in`, `not_in`.

For example, to forward only High and Critical severity notifications:

```yaml
notif_config_filters: '[{"field":"severity","operator":"in","value":["High","Critical"]}]'
notif_config_filters_enabled: "true"
```

For more on configuring filters in the UI, see [Managed Notifications](/workflows/ui-notifications).

---

#### Notification Payload Format

Causely sends the same structured JSON to all webhook destinations, including summaries of causal insights and service impact context.  
For an example payload and explanation of each field, refer to the [Notification Payload Format](/workflows/notification-payload) page.

---

## Grafana Alertmanager

Causely can send causal insights directly to your existing Grafana Alertmanager setup, allowing your team to respond quickly with actionable context. This integration enables you to route high-signal alerts into your existing alerting workflows, no additional rules or duplication needed.

Grafana Alertmanager can be used as a notification destination in Causely.

All Grafana Alertmanager delivery runs through the **Causely Mediator**.

How you define Grafana Alertmanager destinations depends on your setup:

- **Managed Notifications (recommended)**  
  Routing rules and Alertmanager credentials are configured in the Causely UI. The UI stores configuration that the Mediator reads at runtime.

- **Mediator-based configuration (advanced)**  
  Define Alertmanager destinations programmatically with environment variables or Kubernetes secrets instead of the UI. Uses the same Mediator delivery path as managed notifications.

For routing rules and filters in the UI, see [Set up notification routing in the UI](/workflows/ui-notifications).

### Mediator-based configuration (advanced)

If you prefer not to use the UI, define Grafana Alertmanager destinations with a Kubernetes secret in the causely namespace:

```yaml
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: grafana-notification-config
  namespace: causely
  labels:
    causely.ai/notif-config: AlertManager
data:
  notif_config_name: grafana
  notif_config_global: false
  notif_config_token: "user:password"
  notif_config_type: AlertManager
  notif_config_url: https://alertmanager-prod-us-east-0.grafana.net/alertmanager/api/v2/alerts
  notif_config_filters: []
  notif_config_filters_enabled: true
```

:::note

Replace the placeholder values with your actual Alertmanager URL in the "notif_config_url" field and optional credentials in the "notif_config_token" field

If you're using Grafana Cloud, refer to your instance's alerting endpoint.

:::

---

#### Notification Payload Format

Causely sends structured alert payloads to Alertmanager, including service-level impact and diagnostic summaries.
For an example and explanation of each field, refer to the [Notification Payload Format](/workflows/notification-payload) page.

---

## Causely Grafana Plugin

### Causal Insights in Grafana Dashboards

See actual causes and service impact directly in your Grafana dashboards, whether you're using Grafana Cloud or a self-hosted instance.

Causely for Grafana embeds high-context diagnostics alongside your existing metrics, helping teams quickly understand what broke, why it matters, and which services are affected. No code changes or custom alerting rules required.

---

#### Plugin Overview

- View active causes next to dashboards and alerts
- Drill into service relationships and impact
- Works with Grafana Cloud or self-hosted deployments

---

#### Install the Plugin

You can find the plugin on the Grafana Plugin Marketplace including setup guidance and official plugin documentation.

[**Install the Causely Plugin on Grafana**](https://grafana.com/grafana/plugins/esara-causely-app/?tab=installation)

---

## incident.io Integration

Causely can automatically create and resolve incidents in [incident.io](https://incident.io) based on causal inference findings. This integration enables your team to maintain a centralized incident management workflow while leveraging Causely's deep system insights.

When Causely detects critical or high-severity issues, it can automatically create structured incidents in incident.io with relevant context, severity mapping, and resolution tracking.

incident.io can be used as an incident management destination in Causely.

All incident.io delivery runs through the **Causely Mediator**.

How you define incident.io destinations depends on your setup:

- **Managed Notifications (recommended)**  
  Routing rules and incident.io credentials are configured in the Causely UI. The UI stores configuration that the Mediator reads at runtime.

- **Mediator-based configuration (advanced)**  
  Define incident.io destinations programmatically with environment variables or Kubernetes secrets instead of the UI. Uses the same Mediator delivery path as managed notifications.

For routing rules and filters in the UI, see [Set up notification routing in the UI](/workflows/ui-notifications).

---

### Mediator-based configuration (advanced)

If you prefer not to use the UI, define incident.io destinations with a Kubernetes secret in the causely namespace:

```yaml
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: incidentIO-notification-config
  namespace: causely
  labels:
    causely.ai/notif-config: IncidentIO
stringData:
  notif_config_filters: "[]"
  notif_config_filters_enabled: "true"
  notif_config_global: "false"
  notif_config_name: incidentIO
  notif_config_token: "your-incident-io-api-token"
  notif_config_type: IncidentIO
  notif_config_url: ""
```

:::note

Replace `your-incident-io-api-token` with your actual incident.io API token. You can generate one in your incident.io settings under API tokens.

:::

---

#### How It Works

##### Incident Creation

- **Automatic Creation**: Causely automatically creates incidents for **High** and **Critical** severity issues
- **Severity Mapping**: Causely severity levels are mapped to your incident.io severity configuration
- **Rich Context**: Incidents include detailed summaries, affected services, and direct links back to Causely's analysis
- **Idempotency**: Uses Causely's object ID to prevent duplicate incidents for the same issue

##### Incident Resolution

- **Automatic Resolution**: When Causely detects that an issue is resolved, it can automatically update the incident status
- **Configurable Status**: Set the target resolution status name (for example "Fixed", "Resolved")
- **Channel Notifications**: Optionally notify the incident channel when incidents are resolved

---

#### Configuration Options

##### Required Settings

- `api_token`: Your incident.io API token with incident creation permissions

##### Optional Settings

- `incident_type_id`: Specific incident type ID (auto-detected if not provided)
- `severity_id`: Specific severity ID (defaults to "Critical" severity if not provided)
- `resolved_status_name`: Status name for resolved incidents (enables auto-resolution)
- `notify_channel`: Whether to notify the incident channel on resolution (default: false)

##### Advanced Configuration

The Helm chart automatically configures the required environment variables based on your `causely-values.yaml` settings. The following environment variables are set automatically:

- `INCIDENT_IO_API_TOKEN`: Set from the secret containing your API token
- `INCIDENT_IO_INCIDENT_TYPE_ID`: Set from `incident_type_id` if provided
- `INCIDENT_IO_SEVERITY_ID`: Set from `severity_id` if provided
- `INCIDENT_IO_RESOLVED_STATUS_NAME`: Set from `resolved_status_name` if provided
- `INCIDENT_IO_NOTIFY_CHANNEL`: Set from `notify_channel` if provided

---

#### Notification Payload Format

Causely sends rich notification data to incident.io based on its causal inference.
For an example and explanation of each field, refer to the [Notification Payload Format](/workflows/notification-payload) page.

---

## Microsoft Teams

Microsoft Teams can be used as a notification destination in Causely.

All Microsoft Teams delivery runs through the **Causely Mediator**.

How you define Microsoft Teams destinations depends on your setup:

- **Managed Notifications (recommended)**  
  Routing rules and Microsoft Teams credentials are configured in the Causely UI. The UI stores configuration that the Mediator reads at runtime. Microsoft Teams delivery also requires a one-time Teams Workflow setup, described below.

- **Mediator-based configuration (advanced)**  
  Define Microsoft Teams destinations programmatically with environment variables or Kubernetes secrets instead of the UI. Uses the same Mediator delivery path as managed notifications.

For routing rules and filters in the UI, see [Set up notification routing in the UI](/workflows/ui-notifications).

### Managed Notifications (Recommended)

Most teams use Managed Notifications to route Causely insights to Microsoft Teams.  
In this model:

- Notification routing and logic are defined in the Causely UI
- Microsoft Teams is used purely as a delivery destination
- Alerts are delivered as Adaptive Cards

To receive notifications in Microsoft Teams, you must configure a Teams Workflow to accept webhook requests and post Adaptive Cards to a channel.

### Mediator-based configuration (advanced)

If you prefer not to use the UI, define Microsoft Teams destinations with a Kubernetes secret in the causely namespace:

```yaml
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: teams-notification-config
  namespace: causely
  labels:
    causely.ai/notif-config: Teams
stringData:
  notif_config_filters: "[]"
  notif_config_filters_enabled: "true"
  notif_config_global: "false"
  notif_config_name: teams
  notif_config_token: "causely"
  notif_config_type: Teams
  notif_config_url: "https://your-domain.webhook.office.com/webhookb2/..."
```

:::note
Replace the placeholder values with your actual Teams Workflow webhook URL.

Ensure the Microsoft Teams channel webhook is set up to accept incoming messages.
:::

### Microsoft Teams Workflow Setup

Microsoft Teams requires a Teams Workflow to receive webhook requests from Causely and render notifications as Adaptive Cards. This workflow is configured once per Team or Channel.

#### Step 1: Create a New Workflow

1. In Microsoft Teams, navigate to **Workflows**
2. Select **Create**
3. Choose **Create from blank**

#### Step 2: Add the Webhook Trigger

1. Select the trigger **When a Teams webhook request is received**
2. For **Who can trigger the flow**, select **Anyone**

This allows Causely to invoke the workflow using the generated webhook URL.

#### Step 3: Process Adaptive Card Attachments

Causely sends Microsoft Teams notifications as Adaptive Card attachments.

1. Add a new step and select **Apply to each**
2. For the input, select **attachments** from the webhook trigger output

This ensures each Adaptive Card attachment is handled correctly by the workflow.

#### Step 4: Post the Card to a Channel

Inside the **Apply to each** block:

1. Add the action **Post card in a chat or channel**
2. Configure the required fields:
   - **Post as**: Flow bot
   - **Post in**: Channel
   - **Team**: Select your team
   - **Channel**: Select the destination channel
   - **Adaptive Card**: Select **content** from the attachments output

#### Example Workflow Configuration

The example below shows a completed Microsoft Teams Workflow with the webhook trigger, attachment handling, and channel post configured correctly.

#### Result

Once configured, each Causely notification triggers the Teams Workflow and posts a structured Adaptive Card into the selected channel, providing immediate, actionable context without alert noise.

---

#### Notification Payload Format

Causely sends structured alert payloads to Microsoft Teams, including summaries of causal insights and service impact context.  
For an example payload and explanation of each field, refer to the [Notification Payload Format](/workflows/notification-payload) page.

---

## Notification Payload

Causely sends structured JSON payloads to all webhook-based integrations. Whether you're using Slack, Prometheus Alertmanager, Splunk On-Call, Microsoft Teams, or a custom system, the notification format below applies consistently.

#### Diagnosis Detected

The following payload is sent when Causely identifies a Diagnosis (`type: "ProblemDetected"`):

```json
{
  "link": "https://portal.causely.app/rootCauses/b55bd69c-7417-43c7-b1d0-1c06075fa63b",
  "name": "ImagePullErrors",
  "type": "ProblemDetected",
  "entity": {
    "id": "741d88ac-0d47-5f61-b9c1-2d7739a6ebf8",
    "link": "https://portal.causely.app/observe/topology/741d88ac-0d47-5f61-b9c1-2d7739a6ebf8",
    "name": "default/pull-error-demo-2",
    "type": "Controller"
  },
  "labels": {
    "k8s.cluster.uid": "8d92c7e7-db45-4267-a427-2fede28d1218",
    "k8s.cluster.name": "dev",
    "causely.ai/cluster": "dev",
    "k8s.namespace.name": "default",
    "k8s.controller.kind": "Deployment",
    "causely.ai/namespace": "default"
  },
  "objectId": "b55bd69c-7417-43c7-b1d0-1c06075fa63b",
  "severity": "Low",
  "timestamp": "2026-04-28T19:28:31.686364684Z",
  "description": {
    "details": "Image pull errors in Kubernetes often occur due to one of the following reasons:\n\n- **Invalid image name or tag**: The specified image name or tag does not exist in the registry, or has been incorrectly referenced.\n- **Image registry authentication issues**: The Kubernetes node may lack the necessary credentials (e.g., in a private registry) or permissions to pull the image.\n- **Network connectivity issues**: Kubernetes nodes might have restricted access to the internet or specific image registries, possibly due to firewall settings, proxy issues, or DNS misconfigurations.\n- **Rate limiting by image registries**: Some registries, such as Docker Hub, impose pull rate limits, which can prevent nodes from pulling images if exceeded.\n",
    "summary": "Kubernetes controllers may encounter image pull errors when they cannot download container images from a registry, causing Pods to fail in starting or remain in a `ImagePullBackOff` state. This disrupts the deployment of applications and can affect service availability.",
    "remediationOptions": [
      {
        "title": "Check Image Name and Tag",
        "description": "Verify that the image name and tag are correct and the image exists in your registry.\n"
      },
      {
        "title": "Configure ImagePullSecrets for Private Registries",
        "description": "If using a private registry, ensure an `ImagePullSecret` is correctly set up and assigned to the namespace or Pod:\n\n```bash\nkubectl create secret docker-registry myregistrykey \\\n  --docker-server=<registry-url> \\\n  --docker-username=<username> \\\n  --docker-password=<password> \\\n  --docker-email=<email>\n```\n\nAttach this secret to your Pod or namespace to provide credentials for image pulling.\n"
      }
    ]
  },
  "object_type": "defect"
}
```

#### Diagnosis Cleared

The following payload is sent when a previously detected Diagnosis is resolved (`type: "ProblemCleared"`). It includes the same fields as the detected payload, plus `duration_ns` indicating how long the Diagnosis was active:

```json
{
  "link": "https://portal.causely.app/rootCauses/b55bd69c-7417-43c7-b1d0-1c06075fa63b",
  "name": "ImagePullErrors",
  "type": "ProblemCleared",
  "entity": {
    "id": "741d88ac-0d47-5f61-b9c1-2d7739a6ebf8",
    "link": "https://portal.causely.app/observe/topology/741d88ac-0d47-5f61-b9c1-2d7739a6ebf8",
    "name": "default/pull-error-demo-2",
    "type": "Controller"
  },
  "labels": {
    "k8s.cluster.uid": "8d92c7e7-db45-4267-a427-2fede28d1218",
    "k8s.cluster.name": "dev",
    "causely.ai/cluster": "dev",
    "k8s.namespace.name": "default",
    "k8s.controller.kind": "Deployment",
    "causely.ai/namespace": "default"
  },
  "objectId": "b55bd69c-7417-43c7-b1d0-1c06075fa63b",
  "severity": "Low",
  "timestamp": "2026-04-28T19:28:31.686364684Z",
  "description": {
    "details": "Image pull errors in Kubernetes often occur due to one of the following reasons:\n\n- **Invalid image name or tag**: The specified image name or tag does not exist in the registry, or has been incorrectly referenced.\n- **Image registry authentication issues**: The Kubernetes node may lack the necessary credentials (e.g., in a private registry) or permissions to pull the image.\n- **Network connectivity issues**: Kubernetes nodes might have restricted access to the internet or specific image registries, possibly due to firewall settings, proxy issues, or DNS misconfigurations.\n- **Rate limiting by image registries**: Some registries, such as Docker Hub, impose pull rate limits, which can prevent nodes from pulling images if exceeded.\n",
    "summary": "Kubernetes controllers may encounter image pull errors when they cannot download container images from a registry, causing Pods to fail in starting or remain in a `ImagePullBackOff` state. This disrupts the deployment of applications and can affect service availability.",
    "remediationOptions": [
      {
        "title": "Check Image Name and Tag",
        "description": "Verify that the image name and tag are correct and the image exists in your registry.\n"
      },
      {
        "title": "Configure ImagePullSecrets for Private Registries",
        "description": "If using a private registry, ensure an `ImagePullSecret` is correctly set up and assigned to the namespace or Pod:\n\n```bash\nkubectl create secret docker-registry myregistrykey \\\n  --docker-server=<registry-url> \\\n  --docker-username=<username> \\\n  --docker-password=<password> \\\n  --docker-email=<email>\n```\n\nAttach this secret to your Pod or namespace to provide credentials for image pulling.\n"
      }
    ]
  },
  "duration_ns": 601639777492,
  "object_type": "defect"
}
```

#### Payload Fields

- `link`: Direct URL to the Diagnosis in the Causely portal.
- `name`: The name of the Diagnosis.
- `type`: The type of notification. Either `ProblemDetected` (Diagnosis identified) or `ProblemCleared` (Diagnosis resolved).
- `entity`: Details of the affected entity:
  - `id`: Unique identifier of the entity.
  - `link`: Direct URL to the entity in the Causely portal topology view.
  - `name`: Name of the entity.
  - `type`: Type of the entity (for example, `Controller`).
- `labels`: Metadata providing additional context such as cluster name, namespace, and controller kind.
- `objectId`: Unique identifier for the Diagnosis.
- `severity`: Severity level of the Diagnosis (for example, `Low`, `Medium`, `High`, or `Critical`).
- `timestamp`: The timestamp when the event occurred (ISO 8601 format).
- `description`: Structured details about the Diagnosis:
  - `summary`: A concise summary of the Diagnosis and its impact.
  - `details`: A detailed explanation of the Diagnosis, including possible contributing factors.
  - `remediationOptions`: A list of suggested remediation steps. Each item contains a `title` and a `description`.
- `duration_ns`: *(ProblemCleared only)* The duration the Diagnosis was active, in nanoseconds.
- `object_type`: The type of object associated with the event (for example, `defect`).

---

## Observe Export

Once Causely exports [Issues](/reference/terminology#issue) and [Diagnoses](/reference/terminology#diagnosis) to [Observe](https://www.observeinc.com/), Observe's AI SRE can answer reliability questions from the Diagnoses Causely has already identified, instead of inferring the cause from raw telemetry. This makes its answers more consistent and helps users resolve problems faster.

### Why export to Observe

An AI SRE that works only from raw telemetry has to work out the cause of a problem on its own. It reads metrics, logs, and traces, forms a hypothesis, and may reach a different answer each time it is asked.

Causely has already done that work. It identifies the Diagnosis behind each Issue from the relationships between your services and infrastructure. When this data is available in Observe, the AI SRE can start from the cause instead of searching for it.

For AI SRE use cases, this means:

- **Faster triage.** The AI SRE starts from the Diagnosis behind an Issue, so it spends less time searching telemetry for a cause.
- **More consistent answers.** The same Issue leads to the same cause, instead of a new hypothesis each time.
- **Shared context.** Responders and the AI SRE work from the same Diagnosis, so they discuss the same problem.

### What Causely exports

Causely exports the following to Observe:

- [Issues](/reference/terminology#issue)
- [Diagnoses](/reference/terminology#diagnosis)

### Configuration

Exporting to Observe is configured per Causely repository. It is currently available for [BYOC](/getting-started/byoc) deployments of Causely. To learn more about BYOC installation, see [BYOC (Self-Hosted) Deployment](/getting-started/byoc).

To set up the export, you need:

- A Causely deployment that supports export to Observe.
- Your Observe customer ID and a datastream token that accepts the data.

### Related

To discover Kubernetes clusters, application services, metrics, and container logs from Observe, see the [Observe telemetry source](/telemetry-sources/observe).

---

## Push Causal Insights Into Your Workflow

Causely doesn’t just emit alerts, it delivers explainable, causal insight to the right people, in the right context, at the right time.

Rather than forwarding raw metrics, logs, and traces, Causely acts as a reasoning layer between telemetry and action. It continuously reasons about cause and effect across your system to determine what matters, why it matters, and who should care, so teams can move from noise to understanding, and from understanding to action.

Causely supports multiple ways of delivering this insight, depending on how tightly you want understanding and action to be coupled.

#### Native Integrations

**Understand what changed and why, where engineers already work**

Native integrations bring Causely’s causal explanations, including the Diagnosis, blast radius, and service impact, directly into the tools engineers use every day.

This path is optimized for human understanding and collaboration. Engineers can quickly see not just that something is wrong, but what caused it, how it propagates, and what it affects, without switching context or interpreting raw telemetry.

Use this approach when the primary goal is shared situational awareness, faster diagnosis, and better coordination during investigation and response.

Available native integrations:

#### Configuring Notifications

There are two ways to configure where Causely sends notifications, and they offer the **same functionality**. Which one you use is simply a matter of preference for where you'd like to set things up:

- **In the UI**: point-and-click configuration in the Causely product.
- **Via a Kubernetes secret**: configuration as code, applied through the Mediator.

The two stay in sync automatically:

- When you configure a notification in the **UI**, the backing Kubernetes secret is **created for you automatically**.
- When you create a **secret** for notifications, it appears as a **configured notification in the UI**.

Pick whichever fits your workflow, both deliver the same notifications to the same destinations.

#### Managed Notifications (UI-Configured)

**Configure notifications directly in the Causely UI**

Use the Causely UI to choose where notifications are delivered and add filters that control when they're sent, without editing configuration files. When you save a configuration, Causely creates the corresponding Kubernetes secret for you automatically.

**Learn more**: [Managed notifications overview](/workflows/ui-notifications)

#### Mediator-Based Notifications

**Configure notifications with a Kubernetes secret**

Prefer configuration as code? You can define notification destinations by creating a Kubernetes secret (or by setting values in your `values.yaml`) that the Mediator reads at runtime. This is the same delivery path used by UI-configured notifications, a notification created this way also appears as a configured notification in the UI.

To configure mediator-based notifications, enable the executor and specify a webhook URL and token in the `values.yaml` file.

Add the following under the notifications section of your [`causely-values.yaml`](/installation/customize#use-a-custom-values-file) file:

```shell
executor:
  enabled: true

notifications:
  webhook:
    url: "https://webhook.example.com/notify"   # Replace with your webhook URL
    token: "your-secret-token"                  # Replace with your webhook token
    enabled: true
```

:::info

For more details on webhook payload structure and field definitions, see the [Notification Payload Format](/workflows/notification-payload).

:::

Make sure to replace the placeholder values with your actual webhook URL and token.

Click on an integration below to view mediator-based setup instructions for receiving Causely notifications in your preferred tool.

#### How to Choose

- **Want engineers to quickly understand what changed and why?**  
  → Native integrations

- **Prefer to configure notifications with point-and-click?**  
  → Managed notifications (UI-configured)

- **Prefer configuration as code / GitOps?**  
  → Mediator-based notifications (Kubernetes secret)

Managed and mediator-based notifications deliver the same result, choose whichever matches how you like to work.

#### Why This Matters

By separating **causal understanding** from **delivery and action**, Causely ensures teams don’t just receive more alerts, they receive **the right insight, in the right place, with the right context**.

This turns reliability from a reactive exercise into a system of understanding, coordination, and confident action.

---

## Prometheus Alertmanager

Causely can send enriched causal insights directly to your existing Prometheus Alertmanager. This lets you keep using your current alerting pipeline while gaining deeper visibility into what's actually going wrong.

By routing causal insights from Causely to Alertmanager, your team can receive high-signal notifications that include context like impacted services, severity, and SLO relevance—without needing to define new Prometheus alert rules.

Prometheus Alertmanager can be used as a notification destination in Causely.  
How you configure this integration depends on how notifications are managed in your environment.

Causely supports two configuration paths:

- **Managed Notifications (recommended)**  
  Routing rules are configured in the Causely UI based on impact, severity, and context.  
  Prometheus Alertmanager is used purely as a delivery destination, allowing you to inject high-signal causal insights into your existing Alertmanager pipeline without duplicating Prometheus alert rules.

- **Mediator-based Notifications (advanced)**  
  Notifications are configured programmatically using the Causely Mediator and sent directly to your Prometheus Alertmanager webhook endpoint.

Routing and notification rules for Managed Notifications are configured in the Causely UI:
[Set up notification routing in the UI](/workflows/ui-notifications)

#### Mediator-based Configuration

If you would like to configure the Prometheus Alertmanager integration, you can do so by enabling the notifications in the `causely-values.yaml` file.

For example, add the following in your `causely-values.yaml` file like this:

```yaml
mediator:
  notifications:
    enabled: true
```

You can define a Grafana Alertmanager notification integration through the Causely UI, or by creating a secret in your Kubernetes cluster, in the causely namespace:

```yaml
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: prometheus-notification-config
  namespace: causely
  labels:
    causely.ai/notif-config: AlertManager
data:
  notif_config_name: prometheus
  notif_config_global: false
  notif_config_token: "user:password"
  notif_config_type: AlertManager
  notif_config_url: https://alertmanager-prod-us-east-0.grafana.net/alertmanager/api/v2/alerts
  notif_config_filters: []
  notif_config_filters_enabled: true
```

:::note

Replace the placeholder values with your actual Alertmanager URL in the "notif_config_url" field and optional credentials in the "notif_config_token" field

:::

To install the Causely agent with the Alertmanager integration enabled, run:

```bash
causely agent install --cluster-name <cluster-name> --values ./causely-values.yaml
```

---

#### Notification Payload Format

Causely sends structured alert payloads to Alertmanager, including service-level impact and diagnostic summaries.
For an example and explanation of each field, refer to the [Notification Payload Format](/workflows/notification-payload) page.

---

## Slack Notifications

Slack can be used as a notification destination in Causely.

All Slack delivery runs through the **Causely Mediator**.

How you define Slack destinations depends on your setup:

- **Managed Notifications (recommended)**  
  Routing rules and Slack credentials are configured in the Causely UI. The UI stores configuration that the Mediator reads at runtime.

- **Mediator-based configuration (advanced)**  
  Define Slack destinations programmatically with environment variables or Kubernetes secrets instead of the UI. Uses the same Mediator delivery path as managed notifications.

For routing rules and filters in the UI, see [Set up notification routing in the UI](/workflows/ui-notifications).

### Delivery modes

Causely can deliver problem and defect notifications to Slack in two ways. Configure **one mode per notification destination**; you do not need both a webhook and a bot token for the same destination.

| Mode | URL field | Token field | Delivery |
|------|-----------|-------------|----------|
| **Incoming webhook** | `https://hooks.slack.com/services/...` | Leave empty | One combined message per alert |
| **Bot API (threaded)** | Channel ID (for example, `C01234567` or `G01234567` for private channels) | Bot token (`xoxb-...`) | Parent message + threaded replies |

The bot must be a **member of the channel** (invite it with `/invite @YourBotName`). Use the channel ID (for example, `C01234567`) and not the channel name (for example, `#alerts`) in the URL field, as channel names are not supported in the configuration.

The mediator selects the mode automatically from the URL and token values.

---

### Option A: Incoming webhook (simple)

Best when you want a single message per alert and already have (or can create) an incoming webhook.

#### 1. Create a Slack incoming webhook

1. Open [Slack API apps](https://api.slack.com/apps) or use **Slack → Tools → Workflow Builder / Apps** depending on your workspace policy.
2. Create or select an app, then enable **Incoming Webhooks**.
3. **Add New Webhook to Workspace** and pick the target channel.
4. Copy the webhook URL (`https://hooks.slack.com/services/T.../B.../...`).

#### 2. Configure Causely

1. Go to **Settings → Notifications**.
2. If you haven't already selected a processing mode, choose one now. Click **Create**.
3. Complete the notification configuration:

| Field | Value |
|-------|--------|
| **Name** | A descriptive name for the notification |
| **Type** | Slack |
| **URL** | The incoming webhook URL |
| **Token** | Leave blank |

---

Alternatively, configure the Mediator with environment variables or a Kubernetes secret instead of using the UI.

#### Environment variables

Set these on the Causely Mediator deployment. Replace `<NAME>` with your notification config name.

```bash
NOTIFICATION_<NAME>_TYPE=Slack
NOTIFICATION_<NAME>_URL=https://hooks.slack.com/services/T.../B.../...
NOTIFICATION_<NAME>_FILTERS_ENABLED=true
```

#### Kubernetes secret

Create a secret labelled `causely.ai/notif-config: Slack` in the Mediator namespace:

- `notif_config_type`: `Slack`
- `notif_config_name`: your config name
- `notif_config_url`: webhook URL
- `notif_config_filters`: optional JSON filter rules

```yaml
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: slack-notification-config
  namespace: causely
  labels:
    causely.ai/notif-config: Slack
stringData:
  notif_config_filters: "[]"
  notif_config_filters_enabled: "true"
  notif_config_global: "false"
  notif_config_name: slack
  notif_config_token: ""
  notif_config_type: Slack
  notif_config_url: https://hooks.slack.com/services/XXX/YYY/ZZZ
```

#### 3. Verify

Post a test notification from the UI or trigger a real alert. You should see one Block Kit message with header, entity metadata, summary, evidence, remediation, SLOs, labels (when present), and a **View** button.

---

### Option B: Bot token + channel (threaded)

Best when you want a compact parent message and details in a thread (summary, labels, evidence, remediation as separate replies).

#### 1. Create a Slack app and bot token

1. Go to [https://api.slack.com/apps](https://api.slack.com/apps) → **Create New App** → **From scratch**.
2. Open **OAuth & Permissions** and add the bot token scope:
   - `chat:write`: post messages to channels the bot has joined
3. Go to **Install App** and click **Install to Workspace** (or **Reinstall to Workspace** if the app is already installed).
4. Copy the **Bot User OAuth Token** (`xoxb-...`) shown on that page.

#### 2. Create or choose a channel

1. Create or pick a channel (for example, `#causely-alerts`).
2. **Invite the bot** to the channel (required): `/invite @YourBotName`
3. Copy the **channel ID** (required for the Causely URL field):
   - Open the channel → channel name → **About** → scroll down → copy the ID (`C...` for public, `G...` for private).

#### 3. Configure Causely

1. Go to **Settings → Notifications**.
2. If you haven't already selected a processing mode, choose one now. Click **Create**.
3. Complete the notification configuration:

| Field | Value |
|-------|--------|
| **Name** | A descriptive name for the notification |
| **Type** | Slack |
| **URL** | Channel ID (for example, `C0123456789`) |
| **Token** | Bot User OAuth Token (`xoxb-...`) |

The token field is optional for webhooks but **required** when the URL is not a `hooks.slack.com` webhook.

---

Alternatively, configure the Mediator with environment variables or a Kubernetes secret instead of using the UI.

#### Environment variables

Set these on the Causely Mediator deployment. Replace `<NAME>` with your notification config name.

```bash
NOTIFICATION_<NAME>_TYPE=Slack
NOTIFICATION_<NAME>_URL=C0123456789
NOTIFICATION_<NAME>_TOKEN=xoxb-your-bot-token
NOTIFICATION_<NAME>_FILTERS_ENABLED=true
```

A `Bearer` prefix on the token is accepted (for example, `Bearer xoxb-...`).

#### Kubernetes secret

Create a secret labelled `causely.ai/notif-config: Slack` in the Mediator namespace. Set `notif_config_url` to the channel ID and `notif_config_token` to the bot token.

```yaml
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: slack-bot-notification-config
  namespace: causely
  labels:
    causely.ai/notif-config: Slack
stringData:
  notif_config_filters: "[]"
  notif_config_filters_enabled: "true"
  notif_config_global: "false"
  notif_config_name: slack-threaded
  notif_config_token: xoxb-your-bot-token
  notif_config_type: Slack
  notif_config_url: C0123456789
```

#### 4. Verify with curl

```bash
curl -s https://slack.com/api/chat.postMessage \
  -H "Authorization: Bearer xoxb-your-token" \
  -H "Content-Type: application/json" \
  -d '{"channel":"C0123456789","text":"Causely bot test"}'
```

Expect `"ok": true`.

#### 5. Thread layout (bot mode)

For detected problems, Causely posts:

| Message | Content |
|---------|---------|
| **Parent** | Header, entity/severity metadata, **View** button |
| **Thread 1** | Summary |
| **Thread 2** | Labels |
| **Thread 3** | Evidence and impacted SLOs |
| **Thread 4** | Remediation |

Cleared notifications are sent as a single message. Empty sections are skipped (for example, no labels thread if there are no labels).

Failed thread replies are retried with backoff; remaining sections may be combined into a fallback thread message.

---

### Mediator-based configuration (advanced)

Managed notifications in the UI are the recommended path. If you configure the Mediator directly instead, use the **Environment variables** or **Kubernetes secret** sections under [Option A](#option-a-incoming-webhook-simple) or [Option B](#option-b-bot-token--channel-threaded) above.

---

### Filters and object type

Notification destinations support filters (severity, namespace, entity type, etc.) and object type (`defect` vs `issue`) like other Causely notification targets. Configure these in the same create/edit modal or via `notif_config_filters` / `NOTIFICATION_<NAME>_FILTERS`.

---

### Troubleshooting

| Symptom | Likely cause | Fix |
|---------|----------------|-----|
| `not_in_channel` | Bot is not a channel member | `/invite @Bot` in the target channel before sending |
| `channel_not_found` | Wrong channel ID or bot not in channel | Use channel ID (not `#name`); invite bot to private channels |
| `invalid_auth` | Bad or revoked token | Reinstall app; copy fresh `xoxb-` token |
| Webhook works but threading does not | URL still points at webhook | Use channel ID in URL, not `hooks.slack.com` |
| Threaded mode not used | Missing or invalid token | Token must start with `xox` (after optional `Bearer` prefix) |
| Notification marked failed but message in Slack | Partial thread delivery | Check mediator logs; retries/fallback may still leave parent visible |
| UI has no token field | Older UI build | Upgrade UI or set token via env / K8s secret |

---

### Related documentation

- [Managed Notifications](/workflows/ui-notifications)
- Slack incoming webhooks: [https://docs.slack.dev/messaging/sending-messages-using-incoming-webhooks/](https://docs.slack.dev/messaging/sending-messages-using-incoming-webhooks/)
- Slack `chat.postMessage`: [https://docs.slack.dev/reference/methods/chat.postMessage](https://docs.slack.dev/reference/methods/chat.postMessage)

---

#### Notification Payload Format

Causely sends structured alert payloads to Slack, including summaries of causal insights and service impact context.  
For an example payload and explanation of each field, refer to the [Notification Payload Format](/workflows/notification-payload) page.

---

## Splunk On-Call

Causely can send high-context causal insights directly to Splunk On-Call (formerly VictorOps), so your on-call team receives the signal, not the noise. With enriched context like impacted services, problem severity, and diagnostic summaries, alerts become more actionable and reduce time-to-resolution.

Splunk On-Call can be used as a notification destination in Causely.

All Splunk On-Call delivery runs through the **Causely Mediator**.

How you define Splunk On-Call destinations depends on your setup:

- **Managed Notifications (recommended)**  
  Routing rules and Splunk On-Call credentials are configured in the Causely UI. The UI stores configuration that the Mediator reads at runtime.

- **Mediator-based configuration (advanced)**  
  Define Splunk On-Call destinations programmatically with environment variables or Kubernetes secrets instead of the UI. Uses the same Mediator delivery path as managed notifications.

For routing rules and filters in the UI, see [Set up notification routing in the UI](/workflows/ui-notifications).

### Mediator-based configuration (advanced)

If you prefer not to use the UI, define Splunk On-Call destinations with a Kubernetes secret in the causely namespace:

```yaml
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: splunk-notification-config
  namespace: causely
  labels:
    causely.ai/notif-config: Splunk
stringData:
  notif_config_filters: "[]"
  notif_config_filters_enabled: "true"
  notif_config_global: "false"
  notif_config_name: splunk
  notif_config_token: "causely"
  notif_config_type: Splunk
  notif_config_url: https://alert.victorops.com/integrations/generic/12356789/alert/ab12cd34-ef56-7890-ab12-34cd56ef7890
```

:::note

Replace the placeholder values with your actual integration URL and routing key from Splunk On-Call.

:::

---

#### Notification Payload Format

Causely sends structured alert payloads to Splunk On-Call, including rich metadata about the actual causes and their service-level impact.  
For an example payload and explanation of each field, refer to the [Notification Payload Format](/workflows/notification-payload) page.

---

## Managed Notifications

Causely allows you to configure **managed notifications** directly in the UI, so teams can control what gets routed, where, and under which conditions, without managing configuration files or infrastructure.

Managed notifications are the recommended approach for most teams. They let you route causal insights based on impact, severity, and context.

#### Create a Notification Configuration

1. Navigate to Settings → Notifications.
2. Click Create to add a new notification configuration.
3. Choose global or per-cluster processing
4. Enter a name that describes the purpose of the notification (for example, prod-critical-alerts).

The name is used only for identification within Causely and does not affect delivery. Each notification configuration must have a unique name.

#### Choose Global or Per-Cluster Processing

Select how notifications are processed across your environment. Causely supports two modes:

- Global: One cluster handles notification processing for all clusters. Select the processing cluster once; all notification configurations share it.
- Per-cluster: Each cluster processes its own notifications independently. You will select a processing cluster for each notification configuration. This eliminates the need to filter by cluster when scoping notifications.

Notifications are processed by the Causely Mediator running in the selected cluster.

#### Select a Notification Type

Choose where you want to receive notifications. Supported destinations include:

- Slack
- Microsoft Teams
- Prometheus Alertmanager
- incident.io
- Splunk On-Call (VictorOps)
- Generic Webhook

The selected type determines both delivery format and required fields in the next step.

#### Configure Destination-Specific Fields

Depending on the notification type, you may be asked to provide additional details, such as:

- Endpoint URL (for webhooks or alert managers)
- Authentication token (optional, depending on destination)

These fields control how Causely delivers notifications to the selected system.

For the **Generic Webhook** type, the token is sent verbatim as the HTTP `Authorization` header. See [Generic Webhook](/workflows/generic-webhook) for how to format it (for example, `Bearer <token>`).

#### For Per-Cluster Processing Choose a Processing Cluster for 

Select a cluster to handle notification processing.

Notifications are processed by the Causely Mediator running in the selected cluster.

#### Add Filters (Optional but Recommended)

Filters allow you to control which causal insights trigger notifications. You can filter by:

- Entity Type: for example Service, HTTP Path, or Node
- Entity ID: the unique identifier of a specific entity
- Entity Name: the display name of an entity. Supports regex matching (see [Match entity names with regex](#match-entity-names-with-regex) below)
- Namespace
- Diagnosis: for example Code Change Regression: CPU Congestion or Database Malfunction
- Severity: Urgent vs. Non-urgent
- Cluster: Not needed when using per-cluster processing, since each configuration is already scoped to a specific cluster.

Filters help reduce noise and ensure notifications are routed only when they matter to the receiving team.

##### Match entity names with regex

The **Entity Name** filter supports pattern matching with two operations:

- **REGEX**: notify when the entity name matches the pattern.
- **NOT REGEX**: notify when the entity name does **not** match the pattern.

Matching uses Go's standard regular expression engine (the [`regexp`](https://pkg.go.dev/regexp) package, which follows [RE2 syntax](https://github.com/google/re2/wiki/Syntax)). RE2 does not support backreferences or lookaround assertions. Patterns are unanchored by default, so a match anywhere in the name counts, use `^` and `$` to anchor to the start or end of the name.

| Pattern | Matches |
|---|---|
| `checkout` | any entity name containing `checkout` |
| `^checkout` | names that start with `checkout` |
| `-canary$` | names that end with `-canary` |
| `^checkout-service$` | exactly `checkout-service` |
| `(checkout\|payment)` | names containing `checkout` or `payment` |
| `(?i)prod` | `prod` matched case-insensitively (`Prod`, `PROD`, …) |

For example, use the `REGEX` operation with `^prod-` to notify only on entities whose names start with `prod-`, or use `NOT REGEX` with `-canary$` to exclude every entity whose name ends in `-canary`.

#### Save and Manage Notifications

Once saved, the notification configuration appears in the Notifications list, where you can:

- View active filters
- Edit configuration details
- Disable or delete notifications as needed

Changes take effect immediately and do not require redeploying agents or updating configuration files.

#### Test the Notification

After saving, you can send a test notification to verify the configuration.

- Optionally apply test-only filters to simulate specific scenarios.
- Review the exact payload that will be sent to the destination.
- Confirm the notification is received and formatted as expected.

Testing helps validate both routing logic and downstream integrations.

#### When to Use UI-Based Notifications

Managed notifications are best suited for teams that want:

- Centralized, product-managed routing
- Explainable notification logic
- Easy iteration as systems and ownership change

For programmatic or infrastructure-managed setups, notifications can also be configured through the [Causely Mediator](/workflows/#mediator-based-notifications).