Prometheus

Signals Provided

• Metrics - Performance metrics from applications and infrastructure
• Symptoms - Automatic symptom detection from metrics, traces, and external monitoring systems

Causely leverages Prometheus to query time-series metrics from your applications and infrastructure.

To enable Prometheus as a data source with the default exporters, please add the following section to your values.yaml file:

scrapers:
  prometheus:
    enabled: true
    servers:
      prometheus:
        endpoint: http://prometheus-operated.monitoring:9090

Supported Exporters

Causely will automatically pick up metrics from supported exporters.

Supported Exporters:

• cAdvisor - Container resource usage and performance metrics
• CockroachDB - CockroachDB database metrics
• Golang - Go application metrics including mutex and garbage collection
• Java - Java application metrics including JVM memory and garbage collection
• Kafka - Kafka broker and topic metrics
• MongoDB - MongoDB database metrics
• MySQL - MySQL database metrics
• PostgreSQL - PostgreSQL database metrics
• Python - Python application metrics
• RabbitMQ - RabbitMQ message broker metrics
• Redis - Redis cache and database metrics

Prometheus Discovery

When using Prometheus with Kubernetes service discovery, namespace and pod labels are typically added automatically by the Prometheus Helm chart through relabeling. If these labels are missing from your metrics, configure relabeling in your ServiceMonitor or PodMonitor configuration to map Kubernetes metadata (__meta_kubernetes_namespace, __meta_kubernetes_pod_name) to namespace and pod labels.

If you're using the OpenTelemetry Collector with the prometheus/metrics receiver, see the OpenTelemetry Prometheus metrics relabeling configuration for how to configure relabeling in the collector.

How to add custom metrics

To enable Prometheus as a data source with default exporters plus additional custom metrics, please add the following section to your values.yaml file:

scrapers:
  prometheus:
    enabled: true
    servers:
      prometheus:
        endpoint: http://prometheus-operated.monitoring:9090
        exporters:
          - cadvisor
          - causely
          - go-applications
          - java-applications
          - python-applications
          - postgres
          - mongodb
          - cockroachdb
          - redis
          - kafka
          - mysql
          - rabbitmq
          - chaosmania
    exporters:
      chaosmania:
        entities:
          - entity:
              workload: {}

            discovery:
              - kubernetes_pod:
                  namespace: 'namespace'
                  pod_name: 'pod'

            metrics:
              - attribute: TransactionCount
                query: 'sum by (namespace, service, pod) (rate(chaosmania_processed_transactions_duration_count[1m]))'
              - attribute: TransactionDuration
                query: 'sum by (namespace, service, pod) (rate(chaosmania_processed_transactions_duration_sum[1m]) / (rate(chaosmania_processed_transactions_duration_count[1m]) > 0 or (rate(chaosmania_processed_transactions_duration_count[1m]) + 1)))'
              - attribute: DBQueryDuration
                query: 'sum by (namespace, service, pod) (rate(postgres_queries_sum[1m]) / (rate(postgres_queries_count[1m]) > 0 or (rate(postgres_queries_count[1m]) + 1)))'

Example: Kafka Exporter

You can use this helm values.yaml file for the kafka exporter to label the broker name super-heroes-kafka-brokers for the monitored topics:

kafkaServer:
  - super-heroes-kafka-brokers:9092

service:
  annotations:
    prometheus.io/port: '9308'
    prometheus.io/scrape: 'true'
  labels:
    service_name: super-heroes-kafka-brokers

prometheus:
  serviceMonitor:
    enabled: true
    namespace: quarkus-super-heroes
    relabelings:
      - sourceLabels: [__meta_kubernetes_service_label_service_name]
        targetLabel: target

Multi-tenant Mimir

Causely supports multi-tenant Mimir deployments, allowing you to configure multiple tenants and specific exporters for each Mimir instance. To enable Mimir as a data source with multi-tenant support, add the following section to your values.yaml file:

scrapers:
  prometheus:
    enabled: true
    servers:
      mimir:
        endpoint: http://mimir-nginx.mimir:80/prometheus
        tenants:
          - tenant-1
          - tenant-2
        exporters:
          - postgres
          - redis

This configuration allows Causely to:

• Connect to the Mimir instance via the specified endpoint
• Query metrics from multiple tenants (tenant-1 and tenant-2)