Skip to main content

Troubleshooting Telemetry Sources

If entities and connections are not appearing in your Causely topology, check the Integrations page for errors. This guide helps you diagnose and resolve common issues with telemetry source integrations.

Identifying Issues​

Check the Integrations Page​

  1. Navigate to the Integrations page in the Causely UI
  2. Look for any integrations showing error states (red indicators or error messages)
  3. Click on an integration to view detailed error information

Common error indicators include:

  • Initialization errors: The scraper failed to start
  • Authentication errors: Invalid credentials or missing permissions
  • Connection errors: Unable to reach the telemetry source
  • Configuration errors: Missing or incorrect configuration values

Common Issues and Solutions​

Authentication and Credentials​

Invalid or Expired Credentials​

Symptoms:

  • Error messages mentioning "authentication failed", "unauthorized", or "invalid credentials"
  • Integration shows as "Initialization Failed"

Solutions:

  1. Verify credentials are correct:

    • Check that API keys, tokens, or passwords are not expired
    • Ensure credentials match the correct account or environment
    • Verify there are no extra spaces or special characters when copying credentials

  2. For Kubernetes secrets:

    # Verify the secret exists and contains the correct data
    kubectl get secret <secret-name> -n causely -o yaml

    # Check that the secret keys match what Causely expects
    # (e.g., apiKey, appKey, token, etc.)

  3. For cloud provider integrations (AWS, Azure, GCP):

    • Verify IAM roles or service accounts have the required permissions
    • Check that access keys haven't been rotated
    • Ensure service account annotations are correctly set for IRSA (AWS) or Workload Identity (GCP)

Missing Secret Labels (Autodiscovery)​

Symptoms:

  • Integration not appearing in the Integrations page
  • Scraper not starting despite secret existing

Solutions:

  1. Verify the secret has the correct label:

    # Check if the secret has the scraper label
    kubectl get secret <secret-name> -n causely --show-labels

    # Add the label if missing (example for PostgreSQL)
    kubectl --namespace causely label secret <secret-name> "causely.ai/scraper=Postgresql"

  2. Common scraper label values:

    • Postgresql for PostgreSQL
    • MySQL for MySQL
    • Datadog for Datadog
    • AWS for AWS
    • Azure for Azure
    • GCP for GCP
    • Instana for Instana
    • Dynatrace for Dynatrace
    • Confluent for Confluent
    • Snowflake for Snowflake

Network Connectivity​

Cannot Reach Telemetry Source​

Symptoms:

  • Connection timeout errors
  • "Unable to connect" or "network unreachable" messages
  • Integration shows intermittent failures

Solutions:

  1. Verify network connectivity from the mediator pod:

    # Get the mediator pod name
    kubectl get pods -n causely -l app=mediator

    # Test connectivity to the telemetry source
    kubectl exec -n causely <mediator-pod-name> -- curl -v <endpoint-url>

  2. Check firewall rules:

    • Ensure the mediator can reach external APIs (for cloud providers)
    • Verify internal network policies allow communication
    • Check if the telemetry source requires allowlisting the mediator's IP

  3. For Prometheus/Istio integrations:

    • Verify the Prometheus endpoint is accessible from the mediator
    • Check that the endpoint URL is correct (including protocol: http:// or https://)
    • Ensure Prometheus is not behind authentication that requires additional configuration

Configuration Errors​

Missing Required Configuration​

Symptoms:

  • "Configuration error" or "missing required field" messages
  • Integration fails to initialize

Solutions:

  1. Verify all required fields are set:

    • Check the specific telemetry source documentation for required configuration
    • Ensure secrets contain all required keys
    • Verify values.yaml or configuration files have all necessary settings

  2. Common missing fields:

    • Database integrations: Missing host, port, database, or username
    • Cloud providers: Missing region or account ID
    • APM tools: Missing apiKey, appKey, or org (for Datadog)
    • Prometheus: Missing endpoint URL

  3. Check configuration format:

    # Validate Helm values
    helm template causely oci://us-docker.pkg.dev/public-causely/public/causely --values values.yaml --dry-run

    # Check mediator logs for configuration errors
    kubectl logs -n causely -l app=mediator --tail=100

Incorrect Endpoint URLs​

Symptoms:

  • Connection refused errors
  • 404 Not Found errors
  • SSL/TLS certificate errors

Solutions:

  1. Verify endpoint URLs:

    • Ensure URLs include the correct protocol (http:// or https://)
    • Check that ports are correct (for example, :4317 for OTLP, :9090 for Prometheus)
    • Verify hostnames resolve correctly

  2. For HTTPS endpoints:

    • Check if self-signed certificates require additional configuration
    • Verify certificate validity if using custom certificates

Permission Issues​

Insufficient Permissions​

Symptoms:

  • "Access denied" or "Forbidden" errors
  • Integration can connect but cannot read data
  • Partial data discovery (some entities missing)

Solutions:

  1. For cloud provider integrations:

    AWS:

    • Verify IAM role or user has required read permissions
    • Check policies include: ReadOnlyAccess or specific service read permissions
    • For EKS, ensure IRSA (IAM Roles for Service Accounts) is configured correctly

    Azure:

    • Verify service principal has "Reader" role or equivalent
    • Check that the subscription is accessible
    • Ensure Managed Identity is configured if using that method

    GCP:

    • Verify service account has required IAM roles
    • Check Workload Identity is configured for GKE
    • Ensure project-level permissions are correct

  2. For database integrations:

    • Verify database user has SELECT permissions on system tables
    • Check that the user can access pg_stat_statements (PostgreSQL) or equivalent
    • Ensure the user has permissions to query metadata tables

  3. For Kubernetes:

    • Verify the mediator service account has necessary RBAC permissions
    • Check ClusterRole and ClusterRoleBinding are correctly configured
    • Ensure the mediator can list and get resources in target namespaces

Version and Compatibility Issues​

Incompatible Versions​

Symptoms:

  • API errors mentioning version incompatibility
  • Unexpected response formats
  • Features not working as expected

Solutions:

  1. Check Causely version compatibility:

    • Review the telemetry source documentation for version requirements
    • Ensure you're using a supported version of Causely
    • Check changelog for breaking changes

  2. Verify telemetry source versions:

    • Some integrations require minimum versions of the source system
    • Check if API versions have changed
    • Review telemetry source release notes

Platform-Specific Troubleshooting​

Kubernetes​

Common issues:

  • Service account permissions
  • Network policies blocking communication
  • Resource quotas limiting scraper execution

Debug steps:

# Check mediator pod status
kubectl get pods -n causely

# View mediator logs
kubectl logs -n causely -l app=mediator --tail=200

# Check service account
kubectl get serviceaccount -n causely

# Verify RBAC permissions
kubectl get clusterrolebinding | grep causely

Docker​

Common issues:

  • Container networking issues
  • Volume mount permissions
  • Environment variable configuration

Debug steps:

# Check container status
docker ps | grep causely

# View container logs
docker logs <container-name>

# Verify environment variables
docker exec <container-name> env | grep CAUSELY

Nomad​

Common issues:

  • Job allocation failures
  • Network configuration
  • Volume access issues

Debug steps:

# Check job status
nomad job status causely

# View allocation logs
nomad alloc logs <allocation-id>

# Check job specification
nomad job inspect causely

Getting Additional Help​

If you've tried the solutions above and are still experiencing issues:

  1. Check mediator logs for detailed error messages:

    kubectl logs -n causely -l app=mediator --tail=500

  2. Review integration-specific documentation:

  3. Contact Causely Support:

    • Email: support@causely.ai
    • Include:
      • Integration name and error message
      • Relevant log excerpts (redact sensitive information)
      • Configuration details (redact credentials)
      • Steps you've already tried

Prevention Best Practices​

  1. Use autodiscovery labels to ensure secrets are properly recognized
  2. Test connectivity before configuring integrations
  3. Verify credentials are valid and have appropriate permissions
  4. Monitor integration health regularly in the Integrations page
  5. Keep Causely updated to ensure compatibility with latest telemetry sources
  6. Document your configuration to make troubleshooting easier