Services
Services are self-contained units of functionality within a system that perform specific tasks or provide specific capabilities, often accessible through defined interfaces or APIs. Service can be either internal or third-party external services beyond the core application and infrastructure.
Overview
AI Model
Congested
The AI model is experiencing congestion, resulting in high latency for clients. This suggests that the model is unable to handle the current inference load efficiently, causing delays in response times. Congestion often occurs when the model receives more requests than it can process within its capacity, leading to bottlenecks in inference. This may be due to insufficient resources (for example, GPU, memory, or throughput limits), inefficient batching, or a surge in inference demand.
Malfunction
The AI model is experiencing a high rate of errors, causing disruptions for clients. This can lead to degraded performance, failed inference requests, or complete model unavailability, significantly affecting the user experience.
Application Load Balancer
Authentication Misconfiguration
Application Load Balancer (ALB) authentication misconfiguration can disrupt secure traffic routing and lead to widespread configuration issues. This misconfiguration may trigger elevated ELB authentication errors, 504 request timeouts, and target connection errors, ultimately impacting service availability and application performance.
Idle Timeout Misconfiguration
Misconfigured idle timeout settings can lead to unintended connection drops and delays, potentially triggering a high frequency of 504 gateway timeout errors. This misconfiguration may also contribute to broader configuration issues that disrupt seamless connectivity between clients and servers.
Network Policy Misconfiguration
Application Load Balancer (ALB) network policy misconfigurations can block or restrict legitimate traffic, leading to widespread configuration issues. These misconfigurations often result in elevated 504 gateway timeout errors and target connection failures, ultimately disrupting service availability.
Unknown Configuration Failure
Application Load Balancer (ALB) misconfiguration can cause widespread connectivity issues, leading to a high frequency of 504 gateway timeout errors and 5xx server errors. These issues indicate that the load balancer's settings are not properly optimized for handling traffic efficiently and reliably.
Messaging / Event Streaming
Congested Azure Event Hub Namespace
When the Azure Event Hub namespace becomes congested, it reaches a point where its processing capacity is exceeded. This leads to consistent throttling of operations, as the system enforces limits to prevent overload. The high rate of throttling not only impacts event ingestion but also cascades into resource starvation, affecting downstream services that rely on timely event processing. Such congestion is typically caused by high message throughput, suboptimal configuration, or insufficient scaling to handle peak loads.
High Server Errors
Common causes for Azure Event Hub errors include quota exceeded (throughput or message size limits have been breached), partition or offset issues (consumers unable to connect or reading from invalid offsets), networking problems (connectivity issues due to firewall rules, DNS misconfigurations, or latency), service outage (regional Azure service disruption), and misconfigured access policies (incorrect SAS tokens, permissions, or authentication methods).
High User Errors
High user error rates in Azure Event Hub often stem from configuration issues, such as mismatched security credentials (for example Shared Access Signature (SAS) tokens), client SDK version incompatibilities, or throttling from overusing allocated resources. Insufficient permissions or quotas being exceeded can also trigger these errors. Another common cause is incorrect partition or consumer group usage, which can lead to connection limits being breached or messages being inaccessible.
Service
Authentication Rejection
The service is rejecting requests as unauthenticated or forbidden (HTTP 401/403, or gRPC UNAUTHENTICATED/PERMISSION_DENIED) across its callers. The problem is most likely in the service's own authentication or authorization setup rather than in any single client. Common causes are a changed token issuer, audience, or JWKS configuration, an expired or rotated signing key or certificate, a tightened authorization policy, or an identity provider becoming unavailable. This diagnosis also applies to services whose callers cannot be observed, such as an internet-facing gateway, when most requests are rejected. When only a single caller is rejected while others are served normally, see Unauthorized Access on that caller instead. Rejected users are locked out, which can explain a Concurrent Users Drop on the service's Business Application.
Client Connection Disruption
Existing client connections to the service were dropped, for example by a load balancer or proxy change. Clients then reconnect at the same time, which causes a reconnect storm. This predicts an Endpoint Request Burst and lost concurrent users. It can only be identified when the service's Business Application reports concurrent users.
Congested
The service is experiencing congestion, resulting in high latency for clients. This suggests that the system is unable to handle the current load efficiently, causing delays in response times. Congestion often occurs when the service receives more requests than it can handle within its capacity, leading to bottlenecks in processing. This may be due to insufficient resources (for example, CPU, memory, or bandwidth), unoptimized code, or a surge in traffic (for example, due to a sudden increase in demand or DDoS attack).
Malfunction
The Service is experiencing a high rate of errors, causing disruptions for clients. This can lead to degraded performance, failed requests, or complete service unavailability, significantly affecting the user experience.
Policy Rejection
The service is rejecting requests with HTTP 451 "Unavailable For Legal Reasons", meaning one of its own policies refuses to serve them. Clients receive errors even though the service and its dependencies are healthy. A sudden rise in 451 responses usually means a resource-gating, geo-blocking, legal-hold, or feature-flag rule was changed or rolled out and now covers more routes, tenants, or regions than intended. When only a single caller is rejected while others are served normally, see Access Rejected By Policy on that caller instead. Rejected users are locked out, which can explain a Concurrent Users Drop on the service's Business Application.
Throttling Rejection
The service is throttling requests with HTTP 429 "Too Many Requests" across its callers. The problem is most likely in the service's own rate limit or quota rather than in any single client's request rate. Common causes are a lowered rate limit or quota, a limiter that counts requests per instance and was sized for a different number of replicas, a shared limit that one feature or tenant now exhausts, or traffic that has grown past the configured limit. When only a single caller is throttled while others are served normally, see Access Throttled on that caller instead. When most or all callers are rejected, users can be locked out, which can explain a Concurrent Users Drop on the service's Business Application.