Voice API Intermittent Failures (India Region)
Incident
Report for Plivo
Resolved
Incident Summary — Intermittent Voice API Failures (India Region)
Incident Window:
12 January, 10:30 AM IST — 11:46 AM IST
Impact:
Intermittent failures were observed across multiple Voice APIs in the India region, including make call API requests returning HTTP 502 errors. The service was not continuously unavailable, but experienced periodic degradation during the incident window.
Summary
On 12th January between 10:30 AM and 11:46 AM IST, intermittent failures were observed in Voice APIs in the India region. The service remained available for most of the time, but experienced periodic spikes of failures.
The incident was triggered by a spike in CPU utilization on Plivo’s CDR database, which reached 100%, driven by a high volume of list CDR API requests.
To contain the database overload, the rate limit quota for list CDR APIs was reduced. During this containment window, some list CDR API requests returned 5xx responses intermittently. These 5xx responses interacted incorrectly with the API Gateway’s circuit breaker logic.
The circuit breaker marked the upstream service as unhealthy but failed to route traffic to the configured failover upstream. This caused traffic to be dropped instead of being rerouted, resulting in intermittent 502 errors across a wider set of APIs, including the make call API.
Incident Window:
12 January, 10:30 AM IST — 11:46 AM IST
Impact:
Intermittent failures were observed across multiple Voice APIs in the India region, including make call API requests returning HTTP 502 errors. The service was not continuously unavailable, but experienced periodic degradation during the incident window.
Summary
On 12th January between 10:30 AM and 11:46 AM IST, intermittent failures were observed in Voice APIs in the India region. The service remained available for most of the time, but experienced periodic spikes of failures.
The incident was triggered by a spike in CPU utilization on Plivo’s CDR database, which reached 100%, driven by a high volume of list CDR API requests.
To contain the database overload, the rate limit quota for list CDR APIs was reduced. During this containment window, some list CDR API requests returned 5xx responses intermittently. These 5xx responses interacted incorrectly with the API Gateway’s circuit breaker logic.
The circuit breaker marked the upstream service as unhealthy but failed to route traffic to the configured failover upstream. This caused traffic to be dropped instead of being rerouted, resulting in intermittent 502 errors across a wider set of APIs, including the make call API.