ROOT CAUSE ANALYSIS

Issue

Multiple Plivo services, such as SMS API, Inbound MMS API, Console, Number Purchase, and Lookup API, faced full or partial interruptions for different lengths of time. Voice API and Zentrunk services remained unaffected.

Duration

From: 2023-02-02 08:40 AM UTC

To: 2023-02-02 05:15 PM UTC

Services affected

Description

At 2023-02-02 08:40 AM UTC, Plivo deployed a release to its billing platform. Right after this deployment, the service in the billing platform, which is responsible for maintaining Plivo account balance deductions for prepaid customers’ accounts, became unresponsive.

‌

This incident led to several Plivo services (namely: SMS API, Inbound MMS, Console, Phone Number APIs, and Lookup API) not being able to communicate with the billing platform. Without confirmation that the account balance was successfully deducted from prepaid customers’ accounts, the above services started experiencing failures.

‌

Other Plivo services (Voice API and Zentrunk) were not affected because a circuit breaker was implemented to prevent this failure. Unfortunately, the affected Plivo services did not have this failsafe in place.

‌

As part of the follow-up root cause analysis, it was identified that the issue was caused by a very high CPU utilization (99%) on the Redis cluster used by the billing platform despite the overall volume of the Plivo API’s traffic not increasing substantially.

Our team immediately rolled back the last deployment with no effect. The Redis cluster was still showing an abnormally high CPU utilization. As a next step, the team increased the number of nodes in the Redis cluster, but the issue persisted. The team further created a new Redis cluster and switched the billing platform to this new cluster with no success. The CPU utilization was still abnormally high on the new cluster.

‌

While the underlying issue was being investigated, temporary measures were undertaken in parallel to restore services wherever possible. The SMS API, Plivo Console, and Inbound MMS services were restored by skipping the account balance deduction confirmation from the billing platform.

‌

Further investigation revealed that the billing platform service triggered ​​too many Redis commands named ‘COMMAND’. Our team observed that despite having a Redis connection pool set up in the billing platform, new connections to the Redis cluster were constantly created for all Redis commands.

‌

Finally, after further investigation, the team identified that the Golang Redis client (go-redis) version used had several issues related to initializing new connections with Redis cluster engine v6. Upgrading the Golang Redis client version from v6 to v7 post-regression and performance tests fixed the issue around 2023-02-02 05:15 PM UTC.

Root Cause Analysis

Chronology of events

• 2023-02-02 08:40 AM UTC: The Plivo billing platform was updated, which led to failures in credit balance operations, causing multiple Plivo services to be affected.
• 2023-02-02 09:00 AM UTC: The deployment was reverted. However, the billing platform service stayed unresponsive.
• 2023-02-02 09:00 AM UTC: A temporary fix was deployed to restore SMS API & Inbound MMS services by processing messages without balance deductions.
• 2023-02-02 03:40 PM UTC: The engineering team finally identified the root cause: the Go Redis client (go-redis) connection pooling implementation was incompatible with the Redis cluster v6.
• 2023-02-02 04:35 PM UTC: A temporary fix was deployed to make the Plivo Console accessible.2023-02-02 05:15 PM UTC: Corrective upgrades were tested and deployed in production to resolve the incident.

Corrective and preventive action plan

• Conduct compatibility checks across the production setup to ensure the system can handle similar instances without impacting services.
• Conduct performance tests on a production scale, and validate the test metrics, such as latency, functionality, CPU and memory utilization, and throughput, using actual production data.
• Audit all Plivo services and implement the circuit breaker pattern for every service, including SMS API, Inbound MMS, Plivo Console, and Lookup APIs to minimize related services downtime.
• Ensure that impact mitigation is integrated into the incident response process and that it is considered at every stage, from incident detection to resolution.

Technical references

• The Redis COMMAND: https://redis.io/commands/command/
• Go-redis high latency issue: https://www.sobyte.net/post/2022-06/go-redis-server/
• Redis engine v5 -> v6 incompatible changes: https://github.com/redis/redis/commit/c5e717c637cbb1c80e1259560ebf995fb7920628
• Go-redis fix merged in v7 and v8: https://github.com/redis/go-redis/pull/1355