Errors while accessing the management console
Incident
Report for Plivo
Postmortem
ROOT CAUSE ANALYSIS
Issue: Console Outage
Date and Time
From: Oct 16 2018 9:30 am UTC
To: Oct 16 2018 9:35 am UTC
Services affected
Management console
Event Description
9:30 am UTC - We found that Console is showing error message on all the pages. The server started running in development mode after a deployment due to a wrong configuration picked from a stale settings file.
9:31 am UTC - Clean build and deployment was invoked and servers were restarted.
9:35 am UTC - Deployment completed and issue was resolved.
Corrective Action
- As part of build and deployment process, working directory is cleaned up by the build tool, but in this case that cleanup didn’t happen leaving the stale file. To avoid this in future, we are now also doing a explicit cleanup via command line during the build process.
Resolved
We had identified elevated errors while trying to access the management console. This happened due to a deployment that picked up an incorrect configuration.
Start time: Oct 16 2018 9:30 am UTC
End time: Oct 16 2018 9:35 am UTC
Our engineers were able to fix the issue as soon as our systems detected the failures.
Start time: Oct 16 2018 9:30 am UTC
End time: Oct 16 2018 9:35 am UTC
Our engineers were able to fix the issue as soon as our systems detected the failures.