Issue with Spoke Platform due to degradation in US East data center and upstream carrier networks
Incident Report for Spoke Phone
Resolved
The issue has been resolved and all services are operating normally at this time.
Posted Dec 08, 2021 - 21:17 NZDT
Update
Spoke call volumes and answer rates have continued at historic baseline levels for the past 2 hours and we have continued to seeing significantly reduced errors and API latencies. Access to the Spoke account portal has recovered. Customers are able to successfully make and receive calls on the Spoke application. We are moving this incident to "monitoring" and will continue to closely monitor for system stability. We will post an update in the next 4 hours or as soon as more information becomes available.
Posted Dec 08, 2021 - 17:00 NZDT
Monitoring
Our US East region has mitigated the underlying network issue and are now working to recover individual services. Most services that the Spoke platform depends on are fully or almost fully recovered. Spoke call volumes and answer rates have returned to historic baseline levels and we are seeing significantly reduced errors and API latencies. Access to the Spoke account portal has recovered. Customers should be able to successfully make and receive calls on the Spoke application. We are moving this incident to "monitoring" and will continue to closely monitor for system stability.
Posted Dec 08, 2021 - 14:31 NZDT
Update
Our US East data center has mitigated the underlying network issue and are now working to recover individual services. Many services have already recovered however some services that Spoke depends upon are still undergoing recovery. We continue to see reduced error rates across the platform with call disruptions reducing significantly in the last 30 minutes. Access to the Spoke account portal remains intermittent. We continue to monitor the issue and expect to provide another update in 1 hour or as soon as more information becomes available.
Posted Dec 08, 2021 - 12:38 NZDT
Update
Services continue to recover and we are starting to see reduced error rates on the platform, however access to the Spoke applications and account portal is still intermittent. Voice calls remain partially disrupted, with some calls failing to successfully connect. We continue to monitor the issue and expect to provide another update in 1 hour or as soon as more information becomes available.
Posted Dec 08, 2021 - 11:59 NZDT
Update
We are starting to see partial recovery of services across the platform, however access to the Spoke applications and account portal is still intermittent. Calls also continue to be disrupted, with API timeouts due to data center network issues preventing some calls from connecting. We continue to monitor the issue and expect to provide another update in 30 minutes or as soon as more information becomes available.
Posted Dec 08, 2021 - 10:49 NZDT
Update
We are continuing to work on a fix for this issue.
Posted Dec 08, 2021 - 09:43 NZDT
Update
We are starting to see partial recovery of services across the platform, however access to the Spoke applications and account portal is still intermittent. Calls also continue to be disrupted, with API timeouts due to data center network issues preventing some calls from connecting. We continue to monitor the issue and expect to provide another update in 30 minutes or as soon as more information becomes available.
Posted Dec 08, 2021 - 09:43 NZDT
Update
Our data center provider has identified the root cause of the issue which is due to impairment of several network devices in the US East 1 region. This issue is also impacting our upstream carrier. The data center provider is pursuing multiple mitigation paths in parallel and have started to see some signs of recovery. We do not have an ETA for full recovery at this time. We continue to monitor the issue and expect to provide another update in 30 minutes or as soon as more information becomes available.
Posted Dec 08, 2021 - 08:56 NZDT
Update
We have identified the cause of of the issue in our US East data center and with our upstream carrier causing a service interruption with the Spoke platform. We are monitoring the issue and expect to provide another update in 30 minutes or as soon as more information becomes available.
Posted Dec 08, 2021 - 08:22 NZDT
Update
We have identified the cause of of the issue in our US East data center and with our upstream carrier causing a service interruption with the Spoke platform. We are monitoring the issue and expect to provide another update in 30 minutes or as soon as more information becomes available.
Posted Dec 08, 2021 - 07:02 NZDT
Identified
We have identified a service interruption with the Spoke platform due to degradation in our US East data center, and upstream carrier. This interruption impacts access to the Spoke application and account portal. We are monitoring the issue and expect to provide another update in 30 minutes or as soon as more information becomes available.
Posted Dec 08, 2021 - 06:33 NZDT
This incident affected: Spoke Applications (Account Portal, Spoke Mobile Application (iOS), Spoke Mobile Application (Android), Spoke Desktop App for Mac and Windows), Upstream Carriers (Twilio Voice, North America, Twilio Voice, APAC), and Voice (Conferencing, PSTN, Twilio REST API).