ccaas-status - Inbound Call Processing Failure – Incident details

The incident was caused by the number of concurrent active calls across tenants exceeding the capacity of a micro-service responsible for temporary call state storage.

This situation was triggered by an unexpectedly high-volume campaign workload from a tenant, contributing to a significant increase in overall system load.

While the underlying voice infrastructure had sufficient capacity to handle the traffic, the supporting micro-service reached its storage limit. This resulted in a processing blockage that prevented calls from being fully established.

Corrective action plan:
Migrate voice instance state storage from in-memory grain storage to blob-based storage

The underlying process responsible for the blockage was identified and stopped.

The blockage reoccurred, again impacting call handling.

The blockage was cleared, and calls began to be successfully processed. Monitoring continued.

A blockage in the call processing flow was identified, and remediation steps were initiated

Senior members of the development team were engaged for further investigation.

A broader system investigation observed an unusually high volume of outbound calls initiated by a campaign-based tenant, which was noted as a potential contributing factor and investigated further.

After initial checks, SRE rebooted the first affected voice instance and performed test calls. The system did not recover. During the reboot, all Graia tenants were automatically failed over to the secondary instance.