ccaas-status - Notice history

The incident was caused by a third-party voice gateway component becoming overloaded under specific high-load conditions.

This overload triggered a software defect in the component, leading to worker thread conflicts. As a result, voice client connectivity for affected calls was disrupted, causing loss of call control and audio handling failures.

Corrective Action Plan:
Call Probing Implementation
Automated Stuck Call Handling

The first successful call from the recovered service was observed.

A telephony service on one of the EU customer virtual machines experienced a software crash.
Automatic recovery restarted the affected service.

The incident was caused by a third-party voice gateway component becoming overloaded under specific high-load conditions.

This overload triggered a software defect in the component, leading to worker thread conflicts. As a result, voice client connectivity for affected calls was disrupted, causing loss of call control and audio handling failures.

Corrective Action Plan:

Call Probing Implementation

Automated Stuck Call Handling

Agents continued to experience effects from calls that were already active during the incident (e.g. calls stuck, no audio, inability to end or retrieve calls). New calls established after recovery were handled normally.

Successful outbound call from the recovered service was observed.

A telephony service on one of the EU customer virtual machines experienced a software crash.
Automatic recovery restarted the affected service.

The incident was caused by a low-level third-party voice gateway component becoming overloaded, resulting in loss of voice client connectivity for agents. Initial analysis indicates a potential software bug in the affected component related to worker thread collisions under a specific combination of call load conditions.

Although warning logs were generated by the affected service, the monitoring system did not detect the condition in time due to vendor-introduced log syntax changes that no longer matched the existing monitoring pattern.

Corrective action plan:

• Finalize voice gateway component bug assessment and apply fixes.

• Update and validate monitoring rules for the new log entry syntax.

This incident has been resolved. As a precaution, tenants continue to run on the secondary voice instance until the root cause analysis can be completed.

We recovered the affected voice instance and performed test calls. The condition causing the service disruption was identified, and successful test calls after moving a test tenant back to the original instance confirmed that the instance was operating normally again.

Tenants hosted on the affected voice instance were moved to the secondary instance while the investigation continues.

The issue was caused by a backlog in the voice processing layer, which led to internal call queues becoming saturated. As a result, new calls could not be established, and existing sessions could not be properly terminated. 

Further investigation showed that this condition was triggered by a recently deployed minor version update to a third-party telephony component within the voice server (applied during the maintenance window on April 18, 2026). This version exhibited stability issues when handling higher call volumes, leading to the observed behaviour. 

Corrective action plan:
Rollback to the previous stable version of the voice server software.

The incident will now be considered closed with the root cause to be announced when we conclude the investigation. In our status page's terms, this will come as a 'Postmortem' update to the incident.

We recovered functionality of the affected service and are continuing to monitor the system.

We identified the problem and have started working on recovering functionality.

The issue was caused by a backlog in the voice processing layer, which led to internal call queues becoming saturated. As a result, new calls could not be established, and existing sessions could not be properly terminated. 

Further investigation showed that this condition was triggered by a recently deployed minor version update to a third-party telephony component within the voice server (applied during the maintenance window on April 18, 2026). This version exhibited stability issues when handling higher call volumes, leading to the observed behaviour. 

Corrective action plan:
Rollback to the previous stable version of the voice server software.

The incident will now be considered closed with the root cause to be announced when we conclude the investigation. In our status page's terms, this will come as a 'Postmortem' update to the incident.

We recovered functionality of the affected service and are continuing to monitor the system.

Relevant DevOps workitems: Task 148105: GMNT#2026031010000221 — Complete failure Graia