ccaas-status - Notice history

Graia team members observed recovered functionality.

The existing pods that were on the unreachable node were moved away from it.

The node pool was expanded with an additional node to take over the workload of the unresponsive one.

The SRE team discovered the fact that a node is unresponsive by noticing the unusual metrics it is exhibiting.

The incident was caused by a software defect within a third-party voice gateway component that was triggered under specific high-load conditions.

When the defect was triggered, the affected voice server was no longer able to process call-related activities correctly, resulting in disruption to voice client connectivity, call control functions, and audio handling for impacted calls

Service was restored following recovery actions on the affected voice server

Corrective Action Plan:
Improvements to the Call Probing Implementation
Deploy new Voice Gateway version 20.20.0 GA on QA
Enable Active-Active Voice Gateway functionality on QA
Deploy new Voice Gateway version 20.20.0 GA on Production
Enable Active-Active Voice Gateway functionality on Production for Instance 5-6

Follow-up testing confirmed that voice services had returned to normal operation.

Recovery actions were initiated to restore service.

Test calls confirmed an issue affecting voice call processing on the impacted server.

Follow-up testing confirmed that voice services had returned to normal operation.

Recovery actions were initiated to restore service.

Test calls confirmed an issue affecting voice call processing on the impacted server.

The incident was caused by a software defect within a third-party voice gateway component that was triggered under specific high-load conditions.

When the defect was triggered, the affected voice server was no longer able to process call-related activities correctly, resulting in disruption to voice client connectivity, call control functions, and audio handling for impacted calls

Service was restored following recovery actions on the affected voice server

Corrective Action Plan:

Improvements to the Call Probing Implementation

Deploy new Voice Gateway version 20.20.0 GA on QA

Deploy new Voice Gateway version 20.20.0 GA on Production

Enable Active-Active Voice Gateway functionality on QA

Enable Active-Active Voice Gateway functionality on Production

Follow-up testing confirmed that voice services had returned to normal operation.

Recovery actions were initiated to restore service.

Test calls confirmed an issue affecting voice call processing on the impacted server.

The incident was caused by a third-party voice gateway component becoming overloaded under specific high-load conditions.

This overload triggered a software defect in the component, leading to worker thread conflicts. As a result, voice client connectivity for affected calls was disrupted, causing loss of call control and audio handling failures.

Corrective Action Plan:
Call Probing Implementation
Automated Stuck Call Handling

The first successful call from the recovered service was observed.

A telephony service on one of the EU customer virtual machines experienced a software crash.
Automatic recovery restarted the affected service.

The incident was caused by a third-party voice gateway component becoming overloaded under specific high-load conditions.

This overload triggered a software defect in the component, leading to worker thread conflicts. As a result, voice client connectivity for affected calls was disrupted, causing loss of call control and audio handling failures.

Corrective Action Plan:

Call Probing Implementation

Automated Stuck Call Handling

Agents continued to experience effects from calls that were already active during the incident (e.g. calls stuck, no audio, inability to end or retrieve calls). New calls established after recovery were handled normally.

Successful outbound call from the recovered service was observed.

A telephony service on one of the EU customer virtual machines experienced a software crash.
Automatic recovery restarted the affected service.

The incident was caused by a low-level third-party voice gateway component becoming overloaded, resulting in loss of voice client connectivity for agents. Initial analysis indicates a potential software bug in the affected component related to worker thread collisions under a specific combination of call load conditions.

Although warning logs were generated by the affected service, the monitoring system did not detect the condition in time due to vendor-introduced log syntax changes that no longer matched the existing monitoring pattern.

Corrective action plan:

• Finalize voice gateway component bug assessment and apply fixes.

• Update and validate monitoring rules for the new log entry syntax.

This incident has been resolved. As a precaution, tenants continue to run on the secondary voice instance until the root cause analysis can be completed.

We recovered the affected voice instance and performed test calls. The condition causing the service disruption was identified, and successful test calls after moving a test tenant back to the original instance confirmed that the instance was operating normally again.

Tenants hosted on the affected voice instance were moved to the secondary instance while the investigation continues.