ccaas-status - EU02 single-instance voice platform degradation – Incident details

EU02 single-instance voice platform degradation

ResolvedPartial outage 25%1 hour 37 minutes

Affected

Graia Contact Center as a Service

Partial outage from 12:23 PM to 1:00 PM, Operational from 1:00 PM to 2:00 PM

CCaaS Europe - Voice Services

Partial outage from 12:23 PM to 1:00 PM, Operational from 1:00 PM to 2:00 PM

  • PostmortemUTC

    The incident was caused by a low-level third-party voice gateway component becoming overloaded, resulting in loss of voice client connectivity for agents. Initial analysis indicates a potential software bug in the affected component related to worker thread collisions under a specific combination of call load conditions.

    Although warning logs were generated by the affected service, the monitoring system did not detect the condition in time due to vendor-introduced log syntax changes that no longer matched the existing monitoring pattern.


    Corrective action plan:

    • Finalize voice gateway component bug assessment and apply fixes.

    • Update and validate monitoring rules for the new log entry syntax.

  • ResolvedUTC

    This incident has been resolved. As a precaution, tenants continue to run on the secondary voice instance until the root cause analysis can be completed.

  • MonitoringUTC

    We recovered the affected voice instance and performed test calls. The condition causing the service disruption was identified, and successful test calls after moving a test tenant back to the original instance confirmed that the instance was operating normally again.

  • IdentifiedUTC

    Tenants hosted on the affected voice instance were moved to the secondary instance while the investigation continues.

  • InvestigatingUTC

    We are currently investigating the reported incident that some customers are unable to handle inbound voice calls. This is affecting all agents of the affected customers but only customers hosted on one of our voice server instances.