ccaas-status - Notice history

Microsoft Azure incident link: https://portal.azure.com/#view/Microsoft_Azure_Health/DetailsPage.ReactView/fromDeeplink~/false/index~/0/selectedEventSummary~/%7B%22trackingId%22%3A%22YKYN-BWZ%22%2C%22scope%22%3A%22Subscription%22%2C%22impactedSubscriptions%22%3A%5B%22d9441f55-a3fa-4061-b55b-ca448a0e31b5%22%2C%221179af07-7b6c-4e46-9e2b-03098cf8eed4%22%2C%22dc7b7526-8930-46dd-840a-f25b1a7b885e%22%2C%22b70acbd8-a6dd-4f03-b0ab-03ca408c1186%22%2C%22dcffb1fb-e926-4e1e-bbdb-1894128838b3%22%2C%22d447b22c-e460-48e8-9e28-07fec481a256%22%2C%222cf5c8cb-4ffd-4a24-a4f7-28546d926230%22%2C%2268a8d3a9-b4e7-4d9a-9df8-93cadc1c7c9c%22%2C%223334a0fe-ba9d-45e1-a47f-a39cd936a675%22%2C%22be9ef138-8832-49b7-a8c2-e50ffe747418%22%5D%7D/trackingId/YKYN-BWZ/impactedSubs~/%5B%22d9441f55-a3fa-4061-b55b-ca448a0e31b5%22%2C%221179af07-7b6c-4e46-9e2b-03098cf8eed4%22%2C%22dc7b7526-8930-46dd-840a-f25b1a7b885e%22%2C%22b70acbd8-a6dd-4f03-b0ab-03ca408c1186%22%2C%22dcffb1fb-e926-4e1e-bbdb-1894128838b3%22%2C%22d447b22c-e460-48e8-9e28-07fec481a256%22%2C%222cf5c8cb-4ffd-4a24-a4f7-28546d926230%22%2C%2268a8d3a9-b4e7-4d9a-9df8-93cadc1c7c9c%22%2C%223334a0fe-ba9d-45e1-a47f-a39cd936a675%22%2C%22be9ef138-8832-49b7-a8c2-e50ffe747418%22%5D/scope/Subscription

UPDATE 5: We have been seeing the web interfaces (admin portal and agent UI) loading reliably for the past approx. 1 hour. With this, the incident can be considered resolved.

UPDATE 4: At this stage, we anticipate full mitigation within the next four hours as Microsoft continues to recover nodes. This means we expect recovery to happen by 23:20 UTC on 29 October 2025. We will provide another update on our progress within two hours, or sooner if warranted.

UPDATE 3: We have now started to see positive responses when loading the admin portal and agent UI of Graia.

UPDATE 2: The rollback procedure has was initiated and is expected to be fully deployed in about 30 minutes after which we can expect to see signs of recovery.

UPDATE: Microsoft have identified the event that triggered the problem and are now working on rolling back to the last know good state. An ETA for when the rollback will be completed was not yet given, but the next update was estimated to come within the next 30 minutes.

The issue was mitigated on Microsoft side. This incident can be closed.

Microsoft states that the issue was resolved on their end. Keeping under observation.

Microsoft acknowledged the issue and is working on mitigating the impact. MS reference number: QNBQ-5W8

Related DevOps workitems: Incident 138175: Inbound emails not arriving into Graia Bug 136453: emailbots pod using 3.3 GiB

Root Cause: With the release of a bugfix pertaining to the emailbots service, another issue was unfortunately introduced: the “Wait For Customer Reply” email workflow node works in a way that it periodically checks whether the awaited reply has arrived or the timeout reached. If it needs to wait some more, it signals this with delayed queue messages. The introduced bug was that in such cases, the service not only queued a new message but it also requeued the original one. This led to exponential growth of messages in the queue. Because the messages are drained sequentially, newer messages that got introduced after the exponentially duplicated ones were waiting for their turn to be processed. This meant the email conversations were stuck at a point in their workflow that required the processing of messages in the queue.

The incident has been resolved

Related DevOps workitems: Incident 135456: Graia Portal is not loading Product Backlog Item 135490: Make sure Portal can function even if recording manager is down

3 calls on 2 different tenants (MPLUS Turkey and Geomant SI Test) of the production system had been stuck in the active state for an extremely long time (10+ hours). These calls had also been recorded generating files exceeding 1 GB in size. When the recording-manager service tried to process these recordings which involved loading them into memory, the maximum allowed memory usage was reached and crashed the service making it have to restart each time this happened.

The incident has been resolved

We have resolved the problem that was affecting our we interfaces' availability. Furthermore, we have been monitoring the pertinent service and noticed no more relapses in the past 1.5 hours. This incident is now closing. We apologize for the interruption.

We are still experiencing issues with the Graia admin portal. Some users may encounter slow or infinite loading. Our team is actively investigating and working on a fix. We’ll provide further updates as soon as possible.

The problem has been resolved and we're continuing to monitor the system for potential relapses.

Related DevOps workitems: Bug 133816: EU Asterisk1 TP PJSIP/Distributor queue is full and TP Stasis/m:rtp queue is full