ccaas-status - Notice history

Related DevOps workitems: Task 141787: Investigate VE EXT3 connector logs Bug 141817: VE Connector unregisters inbound numbers unexpectedly when SignalR client is reconnecting

Microsoft Azure incident link: https://portal.azure.com/#view/Microsoft_Azure_Health/DetailsPage.ReactView/fromDeeplink~/false/index~/0/selectedEventSummary~/%7B%22trackingId%22%3A%22YKYN-BWZ%22%2C%22scope%22%3A%22Subscription%22%2C%22impactedSubscriptions%22%3A%5B%22d9441f55-a3fa-4061-b55b-ca448a0e31b5%22%2C%221179af07-7b6c-4e46-9e2b-03098cf8eed4%22%2C%22dc7b7526-8930-46dd-840a-f25b1a7b885e%22%2C%22b70acbd8-a6dd-4f03-b0ab-03ca408c1186%22%2C%22dcffb1fb-e926-4e1e-bbdb-1894128838b3%22%2C%22d447b22c-e460-48e8-9e28-07fec481a256%22%2C%222cf5c8cb-4ffd-4a24-a4f7-28546d926230%22%2C%2268a8d3a9-b4e7-4d9a-9df8-93cadc1c7c9c%22%2C%223334a0fe-ba9d-45e1-a47f-a39cd936a675%22%2C%22be9ef138-8832-49b7-a8c2-e50ffe747418%22%5D%7D/trackingId/YKYN-BWZ/impactedSubs~/%5B%22d9441f55-a3fa-4061-b55b-ca448a0e31b5%22%2C%221179af07-7b6c-4e46-9e2b-03098cf8eed4%22%2C%22dc7b7526-8930-46dd-840a-f25b1a7b885e%22%2C%22b70acbd8-a6dd-4f03-b0ab-03ca408c1186%22%2C%22dcffb1fb-e926-4e1e-bbdb-1894128838b3%22%2C%22d447b22c-e460-48e8-9e28-07fec481a256%22%2C%222cf5c8cb-4ffd-4a24-a4f7-28546d926230%22%2C%2268a8d3a9-b4e7-4d9a-9df8-93cadc1c7c9c%22%2C%223334a0fe-ba9d-45e1-a47f-a39cd936a675%22%2C%22be9ef138-8832-49b7-a8c2-e50ffe747418%22%5D/scope/Subscription

UPDATE 5: We have been seeing the web interfaces (admin portal and agent UI) loading reliably for the past approx. 1 hour. With this, the incident can be considered resolved.

UPDATE 4: At this stage, we anticipate full mitigation within the next four hours as Microsoft continues to recover nodes. This means we expect recovery to happen by 23:20 UTC on 29 October 2025. We will provide another update on our progress within two hours, or sooner if warranted.

UPDATE 3: We have now started to see positive responses when loading the admin portal and agent UI of Graia.

UPDATE 2: The rollback procedure has was initiated and is expected to be fully deployed in about 30 minutes after which we can expect to see signs of recovery.

UPDATE: Microsoft have identified the event that triggered the problem and are now working on rolling back to the last know good state. An ETA for when the rollback will be completed was not yet given, but the next update was estimated to come within the next 30 minutes.

The issue was mitigated on Microsoft side. This incident can be closed.

Microsoft states that the issue was resolved on their end. Keeping under observation.

Microsoft acknowledged the issue and is working on mitigating the impact. MS reference number: QNBQ-5W8

Related DevOps workitems: Incident 138175: Inbound emails not arriving into Graia Bug 136453: emailbots pod using 3.3 GiB

Root Cause: With the release of a bugfix pertaining to the emailbots service, another issue was unfortunately introduced: the “Wait For Customer Reply” email workflow node works in a way that it periodically checks whether the awaited reply has arrived or the timeout reached. If it needs to wait some more, it signals this with delayed queue messages. The introduced bug was that in such cases, the service not only queued a new message but it also requeued the original one. This led to exponential growth of messages in the queue. Because the messages are drained sequentially, newer messages that got introduced after the exponentially duplicated ones were waiting for their turn to be processed. This meant the email conversations were stuck at a point in their workflow that required the processing of messages in the queue.

The incident has been resolved