ccaas-status - CCaaS admin portal and agent UI access issues – Incident details

Microsoft Azure incident link: https://portal.azure.com/#view/Microsoft_Azure_Health/DetailsPage.ReactView/fromDeeplink~/false/index~/0/selectedEventSummary~/%7B%22trackingId%22%3A%22YKYN-BWZ%22%2C%22scope%22%3A%22Subscription%22%2C%22impactedSubscriptions%22%3A%5B%22d9441f55-a3fa-4061-b55b-ca448a0e31b5%22%2C%221179af07-7b6c-4e46-9e2b-03098cf8eed4%22%2C%22dc7b7526-8930-46dd-840a-f25b1a7b885e%22%2C%22b70acbd8-a6dd-4f03-b0ab-03ca408c1186%22%2C%22dcffb1fb-e926-4e1e-bbdb-1894128838b3%22%2C%22d447b22c-e460-48e8-9e28-07fec481a256%22%2C%222cf5c8cb-4ffd-4a24-a4f7-28546d926230%22%2C%2268a8d3a9-b4e7-4d9a-9df8-93cadc1c7c9c%22%2C%223334a0fe-ba9d-45e1-a47f-a39cd936a675%22%2C%22be9ef138-8832-49b7-a8c2-e50ffe747418%22%5D%7D/trackingId/YKYN-BWZ/impactedSubs~/%5B%22d9441f55-a3fa-4061-b55b-ca448a0e31b5%22%2C%221179af07-7b6c-4e46-9e2b-03098cf8eed4%22%2C%22dc7b7526-8930-46dd-840a-f25b1a7b885e%22%2C%22b70acbd8-a6dd-4f03-b0ab-03ca408c1186%22%2C%22dcffb1fb-e926-4e1e-bbdb-1894128838b3%22%2C%22d447b22c-e460-48e8-9e28-07fec481a256%22%2C%222cf5c8cb-4ffd-4a24-a4f7-28546d926230%22%2C%2268a8d3a9-b4e7-4d9a-9df8-93cadc1c7c9c%22%2C%223334a0fe-ba9d-45e1-a47f-a39cd936a675%22%2C%22be9ef138-8832-49b7-a8c2-e50ffe747418%22%5D/scope/Subscription

UPDATE 5: We have been seeing the web interfaces (admin portal and agent UI) loading reliably for the past approx. 1 hour. With this, the incident can be considered resolved.

UPDATE 4: At this stage, we anticipate full mitigation within the next four hours as Microsoft continues to recover nodes. This means we expect recovery to happen by 23:20 UTC on 29 October 2025. We will provide another update on our progress within two hours, or sooner if warranted.

UPDATE 3: We have now started to see positive responses when loading the admin portal and agent UI of Graia.

UPDATE 2: The rollback procedure has was initiated and is expected to be fully deployed in about 30 minutes after which we can expect to see signs of recovery.

UPDATE: Microsoft have identified the event that triggered the problem and are now working on rolling back to the last know good state. An ETA for when the rollback will be completed was not yet given, but the next update was estimated to come within the next 30 minutes.