ccaas-status - Notice history

Related DevOps work items: Incident 126596: Inbound is down for Affidea ES and MPLUS Serbia (Ananas)- IVR not routing calls to loggedin staffed agents Bug 126599: Conversations fails to enter the queue

The incident has been resolved

We've discovered the cause of the issue and rolled out a hotfix. Calls should be able to reach agents now.

First occurrence of the pertinent error message in Vivacom's log file "UGBuzzeasyCiscoConnector-20250221.log": 2025-02-21 12:03:25.963 [Error] UGBuzzeasyCiscoConnector.HostedServices.CiscoApplicationService error requesting entry point (1009) configuration (status:Unauthorized) {"code":"401-109","developerMessage":"Invalid or expired 'Authorization' header provided. New access token is required."} Log line in the same file that indicated the renewal of the token. Note that this is missing the beginning of the line but happened some time between 2025-02-21 12:50:39 and 2025-02-21 12:50:20: 6be06f1d09c4","expires_in":86400,"token_type":"Bearer"}

This incident is now resolved.

The incident is affecting only one particular customer. We performed remediation steps and are now testing.

The historical data loading issue has now been resolved. Please try refreshing your webpage if you do not yet see the historical data loading.

Root Cause Communication between CiscoConnector and Cisco was interrupted because the access token used to authenticate communication was considered invalid by one of the parties. CiscoConnector used accompanied its messages with a token that Cisco already considered invalid. For more details, see the RCA document titled "Buzzeasy Root Cause Analysis Report - 2025-01-24 Vivacom undelivered messages.docx"

Root Cause Due to the doubling of the underlying resources (all app service connectors were duplicated between the two instances), chat conversations were distributed between the two logical servers. A conversation existed on one server, meaning that when that conversation was referenced on the other server, it could not be found. For more details see the RCA document titled "Buzzeasy Root Cause Analysis Report - 2025-01-25 United Group projects, undelivered agent messages.docx"

Some customers reported voice call related issues including agents being immediately put into the RONA state when receiving a call. This was caused by a service queue filling up on one of the PBX servers in the EU high availability pair. The incident was noticed with a delay as the impact of the outage was partial and the secondary server, as it was supposed to, handled a lot of the voice traffic in the mean time.

Root Cause Communication between CiscoConnector and Cisco was interrupted because the access token used to authenticate communication was considered invalid by one of the parties. CiscoConnector used accompanied its messages with a token that Cisco already considered invalid. For more details, see the RCA document titled "Buzzeasy Root Cause Analysis Report - 2025-01-24 Vivacom undelivered messages.docx"