ccaas-status - Notice history

First occurrence of the pertinent error message in Vivacom's log file "UGBuzzeasyCiscoConnector-20250221.log": 2025-02-21 12:03:25.963 [Error] UGBuzzeasyCiscoConnector.HostedServices.CiscoApplicationService error requesting entry point (1009) configuration (status:Unauthorized) {"code":"401-109","developerMessage":"Invalid or expired 'Authorization' header provided. New access token is required."} Log line in the same file that indicated the renewal of the token. Note that this is missing the beginning of the line but happened some time between 2025-02-21 12:50:39 and 2025-02-21 12:50:20: 6be06f1d09c4","expires_in":86400,"token_type":"Bearer"}

This incident is now resolved.

The incident is affecting only one particular customer. We performed remediation steps and are now testing.

The historical data loading issue has now been resolved. Please try refreshing your webpage if you do not yet see the historical data loading.

Root Cause Communication between CiscoConnector and Cisco was interrupted because the access token used to authenticate communication was considered invalid by one of the parties. CiscoConnector used accompanied its messages with a token that Cisco already considered invalid. For more details, see the RCA document titled "Buzzeasy Root Cause Analysis Report - 2025-01-24 Vivacom undelivered messages.docx"

Root Cause Due to the doubling of the underlying resources (all app service connectors were duplicated between the two instances), chat conversations were distributed between the two logical servers. A conversation existed on one server, meaning that when that conversation was referenced on the other server, it could not be found. For more details see the RCA document titled "Buzzeasy Root Cause Analysis Report - 2025-01-25 United Group projects, undelivered agent messages.docx"

Some customers reported voice call related issues including agents being immediately put into the RONA state when receiving a call. This was caused by a service queue filling up on one of the PBX servers in the EU high availability pair. The incident was noticed with a delay as the impact of the outage was partial and the secondary server, as it was supposed to, handled a lot of the voice traffic in the mean time.

Root Cause Communication between CiscoConnector and Cisco was interrupted because the access token used to authenticate communication was considered invalid by one of the parties. CiscoConnector used accompanied its messages with a token that Cisco already considered invalid. For more details, see the RCA document titled "Buzzeasy Root Cause Analysis Report - 2025-01-24 Vivacom undelivered messages.docx"

On 17/12/2024 we had a performance issue with our live data processor service that was unable to handle all system messages. This did not affect customer interactions (phone calls, emails, webchat sessions with customers were all delivered to agents), but supervisor reports on Portal Teams report might have shown outdated Workitems (Tasks) on agents' cards or Agent UI showed Agent Performance Dashboard with outdated data. After the internal alerting mechanism triggered an alarm, the wider team got involved and we managed to rectify the fact the there were wrongfully lingering agent tasks on the portal's Teams reports.