ccaas-status - Graia admin portal is slow to load – Incident details

All systems operational

Graia admin portal is slow to load

Resolved
Degraded performance
Started about 1 year agoLasted 2 hours 58 minutes

Affected

Graia Contact Center as a Service

Degraded performance from 8:38 AM to 11:36 AM

CCaaS International - Admin Portal

Degraded performance from 8:38 AM to 11:36 AM

CCaaS International - Agent UI

Degraded performance from 8:38 AM to 11:36 AM

Updates
  • Update
    UTC
    Update

    Related DevOps workitems: Incident 135456: Graia Portal is not loading Product Backlog Item 135490: Make sure Portal can function even if recording manager is down

  • Update
    UTC
    Update

    3 calls on 2 different tenants (MPLUS Turkey and Geomant SI Test) of the production system had been stuck in the active state for an extremely long time (10+ hours). These calls had also been recorded generating files exceeding 1 GB in size. When the recording-manager service tried to process these recordings which involved loading them into memory, the maximum allowed memory usage was reached and crashed the service making it have to restart each time this happened.

  • Update
    UTC
    Update

    The incident has been resolved

  • Update
    UTC
    Update

    We have resolved the problem that was affecting our we interfaces' availability. Furthermore, we have been monitoring the pertinent service and noticed no more relapses in the past 1.5 hours. This incident is now closing. We apologize for the interruption.

  • Update
    UTC
    Update

    We are still experiencing issues with the Graia admin portal. Some users may encounter slow or infinite loading. Our team is actively investigating and working on a fix. We’ll provide further updates as soon as possible.

  • Update
    UTC
    Update

    The problem has been resolved and we're continuing to monitor the system for potential relapses.

  • Investigating
    UTC
    Investigating

    Root cause: A number of unusually long-running calls (10+ hours) produced files that when needing to be processed, due to their sheer size, occupied the services of a component that is also involved in loading the web interfaces. The files are usually small enough to be handled in the matter of less than a seconds. However, the oversized files kept the service occupied for longer resulting in delays in loading the web interfaces. Prevention: We cleared the files that were causing the issue and to prevent this situation in future, we are introducing a conversation duration limit for voice calls. Regards, Graia team