Root cause:
Tunstall Healthcare UK had a voice workflow that included a "Transfer to Channel" node. The node transferred calls to the same channel that used this workflow thus creating a call routing loop. This in itself is generally bad practice in workflow design and leads to undesirable outcomes, but what had an equally contributing part in the degraded performance or all tenants was the fact that a few customer calls (4-5 unique phone numbers) were such that they kept sending DTMF tones non-stop. These varying but somehow always invalid (from the menu's perspective) DTMF choices made the loops taken around the workflow very quick: 1-2 seconds per each loop. Because during a transfer to another channel, a whole new conversation is built, all these transfers each resulted in a new conversation stifling the system's resources with a few thousand calls in a short period.
