Root Cause:
With the release of a bugfix pertaining to the emailbots service, another issue was unfortunately introduced: the “Wait For Customer Reply” email workflow node works in a way that it periodically checks whether the awaited reply has arrived or the timeout reached. If it needs to wait some more, it signals this with delayed queue messages. The introduced bug was that in such cases, the service not only queued a new message but it also requeued the original one. This led to exponential growth of messages in the queue.
Because the messages are drained sequentially, newer messages that got introduced after the exponentially duplicated ones were waiting for their turn to be processed. This meant the email conversations were stuck at a point in their workflow that required the processing of messages in the queue.
