Coveo Rabbit MQ on the Master CES server are getting loaded with over a million items and its causing CES to consume all the harddrive space on the CES Server.
Soon after that the indexes go into the Read only mode.
We tried deleting the million plus queue items but they keep reloading.
Has anyone else faced this?
What we discovered after hours of troubleshooting is that there was a corrupted message in the queue. When we removed that one message, the rest of the messages were processed properly.
I had to re-implement https://www.akshaysura.com/2015/05/01/add-max-message-limit-for-rabbitmq-for-coveo/ on the newer version of the CES and RabbitMQ in order to limit the damage in the future.
When RabbitMQ queues start to pile up messages, it means there are no more consumers connected to the queues. Your Coveo Enterprise Search sources might have encountered errors and disconnected. You can manually reconnect them by disabling and re-enabling live indexing on the Queue Sitecore sources. They will resume their message consumption and the situation will resolve.
Unfortunately, if the index is already in read-only mode due to a lack of disk space, you will have to free up space, and put the index in read-write mode manually beforehand.
