4

Coveo Rabbit MQ on the Master CES server are getting loaded with over a million items and its causing CES to consume all the harddrive space on the CES Server.

Soon after that the indexes go into the Read only mode.

We tried deleting the million plus queue items but they keep reloading.

Has anyone else faced this?

  • does your CES logs show the indexing calls? anything useful there? – Diego Oct 7 '16 at 3:23
  • CES logs and sitecore logs do not show any errors. – ASura Oct 7 '16 at 14:22
6

What we discovered after hours of troubleshooting is that there was a corrupted message in the queue. When we removed that one message, the rest of the messages were processed properly.

I had to re-implement https://www.akshaysura.com/2015/05/01/add-max-message-limit-for-rabbitmq-for-coveo/ on the newer version of the CES and RabbitMQ in order to limit the damage in the future.

| improve this answer | |
  • You consumed the corrupted message in the Queues >> [The queue stuck] >> Get Message section correct? Just for the records. – Simon Langevin Oct 13 '16 at 18:23
  • Yes that is correct. – ASura Oct 13 '16 at 18:26
3

When RabbitMQ queues start to pile up messages, it means there are no more consumers connected to the queues. Your Coveo Enterprise Search sources might have encountered errors and disconnected. You can manually reconnect them by disabling and re-enabling live indexing on the Queue Sitecore sources. They will resume their message consumption and the situation will resolve.

Unfortunately, if the index is already in read-only mode due to a lack of disk space, you will have to free up space, and put the index in read-write mode manually beforehand.

| improve this answer | |
  • CES was running fine, we tried recreating the queues and even the index. – ASura Oct 11 '16 at 15:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.