Has anyone had an issue that you have multiple worker processes under application pool which are trying to write the data under the data/index folder and that are colliding with each other?
Your Sitecore application should never use the same
If they do, they will override their indexes every time while index is rebuilt.
I'm guessing that's exactly what's happening in your scenario - one application tries to update its index and another one tries to perform another operation on index files in the same moment. It's not supported.
Your indexes should be kept in separate location for every Sitecore application.
I think we have finally figured out what is happening.
We have scheduled recycle of the application pool at a certain time during the night but what sometimes happens is that indexing of the database is still in progress. When the recycle happens, worker process doesn't shut down and the second worker process gets created.