The general, the task is to denormalize 3 indexes into 1. Please, don't ask why I use SOLR as a data source here.
Imagine I have 3 normalized indexes:
- index1 with fields: id, name, type, address
- index2 with fields: id, language, description
- index3 with fields: id, image
index1 and index3 doesn't contain info about the language and are not language-related indexes. index2 can contain information for different languages. All the indexes describe different aspects of one entity and are filled differently.
What I need is to combine the data from 3 indexes into 1, denormalized. Let's call it denormalized_index and it has fields: id, name, type, address, language, description, image
Currently, I use a custom Sitecore crawler to fetch the data on denormalized_index rebuild, from 3 indexes, combine them and return the values to the index. The operation of combining is pretty fast, about a minute for 200k documents. But the flushing itself to SOLR takes 50-100 mins depending on the local or distributed env.
I've already tried to finetune batch size and made it use parallel computing, but it doesn't make any significant difference (the parallel does its thing but I am limited by the number of processors, can't make the number larger). I investigated SOLR functionality, too. But the most suitable option is to use "MERGEINDEXES" function of SOLR which just copies the documents, not merging them. Writing my own SOLR module is not an option for me (at least if you can provide a step-by-step guide :D).
Are there any best practices for making this process quicker by changing different settings or by tweaking Sitecore?
