Sitecore keeps a local copy of the lucene index in the file system of each instance and doesn't support sharing in the indexes between instances. How can I achieve this?
You cannot. Lucene is a "local filesystem" technology.
If you want something that is shared between multiple Content Delivery server instances; you need to switch from Lucene to SOLR.
If you're using the recommended Sitecore ContentSearch APIs this is (mostly) a relatively simple matter of switching your configuration files and setting up the SOLR index properly.
Benefits of using Solr:
- It is heavily optimized for search performance with a powerful query cache.
- It can handle a large number of documents.
- There is detailed configuration support for language indexing.
- You can shard indexes across multiple servers.
- With Solr Cloud you can also split an index across multiple physical locations.
I second Mark's answer. In a "huge" content delivery farm you really want to consider a centralized search provider like Solr.
Coveo is another good option as an alternative to Solr. The starter edition is free, and if your indexes are small enough it may do the job for you with minimal fuss. http://www.coveo.com/en/solutions/coveo-for-sitecore/download
You are asking about managing indexes in "a huge content delivery web farm".
It could be a huge in two dimensions - vertically and horizontally.
Horizontally means that you have lots of servers. For that case, you got great answer - move index to out-of-the-process service like
Vertically means that your issue is an index size.
That is very typical situation, and it originates from the fact that out-of-the-box Sitecore is configured for usability, not for performance.
sitecore_web_index are designed for Content Management. These indexes include all versions of all Sitecore items and they used in Item Buckets, Media Selection dialog, Fields with search and etc.
CD requirements are different. You do not need all the items. You need only items that user should see as search results. It could be pages, products, articles, sometimes media-items and similar business objects.
The easiest way to bust performance of index in CD is to limit number of items in index. In typical case, Index rebuild speed will be at least ten times faster.
Update these config sections:
Filter Items by Root Item – sitecore/contentSearch/configuration/indexes/index/locations/crawler/Root
Filter items by Template Type – sitecore/contentSearch/indexConfigurations/yourIndexConfiguration/documentOptions/include
If that is not enough, try Domain Indexes. With that pattern you build custom indexe for every business domain instead of using standard one. You can find more details in my blog post - Indexing Patterns in Sitecore
If you decide to stick with Lucene, this post provides information to help you out.
It describes the following:
1) Use SwitchOnRebuildLuceneIndex to keep a working index while the rebuild is executed in a secondary folder.
2) Update your configuration files so that publishing and index rebuilds on a Content Management server trigger rebuilds on Content Delivery servers.
We use SOLR in our production environment. It removes a lot of load from all the servers.
Few learnings from our setup: The configuration is quite complex (availability, scalability, index config, which fields, which templates, what to exclude, async or sync update, etc.). The out-of-the-box configuration for storing ALL fields in the index for all languages is just not working as it bloats the space and memory needed on the indexing server a lot.
We are applying the Domain Indexes pattern as well and a huge Master Index with special tweaks and computed fields for the authoring search.
Not an easy topic, so don't underestimate this...