Performance issue when inserting item to Lucene index

Question

I have a Sitecore 8 instance with a bucket with aprox. 150K items, and like 200K total.

Publishing is working very slow. I mean, really slow, sometimes 5 items per second. I've stripped all code we had on the pipeline and the issue persists.

I believe this may be related with the insertion of items during publish into the Lucene index. I've tested a manual insertion of an item into the index and I can see it takes a long time, up to two or three seconds sometimes:

var contentSearchIndex = ContentSearchManager.GetIndex("sitecore_master_index");
contentSearchIndex.Refresh((SitecoreIndexableItem)scItem);

I've tried performance tuning the index with this article, but no luck:

https://community.sitecore.net/technical_blogs/b/sitecore_7_development_team/posts/sitecore-7-performance-tuning-part-2

Is there a way of having a fast indexing with a relatively large amount of items? Is there something else that comes to mind that I could try?

Here's a copy of my index config file:

https://drive.google.com/file/d/0B2fC8NS-H2h4TG1lcG9CRUlfN3M/view?usp=sharing

Answer 1

I think your Lucene index needs a trim. Especially if it hasn't been this slow always, but just gradually started degrading little by little.

Since it's not a production server, my first recommendation would be to issue a full Index Rebuild and let it run for however long it needs. If you see a really slow rebuild speed, your problem lies elsewhere unfortunately.

Also: <setting name="ContentSearch.MaxMergeMB" value="512" /> this setting seems very low to me. I've seen Lucene happily use up 6-8 GB of memory for around 125.000 items. Increasing this to 4 or even 8 GB would be my next visiting point.

Answer 2

You could try deleting the Indexes directory and rebuilding all (in case there is some corruption).

I know some who disable Lucene indexing entirely during publishing and then do a re-index once all the dust has settled. Lucene has some cascading actions (Optimize and segment merges) that can be slow, esp if resources are limited. You can disable indexing by changing the config settings, or programmatically via:

Sitecore.ContentSearch.Maintenance.IndexCustodian.PauseIndexing();
IndexCustodian.ResumeIndexing();

You can check the status of indexing with the following:

IndexCustodian.IsIndexingPaused( ISearchIndex index )