I'm rebuilding a link database with more than 10 millions of rows. It is taking 16 hours per database, but the web server and SQL server are almost idle, even when new rows are generated slowly in the links table. Is there any way we can make use of all the resources available? Please note this is Sitecore 6.6 Update 8.
Not sure for 6.6 update 8, but I dug up the following information. Might be worth trying to see if that patch will apply in 6.6 also.
We had recently done upgrade of one of our implementation from SC 7.2 U3 to U6. As part of any upgrades Sitecore recommends to Rebuild Link Database as one of the key activities. Previously in SC 7.2 U3 when we had tried Rebuild Link Database it used to timeout and never used to get executed. This was one of the things that Sitecore fixed as part of SC 7.2 U5
Rebuilding the Link database would sometimes fail and throw a timeout exception. This has been fixed so that link database processing has been split into batches. The size of the batch is controlled by the LinkDatabase.MaximumBatchSize setting. (431417)
This fix did solve the issue we were facing, and rebuild link DB started working without any timeout exceptions. However we it was taking too long to complete. In one of our instance, which is large database – it used to take more than 16+ hours to complete the rebuild link DB for Master DB, and 12+ hours to complete it for Web DB, which was not a desired output in our implementation.
We then worked with Sitecore support which provided a patch#Sitecore.Support.115854.dll which initiated the rebuild link DB in a multiple threads to speed-up the rebuild process. Although this increased the memory utilization during the rebuild process, it did speed-up the rebuild link db process considerably and we were able to rebuild finish the rebuild of Master DB in 6-8 hours.
If you are experiencing similar issues – you can reach out to Sitecore support for the patch#Sitecore.Support.115854.dll which is available for Sitecore 7.2 U6 version.
When inspecting the link database, you might notice that a large part is filled with links on one or two fields (i've seen that).
If these field are not that important to track, you could change the field type, of inherit the field type and skip the update links implementation. See the
CustomCustomField.UpdateLink method for more details.
Other way could be overriding the LinkDatabase provider...