I have a Sitecore 8.1 / Solr 4.10 instance. I know that if I add a new content item or I delete a content item and then publish, that Sitecore is smart enough to do a partial rebuild of the Solr search index and that those items will either be added to or deleted from the Solr index. But what if I have a content item that is currently published to Web and then I decide to change the publishing restrictions on that item so that it is no longer published? I see that it does get removed from the Web database, but I also see that it seems to still show up in the Solr Web index. Is Sitecore smart enough to know that in that case it should also remove that item from the Solr Web index?
1 Answer
I did some research and for simple scenario unpublished content is deleted from the index. In my case, it was Lucene, but that shouldn't be really matter because all relevant code is stored in Sitecore.ContentSearch.dll
.
Scenario that I executed:
- Create the Item
- Publish it
- In Item Publishing Setting uncheck
Publishable
check-box - Publish Item again
Now lets talk about what happens behind the scene when last step from that scenario is executed and how you can trace these actions to see what does not work in your case.
Schedule Publishing
Publishing Start
event is tracked in Core
database.
Checks EventQueue
table in Core
to find recent StartPublishingRemoteEvent
event. I used SQL query below for that:
SELECT TOP 100 *
FROM [MySiteSitecore_Core].[dbo].[EventQueue]
order by Created desc
This is key data from that event:
- InstanceType - Sitecore.Publishing.StartPublishingRemoteEvent, Sitecore.Kernel, Version=10.0.0.0, Culture=neutral, PublicKeyToken=null
- InstanceData - {"ClientLanguage":"en","EventName":"publish:startPublishing","Options":[{"CompareRevisions":false,"Deep":false,"FromDate":"/Date(1478021589129)/","LanguageName":"en","Mode":3,"PublishDate":"/Date(1478051240384)/","PublishRelatedItems":false,"PublishingTargets":["{8E080626-DDC3-4EF4-A1D1-F0BE4A200254}"],"RecoveryId":"c97366e7-7272-4314-9fa3-66accb915a02","RepublishAll":false,"RootItemId":"ef9ec5da-53ca-453a-b57f-4b8ff33166c5","SourceDatabaseName":"master","TargetDatabaseName":"web","UserName":"sitecore\admin"}],"PublishingServer":"DESKTOP-3E7ETU9-Sitecore82","StatusHandle":{"instanceName":"DESKTOP-3E7ETU9-Sitecore82","m_handle":"83a9d418-514c-45fd-ac44-4d4dbc5ec0e9"},"UserName":"sitecore\admin"}
- Created - 2016-11-02 01:47:20.387 (UTC timezone)
That row contains full definition of publishing process.
Publishing
Sitecore takes publishing definition from previous steps and based on that definition it finds all the Items to be affected. With that list it executes modifications over these items in Web
database.
Traces of these modifications stored in EventQueue
table but in Web
database.
We are looking here for DeletedItemRemoteEvent
event.
This is what I found in my case:
- Instance Type - Sitecore.Data.Eventing.Remote.DeletedItemRemoteEvent, Sitecore.Kernel, Version=10.0.0.0, Culture=neutral, PublicKeyToken=null
- InstanceData - {"ItemId":"ef9ec5da-53ca-453a-b57f-4b8ff33166c5","ItemName":"TestItem","LanguageName":"en","TemplateId":"76036f5e-cbce-46d1-af0a-4143f9b557aa","VersionNumber":1,"ParentId":"110d559f-dea5-42ea-9c1c-8a5df7e70ef9"}
- Created - 2016-11-02 01:47:20.860 (UTC timezone)
With that information Sitecore knows what and how was modified in Web
. There is lots of other events to cover other scenarios. These are events that BaseAsynchronousStrategy
is looking for:
eventQueueQuery.EventTypes.Add(typeof(RemovedVersionRemoteEvent));
eventQueueQuery.EventTypes.Add(typeof(SavedItemRemoteEvent));
eventQueueQuery.EventTypes.Add(typeof(DeletedItemRemoteEvent));
eventQueueQuery.EventTypes.Add(typeof(MovedItemRemoteEvent));
eventQueueQuery.EventTypes.Add(typeof(AddedVersionRemoteEvent));
eventQueueQuery.EventTypes.Add(typeof(CopiedItemRemoteEvent));
eventQueueQuery.EventTypes.Add(typeof(RestoreItemCompletedEvent));
Index Update
Sitecore uses OnPublishEndAsynchronousStrategy
to update indexes built on top of Web
database:
<onPublishEndAsync type="Sitecore.ContentSearch.Maintenance.Strategies.OnPublishEndAsynchronousStrategy, Sitecore.ContentSearch">
<param desc="database">web</param>
<CheckForThreshold>true</CheckForThreshold>
</onPublishEndAsync>
With that strategy, Sitecore constantly compares index last update time-stamp and time-stamp of last event in event queue to detect events happened after index being updated. If such events found, Sitecore checks events queue in Web
database to see what data was modified. Based on these findings, OnPublishEndAsynchronousStrategy
generates list of IndexableInfo
objects and later concrete Index provider (Lucene or Solr) pushes data to the Index.
The simplest way to find if that happened is to check Crawling
log. My log contains these entries (set Debug mode in Log config to see all the entries):
12792 21:47:21 DEBUG [Index=sitecore_web_index] OnPublishEndAsynchronousStrategy executing.
12792 21:47:21 INFO [Index=sitecore_web_index] Updating '1' items from Event Queue.
12792 21:47:21 DEBUG IndexCustodian. IncrementalUpdate triggered on index sitecore_web_index. Data=Count=1
ManagedPoolThread #16 21:47:21 DEBUG [Index=sitecore_web_index] Committing: Add: 0; Update:0; DeleteUnique: 0; DeleteGroup: 1
As you can see, Item was deleted. Group is a 'nickname' for Item ID in Content Search world.
I am not sure if this will work for all possible scenarios, but you can use this info to troubleshoot your case.
P.S. All above was checked in Sitecore 8.2.
-
Excellent work Dmitry! I wonder if the question was raised because this process wasn't running as you've outlined -- with Solr, maybe the index update strategy isn't set for example? Regardless, the above should be enough for anyone to trace through in their own implementation. Nov 4, 2016 at 22:22
-
That process is not specific to search engine. Solr code comes to stage later to serialize data and integrate with engine. Nov 4, 2016 at 23:03