If I unpublish an item will Sitecore update the search index?

Question

I have a Sitecore 8.1 / Solr 4.10 instance. I know that if I add a new content item or I delete a content item and then publish, that Sitecore is smart enough to do a partial rebuild of the Solr search index and that those items will either be added to or deleted from the Solr index. But what if I have a content item that is currently published to Web and then I decide to change the publishing restrictions on that item so that it is no longer published? I see that it does get removed from the Web database, but I also see that it seems to still show up in the Solr Web index. Is Sitecore smart enough to know that in that case it should also remove that item from the Solr Web index?

Answer 1

I did some research and for simple scenario unpublished content is deleted from the index. In my case, it was Lucene, but that shouldn't be really matter because all relevant code is stored in Sitecore.ContentSearch.dll.

Scenario that I executed:

1. Create the Item
2. Publish it
3. In Item Publishing Setting uncheck Publishable check-box
4. Publish Item again

Now lets talk about what happens behind the scene when last step from that scenario is executed and how you can trace these actions to see what does not work in your case.

Schedule Publishing

Publishing Start event is tracked in Core database.

Checks EventQueue table in Core to find recent StartPublishingRemoteEvent event. I used SQL query below for that:

SELECT TOP 100 *
FROM [MySiteSitecore_Core].[dbo].[EventQueue]
order by Created desc

This is key data from that event:

• InstanceType - Sitecore.Publishing.StartPublishingRemoteEvent, Sitecore.Kernel, Version=10.0.0.0, Culture=neutral, PublicKeyToken=null
• InstanceData - {"ClientLanguage":"en","EventName":"publish:startPublishing","Options":[{"CompareRevisions":false,"Deep":false,"FromDate":"/Date(1478021589129)/","LanguageName":"en","Mode":3,"PublishDate":"/Date(1478051240384)/","PublishRelatedItems":false,"PublishingTargets":["{8E080626-DDC3-4EF4-A1D1-F0BE4A200254}"],"RecoveryId":"c97366e7-7272-4314-9fa3-66accb915a02","RepublishAll":false,"RootItemId":"ef9ec5da-53ca-453a-b57f-4b8ff33166c5","SourceDatabaseName":"master","TargetDatabaseName":"web","UserName":"sitecore\admin"}],"PublishingServer":"DESKTOP-3E7ETU9-Sitecore82","StatusHandle":{"instanceName":"DESKTOP-3E7ETU9-Sitecore82","m_handle":"83a9d418-514c-45fd-ac44-4d4dbc5ec0e9"},"UserName":"sitecore\admin"}
• Created - 2016-11-02 01:47:20.387 (UTC timezone)

That row contains full definition of publishing process.

Publishing

Sitecore takes publishing definition from previous steps and based on that definition it finds all the Items to be affected. With that list it executes modifications over these items in Web database.

Traces of these modifications stored in EventQueue table but in Web database.

We are looking here for DeletedItemRemoteEvent event.

This is what I found in my case:

• Instance Type - Sitecore.Data.Eventing.Remote.DeletedItemRemoteEvent, Sitecore.Kernel, Version=10.0.0.0, Culture=neutral, PublicKeyToken=null
• InstanceData - {"ItemId":"ef9ec5da-53ca-453a-b57f-4b8ff33166c5","ItemName":"TestItem","LanguageName":"en","TemplateId":"76036f5e-cbce-46d1-af0a-4143f9b557aa","VersionNumber":1,"ParentId":"110d559f-dea5-42ea-9c1c-8a5df7e70ef9"}
• Created - 2016-11-02 01:47:20.860 (UTC timezone)

With that information Sitecore knows what and how was modified in Web. There is lots of other events to cover other scenarios. These are events that BaseAsynchronousStrategy is looking for:

eventQueueQuery.EventTypes.Add(typeof(RemovedVersionRemoteEvent));
eventQueueQuery.EventTypes.Add(typeof(SavedItemRemoteEvent));
eventQueueQuery.EventTypes.Add(typeof(DeletedItemRemoteEvent));
eventQueueQuery.EventTypes.Add(typeof(MovedItemRemoteEvent));
eventQueueQuery.EventTypes.Add(typeof(AddedVersionRemoteEvent));
eventQueueQuery.EventTypes.Add(typeof(CopiedItemRemoteEvent));
eventQueueQuery.EventTypes.Add(typeof(RestoreItemCompletedEvent));

Index Update

Sitecore uses OnPublishEndAsynchronousStrategy to update indexes built on top of Web database:

<onPublishEndAsync type="Sitecore.ContentSearch.Maintenance.Strategies.OnPublishEndAsynchronousStrategy, Sitecore.ContentSearch">
    <param desc="database">web</param>
    <CheckForThreshold>true</CheckForThreshold>
</onPublishEndAsync>

With that strategy, Sitecore constantly compares index last update time-stamp and time-stamp of last event in event queue to detect events happened after index being updated. If such events found, Sitecore checks events queue in Web database to see what data was modified. Based on these findings, OnPublishEndAsynchronousStrategy generates list of IndexableInfo objects and later concrete Index provider (Lucene or Solr) pushes data to the Index.

The simplest way to find if that happened is to check Crawling log. My log contains these entries (set Debug mode in Log config to see all the entries):

12792 21:47:21 DEBUG [Index=sitecore_web_index] OnPublishEndAsynchronousStrategy executing.

12792 21:47:21 INFO [Index=sitecore_web_index] Updating '1' items from Event Queue.

12792 21:47:21 DEBUG IndexCustodian. IncrementalUpdate triggered on index sitecore_web_index. Data=Count=1

ManagedPoolThread #16 21:47:21 DEBUG [Index=sitecore_web_index] Committing: Add: 0; Update:0; DeleteUnique: 0; DeleteGroup: 1

As you can see, Item was deleted. Group is a 'nickname' for Item ID in Content Search world.

I am not sure if this will work for all possible scenarios, but you can use this info to troubleshoot your case.

P.S. All above was checked in Sitecore 8.2.