7

We're using Sitecore 7.2 Update 4, and have 6 Lucene indexes across our sites in a single Sitecore instance.

In the production environment, but not locally or on staging, the web index is not rebuilt when an item is published and we need to rebuild the index manually. Both staging and production use separate content authoring and delivery servers with the web index rebuild on publish being initiated on Staging but not in Production. Apparently there has been a recent backup of databases from Prod to Staging so there shouldnt be significant differences there.

Indexing Strategy on CD: SwitchOnRebuildLuceneIndex

    <strategies hint="list:AddStrategy">
          <strategy ref="contentSearch/indexUpdateStrategies/onPublishEndAsync"/>
          <strategy ref="contentSearch/indexUpdateStrategies/rebuildAfterFullPublish"/>
          <strategy ref="contentSearch/indexUpdateStrategies/remoteRebuild"/>
        </strategies>

And on CM:

<strategies hint="list:AddStrategy">
     <strategy ref="contentSearch/indexUpdateStrategies/syncMaster" />
</strategies>

Errors in Sitecore logs: none

All the conifuration settings in ScalabilitySettings.config exist in the web.config. Although InstanceNames are blank, they are recorded in the EventQueue table successfully using the default format of MachineName-IISSiteName. EnableEventQueues is set to true in web.config.

Im seeing ItemSavedRemote events in the EventQueue in the web db but IndexingStartedEvent is never recorded in the core db.. That seems to be the problem but not sure why the IndexingStartedEvent is not fired.

We have had someone out from Sitecore and they have told us the configs look good. Any suggestions as what may be causing the issue?

UPDATE 3/4/17 - Apparently this has started working again in Prod. When an article is published, the index is rebuilt and display on the website. To me, this is still an open issue as we dont know what caused this issue and it may break again - it has started working again once before and then stopped after a couple of months with no code/config changes. Suggestions as to what that might tell us?

  • I would be interested to know some configuration setup for web_index, like strategies, switchOnRebuild Index if turned on,Database,Root etc. [ Even if it looks good from your Sitecore team] – Balaji Kuppuswamy Mar 1 '17 at 5:40
  • Indexing strategy is SynchronousStrategy and the issue is only on CD i.e. when publish from CM to CD, the index isnt rebuilt but the master index on CM is rebuilt – Steve Ward Mar 1 '17 at 9:08
  • kb.sitecore.net/articles/992608 can you try to apply this patch ? – Vlad Iobagiu Mar 6 '17 at 9:45
  • You mentioned that the rebuild when an item is published is working correctly on Staging, which has a similar infrastructure to Production but that it is not working on Production. Was this ever working on Production, or has this been an issue since launch? – Zachary Kniebel Mar 6 '17 at 17:22
  • Also, can you have a look at the CrawlingLogs for your CD and let us know if you see the following entry for your web index when Sitecore initializes: Initializing OnPublishEndAsynchronousStrategy for index 'sitecore_web_index' (replace sitecore_web_Index with the name of your web index, if different). This will help to confirm that the issue is not with the indexing strategy configuration or initialization. – Zachary Kniebel Mar 6 '17 at 17:38
6
+25

The EventQueue - the medium of communication between your Sitecore instances - is timestamp-based, which can lead to issues if you are using replication and something happened that caused your SQL instance to re-initialize/failover or if the EventQueue's last processed timestamp becomes corrupted.

In order to ensure that this is not the issue, connect to each of your databases (you could do this just for Web, but if it affects one it likely affects all of them) and execute the following SQL commands (order matters; see below for details):

DELETE FROM [EventQueue]; --optionally, you can change this command to delete older than a certain date instead

DELETE FROM [Properties] WHERE [Key] LIKE 'EQStamp_%';

What did I just do?

Cleared the EventQueues

It is safe to clear the EventQueues for your databases, and often times this will result in a performance boost - sometimes noticeable; other times not. When these tables become too large, Sitecore often has trouble processing them, hence why Sitecore includes Cleanup Agents for the EventQueues to delete old records.

Basically, all you are doing here is resetting the stage and giving your environment a "clean slate" as far as events queuing is concerned.

Side-note about the Cleanup Agents: since the agents run on a relatively long default interval, the EventsQueue can get very large on a bigger site. Additionally, since agents can be troublesome when longer intervals are used (reference the section "The Problem with Agents" in this post), I typically create Windows Scheduled tasks to regularly clean out the EventQueues for larger sites, due to the greater reliability of the scheduling.

Cleared the Reported Times the EventQueues were Last Processed

Sitecore stores the value that represents the last time a database's EventQueue was processed in the Properties table. The Key for this value is EQStamp_<MachineName>-<IISAppPoolName>.

By deleting all values from the Properties table with a Key that starts with the value 'EQStamp_' (via the % wildcard and the LIKE operator), you are essentially "resetting" the last recorded time the EventQueues were processed to "never."

As such, the next time Sitecore goes to process the EventQueue for that database it will see any records in the EventQueue table as new events, and will thus begin to process them. As such, it is important that you clear your EventQueues before doing this step. If you do not, your instances will see all of the records in the EventQueue table, including those already and not yet processed - as new events and will begin to process them all. If you have a large amount of records in your EventQueue this can take a very long time, so be sure that your EventQueue is cleared before you clear the last processed timestamp.

Why did I do that thing I just did?

Database-Relative Time and the Last Processed Time

The EventQueue uses the recorded last processed time for determining which events are new, so as to retrieve them for processing. In order to retrieve new events for processing, Sitecore requests all events from the EventQueue that are newer than the last-processed event, as determined by comparing the timestamp values.

The way that the EventQueue's last processed time is recorded is by creating a a copy of the [EventQueue].[Stamp] column value of the last processed event. The [EventQueue].[Stamp] column is of the timestamp type (deprecated since SQL 2008), and its value is copied to the Properties table as a string (stored in an ntext type column). The newly created record in the Properties table is the one that we cleared out, with the Key column value of EQStamp_<MachineName>-<IISAppPoolName>.

The timestamp type stores a database-relative time that is computed by incrementing an 8-byte binary counter on each INSERT or UPDATE operation. According to MSDN:

Each database has a counter that is incremented for each insert or update operation that is performed on a table that contains a rowversion column within the database. This counter is the database rowversion. This tracks a relative time within a database, not an actual time that can be associated with a clock

Essentially, what this means is that when the SQL instance re-initializes it's basically starting out with a fresh database-relative timestamp, meaning that any newly generated timestamp values will be starting from 0. More accurately, the timestamp values will be starting from the point at which the replication started.

Here's where this gets to be a problem: assume that when the replication started, the last processed timestamp was 601283 (which is a number I took from a random web database's Properties table in my local SQL instance). Time passes and the last processed timestamp of the SQL instance being used, SQL A, grows to 1301283. At that point, something happens and SQL A is replaced with the replication set, SQL B. The problem is that SQL B's timestamp value is still 601283 (the value from immediately before replication started), but because SQL B was a replication set its Properties table has the last processed time recorded as 1301283 (the value the timestamp on SQL A grew to, before SQL B took over). As a result, new events are created in SQL B with timestamp values 601288, 601313, 601337, ... and so on (note that these are just examples) meanwhile Sitecore is only requesting events with a timestamp value greater than 1301283. What this means is that the database now thinks that the new events have already been processed, because compared to the recorded last processed timestamp, 1301283, the new events' timestamp values, e.g. 601288, are in the past! This means that new events are effectively ignored by Sitecore.

It should be noted that left unfixed the timestamp value should eventually catch up to the last processed value stored in the Properties table. The amount of time this takes depends on how long your replication has been running. For some instances, this issue might not be that big of a problem, e.g. if your replication has only been running for a few hours, a few days, or sometimes even longer, depending on how heavily the instance is being used and changes are being made. However, if your replication has been running long enough then it could take weeks, months or even years for the timestamp value to catch up.

To sum it all up, this problem can be remedied by simply deleting the EventQueue's recorded last processed time from the Properties table, so as to reset the value that Sitecore assumes a "new" event's Stamp column value must be greater than.

Corruption of the Last Processed Time

While not super likely, it is also possible for the EventQueue's recorded last processed time to become corrupted. This can happen as a result of changes in SQL infrastructure, backups/restores, etc. If any of these scenarios may be relevant, this solution is very low effort and low risk. In general, it pays to execute the queries, specified above, to be safe and eliminate these issues as a possibility.

Credit

Much of the credit for the explanation of why this solution works is due to Per Manniche Bering and his post, Sitecore CD nodes not picking up events after replication re-initialization. I must also give him credit for finding and highlighting (which I matched via emboldening) the excerpt from MSDN that I quoted above, which I found to be of great help in fully understanding the issue (ergo, why I repeated it, here).

  • Thanks for this Zachary. I definitely sounds like something to try and is low risk and doesn't require a code release which is all good. One question, if there are events in the EventQueue which havent yet been processed, wouldnt deleting them mean they aren't actioned ? – Steve Ward Mar 8 '17 at 3:01
  • Yes, but that's no big deal. Your environment's integrity shouldn't depend on event-based logic. In your case, the indexes aren't building in response to the events, so the fist thing I would do after performing the two queries is make a change to and publish a single item to see if the issue is resolved and then manually rebuilding the index one last time (after making sure everything is working), just to make sure everything is up to date. Then you should be good to go ;) – Zachary Kniebel Mar 8 '17 at 3:15
  • We have deleted from the EventQueue and Properties table as suggested but it didnt solve the issue in this case – Steve Ward Mar 10 '17 at 0:24
2

In Sitecore.config (for failing environments), locate this setting.

<eventQueue>
  <!-- Time between checking the queue for newly queued events. If new events are found they will be raised. -->
  <processingInterval>00:00:02</processingInterval>
</eventQueue>

Try and increase the processingInterval. Suspicion being, you may be running into a deadlock type of situation. So for example, set it to this:

<eventQueue>
  <!-- Time between checking the queue for newly queued events. If new events are found they will be raised. -->
  <processingInterval>00:00:30</processingInterval>
</eventQueue>

Reference: Index are not updating after db restore

  • This worked for one of my clients. – phani Mar 10 '17 at 8:19
  • As this is a production environment, I need to have an idea of how this will fix the issue and look at any unintended consequences .. would be so easy to test if we could reproduce in other environments! – Steve Ward Mar 13 '17 at 21:43
  • We tried this change and unfortunately it did not fix our issue – Steve Ward Mar 14 '17 at 5:35
1

Since you have CM and CD Servers, it is required that you enable the scalability settings. This is because it allows the servers to know when the instance specified in the scalability settings is inserted into the event queue, it should triggers the event.

Moreover, some points to note:

  1. Make sure in the properties table, there a re no records of instances that are not CM and CD. For example, instance for Staging is also present in the Properties table of your CM or CD.

  2. Make sure that you don't have verbose logging enable on the servers.

  3. Indexes normally get rebuild once all the servers are in sync. This may be a reason why you need to enable the scalability settings.

  4. Check if indexing events are populated also in the core db of the CD server. If not, it may be that the core db replication is not working accordingly

  • Scalability Settings are not enabled but all of the settings it contains are present in our main web.config. InstanceNames arent defined so they will default to the site name appended to the machine name – Steve Ward Mar 10 '17 at 0:33
  • Yes but since there are different CM and CD, Sitecore will require the scalability settings and also, for events to get triggered, Sitecore servers must first be synchronized. If no Scalability setting is specified, Sitecore will take lot of time to sync – Hishaam Namooya Mar 10 '17 at 3:08
  • All the scalability settings are there in the web.config. You mention Sitecore will take a long time to sync if they aren't. Are you referring to the Instance Names in particular? In our config, we currently have blank (default) values for the instance names .. – Steve Ward Mar 10 '17 at 4:20
  • If left blank, the machine name will be used. I would suggest for only 1 CM server, in the web.config, for the instance name, write CM1 and on the other servers (CM and CD), set the publishing instance value to CM1. – Hishaam Namooya Mar 10 '17 at 4:25
  • Then trigger a publish from the server you had specified the instance name as CM1 – Hishaam Namooya Mar 10 '17 at 4:40
0

I'd hope someone from Sitecore would have checked this, but just in case: Have you looked at your event queue configuration?

It's a while since I worried about stuff like this, but if I remember correctly, triggering an index operation for Lucene runs it locally (on your CM box) and sends out an event to inform all the CD boxes that "an index build is required now". Different rebuild strategies use different events to suit their approach.

Off the top of my head, I can think of a few things that might be wrong here that could cause issues with indexing:

  • Remote events might not be enabled.
    (In the "scalability settings" configuration file I think)
  • The instance names of your servers are odd, so that they don't realise the event being broadcast is for them.
  • Something in the CD server config has detached the index rebuild strategy from the appropriate events, so they fail to respond to the event.

You mentioned using "SynchronousStrategy" - based on the docs, I believe that is making use of the events for item-level actions like "ItemSavedRemote". You might want to check that is triggering on your CD servers?

  • Page 22 of this doc sdn.sitecore.net/upload/sitecore6/64/… says enabling that config file is required to turn the remote events on. My understanding of how the underlying eventing code is working isn't good enough to know if it's really necessary - but it would be my starting point for trying to resolve this. Does turning it on across your cluster change the behaviour? – JermDavis Mar 1 '17 at 22:38
  • Thanks. Jerm, I was going to look at exactly the same things .. The ScalabilitySettings.config isn't enabled but InstanceName in the EventQueue table is recorded OK and EnableEventQueues is set to true in web.config. Im seeing ItemSavedRemote events in the EventQueue in the web db but IndexingStartedEvent is never recorded in the core db.. That is the problem but not sure why the IndexingStartedEvent is not fired – Steve Ward Mar 1 '17 at 23:55
  • A bit of an out-there suggestion: Was looking at the code for index event handling, and it includes a test for "are Item Buckets enabled?". If the test fails it does not initialise the ContentSearch event hub object which SynchronousStrategy appears to depend on. It looks like that initialisation does the binding of DataEngine-level events to the index strategy event handlers... Do you have "BucketConfiguration.ItemBucketsEnabled" disabled? (should be in Sitecore.Buckets.config - but you may be overriding it) – JermDavis Mar 2 '17 at 13:27
  • Yes, BucketConfiguration.ItemBucketsEnabled is true. Sounds like I'm going to need to dig deep into the Sitecore code. Where abouts have you been looking? – Steve Ward Mar 2 '17 at 21:50
  • I started by looking at the SynchronousStrategy class - which lives in Sitecore.ContentSearch.Maintenance.Strategies.dll - and I just clicked on interesting looking dependencies. – JermDavis Mar 3 '17 at 9:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.