Problem found and isolated to Lucene index
So I finally managed to track this one down. It is related to the fact that we run on Lucene indexes so if you're running the recommended SOLR configuration, this problem will not affect you.
The problem lies in OrdersIntervalAsynchronousStrategy
which was defined and deployed to all our instances, CM, CD1 and CD2.
What this strategy does, in simple terms, is this:
- Get a list of orders to index. This list is pulled from Content Authoring
- Index these orders
- Remove these orders from the list
As witnessed by this code:
...
ManagedList ordersToIndex = IndexUtility.GetOrdersToIndex(environment, this.IndexListName, 0, this.ItemsToTake);
...
IndexCustodian.IncrementalUpdate(this.Index, (IEnumerable<IIndexableUniqueId>) list1.Select<Order, IndexableUniqueId<CommerceOrderIndexableItem>>((Func<Order, IndexableUniqueId<CommerceOrderIndexableItem>>) (order => new IndexableUniqueId<CommerceOrderIndexableItem>(this.TranslateOrder(order, artifactId)))).ToList<IndexableUniqueId<CommerceOrderIndexableItem>>());
...
IndexUtility.ClearIndex(environment, this.IndexListName, ordersToIndex.Items.Select<CommerceEntity, string>((Func<CommerceEntity, string>) (item => item.Id)).ToList<string>());
And what was happening was; that since this ran on all instances - a CD server would (sometimes) pick up an order and add it to the index BEFORE the CM server saw the order. And since the strategy REMOVES the order from the list of orders to be indexed - CM would never see it.
What made this extra hard to debug was the fact that, in all cases we had - the log files would throw us off course. CD2 would get an order, and nothing in the logs would tell us why it would not get indexed. Turned out, CD1 picked it up just minutes before CM would have seen it.
The fix
To get around this, the index strategy has been disabled on the CD servers of the environment. As far as I can tell, this index serves no purpose on CD. If this turns out to be a problem, a version of the strategy WITHOUT the list removal will need to be written, and deployed for CD servers.
I modified Y.Commerce.Engine\Sitecore.Commerce.Engine.Connectors.Index.Lucene.config
on CD1 and CD2 and commented out this:
<!-- This cannot be allowed to run on CD servers. It basically picks up new orders to be indexed and indexes them, then removes them from the list of new orders.
Since we run Lucene, this means the CM server never sees these orders and gets them indexed locally -->
<!--<strategies hint="list:AddStrategy">
<strategy ref="contentSearch/indexConfigurations/indexUpdateStrategies/ordersIntervalAsynchronousStrategy" />
</strategies>-->
Effectively disabling the order index strategy on CD servers. Orders are now being picked up on CM servers as expected.
I believe this to be a bug, since "strongly recommend SOLR" and "not supporting Lucene" is not one and the same ;-)