4

Using Sitecore 9.0.1 and DEF 2.0.1, I have a pipeline which reads from an API and then resolves matching Sitecore items using the standard ResolveSitecoreItemStepProcessor and ResolveSitecoreItemStepConverter

Locally, this works fine, and if the Sitecore item exists, it is updated rather than a new one created. On our server, duplicate records are created intermittently when the pipeline is run. The issue appears to happen more often when importing large data sets of 1000 records and then sometimes, only for the last 500 records.

ManagedPoolThread #7 05:41:46 DEBUG [Data Exchange] Value converted for search. (pipeline: Experience from Source to Item Sync Pipeline, pipeline step: Resolve Experience Item, pipeline step identifier: 4b865ee9-78f5-49fe-a34e-b595d5d8b426, field: Uuid, original value: a109b95a-8135-4335-a20d-7edef2321a56, converted value: a109b95a-8135-4335-a20d-7edef2321a56)
ManagedPoolThread #7 05:41:46 DEBUG [Data Exchange] Starting search for item. (pipeline: Experience from Source to Item Sync Pipeline, pipeline step: Resolve Experience Item, pipeline step identifier: 4b865ee9-78f5-49fe-a34e-b595d5d8b426, field: Uuid, value: a109b95a-8135-4335-a20d-7edef2321a56)

i.e. ResolveSitecoreItemStepProcessor is searching for an item with a Uuid (our custom field) of { a109b95a-8135-4335-a20d-7edef2321a56}.

We've decompiled ResolveSitecoreItemStepProcessor and added our own debug. From that, we can see that the search isn't able to find the item in the index even when it exists. Also, when we have a duplicate, if we search in the Content Editor for some content which is duplicated across duplicate items, only one result is found until we re-index via the index manager. It therefore seems to be an indexing issue.

enter image description here

"Iterate Rows from Source and Run Pipeline" in turn calls the " Experience from Source to Item Sync Pipeline" pipeline. We added the "Rebuild Index Pipeline" step to try to address this issue but we still get this issue ..

enter image description here

enter image description here

We are considering a number of approaches to fix this including 1) Using our own implementation of ResolveSitecoreItemStepProcessor that doesn't use the index but looks at the Sitecore tree (might be complex and have unforseen consequences) 2) Batching the import into smaller chunks (of say 100 records) as this happens only with large data sets 3) Possibility of turning off indexing during an import (needs research)

Ideally, we would identify the root cause of this issue, rather than one of these workarounds ..

  • 1
    I've remained quiet since I've no first hand experience with the DEF. However. It is a common mistake when doing migration processes, to rely on values being written to an index and to be later consumed later in the process. Index writes are asynchronous and you can never rely on reading back a value from any index without special precautions. – Mark Cassidy Jun 28 '18 at 7:10
  • Thanks Mark. Im wondering if there is away to disable the indexing while the batch import is running and then do a re-index at the end. Ever tried this? – Steve Ward Jul 2 '18 at 4:56
  • I've tried pausing indexing using Sitecore.ContentSearch.Maintenance.IndexCustodian.PauseIndex() but then Im getting an error "There is no method 'GetResults' on type 'Sitecore.ContentSearch.Linq.QueryableExtensions: which is called internally by the Data Exchange Framework – Steve Ward Jul 2 '18 at 6:06
  • I normally did pause indexing when doing migration runs. However, the way I did it back then is likely not valid any longer. Sitecore.Configuration.Settings.Indexing.Enabled = false; – Mark Cassidy Jul 2 '18 at 8:31
  • If I pause indexing using "IndexCustodian.PauseIndexing", then I get an error "There is no method 'GetResults' on type 'Sitecore.ContentSearch.Linq.QueryableExtensions" when the Data Exchange Framework tries to resolve the next Sitecore item using the index. I'm thinking using our own implementation for the resolve routine that doesnt use the index may be the only way.. – Steve Ward Jul 2 '18 at 23:27
2

Still think the issue in the indexing.

The resolver uses Search method from InProcItemModelRepository.

public virtual IEnumerable<ItemModel> Search(ItemSearchSettings settings)

Search tries to get indexname by databasename

protected virtual string GetSearchIndexNameForDatabase(string databaseName)
    {
        return string.Format("sitecore_{0}_index", databaseName);
    }

Here is configuration for IItemModelRepository:

     <dataExchange>
        ...
        <itemModelRepository type="Sitecore.DataExchange.Local.Repositories.InProcItemModelRepository, Sitecore.DataExchange.Local">
            <databaseName>master</databaseName>
        </itemModelRepository>
        ...
    </dataExchange>
  • Thanks Vlad - My insinct is saying it is related to the indexing issue, but the <databaseName> setting for itemModelRepository is set to "master" and we are rebuilding the sitecore_master_index ... – Steve Ward Jun 21 '18 at 22:44
  • Importing 1000 records - we are only creating duplicates towards the end of the dataset. i.e. if we have 1000 records, and run the import again we end up with 1527 records. The duplicates are all in the second half of the dataset. The weird thing is, looking at the log, I can see 'Item was resolved' even for items that end up as duplicates.. – Steve Ward Jun 25 '18 at 1:15
  • Can it be related issue? sitecore.stackexchange.com/questions/6311/… – Vlad Shpak Jun 25 '18 at 11:35
  • We aren't seeing that error related to the number of fields in the logs so Im not sure it's related. We are using AWS by the way. I've also updated the question with more detail, and added a bounty, so please take a look – Steve Ward Jun 28 '18 at 1:29
  • @SteveWard Can you override "GetSearchIndexNameForDatabase" with your custom index like sitecore_myindex_index and of course add index configuration with root folder – Vlad Shpak Jun 29 '18 at 20:51
2

This is the answer I received from Sitecore support. In the end, we implemented our own version of ItemModel repository as they suggested:

As you probably know, the ResolveSitecoreItemStepProcessor, as well as the other processors of the Sitecore provider, uses InProcItemModelRepository for any item related operations. The InProcItemModelRepository under the hood uses the ItemService API. Whenever your pipeline creates or updates an item InProcItemModelRepository sends the appropriate HTTP request against the ItemService. The problem is that each separate ItemService request gets served in a separate thread on the Sitecore side: same as it works for the other regular HTTP requests. Your sitecore_master_index is configured to use syncMaster strategy, which executes index updates from the corresponding DataEngine events. This all together leads to a situation when a number of parallel ItemService threads may initiate index updates for the performed item operations. But Sitecore does not allow concurrent updates for the same index, thus it queues the index update jobs. And in case of a large amount of content changes the number of jobs queued may grow very quickly.

For example, in order to perform 1000 create/update operations, DEF will execute 1000 separate parallel ItemService requests. As a result, you will get 1000 queued update jobs only for the sitecore_master_index. Processing all these jobs may take some time, and thus the last item updates may be delayed.

When your further DEF processors try to find existing objects then may fail due to the index is not up to date at the moment of processors execution. Unfortunately, this is just how the InProcItemModelRepository was designed. I have registered a feature request to create a new endpoint (ItemModel repository) which does not depend on indexes so that it can be considered for future implementation. To track the future status of this request, please use the reference number 231598. More information about public reference numbers can be found here.

At the moment you can consider developing your own implementation the ItemModel repository. As an option, you can use Sitecore Queries instead of the Content Search queries. Please let me know whether this makes sense for you.

0

After consulting with Sitecore Support on similar issues, one of my collegues received this patch: https://github.com/SitecoreSupport/Sitecore.Support.227828/releases/tag/2.0.1.0

You might want to try this one out.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.