Sitecore fetching duplicate data from Salesforce end when using DEF 3.0.0

Question

I am using Sitecore 9.2 and DEF 3.0.0. I am facing weird problem when fetching data from Salesforce. The ReadObjectStepProcessor of DEF is using ObjectReader class that fetch data from Salesforce end. The problem is when fetching more than 2000 records, it's duplicating the existing records.The dataset count in Salesforce is 264868. First 2000 records are distinct but from 2001 records it again fetch the first 2000 records. I want to push data to SQL database from Salesforce end.

Answer 1

We had a similar issue with the standard Salesforce Provider and DEF 2.0.1. It turned out there was an endless loop in the method ReadObjects() of class Sitecore.DataExchange.Providers.Salesforce.ReadObjects.ObjectReader.

We fixed it by overriding classes ObjectReader and ReadObjectsStepProcessor. You may need to adapt the code below to work with DEF 3.0.0, but the principle is the same.

Please note the last line result = Task.Run<>... - this is what was missing from the original method.

public class CustomObjectReader : Sitecore.DataExchange.Providers.Salesforce.ReadObjects.ObjectReader
{
    public override IEnumerable<ExpandoObject> ReadObjects(ForceClient client, ObjectReaderContext context)
    {
        if (client == null)
            throw new ArgumentNullException(nameof(client));
        if (context == null)
            throw new ArgumentNullException(nameof(context));
        string soql = this.GetSoqlStatement(context);
        QueryResult<ExpandoObject> result = Task.Run(() => client.QueryAsync<ExpandoObject>(soql)).Result;
        while (result != null)
        {
            foreach (ExpandoObject record in result.Records)
                yield return record;
            if (result.Done || string.IsNullOrWhiteSpace(result.NextRecordsUrl))
                break;
            result = Task.Run(() => client.QueryContinuationAsync<ExpandoObject>(result.NextRecordsUrl)).Result;
        }
    }
}

After this you will need to override the class ReadObjectsStepProcessor and use your newly created CustomObjectReader

public class CustomReadObjectsStepProcessor : Sitecore.DataExchange.Providers.Salesforce.ReadObjects.ReadObjectsStepProcessor
{
    private readonly CustomObjectReader customObjectReader =  new CustomObjectReader();

    protected override IObjectReader GetObjectReader(Endpoint endpoint, PipelineStep pipelineStep, PipelineContext pipelineContext, ILogger logger)
    {
        return customObjectReader;
    }
} 

The final step is to reference CustomReadObjectsStepProcessor in the pipeline step. Open your Read Salesforce Objects Pipeline Step and replace Processor Type field value with the custom processor class:

Namespace.CustomReadObjectsStepProcessor, AssemblyName