0

I am using Sitecore 9.2 and DEF 3.0.0. I am facing weird problem when fetching data from Salesforce. The ReadObjectStepProcessor of DEF is using ObjectReader class that fetch data from Salesforce end. The problem is when fetching more than 2000 records, it's duplicating the existing records.The dataset count in Salesforce is 264868. First 2000 records are distinct but from 2001 records it again fetch the first 2000 records. I want to push data to SQL database from Salesforce end. enter image description here

0

We had a similar issue with the standard Salesforce Provider and DEF 2.0.1. It turned out there was an endless loop in the method ReadObjects() of class Sitecore.DataExchange.Providers.Salesforce.ReadObjects.ObjectReader.

We fixed it by overriding classes ObjectReader and ReadObjectsStepProcessor. You may need to adapt the code below to work with DEF 3.0.0, but the principle is the same.

Please note the last line result = Task.Run<>... - this is what was missing from the original method.

public class CustomObjectReader : Sitecore.DataExchange.Providers.Salesforce.ReadObjects.ObjectReader
{
    public override IEnumerable<ExpandoObject> ReadObjects(ForceClient client, ObjectReaderContext context)
    {
        if (client == null)
            throw new ArgumentNullException(nameof(client));
        if (context == null)
            throw new ArgumentNullException(nameof(context));
        string soql = this.GetSoqlStatement(context);
        QueryResult<ExpandoObject> result = Task.Run(() => client.QueryAsync<ExpandoObject>(soql)).Result;
        while (result != null)
        {
            foreach (ExpandoObject record in result.Records)
                yield return record;
            if (result.Done || string.IsNullOrWhiteSpace(result.NextRecordsUrl))
                break;
            result = Task.Run(() => client.QueryContinuationAsync<ExpandoObject>(result.NextRecordsUrl)).Result;
        }
    }
}

After this you will need to override the class ReadObjectsStepProcessor and use your newly created CustomObjectReader

public class CustomReadObjectsStepProcessor : Sitecore.DataExchange.Providers.Salesforce.ReadObjects.ReadObjectsStepProcessor
{
    private readonly CustomObjectReader customObjectReader =  new CustomObjectReader();

    protected override IObjectReader GetObjectReader(Endpoint endpoint, PipelineStep pipelineStep, PipelineContext pipelineContext, ILogger logger)
    {
        return customObjectReader;
    }
} 

The final step is to reference CustomReadObjectsStepProcessor in the pipeline step. Open your Read Salesforce Objects Pipeline Step and replace Processor Type field value with the custom processor class:

Namespace.CustomReadObjectsStepProcessor, AssemblyName
  • Hey Anna, thanks for the reply, but I fixed it already that day 🙂 – StanLee Apr 15 '20 at 22:41
  • @StanLee was it the same issue with an endless loop? – Anna Gevel Apr 16 '20 at 10:22
  • Yeah same issue with endless loop. Its running infinitely when I am hitting Run Pipeline Batch – StanLee Apr 16 '20 at 10:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.