3

I am using Sitecore 9.2 and DEF 3.0.0.

I am facing a weird problem when fetching data from Salesforce. The ReadObjectStepProcessor of DEF is using ObjectReader class that fetch data from the Salesforce end. The problem is when fetching more than 2000 records, it's duplicating the existing records. The dataset count in Salesforce is 264868. The First 2000 records are distinct but from 2001 records it again fetches the first 2000 records. I want to push data to the SQL database from the Salesforce end. enter image description here

1 Answer 1

1

We had a similar issue with the standard Salesforce Provider and DEF 2.0.1. It turned out there was an endless loop in the method ReadObjects() of class Sitecore.DataExchange.Providers.Salesforce.ReadObjects.ObjectReader.

We fixed it by overriding classes ObjectReader and ReadObjectsStepProcessor. You may need to adapt the code below to work with DEF 3.0.0, but the principle is the same.

Please note the last line result = Task.Run<>... - this is what was missing from the original method.

public class CustomObjectReader : Sitecore.DataExchange.Providers.Salesforce.ReadObjects.ObjectReader
{
    public override IEnumerable<ExpandoObject> ReadObjects(ForceClient client, ObjectReaderContext context)
    {
        if (client == null)
            throw new ArgumentNullException(nameof(client));
        if (context == null)
            throw new ArgumentNullException(nameof(context));
        string soql = this.GetSoqlStatement(context);
        QueryResult<ExpandoObject> result = Task.Run(() => client.QueryAsync<ExpandoObject>(soql)).Result;
        while (result != null)
        {
            foreach (ExpandoObject record in result.Records)
                yield return record;
            if (result.Done || string.IsNullOrWhiteSpace(result.NextRecordsUrl))
                break;
            result = Task.Run(() => client.QueryContinuationAsync<ExpandoObject>(result.NextRecordsUrl)).Result;
        }
    }
}

After this you will need to override the class ReadObjectsStepProcessor and use your newly created CustomObjectReader

public class CustomReadObjectsStepProcessor : Sitecore.DataExchange.Providers.Salesforce.ReadObjects.ReadObjectsStepProcessor
{
    private readonly CustomObjectReader customObjectReader =  new CustomObjectReader();

    protected override IObjectReader GetObjectReader(Endpoint endpoint, PipelineStep pipelineStep, PipelineContext pipelineContext, ILogger logger)
    {
        return customObjectReader;
    }
} 

The final step is to reference CustomReadObjectsStepProcessor in the pipeline step. Open your Read Salesforce Objects Pipeline Step and replace Processor Type field value with the custom processor class:

Namespace.CustomReadObjectsStepProcessor, AssemblyName
3
  • Hey Anna, thanks for the reply, but I fixed it already that day 🙂
    – StanLee
    Apr 15, 2020 at 22:41
  • @StanLee was it the same issue with an endless loop?
    – Anna Gevel
    Apr 16, 2020 at 10:22
  • Yeah same issue with endless loop. Its running infinitely when I am hitting Run Pipeline Batch
    – StanLee
    Apr 16, 2020 at 10:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.