I am using Sitecore 9.2 and DEF 3.0.0.
I am facing weird problem when fetching data from Salesforce. The ReadObjectStepProcessor of DEF is using ObjectReader class that fetch data from Salesforce end. The problem is when fetching more than 2000 records, it's duplicating the existing records.The dataset count in Salesforce is 264868. First 2000 records are distinct but from 2001 records it again fetch the first 2000 records. I want to push data to SQL database from Salesforce end.
We had a similar issue with the standard Salesforce Provider and DEF 2.0.1. It turned out there was an endless loop in the method ReadObjects()
of class Sitecore.DataExchange.Providers.Salesforce.ReadObjects.ObjectReader
.
We fixed it by overriding classes ObjectReader
and ReadObjectsStepProcessor
. You may need to adapt the code below to work with DEF 3.0.0, but the principle is the same.
Please note the last line result = Task.Run<>...
- this is what was missing from the original method.
public class CustomObjectReader : Sitecore.DataExchange.Providers.Salesforce.ReadObjects.ObjectReader
{
public override IEnumerable<ExpandoObject> ReadObjects(ForceClient client, ObjectReaderContext context)
{
if (client == null)
throw new ArgumentNullException(nameof(client));
if (context == null)
throw new ArgumentNullException(nameof(context));
string soql = this.GetSoqlStatement(context);
QueryResult<ExpandoObject> result = Task.Run(() => client.QueryAsync<ExpandoObject>(soql)).Result;
while (result != null)
{
foreach (ExpandoObject record in result.Records)
yield return record;
if (result.Done || string.IsNullOrWhiteSpace(result.NextRecordsUrl))
break;
result = Task.Run(() => client.QueryContinuationAsync<ExpandoObject>(result.NextRecordsUrl)).Result;
}
}
}
After this you will need to override the class ReadObjectsStepProcessor
and use your newly created CustomObjectReader
public class CustomReadObjectsStepProcessor : Sitecore.DataExchange.Providers.Salesforce.ReadObjects.ReadObjectsStepProcessor
{
private readonly CustomObjectReader customObjectReader = new CustomObjectReader();
protected override IObjectReader GetObjectReader(Endpoint endpoint, PipelineStep pipelineStep, PipelineContext pipelineContext, ILogger logger)
{
return customObjectReader;
}
}
The final step is to reference CustomReadObjectsStepProcessor in the pipeline step. Open your Read Salesforce Objects Pipeline Step
and replace Processor Type
field value with the custom processor class:
Namespace.CustomReadObjectsStepProcessor, AssemblyName
-
-
-
Yeah same issue with endless loop. Its running infinitely when I am hitting Run Pipeline Batch – StanLee Apr 16 '20 at 10:29