Sitecore DEF - Get the total data count

Question

We would like to log the total records fetched from databases in Data Exchange Framework and log it in log files. How we can get the total count?

Answer 1

For this, you need to add a Pipeline step under your Pipeline. This is called Iterate Data and Run Pipelines Pipeline Step, which will be created from the template called Iterate Data and Run Pipelines Pipeline Step.

The location of this template is

/sitecore/templates/Data Exchange/Framework/Pipeline Steps/Iterate Data and Run Pipelines Pipeline Step

The code is already written in the Processor, which will log the number of items processed. The following code is implemented for this.

if (iterableData == null)
{
    this.Log(new Action<string>(logger.Debug), pipelineContext, "Iterable data is null.", Array.Empty<string>());
}
else
{
    int count = 0;
    try
    {
         Action<object> afterElementHandled = (Action<object>) (element => ++count);
         MultithreadedOperationSettings plugin = pipelineStep.GetPlugin<MultithreadedOperationSettings>();
         if (plugin == null ? this.IterateWithoutThreads(iterableData, pipelinesSettings.Pipelines, afterElementHandled, pipelineStep, pipelineContext, logger) : this.IterateWithThreads(plugin, iterableData, pipelinesSettings.Pipelines, afterElementHandled, pipelineStep, pipelineContext, logger))
              this.Log(new Action<string>(logger.Debug), pipelineContext, string.Format("{0} elements were iterated.", (object) count), Array.Empty<string>());
         else
              this.Log(new Action<string>(logger.Info), pipelineContext, string.Format("{0} elements were iterated before the processor was interrupted.", (object) count), Array.Empty<string>());
          }
          catch (Exception ex)
          {
            this.LogException(ex, new Action<string>(logger.Error), pipelineContext, "Exception while iterating data.", Array.Empty<string>());
            pipelineContext.CriticalError = true;
          }
    }
}

Also if you want to override the logic, then you can create your own Processor class and copy the code from the OOTB class, do your changes, and bind your new Processor under the Iterate Data and Run Pipelines Pipeline Step item in the field called Processor Type under the section Data Exchange Framework.

Hope this help.