Iterable data is null when trying to fetch data from SQL Database by SQL provider in Data exchange framework

Question

I'm facing this error when trying to work with Sitecore 9.1.1 and DEF 2.1.0 and using the out of box SQL provider. I created two pipelines, one for Reading data from Database and other for Iterating the read data and create Items based on reading data from SQL. There is a total of 5 pipeline steps as shown in the image attached. Please note there is data in the database and all other configurations done correctly.

Iterable data is null when trying to fetch data from SQL Database by SQL provider in the Data exchange framework. Logs showing the pipeline steps and Iterable data is null info. I'm following this link

https://neilkillen.com/2017/11/25/sitecore-data-exchange-framework-the-sql-provider/

These are the logs

ManagedPoolThread #9 11:07:51 INFO [Data Exchange] Starting pipeline batch processing. (pipeline batch: SQL Customers to Sitecore Customers Sync)
ManagedPoolThread #9 11:07:51 DEBUG [Data Exchange] Iterable data is null. (pipeline: Read Customers from Database pipeline, pipeline step: Iterate Data and Run Pipelines Pipeline Step, pipeline step identifier: 92fb573e-4771-4d38-bdbd-12ca9237b3aa) 
ManagedPoolThread #9 11:07:51 DEBUG [Data Exchange] Pipeline processor finished processing pipeline steps. (pipeline: Read Customers from Database pipeline, pipeline step: Read Rows from Customers Table, completed pipeline steps: 2) 
ManagedPoolThread #9 11:07:51 DEBUG [Data Exchange] Pipeline batch processor finished processing pipelines. (pipeline batch: SQL Customers to Sitecore Customers Sync, completed pipelines: 1)

Answer 1

A few words about the Data Exchange pipelines. The synchronization configuration can have any number of pipelines to cover a business logic. You can define any order of pipelines because you always define which and when pipeline need to be run. You choose initial pipeline in a Pipeline Batch, then some pipeline step refers another one. But pipeline can't run itself. Things are different with pipeline steps. A pipeline step represents logic that runs during processing a pipeline. And they are run in order they are placed under the Pipeline.

In your case you have Iterate Data and Run Pipelines Pipeline Step above Read Rows from Customers Table which means you read data after you try to iterate it. Try swapping them and it will work as expected.