How can I use the Sitecore Data Exchange Framework to create child items from XML data?

Question

I am building a custom XML provider for SC9.0.1 / DEF 2.0.1. The example XML below shows a simple list of Doctors. And each Doctor has a list of Facilities. I would like to be able to sync the doctors and also have the list of facilities for each doctor live beneath each doctor in Sitecore.

<?xml version='1.0' encoding='UTF-8'?>
<Teachers>
    <Doctor>
        <FirstName>John</FirstName>
        <LastName>Smith</LastName>
        <DoctorID>123678</DoctorID>
        <Facilities>
            <Facility>
                <Name>FacilityA</Name>
                <ID>123</ID>
            </Facility>
            <Facility>
                <Name>FacilityB</Name>
                <ID>234</ID>
            </Facility>
        </Facilities>
    </Doctor>
    <Doctor>
        <FirstName>Jane</FirstName>
        <LastName>Jones</LastName>
        <DoctorID>123978</DoctorID>
        <Facilities>
            <Facility>
                <Name>FacilityC</Name>
                <ID>678</ID>
            </Facility>
            <Facility>
                <Name>FacilityD</Name>
                <ID>289</ID>
            </Facility>
        </Facilities>
    </Doctor>
</Doctors>

I was able to follow the Sitecore documentation and also the example custom provider and handle the sync of Doctors pretty easily. What I can't figure out at all is how to sync the list of Facilities for each Doctor. I get the sense that somehow I should create a separate pipeline to handle the list of facilities for each doctor. But I can't figure out how to set the source for the facilities. I think it has something to do with using the pipeline context maybe.

I read this post - How to handle child data collections. From what I can tell from reading the answer there it seems like it has to do with using a "Copy Object from Context to New Location Pipeline Step". But I can't seem to find any real documentation on how to use a step of that type.

Any ideas?

UPDATE: here is a screen shot showing my current pipeline steps

And here is the processor that handles the XML data:

using Sitecore.DataExchange.Processors.PipelineSteps;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using Sitecore.DataExchange.Contexts;
using Sitecore.DataExchange.Models;
using Sitecore.Services.Core.Diagnostics;
using Sitecore.DataExchange.Attributes;
using System.IO;
using System.Xml;
using Sitecore.DataExchange.Plugins;

namespace MyProject.DataExchange
{
    [RequiredEndpointPlugins(typeof(XmlSystemSettings))]
    public class ReadXmlDataStepProcessor : BaseReadDataStepProcessor
    {
        public ReadXmlDataStepProcessor()
        { }

        protected override void ReadData(Endpoint endpoint, PipelineStep pipelineStep, PipelineContext pipelineContext, ILogger logger)
        {
            if(endpoint == null)
            {
                throw new ArgumentNullException(nameof(endpoint));
        }

        if(pipelineStep == null)
        {
            throw new ArgumentNullException(nameof(pipelineStep));
        }

        if(pipelineContext == null)
        {
            throw new ArgumentNullException(nameof(pipelineContext));
        }

        var settings = endpoint.GetXmlSystemSettings();

        if(settings == null)
        {
            return;
        }

        if(string.IsNullOrWhiteSpace(settings.XmlPath))
        {
            logger.Error("No path is specified on the endpoint. " +
                "(pipeline step: {0}, endpoint: {1})",
                pipelineStep.Name, endpoint.Name);
            return;
        }

        var path = settings.XmlPath;

        Uri uriResult;
        bool result = Uri.TryCreate(path, UriKind.Absolute, out uriResult) && (uriResult.Scheme == Uri.UriSchemeHttp || uriResult.Scheme == Uri.UriSchemeHttps);

        if(!result)
        {
            if(!Path.IsPathRooted(path))
            {
                path = string.Format("{0}{1}", AppDomain.CurrentDomain.BaseDirectory, path);
            }

            if(!File.Exists(path))
            {
                logger.Error("The path specified on the endpoint does not exist. " +
                    "(pipeline step: {0}, endpoint: {1}, path: {2})",
                    pipelineStep.Name, endpoint.Name, path);
                return;
            }
        }

        //Main processing of XML goes here
        var lines = new List<string[]>();
        XmlDocument document = new XmlDocument();
        document.Load(path);
        XmlNodeList xmlNodeList = document.GetElementsByTagName(settings.XmlNodeName);
        XmlNode[] nodeArray = xmlNodeList.Cast<XmlNode>().ToArray();
        for(int i=0; i < xmlNodeList.Count; i++)
        {
            List<string> strs = new List<string>();
            for(int j=0; j < xmlNodeList[i].ChildNodes.Count; j++)
            {
                strs.Add(xmlNodeList[i].ChildNodes[j].InnerText);
            }
            lines.Add(strs.ToArray());
        }

        var dataSettings = new IterableDataSettings(nodeArray);
        logger.Info("{0} rows were read from the file. (pipeline step: {1}, endpoint: {2})",
            lines.Count, pipelineStep.Name, endpoint.Name);

        pipelineContext.AddPlugin(dataSettings);
    }
}

}

UPDATE 2: I am getting closer to figuring this out. The Get Doctor Facility Info step seems to work. I am able to write to the log and see that it is properly iterating through the facilities for a given doctor. Here is a screen shot of my pipelines now:

I am now getting an error in the Handle Individual Facility pipeline in the Apply Mapping step. The error says: Pipeline step processing will abort because no source object could be resolved from the pipeline context.

I believe this means that the previous step (Resolve Facility Item) did not put the item in the right place or something. Below are 2 screen shots of what my Resolve Facility Item looks like. I'm not really sure I have this set up properly. Clearly I don't understand how DEF manages data in the context.

Answer 1

You can resolve this issue in a different way.

I am a bit updated your configuration to explain how I see.

When (7) was executed, means doctor was created. Doctor object still in the Pipeline Context Source location.

(8) is Copy Object from Context to New Location Pipeline Step. Here you need to set Source Object Location and it is a Pipeline Context Source because doctor is still here.
Source Object Value Accessor (that is what you need to create and set) has a reader. The reader should extract Facilities from Doctor. Read data will be located to specified Target Object location. In other words processor takes data from source, pass it to the accessor where accessor reader knows how to read Facilities and pass it to a target.

As you can see in Target Object Location the Pipeline Context Iterable data set. So for the Facilities you have to repeat scenario. It means that you need also iterate them. When you create your reader the reader should return IEnumarable. Just remember we still need the Doctor in Pipeline Context Source. We need it in order to resolver for the single facility can resolve parent for item location and this time it in the pipeline context.

Many pipeline steps have locations settings. Here is an example of available.

When you set one of them processor knows how to read source object. For example:

• Pipeline Context Source. Processor tries to read source from Source property of SynchronizationSettings object.
• Pipeline Context Target. Processor tries to read source from Target property of SynchronizationSettings object
• Pipeline Context Iterable Data. Processor tries to read source from Data property of IterableDataSettings object