Consuming a RESTFUL API with entity relationships

Question

Using DEF 2.01 and Sitecore 9.01, I'm consuming a RESTFUL api, returning JSON, with the methods being

• https://mydomainname/experience-groups
• https://mydomainname/experience-groups/29
• https://mydomainname/experiences
• https://mydomainname/experiences/32

I can read and create experiences and experience-groups from the api calls that return the listings i.e. /experience-groups and /experiences and they fit well with the model of using GetIterableData in the processor as per the example in the DEF cookbook.

I need to also call /experiences/id and experience-groups/id for each item to get additional detail and also as experience-groups/id contains a list of experiences.

Im not sure how to implementing this as I have written my processor to be very generic and not contain any logic for getting child items or establishing relationships. Does anyone have any experience of using DEF with a RESTful api with relationships.

My processor code if that helps:

 [RequiredEndpointPlugins(typeof(JsonServiceEndpointSettings))]
    public class ReadJsonApiProcessor : BaseReadDataStepProcessor
    {
        public ReadJsonApiProcessor()
        {
        }
        protected override void ReadData(
            Endpoint endpoint,
            PipelineStep pipelineStep,
            PipelineContext pipelineContext,
            ILogger logger)
        {

            //add the data that was read from the file to a plugin
            var data = this.GetIterableData(settings, logger, pipelineContext);

            JArray result = ExtractArray(settings, logger, data);
            logger.Info("{0} json objects were read from endpoint. (pipeline step: {1}, endpoint: {2})", result?.Count ?? 0, pipelineStep.Name, endpoint.Name);

            var dataSettings = new IterableDataSettings(result);

            //
            //add the plugin to the pipeline context
            pipelineContext.AddPlugin(dataSettings);
        }


        protected JToken GetIterableData(JsonServiceEndpointSettings endpointSettings,
           ILogger logger, PipelineContext pipelineContext)
        {

            using (var client = new JsonRequestService().GetHttpClient(endpointSettings))
            {

                JToken toReturn = null;
                var url = endpointSettings.Url;
                var response = client.GetAsync(url).Result;
                switch (response.StatusCode)
                {
                    case HttpStatusCode.OK:
                        toReturn = response.Content.ReadAsAsync<JToken>().Result;
                        logger.Debug($"Object loaded (url={url})");
                        break;
                    case HttpStatusCode.NotFound:
                        logger.Info($"Object not found (url={url})");

                        break;
                    default:
                        logger.Warn($"Error reading object information: {(int)response.StatusCode}-{response.ReasonPhrase} (url={url})");
                        pipelineContext.CriticalError = true;

                        break;
                }
                return toReturn;
            }
        }

        private JArray ExtractArray(JsonServiceEndpointSettings readJsonObjectsSettings, ILogger logger, JToken jToken)
        {
            var result = jToken == null ? new JArray() : jToken as JArray;
            if (result == null)
            {
                //select root node
                if (!string.IsNullOrEmpty(readJsonObjectsSettings?.RootJsonPath))
                {
                    try
                    {
                        result = jToken.SelectToken(readJsonObjectsSettings.RootJsonPath) as JArray;
                    }
                    catch (Exception ex)
                    {
                        logger.Error($"Error using '{readJsonObjectsSettings.RootJsonPath}': {ex.Message}");
                    }
                }
            }
            return result;
        }

Answer 1

So, if in order to get additional detail you have to send a new request by id, I can propose 2 options.

Option 1

Create Pipeline step processor that gets all data and loops through them in order to get/send additional request to get data detail by id. (see 4)

Next example demonstrates how to sync all Experience Groups.

Option 2

Create two Pipeline step processors

• Get all data (see 4)
• Get data detail by id (see 6)

Next example demonstrates how to sync all Experiences.

Answer 2

I have implemented this by reading all Sitecore experiences using a Read Sitecore Items Pipeline Step, iterating through it in another step, then another pipeline to call the experience detail api, and finally using the existing pipeline to update the Experience in Sitecore. Job done!