Integration with custom-built Product Management System

Question

I can use some advice on an interesting architectural challenge for an organization in the travel industry :-) Sorry for the long read...

The situation:

• The company has multiple brands that are all migrating towards Sitecore
• Information on countries/regions/cities/accommodations/trips is stored in a custom developed Product Management System (PMS)
• The database contains 100k+ accommodations currently but is expected to grow over time (1 million is not unthinkable)
• They use the PMS to enrich information received from multiple third-party feeds. Think texts, (hotel) images, key/value pairs, properties and so on.
• The PMS contains brand-specific information as the accompanying content/images for label A can differ from label B for the same accommodation
• Not all accommodations/trips are available for all brands
• PMS data is stored in an MS SQL database
• The PMS offers an API that can be called to retrieve data

I have done quite some research and have also read this article (What are some appropriate uses for Sitecore Data Providers?) extensively.

Initially, the client had the plan to keep Sitecore 'dumb' in the sense that they planned to use wildcard pages in Sitecore for countries/regions/cities/accos that directly interact with the PMS API to retrieve the content. As you can imagine this could lead to a lot of inflexibility on the Sitecore end:

• No specific presentation details for individual pages, so workarounds are required for deviations from the generic wildcard presentation details (also for optimization/personalization)
• Sitecore Analytics is also a potential drawback. I'm not sure about the experience profile, but for one I know that the Path Analyzer does not handle wildcard items correctly
• SEO: this is very important for the client. Using wildcard items this should be taken care of in the PMS, which is currently not supported

There are three high-level approaches here IMO, with potentially a lot of hybrid variants in between.

1. Keep the wildcard setup, but take measures to counterfeit the inflexibility. Create custom rules that interact with the PMS API to support deviations/personalization on the generic presentation details;
2. Implement a custom data provider
3. Use the Data Exchange Framework to map/transform/synchronize data from PMS to Sitecore items

Internally we have a discussion mainly between options 1 and 3.

Option 1 would be the easiest to implement. But I'm questioning if this is a sound solution in the long run. Potentially you would have a single accommodation wildcard item with loads and loads of personalization rules. It seems unmanageable from my perspective.

If we would choose option 3, other questions arise. How are we going to map PMS to Sitecore? As I said, the company has multiple brands. The approach I'm thinking about now is to create a specific Data Exchange Framework implementation for each of the brands. While highly similar, it allows to prepare the data in exactly the correct format for use within the brand websites. So I would extract only the data relevant for the specific brand, and transform and map it directly to the page structure required for the specific brand. As I mentioned, it would require to sync a lot of data.

Other options we thought about:

• A hybrid solution where we would create the countries/regions/cities manually in Sitecore, and use a wildcard accommodation item below it. The idea is that most deviations from the generic presentation details would be on the country/region/city level and not so much on specific accommodation level.

• Use option 1, but create specific items for every personalization rule below the wildcard accommodation item. This would require some customizations, but ultimately you would have a list of deviations/personalization rules under the generic accommodation item. Easier to manage, since it would also become possible to use the publication mechanism and publish/unpublish dates

I'd love to get some feedback from you guys. How would you handle this?

Answer 1

While a very detailed question, there are still quite many "it depends" open ends. I'm going to make some assumptions based on experience, and give you my view.

I'll start by shaking the box to try and do some outside thinking ;-)

Shaking the box

• You don't really want to do page level personalisation on 100k scaling to 1m product pages. You want to do it on persona, brand and hotel meta data.
• Neither the CMS nor the PMS can do this without search technology
• Your url structure, something like /brand/hotels/europe/greece/rhodos/la-vida-loca is better suited as a search than a hierarchy. Why? because
• /brand/hotels/europe/greece/all-inclusive/rhodos/la-vida-loca

The fourth option

In your above 3, I say none of them really fits. When your product reaches the desired maturity in your PMS, I would ship it off to SOLR. SOLR scales and can handle your traffic load. You're going to need search technology anyway to serve the CMS needs for searching as well. And it can resolve your URL query at a speed that Sitecore could live with.

Optionally you could sync in these products via the Data Exchange Framework, and then use Sitecore Content Search to build the SOLR index. But don't do this unless you have specific reason to. In many cases this approach is a little bit like crossing the river to get water.

A more likely candidate, is to sync in surrounding meta data around products. That would be brand, region, country information. I would sync in these in their entirety. Use whatever url key is defined in the PMS and store it on the items you create.

Finding relevant hotels then becomes a matter of constructing a query based on these url keys. Like q?brand:brand-urlkey-from-field and region:region-urlkey-from-field and so-on-and-so-forth. This query mechanic will also serve your need for pagination, filtering and facetting - something I assume you need but your question does not mention.

In the above, you could also use Sitecore ContentSearch LINQ query searching if you must.

What's left out

In the above, I offer no solution for filtering out unavailable properties. Or not specifically. Since this information updates in near real-time, you should be looking at some form of distributed memory cache technology to keep this cached. Then inject the filter whenever you display search results. I would need more detail on how you have this data available today to offer up a better solution.

I also don't mention the actual product page. This is where you need to tweak a bit. Let's assume your canonical url ends up like this: /brand/hotels/la-vida-loca-27122. If your product item was in Sitecore, it would be a matter for the url resolver code to parse the url into the component search query, find the matching Sitecore item (based on the response from SOLR) and set the Context.Item. If the products are not in Sitecore, little changes. Likely you would want different layout per brand and possibly per region and so on.

Make this a problem for the Rules Engine.

where brand is brand a
  and country is greece

set product-page to /brand a/product page

Closing thoughts

• Make current search query available as ISearchContext
• Use this to build basic Rules Engine conditions
• Make current product available as IContextProduct
• Use this to build basic Rules Engine conditions
• Use both of the above to achieve personalisation scenarios
• Abstract your search implementation
• Don't try too hard to put Sitecore at the center of the world in this setup. It doesn't naturally fit.

Answer 2

Did some consultancy a few years ago in the travel business - they pushed their data into Sitecore but I'm quite sure they would never do that again..

I would try to go for a hybrid solution where you store some information in Sitecore and keep some (or a lot) in the PMS only. To decide what to sync I would consider:

• try to keep as much "pages" as possible in Sitecore, as that will be easier on analytics, SEO, customization, ...
• do not store data that often changes in Sitecore (you really don't want to sync lots of data)
• remember that you can easily combine an item in Sitecore (used as "page") with data from PMS: store only the minimal and necessary information on your Sitecore item and fetch the rest from the PMS (what this "minimal" is should be part of your analysis - hard to tell without all information)
• definitely try to avoid copying images from your PMS into Sitecore

p.s.: make good use of your indexes ;)

Edit: In theory Sitecore should be able to handle that much (1M) items. But that is theory - in a real world, keeping those items in sync will be very hard. So it actually depends on how volatile that data is. I would never store any texts or so about the acco's in Sitecore, you need your PMS and indexes for that. It could be an option to store acco's as item if you can only store a reference to the data (some id from PMS or so) and the list of acco's is easy to sync (syncing would then just mean deleting/adding items, you really don't want to sync content changes). And use indexes.. I remember that travel queries could become quite complex so a simple Sitecore index might not even be sufficient. That's why I like Mark's last line: Sitecore will be a layer to help you with the site front, your SEO, ... but the "real stuff" should be elsewhere.

Answer 3

I have done some similar solutions in the past – and I would suggest that you integrate the data into Sitecore, but create a custom Sitecore DB (without publishing) to hold the data. Also consider using a “brand” field on templates, to filter which brands has access to certain items, so you enable the solution to share content/products between the different brands.

For live updating stuff like availability or stock, you could implement a strategy pattern, that based on were the user is in the funnel, could implement more aggressive calls.

• Browsing the website – use last updated stock from Sitecore

• Adding to basket (or viewing the product) – If stock > 3 use from Sitecore else do live lookup

• Checkout – Live lookup

This can do wonders in improving response time, and also ensures that the site works, even if the PMS is unavailable for a period of time.