What are some appropriate uses for Sitecore Data Providers?

Question

Data Providers are difficult to develop, given the performance and data mapping challenges they can present. But they are powerful when done correctly:

• Allowing real-time integration with external systems
• Present as Sitecore items that can be used for linking or even to drive pages
• Can enable editing of data in external systems via Sitecore

In what cases would one create a data provider instead of importing content? What are examples of practical integration scenarios which would benefit from a Data Provider implementation?

Answer 1

It's a difficult one; a question I've considered many times over in the years I've been working with Sitecore.

I can start with some numbers. I've done perhaps 30 or so projects that involved data integration of some kind. 29 of which, I ended up importing the data. I've only ever done 1 project where I've implemented a full Data Provider - and I'll highlight why:

Data is often imperfect

This is especially the case for most e-commerce solutions I've worked. This is usually product data, product meta data, category trees and so on. It's very rare that one can source all of the required information (short descriptions, long descriptions, images, product attributes and so on) from the back-end systems. So what I've ended up doing; is doing multiple integration runs - enriching the product catalogue in each run. Then when complete - publish the enriched data.

Back-end systems are too slow for practical website use

This is especially true for ERP and PIM systems. They are designed with data integrity in mind, and only secondly have an eye on performance. If even that. Add to this; the nature of data in these systems is rarely a tree-like structure (rather; relational) - making it difficult to make them fit into the parent/child nature that a Sitecore Data Provider has to follow.

Lastly; if opening up a Data Provider to a back end system - it needs to be available all the time, or the Content Editor users find themselves out of work. And publishing an empty structure... doesn't really end so well.

Sitecore's data model is richer than most back-end systems

I argued this in Working with multiple content databases in Sitecore 6.

Sitecore offers versioning, language versions, workflows, many types of fields - right out of the box. Writing a Data Provider, one often ends up stubbing most of this functionality out, thus detracting from the overall Sitecore experience. The data looks like it's a native part of the Sitecore content tree, but it's really not.

So where?

Well I honestly don't see where I would use one today. Unless writing a very specialized tool - like Kamsar's Unicorn or so. For live use on the website? We have so many other ways of abstracting data - Dependency Injection, Services, Web APIs and so on - I really don't see any added value in feeding this data in via a Data Provider, just so we can use Sitecore Query (brr) to query it. I think Data Providers made sense, when XSLT was still the de-facto standard for rendering Sitecore content.

I think those days are behind us.

EDIT: elaboration

The whole point of a data provider is; to bring in content and have it appear as native Sitecore content. The idea being; you then bring in your content from multiple sources, and then have it work just like had it been created natively in Sitecore. And yes; part of that was based on creating XSLT renderings, and they shouldn't need to know what content was native and what was not - and with XSLT renderings there simply is no easy option to render non-Sitecore data.

So how are these used?

Well presumably your data provider is one-way. You can implement a 2 way provider but I've never had a business case for it. Usually one integrates to a business system and we can't just let any editor simply edit content (by mistake - in Experience Editor this could easily happen) and have it written back to the origin system.

So we feed in data from the provider; and we probably feed it directly into "master" and "web" in many instances, as support for publishing of provided data is extra work. Plus - you data provide to have a real time view to the origin system and publishing would sort of ruin that idea. Now you have to decide how to deal with caches; external link to origin system breaking down (and taking your solution down in the fall)... meh the list just goes on.

So what can we do instead?

Lots. Ruling XSLT out again; say you wanted to reach data from a back end CRM database. Your Helix feature would be something like "CRM Integration" (pardon me if not up to speed on any naming conventions in Helix). And go for it; provide the project layers with ICrmIntegration if you want. And if you desperately crave a Sitecore Item tree, nothing is stopping you from mocking one up at this point. Especially not in new Sitecore releases. Something like

var customersRoot = _crmIntegration.GetCustomersRoot();
foreach(Item customer in customersRoot.GetChildren())...

Possible? Sure. But looking at it like this, it becomes obvious it just doesn't make much sense.

If there is a case for a Data Provider, it would only be a fully implemented 2-way provider. To save you having to write a web interface to update some data. Nothing else rings any kind of sense to me. Not today, not post XSLT. Not with models, Dependency Injection, Layered Architecture. We have so many better options available.

Answer 2

I've used DataProviders fairly extensively and I'd say that they're good if: 1. Your API is fast (not too many calls to get the info, easily cached etc) 2. You need data to be up do date (to reasonable caching margins) 3. You're not going to have masses of items within each folder

The biggest issue I've found is how hungry Sitecore is getting children - this can lead to slowww DPs. Also overuse of the IDTable can cause issues if items with loads of fields (because of the way Sitecore treats all fields as items) - be aware that the IDTable calls aren't cached.

I'm working on a bucket DataProvider at the moment (it's not quite working how I'd like at the moment) so I'll blog about that when it's ready: breaksitecoreblog.wordpress.com

I'd say it's a trade-off between how efficient you can make the DP and how often the data needs to be updated.

Answer 3

A good example is exposing information from a PIM (product information manager) as native Sitecore items in the Sitecore tree. With this, production information can be accessed via the same API's as you'd use for accessing content managed via Stiecore.

Answer 4

I think there are some good individual examples here of useful Data Provider usage, and some good arguments about when/why you should not use them, but I'd like to summarize the situations in which a Data Provider can work well, and provide some concrete examples.

A data provider can potentially be a good option if:

• The data is local to the Sitecore instance, with very fast query/API, and has a hierarchical structure that maps well to Sitecore.
• There are significant advantages to the data being represented as Sitecore items. E.g. being able to browse items for linking or datasources.
• You want to be able to edit the data within Sitecore OR the data you are exposing does not require "enrichment" within Sitecore.

Some examples of useful Data Provider applications:

Commerce Systems

Both uCommerce and Sitecore Commerce use data providers to expose products to the Sitecore content tree. Commerce systems tend to have their own data modeling capabilities, and using a data provider here allows the template model for the product items to be dynamic as well. Having products exposed as items provides huge advantages for linking to products, product URLs, etc. This also allows for product data to be edited in Sitecore as well. If you do allow for editing within Sitecore, you do need to be careful of any locking, caching, etc. that could be happening within the other editing interface.

Product Information Management (PIM) Systems

Similar to commerce systems, PIMs tend to have their own robust data model, and there are big advantages to having those products represented as items. Most PIMs will have some sort of language version concept that can be mapped into Sitecore's data model as well. Typically with a true PIM however you will not want to implement data writes, and you will need to disconnect that data through publishing (i.e. a master db-only Data Provider). The PIM will need to have a fast API, or Content Editor and publishing performance will suffer. Many PIM integrations likely can be handled with an import instead, since publishing means the integration no longer provides "real time" data. But in cases where you have a very dynamic data model, a data provider could have advantages.

Media and Digital Asset Management (DAM)

If you are storing media in a DAM of some sort, a Data Provider implementation could allow you to browse those assets in real time, and use them as if they were native Sitecore media, then publish as needed (and disconnect them from the DAM in the process).

Exposing External Data on Items

This is perhaps more of an edge case. Though the Active Commerce module for Sitecore stores products as native Sitecore items, it stores product stock information in its own database, so that stock levels can be quickly queried and transactionally incremented/decremented. However it also uses a Data Provider to expose those product stock levels in the Content Editor, and allow them to be manually adjusted if need be.

It's also worth noting in this example that Data Providers do not need to own the "whole item" -- they can just provide particular fields. When it comes to performance, this can have advantages when it comes to exposing, editing, and/or publishing external data.

Answer 5

I have used them before to inject custom Security on an item level. In my particular case the security requirements couldn't be fulfilled by Sitecore security, so my data provider would perform the calculation instead and return items appropriately.

I wrote about the implementation details here: http://jockstothecore.com/item-provider-for-security/

Answer 6

In the past, I have recommended a data provider while working with a healthcare client. They had a member database that contained HIPAA (Health Insurance Portability and Accountability Act) protected data that was not suitable for storing in Sitecore, however, was the source of truth for pieces of information that needed to be accessed within the confines of their Sitecore implementation.

For a little background, health insurance company that was building a member portal using Sitecore.

Recommendation was to use a data provider to access the information that we needed without having to do a bulky import process as well as make sure that we weren't bringing over data protected by HIPAA, while at the same time, having reliance that the data coming over was synced and always up-to-date, without having to build automatic task jobs, etc.

As a rule of thumb, I generally tell prospective clients, that Sitecore is a marketing platform, not a database for sensitive information.

End result was a great application of the data provider, and met the needs of the client.

Answer 7

I have used Data Providers for PIMs, employee/personell/staff/etc. directories, and also for large quantities of data points coming from a source that is managed in another system but referenced by Sitecore items.

There are a few really great features of Data Providers that I haven't seen anyone else mention yet:

• You can easily tie the "items" that they "provide" into the built in Content Search.
• Data Providers can be used to add publishing support for content coming from a third-party system
• You can make Sitecore templates for the "provided items", you can use templates stored by the Data Provider, itself, or you can generate dynamic templates to match the data

Key Things to Remember:

1. Items coming in from a Data Provider are not real items, but they are made to look like items and can be referenced like items.
2. A given piece of content should only ever be editable in one location. If you have a Data Provider that is pulling in content from another system where said content is authored (CMS, PIM, CRM, etc.), do not add two-way authoring support.
3. A Data providers can be implemented with multi-lingual support (leveraging OOTB Sitecore multilingual support), but this means that your data store must support this, as well.

Answer 8

We have a situation were we will probably be moving data outside of Sitecore into a custom database and write a data provider on it. Might become a read-only data provider even.

We are facing a tree structure with products that has grown over a million items. The back-end systems with the original data cannot be used in a live environment for several reasons: some are not always available (down for an hour during night), some are external sources and do not allow lots of queries. We already have pricing and stock information outside of Sitecore as that is updated at least every hour.

The main reason we are considering to move away from the Sitecore databases in this case is the import (add and update) of items. Importing is done through files or through services that are send/called once a day. But sometimes the number of items we need to import is so big that processing the data and creating the items if needed takes so long that we get in trouble. Long meaning here that the job isn't done in a day. More and bigger servers might help, but not in the long run. Requirement is set to be able to import whole sections (few hundred thousand items) at once. Knowing that creating items in Sitecore is not the fastest thing to do (we do also require some side effect jobs such as custom indexes, insert/update statements to custom db, ...) made us really consider a data provider on top of something else (custom db in our case).

Answer 9

If you have data in a back-end system that has network access limitations, using a Data Provider allows Sitecore to publish the data as Items, sparing you the need to worry about VPNs or latency when accessing that data from a Content Delivery server.

It's not the only path you can take, and there are many caveats, but for front-end performance, it's occasionally useful.

Answer 10

They are appropriate when you need to integrate with external data sources and want to take advantage of Sitecore's item processing, tracking and manipulation.

One example would be a university: We have many different data sources that we need to work with, one of which is a Course Database. Each course gets it's own page and content associated with it.

One solution would be to have a wildcard page, read the URL for the course id and then render the content manually (by reading it with some database tool)

The problem with this is that we lose the advantages of using sitecore's searching and indexing ability. We can not do site wide HTML rendering and search. There are also secondary tools that are lost out on, namely, Glass Mapper. By integrating the courses directly into Sitecore we can use glass mappers insertion pipeline and reduce the amount of code we need to write (and bugs) by eliminating controllers and rely purely on view renderings. Ultimately, reducing the complexity of our code.

Answer 11

One use case for data providers could when you're creating a module. For instance uCommerce (an e-commerce solution using commerce connect) uses Data providers to make product and catalog data available in Sitecore.