17

I've ran into a number of different scenarios lately around xDB and data retention and archiving.

  1. What is the best practice for archiving data in xDB in general? For example, is there a best practice for how long to keep anonymous data? By archiving or outright deleting that data, rebuilding the reporting database will no longer give full historic data, right?

  2. For PHI environments that require 8-10 years of data retention, what is the best practice for archiving that data? Another Mongo instance? Encrypted to disk? How is this accomplished?

Improve this question
7
  • This is a great question and I'm interested in hearing specifically answers around part two. Just a quick comment to remind you to use encryption when dealing with PHI. Mongo 3.x has support for encryption at rest on the enterprise product. I'm sure you were already aware of that but I did want to remind anyone viewing this question in the future! – Derek C Nov 2 '16 at 13:54
  • 1
    WiredTiger only right? I thought that was only Mongo 3.2? And Sitecore 8.2 is the only version that officially supports Mongo 3.2 as far as I am aware. – sestocker Nov 2 '16 at 14:41
  • 1
    Correct on both counts -- I should have said 3.2 there. However, there options for at-rest encryption in prior versions with Bitlocker (Win Server 2012), for example. – Derek C Nov 2 '16 at 14:46
  • With regard to PHI, I've been well entrenched with dealing with HIPAA over the years, having worked for healthcare companies for a while before getting into the agency space. When dealing with Sitecore clients that have PHI at their disposal (hospital chains, insurance companies, etc.) my immediate response is that Sitecore is not a system to keep PHI in. Leave the storage of PHI information to their original systems, and only expose what endpoints you need in order to achieve desired functionality. Just two cents on this topic. – Pete Navarra Nov 2 '16 at 17:51
  • 2
    Regarding the first set of questions, the answer is heavily dependent upon client needs and expectations. My understanding is that the Mongo Collection is intended to be a the full body of all analytical the client wishes to have reporting on. If they only want to go back two years, then you'll want a process that purges mongo data out that is older than 2 years, and at the same time, rebuild the reporting database. doc.sitecore.net/sitecore_experience_platform/… – Pete Navarra Nov 2 '16 at 18:00
4

MongoDB supports TTL indexes so you could, for example, configure xDB data retention of 6 months (or however long you'd like). The duration would depend on what all you're doing with the xDB data. I talk about this at https://grantkillian.wordpress.com/2016/08/11/sitecore-and-ttl-index-heresy-for-mongodb/

TTL indexes for MongoDB are documented at https://docs.mongodb.com/manual/core/index-ttl/

Improve this answer
3

What you could consider:

It is always possible to rebuild the reporting database if your mongodb data does not get corrupt. Therefor you can choose to keep a copy (or compressed archive) of your data in another location. For the searches that I did, there is no best practice from sitecore yet.

You could consider installing the mongodb ops manager who will keep the backups for you. I personally think you can easily store 10 year of data in mongodb as long as your infrastructure is capable of doing that.

Maybe in future releases there will be features who will assist you, by doing this for you. Keep on hoping.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.