13

We have noticed that our MongoDB's are growing steadily. Which is an obvious result of just capturing all kinds of user interactions. However, we can't have the xDB grow infinitely...

I can't seem to find any config in Sitecore that provides the means to clean up on the collected MongoDB information, provide a TTL for xDB data or anything of that kind.

Furthermore, data is growing way faster than expected since a lot of bot analytics are also captured as the filtering done by ootb Sitecore is not sufficient.

Any general approaches or advice on how to handle this? Because we can hardly pitch this to our customers and inform them that the MongoDB size will keep increasing over time (or their subscription will). And old collected analytics data (older than 2years) is simply not relevant enough anymore. Or should become achievable.

Improve this question
4
  • 2
    This is duplicate of sitecore.stackexchange.com/questions/2121/… which sadly has no answers as yet. The basic gist is, I believe, that you can't do that as an out of the box feature yet. sitecore.stackexchange.com/users/246/pete-navarra has the best suggetion in the other question.
    – Steve Newstead
    Nov 4, 2016 at 15:35
  • @SteveNewstead The questions are indeed similar. I wouldn't consider them duplicates though. This question asks how to remove old data. The OP in the other question seems to want old data to be archived, which means the ability to restore it and use it in the future. Besides, the other question has the PHI aspect to it. I think both posts each have their own merit.
    – Dmytro Shevchenko
    Nov 4, 2016 at 17:56
  • I don't think it would be difficult to do, but for every record you clear out of Mongo, you would need to clear out of the MSSQL reporting DB. There is an association for the user sessions.
    – Chris Auer
    Nov 4, 2016 at 18:21
  • It would be handy to know where archiving / deleting data fits into the Sitecore roadmap, we can then let customers know that data will grow for the next X months before a tool lands
    – Steve Newstead
    Nov 5, 2016 at 9:06

2 Answers 2

Reset to default
4

The new version of the Analytics Database Manager allows to clear historical data in the collection (mongo) database. It allows to specify various conditions to keep only important data. I hope it is helpful for you: https://marketplace.sitecore.net/en/Modules/Analytics_Database_Manager.aspx

Improve this answer
6

Grant Killian wrote on his blog about using MongoDB TTL (Time to Live) Indexes to continuously purge xDB data based on an interval.

The crux of the post that pertains to this:

MongoDB supports “Time to Live (TTL) Indexes” which purge data based on time rules. Data older than 1 week, for example, could automatically be removed with this index type.

...snip...

While pursuing this effort on behalf of session state management, I realized this could be an intriguing solution to data retention challenges with Sitecore’s xDB. Using MongoDB TTL indexes for the xDB collections would prevent that data from growing out of control. Set a TTL value of 180 days, for example, and make use of just the most recent 6 months of user activity as part of content personalization, profiling, etc. Of course, one sacrifices the value of the old data if one expires it after a set time. Remember, I’m acknowledging this is heresy!

Improve this answer
2
  • And we would need to purge the corresponding records from the MS-SQL Reporting database. Leaving them there would result in errors when the user would click on a session that no longer existed in xDB.
    – Chris Auer
    Nov 7, 2016 at 15:47
  • Yeah, you'd have to look at the aggregation pipelines to make sure that the processing services are accurately picking up the changes and updating the reporting DB correctly.
    – Jason St-Cyr
    Nov 7, 2016 at 21:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.