Sitecore XDB cleanup to limit size

Question

On our current system XDB is grown very large, the interactions DB is around 80% of the total database size, followed by KeyBehaviourCache at 5% then GeoIps and Contacts at 4%. We need to reduce the size to keep it maintainable.

Here are some ideas I have had on reducing the size:

• Delete all data in the interactions table over a certain age.
• Only keep X number of interactions per contact (only a good solution if we have a lot of repeat contacts).
• Delete all interaction data and rely only on the key behaviour cache.
• Delete all data over X number of days old.

Has anyone tried any of the above? What other techniques are being used to reduce the size of the XDB Mongo database? What impact has there been on performance when running cleanup tasks?

Answer 1

We have tried this with an adapted version of https://marketplace.sitecore.net/Modules/A/Analytics_Database_Manager.aspx Was going to release the adapted version, but will upgrade it to xConnect first.

Our approach is:

1. Filter our robots in a better way than the default (we have a custom filter in place that works with wildcards, otherwise it is impossible to keep up with a good robots lists). Also don't forget your own "robots" here: custom crawlers (e.g. for search indexes) or monitoring tools
2. Periodically use the cleanup tool. Strategy here depends on the customer. But mostly we delete the data over X days old. It can takes ages.. (weeks). But impact on the production environment is almost none. The Mongo takes more CPU while processing this but can take it. We do need to check server recycles though.. as it recycles, the cleanup job stops so we need to restart it.

Still not a perfect solution. But it did help to remove a few (lot) GB and save some money.