I am using Sitecore 9.3 and xConnect.
The issue is xDB indexes are consuming disk space more than 100GB.
This disk space consumption is increasing day by day. We are using XDB for interactions and showing the aggregated data in power BI. We are not using sitecore analytics dashboard. I plan to delete the xDB indexes.
Is there any impact if i delete the xDb index data?
High disk space consumption in xDB indexes may occur due to a large amount of data being collected and stored. To manage this, you can take several steps to optimize and reduce the size of your xDB indexes without losing valuable data. Here are some suggestions:
Index only required fields: Review the fields being indexed and ensure that you're only indexing the data that's necessary for your reporting and personalization needs. You can update the configuration files to exclude unnecessary fields from indexing.
Configure data retention policies: Set up data retention policies in xConnect to automatically remove data that's no longer needed. You can define the retention period based on your requirements, and the data will be deleted after the specified period. You can find more information here https://doc.sitecore.com/xp/en/developers/93/sitecore-experience-platform/tracking-changes-to-the-xdb-index.html
https://doc.sitecore.com/xp/en/developers/93/sitecore-experience-platform/configuring-the-xconnect-search-indexer.html#sc-xdb-collection-data-sql-xml_body
Think about tracking anonymus users interaction if you really need to track this information. Because it takes ton of space. https://doc.sitecore.com/xp/en/developers/93/sitecore-experience-platform/enable-indexing-of-anonymous-contacts-in-the-xdb-index.html
Monitor index growth: Keep an eye on the index size and growth rate to understand the trends and make adjustments as needed. This will help you proactively manage storage space and ensure that you're not consuming more space than necessary.
Partition the indexes: If your xDB indexes are growing too large, consider partitioning them based on date or other criteria. This will help you manage the indexes more effectively and reduce the overall disk space consumption.
As for deleting indexed data, doing so will result in the loss of historical data and can impact reporting, personalization, and other features that rely on this data. Be cautious and make sure to understand the implications before deleting any indexed data. It is recommended to create backups before performing any data deletion operations.
If you decide to delete the indexed data, you should also rebuild the indexes to ensure that the search index accurately reflects the data in xDB. Note that rebuilding the indexes can be a time-consuming process, especially for large datasets.
xDB is a marketing database. As a rule of thumb, you should only store data in xDB if it's required for Personalization, Targeting and Measurement / Analytics.
You can delete a contact and all associated interactions, facets, and identifiers. Use this functionality to remove data that is no longer valuable to your business - for example, you may want to delete contacts that do not have any interactions from the past two years.
xConnect allows you to delete contacts and all associated data in xDB if you choose to. So, you have the ability to remove the data that isn't useful to you anymore.
xConnect allows you to extend the XConnectDataAdapterProvider, and this allows you to hook into the sweet spot where the session data is being written to the shard databases for storage. You can follow this article to reduce xDB growth.
Extending xConnect To Reduce xDB Growth
When you delete a contact the following data is removed from the xDB Collection database:
Interaction facets
Interactions
Contact facets
Contact
Contact identifier indexes
The xDB Search Indexer removes all contact and interaction data from the xDB index
Hope this helps.
Another thing to consider, which hasnt been mentioned above is deleting users with huge volumes of interactions. I had a problem with this when migrating data between 9.3 and 10.1, whereby one contact had over 150,000 interactions.
Locating contacts
SQL
SELECT TOP (100) ContactId, COUNT(ContactId) as Count
FROM [xdb_collection].[Interactions]
GROUP BY ContactId
ORDER BY Count DESC
Mongo
db.getCollection('Interactions').aggregate(
[
{ $group: { _id: "$_id.parent_id", count: { $sum: 1 } } },
{ $match: { count: {$gt: 1000} } },
{ $sort: { count: -1}}
],{ allowDiskUse: true })
Further info: https://support.sitecore.com/kb?id=kb_article_view&sysparm_article=KB0417184
Blocking Robots
You can then block future visits from getting logged from the same IP address via the Sitecore.Analytics.ExcludeRobots.config file, which has a list of excluded IP addresses.
Delete contacts
To remove the contacts from the collection database.
