2

We are currently engaged in an XP Upgrade Project, migrating from version 9.0.2 to 10.3. As part of this project, we are using the ADM Tool 4.0.2 to clean up unused data from analytics. However, we have encountered a behavior that has raised some questions.

When attempting to remove historical data for a shorter time span using the ADM Tool, it appears to be selecting millions of records for removal. However, upon running a query directly in the database, we discovered that there are only hundreds of records with all interaction data for the same time span.

Our initial investigation has led us to believe that the tool may be placing the data in a queue that is not being cleared properly, which could explain this behavior.

We have made multiple attempts to cancel the task and run it again, but each time, the ADM Tool continues to identify and attempt to remove millions of records.

enter image description here enter image description here

Improve this question

1 Answer 1

Reset to default
1

I have seen similar behaviour with a newer version of the ADM module and having investigated it, I found that it's just how the module works.

You may know that ADM runs the cleanup in two phases: task generation and task processing. The task generation phase is quite complex and includes the following steps:

  • It retrieves all contacts from xDB regardless of the dates you specified
  • Then it starts iterating through all contacts starting from the newest records first
  • For each contact it reads additional data such as interactions and facets to filter out records that do not fall into the specified data range
  • If a contact matches all conditions and should be deleted, its ID will be saved to the table [Tasks] of the [ADM.Tasks] database for later task processing

So even if you specify a short date range in 2018, the module will still have to read all contacts including those from 2019-2023 and run checks for them. As existing SQL indexes are not utilised for filtering by date range, this process takes longer than a simple SQL SELECT query and causes high load on the Shard databases, especially for Data Input/Output.

Depending on your database size and how much data requires removing, you can try letting this process finish to see results and how long it takes. Here are some additional things to consider too:

  • If you have enough server and DB resources and want to speed up the process, you can try increasing settings NumberOfThreads and/or RetrieveDataBatchSize in the config (see section "6. Performance Tuning" in the ADM module documentation)
  • If your server or databases are maxing out, you can try upscaling them just for the cleanup (Premium SKU of Azure SQL databases is better for high Data I/O)
  • If you want to run this process for different date ranges separately, start with the recent dates first, but in fact splitting it into smaller ranges will not make the process faster because ADM will read all contacts each time
  • For older versions of Sitecore I also tried cleaning up data using direct SQL queries, but consider this option only if you are comfortable with SQL and xDB structure as there is a risk of leaving corrupted data in xDB
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.