A/B testing is showing a change of 42% in Top Goal Converted. How Sitecore calculate this change value?

Question

I implemented A/B testing in Sitecore 10.1.2. Now we are getting a result of Top Goal Converter. Now I am not able to understand how this Change value 42% is calculated, please check the image below. The goal it is referring, is new and created for this test only. I checked below URL form Sitecore and from there it means the change between the original experience and the selected experience.

https://doc.sitecore.com/xp/en/users/101/sitecore-experience-platform/content-test-results.html

Can anyone please help me to understand how Sitecore is calculating the value?

Answer 1

When you run an A/B – or multivariate test, Sitecore will assess its success based on the trailing value/visit, which is the average collected engagement value after being exposed to the test. The logic behind this is: If there is a significant engagement value difference between the variations of your content, it can be attributed to the content the visitors see.

The engagement value is an integer that aggregates for each contact over time. The most common way of increasing the engagement value is to create a goal in Sitecore and assign it to an action, like visiting a certain page or submitting a form. The engagement value of a contact tells you how valuable this contact is for your organization, while the value per visit (= average change in engagement value per website visit) tells you how successful your website is in achieving your business goals.

So in your case, it measures the First Page Submission goal is a top converted goal during the test period and it got highrer converstion and engegement value then other goals.

Take a look this for better understanding.

Hope it helps!

Answer 2

Sumit's answer perfectly explains how the Engagement Value and Trailing value/visit numbers are calculated.

When it comes to Goals, Conversion rate and Change columns that are shown on the screenshot, these are calculated differently.

First of all, Conversions is the number of times each goal was triggered for visitors who were exposed to this A/B test. Conversion rate is percentage of visits with goal conversions out of total visits (i.e. the number of goal conversions after viewing the selected experience divided by the number of visits).

Then, for the Change column Sitecore compares conversion rate that is calculated separately for Original and Variant experiences of the A/B test:

Change = (VariantCoversionRate - OriginalConversionRate)/OriginalConversionRate

where VariantCoversionRate = VariantConversions/VariantVisits and OriginalConversionRate = OriginalConversions/OriginalVisits for the same goal.

So the Change column shows the difference in conversion rate between different variants rather than change in the absolute number of conversions or total engagement value.