Page test not splitting 50/50

Question

I am currently running a page test with 2 page variants: Page A (original) Page B (modified with some additional components and personalisation)

I can see that sitecore is currently splitting the traffic 30% to Page A and 70% to Page B which is not what I expected. I have run a few tests previously and all of them have a 50/50 split, but for some reason my current test has an odd 30/70 split.

From what I've read in the sitecore documentation the variants shown should be controlled using the subgroup round robin sticky strategy and since there are only 2 variants in the test the split should roughly be 50/50 but the split is distinctly 30/70 from what I can see. I thought maybe sitecore would equal out the spit allocation but it's been 8 days already an the split is consistently 30/70.

I am currently using the following settings for my page test: Test Strategy: Subgroup Round Robin Sticky Traffic allocation: 100%, Expected effect: Positive, Confidence level: 90%, Duration min: 3 days, Duration max: 30 days

The only difference in set up between this test and my previous test is that I selected No change for the expected effect previously, also my previous test didn't have any personalisation. In this test there is some personalisation to hide certain components if the device is mobile or tablet, however I wouldn't expect this personalisation to effect the split allocation because this is a page test as opposed to a content or component test.

If anyone has an answer to why this might be happening then it would be much appreciated.

Answer 1

The reason why the traffic doesn't split 50/50 when using the Subgroup Round Robin Sticky test strategy is because your page has some components on the page with personalization rules.

This is how Subgroup Round Robin Sticky test strategy is described on the official Sitecore documentation:

The Subgroup Round Robin Sticky test strategy is similar to the Round Robin Sticky strategy: Sitecore presents the experiences alternately, to ensure that an equal number of visitors sees each variant. However, with this strategy, personalization rules are tested effectively because Sitecore distributes experiences evenly within subgroups.

If you don't want to take personalization rules into account, then you need to use the Round Robin Sticky test strategy.

In the Personalization Tests topic documentation page, the following example helps to understand why the traffic is not evenly distributed between the two variances when using the Subgroup Round Robin Sticky test strategy:

During a simple A/B content test, half of the contacts see the original content and half see the new version.

In a personalization test:

• All contacts who do not meet the personalization criteria see the original content.
• Of the contacts that do meet the personalization rule criteria, half see the new variant and half see the original content.