Why are Sitecore roles not persisting quick enough when using a scaled CD?

Question

User scenario is as follows:

1. Anonymous user navigates to website home page
2. Automatically redirected to /login, which detects user is not authenticated & redirects to ADFS
3. User authenticates on ADFS & redirects back to /login
4. Login page processes the ADFS data, logs the user in as a virtual user, adds some Sitecore role & redirects user
5. Users ends up on page they originally requested on step 1

Depending on the route taken, you get a different result

Route 1:

1. CD 1 (home page)
2. CD 1 (/login)
3. External (ADFS)
4. CD 1 (/login) - Sitecore roles applied
5. CD 1 (home page) - user has retained the Sitecore roles applied

Route 2:

1. CD 1 (home page)
2. CD 1 (/login)
3. External (ADFS)
4. CD 1 (/login) - Sitecore roles applied
5. CD 2 (home page) - user is missing the Sitecore roles applied from step 4

If you refresh the page on step 5, and receive CD 2 again the roles will be present. To test a theory, we added a 2 second delay after the roles have been saved to the user, step 4. In that instance, scenario 2 persists the roles after the redirect.

The above scenario & routes have been tested manually and also put through a load test to create higher volume.

Azure Landscape

Application Gateway >> Load Balancer >> 3 x CD (Virtual Machine Scale Set - D4s_v3) >> Redis (Standard C2)

---

Edit 1: Removed notion of Redis as it's clear it's not part of the equation.

Answer 1

After a ticket to Sitecore support and some investigation of our own, we've had to go down the route of using sticky sessions. This keeps the user on the same CD after their initial login.

Sitecore support confirmed my theory that the roles are not being persisted quick enough by the time the redirect is done:

1. Properties of virtual users are stored in in-memory Runtime Settings (CD1)
2. These runtime settings are written to a Client Data Store, the [ClientData] table of a 'core' database in your case, thus virtual user properties are shared among instances.
3. Since CD1 and CD2 instances have their own independent memory (also a cache with user settings), this cache is periodically (not immediately) updated using [EventQueue] records.
4. Considering that your frequency equals 5 seconds, CD2 may be not aware about CD1 virtual user changes up to 5 seconds.

The alternative option to using sticky sessions is to reduce the scheduling frequency, but given the redirect happens so quickly I don't think there would be a guarantee.