404 redirections for valid pages

Question

Our website (Sitecore 10.1.3) is hosted on Sitecore managed cloud. We observed that there are many Failure Requests in Azure App Insights (AAI) and the majority of them are 404 errors.

I have implemented logging to log the "Requested URL" on the incoming request.
Based on the appended query string parameters, I have assumed the original requested page.

The following are the URLs of the 404 page, from AAI and the metric is operation_name = GET /404

Requested URL	Assumed Requested Page (Relative URL)
https://mc-guid-cd.azurewebsites.net/404?item=%2f&layout=%7b00000000-0000-0000-0000-000000000000%7d&device=Default	/
https://mc-guid-cd.azurewebsites.net/404?item=%2Fsing&user=extranet%5CAnonymous&site=website	/sing
https://mc-guid-cd.azurewebsites.net/404?item=%2Fwork-and-stay&user=extranet%5CAnonymous&site=website	/work-and-stay
https://example.com/404?item=%2f&layout=%7b00000000-0000-0000-0000-000000000000%7d&device=Default	/
https://www.example.com/404?item=%2Fsing&user=extranet%5CAnonymous&site=website	/sing
https://example.com/404?item=%2Fwork-and-stay&user=extranet%5CAnonymous&site=website	/work-and-stay

Some of the URLs captured in logging have the Azure App Service URL and some have the website domain.
All those URLs are valid and will load the respective page when browsed and will not redirect to the 404 page.

Could you please help me understand, what could be the reasons that valid pages are being redirected to 404 page or captured as 404 errors in AAI.

Were they really redirected or there is any other reason for this capturing.

How can I troubleshoot?

Thank you

Answer 1

If you look at the last processor that gets called in the httpRequestBegin pipeline Sitecore.Pipelines.HttpRequest.ExecuteRequest you will see the HandleItemNotFound(HttpRequestArgs args) method is called if the Context.Item == null

If you look in that function, you will see that is where the item, user and site parameters are added to the query string, before being redirected to the /404 page.

In order to properly understand why these requests are being redirected to the 404 page, you need to work out why the Context Item is null in these situations. Is it possible some process could be calling these pages via an AJAX call or something? (Reason being - I found a post suggesting it would be null in that case).

Alternatively, you might consider adding some additional logging to the Sitecore.Pipelines.HttpRequest.ItemResolver processor, which is where the context item is found.

For example, you could check to see if the candidatePath (in function below) contains one of your failing strings (i.e. work-and-stay). Then log if and when it is failing to be resolved (recording variuous params etc).

public override void Process(HttpRequestArgs args)
{
....
foreach (string candidatePath in GetCandidatePaths(args))
{
    if (hashSet.Add(candidatePath))
    {
        if (TryResolveItem(candidatePath, args, out item, out permissionDenied))
        {
            text = candidatePath;
            break;
        }
        if (permissionDenied)
        {
            return;
        }
    }
}
...
}

Obviously, I would be very wary of doing too much in that processor as it would be called on every request. Ideally, you should be doing it in a test environment. But often these problems dont tend to show up there...