How can I check for a Bot using the DeviceDetection api

Question

I'm trying to figure out how the DeviceDetection api can be used to detect Bots visiting the website. Sitecore version: 9.0.2

When specific pages get visit, I want to log if it is a bot or not.

var deviceInfo = DeviceDetectionManager.GetDeviceInformation(HttpContext.Current.Request.UserAgent);
return deviceInfo != null ? deviceInfo.DeviceType.ToString() : "";

But after testing, it seems that certain UserAgents always return the DeviceType Computer. Example UserAgent:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.36

While I expect the DeviceType to be Bot in this case. Which it clearly is, it even says Googlebot.

So I started to deassemble the Sitecore.DeviceDetection.CES.dll

1. First off all, the documentation specifies to use DeviceInformation.IsBot because the enum value DeviceType.Bot is obsolete.
   But... the DeviceDetection does not even contain a property IsBot.
2. I looked at the 51Degrees implementation that should set the DeviceType. This is in Sitecore.CES.DeviceDetection.Providers.FiftyOneDegrees.MatchHelper.cs.
   But... there is no code that ever sets the DeviceType to Bot.

So my question is, how can I check for a Bot using the DeviceDetection api?

Can be helpful: 51degrees website to see what you get back from a given user agent string

Answer 1

Generally, checking for bots through the User Agent is not very accurate as it can be easily forged.

Use Robot Detection

Using the Robot Detection feature from Sitecore Analytics, you will have more accurate results as it doesn't only analyze a single request but considers the visitor's behaviour during the current session.

By default, on the first page of the first visit, a user will be classified as bot. HOWEVER... if the visitorIdentification and other indicators (i.E. media item download) are positive, the Session will be classified as Human.

Analyse sessions instead of requests

Because of the above you won't be able to log bot visits on each request, but what you can do is hook into the VisitEnd pipeline, run through the pages that have been visited, check if your page has been visited and check the session's robot classification.

Example:

using System.Linq;
using Sitecore.Analytics;
using Sitecore.Analytics.Core;
using Sitecore.Analytics.Pipelines.VisitEnd;
using Sitecore.Diagnostics;

namespace MyProject.VisitEnd
{
    public class LogRobotVisits : VisitEndProcessor
    {

     public override void Process(VisitEndArgs endArgs)
     {
         Assert.ArgumentNotNull((object)endArgs, nameof(endArgs));

         var classification = Tracker.Current.Contact.System.Classification;
         if (ContactClassification.IsHuman(classification))
         {
             return;
         }

         foreach (var page in Tracker.Current.Interaction.Pages.Where(p => p.Url.Path == "/my-page-to-log"))
         {
             if (ContactClassification.IsMaliciousRobot(classification))
             {
                Log.Warn($"Page '{page.Url}' was visited by a MALICIOUS bot. Useragent: {Tracker.Current.Interaction.UserAgent}", this);
             } else 
             {
                 Log.Warn($"Page '{page.Url}' was visited by a bot. Useragent: {Tracker.Current.Interaction.UserAgent}", this);
             }
         }
     }
  }
}

Patch config:

<sitecore>
<pipelines>
   <visitEnd>
        <processor type="MyProject.VisitEnd.LogRobotVisits,MyProject"/>
   </visitEnd>
</pipelines>
<sitecore>

The code above will run when a session expires. Sessions that have been classified as robots will expire after 1 minute. You can also flush sessions for testing using xConnect helper.

Answer 2

I created a Sitecore Support ticket to find out the correct usage. Use reference 535808
if you contact them.

Apparently in Sitecore 9.0 (maybe also in 9.1), the DeviceDetection.IsBot is not working as intended. The DeviceDetection.IsBot is never set thus checking on it will never give the correct result.

To work around this, you can use the following code:

DeviceDetectionManager.GetExtendedProperty(HttpContext.Current.Request.UserAgent, "IsCrawler");

Also, IsBot is not obsolete. The documentation will be updated to reflect that.