3

I'm trying to figure out how the DeviceDetection api can be used to detect Bots visiting the website. Sitecore version: 9.0.2

When specific pages get visit, I want to log if it is a bot or not.

var deviceInfo = DeviceDetectionManager.GetDeviceInformation(HttpContext.Current.Request.UserAgent);
return deviceInfo != null ? deviceInfo.DeviceType.ToString() : "";

But after testing, it seems that certain UserAgents always return the DeviceType Computer. Example UserAgent:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.36

While I expect the DeviceType to be Bot in this case. Which it clearly is, it even says Googlebot.

So I started to deassemble the Sitecore.DeviceDetection.CES.dll

  1. First off all, the documentation specifies to use DeviceInformation.IsBot because the enum value DeviceType.Bot is obsolete.
    But... the DeviceDetection does not even contain a property IsBot.
  2. I looked at the 51Degrees implementation that should set the DeviceType. This is in Sitecore.CES.DeviceDetection.Providers.FiftyOneDegrees.MatchHelper.cs.
    But... there is no code that ever sets the DeviceType to Bot.

So my question is, how can I check for a Bot using the DeviceDetection api?

Can be helpful: 51degrees website to see what you get back from a given user agent string

  • Have you looked at using Sitecore Analytics Robot Detection for this instead? You can then call Sitecore.Analytics.Core.ContactClassification.IsRobot(currentContact) to check. – Dan Sinclair Jul 11 '19 at 12:01
  • Could be usefull, will readup on it! But I don't like It classifies all visitors on their first visit as robots. Think that could give us some false positives with what we want to do with the data. – Koen Heye Jul 11 '19 at 12:34
  • That could definitely be an issue for the first request, yes. Made worse if the visitor has an adblocker that prevents the callback to the server that verifies they are human. At the very least, though, you could use the Sitecore.Analytics.ExcludeRobots.config as a good reference point for your own robot detection based on user agent... – Dan Sinclair Jul 11 '19 at 13:50
2

Generally, checking for bots through the User Agent is not very accurate as it can be easily forged.

Use Robot Detection

Using the Robot Detection feature from Sitecore Analytics, you will have more accurate results as it doesn't only analyze a single request but considers the visitor's behaviour during the current session.

By default, on the first page of the first visit, a user will be classified as bot. HOWEVER... if the visitorIdentification and other indicators (i.E. media item download) are positive, the Session will be classified as Human.

Analyse sessions instead of requests

Because of the above you won't be able to log bot visits on each request, but what you can do is hook into the VisitEnd pipeline, run through the pages that have been visited, check if your page has been visited and check the session's robot classification.

Example:

using System.Linq;
using Sitecore.Analytics;
using Sitecore.Analytics.Core;
using Sitecore.Analytics.Pipelines.VisitEnd;
using Sitecore.Diagnostics;

namespace MyProject.VisitEnd
{
    public class LogRobotVisits : VisitEndProcessor
    {

     public override void Process(VisitEndArgs endArgs)
     {
         Assert.ArgumentNotNull((object)endArgs, nameof(endArgs));

         var classification = Tracker.Current.Contact.System.Classification;
         if (ContactClassification.IsHuman(classification))
         {
             return;
         }

         foreach (var page in Tracker.Current.Interaction.Pages.Where(p => p.Url.Path == "/my-page-to-log"))
         {
             if (ContactClassification.IsMaliciousRobot(classification))
             {
                Log.Warn($"Page '{page.Url}' was visited by a MALICIOUS bot. Useragent: {Tracker.Current.Interaction.UserAgent}", this);
             } else 
             {
                 Log.Warn($"Page '{page.Url}' was visited by a bot. Useragent: {Tracker.Current.Interaction.UserAgent}", this);
             }
         }
     }
  }
}

Patch config:

<sitecore>
<pipelines>
   <visitEnd>
        <processor type="MyProject.VisitEnd.LogRobotVisits,MyProject"/>
   </visitEnd>
</pipelines>
<sitecore>

The code above will run when a session expires. Sessions that have been classified as robots will expire after 1 minute. You can also flush sessions for testing using xConnect helper.

  • I understand what you propose but it is not a solution for what I'm working with. Good idea to use the visitEnd pipeline. – Koen Heye Jul 16 '19 at 7:56
0

I created a Sitecore Support ticket to find out the correct usage. Use reference 535808
if you contact them.

Apparently in Sitecore 9.0 (maybe also in 9.1), the DeviceDetection.IsBot is not working as intended. The DeviceDetection.IsBot is never set thus checking on it will never give the correct result.

To work around this, you can use the following code:

DeviceDetectionManager.GetExtendedProperty(HttpContext.Current.Request.UserAgent, "IsCrawler");

Also, IsBot is not obsolete. The documentation will be updated to reflect that.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.