0

Sitecore docs says:

You can create a list to assign custom user agents and IP addresses to exclude in the Sitecore.Analytics.ExcludeRobots.config file. You can manually edit this list by adding the user agents that you want to block under the node and IP addresses under the node.

More at: https://doc.sitecore.com/developers/93/sitecore-experience-platform/en/configure-robot-detection-functionality.html#UUID-5fd30313-8795-95cc-fe93-63bb53496d88_N1559896157868_body

But I can't find any information if some kind of wildcard can be used in excludedUserAgents node.

I mean, can I use something like this:

<excludedUserAgents>
*Googlebot*
</excludedUserAgents>

instead of this:

<excludedUserAgents>
Mozilla/5.0+(Linux;+Android+6.0.1;+Nexus+5X+Build/MMB29P)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/80.0.3987.92+Mobile+Safari/537.36+(compatible;+Googlebot/2.1;++http://www.google.com/bot.html)
</excludedUserAgents>

Asking because I have quite a long list of user agents to exclude and most of them contains "crawler" as a substring.

2
  • What version of Sitecore?
    – Chris Auer
    Sep 3, 2020 at 18:45
  • @ChrisAuer Sitecore 9.3 Sep 6, 2020 at 19:49

2 Answers 2

2

The code is below. I uses a .Contains() against a List. It has to be an exact match.

if (args.HttpContext.Request.UserAgent == null || !AnalyticsSettings.Robots.ExcludeList.ContainsUserAgent(args.HttpContext.Request.UserAgent))

Then it is just a simple list Contains

public bool ContainsUserAgent(string userAgent)
{
  Assert.ArgumentNotNull((object) userAgent, nameof (userAgent));
  return this.userAgents.Contains(userAgent);
}

You can override the CheckUserAgent processor in the excludeRobots pipeline and role you own logic for partial.

        <excludeRobots>
            <processor type="Sitecore.Analytics.Pipelines.ExcludeRobots.CheckUserAgent, Sitecore.Analytics" />
        </excludeRobots>
1
  • 1
    Thank you for pointing me to that pipeline. Answering my question: no, there is no way to define wildcards for "excludedUserAgents". By decompiling Sitecore.Analytics.dll I found that ContainsUserAgent method of Sitecore.Analytics.Configuration.ExcludeList is used. And this method verifies if the whole user agent string is defined in "excludedUserAgents" Sep 7, 2020 at 10:18
0

I know I'm late to the party but my module does this. It has also been added as a feature request for future releases of Sitecore. https://github.com/jasonmaclean/exclude-robot-by-agent

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.