0

Sitecore docs says:

You can create a list to assign custom user agents and IP addresses to exclude in the Sitecore.Analytics.ExcludeRobots.config file. You can manually edit this list by adding the user agents that you want to block under the node and IP addresses under the node.

More at: https://doc.sitecore.com/developers/93/sitecore-experience-platform/en/configure-robot-detection-functionality.html#UUID-5fd30313-8795-95cc-fe93-63bb53496d88_N1559896157868_body

But I can't find any information if some kind of wildcard can be used in excludedUserAgents node.

I mean, can I use something like this:

<excludedUserAgents>
*Googlebot*
</excludedUserAgents>

instead of this:

<excludedUserAgents>
Mozilla/5.0+(Linux;+Android+6.0.1;+Nexus+5X+Build/MMB29P)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/80.0.3987.92+Mobile+Safari/537.36+(compatible;+Googlebot/2.1;++http://www.google.com/bot.html)
</excludedUserAgents>

Asking because I have quite a long list of user agents to exclude and most of them contains "crawler" as a substring.

2

The code is below. I uses a .Contains() against a List. It has to be an exact match.

if (args.HttpContext.Request.UserAgent == null || !AnalyticsSettings.Robots.ExcludeList.ContainsUserAgent(args.HttpContext.Request.UserAgent))

Then it is just a simple list Contains

public bool ContainsUserAgent(string userAgent)
{
  Assert.ArgumentNotNull((object) userAgent, nameof (userAgent));
  return this.userAgents.Contains(userAgent);
}

You can override the CheckUserAgent processor in the excludeRobots pipeline and role you own logic for partial.

        <excludeRobots>
            <processor type="Sitecore.Analytics.Pipelines.ExcludeRobots.CheckUserAgent, Sitecore.Analytics" />
        </excludeRobots>
| improve this answer | |
  • 1
    Thank you for pointing me to that pipeline. Answering my question: no, there is no way to define wildcards for "excludedUserAgents". By decompiling Sitecore.Analytics.dll I found that ContainsUserAgent method of Sitecore.Analytics.Configuration.ExcludeList is used. And this method verifies if the whole user agent string is defined in "excludedUserAgents" – Vitalii Ilchenko Sep 7 at 10:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.