4

My network guys have started asking me about a sudden spike in network traffic that they noticed. For the past 4 days I have had about 5 terabytes go between my 3 CD servers and my 1 CM server, which is significantly more than I've ever seen before. I'm on Sitecore 8.1 Original release (and have been on this release since April of this year). At the same time my SOLR disk space requirements have significantly increased, causing me to request extra disk space 2 times in that same time frame. My SOLR Server is on the same machine as my CM server, and my analytics SOLR index has grown very large as well.

I'm not sure where to start looking to troubleshoot this kind of issue. There haven't been exceptions to be logged, there is just a lot more traffic than I was expecting. Our website traffic has not went up significantly in the last week, so I don't think that this is "normal" traffic.

How do I start troubleshooting this issue?

Improve this question
6
  • What do the logs folder look like in this time frame where the traffic has gone up? any large log files? do you see anything useful on logs like exceptions or indexing going on?
    – Diego
    Commented Dec 20, 2016 at 20:11
  • Do you have an on-premise or cloud-hosted solution? Do you have any monitoring tools, like New Relic, installed? Have you checked your IIS logs from the past few days to see if there are any URLs that are being requested with unusually high frequency?
    – Zachary Kniebel
    Commented Dec 20, 2016 at 20:50
  • I have an on-premise solution. I don't have any monitoring tools installed. I am seeing that there is a single SOLR request that continues to be called (select?q=(_template%3a(e07c277f59684cc097bc8b3195159f56)+AND+associated_articles_sm%3a(11fa15b49559444da015c363dd4303fe))&rows=2147483647&fq=_indexname%3a(sitecore_web_index)&version=2.2), but I'm not sure why that is being called, or who is calling it. I'll go through the logs folder and get back to you when I get a better grasp on what that status is.
    – Keith VanderVeen
    Commented Dec 20, 2016 at 21:50
  • I'd start looking at what new/additional data you've gotten in your SOLR because the additional space required could easily translate to more network traffic.
    – Allan S. Hansen
    Commented Dec 21, 2016 at 8:50
  • You mentioned you're now seeing "5 terabytes go between [your] 3 CD servers and [your] 1 CM server". Is that the direction your seeing the data flow? In other words, does it look like analytical info, for example, from your CDs to your CM? Or, is the data flowing both ways or maybe more data from the CM to the CDs?
    – DougCouto
    Commented Dec 21, 2016 at 11:49

1 Answer 1

Reset to default
3

Thank you everyone for your help! Here's what my process ended up looking like. I hope that it will help others:

  1. I checked the Sitecore Search log, and found that I was getting the same query running multiple times in each second.
  2. I then checked my Sitecore (regular) log file and found out that I was getting between 2 and 4 visitors per second to my website (I had some extra logging that helped me identify that).

  3. Next, I checked my SOLR logs and discovered that my Analytics index was getting hit pretty hard. I was getting the following warning messages:

    [sitecore_analytics_index] PERFORMANCE WARNING: Overlapping onDeckSearchers=2

    [sitecore_analytics_index] Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later.

This told me that my SOLR server wasn't able to keep up with the amount of traffic that we were throwing at it.

  1. Finally, I went and looked at my IIS logs and discovered that our on-premise Google Search Appliance was getting stuck on indexing a single page of our website.

So, after I excluded that one page from the GSA's indexing, the traffic went back down to what it should have been.

There are a couple of important take-aways for me, and I hope that this will be helpful to others:

  1. When you see a lot of traffic on your site, check the Sitecore Search log. It may be able to provide you with some insight into what is going on.
  2. When your SOLR server suddenly starts acting like it's under a lot more load, it probably is. Don't assume that there is something wrong with your SOLR server configuration.
  3. If you are seeing a lot more traffic to your site, but you are not seeing any increase in your analytics numbers, that could be because there is a robot hitting your site. Sitecore filters the robots out of the numbers. However, the Analytics index still records/processes those because of how Sitecore figures out if a website user is a robot or not.
Improve this answer
2
  • 1
    Thank you for a thoughtful and detailed answer! I am sure others will find it helpful.
    – Dmytro Shevchenko
    Commented Dec 21, 2016 at 19:38
  • 1
    I can not tell you how many times I've seen GSA's just KILL resources when one starts crawling a site... especially dev sites. Yeah, great find. you might consider adding the GSA User Agent to the Sitecore.Analytics.ExcludeRobots.config which excludes those bots from hitting analytics.
    – Pete Navarra
    Commented Dec 21, 2016 at 22:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.