11

I'm running Sitecore 9.0.1 and trying to rebuild a reporting database. The problem is that the process is ridiculously slow. How slow? Here's a screenshot:

enter image description here

After 26 hours 127 items were processed of 178.839 in total. I went over the logs of the processing and reporting server, they show no error but this:

ManagedPoolThread #8 17:12:01 WARN  Authentication on CES Discovery service failed.
Exception: System.Net.WebException
Message: The remote server returned an error: (403) Forbidden.
Source: System
   at System.Net.HttpWebRequest.GetResponse()
   at Sitecore.CES.Client.WebClient.<>c__DisplayClass6_0.<ExecuteRequest>b__0()
   at Sitecore.CES.Client.WebClient.Execute[T](Func`1 action, String requestUri)
   at Sitecore.CES.Client.ResourceConnector`1.Request(String endpoint, Object[] parameters)
   at Sitecore.CES.Discovery.EndpointSource.GetEndpoint(String serviceName)

ManagedPoolThread #8 17:12:01 ERROR Could not update device detection database
Exception: System.ArgumentNullException
Message: Null ids are not allowed.
Parameter name: endpointUri
Source: Sitecore.Kernel
   at Sitecore.Diagnostics.Assert.ArgumentNotNullOrEmpty(String argument, String argumentName)
   at Sitecore.CES.Client.WebClient.DownloadBinaryContent(String endpointUri, String resourcePath, DownladBinaryContentParameters parameters)
   at Sitecore.CES.DeviceDetection.Providers.FiftyOneDegrees.FiftyOneDeviceDetectionClient.GetNewerVersion()
   at Sitecore.CES.DeviceDetection.Providers.FiftyOneDegrees.FiftyOneDeviceDetectionClient.DoUpdate(String serviceName, String& newDatabasePath, String& newDatabaseVersion)
   at Sitecore.CES.DeviceDetection.Providers.DeviceDetectionClient.Update()

Could this cause this kind of behavior? This problem happens across all Sitecore servers we run for this client, CM, CD, PROC, REP, all. Or is there anything else there?

Does anyone has any tip on what could cause such slowliness?

thank you

Improve this question
7
  • Is this a PaaS solution? If so an in azure, have you looked at DTU for your databases? Are you using Elastic pools?
    – Justin Bradley
    Commented Dec 18, 2018 at 23:54
  • Hey Justin, it's on premise. I've requested the DBA responsible for a DB analysis but all databases sit on the same server and master for example is not presenting the same kind of slowliness.
    – Anderson Fortaleza
    Commented Dec 19, 2018 at 0:29
  • 1
    That CES error looks like an issue that is resolved in a hotfix. Do you have this hotfix applied already? kb.sitecore.net/articles/828414 That said, I don't think that would cause that extreme level of slowness, but worth ruling out. Looking at the processing role metrics and database metrics may give you some insight into where the bottleneck lies.
    – George Tucker
    Commented Dec 19, 2018 at 1:11
  • 1
    I dont think the GeoIP (CES) and Device errors are you issue. You should see those all the time if you have not subscribed to those services. I once ran a rebuild for 10 days.
    – Chris Auer
    Commented Dec 19, 2018 at 1:46
  • 1
    I opened a ticket with Sitecore a few months ago over this same issue. There response was to add more agents to process the rebuild. They referred to this article. doc.sitecore.net/sitecore_experience_platform/82/…
    – Chris Auer
    Commented Dec 19, 2018 at 1:51

1 Answer 1

Reset to default
5

I have applied the hotfix as instructed by George Tucker and increased the number of history workers as instructed by Chris Auer, in my case I have raised from 1 to 500. I'm getting much better speed now, around 1.500 items per minute! I have no idea if the hotfix had any role on this because I did both things before running the rebuild again.

Update

Setting history workers to 500 was too much for the server, Sitecore recommended me to set to 16 as I had a single dedicated server for processing. But one of the problems I was having after raising the number of history workers was that after some time the processing (rate of items processed per minute) was slowing down almost to a halt, independent of the number of history workers.

After contacting Sitecore they said it was something to do with the implementation of the processing server and sent me Sitecore.Support.230550.dll and Sitecore.Support.230550.config. After patching the processing server with these files my rebuild was completed sucessfully after 10 hours of processing (arounf 320K items).

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.