7

I'm upgrading sitecore from 8.0 update 3 to update 7. I have already used this package while upgrading locally with no issues. when I install this package(Sitecore 8.0 rev. 160115.update) using the update installation wizard page on a QA server two strange things happen:

  1. When it's 10% of processing it starts taking a few seconds to process each item(it's almost 7240 items to process) so this is taking hours to process a package that I already installed locally and it took a few minutes.

  2. At some point it timeouts out. I guess the timeout one has to do with this slowness in processing so I guess the question here is how do we increase the timeout in this case?

Also the CPU on the server is not reaching 10% of usage while processing the package. Anyone know possible reasons for both cases?

UPDATE 1

Here is the log for when the installer times out (didn't find anything useful on it :|) :

ManagedPoolThread #3 01:03:13 INFO  Job ended: Core_Database_Agent (units processed: )

ManagedPoolThread #16 01:03:13 INFO  Job started: Master_Database_Agent

ManagedPoolThread #16 01:03:13 INFO  Scheduling.DatabaseAgent started. Database: master

ManagedPoolThread #16 01:03:13 INFO  Examining schedules (count: 4)

ManagedPoolThread #16 01:03:13 INFO  Starting: __Task Schedule

ManagedPoolThread #16 01:03:13 INFO  Ended: __Task Schedule

ManagedPoolThread #16 01:03:13 INFO  Not due: Calculate Statistical Relevancy

ManagedPoolThread #16 01:03:13 INFO  Not due: Rebuild Suggested Tests Index

ManagedPoolThread #16 01:03:13 INFO  Not due: Try Finish Test

ManagedPoolThread #16 01:03:13 INFO  Job ended: Master_Database_Agent (units processed: 4)

4328 01:06:21 ERROR Application error.

Exception: System.Web.HttpException
Message: Request timed out.

5876 01:11:20 INFO  HttpModule is being initialized

4216 01:11:20 INFO  HttpModule is being initialized

Heartbeat 01:13:05 INFO  Health.PrivateBytes: 0

Heartbeat 01:13:05 INFO  Health.CacheInstances: 117

Heartbeat 01:13:05 INFO  Health.CacheTotalCount: 38,989

Heartbeat 01:13:05 INFO  Health.CacheTotalSize: 90,721,502

Heartbeat 01:13:05 INFO  Health.Counter(w3wp, Process\Private Byte

UPDATE 2

RAM is around 35% of usage. Also the server which is running sql server has low cpu and memory usage while the installation happens. XDB has been disabled as well

UPDATE 3

After increasing the ping as suggested by Hishaam it timed out here is the logs of when it happened:

ManagedPoolThread #13 12:38:22 INFO Job started: Sitecore.ListManagement.Analytics.UnlockContactListsAgent ManagedPoolThread #13 12:38:22 INFO Job ended: Sitecore.ListManagement.Analytics.UnlockContactListsAgent (units processed: ) 6100 12:38:28 ERROR Application error. Exception: System.Web.HttpException Message: Request timed out.

ManagedPoolThread #19 12:38:32 INFO Job started: Sitecore.ListManagement.Analytics.UnlockContactListsAgent

Complete log here

UPDATE 4

When it throws the timeout exception I noticed that right about it there is this message:

Exception details: System.Exception: History path is not set at Sitecore.Update.InstallUpdatePackage.SaveInstallationMessages() at Sitecore.Update.InstallUpdatePackage.Install() at Sitecore.Update.InstallUpdatePackage.InstallPackage(MetadataView view) at Sitecore.Update.InstallUpdatePackage.OnLoad(EventArgs e)

One interesting thing is that there is a link to download the log errors and when I hover it I see it pointing to a directory that does not exist(I'm within the server). But if I got to the website/temp/__UpgradeHistory/Upgrade_Failutre_dateoftoday, I see a folder with two files: 1 is the log of all items processed and the other one is a message.xml which is empty. Decompiling the Sitecure.Update.dll this is what it looks like:

public string SaveInstallationMessages()
{
    string installationHistoryRoot = base.InstallationHistoryRoot;
    bool flag = this.EnsureHistoryPath(ref installationHistoryRoot);
    string str = Path.Combine(installationHistoryRoot, "messages.xml");
    FileUtil.EnsureFolder(str);
    using (FileStream fileStream = File.Create(str))
    {
        (new XmlEntrySerializer()).Serialize(this.logMessages, fileStream);
    }
    if (!flag)
    {
        throw new Exception("History path is not set");
    }
    return str;
}

Anyone see this? The file gets generated but without messages. maybe it has to do with the whole problem.

  • 1
    Could be a number of things. Anything happening in the log file while the install is happening? It be a job getting stuck somewhere. – Ian Graham Nov 22 '16 at 20:46
  • How does the RAM look while installing the package? Do you have any profiling or other heavy-weight (and likely debugging-centric) pipelines or event handlers running on the QA server that aren't running on local? This includes custom bucketing strategies, etc. that run it one place but not the other. – Zachary Kniebel Nov 22 '16 at 21:19
  • Check if the connection to the database since there is a large number of items being inserted in the different Sitecore Databases. Also, checks in the logs to see if there are any errors. Moreover, if there are dlls in the update package, each time a dll is copied, the app pool gets restarted. Make sure mongoDB service is running if you have xDB enable – Hishaam Namooya Nov 23 '16 at 9:01
  • As @IanGraham says, check your log files. – Phil Burton Nov 23 '16 at 10:11
  • posted the logs but didn't find it really useful :| – Diego Nov 23 '16 at 13:06
4

The problem is with IIS application pool setting “Ping Maximum Response Time” which is the time in seconds the worker process is given to respond to health monitoring ping. After 5 missed responses the work process is timed out and causes the exception

Exception: System.Web.HttpException
Message: Request timed out.

There are 2 solutions:

  • Set ping enabled to false
  • Set the Ping Maximum Response Time (seconds) to a large value

enter image description here

Please see the following post for more info.

EDIT 1

The main reason the timeout occurs is because the update package is installing both items and files, which reset the app pool. Due to this, Sitecore has to initialize its different pipelines, agents and also the Experience Editor. Moreover, since the introduction of the SPEAK UI, this also contributes in the time taken for Sitecore to be up and running.

  • I have set that and currently trying. Although it seems to me that the cause of it is it taking too long to process. This package has 7240 actions to be processed(items, files, etc) Up to 500 or so it goes well and then it starts getting slow and process one by one(15 seconds to process each). Not sure why since on other environments it went well with this same package – Diego Nov 23 '16 at 15:20
  • I started 40 minutes ago and only 740 items processed so far. The problem was that around 2400 it would crash after hours of processing :| hoopefully this will help make through to the end but I'm still thinking that it's too much time to process something that locally took around 15 or 20 minutes tops – Diego Nov 23 '16 at 15:23
  • I am aligned on your opinion but I think any installation performed on local instance is much quicker than on staging or production environments. Also, the reason it is timing out is because the dll will restart the app pool and Sitecore takes time to re initialize because of the different background jobs being triggered – Hishaam Namooya Nov 23 '16 at 15:36
  • Yes let's see if this IIS change will make it through. Anything else you can think of that could be causing this slowness? – Diego Nov 23 '16 at 15:39
  • 1
    This is the only thing that I can think of because it is a known issue that Sitecore XP initialization takes lot of time when app pool restart. This is because it has to initialize the different jobs and also due to the new experience editor and the use of the SPEAK UI – Hishaam Namooya Nov 23 '16 at 15:42
4
+50

I've found some different steps if you would like to try

1. Use a different browser

Yes, we know you are an engineer and you need logical justification for the same

Don’t worry we have a logical justification for you — posted on my another blog - Long-running processes and browser timeout issue (mainly Internet Explorer)

2. Modification on Web.config / Sitecore.config

In web.config

  • Change httpRuntime executionTimeout to 1800 seconds

httpRuntime maxRequestLength="512000" executionTimeout="18000" enableKernelOutputCache="false"

  • Increase ASXAUTH timeout to 180 minutes

forms name=".ASPXAUTH" cookieless="UseCookies" timeout="180"

  • Adjust sessionState timeout to 180 minutes

sessionState mode="InProc" cookieless="false" timeout="180"

In Sitecore.config

  • Set Authentication.ClientSessionTimeout to 180 minutes

setting name="Authentication.ClientSessionTimeout" value="180"

Please don't forget to backup your configuration before doing it and to revert it back after complete.

3. Change the values of DefaultSQLTimeout and DataProviderTimeout

  • In web.config

setting name=”DefaultSQLTimeout” value=”00:30:00″

setting name=”DataProviderTimeout” value=”00:30:00″

Please don't forget to backup your configuration before doing it and to revert it back after complete.

  • Tried this approach but it still times out it always times out when processing the same item: Changing item '/sitecore/client/Applications/ExperienceAnalytics/Dashboard/Behavior/Page URLs/PageSettings/Page URL/PageSettings' I tried to remove that item from the package to see if that would make any good(I know it's not recommended just looking to see what would happen) but it still failed – Diego Nov 24 '16 at 14:50
  • Assuming you didn't try the first option, I would say to give a shot using Firefox then. – Vinicius Deschamps Nov 24 '16 at 14:54
  • Will try but already saw this on IE and Chrome and both got the same outcome – Diego Nov 24 '16 at 15:25
4

Typically when I run a Sitecore upgrade I'll attach the databases to a clean install of the old version and run the update package on that clean install. That will get the new content into your databases, then you can copy the new files in whatever way is most convenient for you.

This doesn't solve the problem directly but it's a decent workaround. Like I mentioned, this is how I usually execute upgrades. It keeps your custom stuff from interfering and generally goes faster.

3

I'm guessing, based on your edits and the other responses, that you're issue lies in indexing - and that you're probably using Solr (where maybe you had Lucene in your local test). There's several configuration setups that trigger synchronous indexing during package installs, and these manifest as slow throughput without obvious load on either the DB or Sitecore machines (cause it's the indexing server crapping out). I guess Solr because you mentioned there's no obvious CPU load on the local Sitecore box - I wonder if you also checked network performance and/or disk I/O as these can be other culprits? Assuming those are all OK, my thoughts turn to the indexing subsystem.

There's a few things you can do to improve throughput when installing packages (both update and regular Sitecore packages) - from my bag of tricks...

  1. Disable indexing - I have a patch file that sets all the update strategies to "manual"
<strategies hint="list:AddStrategy">
    <manual ref="contentSearch/indexConfigurations/indexUpdateStrategies/manual" patch:instead="*[1]" />
</strategies>

This is really a failsafe to prevent indexing being triggered by the update of items, and to prevent timed refreshed from stealing process time.

  1. Disable the history engine - the history engine logs changes to items for use with incremental publishing. This means every item update is really 2 updates. Not good for performance, but worse than that, the history tables can get really big if you don't do periodic full or incremental publishes.
<databases>
    <database id="core">
        <Engines.HistoryEngine.Storage><patch:delete /></Engines.HistoryEngine.Storage>
        <Engines.HistoryEngine.SaveDotNetCallStack><patch:delete /></Engines.HistoryEngine.SaveDotNetCallStack>
    </database>
    <database id="master">
        <Engines.HistoryEngine.Storage><patch:delete /></Engines.HistoryEngine.Storage>
        <Engines.HistoryEngine.SaveDotNetCallStack><patch:delete /></Engines.HistoryEngine.SaveDotNetCallStack>
    </database>
</databases>
  1. Turn off App Pool recycles on config changes - this means Sitecore will not be able to trigger an app pool recycle by "touching" the web.config - you have to do it manually, but now you have the power!

  2. Remove the indexing events on packageinstall:items:ended - again, you need to rebuild indexes manually, but you can do it when you're finished rather than letting Sitecore decide to launch jobs for you in the background.

<events>
    <event name="packageinstall:items:ended">
        <handler type="Sitecore.ContentSearch.Events.PackagingEventHandler, Sitecore.ContentSearch" method="OnPackageInstallItemsEndHandler">
            <patch:delete />
        </handler>
    </event>
</events>
  1. Disable counters - be honest, you're not using them anyway
<settings>
    <setting name="Counters.Enabled" set:value="false" />
</settings>
  1. Related to indexing and history, disable the threshold checks - these escalate partial index rebuilds to full rebuilds when the number of items in the history table gets over a certain level; I'm not sure about the thinking here - may be a performance tradeoff.
<contentSearch>
    <indexConfigurations>
        <indexUpdateStrategies>
            <onPublishEndAsyncWeb type="Sitecore.ContentSearch.Maintenance.Strategies.OnPublishEndAsynchronousStrategy, Sitecore.ContentSearch">
                <param desc="database">web</param>
                <CheckForThreshold>False</CheckForThreshold>
            </onPublishEndAsyncWeb>
        </indexUpdateStrategies>
    </indexConfigurations>
</contentSearch>

That's just an example - you obviously don't have to do this if you've set all your index strategies to manual. We use this configuration all the time, because our authors rarely use full-site or incremental publishing.

  • thanks for the detailed explanation Richard. In my case I have lucene running on the build and not solr. I have disabled it and tried installing the package facing the same problems :| – Diego Nov 30 '16 at 12:13
  • at the point where it's slowing do you see any spikes in network traffic or i/o? – Richard Hauer Nov 30 '16 at 14:04
  • Also does the AppPool account (IUSR, ApplicationPoolIdentity, NetworkService, etc) have write access to the whole website folder? I would definitely check \Temp but actually it will need to write files all over the place during the update – Richard Hauer Nov 30 '16 at 14:09
  • I have set the Everyone group with full control on the website and data folders. I have monitored both servers(sql server and the qa where sitecore is running) and as far as memory and cpu goes they are fine. The ethernet section of the task manager does show traffic but no constant spikes... – Diego Nov 30 '16 at 14:16
  • can you run ProcessMonitor to check what the w3wp.exe is doing with the file system - look for fails. – Richard Hauer Nov 30 '16 at 14:44
1

You could do what Sitecore support would do in these scenarios.

Use a tool like dotMemory- https://www.jetbrains.com/dotmemory/features/ to get a memory dump.

More tools here:

https://kb.sitecore.net/articles/488758

Take a snapshot of the W3WP process while you are installing the package.

Analyse the results - this will give you some area to focus in in your debuging and you'll be able to find out what triggers your long running process.

1

The root cause of the timeout was infrastructure related. At this moment the IT team within the client is working to pinpoint exactly what the problem was. SQL server was having difficulties to communicate well with the sitecore QA server(not yet sure why maybe firewall rules) but installing sql server within the QA server and running the upgrade package it took 10 minutes and completed without problems.

1

Based on your numerous updates and provided information, I have concluded that that issue at hand is most definitely something to do with infrastructure as I alluded to in my comment above.

Things to Check

  1. I would advise checking the firewall to ensure communication with the SQL server
  2. If the SQL server resides remotely, try installing it locally to see if that resolves external factors (assuming your IT department doesn't complain).
  3. Use the Control Panel ODBC utility to test SQL Server connection on port 1433 (or custom sql port).
  4. Check user permissions on Sitecore folder structure.

Good Luck!

0

Sometimes packages fail when they try to install files. Try following

Rename .update file to .zip
Unzip
Remove files folder from unzip folder
Zip again
Rename the .zip to .update
Install again

For files, we can manually copy them once instsll success

  • it failes before it copies the files. The files are the last thing copied when installing this package – Diego Nov 25 '16 at 12:28
  • sorry.. answer was not formatted correctly earlier. did you try above steps. it should solve the issue – scFootsteps Nov 25 '16 at 13:44
  • I will try this but again there is a total of 7240 actions to be processed(items and files). It fails generally around 2400 while it is processing a sitecore item on core. The files are the last one to process... – Diego Nov 25 '16 at 13:53
  • I have removed the files folders from the package and retried the installation. It failed again with the same error – Diego Nov 25 '16 at 20:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.