Problems while installing an update package

Question

I'm upgrading sitecore from 8.0 update 3 to update 7. I have already used this package while upgrading locally with no issues. when I install this package(Sitecore 8.0 rev. 160115.update) using the update installation wizard page on a QA server two strange things happen:

1. When it's 10% of processing it starts taking a few seconds to process each item(it's almost 7240 items to process) so this is taking hours to process a package that I already installed locally and it took a few minutes.

2. At some point it timeouts out. I guess the timeout one has to do with this slowness in processing so I guess the question here is how do we increase the timeout in this case?

Also the CPU on the server is not reaching 10% of usage while processing the package. Anyone know possible reasons for both cases?

UPDATE 1

Here is the log for when the installer times out (didn't find anything useful on it :|) :

ManagedPoolThread #3 01:03:13 INFO  Job ended: Core_Database_Agent (units processed: )

ManagedPoolThread #16 01:03:13 INFO  Job started: Master_Database_Agent

ManagedPoolThread #16 01:03:13 INFO  Scheduling.DatabaseAgent started. Database: master

ManagedPoolThread #16 01:03:13 INFO  Examining schedules (count: 4)

ManagedPoolThread #16 01:03:13 INFO  Starting: __Task Schedule

ManagedPoolThread #16 01:03:13 INFO  Ended: __Task Schedule

ManagedPoolThread #16 01:03:13 INFO  Not due: Calculate Statistical Relevancy

ManagedPoolThread #16 01:03:13 INFO  Not due: Rebuild Suggested Tests Index

ManagedPoolThread #16 01:03:13 INFO  Not due: Try Finish Test

ManagedPoolThread #16 01:03:13 INFO  Job ended: Master_Database_Agent (units processed: 4)

4328 01:06:21 ERROR Application error.

Exception: System.Web.HttpException
Message: Request timed out.

5876 01:11:20 INFO  HttpModule is being initialized

4216 01:11:20 INFO  HttpModule is being initialized

Heartbeat 01:13:05 INFO  Health.PrivateBytes: 0

Heartbeat 01:13:05 INFO  Health.CacheInstances: 117

Heartbeat 01:13:05 INFO  Health.CacheTotalCount: 38,989

Heartbeat 01:13:05 INFO  Health.CacheTotalSize: 90,721,502

Heartbeat 01:13:05 INFO  Health.Counter(w3wp, Process\Private Byte

UPDATE 2

RAM is around 35% of usage. Also the server which is running sql server has low cpu and memory usage while the installation happens. XDB has been disabled as well

UPDATE 3

After increasing the ping as suggested by Hishaam it timed out here is the logs of when it happened:

ManagedPoolThread #13 12:38:22 INFO Job started: Sitecore.ListManagement.Analytics.UnlockContactListsAgent ManagedPoolThread #13 12:38:22 INFO Job ended: Sitecore.ListManagement.Analytics.UnlockContactListsAgent (units processed: ) 6100 12:38:28 ERROR Application error. Exception: System.Web.HttpException Message: Request timed out.

ManagedPoolThread #19 12:38:32 INFO Job started: Sitecore.ListManagement.Analytics.UnlockContactListsAgent

Complete log here

UPDATE 4

When it throws the timeout exception I noticed that right about it there is this message:

Exception details: System.Exception: History path is not set at Sitecore.Update.InstallUpdatePackage.SaveInstallationMessages() at Sitecore.Update.InstallUpdatePackage.Install() at Sitecore.Update.InstallUpdatePackage.InstallPackage(MetadataView view) at Sitecore.Update.InstallUpdatePackage.OnLoad(EventArgs e)

One interesting thing is that there is a link to download the log errors and when I hover it I see it pointing to a directory that does not exist(I'm within the server). But if I got to the website/temp/__UpgradeHistory/Upgrade_Failutre_dateoftoday, I see a folder with two files: 1 is the log of all items processed and the other one is a message.xml which is empty. Decompiling the Sitecure.Update.dll this is what it looks like:

public string SaveInstallationMessages()
{
    string installationHistoryRoot = base.InstallationHistoryRoot;
    bool flag = this.EnsureHistoryPath(ref installationHistoryRoot);
    string str = Path.Combine(installationHistoryRoot, "messages.xml");
    FileUtil.EnsureFolder(str);
    using (FileStream fileStream = File.Create(str))
    {
        (new XmlEntrySerializer()).Serialize(this.logMessages, fileStream);
    }
    if (!flag)
    {
        throw new Exception("History path is not set");
    }
    return str;
}

Anyone see this? The file gets generated but without messages. maybe it has to do with the whole problem.

Answer 1

The problem is with IIS application pool setting “Ping Maximum Response Time” which is the time in seconds the worker process is given to respond to health monitoring ping. After 5 missed responses the work process is timed out and causes the exception

Exception: System.Web.HttpException
Message: Request timed out.

There are 2 solutions:

• Set ping enabled to false
• Set the Ping Maximum Response Time (seconds) to a large value

Please see the following post for more info.

EDIT 1

The main reason the timeout occurs is because the update package is installing both items and files, which reset the app pool. Due to this, Sitecore has to initialize its different pipelines, agents and also the Experience Editor. Moreover, since the introduction of the SPEAK UI, this also contributes in the time taken for Sitecore to be up and running.

Answer 2

I've found some different steps if you would like to try

1. Use a different browser

• Always install upgrade package in Firefox rather than IE

Yes, we know you are an engineer and you need logical justification for the same

Don’t worry we have a logical justification for you — posted on my another blog - Long-running processes and browser timeout issue (mainly Internet Explorer)

2. Modification on Web.config / Sitecore.config

In web.config

• Change httpRuntime executionTimeout to 1800 seconds

httpRuntime maxRequestLength="512000" executionTimeout="18000" enableKernelOutputCache="false"

• Increase ASXAUTH timeout to 180 minutes

forms name=".ASPXAUTH" cookieless="UseCookies" timeout="180"

• Adjust sessionState timeout to 180 minutes

sessionState mode="InProc" cookieless="false" timeout="180"

In Sitecore.config

• Set Authentication.ClientSessionTimeout to 180 minutes

setting name="Authentication.ClientSessionTimeout" value="180"

Please don't forget to backup your configuration before doing it and to revert it back after complete.

3. Change the values of DefaultSQLTimeout and DataProviderTimeout

• In web.config

setting name=”DefaultSQLTimeout” value=”00:30:00″

setting name=”DataProviderTimeout” value=”00:30:00″

Please don't forget to backup your configuration before doing it and to revert it back after complete.

Answer 3

Typically when I run a Sitecore upgrade I'll attach the databases to a clean install of the old version and run the update package on that clean install. That will get the new content into your databases, then you can copy the new files in whatever way is most convenient for you.

This doesn't solve the problem directly but it's a decent workaround. Like I mentioned, this is how I usually execute upgrades. It keeps your custom stuff from interfering and generally goes faster.

Answer 4

I'm guessing, based on your edits and the other responses, that you're issue lies in indexing - and that you're probably using Solr (where maybe you had Lucene in your local test). There's several configuration setups that trigger synchronous indexing during package installs, and these manifest as slow throughput without obvious load on either the DB or Sitecore machines (cause it's the indexing server crapping out). I guess Solr because you mentioned there's no obvious CPU load on the local Sitecore box - I wonder if you also checked network performance and/or disk I/O as these can be other culprits? Assuming those are all OK, my thoughts turn to the indexing subsystem.

There's a few things you can do to improve throughput when installing packages (both update and regular Sitecore packages) - from my bag of tricks...

1. Disable indexing - I have a patch file that sets all the update strategies to "manual"

<strategies hint="list:AddStrategy">
    <manual ref="contentSearch/indexConfigurations/indexUpdateStrategies/manual" patch:instead="*[1]" />
</strategies>

This is really a failsafe to prevent indexing being triggered by the update of items, and to prevent timed refreshed from stealing process time.

2. Disable the history engine - the history engine logs changes to items for use with incremental publishing. This means every item update is really 2 updates. Not good for performance, but worse than that, the history tables can get really big if you don't do periodic full or incremental publishes.

<databases>
    <database id="core">
        <Engines.HistoryEngine.Storage><patch:delete /></Engines.HistoryEngine.Storage>
        <Engines.HistoryEngine.SaveDotNetCallStack><patch:delete /></Engines.HistoryEngine.SaveDotNetCallStack>
    </database>
    <database id="master">
        <Engines.HistoryEngine.Storage><patch:delete /></Engines.HistoryEngine.Storage>
        <Engines.HistoryEngine.SaveDotNetCallStack><patch:delete /></Engines.HistoryEngine.SaveDotNetCallStack>
    </database>
</databases>

3. Turn off App Pool recycles on config changes - this means Sitecore will not be able to trigger an app pool recycle by "touching" the web.config - you have to do it manually, but now you have the power!

4. Remove the indexing events on packageinstall:items:ended - again, you need to rebuild indexes manually, but you can do it when you're finished rather than letting Sitecore decide to launch jobs for you in the background.

<events>
    <event name="packageinstall:items:ended">
        <handler type="Sitecore.ContentSearch.Events.PackagingEventHandler, Sitecore.ContentSearch" method="OnPackageInstallItemsEndHandler">
            <patch:delete />
        </handler>
    </event>
</events>

5. Disable counters - be honest, you're not using them anyway

<settings>
    <setting name="Counters.Enabled" set:value="false" />
</settings>

6. Related to indexing and history, disable the threshold checks - these escalate partial index rebuilds to full rebuilds when the number of items in the history table gets over a certain level; I'm not sure about the thinking here - may be a performance tradeoff.

<contentSearch>
    <indexConfigurations>
        <indexUpdateStrategies>
            <onPublishEndAsyncWeb type="Sitecore.ContentSearch.Maintenance.Strategies.OnPublishEndAsynchronousStrategy, Sitecore.ContentSearch">
                <param desc="database">web</param>
                <CheckForThreshold>False</CheckForThreshold>
            </onPublishEndAsyncWeb>
        </indexUpdateStrategies>
    </indexConfigurations>
</contentSearch>

That's just an example - you obviously don't have to do this if you've set all your index strategies to manual. We use this configuration all the time, because our authors rarely use full-site or incremental publishing.

Answer 5

You could do what Sitecore support would do in these scenarios.

Use a tool like dotMemory- https://www.jetbrains.com/dotmemory/features/ to get a memory dump.

More tools here:

https://kb.sitecore.net/articles/488758

Take a snapshot of the W3WP process while you are installing the package.

Analyse the results - this will give you some area to focus in in your debuging and you'll be able to find out what triggers your long running process.

Answer 6

The root cause of the timeout was infrastructure related. At this moment the IT team within the client is working to pinpoint exactly what the problem was. SQL server was having difficulties to communicate well with the sitecore QA server(not yet sure why maybe firewall rules) but installing sql server within the QA server and running the upgrade package it took 10 minutes and completed without problems.

Answer 7

Based on your numerous updates and provided information, I have concluded that that issue at hand is most definitely something to do with infrastructure as I alluded to in my comment above.

Things to Check

1. I would advise checking the firewall to ensure communication with the SQL server
2. If the SQL server resides remotely, try installing it locally to see if that resolves external factors (assuming your IT department doesn't complain).
3. Use the Control Panel ODBC utility to test SQL Server connection on port 1433 (or custom sql port).
4. Check user permissions on Sitecore folder structure.

Good Luck!

Answer 8

Sometimes packages fail when they try to install files. Try following

Rename .update file to .zip
Unzip
Remove files folder from unzip folder
Zip again
Rename the .zip to .update
Install again

For files, we can manually copy them once instsll success