Similar to the situation described in this post regarding Solr, we recently had an issue in which our connection attempts to the xDB Cloud 2.0 service were adding hundreds of megabytes of 504 errors to the logs, crippling the site with timeout errors all over the place and effectively taking the site down until an app pool recycle was manually performed.
I am currently discussing the issue with Sitecore support, in order to ensure that nothing on our end caused this, but this seems highly unlikely since the issue occurred on both CD servers at the same time. In either case, I would like to know if there is a way to ensure that the site remains stable in the event of xDB timeout errors.
At this point, my only real guess at how this could be done is to control the amount of time between connection attempts. Does anyone know if this could be done or any other (perhaps better) ways of doing this?
Version info: Sitecore 8.1u2, xDB Cloud 2.0