I have consulted with Sitecore support on this. Regarding resources, all of them are important: Memory, CPU, Disk Latency and Database DTU's.
I have also checked my cache settings and noticed there where 2 caches beyond there limits:
- SqlDataProvider - Prefetch data (core) with 6383 items and a size of 43.3 MB and maxsize of 50MB
- SqlDataProvider - Prefetch data (master) with 8278 items and a size of 46.5 MB and maxsize of 50MB
This surprised me because these prefetch caches are removed by a default installed patch https://kb.sitecore.net/articles/290593
This purpose of this patch is to fix a startup problem needing to much resources from core DB.
In my experience, the patch may resolve the initial issue, but makes startup only slower, the caches are still there but smaller, needing to refetch stuff all the time.
The Solution to improve startup times in my case was to remove the patch and increase the core DB prefetch cach to 250MB.
Note that in my case this didn't reintroduce the initial issue for which the patch was distributed. The reason for this, is that I bundled my databases into an elastic database pool (as described on my blog http://onelittlespark.bartverdonck.be/how-to-boost-your-sitecore-azure-paas-website-for-zero-dollars/). This setup gives more resources to the core database during startup, and makes the patch redundant.