We use the EXM with Sitecore Email Cloud on our project and going to push 4 million emails a day.
Currently we working on setting up a production servers structure.
Is it needed to use dispatch server in case of using Sitecore Email Cloud?
Are there any requirements or recommendation for servers structure for our needs? If someone has an experience of using an EXM to send so many emails, please share which infrastructure you built.
Background: Sitecore 8.2 Update 6 with EXM v.3.5.1 rev. 171103
Yes, Sitecore Email Cloud is simply the mail transit provider. You still need dispatch servers to generate the emails and send to the Sitecore EDS service. The process of generating the emails can be taxing based on how complicated your emails are and if personalization rules are in play.
Scaling to 4 Million emails a day is possible. The experience that I have had in the past got approximately 1 million emails in about 3 hours. Doing 4 in a day is totally do able.
Here is document about scaling EXM: https://doc.sitecore.net/email_experience_manager/setting_up_exm/configuration/configuring_exm_in_a_scaled_environment
The reality is, this document only covers how to scale to one server. But to get to 4 million a day, you're going to need multiple EXM Dispatch Servers. I would recommend starting off with 3-4 Dispatch Servers. Also, I recommend disabling dispatch on the Primary CM role so as not to bog down content authoring.
As for hardware, I would definitely make sure the dispatch servers have a lot of CPU and a good amount of RAM. Cache as much as you can in Sitecore caches to keep from having to make trips to the DB.
Solr optimization is key. EXM spends a lot of time in the indexes (analytics index, master/web indexes, etc). Making sure that your Solr implementation is sufficient and scaled appropriately will help in making sure that your dispatches are optimized. Any latency at all between Solr (and Mongo) will slow down dispatch times.
Ensure that you aren't doing anything CRAZY in the <SendEmail> pipeline. This is the pipeline that every email will call that generates the email. Any customizations to this pipeline will slow down processing. This also calls other pipelines such as <modifyHyperlink> and <modifyImageLink>so you'll want to also make sure that you aren't doing anything crazy here either.
Make sure that you also separate out your Processing and Reporting roles. This will help offset additional processing that 4 million emails will cause on your Primary CM if you do not. You may also find that having only one processing server might not be enough. Consider scaling to 2 if processing times are slow. An indicator of slow processing times is when a dispatch completes, but Sitecore is slow to report the Sent events on contacts and in the Email Report.
On each dispatch server, you can see stats on how the server is performing. You can reach the Dispatch Summary page at /sitecore/admin/dispatchsummary.aspx
On this report, the GetBody() statistic is what you want to try and optimize as much as you can. This method is what each email being generated calls, which reaches out to the CD cluster to generate the email with personalization, and then downloads the HTML into the email body. Any delays in communicating with the CD environment will also impact delivery.
The Primary CM role can generally handle most triggered emails, drip campaigns, and small batches. I have been successful with campaigns as large as 100K contacts on one Primary CM, but it wasn't fast, and authoring was a bit slower due to all of the dispatching and processing occuring.
If you are scaling up slowly, it is fine to start off with just a primary CM. If you find the authoring server begin to choke, separate out the processing and reporting roles. That should allow you to scale the Primary CM up more.
As you increase number of emails going out over time, and authoring chokes again, that is when I would deploy a dedicated dispatch server and then turn dispatch off on primary CM.
Then, to scale up from there, you are really just duplicating dispatch servers, and then down the line maybe add another processing server.
But thats slow growth.
If you have business requirements where you know you are sending out millions of emails on day one. Then plan for multiple dispatch servers out of the gate.
