Why is event data stored in every Sitecore database?

Question

A table named EventQueue is present in every Sitecore database. It seems to contain local and remote events of Sitecore instances.

Why are some of these events stored in the Core, some in the Master, and some in the Web database? What is the reason behind this separation?

Answer 1

I believe the primary reason is each event has an associated target database.

Found a brief explanation here by Kiran Patils on how the event queuing works:

1. The CM instance triggers an event such as "publish:end:remote".
2. The triggered event is added to the EventQueue table in the database that is targeted by the event. For example, the publish:end:remote event will be queued up in the publishing target database, typically the web database (unless you’ve specified custom publishing targets/databases).
3. Every 2 seconds, your CD instances poll the EventQueue table in the database from which they are retrieving content.
4. If they find an event queued up in the EventQueue table, they attempt to handle the event according to the handlers defined in the /sitecore/events section in their respective web.config files. For instance, by default the publish:end:remote event is configured to execute the HtmlCacheClearer handler for the website site. Handlers can be added or removed as needed.

Note: the interval at which your CD instances poll the EventQueue table is configurable in the web.config file under the /sitecore/eventQueue/processingInterval node.

Answer 2

Answer

All databases have an [EventQueue] in order to help spread out responsibilities and reduce the load on a particular database.

Explanation

The [EventQueue] table exists as a means of communication among all web servers that connect to it. One web server can queue an event in any database it has access to. This event is then read by any other web server in a cluster that also has access to the same database. Most typically, this is the System Event Queue. Which is defined in the config as:

/sitecore/eventing/providers/sitecore[systemDatabaseName]

The System Event Queue is special for a few reasons:

• Sitecore makes the assumption that all web servers in a cluster can connect to this database
• Sitecore assumes all web servers in a cluster have the same database set as the systemDatabase
• Queuing a RemoteEvent via EventManager uses the systemDatabase
  • Sitecore.Eventing.EventManager.QueueEvent(TEvent event);

While the System Event Queue is just a bit special, it still gets processed the same as the other databases. Also, please note, the System Event Queue, i.e. the Core database will often hold significantly more records than the other databases.

All web servers poll the [EventQueue] table of all "Sitecore Databases" (Factory.GetDatabases()) they connect to for new RemoteEvents, not just the System Event Queue. Therefore, the following statement is true in many scenarios:

A CD server polls both Core and Web, whereas a CM server polls Core, Master and Web. Specifically, it executes the following pseudo-query each time:

Find all new events where (the InstanceName is equal to my InstanceName and the event is set to RaiseLocally OR the InstanceName is not equal to my InstanceName and the event is set to RaiseGlobally)

I have a thorough post including an animation of an example [EventQueue]. Per this post- know that the animation will occur simultaneously for all databases that a web server has a connection to.

Common Misconceptions

Remote Events vs Local Events

The [EventQueue] only operates with RemoteEvents. It does not require knowledge of local events such as publish:end:remote. While the publish:end:remote handlers do execute, the process for getting there makes a few hops.

Take the following example: A content author executes a publish from a CM server. The publish process completes and the web database has a new RemoteEvent queued up:

Sitecore.Eventing.Remote.PublishEndRemoteEvent

And the raw data passed is (as an example):

{
"CompareRevisions":false,
"Deep":true,
"FromDate":"\/Date(1515701069888)\/",
"LanguageName":"en-GB",
"Mode":3,
"PublishDate":"\/Date(1520448918977)\/",
"PublishingTargets":["{8E080626-DDC3-4EF4-A1D1-F0BE4A200254}"],
"RepublishAll":false,
"RootItemId":"651c7676-dcbd-49f6-a1b7-b9d31e4c6747",
"SourceDatabaseName":"master",
"TargetDatabaseName":"web"
}

Notice that this is a fully qualified type with data. The [EventQueue] intrinsically has no idea what publish:end:remote is- it only operates on a set of defined types that are mapped to methods.

During app start (<initialize>)- the Sitecore.Eventing.Remote.PublishEndRemoteEvent type is mapped to a method- this method:

Sitecore.Eventing.Remote.RemoteEventMap -> SetupRemoteEventSubscribers()

Sets up:

    private static void OnPublishEndRemoteEvent(PublishEndRemoteEvent @event)
    {
      Assert.ArgumentNotNull((object) @event, nameof (@event));
      Event.RaiseEvent("publish:end:remote", (IPassNativeEventArgs) new PublishEndRemoteEventArgs(@event));
    }

Notice that this is what raises the local event publish:end:remote and processes the data that is included with the queued event. publish:end:remote by itself is simply a naming convention and does not imply that an event has to be executed on a remote server.

Side Note: You could also Subscribe to the Sitecore.Eventing.Remote.PublishEndRemoteEvent, which would provide you with the raw publish data, instead of the filtered version that is passed to the publish:end:remote local event. Which fundamentally does the same thing as adding a handler to the publish:end:remote section.

For an excellent explanation of a local event vs a RemoteEvent, view this thorough write-up on the Sitecore Community GitHub.

The [EventQueue] as a unit, thinks like this:

Once a record is added to the [EventQueue] with a type of Sitecore.Eventing.Remote.PublishEndRemoteEvent, execute the method OnPublishEndRemoteEvent and pass to this method the data included in the [EventQueue] record.