How do you programmatically exclude an item from being indexed?

Question

We would like to exclude items from being indexed. Ideally, we want this control to be within Sitecore so that as new pages get created, content authors can determine whether to index them as they desire.

Answer 1

There are a few steps required to make this configurable in Sitecore.

Create a Checkbox Field

Add a checkbox field, likely to your page item template. This field could be added to a base template shared by all "Page" items (or whatever the type of item it is you wish to exclude from indexing).

In this walkthrough, we're assuming the field is called Exclude from Index.

Create a Custom Crawler

Next, create a custom crawler that inherits Sitecore.ContentSearch.SitecoreItemCrawler and override the following methods:

public class CustomItemCrawler : SitecoreItemCrawler
{
    protected override bool IsExcludedFromIndex(SitecoreIndexableItem indexable, bool checkLocation = false)
    {
        var isExcluded = base.IsExcludedFromIndex(indexable, checkLocation);

        if (isExcluded)
            return true;

        Item obj = indexable;

        return obj["Exclude From Index"] == "1"; // feel free to use an ID here
    }

    protected override bool IndexUpdateNeedDelete(SitecoreIndexableItem indexable)
    {
        var needDelete = base.IndexUpdateNeedDelete(indexable);
        if (needDelete)
        {
            return true;
        }

        Item item = indexable;

        return item["Exclude From Index"] == "1";
    }
}

Note

The IndexUpdateNeedDelete method is necessary so that existing items can be physically removed. Without it, existing items are only removed when a full rebuild of the index in question is executed.

Patch in Custom Crawler

<?xml version="1.0" encoding="utf-8" ?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
  <sitecore>
    <contentSearch>
      <configuration>
        <indexes>
          <index id="sitecore_master_index">
            <locations>
              <crawler>
                <patch:attribute name="type">MyProject.CustomItemCrawler, MyProject</patch:attribute>
              </crawler>
            </locations>
          </index>
          <index id="sitecore_web_index">
            <locations>
              <crawler>
                <patch:attribute name="type">MyProject.CustomItemCrawler, MyProject</patch:attribute>
              </crawler>
            </locations>
          </index>
        </indexes>
      </configuration>
    </contentSearch>
  </sitecore>
</configuration>

Note the index id. Update to include all indexes that require this functionality.