Creating media items creates duplicate folder names

Question

I'm importing several thousand of documents in the media library programmatically.
The document are organized in folders.

public MediaItem AddFile(string fileName, string sitecorePath, string mediaItemName)
{
    Sitecore.Resources.Media.MediaCreatorOptions options = new Sitecore.Resources.Media.MediaCreatorOptions();
    // Store the file in the database, not as a file
    options.FileBased = false;
    // Remove file extension from item name
    options.IncludeExtensionInItemName = false;
    // Overwrite any existing file with the same name
    options.OverwriteExisting = true;
    // Do not make a versioned template
    options.Versioned = true;
    // set the path
    options.Destination = sitecorePath + "/" + mediaItemName;
    // Set the database
    options.Database = Sitecore.Configuration.Factory.GetDatabase("master");

    // Now create the file
    Sitecore.Resources.Media.MediaCreator creator = new Sitecore.Resources.Media.MediaCreator();
    MediaItem mediaItem;
    using (new SecurityDisabler()) // Use the SecurityDisabler object to override the security settings
    {
        mediaItem = creator.CreateFromFile(fileName, options);
    }

    return mediaItem;
}

And I'm calling this method in a foreach loop.

var createdMediaItem = AddFile("C:\\path to file on disk", 
    "sitecore/media library/subfolder", "name of document");

In my case, I often - but not always - get duplicate foldernames. So 'subfolder' will be multiple times in my Media Library.

Is there a solution for this?

Answer 1

Here is the code which is used by Sitecore to get parent folder for the new media item:

private Item GetParentFolder(string itemPath, MediaCreatorOptions options)
{
  Assert.ArgumentNotNull((object) itemPath, "itemPath");
  Assert.ArgumentNotNull((object) options, "options");
  string[] strArray = StringUtil.Divide(itemPath, '/', true);
  return this.CreateFolder(strArray.Length > 1 ? strArray[0] : "/sitecore/media library", options);
}

protected virtual Item CreateFolder(string itemPath, MediaCreatorOptions options)
{
  Assert.ArgumentNotNullOrEmpty(itemPath, "itemPath");
  Assert.ArgumentNotNull((object) options, "options");
  Item itemPath1;
  using (new SecurityDisabler())
  {
    TemplateItem folderTemplate = this.GetFolderTemplate(options);
    Database database = this.GetDatabase(options);
    Item obj = database.GetItem(itemPath, options.Language);
    if (obj != null)
      return obj;
    itemPath1 = database.CreateItemPath(itemPath, folderTemplate, folderTemplate);
    Assert.IsNotNull((object) itemPath1, typeof (Item), "Could not create media folder: '{0}'.", (object) itemPath);
  }
  return itemPath1;
}

As you see, it first tries to find the existing parent item using SecurityDisabler and database.GetItem(itemPath, options.Language) method. And from what you wrote ("The duplicated folders all have exactly 1 item in them."), in some cases it cannot find the proper folder.

Now the solution how to debug your problem is to try to use same code as Sitecore does for getting those duplicated folders and see why this code does not return correct parent folder.

Answer 2

Expected Behavior

When Sitecore goes to create a new media item, it also creates all missing folders (comparing by item name) in the new media item's path. In Sitecore, when any new item (not just a media folder item) is created, the new item is automatically added to cache and immediately available for retrieval. As such, when creating a new media item with missing media folders in its path subsequent calls to create another media item at the same parent path shouldn't result in duplicate folder names.

Where this Breaks Down

I have seen the expected behavior break down when trying to bulk import media items inside of a BulkUpdateContext that is paired with a CacheDisabler.

A BulkUpdateContext is analogous to a transaction that commits all of your changes (including item creations) at once. I believe the cause of the breakdown is that when Sitecore goes to create the second media item at the same parent path as the first (which had missing media folders in its path) the newly created folders from the first are not yet available for retrieval from the database when the code looks for them, but rather should only be available from the Cache, which has been disabled. As such, new items with the same names are created.

The above is pretty dense to digest, so I have described it step-by-step below:

Steps in the Breakdown

The following steps describe where I believe the issue occurs inside of a BulkUpdateContext with a CacheDisabler (referencing the methods that @MarekMusielak posted):

1. Custom code makes a call to create a new media item at the path within the media library: /sitecore/media/missing-media-folder/media-item-1
2. Sitecore API checks the path to see if any folders need to be created by calling GetItem on the ancestry of the media item to be created, starting at the parent. This occurs in the GetParentFolder method
3. Sitecore API finds that /sitecore/media/missing-media-folder is missing and thus a media folder item will need to be created with that name
4. Sitecore API calls CreateFolder from the GetParentFolder method in order to "queue" (since we are in a BulkUpdateContext) the creation of the missing parent item, missing-media-folder, and returns the resulting Item. Note that the returned Item representing missing-media-folder is actually not yet truly created since we are still in a BulkUpdateContext
5. The media-item-1 item is "queued" to be created as a child of the returned Item
6. Custom code then makes a call to add another new media item at the following path: /sitecore/media/missing-media-folder/media-item-2. Note that the parent folder has the same name as the one we just "queued" to be created for media-item-1
7. Sitecore API checks the path to see if any folders need to be created by calling GetItem on the ancestry of media-item-2
8. Sitecore API doesn't find missing-media-folder because it has not actually been created yet (still in a BulkUpdateContext)
9. Sitecore API calls to create the missing-media-folder again, because it couldn't find it with GetItem.
10. ... and so on...

Solution

I am a fan of the binary search method of debugging/troubleshooting, so if you are using a BulkUpdateContext try removing it and running your code on a smaller set of data in order to test it first. The code will run much slower without it, but you shouldn't get any more duplicate folder names.

If you now see the expected behavior, try adding the BulkUpdateContext back in and remove the CacheDisabler. This will confirm that the issue is the with the pairing of the two. If you again see the expected behavior then you will have solved your problem with little to no impact on performance (possibly even a gain in performance).

I do recommend that if you remove the CacheDisabler you make sure that you clear the cache (either via code or manually from the http://<yourdomain>/sitecore/admin/cache.aspx page) after running the tool. This will make sure that you don't inflate your cache with a bunch of items in the media folder that it may or may no longer need cached after the script is run.

Answer 3

Some things you could try:

Add a wait state in your for-loop, not that it is a desirable solution but just to confirm it is a timing-sensitive issue. I suspect the problem would stop.

Try using BulkUpdateContext instead of SecurityDisabler

using (new BulkUpdateContext())

The import will run much quicker, but you'll need to re-index relevant indexes when done.

Try using the static using the static media creator? Shouldn't make any difference, but...

mediaItem = Sitecore.Resources.Media.MediaManager.Creator.CreateFromFile(fileName, options);