4

The issue

When I use the Upload files (advanced) feature of the Media Library to upload a zip file with images in it (Sitecore 8.1 Update 3) and check the box to Unpack ZIP archives, uploading fails with the error:

An error occured while uploading:
Exception has been thrown by the target of an invocation

image of error message

What I've tried

I checked the logs, but found no messages at the time that the error occurs (or that looked related).

I have tried tracing the network requests, but everything responds with 200s and there are no error messages in the responses.

I saw this article about handling the upload with SPE, but we don't currently have that installed (and we want content editors to be able to perform the functionality, too).

I saw this article with a promising-looking fix, but the change was already applied in my version of Sitecore.

I tried updating security to give Everyone Full Control to the entire Website directory (and descendants).

  • Have you checked the log files to see what the stack trace of the error is? – Richard Seal Oct 19 '18 at 15:42
  • @RichardSeal, sorry, I forgot to mention that I checked the logs and--even though DEBUG is enabled--there are no messages occurring at that time. – Dan Sinclair Oct 19 '18 at 15:44
  • 4
    Do you have __MACOSX folder in your zip file? Sitecore zipping mechanism doesn't work well with them. If yes, try extracting the archive on PC, remove __MACOSX folder and zip again. – Marek Musielak Oct 19 '18 at 19:03
  • @MarekMusielak, yep, this is the issue. Sitecore has logged it as a bug and I'm looking into whether there is a possible code solution. – Dan Sinclair Oct 19 '18 at 20:07
8

This is a problem with zip archives generated on Macs. Sitecore zipping mechanis doesn't work well with zip archives containing __MACOSX folder.

As a workaround you can unzip your archive on PC, remove __MACOSX folder and zip your files again.

I've tried to look for possible fix for the issue long time ago, but it would require changing multiple Sitecore classes. I don't have a list anymore, but from what I remember, the classes were:

  • two UploadProcessor classes: Save and CheckSize,
  • replacing original MediaUploader.

EDIT: full list of classes which must be changed can be found in Dan's answer

  • If you want to update your answer, the list of classes you have to change is in my answer. :) – Dan Sinclair Oct 24 '18 at 11:22
  • Thanks Dan. I just pointed from my answer to yours :) – Marek Musielak Oct 24 '18 at 11:38
2

Warning

Don't do this unless you have a really good reason to do so. Marek's answer is better practice. The method I've outlined below works, but--as you'll see--requires decompiling a lot of internal Sitecore code. This will mean, at the very least, a lot of testing when upgrading Sitecore in the future and, at the worst, significant rewriting.

You've been warned

So here's how you do it! The underlying problem is that the Sitecore.Zip library doesn't handle the property files that are in the __MACOSX folder of zip files generated on Macs. Luckily, if you're using .NET Framework 4.5 or newer, you can use Microsoft's System.IO.Compression library for handling the files and everything works fine. Everything below is to replace Sitecore's implementation with Microsoft's.

First, you need to decompile three pipelines:

Sitecore.Pipelines.Upload.CheckExtension

This iterates the contents of the zip file and looks for any restricted file extensions.

Note: this one is only necessary if you've added Sitecore's Upload Filter.

using System.Collections.Generic;
using System.IO;
using System.IO.Compression;
using System.Linq;
using System.Web;
using Sitecore.Diagnostics;
using Sitecore.Pipelines.Upload;

namespace Custom.Processors.UiUpload
{
    /// -------------------------------------------------------------------------
    /// NOTE: THIS CLASS IS DECOMPILED FROM Sitecore.Pipelines.Upload.CheckExtension.
    ///     THE ONLY SUBSTANTIVE CHANGE IS TO USE System.IO.Compression INSTEAD 
    ///     OF Sitecore.Zip FOR THE ZIP FILE HANDLING.
    /// -------------------------------------------------------------------------
    public class CustomCheckExtension : UploadProcessor
    {
        private bool _isAllowed;

        private string _extensions;

        public CustomCheckExtension(string allowed, string blocked)
        {
            if (!string.IsNullOrEmpty(allowed))
            {
                this._isAllowed = true;
                this._extensions = allowed.Replace(" ", "").ToLower();
                return;
            }
            if (!string.IsNullOrEmpty(blocked))
            {
                this._isAllowed = false;
                this._extensions = blocked.Replace(" ", "").ToLower();
            }
        }

        private List<string> PrepareExtensions(string csvExtensions)
        {
            string[] strArrays = csvExtensions.Split(',');
            List<string> strs = new List<string>();
            string[] strArrays1 = strArrays;
            for (int i = 0; i < strArrays1.Length; i++)
            {
                strs.Add(strArrays1[i]);
            }
            return strs;
        }

        public void Process(UploadArgs args)
        {
            Assert.ArgumentNotNull(args, nameof(args));
            if (string.IsNullOrEmpty(this._extensions))
            {
                return;
            }
            List<string> strs = this.PrepareExtensions(this._extensions);
            foreach (string file in args.Files)
            {
                HttpPostedFile item = args.Files[file];
                if (string.IsNullOrEmpty(item?.FileName))
                {
                    continue;
                }
                if (IsUnpack(args, item))
                {
                    try
                    {
                        var archive = new ZipArchive(item.InputStream, ZipArchiveMode.Read);
                        foreach (ZipArchiveEntry entry in archive.Entries.Where(e => !e.FullName.ToUpperInvariant().Contains("__MACOSX")))
                        {
                            string str = Path.GetExtension(entry.Name.ToLower()).TrimStart('.');
                            bool flag = strs.Contains(str);
                            if (this._isAllowed && flag || (this._isAllowed || !flag) && (!this._isAllowed || flag))
                            {
                                continue;
                            }
                            HttpContext.Current.Response.Write(string.Concat("<script type=\"text/JavaScript\">alert(\"Uploading files with .", str, " extension is restricted\");</script>"));
                            Log.Audit($"Upload restricted: {entry.Name}", this);
                            new Done().Process(args);
                            args.AbortPipeline();
                            return;
                        }
                    }
                    finally
                    {
                        item.InputStream.Position = 0;
                    }
                }
                string str1 = Path.GetExtension(item.FileName.ToLower()).TrimStart('.');
                bool flag1 = strs.Contains(str1);
                if (this._isAllowed && flag1 || (this._isAllowed || !flag1) && (!this._isAllowed || flag1))
                {
                    continue;
                }
                HttpContext.Current.Response.Write(string.Concat("<script type=\"text/JavaScript\">alert(\"Uploading files with .", str1, " extension is restricted\");</script>"));
                Log.Audit($"Upload restricted: {item.FileName}", this);
                new Done().Process(args);
                args.AbortPipeline();
                return;
            }
        }
    }
}

Sitecore.Pipelines.Upload.CheckSize

This iterates the contents of the zip file and looks for any file that is larger than the allowed size

using System.IO.Compression;
using System.Web;
using Sitecore;
using Sitecore.Configuration;
using Sitecore.Diagnostics;
using Sitecore.Pipelines.Upload;

namespace Custom.Processors.UiUpload
{
    /// <summary>
    /// Resolves the folder.
    /// 
    /// -------------------------------------------------------------------------
    /// NOTE: THIS CLASS IS DECOMPILED FROM Sitecore.Pipelines.Upload.CheckSize.
    ///     THE ONLY SUBSTANTIVE CHANGE IS TO USE System.IO.Compression INSTEAD 
    ///     OF Sitecore.Zip FOR THE ZIP FILE HANDLING.
    /// -------------------------------------------------------------------------
    /// </summary>
    public class CustomCheckSize : UploadProcessor
    {
        /// <summary>
        /// Runs the processor.
        /// </summary>
        /// <param name="args">The arguments.</param>
        public void Process(UploadArgs args)
        {
            Assert.ArgumentNotNull(args, nameof(args));
            if (args.Destination == UploadDestination.File)
            {
                return;
            }
            foreach (string file in args.Files)
            {
                HttpPostedFile item = args.Files[file];
                if (string.IsNullOrEmpty(item?.FileName))
                {
                    continue;
                }
                if (!IsUnpack(args, item))
                {
                    if (item.ContentLength <= Settings.Media.MaxSizeInDatabase)
                    {
                        continue;
                    }
                    string fileName = item.FileName;
                    HttpContext.Current.Response.Write(string.Concat("<html><head><script type=\"text/JavaScript\" language=\"javascript\">window.top.scForm.getTopModalDialog().frames[0].scForm.postRequest(\"\", \"\", \"\", 'ShowFileTooBig(", StringUtil.EscapeJavascriptString(fileName), ")')</script></head><body>Done</body></html>"));
                    args.ErrorText = $"The file \"{fileName}\" is too big to be uploaded. The maximum size for uploading files is {MainUtil.FormatSize(Settings.Media.MaxSizeInDatabase)}.";
                    args.AbortPipeline();
                    return;
                }
                var archive = new ZipArchive(item.InputStream, ZipArchiveMode.Read);
                try
                {
                    foreach (ZipArchiveEntry entry in archive.GetEntriesWithoutMacOsXDirectoryItems())
                    {
                        if (entry.GetIsDirectory() || entry.Length <= Settings.Media.MaxSizeInDatabase)
                        {
                            continue;
                        }
                        string str = string.Concat(item.FileName, "/", entry.Name);
                        HttpContext.Current.Response.Write(string.Concat("<html><head><script type=\"text/JavaScript\" language=\"javascript\">window.top.scForm.getTopModalDialog().frames[0].scForm.postRequest(\"\", \"\", \"\", 'ShowFileTooBig(", StringUtil.EscapeJavascriptString(str), ")')</script></head><body>Done</body></html>"));
                        args.ErrorText = $"The file \"{str}\" is too big to be uploaded. The maximum size for uploading files is {MainUtil.FormatSize(Settings.Media.MaxSizeInDatabase)}.";
                        args.AbortPipeline();
                        return;
                    }
                }
                finally
                {
                    item.InputStream.Position = 0;
                }
            }
        }
    }
}

Sitecore.Pipelines.Upload.Save

This is the one that does the real work (unzips the contents and adds them to the media library)

using System;
using System.Collections.Generic;
using System.IO;
using System.IO.Compression;
using System.Web;
using Sitecore.Data.Items;
using Sitecore.Diagnostics;
using Sitecore.IO;
using Sitecore.Pipelines.Upload;
using Sitecore.SecurityModel;
using Sitecore.Web;

namespace Custom.Processors.UiUpload
{
    /// <summary>
    /// Saves the uploaded files.
    /// -------------------------------------------------------------------------
    /// NOTE: THIS CLASS IS DECOMPILED FROM Sitecore.Pipelines.Upload.Save.
    ///     THE ONLY SUBSTANTIVE CHANGE IS TO USE System.IO.Compression INSTEAD 
    ///     OF Sitecore.Zip FOR THE ZIP FILE HANDLING.
    /// -------------------------------------------------------------------------
    /// </summary>
    public class CustomSave : UploadProcessor
    {
        /// <summary>
        /// Runs the processor.
        /// </summary>
        /// <param name="args">The arguments.</param>
        /// <exception cref="T:System.Exception"><c>Exception</c>.</exception>
        public void Process(UploadArgs args)
        {
            Assert.ArgumentNotNull(args, nameof(args));
            for (int i = 0; i < args.Files.Count; i++)
            {
                HttpPostedFile item = args.Files[i];
                if (!string.IsNullOrEmpty(item.FileName))
                {
                    try
                    {
                        bool flag = IsUnpack(args, item);
                        if (!args.FileOnly)
                        {
                            var mediaUploader = new CustomMediaUploader
                            {
                                File = item,
                                Unpack = flag,
                                Folder = args.Folder,
                                Versioned = args.Versioned,
                                Language = args.Language,
                                AlternateText = args.GetFileParameter(item.FileName, "alt"),
                                Overwrite = args.Overwrite,
                                FileBased = args.Destination == UploadDestination.File
                            };
                            List<CustomMediaUploadResult> mediaUploadResults;
                            using (new SecurityDisabler()) {
                                mediaUploadResults = mediaUploader.SafeUpload();
                            }
                            string[] fileName = { item.FileName };
                            Log.Audit(this, "Upload: {0}", fileName);
                            foreach (CustomMediaUploadResult mediaUploadResult in mediaUploadResults)
                            {
                                this.ProcessItem(args, mediaUploadResult.Item, mediaUploadResult.Path);
                            }
                        }
                        else if (!flag)
                        {
                            string file = this.UploadToFile(args, item);
                            if (i == 0)
                            {
                                args.Properties["filename"] = FileHandle.GetFileHandle(file);
                            }
                        }
                        else
                        {
                            UnpackToFile(args, item);
                        }
                    }
                    catch (Exception exception1)
                    {
                        Exception exception = exception1;
                        Log.Error(string.Concat("Could not save posted file: ", item.FileName), exception, this);
                        throw;
                    }
                }
            }
        }

        /// <summary>
        /// Processes the item.
        /// </summary>
        /// <param name="args">The arguments.</param>
        /// <param name="mediaItem">The media item.</param>
        /// <param name="path">The path.</param>
        private void ProcessItem(UploadArgs args, MediaItem mediaItem, string path)
        {
            Assert.ArgumentNotNull(args, nameof(args));
            Assert.ArgumentNotNull(mediaItem, nameof(mediaItem));
            Assert.ArgumentNotNull(path, nameof(path));
            if (args.Destination != UploadDestination.Database)
            {
                Log.Info(string.Concat("Media Item has been uploaded to file system: ", path), this);
            }
            else
            {
                Log.Info(string.Concat("Media Item has been uploaded to database: ", path), this);
            }
            args.UploadedItems.Add(mediaItem.InnerItem);
        }

        /// <summary>
        /// Unpacks to file.
        /// </summary>
        /// <param name="args">The arguments.</param>
        /// <param name="file">The file.</param>
        private static void UnpackToFile(UploadArgs args, HttpPostedFile file)
        {
            Assert.ArgumentNotNull(args, nameof(args));
            Assert.ArgumentNotNull(file, nameof(file));
            string str = FileUtil.MapPath(TempFolder.GetFilename("temp.zip"));
            file.SaveAs(str);
            using (var archive = new ZipArchive(file.InputStream, ZipArchiveMode.Read))
            {
                foreach (ZipArchiveEntry entry in archive.GetEntriesWithoutMacOsXDirectoryItems())
                {
                    string uniqueFilename = FileUtil.MakePath(args.Folder, entry.Name, '\\');
                    if (!entry.GetIsDirectory())
                    {
                        if (!args.Overwrite)
                        {
                            uniqueFilename = FileUtil.GetUniqueFilename(uniqueFilename);
                        }
                        Directory.CreateDirectory(Path.GetDirectoryName(uniqueFilename));
                        lock (FileUtil.GetFileLock(uniqueFilename))
                        {
                            FileUtil.CreateFile(uniqueFilename, entry.Open(), true);
                        }
                    }
                    else
                    {
                        Directory.CreateDirectory(uniqueFilename);
                    }
                }
            }
        }

        /// <summary>
        /// Uploads to file.
        /// </summary>
        /// <param name="args">The arguments.</param>
        /// <param name="file">The file.</param>
        /// <returns>The name of the uploaded file</returns>
        private string UploadToFile(UploadArgs args, HttpPostedFile file)
        {
            Assert.ArgumentNotNull(args, nameof(args));
            Assert.ArgumentNotNull(file, nameof(file));
            string uniqueFilename = FileUtil.MakePath(args.Folder, Path.GetFileName(file.FileName), '\\');
            if (!args.Overwrite)
            {
                uniqueFilename = FileUtil.GetUniqueFilename(uniqueFilename);
            }
            file.SaveAs(uniqueFilename);
            Log.Info(string.Concat("File has been uploaded: ", uniqueFilename), this);
            return Assert.ResultNotNull(uniqueFilename);
        }
    }
}

And the config patch to put them in place

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
    <sitecore>
        <processors>
            <uiUpload>
                <!-- Again, this one is optional and should only be used if you have the Sitecore Upload Filter configured -->
                <processor mode="on" type="Sitecore.Pipelines.Upload.CheckExtension, Sitecore.UploadFilter">
                    <patch:attribute name="type">Custom.Processors.UiUpload.CustomCheckExtension,Custom</patch:attribute>
                </processor>
                <processor mode="on" type="Custom.Processors.UiUpload.CustomCheckSize,Custom" patch:instead="processor[@type='Sitecore.Pipelines.Upload.CheckSize, Sitecore.Kernel']"/>
                <processor mode="on" type="Custom.Processors.UiUpload.CustomSave,Custom" patch:instead="processor[@type='Sitecore.Pipelines.Upload.Save, Sitecore.Kernel']"/>
            </uiUpload>
        </processors>
    </sitecore>
</configuration>

Next, you need to decompile some supporting classes

These classes are used by the above pipelines.

Sitecore.Resources.Media.MediaUploader

This class is used by the Save pipeline processor to actually upload the contents of the zip file to the database

using System;
using System.Collections.Generic;
using System.IO;
using System.IO.Compression;
using Sitecore.Diagnostics;
using Sitecore.IO;
using Sitecore.Pipelines.GetMediaCreatorOptions;
using Sitecore.Resources.Media;

namespace Custom.Processors.UiUpload
{
    /// -------------------------------------------------------------------------
    /// NOTE: THIS CLASS IS DECOMPILED FROM Sitecore.Resources.Media.MediaUploader.
    ///     THE ONLY SUBSTANTIVE CHANGE IS TO USE System.IO.Compression INSTEAD 
    ///     OF Sitecore.Zip FOR THE ZIP FILE HANDLING.
    /// -------------------------------------------------------------------------
    public class CustomMediaUploader : MediaUploader
    {
        public List<CustomMediaUploadResult> SafeUpload()
        {
            List<CustomMediaUploadResult> list = new List<CustomMediaUploadResult>();
            if (string.Compare(Path.GetExtension(this.File.FileName), ".zip", StringComparison.InvariantCultureIgnoreCase) == 0 && this.Unpack)
                this.UnpackToDatabase(list);
            else
                this.UploadToDatabase(list);
            return list;
        }

        private void UploadToDatabase(List<CustomMediaUploadResult> list)
        {
            Assert.ArgumentNotNull(list, nameof(list));
            CustomMediaUploadResult mediaUploadResult = new CustomMediaUploadResult();
            list.Add(mediaUploadResult);
            mediaUploadResult.Path = FileUtil.MakePath(this.Folder, Path.GetFileName(this.File.FileName), '/');
            mediaUploadResult.ValidMediaPath = MediaPathManager.ProposeValidMediaPath(mediaUploadResult.Path);
            var options = new MediaCreatorOptions
                          {
                              Versioned = this.Versioned,
                              Language = this.Language,
                              OverwriteExisting = this.Overwrite,
                              Destination = mediaUploadResult.ValidMediaPath,
                              FileBased = this.FileBased,
                              AlternateText = this.AlternateText,
                              Database = this.Database
                          };
            options.Build(GetMediaCreatorOptionsArgs.UploadContext);
            mediaUploadResult.Item = MediaManager.Creator.CreateFromStream(this.File.InputStream, mediaUploadResult.Path, options);
        }

        private void UnpackToDatabase(List<CustomMediaUploadResult> list)
        {
            Assert.ArgumentNotNull(list, nameof(list));
            string str = FileUtil.MapPath(TempFolder.GetFilename("temp.zip"));
            string tempFolderPath = str.Substring(0, str.LastIndexOf("\\", StringComparison.OrdinalIgnoreCase));
            this.File.SaveAs(str);
            ZipArchiveEntry lastEntry = null;
            try
            {

                using (ZipArchive archive = ZipFile.OpenRead(str))
                {
                    foreach (ZipArchiveEntry entry in archive.GetEntriesWithoutMacOsXDirectoryItems())
                    {
                        lastEntry = entry;
                        if (!entry.GetIsDirectory())
                        {
                            string filePath = FileUtil.MakePath(tempFolderPath, entry.Name, '/');
                            try
                            {
                                entry.ExtractToFile(filePath, true);
                                var mediaUploadResult = new CustomMediaUploadResult();
                                list.Add(mediaUploadResult);
                                mediaUploadResult.Path = FileUtil.MakePath(this.Folder, entry.Name, '/');
                                mediaUploadResult.ValidMediaPath = MediaPathManager.ProposeValidMediaPath(mediaUploadResult.Path);
                                var options = new MediaCreatorOptions
                                              {
                                                  Language = this.Language,
                                                  Versioned = this.Versioned,
                                                  OverwriteExisting = this.Overwrite,
                                                  Destination = mediaUploadResult.ValidMediaPath,
                                                  FileBased = this.FileBased,
                                                  Database = this.Database
                                              };
                                options.Build(GetMediaCreatorOptionsArgs.UploadContext);
                                mediaUploadResult.Item = MediaManager.Creator.CreateFromFile(filePath, options);
                            }
                            finally
                            {
                                if (System.IO.File.Exists(filePath))
                                    System.IO.File.Delete(filePath);
                            }
                        }
                    }
                }
            }
            catch (Exception ex)
            {
                Log.Error($"[CustomMediaUploader] Error trying to save {lastEntry?.FullName} from {this.File.FileName}", ex, this);
            }
            finally
            {
                FileUtil.Delete(str);
            }
        }
    }
}

Sitecore.Resources.Media.MediaUploadResult

This class is used to provide the result of the file upload back to the MediaUploader and had read-only properties (well, technically the sets are internal)

using Sitecore.Data.Items;

namespace Musc.Common.Web.Processors.UiUpload
{
    /// -------------------------------------------------------------------------
    /// NOTE: THIS CLASS IS DECOMPILED FROM Sitecore.Resources.Media.MediaUploadResult.
    ///     THE ONLY SUBSTANTIVE CHANGE IS TO ALLOW ACCESS TO SET THE PROPERTIES.
    /// -------------------------------------------------------------------------
    public class CustomMediaUploadResult
    {
        public Item Item { get; internal set; }
        public string Path { get; internal set; }
        public string ValidMediaPath { get; internal set; }
    }
}

Last but not least, you may find these extension methods handy

ZipArchiveExtensions

This adds an extension method to get the files in the archive that aren't in a __MACOSX folder

public static class ZipArchiveExtensions
{
    public static IEnumerable<ZipArchiveEntry> GetEntriesWithoutMacOsXDirectoryItems([NotNull] this ZipArchive archive)
    {
        return archive.Entries.Where(e => !e.FullName.ToUpperInvariant().Contains("__MACOSX"));
    }
}

ZipArchiveEntryExtensions

This adds an extension method to see whether the current entry is a directory or not

public static class ZipArchiveEntryExtensions
{
    public static bool GetIsDirectory([NotNull] this ZipArchiveEntry entry)
    {
        return entry.FullName.EndsWith("/", StringComparison.OrdinalIgnoreCase) && string.IsNullOrEmpty(entry.Name);
    }
}

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.