Some contacts are missing in Experience Profile (IndexWorker error)

Question

We have Sitecore 9.1 with Solr-7.2.1 environment. Some contacts in our solution are missing in Experience Profile. All of these contacts are known and have a lot of webvisits.

And everything is OK when I open contact by direct link like (no errors in logs): /sitecore/client/Applications/ExperienceProfile/contact?cid=608BD993-34A6-0000-0000-05B12000A4AC

After investigation I found an weird error in IndexWorker log that appears when I reload Experience Profile:

System.FormatException: Invalid length for a Base-64 char array or string.
   at System.Convert.FromBase64_Decode(Char* startInputPtr, Int32 inputLength, Byte* startDestPtr, Int32 destLength)
   at System.Convert.FromBase64CharPtr(Char* inputPtr, Int32 inputLength)
   at System.Convert.FromBase64String(String s)
   at Sitecore.XConnect.Serialization.XObjectReader.ReadPrimitive(JsonReader reader, XdbPrimitiveTypeKind typeKind)
   at Sitecore.XConnect.Serialization.XObjectReader.ReadXObject(XdbType expectedType)
   at Sitecore.Xdb.Collection.Search.Solr.SolrResults.JsonParseExtensions.ToXObject(JObject jobject, XdbModel model, XdbType expectedType)
   at Sitecore.Xdb.Collection.Search.Solr.SolrResults.JsonParseExtensions.ToXObject(JObject jobject, XdbModel model, EntityType entityType, String facetKey)
   at Sitecore.Xdb.Collection.Search.Solr.DataRecordMapper.CreateFacetPropertyWithLastModified(KeyValuePair`2 facet)
   at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
   at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
   at System.Linq.Enumerable.<ConcatIterator>d__59`1.MoveNext()
   at Sitecore.Xdb.Collection.Search.Solr.DataRecordMapper.<ConvertToXObjects>d__30.MoveNext()
   at Sitecore.Xdb.Collection.Search.Solr.JObjectExpansion.JObjectPropertiesExpander.Expand(IEnumerable`1 properties)
   at Sitecore.Xdb.Collection.Search.Solr.DataRecordMapper.<MapObjects>b__13_0(DataRecord objectToIndex)
   at System.Linq.Enumerable.WhereSelectListIterator`2.MoveNext()
   at Sitecore.Xdb.Collection.Search.Solr.JsonPostCreator.CreateUpdateBatchJson(IEnumerable`1 docsToAdd)
   at Sitecore.Xdb.Collection.Search.Solr.JsonPostCreator.<>c__DisplayClass2_0.<CreateBatchUpdatePosts>b__0(IReadOnlyCollection`1 b)
   at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
   at Sitecore.Xdb.Collection.Search.Solr.SolrWriter.ParallelProcessing[T](IEnumerable`1 dataToProcess, Func`3 asyncFunc, SemaphoreSlim throttle, CancellationToken cancellationToken)
   at Sitecore.Xdb.Collection.Search.Solr.SolrWriter.SendJsonPostsToSolr(Uri updateUri, IEnumerable`1 jsonPosts, CancellationToken cancellationToken)
   at Sitecore.Xdb.Collection.Search.Solr.SolrWriter.<>c__DisplayClass14_0.<Write>b__0()
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.Xdb.Collection.Search.Solr.SolrWriter.<Write>d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.Xdb.Collection.Indexing.IndexWriterCountersDecorator.<Write>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.Xdb.Collection.Indexing.Indexer.<IndexSingleSetAbortingOnSyncWriteFailures>d__15.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.Xdb.Collection.Indexing.AsyncForeachExtensions.<>c__DisplayClass2_0`2.<<GetRunActionWithCancelOnFailure>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.Xdb.Collection.Indexing.AsyncForeachExtensions.<ParallelAsyncForeach>d__4`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.Xdb.Collection.Indexing.AsyncForeachExtensions.<ParallelAsyncForeach>d__0`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.Xdb.Collection.Indexing.IndexerExtensions.<IndexNextChangesSimple>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.Xdb.Collection.Indexing.SingleThreadedIndexer.<IndexNextChangesWithTiming>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Sitecore.Xdb.Collection.Indexing.SingleThreadedIndexer.<RunInThread>d__6.MoveNext()

As I understand from this error, it failed when try to parse some contact facet property. But I don`t even have custom facets, I use only default ones:

• PersonalInformation

• EmailAddress

• PhoneNumber

• Avatar

Has anybody faced this problem? Any advice would be much appreciated.

Answer 1

After investigating I found that this error occured in Avatar facet.

It looks like sitecore indexer bug.

This issue happens when image for avatar has a big/medium resolution. Image data (byte[]) is big and as I understand there are some limits in JsonReader or somewhere else.

After many attempts I found that 128x128 jpeg image is OK and error dissappears, but 256x256 jpeg image still produced this issue (original resolution in Experience Profile is 170x170).

So I just add a code snippet that resize avatar image before saving to xconnect facet.

This is my helper:

public static class ImageHelper
{
    public static byte[] ResizeToJpeg(Stream stream, int width, int height, long qualityPercent)
    {
        Image img = Image.FromStream(stream);
        var bitmap = ResizeImage(img, width, height);
        return Compress(bitmap, qualityPercent);
    }

    private static Bitmap ResizeImage(Image image, int width, int height)
    {
        var destRect = new Rectangle(0, 0, width, height);
        var destImage = new Bitmap(width, height);

        destImage.SetResolution(image.HorizontalResolution, image.VerticalResolution);

        using (var graphics = Graphics.FromImage(destImage))
        {
            graphics.CompositingMode = CompositingMode.SourceCopy;
            graphics.CompositingQuality = CompositingQuality.Default;
            graphics.InterpolationMode = InterpolationMode.Default;
            graphics.SmoothingMode = SmoothingMode.Default;
            graphics.PixelOffsetMode = PixelOffsetMode.Default;

            using (var wrapMode = new ImageAttributes())
            {
                wrapMode.SetWrapMode(WrapMode.TileFlipXY);
                graphics.DrawImage(image, destRect, 0, 0, image.Width, image.Height, GraphicsUnit.Pixel, wrapMode);
            }
        }

        return destImage;
    }

    private static ImageCodecInfo GetEncoder(ImageFormat format)
    {
        ImageCodecInfo[] codecs = ImageCodecInfo.GetImageDecoders();
        foreach (ImageCodecInfo codec in codecs)
        {
            if (codec.FormatID == format.Guid)
            {
                return codec;
            }
        }
        return null;
    }

    private static byte[] Compress(Bitmap bitmap, long qualityPercent)
    {
            var jpgEncoder = GetEncoder(ImageFormat.Jpeg);

            var myEncoderParameters = new EncoderParameters(1);

            var myEncoderParameter = new EncoderParameter(Encoder.Quality, qualityPercent);
            myEncoderParameters.Param[0] = myEncoderParameter;

            MemoryStream memoryStream = new MemoryStream();
            bitmap.Save(memoryStream, jpgEncoder, myEncoderParameters);

            return memoryStream.ToArray();

    }
}

And an example how I use it:

Stream stream = new MemoryStream(imageBytes);

var resizedImageBytes = ImageHelper.ResizeToJpeg(stream, 128, 128, 50L);

Avatar avatar = new Avatar(resizedImageBytes, "image/jpeg");