Coveo for Sitecore (5.0.943.3) - Index rebuild process spits errors in Sitecore logs

Question

I have upgraded Sitecore v9.0.1 to Sitecore v10.0.1. Along with the upgrade, Coveo for Sitecore has been upgraded to 5.0.943.3. When I rebuild the Coveo indexes, it spits a lot of errors in the Sitecore log files.

One of the observation which I noticed is I have nested BEGIN NOINDEX / END NOINDEX tags. Not sure whether this is the root cause. It was working fine in the previous Sitecore version. Could you please provide any suggestion?

Also tried to verify the URL mentioned the stacktrace and it is working fine the server where index is being rebuilt.

<!-- BEGIN NOINDEX -->
Content A
<!-- BEGIN NOINDEX -->
Content B
<!-- END NOINDEX -->
Content C
<!-- END NOINDEX --> 

Error:

ManagedPoolThread #7 14:24:05 ERROR An error occurred while running the ad-hoc pipeline "fetchPageContentHTMLPostProcessing". System.ArgumentOutOfRangeException: Length cannot be less than zero.
Parameter name: length
   at System.String.Substring(Int32 startIndex, Int32 length)
   at Coveo.SearchProvider.Utils.HtmlCleaner.CleanHtmlContent(String p_HtmlContent, String p_StartCommentText, String p_EndCommentText)
   at Coveo.SearchProvider.Processors.FetchPageContent.PostProcessing.CleanHtml.Process(FetchPageContentHtmlPostProcessingArgs p_Args)
   at Coveo.Framework.Pipelines.AdHocPipelineRunner.Run[T](T p_Args, String p_PipelineName, IEnumerable`1 p_Processors)
ManagedPoolThread #7 14:24:05 ERROR An exception occurred while trying to fetch the HTML content of the document id "{93549797-6BE8-48D4-B6DB-1C5EFC279621}" for the URL "https://sitecore.local/en-ca/knowledge". No binary data has been set.
Exception: System.ArgumentOutOfRangeException
Message: Length cannot be less than zero.
Parameter name: length
Source: mscorlib
   at System.String.Substring(Int32 startIndex, Int32 length)
   at Coveo.SearchProvider.Utils.HtmlCleaner.CleanHtmlContent(String p_HtmlContent, String p_StartCommentText, String p_EndCommentText)
   at Coveo.SearchProvider.Processors.FetchPageContent.PostProcessing.CleanHtml.Process(FetchPageContentHtmlPostProcessingArgs p_Args)
   at Coveo.Framework.Pipelines.AdHocPipelineRunner.Run[T](T p_Args, String p_PipelineName, IEnumerable`1 p_Processors)
   at Coveo.SearchProvider.Processors.FetchPageContentProcessor.GetPostProcessedContent(String p_OriginalHtmlContent, IReadOnlyCoveoIndexableItem p_IndexableItem)
   at Coveo.SearchProvider.Processors.FetchPageContent.ResponseBinaryDataFetcher.GetBinaryDataWithHtmlProcessing(IHttpWebResponse p_HttpWebResponse, Func`2 p_HtmlProcessing)
   at Coveo.Framework.CNL.LazyCache`2.get_Item(TKey p_Key)
   at Coveo.SearchProvider.Processors.FetchPageContentProcessor.ProcessItem(IndexableItemBinaryDataFacade p_Item, LazyCache`2 p_BinaryDataCache)

Answer 1

This issue ocurs when using the FetchPageContent processor, specifically during the PostProcessing step, with the CleanHtmlContent processor. If an item contains the "BEGIN NOINDEX" tag but does not have a corresponding "END NOINDEX" tag, an infinite loop will occur as the processor never stops looking for that tag.

Make sure there are end tags for each "BEGIN NOINDEX" tag. For example:

<p>
    This content is included.
    <!--BEGIN NOINDEX-->
        This content is excluded.
    <!--END NOINDEX-->
</p>

See : https://connect.coveo.com/s/article/5661