Unable to complete sitecore_master_index rebuild

Question

When we try to do complete master index rebuild it gives the below error.

Job started: Index_Update_IndexName=sitecore_master_index|#Exception: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> SolrNet.Exceptions.SolrConnectionException: The specified CGI application encountered an error and the server terminated the process. ---> System.Net.WebException: The remote server returned an error: (502) Bad Gateway.
   at System.Net.HttpWebRequest.GetResponse()
   at HttpWebAdapters.Adapters.HttpWebRequestAdapter.GetResponse()
   at SolrNet.Impl.SolrConnection.GetResponse(IHttpWebRequest request)
   at SolrNet.Impl.SolrConnection.PostStream(String relativeUrl, String contentType, Stream content, IEnumerable`1 parameters)
   --- End of inner exception stack trace ---
   at SolrNet.Impl.SolrConnection.PostStream(String relativeUrl, String contentType, Stream content, IEnumerable`1 parameters)
   at SolrNet.Impl.SolrConnection.Post(String relativeUrl, String s)
   at SolrNet.Commands.OptimizeCommand.Execute(ISolrConnection connection)
   at SolrNet.Impl.SolrBasicServer`1.SendAndParseHeader(ISolrCommand cmd)
   at Sitecore.ContentSearch.SolrProvider.SolrSearchIndex.PerformRebuild(Boolean resetIndex, Boolean optimizeOnComplete, IndexingOptions indexingOptions, CancellationToken cancellationToken)
   at Sitecore.ContentSearch.SolrProvider.SolrSearchIndex.Rebuild(Boolean resetIndex, Boolean optimizeOnComplete)
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Sitecore.Reflection.ReflectionUtil.InvokeMethod(MethodInfo method, Object[] parameters, Object obj)
   at Sitecore.Jobs.JobRunner.RunMethod(JobArgs args)
   at (Object , Object[] )
   at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
   at Sitecore.Pipelines.DefaultCorePipelineManager.Run(String pipelineName, PipelineArgs args, String pipelineDomain)
   at Sitecore.Jobs.Job.DoExecute()
   at Sitecore.Jobs.Job.ThreadEntry(Object state)

If anyone experiences the same issue pls help here.

We are using Sitecore 9.0.1, solr 6.6.2, and deployed into Azure.

We have huge data like more than 600,000 to 700,000 items.

Thanks in advance.

Answer 1

Given the amount of content you are indexing, I think you are well past the experimental threshold when running Solr as an App Service in Azure.

Solr is a resource-intensive (Memory, Disk and Network) application. This is especially true when rebuilding one or many indices.

If you have a Solr expert on your team, I’d suggest spinning up 2-3 virtual machines for hosting Solr. You could also consider a set of containers.

If Solr is not a strength area, you can easily spin up a cloud-based instance of Solr using a paid service in a matter of minutes.

My personal experience is with SearchStax https://www.searchstax.com/solutions/sitecore-solr/

Bitnami also has offerings: https://bitnami.com/stack/solr/cloud/azure