1

I was wondering if anyone has come across this solr error when rebuilding indexes that contain coordinate information, and if there is anyway (other than cleaning the data) to mitigate it.

An example of the error that is in Sitecore's crawling log is:

Exception: SolrNet.Exceptions.SolrConnectionException
Message: <?xml version="1.0" encoding="UTF-8"?>
<response>

<lst name="responseHeader">
  <int name="status">400</int>
  <int name="QTime">545</int>
</lst>
<lst name="error">
  <lst name="metadata">
    <str name="error-class">org.apache.solr.common.SolrException</str>
    <str name="root-error-class">java.text.ParseException</str>
  </lst>
  <str name="msg">ERROR: [doc=sitecore://master/{a885844b-34cd-4eb1-a9d0-2ab1bcc8587a}?lang=en&amp;ver=1&amp;ndx=sitecore_master_index] Error adding field 'coordinate_rpt'='-121.444851,37.712654' msg=Unable to parse shape given formats "lat,lon", "x y" or as WKT because java.text.ParseException: Unknown Shape definition [-121.444851,37.712654]</str>
  <int name="code">400</int>
</lst>
</response>

Source: SolrNet
   at SolrNet.Impl.SolrConnection.PostStream(String relativeUrl, String contentType, Stream content, IEnumerable`1 parameters)
   at SolrNet.Impl.SolrConnection.Post(String relativeUrl, String s)
   at SolrNet.Impl.LowLevelSolrServer.SendAndParseHeader(ISolrCommand cmd)
   at Sitecore.ContentSearch.SolrProvider.SolrBatchUpdateContext.AddRange(IEnumerable`1 group, Int32 groupSize)
   at Sitecore.ContentSearch.SolrProvider.SolrBatchUpdateContext.AddDocument(Object itemToAdd, IExecutionContext[] executionContexts)
   at Sitecore.ContentSearch.SolrProvider.SolrIndexOperations.ApplyPermissionsThenIndex(IProviderUpdateContext context, IIndexable version)
   at Sitecore.ContentSearch.SitecoreItemCrawler.DoAdd(IProviderUpdateContext context, SitecoreIndexableItem indexable)
   at Sitecore.ContentSearch.HierarchicalDataCrawler`1.CrawlItem(T indexable, IProviderUpdateContext context, CrawlState`1 state)

Nested Exception

Exception: System.Net.WebException
Message: The remote server returned an error: (400) Bad Request.
Source: System
   at System.Net.HttpWebRequest.GetResponse()
   at HttpWebAdapters.Adapters.HttpWebRequestAdapter.GetResponse()
   at SolrNet.Impl.SolrConnection.GetResponse(IHttpWebRequest request)
   at SolrNet.Impl.SolrConnection.PostStream(String relativeUrl, String contentType, Stream content, IEnumerable`1 parameters)

I appreciate that the coordinate data is incorrect (user data entry issue), however the side affect is that other items get missed from the index when this error is thrown. I suspect that this happening because java is still recovering from the exception while Sitecore continues to send items to be indexed.

Is it possible to have solr handle these errors more gracefully, or do we need to add some Sitecore validation to ensure that bad geo coordinate data does not get into Sitecore?

Sitecore 9.1.1 (rev. 002459) Solr 7.1.0

1

So, in the end I took a look at how Sitecore was indexing the coordinates and it turns out that it uses a computed index field. So I wrote my own with additional validation to replace it (client asked for logging):

public class CoordinateValidationComputedIndex : AbstractComputedIndexField
    {
        private static readonly ILog Logger = LogManager.GetLogger("Sitecore.Diagnostics.Crawling") ?? LoggerFactory.GetLogger(typeof(CrawlingLog));
        public override object ComputeFieldValue(IIndexable indexable)
        {
            Item obj = indexable as SitecoreIndexableItem;
            if (obj == null || !obj.Fields.Contains(new ID(Constants.Latitude)) ||
                !obj.Fields.Contains(new ID(Constants.Longitude)))
            {
                return null;
            }

            if (!double.TryParse(obj[new ID(Constants.Latitude)], NumberStyles.Any, CultureInfo.InvariantCulture,
                    out var lat) || !double.TryParse(obj[new ID(Constants.Longitude)], NumberStyles.Any,
                    CultureInfo.InvariantCulture, out var lon))
            {
                return null;
            }

            //Latitude Check -90 - +90
            //Longitude Check -180 - +180
            if (lat < -90 || lat > 90 || lon < -180 || lon > 180)
            {
                Logger.Warn(
                    $"Coordinate validation failed for {obj.ID.Guid:B}:{obj.Paths.FullPath}\n\rWith value of latitude: {lat}, longitude: {lon}\n\r   Latitude should be in range -90 to +90\n\r   Longitude should be in range -180 to +180");
                return null;
            }

            return new Coordinate(lat, lon).ToString();
        }
    }

Then I created the following patch file:

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:role="http://www.sitecore.net/xmlconfig/role/" xmlns:env="http://www.sitecore.net/xmlconfig/env/">
  <sitecore>
    <contentSearch>
      <indexConfigurations>
        <defaultSolrIndexConfiguration type="Sitecore.ContentSearch.SolrProvider.SolrIndexConfiguration, Sitecore.ContentSearch.SolrProvider">
          <documentOptions type="Sitecore.ContentSearch.SolrProvider.SolrDocumentBuilderOptions, Sitecore.ContentSearch.SolrProvider">
            <fields hint="raw:AddComputedIndexField">
              <field patch:instead="*[@fieldName='coordinate']" fieldName="coordinate" returnType="coordinate" >zzz.Feature.Geolocation.ComputedIndex.CoordinateValidationComputedIndex, zzz.Feature</field>
            </fields>
          </documentOptions>
        </defaultSolrIndexConfiguration>
      </indexConfigurations>
    </contentSearch>
  </sitecore>
</configuration>

And now the index builds without error and I get crawling log entries like:

4176 16:17:47 INFO  [Index=sitecore_master_index] Crawler: Processed 5000 items
7988 16:17:47 WARN  Coordinate validation failed for {570326e8-7ae6-4f7a-9354-62670d99c199} : ***Item Path Removed*** with value of latitude:-95.67611, longitude:-95.67611 - 
   Latitude should be in range -90 to +90
   Longitude should be in range -180 to +180
6564 16:17:51 INFO  [Index=sitecore_master_index] Crawler: Processed 6000 items

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.