Solr query length limit

Question

We are having an issue where queries are too big so we get error 414 (Request-URI Too Long).

From what I can see, sitecore sends a GET request with the query to the server. Is there a way to switch this so it sends a POST request instead? If not, how do you usually handle big queries in SOLR?

PS: we tried modifying the requestBufferSize in jetty.xml and still didn't work.

Using bitnami solr 4.10.3 and Sitecore 8.1 update 1

Answer 1

I don't know if the problem is relevant yet, but it's actually possible to switch Sitecore to using POST queries instead of GET when communicating to Solr.

You need to decompile the IOC integration library that you use from the Sitecore Solr Support Package. On the project where I currently work we use Unity. And here's what we did:

public void Initialize() {
...            
if (SolrContentSearchManager.EnableHttpCache)
{
    this.Container.RegisterType(typeof(ISolrCache), typeof(HttpRuntimeCache), new InjectionMember[0]);
    List<ContainerRegistration> list = this.Container.Registrations.Where<ContainerRegistration>((Func<ContainerRegistration, bool>)(r => r.RegisteredType == typeof(ISolrConnection))).ToList<ContainerRegistration>();
    if (list.Count > 0)
    {
          foreach (ContainerRegistration containerRegistration in list)
          {
               ContainerRegistration registration = containerRegistration;
               SolrServerElement solrServerElement = this.Cores.FirstOrDefault<SolrServerElement>((Func<SolrServerElement, bool>)(core => registration.Name == core.Id + registration.MappedToType.FullName));
               if (solrServerElement == null)
               {
                     Log.Error("The Solr Core configuration for the '" + registration.Name + "' Unity registration could not be found. The HTTP cache for the Solr connection to the Core cannot be configured.", (object)this);
                }
                else
                {
                     InjectionMember[] injectionMemberArray = new InjectionMember[1]
                     {
                         //Change starts
                         (InjectionMember) new InjectionConstructor(new object[2] {
       (object) new SolrConnection(solrServerElement.Url),
       (object) solrServerElement.Url
      })
};                                     

this.Container.RegisterType(typeof(ISolrConnection), typeof(PostSolrConnection), registration.Name, (LifetimeManager)null, injectionMemberArray);
                            //Change ends
                        }
                    }
                }
            }
...
}

When you do this change, you also need to adjust the maxBooleanClauses setting in your solrconfig.xml if you have more than 1024 clauses in any of your queries. See https://wiki.apache.org/solr/SolrConfigXml for more information.

Answer 2

Without rewriting a lot of the Sitecore implementation you won't be able to switch to POST (this is really unfortunate).

With regards to the buffer size, I believe you're using the wrong parameter. Try increasing the headerBufferSize. Increasing the buffer size will solve your problems right now but isn't really a scalable solution as it's linear in that you're just increasing memory.

As you've already mentioned you're looking to reduce the size of the query which is what I would suggest. Depending on what you're doing, you could look to store reduced length values that you can query against.

Answer 3

I had a similar issue and managed to get the connection via POST as explained here.

Create a new SolrStartUp class inheriting from the default one: Here is where you use The PostSolrnConnection

public class PostSolrStartUp: DefaultSolrStartUp
    {
        protected override ISolrConnection CreateConnection(string serverUrl)
        {
            SolrConnection basecon = new SolrConnection(serverUrl) { Timeout = SolrContentSearchManager.ConnectionTimeout };


            FieldInfo field = typeof(DefaultSolrStartUp).GetField("solrCache", BindingFlags.Instance | BindingFlags.NonPublic);
            var basecache = field.GetValue(this);
            if (basecache != null)
            {
                basecon.Cache = (ISolrCache)basecache;
            }
            PostSolrConnection solrConnection = new PostSolrConnection(basecon, serverUrl);
            return solrConnection;
        }
    }

Create a new Sitecore Initializer and use your start up class

public class InitializeSolrProvider
    {
        public InitializeSolrProvider()
        {
        }

        public void Process(PipelineArgs args)
        {
            if (!SolrContentSearchManager.IsEnabled)
            {
                return;
            }
            if (!IntegrationHelper.IsSolrConfigured())
            {
                (new PostSolrStartUp()).Initialize();
                return;
            }
            IntegrationHelper.ReportDoubleSolrConfigurationAttempt(this.GetType());
        }
    }

Replace the default initializer with yours

 <sitecore>
    <pipelines>
      <initialize>
        <processor type="ClearPeople.Sitecore.ContentSearch.SolrProvider.InitializeSolrProvider, ClearPeople.Sitecore" patch:instead="processor[@type='Sitecore.ContentSearch.SolrProvider.Pipelines.Loader.InitializeSolrProvider, Sitecore.ContentSearch.SolrProvider']" />
      </initialize>
    </pipelines>
  </sitecore>

Please, comment any issue you could see with this solution

Answer 4

We had similar issues. I think between Jetty and IIS config we could get our requests upto around 40,000 characters long. The other thing that helped shorten our query length was use Solr Parameter Substitution / Macro Expansion feature.

The steps we used to fix our issues with Solr Queries (using IIS and Jetty) were are follows:

Configure IIS to accept massive requests. Create Registry keys: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HTTP\Parameters

• DWORD - MaxRequestBytes - Value: 131072 (decimal) Very big. Approx twice what Jetty is configured to.
• DWORD - MaxFieldLength - Value: 131072 (decimal) More information here: https://support.microsoft.com/en-us/kb/820129

Restart HTTP stack. From a command line:

• net stop http
• net start http
• iisreset

Configure Solr (Jetty)

• Open the main Jetty.xml file
• Look for "requestHeaderSize" property.
• Change default value from 8192 to 65536

Restart Solr Service

Answer 5

There is no limit on Solr side - may be try with increasing maxFormContentSize if you are using Jetty if you are using POST.

For GET, I think you can increase 1024 in your solrconfig.xml.

Hopefully, it should work for you!