7

We are having an issue where queries are too big so we get error 414 (Request-URI Too Long).

From what I can see, sitecore sends a GET request with the query to the server. Is there a way to switch this so it sends a POST request instead? If not, how do you usually handle big queries in SOLR?

PS: we tried modifying the requestBufferSize in jetty.xml and still didn't work.

Using bitnami solr 4.10.3 and Sitecore 8.1 update 1

  • Can you post the code that is causing this? – Ahmed Okour Dec 12 '16 at 17:11
  • What Sitecore version are you on? – Dmytro Shevchenko Dec 12 '16 at 18:46
  • @DmytroShevchenko Sitecore 8.1 update 1 – josedbaez Dec 12 '16 at 19:03
  • @AhmedOkour, we are optimising the code to figure out what's creating the huge query, but want to figure out if we are able to change from get to post as well in case the optimisation does not reduce the query that much. – josedbaez Dec 12 '16 at 19:05
  • With the SolrNet library, normally, you'd just use PostSolrConnection instead of SolrConnection. But Sitecore's Solr integration libraries have SolrConnection hard-coded everywhere. So it looks like you can't use POST requests. Well, unless you decompile Sitecore dlls, change them to use the right type, and compile them back. – Dmytro Shevchenko Dec 13 '16 at 22:56
9

I don't know if the problem is relevant yet, but it's actually possible to switch Sitecore to using POST queries instead of GET when communicating to Solr.

You need to decompile the IOC integration library that you use from the Sitecore Solr Support Package. On the project where I currently work we use Unity. And here's what we did:

public void Initialize() {
...            
if (SolrContentSearchManager.EnableHttpCache)
{
    this.Container.RegisterType(typeof(ISolrCache), typeof(HttpRuntimeCache), new InjectionMember[0]);
    List<ContainerRegistration> list = this.Container.Registrations.Where<ContainerRegistration>((Func<ContainerRegistration, bool>)(r => r.RegisteredType == typeof(ISolrConnection))).ToList<ContainerRegistration>();
    if (list.Count > 0)
    {
          foreach (ContainerRegistration containerRegistration in list)
          {
               ContainerRegistration registration = containerRegistration;
               SolrServerElement solrServerElement = this.Cores.FirstOrDefault<SolrServerElement>((Func<SolrServerElement, bool>)(core => registration.Name == core.Id + registration.MappedToType.FullName));
               if (solrServerElement == null)
               {
                     Log.Error("The Solr Core configuration for the '" + registration.Name + "' Unity registration could not be found. The HTTP cache for the Solr connection to the Core cannot be configured.", (object)this);
                }
                else
                {
                     InjectionMember[] injectionMemberArray = new InjectionMember[1]
                     {
                         //Change starts
                         (InjectionMember) new InjectionConstructor(new object[2] {
       (object) new SolrConnection(solrServerElement.Url),
       (object) solrServerElement.Url
      })
};                                     

this.Container.RegisterType(typeof(ISolrConnection), typeof(PostSolrConnection), registration.Name, (LifetimeManager)null, injectionMemberArray);
                            //Change ends
                        }
                    }
                }
            }
...
}

When you do this change, you also need to adjust the maxBooleanClauses setting in your solrconfig.xml if you have more than 1024 clauses in any of your queries. See https://wiki.apache.org/solr/SolrConfigXml for more information.

1

Without rewriting a lot of the Sitecore implementation you won't be able to switch to POST (this is really unfortunate).

With regards to the buffer size, I believe you're using the wrong parameter. Try increasing the headerBufferSize. Increasing the buffer size will solve your problems right now but isn't really a scalable solution as it's linear in that you're just increasing memory.

As you've already mentioned you're looking to reduce the size of the query which is what I would suggest. Depending on what you're doing, you could look to store reduced length values that you can query against.

1

I had a similar issue and managed to get the connection via POST as explained here.

Create a new SolrStartUp class inheriting from the default one: Here is where you use The PostSolrnConnection

public class PostSolrStartUp: DefaultSolrStartUp
    {
        protected override ISolrConnection CreateConnection(string serverUrl)
        {
            SolrConnection basecon = new SolrConnection(serverUrl) { Timeout = SolrContentSearchManager.ConnectionTimeout };


            FieldInfo field = typeof(DefaultSolrStartUp).GetField("solrCache", BindingFlags.Instance | BindingFlags.NonPublic);
            var basecache = field.GetValue(this);
            if (basecache != null)
            {
                basecon.Cache = (ISolrCache)basecache;
            }
            PostSolrConnection solrConnection = new PostSolrConnection(basecon, serverUrl);
            return solrConnection;
        }
    }

Create a new Sitecore Initializer and use your start up class

public class InitializeSolrProvider
    {
        public InitializeSolrProvider()
        {
        }

        public void Process(PipelineArgs args)
        {
            if (!SolrContentSearchManager.IsEnabled)
            {
                return;
            }
            if (!IntegrationHelper.IsSolrConfigured())
            {
                (new PostSolrStartUp()).Initialize();
                return;
            }
            IntegrationHelper.ReportDoubleSolrConfigurationAttempt(this.GetType());
        }
    }

Replace the default initializer with yours

 <sitecore>
    <pipelines>
      <initialize>
        <processor type="ClearPeople.Sitecore.ContentSearch.SolrProvider.InitializeSolrProvider, ClearPeople.Sitecore" patch:instead="processor[@type='Sitecore.ContentSearch.SolrProvider.Pipelines.Loader.InitializeSolrProvider, Sitecore.ContentSearch.SolrProvider']" />
      </initialize>
    </pipelines>
  </sitecore>

Please, comment any issue you could see with this solution

0

We had similar issues. I think between Jetty and IIS config we could get our requests upto around 40,000 characters long. The other thing that helped shorten our query length was use Solr Parameter Substitution / Macro Expansion feature.

The steps we used to fix our issues with Solr Queries (using IIS and Jetty) were are follows:

Configure IIS to accept massive requests. Create Registry keys: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HTTP\Parameters

Restart HTTP stack. From a command line:

  • net stop http
  • net start http
  • iisreset

Configure Solr (Jetty)

  • Open the main Jetty.xml file
  • Look for "requestHeaderSize" property.
  • Change default value from 8192 to 65536

Restart Solr Service

0

There is no limit on Solr side - may be try with increasing maxFormContentSize if you are using Jetty if you are using POST.

For GET, I think you can increase 1024 in your solrconfig.xml.

Hopefully, it should work for you!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.