Error While Adding Source to Sitecore Search

Question

I am trying to add source to sitecore search using Request Trigger , but I am getting error

Validation Error: the number of dropped documents is over the 25% of crawled documents
Error crawling request
Cause: Message:'Something went wrong for url: https://security.intuit.com/intuit-cookie-policy/', Type:, Code:0, Data:map[cause:Get "https://security.intuit.com/intuit-cookie-policy/": tls: failed to verify certificate: x509: certificate signed by unknown authority statusCode:0]
 Dropping document since it does not have all the required fields
Missing Fields: type
Required Fields: id, type

I am missing anything ? here

Answer 1

It looks like the document you are crawling does not have required fields i.e. type in your case. You can control field level settings under the Domain setting tab in the Admnistation.

Inside Attributes in the Domain Settings, you can control which fields are required in your content collection.

Inside the document extractor, since type(Content Type) is a required field by default, make sure you set that field as below.

function extract(request, response) {
    $ = response.body;
    return [{
        'type': // logic to store the type field in the document,
    }];
}

Hope this helps!!!