Return to Question

Using Sitecore 10.1 with Commerce 10.1

There is a field Robots Content in this item - /sitecore/content/Sonic/EchoPark/Settings.
This is a container setup and creating a file is not an option.

When a user browses mysite/robots.txt, I want the robots.txt to have whatever is entered in the CMS field.

This outcome is not consistent.

If the content is this:

User-agent:*
Crawl-delay: 10
Sitemap: https://example.com

The output is

Sitemap: https://example.com

There is some logic happening internally here and it is removing content. Is there any documentation explaining this. Is there any way to achieve what I need without overwriting the processor.

UPDATE:

I have realized that the content in the robots.txt page seems to be from cache. I had to do an iisreset for every CMS change.

But this issue remains.

When I add Crawl-delay:10 it is not showing in the robots page. That part is being removed.

Using Sitecore 10.1 with Commerce 10.1

There is a field Robots Content in this item - /sitecore/content/Tenant/MySite/Settings.
This is a container setup and creating a file is not an option.

When a user browses mysite/robots.txt, I want the robots.txt to have whatever is entered in the CMS field.

This outcome is not consistent.

If the content is this:

User-agent:*
Crawl-delay: 10
Sitemap: https://example.com

The output is

Sitemap: https://example.com

There is some logic happening internally here and it is removing content. Is there any documentation explaining this. Is there any way to achieve what I need without overwriting the processor.

UPDATE:

I have realized that the content in the robots.txt page seems to be from cache. I had to do an iisreset for every CMS change.

But this issue remains.

When I add Crawl-delay:10 it is not showing in the robots page. That part is being removed.

Using Sitecore 10.1 with Commerce 10.1

There is a field Robots Content in this item - /sitecore/content/Sonic/EchoPark/Settings.
This is a container setup and creating a file is not an option.

I want the robots.txt to have whatever entered the CMS field when a user browses mysite/robots.txt.

This outcome is not consistent.

If the content is this:

User-agent:*
Crawl-delay: 10
Sitemap: https://example.com

The output is

Sitemap: https://example.com

There is some logic happening internally here and it is removing content. Is there any documentation explaining this. Is there any way to achieve what I need without overwriting the processor.

UPDATE:

I have realized that the content in the robots.txt page seems to be from cache. I had to do an iisreset for every CMS change.

But this issue remains.

When I add Crawl-delay:10 it is not showing in the robots page. That part is being removed.

Using Sitecore 10.1 with Commerce 10.1

There is a field Robots Content in this item - /sitecore/content/Sonic/EchoPark/Settings.
This is a container setup and creating a file is not an option.

When a user browses mysite/robots.txt, I want the robots.txt to have whatever is entered in the CMS field.

This outcome is not consistent.

If the content is this:

User-agent:*
Crawl-delay: 10
Sitemap: https://example.com

The output is

Sitemap: https://example.com

There is some logic happening internally here and it is removing content. Is there any documentation explaining this. Is there any way to achieve what I need without overwriting the processor.

UPDATE:

I have realized that the content in the robots.txt page seems to be from cache. I had to do an iisreset for every CMS change.

But this issue remains.

When I add Crawl-delay:10 it is not showing in the robots page. That part is being removed.

Using Sitecore 10.1 with Commerce 10.1

There is a field Robots Content in this item - /sitecore/content/Sonic/EchoPark/Settings.
This is a container setup and creating a file is not an option.

I want the robots.txt to have whatever entered the CMS field when a user browses mysite/robots.txt.

This outcome is not consistent.

If the content is this:

User-agent:*
Crawl-delay: 10
Sitemap: https://example.com

The output is

Sitemap: https://example.com

There is some logic happening internally here and it is removing content. Is there any documentation explaining this. Is there any way to achieve what I need without overwriting the processor.

Using Sitecore 10.1 with Commerce 10.1

There is a field Robots Content in this item - /sitecore/content/Sonic/EchoPark/Settings.
This is a container setup and creating a file is not an option.

I want the robots.txt to have whatever entered the CMS field when a user browses mysite/robots.txt.

This outcome is not consistent.

If the content is this:

User-agent:*
Crawl-delay: 10
Sitemap: https://example.com

The output is

Sitemap: https://example.com

There is some logic happening internally here and it is removing content. Is there any documentation explaining this. Is there any way to achieve what I need without overwriting the processor.

UPDATE:

I have realized that the content in the robots.txt page seems to be from cache. I had to do an iisreset for every CMS change.

But this issue remains.

When I add Crawl-delay:10 it is not showing in the robots page. That part is being removed.