robots.txt for small sites: what to allow and block

A robots.txt file sits in the root of your site. It tells crawlers which pages they may read.

It is a request, not a lock. Well-behaved crawlers follow it. Bad ones ignore it. Never use it to hide private data.

A safe default

For most small sites, allow everything and point to your sitemap:

• Allow all crawlers.
• Add a line with the address of your sitemap.

What to block

Block pages that give no value in search, such as admin pages, cart pages and internal search results. Do not block your CSS or script files. Search engines need them to show your page the way visitors see it.

AI crawlers

Some crawlers collect pages for AI tools. You choose whether to allow them. If you want AI answers to mention your business, allow them. If you want to keep your text out of AI training, block the crawlers that serve that purpose. The robots.txt generator lets you set each one.

Check your work

A mistake can hide your whole site. After you upload the file, open yourdomain.com/robots.txt and read it. Then list your pages in a sitemap.