robots.txt is a public file at the root of a website that controls how search engines and other compliant crawlers, or web bots, access its URLs. URL (Uniform Resource Locator) means the address of an individual web page. Its main job is to control crawling. It is not password protection or a reliable way to remove a page from Google results.

You will usually find it at:

https://example.fi/robots.txt

What does robots.txt look like?

On a small public business website, it can be very simple:

User-agent: *
Allow: /
Sitemap: https://example.fi/sitemap.xml

Here:

  • User-agent: * refers to all crawlers that follow the rule
  • Allow: / allows crawling of the whole site
  • Sitemap: gives the address of the XML sitemap (XML means Extensible Markup Language), a machine-readable list of site URLs

More complex rules are needed only when there is a real reason to restrict crawling.

Does Disallow keep a page out of Google?

Not necessarily.

This is one of the most common misunderstandings.

Google says robots.txt is not the right tool for keeping a page out of its index. If Google discovers the URL through other links, it can sometimes appear in results even when Google cannot read the page's content.

To keep a public page out of Google Search, use an appropriate noindex directive and allow Googlebot to crawl the page so it can read that directive.

If the content is actually private, protect it with a login or other access controls.

Can robots.txt protect secret addresses?

No.

Robots.txt is public. Anyone can open it in a browser.

Do not put these in it, for example:

  • the address of a secret admin panel
  • paths to private files in the belief that this hides them
  • other information whose existence you do not want to reveal

A crawler rule is not a security measure.

How can robots.txt accidentally harm SEO?

SEO (Search Engine Optimization) can suffer if crawling of important content is accidentally blocked.

One incorrect rule can prevent a search engine from accessing important content.

For example:

User-agent: *
Disallow: /

tells compliant crawlers not to crawl anything on the site.

This can accidentally reach production when development-environment restrictions are copied during launch.

Always check robots.txt when publishing a new site.

What about ChatGPT and Claude?

AI services may use separate user agents for different purposes.

OpenAI uses OAI-SearchBot for ChatGPT search, for example. Anthropic uses Claude-SearchBot to support search and ClaudeBot separately for model development.

If robots.txt generally allows all crawlers and has no more specific blocking rule, these bots can generally follow the wildcard group. The site's CDN (Content Delivery Network) or firewall can still block their traffic separately.

If you want different rules for AI search and model-development crawling, distinguish between the user agents according to their documentation.

How are robots.txt and sitemaps connected?

You can declare the sitemap's address in robots.txt:

Sitemap: https://example.fi/sitemap.xml

They still have different jobs:

  • robots.txt tells a crawler what it may request
  • the sitemap lists the URLs you want search engines to discover

Read more about what a sitemap is.

What is a good robots.txt for a small business website?

Often, a simple file that allows crawling of the public site and gives the sitemap's location is best.

Do not add blocks just in case. Add a rule only when you know which problem it solves.

How does GC handle robots.txt?

GC's website service includes technical indexing settings and a sitemap. An important launch check is ensuring production has no crawl block intended only for development.

See also how to get your website on Google and how your business can be found in ChatGPT.

Sources: Google Search Central, Introduction to robots.txt. OpenAI Help Center, Publishers and Developers – FAQ. Anthropic Help Center, crawler guidance.