Anonymous usage statistics

With your permission we record which pages are visited, using random identifiers, to understand how the site is used. Nothing is recorded unless you accept. Privacy Policy

Free tool

Robots.txt and sitemap generator

Build the two files that guide search engines, check your rules, and test whether a crawler may open a path.

Runs in your browser. What you type is not sent anywhere.

Start from

Crawlers

One crawler name per line. Use * for all of them.

One path per line, such as /admin/ or /*.pdf$.

Paths that stay open inside a blocked one.

Optional. Google ignores it; some other crawlers follow it.

One full address per line, such as https://example.com/sitemap.xml.

The file

User-agent: *Disallow:

Put robots.txt at the root of your site, for example https://example.com/robots.txt.

Checks

No problems found.

Test a path

AllowedNo rule matches this path, so it is allowed.

robots.txt asks crawlers to stay away from paths. It does not hide a page or protect it, and a crawler can ignore it, so never use it to keep something private. Add a last modified date only when it is true; a wrong date makes search engines trust the sitemap less.

How to use it

  1. 01Choose a starting point, then change the crawlers, the blocked paths and the allowed paths.
  2. 02Read the checks, and test a path to see which rule decides it.
  3. 03Download the file, put it on your site, and build the sitemap on the other tab.

Questions

What does robots.txt do?

It tells crawlers which parts of a site they should not request. Search engines such as Google follow it. It is a request, not a lock, so it cannot keep a page private.

Which rule wins when two rules match?

The longest matching rule wins, because it is the most specific. If an Allow and a Disallow match with the same length, the Allow wins. The test box shows which rule decided.

Should I block AI crawlers?

That is your choice. The preset lists crawler names that their companies publish for this, and well-behaved ones follow it. It will not stop a crawler that ignores robots.txt.

Do I need a sitemap?

Small sites that are well linked are usually found without one. A sitemap helps search engines find pages on larger sites, new sites and sites with pages that are hard to reach by links.

Is anything stored or sent?

No. The files are built in your browser and nothing you type is sent anywhere.

Crawl rules are one part of technical SEO

We handle crawling, indexing, canonical and hreflang tags, sitemaps and rendering across whole sites, in Arabic and English.

See SEO services

More free tools

All tools