robots.txt Builder - jquery.app

Loading tool...

What is robots.txt Builder?

A robots.txt file gives crawlers site-level access rules. It can allow or disallow paths, point to the sitemap, and document crawler policies. Static sites often need a small, predictable robots.txt file because build outputs and GitHub Pages deployments only publish what exists in the final folder.

How to use this tool

Enter the public site URL and sitemap URL.
Choose whether normal crawlers should be allowed across the site.
Add disallowed paths only when there is a real reason to block crawling.
Copy the result into robots.txt at the published site root.

What you can use it for

Create a clean robots.txt file for GitHub Pages.
Add a sitemap reference without hand-writing the file.
Document public crawler access before launch.

Use cases

Practical examples

Example

Static site launch

A new static site needs to allow crawling and point search engines to the sitemap. The builder creates a minimal file with the final sitemap URL.

Example

Block generated utility folders

A site can disallow temporary export paths while keeping public pages and tools open to crawlers.

Common mistakes

Blocking important pages and expecting them to rank.
Using robots.txt to remove a page that is already indexed instead of using noindex or removal tools.
Forgetting to publish robots.txt into the final build output.

FAQ