SEO & marketing
Robots.txt Generator
Generate a simple robots.txt file with sitemap, allow, disallow, and crawl-delay directives.
User-agent: * Allow: / Disallow: /admin Disallow: /private Sitemap: https://lumio.pw/sitemap.xml
How to use this tool
Enter your site URL and sitemap URL.
Add allow and disallow paths for crawlers, then review the generated robots.txt file.
Copy the output to `/robots.txt` at the root of your production domain.
Robots.txt is about crawling, not security
Robots.txt tells cooperative crawlers which paths they should or should not crawl.
It does not block users, attackers, browsers, or non-cooperative bots from opening a URL.
Never put secret admin paths, private files, tokens, or hidden staging URLs in robots.txt as a security measure.
Sitemap directive and crawl-delay
The Sitemap directive helps crawlers discover your XML sitemap from a predictable public location.
Crawl-delay is not supported consistently by every search engine, so use it only as a soft hint.
For important SEO pages, allow crawling and rely on canonical tags, redirects, and noindex directives where appropriate.
Examples
Allow the site and point to a sitemap
OutputUser-agent: * Allow: / Sitemap: https://example.com/sitemap.xml
Block internal search pages from crawling
OutputUser-agent: * Disallow: /search Disallow: /*?q=
FAQ
Can robots.txt hide private pages?
No. Robots.txt is public and only gives crawler instructions. Use authentication, authorization, and server rules for private pages.
Where should robots.txt be hosted?
It should be available at the root of the host, for example `https://example.com/robots.txt`.
Should I include my sitemap URL?
Yes, adding a Sitemap directive is a simple way to help crawlers find the canonical XML sitemap.
Does Disallow remove a page from Google?
Not reliably. A blocked URL can still be indexed if other pages link to it. Use noindex on crawlable pages when the goal is removal from search results.