Control who crawls your site

No signup · build, import and test your robots.txt

Robots.txt File Options

Import your live robots.txt

Pull your current file straight from your site, then edit it below. Leave blank to start from scratch.

Want to be found in AI answers (ChatGPT, Perplexity, Gemini)? Allow the AI crawlers. Only block them if you don't want your content used by AI.

Pattern Builder

Pattern Preview

Single URL Test

Bulk URL Testing

Generated robots.txt