Home/Tools/AI robots.txt Generator
Free tool · AI Search (AEO & GEO)

AI robots.txt Generator

Decide exactly which AI crawlers can read your website. Load your current robots.txt or start from a preset, switch 17 AI crawlers on or off, and get a clean file that keeps you in AI answers without feeding model training. Your other rules stay as they are.

FreeNo sign-up17 AI crawlersKeeps your rulesBuilt-in safety check

We load your live file, set the switches to match it and keep every rule that is not about AI crawlers. Or skip this and start from a preset.

Start from a preset

Googlebot and Bingbot are not switched here. They follow your rules for all other crawlers, so your search rankings stay as they are.

Pre-filled with the WordPress default. Allowed AI crawlers follow these rules too.

For crawlers that are not switched above, such as SEO tools. Loading your current file fills this in for you.

robots.txt 

      

      

        
What this file does

        
    
        
      
    
      
    Upload the file to the root of your domain so it opens at yourwebsite.com/robots.txt. On WordPress you can paste it into the robots.txt editor in Yoast SEO or Rank Math.
    
    
    
  
    

    

    
			
    
				
    How it works
    How To Use The AI robots.txt Generator
    01
    Load your current file
    Enter your domain to load your live robots.txt, or skip this and start from a preset. Recommended keeps you in AI answers and blocks training crawlers.
    02
    Choose your crawlers
    Switch any AI crawler on or off. "What this file does" shows the effect as you go and warns you if a rule would block Google.
    03
    Test and upload
    Test the new file in the AI Crawler Checker with one click, then copy or download it and upload it to your site root.
    
			
    
				
    Being readable is step one
    Want ChatGPT And Gemini To Recommend You, Not Just Crawl You?
    Our AEO & GEO team checks which questions your customers ask AI, where you are missing from the answers, and what to fix first.
    Get a free AI visibility auditTalk to a strategist
    
			
    
				
    Questions
    AI robots.txt Generator FAQ
    Straight answers to what people ask about this tool.
    Will this robots.txt change how Google ranks me?
    No. Googlebot and Bingbot are never switched off by a preset. They follow the rules for all other crawlers, which start with the standard WordPress rules or your current ones. If a rule would block Google, the tool warns you before you download.
    Will I lose my existing rules?
    No. When you load your current file, rules for all crawlers are kept in "Rules for all other crawlers", rules for other bots such as SEO tools are kept as written in "Rules for specific crawlers", and every sitemap line is kept. Only the AI crawler groups are rebuilt from the switches.
    What is the recommended setup for most businesses?
    Allow AI search and assistant crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot and similar) so AI tools can cite you, and block training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot and similar) if you do not want your content used to train models.
    What is the difference between GPTBot and OAI-SearchBot?
    GPTBot collects content to train OpenAI models. OAI-SearchBot indexes pages so they can appear in ChatGPT search. Blocking GPTBot does not stop you appearing in ChatGPT answers, as long as OAI-SearchBot is allowed.
    How do I add robots.txt to WordPress?
    In Yoast SEO go to Tools, File editor. In Rank Math go to General Settings, Edit robots.txt. Paste the generated file and save. Without a plugin, upload robots.txt to your site root with your hosting file manager or FTP.
    Does robots.txt stop all AI scraping?
    It stops crawlers that follow the standard, which includes OpenAI, Anthropic, Google, Apple, Perplexity and the others in this tool. It is a request, not a lock. Firewalls can also block crawlers your robots.txt allows, so run the AI Crawler Checker after you upload the file.