Robots.txt and AI Crawlers: The 5-Minute Fix That Makes Your Business Visible

There’s a file on your website that you’ve probably never heard of, and it might be the reason ChatGPT, Claude, and Perplexity don’t know your business exists. It’s called robots.txt, and fixing it takes five minutes.

What is robots.txt?

Every website has a robots.txt file at its root (yoursite.com/robots.txt). It tells search engines and AI crawlers which parts of your site they’re allowed to visit. Think of it as a bouncer for bots.

The problem: many website builders, CMS plugins, or web developers add rules that block AI crawlers without telling you. If your robots.txt blocks GPTBot, your website is invisible to ChatGPT. Block ClaudeBot, and Claude can’t see you. Block PerplexityBot, and Perplexity skips your site entirely.

How to check yours (60 seconds)

1. Open your browser and go to yoursite.com/robots.txt
2. Look for lines that mention any of these bot names:
   • GPTBot — ChatGPT’s crawler
   • ClaudeBot — Claude’s crawler
   • PerplexityBot — Perplexity’s crawler
   • Google-Extended — Google’s AI training crawler
   • CCBot — Used by multiple AI training datasets
3. If you see “Disallow: /” next to any of these names, that crawler is blocked from your entire site.

What good robots.txt looks like

For maximum AI visibility, your robots.txt should explicitly allow all major AI crawlers. Here’s a simple, effective configuration:

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/

Sitemap: https://yoursite.com/sitemap.xml

The key rules: allow AI crawlers to see everything public. Block only genuinely private pages (admin panels, customer portals). Always include your sitemap URL so crawlers can find all your pages efficiently.

Common mistakes to avoid

• Blocking all bots with “User-agent: * / Disallow: /”. This makes your entire site invisible to everything — AI engines, Google, all of it.
• Security plugins that auto-block AI bots. Some WordPress security plugins (like Wordfence or Sucuri) have settings that block AI crawlers. Check your plugin settings, not just robots.txt directly.
• Forgetting to include a sitemap URL. AI crawlers use your sitemap to discover all your pages. Without it, they might only find your homepage.
• Blocking image and asset directories. Some robots.txt files block /wp-content/uploads/ or /images/. This prevents AI from understanding your visual content.

How to fix it on popular platforms

• WordPress: Install the Yoast SEO plugin. Go to SEO → Tools → File Editor. Edit your robots.txt to allow AI crawlers.
• Squarespace: Go to Settings → SEO → Advanced → robots.txt. Add allow rules for AI crawlers.
• Wix: Go to Settings → SEO → robots.txt Editor. Add the allow rules above.
• Shopify: Edit your theme’s robots.txt.liquid file to include AI crawler rules.

After the fix

Once you’ve updated your robots.txt, changes take effect immediately for new crawls. Perplexity, which searches the live web, will pick up the change within days. ChatGPT and Claude incorporate new data during model training updates, which happen periodically.

The fastest way to verify your fix: run an AI visibility scan. If you were previously invisible and your robots.txt was the problem, you should see improvement within a few weeks as AI engines re-crawl your site.