AI Bot & Crawler Opt-OutNext.js 15+ app/robots.tsAEO & SEO Friendly🔒 100% Client-Side

robots.txt & AI Bot Control Generator

Visually configure crawl permissions for Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot and ByteSpider. Download the result instantly as a Next.js 15+ App Router app/robots.ts or a standard robots.txt.

AI & Crawler Policy

robots.txt & AI Bot Control Generator

Visually set the access rules for crawlers such as Googlebot, GPTBot, ClaudeBot and Perplexity.

Crawler (Bot) Access Permissions

GPTBot (OpenAI Model Training)GPTBot

Data crawler OpenAI uses to train its AI models (GPT).

ChatGPT-User (Search & Browsing)ChatGPT-User

Lets ChatGPT users view your website live during a conversation.

ClaudeBot & Anthropic AIClaudeBot

Web scraping and training bot for Anthropic Claude models.

PerplexityBot (AI Search)PerplexityBot

Bot the Perplexity AI search engine uses to build references and answers.

Google-Extended (Gemini Training)Google-Extended

Controls whether Google Gemini AI models use your content for training.

Bytespider (ByteDance AI)Bytespider

AI bot run aggressively by TikTok / ByteDance.

Applebot-Extended (Apple AI)Applebot-Extended

Bot that collects training data for Apple Intelligence models.

CCBot (Common Crawl)CCBot

General dataset crawler used in open-source LLM training.

XML Sitemap URL

Disallowed Paths (comma-separated)

import type { MetadataRoute } from 'next'

// app/robots.ts
export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
    {
      userAgent: '*',
      allow: '/',
      disallow: ['/api/', '/admin/', '/private/'],
    },
    {
      userAgent: 'GPTBot',
      disallow: '/',
    },
    {
      userAgent: 'ClaudeBot',
      disallow: '/',
    },
    {
      userAgent: 'Google-Extended',
      disallow: '/',
    },
    {
      userAgent: 'Bytespider',
      disallow: '/',
    },
    {
      userAgent: 'Applebot-Extended',
      disallow: '/',
    },
    {
      userAgent: 'CCBot',
      disallow: '/',
    }
    ],
    sitemap: 'https://ismailhakkieren.com/sitemap.xml',
  }
}

39 lines of rules💡 Manage AI bots without blocking search engines

Guide to robots.txt and Bot Management in the AI Era

In the past, robots.txt files were only used to guide search engine spiders such as Googlebot, Bingbot and Yandex. Today, AI companies such as OpenAI (GPTBot), Anthropic (ClaudeBot), Perplexity, Google (Google-Extended) and ByteDance (ByteSpider) constantly scrape millions of web pages, both to train large language models (LLMs) and to generate AI search answers. This tool lets you keep your site in Google's index while deciding which AI crawlers may crawl it, in Next.js 15+ app/robots.ts and standard robots.txt format.

1. Popular AI Bots: Model Training vs Live Search

Blocking every AI bot at once can stop AI search engines from citing your site as a source (AEO). What each bot does:

User-Agent	Company / Provider	Purpose	Recommended Policy
Googlebot	Google	Organic search indexing and SERP ranking.	Always Allow
ChatGPT-User	OpenAI	Reads pages on demand and links to them as sources when a user runs a live search in ChatGPT.	Allow (Brings Traffic)
PerplexityBot	Perplexity AI	AI search engine index and direct answer citations.	Allow (Critical for AEO)
GPTBot	OpenAI	General content scraping to train future GPT models.	Optionally Block (Disallow)
ClaudeBot	Anthropic	Training Claude models and building datasets.	Optionally Block (Disallow)
Google-Extended	Google	Training Gemini and Vertex AI models (does not affect Google Search).	Optionally Block (Disallow)
Bytespider / CCBot	ByteDance / Common Crawl	General data scrapers that consume server resources very aggressively.	Block (Eases Server Load)

2. Type-Safe app/robots.ts for the Next.js 15+ App Router

In the Next.js App Router, you can use app/robots.ts instead of a static public/robots.txt file to define dynamic rules based on the environment (development/production):

// app/robots.ts
import type { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  const baseUrl = process.env.NEXT_PUBLIC_SITE_URL || 'https://ismailhakkieren.com'

  return {
    rules: [
      {
        userAgent: '*',
        allow: '/',
        disallow: ['/admin/', '/api/', '/private/'],
      },
      {
        userAgent: ['GPTBot', 'ClaudeBot', 'Google-Extended', 'Bytespider', 'CCBot'],
        disallow: ['/'],
      },
      {
        userAgent: ['ChatGPT-User', 'PerplexityBot'],
        allow: '/',
      },
    ],
    sitemap: `${baseUrl}/sitemap.xml`,
  }
}

3. Standard robots.txt Configuration

A standard robots.txt template for traditional websites, placed in the root directory:

# Allow All Search Engines
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/

# Block Model Training Bots (Opt-Out)
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# Allow Live AI Search Bots
User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

# Sitemap
Sitemap: https://ismailhakkieren.com/sitemap.xml

Frequently Asked Questions (FAQ)

For AI user agents such as GPTBot, ClaudeBot or Bytespider, add a User-agent: GPTBot line followed by a Disallow: / directive.

Yes. You can generate a dynamic robots.txt by returning a MetadataRoute.Robots object from a function in app/robots.ts.

No. GPTBot belongs to OpenAI. Google search rankings are based only on Googlebot crawling and are not affected.

These bots do not train models. They run when a user performs a live AI search, citing your site as a source and sending visitors to it (an AEO advantage).

Absolutely not. GPTBot belongs to OpenAI and ClaudeBot to Anthropic. Your Google search results and organic rankings are determined by Googlebot alone.

No. According to Google's official documentation, Google-Extended only restricts Gemini and Vertex AI model training; it does not stop your site from appearing in standard Google Search or Google Discover.

Yes. robots.txt files are public. Writing Disallow: /top-secret-admin-panel/ points attackers straight at the target. Protect sensitive pages with password protection and a noindex meta tag instead of robots.txt.