SEOツール

無料 AIクローラー robots.txt ビルダー

AIクローラー向けのrobots.txtルールをオープン、選択型、厳格のプリセットから作成します。

ツールを読み込み中...

AIクローラー robots.txt ビルダーとは

A robots.txt file tells crawlers which parts of your site they may access. With the rise of AI training crawlers from OpenAI, Anthropic, Google, Perplexity, and Common Crawl, publishers now use robots.txt to signal whether their content may be used for AI training. This tool builds AI-specific robots.txt blocks with clear categories for training bots versus search/referral bots.

クイックアンサー

Build robots.txt rules to control which AI crawlers access your site. Use a selective policy to block training bots (GPTBot, ClaudeBot, Google-Extended) while allowing search engines (Googlebot, Bingbot). Place specific crawler blocks above general wildcard rules in robots.txt.

制限事項

robots.txt is a voluntary standard — not all crawlers respect it. Some AI data collection happens through means other than web crawling, and robots.txt has no enforcement mechanism.
Some CDNs and WAFs (including Cloudflare Bot Management) can override robots.txt with their own bot-blocking rules. Check your CDN configuration after deploying robots.txt changes.
New AI crawlers appear regularly. This tool includes crawlers known as of early 2026. Check for new crawler names periodically and update your robots.txt accordingly.

使い方

Choose a policy preset: Open (block nothing), Selective (block training bots, allow search engines), or Strict (block all AI crawlers).
Customize individual crawler blocks by checking or unchecking specific bots.
Copy the generated robots.txt blocks and add them to your site's robots.txt file, above any general wildcard rules.

主な用途

Block AI training crawlers from OpenAI, Anthropic, and others while keeping Google and Bing search indexing active.
Create a strict policy that blocks all known AI crawlers from accessing any content.
Add explanatory comments to robots.txt so other developers understand the policy decisions.

用途

使用例

例

Blog with selective blocking

A blog wants to appear in Google and Bing search results but does not want its content used for AI training. Use the Selective preset to allow all bots by default but specifically block GPTBot, ClaudeBot, Google-Extended, and PerplexityBot.

例

Documentation site open to all

An open-source documentation site wants maximum discoverability including AI search. Use the Open preset which adds no AI crawler blocks — all bots including training crawlers have full access.

よくあるミス

Placing AI crawler blocks after a wildcard User-agent: * Allow: / rule — the wildcard rule overrides the specific blocks. Always put specific crawler rules above general rules.
Assuming robots.txt blocks prevent all AI data collection — not all crawlers respect robots.txt, and some content may be collected through other means.
Blocking Google-Extended but expecting normal Google search indexing to work — Google-Extended is a separate crawler for AI training, not the same as Googlebot.

検証

Check your live robots.txt at https://yoursite.com/robots.txt and verify the AI crawler blocks appear above any wildcard User-agent rules.
Use Google Search Console robots.txt Tester to confirm the file is valid and Google can read it correctly.

FAQ

AIクローラー robots.txt ビルダーのFAQ

Does blocking GPTBot in robots.txt prevent OpenAI from using my content?

OpenAI states that GPTBot respects robots.txt directives. Blocking GPTBot should prevent OpenAI from crawling your site for training data going forward. However, this does not remove content that was already included in training data before the block was added, and there is no independent enforcement mechanism beyond OpenAI's stated policy.

What is the difference between Googlebot and Google-Extended?

Googlebot is the main search indexing crawler — blocking it removes your site from Google Search entirely. Google-Extended is a separate crawler used for AI and machine learning model training. You can safely block Google-Extended without affecting your Google Search rankings or indexing.

Should I block Common Crawl (CCBot)?

Common Crawl provides free, open web crawl data used by many AI training datasets, academic research, and web archives. Blocking CCBot prevents your content from appearing in future Common Crawl datasets, which may reduce your content's presence in some open-source AI training corpora. If you want your content in AI training data, allow CCBot. If you do not, block it.