SEOツール

無料 X-Robots-Tag ヘッダージェネレーター

PDFや画像などの非HTMLファイル向けのX-Robots-Tag HTTPヘッダーとロボットメタタグを生成します。

ツールを読み込み中...

X-Robots-Tag ヘッダージェネレーターとは

X-Robots-Tag is an HTTP response header that tells search engines how to handle a resource in their index. It is the HTTP equivalent of the robots meta tag but works on any file type — PDFs, images, videos, feeds, JSON responses, and HTML pages alike. The header accepts the same directives as the meta tag: noindex, nofollow, noarchive, nosnippet, noimageindex, notranslate, and others.

クイックアンサー

X-Robots-Tag is an HTTP header that controls search engine indexing for any file type — PDFs, images, feeds, and HTML. Use it when you need indexing control over non-HTML resources or when you cannot add a meta tag to the page. It accepts the same directives as the robots meta tag.

制限事項

Some bots and crawlers ignore X-Robots-Tag entirely. It is a hint, not a binding directive, and there is no enforcement mechanism beyond voluntary compliance by search engines.
If a page is blocked by robots.txt, crawlers cannot see the X-Robots-Tag header at all. Use robots.txt and X-Robots-Tag as complementary layers, not alternatives.
X-Robots-Tag is not an official IETF standard — it is a de-facto convention supported by Google, Bing, and most major search engines. Smaller or niche crawlers may not recognize it.

使い方

Select the directives you want to apply — common combinations are noindex, nofollow for pages to be excluded, and noarchive for content that should not be cached by search engines.
Optionally specify a user-agent to target specific crawlers, or leave it empty for all bots.
Copy the HTTP header and/or the HTML meta tag into your server configuration or page markup.

主な用途

Noindex PDFs and other downloadable files that should not appear in search results.
Prevent search engines from caching sensitive content with noarchive.
Add noimageindex to a gallery page to prevent individual image indexing while keeping the page indexed.

用途

使用例

例

Prevent PDF indexing

A site offers PDF whitepapers behind a form. Add X-Robots-Tag: noindex, nofollow to the PDF responses so search engines do not index the gated content and do not follow links within the PDFs.

例

Protect image assets

A stock photo site adds X-Robots-Tag: noimageindex to all image responses. The HTML pages remain indexed, but the individual images do not appear in Google Image Search, protecting the unwatermarked files.

よくあるミス

Setting a robots meta tag on a page but also sending a conflicting X-Robots-Tag header — search engines may honor the more restrictive directive.
Using noindex on a page that is blocked by robots.txt — search engines cannot see the noindex directive if they cannot crawl the page.
Forgetting that X-Robots-Tag applies to the entire HTTP response — all embedded resources in that response inherit the header behavior.

検証

Use Google Search Console URL Inspection tool to confirm the status of indexed pages after applying noindex via X-Robots-Tag.
Run curl -I https://yoursite.com/path to verify the X-Robots-Tag header is present and contains the correct directives.

FAQ

X-Robots-Tag ヘッダージェネレーターのFAQ

What is the difference between robots.txt Disallow and X-Robots-Tag noindex?

robots.txt Disallow prevents crawling — the search engine cannot access the resource at all. X-Robots-Tag noindex allows crawling but tells the search engine not to include the resource in its index. To remove a page from search results, use noindex and do NOT block it with robots.txt, so the crawler can read the noindex directive.

Does X-Robots-Tag work on non-HTML files?

Yes, and this is its main advantage over the meta tag. X-Robots-Tag works on PDFs, images, videos, Word documents, RSS feeds, JSON responses, and any other HTTP response. The HTML meta tag only works inside HTML pages.

Can I use X-Robots-Tag for specific user-agents only?

Yes. Most web servers support conditional headers based on user-agent. For example, in Nginx: if the user-agent matches Googlebot, add an X-Robots-Tag header. However, not all bots respect user-agent-specific directives, and the safest approach is to apply directives broadly.