X-Robots-Tag HTTP Documentation

In modern SEO, controlling how search engines crawl and index your website is essential—not only for your HTML pages but for every resource your site serves. While the <meta name="robots"> tag offers control within HTML, it falls short when managing non-HTML files like PDFs, images, videos, or other media. That’s where the X-Robots-Tag HTTP header steps in as a vital tool for webmasters.

The X-Robots-Tag HTTP header allows webmasters to manage search engine crawling and indexing for all types of web resources, including PDFs, images, and videos.

This comprehensive guide covers everything you need to know about the X-Robots-Tag HTTP header — what it is, why it’s crucial for SEO, how to implement it effectively, and the most important directives to tailor your site’s visibility and indexing behavior.

---

What Is the X-Robots-Tag HTTP Header?

The X-Robots-Tag is an HTTP response header used to send indexing and crawling instructions to search engine bots. Unlike the HTML <meta name="robots"> tag, which only applies to HTML pages, the X-Robots-Tag works at the server level and can be applied to any file type or HTTP response, including PDFs, images, videos, and other non-HTML assets.

Why does this matter? Because controlling how search engines treat all your site’s resources helps protect sensitive content, manage crawl budget, and optimize your SEO strategy beyond standard web pages.

---

Why Use X-Robots-Tag? Key SEO Benefits

• Control Indexing of Non-HTML Files: Prevent unwanted PDFs, images, or media files from appearing in search results.
• Server-Level Consistency: Apply uniform SEO rules across your entire website or specific file types.
• Advanced Directives: Utilize powerful instructions not available in meta tags, such as time-based content expiration.
• Crawl Budget Optimization: Limit bots from wasting resources crawling irrelevant or sensitive files.

---

Essential X-Robots-Tag Directives Explained

Here’s a detailed look at the most effective X-Robots-Tag directives and their SEO impact:

1. all

• Meaning: Default directive; allows indexing, crawling, link following, caching, and displaying snippets.
• When to use: Standard behavior for pages and resources.

Example:

X-Robots-Tag: all

---

2. noindex

• Meaning: Prevents the resource from being indexed in search results.
• Use case: Sensitive documents, duplicate content, or temporary files.

Example:

X-Robots-Tag: noindex

---

3. nofollow

• Meaning: Instructs bots not to follow any links on the resource, stopping link equity flow.
• Use case: Pages with untrusted outbound links or content you don’t want to pass SEO value.

Example:

X-Robots-Tag: nofollow

---

4. none

• Meaning: Combination of noindex and nofollow.
• Use case: Strict exclusion of the resource from search indexing and crawling.

Example:

X-Robots-Tag: none

---

5. noarchive

• Meaning: Prevents cached versions from appearing in search engine results.
• Use case: Frequently updated content or confidential information.

Example:

X-Robots-Tag: noarchive

---

6. nosnippet

• Meaning: Blocks search engines from showing text or video snippets for the resource.
• Use case: Protect content previews or proprietary text.

Example:

X-Robots-Tag: nosnippet

---

7. noodp

• Meaning: Prevents search engines from using Open Directory Project (DMOZ) metadata.
• Use case: Avoid outdated or irrelevant descriptions (mostly legacy).

Example:

X-Robots-Tag: noodp

---

8. notranslate

• Meaning: Disables translation options for the resource in search results.
• Use case: Keep content in its original language for branding or legal reasons.

Example:

X-Robots-Tag: notranslate

---

9. noimageindex

• Meaning: Stops images on the page from being indexed in image search.
• Use case: Protect copyright or prevent irrelevant images from appearing in search.

Example:

X-Robots-Tag: noimageindex

---

10. unavailable_after: [date/time]

• Meaning: Tells search engines to stop indexing or showing the resource after the specified date/time (in HTTP date format).
• Use case: Time-sensitive content like event pages, limited promotions, or expired offers.

Example:

X-Robots-Tag: unavailable_after: Fri, 31 Dec 2025 23:59:59 GMT

---

How to Implement X-Robots-Tag Headers

You can set X-Robots-Tag directives either at the server configuration level or dynamically within application code.

Apache .htaccess Example

<FilesMatch "\.(pdf|docx)$">
    Header set X-Robots-Tag "noindex, nofollow"
</FilesMatch>

Prevents indexing and crawling of PDF and DOCX files.

---

Nginx Configuration Example

location ~* \.(pdf|docx)$ {
    add_header X-Robots-Tag "noindex, nofollow";
}

---

PHP Application Example

header("X-Robots-Tag: noindex, nofollow", true);

---

Quick Reference: X-Robots-Tag Directive Summary

Directive	Effect	Common Use Case
all	Allow indexing, crawling, link following	Default behavior
noindex	Exclude from search index	Sensitive or temporary pages
nofollow	Do not follow links	Block passing SEO value
none	noindex + nofollow combined	Strict exclusion
noarchive	Disable cached results	Frequently changing or confidential pages
nosnippet	Prevent text/video snippets	Protect content previews
noodp	Block DMOZ metadata	Avoid outdated directory descriptions
notranslate	Disable translation	Preserve original language
noimageindex	Prevent image indexing	Protect image copyrights
unavailable_after	Stop indexing after given date/time	Time-limited or expiring content

---

Conclusion: Enhance Your SEO Control With X-Robots-Tag

Using the X-Robots-Tag HTTP header is a critical step for advanced SEO management, especially for sites with diverse content types beyond HTML. It gives you full control over how search engines index, crawl, cache, and display your resources—helping protect sensitive data, improve crawl efficiency, and maintain a polished presence in search results.

For best SEO practices, combine X-Robots-Tag headers with traditional <meta name="robots"> tags, tailoring directives per resource type and your website’s unique goals.