📺 JioHotstar Smart TV AdsGrow beyond search — AI SEO · GEO · Smart TV⚡ Free 15-page audit in 48hEvery enquiry gets one — no strings📺 JioHotstar Smart TV AdsGrow beyond search — AI SEO · GEO · Smart TV⚡ Free 15-page audit in 48hEvery enquiry gets one — no strings

What Is X-Robots-Tag? A Guide for UAE Websites

Harsh Rajput · Sr. SEO Executive · 3 years' experience · 16 November 2026 · 11 min read

Key takeaways

  • The X-Robots-Tag is a rule sent in the HTTP response header of a URL.
  • When a crawler asks for a file your server replies with the file and a set of headers.
  • These three tools sound alike but do different jobs.
  • The X-Robots-Tag uses the same directives as the meta robots tag.

A real estate firm in Dubai uploads a price list as a PDF for its agents. Two weeks later the file shows up in Google and a rival quotes it to a buyer. The firm never meant it to be public. The file just sat on the server with no rule to keep it out of search.

Your website holds more than web pages. It holds PDFs, images, videos and downloads. Some should show up in Google. Others should stay hidden. A normal meta tag cannot help because a PDF has no HTML head to put it in. That is where the X-Robots-Tag comes in.

This guide explains it in plain words. You will see the directives and server examples you can copy. You will also learn how it works with AI search and which mistakes to avoid.

What Is X-Robots-Tag?

The X-Robots-Tag is a rule sent in the HTTP response header of a URL. It tells search engines whether to index the file and how to show it in results. It works for any file type and not just HTML pages.

A UAE site owner who asks what is X-Robots-Tag usually found a PDF or an old page in Google that should not be there. Here is the simplest example.

X-Robots-Tag: noindex

The server adds this line when it sends the file. Google reads it before it looks at anything else. The result matches a meta robots tag but it works everywhere.

X-Robots-Tag Key Facts

  • Location: the HTTP response header sent by your server

  • Works on: HTML pages, PDFs, images, videos and other files

  • Directives: the same ones the meta robots tag uses

  • Set up in: server config, CDN or application code

  • Needs crawling: Google must be able to open the URL to see the header

  • Default: index and follow when no header exists

How Does the X-Robots-Tag Work?

When a crawler asks for a file your server replies with the file and a set of headers. The X-Robots-Tag sits inside those headers. The crawler reads it and follows the rules.

Nothing changes on the page itself. A visitor sees the same content. Only the crawler acts on the header. That makes it a clean way to control many files at once.

What Happens Step by Step

  1. Googlebot requests a URL such as a PDF

  2. Your server replies with the file and the header

  3. Googlebot reads the X-Robots-Tag line

  4. It applies the rules to that URL

  5. The file is kept out of or limited in search results after the next crawl

X-Robots-Tag vs Meta Robots Tag vs Robots.txt

These three tools sound alike but do different jobs. Mixing them up is the top cause of indexing trouble.

Robots.txt decides whether bots may crawl a URL. The meta robots tag controls indexing for one HTML page. The X-Robots-Tag does the same job from the server and covers every file type.

Tool

Where it lives

Best for

Robots.txt

One file at the site root

Blocking crawl of whole folders

Meta robots tag

Head of an HTML page

Single web pages

X-Robots-Tag

HTTP response header

PDFs, images and bulk rules

When to Pick the X-Robots-Tag

  1. You need to noindex a PDF or a Word file

  2. You want to control images or videos

  3. You want one rule to cover many URLs at once

  4. You cannot edit the HTML of a page

  5. You want to noindex a whole staging site from the server

A quick note on robots.txt. If a file is blocked there Google never opens it and never sees the header. Pick one tool per job and do not stack them on the same URL.

X-Robots-Tag Directives Explained

The X-Robots-Tag uses the same directives as the meta robots tag. You can combine several in one header with commas.

Main Directives You Should Know

  1. index: allows the file to appear in results and it is the default

  2. noindex: keeps the file out of search results

  3. follow: lets crawlers follow links inside the file and it is the default

  4. nofollow: asks crawlers not to follow links inside the file

  5. none: the same as noindex and nofollow together

  6. all: the same as index and follow together

  7. nosnippet: blocks text snippets and video previews in results

  8. max-snippet:[number]: limits the snippet to a set number of characters

  9. max-image-preview:[none or standard or large]: sets the largest image preview allowed

  10. max-video-preview:[number]: limits video previews to a set number of seconds

  11. noimageindex: stops images on the page from being indexed

  12. notranslate: stops Google from offering a translation

  13. unavailable_after:[date]: removes the file from results after a set date

  14. indexifembedded: lets a noindex file be indexed when it is embedded in another page

If two rules clash Google follows the strictest one. Google no longer uses noarchive since it removed cached pages from results.

Target One Crawler

You can send a rule to one bot by putting its name before the directive.

X-Robots-Tag: googlebot: noindex

X-Robots-Tag: bingbot: nofollow

A header with no bot name applies to every crawler.

X-Robots-Tag Examples You Can Copy

Pick the setup that matches your server. Change the file types and paths to fit your site before you use them.

Apache (.htaccess)

Noindex all PDF files. This needs the mod_headers module.

<FilesMatch "\.pdf$">

  Header set X-Robots-Tag "noindex, nofollow"

</FilesMatch>

Noindex several file types at once:

<FilesMatch "\.(doc|docx|xls|xlsx|zip)$">

  Header set X-Robots-Tag "noindex"

</FilesMatch>

Nginx

Noindex all PDF files:

location ~* \.pdf$ {

  add_header X-Robots-Tag "noindex, nofollow";

}

Next.js

Add a header to a whole folder in next.config.js:

module.exports = {

  async headers() {

    return [

      {

        source: "/private-files/:path*",

        headers: [{ key: "X-Robots-Tag", value: "noindex" }],

      },

    ];

  },

};

Other Ways to Set the Header

  1. IIS: add a custom header in the web.config file

  2. Cloudflare: use a Transform Rule or a Worker to add the header

  3. PHP: call header("X-Robots-Tag: noindex"); before any output

  4. WordPress: use an SEO plugin or a code snippet since it does not handle PDFs by default

Common Uses of the X-Robots-Tag for UAE Sites

The header shines wherever a meta tag cannot reach. UAE sites lean on PDFs more than most. Property brochures, floor plans, clinic forms, tender documents and price lists all live as files.

Best Use Cases

  1. Keep private PDFs like invoices and internal price lists out of Google

  2. Stop old property brochures and expired offers from ranking

  3. Noindex one language copy of a PDF when you publish the same file in Arabic, English and want only one version in results

  4. Noindex image files you do not want in image search

  5. Hide staging and test sites from search

  6. Add an expiry date to a Ramadan or Dubai Shopping Festival offer with unavailable_after

  7. Limit snippet length on files you want shown but not fully quoted

A private file still needs real protection. The header only asks search engines to skip it. Anyone with the link can open it. Use a login for anything sensitive such as customer documents or trade licence copies. The UAE data protection law applies to personal data in those files so a hidden link is not enough.

X-Robots-Tag for AI Overviews, GEO and LLM Platforms

AI answers now sit at the top of many results. Google AI Overviews, AI Mode use pages, files that are indexed and can show a snippet. So the same header rules apply.

If you add noindex to a PDF it leaves the index and cannot be cited. If you use nosnippet or a small max-snippet value Google can still index the file but shows less text from it. If you want a helpful guide PDF to be quoted in AI answers leave it open.

Other AI platforms work differently. Bots like OAI-SearchBot and PerplexityBot mainly follow robots.txt rules. Support for the X-Robots-Tag can vary by company so check each platform's documentation. Teams that offer technical SEO services in UAE usually start by deciding which files should be cited and which should stay private.

How to Stay Visible in AI Answers

  1. Keep useful guides and whitepapers free of noindex

  2. Avoid nosnippet on files you want cited

  3. Use max-snippet:-1 to allow long text previews

  4. Add a clear title and summary inside each PDF

  5. Link to your key files from normal HTML pages

  6. Keep robots.txt open for the search bots you want

How to Check Your X-Robots-Tag

Always test the header after you add it. A rule set on the wrong path can hide files you want ranked.

The quickest way is a command line check. This asks the server for headers only.

curl -I https://www.example.com/files/guide.pdf

Look for the X-Robots-Tag line in the reply. You can also open your browser developer tools and check the Network tab. In Google Search Console the URL Inspection tool shows whether a file is indexed and whether noindex blocks it.

A full scan helps too. A free SEO audit tool can flag pages and files that carry a noindex rule by mistake.

Checking Steps

  1. Run curl -I on the file URL

  2. Find the X-Robots-Tag line in the reply

  3. Inspect the URL in Search Console

  4. Confirm important files show as indexed

  5. Check that the rule hits only the paths you meant

  6. Recheck after every server or CDN change

Common X-Robots-Tag Mistakes

Most problems come from a rule that reaches too far. A single line in a server file can affect thousands of URLs.

Mistakes That Hurt Your Rankings

  1. Setting noindex for the whole site by accident

  2. Blocking the file in robots.txt so Google never sees the header

  3. Using a pattern that also matches HTML pages

  4. Leaving a staging noindex rule live after launch

  5. Forgetting that the CDN may add or remove headers

  6. Setting different rules in the header and the meta tag

  7. Adding nosnippet to files you want cited in AI answers

  8. Trusting the header to protect private files

X-Robots-Tag Best Practices

Keep your rules narrow and simple. A small rule set is easier to test and easier to fix.

Best Practices Checklist

  1. Use the meta robots tag for HTML pages and the header for other files

  2. Target file types or folders and never the whole site

  3. Let Google crawl any URL that carries a noindex header

  4. Keep noindex files out of your XML sitemap

  5. Protect private files with a login

  6. Test with curl -I after each change

  7. Keep a written list of every header rule on your server

  8. Review the rules after a redesign or migration

Server rules can break quietly after updates. Regular website maintenance and security checks catch this kind of drift before it costs you traffic.

Conclusion

The X-Robots-Tag gives you page level control over files that have no HTML. It uses the same directives as the meta robots tag and works from the server. That makes it the right tool for PDFs, images and bulk rules.

Start by listing the files that should stay out of Google. Add noindex only to those. Keep every helpful file open so search engines and AI tools can quote it. Then test each rule with a header check and revisit it after every big site change.

Frequently Asked Questions

1. What is X-Robots-Tag in simple words?

The X-Robots-Tag is a rule your server sends with a file. It tells search engines whether to index the file and how to show it in results.

2. What is the difference between X-Robots-Tag and the meta robots tag?

The meta robots tag sits in the head of an HTML page. The X-Robots-Tag sits in the HTTP header so it also works for PDFs and images.

3. Can I use X-Robots-Tag on PDF files?

Yes. This is its most common use. Add a rule for .pdf files on Apache or Nginx and Google will drop them from results after the next crawl.

4. Does X-Robots-Tag work on images?

Yes. You can set noindex on image files with a server rule so they stay out of Google Images.

5. Do I need robots.txt if I use X-Robots-Tag?

They do different jobs. Robots.txt controls crawling and the header controls indexing. Do not block a URL in robots.txt if you want Google to read its header.

6. Can I use both X-Robots-Tag and a meta robots tag?

Yes. If the rules clash Google follows the strictest one. Keep them consistent to avoid confusion.

7. How do I check if a file has an X-Robots-Tag?

Run curl -I with the file URL and look for the header in the reply. You can also use browser developer tools or the URL Inspection tool in Search Console.

8. Does the X-Robots-Tag affect AI Overviews?

Yes. A noindex header removes the file from the index so it cannot be cited. Nosnippet and max-snippet limit how much text Google can show.

9. Does the X-Robots-Tag keep files private?

No. It only asks search engines to skip the file. Anyone with the link can still open it. Use a password or login for private content.

10. How long does it take for X-Robots-Tag changes to work?

Changes apply after Google crawls the URL again. That can take a few days or a few weeks.

#what is X-Robots-Tag#X-Robots-Tag#X-Robots-Tag noindex#X-Robots-Tag directives#X-Robots-Tag example#noindex PDF#X-Robots-Tag vs meta robots#HTTP header SEO#X-Robots-Tag Apache#X-Robots-Tag Nginx#technical SEO services in UAE#Google indexing#AI search visibility

About the author

Harsh Rajput

Sr. SEO Executive · 3 years' experience

Harsh Rajput is a Senior SEO Executive with 3+ years of experience in SEO, digital marketing and AEO/GEO strategy. He leads a team of SEO executives at Digisutra Solutions, handling keyword research, technical SEO, on-page/off-page optimization, link building and content strategy, while helping brands rank in Google AI Overviews and LLM platforms like ChatGPT, Claude and Gemini. He has worked with clients across India, USA, UAE, and Australia in industries like e-commerce, finance and technology.

Reader reviews

Leave a review

optional

spam-guarded · reviews appear after approval

All articles