What Is X-Robots-Tag? A Guide for UAE Websites

Key takeaways
- The X-Robots-Tag is a rule sent in the HTTP response header of a URL.
- When a crawler asks for a file your server replies with the file and a set of headers.
- These three tools sound alike but do different jobs.
- The X-Robots-Tag uses the same directives as the meta robots tag.
A real estate firm in Dubai uploads a price list as a PDF for its agents. Two weeks later the file shows up in Google and a rival quotes it to a buyer. The firm never meant it to be public. The file just sat on the server with no rule to keep it out of search.
Your website holds more than web pages. It holds PDFs, images, videos and downloads. Some should show up in Google. Others should stay hidden. A normal meta tag cannot help because a PDF has no HTML head to put it in. That is where the X-Robots-Tag comes in.
This guide explains it in plain words. You will see the directives and server examples you can copy. You will also learn how it works with AI search and which mistakes to avoid.
What Is X-Robots-Tag?
The X-Robots-Tag is a rule sent in the HTTP response header of a URL. It tells search engines whether to index the file and how to show it in results. It works for any file type and not just HTML pages.
A UAE site owner who asks what is X-Robots-Tag usually found a PDF or an old page in Google that should not be there. Here is the simplest example.
X-Robots-Tag: noindex
The server adds this line when it sends the file. Google reads it before it looks at anything else. The result matches a meta robots tag but it works everywhere.
X-Robots-Tag Key Facts
Location: the HTTP response header sent by your server
Works on: HTML pages, PDFs, images, videos and other files
Directives: the same ones the meta robots tag uses
Set up in: server config, CDN or application code
Needs crawling: Google must be able to open the URL to see the header
Default: index and follow when no header exists
How Does the X-Robots-Tag Work?
When a crawler asks for a file your server replies with the file and a set of headers. The X-Robots-Tag sits inside those headers. The crawler reads it and follows the rules.
Nothing changes on the page itself. A visitor sees the same content. Only the crawler acts on the header. That makes it a clean way to control many files at once.
What Happens Step by Step
Googlebot requests a URL such as a PDF
Your server replies with the file and the header
Googlebot reads the X-Robots-Tag line
It applies the rules to that URL
The file is kept out of or limited in search results after the next crawl
X-Robots-Tag vs Meta Robots Tag vs Robots.txt
These three tools sound alike but do different jobs. Mixing them up is the top cause of indexing trouble.
Robots.txt decides whether bots may crawl a URL. The meta robots tag controls indexing for one HTML page. The X-Robots-Tag does the same job from the server and covers every file type.
Tool | Where it lives | Best for |
Robots.txt | One file at the site root | Blocking crawl of whole folders |
Meta robots tag | Head of an HTML page | Single web pages |
X-Robots-Tag | HTTP response header | PDFs, images and bulk rules |
When to Pick the X-Robots-Tag
You need to noindex a PDF or a Word file
You want to control images or videos
You want one rule to cover many URLs at once
You cannot edit the HTML of a page
You want to noindex a whole staging site from the server
A quick note on robots.txt. If a file is blocked there Google never opens it and never sees the header. Pick one tool per job and do not stack them on the same URL.
X-Robots-Tag Directives Explained
The X-Robots-Tag uses the same directives as the meta robots tag. You can combine several in one header with commas.
Main Directives You Should Know
index: allows the file to appear in results and it is the default
noindex: keeps the file out of search results
follow: lets crawlers follow links inside the file and it is the default
nofollow: asks crawlers not to follow links inside the file
none: the same as noindex and nofollow together
all: the same as index and follow together
nosnippet: blocks text snippets and video previews in results
max-snippet:[number]: limits the snippet to a set number of characters
max-image-preview:[none or standard or large]: sets the largest image preview allowed
max-video-preview:[number]: limits video previews to a set number of seconds
noimageindex: stops images on the page from being indexed
notranslate: stops Google from offering a translation
unavailable_after:[date]: removes the file from results after a set date
indexifembedded: lets a noindex file be indexed when it is embedded in another page
If two rules clash Google follows the strictest one. Google no longer uses noarchive since it removed cached pages from results.
Target One Crawler
You can send a rule to one bot by putting its name before the directive.
X-Robots-Tag: googlebot: noindex
X-Robots-Tag: bingbot: nofollow
A header with no bot name applies to every crawler.
X-Robots-Tag Examples You Can Copy
Pick the setup that matches your server. Change the file types and paths to fit your site before you use them.
Apache (.htaccess)
Noindex all PDF files. This needs the mod_headers module.
<FilesMatch "\.pdf$">
Header set X-Robots-Tag "noindex, nofollow"
</FilesMatch>
Noindex several file types at once:
<FilesMatch "\.(doc|docx|xls|xlsx|zip)$">
Header set X-Robots-Tag "noindex"
</FilesMatch>
Nginx
Noindex all PDF files:
location ~* \.pdf$ {
add_header X-Robots-Tag "noindex, nofollow";
}
Next.js
Add a header to a whole folder in next.config.js:
module.exports = {
async headers() {
return [
{
source: "/private-files/:path*",
headers: [{ key: "X-Robots-Tag", value: "noindex" }],
},
];
},
};
Other Ways to Set the Header
IIS: add a custom header in the web.config file
Cloudflare: use a Transform Rule or a Worker to add the header
PHP: call header("X-Robots-Tag: noindex"); before any output
WordPress: use an SEO plugin or a code snippet since it does not handle PDFs by default
Common Uses of the X-Robots-Tag for UAE Sites
The header shines wherever a meta tag cannot reach. UAE sites lean on PDFs more than most. Property brochures, floor plans, clinic forms, tender documents and price lists all live as files.
Best Use Cases
Keep private PDFs like invoices and internal price lists out of Google
Stop old property brochures and expired offers from ranking
Noindex one language copy of a PDF when you publish the same file in Arabic, English and want only one version in results
Noindex image files you do not want in image search
Hide staging and test sites from search
Add an expiry date to a Ramadan or Dubai Shopping Festival offer with unavailable_after
Limit snippet length on files you want shown but not fully quoted
A private file still needs real protection. The header only asks search engines to skip it. Anyone with the link can open it. Use a login for anything sensitive such as customer documents or trade licence copies. The UAE data protection law applies to personal data in those files so a hidden link is not enough.
X-Robots-Tag for AI Overviews, GEO and LLM Platforms
AI answers now sit at the top of many results. Google AI Overviews, AI Mode use pages, files that are indexed and can show a snippet. So the same header rules apply.
If you add noindex to a PDF it leaves the index and cannot be cited. If you use nosnippet or a small max-snippet value Google can still index the file but shows less text from it. If you want a helpful guide PDF to be quoted in AI answers leave it open.
Other AI platforms work differently. Bots like OAI-SearchBot and PerplexityBot mainly follow robots.txt rules. Support for the X-Robots-Tag can vary by company so check each platform's documentation. Teams that offer technical SEO services in UAE usually start by deciding which files should be cited and which should stay private.
How to Stay Visible in AI Answers
Keep useful guides and whitepapers free of noindex
Avoid nosnippet on files you want cited
Use max-snippet:-1 to allow long text previews
Add a clear title and summary inside each PDF
Link to your key files from normal HTML pages
Keep robots.txt open for the search bots you want
How to Check Your X-Robots-Tag
Always test the header after you add it. A rule set on the wrong path can hide files you want ranked.
The quickest way is a command line check. This asks the server for headers only.
curl -I https://www.example.com/files/guide.pdf
Look for the X-Robots-Tag line in the reply. You can also open your browser developer tools and check the Network tab. In Google Search Console the URL Inspection tool shows whether a file is indexed and whether noindex blocks it.
A full scan helps too. A free SEO audit tool can flag pages and files that carry a noindex rule by mistake.
Checking Steps
Run curl -I on the file URL
Find the X-Robots-Tag line in the reply
Inspect the URL in Search Console
Confirm important files show as indexed
Check that the rule hits only the paths you meant
Recheck after every server or CDN change
Common X-Robots-Tag Mistakes
Most problems come from a rule that reaches too far. A single line in a server file can affect thousands of URLs.
Mistakes That Hurt Your Rankings
Setting noindex for the whole site by accident
Blocking the file in robots.txt so Google never sees the header
Using a pattern that also matches HTML pages
Leaving a staging noindex rule live after launch
Forgetting that the CDN may add or remove headers
Setting different rules in the header and the meta tag
Adding nosnippet to files you want cited in AI answers
Trusting the header to protect private files
X-Robots-Tag Best Practices
Keep your rules narrow and simple. A small rule set is easier to test and easier to fix.
Best Practices Checklist
Use the meta robots tag for HTML pages and the header for other files
Target file types or folders and never the whole site
Let Google crawl any URL that carries a noindex header
Keep noindex files out of your XML sitemap
Protect private files with a login
Test with curl -I after each change
Keep a written list of every header rule on your server
Review the rules after a redesign or migration
Server rules can break quietly after updates. Regular website maintenance and security checks catch this kind of drift before it costs you traffic.
Conclusion
The X-Robots-Tag gives you page level control over files that have no HTML. It uses the same directives as the meta robots tag and works from the server. That makes it the right tool for PDFs, images and bulk rules.
Start by listing the files that should stay out of Google. Add noindex only to those. Keep every helpful file open so search engines and AI tools can quote it. Then test each rule with a header check and revisit it after every big site change.
Frequently Asked Questions
1. What is X-Robots-Tag in simple words?
The X-Robots-Tag is a rule your server sends with a file. It tells search engines whether to index the file and how to show it in results.
2. What is the difference between X-Robots-Tag and the meta robots tag?
The meta robots tag sits in the head of an HTML page. The X-Robots-Tag sits in the HTTP header so it also works for PDFs and images.
3. Can I use X-Robots-Tag on PDF files?
Yes. This is its most common use. Add a rule for .pdf files on Apache or Nginx and Google will drop them from results after the next crawl.
4. Does X-Robots-Tag work on images?
Yes. You can set noindex on image files with a server rule so they stay out of Google Images.
5. Do I need robots.txt if I use X-Robots-Tag?
They do different jobs. Robots.txt controls crawling and the header controls indexing. Do not block a URL in robots.txt if you want Google to read its header.
6. Can I use both X-Robots-Tag and a meta robots tag?
Yes. If the rules clash Google follows the strictest one. Keep them consistent to avoid confusion.
7. How do I check if a file has an X-Robots-Tag?
Run curl -I with the file URL and look for the header in the reply. You can also use browser developer tools or the URL Inspection tool in Search Console.
8. Does the X-Robots-Tag affect AI Overviews?
Yes. A noindex header removes the file from the index so it cannot be cited. Nosnippet and max-snippet limit how much text Google can show.
9. Does the X-Robots-Tag keep files private?
No. It only asks search engines to skip the file. Anyone with the link can still open it. Use a password or login for private content.
10. How long does it take for X-Robots-Tag changes to work?
Changes apply after Google crawls the URL again. That can take a few days or a few weeks.
About the author

Sr. SEO Executive · 3 years' experience
Harsh Rajput is a Senior SEO Executive with 3+ years of experience in SEO, digital marketing and AEO/GEO strategy. He leads a team of SEO executives at Digisutra Solutions, handling keyword research, technical SEO, on-page/off-page optimization, link building and content strategy, while helping brands rank in Google AI Overviews and LLM platforms like ChatGPT, Claude and Gemini. He has worked with clients across India, USA, UAE, and Australia in industries like e-commerce, finance and technology.
Reader reviews

Up next · SEO & AI search · 13 min
Core Web Vitals SEO Services in UAE to Boost Speed

Related · SEO & AI search · 21 min
Schema Markup SEO Services in UAE: Types and Cost
