X-Robots-Tag Generator – Noindex PDFs, Files & Directories

Nginx: add_header in a nested block silently drops every inherited header. Re-declare them.
Apache: needs mod_headers ; without it the directive is a no-op.
CDNs: a cached response keeps its old headers until you purge.
robots.txt: blocking the URL prevents the header from ever being read — the same trap as meta robots.
Staging leaks: a host-wide noindex that ships to production is a catastrophic and very common outage.

What are you targeting?

File extensions

Directory path

Applies to everything under this prefix, including nested folders.

Exact path

Applies to

Prefixing the value with a bot name scopes the rule to that crawler only.

unavailable_after

Server / platform