Choose the crawler you want to control, toggle the directives that match your goal, and copy the exact tag or HTTP header. No manual syntax, no guesswork.
<!-- No directives selected -> the page defaults to index, follow -->
Use this in your server config to control PDFs, images, videos, and other non-HTML resources.
<!-- No directives selected -> defaults apply -->
Add the X-Robots-Tag to your server config for non-HTML files.
<Files ~ "\.pdf$">
Header set X-Robots-Tag "noindex, nofollow"
</Files>
Search engines assume a page is indexable until you say otherwise. That default sounds harmless, but it means every thin tag page, expired offer, or internal search result you leave alone can end up in the index. Over time, low value pages compete with your real content for crawl attention and dilute the signals that help your best pages rank.
The robots meta tag is how you correct the default. One line in the head of a page tells Google whether to index it, follow its links, or show a snippet. The same logic travels through an HTTP response header for files that have no head section, like PDFs, images, and videos. Publishers who control this layer tend to see cleaner index coverage and more predictable rankings.
There is a behavioural angle worth noting. Users judge a site by what shows up in the results. When a stale or broken page appears above your strongest work, trust drops even if the main content is excellent. Index control is not just technical housekeeping. It is how you decide which pages represent your brand in front of potential customers.
Start with the basic cards. Use noindex when a page should stay visible to visitors but out of search results. Pair it with nofollow when you also want crawlers to ignore the links on it. Use nofollow on its own for pages you want indexed but where link equity should not pass, such as user generated content areas.
The preview cards handle how your page appears. Nosnippet stops search engines from showing a text summary, which matters for pages where the headline should speak for itself. Max snippet sets a character limit when you want some context but not the full text. Image and video preview controls give the same level of choice for visual content.
When the set is complete, generate the tag and drop it into the head section of the page. For files without HTML, apply the HTTP header through your server config. Finish by verifying the page with a robots meta checker so you can confirm crawlers read the directives exactly as intended.