A robots meta tag is an HTML element or HTTP header that tells compliant search crawlers how a specific page should be indexed, followed, and shown in search results. For a beginner, the Meta Robots Generator is the safest way to build that tag because it converts a small set of choices into current Google-supported syntax and refuses to combine directives that contradict each other, so you never paste a tag that Google will quietly ignore. Generation runs entirely in your browser with no settings, URLs, or directives sent to a search service, which means the page you are working on stays private while you decide what the tag should say. Both the page-level meta element and the equivalent X-Robots-Tag header come from the same validated set of choices, so the HTML version and the header version cannot drift apart by accident. This beginner walkthrough explains what each directive means, how to choose between them, where the generated tag must be placed, and how to confirm after deployment that the crawler actually sees it.
What a Robots Meta Tag Actually Does
A robots meta tag is a per-page instruction. Unlike robots.txt, which sits at the root of a domain and applies to whole groups of URLs, a meta robots tag is attached to one HTML document or served as an HTTP response header for one resource. The tag tells a crawler how to treat that single page: whether the page should appear in search results, whether the links on the page should pass ranking signals, and how the result should be displayed if it is shown.
It is a request, not a command. Compliant crawlers honor well-formed rules; crawlers that ignore the rule can still process the page. It is also not access control. Anyone who knows the URL can request the content directly, and Google's documentation explicitly notes that sensitive information must be protected with real authentication, not with a metadata tag. Treating noindex as a privacy lock is the single most common beginner error and the one that produces the most confusing support tickets later.
The tag has two forms. The HTML form is a <meta> element inside the page <head>, identified by a name attribute of robots, googlebot, or googlebot-news. The HTTP form is a response header called X-Robots-Tag, sent by the web server, application, or CDN. Both forms carry the same comma-separated directive list, and the syntax is defined in the Google robots meta tag specification.
Directives You Can Safely Use
Each directive is a single token or a name:value pair. The table below lists the directives a beginner is most likely to use, what each one asks the crawler to do, and a beginner-facing note about how it interacts with the others.
| Directive | What it asks for | Beginner note |
|---|---|---|
| noindex | Page should not appear in search results | Crawler instruction only; not access control. Combine with crawl access in robots.txt for the rule to take effect. |
| nofollow | Links on the page should not be followed for ranking | Crawler instruction only. Crawlers can still discover and crawl the same links from other pages. |
| nosnippet | No text snippet or video preview should be shown | Google also excludes the content from AI Overviews and AI Mode input. A static image thumbnail may still appear. |
| max-snippet:N | Limit text snippet to N characters | Accepts -1 (let Google choose an effective length) or a non-negative integer. Zero is equivalent to nosnippet. |
| max-image-preview | Image preview size: none, standard, or large | None blocks preview, standard permits a default preview, large permits up to viewport width. |
| max-video-preview:N | Seconds of video preview allowed | Accepts -1 (no duration limit) or a non-negative integer of seconds. Zero permits at most a static image. |
| notranslate | Do not offer a translated title or snippet | Google-specific. Other engines may interpret the directive differently or ignore it. |
| noimageindex | Do not index images on the page | Google-specific. Page text can still appear in results. |
| indexifembedded | Index the content when embedded via iframe or similar element despite noindex | Google only honors this when paired with noindex. The generator enforces that dependency. |
Two directives are deliberately missing from this list. noarchive is listed by Google as historical and unused because the cached-link feature no longer exists, so including it would create false confidence. unavailable_after is omitted because correct date-time formatting, time zones, and publication workflows need more context than a small form should guess at.
Build Your First Tag Step by Step
Follow these steps the first time you open the Meta Robots Generator.
- Open the tool and pick the crawler target. Choose between general crawlers, Googlebot for text results, or Googlebot-News. The choice changes the meta name attribute and decides whether the X-Robots-Tag header is prefixed with that user agent; everything else stays identical. Other crawler names and vendor-specific behavior are outside this focused interface.
- Check only the directives that match the page's purpose. For a thank-you page, a staging URL, or an internal search result, noindex is usually the right starting choice. For a press page where outbound links should not pass ranking signals, add nofollow. For an image-led article, set max-image-preview to large. Resist the urge to check every box "just in case."
- Skip the combinations the tool refuses to emit. The generator will not let you pair nosnippet with a numeric max-snippet value because the more restrictive nosnippet makes the number meaningless. It will also block indexifembedded unless noindex is checked, because Google only honors indexifembedded when paired with noindex.
- Copy the HTML element into your page head, or copy the header value into your server config. For HTML pages where you control the markup, use the meta element. For PDFs, images, or video files, or for environments where response headers are the controlled channel, use the X-Robots-Tag header. Pasting the displayed header string into the page body has no effect.
- Save, deploy, and move to verification. The generator cannot guarantee deindexing, timing, or any specific search-result presentation. Search engines process changes on their own crawl schedule, so the next step is to confirm the deployed tag is what you intended.
Where the Tag Goes: Meta Element or HTTP Header
The HTML meta element lives inside the <head> of an HTML document and is rendered when the page is requested. The X-Robots-Tag header is a line in the HTTP response that the web server, application, or CDN returns before the body is sent. Both carry the same comma-separated directive list, and Google's documentation describes them as equivalent ways to express the same request.
The HTML form is the right choice for ordinary HTML pages where you can edit the template or insert a tag through your CMS. The HTTP header form is the right choice for non-HTML resources — a PDF report, an image file, a video clip — because those resources do not have a head element to host a meta tag. The header form is also the right choice when your hosting setup makes response-header configuration easier and more reliable than editing markup, for example on static file servers or behind a CDN.
One detail beginners miss: the header must be configured by software that controls the response. A web server block, an application middleware, or a CDN rule are the typical places. Copying the displayed header string into the visible content of a page does nothing; the value must travel in the HTTP response, not in the HTML body.
Verifying the Tag After Deployment
After the change is deployed, fetch the public URL as the intended crawler could reach it and inspect the source or the response headers. The browser's developer tools, a curl command, or a free header-checking service all work. You are looking for two things: that the tag you generated is present in the form you chose, and that no second, contradictory tag is present alongside it.
Conflicting rules are common because more than one system can add a meta robots tag. A CMS may emit its own default tag, an SEO plugin may add another, a reverse proxy or CDN may inject a header, and your generated tag may join the queue. Google states that the more restrictive rule applies when declarations conflict, so an extra noindex from the CMS will override your intended index directive even if your generated tag says otherwise. The right response is to inventory the existing output first, then add your tag, then re-check the delivered page.
Search engines also process changes on their own crawl schedule. You cannot force an immediate recrawl, and the generator cannot guarantee deindexing or timing. Re-check after a reasonable interval using a site: search, a URL inspection tool, or a header inspection that confirms your directive is still being served.
Beginner Mistakes That Leave the Tag Ignored
A few predictable mistakes account for most beginner confusion with meta robots tags, and the Meta Robots Generator is designed to defuse the most common ones at output time.
Using noindex to hide private content. The tag is a search instruction, not a password. Google documents that noindex and nofollow do not prevent anyone from requesting the page directly. Sensitive data must be protected with authentication and authorization, never with a metadata tag.
Blocking crawl access in robots.txt while serving noindex. If robots.txt disallows crawling, the crawler may never reach the page, may never see the noindex rule, and the URL can remain known from external links while the page content is left unprocessed. To remove an accessible page, permit crawling and serve the directive until the search engine processes it.
Pasting the X-Robots-Tag header into HTML. The header must travel in the HTTP response. A line of text in the page body that looks like a header is just page content and has no effect on crawling or indexing.
Stacking multiple meta robots tags. A second tag added by a plugin or CDN can contradict the one you generated. The more restrictive rule applies, so the unexpected tag wins. Inspect delivered HTML and response headers rather than only your editor setting.
Mixing nosnippet with a numeric max-snippet. Google treats nosnippet as the stronger signal, so the numeric value becomes dead text. The Meta Robots Generator blocks this combination at output time so the rule never ships. For a printable reference on the directive list and syntax, the Meta Robots Generator Cheat Sheet: Directives and Syntax guide keeps the same scope with a printable layout.
If you're weighing options, Documenting AI Bot Robots.txt Check Steps covers this in detail.