A meta robots directive is a page- or resource-level instruction that tells supported crawlers how to index a URL, follow its links, or control how its content may appear in search. It can be delivered as an HTML meta element or an X-Robots-Tag HTTP response header, and equivalent directives can be used in either form. The rule affects one page or resource rather than an entire site; site-wide crawling controls belong in robots.txt. Common requests include noindex, which asks a crawler not to show a resource in search results, and nofollow, which asks it not to follow links on that resource. A crawler must be allowed to fetch the resource before it can read either form, so a robots.txt block can prevent a noindex instruction from being discovered. These directives govern crawler behavior, not privacy or access control: anyone who knows the URL may still request the content, and a crawler that ignores the rule may still process it. For a guided implementation, the Meta Robots Generator converts one policy into validated HTML and HTTP syntax locally, without contacting a search service.
The crawler has three separate jobs
Fetching, indexing, and presentation are separate stages. A crawler first has to retrieve the resource; it may then process it for a search index, follow outbound links, and create a text, image, or video presentation. A noindex request addresses indexing, not fetch access. A nofollow request concerns links on the page and does not remove the page itself. This is why a page can remain accessible, still be discovered through links, and carry a directive that asks a supported engine not to show it in results.
The crawler target also changes who receives the instruction. The generator offers general search crawlers, Googlebot for text results, and Googlebot News. A general rule uses the meta name robots and an unqualified X-Robots-Tag header; Google-specific rules use googlebot or googlebot-news as the meta name and prefix the header value with the same user-agent token. Other crawler names and vendor-specific behavior are outside this focused interface, and other search engines may interpret Google-specific directives differently. When multiple meta tags or headers apply, Google combines their restrictions, so the more restrictive rule applies.
Choose the delivery channel first
Choose the channel that matches the resource and the part of the delivery stack you control. A page rule and a response-header rule can express the same policy, but they are deployed in different places.
| Consideration | HTML meta element | X-Robots-Tag header |
|---|---|---|
| Delivery | Document head and public HTML source | HTTP response headers returned with the resource |
| Best fit | HTML pages whose markup you control | PDFs, images, video, or deployments with reliable header control |
| Configuration | Template, CMS, or page source | Web server, application, or CDN |
| Main check | Inspect the delivered HTML for duplicates or conflicts | Inspect actual response headers; body text has no effect |
For HTML pages, place the element in the document head, where the WHATWG meta element definition governs metadata placement. For a PDF, image, video file, or a site with dependable header management, configure X-Robots-Tag on the response. A header displayed by the generator must be configured by the web server, application, or CDN; pasting it into visible page content does nothing.
The Meta Robots Generator serializes the same selected policy into both forms. For example, a policy containing noindex and nofollow can use <meta name="robots" content="noindex, nofollow"> in HTML or X-Robots-Tag: noindex, nofollow in a response. A Google-specific choice substitutes googlebot or googlebot-news in the meta name and qualifies the header value. Copy the form suited to the deployment; do not add both merely because both are displayed.
Select only directives that fit the policy
Use directives as precise policy statements rather than interchangeable checkboxes. Google’s robots meta tag specification defines the supported values and interactions reproduced by the generator.
| Directive | Requested behavior | Value or constraint |
|---|---|---|
| noindex | Asks that the page or resource not appear in search results. | The resource must be crawlable; this is not authentication or access control. |
| nofollow | Asks the crawler not to follow links on the resource. | It does not make the page private. |
| indexifembedded | Allows content to be indexed when embedded through an iframe or similar element despite noindex. | It only has an effect with noindex, and the generator enforces that dependency before showing output. |
| nosnippet | Asks Google not to show a text snippet or video preview. | A static image thumbnail may still appear. Google says this also prevents page content from serving as direct input for AI Overviews and AI Mode. |
| max-snippet | Limits textual snippet length; -1 lets Google choose an effective length, while 0 is equivalent to no snippet. | Accepts -1 or a non-negative whole number and cannot be combined with nosnippet. |
| max-video-preview | Sets preview duration in seconds; -1 means no duration limit. | 0 permits at most a static image under the image rule; non-integers and values below -1 are rejected. |
| max-image-preview | none requests no image preview, standard permits the default, and large allows a larger preview up to viewport width. | Use only these documented settings. |
| notranslate | Asks Google not to offer a translated title and snippet. | This is a Google presentation request. |
| noimageindex | Asks Google not to index images on the page. | It concerns image indexing rather than image-preview size. |
The generator refuses nosnippet with max-snippet because the broader block would make the numeric choice misleading. It also prevents indexifembedded without noindex. Accepted choices are converted in stable order to Google-documented tokens, keeping the copied directive order consistent.
It deliberately omits noarchive. Google lists that directive as historical and unused because its cached-link feature no longer exists. It also omits unavailable_after because valid date-time formatting, time zones, and publication workflows require more context than a generic form should guess. Familiar but inactive controls are therefore not presented as if they guarantee an effect.
Build, deploy, and inspect the rule
- Choose the crawler target and policy. Select general search crawlers, Googlebot text results, or Googlebot News. Add only the indexing, link-following, snippet, media, translation, and image choices that match the resource. If using indexifembedded, include noindex; do not select nosnippet and max-snippet together. Enter only whole-number preview values in the documented range.
- Generate and copy the appropriate output. The Meta Robots Generator turns the validated choices into stable-order tokens and serializes them as an HTML meta element plus an equivalent X-Robots-Tag option. Generation happens locally without contacting a search service. Copy the HTML for a controlled page or the header for a non-HTML resource or a deployment where response headers are the reliable channel.
- Deploy while preserving crawl access, then verify it. Put the meta element in the served HTML or configure the header at the web server, application, or CDN. Fetch the public URL as the intended crawler can access it, inspect the delivered source or actual headers, and remove duplicate or conflicting declarations. Repeat that inventory until the search engine recrawls the changed resource.
Verify the delivered source and headers
An editor setting is not proof of the final response. A CMS, SEO plugin, reverse proxy, and CDN can each inject a directive, so inspect what the public URL actually returns. For HTML, inventory every robots meta element in the source. For a header-based policy, inspect the real response headers. Also check crawler-specific declarations because a Google-only rule can coexist with a broader rule that changes the combined outcome.
If robots.txt disallows the resource, Google may never fetch the noindex directive. The URL can remain known through external links without the page content being processed. To remove an accessible page from results, allow crawling, keep the directive deployed, and wait for the engine’s own crawl schedule. The generator can produce exact syntax, but it cannot guarantee deindexing, timing, or a particular search-result presentation.
Know where robots metadata stops
Robots metadata is not a security boundary. Noindex can ask a supported crawler to leave a resource out of results, but it does not stop a person from requesting the URL, and a crawler that ignores it may still process the content. Protect sensitive information with authentication and authorization, not a metadata directive.
The generated string is also not a universal content-use switch. Separate permission through structured data, AMP relationships, or licensing can supersede preview limits, and other engines may interpret Google-specific controls differently. Use the rule to express a defined crawler policy, then confirm the delivered response and revisit the directive when permissions, indexing needs, or deployment paths change.
For a deeper look, see Meta Tag Generator for Website: Build and Verify Tags.
For a deeper look, see Robots.txt Generator: Build the File Locally in Your Browser.