Can I use noindex to protect sensitive information?

No. Noindex is a search instruction, not access control. It requests that a page not appear in search results, but anyone who knows the URL can still access it. To protect sensitive information, use authentication and authorization, such as password protection or IP restrictions.

Why doesn’t the Meta Robots Generator include noarchive?

Google lists noarchive as a historical and ignored directive because its cached-link feature no longer exists. Including it in the generator would create false confidence that the rule has an effect, so the tool omits it entirely.

Should I use the meta tag or X-Robots-Tag header?

Use the meta tag for HTML pages where you control the markup directly. Use the X-Robots-Tag header for non-HTML resources like PDFs, images, or videos, or when response-header management is more reliable than markup, such as in dynamic applications or CDN configurations.

What happens if I combine nosnippet with a numeric max-snippet value?

The Meta Robots Generator will reject this combination because nosnippet is more restrictive than a numeric max-snippet value. Google would ignore the numeric value if both were present, so the tool enforces this rule to prevent misleading output.

Can I use indexifembedded without noindex?

No. Indexifembedded only has an effect when accompanied by noindex. The generator enforces this dependency to ensure the directive is valid. If you want content to be indexed when embedded, you must also select noindex.

Create Valid Robots Meta Tags Without an API

A robots meta tag or X-Robots-Tag header is the standard way to request indexing, following, and preview limits for a page or resource without relying on an external API. The Meta Robots Generator creates a validated HTML meta element and an equivalent HTTP header from one set of choices, ensuring no contradictory combinations. This approach works entirely in your browser, so you don’t need to send data to a search service or wait for API responses. Instead, you select the crawler target—general search crawlers, Googlebot for text results, or Googlebot News—then add only the indexing and presentation limits you intend. The tool enforces Google’s documented rules, such as rejecting a numeric snippet limit when nosnippet is selected or requiring noindex for indexifembedded to take effect.

If you’ve used API-based tools for robots meta tags, you might have encountered rate limits, latency, or unexpected formatting issues. A local generator avoids these problems by producing exact syntax you can copy and deploy immediately. For example, if you need to prevent a PDF from appearing in search results, the tool generates an X-Robots-Tag header like X-Robots-Tag: googlebot: noindex, which you can configure in your web server or CDN. Similarly, for an HTML page, it provides the meta tag <meta name="googlebot" content="noindex, nofollow">. Both outputs are derived from the same validated set of choices, so you don’t have to reconcile differences between markup and headers.

This method is especially useful for teams that manage multiple environments—development, staging, and production—where API dependencies can complicate workflows. Since the generator runs locally, you can test configurations offline before deploying them. It also helps avoid common mistakes, such as mixing nosnippet with a numeric max-snippet value or omitting noindex when using indexifembedded. The tool’s interface deliberately omits historical or unused directives like noarchive, which Google no longer supports, and unavailable_after, which requires precise date-time handling beyond a generic form’s scope.

When to Use Robots Meta Tags vs. X-Robots-Tag Headers

Robots meta tags and X-Robots-Tag headers serve the same purpose but are deployed in different contexts. Use the meta tag for HTML pages where you control the markup directly. For example, if you’re editing a blog post or product page in a CMS, you can paste the generated meta element into the <head> section. This approach is straightforward for most publishers and ensures the directive is visible in the page source.

Use the X-Robots-Tag header for non-HTML resources, such as PDFs, images, or videos, or when response-header management is more reliable than markup. For instance, if you’re serving a PDF through a web server, you can configure the server to include the header X-Robots-Tag: noindex in the response. This method is also useful for dynamic applications where headers are easier to manage than embedded markup. However, note that pasting the header text into page content has no effect—the directive must be set by the server or CDN.

The table below summarizes when to use each format:

Format	Best For	Example Use Case	Deployment Method
Robots meta tag	HTML pages	Blog posts, product pages	Paste into <head> section
X-Robots-Tag header	Non-HTML resources or header-controlled environments	PDFs, images, videos, or dynamic apps	Configure in web server, application, or CDN

How to Generate and Deploy Robots Directives

Choose the crawler target and restrictions: Open the Meta Robots Generator and select whether the rule applies to general search crawlers, Googlebot (text results), or Googlebot News. Then, pick only the indexing, following, and preview limits that match your policy. For example, if you want to prevent a page from appearing in search results but allow link following, select noindex without nofollow. If you also want to block text snippets, add nosnippet.
Generate and copy the output: After selecting your options, the tool displays the HTML meta element and the equivalent X-Robots-Tag header. Copy the format you need. For an HTML page, use the meta tag; for a non-HTML resource, use the header. The tool enforces Google’s rules, so it will reject invalid combinations, such as nosnippet with a numeric max-snippet value.
Deploy the directive: For an HTML page, paste the meta tag into the <head> section of the page. For an X-Robots-Tag header, configure your web server, application, or CDN to include the header in the response. For example, in Apache, you might add Header set X-Robots-Tag "noindex, nofollow" to your .htaccess file. In Nginx, you’d use add_header X-Robots-Tag "noindex, nofollow";.
Inspect the public source and headers: After deployment, fetch the public URL as the intended crawler would access it. For an HTML page, view the page source and confirm the meta tag appears in the <head> section. For an X-Robots-Tag header, use your browser’s developer tools (Network tab) or a command-line tool like curl -I to inspect the response headers. Ensure there are no duplicate or conflicting directives from other sources, such as a CMS, SEO plugin, or CDN.
Monitor for recrawling: Search engines process changes on their own schedule, so the directive won’t take effect immediately. Use Google Search Console’s URL Inspection tool to check whether Google has processed the updated page. If the directive isn’t respected, revisit the deployment steps to confirm the tag or header is correctly placed and accessible.

Common Directives and Their Effects

The Meta Robots Generator supports a focused set of directives that align with Google’s current documentation. Each directive controls a specific aspect of how a page or resource appears in search results. Below is a breakdown of the available options and their effects:

Directive	Effect	Example Use Case	Notes
noindex	Requests that the page or resource not appear in search results	Prevent a thank-you page from appearing in search results	Does not block crawl access; anyone who knows the URL can still request it
nofollow	Requests that links on the page not be followed	Prevent search engines from following links on a user-generated content page	Does not prevent the page itself from being indexed
nosnippet	Requests that Google not show a text snippet or video preview	Prevent a page’s content from being used in AI Overviews or as a direct input for AI Mode	May still allow a static image thumbnail
max-snippet:[number]	Limits the textual snippet to the specified number of characters	Limit a snippet to 20 characters for a concise preview	Use -1 to let Google choose the length; 0 is equivalent to no snippet
max-video-preview:[seconds]	Limits the video preview duration to the specified number of seconds	Limit a video preview to 10 seconds	Use -1 for no duration limit; 0 permits only a static image
max-image-preview:[size]	Controls the size of the image preview	Allow a large image preview for a product page	Options: none, standard, or large
notranslate	Asks Google not to offer a translated title and snippet	Prevent a page’s title and snippet from being translated in search results	Does not affect the page’s content or language
noimageindex	Asks Google not to index images on the page	Prevent images on a page from appearing in Google Images	Does not prevent the page itself from being indexed
indexifembedded	Allows content to be indexed when embedded through an iframe or similar element, despite noindex	Allow a video to be indexed when embedded on another site, even if the hosting page is noindex	Only has an effect when accompanied by noindex

Some directives interact with each other or have dependencies. For example, indexifembedded only works if noindex is also selected. Similarly, nosnippet cannot be combined with a numeric max-snippet value, as the former is more restrictive. The generator enforces these rules to prevent invalid combinations that Google would ignore.

Troubleshooting Deployment Issues

Even with a validated directive, deployment issues can prevent search engines from respecting your rules. Below are common problems and how to resolve them:

Crawl access blocked by robots.txt: If your robots.txt file disallows crawling of the page or resource, search engines may never see the robots meta tag or X-Robots-Tag header. To remove a page from search results, ensure crawl access is allowed and serve the noindex directive until the search engine processes it. You can check your robots.txt file using the Robots.txt Generator or Google Search Console’s robots.txt tester.
Duplicate or conflicting directives: Multiple sources, such as a CMS, SEO plugin, reverse proxy, or CDN, can add robots directives to a page. For example, a CMS might add a meta tag, while a CDN adds an X-Robots-Tag header. Conflicting rules are interpreted using combined restrictions, with the more restrictive rule taking precedence. Inspect the delivered HTML and actual response headers to identify duplicates or conflicts. Use your browser’s developer tools or a command-line tool like curl -I to check headers.
Incorrect deployment of X-Robots-Tag: The X-Robots-Tag header must be configured by the web server, application, or CDN. Pasting the header text into page content has no effect. For example, adding X-Robots-Tag: noindex to an HTML page’s body will not work. Instead, configure the header in your server configuration file (e.g., .htaccess for Apache or nginx.conf for Nginx).
Google-specific directives ignored by other crawlers: Directives like googlebot: noindex or googlebot-news: nosnippet are specific to Google. Other search engines may ignore them or interpret them differently. If you need to control how other crawlers handle your content, use the general robots target or consult the crawler’s documentation for supported directives.
Delay in search engine processing: Search engines process changes on their own crawl schedule, which can vary from hours to weeks. Use Google Search Console’s URL Inspection tool to check whether Google has processed the updated page. If the directive isn’t respected, revisit the deployment steps to confirm the tag or header is correctly placed and accessible.

After resolving any issues, monitor the page in Google Search Console to confirm the directive is being respected. If the page remains in search results despite a noindex directive, check for crawl access, duplicate directives, and correct deployment of the tag or header.