A complete list of URLs is one of the most useful assets in SEO because it lets you audit, redirect, or analyze every page a search engine has been told to consider. To extract all URLs from a website, open the site's XML sitemap (often located at /sitemap.xml), copy the entire document, paste it into the Sitemap URL Extractor, and read back a deduplicated one-URL-per-line list that you can copy into any other tool. This approach works because a standards-shaped sitemap puts every page address inside a single <loc> element wrapped by a <urlset> root, and a sitemap index at the same path exposes the child sitemaps that hold those page-level documents.

The reason this matters in practice is that crawlers, audits, and bulk editors all want the same thing: a flat list of addresses, one per line, with no XML baggage. XML sitemaps also include extra elements like <lastmod>, <changefreq>, and <priority> that are useful for context but get in the way of a clean list. Stripping those out by hand on a 5,000-URL site is tedious and error-prone, which is why an automated pass saves real time.

how to extract all urls from a website
how to extract all urls from a website

Where to find the sitemap on any site

Most websites expose a sitemap at the standard location of /sitemap.xml at the root domain. Common variations include /sitemap_index.xml, /sitemap.xml.gz, /sitemap-index.xml, or language-specific paths like /en/sitemap.xml. A few sites also publish separate sitemaps for posts, pages, products, and images, and then link them all from a single sitemap index document.

If the standard URL is empty, look at the robots.txt file at the root of the domain — it almost always declares the sitemap location. The Sitemap directive in robots.txt is a one-line statement, and a real site will name the canonical XML path directly. If even that is missing, the documentation or footer of the site often links to a "Sitemap" page, which can be the entry point into the index.

Extract every URL from a website using the Sitemap URL Extractor

  1. Open the sitemap in your browser, view source, and select the entire XML document from the opening <?xml> declaration to the closing </urlset> or </sitemapindex> tag.
  2. Paste the XML text into the editor area of Sitemap URL Extractor.
  3. Read the summary: the tool reports the detected root type (urlset or sitemapindex), the total number of <loc> entries, the unique count, the duplicate count, and the final one-URL-per-line output.
  4. Copy the deduplicated list using the Copy button and paste it into a plain text editor, spreadsheet, or any downstream tool that accepts line-separated URLs.
  5. If the document was a sitemap index, copy the list of child sitemap URLs first, open each one in turn, and repeat the paste step for each child document to build a combined inventory.

The Sitemap URL Extractor runs entirely in the browser, which means the XML never leaves your machine. That detail matters when the sitemap includes draft URLs, staging paths, or private product feeds that should not be submitted to a third-party crawler. Once the list is in your clipboard, you can move on to verification without re-downloading anything.

How sitemap and sitemap-index documents differ

Understanding the two document shapes helps you decide whether one paste is enough or whether you need to loop over children. The table below summarizes the structural difference and the practical workflow for each.

Aspect urlset sitemap sitemapindex document
Root element <urlset> <sitemapindex>
Direct children <url> elements wrapping one <loc> each <sitemap> elements wrapping one <loc> for each child
What it gives you on first paste Complete page-level URL list List of child sitemap URLs, not the pages themselves
Workflow One paste, then copy the result Paste, then open and paste each child in turn to merge results
Size guidance per sitemaps.org Up to 50,000 URLs, 50 MB uncompressed Up to 50,000 child sitemaps

A common production pattern is a single sitemap index at /sitemap.xml pointing to a dozen child sitemaps such as post-sitemap.xml, page-sitemap.xml, and product-sitemap.xml. Recognizing which type you are looking at keeps the extraction from stopping halfway.

What to do with the URL list once you have it

Extracting URLs is rarely the end goal — it is the start of a series of checks. The list you copied is now a working input for several standard SEO tasks.

  • Status-code sweep: paste the list into a batch HTTP status checker to flag 404s, 301s, 302s, and 5xx responses that should be cleaned up or redirected.
  • Canonical audit: confirm that each URL's rel=canonical element points to itself or to a preferred alternative, especially on paginated and parameterised pages.
  • Robots and meta robots review: cross-check each URL against the site's robots.txt rules and individual meta robots tags to confirm which pages are eligible for crawling and indexing.
  • Indexing verification: spot-check a sample against live search results to confirm that pages marked indexable actually appear.
  • Redirect mapping: feed the list into a redirect planner when migrating to a new domain or restructuring sections.

For a deeper audit context, the XML sitemap guide explains the structural rules a sitemap must follow, which makes it easier to spot malformed entries that the extractor will quietly skip. The canonical tag guide is a useful companion when the canonical audit step above turns up inconsistencies.

Common pitfalls when extracting URLs

Several edge cases trip people up the first time they pull a sitemap apart, and a quick awareness of them saves a round of cleanup work.

  • Gzipped sitemaps: a file ending in .xml.gz must be decompressed before pasting, otherwise the extractor sees binary content and reports zero URLs.
  • Truncated paste: copying only the visible viewport in a browser often cuts off the closing tag and produces a partial list; always view source and select everything.
  • Relative loc values: although the protocol expects absolute URLs, older or broken sitemaps sometimes include paths like /about; flag those for manual fixing rather than treating them as final addresses.
  • Encoding mismatches: sitemaps declared as UTF-8 can contain stray characters from a CMS export, which break parsing; opening the source in a plain text editor before pasting often resolves this.
  • Hidden duplicates: a URL can appear in two child sitemaps at once, especially when an image or news sitemap overlaps with the main one; trust the unique-count number reported by the extractor.

Building a complete inventory across multiple sources

When a website exposes more than one entry point — for example a sitemap index plus a separate news sitemap and an image sitemap — combine them in three steps. First, paste the sitemap index into the extractor and copy the list of child sitemap URLs. Second, paste each child one at a time and append the deduplicated page list to a master document. Third, de-duplicate the master document itself with a sort and uniq pass, or by pasting the merged list back into the extractor to confirm the final unique count.

The protocol that defines this entire pipeline is published by sitemaps.org, and the underlying XML schema is documented at sitemaps.org. For developers who want to understand how <loc> values are wrapped inside <url> elements at the XML level, the relevant reference is the Extensible Markup Language specification from the W3C.

Putting it together

The shortest reliable path is the one this article is built around: find the sitemap, copy the source XML, paste it into the Sitemap URL Extractor, and copy the deduplicated result. From there, treat the list as a working document for the audits you actually need — status codes, canonicals, robots rules, and indexing verification. Because the extraction is client-side, the same workflow works for staging sites, private feeds, and client deliverables without exposing the sitemap to anyone else.

If your audit later requires generating a fresh sitemap from a revised URL list, the XML Sitemap Generator takes a reviewed list and produces a standards-shaped document you can upload. The extractor and the generator are designed to be used as a pair, with one turning XML into a clean list and the other turning that list back into XML once it has been edited.