Extracting URLs from a sitemap's large text paste produces a deduplicated one-URL-per-line list by parsing the loc elements locally in the browser, enforcing a 50,000 unique URL ceiling and a five million UTF-16 code unit input ceiling as hard failures rather than silent truncation. A sitemap text in this context is the actual XML document as it sits in a clipboard or editor — not a crawlable endpoint, not a compressed archive, not a remote URL string — and that distinction is what makes the task solvable by a paste parser rather than by a crawler or an online service. The output the reader usually wants is a deduplicated list of absolute URLs, in the order they appeared in the source, with duplicates that came from host casing, default ports, or trailing slashes collapsed into a single line, and with malformed loc values removed from the run entirely rather than passed through as best-effort strings. Anything beyond that — checking live HTTP status, comparing canonicals, deciding what gets indexed — belongs to a separate check against the live site, and the parser's job ends precisely where that boundary starts.
The job is straightforward when the document is small, but large sitemaps change the equation. A urlset from a content-heavy site can easily cross one hundred thousand lines of XML once optional children such as lastmod and priority are included, and a sitemapindex can reference dozens of child sitemaps. Most off-the-shelf approaches break at that size — spreadsheets truncate, find-and-replace loops over- or under-count, and remote services balk at the upload. The discipline of paste, parse locally, and copy back out scales much better because the entire document sits in memory before the parser makes any commitment.
What "large text" actually means for a sitemap paste
When someone searches for how to extract URLs from sitemap large text, they are usually holding one of three artifacts: a urlset exported from a CMS, a sitemapindex pasted from a developer terminal, or a decompressed copy of a .gz file that does not fit cleanly into a spreadsheet column. The XML in each case can run into several megabytes, and urlset documents in the wild regularly exceed one hundred thousand lines once optional children such as lastmod, changefreq, and priority are present.
Large in this context is measured against the two ceilings the parser enforces: 50,000 unique URLs after deduplication, and five million UTF-16 code units of input text. On a typical urlset with one entry per line, five million code units corresponds to roughly four to five megabytes of UTF-8 — comfortably more than most CMS exports but well below what a compressed production sitemap can grow to once the file is decompressed. Anything larger has to be split into child files referenced by a sitemap index before it can be processed by a paste parser.
Why a browser-only paste parser suits a large XML blob
The Sitemap URL Extractor is built around a single discipline: the XML is pasted as text, parsed in this browser, and never uploaded. That discipline shapes everything else about the experience. The full paste is held in the page rather than streamed to a server, which means the parser has the entire structure in front of it before it commits to a result. For large pastes that approach is exactly what you want — a parser that walks the document once, decides whether the structure is valid, and either returns the complete list or fails with a numbered error pointing at the offending loc or item.
Browser-side parsing also keeps staging and prelaunch inventories private. A production sitemap for a site with hundreds of thousands of URLs is itself a sensitive artifact, and pushing it to a remote endpoint just to pull it back as a list creates an unnecessary audit trail. Because the widget makes no network request, the paste and the resulting list stay on the device unless the user copies them elsewhere.
The list that comes out is not just "every line that looks like a URL." Each entry is what the WHATWG URL Standard calls a serialized absolute URL, with the host lowercased, default ports removed, and non-ASCII path characters percent-encoded. That normalization step is what lets the deduplication report an accurate duplicate count instead of treating example.com and Example.com:80 as separate entries.
How to extract URLs from large sitemap text
- Open the Sitemap URL Extractor in your browser and clear the editor.
- Paste the complete XML text from one urlset or sitemapindex document into the editor. If you started from a .gz file, decompress it first so the editor receives uncompressed XML text; compressed bytes are not parsed.
- Run the extraction. The widget inspects the root element, walks each url or sitemap item, decodes the standard XML entities inside its loc child, and rejects nested markup, DTD declarations, or custom entity references.
- Review the result block. It reports the detected root type (urlset or sitemapindex), the number of unique URLs accepted, the number of normalized duplicates that were removed, and the complete deduplicated list in first-seen order.
- Copy the one-URL-per-line list to your clipboard. Use it directly as a crawler checklist, paste it into a spreadsheet, or feed it into a separate tool.
- Run separate live checks as needed. The extraction confirms only that a valid loc appeared in the pasted structure; HTTP status, canonical tags, robots directives, and indexing still have to be verified against the live site.
What the parser validates in the pasted XML
Before a single URL reaches the output list, the parser enforces a specific contract. A leading byte-order mark, an XML declaration, and comments are stripped at the start. The document is required to declare exactly one root — a urlset or a sitemapindex — with a consistent optional namespace prefix such as sm:urlset, sm:url, and sm:loc, or the common default-namespace form. Inside, every url or sitemap item must contain its direct loc child, and that loc must contain plain text rather than nested XML.
Only the five predefined XML entities and valid decimal or hexadecimal Unicode scalar references are decoded inside loc. Unknown entities, malformed numeric references, DTD declarations, and custom entity declarations cause the entire run to fail. That hard-fail behavior is deliberate: the widget does not silently guess what an entity should resolve to, because hidden entity expansion is the kind of behavior that turns a small tool into a security surprise.
After the XML is decoded, every loc value is run through the browser's WHATWG URL implementation. The parser accepts only absolute http and https URLs, each serialized to fewer than 2,048 characters to match the protocol's own loc constraint. URLs with credentials, fragments, raw whitespace, malformed percent escapes, backslashes, relative paths, or non-http schemes are rejected, and extension locations such as image:loc are not substituted for a missing page loc because they describe a different resource role.
Limits and failure modes for large pastes
The widget enforces two interacting limits on every run. The first is 50,000 unique URLs after deduplication. The second is five million UTF-16 code units of input text. Crossing either bound causes the entire operation to fail without truncation. There is no "first 50,000 only" mode and no warning-then-clip behavior, because presenting a partial list as complete would defeat the audit purpose of the tool.
When a paste fails, the error message is intentionally specific to the numbered loc or item that broke validation, so the source can be corrected in place rather than silently skipped. Common failure modes for large pastes include a stray DTD declaration pasted from a CMS export, a sitemapindex that mixes sitemap entries with image entries, and url entries whose loc contains nested markup such as a CDATA wrapper or an HTML escape that never decoded. Each of these is fixed in the source XML rather than worked around in the tool.
If a production sitemap sits close to these limits, the appropriate response is to validate the actual uncompressed byte size and consider splitting the file into smaller children referenced by a sitemap index. The bulk limits guide walks through how the two ceilings interact with typical XML density. According to Google's sitemap documentation, sitemaps are themselves discovery hints for crawlers rather than indexing guarantees, and a smaller, well-formed child file generally behaves better in practice than a borderline-large parent.
Sitemapindex paste versus urlset paste
| Scenario | What the widget returns | What it does not do |
|---|---|---|
| Paste a urlset from a CMS export | One deduplicated URL per line, root reported as urlset | Fetch the live file or check status codes |
| Paste a sitemapindex from a terminal | One deduplicated sitemap file location per line, root reported as sitemapindex | Expand each child file into its page URLs |
| Paste text that contains a DTD declaration | The run fails with a specific error pointing at the DTD | Silently strip the DTD and continue |
| Paste a decompressed .gz file as text | Parses normally | Decompress a still-compressed .gz blob |
| Paste a remote URL string instead of XML text | URL parse fails on the first loc, run fails | Download the URL behind the scenes |
The table makes the boundary explicit. A sitemapindex is a directory of sitemap files, not a recursive bundle, so the extraction returns the direct sitemap file locations and stops there. Pasting each of those files in turn is a separate operation, not an implicit step inside the first one. Likewise, pasting a remote URL string into the editor does not initiate a download; the editor expects XML text and treats anything else as a malformed loc.
Treating the extracted list as audit evidence, not as indexing proof
An extracted list is one piece of evidence in a larger audit. Compare it with canonical URLs from the content database, with crawler landing pages from server logs, and with the analytics set of pages that actually receive traffic. The deduplication report is useful here: unexpected duplicates that show up in the source often reveal default-port or host-case variants that the source XML accidentally produced through template duplication.
For migration reviews, paste the pre-migration sitemap into the extractor, then paste the post-migration sitemap, and diff the two lists. Because the output preserves first-seen order, both lists stay directly comparable. For redirect audits, the same list feeds a separate batch status checker. For a noindex review, the list feeds a separate meta robots inspector. The tool does not attempt any of those checks itself, and that is by design — keeping the responsibility narrow is what lets it stay correct on the one thing it does.
Finally, keep the limits in mind when interpreting the result. A clean 48,000-line extraction does not mean the site has 48,000 indexable pages, only that 48,000 distinct loc values passed the parser's structural and URL-shape checks against the pasted text. Crawlability, canonical selection, robots directives, and indexing all live with the live site, and they are checked against the live site, not against the paste.
If you're weighing options, Keyword Density Checker on Windows: Paste, Count, Read covers this in detail.