Pulling URLs from an XML sitemap can be done either with a command-line pipeline (curl, grep, sed, awk, or xmllint) or with a browser-based parser, and the right choice depends on whether the audit values automation, privacy, or strict protocol validation more. Command-line pipelines built from familiar tools can fetch a sitemap and print its loc values into a flat list, but they expose every byte of the file to disk, leave deduplication and namespace handling to your own scripts, and demand manual cleanup before the list is usable. A browser-based parser inverts that workflow: you paste the sitemap XML into an editor, run extraction locally, and receive a deduplicated, first-seen-order URL list with the root type, unique count, and removed-duplicate count reported up front. Neither approach contacts a search engine; both still leave HTTP status, canonical, robots, and indexation checks as separate live-site work. What changes is who handles entity decoding, namespace prefixes, host normalization, and the 2,048-character loc constraint — your shell pipeline or the browser parser.
Common Command-Line Approaches for Sitemap URL Extraction
Command-line extraction usually starts with a download. curl or wget grabs the sitemap file from a URL, while cat or zcat handles a file you already have on disk, decompressing .gz sitemaps outside the parser. From there, the choice of locator depends on what you trust most.
The simplest pipelines grep for the loc opening tag and rely on the next line carrying the URL:
curl -s https://example.com/sitemap.xml | grep -oE 'https?://[^<]+'
That pattern works when the sitemap uses the default namespace and loc is on its own line, but it falls over the moment the file contains namespaces such as sm:loc, comments, XML declarations on the same line, or self-closing loc tags. Tighter patterns with grep -oE '<loc>[^<]+' improve accuracy, and switching to sed -n 's/.*<loc>\(.*\)<\/loc>.*/\1/p' or awk '/<loc>/{gsub(/.*<loc>|<\/loc>.*/,""); print}' adds structure. None of these tools, however, decode XML entities, normalize hosts to lowercase, drop default ports, or reject duplicates by serialized URL — they just print text.
For sites that care about protocol conformance, xmllint --xpath '//*[local-name()="loc"]/text()' sitemap.xml is the closest command-line equivalent to a real parser. It respects namespaces, ignores comments, and handles the two root types defined by the protocol, but it still leaves deduplication, host normalization, and the 2,048-character limit to your downstream script.
Where Command-Line Methods Break Down
Command-line pipelines are powerful, but they expose every gap in the data to whoever is running the script. Common failure modes appear whenever a production sitemap strays from the simplest textbook shape.
- Namespace prefixes. A site that emits sm:urlset, sm:url, and sm:loc still passes xmllint and any modern parser, but a naive grep that hard-codes <loc> returns an empty list and silently looks like success.
- Entity decoding. Standard XML entities such as & in URLs need to be decoded once, not twice. A pipeline that prints raw text passes & through to your spreadsheet and corrupts any subsequent request.
- Duplicate variants. The same page can appear with HTTP and HTTPS, with and without a trailing slash, with mixed-case hosts, or with explicit default ports. Grep and awk do not notice these duplicates; you do, much later, when the list does not match your canonical URLs.
- Over-limit inputs. A sitemap that crosses the 50,000-URL mark or the protocol's 2,048-character loc limit does not throw a structured error in most command-line pipelines — it just keeps printing or truncates unexpectedly.
- DTD declarations and custom entities. A sitemap that opens with <!DOCTYPE> or defines custom entities can blow up xmllint or get misread by sed, because both expect predictable tokenization.
Each of those gaps is solvable with another line of shell, but every added line is another place where the audit depends on the operator knowing the protocol well enough to anticipate the failure. When a team is reviewing the work, the list is hard to defend: a reviewer cannot tell whether duplicates were intentionally removed or simply never printed twice.
The Sitemap URL Extractor Approach
The Sitemap URL Extractor takes a different stance. It accepts only one input — pasted XML text from a single urlset or sitemapindex document — and produces only one output: a deduplicated, one-URL-per-line list ready for a clipboard, a spreadsheet, or a redirect audit. Because the parser runs entirely in your browser, the XML never leaves your device, which matters when the sitemap lists prelaunch URLs, staging hosts, or private subdomains.
Under the hood, the parser strips a leading byte-order mark, XML declarations, and comments, rejects DTD and custom entity declarations, then walks a closed urlset/url/loc or sitemapindex/sitemap/loc structure. It accepts the two root types defined by the protocol, including a consistent optional namespace prefix, and decodes only the five predefined XML entities and valid decimal or hexadecimal Unicode scalar references. Anything it cannot parse — nested loc markup, unknown entities, incomplete items, malformed roots — fails the entire run with a numbered pointer to the offending loc or item, so the source can be corrected instead of silently skipped.
Each decoded value is then parsed by the browser's WHATWG URL implementation, the same serializer that powers the address bar, which lowercases hosts, removes default ports, and percent-encodes non-ASCII path characters when required. Credentials, fragments, raw whitespace, malformed percent escapes, backslashes, relative paths, and non-HTTP schemes are all rejected, and every serialized URL must remain under the protocol's 2,048-character limit. URLs that serialize to the same value count as duplicates; only the first occurrence is returned, so the output preserves the first-seen order of the source.
Paste, Extract, Copy: A Three-Step Workflow
The Sitemap URL Extractor follows three operating steps that map directly onto the verified product contract:
- Paste the complete XML text from one urlset or sitemapindex document into the editor. Open your saved sitemap file in any text editor, select everything, and paste it into the input box. If your file is .gz, decompress it first with gunzip or your editor; the widget does not fetch URLs or read compressed archives.
- Extract the direct loc values and review the root type, unique count, duplicate count, and complete output. Run the extraction and read the reported figures: which root structure was detected, how many unique URLs were accepted, and how many normalized duplicates were removed. The full deduplicated list appears below, one URL per line, in the same first-seen order as the source.
- Copy the one-URL-per-line list, then perform separate live status, canonical, robots, and indexing checks as needed. Use the copy button to put the list on your clipboard, then feed it into a crawler checklist, spreadsheet, redirect audit, or migration review. Treat the output as evidence in a larger audit, not as proof of indexation, and pair it with live HTTP, canonical, robots, and indexing checks on the actual pages.
Each step is enforced client-side; nothing is uploaded, fetched, or expanded.
Command Line vs Browser Tool: A Side-by-Side Comparison
The table below lines up the two approaches on the questions an auditor actually asks.
| Consideration | Command-Line Pipeline | Browser-Based Sitemap URL Extractor |
|---|---|---|
| Input source | Fetches via curl or wget, or reads a saved file | Pasted XML text only |
| Network exposure | XML is downloaded to disk or piped through a shell | No network request; parsing is local to the browser |
| Namespace handling | Depends on the regex or xmllint xpath used | Recognizes default and prefixed namespaces consistently |
| Entity decoding | Prints raw text unless scripted | Decodes the five named entities and valid numeric references |
| Duplicate handling | Operator writes dedup logic; case and port variants often slip through | WHATWG URL serialization dedupes in first-seen order |
| Size bounds | Depends on shell memory and command-line limits | 50,000 unique URLs and five million UTF-16 code units; fails rather than truncates |
| Error reporting | Empty output, partial output, or stack trace | Specific numbered pointer to the offending loc or item |
| Output | Raw lines that need downstream cleanup | One URL per line with root type and counts reported |
| Indexation or HTTP checks | Not performed | Not performed; list feeds a separate live-site audit |
Choosing the Right Method for Your Audit
The honest answer to command line versus online is that neither is universally better. The choice tracks the job: who is running it, where the file lives, and what the output has to defend.
Reach for a command-line pipeline when the sitemap is already on disk, the audit is one of dozens you run automatically against a fleet of properties, and the operator understands the protocol well enough to write the namespace and dedup logic from scratch. Scripted extraction also wins when you need to chain the URL list directly into the next stage of automation — a crawl queue, a redirect map generator, or a log diff — without ever copying into a clipboard.
Reach for a browser-based parser such as the Sitemap URL Extractor when the XML contains information you do not want to send to a server, when a teammate who is not a shell user has to verify the list, or when the audit needs a defensible answer to how many duplicates were removed. A browser parser also makes sense as a quick sanity check before a command-line job runs at scale, because the rules it enforces — absolute HTTP or HTTPS locs only, the 2,048-character cap, no DTDs, namespace-aware parsing, dedupe by WHATWG serialization — match the protocol rather than a chosen regex.
Whichever method you use, remember that an extracted URL only confirms that a valid loc appeared in the pasted structure and passed the disclosed validation. It is not evidence that the page is crawlable, canonical, valuable, or indexed, and the Google sitemap documentation describes sitemaps as discovery hints rather than indexing guarantees. Use the list to drive the next round of live checks, and treat any disagreement between the sitemap and your canonical inventory as a signal worth investigating rather than as a bug to suppress.
If you're weighing options, Generate a URL List From a Pattern: A Practical Guide covers this in detail.
If you're weighing options, Does llms.txt Replace robots.txt or sitemap.xml? Explained covers this in detail.