Skip to content
Lizely

Remove Duplicate Lines

Keep the first occurrence of each line with optional case and edge-whitespace comparison, while preserving the retained text exactly.

Privacy: your files never leave your device. All processing happens locally in your browser.

How to use

  1. 1.Paste one value per line.
  2. 2.Choose whether comparison ignores English letter case or edge whitespace.
  3. 3.Remove duplicates, verify the counts, and copy the retained first occurrences.

About Remove Duplicate Lines

Remove Duplicate Lines deletes repeated lines from pasted text while keeping the first occurrence of every comparison key. Paste a list, log excerpt, identifier column, keyword set, or other line-oriented text, choose how equality should work, and process it locally in the browser. The result reports input, output, and removed line counts and shows the exact retained lines. No text is uploaded or stored by Lizely.

Strict mode compares each complete line exactly. Capitalization, leading spaces, trailing spaces, tabs, punctuation, Unicode characters, and internal spacing all participate. Under strict comparison, Apple and apple are different, and a line containing one trailing space differs from the same visible letters without that space. This mode is safest when whitespace or capitalization may carry meaning.

Ignore English letter case creates a comparison key with the browser's English-locale lowercase operation. It keeps the spelling and capitalization of the first occurrence rather than rewriting the output. This is a practical match for ordinary English lists, but it is not accent folding, transliteration, Unicode normalization, fuzzy matching, stemming, or language-aware collation. For example, German sharp s is not automatically treated as the two letters ss.

Ignore edge whitespace trims the comparison key at both ends. It does not trim the retained output. If the first occurrence is surrounded by spaces and a later line contains the same visible value without them, the spaced first occurrence is preserved exactly and the later line is removed. Internal spaces and tabs still matter. Combining case and trim options applies trimming first and English-locale lowercase second.

Line boundaries recognize Windows CRLF, classic carriage return, and line feed. Output boundaries are normalized to LF because the result is a newly generated plain-text string. A CRLF pair counts as one boundary, not two. A trailing boundary creates a final empty line before deduplication, following the same split rule as any other boundary.

Blank lines are valid values. In strict mode the first empty line is kept and later empty lines are duplicates. With trim comparison enabled, a whitespace-only line and an empty line share an empty comparison key, so only whichever appears first remains. Use Remove Empty Lines afterward if the desired output should contain no blank values.

The algorithm processes lines in source order and never sorts them. It creates one comparison key per line, checks a Set of keys already seen, appends only the first unseen line, and joins retained values with LF. It does not search the output again, merge similar values, count word frequency, or infer headers. The operation is deterministic for the input and selected options.

Input is limited to 1,000,000 UTF-16 code units. The exact boundary is accepted and one code unit beyond it is rejected before splitting or allocating comparison keys. Empty input is rejected rather than reporting a misleading one-line result. Editing the text or either option clears the previous result, so stale output cannot be confused with current settings.

Use the tool to clean copied spreadsheet columns, email campaign tags, keyword lists, allowlists, filenames, SKU exports, or log values. Review whether the first occurrence is the one worth keeping; this tool does not choose the newest, longest, most frequent, or otherwise preferred duplicate. It does not parse CSV quoting, JSON arrays, code syntax, or locale-specific records.

For business records, deduplication by a visible line may be weaker than deduplication by a stable database identifier. Confirm the comparison rule before deleting source data, and keep the original input available until the result is checked. This browser utility prepares text; it does not mutate the original file or database.

Methodology & sources

Validate a nonempty input of at most 1,000,000 UTF-16 code units, split CRLF before standalone CR or LF, derive each comparison key with optional edge trim and en-US lowercase, retain a line only when its key has not appeared in a Set, preserve the original first line text, normalize generated boundaries to LF, and report input, retained, and removed counts.

Frequently asked questions

Which duplicate is kept?
The first line for each comparison key is kept, including its original capitalization and whitespace.
Does ignore whitespace change the output line?
No. Trimming applies only to the comparison key; the first retained line stays byte-for-string unchanged apart from output LF boundaries.
Are blank lines deduplicated?
Yes. The first matching blank key remains and later matching blank lines are removed.
Is my list uploaded?
No. Splitting, key comparison, counting, and output generation run locally in the browser.

Related tools

Text Tools guides

View all