GUIDE · LISTS
Removing duplicate lines from a list
Duplicate entries are the most common defect in any list that was built by more than one person, or assembled from more than one file. They inflate counts, send twice the email, and make a checklist look unfinished when a single item appears ticked twice. Removing them is mechanical once you know exactly what the tool is comparing — and the comparison is deliberately stricter than a human would be.
Where duplicates come from
- Merged files. Two exports each contain the same core entries plus their own additions; concatenating them doubles the shared rows.
- Copy-paste loops. Selecting a list, pasting it, then pasting again lower down because you forgot it was already there.
- Re-sent forms. A respondent submits twice, or a form appends a row on every save instead of replacing it.
- Generated lists. A script or export that walks a tree twice, or that merges two sources with overlapping keys.
- Casing and spacing variants. The same entry written slightly differently is not the same text — see the next section.
What counts as a duplicate
Line-based deduplication compares the trimmed text of each line, exactly. Two lines match only when their letters, digits, punctuation and internal spacing are identical after leading and trailing whitespace is removed. So these two lines are duplicates:
apple apple
For the comparison, trimming strips the padding from both lines and the two remaining strings are equal — so they are duplicates. Note what that does not mean: the surviving line is kept exactly as it was written, padding included. Deduplication trims to compare; it does not trim to output. Run Clean spacing first if you want the result to be clean as well as unique. These three lines, by contrast, are not duplicates, even though a person would call them the same word:
apple Apple apple
Capital A is a different character from lowercase a, so the second line never matches. This literalness is a feature, not an oversight: a tool that guessed which differences are unimportant would eventually delete a line you meant to keep. The rule is that if two lines are not character-for-character identical after trimming, both survive.
Case, spacing and invisible characters
Most “duplicates that will not go away” turn out to be variants that differ by something you cannot see. This table shows the usual suspects and how to deal with each:
| Variant | Treated as duplicate? | What to do |
|---|---|---|
| Leading or trailing spaces | Yes — the comparison trims before comparing | Run Clean spacing first: the surviving line keeps its own padding |
| Double space inside the line | No | Run Clean spacing first |
| Different letter case | No | Normalize case with lowercase if case is not meaningful |
| Trailing non-breaking space (U+00A0) | Yes — it trims to empty | Nothing for detection; the note below covers the output |
| Non-breaking space inside the line | No | Not fixed by Clean spacing; see the Unicode guide |
| Smart quotes vs straight quotes | No | Different characters; fix by hand or at the source |
Before and after
A contact list assembled from two exports, with padding, a case variant and one genuine repeat:
alice@example.com bob@example.com alice@example.com carol@example.com Bob@example.com bob@example.com
After Clean spacing the padding is gone. After Remove duplicate lines the genuine repeat of alice@example.com is dropped, while the two variants of Bob remain, because their case differs:
alice@example.com bob@example.com carol@example.com Bob@example.com
Whether that is the right outcome depends on your data. If addresses are case-insensitive — as they are in practice — run lowercase first, then deduplicate, and the two Bob entries collapse into one. If case carries meaning, for example in an identifier or a passcode list, leave it and deduplicate as-is.
The recommended sequence
- One item per line. If entries contain commas or are wrapped in quotes, split them first; one row per line is what the comparison needs.
- Run Clean spacing. Trims each line so padding cannot protect a duplicate from detection.
- Normalize case if case is not meaningful. lowercase is usually the right intermediate for emails, URLs and tags.
- Run Remove duplicate lines. The first occurrence is kept in its original position; later copies are dropped.
- Sort only if you want them ordered. Sort lines A–Z is a separate step, because deduplication preserves input order by design.
Why the first occurrence is kept, and order is preserved
Deduplication keeps the first matching line and drops the later ones. That choice matters when the list has an implicit order: a checklist that was worked through top to bottom, a changelog, a set of instructions. Dropping later copies rather than reordering anything means the surviving list reads in the same sequence you supplied, minus the repeats. If you want the result alphabetized as well, sort afterwards — but do it as a deliberate second step, because sorting a list that has a meaningful order is a destructive edit dressed up as tidying.
When duplicates are intentional
Some lists legitimately repeat. A rota or a schedule may need the same name on several days; a chorus or a refrain repeats on purpose; a shopping list may genuinely need four entries for the same item. Because the comparison is literal, the tool cannot tell a meaningful repeat from an accidental one. When repeats carry meaning, keep the original list and deduplicate a copy — the input panel is not modified, so nothing is lost until you paste over it.
Deduplicating more than plain lists
The same action is useful well beyond tidy word lists. Deduplicating a mailing list before an import avoids sending the same message twice. Deduplicating a list of identifiers or URLs before a crawl avoids fetching the same page twice. Deduplicating a set of tags or keywords shows the true vocabulary of a document, which is often much smaller than the raw count suggests. In each case the sequence is the same: normalize spacing, decide whether case matters, then deduplicate — and check the line count before and after, because the difference is the number of entries you just removed.
Frequently asked questions
Why is “Apple” not removed as a duplicate of “apple”?
Because the comparison is exact. Capital A and lowercase a are different characters. If case is not meaningful for your list, convert to lowercase first and then deduplicate.
Does it remove duplicates that are not next to each other?
Yes. Every line is compared against every earlier line, so a repeat on line 400 of a list is removed even when the first occurrence was on line 2.
Is the first or the last occurrence kept?
The first. Later copies are dropped, and the surviving line stays in its original position and in its original form — padding included, because deduplication trims to compare rather than to output. Run Clean spacing first if you want a tidy result.
Can it merge lines that are only partly similar?
No, and it should not try. Two lines that differ in any character after trimming are both kept. Fuzzy matching would eventually delete something you meant to keep.
What if my file has 50,000 lines?
The comparison runs in your browser and is linear in the number of lines, so large lists are fine. Very large inputs may make the editor feel sluggish; that is a rendering limit, not a correctness limit.