How to remove duplicates from a list (emails, keywords, product codes)

To remove duplicates from a list, put one item per line and run it through a de-duplicator: the first occurrence of each line is kept where it is, and every later repeat is dropped. The part that goes wrong is not the removal but the matching — Anna@Example.com, anna@example.com and anna@example.com are the same address to you and three different strings to a computer.

The quickest route here: paste the list into Remove Duplicate Lines. It keeps the original order, tells you how many repeats it removed, and gives you three switches: Case sensitive, Ignore leading/trailing spaces and Keep empty lines. It runs in your browser, so a customer list never leaves your device.

Choosing a method

Method	Good for	Watch out for
A de-duplicator like this one	Any list you can paste; keeping the original order; seeing the count	You work on text, not on a spreadsheet's other columns
Excel's Remove Duplicates	A column that sits inside a table with other columns	Deletes rows in place, and the comparison ignores case
The UNIQUE function	A live list that keeps updating	Needs Microsoft 365, Excel 2021 or Google Sheets
sort -u or awk '!seen[$0]++'	Files too large to paste	sort -u also sorts; the awk version keeps the order

Excel. Select the range, then Data → Remove Duplicates in the Data Tools group, tick the columns that define a duplicate, and choose OK. Excel reports how many duplicate values it removed and how many unique values remain, and it deletes the rows from the sheet — take a copy first. Its comparison is not case-sensitive, so Apple and apple count as one, which is the opposite of this tool's default. =UNIQUE(A2:A500) leaves the source untouched and spills a de-duplicated list into a new column instead.

Google Sheets. Data → Data cleanup → Remove duplicates, or =UNIQUE(A2:A500) for a live copy.

Step by step

1. Put one item per line. If the list comes from a spreadsheet, copy the column — each cell becomes a line. If the items are buried in running text, such as email addresses inside a thread, pull them out first with Extract Emails, which can de-duplicate, lowercase and sort as it goes. A cell that contained a line break becomes two lines, so check the end of a pasted column before you trust the count.

2. Set the two switches that matter. Turn Case sensitive off for email addresses, domains and keywords, where Anna@Example.com and anna@example.com are the same thing. Leave Ignore leading/trailing spaces on: stray spaces are the most common reason two identical-looking lines are not detected as duplicates. With it on, the output is trimmed as well as compared trimmed.

3. Sort if you need to. Sort Lines orders the result. A → Z (natural) puts item 2 before item 10, and accented words sort next to their base letter, because the comparison uses your browser's language-aware collation rather than raw character codes.

4. Deal with the blank lines. By default, blank lines are treated like any other value: the first one survives and the rest are removed. Tick Keep empty lines to leave them all alone, or drop them completely with Remove Empty Lines, which can also treat a line of spaces as empty.

What the operation does and does not do

• Order is preserved. The first occurrence stays in its original position; only later repeats disappear. This matters when the order carries meaning — a priority list, a sequence of events, a ranked set of keywords.
• You lose the count of each item. De-duplication tells you how many repeats were removed in total, not that paris appeared 14 times. If the frequency is the point, use a pivot table or COUNTIF instead.
• It is a line-by-line comparison. Two lines either match exactly, under the options you chose, or they do not. Nothing is fuzzy: Acme Ltd and Acme Limited are two different entries and always will be.
• Nothing is uploaded. The list is processed in your browser and disappears when you close the page.
• Size is not the limit. Lines are compared through a hash set, so hundreds of thousands of lines are handled at once; pasting them into a browser is the slow part, not the matching.

When duplicates survive the de-duplication

Invisible characters. Non-breaking spaces, zero-width spaces and zero-width joiners come along with text copied from web pages and PDFs, and they make two identical-looking lines different. Text Cleaner removes invisible characters, straightens smart quotes and collapses runs of spaces — run it first, then de-duplicate.

Unicode normalisation. é can be stored as one code point or as e plus a combining acute accent. Both look the same on screen and neither tool nor human can see the difference, but they are different strings, so Méline and Méline can both survive.

Different punctuation. A curly apostrophe against a straight one, an en dash against a hyphen, a trailing full stop. Text Cleaner's Straighten smart quotes and Dashes & ellipsis → ASCII options normalise those before you compare.

Tagged email addresses. anna+news@example.com and anna@example.com reach the same inbox but are different addresses, and no de-duplicator will merge them. Decide deliberately whether you want them merged, and do it with a find-and-replace before de-duplicating.

Trailing separators from a CSV. A column exported with a trailing semicolon or tab gives you paris; and paris. Clean the separator out before you match — see How to open a CSV file in Excel for why exports arrive in that state.

Comparing two lists against each other

De-duplicating one list is not the same as finding what changed between two. For that, sort both lists the same way and compare them with Text Compare: entries present only on the left show as removals, entries only on the right as additions. Sorting first is what makes this readable — two lists with the same entries in a different order otherwise report as entirely different. How to compare two versions of a text covers the options in detail.

Questions people ask

Which occurrence is kept? The first one. Later identical lines are removed and the order of the remaining lines does not change.

Can I paste a column straight from Excel or Google Sheets? Yes. Copying a column puts one cell per line, which is exactly what the tool expects. Paste the result back into a single column.

Does it work on a list separated by commas? Not directly — it compares lines. Replace the commas with line breaks first, de-duplicate, then join it back up if you need a comma-separated list again.

How many lines can it handle? Hundreds of thousands. If your list is bigger than that, or lives in a file you would rather not open, sort -u file.txt or awk '!seen[$0]++' file.txt does the same job from a terminal.

Why does Excel find fewer duplicates than this tool? Usually the opposite: Excel ignores case and this tool respects it by default. Turn Case sensitive off and the two should agree.

Next steps

• Order the result: Sort Lines
• Strip the blank lines: Remove Empty Lines
• Normalise the text before matching: Text Cleaner
• Find what changed between two lists: How to compare two versions of a text