Which Lines Are in Both Files? comm vs grep vs sort|uniq vs Python, Measured
Two files, a million lines each, half a million lines in common. comm finds them in 0.14 seconds. grep needs 2.60. Python needs 0.82. The second-fastest method, sort|uniq -d at 0.49 seconds, is wrong — it reports a line that appears in only one file, because that line appears twice.
Written November 9, 2026 · Tested with Python 3.12.3 · 11 min read
- The table: comm is 8 to 19 times faster than grep
- Why grep is slow here: the pattern load, not the matching
- comm needs sorted input, and the sort is part of the cost
What this guide is
Two text files, a million lines each, with exactly 500,000 lines in common and 500,000 lines unique to each file. Five ways to find the shared lines, each run three times: comm -12 on pre-sorted files, comm -12 with the sort included, sort plus uniq -d, a Python set intersection, and grep -xF -f. Every arm returns 500,000 lines, and the sorted output of every arm hashes to the same value, so the table compares methods that agree on the answer — except one of them, on the trap corpus, does not.
The question is the one people ask when two lists need reconciling: which lines are in both files. The usual answers are comm, grep, or a sort pipeline, and the usual advice is that comm is fastest and grep is simplest. The table confirms the first half and puts a number on it. The second half is where the guide earns its keep: the second-fastest method is wrong on files that contain a duplicate line, and the wrongness is silent — the output looks like an intersection, counts like an intersection, and contains a line that is in only one file.
The table: comm is 8 to 19 times faster than grep
Wall seconds, medians of three runs, fresh tonight.
comm -12on pre-sorted files: fresh median 0.139, reps 0.135, 0.139, 0.145. Night before: 0.172, reps 0.134, 0.172, 0.616.comm -12with the sort included,comm -12 <(sort A.txt) <(sort B.txt): fresh median 0.322, reps 0.314, 0.322, 0.831. Night before: 0.342, reps 0.331, 0.342, 0.818.sort A.txt B.txt | uniq -d: fresh median 0.494, reps 0.472, 0.494, 0.571. Night before: 0.471, reps 0.461, 0.471, 0.475.- Python
setintersection: fresh median 0.824, reps 0.821, 0.824, 0.827. Night before: 0.814, reps 0.801, 0.814, 0.827. grep -xF -f A.txt B.txt: fresh median 2.595, reps 2.595, 2.595, 2.792. Night before: 2.736, reps 2.529, 2.736, 2.794.
Sorting the two files once, untimed in the arms, costs 0.396 seconds total on both nights. That is the honest setup cost of the comm arms when the files arrive unsorted — cheaper than the grep arm alone.
The ratios, fresh tonight: grep is 8.1 times slower than comm with the sort included, and 18.7 times slower than comm on pre-sorted files. Python is 5.9 times slower than comm on pre-sorted files and 2.6 times slower than comm with the sort included. sort|uniq -d is 3.6 times slower than comm on pre-sorted files. The night before, the same ratios were 8.0 and 15.9 for grep, 4.7 and 2.4 for Python, 2.7 for sort|uniq -d. The ranking is identical on both nights; only the comm-presorted ratio moved, because its median moved from 0.172 to 0.139.
The two rep-3 spikes deserve a sentence, not a footnote. On the night before, the pre-sorted comm arm spiked to 0.616 on its third rep; tonight it did not, and all three reps landed between 0.135 and 0.145. But the comm-with-sort arm spiked to 0.831 tonight, almost exactly the 0.818 it spiked to the night before. The spikes are scheduler noise on a machine with other workloads running, and they land on different arms on different nights. Report the ranges, never the medians alone — a median of 0.322 with a max of 0.831 is a different claim from a median of 0.322 with a max of 0.342.
The trap: sort|uniq -d is wrong unless both files are already deduplicated
dup-line-aaa twice and shared-line-bbb once. File B2 holds shared-line-bbb once and other-line-ccc once. The true intersection is one line: shared-line-bbb.comm -12: one line,shared-line-bbb. Correct.grep -xF -f: one line,shared-line-bbb. Correct.- Python
setintersection: one line,shared-line-bbb. Correct. sort A2.txt B2.txt | uniq -d: two lines,dup-line-aaaandshared-line-bbb. Wrong.
uniq -d prints every line that appears twice in the combined input. dup-line-aaa appears twice because it is duplicated inside A2 alone — it is in no file but A2, and it still shows up in the output. The method cannot tell a line that appears once in each file from a line that appears twice in one file, because by the time it sees the combined stream, that distinction is gone. BashFAQ states the precondition plainly, in parentheses after the recipe: "Assuming each of file1 or file2 does not have repeated content." The precondition is exactly what its users skip, because the files that need intersecting are the files nobody has deduplicated.
The fix is sort -u first, which deduplicates each file before the combined sort. That is a different command from the one people type, and it costs a sort pass the method was supposed to avoid. Once the files are deduplicated, sort|uniq -d is correct — and still slower than comm on pre-sorted files, so the trap costs both correctness and speed. The guide's rule is short: never use sort|uniq -d for an intersection unless you can show both files are already deduplicated, and if you have to check, you have already paid for the sort.
Why grep is slow here: the pattern load, not the matching
grep's 2.60 seconds is not the cost of matching a million lines against a million patterns. It is the cost of loading a million patterns. grep -xF -f A.txt B.txt reads all of A into memory as fixed strings, builds its matcher over them, and then scans B. The matching itself is fast — the same grep engine pulls a field out of a million JSON lines in 0.35 seconds in the sibling guide. Here the pattern file is the whole of A, and the load dominates.
That is why the number does not generalize to a small pattern file. A hundred patterns against a million lines would look nothing like this table; the load would be trivial and grep would be competitive. This probe did not measure that case, and the guide does not claim it. What the table does say is that grep is the wrong tool when the pattern file is itself a million-line file, which is exactly the case the intersection question presents. BashFAQ's warning is the citable version of the same point: the file given with -f "will be loaded into memory, so it doesn't scale for very large files." No memory measurement was taken in this probe — the guide reports no RSS number, only the wall time and the documented behavior.
comm needs sorted input, and the sort is part of the cost
comm's speed comes with a precondition of its own: the input files must be sorted in the collating sequence of the LC_COLLATE locale. The GNU manual says it plainly, and adds that sort with no options always produces a file suitable as comm input. The 0.396-second setup sort is therefore part of the honest cost whenever the files arrive unsorted, which is why the guide prints both comm rows: 0.139 seconds if the files are already sorted, 0.322 if they are not.
Unsorted input is not silently accepted. comm diagnoses wrongly sorted input when it finds lines that cannot be paired, and exits nonzero — the output in that case should not be used. The --check-order option makes the failure explicit on any out-of-order line, and --nocheck-order suppresses the check entirely, which the manual warns is not guaranteed to produce any particular output. The practical rule is the manual's: sort first, or use process substitution to sort on the fly, and never pass comm files you have not sorted.
One more precondition, easy to miss: comm compares whole lines, and the comparison honors LC_COLLATE. A file sorted under one locale and compared under another is unsorted input by another name. The probe ran under the box's default locale for both the sort and the comm, so the two agree; a reader sorting in one shell and comparing in another should check the locale matches.
Checklist
- Before intersecting two files with comm, check they are sorted in the same locale you will compare them in, or sort them in the command itself — unsorted input is diagnosed, not silently accepted.
- Before trusting a
sort|uniq -dintersection, check both files are deduplicated — a line duplicated inside one file alone is a false positive, and the output gives no warning. - Before choosing on speed, check the table: comm is 8.1 times faster than grep with the sort included and 18.7 times faster on pre-sorted files, and Python sits between them at 0.82 seconds.
- Before quoting a ratio, check the range: the comm-with-sort arm spiked to 0.831 on its third rep on both nights, so the median is not the whole story.
- Before choosing grep, check the size of the pattern file: its 2.60 seconds is dominated by loading a million patterns, and a small pattern file would look different — unmeasured here.
- Before reusing these numbers, check the setup: WSL2, 32 CPUs, GNU coreutils 9.4, GNU grep 3.11, Python 3.12.3, tmpfs, warm page cache, one thread per arm — the ranking held across two nights, the absolute seconds are this box's.
Related guides on this site
- Pulling One Field Out of a Million JSON Lines: jq vs Python vs grep, Measured
- Numbering a Million Lines, Measured: cat -n vs nl vs awk vs grep -n
- Finding Lines by Prefix: look vs grep vs rg vs awk vs sed, Measured
- Copying, Moving, and Splitting Files: Ten Measured Guides
Sources
- GNU Coreutils, comm invocation — read 2026-11-09: input files must be sorted in the
LC_COLLATEsequence;sortwith no options always produces suitable input;-1,-2,-3suppress the corresponding columns; unsorted input is diagnosed when unpairable lines are found and the command exits nonzero;--check-orderfails on any out-of-order line. - GNU Coreutils, sort invocation — read 2026-11-09: comparisons use the
LC_COLLATEsequence by default;sort -udeduplicates as it sorts; the manual's sort page is the reference for the setup sort this guide times. - GNU Grep 3.12 manual — read 2026-11-09:
-Finterprets patterns as fixed strings,-xselects only matches on the whole line,-fobtains patterns from a file, one per line. Box runs grep 3.11; every cited behavior was exercised by the probe itself. - BashFAQ/036 — read 2026-11-09: the canonical recipe —
comm -12 <(sort file1) <(sort file2)for intersection,grep -xF -f file1 file2as the alternative, with the 20-second anecdote for a 30k-line file against a 400k-line file; thesort|uniq -drecipe carries the precondition "(Assuming each of file1 or file2 does not have repeated content)"; the-ffile "will be loaded into memory, so it doesn't scale for very large files." - tutorialpedia, Fast Way to Find Lines in One File Not in Another — read 2026-11-09: the nearest published table, qualitative only — speed and memory as categories ("Fast (small B)", "Slow (large A+B)"), no timings, no reps, no line counts; its subject is set difference, not intersection, and its duplicate warning is for
uniq -u, not theuniq -dtrap this guide measures. - copyprogramming, Bash Script to Extract Same Lines from 2 Different Files — read 2026-11-09: the nearest published timing table, but the methods were timed on different corpus sizes — comm on 1 GB at 3.2 s, awk on 1 GB at 5.8 s, grep -F on 500 MB at 12.4 s, bash arrays on 100 MB at 8.5 s — with no reps, no line counts, and no stated hardware; this guide's table is the same question on one shared corpus, three reps per arm, with the outputs verified equal.
- Veldica, Pulling One Field Out of a Million JSON Lines: jq vs Python vs grep, Measured — the sibling guide: the same grep engine on the same box, where grep is the fastest of the three at 0.349 seconds on flat NDJSON — the contrast that proves grep's 2.60 seconds here is the pattern load, not the matching; its nested-key trap is the same failure shape as this guide's
uniq -dtrap, a silent extra line.
Keep Exploring
Paste a [Service] section and get a plain-language verdict on every sandboxing setting it finds.
Paste sshd -G output and get a hardening verdict on the values your server actually runs.
Paste a server block and get WebSocket, TLS, HSTS, and proxy-header verdicts.
Paste a systemd timer expression and get the specific reason it fails, the normalized form, and the past-date traps the built-in check hides.
The full sysadmin hub \u2014 systemd, cgroups, file tools, and measured Linux comparisons.