Command line

Finding Lines by Prefix: look vs grep vs rg vs awk vs sed, Measured

Six ways to find the same 1,000 lines by prefix in the same million-line file. look is the fastest arm by a wide margin, and it earns that speed by seeking instead of scanning — but it only works on sorted input, it fails silently when the input is not sorted, and the binary-search flag the BSD manual documents does not exist in the Linux build.

Written November 10, 2026 · 13 min read

By the end of this guide
  • The table: look wins by seeking, not scanning
  • The closest published work has no look arm
  • Two manuals, two looks, and the flag that does not exist
  • The sorted input that look demands

What this guide is

One file, one prefix, six arms, three reps each, run twice: once the night before and once fresh at build time, on the same box, so every number below is one of two medians from the same probe. The file is 1,000,037 lines of userNNNNN names, about 17 MB, sorted under LC_ALL=C, sha256 bdfbebad…. The query is the prefix user042, which matches exactly 1,000 lines, and every timed output was checked against a file Python computed independently from the same corpus.

The six arms are the same question written six ways: look on the sorted file, rg with an anchored pattern, grep with an anchored pattern, grep -F with the same string unanchored, sed -n with an anchored print, and awk with an anchored pattern. The question is the one people ask before searching a sorted word list, a passwd-style file, or a log they already sorted: is the specialist tool actually faster than grep once the file is sorted, and what does it cost when the file is not sorted at all. The manual answers the second part in one sentence, and the probe checks it by running look on the shuffled file rather than taking the manual's word for it.

The answer is a correction in two directions. look is 6.8 times faster than grep and 3.9 times faster than rg on the same 1,000-line answer, 0.0013 seconds against 0.0088 and 0.0051 fresh, and 79.9 times faster than awk. But that speed is bought with a hard requirement: on the shuffled copy of the same file, look returns rc=1 with 0 lines and empty stderr, a silent wrong answer for lines that are present. And the flag that is supposed to turn the search on is not there at all: the BSD manual documents -b, --binary, and tutorials show look -bf, but the installed util-linux 2.39.3 build answers look: invalid option -- 'b'.

The table: look wins by seeking, not scanning

Wall seconds, medians of three runs. Every rep returned rc=0, every timed output matched the independently computed expected file, 1,000 lines on every anchored rep, sha256 ceb67628…, and 1,037 lines on every unanchored rep, sha256 c97c28c0….

The corpus: 1,000,037 lines, user000000 through user999999 plus 37 planted poison lines of the form zz user042NNN tail-NN, sorted with LC_ALL=C sort. The query prefix is user042; the 1,000 matching lines are user042000 through user042999.

  • look user042 words.txt: banked 0.0013 (0.0012 to 0.0013), fresh 0.0013 (0.0011 to 0.0014). Output byte-identical to the expected file on all three reps of both runs.
  • rg '^user042' words.txt: banked 0.0055 (0.0047 to 0.0055), fresh 0.0051 (0.0050 to 0.0524). Output byte-identical to the expected file on all three reps of both runs.
  • grep '^user042' words.txt: banked 0.0082 (0.0081 to 0.0084), fresh 0.0088 (0.0077 to 0.0101). Output byte-identical to the expected file on all three reps of both runs.
  • grep -F 'user042' words.txt (unanchored): banked 0.0089 (0.0084 to 0.0103), fresh 0.0090 (0.0089 to 0.0093). Output byte-identical to the unanchored expected file, 1,037 lines, on all three reps of both runs.
  • sed -n '/^user042/p' words.txt: banked 0.0471 (0.0470 to 0.0508), fresh 0.0503 (0.0480 to 0.0512). Output byte-identical to the expected file on all three reps of both runs.
  • awk '/^user042/' words.txt: banked 0.1010 (0.0996 to 0.1032), fresh 0.1039 (0.1004 to 0.1094). Output byte-identical to the expected file on all three reps of both runs.

Against look, the other arms cost 3.92 times for rg and 6.77 for grep fresh, 4.2 and 6.3 banked; sed costs 38.7 times fresh and 36 banked; awk costs 79.9 times fresh and 78 banked. The comparison that matters between the scanners is rg against grep: rg is 1.73 times faster fresh and 1.49 banked on the same anchored pattern, and awk is 2.07 times slower than sed fresh and 2.14 banked.

One thing to read carefully in the table is what the 0.0013-second look cell actually means. A 17 MB file cannot be scanned in 1.3 milliseconds, so the time is evidence of a seek, not a scan: look jumps to the matching region instead of reading every line. The 6.8-times gap over grep is real on this corpus, but the absolute gap is 7.5 milliseconds, which is irrelevant on a small file and decisive only at scale or in a loop. The ranking (look, then rg, then grep, then sed, then awk) is the stable claim across both runs; the exact ratios are one machine. The rg fresh rep 1 is 0.0524 seconds, ten times its own median, a cold-start spike the other two reps do not show; the grep fresh rep 2 is 0.0101 against a median of 0.0088.

The closest published work has no look arm

The closest published work times the same three scanners on a different job and never runs look. The ComputingForGeeks piece on grep vs awk vs sed times column extraction on a 2-million-line access log: pulling the client IP column takes 0.18 seconds with cut, 0.55 with awk, 2.8 with sed and 10 with grep -o. Its job is extracting a field from every line, not finding lines by prefix, and its slowest arm, grep -o, is slow precisely because it runs the regex engine across every byte to print only the matching text.

The ripgrep-vs-grep pieces are the same shape in the other direction: they time recursive searches across a codebase or a 10 GB log, where the question is which lines match anywhere, and rg wins by 5 to 20 times. Those numbers do not transfer here, because the question is different: on a single sorted file with an anchored prefix, rg beats grep by only 1.73 times fresh, and look beats them both by seeking. A guide that quotes a codebase-search ratio for a sorted-file lookup is quoting the wrong job.

Two manuals, two looks, and the flag that does not exist

The confusion around look starts before the timing, in the manuals. There are two look utilities with the same name and different behavior, and the web mixes them freely.

The BSD manual, the one Debian ships as bsdmainutils, documents -b, --binary, "Use a binary search on the given word list", and then says, in its compatibility section, "look uses a linear search by default instead of a binary search, which is what most other implementations use by default." Its synopsis is look [-bdf] [-t termchar] string [file ...]. Tutorials written against that manual show look -bf as ordinary usage.

The Linux manual, the one util-linux 2.39.3 ships, has no -b flag at all. Its options are -a, -d, -f, -t, -h, -V, confirmed by look --help and by the Ubuntu man page for the same util-linux source. Its description says, "As look performs a binary search, the lines in file must be sorted", with no linear-search line anywhere on the page. Running look -b user042 words.txt answers look: invalid option -- 'b'.

So the correction runs in a specific direction, and the guide states it plainly. The installed Linux binary always searches by seeking, which is why its median is 0.0013 seconds on a 17 MB file, and the BSD manual's "linear search by default" line describes the BSD build, not this one. A reader who copies look -b from a BSD tutorial onto a Linux box gets an error, not a faster search, and a reader who reads the BSD manual's linear-search line and concludes look scans will not understand why it returns nothing on unsorted input. The binary's behavior, not either manual's framing, is what this guide reports, and the two manuals are cited side by side so a reader can see which one their system ships.

One correction to the banking run belongs in the guide itself. The banked file said the installed manual says "linear search by default". That is wrong: the line is in the BSD manual, and the installed util-linux manual says the opposite, "As look performs a binary search". The banking run's probe was right and its citation was wrong; the guide corrects the citation and keeps the measurement, because the measurement never depended on which manual said what.

The sorted input that look demands

The trap file is the same corpus with its lines shuffled, fed straight to look. look user042 words_shuffled.txt exits rc=1 after 0 lines, with empty stderr:

Terminal
$ look user042 words_shuffled.txt
$ echo $?
1

There is no error, no warning, and no partial output. The 1,000 matching lines are all present in the file, and look reports that none of them are there. The same shuffled file through grep returns the full 1,000-line set, because grep scans. The failure is silent and total: a script that feeds look an unsorted file gets "not found" for lines that are present, and nothing in the output distinguishes that from a genuinely absent prefix.

The practical rule follows from the trap, not from the manual: sort the file first, under the same collation look expects, or do not use look. The probe sorts with LC_ALL=C sort, and the sorted-input check (LC_ALL=C sort -c) passes on the corpus on every run. The BSD manual's bugs section says input files must be sorted with LC_COLLATE set to C, and the probe follows that. If the file was sorted under a different collation than look expects, the same silent failure can appear, because the seek lands in the wrong region. This probe never tested a mismatched collation, so the guide does not claim a number for it; it states the rule and the check.

The anchor that grep -F drops

The trap pair is the anchored arm against the unanchored arm on the same corpus. grep '^user042' words.txt returns 1,000 lines. grep -F 'user042' words.txt returns 1,037 lines, and the extra 37 are exactly the planted poison lines, zz user042NNN tail-NN, which contain the prefix mid-line. The two outputs are otherwise identical; the anchor is the whole difference between 1,000 and 1,037.

The practical rule is the one the ComputingForGeeks piece states for its own log: a plain search prints every line that contains the pattern anywhere, so when a term can appear in more than one position, anchor the pattern or test the field directly. Here the cost of dropping the anchor is 37 extra lines out of 1,000, about 4 percent, and on a real log the poison lines would be the ones that look plausible: a user id appearing in a message body, a path, or a comment. The grep -F arm is also no faster than the anchored grep arm, 0.0090 against 0.0088 fresh, so dropping the anchor buys nothing in speed and costs correctness.

The case that look drops

The trap pair is the same query in two cases. look USER042 words.txt returns rc=1 with 0 lines, because look is case-sensitive by default. look -f USER042 words.txt returns the full 1,000-line set, byte-identical to the lowercase query, sha256 ceb67628…. The -f flag is the same flag on both manuals, and it is on by default only when no file is given, when the dictionary file is used instead.

The practical rule is to check the case before the prefix, not after a zero result. A look that returns nothing can mean the prefix is absent, the file is unsorted, or the case differs, and the three failures look identical from the outside. The probe separates them: the unsorted trap and the uppercase trap both return rc=1 with 0 lines, and only the -f rerun distinguishes the case failure from the other two.

Checklist

  • Before using look, check whether the file is sorted under LC_ALL=C: look on unsorted input returns rc=1 with 0 lines and empty stderr, a silent wrong answer, not an error.
  • Before copying look -b from a tutorial, check which look your system ships: util-linux 2.39.3 has no -b flag and answers invalid option, while the BSD manual documents -b and says linear search is the default.
  • Before quoting a look-to-grep ratio from this guide, check which case it is: 6.8 times on this corpus, but the absolute gap is 7.5 milliseconds, so the ranking matters more than the ratio on small files.
  • Before trusting a grep -F count, check whether the pattern is anchored: the unanchored arm returns 1,037 lines against 1,000 anchored, exactly the 37 planted poison lines, at no speed saving.
  • Before trusting a look that returns nothing, check the case: look USER042 returns 0 lines rc=1, while look -f USER042 returns the full 1,000-line set byte-identical to the lowercase query.
  • Before reusing these numbers, check the setup: WSL2, 32 CPUs, look util-linux 2.39.3, GNU grep 3.11, ripgrep 14.1.0, gawk 5.2.1, GNU sed 4.9, one thread per arm, 1,000,037 lines, about 17 MB, one prefix query, outputs written to tmpfs files, warm page cache.

Sources

  • look(1) Linux man page, util-linux — read 2026-11-10: "As look performs a binary search, the lines in file must be sorted"; options -a, -d, -f, -t, -h, -V, no -b flag; the same wording is in the installed util-linux 2.39.3 manual on the build box. The binary-search sentence this guide quotes.
  • look(1) BSD man page, bsdmainutils — read 2026-11-10: synopsis look [-bdf] [-t termchar]; -b, --binary, "Use a binary search on the given word list"; "look uses a linear search by default instead of a binary search, which is what most other implementations use by default"; input files must be sorted with LC_COLLATE set to C. The other look, the source of the -b confusion.
  • ComputingForGeeks, grep vs awk vs sed: Examples and Benchmarks — read in full 2026-11-10: on a 2-million-line access log, pulling the client IP column takes 0.18 seconds with cut, 0.55 with awk, 2.8 with sed and 10 with grep -o; a plain search prints every line that contains the pattern anywhere, which is why its admin search returns four lines instead of three. The closest published work; a different job with no look arm.
  • Veldica, comm vs grep vs sort|uniq vs Python: Set Intersection, Measured — the sibling guide: set intersection on one column, keys only, no merged fields and no prefix search. This guide searches one file by prefix; it does not re-teach the intersection table.
  • Veldica, join vs awk: Keyed Inner Join, Measured — the sibling guide: merging two files on a shared key, with the same sorted-input demand and the same loud failure on unsorted input. This guide searches one file by prefix; it does not re-teach the join table.

Keep Exploring