Pulling One Field Out of a Million JSON Lines: jq vs Python vs grep, Measured
Three tools pull one field out of a million JSON log lines. grep does it in 0.35 seconds, jq in 1.49, Python in 1.78 — grep is more than four times faster than either parser. Then one nested key with the same name adds 1,000 phantom lines to grep's output, and the speed table stops being the whole answer.
Written November 9, 2026 · Tested with jq 1.529 · 12 min read
- The table: one field, one million lines
- grep is faster — and wrong on nested keys
- The pattern bug that shipped in version one
What this guide is
A log file with a million JSON lines, one object per line. You want one field out of every line — the level, info or warn or error — as a plain list. Three tools do the job from a shell prompt: jq with -r '.level', a short Python script that calls json.loads on each line, and grep with -oP and a pattern that matches the key and keeps only the value.
Every tutorial page tells you jq is the right tool for JSON and grep is the wrong one. That advice is about correctness, and it is right as far as it goes. This guide measures both halves: how fast each tool is on the same million lines, and where the fast one quietly gives you the wrong answer.
The measurement ran twice, on two nights, on the same box, with the same probe. The speed table is the same both nights. So is the failure.
The three tools, briefly
jq reads a stream of JSON values and runs a filter over each one. .level is the field-access filter: it takes each object and returns the value under the key level. The -r flag writes a string result straight to standard output, without the JSON quotes around it. Without -r you get "info" with quotes; with -r you get info.
Python reads the file line by line. json.loads turns one line of JSON text into a Python dict, and ["level"] reads the field. The script is eight lines. The point of using a small script instead of python -m json.tool is that json.tool validates and pretty-prints a document. It does not extract a field. The honest Python arm is json.loads in a loop, so that is the arm this guide measures.
grep does not parse anything. It scans bytes for a pattern. The pattern used here is '"level": ?"\K[^"]*'. Read it left to right: the literal text "level":, then an optional space (the ?), then an opening quote, then \K, then [^"]*, every character up to the next quote. \K is a PCRE feature that sets the reported start of the match to that point, so -o, which prints only the matched part of each line, prints the value and not the key. -P tells grep to read the pattern as a Perl-compatible regular expression; without it, \K means nothing.
That difference — parsers that understand the document, a scanner that matches bytes — is the whole guide. The parsers pay to build a data structure for every line and throw most of it away. The scanner pays nothing for structure and gets the speed for it. The scanner also cannot tell a top-level key from a nested key with the same name, and that is the price.
The table: one field, one million lines
The clean corpus is 1,000,000 lines, 130,358,499 bytes, each line one compact JSON object with the keys ts, level, service, user_id, latency_ms, and msg. The field extracted is the top-level level. Medians of three runs, fresh tonight:
- grep
-oP: 0.349 seconds — about 374 MB per second. Reps 0.349, 0.348, 0.350. - jq
-r '.level': 1.492 seconds — about 87 MB per second. Reps 1.492, 1.496, 1.471. - Python
json.loadsper line: 1.783 seconds — about 73 MB per second. Reps 1.845, 1.783, 0.518.
Ratios from the medians: grep is 4.28 times faster than jq and 5.11 times faster than Python; jq is 1.20 times faster than Python.
All three arms returned code 0 on every rep, all three produced exactly 1,000,000 lines, and all three outputs are byte-identical — the same sha256 on every rep, the full value recorded in the probe log. On clean input the three tools agree completely, and grep is four to five times faster.
The night before, the same probe on the same box gave grep 0.350, jq 1.529, Python 1.784 — ratios 4.37x and 5.10x, jq 1.17x faster than Python. Every direction held. The absolute numbers moved by a few percent; the ranking did not move at all.
Two reps in tonight's run are outliers, and both are reported, not dropped. The first rep of the trap and nested corpora ran slow for every arm — Python 2.326 and 2.493 seconds where its other reps are about 0.20, jq 0.507 where its other reps are about 0.15, grep 0.257 where its other reps are 0.038. Those are the first reads of freshly written files, cold page cache, and they hit the first rep of each corpus in order. One Python rep on the clean corpus ran fast instead: 0.518 seconds against 1.783 and 1.845, with byte-identical output. That one has no clean explanation — warm cache does not explain it, since the rep before it was also warm — so it is reported as an unexplained fast outlier. Medians are immune to single-rep outliers in either direction, which is why this guide reports medians and prints every rep.
grep is faster — and wrong on nested keys
Speed is not the whole answer, because the input is not always flat. Two more corpora test that, each 100,000 lines with 1 percent poisoned:
- Trap corpus: 1 percent of lines carry a fake pair inside the message string — the
msgvalue itself contains the text"level": "error". Medians: jq 0.154, Python 0.208, grep 0.039. All three outputs identical, 100,000 lines each. The trap does not break grep, and the next section says why. - Nested corpus: 1 percent of lines carry a nested object,
{"meta": {"level": "debug", "src": "nested"}}. Medians: jq 0.154, Python 0.202, grep 0.038. jq and Python return 100,000 lines. grep returns 101,000 lines — one extradebugfor every poisoned line — and a different sha256 from the parsers, both full values recorded in the probe log.
The nested case is the real failure. grep matches the pattern wherever the bytes appear. A nested "level" key is the same bytes as a top-level one, so grep returns it too, and nothing in the output marks which values came from where. The 1,000 extra lines are silent: grep exits 0, prints no warning, and the output looks exactly like a correct answer with 1,000 more rows than the input has lines.
The tell is the line count. The input has 100,000 lines; a correct extraction has 100,000 values. Count both sides every time:
wc -l logs.ndjson levels.txt
If the counts differ, the extraction is wrong, and the extra lines are the diagnosis. That check costs nothing and it is the only thing standing between the speed table and a corrupted downstream file. Teach the check with the table, not after it.
The escaped-quote trap that does not break grep
"level": "error" pair sitting inside a line. It does not, and the reason is JSON string escaping. The fake pair lives inside the msg string value, so in the file it is written as \"level\": \"error\" — backslash, quote, level, backslash, quote. The grep pattern needs a literal quote immediately before level, and the backslash sits in that spot. The pattern never matches the fake pair.That is a finding about this input shape, not a safety certificate for grep. It says the escaped-quote poison is safe, not that grep is safe in general. A poison that is not inside a string value — a nested object, a second top-level key, a key order change that puts another "level" first — is matched. The trap corpus proves one specific poison harmless; the nested corpus proves the general case is not.
The pattern bug that shipped in version one
The first version of the probe had a bug in the grep arm, and it is worth reporting because it is the exact mistake a reader will make once. The v1 pattern was '"level": "\K[^"]*' — a space after the colon, because that is how pretty-printed JSON looks. The generator writes compact JSON, "level":"info", with no space. grep matched zero lines on both corpora, exited 1, and produced an empty file. The fix is one character: ?, an optional space, so the pattern matches both spacings.
A second slip happened while fixing the first: an intermediate edit changed the value class to [^" ]*, which stops at spaces and would truncate any value containing one. It was corrected to [^"]* before the re-run. Both slips are silent in the same way the nested failure is silent: grep exits, prints nothing alarming, and the output is wrong or empty. The defense is the same in all three cases — check the line count and the first few lines of output before trusting the file.
What the numbers do not say
- Only one field was extracted, and it is a short string. Numbers, nested values, and arrays need different jq filters and different Python code, and the grep pattern would need rewriting for each. The speed ratios are for this shape only.
- The parsers read the whole object and discard five fields out of six. A jq filter that uses more of the object, or a Python loop that keeps the dict, pays the same parse cost for more value. grep's advantage is largest exactly when you want the least out of each line.
- No memory was measured. jq and Python hold one object at a time here, but the probe did not record peak memory, so this guide claims nothing about it.
- One thread per arm, warm page cache except for the first rep of each corpus, no CPU pinning, 32 CPUs. A loaded machine or a cold cache moves the absolute numbers; the ranking held across two nights, which is the part that should travel.
- The timings grow with the input: 100,000 lines take grep 0.038 seconds, 1,000,000 take 0.349 — about ten times the input, about nine times the time. That is a scanner's linear shape, and it is why grep's lead grows with file size rather than shrinking.
- The nearest published tables measure different tool sets. A fastgron benchmark runs a path query on a single 190 MB JSON document: fastgron 0.176 seconds, jj 0.972, jq 3.252. A purejq benchmark compares its CLI against the jq binary on a 93 MB file: single lookup 0.51 seconds against 1.68. Neither table is the plain trio — jq,
python -c,grep -oP— on NDJSON, which is the table this guide adds.
Checklist
- Before pulling a field out of NDJSON with grep, check the shape: one top-level key, no nested object with the same name, or the output will have extra lines and no warning.
- Before trusting a grep extraction, count the lines:
wc -lon the input and on the output must agree, and a mismatch is a wrong answer, not a rounding error. - Before writing the pattern, check the actual bytes: compact JSON has no space after the colon, pretty-printed JSON does, and a pattern that assumes one spacing matches zero lines on the other.
- Before choosing on speed, check the table: grep is 4.28 times faster than jq and 5.11 times faster than Python on a million flat lines, and jq is 1.20 times faster than Python.
- Before choosing on correctness, check the nested case: the parsers return the top-level value only, grep returns every value with that key name, and the extra 1,000 lines per 100,000 are silent.
- Before calling the escaped-quote trap safe, check what it proves: a fake pair inside a string value is escaped and never matches; a real nested key is not escaped and always matches.
- Before reusing these numbers, check the setup: WSL2, 32 CPUs, jq 1.7, Python 3.12.3, GNU grep 3.11, warm page cache, one thread per arm — the ranking held across two nights, the absolute seconds are this box's.
Related guides on this site
- Numbering a Million Lines, Measured: cat -n vs nl vs awk vs grep -n
- Which Lines Are in Both Files? comm vs grep vs sort|uniq vs Python, Measured
- Finding Lines by Prefix: look vs grep vs rg vs awk vs sed, Measured
- grep -F vs -E vs -P, Measured: Same Pattern, Three Engines, One Silent Trap
Sources
- jq manual, jqlang — read 2026-11-09: a jq program is a filter run over a stream of JSON values;
-rwrites a string result directly to standard output instead of formatting it as a JSON string. - GNU Grep 3.12 manual — read 2026-11-09:
-oprints only the matched parts of matching lines, each on its own line;-Pinterprets patterns as Perl-compatible regular expressions, documented in the pcre2syntax and pcre2pattern manual pages. - PCRE2, pcre2syntax — read 2026-11-09:
\Ksets the reported start of the match, under "REPORTED MATCH POINT SETTING". - Python, json — JSON encoder and decoder — read 2026-11-09:
json.loadsdeserializes a string containing a JSON document to a Python object; thepython -m json.toolcommand-line interface validates and pretty-prints a document, it does not extract a field. - fastgron, Performance Benchmarks (DeepWiki) — the nearest published table, a different tool set: a path query on a single 190 MB citylots.json document, fastgron 0.176 s, jj 0.972 s, jq 3.252 s.
- purejq, PyPI — the nearest published table, a different tool set: purejq CLI against the jq 1.8.1 binary, single lookup on a 93 MB file of 1M objects, 0.51 s against 1.68 s.
- Veldica, Regex Catastrophic Backtracking, Measured — the sibling guide: the same grep -P engine, and why a nested-quantifier pattern can cost 100x; the pattern in this guide has no nested quantifier, and its timings grow with the input size.
Keep Exploring
Paste a [Service] section and get a plain-language verdict on every sandboxing setting it finds.
Paste sshd -G output and get a hardening verdict on the values your server actually runs.
Paste a server block and get WebSocket, TLS, HSTS, and proxy-header verdicts.
Paste a systemd timer expression and get the specific reason it fails, the normalized form, and the past-date traps the built-in check hides.
The full sysadmin hub \u2014 systemd, cgroups, file tools, and measured Linux comparisons.