Cutting a Million-Line File Into Ten, Measured: split vs csplit vs awk
The same million-line file splits into ten chunks in 0.031 seconds with split and 0.31 with awk, but the speed table is the least interesting part. The round-robin form scrambles the order, the byte form makes an eleventh chunk out of one newline, and csplit hands you an empty first file when the pattern matches line 1.
Written November 10, 2026 · 14 min read
- The table: the four split arms tie; csplit and awk cost real time
- split -n l/10 balances bytes, not lines
- When to use awk anyway
What this guide is
One file, six arms, three reps each. The file is a million lines of log text, 77,333,891 bytes, with a marker line ---CHUNK--- at lines 100001, 200001, and so on through 900001. Every arm cuts the same file into ten chunks, and every cut is checked the same way: count the lines in each chunk, add them up, and concatenate the chunks in order to see whether the result is byte-identical to the source. A chunk count can be right while the order is wrong, and a line count can be right while the file is cut in the wrong place. The checks catch both.
The question is the one people ask before feeding a big log to ten parallel workers: which tool cuts it, how fast, and what breaks when the cut is not what you assumed. The four split arms tie at roughly 0.03 to 0.06 seconds, so speed does not decide among them. What decides is what each form means: line count, byte-balanced lines, round-robin interleaving, raw bytes, a pattern, or a script. Three of the six arms have a trap that changes the answer, and the traps are the guide.
The table: the four split arms tie; csplit and awk cost real time
Wall seconds, medians of three runs, fresh tonight. Every rep returned rc=0, and every arm except the round-robin one reassembled byte-identically.
split -l 100000: fresh median 0.0425, reps 0.0671, 0.0425, 0.0350. Night before: 0.0370, reps 0.0361, 0.0370, 0.0377. Ten chunks, 100,000 lines each, reassembly byte-identical.split -n l/10: fresh median 0.0305, reps 0.0304, 0.0305, 0.0310. Night before: 0.0323, reps 0.0322, 0.0323, 0.0326. Ten chunks, 99,995 to 100,004 lines each, reassembly byte-identical.split -n r/10: fresh median 0.0637, reps 0.0637, 0.0631, 0.0637. Night before: 0.0643, reps 0.0637, 0.0643, 0.0808. Ten chunks, 100,000 lines each, ordered reassembly DIFFERS. The lines are interleaved.split -b 7733389: fresh median 0.0317, reps 0.0313, 0.0317, 0.0325. Night before: 0.0320, reps 0.0319, 0.0320, 0.0326. Eleven chunks, not ten. Reassembly byte-identical.csplit '/^---CHUNK---$/' '{8}': fresh median 0.1694, reps 0.1690, 0.1694, 0.1724. Night before: 0.1753, reps 0.1744, 0.1753, 0.2141. Ten chunks, 100,000 lines each, reassembly byte-identical.awkchunk by line number: fresh median 0.3052, reps 0.3051, 0.3052, 0.3087. Night before: 0.3104, reps 0.3070, 0.3104, 0.3547. Ten chunks, 100,000 lines each, reassembly byte-identical.
csplit costs 3.99 times split -l tonight (0.1694 against 0.0425) and awk costs 7.18 times (0.3052 against 0.0425). The night before the same ratios were 4.74 and 8.39, because split -l itself moved from 0.0370 to 0.0425, with a first-rep spike to 0.0671 tonight. The exact ratios are one machine's; the gap is not. awk costs 1.80 times csplit on both nights (1.77 the night before), the most stable ratio in the table. The split arms sit near the spawn-and-read floor for a 77 MB file, so the ratios among them are order-of-magnitude, while the csplit and awk gaps are the stable claims.
split -n l/10 balances bytes, not lines
The line-preserving -n form is the one people reach for when they want ten equal chunks without counting lines first, and the manual explains why it cannot deliver exactly that. For l mode, chunks are approximately input size divided by n: the input is partitioned into n regions of approximately equal size, and if a line starts within a partition it is written completely to the corresponding file. Lines are never split, so the files come out larger or smaller than the partition size. The counts prove it: 100000, 100004, 100002, 99998, 99997, 99999, 100000, 100003, 100002, 99995, identical on all three reps on both nights. The form balances chunk bytes, not line counts, and anyone asserting ten equal chunks from -n l/10 is off by up to five lines per chunk.
That is a small error on a million-line file, and it is deterministic, which is what makes it easy to miss. A script that assumes chunk 3 holds exactly lines 200001 to 300000 will be wrong by a few lines at each boundary, and the error is silent: every chunk is a valid file, every line is present exactly once, and only the boundary positions are off. If the chunks feed workers that each process a line range, the ranges overlap or gap by a few lines. Use -l 100000 when the line counts must be exact, and -n l/10 when the chunk sizes must be roughly equal and a few lines either way do not matter.
One published FAQ gets this backwards, calling -n l/n round-robin. It is not. The manual is explicit: l/n generates n files without splitting lines or records, and r/n is like l but uses round-robin distribution. The next section is what round-robin actually does.
The round-robin trap: -n r/10 scrambles your log
split -n r/10 produces ten chunks of exactly 100,000 lines each, and concatenating them in order does not reproduce the source. The lines are interleaved: chunk 1 holds lines 1, 11, 21, and so on, chunk 2 holds lines 2, 12, 22, and the pattern continues through all ten chunks. The sorted content of all ten chunks matches the sorted source exactly, so nothing is lost and nothing is duplicated. The order is gone.The manual warns about this in the same breath as it defines the form: concatenating the output files in sorted order by file name produces the original input file, except for -nr. That exception is the whole point of the form. Round-robin exists to distribute lines evenly across parallel workers, so that each worker gets a representative sample of the whole file rather than one tenth of it. A worker that counts errors per chunk gets a fair share either way. A worker that assumes its chunk is a contiguous slice of the log, or a script that reassembles the chunks with cat and expects the original file back, gets a scrambled file that still has exactly a million lines.
The cost is not speed. Round-robin at 0.0637 seconds is about twice the line-preserving form at 0.0305, and still 2.7 times faster than csplit. The cost is the assumption. Before using -n r/10, check whether anything downstream needs the chunks in order. If it does, use -n l/10 or -l. If the chunks feed independent workers that never reassemble, round-robin is the better distribution, and the table shows it costs almost nothing.
The byte-split trap: split -b makes eleven chunks
split -b 7733389 divides 77,333,891 bytes by ten, and 77,333,891 is not divisible by ten. The file's final newline is one byte past the tenth boundary, so the command makes an eleventh chunk containing exactly that byte: one line, one newline, nothing else. The per-chunk line counts are 100000, 100005, 100003, 99999, 99998, 100000, 100001, 100004, 100003, 99996, 1, identical on all three reps on both nights. Counting lines per chunk the naive way gives a total of 1,000,010, because ten lines are cut mid-line and each cut line is counted once in each of the two fragments it spans. Counting newlines gives exactly 1,000,000, and ordered reassembly is byte-identical.The eleventh chunk is the trap, and the fragment counting is the trap behind it. A script that counts lines per chunk to verify the split will report 1,000,010 lines and conclude the file grew, or that ten lines duplicated. Neither happened. A script that expects exactly ten output files will find eleven and either crash or silently drop the last one, which is a single newline, so dropping it looks harmless until the next file in the pipeline expects the line count to match. And the mid-line cuts are real: chunk boundaries fall inside lines, so no chunk except by luck contains whole records. Byte splitting is for binary files, archives, and uploads, where a cut anywhere is fine and reassembly is a plain concatenation. For text, use -l or -n l/10, or the line-bytes form -C, which packs as many complete lines as fit without exceeding the size.
The csplit trap: a pattern on line 1 leaves an empty first file
csplit splits on content, not on counts: the pattern /^---CHUNK---$/ with {8} repeats makes ten sections of 100,000 lines each, and the marker lines themselves start each new section. That is the right tool when the split points are in the data, a section header or a date stamp, rather than in the arithmetic. It costs 0.1694 seconds against 0.0425 for split -l, about four times as much, because it reads the file looking for matches instead of counting.The trap is what happens when the pattern matches the first line. A pattern that matches line 1 produces a first output file of zero bytes: the section before the first match is empty, and csplit creates the file anyway. The probe ran it on a five-line file whose first line is the marker, and the first output file came back at 0 bytes while the rest held the sections. A script that assumes every chunk is non-empty, or that feeds every chunk to a parser that rejects empty input, chokes on a file that csplit produced correctly by its own rules.
The manual names the fix, and it names the trap in the same paragraph: the -z flag suppresses the generation of zero-length output files, and the manual notes that where section delimiters are supposed to mark the first lines of each section, the first output file will generally be a zero-length file unless you use this option. Add -z when the pattern can match line 1, or when a log starts with a header that is itself a delimiter. The flag is also the reason the copyprogramming piece recommends -z for sparse logs. The empty file is not a bug, and the flag is not a workaround. It is the documented behavior, and the guide states it as such.
When to use awk anyway
awk is the slowest arm in the table at 0.3052 seconds, 7.18 times split -l, and it produces exactly the same ten chunks that split -l produces for a tenth of the time. On this job there is no reason to pay for it. The reason to use awk is a job split cannot express: chunks by value, not by count. Splitting a log into one file per service, or per date, or per error level, needs a condition on the line, and awk is the tool that evaluates it. The close-command trap in that job is a different question, covered by the awk close-command pieces, and this guide does not re-teach it. The rule from the table is simple: if the split is by count, bytes, or pattern, use split or csplit. If the split is by what a line says, use awk, and pay the 7 times knowingly.
The same rule separates this guide from the sibling comm-versus-grep guide, which asks a set question about two files, and from the paste-versus-join guide, which merges two files side by side. Same corpus scale, same harness, different operations. This guide cuts one file into ten. It does not merge, intersect, or compare.
Checklist
- Before splitting a log for parallel workers, check whether anything downstream needs the chunks in order: if it does, never use
split -n r/10, which interleaves the lines and breaks ordered reassembly even though every chunk has exactly 100,000 lines. - Before asserting ten equal chunks from
split -n l/10, check the counts: it balances chunk bytes, not lines, and the chunks came out 99,995 to 100,004 lines on both nights, identical every rep. - Before counting lines per chunk after
split -b, count newlines instead: ten lines are cut mid-line and counted twice as fragments, so the line-count total is 1,000,010 while the newline total is exactly 1,000,000, and there is an eleventh chunk holding one newline. - Before feeding csplit output to a script, check the first file: if the pattern matches line 1, the first output file is 0 bytes, and
-zsuppresses it. - Before choosing between split and csplit on speed alone, check the table: csplit costs 3.99 times
split -ltonight and awk 7.18 times, so the choice is what the split means, not which is faster. - Before quoting a ratio, check the range:
split -lspiked to 0.0671 seconds on its first rep tonight against a 0.0370 median the night before, so the csplit and awk ratios moved with it while the gaps stayed. - Before reusing these numbers, check the setup: WSL2, 32 CPUs, GNU coreutils 9.4, GNU Awk 5.2.1, tmpfs, warm page cache, one thread per arm, one million-line corpus, sha256
a1e51175….
Related guides on this site
- Copying, Moving, and Splitting Files: Ten Measured Guides
- Numbering a Million Lines, Measured: cat -n vs nl vs awk vs grep -n
- Editing a File In Place, Measured: sed -i vs perl -pi vs awk
- Copying a Sparse File, Measured: Which Tools Preserve the Holes
Sources
- GNU Coreutils, split invocation — read 2026-11-10:
-n l/ngenerates n files without splitting lines or records, with chunks approximately input size divided by n and lines never split even if they overlap a partition;-n r/nis likelbut uses round-robin distribution; concatenating output files in sorted order reproduces the input except for-nr. Box runs coreutils 9.4, the manual text is 9.12, and every cited behavior was exercised by the probe itself. - GNU Coreutils, csplit invocation — read 2026-11-10: a
/regexp/pattern creates an output file containing the input up to but not including the next matching line;{repeat}repeats the previous pattern;-zsuppresses zero-length output files, and the manual notes the first output file will generally be zero-length when delimiters mark the first lines of sections unless this option is used. Box runs coreutils 9.4, the manual text is 9.10; the probe exercised it on the same corpus. - LinuxHaxor, A Comprehensive Guide to Splitting Files in Linux — read 2026-11-10: publishes the closest measured table, splitting a 10 GB file on an 8-core Dell R720: split -b 1G in 22 seconds, csplit on a pattern in 29 seconds. One run each, no reps, and different jobs (byte-size split versus pattern split), so it is not a same-corpus, same-job head-to-head. This guide's table is the same job on one shared corpus, three reps per arm, with verified outputs.
- copyprogramming, Mastering Unix Split: Divide Files into N Parts or by Line Patterns — read 2026-11-10: recommends
--number=l/nfor line-based splits and csplit with/^PATTERN/{*}for pattern splits, with code examples and no timing table, no corpus, and no reps. Its FAQ mislabels-n l/nas round-robin; the manual says round-robin isr/n, and the probe shows what round-robin actually does to the order. - Veldica, comm vs grep: Set Intersection, Measured — the sibling guide: the same two-files question family, a set operation rather than a split. This guide cuts one file into ten; it does not intersect two files.
- Veldica, paste vs join: Merging Two Files Side by Side, Measured — the sibling guide: merges two files side by side on the same corpus-scale harness. This guide splits one file apart; it does not merge two.
Keep Exploring
Paste a [Service] section and get a plain-language verdict on every sandboxing setting it finds.
Paste sshd -G output and get a hardening verdict on the values your server actually runs.
Paste a server block and get WebSocket, TLS, HSTS, and proxy-header verdicts.
Paste a systemd timer expression and get the specific reason it fails, the normalized form, and the past-date traps the built-in check hides.
The full sysadmin hub \u2014 systemd, cgroups, file tools, and measured Linux comparisons.