Systems Administration

cgroup v2 MemoryMax: What Actually Happens When a Process Hits the Limit, Measured

A process that crosses a 128 MiB MemoryMax cap dies in about 25 milliseconds, with memory.peak exactly at the cap and almost no pressure stall. The same allocator under a 96 MiB MemoryHigh throttle takes about 101 seconds per 16 MiB step instead of about 17 milliseconds, and never gets killed.

Written November 8, 2026 · Tested with systemd 255 · 18 min read

By the end of this guide
  • The hard cap: MemoryMax kills in about 25 milliseconds
  • What the counters say
  • The second process survives the kill, not the unit
  • The soft cap: MemoryHigh stalls, never kills
What you need
The allocator is a small C program that grows its memory in 16 MiB steps, touching every page of each chunk so the memory is charged, and logging the monotonic time and its resident size at every step. Every run puts it in its own systemd user scope with swap disabled, so there is no swap escape: MemorySwapMax is zero in every run.

What this guide is

This guide measures what actually happens when a process hits a cgroup version 2 memory limit, in two halves that the documentation describes but never times. The first half sets a hard cap, MemoryMax at 128 MiB, and times how fast the kernel kills the process that crosses it, reading the counters the kill leaves behind. The second half sets only a soft cap, MemoryHigh at 96 MiB, and times how slow the same allocator becomes above it, with no kill at all. The headline contrast writes itself: the hard cap kills in about 25 milliseconds with about 1 millisecond of stall, while the soft cap never kills and stalls the process about 96 percent of the time.

The nearest existing measurement is a Docker guide that caps a container at 48 MiB and allocates 80 MiB of Python memory inside it, exiting 137 with memory.events showing max 19, oom 1 and oom_kill 1, peaking at 44 MiB. It is a different tool, Docker rather than systemd-run on a plain cgroup, a different cap, and it measures no kill latency, no pressure stall, no MemoryHigh comparison and no second-process test. A second guide runs real builds under 2, 4 and 8 GiB limits and records memory.peak and memory.events per build, with one build killed at 2 GiB in all three runs. It measures builds, not a controlled allocator, and it times wall-clock build duration, not kill latency or throttle rate. A third piece follows a Go service under a 512 MiB container cap, killed 3 times out of 3, then held with a garbage-collector limit. It is Go and garbage-collector specific, with no latency, pressure or High comparison. A fourth guide, on systemd limits, claims a process hitting the hard cap dies in about a second and shows example pressure numbers with no timed experiment behind them. The fresh median here is 25.0 milliseconds, which contradicts that claim by a factor of forty. A fifth is a diagnostic exercise about a checkout service stalled by MemoryHigh, with real counter readings but no timed experiment. None of them runs the pair of experiments this guide runs: the same allocator, the same steps, once against the hard cap and once against the soft cap, with the kill timed and the stall timed.

The setup

The allocator is a small C program that grows its memory in 16 MiB steps, touching every page of each chunk so the memory is charged, and logging the monotonic time and its resident size at every step. Every run puts it in its own systemd user scope with swap disabled, so there is no swap escape: MemorySwapMax is zero in every run. A small sleeper process shares the scope, about 1.9 MiB, so the cgroup holds two processes and the second-process question can be answered. The scope is the only path that works here: creating a cgroup directly under the filesystem root is denied for the unprivileged user, and the user slice is the way in.

Command
systemd-run --user --scope --no-block --unit=cgprobe-<tag> \
  -p MemoryMax=128M -p MemorySwapMax=0 \
  /home/user/cgprobe/run_one2.sh <tag> 512 16

Kill latency is the time from the last successful step log to the first observation that the allocator process has vanished from the process table, polled every 2 milliseconds. It is an upper bound on the true kernel kill time: the kill happens somewhere inside that window, and the observation can only come after it. The counters are read by a background poller running inside the scope, snapshotting the cgroup files to disk every 10 milliseconds. That pattern matters, because when a process in a scope is killed for out-of-memory, systemd fails the scope and tears down the rest of it, so a reader that runs after the kill is itself killed before it can read anything. The poller snapshots survive because they are already on disk.

The environment is WSL2 throughout: kernel 6.18.33.2-microsoft-standard, systemd 255, Ubuntu 24.04.4, gcc 13.3.0, 24,027 MiB of RAM, 32 virtual CPUs. Every number below is a WSL2 number, not a bare-metal number.

The hard cap: MemoryMax kills in about 25 milliseconds

Three fresh repetitions, all with MemoryMax at 128 MiB, all with the allocator wanting 512 MiB in 16 MiB steps. All three died at the same step, 112 MiB allocated, with the resident size at 116,448 to 116,452 KiB, and all three exited 137, which is 128 plus the kill signal. The kill latency figures are upper bounds, measured from the last step log to the process vanishing.

Rep Last step Kill latency Exit
Rep 1 112 MiB 108.8 ms 137
Rep 2 112 MiB 23.7 ms 137
Rep 3 112 MiB 25.0 ms 137

The fresh median is 25.0 milliseconds, with a range of 23.7 to 108.8 milliseconds. The 108.8 millisecond repetition is the slow one; the other two agree at about 24 to 25 milliseconds. The night before gave 74, 28 and 21 milliseconds, median 28 milliseconds. Across both nights, six timed repetitions, the median is 26.5 milliseconds and the range is 21 to 108.8 milliseconds. The kill is tens of milliseconds, not about a second. The guide that claims about a second is wrong by a factor of forty against the fresh median, and by a factor of nine even against the slowest repetition measured on either night.

The peak is the cleanest number in the whole probe. In all three fresh repetitions, memory.peak read exactly 134,217,728 bytes, which is exactly 128 MiB. The night before captured the peak in two runs, and both read exactly the same value. Five captured peaks across two nights, all exactly at the cap. Usage never exceeds the cap, and the peak lands on it to the byte.

The pressure reading is almost as clean. At the kill, memory.pressure showed totals of 1,103, 2,459 and 946 microseconds across the three fresh runs, about 1.1, 2.5 and 0.9 milliseconds of accumulated stall, with the ten-second averages at zero. The night before read 2,313 microseconds, about 2.3 milliseconds. A hard-cap kill arrives with no pressure warning: the process is not stalled, it is dead. Any monitoring that waits for pressure to rise before acting on a hard cap will see nothing until the process is already gone.

What the counters say

The memory.events counters at the kill separate three numbers that are easy to collapse into one. The max counter counts how many times the cgroup was about to go over the cap. The oom counter counts how many times the kernel declared out-of-memory. The oom_kill counter counts processes actually killed. They are not the same number, and the probe shows why.

Rep max oom oom_kill
Rep 1 (fresh) 86 2 1
Rep 2 (fresh) 39 1 1
Rep 3 (fresh) 39 1 1
Night before 99 4 1

The kernel hit the cap 39 to 99 times and declared out-of-memory 1 to 4 times before the single kill. The max and oom counters vary from run to run; oom_kill is exactly 1 in every run on both nights. One kill, however many times the cap was brushed. The kernel log agrees, three fresh lines on this run, each reading Memory cgroup out of memory: Killed process, with the allocator named, total virtual memory 133,796 KiB, anonymous resident memory 127,744 to 128,512 KiB, and the constraint recorded as the memory cgroup. The kernel killed only the allocator in all three fresh runs, and in all six runs the night before.

The resident size at death is below the cap, and that is expected once the accounting is understood. The allocator itself held about 127 to 129 MiB of anonymous memory at the kill, while the cgroup total includes the sleeper at about 1.9 MiB plus page tables and kernel charges, which together bring the total to exactly 128 MiB. The cap bites on the cgroup total, not on one process resident size alone. A reader who watches only the process resident size will see it die below the number they set, and will think the cap fired early. It did not: the cgroup total is the number that matters.

The second process survives the kill, not the unit

The sleeper is the nuance the counters cannot show. In the night-before runs, the sleeper was still resident immediately after the kill: memory.current read 3,407,872 bytes after the allocator died, which is the sleeper plus the remains. The kernel kills one process. But systemd then fails the scope, recording Failed with result oom-kill in the journal, and tears down the rest of the cgroup. The precise statement is that the bystander survives the kill but does not outlive the unit. Any guide that claims the second process survives is only half right, and the half it omits is the one that decides whether a service keeps running.

Two numbers must not be cited as kill evidence, and this guide does not cite them. The systemd-run command itself exits 137 or 143 when its scope fails with out-of-memory, and the exit code varies from run to run: the fresh runs returned 143, 0 and 143 for the same kill. That exit code is the scope shutdown artifact, not a second kill. The kernel log shows only the allocator killed in every run. The death timestamps from the two failed driver attempts the night before are also unusable: they were recorded about 100 seconds late by a driver bug, and the High-run death timestamps record the operator stopping the scopes, not measurements. Nothing in this guide rests on either.

The soft cap: MemoryHigh stalls, never kills

The same allocator under MemoryHigh at 96 MiB, with no hard cap, behaves like a different machine. The fresh run asked for 128 MiB in 16 MiB steps, wrapped in a 300 second timeout, and it finished: 202.3 seconds from start to done, the first High run on either night to complete its target. The night before, both runs asked for 256 MiB and were stopped by the operator, one at 160 MiB allocated and one at 144 MiB, because at the measured rate the rest would have taken another half hour.

Step Fresh time Night-before time
80 to 96 MiB 0.547 seconds 0.099 seconds
96 to 112 MiB 68.60 seconds 50.74 to 55.66 seconds
112 to 128 MiB 133.09 seconds 131.07 to 141.12 seconds

Below the cap the allocator runs at full speed: steps under 96 MiB take about 9 to 17 milliseconds each, identical to the unconstrained baseline. The step that reaches 96 MiB is where the throttle switches on: 0.547 seconds fresh, 0.099 seconds the night before. Above the cap, each 16 MiB step takes 68.60 seconds and then 133.09 seconds. The fresh median step above High is 100.84 seconds, about 0.159 MiB per second. The night-before median, over seven steps, was 133.06 seconds per step, about 0.12 MiB per second. The first step above High is consistently faster than the later steps on both nights, about 51 to 69 seconds against 131 to 141 seconds, and then the rate settles. With a pure anonymous-memory workload and no swap, MemoryHigh does not slow the service down. It effectively stops it: the process sits in uninterruptible sleep between steps.

The counters confirm the stall and the absence of any kill. At the end of the fresh run, memory.events read high 5,705, max 0, oom 0, oom_kill 0: 5,705 throttle events, zero kills. The night-before runs, sampled mid-run, read high 5,263 and 3,220, also with zero kills. Memory.pressure at the end of the fresh run read some avg10 96.14, full avg10 96.14, total about 174 seconds of accumulated stall out of a 202.3 second run: the cgroup was stalled about 96 percent of the time in the last ten seconds, and about 86 percent of the run overall. The night-before pressure readings, sampled mid-run, read some avg10 94.75 and 98.77, the same shape. And memory.peak sailed far past the soft cap: 137,945,088 bytes, 131.6 MiB, at 128 MiB allocated. The night before it reached 171.5 MiB at 160 MiB allocated. High is a throttle point, not a wall, and the peak is the proof: usage goes past the number and nothing dies.

The baseline: what unconstrained looks like

Without any limit, the same allocator takes 256 MiB in 0.257 seconds on the fresh run, about 996 MiB per second, with a median step of 16.73 milliseconds. The night before it took 0.149 seconds, about 1,720 MiB per second, with a median step of 9.1 milliseconds. Either way, the contrast with the throttled rate is thousands-fold: the fresh High rate of 0.159 MiB per second is about 6,000 times slower than the fresh baseline, and the night-before High rate of 0.12 MiB per second is about 14,000 times slower than its baseline. The exact multiple moves with the baseline; the order of magnitude does not. A soft cap on this workload shape is not a gentler limit. It is a stop with a different name.

Last night said 28 milliseconds; today says 25

Figure 2026-11-07 2026-11-08 (fresh)
Kill latency median 28 ms 25.0 ms
Kill latency range 21 to 74 ms 23.7 to 108.8 ms
memory.peak at kill 134,217,728 134,217,728
memory.events oom_kill 1 1
Pressure total at kill 2,313 microseconds 946 to 2,459 microseconds
Baseline 256 MiB 0.149 seconds 0.257 seconds
High step above cap 133.06 seconds 100.84 seconds
High pressure avg10 94.75 to 98.77 96.14

The kill is tens of milliseconds. The peak lands exactly on the cap. The kill counter reads exactly 1. The pressure at a hard-cap kill is about 1 to 2 milliseconds. The soft cap never kills and stalls about 95 to 99 percent of the time. The numbers that moved are the ones that should move: the slowest kill latency went from 74 to 108.8 milliseconds, the baseline from 0.149 to 0.257 seconds, and the High median step from 133.06 to 100.84 seconds because the fresh run had only two steps above High against seven the night before. Medians absorb the movement; the directions do not change.

What the manuals already tell you

Memory.high is the main mechanism to control memory usage: if a cgroup goes over the high boundary, its processes are throttled and put under heavy reclaim pressure, and going over the high limit never invokes the out-of-memory killer. Memory.max is the final protection mechanism: if usage reaches the limit and cannot be reduced, the out-of-memory killer is invoked in the cgroup. The systemd resource-control manual says the same in unit terms: MemoryHigh is the main control mechanism and MemoryMax is the last line of defense, and it recommends using both together, with High set below Max so the gap acts as a warning zone. The probe adds the numbers the manuals do not give: the last line of defense fires in about 25 milliseconds with about 1 millisecond of stall, and the main control mechanism, on a workload that cannot reclaim anything, stalls the process about 96 percent of the time at about 0.16 MiB per second.

What this guide does not cover

The scope is one workload shape: a single allocator touching anonymous memory in 16 MiB steps, with swap disabled. Workloads with reclaimable file cache, shared memory, or swap available will behave differently under both limits, and this guide does not claim otherwise. The scope is one allocator at a time plus a sleeper: memory.oom.group was left at its default of zero, so the group-kill behavior was not measured, and systemd-oomd is not installed, so the killer here is the kernel cgroup killer, not user space. The High runs used a smaller target on the fresh night, 128 MiB against 256 MiB the night before, so the fresh High median rests on two steps above the cap against seven. The kill latency figures are upper bounds, not exact kill times. And the guide measures what happens after the limit is hit, not how to size the limit: sizing from memory.peak is the use the counters support, and the peak reading exactly at the cap is the number to size against.

Checklist

  • Before trusting an exit code as kill evidence, check memory.events: oom_kill reads exactly 1 for a real cgroup kill, and the systemd-run exit code of 137 or 143 is the scope shutdown, not a second kill.
  • Before sizing a hard cap from a process resident size, check memory.peak: the cap bites on the cgroup total, and the peak lands exactly on the cap while the process itself dies below it.
  • Before waiting for pressure to warn about a hard cap, stop: a MemoryMax kill arrives with about 1 to 2 milliseconds of accumulated stall and a ten-second average of zero.
  • Before treating MemoryHigh as a gentler limit, time it: above the cap this allocator runs at about 0.16 MiB per second, about 6,000 times slower than baseline, and the process sits stalled about 96 percent of the time.
  • Before claiming a bystander survives an out-of-memory kill, check the unit: the kernel kills one process, and systemd then fails the scope and tears down the rest.
  • Before quoting a kill time from one night, run the probe again: the median moved from 28 to 25 milliseconds between nights, and the slowest repetition moved from 74 to 108.8 milliseconds, while every direction held.

Sources

Keep Exploring