Python

Python shlex — Split, Quote, and Join Shell-Like Strings

Scripts constantly meet command lines as text: a line from a config file, a command a user typed, a string to log before it runs. Splitting that text on spaces breaks every quoted argument, and gluing values into a command string by hand breaks the moment a value contains a space or a quote. The shlex module handles both directions with no package to install: split reads shell-like quoting, and quote and join write it back. This guide takes those parts in turn, then the lexer class, operator tokens, and the errors an unbalanced quote raises.

Written November 17, 2026 · Tested with Python 3.12.3 · 10 min read

By the end of this guide
  • The main program: splitting a command line
  • POSIX and non-POSIX modes
  • Building a command string: quote and join
  • The lexer class: comments and whitespace splitting

Written November 17, 2026

Tested with Python 3.12.3

What this guide is

The shlex module is a small lexer for shell-like syntax. It does not run anything: it turns a string into a list of tokens, the way a POSIX shell reads a command line, and it turns values back into a correctly quoted string. That makes it the right tool for parsing a command stored as text and for showing a command to a reader without losing which parts were single arguments. Execution stays with the list form of tools such as subprocess, which takes the token list directly and needs no shell at all.

The main program: splitting a command line

shlex.split reads quotes and escapes the way a shell does. An unquoted path with a space in it splits into two tokens, which is the plain-space problem in one line. Quoting that path keeps it as one token, a backslash escapes a single space outside quotes, and a quoted phrase with punctuation inside stays whole:

Python
import shlex
print(shlex.split("tar -czf backup.tar.gz /home/user/my docs"))
print(shlex.split("tar -czf backup.tar.gz '/home/user/my docs'"))
print(shlex.split(r"open my\ file.txt"))
print(shlex.split('grep -n "error: disk" log.txt'))

The exact output:

Config
['tar', '-czf', 'backup.tar.gz', '/home/user/my', 'docs']
['tar', '-czf', 'backup.tar.gz', '/home/user/my docs']
['open', 'my file.txt']
['grep', '-n', 'error: disk', 'log.txt']

Single and double quotes both group words, and the quote characters themselves are removed in the default mode. The result is a plain list of strings, ready to hand to an argument parser or to a process runner as its argument list. Splitting an empty or all-whitespace string gives an empty list, not a list holding one empty string.

POSIX and non-POSIX modes

split runs in POSIX mode by default. Passing posix=False changes what the tokens keep: the quote characters stay on the tokens, and a backslash outside quotes no longer escapes the space after it, so the escaped form splits into two tokens instead of one:

Python
import shlex
text = "run 'two words' \"three word s\""
print(shlex.split(text))
print(shlex.split(text, posix=False))
print(shlex.split(r"echo a\ b"))
print(shlex.split(r"echo a\ b", posix=False))

The exact output:

Config
['run', 'two words', 'three word s']
['run', "'two words'", '"three word s"']
['echo', 'a b']
['echo', 'a\\', 'b']

POSIX mode is the mode that matches a real shell, and it is the one to use for command text. Non-POSIX mode exists for parsers that want the quote characters preserved as part of each token, for example when a file format borrows shell-like quoting but gives the quotes a meaning of its own. Mixing the two readings of the same text is a quiet source of bugs, so a program picks one mode for a format and stays with it.

Building a command string: quote and join

The reverse direction starts with shlex.quote, which returns a value unchanged when it holds only safe characters and wraps it in single quotes when it does not. A value containing a single quote gets the standard shell treatment of closing the quote, adding an escaped quote in double quotes, and reopening. shlex.join applies quote to every item in a list and joins the results with spaces, and split reads that string back into the same list:

Python
import shlex
for value in ("plain", "two words", "it's", "", "a;b", "$HOME"):
    print(value, "->", shlex.quote(value))
command = ["grep", "-n", "it's broken", "my file.txt"]
line = shlex.join(command)
print(line)
print(shlex.split(line) == command)

The exact output:

Output
plain -> plain
two words -> 'two words'
it's -> 'it'"'"'s'
 -> ''
a;b -> 'a;b'
$HOME -> '$HOME'
grep -n 'it'"'"'s broken' 'my file.txt'
True

Three lines in that output carry the lesson. The empty string quotes to two single quotes, because an unquoted empty value would vanish from the string entirely. A semicolon and a dollar sign both force quoting, because a shell would otherwise read them as an operator and a variable expansion. And the round trip on the last two lines is exact: a joined command splits back into the original list, item for item, which is what makes join the right way to log or display a command a program is about to run in list form.

The lexer class: comments and whitespace splitting

split is a shortcut over the shlex class, and the class matters when a format needs settings of its own. Two defaults differ between them. The class treats # as a comment character by default, so a comment and everything after it on the line never become tokens. split disables comments unless it is asked for them, with comments=True. Setting the class attribute whitespace_split to True makes the lexer split on whitespace alone, which suits simple word formats:

Python
import io, shlex
lexer = shlex.shlex(io.StringIO("set mode=fast # enable fast mode\nrun now"), posix=True)
lexer.whitespace_split = True
print(list(lexer))
print(shlex.split("echo hi # there"))
print(shlex.split("echo hi # there", comments=True))

The exact output:

Config
['set', 'mode=fast', 'run', 'now']
['echo', 'hi', '#', 'there']
['echo', 'hi']

The first line lost its comment to the class default, while the second kept the # as an ordinary token because split turns comments off. A lexer reads from a stream, as the StringIO argument shows, or from a string directly. Iterating it yields the tokens one at a time, and listing it collects them all, which is how a config-file parser walks a file line by line with its own settings on each lexer it builds.

Operator tokens: punctuation_chars

A plain split knows nothing about shell operators: a semicolon glued to a word stays glued to it, and a&&b is a single token. The punctuation_chars argument tells the lexer which characters are operators, so runs of them come back as tokens of their own. Passing True uses the standard set, and passing a string names the exact characters to treat that way:

Python
import shlex
line = "ls -l; cat file | grep x > out.txt"
print(shlex.split(line))
lexer = shlex.shlex(line, posix=True, punctuation_chars=True)
lexer.whitespace_split = True
print(list(lexer))
print(shlex.split("run a&&b"))
lexer = shlex.shlex("run a&&b", posix=True, punctuation_chars="();<>|&")
lexer.whitespace_split = True
print(list(lexer))

The exact output:

Config
['ls', '-l;', 'cat', 'file', '|', 'grep', 'x', '>', 'out.txt']
['ls', '-l', ';', 'cat', 'file', '|', 'grep', 'x', '>', 'out.txt']
['run', 'a&&b']
['run', 'a', '&&', 'b']

With punctuation handling on, the semicolon separates from the option it touched, and && arrives as one operator token rather than two single characters. This is the setting a parser for a small command language wants, where the operators carry structure. For an ordinary argument list it stays off, because a filename or a pattern can legitimately contain those characters as data.

Custom splitting and small cases

The class exposes its character sets as attributes, so a lexer can split on a different separator by naming it as the whitespace set. Adjacent quoted and unquoted pieces join into one token, the way they do in a shell, and a pair of empty quotes produces an empty token rather than nothing:

Python
import shlex
lexer = shlex.shlex("a,b,,c", posix=True)
lexer.whitespace = ","
lexer.whitespace_split = True
print(list(lexer))
print(shlex.split("echo a'b'c"))
print(shlex.split("echo '' x"))
print(shlex.split(""), shlex.split("   "))

The exact output:

Config
['a', 'b', 'c']
['echo', 'abc']
['echo', '', 'x']
[] []

The comma lexer skips the empty field between the doubled commas, because consecutive separators count as one boundary under whitespace splitting. The empty-quotes line is the case that surprises most readers: '' in a command is a real, empty argument, and split keeps it, while a string with nothing in it at all has no arguments to keep.

The traps: unbalanced quotes, single-quote backslashes, and None

Trap
An unterminated quote is an error, not a best effort. split raises ValueError with the message No closing quotation, for either quote character. Inside single quotes a backslash is ordinary data, so it does not escape the quote or anything else, which the third line shows by keeping the backslash in the token. Passing None instead of a string raises ValueError as well:
Python
import shlex
for text in ("echo 'unfinished", 'echo "unfinished'):
    try:
        shlex.split(text)
    except ValueError as e:
        print(type(e).__name__ + ":", e)
print(shlex.split(r"echo 'a\b'"))
try:
    shlex.split(None)
except ValueError as e:
    print(type(e).__name__ + ":", e)

The exact output:

Output
ValueError: No closing quotation
ValueError: No closing quotation
['echo', 'a\\b']
ValueError: s argument must not be None

The single-quote rule explains a common failed fix: writing 'a\'b' to embed a quote does not work, because the backslash stays literal and the quoting falls apart. The working forms are the one quote produces, or double quotes around a value that holds a single quote. Code that reads commands from a file should catch the ValueError around split and report the line, since one unbalanced quote in a stored command is a data error the user can fix, not a reason to guess where the quote was meant to end.

Which choice for which job

Job Choice Why
A command string to an argument list shlex.split Reads shell-like quoting and removes the quote characters
A value to a shell-safe string shlex.quote Quotes only when the value needs it, and handles embedded single quotes
An argument list to a displayable command shlex.join Quotes each item, and split reads the result back exactly
A format with comments or its own separator The shlex class Comment, whitespace, and quote settings are per-lexer attributes
Operators as separate tokens punctuation_chars Runs of operator characters come back as their own tokens
Running the command A list-form runner, not a shell string shlex parses and formats text; it does not execute anything

Checklist

  • Use shlex.split for command text instead of splitting on spaces, and expect the quote characters to be removed in POSIX mode.
  • Pick POSIX mode for real command lines, and keep one mode per format.
  • Build command strings with quote and join, never by wrapping values in quotes by hand.
  • Remember the class enables # comments by default while split disables them.
  • Turn on punctuation_chars only when operators are structure rather than data.
  • Catch ValueError around split for stored text, and never rely on a backslash inside single quotes.

Sources

  • Python Standard Library, shlex — split, join, and quote, the shlex class and its attributes, the parsing rules for POSIX and non-POSIX modes, and punctuation_chars.

Keep Exploring