Error diagnostics

GNU coreutils reports every error as a single line on stderr. That line answers what went wrong, but Not where. For utilities whose arguments form a small language of their own - a test expression, a chmod mode, a sort key - the interesting question is usually which argument, or which character inside an argument, broke the parse.

When stderr is a terminal, uutils renders these errors as a compiler-style report instead: the command line is echoed back as a source line, and a caret points at the part that is at fault, often with a line of advice. Everywhere else - a script, a pipe, a test harness - the plain one-line message is kept, so nothing that reads stderr can tell the difference.

Before and after

Each utility below shows the plain one-line message first - what uutils prints when stderr is not a terminal, and in most cases exactly what GNU prints - then the report rendered on a terminal, colors included.

`test`

Before:

<pre class="diag">$ test 7 -eq zap test: invalid integer 'zap'</pre>

After:

<pre class="diag">$ test 7 -eq zap test: invalid integer 'zap' ╭─[ test:1:7 ] │ 1 │ 7 -eq zap │ ─── │ │ Help: -eq, -ne, -lt, -le, -gt and -ge compare integers; use =, !=, < or > to compare strings │ -eq equal, -ne not equal, -lt less than, -le less than or equal, -gt greater than, -ge greater than or equal ───╯</pre>

Try it in the playground.

`cut`

A list of ranges is often long, and only one item in it is wrong:

Before:

<pre class="diag">$ cut -f 1,4-2,9-12 notes.txt cut: invalid decreasing range</pre>

After:

<pre class="diag">$ cut -f 1,4-2,9-12 notes.txt cut: invalid decreasing range ╭─[ cut:1:10 ] │ 1 │ cut -f 1,4-2,9-12 notes.txt │ ─┬─ │ ╰─── this range ends before it starts │ │ Help: a list is N, N-M, N- or -M, separated by commas, as in -f1,4-6,9- ───╯</pre>

`expr`

Before:

<pre class="diag">$ expr 9 + foo expr: non-integer argument</pre>

After:

<pre class="diag">$ expr 9 + foo expr: non-integer argument ╭─[ expr:1:5 ] │ 1 │ 9 + foo │ ─── │ │ Help: arithmetic operators need integers; use = or != to compare strings instead ───╯</pre>

Try it in the playground.

`chmod`

The caret can point inside an argument, at the exact character that broke the parse:

Before:

<pre class="diag">$ chmod 'g+rw?x' notes.txt chmod: invalid operator (expected +, -, or =, but found ?)</pre>

After:

<pre class="diag">$ chmod 'g+rw?x' notes.txt chmod: invalid operator (expected +, -, or =, but found ?) ╭─[ chmod:1:5 ] │ 1 │ g+rw?x notes.txt │ ─ │ │ Help: a mode is either octal, as in 644, or clauses such as u+rwx,go-w ───╯</pre>

`tr`

Before:

<pre class="diag">$ tr 'qw[y-b]' x tr: range-endpoints of 'y-b' are in reverse collating sequence order</pre>

After:

<pre class="diag">$ tr 'qw[y-b]' x tr: range-endpoints of 'y-b' are in reverse collating sequence order ╭─[ tr:1:7 ] │ 1 │ tr qw[y-b] x │ ─┬─ │ ╰─── did you mean 'b-y'? │ │ Help: a range goes from the lower character to the higher one, as in a-z ───╯</pre>

Try it in the playground.

`csplit`

A pattern is a small language of its own, and the regex engine already knows which character of it it choked on - so the caret can say so too:

Before:

<pre class="diag">$ csplit notes.txt '/a{2,1}/' csplit: '/a{2,1}/': invalid pattern</pre>

After:

<pre class="diag">$ csplit notes.txt '/a{2,1}/' csplit: '/a{2,1}/': invalid pattern ╭─[ csplit:1:20 ] │ 1 │ csplit notes.txt /a{2,1}/ │ ──┬── │ ╰──── invalid repetition count range, the start must be <= the end │ │ Help: a pattern is a line number N, /REGEXP/[OFFSET] or %REGEXP%[OFFSET], each optionally followed by {N} or {*} ───╯</pre>

The label is quoted from the regex engine rather than translated, since that is the only place the wording exists.

`sort`

Before:

<pre class="diag">$ sort -k2.3x notes.txt sort: stray character in field spec: invalid field specification '2.3x'</pre>

After:

<pre class="diag">$ sort -k2.3x notes.txt sort: stray character in field spec: invalid field specification '2.3x' ╭─[ sort:1:11 ] │ 1 │ sort -k2.3x notes.txt │ ─ │ │ Help: a key is FIELD[.CHAR][OPTS][,FIELD[.CHAR][OPTS]], as in -k2.3,4nr ───╯</pre>

Try it in the playground.

`numfmt`

Before:

<pre class="diag">$ numfmt --format=%q 1000 numfmt: invalid format '%q', directive must be %[0]['][-][N][.][N]f</pre>

After:

<pre class="diag">$ numfmt --format=%q 1000 numfmt: invalid format '%q', directive must be %[0]['][-][N][.][N]f ╭─[ numfmt:1:18 ] │ 1 │ numfmt --format=%q 1000 │ ┬ │ ╰── f is the only conversion numfmt has; %d, %e, %g and the other C conversions are not accepted │ │ Help: a format is [PREFIX]%[0]['][-][WIDTH][.PRECISION]f[SUFFIX], as in "%'-10.2f" ───╯</pre>

Try it in the playground.

`printf`

Before:

<pre class="diag">$ printf %5.2c q printf: %5.2c: invalid conversion specification</pre>

After:

<pre class="diag">$ printf %5.2c q printf: %5.2c: invalid conversion specification ╭─[ printf:1:8 ] │ 1 │ printf %5.2c q │ ───── │ │ Help: %d, %s, %x, %f and the other C conversions are accepted, plus %b and %q; a literal % is written %% ───╯</pre>

The same goes for a broken escape: printf 'a\xzb' puts the caret under the \x that is missing its hexadecimal digits.

Try it in the playground.

`env`

env -S takes a whole command line and splits it as a shell would. Its messages name an offset, which is precisely the thing a caret can show instead:

Before:

<pre class="diag">$ env -S 'echo ${1FOO}' env: only ${VARNAME} expansion is supported, error at: ${1FOO}</pre>

After:

<pre class="diag">$ env -S 'echo ${1FOO}' env: only ${VARNAME} expansion is supported, error at: ${1FOO} ╭─[ env:1:14 ] │ 1 │ env -S 'echo ${1FOO}' │ ─┬─ │ ╰─── a variable name cannot start with a digit │ │ Help: only $NAME and ${NAME} are expanded; the other shell forms are not ───╯</pre>

Note that the -S string holds spaces, so it is echoed back quoted - and the caret still points inside it.

`head`

A SIZE is a number and a unit, and the caret says which of the two was rejected:

Before:

<pre class="diag">$ head -c 1fb notes.txt head: invalid number of bytes: '1fb'</pre>

After:

<pre class="diag">$ head -c 1fb notes.txt head: invalid number of bytes: '1fb' ╭─[ head:1:10 ] │ 1 │ head -c 1fb notes.txt │ ─┬ │ ╰── not a known unit │ │ Help: a size is a number and an optional unit: K, M, G and so on for 1024, KB, MB, GB for 1000 ───╯</pre>

`dd`

Every dd operand is a KEY=VALUE pair, and a value can be a comma-separated list of flags, so there are three things the caret can pick out: the key, the whole value, or one flag inside the list. A flag is underlined where it sits in the list rather than wherever its text first turns up, and the advice names the flags that operand accepts:

Before:

<pre class="diag">$ dd conv=ucase,zap dd: invalid conversion: 'zap'</pre>

After:

<pre class="diag">$ dd conv=ucase,zap dd: invalid conversion: 'zap' ╭─[ dd:1:15 ] │ 1 │ dd conv=ucase,zap │ ─┬─ │ ╰─── not a known conversion │ │ Help: conv= is one of ascii, ebcdic, ibm, lcase, ucase, block, unblock, swab, sync, noerror, sparse, excl, nocreat, notrunc, fdatasync or fsync ───╯</pre>

iflag= and oflag= are reported apart, so an output flag is no longer blamed on the input, and each lists its own flags:

Before:

<pre class="diag">$ dd oflag=zap dd: invalid output flag: 'zap'</pre>

After:

<pre class="diag">$ dd oflag=zap dd: invalid output flag: 'zap' ╭─[ dd:1:10 ] │ 1 │ dd oflag=zap │ ─┬─ │ ╰─── not a known output flag │ │ Help: oflag= is one of direct, directory, dsync, sync, nocache, nonblock, noatime, noctty, nofollow, append or seek_bytes ───╯</pre>

An unknown key is underlined without its value, since the value is not what was rejected:

Before:

<pre class="diag">$ dd zap=1 dd: unrecognized operand 'zap=1'</pre>

After:

<pre class="diag">$ dd zap=1 dd: unrecognized operand 'zap=1' ╭─[ dd:1:4 ] │ 1 │ dd zap=1 │ ─── │ │ Help: an operand is KEY=VALUE, as in if=file bs=4k count=10 ───╯</pre>

A number that does not fit is rejected rather than quietly clamped, and the caret covers the whole value:

Before:

<pre class="diag">$ dd count=99999999999999999999999 dd: invalid number: '99999999999999999999999': Value too large for defined data type</pre>

After:

<pre class="diag">$ dd count=99999999999999999999999 dd: invalid number: '99999999999999999999999': Value too large for defined data type ╭─[ dd:1:10 ] │ 1 │ dd count=99999999999999999999999 │ ─────────────────────── │ │ Help: a number may be followed by a multiplier: c, w, b, then K, M, G and so on for 1024, kB, MB, GB for 1000 ───╯</pre>

Try it in the playground.

Compatibility

This is strictly an interactive nicety; nothing that reads our output can tell the difference:

Reports are only rendered when stderr is a terminal, unless UUTILS_DIAG says otherwise (see below). In a script, a pipe, or a test harness, the utility keeps printing its plain one-line message, so existing scripts that match on stderr keep working.
A report is drawn only when the error can be tied to something on the command line. When it cannot - the error is not about any one argument, or the operand it names was rewritten or consumed before it could be located - the plain one-line message is printed instead, even at a terminal. A utility in the table below therefore reports the parse errors it can place, not every error it has.
Exit codes are unchanged, and so is the Try '... --help' hint: an error that is a usage error still prints it under the report.
Colors follow the usual conventions: they are used only on a terminal, and NO_COLOR disables them.
All messages, labels and help lines are localized like the rest of uutils (see Localization).
The rendering can be compiled out entirely: it sits behind the feat_diagnostics cargo feature, which is on by default. Building with --no-default-features (plus a feat_os_* selection) drops the renderer and its ariadne dependency, and every utility keeps its plain one-line messages.

Turning it on and off

The default keys off stderr being a terminal, and nothing else - which is usually what you want, but not always. UUTILS_DIAG overrides it:

Value	Effect
`always`	Draw the report even when stderr is a file or a pipe.
`never`	Keep the plain one-line message even at a terminal.
`auto`, unset, anything else	Decide from stderr, as above.

An unrecognized value is deliberately not an error - this is the kind of variable that gets exported from a shell profile once and forgotten, and no spelling of it should be able to make a utility fail.

always is the one to reach for when the error has to leave the terminal it happened in - a CI log, or a report to paste into a bug:

$ UUTILS_DIAG=always sort -k2.3x notes.txt 2> parse.log

Colors are a separate question, and one the terminal still answers: a report forced into a file is written without them, so nothing has to strip escape sequences back out. NO_COLOR is the middle setting at a terminal - the report is still drawn, just in plain text.

Without the variable, both directions are still one command away. To get the plain line while sitting at a terminal, send stderr somewhere that is not one:

$ sort -k2.3x notes.txt 2>&1 | cat
sort: stray character in field spec: invalid field specification '2.3x'

And to get a report out of a command that has to run under a real terminal, give it a pty with script -qec "..." /dev/null, or unbuffer from expect.

Supported utilities

Utility	What the caret points at	Try it
`test`	the argument that made the expression fail	`test 7 -eq zap`
`expr`	the argument that made the expression fail	`expr 9 + foo`
`chmod`	the failing clause (or character) of an invalid symbolic or octal mode	`chmod 'g+rw?x' fruits.txt`
`mkdir`	the failing part of the mode given to `-m`/`--mode`	`mkdir -m u+q mydir`
`mkfifo`	the failing part of the mode given to `-m`/`--mode`	`mkfifo -m u+q mypipe`
`mknod`	the failing part of the mode given to `-m`/`--mode`	`mknod -m u+q mydev c 1 3`
`install`	the failing part of the mode given to `-m`/`--mode`	`install -m u+q fruits.txt dest`
`tr`	the part of a set that is at fault (bad class, backwards range, bad repeat count, …)	`tr 'qw[y-b]' x`
`sort`	the failing part of a `-k`/`--key` or field specification, or of the SIZE given to `-S`	`sort -k2.3x fruits.txt`
`numfmt`	the failing part of a `--field` or `--format` specification	`numfmt --format=%q 1000`
`printf`	the failing conversion or escape in the format string	`printf %5.2c q`
`seq`	the failing conversion in the format given to `-f`/`--format`	`seq -f %5.2c 1 3`
`stat`	the failing directive of a `-c`/`--format` or `--printf` format	`stat -c %d%.3 fruits.txt`
`env`	the failing part of a `-S`/`--split-string` string	`env -S 'echo ${1FOO}'`
`dd`	the failing key, value or flag of a `KEY=VALUE` operand	`dd conv=ucase,zap`
`join`	the failing field of the output format given to `-o`	`join -o 1.2,2.x fruits.txt fruits.txt`
`cut`	the failing range in the list given to `-b`, `-c`, `-f` or `-F`	`cut -f 1,4-2 fruits.txt`
`csplit`	the failing pattern operand, the character of its regex that broke, or the format given to `-b`/`-n`	`csplit fruits.txt '/a(b/'`
`split`	the failing part of the SIZE given to `-b`, `-C` or `-l`	`split -b 7zq fruits.txt`
`shred`	the failing part of the SIZE given to `-s`/`--size`	`shred -s 4vv fruits.txt`
`head`	the failing part of the SIZE given to `-c` or `-n`	`head -c 1fb fruits.txt`
`tail`	the failing part of the SIZE given to `-c` or `-n`	`tail -c 1fb fruits.txt`
`truncate`	the failing part of the SIZE given to `-s`/`--size`	`truncate -s 10fb fruits.txt`
`od`	the failing part of the SIZE given to `-j`, `-N`, `-S` or `-w`	`od -N 3zz fruits.txt`
`du`	the failing part of the SIZE given to `-B`/`--block-size` or `-t`/`--threshold`	`du -B 1fb`
`df`	the failing part of the SIZE given to `-B`/`--block-size`	`df -B 1fb`
`ls`	the failing part of the SIZE given to `--block-size` (also `dir` and `vdir`)	`ls --block-size=1fb`
`stdbuf`	the failing part of the buffering mode given to `-i`, `-o` or `-e`	`stdbuf -o 6pq head`

How it works

The rendering lives in uucore::features::diagnostics and is built on ariadne. A utility that opts in:

Takes a Snapshot of its argument list. The snapshot joins the arguments into one line (quoting where needed, and rendering non-UTF-8 arguments the way the shell would) and remembers the byte range of each argument.
Maps its own error type to a position, and optionally a label and a line of advice, in a small per-utility diagnostics.rs module. A label is only used when it adds something the message does not say - an expectation, or a fix such as tr's did you mean 'b-y'? - never to restate it; with no label the span is drawn as a bare underline. Everything user-facing is passed in already localized.
Locates the argument the operand came from - with Snapshot::index_of_value for an option's value in whatever spelling it was given (-k 2.3x, -k2.3x, -rk2.3x, --key 2.3x, --key=2.3x), Snapshot::index_of_positional for a positional operand, or an index the utility tracked itself - and calls Snapshot::render to point at the whole argument, or Snapshot::render_inside_at to point at a byte range inside the operand it carries. Because the argument is named rather than searched for, a file, another option, or the program name that happens to share the operand's text can never take the caret. Snapshot::render_option_value does the two steps in one, which is what most option values want.

An argument holding a space is echoed back quoted, and the caret still points inside it: the quotes only wrap the operand, so its bytes are found where they were printed and offsets count from there. An argument that could not be printed as-is - a non-UTF-8 one, or one whose quoting had to be broken up - is underlined as a whole instead, since no offset into it would line up with what the reader sees.

Callers check diagnostics::enabled() first, so the non-interactive path costs nothing and keeps its GNU-compatible message.

Errors that come out of a shared parser are rendered by a shared helper, so that a syntax common to several utilities is explained the same way in all of them, and its labels live in the common locale strings rather than being repeated per utility. Three parsers work this way:

Modes (uucore::mode), for chmod, mkdir, mkfifo, mknod and install.
Range lists (uucore::ranges), for cut's -b, -c and -f and for numfmt --field. Range::from_list reports which item of the list failed and where it sat.
Sizes (uucore::parser::parse_size), for head, tail, truncate, split, shred, stdbuf, sort, od, du, df and ls (with dir and vdir) today, and available to the other callers of the parser. ParseSizeError::span works out from the operand which of its two parts - the number or the unit - was rejected, so the error type keeps the shape its callers build by hand.

Taking modes as the worked example: the parser reports errors as a structured ModeError carrying the byte range at fault, and its rendering places the caret for every utility that takes a mode - ModeError::render_option_value finds the mode as the value of -m/--mode for mkdir, mkfifo, mknod and install, while chmod, which accepts modes clap cannot see (such as chmod -w -r file), tracks where each mode operand sat and passes the index to ModeError::render_at.