docs/requirements/interception-shell-text-parsing.md
A user has the text a build system prints when asked to show commands
without running them - for example make -n output, or a saved build
log - and wants a compilation database from it without paying for a full
build, in one step:
make -n | bear parse-sh
The mode reconstructs the executions the text describes and feeds them through the same semantic analysis and output stage as every other database-producing mode; the compilation database is its only product.
This is a best-effort front-end, not a replacement for interception.
Interception observes real exec() calls and remains the high-fidelity,
recommended path; this mode reconstructs approximate executions from
text a build system chose to print. The user must be told, in the man
page and in this contract's known limitations, that fidelity is bounded
by what the dry-run text contains.
Mode shape:
output-json-compilation-database,
output-append,
output-atomic-write, and the rest of the
output- set). The destination defaults to compile_commands.json,
and appending to an existing database is supported.Parsing contract:
cd (see working-directory tracking
below). The parser does not decide what a compiler is: mv, ar,
ln, and mkdir lines are valid candidates just as compiler lines
are. Which candidates become database entries is decided downstream by
compiler recognition and output validation, exactly as for intercepted
executions; a candidate that is not recognized as a compiler produces
no entry and no diagnostic. Only lines the parser cannot handle are
skipped loudly (see skip reporting below).output-json-compilation-database).VAR=value words are captured as environment overrides for
that command, not treated as the executable. A candidate's environment
is the mode's own environment at parse time overlaid with those
assignments, and it participates in semantic analysis the same way an
intercepted execution's environment does - observably through
environment-derived flags
(output-env-derived-flags).newline, ;, &&, ||, &, and |, where
each side of a pipe is a candidate command;# to end of line) and blank lines, which produce nothing;>, >>, <, 2>, 2>&1, >/dev/null, and the
like), which are recognized and removed from the command's words. A
redirect's whole target word is discarded uninspected: an expansion
or substitution inside one does not skip the line. An unterminated
quote or unterminated substitution in a target skips the line loudly.{ ...; } in command position, which run in the
current shell. The braces are structural and produce no candidate;
the commands between them are parsed with the shared shell state, so
a cd inside a group persists after the closing brace, matching
sh. Groups may nest and may span lines. An unmatched } (a close
with no open group) skips its line loudly, and an input that ends
with a { still open is reported loudly against the opening line;
both are reported as unbalanced braces. This is a subshell contrast,
not a generalization: ( ... ) runs in a child shell and stays
unsupported.( ... ), command
substitution (backticks or $(...)), parameter expansion ($VAR,
${...}), globs in the executable word, here-documents, and the
case/for/while/if keywords - causes the whole line to be
skipped. A skip is reported on standard error with the line number and
the reason. The mode never guesses at the meaning of an unsupported
construct.cd <dir> command updates it for subsequent commands, and is
consumed only for that effect: cd produces no candidate, because
it is a shell builtin that never reaches exec() and so would never
be observed by interception either. cd - restores the previous
working directory when one is known and is a loud skip otherwise;
the - is never treated as a directory name;make[N]: Entering directory '...' and Leaving directory '...'
markers push and pop it, so recursive make -n -w output tracks
directories correctly. These markers are an explicit, documented
extension to the shell subset.cli-exit-codes.Non-guarantees, which hold regardless of parser quality because they are inherent to dry-run text. These must also appear in the man page:
$(shell ...) evaluated at parse
time can differ from a real build.PATH used to resolve bare names are the mode's own at parse time
and may differ from a real build's. Parsing a log captured on another
machine implies a foreign environment and PATH that this mode cannot
reproduce, even when the caller pins the working directory.sh command text only. Non-POSIX shells and Windows
cmd are out of scope. A saved full-build log is accepted only insofar
as its lines happen to be sh commands; interleaved compiler output is
skipped loudly like any other unsupported line.Deliberate exclusion, likely to be proposed again:
ar, ln, a linker line) appear nowhere. The
skip diagnostics on standard error are the only window into the
reconstruction. The reasoning is recorded in
parse-sh-database-product;
a public execution-stream output can be added compatibly if real
demand appears.Given real make -n output from a configured project (the zlib fixture):
When the user pipes it to
bear parse-sh, thencompile_commands.jsonis written at the default destination and covers every compiler invocation in the text, non-compiler lines (ar,mv,mkdir,ln) produce no entries and no diagnostics, and the lines outside the supported subset (subshells, substitutions) are reported on stderr as skips but are not fatal.
Given the line gcc -c foo.c saved in a file named build.sh:
When the user runs the mode with
build.shas the named input anddb.jsonas the named output, thendb.jsoncontains exactly one entry: its file isfoo.c, its compiler is spelledgcc, and its directory is the mode's working directory. Standard output carries nothing.
Given the zlib fixture and a configuration that widens duplicate detection to compare full argument lists:
When the user supplies that configuration to the mode, then the database keeps the entries the default de-duplication folds away - the configuration shaped the semantic stage of this mode just as it shapes the other database-producing modes. The stderr skip report is identical with and without the configuration: configuration never changes how the text is parsed.
Given input whose only line is the non-compiler command mv objs/a.o a.lo:
When the mode parses it, then no line is skipped, no diagnostic names the line, and the database is written empty - a clean parse with no recognized compiler is not an error (the exit code is owned by
cli-exit-codes).
Given input exercising the supported subset together - quotes, an escape, a continuation, separators including a pipe, redirections, a comment, and a blank line:
gcc -DNAME='a b' -c \
foo\ bar.c && ar rc libz.a foo.o | tee log # archive it
gcc -c baz.c >/dev/null 2>&1 || true
When the mode parses it, then the database has exactly two entries: one for
foo bar.cwhose arguments carry the single word-DNAME=a b(the continuation joined, quoting and escape resolved), and one forbaz.cwhose arguments carry no redirection words. Thear,tee, andtruecandidates produce no entries, the comment and the blank line produce nothing, and no line is skipped.
Given the redirect-target pair gcc -c foo.c > $(logdir)/x.log and
cmd > 'oops:
When the mode parses them, then the first yields a clean entry for
foo.cand no skip - the substitution lies entirely inside the discarded target - while the second yields nothing and is skipped loudly as an unterminated quote.
Given a brace group followed by another command, starting in /build:
{ cd sub; gcc -c a.c; }
gcc -c b.c
When the mode parses it, then the braces produce no entry of their own and both compilations appear with directory
/build/sub- the group'scdpersists past the closing brace. A}with no open group (echo a; }) skips its line loudly, and an input that ends with a{still open reports unbalanced braces against the line that{opened on.
Given the line gcc -c foo.c from a log captured elsewhere, and a
caller-set initial working directory of /custom/build:
When the mode parses it, then the entry's directory is
/custom/build, not the mode's own working directory.
Given cd-driven directory changes, starting in /build:
cd sub
gcc -c a.c
cd -
gcc -c b.c
When the mode parses it, then neither
cdline produces an entry,a.cis recorded with directory/build/sub, andb.cback in/build. An input whose first command iscd -(no previous directory is known) has that line skipped loudly and the working directory left unchanged.
Given the line
CPATH=/opt/include gcc -c foo.c -o foo.o >/dev/null 2>&1 and
environment-derived flags at their default (enabled):
When the mode parses it, then the entry for
foo.cnames the compilergcc(the assignment is not the executable), its arguments carry no redirection words, and they include an explicit include flag for/opt/include- the leading assignment reached semantic analysis as that command's environment.
Given a recursive-make log carrying Entering directory and
Leaving directory markers around cd-free compiler lines:
When the mode parses it, then each entry's directory reflects the directory in effect at that line.
Given input in which every line is an unsupported construct (all subshells, all command substitutions):
When the mode runs, then it reports each skipped line on stderr and the run fails (the exit code is owned by
cli-exit-codes).
Given input mixing one supported compile line with one unsupported line
(gcc -c foo.c, then for f in *.c; do gcc -c $f; done):
When the mode runs, then stderr carries the per-line skip report and a summary counting one command parsed and one line skipped - the
forline counts once, not once per command segment inside it. Empty input produces a stderr notice that no commands were found; input with nothing skipped keeps stderr quiet.
Given a request to stream the database:
When the user directs the database to standard output, then the JSON database appears on standard output with diagnostics kept on standard error, and no file is created. Combining standard-output streaming with appending is rejected as a usage error.
Given an existing compilation database and new dry-run text:
When the user runs the mode against the same destination with appending enabled, then the result is the merged set governed by
output-append.