Back to Miller

Miller AI internals

docs/src/ai-support.md

6.20.210.0 KB
Original Source
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. ---> <div> <span class="quicklinks"> Quick links: &nbsp; <a class="quicklink" href="../reference-main-flag-list/index.html">Flags</a> &nbsp; <a class="quicklink" href="../reference-verbs/index.html">Verbs</a> &nbsp; <a class="quicklink" href="../reference-dsl-builtin-functions/index.html">Functions</a> &nbsp; <a class="quicklink" href="../glossary/index.html">Glossary</a> &nbsp; <a class="quicklink" href="../release-docs/index.html">Release docs</a> </span> </div> # Miller AI internals

When you use the Miller agent skill or the Miller MCP server, here are the mlr subcommands your agent runs on your behalf to acquire support. (See also Miller and AI for an introduction.)

The new Miller subcommands as of version 6.20 allow agents to discover information about how to use Miller, constrain attempted solutions to match the data, validate Miller commands before running them, run them, and robustly recover from errors.

If you like, you can run these subcommands yourself, although you don't need to. These AI-support subcommands are documented here for transparency.

Discover: the machine-readable catalog

This is the machine-readable catalog of verbs, DSL functions, flags, and keywords, plus intent-to-capability routing.

These are implemented by mlr help --as-json and mlr which.

mlr help --as-json emits Miller's entire help catalog as one JSON document. The --index form is the cheap first call -- every capability with a one-line summary (here trimmed, and then counted, using Miller itself):

<pre class="pre-highlight-in-pair"> <b>mlr help --as-json --index | mlr --json head -n 2</b> </pre> <pre class="pre-non-highlight-in-pair"> [ { "kind": "verb", "name": "altkv", "summary": "Given fields with values of the form a,b,c,d,e,f emits a=b,c=d,e=f pairs." }, { "kind": "verb", "name": "bar", "summary": "Replaces a numeric field with a number of asterisks, allowing for cheesy" } ] </pre> <pre class="pre-highlight-in-pair"> <b>mlr help --as-json --index | mlr --json count</b> </pre> <pre class="pre-non-highlight-in-pair"> [ { "count": 661 } ] </pre>

From the index, an agent drills into full entries one at a time: mlr help verb sort --as-json, mlr help function splitax --as-json, mlr help flag --ifs --as-json, mlr help keyword ENV --as-json -- each accepting one or more names. A verb entry carries a structured option list -- flag, argument placeholder, type -- alongside the familiar usage text:

<pre class="pre-highlight-in-pair"> <b>mlr help verb decimate --as-json</b> </pre> <pre class="pre-non-highlight-in-pair"> [ { "name": "decimate", "summary": "Passes through one of every n records, optionally by category.", "ignores_input": false, "options": [ { "flag": "-b", "type": "bool", "desc": "Decimate by printing first of every n." }, { "flag": "-e", "type": "bool", "desc": "Decimate by printing last of every n (default)." }, { "flag": "-g", "arg": "{a,b,c}", "type": "csv-list", "desc": "Optional group-by-field names for decimate counts, e.g. a,b,c." }, { "flag": "-n", "arg": "{n}", "type": "int", "desc": "Decimation factor (default 10)." } ], "usage_text": "Usage: mlr decimate [options]\nPasses through one of every n records, optionally by category.\nOptions:\n-b Decimate by printing first of every n.\n-e Decimate by printing last of every n (default).\n-g {a,b,c} Optional group-by-field names for decimate counts, e.g. a,b,c.\n-n {n} Decimation factor (default 10).\n-h|--help Show this message." } ] </pre>

Note that usage_text -- what mlr decimate --help prints -- is rendered from the same structured options, so the human help and the machine help cannot drift apart. Function entries carry name, class, arity, help, and examples; the examples across the whole catalog are exercised by Miller's test suite, so they never rot.

Three properties make the catalog cheap to use:

  • It's a perfect cache key. Every document carries mlr_version and catalog_schema_version. Miller is a static binary, so the catalog changes only when the binary does: fetch once, cache forever, re-fetch on a version bump. No TTLs.
  • It's deterministic. One document per invocation, sorted entries, no colorization -- stable for diffing and for prompt caches.
  • It's opt-in twice over. Per-call via --as-json, or set-once via a truthy MLR_HELP_JSON environment variable.

For routing an intent to a capability -- the reverse of browsing -- mlr which returns ranked candidates:

<pre class="pre-highlight-in-pair"> <b>mlr which "join two files on a key" | mlr --json head -n 2</b> </pre> <pre class="pre-non-highlight-in-pair"> [ { "kind": "verb", "name": "join", "score": 25, "summary": "Joins records from specified left file name with records from all file names" }, { "kind": "function", "name": "joink", "score": 25, "summary": "Makes string from map/array keys. First argument is map/array; second is separator string." } ] </pre>

Its exit code signals confidence -- 0 when a query word matched a capability's name, 2 when it didn't -- so a harness can branch on status without parsing anything.

Constrain: the tool's shape, and the data's shape

This shows field names, types, cardinality, and value domains for your actual input data.

It's implemented by mlr describe.

Agents don't just hallucinate flags; they hallucinate values. Miller attacks that from both sides.

Where an option's domain is fixed by the binary, the catalog says so: type is enum and values is the complete list. Here's one option of the summary verb, extracted from the catalog -- using Miller to query Miller:

<pre class="pre-highlight-in-pair"> <b>mlr help verb summary --as-json | mlr --json put -q 'emit $options[1]'</b> </pre> <pre class="pre-non-highlight-in-pair"> [ { "flag": "-a", "arg": "{mean,sum,etc.}", "type": "enum", "desc": "Use only the specified summarizers.", "values": ["field_type", "count", "null_count", "distinct_count", "mode", "sum", "mean", "stddev", "var", "skewness", "minlen", "maxlen", "min", "p25", "median", "p75", "max", "iqr", "lof", "lif", "uif", "uof"] } ] </pre>

Where the domain depends on your data -- which fields exist, what values filter could compare against, what to pass to -g -- the describe verb profiles the input in one pass: per field, the types seen, counts, cardinality, min/max, and (for low-cardinality fields) every distinct value:

<pre class="pre-highlight-in-pair"> <b>mlr --icsv --ojson describe then head -n 2 example.csv</b> </pre> <pre class="pre-non-highlight-in-pair"> [ { "field_name": "color", "types": { "string": 10 }, "count": 10, "null_count": 0, "distinct_count": 3, "min": "purple", "max": "yellow", "values": ["yellow", "red", "purple"] }, { "field_name": "shape", "types": { "string": 10 }, "count": 10, "null_count": 0, "distinct_count": 3, "min": "circle", "max": "triangle", "values": ["triangle", "square", "circle"] } ] </pre>

The catalog is the tool's shape; describe is the data's shape. An agent that consults both has nothing left to guess.

Validate: check DSL before spending a run

This lets the agent parse and type-check a DSL expression before reading any input files.

It's implemented by mlr put --explain and mlr filter --explain.

mlr put --explain (likewise mlr filter --explain) parses and type-checks an expression, then exits -- without opening any input at all:

<pre class="pre-highlight-in-pair"> <b>mlr put --explain '$z = $x + $y'</b> </pre> <pre class="pre-non-highlight-in-pair"> mlr put: DSL expression is valid. </pre> <pre class="pre-highlight-in-pair"> <b>mlr put --explain '$z = $x +'</b> </pre> <pre class="pre-non-highlight-in-pair"> mlr: cannot parse DSL expression. mlr: parse error: unexpected EOF ("") </pre>

Run and recover: errors as data

Agents are instructed to run Miller commands using mlr with the --errors-json flag so that a failure comes back as a structured document instead of prose.

With --errors-json (or MLR_ERRORS_JSON=true environment variable), errors arrive as a structured document. The kind field gives an agent something to branch on; hint is a runnable next step, not a sentence; and did_you_mean is computed against the same catalog the agent discovered from, closing the self-correction loop:

<pre class="pre-highlight-in-pair"> <b>mlr --errors-json --icsv sorted -f shape example.csv</b> </pre> <pre class="pre-non-highlight-in-pair"> { "error": "mlr: verb \"sorted\" not found. Please use \"mlr -l\" for a list.", "kind": "unknown-verb", "token": "sorted", "hint": "Run 'mlr -l' for a list of verbs, or 'mlr help verb \u003cname\u003e' for details.", "did_you_mean": [ "sort" ] } </pre>

And since Miller's DSL includes system and exec, there's a sandbox: --no-shell (or a truthy MLR_NO_SHELL environment variable) disables all external-command execution -- the DSL system and exec functions, piped redirects, and --prepipe fail cleanly:

<pre class="pre-highlight-in-pair"> <b>mlr --no-shell -n put 'end{print system("hostname")}'</b> </pre> <pre class="pre-non-highlight-in-pair"> (error) </pre>

Summary

A typical agent profile sets all three environment variables once:

<pre class="pre-non-highlight-non-pair"> export MLR_HELP_JSON=1 # help/catalog output as JSON export MLR_ERRORS_JSON=1 # errors as structured JSON export MLR_NO_SHELL=1 # no external-command execution </pre>

Put together, the sections above are a loop -- discover, constrain, validate, run -- where each step feeds the next, and failures route back with structure instead of prose.