* Plan: flesh out PR7 (MCP server + Agent Skill) design
stdio transport (no HTTP port), mlr mcp terminal in the main binary,
SDK-vs-handroll decision, tool list, in-process vs subprocess split,
run-tool safety (--no-shell prerequisite), single-sourced skill, tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add mlr mcp: MCP server + agent playbook; --no-shell gate (#2098 PR7)
New terminal `mlr mcp` runs a Model Context Protocol server over stdio
(spawned by MCP clients; no network port), exposing five tools --
list_capabilities, which, validate_dsl, describe_data, run -- plus an
agent playbook as MCP prompt/resource. Catalog tools are served
in-process from the help registries; the rest subprocess this same
binary with MLR_ERRORS_JSON=1, a timeout, and an output cap.
Prerequisite: a new --no-shell flag / MLR_NO_SHELL env var (one-way
gate) disables the DSL system/exec functions, piped redirects, and
--prepipe/--prepipex; the MCP server sets it on the commands it runs
unless started with --allow-shell.
Adds the github.com/modelcontextprotocol/go-sdk dependency.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Force LF checkout for the embedded SKILL.md (Windows CI fix)
go:embed embeds checkout bytes, so a CRLF checkout on Windows made the
embedded playbook differ per platform and failed
TestPlaybookHasFrontmatter. Pin the file to eol=lf in .gitattributes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Move no-shell test DSL into per-case mlr files (Windows CI fix)
Inline single-quoted DSL in cmd files is mangled by the Windows shell
(single quotes are not quote characters there); the harness's
put -f ${CASEDIR}/mlr pattern avoids shell quoting entirely.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Tier-2 structured verb options: OptionSpec, initial migration (#2098)
PR 3 of the AI-friendly roadmap (plans/plan-2098-llm.md).
Infrastructure:
- Add OptionSpec{Flag,Arg,Type,Desc,Repeatable,Values} to
pkg/transformers/aaa_record_transformer.go alongside TransformerSetup.
Type is one of: bool, string, int, float, csv-list, regex, filename,
format, enum. For type=="enum", Values lists the valid choices.
- Add Options []OptionSpec to TransformerSetup (nil = not yet migrated).
- Emit Options in VerbInfoForJSON (omitempty so unmigrated verbs stay
backward-compatible; agents check key presence for Tier-2 availability).
UsageText is always present as the Tier-1 prose fallback.
- Add VerbOptionsNilCheck() in aaa_verb_options_check.go: progress report
of migrated vs. unmigrated verbs, analogous to FLAG_TABLE.NilCheck().
- Wire verb-options-nil-check into mlr help (internal/docgen section).
Initial migration (5/70 verbs):
- nothing: empty Options (no verb-specific options, explicitly migrated)
- cat: -n (bool), -N (string), -g (csv-list), --filename, --filenum (bool)
- head: -g (csv-list), -n (int)
- tail: -g (csv-list), -n (int)
- tee: -a, -p (bool)
Tests:
- 5 new unit tests in aaa_transformer_json_test.go covering migrated/
unmigrated paths, field population, JSON round-trip, and key-presence.
- Regression test case 0003: mlr help verb-options-nil-check golden output.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Migrate all 70 verbs to structured OptionSpec; bump catalog schema to v2
Completes the Tier-2 migration started in the previous commit. Every verb
in TRANSFORMER_LOOKUP_TABLE now has a non-nil Options field.
- Workflow-migrated all 65 remaining verbs. Each Setup var now carries
Options: []OptionSpec{...} with Flag/Arg/Type/Desc fields. Verbs with
no verb-specific options (altkv, check, group-like, nothing, etc.) use
an empty slice to signal "migrated but no options."
- Drop `omitempty` from VerbInfoForJSON.Options: empty slices were silently
dropped, making migrated-no-option verbs indistinguishable from unmigrated
ones in JSON. Without omitempty: null=unmigrated, []=migrated-no-options,
[...]= migrated-with-options. Bump catalogSchemaVersion 1→2 for this shape
change.
- Replace the two "unmigrated-verb" unit tests (which used stats1 as an
example) with TestAllVerbsFullyMigrated (asserts every verb has non-nil
Options) and TestAllVerbsHaveOptionsKeyInJSON (asserts every migrated
verb emits the "options" key in JSON).
- Regenerate test/cases/cli-help/0003/expout: now reads
"Verb options migration: 70/70 migrated."
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Add --errors-json structured error output (#2098)
PR 4 of the AI-friendly roadmap (plans/plan-2098-llm.md).
Agents previously had to regex-match English prose to branch on error
kind; this PR lets them do it structurally.
Interface:
- mlr --errors-json <bad command> emits a JSON object to stderr and
exits 1; same behavior as prose path but machine-readable.
- MLR_ERRORS_JSON=1 (truthy) is the env-var equivalent.
- Without the flag, prose output is byte-for-byte unchanged.
JSON shape: {error, kind, token, verb, hint, did_you_mean[]}
Error kinds: unknown-verb, unknown-flag, verb-option-error, generic.
Implementation:
- New pkg/climain/errors_json.go: CLIError typed error, StructuredError
DTO, WantErrorsJSON pre-scan, Levenshtein edit distance, topMatches,
EmitStructuredError.
- Convert two direct os.Exit sites in parseCommandLinePassOne (unknown
verb, unknown flag) to return CLIError values; verb-option-error
likewise returns CLIError from pass-one ParseCLIFunc handling.
- parseCommandLinePassOne gains an error return; ParseCommandLine
propagates it.
- entrypoint.go pre-scans os.Args for --errors-json before calling
ParseCommandLine; routes errors to EmitStructuredError or printError.
- did_you_mean: Levenshtein nearest-match over verb catalog, flag
catalog, or the verb own OptionSpec flags (PR3 catalog) for
verb-option-error. Closes the self-correction loop the catalog
enables.
- --errors-json registered in Miscellaneous flags so it appears in
mlr --help and is not treated as an unrecognized flag.
Tests: 16 unit tests covering Levenshtein, topMatches, threshold,
WantErrorsJSON, CLIError interface, and all categorize paths.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* refine the plan
* Fix all staticcheck lint findings (uncapped)
golangci-lint's default max-same-issues=3 was hiding most of the backlog:
the true pre-fix count was 69 staticcheck findings, not 34. This fixes all
of them, driving staticcheck to zero:
- ST1023/QF1011 (37): omit explicit types inferred from the RHS
- S1009/S1031 (15): drop redundant nil checks before len()/range
- SA9003 (9): remove comment-only empty branches, keeping the comments
- QF1007 (3): merge conditional assignment into declaration
- QF1006 (3): lift break conditions into loop conditions
- QF1001 (3): apply De Morgan's law / name the negated predicate
Also updates plans/lintfixes.md with the cap discovery and the corrected
errcheck picture (1202 uncapped, ~949 of them fmt.Fprint*).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Drive errcheck to zero: config for bulk categories, propagate real errors
Adds .golangci.yml with errcheck exclude-functions for fmt.Fprint* (usage
printers), (*bufio.Writer).Write/WriteString (sticky errors, surfaced at the
now-checked final Flush), and (*strings.Builder).WriteString; pins
max-issues-per-linter/max-same-issues to 0 so CI reports true counts.
Real error paths now propagate instead of being dropped:
- Finalize{Reader,Writer}Options in join/put/filter/split/tee and the
repl/script entry points: 'mlr join -i badformat' now errors instead of
silently using wrong separators
- final output-stream Flush in pkg/stream: write failure no longer exits 0
- DSL emit/print/dump redirect writes, matching their sibling branches
- CSV writer WriteCSVRecordMaybeColorized, close-time Flush in file output
handlers, ENV[...] Setenv, REPL record-write and redirect-close errors
- termcvt write-side Close before rename (had "TODO: check return status")
The rest are deliberate ignores, marked with _ = and a comment where the
reason isn't obvious: unset-of-missing-path no-ops, read-side closes,
mid-stream FlushOnEveryRecord, init-time strftime registrations, in-memory
usage-capture pipes, and regtest-harness env/temp-file teardown.
golangci-lint now reports 0 issues on ./cmd/mlr ./pkg/... with all caps off.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Emit Miller's existing help catalog (verbs, functions, flags, keywords)
as structured JSON so AI agents and tooling can model Miller's surface
without scraping prose. The --json token may appear anywhere on a
`mlr help ...` command line; plain text help is unchanged.
mlr help --json # full catalog
mlr help verb cat --json # one or more verbs
mlr help function splitax --json # one or more functions
mlr help flag --ifs --json # one or more flags
mlr help keyword ENV --json # one or more keywords
Functions and flags serialize fully (name/class/arity/help/examples;
section/name/alt_names/arg/help). Verbs carry a summary, ignores_input,
and captured raw usage_text as a Tier-1 fallback, since per-verb options
are prose-only today (each verb hand-writes its UsageFunc). Structured
verb options are a planned follow-on (see #2098).
This is a serialization layer over the existing registries -- no
refactor of the text-help path.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* initial attempt
* fix bash
* fix zsh
* Add shell-completion docs page
Documents the new 'mlr completion {bash,zsh}' feature: the then-chain
context model, install instructions for bash and zsh (including the macOS
bash-3.2 'eval' caveat and zsh compinit self-init), and examples of
context-aware completion. Added to the nav under "Miller in more detail".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Add enum value completion for format and separator flags
Completes the argument value for arg-taking main flags whose values are a
known set: file-format names for -i/-o/--io, separator aliases for
--ifs/--ofs/--ips/etc., and regex-separator aliases for --ifs-regex/--ips-regex.
Other arg-taking flags continue to fall back to filename completion.
Candidate sets come from new cli getters (GetFileFormatNames,
GetSeparatorAliasNames, GetSeparatorRegexAliasNames) that read the same maps
Miller uses at runtime, so there is no separate list to keep in sync. The
command-line walk now records which flag a value position belongs to.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Include format-conversion keystroke-savers in bare-dash completion
Reverts the suppression of --c2j/--x2y-style flags from 'mlr -<TAB>'. The full
set of main flags (297) is now offered, matching what is valid on the command
line. GetFlagNames no longer takes an includeSuppressed argument.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Complete terminal subcommands and top-level help/version flags
'mlr <TAB>' now offers subcommand names (help, version, repl, regtest, script,
completion, terminal-list) alongside verb names, and 'mlr -<TAB>' offers the
top-level terminal flags (-h, --help, --version, --bare-version, and the help
shorthands -g/-l/-L/-f/-F/-k/-K). Subcommand names are offered only as the
first non-flag token, where they are valid.
To let the completion engine know these names without an import cycle
(pkg/terminals imports pkg/terminals/completion), the canonical terminal names
and version-flag spellings are factored into a new leaf package
pkg/terminals/registry, imported by pkg/terminals, pkg/climain, and completion.
The help-flag spellings come from a new help.GetTerminalFlagNames derived from
the existing shorthand table, so nothing drifts.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Complete 'mlr help' topics and topic arguments
'mlr help <TAB>' now completes help topics (flags, verb, function, keyword,
list-verbs, ...), and topics that take a name argument complete it too:
'mlr help verb <TAB>' -> verb names, 'mlr help function <TAB>' -> function
names, 'mlr help keyword <TAB>' -> keyword names, 'mlr help flag <TAB>' ->
flag names. 'mlr completion <TAB>' completes bash/zsh.
A terminal subcommand consumes the rest of the command line, so the walk now
returns a ctxTerminalArgs context carrying the terminal name and the words
typed after it. New getters supply the candidate names without drift:
help.GetTopicNames, help.GetFunctionNames/GetKeywordNames (wrapping new
cst.BuiltinFunctionManager.GetBuiltinFunctionNames and cst.GetKeywordNames).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs-neaten
* Move flag-value-candidate logic into pkg/cli; fix verb-flag collision
The mapping of which flags take a format/separator/regex-separator argument is
flag metadata, so it now lives with the flags in pkg/cli as
cli.FlagValueCandidates, alongside the existing GetFileFormatNames /
GetSeparatorAliasNames getters, replacing the maps that were in
pkg/terminals/completion/value_completion.go (now removed).
This also fixes a bug: value completion now applies only to main flags, not to
identically-spelled verb flags. Previously 'mlr uniq -o <TAB>' offered file
formats because uniq's -o (an output field name) collided with the main -o
format flag; it now correctly falls back to filename completion.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Adds a new Markdown-only flag --omd-aligned (alias --omarkdown-aligned)
that left-justifies cells and pads each column to a uniform width.
The rendered table is unaffected; the goal is readability of the raw
markdown source. The flag implies --omd, so users do not need to pass
both.
Implementation batches records by schema (like pprint), computes max
column widths, then emits header / separator / data rows with bars
vertically aligned. Default markdown writer behavior is unchanged.
* Switch to integer ranges in for loops
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Switch to slices functions where appropriate
A number of utility functions can be replaced outright; since Miller
can technically be used as a library, these are deprecated rather than
removed. go:fix directives ensure that they can be replaced
automatically.
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Switch to reflect.TypeFor
This is slightly more efficient than TypeOf when the type is known at
compile time.
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Switch to strings.SplitSeq instead of strings.Split
SplitSeq results in fewer allocations.
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Drop obsolete build directives
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Use min/max instead of explicit comparisons
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Append slices instead of looping
Signed-off-by: Stephen Kitt <steve@sk2.org>
---------
Signed-off-by: Stephen Kitt <steve@sk2.org>
* reader
* record writer
* pkg/output/record_writer_dcf.go
* test/input/test.dcf
* test/cases/io-dcf/
* pkg/cli/option_parse.go
* make fmt
* make dev
* docs and `make dev`
* docs/src/proposal-yaml-io.md
* initial step
* Add `--y2c` etc
* test cases
* docs
* more testing
* more
* more test cases
* git rm docs/src/proposal-yaml-io.md
* make dev
* ylistwrap -> yarray
* Switch to bufio.Reader, first pass
* temp
* Simplify ILineReader by making it stateless
* Interface not necessary; ILineReader -> TLineReader
* neaten
* iterating
