Phase 4 of plans/exit.md: the aux/terminal sub-entrypoints. pkg/auxents and
pkg/terminals are now os.Exit-free; even the dispatchers return exit codes
instead of exiting.
- auxents.Dispatch returns (handled bool, exitCode int); entrypoint.Main --
the one sanctioned exit point -- exits with the code.
- terminals.Dispatch returns an exit code; the climain caller wraps it in a
lib.ExitRequest, which also retires the 'terminal did not exit the process'
panic.
- Usage functions (hex, unhex, lecat, termcvt, repl, script, regtest) no
longer take an exitCode parameter and exit; call sites print usage and
return the code explicitly.
- Inner I/O helpers (hexDumpFile, unhexFile, lecatFile, termcvtFile) return
errors; each sub-main prints 'mlr {verb}: ...' as before and returns 1.
lecatFile formerly looped forever on a non-EOF read error; it now returns
the error.
- The regtest directory walk (Execute / executeSinglePath /
executeSingleDirectory / hasCaseSubdirectories) plumbs errors up to
RegTestMain; regression_test.go updated for the new Execute signature.
Stderr messages and exit codes are unchanged across mlr aux-list, hex, unhex,
lecat, termcvt, help, version, repl, script, and regtest surfaces (checked by
hand); all 4779 regression cases pass; make lint is clean; docs rebuild with
zero churn.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Remove os.Exit callsites below the entrypoint: phase 2 (plans/exit.md)
Phase 2 of plans/exit.md: the FlagParser signature change.
- FlagParser gains an error return; all 259 inline parser closures in
option_parse.go and NoOpParse1 updated (mechanical rewrite, compiler-checked).
- FlagTable.Parse returns (bool, error); its nine callers (climain passes one
and two, .mlrrc line handling, verb-local flag parsing in tee/join/put/
filter/split, and the repl/script terminals) propagate the error instead of
letting parsers kill the process.
- cli.CheckArgCount returns its missing-argument message as an error rather
than printing and exiting; new cli.FlagErrorf (counterpart of VerbErrorf)
carries user-facing main-flag messages.
- The 14 print-and-exit sites inside parser closures become returned errors;
--list-color-codes/--list-color-names return ExitRequest{0} after printing.
- The repl and script terminals translate ExitRequest into their int return
codes, so 'mlr repl --list-color-codes' still exits 0.
- pkg/cli is now os.Exit-free; stale comments about exiting parsers updated
(the completion introspection accessors remain, since parsers mutate the
options struct).
Stderr messages and exit codes are byte-identical, including the two-line
missing-argument and --seed messages and the trailing-period forms pinned by
test/cases/cli-mfrom/0003.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Merge main (recutils #2201); convert its three new parser closures
Same mechanical rewrite as the rest of phase 2: error return signature plus
final 'return nil' on --irecutils/--orecutils/--recutils.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Rebuild docs: help-catalog index count drifted on main (666 -> 667)
Pre-existing drift from a recent catalog addition; surfaced by make dev.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Remove os.Exit callsites below the entrypoint: phase 1 (plans/exit.md)
Phase 1 of plans/exit.md: the mechanical swaps, plus the single-exit-point
scaffolding.
- New sentinel lib.ExitRequest{Code} (io.EOF-style control flow): returned by
code paths that have already printed what they wanted (--version, -h,
'put --explain' on a valid expression, 'put -x') instead of exiting mid-stack.
- pkg/entrypoint: new exitOnError() maps errors to exit codes in one place:
ErrHelpRequested -> 0, ErrUsagePrinted -> 1, ExitRequest -> its code,
anything else printed (JSON if --errors-json) -> 1.
- pkg/climain: version/help/usage/validation exits become returned errors;
the ErrHelpRequested/ErrUsagePrinted -> exit translation moves up to the
entrypoint; loadMlrrcOrDie becomes error-returning loadMlrrcFiles.
- Transformer ParseCLI/constructor exits -> returned errors (cut,
having_fields, nest, reorder, merge_fields, reshape, rename, split, tee,
subs, put_or_filter). merge_fields' bad-regex message formerly mis-reported
itself as coming from the cut verb; tee's unrecognized-option exit was
formerly silent; split/tee constructor failures formerly exited without
printing the constructor's error.
- YAML/JSON record-writer marshal errors -> returned through Write, matching
the CSV writer's error path.
- lib.WriteTempFileOrDie -> WriteTempFile (string, error); its sole caller
(regtest diff helper) degrades gracefully.
Stderr messages and exit codes are byte-identical for all regression-covered
paths (4779 cases pass); runtime Transform-path exits are phase 3.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add lib.NewExitZeroRequest() constructor for exit-0 sentinel returns
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Add named profile sections to .mlrrc, selected via --profile / -P (#358)
.mlrrc files may now contain INI-style named sections ("profiles"):
# Global settings, applied always:
icsv
[j]
ojson
jvstack
[tsvout]
otsv
Lines before any section header are global settings, applied always, so
existing .mlrrc files behave exactly as before. The new main flag
--profile {name} (alias -P {name}) applies the settings from the [name]
section after the global settings; without it, sections are ignored
entirely (their lines aren't even parsed).
It's a fatal error if a requested profile has no matching section in any
.mlrrc file processed, if no .mlrrc file was found at all, or if
--profile is combined with --norc or MLRRC=__none__.
Also fixes the regression-tester to restore per-case environment
variables to their prior values after each case, rather than setting
them to the empty string -- needed so cases pointing MLRRC at a test
file don't clobber the suite-wide MLRRC=__none__ guard.
Includes unit tests, regression-test cases, and doc/man-page updates.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Reject --profile / -P within .mlrrc files (#358)
Per maintainer feedback on the PR: profiles are selected on the mlr
command line, not from within a .mlrrc file, so --profile / -P inside a
.mlrrc file is now a parse error -- the same way --prepipe is rejected
there -- rather than being silently ignored.
Adds a unit test, a should-fail regression case, and a docs bullet.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Lets an agent type-check a DSL expression before spending a full input
pass. `mlr put --explain '...'` (and filter) runs the existing
parse -> ValidateAST -> CST build -> Resolve path, then:
- valid: prints "mlr {put,filter}: DSL expression is valid." and exits 0
- invalid: returns the build error up the normal path, so --errors-json
emits a structured document; exits 1
- -W with fatal warnings: reports and exits 1
The gate lives in the pass-two constructor, before any input file is
opened, so no input stream is read (verified with a nonexistent input
file still validating OK).
Also categorize bare "parse error: ..." messages from the DSL parser as
kind "dsl-parse-error" rather than "generic" (climain/errors_json.go),
so --explain --errors-json gives an agent a useful error kind. The CSV
reader's "parse error on line ..." is stream-time and never reaches this
command-line-parse categorizer.
Tests: dsl-explain/0001-0004 regression cases (valid put/filter, invalid
plain, invalid --errors-json) and categorize unit tests. Regenerated
verb docs, manpage, and the help usage-verbs golden case.
The older -X ("exit after parsing") still exits 0 even on a parse error;
left as-is since --explain is the correct validation path.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Tier-2 structured verb options: OptionSpec, initial migration (#2098)
PR 3 of the AI-friendly roadmap (plans/plan-2098-llm.md).
Infrastructure:
- Add OptionSpec{Flag,Arg,Type,Desc,Repeatable,Values} to
pkg/transformers/aaa_record_transformer.go alongside TransformerSetup.
Type is one of: bool, string, int, float, csv-list, regex, filename,
format, enum. For type=="enum", Values lists the valid choices.
- Add Options []OptionSpec to TransformerSetup (nil = not yet migrated).
- Emit Options in VerbInfoForJSON (omitempty so unmigrated verbs stay
backward-compatible; agents check key presence for Tier-2 availability).
UsageText is always present as the Tier-1 prose fallback.
- Add VerbOptionsNilCheck() in aaa_verb_options_check.go: progress report
of migrated vs. unmigrated verbs, analogous to FLAG_TABLE.NilCheck().
- Wire verb-options-nil-check into mlr help (internal/docgen section).
Initial migration (5/70 verbs):
- nothing: empty Options (no verb-specific options, explicitly migrated)
- cat: -n (bool), -N (string), -g (csv-list), --filename, --filenum (bool)
- head: -g (csv-list), -n (int)
- tail: -g (csv-list), -n (int)
- tee: -a, -p (bool)
Tests:
- 5 new unit tests in aaa_transformer_json_test.go covering migrated/
unmigrated paths, field population, JSON round-trip, and key-presence.
- Regression test case 0003: mlr help verb-options-nil-check golden output.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Migrate all 70 verbs to structured OptionSpec; bump catalog schema to v2
Completes the Tier-2 migration started in the previous commit. Every verb
in TRANSFORMER_LOOKUP_TABLE now has a non-nil Options field.
- Workflow-migrated all 65 remaining verbs. Each Setup var now carries
Options: []OptionSpec{...} with Flag/Arg/Type/Desc fields. Verbs with
no verb-specific options (altkv, check, group-like, nothing, etc.) use
an empty slice to signal "migrated but no options."
- Drop `omitempty` from VerbInfoForJSON.Options: empty slices were silently
dropped, making migrated-no-option verbs indistinguishable from unmigrated
ones in JSON. Without omitempty: null=unmigrated, []=migrated-no-options,
[...]= migrated-with-options. Bump catalogSchemaVersion 1→2 for this shape
change.
- Replace the two "unmigrated-verb" unit tests (which used stats1 as an
example) with TestAllVerbsFullyMigrated (asserts every verb has non-nil
Options) and TestAllVerbsHaveOptionsKeyInJSON (asserts every migrated
verb emits the "options" key in JSON).
- Regenerate test/cases/cli-help/0003/expout: now reads
"Verb options migration: 70/70 migrated."
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Add --errors-json structured error output (#2098)
PR 4 of the AI-friendly roadmap (plans/plan-2098-llm.md).
Agents previously had to regex-match English prose to branch on error
kind; this PR lets them do it structurally.
Interface:
- mlr --errors-json <bad command> emits a JSON object to stderr and
exits 1; same behavior as prose path but machine-readable.
- MLR_ERRORS_JSON=1 (truthy) is the env-var equivalent.
- Without the flag, prose output is byte-for-byte unchanged.
JSON shape: {error, kind, token, verb, hint, did_you_mean[]}
Error kinds: unknown-verb, unknown-flag, verb-option-error, generic.
Implementation:
- New pkg/climain/errors_json.go: CLIError typed error, StructuredError
DTO, WantErrorsJSON pre-scan, Levenshtein edit distance, topMatches,
EmitStructuredError.
- Convert two direct os.Exit sites in parseCommandLinePassOne (unknown
verb, unknown flag) to return CLIError values; verb-option-error
likewise returns CLIError from pass-one ParseCLIFunc handling.
- parseCommandLinePassOne gains an error return; ParseCommandLine
propagates it.
- entrypoint.go pre-scans os.Args for --errors-json before calling
ParseCommandLine; routes errors to EmitStructuredError or printError.
- did_you_mean: Levenshtein nearest-match over verb catalog, flag
catalog, or the verb own OptionSpec flags (PR3 catalog) for
verb-option-error. Closes the self-correction loop the catalog
enables.
- --errors-json registered in Miscellaneous flags so it appears in
mlr --help and is not treated as an unrecognized flag.
Tests: 16 unit tests covering Levenshtein, topMatches, threshold,
WantErrorsJSON, CLIError interface, and all categorize paths.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* refine the plan
* Fix all staticcheck lint findings (uncapped)
golangci-lint's default max-same-issues=3 was hiding most of the backlog:
the true pre-fix count was 69 staticcheck findings, not 34. This fixes all
of them, driving staticcheck to zero:
- ST1023/QF1011 (37): omit explicit types inferred from the RHS
- S1009/S1031 (15): drop redundant nil checks before len()/range
- SA9003 (9): remove comment-only empty branches, keeping the comments
- QF1007 (3): merge conditional assignment into declaration
- QF1006 (3): lift break conditions into loop conditions
- QF1001 (3): apply De Morgan's law / name the negated predicate
Also updates plans/lintfixes.md with the cap discovery and the corrected
errcheck picture (1202 uncapped, ~949 of them fmt.Fprint*).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Drive errcheck to zero: config for bulk categories, propagate real errors
Adds .golangci.yml with errcheck exclude-functions for fmt.Fprint* (usage
printers), (*bufio.Writer).Write/WriteString (sticky errors, surfaced at the
now-checked final Flush), and (*strings.Builder).WriteString; pins
max-issues-per-linter/max-same-issues to 0 so CI reports true counts.
Real error paths now propagate instead of being dropped:
- Finalize{Reader,Writer}Options in join/put/filter/split/tee and the
repl/script entry points: 'mlr join -i badformat' now errors instead of
silently using wrong separators
- final output-stream Flush in pkg/stream: write failure no longer exits 0
- DSL emit/print/dump redirect writes, matching their sibling branches
- CSV writer WriteCSVRecordMaybeColorized, close-time Flush in file output
handlers, ENV[...] Setenv, REPL record-write and redirect-close errors
- termcvt write-side Close before rename (had "TODO: check return status")
The rest are deliberate ignores, marked with _ = and a comment where the
reason isn't obvious: unset-of-missing-path no-ops, read-side closes,
mid-stream FlushOnEveryRecord, init-time strftime registrations, in-memory
usage-capture pipes, and regtest-harness env/temp-file teardown.
golangci-lint now reports 0 issues on ./cmd/mlr ./pkg/... with all caps off.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Replaces 100+ if/else-if chains on a single variable with tagged switch
statements across 72 files. The bulk are transformer option-parsing loops
(switch on opt string), plus a handful of value-dispatch sites in mlrval,
dsl/cst, repl, lib, auxents, and bifs. One case (surv.go) required a
labeled break to preserve the loop-exit behavior of the original else branch.
Fixes staticcheck QF1003 findings.
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* initial attempt
* fix bash
* fix zsh
* Add shell-completion docs page
Documents the new 'mlr completion {bash,zsh}' feature: the then-chain
context model, install instructions for bash and zsh (including the macOS
bash-3.2 'eval' caveat and zsh compinit self-init), and examples of
context-aware completion. Added to the nav under "Miller in more detail".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Add enum value completion for format and separator flags
Completes the argument value for arg-taking main flags whose values are a
known set: file-format names for -i/-o/--io, separator aliases for
--ifs/--ofs/--ips/etc., and regex-separator aliases for --ifs-regex/--ips-regex.
Other arg-taking flags continue to fall back to filename completion.
Candidate sets come from new cli getters (GetFileFormatNames,
GetSeparatorAliasNames, GetSeparatorRegexAliasNames) that read the same maps
Miller uses at runtime, so there is no separate list to keep in sync. The
command-line walk now records which flag a value position belongs to.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Include format-conversion keystroke-savers in bare-dash completion
Reverts the suppression of --c2j/--x2y-style flags from 'mlr -<TAB>'. The full
set of main flags (297) is now offered, matching what is valid on the command
line. GetFlagNames no longer takes an includeSuppressed argument.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Complete terminal subcommands and top-level help/version flags
'mlr <TAB>' now offers subcommand names (help, version, repl, regtest, script,
completion, terminal-list) alongside verb names, and 'mlr -<TAB>' offers the
top-level terminal flags (-h, --help, --version, --bare-version, and the help
shorthands -g/-l/-L/-f/-F/-k/-K). Subcommand names are offered only as the
first non-flag token, where they are valid.
To let the completion engine know these names without an import cycle
(pkg/terminals imports pkg/terminals/completion), the canonical terminal names
and version-flag spellings are factored into a new leaf package
pkg/terminals/registry, imported by pkg/terminals, pkg/climain, and completion.
The help-flag spellings come from a new help.GetTerminalFlagNames derived from
the existing shorthand table, so nothing drifts.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Complete 'mlr help' topics and topic arguments
'mlr help <TAB>' now completes help topics (flags, verb, function, keyword,
list-verbs, ...), and topics that take a name argument complete it too:
'mlr help verb <TAB>' -> verb names, 'mlr help function <TAB>' -> function
names, 'mlr help keyword <TAB>' -> keyword names, 'mlr help flag <TAB>' ->
flag names. 'mlr completion <TAB>' completes bash/zsh.
A terminal subcommand consumes the rest of the command line, so the walk now
returns a ctxTerminalArgs context carrying the terminal name and the words
typed after it. New getters supply the candidate names without drift:
help.GetTopicNames, help.GetFunctionNames/GetKeywordNames (wrapping new
cst.BuiltinFunctionManager.GetBuiltinFunctionNames and cst.GetKeywordNames).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs-neaten
* Move flag-value-candidate logic into pkg/cli; fix verb-flag collision
The mapping of which flags take a format/separator/regex-separator argument is
flag metadata, so it now lives with the flags in pkg/cli as
cli.FlagValueCandidates, alongside the existing GetFileFormatNames /
GetSeparatorAliasNames getters, replacing the maps that were in
pkg/terminals/completion/value_completion.go (now removed).
This also fixes a bug: value completion now applies only to main flags, not to
identically-spelled verb flags. Previously 'mlr uniq -o <TAB>' offered file
formats because uniq's -o (an output field name) collided with the main -o
format flag; it now correctly falls back to filename completion.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* Default to "cat" verb when none is supplied (#2029)
Invocations like 'mlr --j2y' or 'mlr --c2p' previously failed with
"no verb supplied", forcing users to type the trailing 'cat'
explicitly for pure format conversions. Default the verb to 'cat'
in that case. Bare 'mlr' with no flags, no verb, and no files
still prints the main usage banner.
This handles flag-only invocations (e.g. 'mlr --c2j < input.csv'
or 'mlr --c2j --from input.csv'). File names without a preceding
verb are still parsed as verb candidates and continue to error if
not found; that broader change is out of scope here.
* Use ${MLR} substitution for bare-mlr regression case (#2029)
The regtester only substitutes the mlr executable when the cmd
starts with "mlr " (with a trailing space). The bare-mlr usage-banner
test had a cmd of just "mlr", so on CI -- which invokes regtest with
a relative path like 'test/../mlr' -- the test shelled out to a
literal 'mlr' that isn't on PATH and failed with exit 127.
Switch the cmd to ${MLR} (the regtester's explicit substitution
token) so the case runs the right binary in any invocation context.
* Switch to bufio.Reader, first pass
* temp
* Simplify ILineReader by making it stateless
* Interface not necessary; ILineReader -> TLineReader
* neaten
* iterating
The io/ioutil package has been deprecated as of Go 1.16 [1]. This commit
replaces the existing io/ioutil functions with their new definitions in
io and os packages.
[1]: https://golang.org/doc/go1.16#ioutil
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
