Phase 3 of plans/exit.md: the streaming-interface change, the load-bearing
piece for #341 (DSL exit statement) and #440 (strict mode).
- RecordTransformer.Transform and RecordTransformerFunc now return error.
All 69 Transform implementations and their dispatch helpers updated
(mechanical rewrite, compiler- and errcheck-verified).
- runSingleTransformerBatch, on a Transform error, forwards any output
produced before the failure plus an end-of-stream marker downstream, so
the rest of the chain and the record-writer drain and finish cleanly;
runSingleTransformer then surfaces the error to stream.Stream's select
loop (non-blocking send; first error wins) and signals upstream-done so
the record-reader stops. This is exactly the flush-then-exit sequencing a
future DSL 'exit N' needs.
- dataProcessingErrorChannel and FileOutputHandler.recordErroredChannel are
now chan error instead of chan bool; ChannelWriter still prints write-
error details at the site and sends the 'exiting due to data error'
sentinel, preserving the exact stderr shape pinned by regression cases.
- Mid-stream os.Exit sites converted to returned errors: put/filter DSL
begin/main/end-block errors and the non-boolean filter-expression case,
tee write/close failures, split write/open/close failures, join left-file
ingest failures (both half-streaming and sorted paths, with full error
plumbing through JoinBucketKeeper), histogram/stats2 ingest errors, surv
fit errors, and step stepper allocation (tStepperAllocator now returns
(tStepper, error); bad EWMA coefficients propagate; negative slwin
parameters are reported by the CLI parser via the existing
bad-stepper-name pattern).
- The two genuinely internal join-bucket-keeper states now use
lib.InternalCodingErrorWithMessageIf instead of hand-rolled print+exit.
- pkg/transformers is now os.Exit-free.
Behavior notes: the non-boolean filter message gains the standard 'mlr: '
prefix and a newline (it previously printed with neither); tee errors now
include the underlying cause. All 4779 regression cases pass unchanged;
mlr head early-out latency is unaffected (0.02s over 50M records).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Implements the plan in plans/recutils.md, addressing #378. Adds a
hand-rolled reader/writer pair (pkg/input/record_reader_rec.go,
pkg/output/record_writer_rec.go) supporting recutils' blank-line-separated
"Key: value" records, "+"-continuation and backslash-newline continuation,
and generic comment handling, with recutils' %rec-descriptor schema layer
left unspecialized since Miller has no schema-enforcement concept.
Registers the format under the "recutils" name (--irecutils/--orecutils/
--recutils flags, matching the --idcf/--odcf/--dcf pattern) in the
reader/writer factories and pkg/cli/separators.go, adds docs and a sample
data file, and adds unit tests plus test/cases/io-recutils regression
cases covering round-tripping, continuation lines, comments, and the
hard-error behavior on malformed input.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Rebuild docs: help-catalog index count drifted on main (666 -> 667)
Pre-existing drift from a recent catalog addition; surfaced by make dev.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Remove os.Exit callsites below the entrypoint: phase 1 (plans/exit.md)
Phase 1 of plans/exit.md: the mechanical swaps, plus the single-exit-point
scaffolding.
- New sentinel lib.ExitRequest{Code} (io.EOF-style control flow): returned by
code paths that have already printed what they wanted (--version, -h,
'put --explain' on a valid expression, 'put -x') instead of exiting mid-stack.
- pkg/entrypoint: new exitOnError() maps errors to exit codes in one place:
ErrHelpRequested -> 0, ErrUsagePrinted -> 1, ExitRequest -> its code,
anything else printed (JSON if --errors-json) -> 1.
- pkg/climain: version/help/usage/validation exits become returned errors;
the ErrHelpRequested/ErrUsagePrinted -> exit translation moves up to the
entrypoint; loadMlrrcOrDie becomes error-returning loadMlrrcFiles.
- Transformer ParseCLI/constructor exits -> returned errors (cut,
having_fields, nest, reorder, merge_fields, reshape, rename, split, tee,
subs, put_or_filter). merge_fields' bad-regex message formerly mis-reported
itself as coming from the cut verb; tee's unrecognized-option exit was
formerly silent; split/tee constructor failures formerly exited without
printing the constructor's error.
- YAML/JSON record-writer marshal errors -> returned through Write, matching
the CSV writer's error path.
- lib.WriteTempFileOrDie -> WriteTempFile (string, error); its sole caller
(regtest diff helper) degrades gracefully.
Stderr messages and exit codes are byte-identical for all regression-covered
paths (4779 cases pass); runtime Transform-path exits are phase 3.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add lib.NewExitZeroRequest() constructor for exit-0 sentinel returns
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The DKVPX record reader set dkvpxReader.Comma = ',' unconditionally,
ignoring the resolved --ifs value; the key-value separator '=' was
likewise hard-coded in the dkvpx parser, ignoring --ips. So e.g.
printf 'a=1;b=2\n' | mlr -i dkvpx --ifs semicolon --ojson cat
produced a single field {"a": "1;b=2"} instead of {"a": 1, "b": 2}.
Fixes:
* pkg/dkvpx.Reader gains an Equals field (default '=') alongside Comma.
* The DKVPX record reader now passes the resolved IFS/IPS through,
validating that each is a single character (mirroring the CSV
reader's "IFS can only be a single character" error) rather than
silently ignoring or truncating them.
* The DKVPX record writer already emitted OFS/OPS, but its quoting
decisions were based on the default separators, so with --ofs
semicolon a value containing ';' went out unquoted (ambiguous) while
a value containing ',' was quoted needlessly. Quoting now keys off
the actual OFS/OPS.
Adds unit tests, regression cases under test/cases/io-dkvpx/, and a
note in the separators reference table.
Discovered while investigating #369; tracked in #2170.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
With --right-align-numeric, PPRINT data cells right-align but headers
stayed left-aligned, so a header did not line up with its own column's
data -- the original ask in #380. Now a header is right-aligned when
every value in its column is numeric, for both non-barred and barred
PPRINT output. Mixed columns keep left-aligned headers.
For --omd-aligned, the raw header text of right-aligned columns is now
right-justified too, matching how Markdown viewers render the ---:
marker; this follows the same all-values-numeric per-column rule
already used for the separator markers.
Man-page regeneration also picks up previously-merged reorder help-text
edits that had not been regenerated.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
For PPRINT, right-justifies data cells whose values are numeric,
leaving other cells and header lines left-justified. For Markdown,
emits right-alignment markers (---:) for numeric columns; in
--omd-aligned mode also right-justifies the raw cell text.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Honor --ors CRLF for CSV output (#1810)
The full-CSV record writer validated ORS as newline or
carriage-return/newline, but never propagated the choice to the forked
Go CSV writer's UseCRLF field, so `--ors '\r\n'` (or `--ors crlf`)
silently produced LF line endings. Set UseCRLF from the writer options
so CRLF output is honored. Default behavior (LF) is unchanged, and
other ORS values are still rejected. The CSV-lite and TSV writers
already honored CRLF ORS.
Also: fix a copy-paste "for CSV" in the TSV writer's ORS-validation
message, add unit tests asserting byte-exact line endings (the regtest
harness normalizes CR/LF, so this can't be asserted in test/cases), add
CLI-level regression cases, and update the separators documentation.
This substantially addresses #1722 as well: RFC-4180-style CRLF output
can now be requested on any platform.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix errcheck lint: check Flush() error in CSV writer test
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Plan: flesh out PR7 (MCP server + Agent Skill) design
stdio transport (no HTTP port), mlr mcp terminal in the main binary,
SDK-vs-handroll decision, tool list, in-process vs subprocess split,
run-tool safety (--no-shell prerequisite), single-sourced skill, tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add mlr mcp: MCP server + agent playbook; --no-shell gate (#2098 PR7)
New terminal `mlr mcp` runs a Model Context Protocol server over stdio
(spawned by MCP clients; no network port), exposing five tools --
list_capabilities, which, validate_dsl, describe_data, run -- plus an
agent playbook as MCP prompt/resource. Catalog tools are served
in-process from the help registries; the rest subprocess this same
binary with MLR_ERRORS_JSON=1, a timeout, and an output cap.
Prerequisite: a new --no-shell flag / MLR_NO_SHELL env var (one-way
gate) disables the DSL system/exec functions, piped redirects, and
--prepipe/--prepipex; the MCP server sets it on the commands it runs
unless started with --allow-shell.
Adds the github.com/modelcontextprotocol/go-sdk dependency.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Force LF checkout for the embedded SKILL.md (Windows CI fix)
go:embed embeds checkout bytes, so a CRLF checkout on Windows made the
embedded playbook differ per platform and failed
TestPlaybookHasFrontmatter. Pin the file to eol=lf in .gitattributes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Move no-shell test DSL into per-case mlr files (Windows CI fix)
Inline single-quoted DSL in cmd files is mangled by the Windows shell
(single quotes are not quote characters there); the harness's
put -f ${CASEDIR}/mlr pattern avoids shell quoting entirely.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* refine the plan
* Fix all staticcheck lint findings (uncapped)
golangci-lint's default max-same-issues=3 was hiding most of the backlog:
the true pre-fix count was 69 staticcheck findings, not 34. This fixes all
of them, driving staticcheck to zero:
- ST1023/QF1011 (37): omit explicit types inferred from the RHS
- S1009/S1031 (15): drop redundant nil checks before len()/range
- SA9003 (9): remove comment-only empty branches, keeping the comments
- QF1007 (3): merge conditional assignment into declaration
- QF1006 (3): lift break conditions into loop conditions
- QF1001 (3): apply De Morgan's law / name the negated predicate
Also updates plans/lintfixes.md with the cap discovery and the corrected
errcheck picture (1202 uncapped, ~949 of them fmt.Fprint*).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Drive errcheck to zero: config for bulk categories, propagate real errors
Adds .golangci.yml with errcheck exclude-functions for fmt.Fprint* (usage
printers), (*bufio.Writer).Write/WriteString (sticky errors, surfaced at the
now-checked final Flush), and (*strings.Builder).WriteString; pins
max-issues-per-linter/max-same-issues to 0 so CI reports true counts.
Real error paths now propagate instead of being dropped:
- Finalize{Reader,Writer}Options in join/put/filter/split/tee and the
repl/script entry points: 'mlr join -i badformat' now errors instead of
silently using wrong separators
- final output-stream Flush in pkg/stream: write failure no longer exits 0
- DSL emit/print/dump redirect writes, matching their sibling branches
- CSV writer WriteCSVRecordMaybeColorized, close-time Flush in file output
handlers, ENV[...] Setenv, REPL record-write and redirect-close errors
- termcvt write-side Close before rename (had "TODO: check return status")
The rest are deliberate ignores, marked with _ = and a comment where the
reason isn't obvious: unset-of-missing-path no-ops, read-side closes,
mid-stream FlushOnEveryRecord, init-time strftime registrations, in-memory
usage-capture pipes, and regtest-harness env/temp-file teardown.
golangci-lint now reports 0 issues on ./cmd/mlr ./pkg/... with all caps off.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* plans/lintfixes.md
* plans/lintfixes.md
* Fix remaining govet lint findings
- Rename MarshalJSON -> FormatAsJSON on Mlrval and Mlrmap (govet
stdmethods): the methods shadowed json.Marshaler with an
incompatible signature.
- Remove unreachable return after exhaustive if-else in
pkg/mlrval/mlrval_collections.go (govet unreachable).
- Update plans/lintfixes.md with current status: 84 findings remain
(50 errcheck, 34 staticcheck).
Part of #2109.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Return nil on successful single-index array unset
removeIndexedOnArray removed the element on the in-bounds path but
then fell through to return an "array index out of bounds for unset"
error, so the success path never returned nil. Callers currently
ignore the error, which masked this; return nil on success so that
upcoming errcheck fixes can propagate the error meaningfully. This
matches removeIndexedOnMap, which returns nil on success.
Add unit tests for RemoveIndexed on arrays.
Part of #2109.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
After batch-arena field allocation, profiling cat over 1M-record CSV showed
the remaining ~5M allocations were almost entirely per-record (one each):
the Mlrmap struct, the RecordAndContext wrapper, the CSV writer's []string,
and the go-csv parser's own buffers.
Address the first three:
- mlrval.RecordArena gains NewRecord(), vending the Mlrmap struct itself from
a per-batch slab (respecting --no-hash-records). Rolled out to every
line-based reader (CSV, CSV-lite, TSV, DKVP, NIDX, PPRINT, XTAB, DKVPX) in
place of NewMlrmapAsRecord.
- The CSV reader batch-allocates RecordAndContext wrappers from a per-batch
slab instead of one heap object per record (comment/output-string entries
still allocate individually, but they are rare).
- RecordWriterCSV reuses a single fieldsBuffer []string across records instead
of allocating one per Write; WriteCSVRecordMaybeColorized consumes it
synchronously and the writer is single-goroutine, so this is safe.
Effect (big.*, 1M records, cat, best of 5):
csv 0.26 -> 0.22
dkvp 0.51 -> 0.45 (Mlrmap slab)
For CSV, cat's allocation-object count drops ~5.0M -> ~2.1M. The remaining
~2M are the go-csv parser's per-record backing string and field slice, which
are intrinsic to parsing and would require a zero-copy/batch-slab parser
rework. A CPU profile of cat now shows it is I/O-bound (syscall ~56%, bufio
read+flush), with allocation/GC down to ~10% -- i.e. further allocation
trimming no longer moves cat's wall-clock. GOGC=off confirms (no change).
Verified: go test ./pkg/... and full regression suite pass; output is
byte-identical across all formats including record-retaining verbs (tac),
hashed and --no-hash-records.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* Fix column alignment for wide and combining Unicode chars (#1520, #379)
PPRINT, markdown-aligned, and XTAB writers measured column widths with
utf8.RuneCountInString, which counts codepoints rather than terminal
display columns. East-Asian fullwidth characters (counted as 1 but
displayed as 2) and zero-width combining marks (counted as 1 but
displayed as 0) both caused misalignment.
Add lib.DisplayWidth wrapping uniseg.StringWidth, and use it from the
three writers. The XTAB right-aligned path also drops fmt.Sprintf with
%*s since Go's %*s pads by rune count too.
UTF8Strlen is unchanged so DSL strlen semantics are preserved.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Fix PPRINT alignment with multi-character OFS (#1819)
Previously, writePadding repeated the OFS string for each padding slot, so
non-space or multi-character OFS values produced misaligned columns and
leaked the separator into padding (e.g. --ofs=XY yielded aXYXYXYXYb).
Pad with spaces instead, leaving OFS to act only as the column separator.
The default single-space OFS is unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Use non-space multi-char OFS in test 0021 for Windows portability
PowerShell collapses single-quoted ' ' in command lines, which broke
the test on the windows-latest CI runner. Switch to --ofs XY, which
exercises the same multi-character padding code path without shell
quoting hazards.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PPRINT, markdown-aligned, and XTAB writers measured column widths with
utf8.RuneCountInString, which counts codepoints rather than terminal
display columns. East-Asian fullwidth characters (counted as 1 but
displayed as 2) and zero-width combining marks (counted as 1 but
displayed as 0) both caused misalignment.
Add lib.DisplayWidth wrapping uniseg.StringWidth, and use it from the
three writers. The XTAB right-aligned path also drops fmt.Sprintf with
%*s since Go's %*s pads by rune count too.
UTF8Strlen is unchanged so DSL strlen semantics are preserved.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a new Markdown-only flag --omd-aligned (alias --omarkdown-aligned)
that left-justifies cells and pads each column to a uniform width.
The rendered table is unaffected; the goal is readability of the raw
markdown source. The flag implies --omd, so users do not need to pass
both.
Implementation batches records by schema (like pprint), computes max
column widths, then emits header / separator / data rows with bars
vertically aligned. Default markdown writer behavior is unchanged.
* fix: preserve key order in YAML output
MlrmapToYAMLNative converted the ordered Mlrmap to map[string]interface{},
which loses insertion order. yaml.v3.Marshal then sorts Go map keys
alphabetically, so YAML output always had sorted keys despite other formats
(JSON, pretty-print) correctly preserving order.
Switch to building yaml.Node trees that preserve the Mlrmap's linked-list
iteration order. Use yaml.Node.Encode for scalar values to correctly handle
edge cases (NaN/Inf, non-UTF-8 strings).
Fixes#2028.
* Update pkg/mlrval/mlrval_yaml_test.go with copilot code-review feedback
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: lawrence3699 <lawrence3699@users.noreply.github.com>
Co-authored-by: John Kerl <kerl.john.r@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Switch to integer ranges in for loops
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Switch to slices functions where appropriate
A number of utility functions can be replaced outright; since Miller
can technically be used as a library, these are deprecated rather than
removed. go:fix directives ensure that they can be replaced
automatically.
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Switch to reflect.TypeFor
This is slightly more efficient than TypeOf when the type is known at
compile time.
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Switch to strings.SplitSeq instead of strings.Split
SplitSeq results in fewer allocations.
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Drop obsolete build directives
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Use min/max instead of explicit comparisons
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Append slices instead of looping
Signed-off-by: Stephen Kitt <steve@sk2.org>
---------
Signed-off-by: Stephen Kitt <steve@sk2.org>
* reader
* record writer
* pkg/output/record_writer_dcf.go
* test/input/test.dcf
* test/cases/io-dcf/
* pkg/cli/option_parse.go
* make fmt
* make dev
* docs and `make dev`
* docs/src/proposal-yaml-io.md
* initial step
* Add `--y2c` etc
* test cases
* docs
* more testing
* more
* more test cases
* git rm docs/src/proposal-yaml-io.md
* make dev
* ylistwrap -> yarray
* Auto-unsparsify CSV
* Update unit-test cases
* More unit-test cases
* Key-change handling for CSV output
* Same for TSV, with unit-test and doc updates
