With --right-align-numeric, PPRINT data cells right-align but headers
stayed left-aligned, so a header did not line up with its own column's
data -- the original ask in #380. Now a header is right-aligned when
every value in its column is numeric, for both non-barred and barred
PPRINT output. Mixed columns keep left-aligned headers.
For --omd-aligned, the raw header text of right-aligned columns is now
right-justified too, matching how Markdown viewers render the ---:
marker; this follows the same all-values-numeric per-column rule
already used for the separator markers.
Man-page regeneration also picks up previously-merged reorder help-text
edits that had not been regenerated.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
For PPRINT, right-justifies data cells whose values are numeric,
leaving other cells and header lines left-justified. For Markdown,
emits right-alignment markers (---:) for numeric columns; in
--omd-aligned mode also right-justifies the raw cell text.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Honor --ors CRLF for CSV output (#1810)
The full-CSV record writer validated ORS as newline or
carriage-return/newline, but never propagated the choice to the forked
Go CSV writer's UseCRLF field, so `--ors '\r\n'` (or `--ors crlf`)
silently produced LF line endings. Set UseCRLF from the writer options
so CRLF output is honored. Default behavior (LF) is unchanged, and
other ORS values are still rejected. The CSV-lite and TSV writers
already honored CRLF ORS.
Also: fix a copy-paste "for CSV" in the TSV writer's ORS-validation
message, add unit tests asserting byte-exact line endings (the regtest
harness normalizes CR/LF, so this can't be asserted in test/cases), add
CLI-level regression cases, and update the separators documentation.
This substantially addresses #1722 as well: RFC-4180-style CRLF output
can now be requested on any platform.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix errcheck lint: check Flush() error in CSV writer test
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Plan: flesh out PR7 (MCP server + Agent Skill) design
stdio transport (no HTTP port), mlr mcp terminal in the main binary,
SDK-vs-handroll decision, tool list, in-process vs subprocess split,
run-tool safety (--no-shell prerequisite), single-sourced skill, tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add mlr mcp: MCP server + agent playbook; --no-shell gate (#2098 PR7)
New terminal `mlr mcp` runs a Model Context Protocol server over stdio
(spawned by MCP clients; no network port), exposing five tools --
list_capabilities, which, validate_dsl, describe_data, run -- plus an
agent playbook as MCP prompt/resource. Catalog tools are served
in-process from the help registries; the rest subprocess this same
binary with MLR_ERRORS_JSON=1, a timeout, and an output cap.
Prerequisite: a new --no-shell flag / MLR_NO_SHELL env var (one-way
gate) disables the DSL system/exec functions, piped redirects, and
--prepipe/--prepipex; the MCP server sets it on the commands it runs
unless started with --allow-shell.
Adds the github.com/modelcontextprotocol/go-sdk dependency.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Force LF checkout for the embedded SKILL.md (Windows CI fix)
go:embed embeds checkout bytes, so a CRLF checkout on Windows made the
embedded playbook differ per platform and failed
TestPlaybookHasFrontmatter. Pin the file to eol=lf in .gitattributes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Move no-shell test DSL into per-case mlr files (Windows CI fix)
Inline single-quoted DSL in cmd files is mangled by the Windows shell
(single quotes are not quote characters there); the harness's
put -f ${CASEDIR}/mlr pattern avoids shell quoting entirely.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* refine the plan
* Fix all staticcheck lint findings (uncapped)
golangci-lint's default max-same-issues=3 was hiding most of the backlog:
the true pre-fix count was 69 staticcheck findings, not 34. This fixes all
of them, driving staticcheck to zero:
- ST1023/QF1011 (37): omit explicit types inferred from the RHS
- S1009/S1031 (15): drop redundant nil checks before len()/range
- SA9003 (9): remove comment-only empty branches, keeping the comments
- QF1007 (3): merge conditional assignment into declaration
- QF1006 (3): lift break conditions into loop conditions
- QF1001 (3): apply De Morgan's law / name the negated predicate
Also updates plans/lintfixes.md with the cap discovery and the corrected
errcheck picture (1202 uncapped, ~949 of them fmt.Fprint*).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Drive errcheck to zero: config for bulk categories, propagate real errors
Adds .golangci.yml with errcheck exclude-functions for fmt.Fprint* (usage
printers), (*bufio.Writer).Write/WriteString (sticky errors, surfaced at the
now-checked final Flush), and (*strings.Builder).WriteString; pins
max-issues-per-linter/max-same-issues to 0 so CI reports true counts.
Real error paths now propagate instead of being dropped:
- Finalize{Reader,Writer}Options in join/put/filter/split/tee and the
repl/script entry points: 'mlr join -i badformat' now errors instead of
silently using wrong separators
- final output-stream Flush in pkg/stream: write failure no longer exits 0
- DSL emit/print/dump redirect writes, matching their sibling branches
- CSV writer WriteCSVRecordMaybeColorized, close-time Flush in file output
handlers, ENV[...] Setenv, REPL record-write and redirect-close errors
- termcvt write-side Close before rename (had "TODO: check return status")
The rest are deliberate ignores, marked with _ = and a comment where the
reason isn't obvious: unset-of-missing-path no-ops, read-side closes,
mid-stream FlushOnEveryRecord, init-time strftime registrations, in-memory
usage-capture pipes, and regtest-harness env/temp-file teardown.
golangci-lint now reports 0 issues on ./cmd/mlr ./pkg/... with all caps off.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* plans/lintfixes.md
* plans/lintfixes.md
* Fix remaining govet lint findings
- Rename MarshalJSON -> FormatAsJSON on Mlrval and Mlrmap (govet
stdmethods): the methods shadowed json.Marshaler with an
incompatible signature.
- Remove unreachable return after exhaustive if-else in
pkg/mlrval/mlrval_collections.go (govet unreachable).
- Update plans/lintfixes.md with current status: 84 findings remain
(50 errcheck, 34 staticcheck).
Part of #2109.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Return nil on successful single-index array unset
removeIndexedOnArray removed the element on the in-bounds path but
then fell through to return an "array index out of bounds for unset"
error, so the success path never returned nil. Callers currently
ignore the error, which masked this; return nil on success so that
upcoming errcheck fixes can propagate the error meaningfully. This
matches removeIndexedOnMap, which returns nil on success.
Add unit tests for RemoveIndexed on arrays.
Part of #2109.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
After batch-arena field allocation, profiling cat over 1M-record CSV showed
the remaining ~5M allocations were almost entirely per-record (one each):
the Mlrmap struct, the RecordAndContext wrapper, the CSV writer's []string,
and the go-csv parser's own buffers.
Address the first three:
- mlrval.RecordArena gains NewRecord(), vending the Mlrmap struct itself from
a per-batch slab (respecting --no-hash-records). Rolled out to every
line-based reader (CSV, CSV-lite, TSV, DKVP, NIDX, PPRINT, XTAB, DKVPX) in
place of NewMlrmapAsRecord.
- The CSV reader batch-allocates RecordAndContext wrappers from a per-batch
slab instead of one heap object per record (comment/output-string entries
still allocate individually, but they are rare).
- RecordWriterCSV reuses a single fieldsBuffer []string across records instead
of allocating one per Write; WriteCSVRecordMaybeColorized consumes it
synchronously and the writer is single-goroutine, so this is safe.
Effect (big.*, 1M records, cat, best of 5):
csv 0.26 -> 0.22
dkvp 0.51 -> 0.45 (Mlrmap slab)
For CSV, cat's allocation-object count drops ~5.0M -> ~2.1M. The remaining
~2M are the go-csv parser's per-record backing string and field slice, which
are intrinsic to parsing and would require a zero-copy/batch-slab parser
rework. A CPU profile of cat now shows it is I/O-bound (syscall ~56%, bufio
read+flush), with allocation/GC down to ~10% -- i.e. further allocation
trimming no longer moves cat's wall-clock. GOGC=off confirms (no change).
Verified: go test ./pkg/... and full regression suite pass; output is
byte-identical across all formats including record-retaining verbs (tac),
hashed and --no-hash-records.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* Fix column alignment for wide and combining Unicode chars (#1520, #379)
PPRINT, markdown-aligned, and XTAB writers measured column widths with
utf8.RuneCountInString, which counts codepoints rather than terminal
display columns. East-Asian fullwidth characters (counted as 1 but
displayed as 2) and zero-width combining marks (counted as 1 but
displayed as 0) both caused misalignment.
Add lib.DisplayWidth wrapping uniseg.StringWidth, and use it from the
three writers. The XTAB right-aligned path also drops fmt.Sprintf with
%*s since Go's %*s pads by rune count too.
UTF8Strlen is unchanged so DSL strlen semantics are preserved.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Fix PPRINT alignment with multi-character OFS (#1819)
Previously, writePadding repeated the OFS string for each padding slot, so
non-space or multi-character OFS values produced misaligned columns and
leaked the separator into padding (e.g. --ofs=XY yielded aXYXYXYXYb).
Pad with spaces instead, leaving OFS to act only as the column separator.
The default single-space OFS is unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Use non-space multi-char OFS in test 0021 for Windows portability
PowerShell collapses single-quoted ' ' in command lines, which broke
the test on the windows-latest CI runner. Switch to --ofs XY, which
exercises the same multi-character padding code path without shell
quoting hazards.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PPRINT, markdown-aligned, and XTAB writers measured column widths with
utf8.RuneCountInString, which counts codepoints rather than terminal
display columns. East-Asian fullwidth characters (counted as 1 but
displayed as 2) and zero-width combining marks (counted as 1 but
displayed as 0) both caused misalignment.
Add lib.DisplayWidth wrapping uniseg.StringWidth, and use it from the
three writers. The XTAB right-aligned path also drops fmt.Sprintf with
%*s since Go's %*s pads by rune count too.
UTF8Strlen is unchanged so DSL strlen semantics are preserved.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a new Markdown-only flag --omd-aligned (alias --omarkdown-aligned)
that left-justifies cells and pads each column to a uniform width.
The rendered table is unaffected; the goal is readability of the raw
markdown source. The flag implies --omd, so users do not need to pass
both.
Implementation batches records by schema (like pprint), computes max
column widths, then emits header / separator / data rows with bars
vertically aligned. Default markdown writer behavior is unchanged.
* fix: preserve key order in YAML output
MlrmapToYAMLNative converted the ordered Mlrmap to map[string]interface{},
which loses insertion order. yaml.v3.Marshal then sorts Go map keys
alphabetically, so YAML output always had sorted keys despite other formats
(JSON, pretty-print) correctly preserving order.
Switch to building yaml.Node trees that preserve the Mlrmap's linked-list
iteration order. Use yaml.Node.Encode for scalar values to correctly handle
edge cases (NaN/Inf, non-UTF-8 strings).
Fixes#2028.
* Update pkg/mlrval/mlrval_yaml_test.go with copilot code-review feedback
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: lawrence3699 <lawrence3699@users.noreply.github.com>
Co-authored-by: John Kerl <kerl.john.r@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Switch to integer ranges in for loops
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Switch to slices functions where appropriate
A number of utility functions can be replaced outright; since Miller
can technically be used as a library, these are deprecated rather than
removed. go:fix directives ensure that they can be replaced
automatically.
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Switch to reflect.TypeFor
This is slightly more efficient than TypeOf when the type is known at
compile time.
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Switch to strings.SplitSeq instead of strings.Split
SplitSeq results in fewer allocations.
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Drop obsolete build directives
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Use min/max instead of explicit comparisons
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Append slices instead of looping
Signed-off-by: Stephen Kitt <steve@sk2.org>
---------
Signed-off-by: Stephen Kitt <steve@sk2.org>
* reader
* record writer
* pkg/output/record_writer_dcf.go
* test/input/test.dcf
* test/cases/io-dcf/
* pkg/cli/option_parse.go
* make fmt
* make dev
* docs and `make dev`
* docs/src/proposal-yaml-io.md
* initial step
* Add `--y2c` etc
* test cases
* docs
* more testing
* more
* more test cases
* git rm docs/src/proposal-yaml-io.md
* make dev
* ylistwrap -> yarray
* Auto-unsparsify CSV
* Update unit-test cases
* More unit-test cases
* Key-change handling for CSV output
* Same for TSV, with unit-test and doc updates
