Per maintainer feedback on the PR: profiles are selected on the mlr
command line, not from within a .mlrrc file, so --profile / -P inside a
.mlrrc file is now a parse error -- the same way --prepipe is rejected
there -- rather than being silently ignored.
Adds a unit test, a should-fail regression case, and a docs bullet.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
.mlrrc files may now contain INI-style named sections ("profiles"):
# Global settings, applied always:
icsv
[j]
ojson
jvstack
[tsvout]
otsv
Lines before any section header are global settings, applied always, so
existing .mlrrc files behave exactly as before. The new main flag
--profile {name} (alias -P {name}) applies the settings from the [name]
section after the global settings; without it, sections are ignored
entirely (their lines aren't even parsed).
It's a fatal error if a requested profile has no matching section in any
.mlrrc file processed, if no .mlrrc file was found at all, or if
--profile is combined with --norc or MLRRC=__none__.
Also fixes the regression-tester to restore per-case environment
variables to their prior values after each case, rather than setting
them to the empty string -- needed so cases pointing MLRRC at a test
file don't clobber the suite-wide MLRRC=__none__ guard.
Includes unit tests, regression-test cases, and doc/man-page updates.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Miller's strftime delegates to the full-featured lestrrat-go/strftime
library, but strptime uses an in-tree fork of a small subset-only
package, so strftime output couldn't always be parsed back by
strptime with the same format string. This adds the format codes
needed to round-trip common formats like "%a %b %e %T %Y":
* %a / %A (weekday name) and %h (alias of %b): straightforward
formatMap entries, same pattern as the existing %b/%B.
* %e (space-padded day of month): needed dedicated width-detection
logic, since %e's own optional leading pad space is otherwise
indistinguishable from a literal separator space when the code
searches for the next literal text to bound a field -- this was
the root cause of the round-trip failure in the linked issue.
* %c, %x, %X: shorthand aliases (expanding to %a %b %e %H:%M:%S %Y,
%m/%d/%y, and %H:%M:%S respectively), same mechanism already used
for %T/%D/%F/%R/%r.
Also documents the %D/%F/%r/%R/%T shorthands in the strptime table
in reference-dsl-time.md.in, which were already supported but
missing from the docs -- this was the exact confusion reported in
the discussion linked from #1518.
Round-trip tests added in pkg/bifs/datetime_test.go assert
strptime(strftime(t, fmt), fmt) == t across the newly-supported
formats, plus table-driven cases in
pkg/pbnjay-strptime/strptime_test.go covering %e's edge cases
(padded/unpadded single digit, double digit, adjacent to another
code with no separator, at end of string).
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* stats1: add rank accumulator (#383)
Adds `mlr stats1 -a rank` for standard competition ranking (1,2,2,4,...)
on pre-sorted data, most useful with -s for a rank on every record.
* stats1: make rank order-independent by default, add --rank-sorted opt-in fast path
The rank accumulator previously only compared each value to the immediately
preceding record, silently giving wrong ranks for non-adjacent duplicates
(e.g. unsorted input, or interleaved -g groups). Default to correctly
computing standard competition rank from all values seen so far
(order-independent, buffers values, same approach as percentile
accumulators). Add --rank-sorted for callers who can promise sorted input
and want the previous O(1)-space streaming behavior instead.
* Revert "stats1: make rank order-independent by default, add --rank-sorted opt-in fast path"
This reverts commit aa45a591fe.
* Revert "stats1: add rank accumulator (#383)"
This reverts commit 96deed048a.
* Add mlr rank verb (#383)
Reverts the earlier stats1 -a rank / --rank-sorted approach: stats1 is a
reduce verb (many records -> one summary record per group) and rank is a
per-record annotator, so it never fit cleanly there -- it needed stats1's
-s iterative-stats escape hatch just to be useful, plus a bolted-on
sorted/unsorted split.
mlr rank is a dedicated verb modeled on mlr fraction: -f fields to rank,
-g optional group-by, output field <f>_rank. By default it's a two-pass
algorithm (buffers input, like fraction does) giving standard competition
rank (1,2,2,4,...) that's correct regardless of input order. --sorted
opts into a single-pass, O(1)-space streaming alternative for callers who
can promise pre-sorted input (e.g. via 'mlr sort' first).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Regenerate docs/man pages after merging main (sparkline verb)
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Adds a sparkline(array|map) built-in that renders numeric values as a
Unicode block-character string (▁▂▃▄▅▆▇█), plus a new `mlr sparkline`
verb that summarizes each field's values in record order -- useful for
eyeballing trends without external plotting tools.
Also adds `-s` to `mlr histogram` to sparkline a field's binned counts
(its distribution shape) rather than emitting one record per bin. This
is a different chart from `mlr sparkline` (order-independent binning
vs. record-order values), and the docs for each cross-reference the
other to avoid conflating them.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
The DKVPX record reader set dkvpxReader.Comma = ',' unconditionally,
ignoring the resolved --ifs value; the key-value separator '=' was
likewise hard-coded in the dkvpx parser, ignoring --ips. So e.g.
printf 'a=1;b=2\n' | mlr -i dkvpx --ifs semicolon --ojson cat
produced a single field {"a": "1;b=2"} instead of {"a": 1, "b": 2}.
Fixes:
* pkg/dkvpx.Reader gains an Equals field (default '=') alongside Comma.
* The DKVPX record reader now passes the resolved IFS/IPS through,
validating that each is a single character (mirroring the CSV
reader's "IFS can only be a single character" error) rather than
silently ignoring or truncating them.
* The DKVPX record writer already emitted OFS/OPS, but its quoting
decisions were based on the default separators, so with --ofs
semicolon a value containing ';' went out unquoted (ambiguous) while
a value containing ',' was quoted needlessly. Quoting now keys off
the actual OFS/OPS.
Adds unit tests, regression cases under test/cases/io-dkvpx/, and a
note in the separators reference table.
Discovered while investigating #369; tracked in #2170.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The join verb's left-file ingest silently swallowed read errors: a
nonexistent or malformed left file produced empty (or partial) output
with exit code 0.
Two defects, one per ingest path:
* Unsorted (half-streaming) path, ingestLeftFile in
pkg/transformers/join.go: errors from the record-reader's error
channel were dropped ("TODO: propagate error to caller"), as were
record-reader construction errors. All left-file read failures were
silent.
* Sorted (-s) path, readRecord in
pkg/transformers/utils/join_bucket_keeper.go: the error channel was
consulted, but the record-reader goroutine sends the error and the
end-of-stream marker on separate channels, so the select could see
end-of-stream first and drop the error. A nonexistent left file with
-s exited 0 about half the time (racy).
Both paths now print "mlr: <error>" to stderr and exit 1, matching how
read errors on regular (right) input files are reported. On receipt of
the end-of-stream marker, both consumers now do a final non-blocking
drain of the error channel, which is deterministic since the reader
sends any error before the marker.
Discovered while verifying #377.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
With --right-align-numeric, PPRINT data cells right-align but headers
stayed left-aligned, so a header did not line up with its own column's
data -- the original ask in #380. Now a header is right-aligned when
every value in its column is numeric, for both non-barred and barred
PPRINT output. Mixed columns keep left-aligned headers.
For --omd-aligned, the raw header text of right-aligned columns is now
right-justified too, matching how Markdown viewers render the ---:
marker; this follows the same all-values-numeric per-column rule
already used for the separator markers.
Man-page regeneration also picks up previously-merged reorder help-text
edits that had not been regenerated.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The uniq verb outputs only the group-by columns, and issue #1075 asks
for a way to deduplicate on some fields while keeping the rest. Miller
already supports this via "head -n 1 -g", but that wasn't discoverable
from uniq's help text or its reference-verbs section. Add a note to
"mlr uniq --help" and a short recipe (with live examples) to the uniq
section of the verbs reference, and regenerate the derived man page,
docs, and CLI-help golden files.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
When the skip-trivial-records verb is in the then-chain, the CSV/TSV
record-readers now silently skip trivial (all-fields-empty) input lines
-- notably blank lines at the end of a CSV file -- instead of raising a
fatal header/data length mismatch. The user has explicitly asked for
trivial records to be skipped, so mlr now exits 0 in this case.
The verb's CLI parser sets a new ReaderOptions.SkipTrivialRecords flag,
which the CSV and TSV readers consult only on the would-be-fatal
mismatch path, so:
* Behavior without the verb is unchanged: blank lines still error.
* Genuinely ragged non-trivial records still error even with the verb.
* --allow-ragged-csv-input behavior is unchanged.
Fixes#1535.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
In the explicit-header ragged code paths of the csvlite, TSV, and
barred-pprint readers, the header-longer-than-data fill used
record.PutCopy with the un-deduplicated header name. With duplicate
header names (e.g. "name,4,4" deduped to "name,4,4_2"), the VOID fill
for the missing trailing field overwrote the already-stored value under
the colliding key, so a short row like "bar,sea" lost "sea".
Use record.PutReferenceMaybeDedupe, as the implicit-header paths
already do (where PR #794 had applied it, although implicit headers are
generated unique and never need dedupe).
Adds regression cases under test/cases/io-dedupe-field-names/ for the
csvlite, TSV, and barred-pprint readers.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Plain {} placeholders consume arguments sequentially as before; {N}
refers to the Nth argument (1-based) and may be used to repeat and/or
reorder arguments, e.g. format("{1}/{2}/{1}_{3}.ext", $p1, $p2, $p3).
Mixing is allowed: the {} sequence counter is independent of positional
placeholders (as in Rust format!). Out-of-range indices interpolate the
empty string, consistent with existing too-few-arguments behavior; {0}
is an error value since indices are 1-based.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
With multiple regexes, 'mlr reorder -r' previously emitted all matched
fields in record order, ignoring the order the regexes were given. Now
matched fields are grouped by regex-list order (first regex's matches
first, then the second's, etc.); within each group, fields keep their
record order. A field matching multiple regexes is claimed by the first
one. This makes 'reorder -r' consistent with 'cut -orf' and satisfies
the original request in #1325: -r '^YYY,^XXX' puts YYY-prefixed fields
first, then XXX-prefixed fields, then the rest. Applies to -e, -b, and
-a modes as well.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
For PPRINT, right-justifies data cells whose values are numeric,
leaving other cells and header lines left-justified. For Markdown,
emits right-alignment markers (---:) for numeric columns; in
--omd-aligned mode also right-justifies the raw cell text.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Honor --ors CRLF for CSV output (#1810)
The full-CSV record writer validated ORS as newline or
carriage-return/newline, but never propagated the choice to the forked
Go CSV writer's UseCRLF field, so `--ors '\r\n'` (or `--ors crlf`)
silently produced LF line endings. Set UseCRLF from the writer options
so CRLF output is honored. Default behavior (LF) is unchanged, and
other ORS values are still rejected. The CSV-lite and TSV writers
already honored CRLF ORS.
Also: fix a copy-paste "for CSV" in the TSV writer's ORS-validation
message, add unit tests asserting byte-exact line endings (the regtest
harness normalizes CR/LF, so this can't be asserted in test/cases), add
CLI-level regression cases, and update the separators documentation.
This substantially addresses #1722 as well: RFC-4180-style CRLF output
can now be requested on any platform.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix errcheck lint: check Flush() error in CSV writer test
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The summary verb's field_type column shows values like "string-int" or
"empty-string" for columns containing values of mixed inferred types:
all types encountered across records are printed, hyphen-joined, in the
order first encountered. Add a note to the verb's help text explaining
this, and regenerate the man page, docs, and golden test output.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Miller type-inference accepts float formats which the JSON grammar
disallows -- leading zeros like 004.56, leading plus signs, bare leading
or trailing decimal points like .56 or 4., etc. -- and the JSON writer
passed the original text through verbatim, producing invalid JSON.
Leading-zero integers such as 0123456789 were already handled (inferred
as strings), and hex/binary/octal ints were already re-rendered as
decimal for JSON per #1761; this does the same for floats: when a
float string representation is not a valid JSON number, re-render it
(e.g. 004.56 -> 4.56).
Also documents in reference-main-data-types that CSV double-quoting
does not affect type inference, with -S / --infer-none and string()
as the ways to keep such values as strings.
Addresses #1114 and #1293.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The tee verb passed the same record pointer both to its file-output
handler (which writes asynchronously on another goroutine, and whose
buffering formats like pprint/json can hold records until close) and
downstream in the verb chain, where subsequent verbs mutate records in
place. Downstream mutations could therefore leak into the tee'd output:
mlr tee -p cat then cat -n then nothing <<EOF
a=1,b=2
a=3,b=4
a=5,b=6
EOF
emitted "n=3,a=5,b=6" for the last tee'd record, and with pprint/json
tee formats every record picked up downstream fields.
Fix: give the file-writer its own deep copy of the record. Same fix
applied to the split verb when -v (emit downstream) is used, which
shared records with downstream the same way.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The join verb's help text listed only '-i {one of csv,dkvp,nidx,pprint,xtab}'
for overriding the left-file input format, but the verb also accepts the
--icsv/--ijson-style main-flag shorthands (and formats beyond the five
listed, e.g. json and tsv), since unrecognized verb flags fall through to
the main flag table. Update the usage text to say so, and regenerate the
man page and docs content that embed this help output.
Fixes#444.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The uniq verb writes its group-by fields in the order they are named
with -g or -f, not in the order they appear in the input records
(unlike cut, which preserves input order unless -o is given). Per
discussion on #962, document this in the verb help text rather than
change long-standing output ordering. Regenerate the man page and the
docs pages which embed the verb help.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Plan: flesh out PR7 (MCP server + Agent Skill) design
stdio transport (no HTTP port), mlr mcp terminal in the main binary,
SDK-vs-handroll decision, tool list, in-process vs subprocess split,
run-tool safety (--no-shell prerequisite), single-sourced skill, tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add mlr mcp: MCP server + agent playbook; --no-shell gate (#2098 PR7)
New terminal `mlr mcp` runs a Model Context Protocol server over stdio
(spawned by MCP clients; no network port), exposing five tools --
list_capabilities, which, validate_dsl, describe_data, run -- plus an
agent playbook as MCP prompt/resource. Catalog tools are served
in-process from the help registries; the rest subprocess this same
binary with MLR_ERRORS_JSON=1, a timeout, and an output cap.
Prerequisite: a new --no-shell flag / MLR_NO_SHELL env var (one-way
gate) disables the DSL system/exec functions, piped redirects, and
--prepipe/--prepipex; the MCP server sets it on the commands it runs
unless started with --allow-shell.
Adds the github.com/modelcontextprotocol/go-sdk dependency.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Force LF checkout for the embedded SKILL.md (Windows CI fix)
go:embed embeds checkout bytes, so a CRLF checkout on Windows made the
embedded playbook differ per platform and failed
TestPlaybookHasFrontmatter. Pin the file to eol=lf in .gitattributes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Move no-shell test DSL into per-case mlr files (Windows CI fix)
Inline single-quoted DSL in cmd files is mangled by the Windows shell
(single quotes are not quote characters there); the harness's
put -f ${CASEDIR}/mlr pattern avoids shell quoting entirely.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
One output record per input field: types seen with counts, occurrence
count, null count, cardinality, min/max, and -- for fields within the
-n/--max-values cap -- the complete distinct-value list in first-seen
order. `mlr --ojson describe` is the machine-readable form; nested
types/values flatten in tabular formats.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Lets an agent type-check a DSL expression before spending a full input
pass. `mlr put --explain '...'` (and filter) runs the existing
parse -> ValidateAST -> CST build -> Resolve path, then:
- valid: prints "mlr {put,filter}: DSL expression is valid." and exits 0
- invalid: returns the build error up the normal path, so --errors-json
emits a structured document; exits 1
- -W with fatal warnings: reports and exits 1
The gate lives in the pass-two constructor, before any input file is
opened, so no input stream is read (verified with a nonexistent input
file still validating OK).
Also categorize bare "parse error: ..." messages from the DSL parser as
kind "dsl-parse-error" rather than "generic" (climain/errors_json.go),
so --explain --errors-json gives an agent a useful error kind. The CSV
reader's "parse error on line ..." is stream-time and never reaches this
command-line-parse categorizer.
Tests: dsl-explain/0001-0004 regression cases (valid put/filter, invalid
plain, invalid --errors-json) and categorize unit tests. Regenerated
verb docs, manpage, and the help usage-verbs golden case.
The older -X ("exit after parsing") still exits 0 even on a parse error;
left as-is since --explain is the correct validation path.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Tier-2 structured verb options: OptionSpec, initial migration (#2098)
PR 3 of the AI-friendly roadmap (plans/plan-2098-llm.md).
Infrastructure:
- Add OptionSpec{Flag,Arg,Type,Desc,Repeatable,Values} to
pkg/transformers/aaa_record_transformer.go alongside TransformerSetup.
Type is one of: bool, string, int, float, csv-list, regex, filename,
format, enum. For type=="enum", Values lists the valid choices.
- Add Options []OptionSpec to TransformerSetup (nil = not yet migrated).
- Emit Options in VerbInfoForJSON (omitempty so unmigrated verbs stay
backward-compatible; agents check key presence for Tier-2 availability).
UsageText is always present as the Tier-1 prose fallback.
- Add VerbOptionsNilCheck() in aaa_verb_options_check.go: progress report
of migrated vs. unmigrated verbs, analogous to FLAG_TABLE.NilCheck().
- Wire verb-options-nil-check into mlr help (internal/docgen section).
Initial migration (5/70 verbs):
- nothing: empty Options (no verb-specific options, explicitly migrated)
- cat: -n (bool), -N (string), -g (csv-list), --filename, --filenum (bool)
- head: -g (csv-list), -n (int)
- tail: -g (csv-list), -n (int)
- tee: -a, -p (bool)
Tests:
- 5 new unit tests in aaa_transformer_json_test.go covering migrated/
unmigrated paths, field population, JSON round-trip, and key-presence.
- Regression test case 0003: mlr help verb-options-nil-check golden output.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Migrate all 70 verbs to structured OptionSpec; bump catalog schema to v2
Completes the Tier-2 migration started in the previous commit. Every verb
in TRANSFORMER_LOOKUP_TABLE now has a non-nil Options field.
- Workflow-migrated all 65 remaining verbs. Each Setup var now carries
Options: []OptionSpec{...} with Flag/Arg/Type/Desc fields. Verbs with
no verb-specific options (altkv, check, group-like, nothing, etc.) use
an empty slice to signal "migrated but no options."
- Drop `omitempty` from VerbInfoForJSON.Options: empty slices were silently
dropped, making migrated-no-option verbs indistinguishable from unmigrated
ones in JSON. Without omitempty: null=unmigrated, []=migrated-no-options,
[...]= migrated-with-options. Bump catalogSchemaVersion 1→2 for this shape
change.
- Replace the two "unmigrated-verb" unit tests (which used stats1 as an
example) with TestAllVerbsFullyMigrated (asserts every verb has non-nil
Options) and TestAllVerbsHaveOptionsKeyInJSON (asserts every migrated
verb emits the "options" key in JSON).
- Regenerate test/cases/cli-help/0003/expout: now reads
"Verb options migration: 70/70 migrated."
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Add --errors-json structured error output (#2098)
PR 4 of the AI-friendly roadmap (plans/plan-2098-llm.md).
Agents previously had to regex-match English prose to branch on error
kind; this PR lets them do it structurally.
Interface:
- mlr --errors-json <bad command> emits a JSON object to stderr and
exits 1; same behavior as prose path but machine-readable.
- MLR_ERRORS_JSON=1 (truthy) is the env-var equivalent.
- Without the flag, prose output is byte-for-byte unchanged.
JSON shape: {error, kind, token, verb, hint, did_you_mean[]}
Error kinds: unknown-verb, unknown-flag, verb-option-error, generic.
Implementation:
- New pkg/climain/errors_json.go: CLIError typed error, StructuredError
DTO, WantErrorsJSON pre-scan, Levenshtein edit distance, topMatches,
EmitStructuredError.
- Convert two direct os.Exit sites in parseCommandLinePassOne (unknown
verb, unknown flag) to return CLIError values; verb-option-error
likewise returns CLIError from pass-one ParseCLIFunc handling.
- parseCommandLinePassOne gains an error return; ParseCommandLine
propagates it.
- entrypoint.go pre-scans os.Args for --errors-json before calling
ParseCommandLine; routes errors to EmitStructuredError or printError.
- did_you_mean: Levenshtein nearest-match over verb catalog, flag
catalog, or the verb own OptionSpec flags (PR3 catalog) for
verb-option-error. Closes the self-correction loop the catalog
enables.
- --errors-json registered in Miscellaneous flags so it appears in
mlr --help and is not treated as an unrecognized flag.
Tests: 16 unit tests covering Levenshtein, topMatches, threshold,
WantErrorsJSON, CLIError interface, and all categorize paths.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Tier-2 structured verb options: OptionSpec, initial migration (#2098)
PR 3 of the AI-friendly roadmap (plans/plan-2098-llm.md).
Infrastructure:
- Add OptionSpec{Flag,Arg,Type,Desc,Repeatable,Values} to
pkg/transformers/aaa_record_transformer.go alongside TransformerSetup.
Type is one of: bool, string, int, float, csv-list, regex, filename,
format, enum. For type=="enum", Values lists the valid choices.
- Add Options []OptionSpec to TransformerSetup (nil = not yet migrated).
- Emit Options in VerbInfoForJSON (omitempty so unmigrated verbs stay
backward-compatible; agents check key presence for Tier-2 availability).
UsageText is always present as the Tier-1 prose fallback.
- Add VerbOptionsNilCheck() in aaa_verb_options_check.go: progress report
of migrated vs. unmigrated verbs, analogous to FLAG_TABLE.NilCheck().
- Wire verb-options-nil-check into mlr help (internal/docgen section).
Initial migration (5/70 verbs):
- nothing: empty Options (no verb-specific options, explicitly migrated)
- cat: -n (bool), -N (string), -g (csv-list), --filename, --filenum (bool)
- head: -g (csv-list), -n (int)
- tail: -g (csv-list), -n (int)
- tee: -a, -p (bool)
Tests:
- 5 new unit tests in aaa_transformer_json_test.go covering migrated/
unmigrated paths, field population, JSON round-trip, and key-presence.
- Regression test case 0003: mlr help verb-options-nil-check golden output.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Migrate all 70 verbs to structured OptionSpec; bump catalog schema to v2
Completes the Tier-2 migration started in the previous commit. Every verb
in TRANSFORMER_LOOKUP_TABLE now has a non-nil Options field.
- Workflow-migrated all 65 remaining verbs. Each Setup var now carries
Options: []OptionSpec{...} with Flag/Arg/Type/Desc fields. Verbs with
no verb-specific options (altkv, check, group-like, nothing, etc.) use
an empty slice to signal "migrated but no options."
- Drop `omitempty` from VerbInfoForJSON.Options: empty slices were silently
dropped, making migrated-no-option verbs indistinguishable from unmigrated
ones in JSON. Without omitempty: null=unmigrated, []=migrated-no-options,
[...]= migrated-with-options. Bump catalogSchemaVersion 1→2 for this shape
change.
- Replace the two "unmigrated-verb" unit tests (which used stats1 as an
example) with TestAllVerbsFullyMigrated (asserts every verb has non-nil
Options) and TestAllVerbsHaveOptionsKeyInJSON (asserts every migrated
verb emits the "options" key in JSON).
- Regenerate test/cases/cli-help/0003/expout: now reads
"Verb options migration: 70/70 migrated."
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* remove a transitional helper
* git rms
* Render verb usage Options blocks from structured OptionSpec
Each verb's usage message and its Tier-2 OptionSpec list previously
duplicated the option text. New WriteVerbOptions (aaa_verb_usage.go)
renders the "Options:" block from the specs: aligned flag column,
descriptions word-wrapped at 80, uniform trailing -h|--help line.
- OptionSpec gains Aliases (JSON "aliases") so long-form spellings
like join's --lk|--left-keep-field-names survive in both outputs
- All 70 verbs migrated; options literals hoisted to package-level
vars (usage funcs can't reference their Setup var without a Go
init cycle)
- Hand-written per-option details the specs had condensed away are
merged into Desc, enriching the JSON catalog
- Non-option prose (examples, cross-references, dynamic accumulator
listings) kept verbatim
- Regenerated the six usage-embedding regression expectations and
the two affected doc pages
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix pre-existing usage-text bugs surfaced by the OptionSpec migration
- gap: usage said "One of -f or -g is required" but the parser takes
-n or -g
- seqgen: drop description line copy-pasted from cat ("Passes input
records directly to output...") which contradicted "Discards the
input record stream"
- utf8-to-latin1: description read inverted ("from Latin-1 to UTF-8")
- sec2gmtdate: usage said "../c/mlr" instead of "mlr"
- top: document the accepted-but-undocumented --max flag
- stats2: add linreg-pca to the -a enum values, matching the runtime
accumulator table
Regression expectations and docs regenerated accordingly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix check usage sentence order; stats1 usage blank line to usage stream
- check: the description's second and third lines were swapped,
reading "Consumes records without printing any output, / Useful for
doing a well-formatted check on input data. / with the exception
that warnings are printed to stderr."
- stats1: a bare fmt.Println() in the usage func wrote its blank line
to process stdout instead of the usage output stream
Regression expectation and docs regenerated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Add MT_BYTES mlrval type: foundation and disposition tables
First step toward a first-class bytes type in the DSL (#1231).
Adds MT_BYTES (payload []byte, rendered as lowercase hex in all output
formats, JSON-encoded as a hex string), extends every disposition
matrix/vector with the new row/column -- real cells for comparison,
sorting, and dot-concat of bytes with bytes; type-error stubs
elsewhere -- and adds sweep tests asserting no table has nil cells,
since Go zero-fills short array literals when MT_DIM grows.
Bytes values are not yet constructible from the DSL; b"..." literals
and constructor/codec functions follow in subsequent commits.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add b"..." bytes-literal syntax to the DSL
Adds a bytes_literal token to the grammar (regenerating the PGPG lexer
and parser) and a BytesLiteralNode in the CST which evaluates to an
MT_BYTES mlrval. Escape handling reuses UnbackslashStringLiteral,
which is already byte-oriented: b"\xff" is the single byte 0xff.
Unlike string literals, bytes literals never participate in
regex-capture replacement. A bare identifier b is unaffected.
Part of #1231.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add bytes DSL functions: conversions, codecs, and bytes-aware built-ins
- bytes(x) converts strings to bytes; string(b) reinterprets raw bytes
as UTF-8 text (the reverse)
- base64_decode now always returns bytes (superseding the interim
string-or-hex behavior); base64_encode accepts string or bytes
- New hex_encode/hex_decode functions
- is_bytes and asserting_bytes predicates
- md5/sha1/sha256/sha512 accept bytes, hashing the raw payload
- strlen of bytes is the byte count; substr/substr0/substr1 on bytes
slice by byte position and return bytes
The Cyrillic-LDAP scenario from #1231 now works without exec
workarounds: string(base64_decode($x)) recovers the text, and binary
payloads survive undamaged as bytes.
Closes#1231.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add bytes-type docs and regression cases
Documents the bytes type on the data-types page, regenerates the
function-reference/man-page material, and adds regression coverage:
literal escape forms, operators (concat/compare/slice/sort and
type errors), conversions and codec round-trips, and CSV-to-JSON
output rendering of bytes fields.
Part of #1231.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Reposition MT_BYTES to sort adjacent to MT_STRING in the type enum
MT_BYTES was appended after MT_ABSENT for index stability; move it
right after MT_STRING instead, since that's where it conceptually
belongs and where it already sorts in the cmp disposition matrices.
Mechanically re-derive all ~40 disposition tables in pkg/bifs and
pkg/mlrval accordingly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fix windows CI
* fix merge
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* refine the plan
* Fix all staticcheck lint findings (uncapped)
golangci-lint's default max-same-issues=3 was hiding most of the backlog:
the true pre-fix count was 69 staticcheck findings, not 34. This fixes all
of them, driving staticcheck to zero:
- ST1023/QF1011 (37): omit explicit types inferred from the RHS
- S1009/S1031 (15): drop redundant nil checks before len()/range
- SA9003 (9): remove comment-only empty branches, keeping the comments
- QF1007 (3): merge conditional assignment into declaration
- QF1006 (3): lift break conditions into loop conditions
- QF1001 (3): apply De Morgan's law / name the negated predicate
Also updates plans/lintfixes.md with the cap discovery and the corrected
errcheck picture (1202 uncapped, ~949 of them fmt.Fprint*).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Drive errcheck to zero: config for bulk categories, propagate real errors
Adds .golangci.yml with errcheck exclude-functions for fmt.Fprint* (usage
printers), (*bufio.Writer).Write/WriteString (sticky errors, surfaced at the
now-checked final Flush), and (*strings.Builder).WriteString; pins
max-issues-per-linter/max-same-issues to 0 so CI reports true counts.
Real error paths now propagate instead of being dropped:
- Finalize{Reader,Writer}Options in join/put/filter/split/tee and the
repl/script entry points: 'mlr join -i badformat' now errors instead of
silently using wrong separators
- final output-stream Flush in pkg/stream: write failure no longer exits 0
- DSL emit/print/dump redirect writes, matching their sibling branches
- CSV writer WriteCSVRecordMaybeColorized, close-time Flush in file output
handlers, ENV[...] Setenv, REPL record-write and redirect-close errors
- termcvt write-side Close before rename (had "TODO: check return status")
The rest are deliberate ignores, marked with _ = and a comment where the
reason isn't obvious: unset-of-missing-path no-ops, read-side closes,
mid-stream FlushOnEveryRecord, init-time strftime registrations, in-memory
usage-capture pipes, and regtest-harness env/temp-file teardown.
golangci-lint now reports 0 issues on ./cmd/mlr ./pkg/... with all caps off.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* plans/lintfixes.md
* plans/lintfixes.md
* Fix remaining govet lint findings
- Rename MarshalJSON -> FormatAsJSON on Mlrval and Mlrmap (govet
stdmethods): the methods shadowed json.Marshaler with an
incompatible signature.
- Remove unreachable return after exhaustive if-else in
pkg/mlrval/mlrval_collections.go (govet unreachable).
- Update plans/lintfixes.md with current status: 84 findings remain
(50 errcheck, 34 staticcheck).
Part of #2109.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Return nil on successful single-index array unset
removeIndexedOnArray removed the element on the in-bounds path but
then fell through to return an "array index out of bounds for unset"
error, so the success path never returned nil. Callers currently
ignore the error, which masked this; return nil on success so that
upcoming errcheck fixes can propagate the error meaningfully. This
matches removeIndexedOnMap, which returns nil on success.
Add unit tests for RemoveIndexed on arrays.
Part of #2109.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Found with golang.org/x/tools/cmd/deadcode (rooted at cmd/mlr + tests)
and staticcheck U1000; each finding verified by hand before deletion.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Replaces 100+ if/else-if chains on a single variable with tagged switch
statements across 72 files. The bulk are transformer option-parsing loops
(switch on opt string), plus a handful of value-dispatch sites in mlrval,
dsl/cst, repl, lib, auxents, and bifs. One case (surv.go) required a
labeled break to preserve the loop-exit behavior of the original else branch.
Fixes staticcheck QF1003 findings.
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Add JSON index and mlr which capability router (#2098)
PR 2 of the AI-friendly roadmap (plans/plan-2098-llm.md):
- `mlr help --as-json --index` emits a lightweight [{kind,name,summary}]
index across all 651 catalog items (verbs, functions, flags, keywords),
sorted by kind then name. Agents use this as a cheap first call to pick
a verb before fetching its full entry.
- `mlr which "<query>"` is a new terminal that tokenizes a natural-language
query, scores every catalog item (name match +20/token, body match +5/token),
and returns ranked JSON [{kind,name,score,summary}]. Exit code 0 means a
confident match (at least one token hit the item name); exit code 2 means
low confidence. Agents branch on the exit code rather than parsing prose.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Move mlr which into pkg/terminals/help, deduplicate firstLine
pkg/terminals/which/ was a misplaced package: it imported the same four
catalog registries as pkg/terminals/help/ and duplicated the firstLine
helper. Moving the logic into help/entry_which.go fixes both issues:
- WhichMain and all which helpers now live alongside the other --as-json
catalog machinery in pkg/terminals/help/
- indexFirstLine (entry_json.go) and firstLine (which/entry.go) collapse
into a single firstLine shared by both files
- pkg/terminals/terminals.go calls help.WhichMain directly; the
pkg/terminals/which/ package is deleted
No behavior change.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Fix five Go-purity issues found in code review
1. Exit-code false positive: whichScore now returns (int, bool) where the
bool records whether any token hit the item name. WhichMain uses
results[0].nameHit for exit-code 0, so 4 body-only token hits (4×5=20)
no longer incorrectly signal a confident match.
2. Flag Summary inconsistency: whichSearch was setting Summary: fl.Help
directly for flags while using firstLine(...) for functions and keywords.
Changed to firstLine(fl.Help) so all four kinds behave consistently.
3+5. kindOrder/whichKindRank duplication: the verb<function<flag<keyword
ordering was encoded twice — as a local map[string]int in buildIndex and
as a switch in whichKindRank. Replaced both with a single package-level
kindRank() function. The map lookup also silently returned 0 (= verb rank)
for unknown kinds; the switch correctly returns 4 (sorts last).
4. extractIndexFlag/extractAsJSONFlag duplication: both had identical loop
bodies differing only in the sentinel string. Introduced a generic
extractFlag(args, flag) helper; both are now one-liners.
Also promoted whichStopwords to a package-level var so whichTokenize does
not allocate a new map on every call.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Emit Miller's existing help catalog (verbs, functions, flags, keywords)
as structured JSON so AI agents and tooling can model Miller's surface
without scraping prose. The --json token may appear anywhere on a
`mlr help ...` command line; plain text help is unchanged.
mlr help --json # full catalog
mlr help verb cat --json # one or more verbs
mlr help function splitax --json # one or more functions
mlr help flag --ifs --json # one or more flags
mlr help keyword ENV --json # one or more keywords
Functions and flags serialize fully (name/class/arity/help/examples;
section/name/alt_names/arg/help). Verbs carry a summary, ignores_input,
and captured raw usage_text as a Tier-1 fallback, since per-verb options
are prose-only today (each verb hand-writes its UsageFunc). Structured
verb options are a planned follow-on (see #2098).
This is a serialization layer over the existing registries -- no
refactor of the text-help path.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* initial attempt
* fix bash
* fix zsh
* Add shell-completion docs page
Documents the new 'mlr completion {bash,zsh}' feature: the then-chain
context model, install instructions for bash and zsh (including the macOS
bash-3.2 'eval' caveat and zsh compinit self-init), and examples of
context-aware completion. Added to the nav under "Miller in more detail".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Add enum value completion for format and separator flags
Completes the argument value for arg-taking main flags whose values are a
known set: file-format names for -i/-o/--io, separator aliases for
--ifs/--ofs/--ips/etc., and regex-separator aliases for --ifs-regex/--ips-regex.
Other arg-taking flags continue to fall back to filename completion.
Candidate sets come from new cli getters (GetFileFormatNames,
GetSeparatorAliasNames, GetSeparatorRegexAliasNames) that read the same maps
Miller uses at runtime, so there is no separate list to keep in sync. The
command-line walk now records which flag a value position belongs to.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Include format-conversion keystroke-savers in bare-dash completion
Reverts the suppression of --c2j/--x2y-style flags from 'mlr -<TAB>'. The full
set of main flags (297) is now offered, matching what is valid on the command
line. GetFlagNames no longer takes an includeSuppressed argument.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Complete terminal subcommands and top-level help/version flags
'mlr <TAB>' now offers subcommand names (help, version, repl, regtest, script,
completion, terminal-list) alongside verb names, and 'mlr -<TAB>' offers the
top-level terminal flags (-h, --help, --version, --bare-version, and the help
shorthands -g/-l/-L/-f/-F/-k/-K). Subcommand names are offered only as the
first non-flag token, where they are valid.
To let the completion engine know these names without an import cycle
(pkg/terminals imports pkg/terminals/completion), the canonical terminal names
and version-flag spellings are factored into a new leaf package
pkg/terminals/registry, imported by pkg/terminals, pkg/climain, and completion.
The help-flag spellings come from a new help.GetTerminalFlagNames derived from
the existing shorthand table, so nothing drifts.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Complete 'mlr help' topics and topic arguments
'mlr help <TAB>' now completes help topics (flags, verb, function, keyword,
list-verbs, ...), and topics that take a name argument complete it too:
'mlr help verb <TAB>' -> verb names, 'mlr help function <TAB>' -> function
names, 'mlr help keyword <TAB>' -> keyword names, 'mlr help flag <TAB>' ->
flag names. 'mlr completion <TAB>' completes bash/zsh.
A terminal subcommand consumes the rest of the command line, so the walk now
returns a ctxTerminalArgs context carrying the terminal name and the words
typed after it. New getters supply the candidate names without drift:
help.GetTopicNames, help.GetFunctionNames/GetKeywordNames (wrapping new
cst.BuiltinFunctionManager.GetBuiltinFunctionNames and cst.GetKeywordNames).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs-neaten
* Move flag-value-candidate logic into pkg/cli; fix verb-flag collision
The mapping of which flags take a format/separator/regex-separator argument is
flag metadata, so it now lives with the flags in pkg/cli as
cli.FlagValueCandidates, alongside the existing GetFileFormatNames /
GetSeparatorAliasNames getters, replacing the maps that were in
pkg/terminals/completion/value_completion.go (now removed).
This also fixes a bug: value completion now applies only to main flags, not to
identically-spelled verb flags. Previously 'mlr uniq -o <TAB>' offered file
formats because uniq's -o (an output field name) collided with the main -o
format flag; it now correctly falls back to filename completion.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Inspired by GNU head & tail, they match their behavior while supporting
the usual grouping operations.
Co-authored-by: John Kerl <kerl.john.r@gmail.com>
seqgen.Transform was generating its full sequence on every call,
including once per upstream record. When chaining two seqgens of
N records each, the second produced N+1 copies of N records in
memory (O(N²)) before writing only the first N. Fix: return
immediately for non-EOS input; generate the sequence only when
the upstream end-of-stream arrives, which is the correct
semantics for a verb that discards its input record stream.
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Batch-allocate per-record objects; reuse CSV writer field buffer
After batch-arena field allocation, profiling cat over 1M-record CSV showed
the remaining ~5M allocations were almost entirely per-record (one each):
the Mlrmap struct, the RecordAndContext wrapper, the CSV writer's []string,
and the go-csv parser's own buffers.
Address the first three:
- mlrval.RecordArena gains NewRecord(), vending the Mlrmap struct itself from
a per-batch slab (respecting --no-hash-records). Rolled out to every
line-based reader (CSV, CSV-lite, TSV, DKVP, NIDX, PPRINT, XTAB, DKVPX) in
place of NewMlrmapAsRecord.
- The CSV reader batch-allocates RecordAndContext wrappers from a per-batch
slab instead of one heap object per record (comment/output-string entries
still allocate individually, but they are rare).
- RecordWriterCSV reuses a single fieldsBuffer []string across records instead
of allocating one per Write; WriteCSVRecordMaybeColorized consumes it
synchronously and the writer is single-goroutine, so this is safe.
Effect (big.*, 1M records, cat, best of 5):
csv 0.26 -> 0.22
dkvp 0.51 -> 0.45 (Mlrmap slab)
For CSV, cat's allocation-object count drops ~5.0M -> ~2.1M. The remaining
~2M are the go-csv parser's per-record backing string and field slice, which
are intrinsic to parsing and would require a zero-copy/batch-slab parser
rework. A CPU profile of cat now shows it is I/O-bound (syscall ~56%, bufio
read+flush), with allocation/GC down to ~10% -- i.e. further allocation
trimming no longer moves cat's wall-clock. GOGC=off confirms (no change).
Verified: go test ./pkg/... and full regression suite pass; output is
byte-identical across all formats including record-retaining verbs (tac),
hashed and --no-hash-records.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Pool DSL stack frames across records (~8-9% on put)
A StackFrameSet lives on the persistent runtime.State and is reused across
all records, but every block entry (StatementBlockNode.Execute does
PushStackFrame/PopStackFrame, which runs once per record for the main block,
plus once per if/for/etc.) allocated a fresh StackFrame -- a []*var slice and
a map[string]int -- and discarded it on exit. For `put`/`filter` that is
millions of throwaway allocations.
Since push/pop is strictly LIFO, retain popped frames in a per-frameset free
list and clear-and-reuse them on the next push. After the first record
establishes the max block-nesting depth, per-record block execution is
allocation-free for frames. len(stackFrames) remains the logical depth, so
get/set/defineTyped/unset/etc. are unchanged.
Measured (big.csv, 1M rows, best of 4):
put chain-1 0.78 -> 0.72 (~8%)
put chain-4 0.96 -> 0.87 (~9%)
Allocation objects for put chain-1 drop ~23.1M -> ~20.0M (the per-record
newStackFrame churn, ~2.86M, is eliminated). UDF calls still allocate a fresh
frameset per call (PushStackFrameSet); pooling those is a separate change.
The dominant remaining DSL allocator is FromFloat (~6.8M, interior arithmetic
temporaries); eliminating it needs node-owned result slots + in-place bif
variants, a much larger and aliasing-sensitive change, left for follow-up.
Verified: go test ./pkg/... and full regression suite pass; put output is
byte-identical, including UDFs with locals/loops/blocks.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Pool DSL stack-frame *sets* across UDF/subr calls (~31% on function-heavy put)
Companion to the per-block frame pooling: that left PushStackFrameSet /
PopStackFrameSet (entered once per user-defined function or subroutine call)
allocating. Each call did newStackFrameSet() -- a StackFrameSet plus its
initial StackFrame (a slice and a map) -- AND, worse, prepended it with
append([]*StackFrameSet{head}, sets...), allocating a fresh backing slice and
copying the whole save-stack every call.
Two changes:
- Treat the frameset save-stack as a tail stack (append to push, truncate to
pop) instead of prepending at index 0. get/set only ever touch the cached
head, so list order is irrelevant; this removes the per-call slice
realloc + O(depth) copy.
- Pool popped framesets (LIFO) and reset-and-reuse them on the next push,
mirroring the per-frameset frame free list. A reset trims back to one
cleared base frame (extras go to the frame pool). After warmup, repeated
calls allocate no framesets or frames.
Measured (big.csv, 1M rows, best of 5):
put, 2 nested func calls/record: 2.73 -> 1.87 (~31%)
GC cycles 25 -> 16; newStackFrameSet/newStackFrame fall out of the allocation
profile entirely. (chain-1 etc. have no UDFs and are unaffected.)
Verified: go test ./pkg/... and full regression suite pass; recursion
(fact/fib), local-scope isolation, and subroutine+oosvar all correct.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Drop redundant deep-copy of UDF return values (~3-16% on UDF put)
A user-defined function's return value was deep-copied twice on the way out:
once in ReturnNode.Execute (returnValue.Copy() when building the block-exit
payload) and again in UDFCallsite.EvaluateWithArguments
(blockReturnValue.Copy() at the end).
The ReturnNode copy is the necessary one: it detaches the value from the
callee's frame so it survives the frame being popped (and, since perf-try-7,
pooled/reused). By the time EvaluateWithArguments returns, blockReturnValue is
therefore already an independent deep copy, so the second copy is pure waste --
and callers that retain the result copy again anyway (field/oosvar/local
assignment all PutCopy/Copy). The other return paths (implicit-absent, error)
don't use blockReturnValue, so this only affects the BLOCK_EXIT_RETURN_VALUE
path.
Return blockReturnValue directly.
Measured (big.csv, 1M rows, best of 5):
put, 2 nested scalar-returning calls/record: 1.89 -> 1.83 (~3%)
put, map-returning func per record: 2.34 -> 1.97 (~16%)
Win scales with return-value size (the avoided copy is deep). All UDF/HOF
callsites (apply/reduce/sort/select/fold/...) go through this path.
Verified: go test ./pkg/... and full regression suite pass; recursion, HOFs,
and returned-map isolation (mutating a returned map does not affect a
subsequent call) all correct.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Bind scalar locals/params by reference, not by copy (~4-9% on DSL)
NewTypeGatedMlrvalVariable and TypeGatedMlrvalVariable.Assign deep-copied every
value bound to a local variable or function parameter -- ~6.9M allocations on a
UDF-heavy put. For scalars that copy is unnecessary:
Aliasing audit. Assignment everywhere REPLACES pointers rather than mutating
Mlrvals in place: Mlrmap.PutCopy reassigns pe.Value, Assign reassigns
tvar.value. The only in-place mutation a scalar undergoes is idempotent
type-inference caching (printrep -> typed). So a local/param bound by reference
to a scalar source can never observe its source change, and reassigning the
local replaces its own pointer without touching the source -- capture-by-value
semantics are preserved. Maps and arrays, by contrast, ARE mutated in place by
indexed assignment (m[k]=v), so an aliased collection would corrupt its source;
those must still be deep-copied.
So copyForBind copies only arrays/maps and binds scalars by reference. (Return
values are independently safe: ReturnNode.Execute still deep-copies them.)
Measured (big.csv, 1M rows, best of 5):
UDF-heavy put (scalar args/locals): 1.84 -> 1.68 (~9%)
x = $a+$b; $s = x*2 (no UDF): 0.50 -> 0.48 (~4%)
Verified: go test ./pkg/... and full regression suite pass, plus targeted
alias-then-mutate tests: scalar locals capture-by-value (source change after
bind not observed; reassigning one of two aliases leaves the other intact;
mutating a scalar param leaves the caller field intact), and collections stay
independent (local/param/oosvar-element map copies isolate in-place mutation).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* Batch-allocate per-record objects; reuse CSV writer field buffer
After batch-arena field allocation, profiling cat over 1M-record CSV showed
the remaining ~5M allocations were almost entirely per-record (one each):
the Mlrmap struct, the RecordAndContext wrapper, the CSV writer's []string,
and the go-csv parser's own buffers.
Address the first three:
- mlrval.RecordArena gains NewRecord(), vending the Mlrmap struct itself from
a per-batch slab (respecting --no-hash-records). Rolled out to every
line-based reader (CSV, CSV-lite, TSV, DKVP, NIDX, PPRINT, XTAB, DKVPX) in
place of NewMlrmapAsRecord.
- The CSV reader batch-allocates RecordAndContext wrappers from a per-batch
slab instead of one heap object per record (comment/output-string entries
still allocate individually, but they are rare).
- RecordWriterCSV reuses a single fieldsBuffer []string across records instead
of allocating one per Write; WriteCSVRecordMaybeColorized consumes it
synchronously and the writer is single-goroutine, so this is safe.
Effect (big.*, 1M records, cat, best of 5):
csv 0.26 -> 0.22
dkvp 0.51 -> 0.45 (Mlrmap slab)
For CSV, cat's allocation-object count drops ~5.0M -> ~2.1M. The remaining
~2M are the go-csv parser's per-record backing string and field slice, which
are intrinsic to parsing and would require a zero-copy/batch-slab parser
rework. A CPU profile of cat now shows it is I/O-bound (syscall ~56%, bufio
read+flush), with allocation/GC down to ~10% -- i.e. further allocation
trimming no longer moves cat's wall-clock. GOGC=off confirms (no change).
Verified: go test ./pkg/... and full regression suite pass; output is
byte-identical across all formats including record-retaining verbs (tac),
hashed and --no-hash-records.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Pool DSL stack frames across records (~8-9% on put)
A StackFrameSet lives on the persistent runtime.State and is reused across
all records, but every block entry (StatementBlockNode.Execute does
PushStackFrame/PopStackFrame, which runs once per record for the main block,
plus once per if/for/etc.) allocated a fresh StackFrame -- a []*var slice and
a map[string]int -- and discarded it on exit. For `put`/`filter` that is
millions of throwaway allocations.
Since push/pop is strictly LIFO, retain popped frames in a per-frameset free
list and clear-and-reuse them on the next push. After the first record
establishes the max block-nesting depth, per-record block execution is
allocation-free for frames. len(stackFrames) remains the logical depth, so
get/set/defineTyped/unset/etc. are unchanged.
Measured (big.csv, 1M rows, best of 4):
put chain-1 0.78 -> 0.72 (~8%)
put chain-4 0.96 -> 0.87 (~9%)
Allocation objects for put chain-1 drop ~23.1M -> ~20.0M (the per-record
newStackFrame churn, ~2.86M, is eliminated). UDF calls still allocate a fresh
frameset per call (PushStackFrameSet); pooling those is a separate change.
The dominant remaining DSL allocator is FromFloat (~6.8M, interior arithmetic
temporaries); eliminating it needs node-owned result slots + in-place bif
variants, a much larger and aliasing-sensitive change, left for follow-up.
Verified: go test ./pkg/... and full regression suite pass; put output is
byte-identical, including UDFs with locals/loops/blocks.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Pool DSL stack-frame *sets* across UDF/subr calls (~31% on function-heavy put)
Companion to the per-block frame pooling: that left PushStackFrameSet /
PopStackFrameSet (entered once per user-defined function or subroutine call)
allocating. Each call did newStackFrameSet() -- a StackFrameSet plus its
initial StackFrame (a slice and a map) -- AND, worse, prepended it with
append([]*StackFrameSet{head}, sets...), allocating a fresh backing slice and
copying the whole save-stack every call.
Two changes:
- Treat the frameset save-stack as a tail stack (append to push, truncate to
pop) instead of prepending at index 0. get/set only ever touch the cached
head, so list order is irrelevant; this removes the per-call slice
realloc + O(depth) copy.
- Pool popped framesets (LIFO) and reset-and-reuse them on the next push,
mirroring the per-frameset frame free list. A reset trims back to one
cleared base frame (extras go to the frame pool). After warmup, repeated
calls allocate no framesets or frames.
Measured (big.csv, 1M rows, best of 5):
put, 2 nested func calls/record: 2.73 -> 1.87 (~31%)
GC cycles 25 -> 16; newStackFrameSet/newStackFrame fall out of the allocation
profile entirely. (chain-1 etc. have no UDFs and are unaffected.)
Verified: go test ./pkg/... and full regression suite pass; recursion
(fact/fib), local-scope isolation, and subroutine+oosvar all correct.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Drop redundant deep-copy of UDF return values (~3-16% on UDF put)
A user-defined function's return value was deep-copied twice on the way out:
once in ReturnNode.Execute (returnValue.Copy() when building the block-exit
payload) and again in UDFCallsite.EvaluateWithArguments
(blockReturnValue.Copy() at the end).
The ReturnNode copy is the necessary one: it detaches the value from the
callee's frame so it survives the frame being popped (and, since perf-try-7,
pooled/reused). By the time EvaluateWithArguments returns, blockReturnValue is
therefore already an independent deep copy, so the second copy is pure waste --
and callers that retain the result copy again anyway (field/oosvar/local
assignment all PutCopy/Copy). The other return paths (implicit-absent, error)
don't use blockReturnValue, so this only affects the BLOCK_EXIT_RETURN_VALUE
path.
Return blockReturnValue directly.
Measured (big.csv, 1M rows, best of 5):
put, 2 nested scalar-returning calls/record: 1.89 -> 1.83 (~3%)
put, map-returning func per record: 2.34 -> 1.97 (~16%)
Win scales with return-value size (the avoided copy is deep). All UDF/HOF
callsites (apply/reduce/sort/select/fold/...) go through this path.
Verified: go test ./pkg/... and full regression suite pass; recursion, HOFs,
and returned-map isolation (mutating a returned map does not affect a
subsequent call) all correct.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
