* docs: add "Miller and AI agents" quick-start page (#2098)
Umbrella page for the AI-friendly feature stack: one-line MCP setup as
the fast path, plus the plain-CLI path (which, help --as-json,
describe, --explain, --errors-json, --no-shell) with live CI-tested
examples, and the discover -> constrain -> validate -> run loop.
Cross-linked with the MCP server page; listed under Getting started.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: reframe AI page as "Miller and AI" (#2098)
Rename ai-agents.md to ai.md and restructure around the pre-MCP
feature stack, organized as the loop each feature serves: Discover
(catalog/index/which, cache keys, single-sourced usage text),
Constrain (enum value-sets + describe: tool shape vs data shape),
Validate (--explain), Run and recover (--errors-json, --no-shell,
env-var trio). MCP is now one closing section pointing at the
mcp-server.md detail page. All examples are live and CI-tested,
including Miller querying its own catalog.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: retitle MCP page to "The MCP server" under the Miller-and-AI umbrella
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: add bare-minimum getting-started section to Miller-and-AI page
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: AI features land in Miller 6.20
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: hyperlink SKILL.md references to the repo
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: rename section to 'The essentials'
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Plan: flesh out PR7 (MCP server + Agent Skill) design
stdio transport (no HTTP port), mlr mcp terminal in the main binary,
SDK-vs-handroll decision, tool list, in-process vs subprocess split,
run-tool safety (--no-shell prerequisite), single-sourced skill, tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add mlr mcp: MCP server + agent playbook; --no-shell gate (#2098 PR7)
New terminal `mlr mcp` runs a Model Context Protocol server over stdio
(spawned by MCP clients; no network port), exposing five tools --
list_capabilities, which, validate_dsl, describe_data, run -- plus an
agent playbook as MCP prompt/resource. Catalog tools are served
in-process from the help registries; the rest subprocess this same
binary with MLR_ERRORS_JSON=1, a timeout, and an output cap.
Prerequisite: a new --no-shell flag / MLR_NO_SHELL env var (one-way
gate) disables the DSL system/exec functions, piped redirects, and
--prepipe/--prepipex; the MCP server sets it on the commands it runs
unless started with --allow-shell.
Adds the github.com/modelcontextprotocol/go-sdk dependency.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Force LF checkout for the embedded SKILL.md (Windows CI fix)
go:embed embeds checkout bytes, so a CRLF checkout on Windows made the
embedded playbook differ per platform and failed
TestPlaybookHasFrontmatter. Pin the file to eol=lf in .gitattributes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Move no-shell test DSL into per-case mlr files (Windows CI fix)
Inline single-quoted DSL in cmd files is mangled by the Windows shell
(single quotes are not quote characters there); the harness's
put -f ${CASEDIR}/mlr pattern avoids shell quoting entirely.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
One output record per input field: types seen with counts, occurrence
count, null count, cardinality, min/max, and -- for fields within the
-n/--max-values cap -- the complete distinct-value list in first-seen
order. `mlr --ojson describe` is the machine-readable form; nested
types/values flatten in tabular formats.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Lets an agent type-check a DSL expression before spending a full input
pass. `mlr put --explain '...'` (and filter) runs the existing
parse -> ValidateAST -> CST build -> Resolve path, then:
- valid: prints "mlr {put,filter}: DSL expression is valid." and exits 0
- invalid: returns the build error up the normal path, so --errors-json
emits a structured document; exits 1
- -W with fatal warnings: reports and exits 1
The gate lives in the pass-two constructor, before any input file is
opened, so no input stream is read (verified with a nonexistent input
file still validating OK).
Also categorize bare "parse error: ..." messages from the DSL parser as
kind "dsl-parse-error" rather than "generic" (climain/errors_json.go),
so --explain --errors-json gives an agent a useful error kind. The CSV
reader's "parse error on line ..." is stream-time and never reaches this
command-line-parse categorizer.
Tests: dsl-explain/0001-0004 regression cases (valid put/filter, invalid
plain, invalid --errors-json) and categorize unit tests. Regenerated
verb docs, manpage, and the help usage-verbs golden case.
The older -X ("exit after parsing") still exits 0 even on a parse error;
left as-is since --explain is the correct validation path.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Tier-2 structured verb options: OptionSpec, initial migration (#2098)
PR 3 of the AI-friendly roadmap (plans/plan-2098-llm.md).
Infrastructure:
- Add OptionSpec{Flag,Arg,Type,Desc,Repeatable,Values} to
pkg/transformers/aaa_record_transformer.go alongside TransformerSetup.
Type is one of: bool, string, int, float, csv-list, regex, filename,
format, enum. For type=="enum", Values lists the valid choices.
- Add Options []OptionSpec to TransformerSetup (nil = not yet migrated).
- Emit Options in VerbInfoForJSON (omitempty so unmigrated verbs stay
backward-compatible; agents check key presence for Tier-2 availability).
UsageText is always present as the Tier-1 prose fallback.
- Add VerbOptionsNilCheck() in aaa_verb_options_check.go: progress report
of migrated vs. unmigrated verbs, analogous to FLAG_TABLE.NilCheck().
- Wire verb-options-nil-check into mlr help (internal/docgen section).
Initial migration (5/70 verbs):
- nothing: empty Options (no verb-specific options, explicitly migrated)
- cat: -n (bool), -N (string), -g (csv-list), --filename, --filenum (bool)
- head: -g (csv-list), -n (int)
- tail: -g (csv-list), -n (int)
- tee: -a, -p (bool)
Tests:
- 5 new unit tests in aaa_transformer_json_test.go covering migrated/
unmigrated paths, field population, JSON round-trip, and key-presence.
- Regression test case 0003: mlr help verb-options-nil-check golden output.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Migrate all 70 verbs to structured OptionSpec; bump catalog schema to v2
Completes the Tier-2 migration started in the previous commit. Every verb
in TRANSFORMER_LOOKUP_TABLE now has a non-nil Options field.
- Workflow-migrated all 65 remaining verbs. Each Setup var now carries
Options: []OptionSpec{...} with Flag/Arg/Type/Desc fields. Verbs with
no verb-specific options (altkv, check, group-like, nothing, etc.) use
an empty slice to signal "migrated but no options."
- Drop `omitempty` from VerbInfoForJSON.Options: empty slices were silently
dropped, making migrated-no-option verbs indistinguishable from unmigrated
ones in JSON. Without omitempty: null=unmigrated, []=migrated-no-options,
[...]= migrated-with-options. Bump catalogSchemaVersion 1→2 for this shape
change.
- Replace the two "unmigrated-verb" unit tests (which used stats1 as an
example) with TestAllVerbsFullyMigrated (asserts every verb has non-nil
Options) and TestAllVerbsHaveOptionsKeyInJSON (asserts every migrated
verb emits the "options" key in JSON).
- Regenerate test/cases/cli-help/0003/expout: now reads
"Verb options migration: 70/70 migrated."
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* remove a transitional helper
* git rms
* Render verb usage Options blocks from structured OptionSpec
Each verb's usage message and its Tier-2 OptionSpec list previously
duplicated the option text. New WriteVerbOptions (aaa_verb_usage.go)
renders the "Options:" block from the specs: aligned flag column,
descriptions word-wrapped at 80, uniform trailing -h|--help line.
- OptionSpec gains Aliases (JSON "aliases") so long-form spellings
like join's --lk|--left-keep-field-names survive in both outputs
- All 70 verbs migrated; options literals hoisted to package-level
vars (usage funcs can't reference their Setup var without a Go
init cycle)
- Hand-written per-option details the specs had condensed away are
merged into Desc, enriching the JSON catalog
- Non-option prose (examples, cross-references, dynamic accumulator
listings) kept verbatim
- Regenerated the six usage-embedding regression expectations and
the two affected doc pages
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix pre-existing usage-text bugs surfaced by the OptionSpec migration
- gap: usage said "One of -f or -g is required" but the parser takes
-n or -g
- seqgen: drop description line copy-pasted from cat ("Passes input
records directly to output...") which contradicted "Discards the
input record stream"
- utf8-to-latin1: description read inverted ("from Latin-1 to UTF-8")
- sec2gmtdate: usage said "../c/mlr" instead of "mlr"
- top: document the accepted-but-undocumented --max flag
- stats2: add linreg-pca to the -a enum values, matching the runtime
accumulator table
Regression expectations and docs regenerated accordingly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix check usage sentence order; stats1 usage blank line to usage stream
- check: the description's second and third lines were swapped,
reading "Consumes records without printing any output, / Useful for
doing a well-formatted check on input data. / with the exception
that warnings are printed to stderr."
- stats1: a bare fmt.Println() in the usage func wrote its blank line
to process stdout instead of the usage output stream
Regression expectation and docs regenerated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Add MT_BYTES mlrval type: foundation and disposition tables
First step toward a first-class bytes type in the DSL (#1231).
Adds MT_BYTES (payload []byte, rendered as lowercase hex in all output
formats, JSON-encoded as a hex string), extends every disposition
matrix/vector with the new row/column -- real cells for comparison,
sorting, and dot-concat of bytes with bytes; type-error stubs
elsewhere -- and adds sweep tests asserting no table has nil cells,
since Go zero-fills short array literals when MT_DIM grows.
Bytes values are not yet constructible from the DSL; b"..." literals
and constructor/codec functions follow in subsequent commits.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add b"..." bytes-literal syntax to the DSL
Adds a bytes_literal token to the grammar (regenerating the PGPG lexer
and parser) and a BytesLiteralNode in the CST which evaluates to an
MT_BYTES mlrval. Escape handling reuses UnbackslashStringLiteral,
which is already byte-oriented: b"\xff" is the single byte 0xff.
Unlike string literals, bytes literals never participate in
regex-capture replacement. A bare identifier b is unaffected.
Part of #1231.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add bytes DSL functions: conversions, codecs, and bytes-aware built-ins
- bytes(x) converts strings to bytes; string(b) reinterprets raw bytes
as UTF-8 text (the reverse)
- base64_decode now always returns bytes (superseding the interim
string-or-hex behavior); base64_encode accepts string or bytes
- New hex_encode/hex_decode functions
- is_bytes and asserting_bytes predicates
- md5/sha1/sha256/sha512 accept bytes, hashing the raw payload
- strlen of bytes is the byte count; substr/substr0/substr1 on bytes
slice by byte position and return bytes
The Cyrillic-LDAP scenario from #1231 now works without exec
workarounds: string(base64_decode($x)) recovers the text, and binary
payloads survive undamaged as bytes.
Closes#1231.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add bytes-type docs and regression cases
Documents the bytes type on the data-types page, regenerates the
function-reference/man-page material, and adds regression coverage:
literal escape forms, operators (concat/compare/slice/sort and
type errors), conversions and codec round-trips, and CSV-to-JSON
output rendering of bytes fields.
Part of #1231.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Reposition MT_BYTES to sort adjacent to MT_STRING in the type enum
MT_BYTES was appended after MT_ABSENT for index stability; move it
right after MT_STRING instead, since that's where it conceptually
belongs and where it already sorts in the cmp disposition matrices.
Mechanically re-derive all ~40 disposition tables in pkg/bifs and
pkg/mlrval accordingly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fix windows CI
* fix merge
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* initial attempt
* fix bash
* fix zsh
* Add shell-completion docs page
Documents the new 'mlr completion {bash,zsh}' feature: the then-chain
context model, install instructions for bash and zsh (including the macOS
bash-3.2 'eval' caveat and zsh compinit self-init), and examples of
context-aware completion. Added to the nav under "Miller in more detail".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Add enum value completion for format and separator flags
Completes the argument value for arg-taking main flags whose values are a
known set: file-format names for -i/-o/--io, separator aliases for
--ifs/--ofs/--ips/etc., and regex-separator aliases for --ifs-regex/--ips-regex.
Other arg-taking flags continue to fall back to filename completion.
Candidate sets come from new cli getters (GetFileFormatNames,
GetSeparatorAliasNames, GetSeparatorRegexAliasNames) that read the same maps
Miller uses at runtime, so there is no separate list to keep in sync. The
command-line walk now records which flag a value position belongs to.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Include format-conversion keystroke-savers in bare-dash completion
Reverts the suppression of --c2j/--x2y-style flags from 'mlr -<TAB>'. The full
set of main flags (297) is now offered, matching what is valid on the command
line. GetFlagNames no longer takes an includeSuppressed argument.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Complete terminal subcommands and top-level help/version flags
'mlr <TAB>' now offers subcommand names (help, version, repl, regtest, script,
completion, terminal-list) alongside verb names, and 'mlr -<TAB>' offers the
top-level terminal flags (-h, --help, --version, --bare-version, and the help
shorthands -g/-l/-L/-f/-F/-k/-K). Subcommand names are offered only as the
first non-flag token, where they are valid.
To let the completion engine know these names without an import cycle
(pkg/terminals imports pkg/terminals/completion), the canonical terminal names
and version-flag spellings are factored into a new leaf package
pkg/terminals/registry, imported by pkg/terminals, pkg/climain, and completion.
The help-flag spellings come from a new help.GetTerminalFlagNames derived from
the existing shorthand table, so nothing drifts.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Complete 'mlr help' topics and topic arguments
'mlr help <TAB>' now completes help topics (flags, verb, function, keyword,
list-verbs, ...), and topics that take a name argument complete it too:
'mlr help verb <TAB>' -> verb names, 'mlr help function <TAB>' -> function
names, 'mlr help keyword <TAB>' -> keyword names, 'mlr help flag <TAB>' ->
flag names. 'mlr completion <TAB>' completes bash/zsh.
A terminal subcommand consumes the rest of the command line, so the walk now
returns a ctxTerminalArgs context carrying the terminal name and the words
typed after it. New getters supply the candidate names without drift:
help.GetTopicNames, help.GetFunctionNames/GetKeywordNames (wrapping new
cst.BuiltinFunctionManager.GetBuiltinFunctionNames and cst.GetKeywordNames).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs-neaten
* Move flag-value-candidate logic into pkg/cli; fix verb-flag collision
The mapping of which flags take a format/separator/regex-separator argument is
flag metadata, so it now lives with the flags in pkg/cli as
cli.FlagValueCandidates, alongside the existing GetFileFormatNames /
GetSeparatorAliasNames getters, replacing the maps that were in
pkg/terminals/completion/value_completion.go (now removed).
This also fixes a bug: value completion now applies only to main flags, not to
identically-spelled verb flags. Previously 'mlr uniq -o <TAB>' offered file
formats because uniq's -o (an output field name) collided with the main -o
format flag; it now correctly falls back to filename completion.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Inspired by GNU head & tail, they match their behavior while supporting
the usual grouping operations.
Co-authored-by: John Kerl <kerl.john.r@gmail.com>
* Batch-allocate per-record objects; reuse CSV writer field buffer
After batch-arena field allocation, profiling cat over 1M-record CSV showed
the remaining ~5M allocations were almost entirely per-record (one each):
the Mlrmap struct, the RecordAndContext wrapper, the CSV writer's []string,
and the go-csv parser's own buffers.
Address the first three:
- mlrval.RecordArena gains NewRecord(), vending the Mlrmap struct itself from
a per-batch slab (respecting --no-hash-records). Rolled out to every
line-based reader (CSV, CSV-lite, TSV, DKVP, NIDX, PPRINT, XTAB, DKVPX) in
place of NewMlrmapAsRecord.
- The CSV reader batch-allocates RecordAndContext wrappers from a per-batch
slab instead of one heap object per record (comment/output-string entries
still allocate individually, but they are rare).
- RecordWriterCSV reuses a single fieldsBuffer []string across records instead
of allocating one per Write; WriteCSVRecordMaybeColorized consumes it
synchronously and the writer is single-goroutine, so this is safe.
Effect (big.*, 1M records, cat, best of 5):
csv 0.26 -> 0.22
dkvp 0.51 -> 0.45 (Mlrmap slab)
For CSV, cat's allocation-object count drops ~5.0M -> ~2.1M. The remaining
~2M are the go-csv parser's per-record backing string and field slice, which
are intrinsic to parsing and would require a zero-copy/batch-slab parser
rework. A CPU profile of cat now shows it is I/O-bound (syscall ~56%, bufio
read+flush), with allocation/GC down to ~10% -- i.e. further allocation
trimming no longer moves cat's wall-clock. GOGC=off confirms (no change).
Verified: go test ./pkg/... and full regression suite pass; output is
byte-identical across all formats including record-retaining verbs (tac),
hashed and --no-hash-records.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Pool DSL stack frames across records (~8-9% on put)
A StackFrameSet lives on the persistent runtime.State and is reused across
all records, but every block entry (StatementBlockNode.Execute does
PushStackFrame/PopStackFrame, which runs once per record for the main block,
plus once per if/for/etc.) allocated a fresh StackFrame -- a []*var slice and
a map[string]int -- and discarded it on exit. For `put`/`filter` that is
millions of throwaway allocations.
Since push/pop is strictly LIFO, retain popped frames in a per-frameset free
list and clear-and-reuse them on the next push. After the first record
establishes the max block-nesting depth, per-record block execution is
allocation-free for frames. len(stackFrames) remains the logical depth, so
get/set/defineTyped/unset/etc. are unchanged.
Measured (big.csv, 1M rows, best of 4):
put chain-1 0.78 -> 0.72 (~8%)
put chain-4 0.96 -> 0.87 (~9%)
Allocation objects for put chain-1 drop ~23.1M -> ~20.0M (the per-record
newStackFrame churn, ~2.86M, is eliminated). UDF calls still allocate a fresh
frameset per call (PushStackFrameSet); pooling those is a separate change.
The dominant remaining DSL allocator is FromFloat (~6.8M, interior arithmetic
temporaries); eliminating it needs node-owned result slots + in-place bif
variants, a much larger and aliasing-sensitive change, left for follow-up.
Verified: go test ./pkg/... and full regression suite pass; put output is
byte-identical, including UDFs with locals/loops/blocks.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Pool DSL stack-frame *sets* across UDF/subr calls (~31% on function-heavy put)
Companion to the per-block frame pooling: that left PushStackFrameSet /
PopStackFrameSet (entered once per user-defined function or subroutine call)
allocating. Each call did newStackFrameSet() -- a StackFrameSet plus its
initial StackFrame (a slice and a map) -- AND, worse, prepended it with
append([]*StackFrameSet{head}, sets...), allocating a fresh backing slice and
copying the whole save-stack every call.
Two changes:
- Treat the frameset save-stack as a tail stack (append to push, truncate to
pop) instead of prepending at index 0. get/set only ever touch the cached
head, so list order is irrelevant; this removes the per-call slice
realloc + O(depth) copy.
- Pool popped framesets (LIFO) and reset-and-reuse them on the next push,
mirroring the per-frameset frame free list. A reset trims back to one
cleared base frame (extras go to the frame pool). After warmup, repeated
calls allocate no framesets or frames.
Measured (big.csv, 1M rows, best of 5):
put, 2 nested func calls/record: 2.73 -> 1.87 (~31%)
GC cycles 25 -> 16; newStackFrameSet/newStackFrame fall out of the allocation
profile entirely. (chain-1 etc. have no UDFs and are unaffected.)
Verified: go test ./pkg/... and full regression suite pass; recursion
(fact/fib), local-scope isolation, and subroutine+oosvar all correct.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Drop redundant deep-copy of UDF return values (~3-16% on UDF put)
A user-defined function's return value was deep-copied twice on the way out:
once in ReturnNode.Execute (returnValue.Copy() when building the block-exit
payload) and again in UDFCallsite.EvaluateWithArguments
(blockReturnValue.Copy() at the end).
The ReturnNode copy is the necessary one: it detaches the value from the
callee's frame so it survives the frame being popped (and, since perf-try-7,
pooled/reused). By the time EvaluateWithArguments returns, blockReturnValue is
therefore already an independent deep copy, so the second copy is pure waste --
and callers that retain the result copy again anyway (field/oosvar/local
assignment all PutCopy/Copy). The other return paths (implicit-absent, error)
don't use blockReturnValue, so this only affects the BLOCK_EXIT_RETURN_VALUE
path.
Return blockReturnValue directly.
Measured (big.csv, 1M rows, best of 5):
put, 2 nested scalar-returning calls/record: 1.89 -> 1.83 (~3%)
put, map-returning func per record: 2.34 -> 1.97 (~16%)
Win scales with return-value size (the avoided copy is deep). All UDF/HOF
callsites (apply/reduce/sort/select/fold/...) go through this path.
Verified: go test ./pkg/... and full regression suite pass; recursion, HOFs,
and returned-map isolation (mutating a returned map does not affect a
subsequent call) all correct.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Bind scalar locals/params by reference, not by copy (~4-9% on DSL)
NewTypeGatedMlrvalVariable and TypeGatedMlrvalVariable.Assign deep-copied every
value bound to a local variable or function parameter -- ~6.9M allocations on a
UDF-heavy put. For scalars that copy is unnecessary:
Aliasing audit. Assignment everywhere REPLACES pointers rather than mutating
Mlrvals in place: Mlrmap.PutCopy reassigns pe.Value, Assign reassigns
tvar.value. The only in-place mutation a scalar undergoes is idempotent
type-inference caching (printrep -> typed). So a local/param bound by reference
to a scalar source can never observe its source change, and reassigning the
local replaces its own pointer without touching the source -- capture-by-value
semantics are preserved. Maps and arrays, by contrast, ARE mutated in place by
indexed assignment (m[k]=v), so an aliased collection would corrupt its source;
those must still be deep-copied.
So copyForBind copies only arrays/maps and binds scalars by reference. (Return
values are independently safe: ReturnNode.Execute still deep-copies them.)
Measured (big.csv, 1M rows, best of 5):
UDF-heavy put (scalar args/locals): 1.84 -> 1.68 (~9%)
x = $a+$b; $s = x*2 (no UDF): 0.50 -> 0.48 (~4%)
Verified: go test ./pkg/... and full regression suite pass, plus targeted
alias-then-mutate tests: scalar locals capture-by-value (source change after
bind not observed; reassigning one of two aliases leaves the other intact;
mutating a scalar param leaves the caller field intact), and collections stay
independent (local/param/oosvar-element map copies isolate in-place mutation).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs: add "Allocation/GC optimizations: June 2026" perf section
Documents the mid-2026 allocation-reduction work on the performance page:
before/after wall-clock and peak-RSS tables (best-of-five, M1, the same
million-record files used elsewhere on the page), plus notes on why streaming
I/O verbs and DSL/UDF workloads benefit most and why sort's time improves
while its memory does not. Added to both performance.md.in and the generated
performance.md (the page has no live-code GENMD blocks).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The --lp/--rp prefixes and -j output-field rename were applied only to
paired records, leaving unpaired records emitted via --ul/--ur with their
original (left/right) field names. This broke downstream operations like
unsparsify that expected consistent column names across paired and
unpaired output.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Default to "cat" verb when none is supplied (#2029)
Invocations like 'mlr --j2y' or 'mlr --c2p' previously failed with
"no verb supplied", forcing users to type the trailing 'cat'
explicitly for pure format conversions. Default the verb to 'cat'
in that case. Bare 'mlr' with no flags, no verb, and no files
still prints the main usage banner.
This handles flag-only invocations (e.g. 'mlr --c2j < input.csv'
or 'mlr --c2j --from input.csv'). File names without a preceding
verb are still parsed as verb candidates and continue to error if
not found; that broader change is out of scope here.
* Use ${MLR} substitution for bare-mlr regression case (#2029)
The regtester only substitutes the mlr executable when the cmd
starts with "mlr " (with a trailing space). The bare-mlr usage-banner
test had a cmd of just "mlr", so on CI -- which invokes regtest with
a relative path like 'test/../mlr' -- the test shelled out to a
literal 'mlr' that isn't on PATH and failed with exit 127.
Switch the cmd to ${MLR} (the regtester's explicit substitution
token) so the case runs the right binary in any invocation context.
The shebang example `#!/usr/bin/env mlr -s` does not work on Linux
because env(1) does not accept arguments to the interpreter; use
`#!/usr/bin/env -S mlr -s` instead.
`contains` and `index` previously stringified their inputs silently,
so `contains([1,2], $foo)` would match against the literal string
`[1, 2]` and produce surprising results. Both now return a type-error
when either argument is an array or map; help text points users at
`any(arr, func(e){return e == x})` for membership tests.
Adds a new Markdown-only flag --omd-aligned (alias --omarkdown-aligned)
that left-justifies cells and pads each column to a uniform width.
The rendered table is unaffected; the goal is readability of the raw
markdown source. The flag implies --omd, so users do not need to pass
both.
Implementation batches records by schema (like pprint), computes max
column widths, then emits header / separator / data rows with bars
vertically aligned. Default markdown writer behavior is unchanged.
