Records (NewMlrmapAsRecord) eagerly allocated and populated a
map[string]*MlrmapEntry on construction whenever hashRecords was true
(the default). For streaming verbs that never look records up by key
(e.g. `mlr cat`) that map is pure overhead: a heap allocation plus N
map-inserts per record, and N more pointer-heavy objects for the GC to
scan. Profiling 1M-record CSV shows runtime allocation/GC machinery
dominating every workload, and `--no-hash-records` was 25-30% faster --
but that flag makes wide-record lookups O(n), the regression that
motivated hashing in #1506.
Make record hashing lazy instead: allocate no index up front; build it
in findEntry on the first lookup, and only when the record is wide
enough (FieldCount >= mlrmapHashThreshold) that linear search would
hurt. Narrow records and never-looked-up records never pay for a map;
wide records that are actually queried still get hash-accelerated
lookups, matching the old eager-hash default. DSL maps (NewMlrmap) keep
eager hashing to limit the behavioral surface.
This is transparent: findEntry already fell back to linear scan when
keysToEntries was nil, and every mutator already guarded on
keysToEntries != nil.
Measured (big.csv, 1M x 7 cols, default flags, best of 3):
cat 0.62 -> 0.47 (~24%)
put 1.08 -> 0.82 (~24%)
stats1 0.66 -> 0.57 (~14%)
sort 2.9 -> 2.0 (~30%)
Wide-column case protected: 60-col file with field lookups, lazy (1.42s)
matches old eager default (1.40s) and beats pure linear (1.55s).
Verified: go test ./pkg/... and full regression suite pass; output is
byte-identical to forced --hash-records for sort, stats1, cut,
wide-column put, and duplicate-key dedupe.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* Fix column alignment for wide and combining Unicode chars (#1520, #379)
PPRINT, markdown-aligned, and XTAB writers measured column widths with
utf8.RuneCountInString, which counts codepoints rather than terminal
display columns. East-Asian fullwidth characters (counted as 1 but
displayed as 2) and zero-width combining marks (counted as 1 but
displayed as 0) both caused misalignment.
Add lib.DisplayWidth wrapping uniseg.StringWidth, and use it from the
three writers. The XTAB right-aligned path also drops fmt.Sprintf with
%*s since Go's %*s pads by rune count too.
UTF8Strlen is unchanged so DSL strlen semantics are preserved.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Fix PPRINT alignment with multi-character OFS (#1819)
Previously, writePadding repeated the OFS string for each padding slot, so
non-space or multi-character OFS values produced misaligned columns and
leaked the separator into padding (e.g. --ofs=XY yielded aXYXYXYXYb).
Pad with spaces instead, leaving OFS to act only as the column separator.
The default single-space OFS is unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Use non-space multi-char OFS in test 0021 for Windows portability
PowerShell collapses single-quoted ' ' in command lines, which broke
the test on the windows-latest CI runner. Switch to --ofs XY, which
exercises the same multi-character padding code path without shell
quoting hazards.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The --lp/--rp prefixes and -j output-field rename were applied only to
paired records, leaving unpaired records emitted via --ul/--ur with their
original (left/right) field names. This broke downstream operations like
unsparsify that expected consistent column names across paired and
unpaired output.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Default to "cat" verb when none is supplied (#2029)
Invocations like 'mlr --j2y' or 'mlr --c2p' previously failed with
"no verb supplied", forcing users to type the trailing 'cat'
explicitly for pure format conversions. Default the verb to 'cat'
in that case. Bare 'mlr' with no flags, no verb, and no files
still prints the main usage banner.
This handles flag-only invocations (e.g. 'mlr --c2j < input.csv'
or 'mlr --c2j --from input.csv'). File names without a preceding
verb are still parsed as verb candidates and continue to error if
not found; that broader change is out of scope here.
* Use ${MLR} substitution for bare-mlr regression case (#2029)
The regtester only substitutes the mlr executable when the cmd
starts with "mlr " (with a trailing space). The bare-mlr usage-banner
test had a cmd of just "mlr", so on CI -- which invokes regtest with
a relative path like 'test/../mlr' -- the test shelled out to a
literal 'mlr' that isn't on PATH and failed with exit 127.
Switch the cmd to ${MLR} (the regtester's explicit substitution
token) so the case runs the right binary in any invocation context.
PPRINT, markdown-aligned, and XTAB writers measured column widths with
utf8.RuneCountInString, which counts codepoints rather than terminal
display columns. East-Asian fullwidth characters (counted as 1 but
displayed as 2) and zero-width combining marks (counted as 1 but
displayed as 0) both caused misalignment.
Add lib.DisplayWidth wrapping uniseg.StringWidth, and use it from the
three writers. The XTAB right-aligned path also drops fmt.Sprintf with
%*s since Go's %*s pads by rune count too.
UTF8Strlen is unchanged so DSL strlen semantics are preserved.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The implicit-header readers for TSV and CSV-lite were missing the
`for` loop in their `nh < nd` branch, so when a data row had more
fields than the auto-generated header, only one extra field was
captured and the rest were dropped. Restore the loop, matching the
explicit-header readers and the pprint reader.
The shebang example `#!/usr/bin/env mlr -s` does not work on Linux
because env(1) does not accept arguments to the interpreter; use
`#!/usr/bin/env -S mlr -s` instead.
`contains` and `index` previously stringified their inputs silently,
so `contains([1,2], $foo)` would match against the literal string
`[1, 2]` and produce surprising results. Both now return a type-error
when either argument is an array or map; help text points users at
`any(arr, func(e){return e == x})` for membership tests.
Adds a new Markdown-only flag --omd-aligned (alias --omarkdown-aligned)
that left-justifies cells and pads each column to a uniform width.
The rendered table is unaffected; the goal is readability of the raw
markdown source. The flag implies --omd, so users do not need to pass
both.
Implementation batches records by schema (like pprint), computes max
column widths, then emits header / separator / data rows with bars
vertically aligned. Default markdown writer behavior is unchanged.
When using --fr (regex field selector) with stats1 -a null_count, void
(empty) field values were unconditionally skipped in
ingestWithValueFieldRegexes, causing null_count to always report 0.
The non-regex path (ingestWithoutValueFieldRegexes) already had a special
case that allows void values through for null_count accumulators. This
commit adds the same exception to the regex path.
Fixes#1639
Co-authored-by: cobyfrombrooklyn-bot <cobyfrombrooklyn-bot@users.noreply.github.com>
* fix: preserve key order in YAML output
MlrmapToYAMLNative converted the ordered Mlrmap to map[string]interface{},
which loses insertion order. yaml.v3.Marshal then sorts Go map keys
alphabetically, so YAML output always had sorted keys despite other formats
(JSON, pretty-print) correctly preserving order.
Switch to building yaml.Node trees that preserve the Mlrmap's linked-list
iteration order. Use yaml.Node.Encode for scalar values to correctly handle
edge cases (NaN/Inf, non-UTF-8 strings).
Fixes#2028.
* Update pkg/mlrval/mlrval_yaml_test.go with copilot code-review feedback
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: lawrence3699 <lawrence3699@users.noreply.github.com>
Co-authored-by: John Kerl <kerl.john.r@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
