Prototype of a zero-copy CSV record reader (ZeroCopyCSVReader): reads input
in large persistent blocks and returns field strings as unsafe.String views
directly into the block, eliminating the stdlib-derived parser's per-record
string() copy and recordBuffer accumulation. Unquoted records are fully
zero-copy; quoted records delegate to the go-csv parser for correctness.
Correctness: full regression suite passes, including a deterministic stress
run with a 3-byte forced block size (maximal boundary splitting). Required
fixing comment trailing-newline handling, comment-before-quote ordering, and
absolute error-line-number rewriting for delegated quoted records.
Performance: NEGATIVE RESULT -- do not merge.
- Wall-clock neutral: CSV cat is I/O-bound (syscall ~56%) after the
allocation-batching work, so removing the per-record copy saves CPU that
already overlapped with I/O on idle cores (confirmed under GOGC variations
and across cat/stats1/cut).
- Peak RSS 2-2.6x WORSE (162MB -> 357-423MB) at every block size: zero-copy
pins whole input blocks while any field references them, and the pipeline
buffers up to 500 batches, so many blocks stay live at once. The stdlib
parser's per-record strings keep memory proportional to live data.
Committed for the record; tracked in a GitHub issue. Not for merge.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
After batch-arena field allocation, profiling cat over 1M-record CSV showed
the remaining ~5M allocations were almost entirely per-record (one each):
the Mlrmap struct, the RecordAndContext wrapper, the CSV writer's []string,
and the go-csv parser's own buffers.
Address the first three:
- mlrval.RecordArena gains NewRecord(), vending the Mlrmap struct itself from
a per-batch slab (respecting --no-hash-records). Rolled out to every
line-based reader (CSV, CSV-lite, TSV, DKVP, NIDX, PPRINT, XTAB, DKVPX) in
place of NewMlrmapAsRecord.
- The CSV reader batch-allocates RecordAndContext wrappers from a per-batch
slab instead of one heap object per record (comment/output-string entries
still allocate individually, but they are rare).
- RecordWriterCSV reuses a single fieldsBuffer []string across records instead
of allocating one per Write; WriteCSVRecordMaybeColorized consumes it
synchronously and the writer is single-goroutine, so this is safe.
Effect (big.*, 1M records, cat, best of 5):
csv 0.26 -> 0.22
dkvp 0.51 -> 0.45 (Mlrmap slab)
For CSV, cat's allocation-object count drops ~5.0M -> ~2.1M. The remaining
~2M are the go-csv parser's per-record backing string and field slice, which
are intrinsic to parsing and would require a zero-copy/batch-slab parser
rework. A CPU profile of cat now shows it is I/O-bound (syscall ~56%, bufio
read+flush), with allocation/GC down to ~10% -- i.e. further allocation
trimming no longer moves cat's wall-clock. GOGC=off confirms (no change).
Verified: go test ./pkg/... and full regression suite pass; output is
byte-identical across all formats including record-retaining verbs (tac),
hashed and --no-hash-records.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Following the lazy-hashing commit, profiling showed the dominant remaining
cost in read/write-bound workloads is allocation *operations* (not bytes):
each input field allocated two heap objects -- an Mlrval (FromDeferredType)
and an MlrmapEntry. For 1M x 7-field CSV that is ~13.4M of ~18.6M total
allocations.
Introduce mlrval.RecordArena, a per-batch slab allocator: a reader draws
each field's entry and value from contiguous []MlrmapEntry / []Mlrval slabs,
turning two allocations per field into roughly two per slab. The arena grows
on demand, so the size hint need not be exact; on duplicate keys it mirrors
PutReferenceMaybeDedupe semantics. findEntry/linkNewEntry already supported
externally-constructed entries, so this is transparent.
Wired into every line-based reader that builds records from deferred-type
strings: CSV, CSV-lite, TSV, DKVP, NIDX, PPRINT, XTAB, DKVPX. (JSON values
arrive already typed and are unaffected.) Readers with inline batch loops use
a local arena; those that build records via a helper (DKVP/NIDX line
splitter, XTAB stanza) hold the arena on the reader struct, reset per batch
and also initialized in the constructor so direct/test callers never see nil.
Measured (big.*, 1M records, default flags, cat, best of 3):
csv 0.46 -> 0.27 (~41%)
dkvp 0.75 -> 0.46 (~39%)
nidx 1.92 -> 1.58 (~18%)
(xtab ~flat: dominated by stanza parse/emit, not field allocation)
For cat the allocation-object count drops from ~18.6M to ~4.85M and peak RSS
from ~402MB to ~237MB (slabs are compact and freed as units). Alloc *bytes*
are essentially unchanged -- confirming the cost was per-allocation overhead,
not volume. Streaming and accumulating verbs (put/sort) are unchanged: their
bottleneck is DSL-side allocation / heap scanning, not field construction.
Verified: go test ./pkg/... and full regression suite pass; output is
byte-identical across all formats (hashed and --no-hash-records).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Records (NewMlrmapAsRecord) eagerly allocated and populated a
map[string]*MlrmapEntry on construction whenever hashRecords was true
(the default). For streaming verbs that never look records up by key
(e.g. `mlr cat`) that map is pure overhead: a heap allocation plus N
map-inserts per record, and N more pointer-heavy objects for the GC to
scan. Profiling 1M-record CSV shows runtime allocation/GC machinery
dominating every workload, and `--no-hash-records` was 25-30% faster --
but that flag makes wide-record lookups O(n), the regression that
motivated hashing in #1506.
Make record hashing lazy instead: allocate no index up front; build it
in findEntry on the first lookup, and only when the record is wide
enough (FieldCount >= mlrmapHashThreshold) that linear search would
hurt. Narrow records and never-looked-up records never pay for a map;
wide records that are actually queried still get hash-accelerated
lookups, matching the old eager-hash default. DSL maps (NewMlrmap) keep
eager hashing to limit the behavioral surface.
This is transparent: findEntry already fell back to linear scan when
keysToEntries was nil, and every mutator already guarded on
keysToEntries != nil.
Measured (big.csv, 1M x 7 cols, default flags, best of 3):
cat 0.62 -> 0.47 (~24%)
put 1.08 -> 0.82 (~24%)
stats1 0.66 -> 0.57 (~14%)
sort 2.9 -> 2.0 (~30%)
Wide-column case protected: 60-col file with field lookups, lazy (1.42s)
matches old eager default (1.40s) and beats pure linear (1.55s).
Verified: go test ./pkg/... and full regression suite pass; output is
byte-identical to forced --hash-records for sort, stats1, cut,
wide-column put, and duplicate-key dedupe.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Fix column alignment for wide and combining Unicode chars (#1520, #379)
PPRINT, markdown-aligned, and XTAB writers measured column widths with
utf8.RuneCountInString, which counts codepoints rather than terminal
display columns. East-Asian fullwidth characters (counted as 1 but
displayed as 2) and zero-width combining marks (counted as 1 but
displayed as 0) both caused misalignment.
Add lib.DisplayWidth wrapping uniseg.StringWidth, and use it from the
three writers. The XTAB right-aligned path also drops fmt.Sprintf with
%*s since Go's %*s pads by rune count too.
UTF8Strlen is unchanged so DSL strlen semantics are preserved.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Fix PPRINT alignment with multi-character OFS (#1819)
Previously, writePadding repeated the OFS string for each padding slot, so
non-space or multi-character OFS values produced misaligned columns and
leaked the separator into padding (e.g. --ofs=XY yielded aXYXYXYXYb).
Pad with spaces instead, leaving OFS to act only as the column separator.
The default single-space OFS is unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Use non-space multi-char OFS in test 0021 for Windows portability
PowerShell collapses single-quoted ' ' in command lines, which broke
the test on the windows-latest CI runner. Switch to --ofs XY, which
exercises the same multi-character padding code path without shell
quoting hazards.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The --lp/--rp prefixes and -j output-field rename were applied only to
paired records, leaving unpaired records emitted via --ul/--ur with their
original (left/right) field names. This broke downstream operations like
unsparsify that expected consistent column names across paired and
unpaired output.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Default to "cat" verb when none is supplied (#2029)
Invocations like 'mlr --j2y' or 'mlr --c2p' previously failed with
"no verb supplied", forcing users to type the trailing 'cat'
explicitly for pure format conversions. Default the verb to 'cat'
in that case. Bare 'mlr' with no flags, no verb, and no files
still prints the main usage banner.
This handles flag-only invocations (e.g. 'mlr --c2j < input.csv'
or 'mlr --c2j --from input.csv'). File names without a preceding
verb are still parsed as verb candidates and continue to error if
not found; that broader change is out of scope here.
* Use ${MLR} substitution for bare-mlr regression case (#2029)
The regtester only substitutes the mlr executable when the cmd
starts with "mlr " (with a trailing space). The bare-mlr usage-banner
test had a cmd of just "mlr", so on CI -- which invokes regtest with
a relative path like 'test/../mlr' -- the test shelled out to a
literal 'mlr' that isn't on PATH and failed with exit 127.
Switch the cmd to ${MLR} (the regtester's explicit substitution
token) so the case runs the right binary in any invocation context.
PPRINT, markdown-aligned, and XTAB writers measured column widths with
utf8.RuneCountInString, which counts codepoints rather than terminal
display columns. East-Asian fullwidth characters (counted as 1 but
displayed as 2) and zero-width combining marks (counted as 1 but
displayed as 0) both caused misalignment.
Add lib.DisplayWidth wrapping uniseg.StringWidth, and use it from the
three writers. The XTAB right-aligned path also drops fmt.Sprintf with
%*s since Go's %*s pads by rune count too.
UTF8Strlen is unchanged so DSL strlen semantics are preserved.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The implicit-header readers for TSV and CSV-lite were missing the
`for` loop in their `nh < nd` branch, so when a data row had more
fields than the auto-generated header, only one extra field was
captured and the rest were dropped. Restore the loop, matching the
explicit-header readers and the pprint reader.
The shebang example `#!/usr/bin/env mlr -s` does not work on Linux
because env(1) does not accept arguments to the interpreter; use
`#!/usr/bin/env -S mlr -s` instead.
`contains` and `index` previously stringified their inputs silently,
so `contains([1,2], $foo)` would match against the literal string
`[1, 2]` and produce surprising results. Both now return a type-error
when either argument is an array or map; help text points users at
`any(arr, func(e){return e == x})` for membership tests.
Adds a new Markdown-only flag --omd-aligned (alias --omarkdown-aligned)
that left-justifies cells and pads each column to a uniform width.
The rendered table is unaffected; the goal is readability of the raw
markdown source. The flag implies --omd, so users do not need to pass
both.
Implementation batches records by schema (like pprint), computes max
column widths, then emits header / separator / data rows with bars
vertically aligned. Default markdown writer behavior is unchanged.
When using --fr (regex field selector) with stats1 -a null_count, void
(empty) field values were unconditionally skipped in
ingestWithValueFieldRegexes, causing null_count to always report 0.
The non-regex path (ingestWithoutValueFieldRegexes) already had a special
case that allows void values through for null_count accumulators. This
commit adds the same exception to the regex path.
Fixes#1639
Co-authored-by: cobyfrombrooklyn-bot <cobyfrombrooklyn-bot@users.noreply.github.com>
* fix: preserve key order in YAML output
MlrmapToYAMLNative converted the ordered Mlrmap to map[string]interface{},
which loses insertion order. yaml.v3.Marshal then sorts Go map keys
alphabetically, so YAML output always had sorted keys despite other formats
(JSON, pretty-print) correctly preserving order.
Switch to building yaml.Node trees that preserve the Mlrmap's linked-list
iteration order. Use yaml.Node.Encode for scalar values to correctly handle
edge cases (NaN/Inf, non-UTF-8 strings).
Fixes#2028.
* Update pkg/mlrval/mlrval_yaml_test.go with copilot code-review feedback
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: lawrence3699 <lawrence3699@users.noreply.github.com>
Co-authored-by: John Kerl <kerl.john.r@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
