After batch-arena field allocation, profiling cat over 1M-record CSV showed
the remaining ~5M allocations were almost entirely per-record (one each):
the Mlrmap struct, the RecordAndContext wrapper, the CSV writer's []string,
and the go-csv parser's own buffers.
Address the first three:
- mlrval.RecordArena gains NewRecord(), vending the Mlrmap struct itself from
a per-batch slab (respecting --no-hash-records). Rolled out to every
line-based reader (CSV, CSV-lite, TSV, DKVP, NIDX, PPRINT, XTAB, DKVPX) in
place of NewMlrmapAsRecord.
- The CSV reader batch-allocates RecordAndContext wrappers from a per-batch
slab instead of one heap object per record (comment/output-string entries
still allocate individually, but they are rare).
- RecordWriterCSV reuses a single fieldsBuffer []string across records instead
of allocating one per Write; WriteCSVRecordMaybeColorized consumes it
synchronously and the writer is single-goroutine, so this is safe.
Effect (big.*, 1M records, cat, best of 5):
csv 0.26 -> 0.22
dkvp 0.51 -> 0.45 (Mlrmap slab)
For CSV, cat's allocation-object count drops ~5.0M -> ~2.1M. The remaining
~2M are the go-csv parser's per-record backing string and field slice, which
are intrinsic to parsing and would require a zero-copy/batch-slab parser
rework. A CPU profile of cat now shows it is I/O-bound (syscall ~56%, bufio
read+flush), with allocation/GC down to ~10% -- i.e. further allocation
trimming no longer moves cat's wall-clock. GOGC=off confirms (no change).
Verified: go test ./pkg/... and full regression suite pass; output is
byte-identical across all formats including record-retaining verbs (tac),
hashed and --no-hash-records.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Following the lazy-hashing commit, profiling showed the dominant remaining
cost in read/write-bound workloads is allocation *operations* (not bytes):
each input field allocated two heap objects -- an Mlrval (FromDeferredType)
and an MlrmapEntry. For 1M x 7-field CSV that is ~13.4M of ~18.6M total
allocations.
Introduce mlrval.RecordArena, a per-batch slab allocator: a reader draws
each field's entry and value from contiguous []MlrmapEntry / []Mlrval slabs,
turning two allocations per field into roughly two per slab. The arena grows
on demand, so the size hint need not be exact; on duplicate keys it mirrors
PutReferenceMaybeDedupe semantics. findEntry/linkNewEntry already supported
externally-constructed entries, so this is transparent.
Wired into every line-based reader that builds records from deferred-type
strings: CSV, CSV-lite, TSV, DKVP, NIDX, PPRINT, XTAB, DKVPX. (JSON values
arrive already typed and are unaffected.) Readers with inline batch loops use
a local arena; those that build records via a helper (DKVP/NIDX line
splitter, XTAB stanza) hold the arena on the reader struct, reset per batch
and also initialized in the constructor so direct/test callers never see nil.
Measured (big.*, 1M records, default flags, cat, best of 3):
csv 0.46 -> 0.27 (~41%)
dkvp 0.75 -> 0.46 (~39%)
nidx 1.92 -> 1.58 (~18%)
(xtab ~flat: dominated by stanza parse/emit, not field allocation)
For cat the allocation-object count drops from ~18.6M to ~4.85M and peak RSS
from ~402MB to ~237MB (slabs are compact and freed as units). Alloc *bytes*
are essentially unchanged -- confirming the cost was per-allocation overhead,
not volume. Streaming and accumulating verbs (put/sort) are unchanged: their
bottleneck is DSL-side allocation / heap scanning, not field construction.
Verified: go test ./pkg/... and full regression suite pass; output is
byte-identical across all formats (hashed and --no-hash-records).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Switch to integer ranges in for loops
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Switch to slices functions where appropriate
A number of utility functions can be replaced outright; since Miller
can technically be used as a library, these are deprecated rather than
removed. go:fix directives ensure that they can be replaced
automatically.
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Switch to reflect.TypeFor
This is slightly more efficient than TypeOf when the type is known at
compile time.
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Switch to strings.SplitSeq instead of strings.Split
SplitSeq results in fewer allocations.
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Drop obsolete build directives
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Use min/max instead of explicit comparisons
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Append slices instead of looping
Signed-off-by: Stephen Kitt <steve@sk2.org>
---------
Signed-off-by: Stephen Kitt <steve@sk2.org>
