mirror of
https://github.com/johnkerl/miller.git
synced 2026-07-18 00:45:47 +00:00
d00de0f71a
After batch-arena field allocation, profiling cat over 1M-record CSV showed the remaining ~5M allocations were almost entirely per-record (one each): the Mlrmap struct, the RecordAndContext wrapper, the CSV writer's []string, and the go-csv parser's own buffers. Address the first three: - mlrval.RecordArena gains NewRecord(), vending the Mlrmap struct itself from a per-batch slab (respecting --no-hash-records). Rolled out to every line-based reader (CSV, CSV-lite, TSV, DKVP, NIDX, PPRINT, XTAB, DKVPX) in place of NewMlrmapAsRecord. - The CSV reader batch-allocates RecordAndContext wrappers from a per-batch slab instead of one heap object per record (comment/output-string entries still allocate individually, but they are rare). - RecordWriterCSV reuses a single fieldsBuffer []string across records instead of allocating one per Write; WriteCSVRecordMaybeColorized consumes it synchronously and the writer is single-goroutine, so this is safe. Effect (big.*, 1M records, cat, best of 5): csv 0.26 -> 0.22 dkvp 0.51 -> 0.45 (Mlrmap slab) For CSV, cat's allocation-object count drops ~5.0M -> ~2.1M. The remaining ~2M are the go-csv parser's per-record backing string and field slice, which are intrinsic to parsing and would require a zero-copy/batch-slab parser rework. A CPU profile of cat now shows it is I/O-bound (syscall ~56%, bufio read+flush), with allocation/GC down to ~10% -- i.e. further allocation trimming no longer moves cat's wall-clock. GOGC=off confirms (no change). Verified: go test ./pkg/... and full regression suite pass; output is byte-identical across all formats including record-retaining verbs (tac), hashed and --no-hash-records. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| mlrmap.go | ||
| mlrmap_accessors.go | ||
| mlrmap_accessors_test.go | ||
| mlrmap_flatten_unflatten.go | ||
| mlrmap_json.go | ||
| mlrmap_new_test.go | ||
| mlrmap_print.go | ||
| mlrval_accessors.go | ||
| mlrval_benchmark_test.go | ||
| mlrval_cmp.go | ||
| mlrval_cmp_test.go | ||
| mlrval_collections.go | ||
| mlrval_constants.go | ||
| mlrval_copy.go | ||
| mlrval_format.go | ||
| mlrval_format_test.go | ||
| mlrval_get.go | ||
| mlrval_get_test.go | ||
| mlrval_infer.go | ||
| mlrval_infer_test.go | ||
| mlrval_is.go | ||
| mlrval_is_test.go | ||
| mlrval_json.go | ||
| mlrval_new.go | ||
| mlrval_new_test.go | ||
| mlrval_output.go | ||
| mlrval_output_test.go | ||
| mlrval_sort.go | ||
| mlrval_sort_test.go | ||
| mlrval_type.go | ||
| mlrval_type_test.go | ||
| mlrval_yaml.go | ||
| mlrval_yaml_test.go | ||
| record_arena.go | ||