Mirrors/miller
1
0
Fork 0
mirror of https://github.com/johnkerl/miller.git synced 2026-07-18 08:55:41 +00:00
Code Issues Projects Releases Wiki Activity
miller/pkg/input
History
John Kerl d2acbfeea3 Batch-allocate per-record objects; reuse CSV writer field buffer 
After batch-arena field allocation, profiling cat over 1M-record CSV showed
the remaining ~5M allocations were almost entirely per-record (one each):
the Mlrmap struct, the RecordAndContext wrapper, the CSV writer's []string,
and the go-csv parser's own buffers.

Address the first three:

- mlrval.RecordArena gains NewRecord(), vending the Mlrmap struct itself from
  a per-batch slab (respecting --no-hash-records). Rolled out to every
  line-based reader (CSV, CSV-lite, TSV, DKVP, NIDX, PPRINT, XTAB, DKVPX) in
  place of NewMlrmapAsRecord.

- The CSV reader batch-allocates RecordAndContext wrappers from a per-batch
  slab instead of one heap object per record (comment/output-string entries
  still allocate individually, but they are rare).

- RecordWriterCSV reuses a single fieldsBuffer []string across records instead
  of allocating one per Write; WriteCSVRecordMaybeColorized consumes it
  synchronously and the writer is single-goroutine, so this is safe.

Effect (big.*, 1M records, cat, best of 5):
  csv   0.26 -> 0.22
  dkvp  0.51 -> 0.45  (Mlrmap slab)

For CSV, cat's allocation-object count drops ~5.0M -> ~2.1M. The remaining
~2M are the go-csv parser's per-record backing string and field slice, which
are intrinsic to parsing and would require a zero-copy/batch-slab parser
rework. A CPU profile of cat now shows it is I/O-bound (syscall ~56%, bufio
read+flush), with allocation/GC down to ~10% -- i.e. further allocation
trimming no longer moves cat's wall-clock. GOGC=off confirms (no change).

Verified: go test ./pkg/... and full regression suite pass; output is
byte-identical across all formats including record-retaining verbs (tac),
hashed and --no-hash-records.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-19 12:27:22 -04:00
..
constants.go Separate out ILineReader abstraction (#1504) 2024-02-24 22:07:56 -05:00
doc.go Export library code in pkg/ (#1391) 2023-09-10 17:15:13 -04:00
fixed_width_splitter.go Some fixes for staticcheck (#2006) 2026-03-03 09:27:03 -05:00
fixed_width_splitter_test.go Add fixed width support for pprint reader (#1999) 2026-03-01 11:25:17 -05:00
line_reader.go Replace list.List with Go slices (#1950) 2026-02-01 17:22:28 -05:00
pseudo_reader_gen.go Multiple style updates (#1974) 2026-02-16 15:49:21 -05:00
README.md Export library code in pkg/ (#1391) 2023-09-10 17:15:13 -04:00
record_reader.go Replace list.List with Go slices (#1950) 2026-02-01 17:22:28 -05:00
record_reader_benchmark_test.go Replace list.List with Go slices (#1950) 2026-02-01 17:22:28 -05:00
record_reader_csv.go Batch-allocate per-record objects; reuse CSV writer field buffer 2026-06-19 12:27:22 -04:00
record_reader_csvlite.go Batch-allocate per-record objects; reuse CSV writer field buffer 2026-06-19 12:27:22 -04:00
record_reader_dcf.go Add DCF (Debian Control Format) as supported file type (#1970) 2026-02-16 11:10:39 -05:00
record_reader_dkvp_nidx.go Batch-allocate per-record objects; reuse CSV writer field buffer 2026-06-19 12:27:22 -04:00
record_reader_dkvp_test.go The package version must match the major tag version (#1654) 2024-09-20 12:10:11 -04:00
record_reader_dkvpx.go Batch-allocate per-record objects; reuse CSV writer field buffer 2026-06-19 12:27:22 -04:00
record_reader_dkvpx_test.go Add DKVPX file format (#2002) 2026-03-02 22:35:08 -05:00
record_reader_factory.go Add DKVPX file format (#2002) 2026-03-02 22:35:08 -05:00
record_reader_json.go Performance and style fixes (#1981) 2026-02-18 09:19:31 -05:00
record_reader_markdown.go The package version must match the major tag version (#1654) 2024-09-20 12:10:11 -04:00
record_reader_pprint.go Batch-allocate per-record objects; reuse CSV writer field buffer 2026-06-19 12:27:22 -04:00
record_reader_tsv.go Batch-allocate per-record objects; reuse CSV writer field buffer 2026-06-19 12:27:22 -04:00
record_reader_xtab.go Batch-allocate per-record objects; reuse CSV writer field buffer 2026-06-19 12:27:22 -04:00
record_reader_yaml.go Support YAML I/O (#1963) 2026-02-15 09:24:38 -05:00
splitters.go Multiple style updates (#1974) 2026-02-16 15:49:21 -05:00

README.md

Logic for parsing disk-file/standard-input byte streams into records.

There is one record-reader type per supported input file format, and a factory method.