mirror of
https://github.com/johnkerl/miller.git
synced 2026-07-18 00:45:47 +00:00
5b29ad1596
Prototype of a zero-copy CSV record reader (ZeroCopyCSVReader): reads input
in large persistent blocks and returns field strings as unsafe.String views
directly into the block, eliminating the stdlib-derived parser's per-record
string() copy and recordBuffer accumulation. Unquoted records are fully
zero-copy; quoted records delegate to the go-csv parser for correctness.
Correctness: full regression suite passes, including a deterministic stress
run with a 3-byte forced block size (maximal boundary splitting). Required
fixing comment trailing-newline handling, comment-before-quote ordering, and
absolute error-line-number rewriting for delegated quoted records.
Performance: NEGATIVE RESULT -- do not merge.
- Wall-clock neutral: CSV cat is I/O-bound (syscall ~56%) after the
allocation-batching work, so removing the per-record copy saves CPU that
already overlapped with I/O on idle cores (confirmed under GOGC variations
and across cat/stats1/cut).
- Peak RSS 2-2.6x WORSE (162MB -> 357-423MB) at every block size: zero-copy
pins whole input blocks while any field references them, and the pipeline
buffers up to 500 batches, so many blocks stay live at once. The stdlib
parser's per-record strings keep memory proportional to live data.
Committed for the record; tracked in a GitHub issue. Not for merge.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|---|---|---|
| .. | ||
| constants.go | ||
| csv_zerocopy_reader.go | ||
| doc.go | ||
| fixed_width_splitter.go | ||
| fixed_width_splitter_test.go | ||
| line_reader.go | ||
| pseudo_reader_gen.go | ||
| README.md | ||
| record_reader.go | ||
| record_reader_benchmark_test.go | ||
| record_reader_csv.go | ||
| record_reader_csvlite.go | ||
| record_reader_dcf.go | ||
| record_reader_dkvp_nidx.go | ||
| record_reader_dkvp_test.go | ||
| record_reader_dkvpx.go | ||
| record_reader_dkvpx_test.go | ||
| record_reader_factory.go | ||
| record_reader_json.go | ||
| record_reader_markdown.go | ||
| record_reader_pprint.go | ||
| record_reader_tsv.go | ||
| record_reader_xtab.go | ||
| record_reader_yaml.go | ||
| splitters.go | ||
Logic for parsing disk-file/standard-input byte streams into records.
There is one record-reader type per supported input file format, and a factory method.