mirror of
https://github.com/johnkerl/miller.git
synced 2026-07-17 16:38:54 +00:00
5b29ad1596
Prototype of a zero-copy CSV record reader (ZeroCopyCSVReader): reads input
in large persistent blocks and returns field strings as unsafe.String views
directly into the block, eliminating the stdlib-derived parser's per-record
string() copy and recordBuffer accumulation. Unquoted records are fully
zero-copy; quoted records delegate to the go-csv parser for correctness.
Correctness: full regression suite passes, including a deterministic stress
run with a 3-byte forced block size (maximal boundary splitting). Required
fixing comment trailing-newline handling, comment-before-quote ordering, and
absolute error-line-number rewriting for delegated quoted records.
Performance: NEGATIVE RESULT -- do not merge.
- Wall-clock neutral: CSV cat is I/O-bound (syscall ~56%) after the
allocation-batching work, so removing the per-record copy saves CPU that
already overlapped with I/O on idle cores (confirmed under GOGC variations
and across cat/stats1/cut).
- Peak RSS 2-2.6x WORSE (162MB -> 357-423MB) at every block size: zero-copy
pins whole input blocks while any field references them, and the pipeline
buffers up to 500 batches, so many blocks stay live at once. The stdlib
parser's per-record strings keep memory proportional to live data.
Committed for the record; tracked in a GitHub issue. Not for merge.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|---|---|---|
| .. | ||
| auxents | ||
| bifs | ||
| cli | ||
| climain | ||
| colorizer | ||
| dkvpx | ||
| dsl | ||
| entrypoint | ||
| go-csv | ||
| input | ||
| lib | ||
| mlrval | ||
| output | ||
| parsing | ||
| pbnjay-strptime | ||
| platform | ||
| runtime | ||
| scan | ||
| stream | ||
| terminals | ||
| transformers | ||
| types | ||
| version | ||
| README.md | ||
Please see ../../README-dev.md for an overview; please see each subdirectory for details about it.