mirror of
https://github.com/johnkerl/miller.git
synced 2026-07-18 00:45:47 +00:00
e8d19f6892
Records (NewMlrmapAsRecord) eagerly allocated and populated a map[string]*MlrmapEntry on construction whenever hashRecords was true (the default). For streaming verbs that never look records up by key (e.g. `mlr cat`) that map is pure overhead: a heap allocation plus N map-inserts per record, and N more pointer-heavy objects for the GC to scan. Profiling 1M-record CSV shows runtime allocation/GC machinery dominating every workload, and `--no-hash-records` was 25-30% faster -- but that flag makes wide-record lookups O(n), the regression that motivated hashing in #1506. Make record hashing lazy instead: allocate no index up front; build it in findEntry on the first lookup, and only when the record is wide enough (FieldCount >= mlrmapHashThreshold) that linear search would hurt. Narrow records and never-looked-up records never pay for a map; wide records that are actually queried still get hash-accelerated lookups, matching the old eager-hash default. DSL maps (NewMlrmap) keep eager hashing to limit the behavioral surface. This is transparent: findEntry already fell back to linear scan when keysToEntries was nil, and every mutator already guarded on keysToEntries != nil. Measured (big.csv, 1M x 7 cols, default flags, best of 3): cat 0.62 -> 0.47 (~24%) put 1.08 -> 0.82 (~24%) stats1 0.66 -> 0.57 (~14%) sort 2.9 -> 2.0 (~30%) Wide-column case protected: 60-col file with field lookups, lazy (1.42s) matches old eager default (1.40s) and beats pure linear (1.55s). Verified: go test ./pkg/... and full regression suite pass; output is byte-identical to forced --hash-records for sort, stats1, cut, wide-column put, and duplicate-key dedupe. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| auxents | ||
| bifs | ||
| cli | ||
| climain | ||
| colorizer | ||
| dkvpx | ||
| dsl | ||
| entrypoint | ||
| go-csv | ||
| input | ||
| lib | ||
| mlrval | ||
| output | ||
| parsing | ||
| pbnjay-strptime | ||
| platform | ||
| runtime | ||
| scan | ||
| stream | ||
| terminals | ||
| transformers | ||
| types | ||
| version | ||
| README.md | ||
Please see ../../README-dev.md for an overview; please see each subdirectory for details about it.