The reader's three-pass pipeline (backslash-join -> fold "+"-continuations
onto raw lines -> split each folded line on ": ") broke on fields written
as bare "Name:" (colon, no trailing space, no value) whose value is
supplied entirely by following "+" lines -- the idiomatic recutils pattern
for e.g. a "%doc:" field. Folding the continuation onto the raw line
destroyed the ": " substring needed for the later split, so parsing such a
field crashed with a "missing field separator" error, even on well-formed
input.
Found by testing against GNU recutils' own tutorial example (books.rec,
from the manual's "A Little Example" page), now added as a fixture.
Restructured parsing to split each line into a key/value field before
folding continuations, so folding happens on the value rather than the
raw line; when the preceding value is empty, the continuation becomes the
value outright instead of gaining a spurious leading newline.
Adds regression coverage (test/cases/io-recutils/0006, 0007) and a docs
section covering the empty-value + continuation pattern.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Implements the plan in plans/recutils.md, addressing #378. Adds a
hand-rolled reader/writer pair (pkg/input/record_reader_rec.go,
pkg/output/record_writer_rec.go) supporting recutils' blank-line-separated
"Key: value" records, "+"-continuation and backslash-newline continuation,
and generic comment handling, with recutils' %rec-descriptor schema layer
left unspecialized since Miller has no schema-enforcement concept.
Registers the format under the "recutils" name (--irecutils/--orecutils/
--recutils flags, matching the --idcf/--odcf/--dcf pattern) in the
reader/writer factories and pkg/cli/separators.go, adds docs and a sample
data file, and adds unit tests plus test/cases/io-recutils regression
cases covering round-tripping, continuation lines, comments, and the
hard-error behavior on malformed input.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
The DKVPX record reader set dkvpxReader.Comma = ',' unconditionally,
ignoring the resolved --ifs value; the key-value separator '=' was
likewise hard-coded in the dkvpx parser, ignoring --ips. So e.g.
printf 'a=1;b=2\n' | mlr -i dkvpx --ifs semicolon --ojson cat
produced a single field {"a": "1;b=2"} instead of {"a": 1, "b": 2}.
Fixes:
* pkg/dkvpx.Reader gains an Equals field (default '=') alongside Comma.
* The DKVPX record reader now passes the resolved IFS/IPS through,
validating that each is a single character (mirroring the CSV
reader's "IFS can only be a single character" error) rather than
silently ignoring or truncating them.
* The DKVPX record writer already emitted OFS/OPS, but its quoting
decisions were based on the default separators, so with --ofs
semicolon a value containing ';' went out unquoted (ambiguous) while
a value containing ',' was quoted needlessly. Quoting now keys off
the actual OFS/OPS.
Adds unit tests, regression cases under test/cases/io-dkvpx/, and a
note in the separators reference table.
Discovered while investigating #369; tracked in #2170.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
When the skip-trivial-records verb is in the then-chain, the CSV/TSV
record-readers now silently skip trivial (all-fields-empty) input lines
-- notably blank lines at the end of a CSV file -- instead of raising a
fatal header/data length mismatch. The user has explicitly asked for
trivial records to be skipped, so mlr now exits 0 in this case.
The verb's CLI parser sets a new ReaderOptions.SkipTrivialRecords flag,
which the CSV and TSV readers consult only on the would-be-fatal
mismatch path, so:
* Behavior without the verb is unchanged: blank lines still error.
* Genuinely ragged non-trivial records still error even with the verb.
* --allow-ragged-csv-input behavior is unchanged.
Fixes#1535.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
In the explicit-header ragged code paths of the csvlite, TSV, and
barred-pprint readers, the header-longer-than-data fill used
record.PutCopy with the un-deduplicated header name. With duplicate
header names (e.g. "name,4,4" deduped to "name,4,4_2"), the VOID fill
for the missing trailing field overwrote the already-stored value under
the colliding key, so a short row like "bar,sea" lost "sea".
Use record.PutReferenceMaybeDedupe, as the implicit-header paths
already do (where PR #794 had applied it, although implicit headers are
generated unique and never need dedupe).
Adds regression cases under test/cases/io-dedupe-field-names/ for the
csvlite, TSV, and barred-pprint readers.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* refine the plan
* Fix all staticcheck lint findings (uncapped)
golangci-lint's default max-same-issues=3 was hiding most of the backlog:
the true pre-fix count was 69 staticcheck findings, not 34. This fixes all
of them, driving staticcheck to zero:
- ST1023/QF1011 (37): omit explicit types inferred from the RHS
- S1009/S1031 (15): drop redundant nil checks before len()/range
- SA9003 (9): remove comment-only empty branches, keeping the comments
- QF1007 (3): merge conditional assignment into declaration
- QF1006 (3): lift break conditions into loop conditions
- QF1001 (3): apply De Morgan's law / name the negated predicate
Also updates plans/lintfixes.md with the cap discovery and the corrected
errcheck picture (1202 uncapped, ~949 of them fmt.Fprint*).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Drive errcheck to zero: config for bulk categories, propagate real errors
Adds .golangci.yml with errcheck exclude-functions for fmt.Fprint* (usage
printers), (*bufio.Writer).Write/WriteString (sticky errors, surfaced at the
now-checked final Flush), and (*strings.Builder).WriteString; pins
max-issues-per-linter/max-same-issues to 0 so CI reports true counts.
Real error paths now propagate instead of being dropped:
- Finalize{Reader,Writer}Options in join/put/filter/split/tee and the
repl/script entry points: 'mlr join -i badformat' now errors instead of
silently using wrong separators
- final output-stream Flush in pkg/stream: write failure no longer exits 0
- DSL emit/print/dump redirect writes, matching their sibling branches
- CSV writer WriteCSVRecordMaybeColorized, close-time Flush in file output
handlers, ENV[...] Setenv, REPL record-write and redirect-close errors
- termcvt write-side Close before rename (had "TODO: check return status")
The rest are deliberate ignores, marked with _ = and a comment where the
reason isn't obvious: unset-of-missing-path no-ops, read-side closes,
mid-stream FlushOnEveryRecord, init-time strftime registrations, in-memory
usage-capture pipes, and regtest-harness env/temp-file teardown.
golangci-lint now reports 0 issues on ./cmd/mlr ./pkg/... with all caps off.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
After batch-arena field allocation, profiling cat over 1M-record CSV showed
the remaining ~5M allocations were almost entirely per-record (one each):
the Mlrmap struct, the RecordAndContext wrapper, the CSV writer's []string,
and the go-csv parser's own buffers.
Address the first three:
- mlrval.RecordArena gains NewRecord(), vending the Mlrmap struct itself from
a per-batch slab (respecting --no-hash-records). Rolled out to every
line-based reader (CSV, CSV-lite, TSV, DKVP, NIDX, PPRINT, XTAB, DKVPX) in
place of NewMlrmapAsRecord.
- The CSV reader batch-allocates RecordAndContext wrappers from a per-batch
slab instead of one heap object per record (comment/output-string entries
still allocate individually, but they are rare).
- RecordWriterCSV reuses a single fieldsBuffer []string across records instead
of allocating one per Write; WriteCSVRecordMaybeColorized consumes it
synchronously and the writer is single-goroutine, so this is safe.
Effect (big.*, 1M records, cat, best of 5):
csv 0.26 -> 0.22
dkvp 0.51 -> 0.45 (Mlrmap slab)
For CSV, cat's allocation-object count drops ~5.0M -> ~2.1M. The remaining
~2M are the go-csv parser's per-record backing string and field slice, which
are intrinsic to parsing and would require a zero-copy/batch-slab parser
rework. A CPU profile of cat now shows it is I/O-bound (syscall ~56%, bufio
read+flush), with allocation/GC down to ~10% -- i.e. further allocation
trimming no longer moves cat's wall-clock. GOGC=off confirms (no change).
Verified: go test ./pkg/... and full regression suite pass; output is
byte-identical across all formats including record-retaining verbs (tac),
hashed and --no-hash-records.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* Lazy per-record hashing: ~15-30% faster on common workloads
Records (NewMlrmapAsRecord) eagerly allocated and populated a
map[string]*MlrmapEntry on construction whenever hashRecords was true
(the default). For streaming verbs that never look records up by key
(e.g. `mlr cat`) that map is pure overhead: a heap allocation plus N
map-inserts per record, and N more pointer-heavy objects for the GC to
scan. Profiling 1M-record CSV shows runtime allocation/GC machinery
dominating every workload, and `--no-hash-records` was 25-30% faster --
but that flag makes wide-record lookups O(n), the regression that
motivated hashing in #1506.
Make record hashing lazy instead: allocate no index up front; build it
in findEntry on the first lookup, and only when the record is wide
enough (FieldCount >= mlrmapHashThreshold) that linear search would
hurt. Narrow records and never-looked-up records never pay for a map;
wide records that are actually queried still get hash-accelerated
lookups, matching the old eager-hash default. DSL maps (NewMlrmap) keep
eager hashing to limit the behavioral surface.
This is transparent: findEntry already fell back to linear scan when
keysToEntries was nil, and every mutator already guarded on
keysToEntries != nil.
Measured (big.csv, 1M x 7 cols, default flags, best of 3):
cat 0.62 -> 0.47 (~24%)
put 1.08 -> 0.82 (~24%)
stats1 0.66 -> 0.57 (~14%)
sort 2.9 -> 2.0 (~30%)
Wide-column case protected: 60-col file with field lookups, lazy (1.42s)
matches old eager default (1.40s) and beats pure linear (1.55s).
Verified: go test ./pkg/... and full regression suite pass; output is
byte-identical to forced --hash-records for sort, stats1, cut,
wide-column put, and duplicate-key dedupe.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Batch-arena field allocation for line-based readers (approach B)
Following the lazy-hashing commit, profiling showed the dominant remaining
cost in read/write-bound workloads is allocation *operations* (not bytes):
each input field allocated two heap objects -- an Mlrval (FromDeferredType)
and an MlrmapEntry. For 1M x 7-field CSV that is ~13.4M of ~18.6M total
allocations.
Introduce mlrval.RecordArena, a per-batch slab allocator: a reader draws
each field's entry and value from contiguous []MlrmapEntry / []Mlrval slabs,
turning two allocations per field into roughly two per slab. The arena grows
on demand, so the size hint need not be exact; on duplicate keys it mirrors
PutReferenceMaybeDedupe semantics. findEntry/linkNewEntry already supported
externally-constructed entries, so this is transparent.
Wired into every line-based reader that builds records from deferred-type
strings: CSV, CSV-lite, TSV, DKVP, NIDX, PPRINT, XTAB, DKVPX. (JSON values
arrive already typed and are unaffected.) Readers with inline batch loops use
a local arena; those that build records via a helper (DKVP/NIDX line
splitter, XTAB stanza) hold the arena on the reader struct, reset per batch
and also initialized in the constructor so direct/test callers never see nil.
Measured (big.*, 1M records, default flags, cat, best of 3):
csv 0.46 -> 0.27 (~41%)
dkvp 0.75 -> 0.46 (~39%)
nidx 1.92 -> 1.58 (~18%)
(xtab ~flat: dominated by stanza parse/emit, not field allocation)
For cat the allocation-object count drops from ~18.6M to ~4.85M and peak RSS
from ~402MB to ~237MB (slabs are compact and freed as units). Alloc *bytes*
are essentially unchanged -- confirming the cost was per-allocation overhead,
not volume. Streaming and accumulating verbs (put/sort) are unchanged: their
bottleneck is DSL-side allocation / heap scanning, not field construction.
Verified: go test ./pkg/... and full regression suite pass; output is
byte-identical across all formats (hashed and --no-hash-records).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The implicit-header readers for TSV and CSV-lite were missing the
`for` loop in their `nh < nd` branch, so when a data row had more
fields than the auto-generated header, only one extra field was
captured and the rest were dropped. Restore the loop, matching the
explicit-header readers and the pprint reader.
* Switch to integer ranges in for loops
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Switch to slices functions where appropriate
A number of utility functions can be replaced outright; since Miller
can technically be used as a library, these are deprecated rather than
removed. go:fix directives ensure that they can be replaced
automatically.
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Switch to reflect.TypeFor
This is slightly more efficient than TypeOf when the type is known at
compile time.
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Switch to strings.SplitSeq instead of strings.Split
SplitSeq results in fewer allocations.
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Drop obsolete build directives
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Use min/max instead of explicit comparisons
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Append slices instead of looping
Signed-off-by: Stephen Kitt <steve@sk2.org>
---------
Signed-off-by: Stephen Kitt <steve@sk2.org>
* reader
* record writer
* pkg/output/record_writer_dcf.go
* test/input/test.dcf
* test/cases/io-dcf/
* pkg/cli/option_parse.go
* make fmt
* make dev
* docs and `make dev`
* docs/src/proposal-yaml-io.md
* initial step
* Add `--y2c` etc
* test cases
* docs
* more testing
* more
* more test cases
* git rm docs/src/proposal-yaml-io.md
* make dev
* ylistwrap -> yarray
* Switch to bufio.Reader, first pass
* temp
* Simplify ILineReader by making it stateless
* Interface not necessary; ILineReader -> TLineReader
* neaten
* iterating
