Miller type-inference accepts float formats which the JSON grammar
disallows -- leading zeros like 004.56, leading plus signs, bare leading
or trailing decimal points like .56 or 4., etc. -- and the JSON writer
passed the original text through verbatim, producing invalid JSON.
Leading-zero integers such as 0123456789 were already handled (inferred
as strings), and hex/binary/octal ints were already re-rendered as
decimal for JSON per #1761; this does the same for floats: when a
float string representation is not a valid JSON number, re-render it
(e.g. 004.56 -> 4.56).
Also documents in reference-main-data-types that CSV double-quoting
does not affect type inference, with -S / --infer-none and string()
as the ways to keep such values as strings.
Addresses #1114 and #1293.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Add MT_BYTES mlrval type: foundation and disposition tables
First step toward a first-class bytes type in the DSL (#1231).
Adds MT_BYTES (payload []byte, rendered as lowercase hex in all output
formats, JSON-encoded as a hex string), extends every disposition
matrix/vector with the new row/column -- real cells for comparison,
sorting, and dot-concat of bytes with bytes; type-error stubs
elsewhere -- and adds sweep tests asserting no table has nil cells,
since Go zero-fills short array literals when MT_DIM grows.
Bytes values are not yet constructible from the DSL; b"..." literals
and constructor/codec functions follow in subsequent commits.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add b"..." bytes-literal syntax to the DSL
Adds a bytes_literal token to the grammar (regenerating the PGPG lexer
and parser) and a BytesLiteralNode in the CST which evaluates to an
MT_BYTES mlrval. Escape handling reuses UnbackslashStringLiteral,
which is already byte-oriented: b"\xff" is the single byte 0xff.
Unlike string literals, bytes literals never participate in
regex-capture replacement. A bare identifier b is unaffected.
Part of #1231.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add bytes DSL functions: conversions, codecs, and bytes-aware built-ins
- bytes(x) converts strings to bytes; string(b) reinterprets raw bytes
as UTF-8 text (the reverse)
- base64_decode now always returns bytes (superseding the interim
string-or-hex behavior); base64_encode accepts string or bytes
- New hex_encode/hex_decode functions
- is_bytes and asserting_bytes predicates
- md5/sha1/sha256/sha512 accept bytes, hashing the raw payload
- strlen of bytes is the byte count; substr/substr0/substr1 on bytes
slice by byte position and return bytes
The Cyrillic-LDAP scenario from #1231 now works without exec
workarounds: string(base64_decode($x)) recovers the text, and binary
payloads survive undamaged as bytes.
Closes#1231.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add bytes-type docs and regression cases
Documents the bytes type on the data-types page, regenerates the
function-reference/man-page material, and adds regression coverage:
literal escape forms, operators (concat/compare/slice/sort and
type errors), conversions and codec round-trips, and CSV-to-JSON
output rendering of bytes fields.
Part of #1231.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Reposition MT_BYTES to sort adjacent to MT_STRING in the type enum
MT_BYTES was appended after MT_ABSENT for index stability; move it
right after MT_STRING instead, since that's where it conceptually
belongs and where it already sorts in the cmp disposition matrices.
Mechanically re-derive all ~40 disposition tables in pkg/bifs and
pkg/mlrval accordingly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fix windows CI
* fix merge
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* refine the plan
* Fix all staticcheck lint findings (uncapped)
golangci-lint's default max-same-issues=3 was hiding most of the backlog:
the true pre-fix count was 69 staticcheck findings, not 34. This fixes all
of them, driving staticcheck to zero:
- ST1023/QF1011 (37): omit explicit types inferred from the RHS
- S1009/S1031 (15): drop redundant nil checks before len()/range
- SA9003 (9): remove comment-only empty branches, keeping the comments
- QF1007 (3): merge conditional assignment into declaration
- QF1006 (3): lift break conditions into loop conditions
- QF1001 (3): apply De Morgan's law / name the negated predicate
Also updates plans/lintfixes.md with the cap discovery and the corrected
errcheck picture (1202 uncapped, ~949 of them fmt.Fprint*).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Drive errcheck to zero: config for bulk categories, propagate real errors
Adds .golangci.yml with errcheck exclude-functions for fmt.Fprint* (usage
printers), (*bufio.Writer).Write/WriteString (sticky errors, surfaced at the
now-checked final Flush), and (*strings.Builder).WriteString; pins
max-issues-per-linter/max-same-issues to 0 so CI reports true counts.
Real error paths now propagate instead of being dropped:
- Finalize{Reader,Writer}Options in join/put/filter/split/tee and the
repl/script entry points: 'mlr join -i badformat' now errors instead of
silently using wrong separators
- final output-stream Flush in pkg/stream: write failure no longer exits 0
- DSL emit/print/dump redirect writes, matching their sibling branches
- CSV writer WriteCSVRecordMaybeColorized, close-time Flush in file output
handlers, ENV[...] Setenv, REPL record-write and redirect-close errors
- termcvt write-side Close before rename (had "TODO: check return status")
The rest are deliberate ignores, marked with _ = and a comment where the
reason isn't obvious: unset-of-missing-path no-ops, read-side closes,
mid-stream FlushOnEveryRecord, init-time strftime registrations, in-memory
usage-capture pipes, and regtest-harness env/temp-file teardown.
golangci-lint now reports 0 issues on ./cmd/mlr ./pkg/... with all caps off.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* plans/lintfixes.md
* plans/lintfixes.md
* Fix remaining govet lint findings
- Rename MarshalJSON -> FormatAsJSON on Mlrval and Mlrmap (govet
stdmethods): the methods shadowed json.Marshaler with an
incompatible signature.
- Remove unreachable return after exhaustive if-else in
pkg/mlrval/mlrval_collections.go (govet unreachable).
- Update plans/lintfixes.md with current status: 84 findings remain
(50 errcheck, 34 staticcheck).
Part of #2109.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Return nil on successful single-index array unset
removeIndexedOnArray removed the element on the in-bounds path but
then fell through to return an "array index out of bounds for unset"
error, so the success path never returned nil. Callers currently
ignore the error, which masked this; return nil on success so that
upcoming errcheck fixes can propagate the error meaningfully. This
matches removeIndexedOnMap, which returns nil on success.
Add unit tests for RemoveIndexed on arrays.
Part of #2109.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Found with golang.org/x/tools/cmd/deadcode (rooted at cmd/mlr + tests)
and staticcheck U1000; each finding verified by hand before deletion.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Replaces 100+ if/else-if chains on a single variable with tagged switch
statements across 72 files. The bulk are transformer option-parsing loops
(switch on opt string), plus a handful of value-dispatch sites in mlrval,
dsl/cst, repl, lib, auxents, and bifs. One case (surv.go) required a
labeled break to preserve the loop-exit behavior of the original else branch.
Fixes staticcheck QF1003 findings.
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
After batch-arena field allocation, profiling cat over 1M-record CSV showed
the remaining ~5M allocations were almost entirely per-record (one each):
the Mlrmap struct, the RecordAndContext wrapper, the CSV writer's []string,
and the go-csv parser's own buffers.
Address the first three:
- mlrval.RecordArena gains NewRecord(), vending the Mlrmap struct itself from
a per-batch slab (respecting --no-hash-records). Rolled out to every
line-based reader (CSV, CSV-lite, TSV, DKVP, NIDX, PPRINT, XTAB, DKVPX) in
place of NewMlrmapAsRecord.
- The CSV reader batch-allocates RecordAndContext wrappers from a per-batch
slab instead of one heap object per record (comment/output-string entries
still allocate individually, but they are rare).
- RecordWriterCSV reuses a single fieldsBuffer []string across records instead
of allocating one per Write; WriteCSVRecordMaybeColorized consumes it
synchronously and the writer is single-goroutine, so this is safe.
Effect (big.*, 1M records, cat, best of 5):
csv 0.26 -> 0.22
dkvp 0.51 -> 0.45 (Mlrmap slab)
For CSV, cat's allocation-object count drops ~5.0M -> ~2.1M. The remaining
~2M are the go-csv parser's per-record backing string and field slice, which
are intrinsic to parsing and would require a zero-copy/batch-slab parser
rework. A CPU profile of cat now shows it is I/O-bound (syscall ~56%, bufio
read+flush), with allocation/GC down to ~10% -- i.e. further allocation
trimming no longer moves cat's wall-clock. GOGC=off confirms (no change).
Verified: go test ./pkg/... and full regression suite pass; output is
byte-identical across all formats including record-retaining verbs (tac),
hashed and --no-hash-records.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* Lazy per-record hashing: ~15-30% faster on common workloads
Records (NewMlrmapAsRecord) eagerly allocated and populated a
map[string]*MlrmapEntry on construction whenever hashRecords was true
(the default). For streaming verbs that never look records up by key
(e.g. `mlr cat`) that map is pure overhead: a heap allocation plus N
map-inserts per record, and N more pointer-heavy objects for the GC to
scan. Profiling 1M-record CSV shows runtime allocation/GC machinery
dominating every workload, and `--no-hash-records` was 25-30% faster --
but that flag makes wide-record lookups O(n), the regression that
motivated hashing in #1506.
Make record hashing lazy instead: allocate no index up front; build it
in findEntry on the first lookup, and only when the record is wide
enough (FieldCount >= mlrmapHashThreshold) that linear search would
hurt. Narrow records and never-looked-up records never pay for a map;
wide records that are actually queried still get hash-accelerated
lookups, matching the old eager-hash default. DSL maps (NewMlrmap) keep
eager hashing to limit the behavioral surface.
This is transparent: findEntry already fell back to linear scan when
keysToEntries was nil, and every mutator already guarded on
keysToEntries != nil.
Measured (big.csv, 1M x 7 cols, default flags, best of 3):
cat 0.62 -> 0.47 (~24%)
put 1.08 -> 0.82 (~24%)
stats1 0.66 -> 0.57 (~14%)
sort 2.9 -> 2.0 (~30%)
Wide-column case protected: 60-col file with field lookups, lazy (1.42s)
matches old eager default (1.40s) and beats pure linear (1.55s).
Verified: go test ./pkg/... and full regression suite pass; output is
byte-identical to forced --hash-records for sort, stats1, cut,
wide-column put, and duplicate-key dedupe.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Batch-arena field allocation for line-based readers (approach B)
Following the lazy-hashing commit, profiling showed the dominant remaining
cost in read/write-bound workloads is allocation *operations* (not bytes):
each input field allocated two heap objects -- an Mlrval (FromDeferredType)
and an MlrmapEntry. For 1M x 7-field CSV that is ~13.4M of ~18.6M total
allocations.
Introduce mlrval.RecordArena, a per-batch slab allocator: a reader draws
each field's entry and value from contiguous []MlrmapEntry / []Mlrval slabs,
turning two allocations per field into roughly two per slab. The arena grows
on demand, so the size hint need not be exact; on duplicate keys it mirrors
PutReferenceMaybeDedupe semantics. findEntry/linkNewEntry already supported
externally-constructed entries, so this is transparent.
Wired into every line-based reader that builds records from deferred-type
strings: CSV, CSV-lite, TSV, DKVP, NIDX, PPRINT, XTAB, DKVPX. (JSON values
arrive already typed and are unaffected.) Readers with inline batch loops use
a local arena; those that build records via a helper (DKVP/NIDX line
splitter, XTAB stanza) hold the arena on the reader struct, reset per batch
and also initialized in the constructor so direct/test callers never see nil.
Measured (big.*, 1M records, default flags, cat, best of 3):
csv 0.46 -> 0.27 (~41%)
dkvp 0.75 -> 0.46 (~39%)
nidx 1.92 -> 1.58 (~18%)
(xtab ~flat: dominated by stanza parse/emit, not field allocation)
For cat the allocation-object count drops from ~18.6M to ~4.85M and peak RSS
from ~402MB to ~237MB (slabs are compact and freed as units). Alloc *bytes*
are essentially unchanged -- confirming the cost was per-allocation overhead,
not volume. Streaming and accumulating verbs (put/sort) are unchanged: their
bottleneck is DSL-side allocation / heap scanning, not field construction.
Verified: go test ./pkg/... and full regression suite pass; output is
byte-identical across all formats (hashed and --no-hash-records).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Records (NewMlrmapAsRecord) eagerly allocated and populated a
map[string]*MlrmapEntry on construction whenever hashRecords was true
(the default). For streaming verbs that never look records up by key
(e.g. `mlr cat`) that map is pure overhead: a heap allocation plus N
map-inserts per record, and N more pointer-heavy objects for the GC to
scan. Profiling 1M-record CSV shows runtime allocation/GC machinery
dominating every workload, and `--no-hash-records` was 25-30% faster --
but that flag makes wide-record lookups O(n), the regression that
motivated hashing in #1506.
Make record hashing lazy instead: allocate no index up front; build it
in findEntry on the first lookup, and only when the record is wide
enough (FieldCount >= mlrmapHashThreshold) that linear search would
hurt. Narrow records and never-looked-up records never pay for a map;
wide records that are actually queried still get hash-accelerated
lookups, matching the old eager-hash default. DSL maps (NewMlrmap) keep
eager hashing to limit the behavioral surface.
This is transparent: findEntry already fell back to linear scan when
keysToEntries was nil, and every mutator already guarded on
keysToEntries != nil.
Measured (big.csv, 1M x 7 cols, default flags, best of 3):
cat 0.62 -> 0.47 (~24%)
put 1.08 -> 0.82 (~24%)
stats1 0.66 -> 0.57 (~14%)
sort 2.9 -> 2.0 (~30%)
Wide-column case protected: 60-col file with field lookups, lazy (1.42s)
matches old eager default (1.40s) and beats pure linear (1.55s).
Verified: go test ./pkg/... and full regression suite pass; output is
byte-identical to forced --hash-records for sort, stats1, cut,
wide-column put, and duplicate-key dedupe.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix: preserve key order in YAML output
MlrmapToYAMLNative converted the ordered Mlrmap to map[string]interface{},
which loses insertion order. yaml.v3.Marshal then sorts Go map keys
alphabetically, so YAML output always had sorted keys despite other formats
(JSON, pretty-print) correctly preserving order.
Switch to building yaml.Node trees that preserve the Mlrmap's linked-list
iteration order. Use yaml.Node.Encode for scalar values to correctly handle
edge cases (NaN/Inf, non-UTF-8 strings).
Fixes#2028.
* Update pkg/mlrval/mlrval_yaml_test.go with copilot code-review feedback
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: lawrence3699 <lawrence3699@users.noreply.github.com>
Co-authored-by: John Kerl <kerl.john.r@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Move `$[[...]]` and `$[[[...]]]` out of the grammar and into Go
* Update pkg/parsing/mlr.bnf
* Update Go source
* update test files
* Fix NPE
* .gitignore
* more
* Switch to integer ranges in for loops
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Switch to slices functions where appropriate
A number of utility functions can be replaced outright; since Miller
can technically be used as a library, these are deprecated rather than
removed. go:fix directives ensure that they can be replaced
automatically.
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Switch to reflect.TypeFor
This is slightly more efficient than TypeOf when the type is known at
compile time.
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Switch to strings.SplitSeq instead of strings.Split
SplitSeq results in fewer allocations.
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Drop obsolete build directives
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Use min/max instead of explicit comparisons
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Append slices instead of looping
Signed-off-by: Stephen Kitt <steve@sk2.org>
---------
Signed-off-by: Stephen Kitt <steve@sk2.org>
* docs/src/proposal-yaml-io.md
* initial step
* Add `--y2c` etc
* test cases
* docs
* more testing
* more
* more test cases
* git rm docs/src/proposal-yaml-io.md
* make dev
* ylistwrap -> yarray
