By default, records with an empty-string value in a join field are
paired just like any other value, so two records both missing an ID
get matched with each other -- rarely the intended behavior.
--ignore-empty treats an empty-string join-field value as if the
field were absent, on both the left and right files, so such records
fall through to unpaired handling (--np/--ul/--ur) instead of
cross-joining on the empty string.
Wires the check through both the default half-streaming join and the
-s/--sorted-input doubly-streaming join, which track left-file
buckets independently and needed the same empty-aware key-presence
check in JoinBucketKeeper.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Phase 3 of plans/exit.md: the streaming-interface change, the load-bearing
piece for #341 (DSL exit statement) and #440 (strict mode).
- RecordTransformer.Transform and RecordTransformerFunc now return error.
All 69 Transform implementations and their dispatch helpers updated
(mechanical rewrite, compiler- and errcheck-verified).
- runSingleTransformerBatch, on a Transform error, forwards any output
produced before the failure plus an end-of-stream marker downstream, so
the rest of the chain and the record-writer drain and finish cleanly;
runSingleTransformer then surfaces the error to stream.Stream's select
loop (non-blocking send; first error wins) and signals upstream-done so
the record-reader stops. This is exactly the flush-then-exit sequencing a
future DSL 'exit N' needs.
- dataProcessingErrorChannel and FileOutputHandler.recordErroredChannel are
now chan error instead of chan bool; ChannelWriter still prints write-
error details at the site and sends the 'exiting due to data error'
sentinel, preserving the exact stderr shape pinned by regression cases.
- Mid-stream os.Exit sites converted to returned errors: put/filter DSL
begin/main/end-block errors and the non-boolean filter-expression case,
tee write/close failures, split write/open/close failures, join left-file
ingest failures (both half-streaming and sorted paths, with full error
plumbing through JoinBucketKeeper), histogram/stats2 ingest errors, surv
fit errors, and step stepper allocation (tStepperAllocator now returns
(tStepper, error); bad EWMA coefficients propagate; negative slwin
parameters are reported by the CLI parser via the existing
bad-stepper-name pattern).
- The two genuinely internal join-bucket-keeper states now use
lib.InternalCodingErrorWithMessageIf instead of hand-rolled print+exit.
- pkg/transformers is now os.Exit-free.
Behavior notes: the non-boolean filter message gains the standard 'mlr: '
prefix and a newline (it previously printed with neither); tee errors now
include the underlying cause. All 4779 regression cases pass unchanged;
mlr head early-out latency is unaffected (0.02s over 50M records).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Remove os.Exit callsites below the entrypoint: phase 2 (plans/exit.md)
Phase 2 of plans/exit.md: the FlagParser signature change.
- FlagParser gains an error return; all 259 inline parser closures in
option_parse.go and NoOpParse1 updated (mechanical rewrite, compiler-checked).
- FlagTable.Parse returns (bool, error); its nine callers (climain passes one
and two, .mlrrc line handling, verb-local flag parsing in tee/join/put/
filter/split, and the repl/script terminals) propagate the error instead of
letting parsers kill the process.
- cli.CheckArgCount returns its missing-argument message as an error rather
than printing and exiting; new cli.FlagErrorf (counterpart of VerbErrorf)
carries user-facing main-flag messages.
- The 14 print-and-exit sites inside parser closures become returned errors;
--list-color-codes/--list-color-names return ExitRequest{0} after printing.
- The repl and script terminals translate ExitRequest into their int return
codes, so 'mlr repl --list-color-codes' still exits 0.
- pkg/cli is now os.Exit-free; stale comments about exiting parsers updated
(the completion introspection accessors remain, since parsers mutate the
options struct).
Stderr messages and exit codes are byte-identical, including the two-line
missing-argument and --seed messages and the trailing-period forms pinned by
test/cases/cli-mfrom/0003.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Merge main (recutils #2201); convert its three new parser closures
Same mechanical rewrite as the rest of phase 2: error return signature plus
final 'return nil' on --irecutils/--orecutils/--recutils.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The join verb's left-file ingest silently swallowed read errors: a
nonexistent or malformed left file produced empty (or partial) output
with exit code 0.
Two defects, one per ingest path:
* Unsorted (half-streaming) path, ingestLeftFile in
pkg/transformers/join.go: errors from the record-reader's error
channel were dropped ("TODO: propagate error to caller"), as were
record-reader construction errors. All left-file read failures were
silent.
* Sorted (-s) path, readRecord in
pkg/transformers/utils/join_bucket_keeper.go: the error channel was
consulted, but the record-reader goroutine sends the error and the
end-of-stream marker on separate channels, so the select could see
end-of-stream first and drop the error. A nonexistent left file with
-s exited 0 about half the time (racy).
Both paths now print "mlr: <error>" to stderr and exit 1, matching how
read errors on regular (right) input files are reported. On receipt of
the end-of-stream marker, both consumers now do a final non-blocking
drain of the error channel, which is deterministic since the reader
sends any error before the marker.
Discovered while verifying #377.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The join verb's help text listed only '-i {one of csv,dkvp,nidx,pprint,xtab}'
for overriding the left-file input format, but the verb also accepts the
--icsv/--ijson-style main-flag shorthands (and formats beyond the five
listed, e.g. json and tsv), since unrecognized verb flags fall through to
the main flag table. Update the usage text to say so, and regenerate the
man page and docs content that embed this help output.
Fixes#444.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Tier-2 structured verb options: OptionSpec, initial migration (#2098)
PR 3 of the AI-friendly roadmap (plans/plan-2098-llm.md).
Infrastructure:
- Add OptionSpec{Flag,Arg,Type,Desc,Repeatable,Values} to
pkg/transformers/aaa_record_transformer.go alongside TransformerSetup.
Type is one of: bool, string, int, float, csv-list, regex, filename,
format, enum. For type=="enum", Values lists the valid choices.
- Add Options []OptionSpec to TransformerSetup (nil = not yet migrated).
- Emit Options in VerbInfoForJSON (omitempty so unmigrated verbs stay
backward-compatible; agents check key presence for Tier-2 availability).
UsageText is always present as the Tier-1 prose fallback.
- Add VerbOptionsNilCheck() in aaa_verb_options_check.go: progress report
of migrated vs. unmigrated verbs, analogous to FLAG_TABLE.NilCheck().
- Wire verb-options-nil-check into mlr help (internal/docgen section).
Initial migration (5/70 verbs):
- nothing: empty Options (no verb-specific options, explicitly migrated)
- cat: -n (bool), -N (string), -g (csv-list), --filename, --filenum (bool)
- head: -g (csv-list), -n (int)
- tail: -g (csv-list), -n (int)
- tee: -a, -p (bool)
Tests:
- 5 new unit tests in aaa_transformer_json_test.go covering migrated/
unmigrated paths, field population, JSON round-trip, and key-presence.
- Regression test case 0003: mlr help verb-options-nil-check golden output.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Migrate all 70 verbs to structured OptionSpec; bump catalog schema to v2
Completes the Tier-2 migration started in the previous commit. Every verb
in TRANSFORMER_LOOKUP_TABLE now has a non-nil Options field.
- Workflow-migrated all 65 remaining verbs. Each Setup var now carries
Options: []OptionSpec{...} with Flag/Arg/Type/Desc fields. Verbs with
no verb-specific options (altkv, check, group-like, nothing, etc.) use
an empty slice to signal "migrated but no options."
- Drop `omitempty` from VerbInfoForJSON.Options: empty slices were silently
dropped, making migrated-no-option verbs indistinguishable from unmigrated
ones in JSON. Without omitempty: null=unmigrated, []=migrated-no-options,
[...]= migrated-with-options. Bump catalogSchemaVersion 1→2 for this shape
change.
- Replace the two "unmigrated-verb" unit tests (which used stats1 as an
example) with TestAllVerbsFullyMigrated (asserts every verb has non-nil
Options) and TestAllVerbsHaveOptionsKeyInJSON (asserts every migrated
verb emits the "options" key in JSON).
- Regenerate test/cases/cli-help/0003/expout: now reads
"Verb options migration: 70/70 migrated."
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* remove a transitional helper
* git rms
* Render verb usage Options blocks from structured OptionSpec
Each verb's usage message and its Tier-2 OptionSpec list previously
duplicated the option text. New WriteVerbOptions (aaa_verb_usage.go)
renders the "Options:" block from the specs: aligned flag column,
descriptions word-wrapped at 80, uniform trailing -h|--help line.
- OptionSpec gains Aliases (JSON "aliases") so long-form spellings
like join's --lk|--left-keep-field-names survive in both outputs
- All 70 verbs migrated; options literals hoisted to package-level
vars (usage funcs can't reference their Setup var without a Go
init cycle)
- Hand-written per-option details the specs had condensed away are
merged into Desc, enriching the JSON catalog
- Non-option prose (examples, cross-references, dynamic accumulator
listings) kept verbatim
- Regenerated the six usage-embedding regression expectations and
the two affected doc pages
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix pre-existing usage-text bugs surfaced by the OptionSpec migration
- gap: usage said "One of -f or -g is required" but the parser takes
-n or -g
- seqgen: drop description line copy-pasted from cat ("Passes input
records directly to output...") which contradicted "Discards the
input record stream"
- utf8-to-latin1: description read inverted ("from Latin-1 to UTF-8")
- sec2gmtdate: usage said "../c/mlr" instead of "mlr"
- top: document the accepted-but-undocumented --max flag
- stats2: add linreg-pca to the -a enum values, matching the runtime
accumulator table
Regression expectations and docs regenerated accordingly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix check usage sentence order; stats1 usage blank line to usage stream
- check: the description's second and third lines were swapped,
reading "Consumes records without printing any output, / Useful for
doing a well-formatted check on input data. / with the exception
that warnings are printed to stderr."
- stats1: a bare fmt.Println() in the usage func wrote its blank line
to process stdout instead of the usage output stream
Regression expectation and docs regenerated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* refine the plan
* Fix all staticcheck lint findings (uncapped)
golangci-lint's default max-same-issues=3 was hiding most of the backlog:
the true pre-fix count was 69 staticcheck findings, not 34. This fixes all
of them, driving staticcheck to zero:
- ST1023/QF1011 (37): omit explicit types inferred from the RHS
- S1009/S1031 (15): drop redundant nil checks before len()/range
- SA9003 (9): remove comment-only empty branches, keeping the comments
- QF1007 (3): merge conditional assignment into declaration
- QF1006 (3): lift break conditions into loop conditions
- QF1001 (3): apply De Morgan's law / name the negated predicate
Also updates plans/lintfixes.md with the cap discovery and the corrected
errcheck picture (1202 uncapped, ~949 of them fmt.Fprint*).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Drive errcheck to zero: config for bulk categories, propagate real errors
Adds .golangci.yml with errcheck exclude-functions for fmt.Fprint* (usage
printers), (*bufio.Writer).Write/WriteString (sticky errors, surfaced at the
now-checked final Flush), and (*strings.Builder).WriteString; pins
max-issues-per-linter/max-same-issues to 0 so CI reports true counts.
Real error paths now propagate instead of being dropped:
- Finalize{Reader,Writer}Options in join/put/filter/split/tee and the
repl/script entry points: 'mlr join -i badformat' now errors instead of
silently using wrong separators
- final output-stream Flush in pkg/stream: write failure no longer exits 0
- DSL emit/print/dump redirect writes, matching their sibling branches
- CSV writer WriteCSVRecordMaybeColorized, close-time Flush in file output
handlers, ENV[...] Setenv, REPL record-write and redirect-close errors
- termcvt write-side Close before rename (had "TODO: check return status")
The rest are deliberate ignores, marked with _ = and a comment where the
reason isn't obvious: unset-of-missing-path no-ops, read-side closes,
mid-stream FlushOnEveryRecord, init-time strftime registrations, in-memory
usage-capture pipes, and regtest-harness env/temp-file teardown.
golangci-lint now reports 0 issues on ./cmd/mlr ./pkg/... with all caps off.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Replaces 100+ if/else-if chains on a single variable with tagged switch
statements across 72 files. The bulk are transformer option-parsing loops
(switch on opt string), plus a handful of value-dispatch sites in mlrval,
dsl/cst, repl, lib, auxents, and bifs. One case (surv.go) required a
labeled break to preserve the loop-exit behavior of the original else branch.
Fixes staticcheck QF1003 findings.
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
The --lp/--rp prefixes and -j output-field rename were applied only to
paired records, leaving unpaired records emitted via --ul/--ur with their
original (left/right) field names. This broke downstream operations like
unsparsify that expected consistent column names across paired and
unpaired output.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Switch to integer ranges in for loops
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Switch to slices functions where appropriate
A number of utility functions can be replaced outright; since Miller
can technically be used as a library, these are deprecated rather than
removed. go:fix directives ensure that they can be replaced
automatically.
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Switch to reflect.TypeFor
This is slightly more efficient than TypeOf when the type is known at
compile time.
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Switch to strings.SplitSeq instead of strings.Split
SplitSeq results in fewer allocations.
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Drop obsolete build directives
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Use min/max instead of explicit comparisons
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Append slices instead of looping
Signed-off-by: Stephen Kitt <steve@sk2.org>
---------
Signed-off-by: Stephen Kitt <steve@sk2.org>
