By default, records with an empty-string value in a join field are
paired just like any other value, so two records both missing an ID
get matched with each other -- rarely the intended behavior.
--ignore-empty treats an empty-string join-field value as if the
field were absent, on both the left and right files, so such records
fall through to unpaired handling (--np/--ul/--ur) instead of
cross-joining on the empty string.
Wires the check through both the default half-streaming join and the
-s/--sorted-input doubly-streaming join, which track left-file
buckets independently and needed the same empty-aware key-presence
check in JoinBucketKeeper.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Phase 3 of plans/exit.md: the streaming-interface change, the load-bearing
piece for #341 (DSL exit statement) and #440 (strict mode).
- RecordTransformer.Transform and RecordTransformerFunc now return error.
All 69 Transform implementations and their dispatch helpers updated
(mechanical rewrite, compiler- and errcheck-verified).
- runSingleTransformerBatch, on a Transform error, forwards any output
produced before the failure plus an end-of-stream marker downstream, so
the rest of the chain and the record-writer drain and finish cleanly;
runSingleTransformer then surfaces the error to stream.Stream's select
loop (non-blocking send; first error wins) and signals upstream-done so
the record-reader stops. This is exactly the flush-then-exit sequencing a
future DSL 'exit N' needs.
- dataProcessingErrorChannel and FileOutputHandler.recordErroredChannel are
now chan error instead of chan bool; ChannelWriter still prints write-
error details at the site and sends the 'exiting due to data error'
sentinel, preserving the exact stderr shape pinned by regression cases.
- Mid-stream os.Exit sites converted to returned errors: put/filter DSL
begin/main/end-block errors and the non-boolean filter-expression case,
tee write/close failures, split write/open/close failures, join left-file
ingest failures (both half-streaming and sorted paths, with full error
plumbing through JoinBucketKeeper), histogram/stats2 ingest errors, surv
fit errors, and step stepper allocation (tStepperAllocator now returns
(tStepper, error); bad EWMA coefficients propagate; negative slwin
parameters are reported by the CLI parser via the existing
bad-stepper-name pattern).
- The two genuinely internal join-bucket-keeper states now use
lib.InternalCodingErrorWithMessageIf instead of hand-rolled print+exit.
- pkg/transformers is now os.Exit-free.
Behavior notes: the non-boolean filter message gains the standard 'mlr: '
prefix and a newline (it previously printed with neither); tee errors now
include the underlying cause. All 4779 regression cases pass unchanged;
mlr head early-out latency is unaffected (0.02s over 50M records).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* stats1: add rank accumulator (#383)
Adds `mlr stats1 -a rank` for standard competition ranking (1,2,2,4,...)
on pre-sorted data, most useful with -s for a rank on every record.
* stats1: make rank order-independent by default, add --rank-sorted opt-in fast path
The rank accumulator previously only compared each value to the immediately
preceding record, silently giving wrong ranks for non-adjacent duplicates
(e.g. unsorted input, or interleaved -g groups). Default to correctly
computing standard competition rank from all values seen so far
(order-independent, buffers values, same approach as percentile
accumulators). Add --rank-sorted for callers who can promise sorted input
and want the previous O(1)-space streaming behavior instead.
* Revert "stats1: make rank order-independent by default, add --rank-sorted opt-in fast path"
This reverts commit aa45a591fe.
* Revert "stats1: add rank accumulator (#383)"
This reverts commit 96deed048a.
* Add mlr rank verb (#383)
Reverts the earlier stats1 -a rank / --rank-sorted approach: stats1 is a
reduce verb (many records -> one summary record per group) and rank is a
per-record annotator, so it never fit cleanly there -- it needed stats1's
-s iterative-stats escape hatch just to be useful, plus a bolted-on
sorted/unsorted split.
mlr rank is a dedicated verb modeled on mlr fraction: -f fields to rank,
-g optional group-by, output field <f>_rank. By default it's a two-pass
algorithm (buffers input, like fraction does) giving standard competition
rank (1,2,2,4,...) that's correct regardless of input order. --sorted
opts into a single-pass, O(1)-space streaming alternative for callers who
can promise pre-sorted input (e.g. via 'mlr sort' first).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Regenerate docs/man pages after merging main (sparkline verb)
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
The join verb's left-file ingest silently swallowed read errors: a
nonexistent or malformed left file produced empty (or partial) output
with exit code 0.
Two defects, one per ingest path:
* Unsorted (half-streaming) path, ingestLeftFile in
pkg/transformers/join.go: errors from the record-reader's error
channel were dropped ("TODO: propagate error to caller"), as were
record-reader construction errors. All left-file read failures were
silent.
* Sorted (-s) path, readRecord in
pkg/transformers/utils/join_bucket_keeper.go: the error channel was
consulted, but the record-reader goroutine sends the error and the
end-of-stream marker on separate channels, so the select could see
end-of-stream first and drop the error. A nonexistent left file with
-s exited 0 about half the time (racy).
Both paths now print "mlr: <error>" to stderr and exit 1, matching how
read errors on regular (right) input files are reported. On receipt of
the end-of-stream marker, both consumers now do a final non-blocking
drain of the error channel, which is deterministic since the reader
sends any error before the marker.
Discovered while verifying #377.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Replaces 100+ if/else-if chains on a single variable with tagged switch
statements across 72 files. The bulk are transformer option-parsing loops
(switch on opt string), plus a handful of value-dispatch sites in mlrval,
dsl/cst, repl, lib, auxents, and bifs. One case (surv.go) required a
labeled break to preserve the loop-exit behavior of the original else branch.
Fixes staticcheck QF1003 findings.
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Switch to integer ranges in for loops
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Switch to slices functions where appropriate
A number of utility functions can be replaced outright; since Miller
can technically be used as a library, these are deprecated rather than
removed. go:fix directives ensure that they can be replaced
automatically.
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Switch to reflect.TypeFor
This is slightly more efficient than TypeOf when the type is known at
compile time.
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Switch to strings.SplitSeq instead of strings.Split
SplitSeq results in fewer allocations.
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Drop obsolete build directives
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Use min/max instead of explicit comparisons
Signed-off-by: Stephen Kitt <steve@sk2.org>
* Append slices instead of looping
Signed-off-by: Stephen Kitt <steve@sk2.org>
---------
Signed-off-by: Stephen Kitt <steve@sk2.org>
