When a transformer fails mid-stream (e.g. join -s with a malformed left
file), the error could be lost, yielding exit 0 with no stderr message:
- runSingleTransformerBatch forwarded the end-of-stream marker downstream
before runSingleTransformer sent the error to
dataProcessingErrorChannel, so the record-writer could finish and signal
done-writing while the error was still unsent.
- Even with the error buffered, stream.Stream's select loop chooses among
simultaneously-ready channels at random, and exiting on the done-writing
signal dropped the buffered error.
Send the error before forwarding the end-of-stream marker (so it is
always buffered before the writer can finish), and drain the error
channels after the select loop exits.
Observed as a one-off Windows CI failure of
test/cases/verb-join/left-file-malformed-sorted, where the same case
passed on automatic rerun within the same job. Reproduced locally by
widening the deschedule window with a sleep between the end-of-stream
forward and the error send: 5/5 runs exited 0 with empty stderr; with
this fix, 200/200 runs exit 1 with the expected message even with
adversarial delays injected on both sides of the end-of-stream forward.
Follow-up to the os.Exit-removal refactor (plans/exit.md, #2204/#2205).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Phase 3 of plans/exit.md: the streaming-interface change, the load-bearing
piece for #341 (DSL exit statement) and #440 (strict mode).
- RecordTransformer.Transform and RecordTransformerFunc now return error.
All 69 Transform implementations and their dispatch helpers updated
(mechanical rewrite, compiler- and errcheck-verified).
- runSingleTransformerBatch, on a Transform error, forwards any output
produced before the failure plus an end-of-stream marker downstream, so
the rest of the chain and the record-writer drain and finish cleanly;
runSingleTransformer then surfaces the error to stream.Stream's select
loop (non-blocking send; first error wins) and signals upstream-done so
the record-reader stops. This is exactly the flush-then-exit sequencing a
future DSL 'exit N' needs.
- dataProcessingErrorChannel and FileOutputHandler.recordErroredChannel are
now chan error instead of chan bool; ChannelWriter still prints write-
error details at the site and sends the 'exiting due to data error'
sentinel, preserving the exact stderr shape pinned by regression cases.
- Mid-stream os.Exit sites converted to returned errors: put/filter DSL
begin/main/end-block errors and the non-boolean filter-expression case,
tee write/close failures, split write/open/close failures, join left-file
ingest failures (both half-streaming and sorted paths, with full error
plumbing through JoinBucketKeeper), histogram/stats2 ingest errors, surv
fit errors, and step stepper allocation (tStepperAllocator now returns
(tStepper, error); bad EWMA coefficients propagate; negative slwin
parameters are reported by the CLI parser via the existing
bad-stepper-name pattern).
- The two genuinely internal join-bucket-keeper states now use
lib.InternalCodingErrorWithMessageIf instead of hand-rolled print+exit.
- pkg/transformers is now os.Exit-free.
Behavior notes: the non-boolean filter message gains the standard 'mlr: '
prefix and a newline (it previously printed with neither); tee errors now
include the underlying cause. All 4779 regression cases pass unchanged;
mlr head early-out latency is unaffected (0.02s over 50M records).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* refine the plan
* Fix all staticcheck lint findings (uncapped)
golangci-lint's default max-same-issues=3 was hiding most of the backlog:
the true pre-fix count was 69 staticcheck findings, not 34. This fixes all
of them, driving staticcheck to zero:
- ST1023/QF1011 (37): omit explicit types inferred from the RHS
- S1009/S1031 (15): drop redundant nil checks before len()/range
- SA9003 (9): remove comment-only empty branches, keeping the comments
- QF1007 (3): merge conditional assignment into declaration
- QF1006 (3): lift break conditions into loop conditions
- QF1001 (3): apply De Morgan's law / name the negated predicate
Also updates plans/lintfixes.md with the cap discovery and the corrected
errcheck picture (1202 uncapped, ~949 of them fmt.Fprint*).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Drive errcheck to zero: config for bulk categories, propagate real errors
Adds .golangci.yml with errcheck exclude-functions for fmt.Fprint* (usage
printers), (*bufio.Writer).Write/WriteString (sticky errors, surfaced at the
now-checked final Flush), and (*strings.Builder).WriteString; pins
max-issues-per-linter/max-same-issues to 0 so CI reports true counts.
Real error paths now propagate instead of being dropped:
- Finalize{Reader,Writer}Options in join/put/filter/split/tee and the
repl/script entry points: 'mlr join -i badformat' now errors instead of
silently using wrong separators
- final output-stream Flush in pkg/stream: write failure no longer exits 0
- DSL emit/print/dump redirect writes, matching their sibling branches
- CSV writer WriteCSVRecordMaybeColorized, close-time Flush in file output
handlers, ENV[...] Setenv, REPL record-write and redirect-close errors
- termcvt write-side Close before rename (had "TODO: check return status")
The rest are deliberate ignores, marked with _ = and a comment where the
reason isn't obvious: unset-of-missing-path no-ops, read-side closes,
mid-stream FlushOnEveryRecord, init-time strftime registrations, in-memory
usage-capture pipes, and regtest-harness env/temp-file teardown.
golangci-lint now reports 0 issues on ./cmd/mlr ./pkg/... with all caps off.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
