Mirrors/miller
1
0
Fork 0
mirror of https://github.com/johnkerl/miller.git synced 2026-07-17 16:38:54 +00:00
Code Issues Projects Releases Wiki Activity
miller/pkg/transformers
History
John Kerl 5ede886106
Fix mid-stream error-loss race (flaky exit codes on transformer errors) (#2211) 
When a transformer fails mid-stream (e.g. join -s with a malformed left
file), the error could be lost, yielding exit 0 with no stderr message:

- runSingleTransformerBatch forwarded the end-of-stream marker downstream
  before runSingleTransformer sent the error to
  dataProcessingErrorChannel, so the record-writer could finish and signal
  done-writing while the error was still unsent.
- Even with the error buffered, stream.Stream's select loop chooses among
  simultaneously-ready channels at random, and exiting on the done-writing
  signal dropped the buffered error.

Send the error before forwarding the end-of-stream marker (so it is
always buffered before the writer can finish), and drain the error
channels after the select loop exits.

Observed as a one-off Windows CI failure of
test/cases/verb-join/left-file-malformed-sorted, where the same case
passed on automatic rerun within the same job. Reproduced locally by
widening the deschedule window with a sleep between the end-of-stream
forward and the error send: 5/5 runs exited 0 with empty stderr; with
this fix, 200/200 runs exit 1 with the expected message even with
adversarial delays injected on both sides of the end-of-stream forward.

Follow-up to the os.Exit-removal refactor (plans/exit.md, #2204/#2205).

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-15 18:26:56 -04:00
..
utils join: add --ignore-empty to skip pairing on empty-string join keys (#1194) (#2210) 2026-07-15 16:42:39 -04:00
aaa_chain_transformer.go Fix mid-stream error-loss race (flaky exit codes on transformer errors) (#2211) 2026-07-15 18:26:56 -04:00
aaa_record_transformer.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
aaa_transformer_json.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
aaa_transformer_json_test.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
aaa_transformer_table.go Add bootstrap-ci verb for bootstrap-resampling confidence intervals (#1670) (#2189) 2026-07-14 17:00:34 -04:00
aaa_verb_usage.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
altkv.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
bar.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
bootstrap.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
bootstrap_ci.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
bootstrap_ci_test.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
case.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
cat.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
check.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
clean_whitespace.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
count.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
count_similar.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
cut.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
decimate.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
describe.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
doc.go
fill_down.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
fill_empty.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
flatten.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
format_values.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
fraction.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
gap.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
grep.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
group_by.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
group_like.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
having_fields.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
head.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
histogram.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
join.go join: add --ignore-empty to skip pairing on empty-string join keys (#1194) (#2210) 2026-07-15 16:42:39 -04:00
json_parse.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
json_stringify.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
label.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
latin1_to_utf8.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
merge_fields.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
most_or_least_frequent.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
nest.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
nothing.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
put_or_filter.go plans/exit.md: record final status; defer phase 5; convert stragglers (#2207) 2026-07-15 16:05:53 -04:00
rank.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
README.md
regularize.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
remove_empty_columns.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
rename.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
reorder.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
repeat.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
reshape.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
sample.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
sec2gmt.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
sec2gmtdate.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
seqgen.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
shuffle.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
skip_trivial_records.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
sort.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
sort_within_records.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
sparkline.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
sparsify.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
split.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
stats1.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
stats1_test.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
stats2.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
step.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
step_test.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
subs.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
summary.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
surv.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
tac.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
tail.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
tee.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
template.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
top.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
unflatten.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
uniq.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
unspace.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
unsparsify.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00
utf8_to_latin1.go Remove os.Exit callsites below the entrypoint: phase 3 (plans/exit.md) (#2204) 2026-07-15 15:07:09 -04:00

README.md

Logic for transforming input records into output records as requested by the user (sort, filter, etc.).:

  • The IRecordTransformer abstract record-transformer interface datatype, as well as the ChainTransformer Go-channel chaining mechanism for piping one transformer into the next.
  • The transformer lookup table, used for Miller command-line parsing, verb construction, and online help.
  • All the concrete record-transformers such as cat, tac, sort, put, and so on.