miller/pkg/transformers
John Kerl f9afae491f
Add mlr rank verb (#2178)
* stats1: add rank accumulator (#383)

Adds `mlr stats1 -a rank` for standard competition ranking (1,2,2,4,...)
on pre-sorted data, most useful with -s for a rank on every record.

* stats1: make rank order-independent by default, add --rank-sorted opt-in fast path

The rank accumulator previously only compared each value to the immediately
preceding record, silently giving wrong ranks for non-adjacent duplicates
(e.g. unsorted input, or interleaved -g groups). Default to correctly
computing standard competition rank from all values seen so far
(order-independent, buffers values, same approach as percentile
accumulators). Add --rank-sorted for callers who can promise sorted input
and want the previous O(1)-space streaming behavior instead.

* Revert "stats1: make rank order-independent by default, add --rank-sorted opt-in fast path"

This reverts commit aa45a591fe.

* Revert "stats1: add rank accumulator (#383)"

This reverts commit 96deed048a.

* Add mlr rank verb (#383)

Reverts the earlier stats1 -a rank / --rank-sorted approach: stats1 is a
reduce verb (many records -> one summary record per group) and rank is a
per-record annotator, so it never fit cleanly there -- it needed stats1's
-s iterative-stats escape hatch just to be useful, plus a bolted-on
sorted/unsorted split.

mlr rank is a dedicated verb modeled on mlr fraction: -f fields to rank,
-g optional group-by, output field <f>_rank. By default it's a two-pass
algorithm (buffers input, like fraction does) giving standard competition
rank (1,2,2,4,...) that's correct regardless of input order. --sorted
opts into a single-pass, O(1)-space streaming alternative for callers who
can promise pre-sorted input (e.g. via 'mlr sort' first).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* Regenerate docs/man pages after merging main (sparkline verb)

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-07 17:35:18 -04:00
..
utils Add mlr rank verb (#2178) 2026-07-07 17:35:18 -04:00
aaa_chain_transformer.go Performance and style fixes (#1981) 2026-02-18 09:19:31 -05:00
aaa_record_transformer.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
aaa_transformer_json.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
aaa_transformer_json_test.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
aaa_transformer_table.go Add mlr rank verb (#2178) 2026-07-07 17:35:18 -04:00
aaa_verb_usage.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
altkv.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
bar.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
bootstrap.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
case.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
cat.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
check.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
clean_whitespace.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
count.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
count_similar.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
cut.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
decimate.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
describe.go Add mlr describe schema/shape introspection verb (#2098 PR6) (#2132) 2026-07-03 18:43:49 -04:00
doc.go Export library code in pkg/ (#1391) 2023-09-10 17:15:13 -04:00
fill_down.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
fill_empty.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
flatten.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
format_values.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
fraction.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
gap.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
grep.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
group_by.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
group_like.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
having_fields.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
head.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
histogram.go Add sparkline DSL function and mlr sparkline verb (#166) (#2177) 2026-07-07 15:47:01 -04:00
join.go join: exit nonzero when the left file cannot be read (#2169) (#2171) 2026-07-06 17:34:33 -04:00
json_parse.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
json_stringify.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
label.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
latin1_to_utf8.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
merge_fields.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
most_or_least_frequent.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
nest.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
nothing.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
put_or_filter.go Add DSL validate/dry-run: put/filter --explain (#2098 PR5) (#2131) 2026-07-03 18:12:23 -04:00
rank.go Add mlr rank verb (#2178) 2026-07-07 17:35:18 -04:00
README.md Export library code in pkg/ (#1391) 2023-09-10 17:15:13 -04:00
regularize.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
remove_empty_columns.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
rename.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
reorder.go reorder -r: group matched fields by regex order, not record order (#2159) 2026-07-06 15:07:16 -04:00
repeat.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
reshape.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
sample.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
sec2gmt.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
sec2gmtdate.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
seqgen.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
shuffle.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
skip_trivial_records.go Exit 0 when skip-trivial-records is used and CSV/TSV ends with blank lines (#1535) (#2146) 2026-07-06 15:22:51 -04:00
sort.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
sort_within_records.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
sparkline.go Add sparkline DSL function and mlr sparkline verb (#166) (#2177) 2026-07-07 15:47:01 -04:00
sparsify.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
split.go Copy records handed to async file-writers in tee and split verbs (#1671) (#2152) 2026-07-05 20:17:32 -04:00
stats1.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
stats2.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
step.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
subs.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
summary.go Document meaning of hyphenated field_type values in mlr summary (#1082) (#2149) 2026-07-05 20:34:16 -04:00
surv.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
tac.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
tail.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
tee.go Copy records handed to async file-writers in tee and split verbs (#1671) (#2152) 2026-07-05 20:17:32 -04:00
template.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
top.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
unflatten.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
uniq.go Cross-reference head -n 1 -g from uniq for keep-all-columns dedupe (#1075) (#2158) 2026-07-06 15:26:39 -04:00
unspace.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
unsparsify.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00
utf8_to_latin1.go Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111) 2026-07-03 14:27:23 -04:00

Logic for transforming input records into output records as requested by the user (sort, filter, etc.).:

  • The IRecordTransformer abstract record-transformer interface datatype, as well as the ChainTransformer Go-channel chaining mechanism for piping one transformer into the next.
  • The transformer lookup table, used for Miller command-line parsing, verb construction, and online help.
  • All the concrete record-transformers such as cat, tac, sort, put, and so on.