Add mlr rank verb (#2178)

* stats1: add rank accumulator (#383)

Adds `mlr stats1 -a rank` for standard competition ranking (1,2,2,4,...)
on pre-sorted data, most useful with -s for a rank on every record.

* stats1: make rank order-independent by default, add --rank-sorted opt-in fast path

The rank accumulator previously only compared each value to the immediately
preceding record, silently giving wrong ranks for non-adjacent duplicates
(e.g. unsorted input, or interleaved -g groups). Default to correctly
computing standard competition rank from all values seen so far
(order-independent, buffers values, same approach as percentile
accumulators). Add --rank-sorted for callers who can promise sorted input
and want the previous O(1)-space streaming behavior instead.

* Revert "stats1: make rank order-independent by default, add --rank-sorted opt-in fast path"

This reverts commit aa45a591fe.

* Revert "stats1: add rank accumulator (#383)"

This reverts commit 96deed048a.

* Add mlr rank verb (#383)

Reverts the earlier stats1 -a rank / --rank-sorted approach: stats1 is a
reduce verb (many records -> one summary record per group) and rank is a
per-record annotator, so it never fit cleanly there -- it needed stats1's
-s iterative-stats escape hatch just to be useful, plus a bolted-on
sorted/unsorted split.

mlr rank is a dedicated verb modeled on mlr fraction: -f fields to rank,
-g optional group-by, output field <f>_rank. By default it's a two-pass
algorithm (buffers input, like fraction does) giving standard competition
rank (1,2,2,4,...) that's correct regardless of input order. --sorted
opts into a single-pass, O(1)-space streaming alternative for callers who
can promise pre-sorted input (e.g. via 'mlr sort' first).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* Regenerate docs/man pages after merging main (sparkline verb)

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
John Kerl 2026-07-07 17:35:18 -04:00 committed by GitHub
parent a55327c7ef
commit f9afae491f
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
36 changed files with 680 additions and 23 deletions

View file

@ -207,11 +207,11 @@
count-similar cut decimate describe fill-down fill-empty filter flatten
format-values fraction gap grep group-by group-like gsub having-fields head
histogram json-parse json-stringify join label latin1-to-utf8 least-frequent
merge-fields most-frequent nest nothing put regularize remove-empty-columns
rename reorder repeat reshape sample sec2gmtdate sec2gmt seqgen shuffle
skip-trivial-records sort sort-within-records sparkline sparsify split ssub
stats1 stats2 step sub summary surv tac tail tee template top utf8-to-latin1
unflatten uniq unspace unsparsify
merge-fields most-frequent nest nothing put rank regularize
remove-empty-columns rename reorder repeat reshape sample sec2gmtdate sec2gmt
seqgen shuffle skip-trivial-records sort sort-within-records sparkline
sparsify split ssub stats1 stats2 step sub summary surv tac tail tee template
top utf8-to-latin1 unflatten uniq unspace unsparsify
1mFUNCTION LIST0m
abs acos acosh antimode any append apply arrayify asin asinh asserting_absent
@ -1791,6 +1791,34 @@
See also https://miller.readthedocs.io/reference-dsl for more context.
1mrank0m
Usage: mlr rank [options]
For each record's value in specified fields, computes the standard
competition rank (1,2,2,4,...) of that value among all input records,
optionally within groups.
E.g. with input records x=10, x=20, x=20, and x=30, emits output records
x=10,x_rank=1 x=20,x_rank=2 x=20,x_rank=2 and x=30,x_rank=4.
Note: by default this is a two-pass algorithm: on the first pass it retains
input records and their values; on the second pass it computes ranks and
emits output records, in original input order. This means it produces no
output until all input is read, but gives correct ranks regardless of input
order. Use --sorted for a single-pass streaming alternative.
Options:
-f {a,b,c} Field name(s) to rank.
-g {d,e,f} Optional group-by-field name(s).
--sorted Promise that the input is already sorted by the field(s) being ranked
(within each group, if -g is given). This computes rank in a single
streaming pass and O(1) space, by comparing each record's value only
to the immediately preceding one, rather than buffering all records
to compute an order-independent rank. Produces wrong output if the
input is not in fact sorted.
-h|--help Show this message.
Example: mlr rank -f x data/rank-example.csv
Example: mlr rank -f x -g g data/rank-example.csv
Example: mlr sort -f x then rank -f x --sorted data/rank-example.csv
1mregularize0m
Usage: mlr regularize [options]
Outputs records sorted lexically ascending by keys.

View file

@ -249,11 +249,11 @@ altkv bar bootstrap case cat check clean-whitespace count-distinct count
count-similar cut decimate describe fill-down fill-empty filter flatten
format-values fraction gap grep group-by group-like gsub having-fields head
histogram json-parse json-stringify join label latin1-to-utf8 least-frequent
merge-fields most-frequent nest nothing put regularize remove-empty-columns
rename reorder repeat reshape sample sec2gmtdate sec2gmt seqgen shuffle
skip-trivial-records sort sort-within-records sparkline sparsify split ssub
stats1 stats2 step sub summary surv tac tail tee template top utf8-to-latin1
unflatten uniq unspace unsparsify
merge-fields most-frequent nest nothing put rank regularize
remove-empty-columns rename reorder repeat reshape sample sec2gmtdate sec2gmt
seqgen shuffle skip-trivial-records sort sort-within-records sparkline
sparsify split ssub stats1 stats2 step sub summary surv tac tail tee template
top utf8-to-latin1 unflatten uniq unspace unsparsify
.fi
.if n \{\
.RE
@ -2209,6 +2209,40 @@ See also https://miller.readthedocs.io/reference-dsl for more context.
.fi
.if n \{\
.RE
.SS "rank"
.if n \{\
.RS 0
.\}
.nf
Usage: mlr rank [options]
For each record's value in specified fields, computes the standard
competition rank (1,2,2,4,...) of that value among all input records,
optionally within groups.
E.g. with input records x=10, x=20, x=20, and x=30, emits output records
x=10,x_rank=1 x=20,x_rank=2 x=20,x_rank=2 and x=30,x_rank=4.
Note: by default this is a two-pass algorithm: on the first pass it retains
input records and their values; on the second pass it computes ranks and
emits output records, in original input order. This means it produces no
output until all input is read, but gives correct ranks regardless of input
order. Use --sorted for a single-pass streaming alternative.
Options:
-f {a,b,c} Field name(s) to rank.
-g {d,e,f} Optional group-by-field name(s).
--sorted Promise that the input is already sorted by the field(s) being ranked
(within each group, if -g is given). This computes rank in a single
streaming pass and O(1) space, by comparing each record's value only
to the immediately preceding one, rather than buffering all records
to compute an order-independent rank. Produces wrong output if the
input is not in fact sorted.
-h|--help Show this message.
Example: mlr rank -f x data/rank-example.csv
Example: mlr rank -f x -g g data/rank-example.csv
Example: mlr sort -f x then rank -f x --sorted data/rank-example.csv
.fi
.if n \{\
.RE
.SS "regularize"
.if n \{\
.RS 0