Add mlr rank verb (#2178)

* stats1: add rank accumulator (#383)

Adds `mlr stats1 -a rank` for standard competition ranking (1,2,2,4,...)
on pre-sorted data, most useful with -s for a rank on every record.

* stats1: make rank order-independent by default, add --rank-sorted opt-in fast path

The rank accumulator previously only compared each value to the immediately
preceding record, silently giving wrong ranks for non-adjacent duplicates
(e.g. unsorted input, or interleaved -g groups). Default to correctly
computing standard competition rank from all values seen so far
(order-independent, buffers values, same approach as percentile
accumulators). Add --rank-sorted for callers who can promise sorted input
and want the previous O(1)-space streaming behavior instead.

* Revert "stats1: make rank order-independent by default, add --rank-sorted opt-in fast path"

This reverts commit aa45a591fe.

* Revert "stats1: add rank accumulator (#383)"

This reverts commit 96deed048a.

* Add mlr rank verb (#383)

Reverts the earlier stats1 -a rank / --rank-sorted approach: stats1 is a
reduce verb (many records -> one summary record per group) and rank is a
per-record annotator, so it never fit cleanly there -- it needed stats1's
-s iterative-stats escape hatch just to be useful, plus a bolted-on
sorted/unsorted split.

mlr rank is a dedicated verb modeled on mlr fraction: -f fields to rank,
-g optional group-by, output field <f>_rank. By default it's a two-pass
algorithm (buffers input, like fraction does) giving standard competition
rank (1,2,2,4,...) that's correct regardless of input order. --sorted
opts into a single-pass, O(1)-space streaming alternative for callers who
can promise pre-sorted input (e.g. via 'mlr sort' first).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* Regenerate docs/man pages after merging main (sparkline verb)

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

docs/src/ai-support.md

@@ -64,7 +64,7 @@ one-line summary (here trimmed, and then counted, using Miller itself):
 <pre class="pre-non-highlight-in-pair">
 [
 {
-  "count": 664
+  "count": 665
 }
 ]
 </pre>

docs/src/data/rank-example.csv

@@ -0,0 +1,8 @@
+g,x
+a,10
+a,20
+a,20
+a,30
+b,5
+b,5
+b,9

docs/src/manpage.md

@@ -228,11 +228,11 @@ This is simply a copy of what you should see on running `man mlr` at a command p
        count-similar cut decimate describe fill-down fill-empty filter flatten
        format-values fraction gap grep group-by group-like gsub having-fields head
        histogram json-parse json-stringify join label latin1-to-utf8 least-frequent
-       merge-fields most-frequent nest nothing put regularize remove-empty-columns
-       rename reorder repeat reshape sample sec2gmtdate sec2gmt seqgen shuffle
-       skip-trivial-records sort sort-within-records sparsify split ssub stats1
-       stats2 step sub summary surv tac tail tee template top utf8-to-latin1
-       unflatten uniq unspace unsparsify
+       merge-fields most-frequent nest nothing put rank regularize
+       remove-empty-columns rename reorder repeat reshape sample sec2gmtdate sec2gmt
+       seqgen shuffle skip-trivial-records sort sort-within-records sparkline
+       sparsify split ssub stats1 stats2 step sub summary surv tac tail tee template
+       top utf8-to-latin1 unflatten uniq unspace unsparsify
 
 1mFUNCTION LIST0m
        abs acos acosh antimode any append apply arrayify asin asinh asserting_absent
@@ -1812,6 +1812,34 @@ This is simply a copy of what you should see on running `man mlr` at a command p
 
        See also https://miller.readthedocs.io/reference-dsl for more context.
 
+   1mrank0m
+       Usage: mlr rank [options]
+       For each record's value in specified fields, computes the standard
+       competition rank (1,2,2,4,...) of that value among all input records,
+       optionally within groups.
+       E.g. with input records x=10, x=20, x=20, and x=30, emits output records
+       x=10,x_rank=1  x=20,x_rank=2  x=20,x_rank=2  and  x=30,x_rank=4.
+
+       Note: by default this is a two-pass algorithm: on the first pass it retains
+       input records and their values; on the second pass it computes ranks and
+       emits output records, in original input order. This means it produces no
+       output until all input is read, but gives correct ranks regardless of input
+       order. Use --sorted for a single-pass streaming alternative.
+
+       Options:
+       -f {a,b,c} Field name(s) to rank.
+       -g {d,e,f} Optional group-by-field name(s).
+       --sorted   Promise that the input is already sorted by the field(s) being ranked
+                  (within each group, if -g is given). This computes rank in a single
+                  streaming pass and O(1) space, by comparing each record's value only
+                  to the immediately preceding one, rather than buffering all records
+                  to compute an order-independent rank. Produces wrong output if the
+                  input is not in fact sorted.
+       -h|--help  Show this message.
+       Example: mlr rank -f x data/rank-example.csv
+       Example: mlr rank -f x -g g data/rank-example.csv
+       Example: mlr sort -f x then rank -f x --sorted data/rank-example.csv
+
    1mregularize0m
        Usage: mlr regularize [options]
        Outputs records sorted lexically ascending by keys.
@@ -2087,6 +2115,16 @@ This is simply a copy of what you should see on running `man mlr` at a command p
        -n         Sort field names naturally (e.g. 2 before 12). Combines with -f/-r.
        -h|--help  Show this message.
 
+   1msparkline0m
+       Usage: mlr sparkline [options]
+       Reduces numeric field(s), across all records in input order, to a compact
+       Unicode sparkline -- one block character per record -- for visualizing
+       trends. Emits one output record per field. Holds all records in memory
+       before producing any output.
+       Options:
+       -f {a,b,c} Field names to sparkline.
+       -h|--help  Show this message.
+
    1msparsify0m
        Usage: mlr sparsify [options]
        Unsets fields for which the key is the empty string (or, optionally, another

docs/src/manpage.txt

@@ -207,11 +207,11 @@
        count-similar cut decimate describe fill-down fill-empty filter flatten
        format-values fraction gap grep group-by group-like gsub having-fields head
        histogram json-parse json-stringify join label latin1-to-utf8 least-frequent
-       merge-fields most-frequent nest nothing put regularize remove-empty-columns
-       rename reorder repeat reshape sample sec2gmtdate sec2gmt seqgen shuffle
-       skip-trivial-records sort sort-within-records sparkline sparsify split ssub
-       stats1 stats2 step sub summary surv tac tail tee template top utf8-to-latin1
-       unflatten uniq unspace unsparsify
+       merge-fields most-frequent nest nothing put rank regularize
+       remove-empty-columns rename reorder repeat reshape sample sec2gmtdate sec2gmt
+       seqgen shuffle skip-trivial-records sort sort-within-records sparkline
+       sparsify split ssub stats1 stats2 step sub summary surv tac tail tee template
+       top utf8-to-latin1 unflatten uniq unspace unsparsify
 
 1mFUNCTION LIST0m
        abs acos acosh antimode any append apply arrayify asin asinh asserting_absent
@@ -1791,6 +1791,34 @@
 
        See also https://miller.readthedocs.io/reference-dsl for more context.
 
+   1mrank0m
+       Usage: mlr rank [options]
+       For each record's value in specified fields, computes the standard
+       competition rank (1,2,2,4,...) of that value among all input records,
+       optionally within groups.
+       E.g. with input records x=10, x=20, x=20, and x=30, emits output records
+       x=10,x_rank=1  x=20,x_rank=2  x=20,x_rank=2  and  x=30,x_rank=4.
+
+       Note: by default this is a two-pass algorithm: on the first pass it retains
+       input records and their values; on the second pass it computes ranks and
+       emits output records, in original input order. This means it produces no
+       output until all input is read, but gives correct ranks regardless of input
+       order. Use --sorted for a single-pass streaming alternative.
+
+       Options:
+       -f {a,b,c} Field name(s) to rank.
+       -g {d,e,f} Optional group-by-field name(s).
+       --sorted   Promise that the input is already sorted by the field(s) being ranked
+                  (within each group, if -g is given). This computes rank in a single
+                  streaming pass and O(1) space, by comparing each record's value only
+                  to the immediately preceding one, rather than buffering all records
+                  to compute an order-independent rank. Produces wrong output if the
+                  input is not in fact sorted.
+       -h|--help  Show this message.
+       Example: mlr rank -f x data/rank-example.csv
+       Example: mlr rank -f x -g g data/rank-example.csv
+       Example: mlr sort -f x then rank -f x --sorted data/rank-example.csv
+
    1mregularize0m
        Usage: mlr regularize [options]
        Outputs records sorted lexically ascending by keys.

docs/src/reference-verbs.md

@@ -54,7 +54,7 @@ These fall into categories as follows:
 
 * `awk`-like functionality: [filter](reference-verbs.md#filter), [put](reference-verbs.md#put), [sec2gmt](reference-verbs.md#sec2gmt), [sec2gmtdate](reference-verbs.md#sec2gmtdate), [step](reference-verbs.md#step), [tee](reference-verbs.md#tee).
 
-* Statistically oriented: [bar](reference-verbs.md#bar), [bootstrap](reference-verbs.md#bootstrap), [decimate](reference-verbs.md#decimate), [histogram](reference-verbs.md#histogram), [least-frequent](reference-verbs.md#least-frequent), [most-frequent](reference-verbs.md#most-frequent), [sample](reference-verbs.md#sample), [shuffle](reference-verbs.md#shuffle), [sparkline](reference-verbs.md#sparkline), [stats1](reference-verbs.md#stats1), [stats2](reference-verbs.md#stats2).
+* Statistically oriented: [bar](reference-verbs.md#bar), [bootstrap](reference-verbs.md#bootstrap), [decimate](reference-verbs.md#decimate), [histogram](reference-verbs.md#histogram), [least-frequent](reference-verbs.md#least-frequent), [most-frequent](reference-verbs.md#most-frequent), [rank](reference-verbs.md#rank), [sample](reference-verbs.md#sample), [shuffle](reference-verbs.md#shuffle), [sparkline](reference-verbs.md#sparkline), [stats1](reference-verbs.md#stats1), [stats2](reference-verbs.md#stats2).
 
 * Particularly oriented toward [Record Heterogeneity](record-heterogeneity.md), although all Miller commands can handle heterogeneous records: [group-by](reference-verbs.md#group-by), [group-like](reference-verbs.md#group-like), [having-fields](reference-verbs.md#having-fields).
 
@@ -2628,6 +2628,106 @@ See also https://miller.readthedocs.io/reference-dsl for more context.
 
 Please see the [DSL reference](reference-dsl.md) for more information about the expression language for `mlr put`.
 
+## rank
+
+<pre class="pre-highlight-in-pair">
+<b>mlr rank --help</b>
+</pre>
+<pre class="pre-non-highlight-in-pair">
+Usage: mlr rank [options]
+For each record's value in specified fields, computes the standard
+competition rank (1,2,2,4,...) of that value among all input records,
+optionally within groups.
+E.g. with input records x=10, x=20, x=20, and x=30, emits output records
+x=10,x_rank=1  x=20,x_rank=2  x=20,x_rank=2  and  x=30,x_rank=4.
+
+Note: by default this is a two-pass algorithm: on the first pass it retains
+input records and their values; on the second pass it computes ranks and
+emits output records, in original input order. This means it produces no
+output until all input is read, but gives correct ranks regardless of input
+order. Use --sorted for a single-pass streaming alternative.
+
+Options:
+-f {a,b,c} Field name(s) to rank.
+-g {d,e,f} Optional group-by-field name(s).
+--sorted   Promise that the input is already sorted by the field(s) being ranked
+           (within each group, if -g is given). This computes rank in a single
+           streaming pass and O(1) space, by comparing each record's value only
+           to the immediately preceding one, rather than buffering all records
+           to compute an order-independent rank. Produces wrong output if the
+           input is not in fact sorted.
+-h|--help  Show this message.
+Example: mlr rank -f x data/rank-example.csv
+Example: mlr rank -f x -g g data/rank-example.csv
+Example: mlr sort -f x then rank -f x --sorted data/rank-example.csv
+</pre>
+
+For example, suppose you have the following CSV file:
+
+<pre class="pre-non-highlight-non-pair">
+g,x
+a,10
+a,20
+a,20
+a,30
+b,5
+b,5
+b,9
+</pre>
+
+Then we can rank each record's `x` among all records:
+
+<pre class="pre-highlight-in-pair">
+<b>mlr --icsv --opprint rank -f x data/rank-example.csv</b>
+</pre>
+<pre class="pre-non-highlight-in-pair">
+g x  x_rank
+a 10 4
+a 20 5
+a 20 5
+a 30 7
+b 5  1
+b 5  1
+b 9  3
+</pre>
+
+Using `-g` we can rank within each group instead:
+
+<pre class="pre-highlight-in-pair">
+<b>mlr --icsv --opprint rank -f x -g g data/rank-example.csv</b>
+</pre>
+<pre class="pre-non-highlight-in-pair">
+g x  x_rank
+a 10 1
+a 20 2
+a 20 2
+a 30 4
+b 5  1
+b 5  1
+b 9  3
+</pre>
+
+By default `rank` is a two-pass algorithm: it retains all input records so that it can compute
+ranks which don't depend on input order, even when same-valued records aren't adjacent to one
+another. If you know your input is already sorted on the field(s) you're ranking by -- e.g. by
+piping through `mlr sort` first -- then `--sorted` computes ranks in a single streaming pass
+and O(1) space instead, by comparing each record only to the one immediately before it (within
+its group, if `-g` is given):
+
+<pre class="pre-highlight-in-pair">
+<b>mlr --icsv --opprint sort -f g -nf x then rank -f x -g g --sorted data/rank-example.csv</b>
+</pre>
+<pre class="pre-non-highlight-in-pair">
+g x  x_rank
+a 10 1
+a 20 2
+a 20 2
+a 30 4
+b 5  1
+b 5  1
+b 9  3
+</pre>
+
 ## regularize
 
 <pre class="pre-highlight-in-pair">

docs/src/reference-verbs.md.in

@@ -27,7 +27,7 @@ These fall into categories as follows:
 
 * `awk`-like functionality: [filter](reference-verbs.md#filter), [put](reference-verbs.md#put), [sec2gmt](reference-verbs.md#sec2gmt), [sec2gmtdate](reference-verbs.md#sec2gmtdate), [step](reference-verbs.md#step), [tee](reference-verbs.md#tee).
 
-* Statistically oriented: [bar](reference-verbs.md#bar), [bootstrap](reference-verbs.md#bootstrap), [decimate](reference-verbs.md#decimate), [histogram](reference-verbs.md#histogram), [least-frequent](reference-verbs.md#least-frequent), [most-frequent](reference-verbs.md#most-frequent), [sample](reference-verbs.md#sample), [shuffle](reference-verbs.md#shuffle), [sparkline](reference-verbs.md#sparkline), [stats1](reference-verbs.md#stats1), [stats2](reference-verbs.md#stats2).
+* Statistically oriented: [bar](reference-verbs.md#bar), [bootstrap](reference-verbs.md#bootstrap), [decimate](reference-verbs.md#decimate), [histogram](reference-verbs.md#histogram), [least-frequent](reference-verbs.md#least-frequent), [most-frequent](reference-verbs.md#most-frequent), [rank](reference-verbs.md#rank), [sample](reference-verbs.md#sample), [shuffle](reference-verbs.md#shuffle), [sparkline](reference-verbs.md#sparkline), [stats1](reference-verbs.md#stats1), [stats2](reference-verbs.md#stats2).
 
 * Particularly oriented toward [Record Heterogeneity](record-heterogeneity.md), although all Miller commands can handle heterogeneous records: [group-by](reference-verbs.md#group-by), [group-like](reference-verbs.md#group-like), [having-fields](reference-verbs.md#having-fields).
 
@@ -796,6 +796,39 @@ GENMD-EOF
 
 Please see the [DSL reference](reference-dsl.md) for more information about the expression language for `mlr put`.
 
+## rank
+
+GENMD-RUN-COMMAND
+mlr rank --help
+GENMD-EOF
+
+For example, suppose you have the following CSV file:
+
+GENMD-INCLUDE-ESCAPED(data/rank-example.csv)
+
+Then we can rank each record's `x` among all records:
+
+GENMD-RUN-COMMAND
+mlr --icsv --opprint rank -f x data/rank-example.csv
+GENMD-EOF
+
+Using `-g` we can rank within each group instead:
+
+GENMD-RUN-COMMAND
+mlr --icsv --opprint rank -f x -g g data/rank-example.csv
+GENMD-EOF
+
+By default `rank` is a two-pass algorithm: it retains all input records so that it can compute
+ranks which don't depend on input order, even when same-valued records aren't adjacent to one
+another. If you know your input is already sorted on the field(s) you're ranking by -- e.g. by
+piping through `mlr sort` first -- then `--sorted` computes ranks in a single streaming pass
+and O(1) space instead, by comparing each record only to the one immediately before it (within
+its group, if `-g` is given):
+
+GENMD-RUN-COMMAND
+mlr --icsv --opprint sort -f g -nf x then rank -f x -g g --sorted data/rank-example.csv
+GENMD-EOF
+
 ## regularize
 
 GENMD-RUN-COMMAND

docs/src/streaming-and-memory.md

@@ -100,6 +100,7 @@ They are memory-friendly, and they don't wait for end of input to produce their
 * [merge-fields](reference-verbs.md#merge-fields)
 * [nest](reference-verbs.md#nest) -- if not `implode-values-across-records`
 * [nothing](reference-verbs.md#nothing)
+* [rank](reference-verbs.md#rank) -- if `--sorted`
 * [regularize](reference-verbs.md#regularize)
 * [rename](reference-verbs.md#rename)
 * [reorder](reference-verbs.md#reorder)
@@ -131,6 +132,7 @@ They are memory-unfriendly, and they wait for end of input to produce their outp
 * [least-frequent](reference-verbs.md#least-frequent)
 * [most-frequent](reference-verbs.md#most-frequent)
 * [nest](reference-verbs.md#nest) -- if `implode-values-across-records`
+* [rank](reference-verbs.md#rank) -- if not `--sorted`
 * [remove-empty-columns](reference-verbs.md#remove-empty-columns)
 * [reshape](reference-verbs.md#reshape) -- if long-to-wide
 * [sample](reference-verbs.md#sample)

docs/src/streaming-and-memory.md.in

@@ -84,6 +84,7 @@ They are memory-friendly, and they don't wait for end of input to produce their
 * [merge-fields](reference-verbs.md#merge-fields)
 * [nest](reference-verbs.md#nest) -- if not `implode-values-across-records`
 * [nothing](reference-verbs.md#nothing)
+* [rank](reference-verbs.md#rank) -- if `--sorted`
 * [regularize](reference-verbs.md#regularize)
 * [rename](reference-verbs.md#rename)
 * [reorder](reference-verbs.md#reorder)
@@ -115,6 +116,7 @@ They are memory-unfriendly, and they wait for end of input to produce their outp
 * [least-frequent](reference-verbs.md#least-frequent)
 * [most-frequent](reference-verbs.md#most-frequent)
 * [nest](reference-verbs.md#nest) -- if `implode-values-across-records`
+* [rank](reference-verbs.md#rank) -- if not `--sorted`
 * [remove-empty-columns](reference-verbs.md#remove-empty-columns)
 * [reshape](reference-verbs.md#reshape) -- if long-to-wide
 * [sample](reference-verbs.md#sample)

man/manpage.txt

@@ -207,11 +207,11 @@
        count-similar cut decimate describe fill-down fill-empty filter flatten
        format-values fraction gap grep group-by group-like gsub having-fields head
        histogram json-parse json-stringify join label latin1-to-utf8 least-frequent
-       merge-fields most-frequent nest nothing put regularize remove-empty-columns
-       rename reorder repeat reshape sample sec2gmtdate sec2gmt seqgen shuffle
-       skip-trivial-records sort sort-within-records sparkline sparsify split ssub
-       stats1 stats2 step sub summary surv tac tail tee template top utf8-to-latin1
-       unflatten uniq unspace unsparsify
+       merge-fields most-frequent nest nothing put rank regularize
+       remove-empty-columns rename reorder repeat reshape sample sec2gmtdate sec2gmt
+       seqgen shuffle skip-trivial-records sort sort-within-records sparkline
+       sparsify split ssub stats1 stats2 step sub summary surv tac tail tee template
+       top utf8-to-latin1 unflatten uniq unspace unsparsify
 
 1mFUNCTION LIST0m
        abs acos acosh antimode any append apply arrayify asin asinh asserting_absent
@@ -1791,6 +1791,34 @@
 
        See also https://miller.readthedocs.io/reference-dsl for more context.
 
+   1mrank0m
+       Usage: mlr rank [options]
+       For each record's value in specified fields, computes the standard
+       competition rank (1,2,2,4,...) of that value among all input records,
+       optionally within groups.
+       E.g. with input records x=10, x=20, x=20, and x=30, emits output records
+       x=10,x_rank=1  x=20,x_rank=2  x=20,x_rank=2  and  x=30,x_rank=4.
+
+       Note: by default this is a two-pass algorithm: on the first pass it retains
+       input records and their values; on the second pass it computes ranks and
+       emits output records, in original input order. This means it produces no
+       output until all input is read, but gives correct ranks regardless of input
+       order. Use --sorted for a single-pass streaming alternative.
+
+       Options:
+       -f {a,b,c} Field name(s) to rank.
+       -g {d,e,f} Optional group-by-field name(s).
+       --sorted   Promise that the input is already sorted by the field(s) being ranked
+                  (within each group, if -g is given). This computes rank in a single
+                  streaming pass and O(1) space, by comparing each record's value only
+                  to the immediately preceding one, rather than buffering all records
+                  to compute an order-independent rank. Produces wrong output if the
+                  input is not in fact sorted.
+       -h|--help  Show this message.
+       Example: mlr rank -f x data/rank-example.csv
+       Example: mlr rank -f x -g g data/rank-example.csv
+       Example: mlr sort -f x then rank -f x --sorted data/rank-example.csv
+
    1mregularize0m
        Usage: mlr regularize [options]
        Outputs records sorted lexically ascending by keys.

man/mlr.1

@@ -249,11 +249,11 @@ altkv bar bootstrap case cat check clean-whitespace count-distinct count
 count-similar cut decimate describe fill-down fill-empty filter flatten
 format-values fraction gap grep group-by group-like gsub having-fields head
 histogram json-parse json-stringify join label latin1-to-utf8 least-frequent
-merge-fields most-frequent nest nothing put regularize remove-empty-columns
-rename reorder repeat reshape sample sec2gmtdate sec2gmt seqgen shuffle
-skip-trivial-records sort sort-within-records sparkline sparsify split ssub
-stats1 stats2 step sub summary surv tac tail tee template top utf8-to-latin1
-unflatten uniq unspace unsparsify
+merge-fields most-frequent nest nothing put rank regularize
+remove-empty-columns rename reorder repeat reshape sample sec2gmtdate sec2gmt
+seqgen shuffle skip-trivial-records sort sort-within-records sparkline
+sparsify split ssub stats1 stats2 step sub summary surv tac tail tee template
+top utf8-to-latin1 unflatten uniq unspace unsparsify
 .fi
 .if n \{\
 .RE
@@ -2209,6 +2209,40 @@ See also https://miller.readthedocs.io/reference-dsl for more context.
 .fi
 .if n \{\
 .RE
+.SS "rank"
+.if n \{\
+.RS 0
+.\}
+.nf
+Usage: mlr rank [options]
+For each record's value in specified fields, computes the standard
+competition rank (1,2,2,4,...) of that value among all input records,
+optionally within groups.
+E.g. with input records x=10, x=20, x=20, and x=30, emits output records
+x=10,x_rank=1  x=20,x_rank=2  x=20,x_rank=2  and  x=30,x_rank=4.
+
+Note: by default this is a two-pass algorithm: on the first pass it retains
+input records and their values; on the second pass it computes ranks and
+emits output records, in original input order. This means it produces no
+output until all input is read, but gives correct ranks regardless of input
+order. Use --sorted for a single-pass streaming alternative.
+
+Options:
+-f {a,b,c} Field name(s) to rank.
+-g {d,e,f} Optional group-by-field name(s).
+--sorted   Promise that the input is already sorted by the field(s) being ranked
+           (within each group, if -g is given). This computes rank in a single
+           streaming pass and O(1) space, by comparing each record's value only
+           to the immediately preceding one, rather than buffering all records
+           to compute an order-independent rank. Produces wrong output if the
+           input is not in fact sorted.
+-h|--help  Show this message.
+Example: mlr rank -f x data/rank-example.csv
+Example: mlr rank -f x -g g data/rank-example.csv
+Example: mlr sort -f x then rank -f x --sorted data/rank-example.csv
+.fi
+.if n \{\
+.RE
 .SS "regularize"
 .if n \{\
 .RS 0

pkg/transformers/aaa_transformer_table.go

@@ -48,6 +48,7 @@ var TRANSFORMER_LOOKUP_TABLE = []TransformerSetup{
 	NestSetup,
 	NothingSetup,
 	PutSetup,
+	RankSetup,
 	RegularizeSetup,
 	RemoveEmptyColumnsSetup,
 	RenameSetup,

pkg/transformers/rank.go

@@ -0,0 +1,284 @@
+package transformers
+
+import (
+	"fmt"
+	"os"
+	"strings"
+
+	"github.com/johnkerl/miller/v6/pkg/cli"
+	"github.com/johnkerl/miller/v6/pkg/mlrval"
+	"github.com/johnkerl/miller/v6/pkg/transformers/utils"
+	"github.com/johnkerl/miller/v6/pkg/types"
+)
+
+const verbNameRank = "rank"
+
+var rankOptions = []OptionSpec{
+	{Flag: "-f", Arg: "{a,b,c}", Type: "csv-list", Desc: "Field name(s) to rank."},
+	{Flag: "-g", Arg: "{d,e,f}", Type: "csv-list", Desc: "Optional group-by-field name(s)."},
+	{Flag: "--sorted", Type: "bool", Desc: "Promise that the input is already sorted by the field(s) being ranked (within each group, if -g is given). This computes rank in a single streaming pass and O(1) space, by comparing each record's value only to the immediately preceding one, rather than buffering all records to compute an order-independent rank. Produces wrong output if the input is not in fact sorted."},
+}
+
+var RankSetup = TransformerSetup{
+	Verb:         verbNameRank,
+	UsageFunc:    transformerRankUsage,
+	ParseCLIFunc: transformerRankParseCLI,
+	IgnoresInput: false,
+	Options:      rankOptions,
+}
+
+func transformerRankUsage(
+	o *os.File,
+) {
+	argv0 := "mlr"
+	verb := verbNameRank
+	fmt.Fprintf(o, "Usage: %s %s [options]\n", argv0, verb)
+	fmt.Fprintf(o, "For each record's value in specified fields, computes the standard\n")
+	fmt.Fprintf(o, "competition rank (1,2,2,4,...) of that value among all input records,\n")
+	fmt.Fprintf(o, "optionally within groups.\n")
+	fmt.Fprintf(o, "E.g. with input records x=10, x=20, x=20, and x=30, emits output records\n")
+	fmt.Fprintf(o, "x=10,x_rank=1  x=20,x_rank=2  x=20,x_rank=2  and  x=30,x_rank=4.\n")
+	fmt.Fprintf(o, "\n")
+	fmt.Fprintf(o, "Note: by default this is a two-pass algorithm: on the first pass it retains\n")
+	fmt.Fprintf(o, "input records and their values; on the second pass it computes ranks and\n")
+	fmt.Fprintf(o, "emits output records, in original input order. This means it produces no\n")
+	fmt.Fprintf(o, "output until all input is read, but gives correct ranks regardless of input\n")
+	fmt.Fprintf(o, "order. Use --sorted for a single-pass streaming alternative.\n")
+	fmt.Fprintf(o, "\n")
+	WriteVerbOptions(o, rankOptions)
+	fmt.Fprintln(o, "Example: mlr rank -f x data/rank-example.csv")
+	fmt.Fprintln(o, "Example: mlr rank -f x -g g data/rank-example.csv")
+	fmt.Fprintln(o, "Example: mlr sort -f x then rank -f x --sorted data/rank-example.csv")
+}
+
+func transformerRankParseCLI(
+	pargi *int,
+	argc int,
+	args []string,
+	_ *cli.TOptions,
+	doConstruct bool, // false for first pass of CLI-parse, true for second pass
+) (RecordTransformer, error) {
+
+	// Skip the verb name from the current spot in the mlr command line
+	argi := *pargi
+	verb := args[argi]
+	argi++
+
+	var rankFieldNames []string = nil
+	var groupByFieldNames []string = nil
+	doSorted := false
+
+	var err error
+	for argi < argc /* variable increment: 1 or 2 depending on flag */ {
+		opt := args[argi]
+		if !strings.HasPrefix(opt, "-") {
+			break // No more flag options to process
+		}
+		if args[argi] == "--" {
+			break // All transformers must do this so main-flags can follow verb-flags
+		}
+		argi++
+
+		switch opt {
+		case "-h", "--help":
+			transformerRankUsage(os.Stdout)
+			return nil, cli.ErrHelpRequested
+
+		case "-f":
+			rankFieldNames, err = cli.VerbGetStringArrayArg(verb, opt, args, &argi, argc)
+			if err != nil {
+				return nil, err
+			}
+
+		case "-g":
+			groupByFieldNames, err = cli.VerbGetStringArrayArg(verb, opt, args, &argi, argc)
+			if err != nil {
+				return nil, err
+			}
+
+		case "--sorted":
+			doSorted = true
+
+		default:
+			return nil, cli.VerbErrorf(verb, "option \"%s\" not recognized", opt)
+		}
+	}
+
+	if rankFieldNames == nil {
+		return nil, cli.VerbErrorf(verb, "-f field names required")
+	}
+
+	*pargi = argi
+	if !doConstruct { // All transformers must do this for main command-line parsing
+		return nil, nil
+	}
+
+	transformer, err := NewTransformerRank(
+		rankFieldNames,
+		groupByFieldNames,
+		doSorted,
+	)
+	if err != nil {
+		return nil, err
+	}
+
+	return transformer, nil
+}
+
+type TransformerRank struct {
+	rankFieldNames    []string
+	groupByFieldNames []string
+	doSorted          bool
+
+	// Default (unsorted) mode: two-pass. Records are retained on the first
+	// pass, along with per-group-per-field percentile-keepers; on the
+	// second pass (end of stream) the retained records are decorated with
+	// rank fields and emitted in original input order.
+	recordsAndContexts []*types.RecordAndContext
+	keepers            map[string]map[string]*utils.PercentileKeeper // grouping-key -> field-name -> keeper
+
+	// --sorted mode: single streaming pass, O(1) space. Same shape as the
+	// keepers map above, but holding lightweight adjacency state instead of
+	// buffered/sorted values.
+	sortedStates map[string]map[string]*tRankSortedFieldState // grouping-key -> field-name -> state
+}
+
+type tRankSortedFieldState struct {
+	count               int64
+	rank                int64
+	havePreviousValue   bool
+	previousValueString string
+}
+
+func NewTransformerRank(
+	rankFieldNames []string,
+	groupByFieldNames []string,
+	doSorted bool,
+) (*TransformerRank, error) {
+	return &TransformerRank{
+		rankFieldNames:     rankFieldNames,
+		groupByFieldNames:  groupByFieldNames,
+		doSorted:           doSorted,
+		recordsAndContexts: []*types.RecordAndContext{},
+		keepers:            make(map[string]map[string]*utils.PercentileKeeper),
+		sortedStates:       make(map[string]map[string]*tRankSortedFieldState),
+	}, nil
+}
+
+func (tr *TransformerRank) Transform(
+	inrecAndContext *types.RecordAndContext,
+	outputRecordsAndContexts *[]*types.RecordAndContext, // list of *types.RecordAndContext
+	inputDownstreamDoneChannel <-chan bool,
+	outputDownstreamDoneChannel chan<- bool,
+) {
+	HandleDefaultDownstreamDone(inputDownstreamDoneChannel, outputDownstreamDoneChannel)
+	if tr.doSorted {
+		tr.transformSorted(inrecAndContext, outputRecordsAndContexts)
+	} else {
+		tr.transformUnsorted(inrecAndContext, outputRecordsAndContexts)
+	}
+}
+
+// transformSorted computes rank in a single pass, O(1) space, by comparing
+// each record's value only to the immediately preceding one within its
+// group. This is only correct if the caller has ensured the input is
+// already sorted by the ranked field(s), e.g. via 'mlr sort'.
+func (tr *TransformerRank) transformSorted(
+	inrecAndContext *types.RecordAndContext,
+	outputRecordsAndContexts *[]*types.RecordAndContext,
+) {
+	if !inrecAndContext.EndOfStream {
+		inrec := inrecAndContext.Record
+
+		groupingKey, hasAll := inrec.GetSelectedValuesJoined(tr.groupByFieldNames)
+		if hasAll {
+			statesForGroup := tr.sortedStates[groupingKey]
+			if statesForGroup == nil {
+				statesForGroup = make(map[string]*tRankSortedFieldState)
+				tr.sortedStates[groupingKey] = statesForGroup
+			}
+			for _, rankFieldName := range tr.rankFieldNames {
+				value := inrec.Get(rankFieldName)
+				if value == nil {
+					continue
+				}
+				state := statesForGroup[rankFieldName]
+				if state == nil {
+					state = &tRankSortedFieldState{}
+					statesForGroup[rankFieldName] = state
+				}
+				state.count++
+				valueString := value.String() // 1, 1.0, and 1.000 are distinct
+				if !state.havePreviousValue || valueString != state.previousValueString {
+					state.rank = state.count
+					state.previousValueString = valueString
+					state.havePreviousValue = true
+				}
+				inrec.PutCopy(rankFieldName+"_rank", mlrval.FromInt(state.rank))
+			}
+		}
+	}
+	*outputRecordsAndContexts = append(*outputRecordsAndContexts, inrecAndContext)
+}
+
+// transformUnsorted computes order-independent standard competition rank:
+// on the first pass it retains records and accumulates per-group-per-field
+// value sets; on the second pass (end of stream) it decorates the retained
+// records with rank fields and emits them in original input order.
+func (tr *TransformerRank) transformUnsorted(
+	inrecAndContext *types.RecordAndContext,
+	outputRecordsAndContexts *[]*types.RecordAndContext,
+) {
+	if !inrecAndContext.EndOfStream { // Not end of stream; pass 1
+		inrec := inrecAndContext.Record
+
+		// Append records into a single output list (so that this verb is order-preserving).
+		tr.recordsAndContexts = append(tr.recordsAndContexts, inrecAndContext)
+
+		groupingKey, hasAll := inrec.GetSelectedValuesJoined(tr.groupByFieldNames)
+		if hasAll {
+			keepersForGroup := tr.keepers[groupingKey]
+			if keepersForGroup == nil {
+				keepersForGroup = make(map[string]*utils.PercentileKeeper)
+				tr.keepers[groupingKey] = keepersForGroup
+			}
+			for _, rankFieldName := range tr.rankFieldNames {
+				value := inrec.Get(rankFieldName)
+				if value == nil {
+					continue
+				}
+				keeper := keepersForGroup[rankFieldName]
+				if keeper == nil {
+					keeper = utils.NewPercentileKeeper(false)
+					keepersForGroup[rankFieldName] = keeper
+				}
+				keeper.Ingest(value)
+			}
+		}
+
+	} else { // End of stream; pass 2
+		// Iterate over the retained records, decorating them with rank fields.
+		endOfStreamContext := inrecAndContext.Context
+
+		for _, recordAndContext := range tr.recordsAndContexts {
+			outrec := recordAndContext.Record
+
+			groupingKey, hasAll := outrec.GetSelectedValuesJoined(tr.groupByFieldNames)
+			if hasAll {
+				keepersForGroup := tr.keepers[groupingKey]
+				for _, rankFieldName := range tr.rankFieldNames {
+					value := outrec.Get(rankFieldName)
+					if value == nil {
+						continue
+					}
+					keeper := keepersForGroup[rankFieldName]
+					outrec.PutCopy(rankFieldName+"_rank", keeper.EmitRank(value))
+				}
+			}
+
+			*outputRecordsAndContexts = append(*outputRecordsAndContexts, types.NewRecordAndContext(outrec, &endOfStreamContext))
+		}
+		tr.recordsAndContexts = tr.recordsAndContexts[:0]
+		*outputRecordsAndContexts = append(*outputRecordsAndContexts, inrecAndContext) // end-of-stream marker
+	}
+}

pkg/transformers/utils/percentile_keeper.go

@@ -82,6 +82,27 @@ func (keeper *PercentileKeeper) EmitLinearlyInterpolated(percentile float64) *ml
 	return bifs.GetPercentileLinearlyInterpolated(keeper.data, int(len(keeper.data)), percentile)
 }
 
+// EmitRank returns the standard competition rank (1,2,2,4,...) of value
+// among all values ingested so far: one plus the number of ingested values
+// strictly less than it.
+func (keeper *PercentileKeeper) EmitRank(value *mlrval.Mlrval) *mlrval.Mlrval {
+	if len(keeper.data) == 0 {
+		return mlrval.VOID
+	}
+	keeper.sortIfNecessary()
+	n := len(keeper.data)
+	lo, hi := 0, n
+	for lo < hi {
+		mid := (lo + hi) / 2
+		if mlrval.LessThan(keeper.data[mid], value) {
+			lo = mid + 1
+		} else {
+			hi = mid
+		}
+	}
+	return mlrval.FromInt(int64(lo + 1))
+}
+
 // TODO: COMMENT
 func (keeper *PercentileKeeper) EmitNamed(name string) *mlrval.Mlrval {
 	switch name {

test/cases/cli-help/0001/expout

@@ -835,6 +835,35 @@ More example put expressions:
 
 See also https://miller.readthedocs.io/reference-dsl for more context.
 
+================================================================
+rank
+Usage: mlr rank [options]
+For each record's value in specified fields, computes the standard
+competition rank (1,2,2,4,...) of that value among all input records,
+optionally within groups.
+E.g. with input records x=10, x=20, x=20, and x=30, emits output records
+x=10,x_rank=1  x=20,x_rank=2  x=20,x_rank=2  and  x=30,x_rank=4.
+
+Note: by default this is a two-pass algorithm: on the first pass it retains
+input records and their values; on the second pass it computes ranks and
+emits output records, in original input order. This means it produces no
+output until all input is read, but gives correct ranks regardless of input
+order. Use --sorted for a single-pass streaming alternative.
+
+Options:
+-f {a,b,c} Field name(s) to rank.
+-g {d,e,f} Optional group-by-field name(s).
+--sorted   Promise that the input is already sorted by the field(s) being ranked
+           (within each group, if -g is given). This computes rank in a single
+           streaming pass and O(1) space, by comparing each record's value only
+           to the immediately preceding one, rather than buffering all records
+           to compute an order-independent rank. Produces wrong output if the
+           input is not in fact sorted.
+-h|--help  Show this message.
+Example: mlr rank -f x data/rank-example.csv
+Example: mlr rank -f x -g g data/rank-example.csv
+Example: mlr sort -f x then rank -f x --sorted data/rank-example.csv
+
 ================================================================
 regularize
 Usage: mlr regularize [options]

test/cases/verb-rank/0001/cmd

@@ -0,0 +1 @@
+mlr --icsv --ocsv rank -f x test/input/rank-data.csv

test/cases/verb-rank/0001/expout

@@ -0,0 +1,5 @@
+x,x_rank
+10,1
+20,2
+20,2
+30,4

test/cases/verb-rank/0002/cmd

@@ -0,0 +1 @@
+mlr --icsv --ocsv rank -f x test/input/rank-data-unsorted.csv

test/cases/verb-rank/0002/expout

@@ -0,0 +1,5 @@
+x,x_rank
+20,2
+10,1
+20,2
+30,4

test/cases/verb-rank/0003/cmd

@@ -0,0 +1 @@
+mlr --icsv --ocsv rank -f x -g g test/input/rank-data-grouped.csv

test/cases/verb-rank/0003/expout

@@ -0,0 +1,6 @@
+g,x,x_rank
+a,20,2
+b,5,1
+a,10,1
+b,5,1
+a,20,2

test/cases/verb-rank/0004/cmd

@@ -0,0 +1 @@
+mlr --icsv --ocsv rank -f x --sorted test/input/rank-data.csv

test/cases/verb-rank/0004/expout

@@ -0,0 +1,5 @@
+x,x_rank
+10,1
+20,2
+20,2
+30,4

test/cases/verb-rank/0005/cmd

@@ -0,0 +1 @@
+mlr --icsv --ocsv sort -f x then rank -f x --sorted test/input/rank-data-unsorted.csv

test/cases/verb-rank/0005/expout

@@ -0,0 +1,5 @@
+x,x_rank
+10,1
+20,2
+20,2
+30,4

test/cases/verb-rank/0006/cmd

@@ -0,0 +1 @@
+mlr rank -g g

test/cases/verb-rank/0006/experr

@@ -0,0 +1 @@
+mlr rank: -f field names required

test/input/rank-data-grouped.csv

@@ -0,0 +1,6 @@
+g,x
+a,20
+b,5
+a,10
+b,5
+a,20

test/input/rank-data-unsorted.csv

@@ -0,0 +1,5 @@
+x
+20
+10
+20
+30

test/input/rank-data.csv

@@ -0,0 +1,5 @@
+x
+10
+20
+20
+30