diff --git a/docs/src/ai-support.md b/docs/src/ai-support.md
index 105b713c1..509e1f0f0 100644
--- a/docs/src/ai-support.md
+++ b/docs/src/ai-support.md
@@ -64,7 +64,7 @@ one-line summary (here trimmed, and then counted, using Miller itself):
[
{
- "count": 664
+ "count": 665
}
]
diff --git a/docs/src/data/rank-example.csv b/docs/src/data/rank-example.csv
new file mode 100644
index 000000000..6d976d274
--- /dev/null
+++ b/docs/src/data/rank-example.csv
@@ -0,0 +1,8 @@
+g,x
+a,10
+a,20
+a,20
+a,30
+b,5
+b,5
+b,9
diff --git a/docs/src/manpage.md b/docs/src/manpage.md
index 0f6e75a95..ad1848539 100644
--- a/docs/src/manpage.md
+++ b/docs/src/manpage.md
@@ -228,11 +228,11 @@ This is simply a copy of what you should see on running `man mlr` at a command p
count-similar cut decimate describe fill-down fill-empty filter flatten
format-values fraction gap grep group-by group-like gsub having-fields head
histogram json-parse json-stringify join label latin1-to-utf8 least-frequent
- merge-fields most-frequent nest nothing put regularize remove-empty-columns
- rename reorder repeat reshape sample sec2gmtdate sec2gmt seqgen shuffle
- skip-trivial-records sort sort-within-records sparsify split ssub stats1
- stats2 step sub summary surv tac tail tee template top utf8-to-latin1
- unflatten uniq unspace unsparsify
+ merge-fields most-frequent nest nothing put rank regularize
+ remove-empty-columns rename reorder repeat reshape sample sec2gmtdate sec2gmt
+ seqgen shuffle skip-trivial-records sort sort-within-records sparkline
+ sparsify split ssub stats1 stats2 step sub summary surv tac tail tee template
+ top utf8-to-latin1 unflatten uniq unspace unsparsify
1mFUNCTION LIST0m
abs acos acosh antimode any append apply arrayify asin asinh asserting_absent
@@ -1812,6 +1812,34 @@ This is simply a copy of what you should see on running `man mlr` at a command p
See also https://miller.readthedocs.io/reference-dsl for more context.
+ 1mrank0m
+ Usage: mlr rank [options]
+ For each record's value in specified fields, computes the standard
+ competition rank (1,2,2,4,...) of that value among all input records,
+ optionally within groups.
+ E.g. with input records x=10, x=20, x=20, and x=30, emits output records
+ x=10,x_rank=1 x=20,x_rank=2 x=20,x_rank=2 and x=30,x_rank=4.
+
+ Note: by default this is a two-pass algorithm: on the first pass it retains
+ input records and their values; on the second pass it computes ranks and
+ emits output records, in original input order. This means it produces no
+ output until all input is read, but gives correct ranks regardless of input
+ order. Use --sorted for a single-pass streaming alternative.
+
+ Options:
+ -f {a,b,c} Field name(s) to rank.
+ -g {d,e,f} Optional group-by-field name(s).
+ --sorted Promise that the input is already sorted by the field(s) being ranked
+ (within each group, if -g is given). This computes rank in a single
+ streaming pass and O(1) space, by comparing each record's value only
+ to the immediately preceding one, rather than buffering all records
+ to compute an order-independent rank. Produces wrong output if the
+ input is not in fact sorted.
+ -h|--help Show this message.
+ Example: mlr rank -f x data/rank-example.csv
+ Example: mlr rank -f x -g g data/rank-example.csv
+ Example: mlr sort -f x then rank -f x --sorted data/rank-example.csv
+
1mregularize0m
Usage: mlr regularize [options]
Outputs records sorted lexically ascending by keys.
@@ -2087,6 +2115,16 @@ This is simply a copy of what you should see on running `man mlr` at a command p
-n Sort field names naturally (e.g. 2 before 12). Combines with -f/-r.
-h|--help Show this message.
+ 1msparkline0m
+ Usage: mlr sparkline [options]
+ Reduces numeric field(s), across all records in input order, to a compact
+ Unicode sparkline -- one block character per record -- for visualizing
+ trends. Emits one output record per field. Holds all records in memory
+ before producing any output.
+ Options:
+ -f {a,b,c} Field names to sparkline.
+ -h|--help Show this message.
+
1msparsify0m
Usage: mlr sparsify [options]
Unsets fields for which the key is the empty string (or, optionally, another
diff --git a/docs/src/manpage.txt b/docs/src/manpage.txt
index 56e848bb8..ebbd0a727 100644
--- a/docs/src/manpage.txt
+++ b/docs/src/manpage.txt
@@ -207,11 +207,11 @@
count-similar cut decimate describe fill-down fill-empty filter flatten
format-values fraction gap grep group-by group-like gsub having-fields head
histogram json-parse json-stringify join label latin1-to-utf8 least-frequent
- merge-fields most-frequent nest nothing put regularize remove-empty-columns
- rename reorder repeat reshape sample sec2gmtdate sec2gmt seqgen shuffle
- skip-trivial-records sort sort-within-records sparkline sparsify split ssub
- stats1 stats2 step sub summary surv tac tail tee template top utf8-to-latin1
- unflatten uniq unspace unsparsify
+ merge-fields most-frequent nest nothing put rank regularize
+ remove-empty-columns rename reorder repeat reshape sample sec2gmtdate sec2gmt
+ seqgen shuffle skip-trivial-records sort sort-within-records sparkline
+ sparsify split ssub stats1 stats2 step sub summary surv tac tail tee template
+ top utf8-to-latin1 unflatten uniq unspace unsparsify
1mFUNCTION LIST0m
abs acos acosh antimode any append apply arrayify asin asinh asserting_absent
@@ -1791,6 +1791,34 @@
See also https://miller.readthedocs.io/reference-dsl for more context.
+ 1mrank0m
+ Usage: mlr rank [options]
+ For each record's value in specified fields, computes the standard
+ competition rank (1,2,2,4,...) of that value among all input records,
+ optionally within groups.
+ E.g. with input records x=10, x=20, x=20, and x=30, emits output records
+ x=10,x_rank=1 x=20,x_rank=2 x=20,x_rank=2 and x=30,x_rank=4.
+
+ Note: by default this is a two-pass algorithm: on the first pass it retains
+ input records and their values; on the second pass it computes ranks and
+ emits output records, in original input order. This means it produces no
+ output until all input is read, but gives correct ranks regardless of input
+ order. Use --sorted for a single-pass streaming alternative.
+
+ Options:
+ -f {a,b,c} Field name(s) to rank.
+ -g {d,e,f} Optional group-by-field name(s).
+ --sorted Promise that the input is already sorted by the field(s) being ranked
+ (within each group, if -g is given). This computes rank in a single
+ streaming pass and O(1) space, by comparing each record's value only
+ to the immediately preceding one, rather than buffering all records
+ to compute an order-independent rank. Produces wrong output if the
+ input is not in fact sorted.
+ -h|--help Show this message.
+ Example: mlr rank -f x data/rank-example.csv
+ Example: mlr rank -f x -g g data/rank-example.csv
+ Example: mlr sort -f x then rank -f x --sorted data/rank-example.csv
+
1mregularize0m
Usage: mlr regularize [options]
Outputs records sorted lexically ascending by keys.
diff --git a/docs/src/reference-verbs.md b/docs/src/reference-verbs.md
index a2fa85aee..547bb2938 100644
--- a/docs/src/reference-verbs.md
+++ b/docs/src/reference-verbs.md
@@ -54,7 +54,7 @@ These fall into categories as follows:
* `awk`-like functionality: [filter](reference-verbs.md#filter), [put](reference-verbs.md#put), [sec2gmt](reference-verbs.md#sec2gmt), [sec2gmtdate](reference-verbs.md#sec2gmtdate), [step](reference-verbs.md#step), [tee](reference-verbs.md#tee).
-* Statistically oriented: [bar](reference-verbs.md#bar), [bootstrap](reference-verbs.md#bootstrap), [decimate](reference-verbs.md#decimate), [histogram](reference-verbs.md#histogram), [least-frequent](reference-verbs.md#least-frequent), [most-frequent](reference-verbs.md#most-frequent), [sample](reference-verbs.md#sample), [shuffle](reference-verbs.md#shuffle), [sparkline](reference-verbs.md#sparkline), [stats1](reference-verbs.md#stats1), [stats2](reference-verbs.md#stats2).
+* Statistically oriented: [bar](reference-verbs.md#bar), [bootstrap](reference-verbs.md#bootstrap), [decimate](reference-verbs.md#decimate), [histogram](reference-verbs.md#histogram), [least-frequent](reference-verbs.md#least-frequent), [most-frequent](reference-verbs.md#most-frequent), [rank](reference-verbs.md#rank), [sample](reference-verbs.md#sample), [shuffle](reference-verbs.md#shuffle), [sparkline](reference-verbs.md#sparkline), [stats1](reference-verbs.md#stats1), [stats2](reference-verbs.md#stats2).
* Particularly oriented toward [Record Heterogeneity](record-heterogeneity.md), although all Miller commands can handle heterogeneous records: [group-by](reference-verbs.md#group-by), [group-like](reference-verbs.md#group-like), [having-fields](reference-verbs.md#having-fields).
@@ -2628,6 +2628,106 @@ See also https://miller.readthedocs.io/reference-dsl for more context.
Please see the [DSL reference](reference-dsl.md) for more information about the expression language for `mlr put`.
+## rank
+
+
+mlr rank --help
+
+
+Usage: mlr rank [options]
+For each record's value in specified fields, computes the standard
+competition rank (1,2,2,4,...) of that value among all input records,
+optionally within groups.
+E.g. with input records x=10, x=20, x=20, and x=30, emits output records
+x=10,x_rank=1 x=20,x_rank=2 x=20,x_rank=2 and x=30,x_rank=4.
+
+Note: by default this is a two-pass algorithm: on the first pass it retains
+input records and their values; on the second pass it computes ranks and
+emits output records, in original input order. This means it produces no
+output until all input is read, but gives correct ranks regardless of input
+order. Use --sorted for a single-pass streaming alternative.
+
+Options:
+-f {a,b,c} Field name(s) to rank.
+-g {d,e,f} Optional group-by-field name(s).
+--sorted Promise that the input is already sorted by the field(s) being ranked
+ (within each group, if -g is given). This computes rank in a single
+ streaming pass and O(1) space, by comparing each record's value only
+ to the immediately preceding one, rather than buffering all records
+ to compute an order-independent rank. Produces wrong output if the
+ input is not in fact sorted.
+-h|--help Show this message.
+Example: mlr rank -f x data/rank-example.csv
+Example: mlr rank -f x -g g data/rank-example.csv
+Example: mlr sort -f x then rank -f x --sorted data/rank-example.csv
+
+
+For example, suppose you have the following CSV file:
+
+
+g,x
+a,10
+a,20
+a,20
+a,30
+b,5
+b,5
+b,9
+
+
+Then we can rank each record's `x` among all records:
+
+
+mlr --icsv --opprint rank -f x data/rank-example.csv
+
+
+g x x_rank
+a 10 4
+a 20 5
+a 20 5
+a 30 7
+b 5 1
+b 5 1
+b 9 3
+
+
+Using `-g` we can rank within each group instead:
+
+
+mlr --icsv --opprint rank -f x -g g data/rank-example.csv
+
+
+g x x_rank
+a 10 1
+a 20 2
+a 20 2
+a 30 4
+b 5 1
+b 5 1
+b 9 3
+
+
+By default `rank` is a two-pass algorithm: it retains all input records so that it can compute
+ranks which don't depend on input order, even when same-valued records aren't adjacent to one
+another. If you know your input is already sorted on the field(s) you're ranking by -- e.g. by
+piping through `mlr sort` first -- then `--sorted` computes ranks in a single streaming pass
+and O(1) space instead, by comparing each record only to the one immediately before it (within
+its group, if `-g` is given):
+
+
+mlr --icsv --opprint sort -f g -nf x then rank -f x -g g --sorted data/rank-example.csv
+
+
+g x x_rank
+a 10 1
+a 20 2
+a 20 2
+a 30 4
+b 5 1
+b 5 1
+b 9 3
+
+
## regularize
diff --git a/docs/src/reference-verbs.md.in b/docs/src/reference-verbs.md.in
index ae897cf51..1d4b74da3 100644
--- a/docs/src/reference-verbs.md.in
+++ b/docs/src/reference-verbs.md.in
@@ -27,7 +27,7 @@ These fall into categories as follows:
* `awk`-like functionality: [filter](reference-verbs.md#filter), [put](reference-verbs.md#put), [sec2gmt](reference-verbs.md#sec2gmt), [sec2gmtdate](reference-verbs.md#sec2gmtdate), [step](reference-verbs.md#step), [tee](reference-verbs.md#tee).
-* Statistically oriented: [bar](reference-verbs.md#bar), [bootstrap](reference-verbs.md#bootstrap), [decimate](reference-verbs.md#decimate), [histogram](reference-verbs.md#histogram), [least-frequent](reference-verbs.md#least-frequent), [most-frequent](reference-verbs.md#most-frequent), [sample](reference-verbs.md#sample), [shuffle](reference-verbs.md#shuffle), [sparkline](reference-verbs.md#sparkline), [stats1](reference-verbs.md#stats1), [stats2](reference-verbs.md#stats2).
+* Statistically oriented: [bar](reference-verbs.md#bar), [bootstrap](reference-verbs.md#bootstrap), [decimate](reference-verbs.md#decimate), [histogram](reference-verbs.md#histogram), [least-frequent](reference-verbs.md#least-frequent), [most-frequent](reference-verbs.md#most-frequent), [rank](reference-verbs.md#rank), [sample](reference-verbs.md#sample), [shuffle](reference-verbs.md#shuffle), [sparkline](reference-verbs.md#sparkline), [stats1](reference-verbs.md#stats1), [stats2](reference-verbs.md#stats2).
* Particularly oriented toward [Record Heterogeneity](record-heterogeneity.md), although all Miller commands can handle heterogeneous records: [group-by](reference-verbs.md#group-by), [group-like](reference-verbs.md#group-like), [having-fields](reference-verbs.md#having-fields).
@@ -796,6 +796,39 @@ GENMD-EOF
Please see the [DSL reference](reference-dsl.md) for more information about the expression language for `mlr put`.
+## rank
+
+GENMD-RUN-COMMAND
+mlr rank --help
+GENMD-EOF
+
+For example, suppose you have the following CSV file:
+
+GENMD-INCLUDE-ESCAPED(data/rank-example.csv)
+
+Then we can rank each record's `x` among all records:
+
+GENMD-RUN-COMMAND
+mlr --icsv --opprint rank -f x data/rank-example.csv
+GENMD-EOF
+
+Using `-g` we can rank within each group instead:
+
+GENMD-RUN-COMMAND
+mlr --icsv --opprint rank -f x -g g data/rank-example.csv
+GENMD-EOF
+
+By default `rank` is a two-pass algorithm: it retains all input records so that it can compute
+ranks which don't depend on input order, even when same-valued records aren't adjacent to one
+another. If you know your input is already sorted on the field(s) you're ranking by -- e.g. by
+piping through `mlr sort` first -- then `--sorted` computes ranks in a single streaming pass
+and O(1) space instead, by comparing each record only to the one immediately before it (within
+its group, if `-g` is given):
+
+GENMD-RUN-COMMAND
+mlr --icsv --opprint sort -f g -nf x then rank -f x -g g --sorted data/rank-example.csv
+GENMD-EOF
+
## regularize
GENMD-RUN-COMMAND
diff --git a/docs/src/streaming-and-memory.md b/docs/src/streaming-and-memory.md
index b6097e755..e552d2cde 100644
--- a/docs/src/streaming-and-memory.md
+++ b/docs/src/streaming-and-memory.md
@@ -100,6 +100,7 @@ They are memory-friendly, and they don't wait for end of input to produce their
* [merge-fields](reference-verbs.md#merge-fields)
* [nest](reference-verbs.md#nest) -- if not `implode-values-across-records`
* [nothing](reference-verbs.md#nothing)
+* [rank](reference-verbs.md#rank) -- if `--sorted`
* [regularize](reference-verbs.md#regularize)
* [rename](reference-verbs.md#rename)
* [reorder](reference-verbs.md#reorder)
@@ -131,6 +132,7 @@ They are memory-unfriendly, and they wait for end of input to produce their outp
* [least-frequent](reference-verbs.md#least-frequent)
* [most-frequent](reference-verbs.md#most-frequent)
* [nest](reference-verbs.md#nest) -- if `implode-values-across-records`
+* [rank](reference-verbs.md#rank) -- if not `--sorted`
* [remove-empty-columns](reference-verbs.md#remove-empty-columns)
* [reshape](reference-verbs.md#reshape) -- if long-to-wide
* [sample](reference-verbs.md#sample)
diff --git a/docs/src/streaming-and-memory.md.in b/docs/src/streaming-and-memory.md.in
index 07e3018dc..f55e18b33 100644
--- a/docs/src/streaming-and-memory.md.in
+++ b/docs/src/streaming-and-memory.md.in
@@ -84,6 +84,7 @@ They are memory-friendly, and they don't wait for end of input to produce their
* [merge-fields](reference-verbs.md#merge-fields)
* [nest](reference-verbs.md#nest) -- if not `implode-values-across-records`
* [nothing](reference-verbs.md#nothing)
+* [rank](reference-verbs.md#rank) -- if `--sorted`
* [regularize](reference-verbs.md#regularize)
* [rename](reference-verbs.md#rename)
* [reorder](reference-verbs.md#reorder)
@@ -115,6 +116,7 @@ They are memory-unfriendly, and they wait for end of input to produce their outp
* [least-frequent](reference-verbs.md#least-frequent)
* [most-frequent](reference-verbs.md#most-frequent)
* [nest](reference-verbs.md#nest) -- if `implode-values-across-records`
+* [rank](reference-verbs.md#rank) -- if not `--sorted`
* [remove-empty-columns](reference-verbs.md#remove-empty-columns)
* [reshape](reference-verbs.md#reshape) -- if long-to-wide
* [sample](reference-verbs.md#sample)
diff --git a/man/manpage.txt b/man/manpage.txt
index 56e848bb8..ebbd0a727 100644
--- a/man/manpage.txt
+++ b/man/manpage.txt
@@ -207,11 +207,11 @@
count-similar cut decimate describe fill-down fill-empty filter flatten
format-values fraction gap grep group-by group-like gsub having-fields head
histogram json-parse json-stringify join label latin1-to-utf8 least-frequent
- merge-fields most-frequent nest nothing put regularize remove-empty-columns
- rename reorder repeat reshape sample sec2gmtdate sec2gmt seqgen shuffle
- skip-trivial-records sort sort-within-records sparkline sparsify split ssub
- stats1 stats2 step sub summary surv tac tail tee template top utf8-to-latin1
- unflatten uniq unspace unsparsify
+ merge-fields most-frequent nest nothing put rank regularize
+ remove-empty-columns rename reorder repeat reshape sample sec2gmtdate sec2gmt
+ seqgen shuffle skip-trivial-records sort sort-within-records sparkline
+ sparsify split ssub stats1 stats2 step sub summary surv tac tail tee template
+ top utf8-to-latin1 unflatten uniq unspace unsparsify
1mFUNCTION LIST0m
abs acos acosh antimode any append apply arrayify asin asinh asserting_absent
@@ -1791,6 +1791,34 @@
See also https://miller.readthedocs.io/reference-dsl for more context.
+ 1mrank0m
+ Usage: mlr rank [options]
+ For each record's value in specified fields, computes the standard
+ competition rank (1,2,2,4,...) of that value among all input records,
+ optionally within groups.
+ E.g. with input records x=10, x=20, x=20, and x=30, emits output records
+ x=10,x_rank=1 x=20,x_rank=2 x=20,x_rank=2 and x=30,x_rank=4.
+
+ Note: by default this is a two-pass algorithm: on the first pass it retains
+ input records and their values; on the second pass it computes ranks and
+ emits output records, in original input order. This means it produces no
+ output until all input is read, but gives correct ranks regardless of input
+ order. Use --sorted for a single-pass streaming alternative.
+
+ Options:
+ -f {a,b,c} Field name(s) to rank.
+ -g {d,e,f} Optional group-by-field name(s).
+ --sorted Promise that the input is already sorted by the field(s) being ranked
+ (within each group, if -g is given). This computes rank in a single
+ streaming pass and O(1) space, by comparing each record's value only
+ to the immediately preceding one, rather than buffering all records
+ to compute an order-independent rank. Produces wrong output if the
+ input is not in fact sorted.
+ -h|--help Show this message.
+ Example: mlr rank -f x data/rank-example.csv
+ Example: mlr rank -f x -g g data/rank-example.csv
+ Example: mlr sort -f x then rank -f x --sorted data/rank-example.csv
+
1mregularize0m
Usage: mlr regularize [options]
Outputs records sorted lexically ascending by keys.
diff --git a/man/mlr.1 b/man/mlr.1
index 1d955610a..efcfdf33e 100644
--- a/man/mlr.1
+++ b/man/mlr.1
@@ -249,11 +249,11 @@ altkv bar bootstrap case cat check clean-whitespace count-distinct count
count-similar cut decimate describe fill-down fill-empty filter flatten
format-values fraction gap grep group-by group-like gsub having-fields head
histogram json-parse json-stringify join label latin1-to-utf8 least-frequent
-merge-fields most-frequent nest nothing put regularize remove-empty-columns
-rename reorder repeat reshape sample sec2gmtdate sec2gmt seqgen shuffle
-skip-trivial-records sort sort-within-records sparkline sparsify split ssub
-stats1 stats2 step sub summary surv tac tail tee template top utf8-to-latin1
-unflatten uniq unspace unsparsify
+merge-fields most-frequent nest nothing put rank regularize
+remove-empty-columns rename reorder repeat reshape sample sec2gmtdate sec2gmt
+seqgen shuffle skip-trivial-records sort sort-within-records sparkline
+sparsify split ssub stats1 stats2 step sub summary surv tac tail tee template
+top utf8-to-latin1 unflatten uniq unspace unsparsify
.fi
.if n \{\
.RE
@@ -2209,6 +2209,40 @@ See also https://miller.readthedocs.io/reference-dsl for more context.
.fi
.if n \{\
.RE
+.SS "rank"
+.if n \{\
+.RS 0
+.\}
+.nf
+Usage: mlr rank [options]
+For each record's value in specified fields, computes the standard
+competition rank (1,2,2,4,...) of that value among all input records,
+optionally within groups.
+E.g. with input records x=10, x=20, x=20, and x=30, emits output records
+x=10,x_rank=1 x=20,x_rank=2 x=20,x_rank=2 and x=30,x_rank=4.
+
+Note: by default this is a two-pass algorithm: on the first pass it retains
+input records and their values; on the second pass it computes ranks and
+emits output records, in original input order. This means it produces no
+output until all input is read, but gives correct ranks regardless of input
+order. Use --sorted for a single-pass streaming alternative.
+
+Options:
+-f {a,b,c} Field name(s) to rank.
+-g {d,e,f} Optional group-by-field name(s).
+--sorted Promise that the input is already sorted by the field(s) being ranked
+ (within each group, if -g is given). This computes rank in a single
+ streaming pass and O(1) space, by comparing each record's value only
+ to the immediately preceding one, rather than buffering all records
+ to compute an order-independent rank. Produces wrong output if the
+ input is not in fact sorted.
+-h|--help Show this message.
+Example: mlr rank -f x data/rank-example.csv
+Example: mlr rank -f x -g g data/rank-example.csv
+Example: mlr sort -f x then rank -f x --sorted data/rank-example.csv
+.fi
+.if n \{\
+.RE
.SS "regularize"
.if n \{\
.RS 0
diff --git a/pkg/transformers/aaa_transformer_table.go b/pkg/transformers/aaa_transformer_table.go
index bb49e0098..b9bbe7f64 100644
--- a/pkg/transformers/aaa_transformer_table.go
+++ b/pkg/transformers/aaa_transformer_table.go
@@ -48,6 +48,7 @@ var TRANSFORMER_LOOKUP_TABLE = []TransformerSetup{
NestSetup,
NothingSetup,
PutSetup,
+ RankSetup,
RegularizeSetup,
RemoveEmptyColumnsSetup,
RenameSetup,
diff --git a/pkg/transformers/rank.go b/pkg/transformers/rank.go
new file mode 100644
index 000000000..ae2855709
--- /dev/null
+++ b/pkg/transformers/rank.go
@@ -0,0 +1,284 @@
+package transformers
+
+import (
+ "fmt"
+ "os"
+ "strings"
+
+ "github.com/johnkerl/miller/v6/pkg/cli"
+ "github.com/johnkerl/miller/v6/pkg/mlrval"
+ "github.com/johnkerl/miller/v6/pkg/transformers/utils"
+ "github.com/johnkerl/miller/v6/pkg/types"
+)
+
+const verbNameRank = "rank"
+
+var rankOptions = []OptionSpec{
+ {Flag: "-f", Arg: "{a,b,c}", Type: "csv-list", Desc: "Field name(s) to rank."},
+ {Flag: "-g", Arg: "{d,e,f}", Type: "csv-list", Desc: "Optional group-by-field name(s)."},
+ {Flag: "--sorted", Type: "bool", Desc: "Promise that the input is already sorted by the field(s) being ranked (within each group, if -g is given). This computes rank in a single streaming pass and O(1) space, by comparing each record's value only to the immediately preceding one, rather than buffering all records to compute an order-independent rank. Produces wrong output if the input is not in fact sorted."},
+}
+
+var RankSetup = TransformerSetup{
+ Verb: verbNameRank,
+ UsageFunc: transformerRankUsage,
+ ParseCLIFunc: transformerRankParseCLI,
+ IgnoresInput: false,
+ Options: rankOptions,
+}
+
+func transformerRankUsage(
+ o *os.File,
+) {
+ argv0 := "mlr"
+ verb := verbNameRank
+ fmt.Fprintf(o, "Usage: %s %s [options]\n", argv0, verb)
+ fmt.Fprintf(o, "For each record's value in specified fields, computes the standard\n")
+ fmt.Fprintf(o, "competition rank (1,2,2,4,...) of that value among all input records,\n")
+ fmt.Fprintf(o, "optionally within groups.\n")
+ fmt.Fprintf(o, "E.g. with input records x=10, x=20, x=20, and x=30, emits output records\n")
+ fmt.Fprintf(o, "x=10,x_rank=1 x=20,x_rank=2 x=20,x_rank=2 and x=30,x_rank=4.\n")
+ fmt.Fprintf(o, "\n")
+ fmt.Fprintf(o, "Note: by default this is a two-pass algorithm: on the first pass it retains\n")
+ fmt.Fprintf(o, "input records and their values; on the second pass it computes ranks and\n")
+ fmt.Fprintf(o, "emits output records, in original input order. This means it produces no\n")
+ fmt.Fprintf(o, "output until all input is read, but gives correct ranks regardless of input\n")
+ fmt.Fprintf(o, "order. Use --sorted for a single-pass streaming alternative.\n")
+ fmt.Fprintf(o, "\n")
+ WriteVerbOptions(o, rankOptions)
+ fmt.Fprintln(o, "Example: mlr rank -f x data/rank-example.csv")
+ fmt.Fprintln(o, "Example: mlr rank -f x -g g data/rank-example.csv")
+ fmt.Fprintln(o, "Example: mlr sort -f x then rank -f x --sorted data/rank-example.csv")
+}
+
+func transformerRankParseCLI(
+ pargi *int,
+ argc int,
+ args []string,
+ _ *cli.TOptions,
+ doConstruct bool, // false for first pass of CLI-parse, true for second pass
+) (RecordTransformer, error) {
+
+ // Skip the verb name from the current spot in the mlr command line
+ argi := *pargi
+ verb := args[argi]
+ argi++
+
+ var rankFieldNames []string = nil
+ var groupByFieldNames []string = nil
+ doSorted := false
+
+ var err error
+ for argi < argc /* variable increment: 1 or 2 depending on flag */ {
+ opt := args[argi]
+ if !strings.HasPrefix(opt, "-") {
+ break // No more flag options to process
+ }
+ if args[argi] == "--" {
+ break // All transformers must do this so main-flags can follow verb-flags
+ }
+ argi++
+
+ switch opt {
+ case "-h", "--help":
+ transformerRankUsage(os.Stdout)
+ return nil, cli.ErrHelpRequested
+
+ case "-f":
+ rankFieldNames, err = cli.VerbGetStringArrayArg(verb, opt, args, &argi, argc)
+ if err != nil {
+ return nil, err
+ }
+
+ case "-g":
+ groupByFieldNames, err = cli.VerbGetStringArrayArg(verb, opt, args, &argi, argc)
+ if err != nil {
+ return nil, err
+ }
+
+ case "--sorted":
+ doSorted = true
+
+ default:
+ return nil, cli.VerbErrorf(verb, "option \"%s\" not recognized", opt)
+ }
+ }
+
+ if rankFieldNames == nil {
+ return nil, cli.VerbErrorf(verb, "-f field names required")
+ }
+
+ *pargi = argi
+ if !doConstruct { // All transformers must do this for main command-line parsing
+ return nil, nil
+ }
+
+ transformer, err := NewTransformerRank(
+ rankFieldNames,
+ groupByFieldNames,
+ doSorted,
+ )
+ if err != nil {
+ return nil, err
+ }
+
+ return transformer, nil
+}
+
+type TransformerRank struct {
+ rankFieldNames []string
+ groupByFieldNames []string
+ doSorted bool
+
+ // Default (unsorted) mode: two-pass. Records are retained on the first
+ // pass, along with per-group-per-field percentile-keepers; on the
+ // second pass (end of stream) the retained records are decorated with
+ // rank fields and emitted in original input order.
+ recordsAndContexts []*types.RecordAndContext
+ keepers map[string]map[string]*utils.PercentileKeeper // grouping-key -> field-name -> keeper
+
+ // --sorted mode: single streaming pass, O(1) space. Same shape as the
+ // keepers map above, but holding lightweight adjacency state instead of
+ // buffered/sorted values.
+ sortedStates map[string]map[string]*tRankSortedFieldState // grouping-key -> field-name -> state
+}
+
+type tRankSortedFieldState struct {
+ count int64
+ rank int64
+ havePreviousValue bool
+ previousValueString string
+}
+
+func NewTransformerRank(
+ rankFieldNames []string,
+ groupByFieldNames []string,
+ doSorted bool,
+) (*TransformerRank, error) {
+ return &TransformerRank{
+ rankFieldNames: rankFieldNames,
+ groupByFieldNames: groupByFieldNames,
+ doSorted: doSorted,
+ recordsAndContexts: []*types.RecordAndContext{},
+ keepers: make(map[string]map[string]*utils.PercentileKeeper),
+ sortedStates: make(map[string]map[string]*tRankSortedFieldState),
+ }, nil
+}
+
+func (tr *TransformerRank) Transform(
+ inrecAndContext *types.RecordAndContext,
+ outputRecordsAndContexts *[]*types.RecordAndContext, // list of *types.RecordAndContext
+ inputDownstreamDoneChannel <-chan bool,
+ outputDownstreamDoneChannel chan<- bool,
+) {
+ HandleDefaultDownstreamDone(inputDownstreamDoneChannel, outputDownstreamDoneChannel)
+ if tr.doSorted {
+ tr.transformSorted(inrecAndContext, outputRecordsAndContexts)
+ } else {
+ tr.transformUnsorted(inrecAndContext, outputRecordsAndContexts)
+ }
+}
+
+// transformSorted computes rank in a single pass, O(1) space, by comparing
+// each record's value only to the immediately preceding one within its
+// group. This is only correct if the caller has ensured the input is
+// already sorted by the ranked field(s), e.g. via 'mlr sort'.
+func (tr *TransformerRank) transformSorted(
+ inrecAndContext *types.RecordAndContext,
+ outputRecordsAndContexts *[]*types.RecordAndContext,
+) {
+ if !inrecAndContext.EndOfStream {
+ inrec := inrecAndContext.Record
+
+ groupingKey, hasAll := inrec.GetSelectedValuesJoined(tr.groupByFieldNames)
+ if hasAll {
+ statesForGroup := tr.sortedStates[groupingKey]
+ if statesForGroup == nil {
+ statesForGroup = make(map[string]*tRankSortedFieldState)
+ tr.sortedStates[groupingKey] = statesForGroup
+ }
+ for _, rankFieldName := range tr.rankFieldNames {
+ value := inrec.Get(rankFieldName)
+ if value == nil {
+ continue
+ }
+ state := statesForGroup[rankFieldName]
+ if state == nil {
+ state = &tRankSortedFieldState{}
+ statesForGroup[rankFieldName] = state
+ }
+ state.count++
+ valueString := value.String() // 1, 1.0, and 1.000 are distinct
+ if !state.havePreviousValue || valueString != state.previousValueString {
+ state.rank = state.count
+ state.previousValueString = valueString
+ state.havePreviousValue = true
+ }
+ inrec.PutCopy(rankFieldName+"_rank", mlrval.FromInt(state.rank))
+ }
+ }
+ }
+ *outputRecordsAndContexts = append(*outputRecordsAndContexts, inrecAndContext)
+}
+
+// transformUnsorted computes order-independent standard competition rank:
+// on the first pass it retains records and accumulates per-group-per-field
+// value sets; on the second pass (end of stream) it decorates the retained
+// records with rank fields and emits them in original input order.
+func (tr *TransformerRank) transformUnsorted(
+ inrecAndContext *types.RecordAndContext,
+ outputRecordsAndContexts *[]*types.RecordAndContext,
+) {
+ if !inrecAndContext.EndOfStream { // Not end of stream; pass 1
+ inrec := inrecAndContext.Record
+
+ // Append records into a single output list (so that this verb is order-preserving).
+ tr.recordsAndContexts = append(tr.recordsAndContexts, inrecAndContext)
+
+ groupingKey, hasAll := inrec.GetSelectedValuesJoined(tr.groupByFieldNames)
+ if hasAll {
+ keepersForGroup := tr.keepers[groupingKey]
+ if keepersForGroup == nil {
+ keepersForGroup = make(map[string]*utils.PercentileKeeper)
+ tr.keepers[groupingKey] = keepersForGroup
+ }
+ for _, rankFieldName := range tr.rankFieldNames {
+ value := inrec.Get(rankFieldName)
+ if value == nil {
+ continue
+ }
+ keeper := keepersForGroup[rankFieldName]
+ if keeper == nil {
+ keeper = utils.NewPercentileKeeper(false)
+ keepersForGroup[rankFieldName] = keeper
+ }
+ keeper.Ingest(value)
+ }
+ }
+
+ } else { // End of stream; pass 2
+ // Iterate over the retained records, decorating them with rank fields.
+ endOfStreamContext := inrecAndContext.Context
+
+ for _, recordAndContext := range tr.recordsAndContexts {
+ outrec := recordAndContext.Record
+
+ groupingKey, hasAll := outrec.GetSelectedValuesJoined(tr.groupByFieldNames)
+ if hasAll {
+ keepersForGroup := tr.keepers[groupingKey]
+ for _, rankFieldName := range tr.rankFieldNames {
+ value := outrec.Get(rankFieldName)
+ if value == nil {
+ continue
+ }
+ keeper := keepersForGroup[rankFieldName]
+ outrec.PutCopy(rankFieldName+"_rank", keeper.EmitRank(value))
+ }
+ }
+
+ *outputRecordsAndContexts = append(*outputRecordsAndContexts, types.NewRecordAndContext(outrec, &endOfStreamContext))
+ }
+ tr.recordsAndContexts = tr.recordsAndContexts[:0]
+ *outputRecordsAndContexts = append(*outputRecordsAndContexts, inrecAndContext) // end-of-stream marker
+ }
+}
diff --git a/pkg/transformers/utils/percentile_keeper.go b/pkg/transformers/utils/percentile_keeper.go
index 37725e051..811a0586b 100644
--- a/pkg/transformers/utils/percentile_keeper.go
+++ b/pkg/transformers/utils/percentile_keeper.go
@@ -82,6 +82,27 @@ func (keeper *PercentileKeeper) EmitLinearlyInterpolated(percentile float64) *ml
return bifs.GetPercentileLinearlyInterpolated(keeper.data, int(len(keeper.data)), percentile)
}
+// EmitRank returns the standard competition rank (1,2,2,4,...) of value
+// among all values ingested so far: one plus the number of ingested values
+// strictly less than it.
+func (keeper *PercentileKeeper) EmitRank(value *mlrval.Mlrval) *mlrval.Mlrval {
+ if len(keeper.data) == 0 {
+ return mlrval.VOID
+ }
+ keeper.sortIfNecessary()
+ n := len(keeper.data)
+ lo, hi := 0, n
+ for lo < hi {
+ mid := (lo + hi) / 2
+ if mlrval.LessThan(keeper.data[mid], value) {
+ lo = mid + 1
+ } else {
+ hi = mid
+ }
+ }
+ return mlrval.FromInt(int64(lo + 1))
+}
+
// TODO: COMMENT
func (keeper *PercentileKeeper) EmitNamed(name string) *mlrval.Mlrval {
switch name {
diff --git a/test/cases/cli-help/0001/expout b/test/cases/cli-help/0001/expout
index 9399b12b3..a6a2d3275 100644
--- a/test/cases/cli-help/0001/expout
+++ b/test/cases/cli-help/0001/expout
@@ -835,6 +835,35 @@ More example put expressions:
See also https://miller.readthedocs.io/reference-dsl for more context.
+================================================================
+rank
+Usage: mlr rank [options]
+For each record's value in specified fields, computes the standard
+competition rank (1,2,2,4,...) of that value among all input records,
+optionally within groups.
+E.g. with input records x=10, x=20, x=20, and x=30, emits output records
+x=10,x_rank=1 x=20,x_rank=2 x=20,x_rank=2 and x=30,x_rank=4.
+
+Note: by default this is a two-pass algorithm: on the first pass it retains
+input records and their values; on the second pass it computes ranks and
+emits output records, in original input order. This means it produces no
+output until all input is read, but gives correct ranks regardless of input
+order. Use --sorted for a single-pass streaming alternative.
+
+Options:
+-f {a,b,c} Field name(s) to rank.
+-g {d,e,f} Optional group-by-field name(s).
+--sorted Promise that the input is already sorted by the field(s) being ranked
+ (within each group, if -g is given). This computes rank in a single
+ streaming pass and O(1) space, by comparing each record's value only
+ to the immediately preceding one, rather than buffering all records
+ to compute an order-independent rank. Produces wrong output if the
+ input is not in fact sorted.
+-h|--help Show this message.
+Example: mlr rank -f x data/rank-example.csv
+Example: mlr rank -f x -g g data/rank-example.csv
+Example: mlr sort -f x then rank -f x --sorted data/rank-example.csv
+
================================================================
regularize
Usage: mlr regularize [options]
diff --git a/test/cases/verb-rank/0001/cmd b/test/cases/verb-rank/0001/cmd
new file mode 100644
index 000000000..46ddbeaf4
--- /dev/null
+++ b/test/cases/verb-rank/0001/cmd
@@ -0,0 +1 @@
+mlr --icsv --ocsv rank -f x test/input/rank-data.csv
diff --git a/test/cases/verb-rank/0001/experr b/test/cases/verb-rank/0001/experr
new file mode 100644
index 000000000..e69de29bb
diff --git a/test/cases/verb-rank/0001/expout b/test/cases/verb-rank/0001/expout
new file mode 100644
index 000000000..543c233c9
--- /dev/null
+++ b/test/cases/verb-rank/0001/expout
@@ -0,0 +1,5 @@
+x,x_rank
+10,1
+20,2
+20,2
+30,4
diff --git a/test/cases/verb-rank/0002/cmd b/test/cases/verb-rank/0002/cmd
new file mode 100644
index 000000000..d8ef5e099
--- /dev/null
+++ b/test/cases/verb-rank/0002/cmd
@@ -0,0 +1 @@
+mlr --icsv --ocsv rank -f x test/input/rank-data-unsorted.csv
diff --git a/test/cases/verb-rank/0002/experr b/test/cases/verb-rank/0002/experr
new file mode 100644
index 000000000..e69de29bb
diff --git a/test/cases/verb-rank/0002/expout b/test/cases/verb-rank/0002/expout
new file mode 100644
index 000000000..dc98cbe30
--- /dev/null
+++ b/test/cases/verb-rank/0002/expout
@@ -0,0 +1,5 @@
+x,x_rank
+20,2
+10,1
+20,2
+30,4
diff --git a/test/cases/verb-rank/0003/cmd b/test/cases/verb-rank/0003/cmd
new file mode 100644
index 000000000..81c63e31b
--- /dev/null
+++ b/test/cases/verb-rank/0003/cmd
@@ -0,0 +1 @@
+mlr --icsv --ocsv rank -f x -g g test/input/rank-data-grouped.csv
diff --git a/test/cases/verb-rank/0003/experr b/test/cases/verb-rank/0003/experr
new file mode 100644
index 000000000..e69de29bb
diff --git a/test/cases/verb-rank/0003/expout b/test/cases/verb-rank/0003/expout
new file mode 100644
index 000000000..49fcf4769
--- /dev/null
+++ b/test/cases/verb-rank/0003/expout
@@ -0,0 +1,6 @@
+g,x,x_rank
+a,20,2
+b,5,1
+a,10,1
+b,5,1
+a,20,2
diff --git a/test/cases/verb-rank/0004/cmd b/test/cases/verb-rank/0004/cmd
new file mode 100644
index 000000000..71d4149da
--- /dev/null
+++ b/test/cases/verb-rank/0004/cmd
@@ -0,0 +1 @@
+mlr --icsv --ocsv rank -f x --sorted test/input/rank-data.csv
diff --git a/test/cases/verb-rank/0004/experr b/test/cases/verb-rank/0004/experr
new file mode 100644
index 000000000..e69de29bb
diff --git a/test/cases/verb-rank/0004/expout b/test/cases/verb-rank/0004/expout
new file mode 100644
index 000000000..543c233c9
--- /dev/null
+++ b/test/cases/verb-rank/0004/expout
@@ -0,0 +1,5 @@
+x,x_rank
+10,1
+20,2
+20,2
+30,4
diff --git a/test/cases/verb-rank/0005/cmd b/test/cases/verb-rank/0005/cmd
new file mode 100644
index 000000000..9fad3ad0e
--- /dev/null
+++ b/test/cases/verb-rank/0005/cmd
@@ -0,0 +1 @@
+mlr --icsv --ocsv sort -f x then rank -f x --sorted test/input/rank-data-unsorted.csv
diff --git a/test/cases/verb-rank/0005/experr b/test/cases/verb-rank/0005/experr
new file mode 100644
index 000000000..e69de29bb
diff --git a/test/cases/verb-rank/0005/expout b/test/cases/verb-rank/0005/expout
new file mode 100644
index 000000000..543c233c9
--- /dev/null
+++ b/test/cases/verb-rank/0005/expout
@@ -0,0 +1,5 @@
+x,x_rank
+10,1
+20,2
+20,2
+30,4
diff --git a/test/cases/verb-rank/0006/cmd b/test/cases/verb-rank/0006/cmd
new file mode 100644
index 000000000..530a8c3d2
--- /dev/null
+++ b/test/cases/verb-rank/0006/cmd
@@ -0,0 +1 @@
+mlr rank -g g
diff --git a/test/cases/verb-rank/0006/experr b/test/cases/verb-rank/0006/experr
new file mode 100644
index 000000000..3525047c2
--- /dev/null
+++ b/test/cases/verb-rank/0006/experr
@@ -0,0 +1 @@
+mlr rank: -f field names required
diff --git a/test/cases/verb-rank/0006/expout b/test/cases/verb-rank/0006/expout
new file mode 100644
index 000000000..e69de29bb
diff --git a/test/cases/verb-rank/0006/should-fail b/test/cases/verb-rank/0006/should-fail
new file mode 100644
index 000000000..e69de29bb
diff --git a/test/input/rank-data-grouped.csv b/test/input/rank-data-grouped.csv
new file mode 100644
index 000000000..c4d1092ca
--- /dev/null
+++ b/test/input/rank-data-grouped.csv
@@ -0,0 +1,6 @@
+g,x
+a,20
+b,5
+a,10
+b,5
+a,20
diff --git a/test/input/rank-data-unsorted.csv b/test/input/rank-data-unsorted.csv
new file mode 100644
index 000000000..9d68f16b0
--- /dev/null
+++ b/test/input/rank-data-unsorted.csv
@@ -0,0 +1,5 @@
+x
+20
+10
+20
+30
diff --git a/test/input/rank-data.csv b/test/input/rank-data.csv
new file mode 100644
index 000000000..4617ce85c
--- /dev/null
+++ b/test/input/rank-data.csv
@@ -0,0 +1,5 @@
+x
+10
+20
+20
+30