stats1: add rank accumulator (#383)

Adds `mlr stats1 -a rank` for standard competition ranking (1,2,2,4,...)
on pre-sorted data, most useful with -s for a rank on every record.
This commit is contained in:
John Kerl 2026-07-07 15:49:53 -04:00
parent d9d72a3ad1
commit 96deed048a
16 changed files with 118 additions and 6 deletions

View file

@ -1626,6 +1626,7 @@ This is simply a copy of what you should see on running `man mlr` at a command p
kurtosis Compute sample kurtosis of specified fields
min Compute minimum values of specified fields
max Compute maximum values of specified fields
rank Compute rank 1,2,2,4,... of specified fields, assuming input is sorted by that field; use with -s
minlen Compute minimum string-lengths of specified fields
maxlen Compute maximum string-lengths of specified fields
@ -2202,6 +2203,7 @@ This is simply a copy of what you should see on running `man mlr` at a command p
kurtosis Compute sample kurtosis of specified fields
min Compute minimum values of specified fields
max Compute maximum values of specified fields
rank Compute rank 1,2,2,4,... of specified fields, assuming input is sorted by that field; use with -s
minlen Compute minimum string-lengths of specified fields
maxlen Compute maximum string-lengths of specified fields
Example: mlr stats1 -a min,p10,p50,p90,max -f value -g size,shape
@ -2221,6 +2223,10 @@ This is simply a copy of what you should see on running `man mlr` at a command p
* count and mode allow text input; the rest require numeric input.
In particular, 1 and 1.0 are distinct text for count and mode.
* When there are mode ties, the first-encountered datum wins.
* rank assumes the input is already sorted (e.g. by mlr sort) on the field
being ranked, and assigns standard competition rank (1,2,2,4,...) by
comparing each record's value to the immediately preceding one. Use with
-s to get a rank on every input record.
1mstats20m
Usage: mlr stats2 [options]
@ -4062,5 +4068,5 @@ This is simply a copy of what you should see on running `man mlr` at a command p
MIME Type for Comma-Separated Values (CSV) Files, the Miller docsite
https://miller.readthedocs.io
2026-07-06 4mMILLER24m(1)
2026-07-07 4mMILLER24m(1)
</pre>

View file

@ -1605,6 +1605,7 @@
kurtosis Compute sample kurtosis of specified fields
min Compute minimum values of specified fields
max Compute maximum values of specified fields
rank Compute rank 1,2,2,4,... of specified fields, assuming input is sorted by that field; use with -s
minlen Compute minimum string-lengths of specified fields
maxlen Compute maximum string-lengths of specified fields
@ -2181,6 +2182,7 @@
kurtosis Compute sample kurtosis of specified fields
min Compute minimum values of specified fields
max Compute maximum values of specified fields
rank Compute rank 1,2,2,4,... of specified fields, assuming input is sorted by that field; use with -s
minlen Compute minimum string-lengths of specified fields
maxlen Compute maximum string-lengths of specified fields
Example: mlr stats1 -a min,p10,p50,p90,max -f value -g size,shape
@ -2200,6 +2202,10 @@
* count and mode allow text input; the rest require numeric input.
In particular, 1 and 1.0 are distinct text for count and mode.
* When there are mode ties, the first-encountered datum wins.
* rank assumes the input is already sorted (e.g. by mlr sort) on the field
being ranked, and assigns standard competition rank (1,2,2,4,...) by
comparing each record's value to the immediately preceding one. Use with
-s to get a rank on every input record.
1mstats20m
Usage: mlr stats2 [options]
@ -4041,4 +4047,4 @@
MIME Type for Comma-Separated Values (CSV) Files, the Miller docsite
https://miller.readthedocs.io
2026-07-06 4mMILLER24m(1)
2026-07-07 4mMILLER24m(1)

View file

@ -2317,6 +2317,7 @@ Accumulators for -a:
kurtosis Compute sample kurtosis of specified fields
min Compute minimum values of specified fields
max Compute maximum values of specified fields
rank Compute rank 1,2,2,4,... of specified fields, assuming input is sorted by that field; use with -s
minlen Compute minimum string-lengths of specified fields
maxlen Compute maximum string-lengths of specified fields
@ -3568,6 +3569,7 @@ Names of accumulators for -a, one or more of:
kurtosis Compute sample kurtosis of specified fields
min Compute minimum values of specified fields
max Compute maximum values of specified fields
rank Compute rank 1,2,2,4,... of specified fields, assuming input is sorted by that field; use with -s
minlen Compute minimum string-lengths of specified fields
maxlen Compute maximum string-lengths of specified fields
Example: mlr stats1 -a min,p10,p50,p90,max -f value -g size,shape
@ -3587,6 +3589,10 @@ Notes:
* count and mode allow text input; the rest require numeric input.
In particular, 1 and 1.0 are distinct text for count and mode.
* When there are mode ties, the first-encountered datum wins.
* rank assumes the input is already sorted (e.g. by mlr sort) on the field
being ranked, and assigns standard competition rank (1,2,2,4,...) by
comparing each record's value to the immediately preceding one. Use with
-s to get a rank on every input record.
</pre>
These are simple univariate statistics on one or more number-valued fields

View file

@ -1605,6 +1605,7 @@
kurtosis Compute sample kurtosis of specified fields
min Compute minimum values of specified fields
max Compute maximum values of specified fields
rank Compute rank 1,2,2,4,... of specified fields, assuming input is sorted by that field; use with -s
minlen Compute minimum string-lengths of specified fields
maxlen Compute maximum string-lengths of specified fields
@ -2181,6 +2182,7 @@
kurtosis Compute sample kurtosis of specified fields
min Compute minimum values of specified fields
max Compute maximum values of specified fields
rank Compute rank 1,2,2,4,... of specified fields, assuming input is sorted by that field; use with -s
minlen Compute minimum string-lengths of specified fields
maxlen Compute maximum string-lengths of specified fields
Example: mlr stats1 -a min,p10,p50,p90,max -f value -g size,shape
@ -2200,6 +2202,10 @@
* count and mode allow text input; the rest require numeric input.
In particular, 1 and 1.0 are distinct text for count and mode.
* When there are mode ties, the first-encountered datum wins.
* rank assumes the input is already sorted (e.g. by mlr sort) on the field
being ranked, and assigns standard competition rank (1,2,2,4,...) by
comparing each record's value to the immediately preceding one. Use with
-s to get a rank on every input record.
1mstats20m
Usage: mlr stats2 [options]
@ -4041,4 +4047,4 @@
MIME Type for Comma-Separated Values (CSV) Files, the Miller docsite
https://miller.readthedocs.io
2026-07-06 4mMILLER24m(1)
2026-07-07 4mMILLER24m(1)

View file

@ -2,12 +2,12 @@
.\" Title: mlr
.\" Author: [see the "AUTHOR" section]
.\" Generator: ./mkman.rb
.\" Date: 2026-07-06
.\" Date: 2026-07-07
.\" Manual: \ \&
.\" Source: \ \&
.\" Language: English
.\"
.TH "MILLER" "1" "2026-07-06" "\ \&" "\ \&"
.TH "MILLER" "1" "2026-07-07" "\ \&" "\ \&"
.\" -----------------------------------------------------------------
.\" * Portability definitions
.\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@ -1997,6 +1997,7 @@ Accumulators for -a:
kurtosis Compute sample kurtosis of specified fields
min Compute minimum values of specified fields
max Compute maximum values of specified fields
rank Compute rank 1,2,2,4,... of specified fields, assuming input is sorted by that field; use with -s
minlen Compute minimum string-lengths of specified fields
maxlen Compute maximum string-lengths of specified fields
@ -2705,6 +2706,7 @@ Names of accumulators for -a, one or more of:
kurtosis Compute sample kurtosis of specified fields
min Compute minimum values of specified fields
max Compute maximum values of specified fields
rank Compute rank 1,2,2,4,... of specified fields, assuming input is sorted by that field; use with -s
minlen Compute minimum string-lengths of specified fields
maxlen Compute maximum string-lengths of specified fields
Example: mlr stats1 -a min,p10,p50,p90,max -f value -g size,shape
@ -2724,6 +2726,10 @@ Notes:
* count and mode allow text input; the rest require numeric input.
In particular, 1 and 1.0 are distinct text for count and mode.
* When there are mode ties, the first-encountered datum wins.
* rank assumes the input is already sorted (e.g. by mlr sort) on the field
being ranked, and assigns standard competition rank (1,2,2,4,...) by
comparing each record's value to the immediately preceding one. Use with
-s to get a rank on every input record.
.fi
.if n \{\
.RE

View file

@ -18,7 +18,7 @@ import (
const verbNameStats1 = "stats1"
var stats1Options = []OptionSpec{
{Flag: "-a", Arg: "{sum,count,...}", Type: "enum", Desc: "Names of accumulators: one or more of the listed values. Also accepts median (same as p50) and percentiles p{n} for n in 0..100, e.g. p10 p25.2 p50 p98 p100.", Values: []string{"count", "null_count", "distinct_count", "mode", "antimode", "sum", "mean", "mad", "var", "stddev", "meaneb", "skewness", "kurtosis", "min", "max", "minlen", "maxlen"}},
{Flag: "-a", Arg: "{sum,count,...}", Type: "enum", Desc: "Names of accumulators: one or more of the listed values. Also accepts median (same as p50) and percentiles p{n} for n in 0..100, e.g. p10 p25.2 p50 p98 p100.", Values: []string{"count", "null_count", "distinct_count", "mode", "antimode", "sum", "mean", "mad", "var", "stddev", "meaneb", "skewness", "kurtosis", "min", "max", "rank", "minlen", "maxlen"}},
{Flag: "-f", Arg: "{a,b,c}", Type: "csv-list", Desc: "Value-field names on which to compute statistics."},
{Flag: "--fr", Arg: "{regex}", Type: "regex", Desc: "Regex for value-field names on which to compute statistics (compute statistics on values in all field names matching the regex)."},
{Flag: "--fx", Arg: "{regex}", Type: "regex", Desc: "Inverted regex for value-field names on which to compute statistics (compute statistics on values in all field names not matching the regex)."},
@ -79,6 +79,10 @@ the input record stream.
* count and mode allow text input; the rest require numeric input.
In particular, 1 and 1.0 are distinct text for count and mode.
* When there are mode ties, the first-encountered datum wins.
* rank assumes the input is already sorted (e.g. by mlr sort) on the field
being ranked, and assigns standard competition rank (1,2,2,4,...) by
comparing each record's value to the immediately preceding one. Use with
-s to get a rank on every input record.
`)
}

View file

@ -111,6 +111,12 @@ var stats1AccumulatorInfos []stats1AccumulatorInfo = []stats1AccumulatorInfo{
NewStats1MaxAccumulator,
},
{
"rank",
"Compute rank 1,2,2,4,... of specified fields, assuming input is sorted by that field; use with -s",
NewStats1RankAccumulator,
},
{
"minlen",
"Compute minimum string-lengths of specified fields",
@ -533,6 +539,48 @@ func (acc *Stats1MeanAbsDevAccumulator) Reset() {
acc.samples = make([]*mlrval.Mlrval, 0, 1000)
}
// Stats1RankAccumulator implements standard competition ranking (1,2,2,4,...)
// on a sequence of values, e.g. as produced by 'mlr sort'. This assumes
// same-valued items are adjacent in the input stream: it compares each
// ingested value only to the immediately preceding one. This accumulator is
// most useful with 'stats1 -s' so that a rank is emitted for every input
// record, rather than only once at end of stream.
type Stats1RankAccumulator struct {
count int64
rank int64
havePreviousValue bool
previousValueString string
}
func NewStats1RankAccumulator() IStats1Accumulator {
return &Stats1RankAccumulator{
count: 0,
rank: 0,
havePreviousValue: false,
}
}
func (acc *Stats1RankAccumulator) Ingest(value *mlrval.Mlrval) {
acc.count++
valueString := value.String() // 1, 1.0, and 1.000 are distinct
if !acc.havePreviousValue || valueString != acc.previousValueString {
acc.rank = acc.count
acc.previousValueString = valueString
acc.havePreviousValue = true
}
}
func (acc *Stats1RankAccumulator) Emit() *mlrval.Mlrval {
if acc.count == 0 {
return mlrval.VOID
}
return mlrval.FromInt(acc.rank)
}
func (acc *Stats1RankAccumulator) Reset() {
acc.count = 0
acc.rank = 0
acc.havePreviousValue = false
acc.previousValueString = ""
}
type Stats1MinAccumulator struct {
min *mlrval.Mlrval
}

View file

@ -645,6 +645,7 @@ Accumulators for -a:
kurtosis Compute sample kurtosis of specified fields
min Compute minimum values of specified fields
max Compute maximum values of specified fields
rank Compute rank 1,2,2,4,... of specified fields, assuming input is sorted by that field; use with -s
minlen Compute minimum string-lengths of specified fields
maxlen Compute maximum string-lengths of specified fields
@ -1243,6 +1244,7 @@ Names of accumulators for -a, one or more of:
kurtosis Compute sample kurtosis of specified fields
min Compute minimum values of specified fields
max Compute maximum values of specified fields
rank Compute rank 1,2,2,4,... of specified fields, assuming input is sorted by that field; use with -s
minlen Compute minimum string-lengths of specified fields
maxlen Compute maximum string-lengths of specified fields
Example: mlr stats1 -a min,p10,p50,p90,max -f value -g size,shape
@ -1262,6 +1264,10 @@ Notes:
* count and mode allow text input; the rest require numeric input.
In particular, 1 and 1.0 are distinct text for count and mode.
* When there are mode ties, the first-encountered datum wins.
* rank assumes the input is already sorted (e.g. by mlr sort) on the field
being ranked, and assigns standard competition rank (1,2,2,4,...) by
comparing each record's value to the immediately preceding one. Use with
-s to get a rank on every input record.
================================================================
stats2

View file

@ -0,0 +1 @@
mlr --icsv --ocsv stats1 -a rank -f x -s test/input/rank-data.csv

View file

View file

@ -0,0 +1,5 @@
x,x_rank
10,1
20,2
20,2
30,4

View file

@ -0,0 +1 @@
mlr --icsv --ocsv stats1 -a rank -f x -g g -s test/input/rank-data-grouped.csv

View file

View file

@ -0,0 +1,6 @@
g,x,x_rank
a,10,1
a,10,1
a,20,3
b,5,1
b,5,1

View file

@ -0,0 +1,6 @@
g,x
a,10
a,10
a,20
b,5
b,5
1 g x
2 a 10
3 a 10
4 a 20
5 b 5
6 b 5

5
test/input/rank-data.csv Normal file
View file

@ -0,0 +1,5 @@
x
10
20
20
30
1 x
2 10
3 20
4 20
5 30