mirror of
https://github.com/johnkerl/miller.git
synced 2026-07-17 16:38:54 +00:00
stats1: add rank accumulator (#383)
Adds `mlr stats1 -a rank` for standard competition ranking (1,2,2,4,...) on pre-sorted data, most useful with -s for a rank on every record.
This commit is contained in:
parent
d9d72a3ad1
commit
96deed048a
16 changed files with 118 additions and 6 deletions
|
|
@ -1626,6 +1626,7 @@ This is simply a copy of what you should see on running `man mlr` at a command p
|
|||
kurtosis Compute sample kurtosis of specified fields
|
||||
min Compute minimum values of specified fields
|
||||
max Compute maximum values of specified fields
|
||||
rank Compute rank 1,2,2,4,... of specified fields, assuming input is sorted by that field; use with -s
|
||||
minlen Compute minimum string-lengths of specified fields
|
||||
maxlen Compute maximum string-lengths of specified fields
|
||||
|
||||
|
|
@ -2202,6 +2203,7 @@ This is simply a copy of what you should see on running `man mlr` at a command p
|
|||
kurtosis Compute sample kurtosis of specified fields
|
||||
min Compute minimum values of specified fields
|
||||
max Compute maximum values of specified fields
|
||||
rank Compute rank 1,2,2,4,... of specified fields, assuming input is sorted by that field; use with -s
|
||||
minlen Compute minimum string-lengths of specified fields
|
||||
maxlen Compute maximum string-lengths of specified fields
|
||||
Example: mlr stats1 -a min,p10,p50,p90,max -f value -g size,shape
|
||||
|
|
@ -2221,6 +2223,10 @@ This is simply a copy of what you should see on running `man mlr` at a command p
|
|||
* count and mode allow text input; the rest require numeric input.
|
||||
In particular, 1 and 1.0 are distinct text for count and mode.
|
||||
* When there are mode ties, the first-encountered datum wins.
|
||||
* rank assumes the input is already sorted (e.g. by mlr sort) on the field
|
||||
being ranked, and assigns standard competition rank (1,2,2,4,...) by
|
||||
comparing each record's value to the immediately preceding one. Use with
|
||||
-s to get a rank on every input record.
|
||||
|
||||
1mstats20m
|
||||
Usage: mlr stats2 [options]
|
||||
|
|
@ -4062,5 +4068,5 @@ This is simply a copy of what you should see on running `man mlr` at a command p
|
|||
MIME Type for Comma-Separated Values (CSV) Files, the Miller docsite
|
||||
https://miller.readthedocs.io
|
||||
|
||||
2026-07-06 4mMILLER24m(1)
|
||||
2026-07-07 4mMILLER24m(1)
|
||||
</pre>
|
||||
|
|
|
|||
|
|
@ -1605,6 +1605,7 @@
|
|||
kurtosis Compute sample kurtosis of specified fields
|
||||
min Compute minimum values of specified fields
|
||||
max Compute maximum values of specified fields
|
||||
rank Compute rank 1,2,2,4,... of specified fields, assuming input is sorted by that field; use with -s
|
||||
minlen Compute minimum string-lengths of specified fields
|
||||
maxlen Compute maximum string-lengths of specified fields
|
||||
|
||||
|
|
@ -2181,6 +2182,7 @@
|
|||
kurtosis Compute sample kurtosis of specified fields
|
||||
min Compute minimum values of specified fields
|
||||
max Compute maximum values of specified fields
|
||||
rank Compute rank 1,2,2,4,... of specified fields, assuming input is sorted by that field; use with -s
|
||||
minlen Compute minimum string-lengths of specified fields
|
||||
maxlen Compute maximum string-lengths of specified fields
|
||||
Example: mlr stats1 -a min,p10,p50,p90,max -f value -g size,shape
|
||||
|
|
@ -2200,6 +2202,10 @@
|
|||
* count and mode allow text input; the rest require numeric input.
|
||||
In particular, 1 and 1.0 are distinct text for count and mode.
|
||||
* When there are mode ties, the first-encountered datum wins.
|
||||
* rank assumes the input is already sorted (e.g. by mlr sort) on the field
|
||||
being ranked, and assigns standard competition rank (1,2,2,4,...) by
|
||||
comparing each record's value to the immediately preceding one. Use with
|
||||
-s to get a rank on every input record.
|
||||
|
||||
1mstats20m
|
||||
Usage: mlr stats2 [options]
|
||||
|
|
@ -4041,4 +4047,4 @@
|
|||
MIME Type for Comma-Separated Values (CSV) Files, the Miller docsite
|
||||
https://miller.readthedocs.io
|
||||
|
||||
2026-07-06 4mMILLER24m(1)
|
||||
2026-07-07 4mMILLER24m(1)
|
||||
|
|
|
|||
|
|
@ -2317,6 +2317,7 @@ Accumulators for -a:
|
|||
kurtosis Compute sample kurtosis of specified fields
|
||||
min Compute minimum values of specified fields
|
||||
max Compute maximum values of specified fields
|
||||
rank Compute rank 1,2,2,4,... of specified fields, assuming input is sorted by that field; use with -s
|
||||
minlen Compute minimum string-lengths of specified fields
|
||||
maxlen Compute maximum string-lengths of specified fields
|
||||
|
||||
|
|
@ -3568,6 +3569,7 @@ Names of accumulators for -a, one or more of:
|
|||
kurtosis Compute sample kurtosis of specified fields
|
||||
min Compute minimum values of specified fields
|
||||
max Compute maximum values of specified fields
|
||||
rank Compute rank 1,2,2,4,... of specified fields, assuming input is sorted by that field; use with -s
|
||||
minlen Compute minimum string-lengths of specified fields
|
||||
maxlen Compute maximum string-lengths of specified fields
|
||||
Example: mlr stats1 -a min,p10,p50,p90,max -f value -g size,shape
|
||||
|
|
@ -3587,6 +3589,10 @@ Notes:
|
|||
* count and mode allow text input; the rest require numeric input.
|
||||
In particular, 1 and 1.0 are distinct text for count and mode.
|
||||
* When there are mode ties, the first-encountered datum wins.
|
||||
* rank assumes the input is already sorted (e.g. by mlr sort) on the field
|
||||
being ranked, and assigns standard competition rank (1,2,2,4,...) by
|
||||
comparing each record's value to the immediately preceding one. Use with
|
||||
-s to get a rank on every input record.
|
||||
</pre>
|
||||
|
||||
These are simple univariate statistics on one or more number-valued fields
|
||||
|
|
|
|||
|
|
@ -1605,6 +1605,7 @@
|
|||
kurtosis Compute sample kurtosis of specified fields
|
||||
min Compute minimum values of specified fields
|
||||
max Compute maximum values of specified fields
|
||||
rank Compute rank 1,2,2,4,... of specified fields, assuming input is sorted by that field; use with -s
|
||||
minlen Compute minimum string-lengths of specified fields
|
||||
maxlen Compute maximum string-lengths of specified fields
|
||||
|
||||
|
|
@ -2181,6 +2182,7 @@
|
|||
kurtosis Compute sample kurtosis of specified fields
|
||||
min Compute minimum values of specified fields
|
||||
max Compute maximum values of specified fields
|
||||
rank Compute rank 1,2,2,4,... of specified fields, assuming input is sorted by that field; use with -s
|
||||
minlen Compute minimum string-lengths of specified fields
|
||||
maxlen Compute maximum string-lengths of specified fields
|
||||
Example: mlr stats1 -a min,p10,p50,p90,max -f value -g size,shape
|
||||
|
|
@ -2200,6 +2202,10 @@
|
|||
* count and mode allow text input; the rest require numeric input.
|
||||
In particular, 1 and 1.0 are distinct text for count and mode.
|
||||
* When there are mode ties, the first-encountered datum wins.
|
||||
* rank assumes the input is already sorted (e.g. by mlr sort) on the field
|
||||
being ranked, and assigns standard competition rank (1,2,2,4,...) by
|
||||
comparing each record's value to the immediately preceding one. Use with
|
||||
-s to get a rank on every input record.
|
||||
|
||||
1mstats20m
|
||||
Usage: mlr stats2 [options]
|
||||
|
|
@ -4041,4 +4047,4 @@
|
|||
MIME Type for Comma-Separated Values (CSV) Files, the Miller docsite
|
||||
https://miller.readthedocs.io
|
||||
|
||||
2026-07-06 4mMILLER24m(1)
|
||||
2026-07-07 4mMILLER24m(1)
|
||||
|
|
|
|||
10
man/mlr.1
10
man/mlr.1
|
|
@ -2,12 +2,12 @@
|
|||
.\" Title: mlr
|
||||
.\" Author: [see the "AUTHOR" section]
|
||||
.\" Generator: ./mkman.rb
|
||||
.\" Date: 2026-07-06
|
||||
.\" Date: 2026-07-07
|
||||
.\" Manual: \ \&
|
||||
.\" Source: \ \&
|
||||
.\" Language: English
|
||||
.\"
|
||||
.TH "MILLER" "1" "2026-07-06" "\ \&" "\ \&"
|
||||
.TH "MILLER" "1" "2026-07-07" "\ \&" "\ \&"
|
||||
.\" -----------------------------------------------------------------
|
||||
.\" * Portability definitions
|
||||
.\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
|
@ -1997,6 +1997,7 @@ Accumulators for -a:
|
|||
kurtosis Compute sample kurtosis of specified fields
|
||||
min Compute minimum values of specified fields
|
||||
max Compute maximum values of specified fields
|
||||
rank Compute rank 1,2,2,4,... of specified fields, assuming input is sorted by that field; use with -s
|
||||
minlen Compute minimum string-lengths of specified fields
|
||||
maxlen Compute maximum string-lengths of specified fields
|
||||
|
||||
|
|
@ -2705,6 +2706,7 @@ Names of accumulators for -a, one or more of:
|
|||
kurtosis Compute sample kurtosis of specified fields
|
||||
min Compute minimum values of specified fields
|
||||
max Compute maximum values of specified fields
|
||||
rank Compute rank 1,2,2,4,... of specified fields, assuming input is sorted by that field; use with -s
|
||||
minlen Compute minimum string-lengths of specified fields
|
||||
maxlen Compute maximum string-lengths of specified fields
|
||||
Example: mlr stats1 -a min,p10,p50,p90,max -f value -g size,shape
|
||||
|
|
@ -2724,6 +2726,10 @@ Notes:
|
|||
* count and mode allow text input; the rest require numeric input.
|
||||
In particular, 1 and 1.0 are distinct text for count and mode.
|
||||
* When there are mode ties, the first-encountered datum wins.
|
||||
* rank assumes the input is already sorted (e.g. by mlr sort) on the field
|
||||
being ranked, and assigns standard competition rank (1,2,2,4,...) by
|
||||
comparing each record's value to the immediately preceding one. Use with
|
||||
-s to get a rank on every input record.
|
||||
.fi
|
||||
.if n \{\
|
||||
.RE
|
||||
|
|
|
|||
|
|
@ -18,7 +18,7 @@ import (
|
|||
const verbNameStats1 = "stats1"
|
||||
|
||||
var stats1Options = []OptionSpec{
|
||||
{Flag: "-a", Arg: "{sum,count,...}", Type: "enum", Desc: "Names of accumulators: one or more of the listed values. Also accepts median (same as p50) and percentiles p{n} for n in 0..100, e.g. p10 p25.2 p50 p98 p100.", Values: []string{"count", "null_count", "distinct_count", "mode", "antimode", "sum", "mean", "mad", "var", "stddev", "meaneb", "skewness", "kurtosis", "min", "max", "minlen", "maxlen"}},
|
||||
{Flag: "-a", Arg: "{sum,count,...}", Type: "enum", Desc: "Names of accumulators: one or more of the listed values. Also accepts median (same as p50) and percentiles p{n} for n in 0..100, e.g. p10 p25.2 p50 p98 p100.", Values: []string{"count", "null_count", "distinct_count", "mode", "antimode", "sum", "mean", "mad", "var", "stddev", "meaneb", "skewness", "kurtosis", "min", "max", "rank", "minlen", "maxlen"}},
|
||||
{Flag: "-f", Arg: "{a,b,c}", Type: "csv-list", Desc: "Value-field names on which to compute statistics."},
|
||||
{Flag: "--fr", Arg: "{regex}", Type: "regex", Desc: "Regex for value-field names on which to compute statistics (compute statistics on values in all field names matching the regex)."},
|
||||
{Flag: "--fx", Arg: "{regex}", Type: "regex", Desc: "Inverted regex for value-field names on which to compute statistics (compute statistics on values in all field names not matching the regex)."},
|
||||
|
|
@ -79,6 +79,10 @@ the input record stream.
|
|||
* count and mode allow text input; the rest require numeric input.
|
||||
In particular, 1 and 1.0 are distinct text for count and mode.
|
||||
* When there are mode ties, the first-encountered datum wins.
|
||||
* rank assumes the input is already sorted (e.g. by mlr sort) on the field
|
||||
being ranked, and assigns standard competition rank (1,2,2,4,...) by
|
||||
comparing each record's value to the immediately preceding one. Use with
|
||||
-s to get a rank on every input record.
|
||||
`)
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -111,6 +111,12 @@ var stats1AccumulatorInfos []stats1AccumulatorInfo = []stats1AccumulatorInfo{
|
|||
NewStats1MaxAccumulator,
|
||||
},
|
||||
|
||||
{
|
||||
"rank",
|
||||
"Compute rank 1,2,2,4,... of specified fields, assuming input is sorted by that field; use with -s",
|
||||
NewStats1RankAccumulator,
|
||||
},
|
||||
|
||||
{
|
||||
"minlen",
|
||||
"Compute minimum string-lengths of specified fields",
|
||||
|
|
@ -533,6 +539,48 @@ func (acc *Stats1MeanAbsDevAccumulator) Reset() {
|
|||
acc.samples = make([]*mlrval.Mlrval, 0, 1000)
|
||||
}
|
||||
|
||||
// Stats1RankAccumulator implements standard competition ranking (1,2,2,4,...)
|
||||
// on a sequence of values, e.g. as produced by 'mlr sort'. This assumes
|
||||
// same-valued items are adjacent in the input stream: it compares each
|
||||
// ingested value only to the immediately preceding one. This accumulator is
|
||||
// most useful with 'stats1 -s' so that a rank is emitted for every input
|
||||
// record, rather than only once at end of stream.
|
||||
type Stats1RankAccumulator struct {
|
||||
count int64
|
||||
rank int64
|
||||
havePreviousValue bool
|
||||
previousValueString string
|
||||
}
|
||||
|
||||
func NewStats1RankAccumulator() IStats1Accumulator {
|
||||
return &Stats1RankAccumulator{
|
||||
count: 0,
|
||||
rank: 0,
|
||||
havePreviousValue: false,
|
||||
}
|
||||
}
|
||||
func (acc *Stats1RankAccumulator) Ingest(value *mlrval.Mlrval) {
|
||||
acc.count++
|
||||
valueString := value.String() // 1, 1.0, and 1.000 are distinct
|
||||
if !acc.havePreviousValue || valueString != acc.previousValueString {
|
||||
acc.rank = acc.count
|
||||
acc.previousValueString = valueString
|
||||
acc.havePreviousValue = true
|
||||
}
|
||||
}
|
||||
func (acc *Stats1RankAccumulator) Emit() *mlrval.Mlrval {
|
||||
if acc.count == 0 {
|
||||
return mlrval.VOID
|
||||
}
|
||||
return mlrval.FromInt(acc.rank)
|
||||
}
|
||||
func (acc *Stats1RankAccumulator) Reset() {
|
||||
acc.count = 0
|
||||
acc.rank = 0
|
||||
acc.havePreviousValue = false
|
||||
acc.previousValueString = ""
|
||||
}
|
||||
|
||||
type Stats1MinAccumulator struct {
|
||||
min *mlrval.Mlrval
|
||||
}
|
||||
|
|
|
|||
|
|
@ -645,6 +645,7 @@ Accumulators for -a:
|
|||
kurtosis Compute sample kurtosis of specified fields
|
||||
min Compute minimum values of specified fields
|
||||
max Compute maximum values of specified fields
|
||||
rank Compute rank 1,2,2,4,... of specified fields, assuming input is sorted by that field; use with -s
|
||||
minlen Compute minimum string-lengths of specified fields
|
||||
maxlen Compute maximum string-lengths of specified fields
|
||||
|
||||
|
|
@ -1243,6 +1244,7 @@ Names of accumulators for -a, one or more of:
|
|||
kurtosis Compute sample kurtosis of specified fields
|
||||
min Compute minimum values of specified fields
|
||||
max Compute maximum values of specified fields
|
||||
rank Compute rank 1,2,2,4,... of specified fields, assuming input is sorted by that field; use with -s
|
||||
minlen Compute minimum string-lengths of specified fields
|
||||
maxlen Compute maximum string-lengths of specified fields
|
||||
Example: mlr stats1 -a min,p10,p50,p90,max -f value -g size,shape
|
||||
|
|
@ -1262,6 +1264,10 @@ Notes:
|
|||
* count and mode allow text input; the rest require numeric input.
|
||||
In particular, 1 and 1.0 are distinct text for count and mode.
|
||||
* When there are mode ties, the first-encountered datum wins.
|
||||
* rank assumes the input is already sorted (e.g. by mlr sort) on the field
|
||||
being ranked, and assigns standard competition rank (1,2,2,4,...) by
|
||||
comparing each record's value to the immediately preceding one. Use with
|
||||
-s to get a rank on every input record.
|
||||
|
||||
================================================================
|
||||
stats2
|
||||
|
|
|
|||
1
test/cases/verb-stats1/0020/cmd
Normal file
1
test/cases/verb-stats1/0020/cmd
Normal file
|
|
@ -0,0 +1 @@
|
|||
mlr --icsv --ocsv stats1 -a rank -f x -s test/input/rank-data.csv
|
||||
0
test/cases/verb-stats1/0020/experr
Normal file
0
test/cases/verb-stats1/0020/experr
Normal file
5
test/cases/verb-stats1/0020/expout
Normal file
5
test/cases/verb-stats1/0020/expout
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
x,x_rank
|
||||
10,1
|
||||
20,2
|
||||
20,2
|
||||
30,4
|
||||
1
test/cases/verb-stats1/0021/cmd
Normal file
1
test/cases/verb-stats1/0021/cmd
Normal file
|
|
@ -0,0 +1 @@
|
|||
mlr --icsv --ocsv stats1 -a rank -f x -g g -s test/input/rank-data-grouped.csv
|
||||
0
test/cases/verb-stats1/0021/experr
Normal file
0
test/cases/verb-stats1/0021/experr
Normal file
6
test/cases/verb-stats1/0021/expout
Normal file
6
test/cases/verb-stats1/0021/expout
Normal file
|
|
@ -0,0 +1,6 @@
|
|||
g,x,x_rank
|
||||
a,10,1
|
||||
a,10,1
|
||||
a,20,3
|
||||
b,5,1
|
||||
b,5,1
|
||||
6
test/input/rank-data-grouped.csv
Normal file
6
test/input/rank-data-grouped.csv
Normal file
|
|
@ -0,0 +1,6 @@
|
|||
g,x
|
||||
a,10
|
||||
a,10
|
||||
a,20
|
||||
b,5
|
||||
b,5
|
||||
|
5
test/input/rank-data.csv
Normal file
5
test/input/rank-data.csv
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
x
|
||||
10
|
||||
20
|
||||
20
|
||||
30
|
||||
|
Loading…
Add table
Add a link
Reference in a new issue