Mirrors/miller
1
0
Fork 0
mirror of https://github.com/johnkerl/miller.git synced 2026-07-17 16:38:54 +00:00
Code Issues Projects Releases Wiki Activity

Add sliding-window mode to mlr stats1 via -w {n} (#1017) (#2186)

Browse source 
* Add sliding-window mode to stats1 via -w {n} (#1017)

This adds a generic sliding-window option to the stats1 verb, per the
feature request in #1017. With 'mlr stats1 ... -w {n}', statistics are
computed over a trailing window of up to n records -- per group when -g
is used -- and one output record is emitted per input record, with the
windowed statistics appended to it.

The implementation retains, per grouping key, copies of only the
relevant value fields from the last up-to-n records. On each input
record the group's accumulators are reset and re-fed from the window.
This is O(window size) per record, but is fully generic: it works
uniformly for every stats1 accumulator, including order-sensitive ones
(mode, antimode) and non-invertible ones (percentiles, distinct_count),
and composes with -g, --fr/--fx, and --gr/--gx.

Includes unit tests, regression cases (test/cases/verb-stats1/0020-0025),
and regenerated docs/man/help-text artifacts.

Addresses #1017.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* neaten

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
John Kerl 2026-07-14 10:24:41 -04:00 • committed by GitHub
parent 806c51ebeb
commit 10939349d2
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
28 changed files with 564 additions and 6 deletions
Download patch file Download diff file Expand all files Collapse all files

11
docs/src/manpage.md
View file

@ -2223,6 +2223,12 @@ This is simply a copy of what you should see on running `man mlr` at a command p
case please avoid pprint-format output since end of input
stream will never be seen. Likewise, if input is coming from
`tail -f` be sure to use `--records-per-batch 1`.
-w {n} Sliding-window mode: compute statistics over a trailing
window of up to n records (including the current one), rather
than over the whole record stream. Windows are kept per group
when -g is used. One output record is emitted per input
record, with the windowed statistics appended to it. Not
compatible with -s.
-S No-op flag for backward compatibility with Miller 5.
-F No-op flag for backward compatibility with Miller 5.
-h|--help Show this message.
@ -2249,6 +2255,9 @@ This is simply a copy of what you should see on running `man mlr` at a command p
Example: mlr stats1 -a min,p10,p50,p90,max -f value -g size,shape
Example: mlr stats1 -a count,mode -f size
Example: mlr stats1 -a count,mode -f size -g shape
Example: mlr stats1 -a mean,min,max -f quantity -g name -w 7
This emits one output record per input record, with sliding-window
statistics over the last up-to-7 records for each name.
Example: mlr stats1 -a count,mode --fr '^[a-h].*$' --gr '^k.*$'
This computes count and mode statistics on all field names beginning
with a through h, grouped by all field names starting with k.
@ -4110,5 +4119,5 @@ This is simply a copy of what you should see on running `man mlr` at a command p
MIME Type for Comma-Separated Values (CSV) Files, the Miller docsite
https://miller.readthedocs.io
2026-07-08 4mMILLER24m(1)
2026-07-14 4mMILLER24m(1)
</pre>

11
docs/src/manpage.txt
View file

@ -2202,6 +2202,12 @@
case please avoid pprint-format output since end of input
stream will never be seen. Likewise, if input is coming from
`tail -f` be sure to use `--records-per-batch 1`.
-w {n} Sliding-window mode: compute statistics over a trailing
window of up to n records (including the current one), rather
than over the whole record stream. Windows are kept per group
when -g is used. One output record is emitted per input
record, with the windowed statistics appended to it. Not
compatible with -s.
-S No-op flag for backward compatibility with Miller 5.
-F No-op flag for backward compatibility with Miller 5.
-h|--help Show this message.
@ -2228,6 +2234,9 @@
Example: mlr stats1 -a min,p10,p50,p90,max -f value -g size,shape
Example: mlr stats1 -a count,mode -f size
Example: mlr stats1 -a count,mode -f size -g shape
Example: mlr stats1 -a mean,min,max -f quantity -g name -w 7
This emits one output record per input record, with sliding-window
statistics over the last up-to-7 records for each name.
Example: mlr stats1 -a count,mode --fr '^[a-h].*$' --gr '^k.*$'
This computes count and mode statistics on all field names beginning
with a through h, grouped by all field names starting with k.
@ -4089,4 +4098,4 @@
MIME Type for Comma-Separated Values (CSV) Files, the Miller docsite
https://miller.readthedocs.io
2026-07-08 4mMILLER24m(1)
2026-07-14 4mMILLER24m(1)

47
docs/src/reference-verbs.md
View file

@ -3697,6 +3697,12 @@ Options:
case please avoid pprint-format output since end of input
stream will never be seen. Likewise, if input is coming from
`tail -f` be sure to use `--records-per-batch 1`.
-w {n} Sliding-window mode: compute statistics over a trailing
window of up to n records (including the current one), rather
than over the whole record stream. Windows are kept per group
when -g is used. One output record is emitted per input
record, with the windowed statistics appended to it. Not
compatible with -s.
-S No-op flag for backward compatibility with Miller 5.
-F No-op flag for backward compatibility with Miller 5.
-h|--help Show this message.
@ -3723,6 +3729,9 @@ Names of accumulators for -a, one or more of:
Example: mlr stats1 -a min,p10,p50,p90,max -f value -g size,shape
Example: mlr stats1 -a count,mode -f size
Example: mlr stats1 -a count,mode -f size -g shape
Example: mlr stats1 -a mean,min,max -f quantity -g name -w 7
This emits one output record per input record, with sliding-window
statistics over the last up-to-7 records for each name.
Example: mlr stats1 -a count,mode --fr '^[a-h].*$' --gr '^k.*$'
This computes count and mode statistics on all field names beginning
with a through h, grouped by all field names starting with k.
@ -3812,6 +3821,44 @@ square red
circle red
</pre>
With `-w {n}`, statistics are computed over a sliding window of the last up-to-`n`
records -- within each group, when `-g` is used -- and one output record is emitted
per input record, with the windowed statistics appended to it:
<pre class="pre-highlight-in-pair">
<b>mlr --icsv --opprint --from example.csv stats1 -a mean,min,max -f quantity -w 4</b>
</pre>
<pre class="pre-non-highlight-in-pair">
color shape flag k index quantity rate quantity_mean quantity_min quantity_max
yellow triangle true 1 11 43.6498 9.8870 43.6498 43.6498 43.6498
red square true 2 15 79.2778 0.0130 61.4638 43.6498 79.2778
red circle true 3 16 13.8103 2.9010 45.579299999999996 13.8103 79.2778
red square false 4 48 77.5542 7.4670 53.573025 13.8103 79.2778
purple triangle false 5 51 81.2290 8.5910 62.96782499999999 13.8103 81.229
red square false 6 64 77.1991 9.5310 62.44815 13.8103 81.229
purple triangle false 7 65 80.1405 5.8240 79.0307 77.1991 81.229
yellow circle true 8 73 63.9785 4.2370 75.636775 63.9785 81.229
yellow circle true 9 87 63.5058 8.3350 71.20597500000001 63.5058 80.1405
purple square false 10 91 72.3735 8.2430 69.999575 63.5058 80.1405
</pre>
<pre class="pre-highlight-in-pair">
<b>mlr --icsv --opprint --from example.csv stats1 -a mean -f quantity -g shape -w 2</b>
</pre>
<pre class="pre-non-highlight-in-pair">
color shape flag k index quantity rate quantity_mean
yellow triangle true 1 11 43.6498 9.8870 43.6498
red square true 2 15 79.2778 0.0130 79.2778
red circle true 3 16 13.8103 2.9010 13.8103
red square false 4 48 77.5542 7.4670 78.416
purple triangle false 5 51 81.2290 8.5910 62.4394
red square false 6 64 77.1991 9.5310 77.37665
purple triangle false 7 65 80.1405 5.8240 80.68475000000001
yellow circle true 8 73 63.9785 4.2370 38.8944
yellow circle true 9 87 63.5058 8.3350 63.742149999999995
purple square false 10 91 72.3735 8.2430 74.78630000000001
</pre>
## stats2
<pre class="pre-highlight-in-pair">

12
docs/src/reference-verbs.md.in
View file

@ -1132,6 +1132,18 @@ GENMD-RUN-COMMAND
mlr --c2p stats1 -a mode -f color -g shape data/colored-shapes.csv
GENMD-EOF
With `-w {n}`, statistics are computed over a sliding window of the last up-to-`n`
records -- within each group, when `-g` is used -- and one output record is emitted
per input record, with the windowed statistics appended to it:
GENMD-RUN-COMMAND
mlr --icsv --opprint --from example.csv stats1 -a mean,min,max -f quantity -w 4
GENMD-EOF
GENMD-RUN-COMMAND
mlr --icsv --opprint --from example.csv stats1 -a mean -f quantity -g shape -w 2
GENMD-EOF
## stats2
GENMD-RUN-COMMAND

11
man/manpage.txt
View file

@ -2202,6 +2202,12 @@
case please avoid pprint-format output since end of input
stream will never be seen. Likewise, if input is coming from
`tail -f` be sure to use `--records-per-batch 1`.
-w {n} Sliding-window mode: compute statistics over a trailing
window of up to n records (including the current one), rather
than over the whole record stream. Windows are kept per group
when -g is used. One output record is emitted per input
record, with the windowed statistics appended to it. Not
compatible with -s.
-S No-op flag for backward compatibility with Miller 5.
-F No-op flag for backward compatibility with Miller 5.
-h|--help Show this message.
@ -2228,6 +2234,9 @@
Example: mlr stats1 -a min,p10,p50,p90,max -f value -g size,shape
Example: mlr stats1 -a count,mode -f size
Example: mlr stats1 -a count,mode -f size -g shape
Example: mlr stats1 -a mean,min,max -f quantity -g name -w 7
This emits one output record per input record, with sliding-window
statistics over the last up-to-7 records for each name.
Example: mlr stats1 -a count,mode --fr '^[a-h].*$' --gr '^k.*$'
This computes count and mode statistics on all field names beginning
with a through h, grouped by all field names starting with k.
@ -4089,4 +4098,4 @@
MIME Type for Comma-Separated Values (CSV) Files, the Miller docsite
https://miller.readthedocs.io
2026-07-08 4mMILLER24m(1)
2026-07-14 4mMILLER24m(1)

13
man/mlr.1
View file

@ -2,12 +2,12 @@
.\" Title: mlr
.\" Author: [see the "AUTHOR" section]
.\" Generator: ./mkman.rb
.\" Date: 2026-07-08
.\" Date: 2026-07-14
.\" Manual: \ \&
.\" Source: \ \&
.\" Language: English
.\"
.TH "MILLER" "1" "2026-07-08" "\ \&" "\ \&"
.TH "MILLER" "1" "2026-07-14" "\ \&" "\ \&"
.\" -----------------------------------------------------------------
.\" * Portability definitions
.\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@ -2738,6 +2738,12 @@ Options:
case please avoid pprint-format output since end of input
stream will never be seen. Likewise, if input is coming from
`tail -f` be sure to use `--records-per-batch 1`.
-w {n} Sliding-window mode: compute statistics over a trailing
window of up to n records (including the current one), rather
than over the whole record stream. Windows are kept per group
when -g is used. One output record is emitted per input
record, with the windowed statistics appended to it. Not
compatible with -s.
-S No-op flag for backward compatibility with Miller 5.
-F No-op flag for backward compatibility with Miller 5.
-h|--help Show this message.
@ -2764,6 +2770,9 @@ Names of accumulators for -a, one or more of:
Example: mlr stats1 -a min,p10,p50,p90,max -f value -g size,shape
Example: mlr stats1 -a count,mode -f size
Example: mlr stats1 -a count,mode -f size -g shape
Example: mlr stats1 -a mean,min,max -f quantity -g name -w 7
This emits one output record per input record, with sliding-window
statistics over the last up-to-7 records for each name.
Example: mlr stats1 -a count,mode --fr '^[a-h].*$' --gr '^k.*$'
This computes count and mode statistics on all field names beginning
with a through h, grouped by all field names starting with k.

143
pkg/transformers/stats1.go
View file

@ -28,6 +28,7 @@ var stats1Options = []OptionSpec{
{Flag: "--grfx", Arg: "{regex}", Type: "regex", Desc: "Shorthand for --gr {regex} --fx {that same regex}."},
{Flag: "-i", Type: "bool", Desc: "Use interpolated percentiles, like R's type=7; default like type=1. Not sensical for string-valued fields."},
{Flag: "-s", Type: "bool", Desc: "Print iterative stats. Useful in tail -f contexts, in which case please avoid pprint-format output since end of input stream will never be seen. Likewise, if input is coming from `tail -f` be sure to use `--records-per-batch 1`."},
{Flag: "-w", Arg: "{n}", Type: "int", Desc: "Sliding-window mode: compute statistics over a trailing window of up to n records (including the current one), rather than over the whole record stream. Windows are kept per group when -g is used. One output record is emitted per input record, with the windowed statistics appended to it. Not compatible with -s."},
{Flag: "-S", Type: "bool", Desc: "No-op flag for backward compatibility with Miller 5."},
{Flag: "-F", Type: "bool", Desc: "No-op flag for backward compatibility with Miller 5."},
}
@ -62,6 +63,11 @@ the input record stream.
"Example: mlr stats1 -a count,mode -f size")
fmt.Fprintln(o,
"Example: mlr stats1 -a count,mode -f size -g shape")
fmt.Fprintln(o,
"Example: mlr stats1 -a mean,min,max -f quantity -g name -w 7")
fmt.Fprintln(o,
` This emits one output record per input record, with sliding-window
statistics over the last up-to-7 records for each name.`)
fmt.Fprintln(o,
"Example: mlr stats1 -a count,mode --fr '^[a-h].*$' --gr '^k.*$'")
fmt.Fprintln(o,
@ -106,6 +112,7 @@ func transformerStats1ParseCLI(
doInterpolatedPercentiles := false
doIterativeStats := false
slidingWindowSize := int64(0)
var err error
for argi < argc /* variable increment: 1 or 2 depending on flag */ {
@ -185,6 +192,15 @@ func transformerStats1ParseCLI(
case "-s":
doIterativeStats = true
case "-w":
slidingWindowSize, err = cli.VerbGetIntArg(verb, opt, args, &argi, argc)
if err != nil {
return nil, err
}
if slidingWindowSize < 1 {
return nil, cli.VerbErrorf(verbNameStats1, "-w argument must be positive; got %d", slidingWindowSize)
}
case "-S", "-F":
// No-op pass-through for backward compatibility with Miller 5
@ -200,6 +216,9 @@ func transformerStats1ParseCLI(
if len(valueFieldNameList) == 0 {
return nil, cli.VerbErrorf(verbNameStats1, "-f option is required")
}
if doIterativeStats && slidingWindowSize > 0 {
return nil, cli.VerbErrorf(verbNameStats1, "-s and -w may not be used together")
}
*pargi = argi
if !doConstruct { // All transformers must do this for main command-line parsing
@ -218,6 +237,7 @@ func transformerStats1ParseCLI(
doInterpolatedPercentiles,
doIterativeStats,
slidingWindowSize,
)
if err != nil {
return nil, err
@ -250,9 +270,19 @@ type TransformerStats1 struct {
doInterpolatedPercentiles bool
doIterativeStats bool
// If positive, statistics are computed over a trailing window of up to
// this many records (per grouping key), with one output record emitted per
// input record.
slidingWindowSize int64
// State:
accumulatorFactory *utils.Stats1AccumulatorFactory
// For sliding-window mode: per grouping key, the last (up to)
// slidingWindowSize windowed entries. Each entry is a small record holding
// copies of only the relevant value fields from one input record.
slidingWindows map[string][]*mlrval.Mlrmap
// Accumulators are indexed by
// groupByFieldName -> valueFieldName -> accumulatorName -> accumulator object
// This would be
@ -319,6 +349,7 @@ func NewTransformerStats1(
doInterpolatedPercentiles bool,
doIterativeStats bool,
slidingWindowSize int64,
) (*TransformerStats1, error) {
for _, name := range accumulatorNameList {
if !utils.ValidateStats1AccumulatorName(name) {
@ -339,7 +370,9 @@ func NewTransformerStats1(
doInterpolatedPercentiles: doInterpolatedPercentiles,
doIterativeStats: doIterativeStats,
slidingWindowSize: slidingWindowSize,
accumulatorFactory: utils.NewStats1AccumulatorFactory(),
slidingWindows: make(map[string][]*mlrval.Mlrmap),
namedAccumulators: lib.NewOrderedMap[*lib.OrderedMap[*lib.OrderedMap[*utils.Stats1NamedAccumulator]]](),
groupingKeysToGroupByFieldValues: make(map[string]*lib.OrderedMap[*mlrval.Mlrval]),
}
@ -378,6 +411,17 @@ func (tr *TransformerStats1) Transform(
func (tr *TransformerStats1) handleInputRecord(
inrecAndContext *types.RecordAndContext,
outputRecordsAndContexts *[]*types.RecordAndContext, // list of *types.RecordAndContext
) {
if tr.slidingWindowSize > 0 {
tr.handleInputRecordWindowed(inrecAndContext, outputRecordsAndContexts)
} else {
tr.handleInputRecordNonWindowed(inrecAndContext, outputRecordsAndContexts)
}
}
func (tr *TransformerStats1) handleInputRecordNonWindowed(
inrecAndContext *types.RecordAndContext,
outputRecordsAndContexts *[]*types.RecordAndContext, // list of *types.RecordAndContext
) {
inrec := inrecAndContext.Record
@ -430,6 +474,103 @@ func (tr *TransformerStats1) handleInputRecord(
}
}
// handleInputRecordWindowed processes one input record in sliding-window mode
// (-w {n}). For each grouping key we retain the relevant value fields of the
// last up-to-n records; on every input record the accumulators for that
// grouping key are reset and re-fed from the window, then the windowed
// statistics are appended to the record, which is emitted immediately.
func (tr *TransformerStats1) handleInputRecordWindowed(
inrecAndContext *types.RecordAndContext,
outputRecordsAndContexts *[]*types.RecordAndContext, // list of *types.RecordAndContext
) {
inrec := inrecAndContext.Record
// E.g. if grouping by "a" and "b", and the current record has a=circle, b=blue,
// then groupingKey is the string "circle,blue".
var groupingKey string
var groupByFieldValues *lib.OrderedMap[*mlrval.Mlrval] // OrderedMap[string]*mlrval.Mlrval
var ok bool
if tr.doRegexGroupByFieldNames {
groupingKey, groupByFieldValues, ok = tr.getGroupByFieldNamesWithRegexes(inrec)
} else {
groupingKey, ok = tr.getGroupingKeyWithoutRegexes(inrec)
}
if !ok {
return
}
level2 := tr.namedAccumulators.Get(groupingKey)
if level2 == nil {
level2 = lib.NewOrderedMap[*lib.OrderedMap[*utils.Stats1NamedAccumulator]]()
tr.namedAccumulators.Put(groupingKey, level2)
if !tr.doRegexGroupByFieldNames {
groupByFieldValues, ok = tr.buildGroupByFieldValuesWithoutRegexes(inrec)
if !ok {
return
}
}
tr.groupingKeysToGroupByFieldValues[groupingKey] = groupByFieldValues
} else if !tr.doRegexGroupByFieldNames {
groupByFieldValues = tr.groupingKeysToGroupByFieldValues[groupingKey]
}
// Update this grouping key's window with the current record's value fields.
window := tr.slidingWindows[groupingKey]
if int64(len(window)) >= tr.slidingWindowSize {
window = window[1:]
}
window = append(window, tr.buildWindowEntry(inrec))
tr.slidingWindows[groupingKey] = window
// Reset the accumulators for this grouping key, then re-ingest the values
// retained in the window. This is O(window size) per record but is fully
// generic: it works for any accumulator, including order-sensitive ones
// like mode/antimode and non-invertible ones like percentiles.
for pb := level2.Head; pb != nil; pb = pb.Next {
for pc := pb.Value.Head; pc != nil; pc = pc.Next {
pc.Value.Reset()
}
}
for _, windowEntry := range window {
if tr.doRegexValueFieldNames {
tr.ingestWithValueFieldRegexes(windowEntry, groupingKey, level2)
} else {
tr.ingestWithoutValueFieldRegexes(windowEntry, groupingKey, level2)
}
}
tr.emitIntoOutputRecord(
inrec,
groupByFieldValues,
level2,
inrec,
)
*outputRecordsAndContexts = append(*outputRecordsAndContexts, inrecAndContext)
}
// buildWindowEntry makes a small record holding copies of only the relevant
// value fields from the given input record, for retention in a sliding window.
func (tr *TransformerStats1) buildWindowEntry(
inrec *mlrval.Mlrmap,
) *mlrval.Mlrmap {
windowEntry := mlrval.NewMlrmapAsRecord()
if tr.doRegexValueFieldNames {
for pe := inrec.Head; pe != nil; pe = pe.Next {
if tr.matchValueFieldName(pe.Key) {
windowEntry.PutCopy(pe.Key, pe.Value)
}
}
} else {
for _, valueFieldName := range tr.valueFieldNameList {
valueFieldValue := inrec.Get(valueFieldName)
if valueFieldValue != nil {
windowEntry.PutCopy(valueFieldName, valueFieldValue)
}
}
}
return windowEntry
}
// E.g. if grouping by "a" and "b", and the current record has a=circle,
// b=blue, then groupingKey is the string "circle,blue". For grouping without
// regexed group-by field names, the group-by field names/values are the same
@ -610,7 +751,7 @@ func (tr *TransformerStats1) handleEndOfRecordStream(
inrecAndContext *types.RecordAndContext,
outputRecordsAndContexts *[]*types.RecordAndContext, // list of *types.RecordAndContext
) {
if tr.doIterativeStats {
if tr.doIterativeStats || tr.slidingWindowSize > 0 {
*outputRecordsAndContexts = append(*outputRecordsAndContexts, inrecAndContext) // end-of-stream marker
return
}

150
pkg/transformers/stats1_test.go Normal file
View file

@ -0,0 +1,150 @@
package transformers
import (
"testing"
"github.com/johnkerl/miller/v6/pkg/mlrval"
"github.com/johnkerl/miller/v6/pkg/types"
)
// makeStats1TestRecord builds a record from alternating key/value pairs, e.g.
// makeStats1TestRecord("x", "1", "g", "a").
func makeStats1TestRecord(kvPairs ...string) *types.RecordAndContext {
record := mlrval.NewMlrmapAsRecord()
for i := 0; i < len(kvPairs); i += 2 {
record.PutReference(kvPairs[i], mlrval.FromInferredType(kvPairs[i+1]))
}
context := types.NewNilContext()
return types.NewRecordAndContext(record, context)
}
func runStats1TestTransformer(
tr RecordTransformer,
inputs []*types.RecordAndContext,
) []*types.RecordAndContext {
inputDownstreamDoneChannel := make(chan bool, 1)
outputDownstreamDoneChannel := make(chan bool, 1)
outputs := make([]*types.RecordAndContext, 0)
for _, input := range inputs {
tr.Transform(input, &outputs, inputDownstreamDoneChannel, outputDownstreamDoneChannel)
}
// End-of-stream marker
eos := types.NewRecordAndContext(nil, types.NewNilContext())
eos.EndOfStream = true
tr.Transform(eos, &outputs, inputDownstreamDoneChannel, outputDownstreamDoneChannel)
return outputs
}
// TestStats1SlidingWindow exercises 'mlr stats1 -a sum,mean,min,max -f x -w 3'.
func TestStats1SlidingWindow(t *testing.T) {
tr, err := NewTransformerStats1(
[]string{"sum", "mean", "min", "max"},
[]string{"x"},
[]string{}, // groupByFieldNameList
false, false, // doRegexValueFieldNames, doRegexGroupByFieldNames
false, false, // invertRegexValueFieldNames, invertRegexGroupByFieldNames
false, // doInterpolatedPercentiles
false, // doIterativeStats
3, // slidingWindowSize
)
if err != nil {
t.Fatal(err)
}
inputs := []*types.RecordAndContext{
makeStats1TestRecord("x", "1"),
makeStats1TestRecord("x", "2"),
makeStats1TestRecord("x", "3"),
makeStats1TestRecord("x", "4"),
makeStats1TestRecord("x", "5"),
}
outputs := runStats1TestTransformer(tr, inputs)
// 5 data records plus end-of-stream marker
if len(outputs) != 6 {
t.Fatalf("got %d outputs, want 6", len(outputs))
}
expectations := []struct{ sum, mean, min, max string }{
{"1", "1", "1", "1"}, // window [1]
{"3", "1.5", "1", "2"}, // window [1,2]
{"6", "2", "1", "3"}, // window [1,2,3]
{"9", "3", "2", "4"}, // window [2,3,4]
{"12", "4", "3", "5"}, // window [3,4,5]
}
for i, expectation := range expectations {
record := outputs[i].Record
for name, want := range map[string]string{
"x_sum": expectation.sum,
"x_mean": expectation.mean,
"x_min": expectation.min,
"x_max": expectation.max,
} {
value := record.Get(name)
if value == nil {
t.Fatalf("record %d: field %s missing", i, name)
}
if value.String() != want {
t.Errorf("record %d: %s got %s, want %s", i, name, value.String(), want)
}
}
}
if !outputs[5].EndOfStream {
t.Errorf("last output should be the end-of-stream marker")
}
}
// TestStats1SlidingWindowGrouped exercises 'mlr stats1 -a count,sum -f x -g g -w 2':
// windows are maintained per group.
func TestStats1SlidingWindowGrouped(t *testing.T) {
tr, err := NewTransformerStats1(
[]string{"count", "sum"},
[]string{"x"},
[]string{"g"}, // groupByFieldNameList
false, false, // doRegexValueFieldNames, doRegexGroupByFieldNames
false, false, // invertRegexValueFieldNames, invertRegexGroupByFieldNames
false, // doInterpolatedPercentiles
false, // doIterativeStats
2, // slidingWindowSize
)
if err != nil {
t.Fatal(err)
}
inputs := []*types.RecordAndContext{
makeStats1TestRecord("g", "a", "x", "1"),
makeStats1TestRecord("g", "b", "x", "10"),
makeStats1TestRecord("g", "a", "x", "2"),
makeStats1TestRecord("g", "b", "x", "20"),
makeStats1TestRecord("g", "a", "x", "3"),
}
outputs := runStats1TestTransformer(tr, inputs)
// 5 data records plus end-of-stream marker
if len(outputs) != 6 {
t.Fatalf("got %d outputs, want 6", len(outputs))
}
expectations := []struct{ count, sum string }{
{"1", "1"}, // group a, window [1]
{"1", "10"}, // group b, window [10]
{"2", "3"}, // group a, window [1,2]
{"2", "30"}, // group b, window [10,20]
{"2", "5"}, // group a, window [2,3]
}
for i, expectation := range expectations {
record := outputs[i].Record
count := record.Get("x_count")
sum := record.Get("x_sum")
if count == nil || sum == nil {
t.Fatalf("record %d: missing x_count or x_sum", i)
}
if count.String() != expectation.count {
t.Errorf("record %d: x_count got %s, want %s", i, count.String(), expectation.count)
}
if sum.String() != expectation.sum {
t.Errorf("record %d: x_sum got %s, want %s", i, sum.String(), expectation.sum)
}
}
}

9
test/cases/cli-help/0001/expout
View file

@ -1266,6 +1266,12 @@ Options:
case please avoid pprint-format output since end of input
stream will never be seen. Likewise, if input is coming from
`tail -f` be sure to use `--records-per-batch 1`.
-w {n} Sliding-window mode: compute statistics over a trailing
window of up to n records (including the current one), rather
than over the whole record stream. Windows are kept per group
when -g is used. One output record is emitted per input
record, with the windowed statistics appended to it. Not
compatible with -s.
-S No-op flag for backward compatibility with Miller 5.
-F No-op flag for backward compatibility with Miller 5.
-h|--help Show this message.
@ -1292,6 +1298,9 @@ Names of accumulators for -a, one or more of:
Example: mlr stats1 -a min,p10,p50,p90,max -f value -g size,shape
Example: mlr stats1 -a count,mode -f size
Example: mlr stats1 -a count,mode -f size -g shape
Example: mlr stats1 -a mean,min,max -f quantity -g name -w 7
This emits one output record per input record, with sliding-window
statistics over the last up-to-7 records for each name.
Example: mlr stats1 -a count,mode --fr '^[a-h].*$' --gr '^k.*$'
This computes count and mode statistics on all field names beginning
with a through h, grouped by all field names starting with k.

1
test/cases/verb-stats1/0020/cmd Normal file
View file

@ -0,0 +1 @@
mlr --opprint --from test/input/abixy stats1 -a mean,sum,min,max,count -f i,x -w 3

0
test/cases/verb-stats1/0020/experr Normal file
View file

11
test/cases/verb-stats1/0020/expout Normal file
View file

@ -0,0 +1,11 @@
a b i x y i_mean i_sum i_min i_max i_count x_mean x_sum x_min x_max x_count
pan pan 1 0.34679014 0.72680286 1 1 1 1 1 0.34679014 0.34679014 0.34679014 0.34679014 1
eks pan 2 0.75867996 0.52215111 1.50000000 3 1 2 2 0.55273505 1.10547011 0.34679014 0.75867996 2
wye wye 3 0.20460331 0.33831853 2 6 1 3 3 0.43669114 1.31007341 0.20460331 0.75867996 3
eks wye 4 0.38139939 0.13418874 3 9 2 4 3 0.44822755 1.34468266 0.20460331 0.75867996 3
wye pan 5 0.57328892 0.86362447 4 12 3 5 3 0.38643054 1.15929162 0.20460331 0.57328892 3
zee pan 6 0.52712616 0.49322129 5 15 4 6 3 0.49393816 1.48181447 0.38139939 0.57328892 3
eks zee 7 0.61178406 0.18788492 6 18 5 7 3 0.57073305 1.71219914 0.52712616 0.61178406 3
zee wye 8 0.59855401 0.97618139 7 21 6 8 3 0.57915474 1.73746423 0.52712616 0.61178406 3
hat wye 9 0.03144188 0.74955076 8 24 7 9 3 0.41392665 1.24177995 0.03144188 0.61178406 3
pan wye 10 0.50262601 0.95261836 9 27 8 10 3 0.37754063 1.13262189 0.03144188 0.59855401 3

1
test/cases/verb-stats1/0021/cmd Normal file
View file

@ -0,0 +1 @@
mlr --opprint --from test/input/abixy stats1 -a count,mean -f i -g a -w 2

0
test/cases/verb-stats1/0021/experr Normal file
View file

11
test/cases/verb-stats1/0021/expout Normal file
View file

@ -0,0 +1,11 @@
a b i x y i_count i_mean
pan pan 1 0.34679014 0.72680286 1 1
eks pan 2 0.75867996 0.52215111 1 2
wye wye 3 0.20460331 0.33831853 1 3
eks wye 4 0.38139939 0.13418874 2 3
wye pan 5 0.57328892 0.86362447 2 4
zee pan 6 0.52712616 0.49322129 1 6
eks zee 7 0.61178406 0.18788492 2 5.50000000
zee wye 8 0.59855401 0.97618139 2 7
hat wye 9 0.03144188 0.74955076 1 9
pan wye 10 0.50262601 0.95261836 2 5.50000000

1
test/cases/verb-stats1/0022/cmd Normal file
View file

@ -0,0 +1 @@
mlr --opprint --from test/input/abixy stats1 -a p25,median,p75,distinct_count,mode -f i -w 4

0
test/cases/verb-stats1/0022/experr Normal file
View file

11
test/cases/verb-stats1/0022/expout Normal file
View file

@ -0,0 +1,11 @@
a b i x y i_p25 i_median i_p75 i_distinct_count i_mode
pan pan 1 0.34679014 0.72680286 1 1 1 1 1
eks pan 2 0.75867996 0.52215111 1 2 2 2 1
wye wye 3 0.20460331 0.33831853 1 2 3 3 1
eks wye 4 0.38139939 0.13418874 2 3 4 4 1
wye pan 5 0.57328892 0.86362447 3 4 5 4 2
zee pan 6 0.52712616 0.49322129 4 5 6 4 3
eks zee 7 0.61178406 0.18788492 5 6 7 4 4
zee wye 8 0.59855401 0.97618139 6 7 8 4 5
hat wye 9 0.03144188 0.74955076 7 8 9 4 6
pan wye 10 0.50262601 0.95261836 8 9 10 4 7

1
test/cases/verb-stats1/0023/cmd Normal file
View file

@ -0,0 +1 @@
mlr --ojson --from test/input/abixy-het stats1 -a sum,count -f x,y -w 3

0
test/cases/verb-stats1/0023/experr Normal file
View file

112
test/cases/verb-stats1/0023/expout Normal file
View file

@ -0,0 +1,112 @@
[
{
"a": "pan",
"b": "pan",
"i": 1,
"x": 0.34679014,
"y": 0.72680286,
"x_sum": 0.34679014,
"x_count": 1,
"y_sum": 0.72680286,
"y_count": 1
},
{
"a": "eks",
"b": "pan",
"i": 2,
"x": 0.75867996,
"y": 0.52215111,
"x_sum": 1.10547011,
"x_count": 2,
"y_sum": 1.24895397,
"y_count": 2
},
{
"aaa": "wye",
"b": "wye",
"i": 3,
"x": 0.20460331,
"y": 0.33831853,
"x_sum": 1.31007341,
"x_count": 3,
"y_sum": 1.58727250,
"y_count": 3
},
{
"a": "eks",
"bbb": "wye",
"i": 4,
"x": 0.38139939,
"y": 0.13418874,
"x_sum": 1.34468266,
"x_count": 3,
"y_sum": 0.99465838,
"y_count": 3
},
{
"a": "wye",
"b": "pan",
"i": 5,
"xxx": 0.57328892,
"y": 0.86362447,
"x_sum": 0.58600270,
"x_count": 2,
"y_sum": 1.33613174,
"y_count": 3
},
{
"a": "zee",
"b": "pan",
"i": 6,
"x": 0.52712616,
"y": 0.49322129,
"x_sum": 0.90852555,
"x_count": 2,
"y_sum": 1.49103450,
"y_count": 3
},
{
"a": "eks",
"b": "zee",
"iii": 7,
"x": 0.61178406,
"y": 0.18788492,
"x_sum": 1.13891022,
"x_count": 2,
"y_sum": 1.54473068,
"y_count": 3
},
{
"a": "zee",
"b": "wye",
"i": 8,
"x": 0.59855401,
"yyy": 0.97618139,
"x_sum": 1.73746423,
"x_count": 3,
"y_sum": 0.68110621,
"y_count": 2
},
{
"aaa": "hat",
"bbb": "wye",
"i": 9,
"x": 0.03144188,
"y": 0.74955076,
"x_sum": 1.24177995,
"x_count": 3,
"y_sum": 0.93743568,
"y_count": 2
},
{
"a": "pan",
"b": "wye",
"i": 10,
"x": 0.50262601,
"y": 0.95261836,
"x_sum": 1.13262189,
"x_count": 3,
"y_sum": 1.70216912,
"y_count": 2
}
]

1
test/cases/verb-stats1/0024/cmd Normal file
View file

@ -0,0 +1 @@
mlr --opprint --from test/input/abixy stats1 -a sum --fr '^[xy]$' -w 3

0
test/cases/verb-stats1/0024/experr Normal file
View file

11
test/cases/verb-stats1/0024/expout Normal file
View file

@ -0,0 +1,11 @@
a b i x y x_sum y_sum
pan pan 1 0.34679014 0.72680286 0.34679014 0.72680286
eks pan 2 0.75867996 0.52215111 1.10547011 1.24895397
wye wye 3 0.20460331 0.33831853 1.31007341 1.58727250
eks wye 4 0.38139939 0.13418874 1.34468266 0.99465838
wye pan 5 0.57328892 0.86362447 1.15929162 1.33613174
zee pan 6 0.52712616 0.49322129 1.48181447 1.49103450
eks zee 7 0.61178406 0.18788492 1.71219914 1.54473068
zee wye 8 0.59855401 0.97618139 1.73746423 1.65728759
hat wye 9 0.03144188 0.74955076 1.24177995 1.91361707
pan wye 10 0.50262601 0.95261836 1.13262189 2.67835051

1
test/cases/verb-stats1/0025/cmd Normal file
View file

@ -0,0 +1 @@
mlr --from test/input/abixy stats1 -a mean -f x -s -w 3

1
test/cases/verb-stats1/0025/experr Normal file
View file

@ -0,0 +1 @@
mlr stats1: -s and -w may not be used together

0
test/cases/verb-stats1/0025/expout Normal file
View file

0
test/cases/verb-stats1/0025/should-fail Normal file
View file