Mirrors/miller
1
0
Fork 0
mirror of https://github.com/johnkerl/miller.git synced 2026-01-23 02:14:13 +00:00
Code Issues Projects Releases Wiki Activity

Implement shift-lead option for mlr step (#893)

Browse source 
* iterating

* stepper-input refactor in prep for sliding-window PR

* window-keeper util class

* integrate window-keeper into step-transformer
This commit is contained in:
John Kerl 2022-01-23 00:54:39 -05:00 • committed by GitHub
parent 35bc852cd7
commit a2a9118ad8
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
31 changed files with 1237 additions and 281 deletions
Download patch file Download diff file Expand all files Collapse all files

1
.vimrc
View file

@ -1,3 +1,4 @@
map \d :w<C-m>:!clear;echo Building ...; echo; make mlr<C-m>
map \f :w<C-m>:!clear;echo Building ...; echo; make ut<C-m>
map \r :w<C-m>:!clear;echo Building ...; echo; make ut-scan ut-mlv<C-m>
map \t :w<C-m>:!clear;go test github.com/johnkerl/miller/internal/pkg/transformers/...<C-m>

48
docs/src/manpage.md
View file

@ -1825,31 +1825,33 @@ VERBS
step
Usage: mlr step [options]
Computes values dependent on the previous record, optionally grouped by category.
Computes values dependent on earlier/later records, optionally grouped by category.
Options:
-a {delta,rsum,...} Names of steppers: comma-separated, one or more of:
delta Compute differences in field(s) between successive records
shift Include value(s) in field(s) from previous record, if any
-a {delta,rsum,...} Names of steppers: comma-separated, one or more of:
counter Count instances of field(s) between successive records
delta Compute differences in field(s) between successive records
ewma Exponentially weighted moving average over successive records
from-first Compute differences in field(s) from first record
ratio Compute ratios in field(s) between successive records
rsum Compute running sums of field(s) between successive records
counter Count instances of field(s) between successive records
ewma Exponentially weighted moving average over successive records
ratio Compute ratios in field(s) between successive records
rsum Compute running sums of field(s) between successive records
shift Alias for shift-lag
shift-lag Include value(s) in field(s) from the previous record, if any
shift-lead Include value(s) in field(s) from the next record, if any
-f {a,b,c} Value-field names on which to compute statistics
-g {d,e,f} Optional group-by-field names
-F Computes integerable things (e.g. counter) in floating point.
As of Miller 6 this happens automatically, but the flag is accepted
as a no-op for backward compatibility with Miller 5 and below.
-d {x,y,z} Weights for ewma. 1 means current sample gets all weight (no
smoothing), near under under 1 is light smoothing, near over 0 is
heavy smoothing. Multiple weights may be specified, e.g.
"mlr step -a ewma -f sys_load -d 0.01,0.1,0.9". Default if omitted
is "-d 0.5".
-o {a,b,c} Custom suffixes for EWMA output fields. If omitted, these default to
the -d values. If supplied, the number of -o values must be the same
as the number of -d values.
-h|--help Show this message.
-f {a,b,c} Value-field names on which to compute statistics
-g {d,e,f} Optional group-by-field names
-F Computes integerable things (e.g. counter) in floating point.
As of Miller 6 this happens automatically, but the flag is accepted
as a no-op for backward compatibility with Miller 5 and below.
-d {x,y,z} Weights for EWMA. 1 means current sample gets all weight (no
smoothing), near under under 1 is light smoothing, near over 0 is
heavy smoothing. Multiple weights may be specified, e.g.
"mlr step -a ewma -f sys_load -d 0.01,0.1,0.9". Default if omitted
is "-d 0.5".
-o {a,b,c} Custom suffixes for EWMA output fields. If omitted, these default to
the -d values. If supplied, the number of -o values must be the same
as the number of -d values.
-h|--help S how this message.
Examples:
mlr step -a rsum -f request_size
@ -3086,5 +3088,5 @@ SEE ALSO
2022-01-20 MILLER(1)
2022-01-23 MILLER(1)
</pre>

48
docs/src/manpage.txt
View file

@ -1804,31 +1804,33 @@ VERBS
step
Usage: mlr step [options]
Computes values dependent on the previous record, optionally grouped by category.
Computes values dependent on earlier/later records, optionally grouped by category.
Options:
-a {delta,rsum,...} Names of steppers: comma-separated, one or more of:
delta Compute differences in field(s) between successive records
shift Include value(s) in field(s) from previous record, if any
-a {delta,rsum,...} Names of steppers: comma-separated, one or more of:
counter Count instances of field(s) between successive records
delta Compute differences in field(s) between successive records
ewma Exponentially weighted moving average over successive records
from-first Compute differences in field(s) from first record
ratio Compute ratios in field(s) between successive records
rsum Compute running sums of field(s) between successive records
counter Count instances of field(s) between successive records
ewma Exponentially weighted moving average over successive records
ratio Compute ratios in field(s) between successive records
rsum Compute running sums of field(s) between successive records
shift Alias for shift-lag
shift-lag Include value(s) in field(s) from the previous record, if any
shift-lead Include value(s) in field(s) from the next record, if any
-f {a,b,c} Value-field names on which to compute statistics
-g {d,e,f} Optional group-by-field names
-F Computes integerable things (e.g. counter) in floating point.
As of Miller 6 this happens automatically, but the flag is accepted
as a no-op for backward compatibility with Miller 5 and below.
-d {x,y,z} Weights for ewma. 1 means current sample gets all weight (no
smoothing), near under under 1 is light smoothing, near over 0 is
heavy smoothing. Multiple weights may be specified, e.g.
"mlr step -a ewma -f sys_load -d 0.01,0.1,0.9". Default if omitted
is "-d 0.5".
-o {a,b,c} Custom suffixes for EWMA output fields. If omitted, these default to
the -d values. If supplied, the number of -o values must be the same
as the number of -d values.
-h|--help Show this message.
-f {a,b,c} Value-field names on which to compute statistics
-g {d,e,f} Optional group-by-field names
-F Computes integerable things (e.g. counter) in floating point.
As of Miller 6 this happens automatically, but the flag is accepted
as a no-op for backward compatibility with Miller 5 and below.
-d {x,y,z} Weights for EWMA. 1 means current sample gets all weight (no
smoothing), near under under 1 is light smoothing, near over 0 is
heavy smoothing. Multiple weights may be specified, e.g.
"mlr step -a ewma -f sys_load -d 0.01,0.1,0.9". Default if omitted
is "-d 0.5".
-o {a,b,c} Custom suffixes for EWMA output fields. If omitted, these default to
the -d values. If supplied, the number of -o values must be the same
as the number of -d values.
-h|--help S how this message.
Examples:
mlr step -a rsum -f request_size
@ -3065,4 +3067,4 @@ SEE ALSO
2022-01-20 MILLER(1)
2022-01-23 MILLER(1)

46
docs/src/reference-verbs.md
View file

@ -3280,31 +3280,33 @@ donesec 25.10852919630297
</pre>
<pre class="pre-non-highlight-in-pair">
Usage: mlr step [options]
Computes values dependent on the previous record, optionally grouped by category.
Computes values dependent on earlier/later records, optionally grouped by category.
Options:
-a {delta,rsum,...} Names of steppers: comma-separated, one or more of:
delta Compute differences in field(s) between successive records
shift Include value(s) in field(s) from previous record, if any
-a {delta,rsum,...} Names of steppers: comma-separated, one or more of:
counter Count instances of field(s) between successive records
delta Compute differences in field(s) between successive records
ewma Exponentially weighted moving average over successive records
from-first Compute differences in field(s) from first record
ratio Compute ratios in field(s) between successive records
rsum Compute running sums of field(s) between successive records
counter Count instances of field(s) between successive records
ewma Exponentially weighted moving average over successive records
ratio Compute ratios in field(s) between successive records
rsum Compute running sums of field(s) between successive records
shift Alias for shift-lag
shift-lag Include value(s) in field(s) from the previous record, if any
shift-lead Include value(s) in field(s) from the next record, if any
-f {a,b,c} Value-field names on which to compute statistics
-g {d,e,f} Optional group-by-field names
-F Computes integerable things (e.g. counter) in floating point.
As of Miller 6 this happens automatically, but the flag is accepted
as a no-op for backward compatibility with Miller 5 and below.
-d {x,y,z} Weights for ewma. 1 means current sample gets all weight (no
smoothing), near under under 1 is light smoothing, near over 0 is
heavy smoothing. Multiple weights may be specified, e.g.
"mlr step -a ewma -f sys_load -d 0.01,0.1,0.9". Default if omitted
is "-d 0.5".
-o {a,b,c} Custom suffixes for EWMA output fields. If omitted, these default to
the -d values. If supplied, the number of -o values must be the same
as the number of -d values.
-h|--help Show this message.
-f {a,b,c} Value-field names on which to compute statistics
-g {d,e,f} Optional group-by-field names
-F Computes integerable things (e.g. counter) in floating point.
As of Miller 6 this happens automatically, but the flag is accepted
as a no-op for backward compatibility with Miller 5 and below.
-d {x,y,z} Weights for EWMA. 1 means current sample gets all weight (no
smoothing), near under under 1 is light smoothing, near over 0 is
heavy smoothing. Multiple weights may be specified, e.g.
"mlr step -a ewma -f sys_load -d 0.01,0.1,0.9". Default if omitted
is "-d 0.5".
-o {a,b,c} Custom suffixes for EWMA output fields. If omitted, these default to
the -d values. If supplied, the number of -o values must be the same
as the number of -d values.
-h|--help S how this message.
Examples:
mlr step -a rsum -f request_size

770
internal/pkg/transformers/step.go
View file

File diff suppressed because it is too large Load diff

70
internal/pkg/transformers/utils/window-keeper.go Normal file
View file

@ -0,0 +1,70 @@
package utils
import (
"github.com/johnkerl/miller/internal/pkg/lib"
)
// WindowKeeper is a sliding-window container, nominally for use by mlr step,
// for holding a number of records before the current one, the current one, and
// a number of records after. The payload is interface{}, not *mlrval.Mlrmap,
// for ease of unit-testing -- and also, since nothing here inspects the
// payload, so that this code could be repurposed.
type TWindowKeeper struct {
numBackward int
numForward int
itemsBackward []interface{}
currentItem interface{}
itemsForward []interface{}
}
func NewWindowKeeper(
numBackward int,
numForward int,
) *TWindowKeeper {
return &TWindowKeeper{
numBackward: numBackward,
numForward: numForward,
itemsBackward: make([]interface{}, numBackward),
currentItem: nil,
itemsForward: make([]interface{}, numForward),
}
}
func (wk *TWindowKeeper) Ingest(
inrec interface{},
) {
for i := wk.numBackward - 1; i > 0; i-- {
wk.itemsBackward[i] = wk.itemsBackward[i-1]
}
if wk.numBackward > 0 {
wk.itemsBackward[0] = wk.currentItem
}
if wk.numForward > 0 {
wk.currentItem = wk.itemsForward[0]
for i := 0; i < wk.numForward-1; i++ {
wk.itemsForward[i] = wk.itemsForward[i+1]
}
wk.itemsForward[wk.numForward-1] = inrec
} else {
wk.currentItem = inrec
}
}
// Get maps a user-visible indexing ..., -3, -2, -1, 0, 1, 2, 3, ...
// into this struct's zero-index array storage.
func (wk *TWindowKeeper) Get(
index int,
) interface{} {
if index == 0 {
return wk.currentItem
} else if index > 0 {
lib.InternalCodingErrorIf(index > wk.numForward)
return wk.itemsForward[index-1]
} else {
index = -index
lib.InternalCodingErrorIf(index > wk.numBackward)
return wk.itemsBackward[index-1]
}
}

157
internal/pkg/transformers/utils/window_keeper_test.go Normal file
View file

@ -0,0 +1,157 @@
package utils
import (
"testing"
"github.com/stretchr/testify/assert"
)
func Test00(t *testing.T) {
wk := NewWindowKeeper(0, 0)
wk.Ingest("a")
assert.Equal(t, "a", wk.Get(0).(string))
wk.Ingest("b")
assert.Equal(t, "b", wk.Get(0).(string))
}
func Test10(t *testing.T) {
wk := NewWindowKeeper(1, 0)
wk.Ingest("a")
assert.Equal(t, "a", wk.Get(0).(string))
assert.Equal(t, nil, wk.Get(-1))
wk.Ingest("b")
assert.Equal(t, "b", wk.Get(0).(string))
assert.Equal(t, "a", wk.Get(-1).(string))
wk.Ingest("c")
assert.Equal(t, "c", wk.Get(0).(string))
assert.Equal(t, "b", wk.Get(-1).(string))
}
func Test20(t *testing.T) {
wk := NewWindowKeeper(2, 0)
wk.Ingest("a")
assert.Equal(t, "a", wk.Get(0).(string))
assert.Equal(t, nil, wk.Get(-1))
assert.Equal(t, nil, wk.Get(-2))
wk.Ingest("b")
assert.Equal(t, "b", wk.Get(0).(string))
assert.Equal(t, "a", wk.Get(-1).(string))
assert.Equal(t, nil, wk.Get(-2))
wk.Ingest("c")
assert.Equal(t, "c", wk.Get(0).(string))
assert.Equal(t, "b", wk.Get(-1).(string))
assert.Equal(t, "a", wk.Get(-2).(string))
wk.Ingest("d")
assert.Equal(t, "d", wk.Get(0).(string))
assert.Equal(t, "c", wk.Get(-1).(string))
assert.Equal(t, "b", wk.Get(-2).(string))
}
func Test01(t *testing.T) {
wk := NewWindowKeeper(0, 1)
wk.Ingest("a")
assert.Equal(t, "a", wk.Get(1).(string))
assert.Equal(t, nil, wk.Get(0))
wk.Ingest("b")
assert.Equal(t, "b", wk.Get(1).(string))
assert.Equal(t, "a", wk.Get(0).(string))
wk.Ingest("c")
assert.Equal(t, "c", wk.Get(1).(string))
assert.Equal(t, "b", wk.Get(0).(string))
}
func Test02(t *testing.T) {
wk := NewWindowKeeper(0, 2)
wk.Ingest("a")
assert.Equal(t, "a", wk.Get(2).(string))
assert.Equal(t, nil, wk.Get(1))
assert.Equal(t, nil, wk.Get(0))
wk.Ingest("b")
assert.Equal(t, "b", wk.Get(2).(string))
assert.Equal(t, "a", wk.Get(1).(string))
assert.Equal(t, nil, wk.Get(0))
wk.Ingest("c")
assert.Equal(t, "c", wk.Get(2).(string))
assert.Equal(t, "b", wk.Get(1).(string))
assert.Equal(t, "a", wk.Get(0).(string))
wk.Ingest("d")
assert.Equal(t, "d", wk.Get(2).(string))
assert.Equal(t, "c", wk.Get(1).(string))
assert.Equal(t, "b", wk.Get(0).(string))
}
func Test32(t *testing.T) {
wk := NewWindowKeeper(3, 2)
wk.Ingest("a")
assert.Equal(t, "a", wk.Get(2).(string))
assert.Equal(t, nil, wk.Get(1))
assert.Equal(t, nil, wk.Get(0))
assert.Equal(t, nil, wk.Get(-1))
assert.Equal(t, nil, wk.Get(-2))
assert.Equal(t, nil, wk.Get(-3))
wk.Ingest("b")
assert.Equal(t, "b", wk.Get(2).(string))
assert.Equal(t, "a", wk.Get(1).(string))
assert.Equal(t, nil, wk.Get(0))
assert.Equal(t, nil, wk.Get(-1))
assert.Equal(t, nil, wk.Get(-2))
assert.Equal(t, nil, wk.Get(-3))
wk.Ingest("c")
assert.Equal(t, "c", wk.Get(2).(string))
assert.Equal(t, "b", wk.Get(1).(string))
assert.Equal(t, "a", wk.Get(0).(string))
assert.Equal(t, nil, wk.Get(-1))
assert.Equal(t, nil, wk.Get(-2))
assert.Equal(t, nil, wk.Get(-3))
wk.Ingest("d")
assert.Equal(t, "d", wk.Get(2).(string))
assert.Equal(t, "c", wk.Get(1).(string))
assert.Equal(t, "b", wk.Get(0).(string))
assert.Equal(t, "a", wk.Get(-1).(string))
assert.Equal(t, nil, wk.Get(-2))
assert.Equal(t, nil, wk.Get(-3))
wk.Ingest("e")
assert.Equal(t, "e", wk.Get(2).(string))
assert.Equal(t, "d", wk.Get(1).(string))
assert.Equal(t, "c", wk.Get(0).(string))
assert.Equal(t, "b", wk.Get(-1).(string))
assert.Equal(t, "a", wk.Get(-2).(string))
assert.Equal(t, nil, wk.Get(-3))
wk.Ingest("f")
assert.Equal(t, "f", wk.Get(2).(string))
assert.Equal(t, "e", wk.Get(1).(string))
assert.Equal(t, "d", wk.Get(0).(string))
assert.Equal(t, "c", wk.Get(-1).(string))
assert.Equal(t, "b", wk.Get(-2).(string))
assert.Equal(t, "a", wk.Get(-3).(string))
wk.Ingest("g")
assert.Equal(t, "g", wk.Get(2).(string))
assert.Equal(t, "f", wk.Get(1).(string))
assert.Equal(t, "e", wk.Get(0).(string))
assert.Equal(t, "d", wk.Get(-1).(string))
assert.Equal(t, "c", wk.Get(-2).(string))
assert.Equal(t, "b", wk.Get(-3).(string))
}

48
man/manpage.txt
View file

@ -1804,31 +1804,33 @@ VERBS
step
Usage: mlr step [options]
Computes values dependent on the previous record, optionally grouped by category.
Computes values dependent on earlier/later records, optionally grouped by category.
Options:
-a {delta,rsum,...} Names of steppers: comma-separated, one or more of:
delta Compute differences in field(s) between successive records
shift Include value(s) in field(s) from previous record, if any
-a {delta,rsum,...} Names of steppers: comma-separated, one or more of:
counter Count instances of field(s) between successive records
delta Compute differences in field(s) between successive records
ewma Exponentially weighted moving average over successive records
from-first Compute differences in field(s) from first record
ratio Compute ratios in field(s) between successive records
rsum Compute running sums of field(s) between successive records
counter Count instances of field(s) between successive records
ewma Exponentially weighted moving average over successive records
ratio Compute ratios in field(s) between successive records
rsum Compute running sums of field(s) between successive records
shift Alias for shift-lag
shift-lag Include value(s) in field(s) from the previous record, if any
shift-lead Include value(s) in field(s) from the next record, if any
-f {a,b,c} Value-field names on which to compute statistics
-g {d,e,f} Optional group-by-field names
-F Computes integerable things (e.g. counter) in floating point.
As of Miller 6 this happens automatically, but the flag is accepted
as a no-op for backward compatibility with Miller 5 and below.
-d {x,y,z} Weights for ewma. 1 means current sample gets all weight (no
smoothing), near under under 1 is light smoothing, near over 0 is
heavy smoothing. Multiple weights may be specified, e.g.
"mlr step -a ewma -f sys_load -d 0.01,0.1,0.9". Default if omitted
is "-d 0.5".
-o {a,b,c} Custom suffixes for EWMA output fields. If omitted, these default to
the -d values. If supplied, the number of -o values must be the same
as the number of -d values.
-h|--help Show this message.
-f {a,b,c} Value-field names on which to compute statistics
-g {d,e,f} Optional group-by-field names
-F Computes integerable things (e.g. counter) in floating point.
As of Miller 6 this happens automatically, but the flag is accepted
as a no-op for backward compatibility with Miller 5 and below.
-d {x,y,z} Weights for EWMA. 1 means current sample gets all weight (no
smoothing), near under under 1 is light smoothing, near over 0 is
heavy smoothing. Multiple weights may be specified, e.g.
"mlr step -a ewma -f sys_load -d 0.01,0.1,0.9". Default if omitted
is "-d 0.5".
-o {a,b,c} Custom suffixes for EWMA output fields. If omitted, these default to
the -d values. If supplied, the number of -o values must be the same
as the number of -d values.
-h|--help S how this message.
Examples:
mlr step -a rsum -f request_size
@ -3065,4 +3067,4 @@ SEE ALSO
2022-01-20 MILLER(1)
2022-01-23 MILLER(1)

50
man/mlr.1
View file

@ -2,12 +2,12 @@
.\" Title: mlr
.\" Author: [see the "AUTHOR" section]
.\" Generator: ./mkman.rb
.\" Date: 2022-01-20
.\" Date: 2022-01-23
.\" Manual: \ \&
.\" Source: \ \&
.\" Language: English
.\"
.TH "MILLER" "1" "2022-01-20" "\ \&" "\ \&"
.TH "MILLER" "1" "2022-01-23" "\ \&" "\ \&"
.\" -----------------------------------------------------------------
.\" * Portability definitions
.\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@ -2273,31 +2273,33 @@ Example: mlr stats2 -a corr -f x,y
.\}
.nf
Usage: mlr step [options]
Computes values dependent on the previous record, optionally grouped by category.
Computes values dependent on earlier/later records, optionally grouped by category.
Options:
-a {delta,rsum,...} Names of steppers: comma-separated, one or more of:
delta Compute differences in field(s) between successive records
shift Include value(s) in field(s) from previous record, if any
-a {delta,rsum,...} Names of steppers: comma-separated, one or more of:
counter Count instances of field(s) between successive records
delta Compute differences in field(s) between successive records
ewma Exponentially weighted moving average over successive records
from-first Compute differences in field(s) from first record
ratio Compute ratios in field(s) between successive records
rsum Compute running sums of field(s) between successive records
counter Count instances of field(s) between successive records
ewma Exponentially weighted moving average over successive records
ratio Compute ratios in field(s) between successive records
rsum Compute running sums of field(s) between successive records
shift Alias for shift-lag
shift-lag Include value(s) in field(s) from the previous record, if any
shift-lead Include value(s) in field(s) from the next record, if any
-f {a,b,c} Value-field names on which to compute statistics
-g {d,e,f} Optional group-by-field names
-F Computes integerable things (e.g. counter) in floating point.
As of Miller 6 this happens automatically, but the flag is accepted
as a no-op for backward compatibility with Miller 5 and below.
-d {x,y,z} Weights for ewma. 1 means current sample gets all weight (no
smoothing), near under under 1 is light smoothing, near over 0 is
heavy smoothing. Multiple weights may be specified, e.g.
"mlr step -a ewma -f sys_load -d 0.01,0.1,0.9". Default if omitted
is "-d 0.5".
-o {a,b,c} Custom suffixes for EWMA output fields. If omitted, these default to
the -d values. If supplied, the number of -o values must be the same
as the number of -d values.
-h|--help Show this message.
-f {a,b,c} Value-field names on which to compute statistics
-g {d,e,f} Optional group-by-field names
-F Computes integerable things (e.g. counter) in floating point.
As of Miller 6 this happens automatically, but the flag is accepted
as a no-op for backward compatibility with Miller 5 and below.
-d {x,y,z} Weights for EWMA. 1 means current sample gets all weight (no
smoothing), near under under 1 is light smoothing, near over 0 is
heavy smoothing. Multiple weights may be specified, e.g.
"mlr step -a ewma -f sys_load -d 0.01,0.1,0.9". Default if omitted
is "-d 0.5".
-o {a,b,c} Custom suffixes for EWMA output fields. If omitted, these default to
the -d values. If supplied, the number of -o values must be the same
as the number of -d values.
-h|--help S how this message.
Examples:
mlr step -a rsum -f request_size

46
test/cases/cli-help/0001/expout
View file

@ -1021,31 +1021,33 @@ Example: mlr stats2 -a corr -f x,y
================================================================
step
Usage: mlr step [options]
Computes values dependent on the previous record, optionally grouped by category.
Computes values dependent on earlier/later records, optionally grouped by category.
Options:
-a {delta,rsum,...} Names of steppers: comma-separated, one or more of:
delta Compute differences in field(s) between successive records
shift Include value(s) in field(s) from previous record, if any
-a {delta,rsum,...} Names of steppers: comma-separated, one or more of:
counter Count instances of field(s) between successive records
delta Compute differences in field(s) between successive records
ewma Exponentially weighted moving average over successive records
from-first Compute differences in field(s) from first record
ratio Compute ratios in field(s) between successive records
rsum Compute running sums of field(s) between successive records
counter Count instances of field(s) between successive records
ewma Exponentially weighted moving average over successive records
ratio Compute ratios in field(s) between successive records
rsum Compute running sums of field(s) between successive records
shift Alias for shift-lag
shift-lag Include value(s) in field(s) from the previous record, if any
shift-lead Include value(s) in field(s) from the next record, if any
-f {a,b,c} Value-field names on which to compute statistics
-g {d,e,f} Optional group-by-field names
-F Computes integerable things (e.g. counter) in floating point.
As of Miller 6 this happens automatically, but the flag is accepted
as a no-op for backward compatibility with Miller 5 and below.
-d {x,y,z} Weights for ewma. 1 means current sample gets all weight (no
smoothing), near under under 1 is light smoothing, near over 0 is
heavy smoothing. Multiple weights may be specified, e.g.
"mlr step -a ewma -f sys_load -d 0.01,0.1,0.9". Default if omitted
is "-d 0.5".
-o {a,b,c} Custom suffixes for EWMA output fields. If omitted, these default to
the -d values. If supplied, the number of -o values must be the same
as the number of -d values.
-h|--help Show this message.
-f {a,b,c} Value-field names on which to compute statistics
-g {d,e,f} Optional group-by-field names
-F Computes integerable things (e.g. counter) in floating point.
As of Miller 6 this happens automatically, but the flag is accepted
as a no-op for backward compatibility with Miller 5 and below.
-d {x,y,z} Weights for EWMA. 1 means current sample gets all weight (no
smoothing), near under under 1 is light smoothing, near over 0 is
heavy smoothing. Multiple weights may be specified, e.g.
"mlr step -a ewma -f sys_load -d 0.01,0.1,0.9". Default if omitted
is "-d 0.5".
-o {a,b,c} Custom suffixes for EWMA output fields. If omitted, these default to
the -d values. If supplied, the number of -o values must be the same
as the number of -d values.
-h|--help S how this message.
Examples:
mlr step -a rsum -f request_size

2
test/cases/verb-step/0005/cmd
View file

@ -1 +1 @@
mlr --odkvp step -a rsum,shift,delta,counter -f x,y test/input/abixy-het
mlr --ojson step -a rsum,shift,delta,counter -f x,y test/input/abixy-het

154
test/cases/verb-step/0005/expout
View file

@ -1,10 +1,144 @@
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533,x_rsum=0.3467901443380824,x_shift=,x_delta=0,x_counter=1,y_rsum=0.7268028627434533,y_shift=,y_delta=0,y_counter=1
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797,x_rsum=1.105470109128046,x_shift=0.3467901443380824,x_delta=0.41188982045188116,x_counter=2,y_rsum=1.2489539710769328,y_shift=0.7268028627434533,y_delta=-0.20465175440997363,y_counter=2
aaa=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776,x_rsum=1.3100734148943491,x_shift=0.7586799647899636,x_delta=-0.5540766590236605,x_counter=3,y_rsum=1.5872724965935805,y_shift=0.5221511083334797,y_delta=-0.1838325828168319,y_counter=3
a=eks,bbb=wye,i=4,x=0.38139939387114097,y=0.13418874328430463,x_rsum=1.6914728087654902,x_shift=0.20460330576630303,x_delta=0.17679608810483793,x_counter=4,y_rsum=1.7214612398778852,y_shift=0.33831852551664776,y_delta=-0.20412978223234313,y_counter=4
a=wye,b=pan,i=5,xxx=0.5732889198020006,y=0.8636244699032729,y_rsum=2.585085709781158,y_shift=0.13418874328430463,y_delta=0.7294357266189683,y_counter=5
a=zee,b=pan,i=6,x=0.5271261600918548,y=0.49322128674835697,x_rsum=2.218598968857345,x_shift=0.38139939387114097,x_delta=0.1457267662207138,x_counter=5,y_rsum=3.0783069965295153,y_shift=0.8636244699032729,y_delta=-0.37040318315491594,y_counter=6
a=eks,b=zee,iii=7,x=0.6117840605678454,y=0.1878849191181694,x_rsum=2.8303830294251906,x_shift=0.5271261600918548,x_delta=0.08465790047599064,x_counter=6,y_rsum=3.266191915647685,y_shift=0.49322128674835697,y_delta=-0.30533636763018757,y_counter=7
a=zee,b=wye,i=8,x=0.5985540091064224,yyy=0.976181385699006,x_rsum=3.428937038531613,x_shift=0.6117840605678454,x_delta=-0.013230051461422976,x_counter=7
aaa=hat,bbb=wye,i=9,x=0.03144187646093577,y=0.7495507603507059,x_rsum=3.460378914992549,x_shift=0.5985540091064224,x_delta=-0.5671121326454867,x_counter=8,y_rsum=4.015742675998391,y_shift=0.1878849191181694,y_delta=0.5616658412325365,y_counter=8
a=pan,b=wye,i=10,x=0.5026260055412137,y=0.9526183602969864,x_rsum=3.9630049205337627,x_shift=0.03144187646093577,x_delta=0.47118412908027796,x_counter=9,y_rsum=4.968361036295377,y_shift=0.7495507603507059,y_delta=0.20306759994628054,y_counter=9
[
{
"a": "pan",
"b": "pan",
"i": 1,
"x": 0.3467901443380824,
"y": 0.7268028627434533,
"x_rsum": 0.3467901443380824,
"x_shift": "",
"x_delta": 0,
"x_counter": 1,
"y_rsum": 0.7268028627434533,
"y_shift": "",
"y_delta": 0,
"y_counter": 1
},
{
"a": "eks",
"b": "pan",
"i": 2,
"x": 0.7586799647899636,
"y": 0.5221511083334797,
"x_rsum": 1.105470109128046,
"x_shift": 0.3467901443380824,
"x_delta": 0.41188982045188116,
"x_counter": 2,
"y_rsum": 1.2489539710769328,
"y_shift": 0.7268028627434533,
"y_delta": -0.20465175440997363,
"y_counter": 2
},
{
"aaa": "wye",
"b": "wye",
"i": 3,
"x": 0.20460330576630303,
"y": 0.33831852551664776,
"x_rsum": 1.3100734148943491,
"x_shift": 0.7586799647899636,
"x_delta": -0.5540766590236605,
"x_counter": 3,
"y_rsum": 1.5872724965935805,
"y_shift": 0.5221511083334797,
"y_delta": -0.1838325828168319,
"y_counter": 3
},
{
"a": "eks",
"bbb": "wye",
"i": 4,
"x": 0.38139939387114097,
"y": 0.13418874328430463,
"x_rsum": 1.6914728087654902,
"x_shift": 0.20460330576630303,
"x_delta": 0.17679608810483793,
"x_counter": 4,
"y_rsum": 1.7214612398778852,
"y_shift": 0.33831852551664776,
"y_delta": -0.20412978223234313,
"y_counter": 4
},
{
"a": "wye",
"b": "pan",
"i": 5,
"xxx": 0.5732889198020006,
"y": 0.8636244699032729,
"y_rsum": 2.585085709781158,
"y_shift": 0.13418874328430463,
"y_delta": 0.7294357266189683,
"y_counter": 5
},
{
"a": "zee",
"b": "pan",
"i": 6,
"x": 0.5271261600918548,
"y": 0.49322128674835697,
"x_rsum": 2.218598968857345,
"x_shift": "",
"x_delta": 0,
"x_counter": 5,
"y_rsum": 3.0783069965295153,
"y_shift": 0.8636244699032729,
"y_delta": -0.37040318315491594,
"y_counter": 6
},
{
"a": "eks",
"b": "zee",
"iii": 7,
"x": 0.6117840605678454,
"y": 0.1878849191181694,
"x_rsum": 2.8303830294251906,
"x_shift": 0.5271261600918548,
"x_delta": 0.08465790047599064,
"x_counter": 6,
"y_rsum": 3.266191915647685,
"y_shift": 0.49322128674835697,
"y_delta": -0.30533636763018757,
"y_counter": 7
},
{
"a": "zee",
"b": "wye",
"i": 8,
"x": 0.5985540091064224,
"yyy": 0.976181385699006,
"x_rsum": 3.428937038531613,
"x_shift": 0.6117840605678454,
"x_delta": -0.013230051461422976,
"x_counter": 7
},
{
"aaa": "hat",
"bbb": "wye",
"i": 9,
"x": 0.03144187646093577,
"y": 0.7495507603507059,
"x_rsum": 3.460378914992549,
"x_shift": 0.5985540091064224,
"x_delta": -0.5671121326454867,
"x_counter": 8,
"y_rsum": 4.015742675998391,
"y_shift": "",
"y_delta": 0,
"y_counter": 8
},
{
"a": "pan",
"b": "wye",
"i": 10,
"x": 0.5026260055412137,
"y": 0.9526183602969864,
"x_rsum": 3.9630049205337627,
"x_shift": 0.03144187646093577,
"x_delta": 0.47118412908027796,
"x_counter": 9,
"y_rsum": 4.968361036295377,
"y_shift": 0.7495507603507059,
"y_delta": 0.20306759994628054,
"y_counter": 9
}
]

1
test/cases/verb-step/0011/cmd Normal file
View file

@ -0,0 +1 @@
mlr --opprint --from test/input/abixy step -a shift-lag -f i

0
test/cases/verb-step/0011/experr Normal file
View file

11
test/cases/verb-step/0011/expout Normal file
View file

@ -0,0 +1,11 @@
a b i x y i_shift_lag
pan pan 1 0.3467901443380824 0.7268028627434533 -
eks pan 2 0.7586799647899636 0.5221511083334797 1
wye wye 3 0.20460330576630303 0.33831852551664776 2
eks wye 4 0.38139939387114097 0.13418874328430463 3
wye pan 5 0.5732889198020006 0.8636244699032729 4
zee pan 6 0.5271261600918548 0.49322128674835697 5
eks zee 7 0.6117840605678454 0.1878849191181694 6
zee wye 8 0.5985540091064224 0.976181385699006 7
hat wye 9 0.03144187646093577 0.7495507603507059 8
pan wye 10 0.5026260055412137 0.9526183602969864 9

1
test/cases/verb-step/0012/cmd Normal file
View file

@ -0,0 +1 @@
mlr --opprint --from test/input/abixy step -a shift-lead -f i

0
test/cases/verb-step/0012/experr Normal file
View file

11
test/cases/verb-step/0012/expout Normal file
View file

@ -0,0 +1,11 @@
a b i x y i_shift_lead
pan pan 1 0.3467901443380824 0.7268028627434533 2
eks pan 2 0.7586799647899636 0.5221511083334797 3
wye wye 3 0.20460330576630303 0.33831852551664776 4
eks wye 4 0.38139939387114097 0.13418874328430463 5
wye pan 5 0.5732889198020006 0.8636244699032729 6
zee pan 6 0.5271261600918548 0.49322128674835697 7
eks zee 7 0.6117840605678454 0.1878849191181694 8
zee wye 8 0.5985540091064224 0.976181385699006 9
hat wye 9 0.03144187646093577 0.7495507603507059 10
pan wye 10 0.5026260055412137 0.9526183602969864 -

1
test/cases/verb-step/0013/cmd Normal file
View file

@ -0,0 +1 @@
mlr --opprint --from test/input/abixy step -a shift-lag,shift-lead -f i

0
test/cases/verb-step/0013/experr Normal file
View file

11
test/cases/verb-step/0013/expout Normal file
View file

@ -0,0 +1,11 @@
a b i x y i_shift_lag i_shift_lead
pan pan 1 0.3467901443380824 0.7268028627434533 - 2
eks pan 2 0.7586799647899636 0.5221511083334797 1 3
wye wye 3 0.20460330576630303 0.33831852551664776 2 4
eks wye 4 0.38139939387114097 0.13418874328430463 3 5
wye pan 5 0.5732889198020006 0.8636244699032729 4 6
zee pan 6 0.5271261600918548 0.49322128674835697 5 7
eks zee 7 0.6117840605678454 0.1878849191181694 6 8
zee wye 8 0.5985540091064224 0.976181385699006 7 9
hat wye 9 0.03144187646093577 0.7495507603507059 8 10
pan wye 10 0.5026260055412137 0.9526183602969864 9 -

1
test/cases/verb-step/0014/cmd Normal file
View file

@ -0,0 +1 @@
mlr --opprint --from test/input/abixy step -a shift-lag -f i -g a then sort -f a

0
test/cases/verb-step/0014/experr Normal file
View file

11
test/cases/verb-step/0014/expout Normal file
View file

@ -0,0 +1,11 @@
a b i x y i_shift_lag
eks pan 2 0.7586799647899636 0.5221511083334797 -
eks wye 4 0.38139939387114097 0.13418874328430463 2
eks zee 7 0.6117840605678454 0.1878849191181694 4
hat wye 9 0.03144187646093577 0.7495507603507059 -
pan pan 1 0.3467901443380824 0.7268028627434533 -
pan wye 10 0.5026260055412137 0.9526183602969864 1
wye wye 3 0.20460330576630303 0.33831852551664776 -
wye pan 5 0.5732889198020006 0.8636244699032729 3
zee pan 6 0.5271261600918548 0.49322128674835697 -
zee wye 8 0.5985540091064224 0.976181385699006 6

1
test/cases/verb-step/0015/cmd Normal file
View file

@ -0,0 +1 @@
mlr --opprint --from test/input/abixy step -a shift-lead -f i -g a then sort -f a

0
test/cases/verb-step/0015/experr Normal file
View file

11
test/cases/verb-step/0015/expout Normal file
View file

@ -0,0 +1,11 @@
a b i x y i_shift_lead
eks pan 2 0.7586799647899636 0.5221511083334797 4
eks wye 4 0.38139939387114097 0.13418874328430463 7
eks zee 7 0.6117840605678454 0.1878849191181694 -
hat wye 9 0.03144187646093577 0.7495507603507059 -
pan pan 1 0.3467901443380824 0.7268028627434533 10
pan wye 10 0.5026260055412137 0.9526183602969864 -
wye wye 3 0.20460330576630303 0.33831852551664776 5
wye pan 5 0.5732889198020006 0.8636244699032729 -
zee pan 6 0.5271261600918548 0.49322128674835697 8
zee wye 8 0.5985540091064224 0.976181385699006 -

1
test/cases/verb-step/0016/cmd Normal file
View file

@ -0,0 +1 @@
mlr --opprint --from test/input/abixy step -a shift-lag,shift-lead -f i -g a then sort -f a

0
test/cases/verb-step/0016/experr Normal file
View file

11
test/cases/verb-step/0016/expout Normal file
View file

@ -0,0 +1,11 @@
a b i x y i_shift_lag i_shift_lead
eks pan 2 0.7586799647899636 0.5221511083334797 - 4
eks wye 4 0.38139939387114097 0.13418874328430463 2 7
eks zee 7 0.6117840605678454 0.1878849191181694 4 -
hat wye 9 0.03144187646093577 0.7495507603507059 - -
pan pan 1 0.3467901443380824 0.7268028627434533 - 10
pan wye 10 0.5026260055412137 0.9526183602969864 1 -
wye wye 3 0.20460330576630303 0.33831852551664776 - 5
wye pan 5 0.5732889198020006 0.8636244699032729 3 -
zee pan 6 0.5271261600918548 0.49322128674835697 - 8
zee wye 8 0.5985540091064224 0.976181385699006 6 -

6
todo.txt
View file

@ -24,13 +24,15 @@ k better print-interpolate with {} etc
----------------------------------------------------------------
! strmatch https://github.com/johnkerl/miller/issues/77#issuecomment-538790927
----------------------------------------------------------------
a-b.go -> a_b.go renamer PR
----------------------------------------------------------------
! sliding window / moving average
o port u/window*.mlr from mlrc to mlr (actually, fix mlr of course)
o sliding-window averages into mapper step (C + Go)
----------------------------------------------------------------
! shift_lead and shift_lag steps
! make a lag-by-n and lead-by-n
----------------------------------------------------------------
! rank