mirror of
https://github.com/johnkerl/miller.git
synced 2026-07-22 15:37:59 +00:00
203 lines
6.4 KiB
Markdown
203 lines
6.4 KiB
Markdown
# Two-pass algorithms
|
|
|
|
## Overview
|
|
|
|
Miller is a streaming record processor; commands are performed once per record. This makes Miller particularly suitable for single-pass algorithms, allowing many of its verbs to process files that are (much) larger than the amount of RAM present in your system. (Of course, Miller verbs such as `sort`, `tac`, etc. all must ingest and retain all input records before emitting any output records.) You can also use out-of-stream variables to perform multi-pass computations, at the price of retaining all input records in memory.
|
|
|
|
One of Miller's strengths is its compact notation: for example, given input of the form
|
|
|
|
GENMD_RUN_COMMAND
|
|
head -n 5 ./data/medium
|
|
GENMD_EOF
|
|
|
|
you can simply do
|
|
|
|
GENMD_RUN_COMMAND
|
|
mlr --oxtab stats1 -a sum -f x ./data/medium
|
|
GENMD_EOF
|
|
|
|
or
|
|
|
|
GENMD_RUN_COMMAND
|
|
mlr --opprint stats1 -a sum -f x -g b ./data/medium
|
|
GENMD_EOF
|
|
|
|
rather than the more tedious
|
|
|
|
GENMD_INCLUDE_AND_RUN_ESCAPED(oosvar-example-sum.sh)
|
|
|
|
or
|
|
|
|
GENMD_INCLUDE_AND_RUN_ESCAPED(oosvar-example-sum-grouped.sh)
|
|
|
|
The former (`mlr stats1` et al.) has the advantages of being easier to type, being less error-prone to type, and running faster.
|
|
|
|
Nonetheless, out-of-stream variables (which I whimsically call *oosvars*), begin/end blocks, and emit statements give you the ability to implement logic -- if you wish to do so -- which isn't present in other Miller verbs. (If you find yourself often using the same out-of-stream-variable logic over and over, please file a request at [https://github.com/johnkerl/miller/issues](https://github.com/johnkerl/miller/issues) to get it implemented directly in Go as a Miller verb of its own.)
|
|
|
|
The following examples compute some things using oosvars which are already computable using Miller verbs, by way of providing food for thought.
|
|
|
|
## Computation of percentages
|
|
|
|
For example, mapping numeric values down a column to the percentage between their min and max values is two-pass: on the first pass you find the min and max values, then on the second, map each record's value to a percentage.
|
|
|
|
GENMD_INCLUDE_AND_RUN_ESCAPED(data/two-pass-percentage.sh)
|
|
|
|
## Line-number ratios
|
|
|
|
Similarly, finding the total record count requires first reading through all the data:
|
|
|
|
GENMD_INCLUDE_AND_RUN_ESCAPED(data/two-pass-record-numbers.sh)
|
|
|
|
## Records having max value
|
|
|
|
The idea is to retain records having the largest value of `n` in the following data:
|
|
|
|
GENMD_RUN_COMMAND
|
|
mlr --itsv --opprint cat data/maxrows.tsv
|
|
GENMD_EOF
|
|
|
|
Of course, the largest value of `n` isn't known until after all data have been read. Using an out-of-stream variable we can retain all records as they are read, then filter them at the end:
|
|
|
|
GENMD_RUN_COMMAND
|
|
cat data/maxrows.mlr
|
|
GENMD_EOF
|
|
|
|
GENMD_RUN_COMMAND
|
|
mlr --itsv --opprint put -q -f data/maxrows.mlr data/maxrows.tsv
|
|
GENMD_EOF
|
|
|
|
## Feature-counting
|
|
|
|
Suppose you have some heterogeneous data like this:
|
|
|
|
GENMD_INCLUDE_ESCAPED(data/features.json)
|
|
|
|
A reasonable question to ask is, how many occurrences of each field are there? And, what percentage of total row count has each of them? Since the denominator of the percentage is not known until the end, this is a two-pass algorithm:
|
|
|
|
GENMD_INCLUDE_ESCAPED(data/feature-count.mlr)
|
|
|
|
Then
|
|
|
|
GENMD_RUN_COMMAND
|
|
mlr --json put -q -f data/feature-count.mlr data/features.json
|
|
GENMD_EOF
|
|
|
|
GENMD_RUN_COMMAND
|
|
mlr --ijson --opprint put -q -f data/feature-count.mlr data/features.json
|
|
GENMD_EOF
|
|
|
|
## Unsparsing
|
|
|
|
The previous section discussed how to fill out missing data fields within CSV with full header line -- so the list of all field names is present within the header line. Next, let's look at a related problem: we have data where each record has various key names but we want to produce rectangular output having the union of all key names.
|
|
|
|
For example, suppose you have JSON input like this:
|
|
|
|
GENMD_RUN_COMMAND
|
|
cat data/sparse.json
|
|
GENMD_EOF
|
|
|
|
There are field names `a`, `b`, `v`, `u`, `x`, `w` in the data -- but not all in every record. Since we don't know the names of all the keys until we've read them all, this needs to be a two-pass algorithm. On the first pass, remember all the unique key names and all the records; on the second pass, loop through the records filling in absent values, then producing output. Use `put -q` since we don't want to produce per-record output, only emitting output in the `end` block:
|
|
|
|
GENMD_RUN_COMMAND
|
|
cat data/unsparsify.mlr
|
|
GENMD_EOF
|
|
|
|
GENMD_RUN_COMMAND
|
|
mlr --json put -q -f data/unsparsify.mlr data/sparse.json
|
|
GENMD_EOF
|
|
|
|
GENMD_RUN_COMMAND
|
|
mlr --ijson --ocsv put -q -f data/unsparsify.mlr data/sparse.json
|
|
GENMD_EOF
|
|
|
|
GENMD_RUN_COMMAND
|
|
mlr --ijson --opprint put -q -f data/unsparsify.mlr data/sparse.json
|
|
GENMD_EOF
|
|
|
|
There is a keystroke-saving verb for this: [unsparsify](reference-verbs.md#unsparsify).
|
|
|
|
## Mean without/with oosvars
|
|
|
|
GENMD_RUN_COMMAND
|
|
mlr --opprint stats1 -a mean -f x data/medium
|
|
GENMD_EOF
|
|
|
|
GENMD_INCLUDE_AND_RUN_ESCAPED(data/mean-with-oosvars.sh)
|
|
|
|
## Keyed mean without/with oosvars
|
|
|
|
GENMD_RUN_COMMAND
|
|
mlr --opprint stats1 -a mean -f x -g a,b data/medium
|
|
GENMD_EOF
|
|
|
|
GENMD_INCLUDE_AND_RUN_ESCAPED(data/keyed-mean-with-oosvars.sh)
|
|
|
|
## Variance and standard deviation without/with oosvars
|
|
|
|
GENMD_RUN_COMMAND
|
|
mlr --oxtab stats1 -a count,sum,mean,var,stddev -f x data/medium
|
|
GENMD_EOF
|
|
|
|
GENMD_RUN_COMMAND
|
|
cat variance.mlr
|
|
GENMD_EOF
|
|
|
|
GENMD_RUN_COMMAND
|
|
mlr --oxtab put -q -f variance.mlr data/medium
|
|
GENMD_EOF
|
|
|
|
You can also do this keyed, of course, imitating the keyed-mean example above.
|
|
|
|
## Min/max without/with oosvars
|
|
|
|
GENMD_RUN_COMMAND
|
|
mlr --oxtab stats1 -a min,max -f x data/medium
|
|
GENMD_EOF
|
|
|
|
GENMD_RUN_COMMAND
|
|
mlr --oxtab put -q '
|
|
@x_min = min(@x_min, $x);
|
|
@x_max = max(@x_max, $x);
|
|
end{emitf @x_min, @x_max}
|
|
' data/medium
|
|
GENMD_EOF
|
|
|
|
## Keyed min/max without/with oosvars
|
|
|
|
GENMD_RUN_COMMAND
|
|
mlr --opprint stats1 -a min,max -f x -g a data/medium
|
|
GENMD_EOF
|
|
|
|
GENMD_INCLUDE_AND_RUN_ESCAPED(data/keyed-min-max-with-oosvars.sh)
|
|
|
|
## Delta without/with oosvars
|
|
|
|
GENMD_RUN_COMMAND
|
|
mlr --opprint step -a delta -f x data/small
|
|
GENMD_EOF
|
|
|
|
GENMD_RUN_COMMAND
|
|
mlr --opprint put '
|
|
$x_delta = is_present(@last) ? $x - @last : 0;
|
|
@last = $x
|
|
' data/small
|
|
GENMD_EOF
|
|
|
|
## Keyed delta without/with oosvars
|
|
|
|
GENMD_RUN_COMMAND
|
|
mlr --opprint step -a delta -f x -g a data/small
|
|
GENMD_EOF
|
|
|
|
GENMD_RUN_COMMAND
|
|
mlr --opprint put '
|
|
$x_delta = is_present(@last[$a]) ? $x - @last[$a] : 0;
|
|
@last[$a]=$x
|
|
' data/small
|
|
GENMD_EOF
|
|
|
|
## Exponentially weighted moving averages without/with oosvars
|
|
|
|
GENMD_INCLUDE_AND_RUN_ESCAPED(verb-example-ewma.sh)
|
|
|
|
GENMD_INCLUDE_AND_RUN_ESCAPED(oosvar-example-ewma.sh)
|