Miller
User info:
• About
• File formats
• Miller features in the context of the Unix toolkit
• Record-heterogeneity
• Performance
• Why call it Miller?
• How original is Miller?
• Reference
• Data examples
• Things to do
Developer info:
• Compiling, portability, dependencies, and testing
• Why C?
• Contact information
• GitHub repo

Reference

Contents: • Command overview
• On-line help
• then-chaining
• I/O options
    • Formats
    • Record/field/pair separators
    • Number formatting
• Data transformations
    • cat
    • count-distinct
    • cut
    • filter
    • group-by
    • group-like
    • having-fields
    • head
    • histogram
    • put
    • rename
    • reorder
    • sort
    • stats1
    • stats2
    • step
    • tac
    • tail
    • top
    • uniq
• Functions for filter and put

Command overview

Whereas the Unix toolkit is made of the separate executables cat, tail, cut, sort, etc., Miller has subcommands, invoked as follows:

mlr tac *.dat
mlr cut --complement -f os_version *.dat
mlr sort -f hostname,uptime *.dat

These falls into categories as follows:

Commands	Description
`cat`, `cut`, `head`, `sort`, `tac`, `tail`, `top`, `uniq`	Analogs of their Unix-toolkit namesakes, discussed below as well as in Miller features in the context of the Unix toolkit
`filter`, `put`, `step`	`awk`-like functionality
`histogram`, `stats1`, `stats2`	Statistically oriented
`group-by`, `group-like`, `having-fields`	Particularly oriented toward Record-heterogeneity, although all Miller commands can handle heterogeneous records
`count-distinct`, `rename`	These draw from other sources (see also How original is Miller?): `count-distinct` is SQL-ish, and `rename` can be done by `sed` (which does it faster: see Performance).

On-line help

Examples:

$ mlr --help | fold -s
Usage: mlr [I/O options] {verb} [verb-dependent options ...] {file names}
verbs:
cat check count-distinct cut filter group-by group-like having-fields head
histogram put rename reorder sort stats1 stats2 step tac tail top uniq
Please use "mlr {verb name} --help" for verb-specific help.

I/O options:
  --rs      --irs     --ors
  --fs      --ifs     --ofs    --repifs
  --ps      --ips     --ops
  --dkvp    --idkvp   --odkvp
  --nidx    --inidx   --onidx
  --csv     --icsv    --ocsv
  --pprint  --ipprint --opprint --right
  --xtab    --ixtab   --oxtab
  --ofmt

$ mlr sort --help
Usage: mlr sort {flags}
Flags:
  -f  {comma-separated field names}  Lexical ascending
  -n  {comma-separated field names}  Numerical ascending
  -nf {comma-separated field names}  Numerical ascending
  -r  {comma-separated field names}  Lexical descending
  -nr {comma-separated field names}  Numerical descending
Example:
  mlr sort -f a,b -nr x,y,z

then-chaining

In accord with the Unix philosophy, you can pipe data into or out of Miller. For example:

mlr cut --complement -f os_version *.dat | mlr sort -f hostname,uptime

For better performance (avoiding redundant string-parsing and string-formatting when you pipe Miller commands together) you can, if you like, instead simply chain commands together using the then keyword:

mlr cut --complement -f os_version then sort -f hostname,uptime *.dat

I/O options

Formats

Options:

  --dkvp    --idkvp   --odkvp
  --nidx    --inidx   --onidx
  --csv     --icsv    --ocsv
  --pprint  --ipprint --ppprint --right
  --xtab    --ixtab   --oxtab

These are as discussed in File formats, with the exception of --right which makes pretty-printed output right-aligned:

$ mlr --opprint cat data/small
a   b   i x                   y
pan pan 1 0.3467901443380824  0.7268028627434533
eks pan 2 0.7586799647899636  0.5221511083334797
wye wye 3 0.20460330576630303 0.33831852551664776
eks wye 4 0.38139939387114097 0.13418874328430463
wye pan 5 0.5732889198020006  0.8636244699032729

$ mlr --opprint --right cat data/small
  a   b i                   x                   y
pan pan 1  0.3467901443380824  0.7268028627434533
eks pan 2  0.7586799647899636  0.5221511083334797
wye wye 3 0.20460330576630303 0.33831852551664776
eks wye 4 0.38139939387114097 0.13418874328430463
wye pan 5  0.5732889198020006  0.8636244699032729

Additional notes:

Use --csv, --pprint, etc. when the input and output formats are the same.
Use --icsv --opprint, etc. when you want format conversion as part of what Miller does to your data.
DKVP (key-value-pair) format is the default for input and output. So, --oxtab is the same as --idkvp --oxtab.

Record/field/pair separators

Miller has record separators IRS and ORS, field separators IFS and OFS, and pair separators IPS and OPS. For example, in the DKVP line a=1,b=2,c=3, the record separator is newline, field separator is comma, and pair separator is the equals sign. These are the default values.

Options:

  --rs --irs --ors
  --fs --ifs --ofs --repifs
  --ps --ips --ops

You can change a separator from input to output via e.g. --ifs = --ofs :. Or, you can specify that the same separator is to be used for input and output via e.g. --fs :.
The pair separator is only relevant to DKVP format.
Pretty-print and xtab formats ignore the separator arguments altogether.
The --repifs means that multiple successive occurrences of the field separator count as one. For example, in CSV data we often signify nulls by empty strings, e.g. 2,9,,,,,6,5,4. On the other hand, if the field separator is a space, it might be more natural to parse 2 4 5 the same as 2 4 5: --repifs --ifs ' ' lets this happen. In fact, the --ipprint option above is internally implemented in terms of --repifs.
Just write out the desired separator, e.g. --ofs '|'. But you may use the symbolic names newline, space, tab, pipe, or semicolon if you like.

Number formatting

Options:

  --ofmt {format string}

This is the global number format for commands which generate numeric output, e.g. stats1, stats2, histogram, and step. Examples:

--ofmt %.9le  --ofmt %.6lf  --ofmt %.0lf

These are just C printf formats applied to double-precision numbers. Please don’t use %s or %d. Additionally, if you use leading with (e.g. %18.12lf) then the output will contain embedded whitespace, which may not be what you want if you pipe the output to something else.

Data transformations

cat

Most useful for format conversions (see File formats), and concatenating multiple same-schema CSV files to have the same header:

$ cat a.csv
a,b,c
1,2,3
4,5,6

$ cat b.csv
a,b,c
7,8,9

$ mlr --csv cat a.csv b.csv
a,b,c
1,2,3
4,5,6
7,8,9

$ mlr --icsv --oxtab cat a.csv b.csv
a 1
b 2
c 3

a 4
b 5
c 6

a 7
b 8
c 9

count-distinct

$ mlr count-distinct --help
Usage: mlr count-distinct [options]
-f {a,b,c}   Field names for distinct count.

$ mlr count-distinct -f a,b then sort -nr count data/medium
a=zee,b=wye,count=455
a=pan,b=eks,count=429
a=pan,b=pan,count=427
a=wye,b=hat,count=426
a=hat,b=wye,count=423
a=pan,b=hat,count=417
a=eks,b=hat,count=417
a=eks,b=eks,count=413
a=pan,b=zee,count=413
a=zee,b=hat,count=409
a=eks,b=wye,count=407
a=zee,b=zee,count=403
a=pan,b=wye,count=395
a=wye,b=pan,count=392
a=zee,b=eks,count=391
a=zee,b=pan,count=389
a=hat,b=eks,count=389
a=wye,b=eks,count=386
a=hat,b=zee,count=385
a=wye,b=zee,count=385
a=hat,b=hat,count=381
a=wye,b=wye,count=377
a=eks,b=pan,count=371
a=hat,b=pan,count=363
a=eks,b=zee,count=357

cut

$ mlr cut --help
Usage: mlr cut [options]
-f {a,b,c}       Field names to cut.
-x|--complement  Exclude, rather that include, field names specified by -f.

Note that cut doesn’t reorder field names — for that, use reorder.

$ mlr --opprint cat data/small
a   b   i x                   y
pan pan 1 0.3467901443380824  0.7268028627434533
eks pan 2 0.7586799647899636  0.5221511083334797
wye wye 3 0.20460330576630303 0.33831852551664776
eks wye 4 0.38139939387114097 0.13418874328430463
wye pan 5 0.5732889198020006  0.8636244699032729

$ mlr --opprint cut -f y,x,i data/small
i x                   y
1 0.3467901443380824  0.7268028627434533
2 0.7586799647899636  0.5221511083334797
3 0.20460330576630303 0.33831852551664776
4 0.38139939387114097 0.13418874328430463
5 0.5732889198020006  0.8636244699032729

filter

$ mlr filter --help
Usage: mlr filter [options]
[-v] {expression} xxx needs more doc here please.

Field names must be specified using a $ in filter expressions, even though they don’t appear in the data stream. For integer-indexed data, this looks like awk’s $1,$2,$3.

$ ruby -e '10.times{|i|puts "i=#{i}"}' | mlr --opprint put '$j=$i+1;$k=$i+$j'
i j         k
0 1.000000  1.000000
1 2.000000  3.000000
2 3.000000  5.000000
3 4.000000  7.000000
4 5.000000  9.000000
5 6.000000  11.000000
6 7.000000  13.000000
7 8.000000  15.000000
8 9.000000  17.000000
9 10.000000 19.000000

The filter command supports the same built-in variables as for put, all awk-inspired: NF, NR, FNR, FILENUM, and FILENAME. This selects the 2nd record from each matching file:

$ mlr filter 'FNR == 2' data/small*
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
pan=pan,1=1,0.3467901443380824=0.3467901443380824,0.7268028627434533=0.7268028627434533
a=wye,b=eks,i=10000,x=0.734806020620654365,y=0.884788571337605134

Expressions may be arbitrarily complex:

$ mlr --opprint filter '$a == "pan" || $b == "wye"' data/small
a   b   i x                   y
pan pan 1 0.3467901443380824  0.7268028627434533
wye wye 3 0.20460330576630303 0.33831852551664776
eks wye 4 0.38139939387114097 0.13418874328430463

$ mlr --opprint filter '($x > 0.5 && $y > 0.5) || ($x < 0.5 && $y < 0.5)' then stats2 -a corr -f x,y data/medium
x_y_corr
0.756439

$ mlr --opprint filter '($x > 0.5 && $y < 0.5) || ($x < 0.5 && $y > 0.5)' then stats2 -a corr -f x,y data/medium
x_y_corr
-0.747994

group-by

$ mlr group-by --help
Usage: mlr group-by {comma-separated field names}

This is similar to sort but with less work. Namely, Miller’s sort has three steps: read through the data and append linked lists of records, one for each unique combination of the key-field values; after all records are read, sort the key-field values; then print each record-list. The group-by operation simply omits the middle sort. An example should make this more clear.

$ mlr --opprint group-by a data/small
a   b   i x                   y
pan pan 1 0.3467901443380824  0.7268028627434533
eks pan 2 0.7586799647899636  0.5221511083334797
eks wye 4 0.38139939387114097 0.13418874328430463
wye wye 3 0.20460330576630303 0.33831852551664776
wye pan 5 0.5732889198020006  0.8636244699032729

$ mlr --opprint sort -f a data/small
a   b   i x                   y
eks pan 2 0.7586799647899636  0.5221511083334797
eks wye 4 0.38139939387114097 0.13418874328430463
pan pan 1 0.3467901443380824  0.7268028627434533
wye wye 3 0.20460330576630303 0.33831852551664776
wye pan 5 0.5732889198020006  0.8636244699032729

In this example, since the sort is on field a, the first step is to group together all records having the same value for field a; the second step is to sort the distinct a-field values pan, eks, and wye into eks, pan, and wye; the third step is to print out the record-list for a=eks, then the record-list for a=pan, then the record-list for a=wye. The group-by operation omits the middle sort and just puts like records together, for those times when a sort isn’t desired. In particular, the ordering of group-by fields for group-by is the order in which they were encountered in the data stream, which in some cases may be more interesting to you.

group-like

$ mlr group-like --help
Usage: mlr group-like

This groups together records having the same schema (i.e. same ordered list of field names) which is useful for making sense of time-ordered output as described in Record-heterogeneity — in particular, in preparation for CSV or pretty-print output.

$ mlr cat data/het.dkvp
resource=/path/to/file,loadsec=0.45,ok=true
record_count=100,resource=/path/to/file
resource=/path/to/second/file,loadsec=0.32,ok=true
record_count=150,resource=/path/to/second/file
resource=/some/other/path,loadsec=0.97,ok=false

$ mlr --opprint group-like data/het.dkvp
resource             loadsec ok
/path/to/file        0.45    true
/path/to/second/file 0.32    true
/some/other/path     0.97    false

record_count resource
100          /path/to/file
150          /path/to/second/file

having-fields

$ mlr having-fields --help
Usage: mlr having-fields [options]
--at-least {a,b,c}
--which-are {a,b,c}
--at-most {a,b,c}

Similar to group-like, this retains records with specified schema.

$ mlr cat data/het.dkvp
resource=/path/to/file,loadsec=0.45,ok=true
record_count=100,resource=/path/to/file
resource=/path/to/second/file,loadsec=0.32,ok=true
record_count=150,resource=/path/to/second/file
resource=/some/other/path,loadsec=0.97,ok=false

$ mlr having-fields --at-least resource data/het.dkvp
resource=/path/to/file,loadsec=0.45,ok=true
record_count=100,resource=/path/to/file
resource=/path/to/second/file,loadsec=0.32,ok=true
record_count=150,resource=/path/to/second/file
resource=/some/other/path,loadsec=0.97,ok=false

$ mlr having-fields --which-are resource,ok,loadsec data/het.dkvp
resource=/path/to/file,loadsec=0.45,ok=true
resource=/path/to/second/file,loadsec=0.32,ok=true
resource=/some/other/path,loadsec=0.97,ok=false

head

$ mlr head --help
Usage: mlr head [options]
-n {count}    Head count to print; default 10
-g {a,b,c}    Group-by-field names for head counts

Note that head is distinct from top — head shows fields which appear first in the data stream; top shows fields which are numerically largest (or smallest).

$ mlr --opprint head -n 4 data/medium
a   b   i x                   y
pan pan 1 0.3467901443380824  0.7268028627434533
eks pan 2 0.7586799647899636  0.5221511083334797
wye wye 3 0.20460330576630303 0.33831852551664776
eks wye 4 0.38139939387114097 0.13418874328430463

$ mlr --opprint head -n 1 -g b data/medium
a   b   i  x                   y
pan pan 1  0.3467901443380824  0.7268028627434533
wye wye 3  0.20460330576630303 0.33831852551664776
eks zee 7  0.6117840605678454  0.1878849191181694
zee eks 17 0.29081949506712723 0.054478717073354166
wye hat 24 0.7286126830627567  0.19441962592638418

histogram

$ mlr histogram --help
Usage: mlr histogram [options]
-f {a,b,c}    Value-field names for histogram counts
--lo {lo}     Histogram low value
--hi {hi}     Histogram high value
--nbins {n}   Number of histogram bins

This is just a histogram; there’s not too much to say here. A note about binning, by example: Suppose you use --lo 0.0 --hi 1.0 --nbins 10 -f x. The input numbers less than 0 or greater than 1 aren’t counted in any bin. Input numbers equal to 1 are counted in the last bin. That is, bin 0 has 0.0 ≤ x < 0.1, bin 1 has 0.1 ≤ x < 0.2, etc., but bin 9 has 0.9 ≤ x ≤ 1.0.

$ mlr --opprint put '$x2=$x**2;$x3=$x2*$x' then histogram -f x,x2,x3 --lo 0 --hi 1 --nbins 10 data/medium
bin_lo   bin_hi   x_count x2_count x3_count
0.000000 0.100000 1072    3231     4661
0.100000 0.200000 938     1254     1184
0.200000 0.300000 1037    988      845
0.300000 0.400000 988     832      676
0.400000 0.500000 950     774      576
0.500000 0.600000 1002    692      476
0.600000 0.700000 1007    591      438
0.700000 0.800000 1007    560      420
0.800000 0.900000 986     571      383
0.900000 1.000000 1013    507      341

put

$ mlr put --help
Usage: mlr put [options]
[-v] {expression} xxx needs more doc here please.

Field names must be specified using a $ in put expressions, even though they don’t appear in the data stream. For integer-indexed data, this looks like awk’s $1,$2,$3. Multiple expressions may be given, separated by semicolons, and each may refer to the ones before.

$ ruby -e '10.times{|i|puts "i=#{i}"}' | mlr --opprint put '$j=$i+1;$k=$i+$j'
i j         k
0 1.000000  1.000000
1 2.000000  3.000000
2 3.000000  5.000000
3 4.000000  7.000000
4 5.000000  9.000000
5 6.000000  11.000000
6 7.000000  13.000000
7 8.000000  15.000000
8 9.000000  17.000000
9 10.000000 19.000000

Miller supports the following five built-in variables for filter and put, all awk-inspired: NF, NR, FNR, FILENUM, and FILENAME.

$ mlr --opprint put '$nf=NF; $nr=NR; $fnr=FNR; $filenum=FILENUM; $filename=FILENAME' data/small data/small2
a   b   i     x                    y                    nf nr fnr filenum filename
pan pan 1     0.3467901443380824   0.7268028627434533   5  1  1   1       data/small
eks pan 2     0.7586799647899636   0.5221511083334797   5  2  2   1       data/small
wye wye 3     0.20460330576630303  0.33831852551664776  5  3  3   1       data/small
eks wye 4     0.38139939387114097  0.13418874328430463  5  4  4   1       data/small
wye pan 5     0.5732889198020006   0.8636244699032729   5  5  5   1       data/small
pan eks 9999  0.267481232652199086 0.557077185510228001 5  6  1   2       data/small2
wye eks 10000 0.734806020620654365 0.884788571337605134 5  7  2   2       data/small2
pan wye 10001 0.870530722602517626 0.009854780514656930 5  8  3   2       data/small2
hat wye 10002 0.321507044286237609 0.568893318795083758 5  9  4   2       data/small2
pan zee 10003 0.272054845593895200 0.425789896597056627 5  10 5   2       data/small2

rename

$ mlr rename --help
Usage: mlr rename {old1,new1,old2,new2,...}

$ mlr --opprint cat data/small
a   b   i x                   y
pan pan 1 0.3467901443380824  0.7268028627434533
eks pan 2 0.7586799647899636  0.5221511083334797
wye wye 3 0.20460330576630303 0.33831852551664776
eks wye 4 0.38139939387114097 0.13418874328430463
wye pan 5 0.5732889198020006  0.8636244699032729

$ mlr --opprint rename i,INDEX,b,COLUMN2 data/small
a   COLUMN2 INDEX x                   y
pan pan     1     0.3467901443380824  0.7268028627434533
eks pan     2     0.7586799647899636  0.5221511083334797
wye wye     3     0.20460330576630303 0.33831852551664776
eks wye     4     0.38139939387114097 0.13418874328430463
wye pan     5     0.5732889198020006  0.8636244699032729

As discussed in Performance, sed is significantly faster than Miller at doing this. However, Miller is format-aware, so it knows to do renames only within specified field keys and not any others, nor in field values which may happen to contain the same pattern. Example:

$ sed 's/y/COLUMN5/g' data/small
a=pan,b=pan,i=1,x=0.3467901443380824,COLUMN5=0.7268028627434533
a=eks,b=pan,i=2,x=0.7586799647899636,COLUMN5=0.5221511083334797
a=wCOLUMN5e,b=wCOLUMN5e,i=3,x=0.20460330576630303,COLUMN5=0.33831852551664776
a=eks,b=wCOLUMN5e,i=4,x=0.38139939387114097,COLUMN5=0.13418874328430463
a=wCOLUMN5e,b=pan,i=5,x=0.5732889198020006,COLUMN5=0.8636244699032729

$ mlr rename y,COLUMN5 data/small
a=pan,b=pan,i=1,x=0.3467901443380824,COLUMN5=0.7268028627434533
a=eks,b=pan,i=2,x=0.7586799647899636,COLUMN5=0.5221511083334797
a=wye,b=wye,i=3,x=0.20460330576630303,COLUMN5=0.33831852551664776
a=eks,b=wye,i=4,x=0.38139939387114097,COLUMN5=0.13418874328430463
a=wye,b=pan,i=5,x=0.5732889198020006,COLUMN5=0.8636244699032729

reorder

$ mlr reorder --help
Usage: mlr reorder [options]
-f {a,b,c}       Field names to reorder.
-e           Put specified field names at record end: default is to put at record start.
Example: mlr reorder    -f a,b sends input record d=4,b=2,a=1,c=3 to a=1,b=2,d=4,c=3.
Example: mlr reorder -e -f a,b sends input record d=4,b=2,a=1,c=3 to d=4,c=3,a=1,b=2.

This pivots specified field names to the start or end of the record — for example when you have highly multi-column data and you want to bring a field or two to the front of line where you can give a quick visual scan.

$ mlr --opprint cat data/small
a   b   i x                   y
pan pan 1 0.3467901443380824  0.7268028627434533
eks pan 2 0.7586799647899636  0.5221511083334797
wye wye 3 0.20460330576630303 0.33831852551664776
eks wye 4 0.38139939387114097 0.13418874328430463
wye pan 5 0.5732889198020006  0.8636244699032729

$ mlr --opprint reorder -f i,b data/small
i b   a   x                   y
1 pan pan 0.3467901443380824  0.7268028627434533
2 pan eks 0.7586799647899636  0.5221511083334797
3 wye wye 0.20460330576630303 0.33831852551664776
4 wye eks 0.38139939387114097 0.13418874328430463
5 pan wye 0.5732889198020006  0.8636244699032729

$ mlr --opprint reorder -e -f i,b data/small
a   x                   y                   i b
pan 0.3467901443380824  0.7268028627434533  1 pan
eks 0.7586799647899636  0.5221511083334797  2 pan
wye 0.20460330576630303 0.33831852551664776 3 wye
eks 0.38139939387114097 0.13418874328430463 4 wye
wye 0.5732889198020006  0.8636244699032729  5 pan

sort

$ mlr sort --help
Usage: mlr sort {flags}
Flags:
  -f  {comma-separated field names}  Lexical ascending
  -n  {comma-separated field names}  Numerical ascending
  -nf {comma-separated field names}  Numerical ascending
  -r  {comma-separated field names}  Lexical descending
  -nr {comma-separated field names}  Numerical descending
Example:
  mlr sort -f a,b -nr x,y,z

xxx write up after -n/-r.

stats1

$ mlr stats1 --help
Usage: mlr stats1 [options]
-a {sum,count,...}    Names of accumulators: one or more of
                      count sum avg stddev avgeb min max
-f {a,b,c}            Value-field names on which to compute statistics
-g {d,e,f}            Group-by-field names

These are simple univariate statistics on one or more number-valued fields, optionally categorized by one or more fields.

$ mlr --oxtab stats1 -a count,sum,avg -f x,y data/medium
x_count 10000
x_sum   4986.019682
x_avg   0.498602
y_count 10000
y_sum   5062.057445
y_avg   0.506206

$ mlr --opprint stats1 -a avg -f x,y -g b then sort -f b data/medium
b   x_avg    y_avg
eks 0.506361 0.510293
hat 0.487899 0.513118
pan 0.497304 0.499599
wye 0.497593 0.504596
zee 0.504242 0.502997

stats2

$ mlr stats2 --help
Usage: mlr stats2 [options]
-a {linreg-ols,corr,...}    Names of accumulators: one or more of
                      linreg-ols r2 corr cov covx linreg-pca                      r2 is a quality metric for linreg-ols; linrec-pca outputs its own quality metric.
-f {a,b,c,d}          Value-field names on which to compute statistics.
                      There must be an even number of these.
-g {d,e,f}            Group-by-field names
-v                    Print additional output for linreg-pca.

These are simple bivariate statistics on one or more pairs of number-valued fields, optionally categorized by one or more fields.

$ mlr --oxtab put '$x2=$x*$x; $xy=$x*$y; $y2=$y**2' then stats2 -a cov,corr -f x,y,y,y,x2,xy,x2,y2 data/medium
x_y_cov    0.000043
x_y_corr   0.000504
y_y_cov    0.084611
y_y_corr   1.000000
x2_xy_cov  0.041884
x2_xy_corr 0.630174
x2_y2_cov  -0.000310
x2_y2_corr -0.003425

$ mlr --opprint put '$x2=$x*$x; $xy=$x*$y; $y2=$y**2' then stats2 -a linreg-ols,r2 -f x,y,y,y,xy,y2 -g a data/medium
a   x_y_ols_m x_y_ols_b x_y_r2   y_y_ols_m y_y_ols_b y_y_r2   xy_y2_ols_m xy_y2_ols_b xy_y2_r2
pan 0.017026  0.500403  0.000287 1.000000  0.000000  1.000000 0.878132    0.119082    0.417498
eks 0.040780  0.481402  0.001646 1.000000  0.000000  1.000000 0.897873    0.107341    0.455632
wye -0.039153 0.525510  0.001505 1.000000  0.000000  1.000000 0.853832    0.126745    0.389917
zee 0.002781  0.504307  0.000008 1.000000  0.000000  1.000000 0.852444    0.124017    0.393566
hat -0.018621 0.517901  0.000352 1.000000  0.000000  1.000000 0.841230    0.135573    0.368794

Here’s an example simple line-fit. The x and y fields of the data/medium dataset are just independent uniformly distributed on the unit interval. Here we remove half the data and fit a line to it.


mlr filter '($x<.5 && $y<.5) || ($x>.5 && $y>.5)' data/medium > data/medium-squares

mlr --ofs newline stats2 -a linreg-pca -f x,y data/medium-squares
x_y_pca_m=1.014419
x_y_pca_b=0.000308
x_y_pca_quality=0.861354

# Set x_y_pca_m and x_y_pca_b as shell variables
eval $(mlr --ofs newline stats2 -a linreg-pca -f x,y data/medium-squares)

# In addition to x and y, make a new yfit which is the line fit. Plot using your favorite tool.
mlr --onidx put '$yfit='$x_y_pca_m'*$x+'$x_y_pca_b then cut -x -f a,b,i data/medium-squares \
  | pgr -p -title 'linreg-pca example' -xmin 0 -xmax 1 -ymin 0 -ymax 1

I use pgr for plotting; here’s a screenshot.

(Thanks Drew Kunas for a good conversation about PCA!)

step

$ mlr step --help
Usage: mlr step [options]
-a {delta,rsum,...}    Names of steppers: one or more of
                      delta rsum counter
-f {a,b,c}            Value-field names on which to compute statistics
-g {d,e,f}            Group-by-field names

Most Miller commands are record-at-a-time, with the exception of stats1, stats2, and histogram which compute aggregate output. The step command is intermediate: it allows the option of adding fields which are functions of fields from previous records. Rsum is short for running sum.

$ mlr --opprint step -a delta,rsum,counter -f x data/medium | head -15
a   b   i     x                      y                      x_delta   x_rsum      x_counter
pan pan 1     0.3467901443380824     0.7268028627434533     0.346790  0.346790    1
eks pan 2     0.7586799647899636     0.5221511083334797     0.411890  1.105470    2
wye wye 3     0.20460330576630303    0.33831852551664776    -0.554077 1.310073    3
eks wye 4     0.38139939387114097    0.13418874328430463    0.176796  1.691473    4
wye pan 5     0.5732889198020006     0.8636244699032729     0.191890  2.264762    5
zee pan 6     0.5271261600918548     0.49322128674835697    -0.046163 2.791888    6
eks zee 7     0.6117840605678454     0.1878849191181694     0.084658  3.403672    7
zee wye 8     0.5985540091064224     0.976181385699006      -0.013230 4.002226    8
hat wye 9     0.03144187646093577    0.7495507603507059     -0.567112 4.033668    9
pan wye 10    0.5026260055412137     0.9526183602969864     0.471184  4.536294    10
pan pan 11    0.7930488423451967     0.6505816637259333     0.290423  5.329343    11
zee pan 12    0.3676141320555616     0.23614420670296965    -0.425435 5.696957    12
eks pan 13    0.4915175580479536     0.7709126592971468     0.123903  6.188474    13
eks zee 14    0.5207382318405251     0.34141681118811673    0.029221  6.709213    14

$ mlr --opprint step -a delta,rsum,counter -f x -g a data/medium | head -15
a   b   i     x                      y                      x_delta   x_rsum      x_counter
pan pan 1     0.3467901443380824     0.7268028627434533     0.346790  0.346790    1
eks pan 2     0.7586799647899636     0.5221511083334797     0.758680  0.758680    1
wye wye 3     0.20460330576630303    0.33831852551664776    0.204603  0.204603    1
eks wye 4     0.38139939387114097    0.13418874328430463    -0.377281 1.140079    2
wye pan 5     0.5732889198020006     0.8636244699032729     0.368686  0.777892    2
zee pan 6     0.5271261600918548     0.49322128674835697    0.527126  0.527126    1
eks zee 7     0.6117840605678454     0.1878849191181694     0.230385  1.751863    3
zee wye 8     0.5985540091064224     0.976181385699006      0.071428  1.125680    2
hat wye 9     0.03144187646093577    0.7495507603507059     0.031442  0.031442    1
pan wye 10    0.5026260055412137     0.9526183602969864     0.155836  0.849416    2
pan pan 11    0.7930488423451967     0.6505816637259333     0.290423  1.642465    3
zee pan 12    0.3676141320555616     0.23614420670296965    -0.230940 1.493294    3
eks pan 13    0.4915175580479536     0.7709126592971468     -0.120267 2.243381    4
eks zee 14    0.5207382318405251     0.34141681118811673    0.029221  2.764119    5

tac

$ mlr tac --help
Usage: mlr tac

Prints the records in the input stream in reverse order. Note: this requires Miller to retain all input records in memory before any output records are produced.

$ mlr --icsv --opprint cat a.csv
a b c
1 2 3
4 5 6

$ mlr --icsv --opprint cat b.csv
a b c
7 8 9

$ mlr --icsv --opprint tac a.csv b.csv
a b c
7 8 9
4 5 6
1 2 3

$ mlr --icsv --opprint put '$filename=FILENAME' then tac a.csv b.csv
a b c filename
7 8 9 b.csv
4 5 6 a.csv
1 2 3 a.csv

tail

$ mlr tail --help
Usage: mlr tail [options]
-n {count}    Tail count to print; default 10
-g {a,b,c}    Group-by-field names for tail counts

Prints the last n records in the input stream, optionally by category.

$ mlr --opprint tail -n 4 data/colored-shapes.dkvp
color  shape    flag i      u                  v                  w                   x
yellow circle   1    99997  0.5228034832314841 0.7478634261534541 0.49477944033468396 6.085638633037881
red    triangle 0    99998  0.8566019561040149 0.5583785393850178 0.4993735796215503  6.393409471109115
yellow triangle 1    99999  0.5369350176939407 0.5197619334387739 0.5064468446479313  3.2682256831629695
green  square   0    100000 0.0277485352321325 0.5303062901341336 0.5274344049261097  5.806843329974349

$ mlr --opprint tail -n 1 -g shape data/colored-shapes.dkvp
color  shape    flag i      u                  v                  w                   x
yellow circle   1    99997  0.5228034832314841 0.7478634261534541 0.49477944033468396 6.085638633037881
green  square   0    100000 0.0277485352321325 0.5303062901341336 0.5274344049261097  5.806843329974349
yellow triangle 1    99999  0.5369350176939407 0.5197619334387739 0.5064468446479313  3.2682256831629695

top

$ mlr top --help
Usage: mlr top [options]
-f {a,b,c}    Value-field names for top counts
-g {d,e,f}    Group-by-field names for top counts
-n {count}    Top n records to print; default 1
-a            Print all fields for top-value records; default is
              to print only value and group-by fields.
--min         Print top smallest values; default is top largest values

Note that top is distinct from head — head shows fields which appear first in the data stream; top shows fields which are numerically largest (or smallest).

$ mlr --opprint top -n 4 -f x data/medium
top_idx x_top
1       0.999953
2       0.999823
3       0.999733
4       0.999563

$ mlr --opprint top -n 2 -f x -g a then sort -f a data/medium
a   top_idx x_top
eks 1       0.998811
eks 2       0.998534
hat 1       0.999953
hat 2       0.999733
pan 1       0.999403
pan 2       0.999044
wye 1       0.999823
wye 2       0.999264
zee 1       0.999490
zee 2       0.999438

uniq

$ mlr uniq --help
Usage: mlr uniq [options]
-g {d,e,f}    Group-by-field names for uniq counts
-c            Show repeat counts in addition to unique values

$ wc -l data/colored-shapes.dkvp
  100000 data/colored-shapes.dkvp

$ mlr uniq -g color,shape data/colored-shapes.dkvp
color=green,shape=circle
color=red,shape=square
color=yellow,shape=circle
color=red,shape=circle
color=blue,shape=circle
color=purple,shape=triangle
color=blue,shape=triangle
color=green,shape=square
color=red,shape=triangle
color=yellow,shape=triangle
color=purple,shape=square
color=blue,shape=square
color=yellow,shape=square
color=green,shape=triangle
color=purple,shape=circle
color=orange,shape=triangle
color=orange,shape=square
color=orange,shape=circle

$ mlr --opprint uniq -g color,shape -c then sort -f color,shape data/colored-shapes.dkvp
color  shape    count
blue   circle   3578
blue   square   6016
blue   triangle 4843
green  circle   2832
green  square   4678
green  triangle 3924
orange circle   705
orange square   1196
orange triangle 954
purple circle   2861
purple square   4808
purple triangle 3841
red    circle   11477
red    square   19051
red    triangle 15248
yellow circle   3482
yellow square   5839
yellow triangle 4667

Functions for filter and put

Miller’s filter and put support the following operators and functions:

Operators/functions	Description
`==`, `!=`, `<`, `<=`, `>`, `>=`, `&&`, `\|\|`	Filter-only. Comparisons are string-valued or number-valued depending on absence/presence of double-quotes in the literal value: `mlr filter '$color != "blue" && $value > 4.2'`
`+`, `-` (unary or binary), ``, `/`, `*`	Number-valued.
`.`	String concatenation
`systime` (seconds since epoch), `urand`	Functions of no arguments returning numbers
`abs`, `ceil`, `cos`, `exp`, `floor`, `log`, `log10`, `pow`, `round`, `sin`, `sqrt`, `tan`	Number-to-number functions with one argument
`atan2`	Number-to-number functions with two arguments
`tolower`, `toupper`	String-to-string functions with one argument

See also the awk-like built-in variables NF, NR, FNR, FILENUM, and FILENAME as described in the section on put.