miller/doc/content-for-reference.html

POKI_PUT_TOC_HERE

<h1>Command overview</h1>

<p>
Whereas the Unix toolkit is made of the separate executables <tt>cat</tt>, <tt>tail</tt>, <tt>cut</tt>,
<tt>sort</tt>, etc., Miller has subcommands, invoked as follows:

POKI_INCLUDE_ESCAPED(data/subcommand-example.txt)HERE

<p/>These fall into categories as follows:

<table border=1>
  <tr class="mlrbg">
    <th>Commands </th>
    <th>Description</th>
  </tr>
  <tr>
    <td>
      <a href="#cat"><tt>cat</tt></a>,
      <a href="#cut"><tt>cut</tt></a>,
      <a href="#head"><tt>head</tt></a>,
      <a href="#sort"><tt>sort</tt></a>,
      <a href="#tac"><tt>tac</tt></a>,
      <a href="#tail"><tt>tail</tt></a>,
      <a href="#top"><tt>top</tt></a>,
      <a href="#uniq"><tt>uniq</tt></a>
    </td>
    <td> Analogs of their Unix-toolkit namesakes, discussed below as well as in
    POKI_PUT_LINK_FOR_PAGE(feature-comparison.html)HERE </td>
  </tr>

  <tr>
    <td>
      <a href="#filter"><tt>filter</tt></a>,
      <a href="#put"><tt>put</tt></a>,
      <a href="#step"><tt>step</tt></a>
    </td>
    <td> <tt>awk</tt>-like functionality </td>
  </tr>

  <tr>
    <td>
      <a href="#histogram"><tt>histogram</tt></a>,
      <a href="#stats1"><tt>stats1</tt></a>,
      <a href="#stats2"><tt>stats2</tt></a>
    </td>
    <td> Statistically oriented </td>
  </tr>

  <tr>
    <td>
      <a href="#group-by"><tt>group-by</tt></a>,
      <a href="#group-like"><tt>group-like</tt></a>,
      <a href="#having-fields"><tt>having-fields</tt></a>
    </td>
    <td> Particularly oriented toward POKI_PUT_LINK_FOR_PAGE(record-heterogeneity.html)HERE, although
      all Miller commands can handle heterogeneous records
  </tr>

  <tr>
    <td>
      <a href="#count-distinct"><tt>count-distinct</tt></a>,
      <a href="#label"><tt>label</tt></a>,
      <a href="#regularize"><tt>rename</tt></a>,
      <a href="#rename"><tt>rename</tt></a>,
      <a href="#reorder"><tt>reorder</tt></a>
    </td>
    <td> These draw from other sources (see also POKI_PUT_LINK_FOR_PAGE(originality.html)HERE):
      <a href="#count-distinct"><tt>count-distinct</tt></a> is SQL-ish, and
      <a href="#rename"><tt>rename</tt></a> can be done by <tt>sed</tt> (which does it faster:
        see POKI_PUT_LINK_FOR_PAGE(performance.html)HERE).
    </td>
  </tr>

</table>

<h1>On-line help</h1>

<p/>Examples:<p/>

POKI_RUN_COMMAND{{mlr --help}}HERE

POKI_RUN_COMMAND{{mlr sort --help}}HERE

<!-- ================================================================ -->
<h1>I/O options</h1>

<!-- ================================================================ -->
<h2>Formats</h2>

<p/> Options:

<pre>
  --dkvp    --idkvp    --odkvp
  --nidx    --inidx    --onidx
  --csv     --icsv     --ocsv
  --csvlite --icsvlite --ocsvlite
  --pprint  --ipprint  --ppprint  --right
  --xtab    --ixtab    --oxtab
  --json    --ijson    --ojson
</pre>

<p/> These are as discussed in POKI_PUT_LINK_FOR_PAGE(file-formats.html)HERE, with the exception of <tt>--right</tt>
which makes pretty-printed output right-aligned:

<table><tr><td>
POKI_RUN_COMMAND{{mlr --opprint cat data/small}}HERE
</td><td>
POKI_RUN_COMMAND{{mlr --opprint --right cat data/small}}HERE
</td></tr></table>

<p/>Additional notes:

<ul>

<li/> Use <tt>--csv</tt>, <tt>--pprint</tt>, etc. when the input and output formats are the same.

<li/> Use <tt>--icsv --opprint</tt>, etc. when you want format conversion as part of what Miller does to your data.

<li/> DKVP (key-value-pair) format is the default for input and output. So,
<tt>--oxtab</tt> is the same as <tt>--idkvp --oxtab</tt>.

</ul>

<!-- ================================================================ -->
<h2>Compression</h2>

<p/> Options:

<pre>
  --prepipe {command}
</pre>

<p/>The prepipe command is anything which reads from standard input and produces data acceptable to
Miller. Nominally this allows you to use whichever decompression utilities you have installed on your
system, on a per-file basis. If the command has flags, quote them: e.g. <tt>mlr --prepipe 'zcat -cf'</tt>. Examples:

<p/>
<div class="pokipanel">
<pre>
# These two produce the same output:
$ gunzip &lt; myfile1.csv.gz | mlr cut -f hostname,uptime
$ mlr --prepipe gunzip cut -f hostname,uptime myfile1.csv.gz
# With multiple input files you need --prepipe:
$ mlr --prepipe gunzip cut -f hostname,uptime myfile1.csv.gz myfile2.csv.gz
$ mlr --prepipe gunzip --idkvp --oxtab cut -f hostname,uptime myfile1.dat.gz myfile2.dat.gz

# Similar to the above, but with compressed output as well as input:
$ gunzip &lt; myfile1.csv.gz | mlr cut -f hostname,uptime | gzip &gt; outfile.csv.gz
$ mlr --prepipe gunzip cut -f hostname,uptime myfile1.csv.gz | gzip &gt; outfile.csv.gz
$ mlr --prepipe gunzip cut -f hostname,uptime myfile1.csv.gz myfile2.csv.gz | gzip &gt; outfile.csv.gz

# Similar to the above, but with different compression tools for input and output:
$ gunzip &lt; myfile1.csv.gz | mlr cut -f hostname,uptime | xz -z &gt; outfile.csv.xz
$ xz -cd &lt; myfile1.csv.xz | mlr cut -f hostname,uptime | gzip &gt; outfile.csv.xz
$ mlr --prepipe 'xz -cd' cut -f hostname,uptime myfile1.csv.xz myfile2.csv.xz | xz -z &gt; outfile.csv.xz

... etc.
</pre>
</div>

<!-- ================================================================ -->
<h2>Record/field/pair separators</h2>

<p/> Miller has record separators <tt>IRS</tt> and <tt>ORS</tt>, field
separators <tt>IFS</tt> and <tt>OFS</tt>, and pair separators <tt>IPS</tt> and
<tt>OPS</tt>.  For example, in the DKVP line <tt>a=1,b=2,c=3</tt>, the record
separator is newline, field separator is comma, and pair separator is the
equals sign. These are the default values.

<p/> Options:
<pre>
  --rs --irs --ors
  --fs --ifs --ofs --repifs
  --ps --ips --ops
</pre>

<ul>

<li/> You can change a separator from input to output via e.g. <tt>--ifs =
--ofs :</tt>. Or, you can specify that the same separator is to be used for
input and output via e.g. <tt>--fs :</tt>.

<li/> The pair separator is only relevant to DKVP format.

<li/> Pretty-print and xtab formats ignore the separator arguments altogether.

<li/> The <tt>--repifs</tt> means that multiple successive occurrences of the
field separator count as one.  For example, in CSV data we often signify nulls
by empty strings, e.g. <tt>2,9,,,,,6,5,4</tt>. On the other hand, if the field
separator is a space, it might be more natural to parse <tt>2 4    5</tt> the
same as <tt>2 4 5</tt>: <tt>--repifs --ifs ' '</tt> lets this happen.  In fact,
the <tt>--ipprint</tt> option above is internally implemented in terms of
<tt>--repifs</tt>.

<li/> Just write out the desired separator, e.g. <tt>--ofs '|'</tt>. But you
may use the symbolic names <tt>newline</tt>, <tt>space</tt>, <tt>tab</tt>,
<tt>pipe</tt>, or <tt>semicolon</tt> if you like.

</ul>

<!-- ================================================================ -->
<h2>Number formatting</h2>

<p/> The command-line option <tt>--ofmt {format string}</tt> is the global
number format for commands which generate numeric output, e.g.
<tt>stats1</tt>, <tt>stats2</tt>, <tt>histogram</tt>, and <tt>step</tt>, as
well as <tt>mlr put</tt>. Examples:

POKI_CARDIFY(--ofmt %.9le  --ofmt %.6lf  --ofmt %.0lf)HERE

<p/> These are just C <tt>printf</tt> formats applied to double-precision
numbers.  Please don&rsquo;t use <tt>%s</tt> or <tt>%d</tt>. Additionally, if
you use leading width (e.g. <tt>%18.12lf</tt>) then the output will contain
embedded whitespace, which may not be what you want if you pipe the output to
something else, particularly CSV. I use Miller&rsquo;s pretty-print format
(<tt>mlr --opprint</tt>) to column-align numerical data.

<p/> To apply formatting to a single field, overriding the global
<tt>ofmt</tt>, use <tt>fmtnum</tt> function within <tt>mlr put</tt>. For example:
POKI_RUN_COMMAND{{echo 'x=3.1,y=4.3' | mlr put '$z=fmtnum($x*$y,"%08lf")'}}HERE
POKI_RUN_COMMAND{{echo 'x=0xffff,y=0xff' | mlr put '$z=fmtnum(int($x*$y),"%08llx")'}}HERE

<p/>Input conversion from hexadecimal is done automatically on fields handled
by <tt>mlr put</tt> and <tt>mlr filter</tt> as long as the field value begins
with "0x".  To apply output conversion to hexadecimal on a single column, you
may use <tt>fmtnum</tt>, or the keystroke-saving <tt>hexfmt</tt> function.
Example:

POKI_RUN_COMMAND{{echo 'x=0xffff,y=0xff' | mlr put '$z=hexfmt($x*$y)'}}HERE

<!-- ================================================================ -->
<h1>Data transformations</h1>

<!-- ================================================================ -->
<h2>bar</h2>

<p/> Cheesy bar-charting.

POKI_RUN_COMMAND{{mlr bar -h}}HERE

POKI_RUN_COMMAND{{mlr --opprint cat data/small}}HERE

POKI_RUN_COMMAND{{mlr --opprint bar --lo 0 --hi 1 -f x,y data/small}}HERE

POKI_RUN_COMMAND{{mlr --opprint bar --lo 0.4 --hi 0.6 -f x,y data/small}}HERE

POKI_RUN_COMMAND{{mlr --opprint bar --auto -f x,y data/small}}HERE

<!-- ================================================================ -->
<h2>bootstrap</h2>

POKI_RUN_COMMAND{{mlr bootstrap --help}}HERE

<p/> The canonical use for bootstrap sampling is to put error bars on statistical quantities, such as mean. For example:

<p/>
<div class="pokipanel">
<pre>
$ mlr --opprint stats1 -a mean,count -f u -g color data/colored-shapes.dkvp
color  u_mean   u_count
yellow 0.497129 1413
red    0.492560 4641
purple 0.494005 1142
green  0.504861 1109
blue   0.517717 1470
orange 0.490532 303

$ mlr --opprint bootstrap then stats1 -a mean,count -f u -g color data/colored-shapes.dkvp
color  u_mean   u_count
yellow 0.500651 1380
purple 0.501556 1111
green  0.503272 1068
red    0.493895 4702
blue   0.512529 1496
orange 0.521030 321

$ mlr --opprint bootstrap then stats1 -a mean,count -f u -g color data/colored-shapes.dkvp
color  u_mean   u_count
yellow 0.498046 1485
blue   0.513576 1417
red    0.492870 4595
orange 0.507697 307
green  0.496803 1075
purple 0.486337 1199

$ mlr --opprint bootstrap then stats1 -a mean,count -f u -g color data/colored-shapes.dkvp
color  u_mean   u_count
blue   0.522921 1447
red    0.490717 4617
yellow 0.496450 1419
purple 0.496523 1192
green  0.507569 1111
orange 0.468014 292
</pre>
</div>
<p/>

<!-- ================================================================ -->
<h2>cat</h2>

<p/> Most useful for format conversions (see
POKI_PUT_LINK_FOR_PAGE(file-formats.html)HERE), and concatenating multiple
same-schema CSV files to have the same header:

POKI_RUN_COMMAND{{mlr cat -h}}HERE

<table><tr><td>
POKI_RUN_COMMAND{{cat data/a.csv}}HERE
</td> <td>
POKI_RUN_COMMAND{{cat data/b.csv}}HERE
</td> <td>
POKI_RUN_COMMAND{{mlr --csv cat data/a.csv data/b.csv}}HERE
</td></tr></table>

<table><tr><td>
</td> <td>
POKI_RUN_COMMAND{{mlr --icsv --oxtab cat data/a.csv data/b.csv}}HERE
</td> <td>
POKI_RUN_COMMAND{{mlr --csv cat -n data/a.csv data/b.csv}}HERE
</td></tr></table>

<!-- ================================================================ -->
<h2>check</h2>

POKI_RUN_COMMAND{{mlr check --help}}HERE

<!-- ================================================================ -->
<h2>count-distinct</h2>

POKI_RUN_COMMAND{{mlr count-distinct --help}}HERE

POKI_RUN_COMMAND{{mlr count-distinct -f a,b then sort -nr count data/medium}}HERE

POKI_RUN_COMMAND{{mlr count-distinct -n -f a,b data/medium}}HERE

<!-- ================================================================ -->
<h2>cut</h2>

POKI_RUN_COMMAND{{mlr cut --help}}HERE

<table><tr><td>
POKI_RUN_COMMAND{{mlr --opprint cat data/small}}HERE
</td><td>
POKI_RUN_COMMAND{{mlr --opprint cut -f y,x,i data/small}}HERE
</td></tr><tr><td>
POKI_RUN_COMMAND{{echo 'a=1,b=2,c=3' | mlr cut -f b,c,a}}HERE
</td><td>
POKI_RUN_COMMAND{{echo 'a=1,b=2,c=3' | mlr cut -o -f b,c,a}}HERE
</td></tr></table>

<!-- ================================================================ -->
<h2>decimate</h2>

POKI_RUN_COMMAND{{mlr decimate --help}}HERE

<p/>
<!-- ================================================================ -->
<h2>filter</h2>

POKI_RUN_COMMAND{{mlr filter --help}}HERE

<h3>Features which filter shares with put</h3>

<p/>Please see <a href="#Expression_language_for_filter_and_put">Expression
language for filter and put</a> for more information about the expression
language for <tt>mlr filter</tt>.

<!-- ================================================================ -->
<h2>grep</h2>

POKI_RUN_COMMAND{{mlr grep -h}}HERE

<!-- ================================================================ -->
<h2>group-by</h2>

POKI_RUN_COMMAND{{mlr group-by --help}}HERE

<p/>This is similar to <tt>sort</tt> but with less work. Namely, Miller&rsquo;s
sort has three steps: read through the data and append linked lists of records,
one for each unique combination of the key-field values; after all records
are read, sort the key-field values; then print each record-list. The group-by
operation simply omits the middle sort.  An example should make this more
clear.

<table><tr> <td>
POKI_RUN_COMMAND{{mlr --opprint group-by a data/small}}HERE
</td> <td>
POKI_RUN_COMMAND{{mlr --opprint sort -f a data/small}}HERE
</td> </tr></table>

<p/>In this example, since the sort is on field <tt>a</tt>, the first step is
to group together all records having the same value for field <tt>a</tt>; the
second step is to sort the distinct <tt>a</tt>-field values <tt>pan</tt>,
<tt>eks</tt>, and <tt>wye</tt> into <tt>eks</tt>, <tt>pan</tt>, and
<tt>wye</tt>; the third step is to print out the record-list for
<tt>a=eks</tt>, then the record-list for <tt>a=pan</tt>, then the record-list
for <tt>a=wye</tt>.  The group-by operation omits the middle sort and just puts
like records together, for those times when a sort isn&rsquo;t desired. In
particular, the ordering of group-by fields for group-by is the order in which
they were encountered in the data stream, which in some cases may be more interesting
to you.

<!-- ================================================================ -->
<h2>group-like</h2>

POKI_RUN_COMMAND{{mlr group-like --help}}HERE

<p/> This groups together records having the same schema (i.e. same ordered list of field names)
which is useful for making sense of time-ordered output as described in
POKI_PUT_LINK_FOR_PAGE(record-heterogeneity.html)HERE &mdash; in particular, in
preparation for CSV or pretty-print output.

<table><tr><td>
POKI_RUN_COMMAND{{mlr cat data/het.dkvp}}HERE
</td><td>
POKI_RUN_COMMAND{{mlr --opprint group-like data/het.dkvp}}HERE
</td></tr></table>

<!-- ================================================================ -->
<h2>having-fields</h2>

POKI_RUN_COMMAND{{mlr having-fields --help}}HERE

<p/> Similar to <a href="#group-like"><tt>group-like</tt></a>, this retains records with specified schema.

<table><tr><td>
POKI_RUN_COMMAND{{mlr cat data/het.dkvp}}HERE
</td></tr><tr><td>
POKI_RUN_COMMAND{{mlr having-fields --at-least resource data/het.dkvp}}HERE
</td></tr><tr><td>
POKI_RUN_COMMAND{{mlr having-fields --which-are resource,ok,loadsec data/het.dkvp}}HERE
</td></tr></table>

<!-- ================================================================ -->
<h2>head</h2>

POKI_RUN_COMMAND{{mlr head --help}}HERE

Note that <tt>head</tt> is distinct from <a href="#top"><tt>top</tt></a>
&mdash; <tt>head</tt> shows fields which appear first in the data stream;
<tt>top</tt> shows fields which are numerically largest (or smallest).

<table><tr><td>
POKI_RUN_COMMAND{{mlr --opprint head -n 4 data/medium}}HERE
</td><td>
POKI_RUN_COMMAND{{mlr --opprint head -n 1 -g b data/medium}}HERE
</td></tr></table>

<!-- ================================================================ -->
<h2>histogram</h2>

POKI_RUN_COMMAND{{mlr histogram --help}}HERE

This is just a histogram; there&rsquo;s not too much to say here. A note about
binning, by example: Suppose you use <tt>--lo 0.0 --hi 1.0 --nbins 10 -f
x</tt>.  The input numbers less than 0 or greater than 1 aren&rsquo;t counted
in any bin.  Input numbers equal to 1 are counted in the last bin. That is, bin
0 has <tt>0.0 &le; x &lt; 0.1</tt>, bin 1 has <tt>0.1 &le; x &lt; 0.2</tt>,
etc., but bin 9 has <tt>0.9 &le; x &le; 1.0</tt>.

POKI_RUN_COMMAND{{mlr --opprint put '$x2=$x**2;$x3=$x2*$x' then histogram -f x,x2,x3 --lo 0 --hi 1 --nbins 10 data/medium}}HERE

<!-- ================================================================ -->
<h2>join</h2>

POKI_RUN_COMMAND{{mlr join --help}}HERE

Examples:

<p/>Join larger table with IDs with smaller ID-to-name lookup table, showing only paired records:

<table><tr><td>
POKI_RUN_COMMAND{{mlr --icsvlite --opprint cat data/join-left-example.csv}}HERE
</td></tr><tr><td>
POKI_RUN_COMMAND{{mlr --icsvlite --opprint cat data/join-right-example.csv}}HERE
</td></tr><tr><td>
POKI_RUN_COMMAND{{mlr --icsvlite --opprint join -u -j id -r idcode -f data/join-left-example.csv data/join-right-example.csv}}HERE
</td></tr></table>

<p/>Same, but with sorting the input first:
<table><tr><td>
POKI_RUN_COMMAND{{mlr --icsvlite --opprint sort -f idcode then join -j id -r idcode -f data/join-left-example.csv data/join-right-example.csv}}HERE
</td></tr></table>

<p/>Same, but showing only unpaired records:
<table><tr><td>
POKI_RUN_COMMAND{{mlr --icsvlite --opprint join --np --ul --ur -u -j id -r idcode -f data/join-left-example.csv data/join-right-example.csv}}HERE
</td></tr></table>

<p/>Use prefixing options to disambiguate between otherwise identical non-join field names:
<table><tr><td>
POKI_RUN_COMMAND{{mlr --csvlite --opprint cat data/self-join.csv data/self-join.csv}}HERE
</td></tr><tr><td>
POKI_RUN_COMMAND{{mlr --csvlite --opprint join -j a --lp left_ --rp right_ -f data/self-join.csv data/self-join.csv}}HERE
</td></tr></table>

<p/>Use zero join columns:
<table><tr><td>
POKI_RUN_COMMAND{{mlr --csvlite --opprint join -j "" --lp left_ --rp right_ -f data/self-join.csv data/self-join.csv}}HERE
</td></tr></table>

<!-- ================================================================ -->
<h2>label</h2>

POKI_RUN_COMMAND{{mlr label --help}}HERE

See also <a href="#rename"><tt>rename</tt></a>.

<p/>Example: Files such as <tt>/etc/passwd</tt>, <tt>/etc/group</tt>, and so on
have implicit field names which are found in section-5 manpages. These field names may be made explicit as follows:

POKI_INCLUDE_ESCAPED(data/label-example.txt)HERE

<p/>Likewise, if you have CSV/CSV-lite input data which has somehow been bereft of its header line, you can re-add a header line using <tt>--implicit-csv-header</tt> and <tt>label</tt>:

POKI_RUN_COMMAND{{cat data/headerless.csv}}HERE
POKI_RUN_COMMAND{{mlr --csv --rs lf --implicit-csv-header cat data/headerless.csv}}HERE
POKI_RUN_COMMAND{{mlr --csv --rs lf --implicit-csv-header label name,age,status data/headerless.csv}}HERE
POKI_RUN_COMMAND{{mlr --icsv --rs lf --implicit-csv-header --opprint label name,age,status data/headerless.csv}}HERE

<!-- ================================================================ -->
<h2>least-frequent</h2>

POKI_RUN_COMMAND{{mlr least-frequent -h}}HERE

POKI_RUN_COMMAND{{mlr --opprint --from data/colored-shapes.dkvp least-frequent -f shape -n 5}}HERE

POKI_RUN_COMMAND{{mlr --opprint --from data/colored-shapes.dkvp least-frequent -f shape,color -n 5}}HERE

POKI_RUN_COMMAND{{mlr --opprint --from data/colored-shapes.dkvp least-frequent -f shape,color -n 5 -b}}HERE

See also <a href="#most-frequent">most-frequent</a>.

<!-- ================================================================ -->
<h2>merge-fields</h2>

POKI_RUN_COMMAND{{mlr merge-fields --help}}HERE

<p/>This is like <tt>mlr stats1</tt> but all accumulation is done across fields
within each given record: horizontal rather than vertical statistics, if you
will.

<p/>Examples:

POKI_RUN_COMMAND{{mlr --csvlite --opprint cat data/inout.csv}}HERE
POKI_RUN_COMMAND{{mlr --csvlite --opprint merge-fields -a min,max,sum -c _in,_out data/inout.csv}}HERE
POKI_RUN_COMMAND{{mlr --csvlite --opprint merge-fields -k -a sum -c _in,_out data/inout.csv}}HERE

<!-- ================================================================ -->
<h2>most-frequent</h2>

POKI_RUN_COMMAND{{mlr most-frequent -h}}HERE

POKI_RUN_COMMAND{{mlr --opprint --from data/colored-shapes.dkvp most-frequent -f shape -n 5}}HERE

POKI_RUN_COMMAND{{mlr --opprint --from data/colored-shapes.dkvp most-frequent -f shape,color -n 5}}HERE

POKI_RUN_COMMAND{{mlr --opprint --from data/colored-shapes.dkvp most-frequent -f shape,color -n 5 -b}}HERE

See also <a href="#least-frequent">least-frequent</a>.

<!-- ================================================================ -->
<h2>nest</h2>

POKI_RUN_COMMAND{{mlr nest -h}}HERE

<!-- ================================================================ -->
<h2>nothing</h2>

POKI_RUN_COMMAND{{mlr nothing -h}}HERE

<!-- ================================================================ -->
<h2>put</h2>

POKI_RUN_COMMAND{{mlr put --help}}HERE

<h3>Features which put shares with filter</h3>

<p/>Please see <a href="#Expression_language_for_filter_and_put">Expression
language for filter and put</a> for more information about the expression
language for <tt>mlr put</tt>.

<!-- ================================================================ -->
<h2>regularize</h2>

POKI_RUN_COMMAND{{mlr regularize --help}}HERE

<p/>This exists since hash-map software in various languages and tools
encountered in the wild does not always print similar rows with fields in the
same order: <tt>mlr regularize</tt> helps clean that up.

<p/>See also <a href="#reorder"><tt>reorder</tt></a>.

<!-- ================================================================ -->
<h2>rename</h2>

POKI_RUN_COMMAND{{mlr rename --help}}HERE

<table><tr><td>
POKI_RUN_COMMAND{{mlr --opprint cat data/small}}HERE
</td><td>
POKI_RUN_COMMAND{{mlr --opprint rename i,INDEX,b,COLUMN2 data/small}}HERE
</td></tr></table>

<p/>As discussed in POKI_PUT_LINK_FOR_PAGE(performance.html)HERE, <tt>sed</tt>
is significantly faster than Miller at doing this. However, Miller is
format-aware, so it knows to do renames only within specified field keys and
not any others, nor in field values which may happen to contain the same
pattern. Example:

<table><tr><td>
POKI_RUN_COMMAND{{sed 's/y/COLUMN5/g' data/small}}HERE
</td></tr><tr><td>
POKI_RUN_COMMAND{{mlr rename y,COLUMN5 data/small}}HERE
</td></tr></table>

See also <a href="#label"><tt>label</tt></a>.

<!-- ================================================================ -->
<h2>reorder</h2>

POKI_RUN_COMMAND{{mlr reorder --help}}HERE

This pivots specified field names to the start or end of the record &mdash; for
example when you have highly multi-column data and you want to bring a field or
two to the front of line where you can give a quick visual scan.

<table><tr><td>
POKI_RUN_COMMAND{{mlr --opprint cat data/small}}HERE
</td></tr><tr><td>
POKI_RUN_COMMAND{{mlr --opprint reorder -f i,b data/small}}HERE
</td><td>
POKI_RUN_COMMAND{{mlr --opprint reorder -e -f i,b data/small}}HERE
</td></tr></table>

<!-- ================================================================ -->
<h2>repeat</h2>

POKI_RUN_COMMAND{{mlr repeat --help}}HERE

<p>This is useful in at least two ways: one, as a data-generator as in the
above example using <tt>urand()</tt>; two, for reconstructing individual
samples from data which has been count-aggregated:

POKI_RUN_COMMAND{{cat data/repeat-example.dat}}HERE

POKI_RUN_COMMAND{{mlr repeat -f count then cut -x -f count data/repeat-example.dat}}HERE

<p>After expansion with <tt>repeat</tt>, such data can then be sent on to
<tt>stats1 -a mode</tt>, or (if the data are numeric) to <tt>stats1 -a
p10,p50,p90</tt>, etc.

<!-- ================================================================ -->
<h2>reshape</h2>

POKI_RUN_COMMAND{{mlr reshape --help}}HERE

<!-- ================================================================ -->
<h2>sample</h2>

POKI_RUN_COMMAND{{mlr sample --help}}HERE

<p/>This is reservoir-sampling: select <i>k</i> items from <i>n</i> with
uniform probability and no repeats in the sample. (If <i>n</i> is less than
<i>k</i>, then of course only <i>n</i> samples are produced.) With <tt>-g
{field names}</tt>, produce a <i>k</i>-sample for each distinct value of the
specified field names.

POKI_INCLUDE_ESCAPED(data/sample-example.txt)HERE

<p/>Note that no output is produced until all inputs are in. Another way to do
sampling, which works in the streaming case, is <tt>mlr filter 'urand() &amp;
0.001'</tt> where you tune the 0.001 to meet your needs.

<!-- ================================================================ -->
<h2>sec2gmt</h2>

POKI_RUN_COMMAND{{mlr sec2gmt -h}}HERE

<!-- ================================================================ -->
<h2>sec2gmtdate</h2>

POKI_RUN_COMMAND{{mlr sec2gmtdate -h}}HERE

<!-- ================================================================ -->
<h2>seqgen</h2>

POKI_RUN_COMMAND{{mlr seqgen -h}}HERE

POKI_RUN_COMMAND{{mlr seqgen --stop 10}}HERE
POKI_RUN_COMMAND{{mlr seqgen --start 20 --stop 40 --step 4}}HERE
POKI_RUN_COMMAND{{mlr seqgen --start 40 --stop 20 --step -4}}HERE

<!-- ================================================================ -->
<h2>shuffle</h2>

POKI_RUN_COMMAND{{mlr shuffle -h}}HERE

<!-- ================================================================ -->
<h2>sort</h2>

POKI_RUN_COMMAND{{mlr sort --help}}HERE

<p/>Example:
POKI_RUN_COMMAND{{mlr --opprint sort -f a -nr x data/small}}HERE

<p/>Here&rsquo;s an example filtering log data: suppose multiple threads (labeled here by color) are all logging progress counts to a single log file. The log file is (by nature) chronological, so the progress of various threads is interleaved:

POKI_RUN_COMMAND{{head -n 10 data/multicountdown.dat}}HERE

<p/> We can group these by thread by sorting on the thread ID (here,
<tt>color</tt>). Since Miller&rsquo;s sort is stable, this means that
timestamps within each thread&rsquo;s log data are still chronological:

POKI_RUN_COMMAND{{head -n 20 data/multicountdown.dat | mlr --opprint sort -f color}}HERE

<p/>Any records not having all specified sort keys will appear at the end of the output, in the order they
were encountered, regardless of the specified sort order:

POKI_RUN_COMMAND{{mlr sort -n  x data/sort-missing.dkvp}}HERE
POKI_RUN_COMMAND{{mlr sort -nr x data/sort-missing.dkvp}}HERE

<!-- ================================================================ -->
<h2>stats1</h2>

POKI_RUN_COMMAND{{mlr stats1 --help}}HERE

These are simple univariate statistics on one or more number-valued fields
(<tt>count</tt> and <tt>mode</tt> apply to non-numeric fields as well),
optionally categorized by one or more other fields.

<table><tr><td>
POKI_RUN_COMMAND{{mlr --oxtab stats1 -a count,sum,min,p10,p50,mean,p90,max -f x,y data/medium}}HERE
</td></tr><tr><td>
POKI_RUN_COMMAND{{mlr --opprint stats1 -a mean -f x,y -g b then sort -f b data/medium}}HERE
</td></tr><tr><td>
POKI_RUN_COMMAND{{mlr --opprint stats1 -a p50,p99 -f u,v -g color then put '$ur=$u_p99/$u_p50;$vr=$v_p99/$v_p50' data/colored-shapes.dkvp}}HERE
</td></tr><tr><td>
POKI_RUN_COMMAND{{mlr --opprint count-distinct -f shape then sort -nr count data/colored-shapes.dkvp}}HERE
</td></tr><tr><td>
POKI_RUN_COMMAND{{mlr --opprint stats1 -a mode -f color -g shape data/colored-shapes.dkvp}}HERE
</td></tr></table>

<!-- ================================================================ -->
<h2>stats2</h2>

POKI_RUN_COMMAND{{mlr stats2 --help}}HERE

These are simple bivariate statistics on one or more pairs of number-valued
fields, optionally categorized by one or more fields.

<table><tr><td>
POKI_RUN_COMMAND{{mlr --oxtab put '$x2=$x*$x; $xy=$x*$y; $y2=$y**2' then stats2 -a cov,corr -f x,y,y,y,x2,xy,x2,y2 data/medium}}HERE
</td></tr><tr><td>
POKI_RUN_COMMAND{{mlr --opprint put '$x2=$x*$x; $xy=$x*$y; $y2=$y**2' then stats2 -a linreg-ols,r2 -f x,y,y,y,xy,y2 -g a data/medium}}HERE
</td></tr></table>

<p/>Here&rsquo;s an example simple line-fit. The <tt>x</tt> and <tt>y</tt>
fields of the <tt>data/medium</tt> dataset are just independent uniformly
distributed on the unit interval. Here we remove half the data and fit a line to it.

POKI_INCLUDE_ESCAPED(data/linreg-example.txt)HERE

<p/>I use <a href="https://github.com/johnkerl/pgr"><tt>pgr</tt></a> for
plotting; here&rsquo;s a screenshot.

<center>
<img src="data/linreg-example.jpg"/>
</center>

<p/> (Thanks Drew Kunas for a good conversation about PCA!)

<p/> Here&rsquo;s an example estimating time-to-completion for a set of jobs.
Input data comes from a log file, with number of work units left to do in the
<tt>count</tt> field and accumulated seconds in the <tt>upsec</tt> field,
labeled by the <tt>color</tt> field:

POKI_RUN_COMMAND{{head -n 10 data/multicountdown.dat}}HERE

We can do a linear regression on count remaining as a function of time: with <tt>c = m*u+b</tt> we want to find the
time when the count goes to zero, i.e. <tt>u=-b/m</tt>.

POKI_RUN_COMMAND{{mlr --oxtab stats2 -a linreg-pca -f upsec,count -g color then put '$donesec = -$upsec_count_pca_b/$upsec_count_pca_m' data/multicountdown.dat}}HERE

<!-- ================================================================ -->
<h2>step</h2>

POKI_RUN_COMMAND{{mlr step --help}}HERE

Most Miller commands are record-at-a-time, with the exception of <tt>stats1</tt>,
<tt>stats2</tt>, and <tt>histogram</tt> which compute aggregate output. The
<tt>step</tt> command is intermediate: it allows the option of adding fields
which are functions of fields from previous records. Rsum is short for <i>running sum</i>.

<table><tr><td>
POKI_RUN_COMMAND{{mlr --opprint step -a shift,delta,rsum,counter -f x data/medium | head -15}}HERE
</td></tr><tr><td>
POKI_RUN_COMMAND{{mlr --opprint step -a shift,delta,rsum,counter -f x -g a data/medium | head -15}}HERE
</td></tr><tr><td>
POKI_RUN_COMMAND{{mlr --opprint step -a ewma -f x -d 0.1,0.9 ../doc/data/medium | head -15}}HERE
</td></tr><tr><td>
POKI_RUN_COMMAND{{mlr --opprint step -a ewma -f x -d 0.1,0.9 -o smooth,rough ../doc/data/medium | head -15}}HERE
</td></tr></table>

Example deriving uptime-delta from system uptime:

POKI_INCLUDE_ESCAPED(data/ping-delta-example.txt)HERE

<!-- ================================================================ -->
<h2>tac</h2>

POKI_RUN_COMMAND{{mlr tac --help}}HERE

<p/>Prints the records in the input stream in reverse order. Note: this
requires Miller to retain all input records in memory before any output records
are produced.

<table><tr><td>
POKI_RUN_COMMAND{{mlr --icsv --opprint cat data/a.csv}}HERE
</td><td>
POKI_RUN_COMMAND{{mlr --icsv --opprint cat data/b.csv}}HERE
</td><td>
POKI_RUN_COMMAND{{mlr --icsv --opprint tac data/a.csv data/b.csv}}HERE
</td></tr></table>
<table><tr><td>
POKI_RUN_COMMAND{{mlr --icsv --opprint put '$filename=FILENAME' then tac data/a.csv data/b.csv}}HERE
</td></tr></table>

<!-- ================================================================ -->
<h2>tail</h2>

POKI_RUN_COMMAND{{mlr tail --help}}HERE

<p/> Prints the last <i>n</i> records in the input stream, optionally by category.

<table><tr><td>
POKI_RUN_COMMAND{{mlr --opprint tail -n 4 data/colored-shapes.dkvp}}HERE
</td></tr><tr><td>
POKI_RUN_COMMAND{{mlr --opprint tail -n 1 -g shape data/colored-shapes.dkvp}}HERE
</td></tr></table>

<!-- ================================================================ -->
<h2>tee</h2>

POKI_RUN_COMMAND{{mlr tee --help}}HERE

<!-- ================================================================ -->
<h2>top</h2>

POKI_RUN_COMMAND{{mlr top --help}}HERE

Note that <tt>top</tt> is distinct from <a href="#head"><tt>head</tt></a>
&mdash; <tt>head</tt> shows fields which appear first in the data stream;
<tt>top</tt> shows fields which are numerically largest (or smallest).

<table><tr><td>
POKI_RUN_COMMAND{{mlr --opprint top -n 4 -f x data/medium}}HERE
</td><td>
POKI_RUN_COMMAND{{mlr --opprint top -n 2 -f x -g a then sort -f a data/medium}}HERE
</td></tr></table>

<!-- ================================================================ -->
<h2>uniq</h2>

POKI_RUN_COMMAND{{mlr uniq --help}}HERE

<table><tr><td>
POKI_RUN_COMMAND{{wc -l data/colored-shapes.dkvp}}HERE
</td></tr><tr><td>
POKI_RUN_COMMAND{{mlr uniq -g color,shape data/colored-shapes.dkvp}}HERE
</td></tr><tr><td>
POKI_RUN_COMMAND{{mlr --opprint uniq -g color,shape -c then sort -f color,shape data/colored-shapes.dkvp}}HERE
</td></tr><tr><td>
POKI_RUN_COMMAND{{mlr --opprint uniq -n -g color,shape data/colored-shapes.dkvp}}HERE
</td></tr></table>

<!-- ================================================================ -->
<h1>Expression language for filter and put</h1>

The basic idea of <tt>mlr filter</tt> and <tt>mlr put</tt> are for
record-selection and record-updating expressions, respectively. For example:

POKI_RUN_COMMAND{{cat data/small}}HERE
POKI_RUN_COMMAND{{mlr filter '$a == "eks"' data/small}}HERE
POKI_RUN_COMMAND{{mlr put '$ab = $a . "_" . $b ' data/small}}HERE

<p/>The two are essentially the same command. The only differences are:
expressions sent to <tt>mlr filter</tt> must end with a boolean expression,
which is the filtering criterion; <tt>mlr filter</tt> expressions may not
reference the <tt>filter</tt> keyword within them; and <tt>mlr filter</tt>
expressions may not use <tt>tee</tt>, <tt>emit</tt>, <tt>emitp</tt>, or
<tt>emitf</tt>.  All the rest is the same: in particular, you can define and
invoke functions and subroutines to help produce the final boolean statement,
and record fields may be assigned to in the statements preceding the final
boolean statement.

<p/>There are more details and more choices, of course, as detailed in the following sections.

<!-- ================================================================ -->
<h2>Syntax</h2>

<!-- ================================================================ -->
<h3>Expression formatting</h3>

<p/>Multiple expressions may be given, separated by semicolons, and each may refer to the ones before:

POKI_RUN_COMMAND{{ruby -e '10.times{|i|puts "i=#{i}"}' | mlr --opprint put '$j = $i + 1; $k = $i +$j'}}HERE

Newlines within the expression are ignored, which can help increase legibility of complex expressions:

POKI_INCLUDE_AND_RUN_ESCAPED(data/put-multiline-example.txt)HERE

POKI_RUN_COMMAND{{mlr --opprint filter '($x > 0.5 && $y < 0.5) || ($x < 0.5 && $y > 0.5)' then stats2 -a corr -f x,y data/medium}}HERE

<!-- ================================================================ -->
<h3>Expressions from files</h3>

<p/>The simplest way to enter expressions for <tt>put</tt> and <tt>filter</tt> is between single quotes on the command line, e.g.

POKI_INCLUDE_AND_RUN_ESCAPED(data/fe-example-1.sh)HERE
POKI_INCLUDE_AND_RUN_ESCAPED(data/fe-example-2.sh)HERE

<p/>You may, though, find it convenient to put expressions into files for reuse, and read them
<b>using the -f option</b>. For example:

POKI_RUN_COMMAND{{cat data/fe-example-3.mlr}}HERE
POKI_RUN_COMMAND{{mlr --from data/small put -f data/fe-example-3.mlr}}HERE

<p/>If you have some of the logic in a file and you want to write the rest on the command line, you
can <b>use the -f and -e options</b>:

POKI_RUN_COMMAND{{cat data/fe-example-4.mlr}}HERE
POKI_RUN_COMMAND{{mlr --from data/small put -f data/fe-example-4.mlr -e '$xy = f($x, $y)'}}HERE

<p/>A suggested use-case here is defining functions in files, and calling them from command-line expressions.

<p/>Moreover, you can have one or more <tt>-f</tt> expressions (maybe one
function per file, for example) and one or more <tt>-e</tt> expressions on the
command line.  All the <tt>-f</tt>&rsquo;s will be evaluated in order, then all
the <tt>-e</tt>&rsquo;s will be evaluated in order.

<!-- ================================================================ -->
<h3>Semicolons, newlines, and curly braces</h3>

<p/>Miller uses semicolons as statement separators, not statement terminators. This means you can write:

POKI_INCLUDE_ESCAPED(data/semicolon-example.txt)HERE

<p/>Semicolons are optional after closing curly braces (which close conditionals and loops as discussed below).

POKI_RUN_COMMAND{{echo x=1,y=2 | mlr put 'while (NF < 10) { $[NF+1] = ""}  $foo = "bar"'}}HERE
POKI_RUN_COMMAND{{echo x=1,y=2 | mlr put 'while (NF < 10) { $[NF+1] = ""}; $foo = "bar"'}}HERE

<p/>Semicolons are required between statements even if those statements are on
separate lines.  Newlines are for your convenience but have no syntactic
meaning: line endings do not terminate statements. For example, adjacent
assignment statements must be separated by semicolons even if those statements
are on separate lines:

POKI_INCLUDE_ESCAPED(data/newline-example.txt)HERE

<p/>Bodies for all compound statements must be enclosed in curly braces, even if the body is a single statement:
POKI_CARDIFY{{mlr put 'if ($x == 1) $y = 2' # Syntax error}}HERE
POKI_CARDIFY{{mlr put 'if ($x == 1) { $y = 2 }' # This is OK}}HERE

<p/>Bodies for compound statements may be empty:

POKI_CARDIFY{{mlr put 'if ($x == 1) { }' # This no-op is syntactically acceptable}}HERE

<!-- ================================================================ -->
<h2>Variables</h2>

<p/>Miller has the following kinds of variables:

<p/> <b>Built-in variables</b> such as <tt>NF</tt>, <tt>NF</tt>,
<tt>FILENAME</tt>, <tt>PI</tt>, and <tt>E</tt>.  These are all capital letters
and are read-only (although some of them change value from one record to
another).

<p/> <b>Fields of stream records</b>, accessed using the <tt>$</tt> prefix.
These refer to fields of the current data-stream record. For example, in
<tt>echo x=1,y=2 | mlr put '$z = $x + $y'</tt>, <tt>$x</tt> and <tt>$y</tt>
refer to input fields, and <tt>$z</tt> refers to a new, computed output field.
In a few contexts, presented below, you can refer to the entire record as
<tt>$*</tt>.

<p/> <b>Out-of-stream variables</b> accessed using the <tt>@</tt> prefix. These
refer to data which persist from one record to the next, including in
<tt>begin</tt> and <tt>end</tt> blocks (which execute before/after the record
stream is consumed, respectively). You use them to remember values across
records, such as sums, differences, counters, and so on.  In a few contexts,
presented below, you can refer to the entire out-of-stream-variables collection
as <tt>@*</tt>.

<p/> <b>Local variables</b> are limited in scope and extent to the current
statements being executed: these include function arguments, bound variables in
for loops, and explicitly declared <tt>local</tt> variables.

<p/> <b>Keywords</b> are not variables, but since their names are reserved, you
cannot use these names for local variables.

<!-- ================================================================ -->
<h3>Built-in variables</h3>

<p/> These are written all in capital letters, such as <tt>NR</tt>,
<tt>NF</tt>, <tt>FILENAME</tt>, and only a small, specific set of them is
defined by Miller.

<p/>Miller supports the following five built-in variables for <a
href="#filter"><tt>filter</tt></a> and <tt>put</tt>, all <tt>awk</tt>-inspired:
<tt>NF</tt>, <tt>NR</tt>, <tt>FNR</tt>, <tt>FILENUM</tt>, and
<tt>FILENAME</tt>, as well as the mathematical constants <tt>PI</tt> and
<tt>E</tt>.  Lastly, the <tt>ENV</tt> hashmap allows read access to environment
variables, e.g.  <tt>ENV["HOME"]</tt> or <tt>ENV["foo_".$hostname]</tt>.

POKI_RUN_COMMAND{{mlr filter 'FNR == 2' data/small*}}HERE

POKI_RUN_COMMAND{{mlr put '$fnr = FNR' data/small*}}HERE

<p/> Their values of <tt>NF</tt>, <tt>NR</tt>, <tt>FNR</tt>, <tt>FILENUM</tt>,
and <tt>FILENAME</tt> change from one record to the next as Miller scans
through your input data stream. The mathematical constants, of course, do not
change; <tt>ENV</tt> is populated from the system environment variables at the
time Miller starts and is read-only for the remainder of program execution.

<p/> Their <b>scope is global</b>: you can refer to them in any <tt>filter</tt>
or <tt>put</tt> statement. Their values are assigned by the input-record
reader:

POKI_RUN_COMMAND{{mlr --csv put '$nr = NR' data/a.csv}}HERE
POKI_RUN_COMMAND{{mlr --csv repeat -n 3 then put '$nr = NR' data/a.csv}}HERE

<p/> The <b>extent</b> is for the duration of the put/filter: in a
<tt>begin</tt> statement (which executes before the first input record is
consumed) you will find <tt>NR=1</tt> and in an <tt>end</tt>statement (which is
executed after the last input record is consumed) you will find <tt>NR</tt> to
be the total number of records ingested.

<p/> These are all <b>read-only</b> for the <tt>mlr put</tt> and <tt>mlr
filter</tt> DSLs: they may be assigned from, e.g. <tt>$nr=NR</tt>, but they may
not be assigned to: <tt>NR=100</tt> is a syntax error.

<!-- ================================================================ -->
<h3>Field names</h3>

<p/>Field names must be specified using a <tt>$</tt> in <tt>filter</tt> and <a
href="#put"><tt>put</tt></a> expressions, even though the dollar signs
don&rsquo;t appear in the data stream. For integer-indexed data, this looks
like <tt>awk</tt>&rsquo;s <tt>$1,$2,$3</tt>, except that Miller allows
non-numeric names such as <tt>$quantity</tt> or <tt>$hostname</tt>.  (Likewise,
enclose string literals in double quotes in <tt>filter</tt> expressions even
though they don&rsquo;t appear in file data.  In particular, <tt>mlr filter
'$x=="abc"'</tt> passes through the record <tt>x=abc</tt>.)

<p/>If field names have <b>special characters</b> such as <tt>.</tt> then you can use
braces, e.g. <tt>'${field.name}'</tt>.

<p/>You may also use a <b>computed field name</b> in square brackets, e.g.

POKI_RUN_COMMAND{{echo a=3,b=4 | mlr filter '$["x"] < 0.5'}}HERE

POKI_RUN_COMMAND{{echo s=green,t=blue,a=3,b=4 | mlr put '$[$s."_".$t] = $a * $b'}}HERE

<p/> The names of record fields depend on the contents of your input data stream, and their
values change from one record to the next as Miller scans through your input
data stream.

<p/> Their <b>extent</b> is limited to the current record; their <b>scope</b>
is the <tt>filter</tt> or <tt>put</tt> command in which they appear.

<p/> These are <b>read-write</b>: you can do <tt>$y=2*$x</tt>,
<tt>$x=$x+1</tt>, etc.

<p/> Records are Miller&rsquo;s output: field names present in the input
stream are passed through to output (written to standard output) unless fields
are removed with <tt>cut</tt>, or records are excluded with <tt>filter</tt> or
<tt>put -q</tt>, etc. Simply assign a value to a field and it will be output.

<!-- ================================================================ -->
<h3>Local variables</h3>

<p/>There are three kinds of local variables: <b>arguments</b> to
functions/subroutines, <b>variables bound within for-loops</b>, and
<b>locals</b> defined within control blocks.

<p/>This example shows all three:

POKI_INCLUDE_AND_RUN_ESCAPED(data/local-example-1.sh)HERE

<p/>Notes:

<ul>

<li/> Parameter names are bound to their arguments but can be reassigned, e.g.
if there is a parameter named <tt>a</tt> then you can reassign the value of
<tt>a</tt> to be something else within the function if you like.

<li/> All argument-passing is positional rather than by name; arguments are
passed by value, not by reference.

<li/> You can define locals (using the <tt>local</tt> keyword) at any scope
(if-statementst, else-statements, while-loops, for-loops, or the top-level
scope), and nested scopes will have access (more details on scope in the next
section).  If you define a local variable with the same name inside an inner
scope, then a new variable is created with the narrower scope.

<li/> If you assign to a local variable for the first time in a scope without
declaring it as <tt>local</tt> then: if it exists in an outer scope, that
outer-scope variable will be updated; if not, it will be defined in the current
scope as if <tt>local</tt> had been used. I recommend always declaring
variables with <tt>local</tt> to make the intended scoping clear.

<li/> Functions and subroutines never have access to locals from their callee
(unless passed by value as arguments).

<li/> If a for-loop variable is defined using <tt>local</tt> then it is scoped
to that for-loop.

</ul>

<p/> The following example demonstrates the scope rules:

POKI_RUN_COMMAND{{cat data/scope-example.mlr}}HERE
POKI_RUN_COMMAND{{cat data/scope-example.dat}}HERE
POKI_RUN_COMMAND{{mlr --oxtab --from data/scope-example.dat put -f data/scope-example.mlr}}HERE

<!-- ================================================================ -->
<h3>Out-of-stream variables</h3>

<p/> These are prefixed with an at-sign, e.g. <tt>@sum</tt>.  Furthermore,
unlike built-in variables and stream-record fields, they are maintained in an
arbitrarily nested hashmap: you can do <tt>@sum += $quanity</tt>, or
<tt>@sum[$color] += $quanity</tt>, or <tt>@sum[$color][$shape] +=
$quanity</tt>. The keys for the multi-level hashmap can be any expression which
evaluates to string or integer: e.g.  <tt>@sum[NR] = $a + $b</tt>,
<tt>@sum[$a."-".$b] = $x</tt>, etc.

<p/> Their names and their values are entirely under your control; they change
only when you assign to them.

<p/> Just as for field names in stream records, if you want to define out-of-stream variables
with <b>special characters</b> such as <tt>.</tt> then you can use braces, e.g. <tt>'@{variable.name}["index"]'</tt>.

<p/>You may use a <b>computed key </b> in square brackets, e.g.

POKI_RUN_COMMAND{{echo s=green,t=blue,a=3,b=4 | mlr put -q '@[$s."_".$t] = $a * $b; emit all'}}HERE

<p/> Out-of-stream variables are <b>scoped</b> to the <tt>put</tt> command in
which they appear.  In particular, if you have two or more <tt>put</tt>
commands separated by <tt>then</tt>, each put will have its own set of
out-of-stream variables:

POKI_RUN_COMMAND{{cat data/a.dkvp}}HERE
POKI_RUN_COMMAND{{mlr put '@sum += $a; end {emit @sum}' then put 'ispresent($a) {$a=10*$a; @sum += $a}; end {emit @sum}' data/a.dkvp}}HERE

<p/> Out-of-stream variables are read-write: you can do <tt>$sum=@sum</tt>, <tt>@sum=$sum</tt>,
etc.

<!-- ================================================================ -->
<h3>Indexed out-of-stream variables</h3>

<p/>Using an index on the <tt>@count</tt> and <tt>@sum</tt> variables, we get the benefit of the
<tt>-g</tt> (group-by) option which <tt>mlr stats1</tt> and various other Miller commands have:

POKI_INCLUDE_AND_RUN_ESCAPED(data/begin-end-example-6.sh)HERE
POKI_INCLUDE_AND_RUN_ESCAPED(data/begin-end-example-7.sh)HERE

<p/>Indices can be arbitrarily deep &mdash; here there are two or more of them:

POKI_INCLUDE_AND_RUN_ESCAPED(data/begin-end-example-6a.sh)HERE

The idea is that <tt>stats1</tt>, and other Miller commands, encapsulate
frequently-used patterns with a minimum of keystroking (and run a little
faster), whereas using out-of-stream variables you have more flexibility and
control in what you do.

<p/>Begin/end blocks can be mixed with pattern/action blocks. For example:

POKI_INCLUDE_AND_RUN_ESCAPED(data/begin-end-example-8.sh)HERE

<!-- ================================================================ -->
<h3>Aggregate variable assignments</h3>

<p/>There are three remaining kinds of variable assignment using out-of-stream
variables, the last two of which use the <tt>$*</tt> syntax:
<ul>
<li/> Recursive copy of out-of-stream variables
<li/> Out-of-stream variable assigned to full stream record
<li/> Full stream record assigned to an out-of-stream variable
</ul>

<p/> Example recursive copy of out-of-stream variables:

POKI_RUN_COMMAND{{mlr --opprint put -q '@v["sum"] += $x; @v["count"] += 1; end{dump; @w = @v; dump}' data/small}}HERE

<p/>Example of out-of-stream variable assigned to full stream record, where the 2nd record is stashed, and the 4th record is overwritten with that:

POKI_RUN_COMMAND{{mlr put 'NR == 2 {@keep = $*}; NR == 4 {$* = @keep}' data/small}}HERE

<p/>Example of full stream record assigned to an out-of-stream variable, finding
the record for which the <tt>x</tt> field has the largest value in the input
stream:

POKI_RUN_COMMAND{{cat data/small}}HERE
POKI_RUN_COMMAND{{mlr --opprint put -q 'isnull(@xmax) || $x > @xmax {@xmax=$x; @recmax=$*}; end {emit @recmax}' data/small}}HERE

<!-- ================================================================ -->
<h3>Keywords for filter and put</h3>

POKI_RUN_COMMAND{{mlr --help-all-keywords}}HERE

<!-- ================================================================ -->
<h2>Control structures</h2>

<!-- ================================================================ -->
<h3>Pattern-action blocks</h3>

<p/>These are reminiscent of <tt>awk</tt> syntax.  They can be used to allow
assignments to be done only when appropriate &mdash; e.g. for math-function
domain restrictions, regex-matching, and so on:

POKI_RUN_COMMAND{{mlr cat data/put-gating-example-1.dkvp}}HERE
POKI_RUN_COMMAND{{mlr put '$x > 0.0 { $y = log10($x); $z = sqrt($y) }' data/put-gating-example-1.dkvp}}HERE
POKI_RUN_COMMAND{{mlr cat data/put-gating-example-2.dkvp}}HERE
POKI_RUN_COMMAND{{mlr put '$a =~ "([a-z]+)_([0-9]+)" { $b = "left_\1"; $c = "right_\2" }' data/put-gating-example-2.dkvp}}HERE

<p/>This produces heteregenous output which Miller, of course, has no problems
with (see POKI_PUT_LINK_FOR_PAGE(record-heterogeneity.html)HERE).  But if you
want homogeneous output, the curly braces can be replaced with a semicolon
between the expression and the body statements.  This causes <tt>put</tt> to
evaluate the boolean expression (along with any side effects, namely,
regex-captures <tt>\1</tt>, <tt>\2</tt>, etc.) but doesn&rsquo;t use it as a
criterion for whether subsequent assignments should be executed. Instead,
subsequent assignments are done unconditionally:

POKI_RUN_COMMAND{{mlr put '$x > 0.0; $y = log10($x); $z = sqrt($y)' data/put-gating-example-1.dkvp}}HERE
POKI_RUN_COMMAND{{mlr put '$a =~ "([a-z]+)_([0-9]+)"; $b = "left_\1"; $c = "right_\2"' data/put-gating-example-2.dkvp}}HERE

<!-- ================================================================ -->
<h3>If-statements</h3>

<p/>These are again reminiscent of <tt>awk</tt>. Pattern-action blocks are a special case of <tt>if</tt> with no
<tt>elif</tt> or <tt>else</tt> blocks, no <tt>if</tt> keyword, and parentheses optional around the boolean expression:

POKI_CARDIFY{{mlr put 'NR == 4 {$foo = "bar"}'}}HERE
POKI_CARDIFY{{mlr put 'if (NR == 4) {$foo = "bar"}'}}HERE

<p/>Compound statements use <tt>elif</tt> (rather than <tt>elsif</tt> or <tt>else if</tt>):

POKI_INCLUDE_ESCAPED(data/if-chain.sh)HERE

<!-- ================================================================ -->
<h3>While and do-while loops</h3>

<p/>Miller&rsquo;s <tt>while</tt> and <tt>do-while</tt> are unsurprising in
comparison to various languages, as are <tt>break</tt> and <tt>continue</tt>:

POKI_INCLUDE_AND_RUN_ESCAPED(data/while-example-1.sh)HERE
POKI_INCLUDE_AND_RUN_ESCAPED(data/while-example-2.sh)HERE

<p/> A <tt>break</tt> or <tt>continue</tt> within nested conditional blocks or
if-statements will, of course, propagate to the innermost loop enclosing them,
if any. A <tt>break</tt> or <tt>continue</tt> outside a loop is a syntax error
that will be flagged as soon as the expression is parsed, before any input
records are ingested.

<p/> The existence of <tt>while</tt>, <tt>do-while</tt>, and <tt>for</tt> loops
in Miller&rsquo;s DSL means that you can create infinite-loop scenarios
inadvertently.  In particular, please recall that DSL statements are executed
once if in <tt>begin</tt> or <tt>end</tt> blocks, and once <i>per record</i>
otherwise. For example, <b><tt>while (NR < 10)</tt> will never terminate as
<tt>NR</tt> is only incremented between records</b>.

<!-- ================================================================ -->
<h3>For-loops</h3>

<p/>While Miller&rsquo;s <tt>while</tt> and <tt>do-while</tt> statements are
much as in many other languages, <tt>for</tt> loops are more idiosyncratic to
Miller. They are loops over key-value pairs, whether in stream records or
out-of-stream variables: more reminiscent of <tt>foreach</tt>, as in (for
example) PHP.

<p/> There are three variants: <b>for-loop over key-value pairs in the current
stream record</b>, <b>for-loop over key-value pairs in an out-of-stream
variable</b>, and <b>C-style triple-for loops</b>.  In each of the first two
cases the <tt>in</tt> keyword specifies the hashmap being iterated over, and
the variable names between <tt>for</tt> and <tt>in</tt> are bound to the keys
and values, respectively, of the hashmap&rsquo;s key-value pairs on each loop
iteration.  As with <tt>while</tt> and <tt>do-while</tt>, a <tt>break</tt> or
<tt>continue</tt> within nested control structures will propagate to the
innermost loop enclosing them, if any, and a <tt>break</tt> or
<tt>continue</tt> outside a loop is a syntax error that will be flagged as soon
as the expression is parsed, before any input records are ingested.

<p/><b>For-loop over the current stream record</b>:

POKI_RUN_COMMAND{{cat data/for-srec-example.tbl}}HERE
POKI_INCLUDE_AND_RUN_ESCAPED(data/for-srec-example-1.sh)HERE
POKI_RUN_COMMAND{{mlr --from data/small --opprint put 'for (k,v in $*) { $[k."_type"] = typeof(v) }'}}HERE

<p/>Note that the value of the current field in the for-loop can be gotten either using the bound
variable <tt>value</tt>, or through a <b>computed field name</b> using square brackets as in <tt>$[key]</tt>.

<p/>Important note: to avoid inconsistent looping behavior in case you&rsquo;re
setting new fields (and/or unsetting existing ones) while looping over the
record, <b>Miller makes a copy of the record before the loop: loop variables
are bound from the copy and all other reads/writes involve the record
itself</b>:

POKI_INCLUDE_AND_RUN_ESCAPED(data/for-srec-example-2.sh)HERE

It can be confusing to modify the stream record while iterating over a copy of it, so
instead you might find it simpler to use an out-of-stream variable in the loop and only update
the stream record after the loop:

POKI_INCLUDE_AND_RUN_ESCAPED(data/for-srec-example-3.sh)HERE

<p/><b>For-loop over out-of-stream variable</b>: This is similar to looping
over the current stream record except for additional degrees of freedom: you
can start iterating on sub-hashmaps of an out-of-stream variable; you can loop
over nested keys; you can loop over all out-of-stream variables.  As with
for-loops over stream records, the bound variables are bound to a copy of the
sub-hashmap as it was before the loop started.  The sub-hashmap is specified by
square-bracketed indices after <tt>in</tt>, and additional deeper indices are
bound to loop key-variables. The terminal values are bound to the loop
value-variable whenever the keys are neither too shallow, nor too deep. Example
indexing is as follows:

POKI_INCLUDE_ESCAPED(data/for-oosvar-example-0a.txt)HERE

<p/>That&rsquo;s confusing in the abstract, so a concrete example is in order.
Suppose the out-of-stream variable <tt>@myvar</tt> is populated as follows:

POKI_INCLUDE_AND_RUN_ESCAPED(data/for-oosvar-example-0b.sh)HERE

<p/> Then the too-shallow parts &mdash; indexed by the basename <tt>myvar</tt>
and the index <tt>"nesting-is-too-shallow"</tt> &mdash; have depth two
(basename and one index specify a terminal value) and can be gotten as follows:

POKI_INCLUDE_AND_RUN_ESCAPED(data/for-oosvar-example-0c.sh)HERE
POKI_INCLUDE_AND_RUN_ESCAPED(data/for-oosvar-example-0d.sh)HERE

<p/>Note that it would take more than these two indices to reach the deeper values in the hashmap so they
aren&rsquo;t bound in either of these for-loops.

<p/>By contrast, the <tt>"just-right"</tt> parts have depth three (basename and
two indices specify a terminal value) and can be gotten at by any of the
following:

POKI_INCLUDE_AND_RUN_ESCAPED(data/for-oosvar-example-0e.sh)HERE
POKI_INCLUDE_AND_RUN_ESCAPED(data/for-oosvar-example-0f.sh)HERE
POKI_INCLUDE_AND_RUN_ESCAPED(data/for-oosvar-example-0g.sh)HERE

<p/> Note that three key levels are specified here: basename and two indices.
So these for-loops don&rsquo;t produce the depth-two or depth-four entries in
the hashmap.

<p/><b>C-style triple-for loops</b> are supported as follows:

POKI_INCLUDE_AND_RUN_ESCAPED(data/triple-for-example.sh)HERE

Notes:
<ul>

<li/> In <tt>for (start; continuation; update) { body }</tt>, the start,
continuation, and update statements may be empty, single statements, or
multiple comma-separated statements. If the continuation is empty it defaults
to true.

<li/> In particular, you may use <tt>$</tt>-variables and/or
<tt>@</tt>-variables in the start, continuation, and/or update steps (as well
as the body, of course).

<li/> As with all for/if/while statements in Miller, the curly braces are
required even if the body is a single statement, or empty.

</ul>

<!-- ================================================================ -->
<h3>Begin/end blocks</h3>

<p/>Miller supports an <tt>awk</tt>-like <tt>begin/end</tt> syntax.  The
statements in the <tt>begin</tt> block are executed before any input records
are read; the statements in the <tt>end</tt> block are executed after the last
input record is read.  (If you want to execute some statement at the start of
each file, not at the start of the first file as with <tt>begin</tt>, you might
use a pattern/action block of the form <tt>FNR == 1 { ... }</tt>.) All
statements outside of <tt>begin</tt> or <tt>end</tt> are, of course, executed
on every input record. Semicolons separate statements inside or outside of
begin/end blocks; semicolons are required between begin/end block bodies and
any subsequent statement.  For example:

POKI_INCLUDE_AND_RUN_ESCAPED(data/begin-end-example-1.sh)HERE

<p/>Since uninitialized out-of-stream variables default to 0 for
addition/substraction and 1 for multiplication when they appear on expression
right-hand sides (as in <tt>awk</tt>), the above can be written more succinctly
as

POKI_INCLUDE_AND_RUN_ESCAPED(data/begin-end-example-2.sh)HERE

<p/>The <b>put -q</b> option is a shorthand which suppresses printing of each
output record, with only <tt>emit</tt> statements being output. So to get only
summary outputs, one could write

POKI_INCLUDE_AND_RUN_ESCAPED(data/begin-end-example-3.sh)HERE

<p/>We can do similarly with multiple out-of-stream variables:

POKI_INCLUDE_AND_RUN_ESCAPED(data/begin-end-example-4.sh)HERE

This is of course not much different than

POKI_INCLUDE_AND_RUN_ESCAPED(data/begin-end-example-5.sh)HERE

<p/>Note that it&rsquo;s a syntax error for begin/end blocks to refer to field
names (beginning with <tt>$</tt>), since these execute outside the context of
input records.

<!-- ================================================================ -->
<h2>Output statements</h2>

<p/> xxx grist: You can output built-in variables indirectly, by assigning them
to a non-built-in variable: e.g. <tt>$nr = NR</tt> adds a field named
<tt>nr</tt> to each output record, containing the value of <tt>NR</tt> as of
when that record was ingested.

<p/> xxx grist: You can <b>output</b> these in <b>four ways</b>: (1)
<b>assign</b> them to stream-record fields, e.g.  <tt>$cumulative_sum =
@sum</tt>; (2) use <b>emit</b>/<b>emitp</b>/<b>emitf</b>, e.g.  <tt>@sum += $x;
emit @sum</tt> which produces an extra output record such as
<tt>sum=3.1648382</tt>; (3) use the <b>dump</b> or <b>edump</b> keywords, which
immediately print all out-of-stream variables as a JSON data structure to the
standard output or standard error (respectively), or (4) use the <b>print</b>
or <b>eprint</b> keywords which immediately print an expression to standard
output or standard error, respectively. Note that <tt>dump</tt>,
<tt>edump</tt>, <tt>print</tt>, and <tt>eprint</tt> don&rsquo;t output records
which participate in <tt>then</tt>-chaining; rather, they&rsquo;re just
immediate prints to stdout/stderr. The <tt>printn</tt> and <tt>eprintn</tt>
keywords are the same except that they don&rsquo;t print final newlines.

<!-- ================================================================ -->
<h3>Emit statements</h3>

<p/>As noted above, there are three ways to output out-of-stream variables:
(1) Assign them to stream-record fields, e.g.  <tt>$cumulative_sum = @sum</tt>;
(2) Use <tt>emit</tt>, e.g.  <tt>@sum += $x; emit @sum</tt> which produces an
extra output record such as <tt>sum=3.1648382</tt>; (3) Use the <tt>dump</tt>
keyword, which immediately prints all out-of-stream variables to the standard
output as a JSON data structure. Note that the latter aren&rsquo;t output records
which participate in <tt>then</tt>-chaining; rather, they&rsquo;re just an
immediate print to stdout. This section is about <tt>emit</tt>.

<p/>There are three variants: <tt>emitf</tt>, <tt>emit</tt>, and
<tt>emitp</tt>. Keep in mind that out-of-stream variables are a nested,
multi-level hashmap (directly viewable as JSON using <tt>dump</tt>), whereas
Miller output records are lists of single-level key-value pairs. The three emit
variants allow you to control how the multilevel hashmaps are flatten down to
output records.

<p/>Use <b>emitf</b> to output several out-of-stream variables side-by-side in the same output record.
For <tt>emitf</tt> these mustn&rsquo;t have indexing using <tt>@name[...]</tt>. Example:

POKI_RUN_COMMAND{{mlr put -q '@count += 1; @x_sum += $x; @y_sum += $y; end { emitf @count, @x_sum, @y_sum}' data/small}}HERE

<p/>Use <b>emit</b> to output an out-of-stream variable. If it&rsquo;s non-indexed you&rsquo;ll get a simple key-value pair:

POKI_RUN_COMMAND{{cat data/small}}HERE
POKI_RUN_COMMAND{{mlr put -q '@sum += $x; end { dump }' data/small}}HERE
POKI_RUN_COMMAND{{mlr put -q '@sum += $x; end { emit @sum }' data/small}}HERE

<p/>If it&rsquo;s indexed then use as many names after <tt>emit</tt> as there are indices:

POKI_RUN_COMMAND{{mlr put -q '@sum[$a] += $x; end { dump }' data/small}}HERE
POKI_RUN_COMMAND{{mlr put -q '@sum[$a] += $x; end { emit @sum, "a" }' data/small}}HERE

POKI_RUN_COMMAND{{mlr put -q '@sum[$a][$b] += $x; end { dump }' data/small}}HERE
POKI_RUN_COMMAND{{mlr put -q '@sum[$a][$b] += $x; end { emit @sum, "a", "b" }' data/small}}HERE

POKI_RUN_COMMAND{{mlr put -q '@sum[$a][$b][$i] += $x; end { dump }' data/small}}HERE
POKI_RUN_COMMAND{{mlr put -q '@sum[$a][$b][$i] += $x; end { emit @sum, "a", "b", "i" }' data/small}}HERE

<p/>Now for <b>emitp</b>: if you have as many names following <tt>emit</tt> as
there are levels in the out-of-stream variable&rsquo;s hashmap, then <tt>emit</tt> and <tt>emitp</tt> do the same
thing. Where they differ is when you don&rsquo;t specify as many names as there are hashmap levels. In this
case, Miller needs to flatten multiple map indices down to output-record keys: <tt>emitp</tt> includes full
prefixing (hence the <tt>p</tt> in <tt>emitp</tt>) while <tt>emit</tt> takes the deepest hashmap key as the
output-record key:

POKI_RUN_COMMAND{{mlr put -q '@sum[$a][$b] += $x; end { dump }' data/small}}HERE
POKI_RUN_COMMAND{{mlr put -q '@sum[$a][$b] += $x; end { emit @sum, "a" }' data/small}}HERE
POKI_RUN_COMMAND{{mlr put -q '@sum[$a][$b] += $x; end { emit @sum }' data/small}}HERE
POKI_RUN_COMMAND{{mlr put -q '@sum[$a][$b] += $x; end { emitp @sum, "a" }' data/small}}HERE
POKI_RUN_COMMAND{{mlr put -q '@sum[$a][$b] += $x; end { emitp @sum }' data/small}}HERE
POKI_RUN_COMMAND{{mlr --oxtab put -q '@sum[$a][$b] += $x; end { emitp @sum }' data/small}}HERE

<p/>Use <b>--oflatsep</b> to specify the character which joins multilevel
keys for <tt>emitp</tt> (it defaults to a colon):

POKI_RUN_COMMAND{{mlr put -q --oflatsep / '@sum[$a][$b] += $x; end { emitp @sum, "a" }' data/small}}HERE
POKI_RUN_COMMAND{{mlr put -q --oflatsep / '@sum[$a][$b] += $x; end { emitp @sum }' data/small}}HERE
POKI_RUN_COMMAND{{mlr --oxtab put -q --oflatsep / '@sum[$a][$b] += $x; end { emitp @sum }' data/small}}HERE

<!-- ================================================================ -->
<h3>Multi-emit statements</h3>

<p/>You can emit <b>multiple out-of-stream variables side-by-side</b> by including their names in parentheses:

POKI_INCLUDE_AND_RUN_ESCAPED(data/emit-lashed.sh)HERE

What this does is walk through the first out-of-stream variable
(<tt>@x_sum</tt> in this example) as usual, then for each keylist found (e.g.
<tt>pan,wye</tt>), include the values for the remaining out-of-stream variables
(here, <tt>@x_count</tt> and <tt>@x_mean</tt>). You should use this when all
out-of-stream variables in the emit statement have the same shape and the same
keylists.

<!-- ================================================================ -->
<h3>Emit-all statements</h3>

<p/>Use <b>emit all</b> (or <tt>emit @*</tt> which is synonumous) to output all
out-of-stream variables. You can use the following idiom to get various
accumulators output side-by-side (reminiscent of <tt>mlr stats1</tt>):

POKI_RUN_COMMAND{{mlr --from data/small --opprint put -q '@v[$a][$b]["sum"] += $x; @v[$a][$b]["count"] += 1; end{emit @*,"a","b"}'}}HERE
POKI_RUN_COMMAND{{mlr --from data/small --opprint put -q '@sum[$a][$b] += $x; @count[$a][$b] += 1; end{emit @*,"a","b"}'}}HERE
POKI_RUN_COMMAND{{mlr --from data/small --opprint put -q '@sum[$a][$b] += $x; @count[$a][$b] += 1; end{emit (@sum, @count),"a","b"}'}}HERE

<!-- ================================================================ -->
<h3>Redirected-output statements</h3>

The <b>tee</b>, <b>emitf</b>, <b>emitp</b>, <b>emit</b>, <b>print</b>, and
<b>dump</b> keywords all allow you to redirect output to one or more files or
pipe-to commands. The filenames/commands are strings which can be constructed
using record-dependent values, so you can do things like splitting a table into
multiple files, one for each account ID, and so on.

<p/> Details:

<ul>

<li/> <tt>mlr put</tt> sends the current record (possibly modified by the
<tt>put</tt> expression) to the output record stream. Records are then input to
the following verb in a <tt>then</tt>-chain (if any), else printed to standard
output (unless <tt>put -q</tt>). The <b>tee</b> keyword <i>additionally</i>
writes the output record to specified file(s) or pipe-to command, or
immediately to <tt>stdout</tt>/<tt>stderr</tt>.

POKI_RUN_COMMAND{{mlr --help-keyword tee}}HERE

<li/> <tt>mlr put</tt>&rsquo;s <tt>emitf</tt>, <tt>emitp</tt>, and
<tt>emit</tt> send out-of-stream variables to the output record stream. These
are then input to the following verb in a <tt>then</tt>-chain (if any), else
printed to standard output. When redirected with <tt>&gt;</tt>,
<tt>&gt;&gt;</tt>, or <tt>|</tt>, they <i>instead</i> write the out-of-stream
variable(s) to specified file(s) or pipe-to command, or immediately to
<tt>stdout</tt>/<tt>stderr</tt>.

POKI_RUN_COMMAND{{mlr --help-keyword emitf}}HERE
POKI_RUN_COMMAND{{mlr --help-keyword emitp}}HERE
POKI_RUN_COMMAND{{mlr --help-keyword emit}}HERE

<li/> The <tt>print</tt> and <tt>dump</tt> keywords produce output immediately
to standard output, or to specified file(s) or pipe-to command if present.

POKI_RUN_COMMAND{{mlr --help-keyword print}}HERE
POKI_RUN_COMMAND{{mlr --help-keyword dump}}HERE

</ul>

<!-- ================================================================ -->
<h2>Unset statements</h2>

<p/>You can clear a map key by assigning the empty string as its value: <tt>$x=""</tt> or <tt>@x=""</tt>.
Using <tt>unset</tt> you can remove the key entirely. Examples:

POKI_RUN_COMMAND{{cat data/small}}HERE
POKI_RUN_COMMAND{{mlr put 'unset $x, $a' data/small}}HERE

<p/>This can also be done, of course, using <tt>mlr cut -x</tt>. You can also clear out-of-stream variables, at the base name level, or at an indexed sublevel:

POKI_RUN_COMMAND{{mlr put -q '@sum[$a][$b] += $x; end { dump; unset @sum; dump }' data/small}}HERE
POKI_RUN_COMMAND{{mlr put -q '@sum[$a][$b] += $x; end { dump; unset @sum["eks"]; dump }' data/small}}HERE

<p/>If you use <tt>unset all</tt> (or <tt>unset @*</tt> which is synonymous), that will unset all out-of-stream
variables which have been defined up to that point.

<!-- ================================================================ -->
<h2>Filter statements</h2>

<p/> You can use <tt>filter</tt> within <tt>put</tt>. In fact, the
following two are synonymous:

POKI_RUN_COMMAND{{mlr filter 'NR==2 || NR==3' data/small}}HERE
POKI_RUN_COMMAND{{mlr put 'filter NR==2 || NR==3' data/small}}HERE

<p/>The former, of course, is much easier to type. But the latter allows you to define more complex expressions
for the filter, and/or do other things in addition to the filter:

POKI_RUN_COMMAND{{mlr put '@running_sum += $x; filter @running_sum > 1.3' data/small}}HERE
POKI_RUN_COMMAND{{mlr put '$z = $x * $y; filter $z > 0.3' data/small}}HERE

<!-- ================================================================ -->
<h2>Built-in functions for filter and put</h2>

<p/>Each function takes a specific number of arguments, as shown below, except
for functions marked as variadic such as <tt>min</tt> and <tt>max</tt>. (The
latter compute min and max of any number of numerical arguments.) There is no
notion of optional or default-on-absent arguments. All argument-passing is
positional rather than by name; arguments are passed by value, not by
reference.

POKI_RUN_COMMAND{{mlr --help-all-functions}}HERE

<!-- ================================================================ -->
<h2>User-defined functions and subroutines</h2>

<p/> As of Miller 4.6.0 you can define your own functions, as well as subroutines.

<!-- ================================================================ -->
<h3>User-defined functions</h3>

<p/>Here&rsquo;s the obligatory example of a recursive function to compute the factorial function:

POKI_INCLUDE_AND_RUN_ESCAPED(data/factorial-example.sh)HERE

<p/>Properties of user-defined functions:

<ul>

<li/> xxx example definition. outside begin/end. before/after body defs (resolved after parse).
<li/> filter and put both.

<li/> xxx xref to -f/-e for each of use.
<li/> xxx no redefines of built-ins or of one another.
<li/> xxx no nested definitions.
<li/> xxx access to arguments (positional/value); $/@ state as RHS.
<li/> xxx all RHS stuff &mdash; no assigns to $/@; only to localvars. no subr calls.
<li/> xxx recursive/mutually so ok. 
<li/> xxx localvars as in fors. args may be reassigned.
<li/> xxx xref to shared section for func/subr localvars (scope/extent/etc.)
<li/> xxx missing return leads to absent-null (which xref). return value (single).

<li/> <a href="#Local_variables">Local variables</a>

</ul>

<p/>xxx uncached/cached fibo examples.

<!-- ================================================================ -->
<h3>User-defined subroutines</h3>

<p/> xxx this section is under construction.

<ul>
<li/> xxx example definition. outside begin/end. before/after body defs (resolved after parse).
<li/> put only, not filter
<li/> xxx xref to -f/-e for each of use.
<li/> xxx no redefines of built-ins or of one another.
<li/> xxx no nested definitions.
<li/> xxx access to arguments (positional/value); $/@ state as RHS.
<li/> xxx all RHS stuff &mdash; and assigns to $/@.
<li/> xxx recursive/mutually so ok. 
<li/> xxx localvars as in fors. args may be reassigned.
<li/> xxx xref to shared section for func/subr localvars (scope/extent/etc.)
<li/> xxx rename
<li/> <a href="#Local_variables">Local variables</a>
</ul>

<!-- ================================================================ -->
<h2>A note on the complexity of Miller&rsquo;s expression language</h2>

<p/> One of Miller&rsquo;s strengths is its brevity: it&rsquo;s much quicker
&mdash; and less error-prone &mdash; to type <tt>mlr stats1 -a sum -f x,y -g
a,b</tt> than having to track summation variables as in <tt>awk</tt>, or using
Miller&rsquo;s out-of-stream variables. And the more language features
Miller&rsquo;s put-DSL has (for-loops, if-statements, nested control
structures, etc.) then the <i>less</i> powerful it begins to seem: because of
the other programming-language features it <i>doesn&rsquo;t</i> have.

<p/> When I was originally prototyping Miller in 2015, the decision I had was
whether to hand-code in a low-level language like C or Rust, with my own
hand-rolled DSL, or whether to use a higher-level language (like Python or Lua
or Nim) and let the <tt>put</tt> statements be handled by the implementation
language&rsquo;s own <tt>eval</tt>: the implementation language would take the
place of a DSL. Multiple performance experiments showed me I could get better
throughput using the former, and using C in particular &mdash; by a wide margin. So
Miller is C under the hood with a hand-rolled DSL.

<p/> I do want to keep focusing on what Miller is good at &mdash; concise notation,
low latency, and high throughput &mdash; and not add too much in terms of
high-level-language features to the DSL.  That said, some sort of looping over
field names is a basic thing to want. As of 4.1.0 we have recursive
for/while/if structures on about the same complexity level as <tt>awk</tt>.
While I&rsquo;m excited by these powerful language features, I hope to keep new
features beyond 4.1.0 focused on Miller&rsquo;s sweet spot which is speed plus
simplicity.

<!-- ================================================================ -->
<h1>then-chaining</h1>

<p/>
In accord with the
<a href="http://en.wikipedia.org/wiki/Unix_philosophy">Unix philosophy</a>, you can pipe data into or out of
Miller. For example:

POKI_CARDIFY(mlr cut --complement -f os_version *.dat | mlr sort -f hostname,uptime)HERE

<p/>
You can, if you like, instead simply chain commands together using the
<tt>then</tt> keyword:

POKI_CARDIFY(mlr cut --complement -f os_version then sort -f hostname,uptime *.dat)HERE

Here&rsquo;s a performance comparison:

POKI_INCLUDE_ESCAPED(data/then-chaining-performance.txt)HERE

There are two reasons to use then-chaining: one is for performance, although I
don&rsquo;t expect this to be a win in all cases.  Using then-chaining avoids
redundant string-parsing and string-formatting at each pipeline step: instead
input records are parsed once, they are fed through each pipeline stage in
memory, and then output records are formatted once. On the other hand, Miller
is single-threaded, while modern systems are usually multi-processor, and when
streaming-data programs operate through pipes, each one can use a CPU.  Rest
assured you get the same results either way.

<p/>The other reason to use then-chaining is for simplicity: you don&rsquo;t
have re-type formatting flags (e.g. <tt>--csv --rs lf --fs tab</tt>) at every
pipeline stage.

<!-- ================================================================ -->
<h1>Data types</h1>

<p/> Miller&rsquo;s input and output are all string-oriented: there is (as of
August 2015 anyway) no support for binary record packing. In this sense,
everything is a string in and out of Miller.  During processing, field names
are always strings, even if they have names like "3"; field values are usually
strings.  Field values&rsquo; ability to be interpreted as a non-string type
only has meaning when comparison or function operations are done on them.  And
it is an error condition if Miller encounters non-numeric (or otherwise
mistyped) data in a field in which it has been asked to do numeric (or
otherwise type-specific) operations.

<p/> Field values are treated as numeric for the following:
<ul>
<li/> Numeric sort: <tt>mlr sort -n</tt>, <tt>mlr sort -nr</tt>.
<li/> Statistics: <tt>mlr histogram</tt>, <tt>mlr stats1</tt>, <tt>mlr stats2</tt>.
<li/> Cross-record arithmetic: <tt>mlr step</tt>.
</ul>

<p/>For <tt>mlr put</tt> and <tt>mlr filter</tt>:

<ul>

<li/> Miller&rsquo;s types for function processing are <b>null</b> (empty
string), <b>error</b>, <b>string</b>, <b>float</b> (double-precision),
<b>int</b> (64-bit signed), and <b>boolean</b>.

<li/> On input, string values representable as numbers, e.g. "3" or "3.1", are
treated as int or float, respectively. If a record has <tt>x=1,y=2</tt> then
<tt>mlr put '$z=$x+$y'</tt> will produce <tt>x=1,y=2,z=3</tt>, and <tt>mlr put
'$z=$x.$y'</tt> gives an error.  To coerce back to string for processing, use
the <tt>string</tt> function: <tt>mlr put '$z=string($x).string($y)'</tt> will
produce <tt>x=1,y=2,z=12</tt>.

<li/> On input, string values representable as boolean  (e.g. <tt>"true"</tt>,
<tt>"false"</tt>) are <i>not</i> automatically treated as boolean.  (This is
because <tt>"true"</tt> and <tt>"false"</tt> are ordinary words, and auto
string-to-boolean on a column consisting of words would result in some strings
mixed with some booleans.) Use the <tt>boolean</tt> function to coerce: e.g.
giving the record <tt>x=1,y=2,w=false</tt> to <tt>mlr put '$z=($x&lt;$y) ||
boolean($w)'</tt>.

<li/> Functions take types as described in <tt>mlr --help-all-functions</tt>:
for example, <tt>log10</tt> takes float input and produces float output,
<tt>gmt2sec</tt> maps string to int, and <tt>sec2gmt</tt> maps int to string.

<li/> All math functions described in <tt>mlr --help-all-functions</tt> take
integer as well as float input.

</ul>

<!-- ================================================================ -->
<h1>Null data: empty and absent</h1>

<p/> One of Miller&rsquo;s key features is its support for <b>heterogeneous</b>
data.  For example, take <tt>mlr sort</tt>: if you try to sort on field
<tt>hostname</tt> when not all records in the data stream <i>have</i> a field
named <tt>hostname</tt>, it is not an error (although you could pre-filter the
data stream using <tt>mlr having-fields --at-least hostname then sort
...</tt>).  Rather, records lacking one or more sort keys are simply output
contiguously by <tt>mlr sort</tt>.

<p/> Miller has two kinds of null data:

<ul>

<li/> <b>Empty</b>: a field name is present in a record (or in an out-of-stream
variable) with empty value: e.g. <tt>x=,y=2</tt> in the data input stream, or
assignment <tt>$x=""</tt> or <tt>@x=""</tt> in <tt>mlr put</tt>.

<li/> <b>Absent</b>: a field name is not present, e.g. input record is
<tt>x=1,y=2</tt> and a <tt>put</tt> or <tt>filter</tt> expression refers to
<tt>$z</tt>. Or, reading an out-of-stream variable which hasn&rsquo;t been
assigned a value yet,
e.g.  <tt>mlr put -q '@sum += $x'; end{emit @sum}'</tt> or <tt>mlr put -q
'@sum[$a][$b] += $x'; end{emit @sum, "a", "b"}'</tt>.

</ul>

<p/>You can test these programatically using the functions
<tt>isempty</tt>/<tt>isnotempty</tt>, <tt>isabsent</tt>/<tt>ispresent</tt>, and
<tt>isnull</tt>/<tt>isnotnull</tt>. For the last pair, note that null means
either empty or absent.

<p/>
Rules for null-handling:

<ul>

<li> Records with one or more empty sort-field values sort after records with
all sort-field values present:
POKI_RUN_COMMAND{{mlr cat data/sort-null.dat}}HERE
POKI_RUN_COMMAND{{mlr sort -n  a data/sort-null.dat}}HERE
POKI_RUN_COMMAND{{mlr sort -nr a data/sort-null.dat}}HERE

<li> Functions/operators which have one or more <i>empty</i> arguments produce empty output: e.g.
POKI_RUN_COMMAND{{echo 'x=2,y=3' | mlr put '$a=$x+$y'}}HERE
POKI_RUN_COMMAND{{echo 'x=,y=3' | mlr put '$a=$x+$y'}}HERE
POKI_RUN_COMMAND{{echo 'x=,y=3' | mlr put '$a=log($x);$b=log($y)'}}HERE

with the exception that the <tt>min</tt> and <tt>max</tt> functions are
special: if one argument is non-null, it wins:
POKI_RUN_COMMAND{{echo 'x=,y=3' | mlr put '$a=min($x,$y);$b=max($x,$y)'}}HERE

<li> Functions of <i>absent</i> variables (e.g. <tt>mlr put '$y =
log10($nonesuch)'</tt>) evaluate to absent, and arithmetic/bitwise/boolean
operators with both operands being absent evaluate to absent.
Arithmetic operators with one absent operand return the other operand.
More specifically, absent values act like zero for addition/subtraction, and
one for multiplication: Furthermore, <b>any expression which evaluates to
absent is not stored in the output record</b>:

POKI_RUN_COMMAND{{echo 'x=2,y=3' | mlr put '$a=$u+$v; $b=$u+$y; $c=$x+$y'}}HERE
POKI_RUN_COMMAND{{echo 'x=2,y=3' | mlr put '$a=min($x,$v);$b=max($u,$y);$c=min($u,$v)'}}HERE

</ul>

The reasoning is as follows:

<ul>

<li/> Empty values are explicit in the data so they should explicitly affect accumulations:
<tt>mlr put '@sum += $x'</tt>
should accumulate numeric <tt>x</tt> values into the sum but an empty
<tt>x</tt>, when encountered in the input data stream, should make the sum
non-numeric. To work around this you can use the
<tt>isnotnull</tt> function as follows:
<tt>mlr put 'isnotnull($x) { @sum += $x }'</tt>

<li/> Absent stream-record values should not break accumulations, since Miller
by design handles heterogenous data: the running <tt>@sum</tt> in
<tt>mlr put '@sum += $x'</tt>
should not be invalidated for records which have no <tt>x</tt>.

<li/> Absent out-of-stream-variable values are precisely what allow you to write
<tt>mlr put '@sum += $x'</tt>. Otherwise you would have to write
<tt>mlr put 'begin{@sum = 0}; @sum += $x'</tt> &mdash;
which is tolerable &mdash; but for
<tt>mlr put 'begin{...}; @sum[$a][$b] += $x'</tt>
you&rsquo;d have to pre-initialize <tt>@sum</tt> for all values of <tt>$a</tt> and <tt>$b</tt> in your
input data stream, which is intolerable.

<li/> The penalty for the absent feature is that misspelled variables can be hard to find:
e.g. in <tt>mlr put 'begin{@sumx = 10}; ...; update @sumx somehow per-record; ...; end {@something = @sum * 2}'</tt>
the accumulator is spelt <tt>@sumx</tt> in the begin-block but <tt>@sum</tt> in the end-block, where since it
is absent, <tt>@sum*2</tt> evaluates to 2.

</ul>

<p/>Since absent plus absent is absent (and likewise for other operators),
accumulations such as <tt>@sum += $x</tt> work correctly on heterogenous data,
as do within-record formulas if both operands are absent. If one operand is
present, you may get behavior you don&rsquo;t desire.  To work around this
&mdash; namely, to set an output field only for records which have all the
inputs present &mdash; you can use a pattern-action block with
<tt>ispresent</tt>:

POKI_RUN_COMMAND{{mlr cat data/het.dkvp}}HERE
POKI_RUN_COMMAND{{mlr put 'ispresent($loadsec) { $loadmillis = $loadsec * 1000 }' data/het.dkvp}}HERE
POKI_RUN_COMMAND{{mlr put '$loadmillis = (ispresent($loadsec) ? $loadsec : 0.0) * 1000' data/het.dkvp}}HERE

<p/> If you&rsquo;re interested in a formal description of how empty and absent
fields participate in arithmetic, here&rsquo;s a table for plus (other
arithmetic/boolean/bitwise operators are similar):

POKI_RUN_COMMAND{{mlr --print-type-arithmetic-info}}HERE

<!-- ================================================================ -->
<h1>String literals</h1>

<p/>
You can use the following backslash escapes for strings such as between the double quotes in contexts such as
<tt>mlr filter '$name =~ "..."'</tt>,
<tt>mlr put '$name = $othername . "..."'</tt>,
<tt>mlr put '$name = sub($name, "...", "...")</tt>, etc.:

<ul>
<li/> <tt>\a</tt>: ASCII code 0x07 (alarm/bell)
<li/> <tt>\b</tt>: ASCII code 0x08 (backspace)
<li/> <tt>\f</tt>: ASCII code 0x0c (formfeed)
<li/> <tt>\n</tt>: ASCII code 0x0a (LF/linefeed/newline)
<li/> <tt>\r</tt>: ASCII code 0x0d (CR/carriage return)
<li/> <tt>\t</tt>: ASCII code 0x09 (tab)
<li/> <tt>\v</tt>: ASCII code 0x0b (vertical tab)
<li/> <tt>\\</tt>: backslash
<li/> <tt>\"</tt>: double quote
<li/> <tt>\123</tt>: Octal 123, etc. for <tt>\000</tt> up to <tt>\377</tt>
<li/> <tt>\x7f</tt>: Hexadecimal 7f, etc. for <tt>\x00</tt> up to <tt>\xff</tt>
</ul>

<p/>See also <a href="https://en.wikipedia.org/wiki/Escape_sequences_in_C">https://en.wikipedia.org/wiki/Escape_sequences_in_C</a>.

<p/>These replacements apply only to strings you key in for the DSL expressions for <tt>filter</tt> and <tt>put</tt>:
that is, if you type <tt>\t</tt> in a string literal for a <tt>filter</tt>/<tt>put</tt> expression, it will be turned into a tab character. If you want a backslash followed by a <tt>t</tt>, then please type <tt>\\t</tt>.

<p/>However, these replacements are not done automatically within your data stream. If you wish to make these
replacements, you can do, for example, for a field named <tt>field</tt>, <tt> mlr put '$field = gsub($field, "\\t",
"\t")'</tt>. If you need to make such a replacement for all fields in your data, you should probably simply use the
system <tt>sed</tt> command.

<h1>Regular expressions</h1>

<p/>Miller lets you use regular expressions (of type POSIX.2) in the following contexts:

<ul>

<li/> In <tt>mlr filter</tt> with <tt>=~</tt> or <tt>!=~</tt>, e.g. <tt>mlr
filter '$url =~ "http.*com"'</tt>

<li/> In <tt>mlr put</tt> with <tt>sub</tt> or <tt>gsub</tt>, e.g. <tt>mlr put
'$url = sub($url, "http.*com", "")'</tt>

<li/> In <tt>mlr having-fields</tt>, e.g. <tt>mlr having-fields
--any-matching '^sda[0-9]'</tt>

<li/> In <tt>mlr cut</tt>, e.g. <tt>mlr cut -r -f '^status$,^sda[0-9]'</tt>

<li/> In <tt>mlr rename</tt>, e.g. <tt>mlr rename -r '^(sda[0-9]).*$,dev/\1'</tt>

<li/> In <tt>mlr grep</tt>, e.g. <tt>mlr --csv grep 00188555487 myfiles*.csv</tt>

</ul>

<p/>Points demonstrated by the above examples:

<ul>

<li/> There are no implicit start-of-string or end-of-string anchors; please
use <tt>^</tt> and/or <tt>$</tt> explicitly.

<li/> Miller regexes are wrapped with double quotes rather than slashes.

<li/> The <tt>i</tt> after the ending double quote indicates a case-insensitive
regex.

<li/> Capture groups are wrapped with <tt>(...)</tt> rather than
<tt>\(...\)</tt>; use <tt>\(</tt> and <tt>\)</tt> to match against parentheses.

</ul>

<p/>For <tt>filter</tt> and <tt>put</tt>, if the regular expression is a string
literal (the normal case), it is precompiled at process start and reused
thereafter, which is efficient. If the regular expression is a more complex
expression, including string concatenation using <tt>.</tt>, or a column name
(in which case you can take regular expressions from input data!), then regexes
are compiled on each record which works but is less efficient. As well, in this
case there is no way to specify case-insensitive matching.

<p/>Example:

POKI_RUN_COMMAND{{cat data/regex-in-data.dat}}HERE
POKI_RUN_COMMAND{{mlr filter '$name =~ $regex' data/regex-in-data.dat}}HERE

<h2>Regex captures</h2>

<p/>Regex captures of the form <tt>\0</tt> through <tt>\9</tt> are supported as
follows: <ul>

<li/> Captures have in-function context for <tt>sub</tt> and <tt>gsub</tt>.
For example, the first <tt>\1,\2</tt> pair belong to the first <tt>sub</tt> and
the second <tt>\1,\2</tt> pair belong to the second <tt>sub</tt>:

<p/>
<div class=pokipanel>
<pre>
mlr put '$b = sub($a, "(..)_(...)", "\2-\1"); $c = sub($a, "(..)_(.)(..)", ":\1:\2:\3")'
</pre>
</div>

<li/> Captures endure for the entirety of a <tt>put</tt> for the <tt>=~</tt>
and <tt>!=~</tt> operators. For example, here the <tt>\1,\2</tt> are set by the
<tt>=~</tt> operator and are used by both subsequent assignment statements:

<p/>
<div class=pokipanel>
<pre>
mlr put '$a =~ "(..)_(....); $b = "left_\1"; $c = "right_\2"'
</pre>
</div>

<li/>The captures are not retained across multiple puts. For example, here the
<tt>\1,\2</tt> won&rsquo;t be expanded from the regex capture:

<p/>
<div class=pokipanel>
<pre>
mlr put '$a =~ "(..)_(....)' then {... something else ...} then put '$b = "left_\1"; $c = "right_\2"'
</pre>
</div>

<li/> Captures are ignored in <tt>filter</tt> for the <tt>=~</tt> and
<tt>!=~</tt> operators. For example, there is no mechanism provided to refer to
the first <tt>(..)</tt> as <tt>\1</tt> or to the second <tt>(....)</tt> as
<tt>\2</tt> in the following filter statement:

<p/>
<div class=pokipanel>
<pre>
mlr filter '$a =~ "(..)_(....)'
</pre>
</div>

<li/> Up to nine matches are supported: <tt>\1</tt> through <tt>\9</tt>, while
<tt>\0</tt> is the entire match string; <tt>\15</tt> is treated as <tt>\1</tt>
followed by an unrelated <tt>5</tt>.
</ul>

<!-- ================================================================ -->
<h1>Operator precedence</h1>

<p/>Operators are listed in order of decreasing precedence, highest first.

<p/>
<div class="pokipanel">
<pre>
Operators              Associativity
---------              -------------
()                     left to right
**                     right to left
! ~ unary+ unary- &amp;    right to left
binary* / // %         left to right
binary+ binary- .      left to right
&lt;&lt; &gt;&gt;                  left to right
&amp;                      left to right
^                      left to right
|                      left to right
&lt; &lt;= &gt; &gt;=              left to right
== != =~ !=~           left to right
&amp;&amp;                     left to right
^^                     left to right
||                     left to right
? :                    right to left
=                      N/A for Miller (there is no $a=$b=$c)
</pre>
</div>
<p/>

<!-- ================================================================ -->
<h1>Operator and function semantics</h1>

<ul>

<li/> Functions are in general pass-throughs straight to the system-standard C
library.

<li/> The <tt>min</tt> and <tt>max</tt> functions are different from other
multi-argument functions which return null if any of their inputs are null: for
<tt>min</tt> and <tt>max</tt>, by contrast, if one argument is null, the other
is returned.

<li/> Symmetrically with respect to the bitwise OR, XOR, and AND operators
<tt>|</tt>, <tt>^</tt>, <tt>&amp;</tt>, Miller has logical operators
<tt>||</tt>, <tt>^^</tt>, <tt>&amp;&amp;</tt>: the logical XOR not existing in
C.

<li/> The exponentiation operator <tt>**</tt> is familiar from many languages.

<li/> The regex-match and regex-not-match operators <tt>=~</tt> and
<tt>!=~</tt> are similar to those in Ruby and Perl.

</ul>

<!-- ================================================================ -->
<h1>Arithmetic</h1>

<h2>Input scanning</h2>

<p/>Numbers in Miller are double-precision float or 64-bit signed integers.
Anything scannable as int, e.g <tt>123</tt> or <tt>0xabcd</tt>, is treated as
an integer; otherwise, input scannable as float (<tt>4.56</tt> or <tt>8e9</tt>)
is treated as float; everything else is a string.

<p/>If you want all numbers to be treated as floats, then you may use
<tt>float()</tt> in your filter/put expressions (e.g. replacing <tt>$c = $a *
$b</tt> with <tt>$c = float($a) * float($b)</tt>) &mdash; or, more simply, use
<tt>mlr filter -F</tt> and <tt>mlr put -F</tt> which forces all numeric input,
whether from expression literals or field values, to float. Likewise <tt>mlr
stats1 -F</tt> and <tt>mlr step -F</tt> force integerable accumulators (such as
<tt>count</tt>) to be done in floating-point.

<h2>Conversion by math routines</h2>

<p/>For most math functions, integers are cast to float on input, and produce
float output: e.g. <tt>exp(0) = 1.0</tt> rather than <tt>1</tt>.  The
following, however, produce integer output if their inputs are integers:
<tt>+</tt> <tt>-</tt> <tt>*</tt> <tt>/</tt> <tt>//</tt> <tt>%</tt> <tt>abs</tt>
<tt>ceil</tt> <tt>floor</tt> <tt>max</tt> <tt>min</tt> <tt>round</tt>
<tt>roundm</tt> <tt>sgn</tt>. As well, <tt>stats1 -a min</tt>, <tt>stats1 -a
max</tt>, <tt>stats1 -a sum</tt>, <tt>step -a delta</tt>, and <tt>step -a
rsum</tt> produce integer output if their inputs are integers.

<h2>Conversion by arithmetic operators</h2>

<p/>The sum, difference, and product of integers is again integer, except for
when that would overflow a 64-bit integer at which point Miller converts the
result to float.

<p/>The short of it is that Miller does this transparently for you so you
needn&rsquo;t think about it.

<p/>Implementation details of this, for the interested: integer adds and
subtracts overflow by at most one bit so it suffices to check sign-changes.
Thus, Miller allows you to add and subtract arbitrary 64-bit signed integers,
converting only to float precisely when the result is less than -2<sup>63</sup>
or greater than 2<sup>63</sup>-1.  Multiplies, on the other hand, can overflow
by a word size and a sign-change technique does not suffice to detect overflow.
Instead Miller tests whether the floating-point product exceeds the
representable integer range. Now, 64-bit integers have 64-bit precision while
IEEE-doubles have only 52-bit mantissas &mdash; so, there are 53 bits including
implicit leading one.  The following experiment explicitly demonstrates the
resolution at this range:

<div class=pokipanel>
<pre>
   64-bit integer     64-bit integer     Casted to double           Back to 64-bit
       in hex           in decimal                                    integer
0x7ffffffffffff9ff 9223372036854774271 9223372036854773760.000000 0x7ffffffffffff800
0x7ffffffffffffa00 9223372036854774272 9223372036854773760.000000 0x7ffffffffffff800
0x7ffffffffffffbff 9223372036854774783 9223372036854774784.000000 0x7ffffffffffffc00
0x7ffffffffffffc00 9223372036854774784 9223372036854774784.000000 0x7ffffffffffffc00
0x7ffffffffffffdff 9223372036854775295 9223372036854774784.000000 0x7ffffffffffffc00
0x7ffffffffffffe00 9223372036854775296 9223372036854775808.000000 0x8000000000000000
0x7ffffffffffffffe 9223372036854775806 9223372036854775808.000000 0x8000000000000000
0x7fffffffffffffff 9223372036854775807 9223372036854775808.000000 0x8000000000000000
</pre>
</div>

<p/>That is, one cannot check an integer product to see if it is precisely
greater than 2<sup>63</sup>-1 or less than -2<sup>63</sup> using either integer
arithmetic (it may have already overflowed) or using double-precision (due to
granularity).  Instead Miller checks for overflow in 64-bit integer
multiplication by seeing whether the absolute value of the double-precision
product exceeds the largest representable IEEE double less than 2<sup>63</sup>,
which we see from the listing above is 9223372036854774784. (An alternative
would be to do all integer multiplies using handcrafted multi-word 128-bit
arithmetic.  This approach is not taken.)

<h2>Pythonic division</h2>

<p/>Division and remainder are
<a href="http://python-history.blogspot.com/2010/08/why-pythons-integer-division-floors.html">
pythonic</a>:
  <ul>
  <li/> Quotient of integers is floating-point: <tt>7/2</tt> is <tt>3.5</tt>.
  <li/> Integer division is done with <tt>//</tt>: <tt>7/2</tt> is <tt>3</tt>.
    This rounds toward the negative.
  <li/> Remainders are non-negative.
  </ul>