miller/doc/content-for-reference.html

POKI_PUT_TOC_HERE

<h1>Command overview</h1>

<p>
Whereas the Unix toolkit is made of the separate executables <tt>cat</tt>, <tt>tail</tt>, <tt>cut</tt>,
<tt>sort</tt>, etc., Miller has subcommands, invoked as follows:

POKI_INCLUDE_ESCAPED(data/subcommand-example.txt)HERE

<p/>These falls into categories as follows:

<table border=1>
  <tr class="mlrbg">
    <th>Commands </th>
    <th>Description</th>
  </tr>
  <tr>
    <td>
      <a href="#cat"><tt>cat</tt></a>,
      <a href="#cut"><tt>cut</tt></a>,
      <a href="#head"><tt>head</tt></a>,
      <a href="#sort"><tt>sort</tt></a>,
      <a href="#tac"><tt>tac</tt></a>,
      <a href="#tail"><tt>tail</tt></a>,
      <a href="#top"><tt>top</tt></a>,
      <a href="#uniq"><tt>uniq</tt></a>
    </td>
    <td> Analogs of their Unix-toolkit namesakes, discussed below as well as in
    POKI_PUT_LINK_FOR_PAGE(feature-comparison.html)HERE </td>
  </tr>

  <tr>
    <td>
      <a href="#filter"><tt>filter</tt></a>,
      <a href="#put"><tt>put</tt></a>,
      <a href="#step"><tt>step</tt></a>
    </td>
    <td> <tt>awk</tt>-like functionality </td>
  </tr>

  <tr>
    <td>
      <a href="#histogram"><tt>histogram</tt></a>,
      <a href="#stats1"><tt>stats1</tt></a>,
      <a href="#stats2"><tt>stats2</tt></a>
    </td>
    <td> Statistically oriented </td>
  </tr>

  <tr>
    <td>
      <a href="#group-by"><tt>group-by</tt></a>,
      <a href="#group-like"><tt>group-like</tt></a>,
      <a href="#having-fields"><tt>having-fields</tt></a>
    </td>
    <td> Particularly oriented toward POKI_PUT_LINK_FOR_PAGE(record-heterogeneity.html)HERE, although
      all Miller commands can handle heterogeneous records
  </tr>

  <tr>
    <td>
      <a href="#count-distinct"><tt>count-distinct</tt></a>,
      <a href="#label"><tt>label</tt></a>,
      <a href="#regularize"><tt>rename</tt></a>,
      <a href="#rename"><tt>rename</tt></a>,
      <a href="#reorder"><tt>reorder</tt></a>
    </td>
    <td> These draw from other sources (see also POKI_PUT_LINK_FOR_PAGE(originality.html)HERE):
      <a href="#count-distinct"><tt>count-distinct</tt></a> is SQL-ish, and
      <a href="#rename"><tt>rename</tt></a> can be done by <tt>sed</tt> (which does it faster:
        see POKI_PUT_LINK_FOR_PAGE(performance.html)HERE).
    </td>
  </tr>

</table>

<h1>On-line help</h1>

<p/>Examples:<p/>

POKI_RUN_COMMAND{{mlr --help}}HERE

POKI_RUN_COMMAND{{mlr sort --help}}HERE

<!-- ================================================================ -->
<h1>Data types</h1>

<p/> Miller&rsquo;s input and output are all string-oriented: there is (as of
August 2015 anyway) no support for binary record packing. In this sense,
everything is a string in and out of Miller.  During processing, field names
are always strings, even if they have names like "3"; field values are usually
strings.  Field values&rsquo; ability to be interpreted as a non-string type
only has meaning when comparison or function operations are done on them.  And
it is an error condition if Miller encounters non-numeric (or otherwise
mistyped) data in a field in which it has been asked to do numeric (or
otherwise type-specific) operations.

<p/> Field values are treated as numeric for the following:
<ul>
<li/> Numeric sort: <tt>mlr sort -n</tt>, <tt>mlr sort -nr</tt>.
<li/> Statistics: <tt>mlr histogram</tt>, <tt>mlr stats1</tt>, <tt>mlr stats2</tt>.
<li/> Cross-record arithmetic: <tt>mlr step</tt>.
</ul>

<p/>For <tt>mlr put</tt> and <tt>mlr filter</tt>:

<ul>

<li/> Miller&rsquo;s types for function processing are <b>null</b> (empty
string), <b>error</b>, <b>string</b>, <b>float</b> (double-precision),
<b>int</b> (64-bit signed), and <b>boolean</b>.

<li/> On input, string values representable as numbers, e.g. "3" or "3.1", are
treated as int or float, respectively. If a record has <tt>x=1,y=2</tt> then
<tt>mlr put '$z=$x+$y'</tt> will produce <tt>x=1,y=2,z=3</tt>, and <tt>mlr put
'$z=$x.$y'</tt> gives an error.  To coerce back to string for processing, use
the <tt>string</tt> function: <tt>mlr put '$z=string($x).string($y)'</tt> will
produce <tt>x=1,y=2,z=12</tt>.

<li/> On input, string values representable as boolean  (e.g. <tt>"true"</tt>,
<tt>"false"</tt>) are <i>not</i> automatically treated as boolean.  (This is
because <tt>"true"</tt> and <tt>"false"</tt> are ordinary words, and auto
string-to-boolean on a column consisting of words would result in some strings
mixed with some booleans.) Use the <tt>boolean</tt> function to coerce: e.g.
giving the record <tt>x=1,y=2,w=false</tt> to <tt>mlr put '$z=($x&lt;$y) ||
boolean($w)'</tt>.

<li/> Functions take types as described in <tt>mlr --help-all-functions</tt>:
for example, <tt>log10</tt> takes float input and produces float output,
<tt>gmt2sec</tt> maps string to int, and <tt>sec2gmt</tt> maps int to string.

<li/> All math functions described in <tt>mlr --help-all-functions</tt> take
integer as well as float input.

</ul>

<!-- ================================================================ -->
<h1>Null data</h1>

<p/> One of Miller&rsquo;s key features is its support for <b>heterogeneous</b> data.
Accordingly, if you try to sort on field <tt>hostname</tt> when not all records in the data
stream <i>have</i> a field named <tt>hostname</tt>, it is not an error (although you could
pre-filter the data stream using <tt>mlr having-fields --at-least hostname then sort ...</tt>).
Rather, records lacking one or more sort keys are simply output contiguously by <tt>mlr sort</tt>.

<p/> Field values may also be null by being
specified with present key but empty value: e.g. sending <tt>x=,y=2</tt> to <tt>mlr put '$z=$x+$y'</tt>.

<p/>
Rules for null-handling:
<ul>
<li> Records with one or more null sort-field values sort after records with all sort-field values present:
POKI_RUN_COMMAND{{mlr --opprint cat data/sort-null.dat}}HERE
POKI_RUN_COMMAND{{mlr --opprint sort -n  a data/sort-null.dat}}HERE
POKI_RUN_COMMAND{{mlr --opprint sort -nr a data/sort-null.dat}}HERE
<li> Functions which have one or more null arguments produce null output: e.g.
POKI_RUN_COMMAND{{echo 'x=2,y=3' | mlr put '$a=$x+$y'}}HERE
POKI_RUN_COMMAND{{echo 'x=,y=3' | mlr put '$a=$x+$y'}}HERE
POKI_RUN_COMMAND{{echo 'x=,y=3' | mlr put '$a=log($x);$b=log($y)'}}HERE
<li> The <tt>min</tt> and <tt>max</tt> functions are special: if one argument is non-null, it wins:
POKI_RUN_COMMAND{{echo 'x=,y=3' | mlr put '$a=min($x,$y);$b=max($x,$y)'}}HERE
</ul>

<!-- ================================================================ -->
<h1>I/O options</h1>

<!-- ================================================================ -->
<h2>Formats</h2>

<p/> Options:

<pre>
  --dkvp    --idkvp    --odkvp
  --nidx    --inidx    --onidx
  --csv     --icsv     --ocsv
  --csvlite --icsvlite --ocsvlite
  --pprint  --ipprint  --ppprint  --right
  --xtab    --ixtab    --oxtab
  --json    --ijson    --ojson
</pre>

<p/> These are as discussed in POKI_PUT_LINK_FOR_PAGE(file-formats.html)HERE, with the exception of <tt>--right</tt>
which makes pretty-printed output right-aligned:

<table><tr><td>
POKI_RUN_COMMAND{{mlr --opprint cat data/small}}HERE
</td><td>
POKI_RUN_COMMAND{{mlr --opprint --right cat data/small}}HERE
</td></tr></table>

<p/>Additional notes:

<ul>

<li/> Use <tt>--csv</tt>, <tt>--pprint</tt>, etc. when the input and output formats are the same.

<li/> Use <tt>--icsv --opprint</tt>, etc. when you want format conversion as part of what Miller does to your data.

<li/> DKVP (key-value-pair) format is the default for input and output. So,
<tt>--oxtab</tt> is the same as <tt>--idkvp --oxtab</tt>.

</ul>

<!-- ================================================================ -->
<h2>Compression</h2>

<p/> Options:

<pre>
  --prepipe {command}
</pre>

<p/>The prepipe command is anything which reads from standard input and produces data acceptable to
Miller. Nominally this allows you to use whichever decompression utilities you have installed on your
system, on a per-file basis. If the command has flags, quote them: e.g. <tt>mlr --prepipe 'zcat -cf'</tt>. Examples:

<p/>
<div class="pokipanel">
<pre>
# These two produce the same output:
$ gunzip &lt; myfile1.csv.gz | mlr cut -f hostname,uptime
$ mlr --prepipe gunzip cut -f hostname,uptime myfile1.csv.gz
# With multiple input files you need --prepipe:
$ mlr --prepipe gunzip cut -f hostname,uptime myfile1.csv.gz myfile2.csv.gz
$ mlr --prepipe gunzip --idkvp --oxtab cut -f hostname,uptime myfile1.dat.gz myfile2.dat.gz

# Similar to the above, but with compressed output as well as input:
$ gunzip &lt; myfile1.csv.gz | mlr cut -f hostname,uptime | gzip &gt; outfile.csv.gz
$ mlr --prepipe gunzip cut -f hostname,uptime myfile1.csv.gz | gzip &gt; outfile.csv.gz
$ mlr --prepipe gunzip cut -f hostname,uptime myfile1.csv.gz myfile2.csv.gz | gzip &gt; outfile.csv.gz

# Similar to the above, but with different compression tools for input and output:
$ gunzip &lt; myfile1.csv.gz | mlr cut -f hostname,uptime | xz -z &gt; outfile.csv.xz
$ xz -cd &lt; myfile1.csv.xz | mlr cut -f hostname,uptime | gzip &gt; outfile.csv.xz
$ mlr --prepipe 'xz -cd' cut -f hostname,uptime myfile1.csv.xz myfile2.csv.xz | xz -z &gt; outfile.csv.xz

... etc.
</pre>
</div>

<!-- ================================================================ -->
<h2>Record/field/pair separators</h2>

<p/> Miller has record separators <tt>IRS</tt> and <tt>ORS</tt>, field
separators <tt>IFS</tt> and <tt>OFS</tt>, and pair separators <tt>IPS</tt> and
<tt>OPS</tt>.  For example, in the DKVP line <tt>a=1,b=2,c=3</tt>, the record
separator is newline, field separator is comma, and pair separator is the
equals sign. These are the default values.

<p/> Options:
<pre>
  --rs --irs --ors
  --fs --ifs --ofs --repifs
  --ps --ips --ops
</pre>

<ul>

<li/> You can change a separator from input to output via e.g. <tt>--ifs =
--ofs :</tt>. Or, you can specify that the same separator is to be used for
input and output via e.g. <tt>--fs :</tt>.

<li/> The pair separator is only relevant to DKVP format.

<li/> Pretty-print and xtab formats ignore the separator arguments altogether.

<li/> The <tt>--repifs</tt> means that multiple successive occurrences of the
field separator count as one.  For example, in CSV data we often signify nulls
by empty strings, e.g. <tt>2,9,,,,,6,5,4</tt>. On the other hand, if the field
separator is a space, it might be more natural to parse <tt>2 4    5</tt> the
same as <tt>2 4 5</tt>: <tt>--repifs --ifs ' '</tt> lets this happen.  In fact,
the <tt>--ipprint</tt> option above is internally implemented in terms of
<tt>--repifs</tt>.

<li/> Just write out the desired separator, e.g. <tt>--ofs '|'</tt>. But you
may use the symbolic names <tt>newline</tt>, <tt>space</tt>, <tt>tab</tt>,
<tt>pipe</tt>, or <tt>semicolon</tt> if you like.

</ul>

<!-- ================================================================ -->
<h2>Number formatting</h2>

<p/> The command-line option <tt>--ofmt {format string}</tt> is the global
number format for commands which generate numeric output, e.g.
<tt>stats1</tt>, <tt>stats2</tt>, <tt>histogram</tt>, and <tt>step</tt>, as
well as <tt>mlr put</tt>. Examples:

POKI_CARDIFY(--ofmt %.9le  --ofmt %.6lf  --ofmt %.0lf)HERE

<p/> These are just C <tt>printf</tt> formats applied to double-precision
numbers.  Please don&rsquo;t use <tt>%s</tt> or <tt>%d</tt>. Additionally, if
you use leading width (e.g. <tt>%18.12lf</tt>) then the output will contain
embedded whitespace, which may not be what you want if you pipe the output to
something else, particularly CSV. I use Miller&rsquo;s pretty-print format
(<tt>mlr --opprint</tt>) to column-align numerical data.

<p/> To apply formatting to a single field, overriding the global
<tt>ofmt</tt>, use <tt>fmtnum</tt> function within <tt>mlr put</tt>. For example:
POKI_RUN_COMMAND{{echo 'x=3.1,y=4.3' | mlr put '$z=fmtnum($x*$y,"%08lf")'}}HERE
POKI_RUN_COMMAND{{echo 'x=0xffff,y=0xff' | mlr put '$z=fmtnum(int($x*$y),"%08llx")'}}HERE

<p/>Input conversion from hexadecimal is done automatically on fields handled
by <tt>mlr put</tt> and <tt>mlr filter</tt> as long as the field value begins
with "0x".  To apply output conversion to hexadecimal on a single column, you
may use <tt>fmtnum</tt>, or the keystroke-saving <tt>hexfmt</tt> function.
Example:

POKI_RUN_COMMAND{{echo 'x=0xffff,y=0xff' | mlr put '$z=hexfmt($x*$y)'}}HERE

<!-- ================================================================ -->
<h1>Data transformations</h1>

<!-- ================================================================ -->
<h2>bar</h2>

<p/> Cheesy bar-charting.

POKI_RUN_COMMAND{{mlr bar -h}}HERE

POKI_RUN_COMMAND{{mlr --opprint cat data/small}}HERE

POKI_RUN_COMMAND{{mlr --opprint bar --lo 0 --hi 1 -f x,y data/small}}HERE

POKI_RUN_COMMAND{{mlr --opprint bar --lo 0.4 --hi 0.6 -f x,y data/small}}HERE

POKI_RUN_COMMAND{{mlr --opprint bar --auto -f x,y data/small}}HERE

<!-- ================================================================ -->
<h2>bootstrap</h2>

POKI_RUN_COMMAND{{mlr bootstrap --help}}HERE

<p/> The canonical use for bootstrap sampling is to put error bars on statistical quantities, such as mean. For example:

<p/>
<div class="pokipanel">
<pre>
$ mlr --opprint stats1 -a mean,count -f u -g color data/colored-shapes.dkvp 
color  u_mean   u_count
yellow 0.497129 1413
red    0.492560 4641
purple 0.494005 1142
green  0.504861 1109
blue   0.517717 1470
orange 0.490532 303

$ mlr --opprint bootstrap then stats1 -a mean,count -f u -g color data/colored-shapes.dkvp 
color  u_mean   u_count
yellow 0.500651 1380
purple 0.501556 1111
green  0.503272 1068
red    0.493895 4702
blue   0.512529 1496
orange 0.521030 321

$ mlr --opprint bootstrap then stats1 -a mean,count -f u -g color data/colored-shapes.dkvp 
color  u_mean   u_count
yellow 0.498046 1485
blue   0.513576 1417
red    0.492870 4595
orange 0.507697 307
green  0.496803 1075
purple 0.486337 1199

$ mlr --opprint bootstrap then stats1 -a mean,count -f u -g color data/colored-shapes.dkvp 
color  u_mean   u_count
blue   0.522921 1447
red    0.490717 4617
yellow 0.496450 1419
purple 0.496523 1192
green  0.507569 1111
orange 0.468014 292
</pre>
</div>
<p/>

<!-- ================================================================ -->
<h2>cat</h2>

<p/> Most useful for format conversions (see
POKI_PUT_LINK_FOR_PAGE(file-formats.html)HERE), and concatenating multiple
same-schema CSV files to have the same header:

POKI_RUN_COMMAND{{mlr cat -h}}HERE

<table><tr><td>
POKI_RUN_COMMAND{{cat a.csv}}HERE
</td> <td>
POKI_RUN_COMMAND{{cat b.csv}}HERE
</td> <td>
POKI_RUN_COMMAND{{mlr --csv cat a.csv b.csv}}HERE
</td></tr></table>

<table><tr><td>
</td> <td>
POKI_RUN_COMMAND{{mlr --icsv --oxtab cat a.csv b.csv}}HERE
</td> <td>
POKI_RUN_COMMAND{{mlr --csv cat -n a.csv b.csv}}HERE
</td></tr></table>

<!-- ================================================================ -->
<h2>check</h2>

POKI_RUN_COMMAND{{mlr check --help}}HERE

<!-- ================================================================ -->
<h2>decimate</h2>

POKI_RUN_COMMAND{{mlr decimate --help}}HERE

<!-- ================================================================ -->
<h2>count-distinct</h2>

POKI_RUN_COMMAND{{mlr count-distinct --help}}HERE

POKI_RUN_COMMAND{{mlr count-distinct -f a,b then sort -nr count data/medium}}HERE

POKI_RUN_COMMAND{{mlr count-distinct -n -f a,b data/medium}}HERE

<!-- ================================================================ -->
<h2>cut</h2>

POKI_RUN_COMMAND{{mlr cut --help}}HERE

<table><tr><td>
POKI_RUN_COMMAND{{mlr --opprint cat data/small}}HERE
</td><td>
POKI_RUN_COMMAND{{mlr --opprint cut -f y,x,i data/small}}HERE
</td></tr><tr><td>
POKI_RUN_COMMAND{{echo 'a=1,b=2,c=3' | mlr cut -f b,c,a}}HERE
</td><td>
POKI_RUN_COMMAND{{echo 'a=1,b=2,c=3' | mlr cut -o -f b,c,a}}HERE
</td></tr></table>

<p/>
<!-- ================================================================ -->
<h2>filter</h2>

POKI_RUN_COMMAND{{mlr filter --help}}HERE

<h3>Field names for filter</h3>

<p/>Field names must be specified using a <tt>$</tt> in <tt>filter</tt> and <a
href="#put"><tt>put</tt></a> expressions, even though they don&rsquo;t appear
in the data stream. For integer-indexed data, this looks like
<tt>awk</tt>&rsquo;s <tt>$1,$2,$3</tt>.  Likewise, enclose string literals in
double quotes in <tt>filter</tt> expressions even though they don&rsquo;t
appear in file data.  In particular, <tt>mlr filter '$x=="abc"'</tt> passes
through the record <tt>x=abc</tt>.  If field names have special characters such
as <tt>.</tt> then you can use braces, e.g. <tt>'${field.name}'</tt>.

<h3>Built-in variables for filter</h3>

<p/>The <tt>filter</tt> command supports the same built-in variables as for <a
href="#put"><tt>put</tt></a>, all <tt>awk</tt>-inspired: <tt>NF</tt>,
<tt>NR</tt>, <tt>FNR</tt>, <tt>FILENUM</tt>, and <tt>FILENAME</tt>, as well as
the mathematical constants <tt>PI</tt> and <tt>E</tt>. This selects the 2nd
record from each matching file:

POKI_RUN_COMMAND{{mlr filter 'FNR == 2' data/small*}}HERE

<h3>Expression formatting for filter</h3>

<p/>Expressions may be arbitrarily complex:

POKI_RUN_COMMAND{{mlr --opprint filter '$a == "pan" || $b == "wye"' data/small}}HERE

<table><tr><td>
POKI_RUN_COMMAND{{mlr --opprint filter '($x > 0.5 && $y > 0.5) || ($x < 0.5 && $y < 0.5)' then stats2 -a corr -f x,y data/medium}}HERE
</td></tr><tr><td>
POKI_RUN_COMMAND{{mlr --opprint filter '($x > 0.5 && $y < 0.5) || ($x < 0.5 && $y > 0.5)' then stats2 -a corr -f x,y data/medium}}HERE
</td></tr></table>

Newlines within the expression are ignored, which can help increase legibility of complex expressions:

POKI_INCLUDE_ESCAPED(filter-multiline-example.txt)HERE

<!-- ================================================================ -->
<h2>grep</h2>

POKI_RUN_COMMAND{{mlr grep -h}}HERE

<!-- ================================================================ -->
<h2>group-by</h2>

POKI_RUN_COMMAND{{mlr group-by --help}}HERE

<p/>This is similar to <tt>sort</tt> but with less work. Namely, Miller&rsquo;s
sort has three steps: read through the data and append linked lists of records,
one for each unique combination of the key-field values; after all records
are read, sort the key-field values; then print each record-list. The group-by
operation simply omits the middle sort.  An example should make this more
clear.

<table><tr> <td>
POKI_RUN_COMMAND{{mlr --opprint group-by a data/small}}HERE
</td> <td>
POKI_RUN_COMMAND{{mlr --opprint sort -f a data/small}}HERE
</td> </tr></table>

<p/>In this example, since the sort is on field <tt>a</tt>, the first step is
to group together all records having the same value for field <tt>a</tt>; the
second step is to sort the distinct <tt>a</tt>-field values <tt>pan</tt>,
<tt>eks</tt>, and <tt>wye</tt> into <tt>eks</tt>, <tt>pan</tt>, and
<tt>wye</tt>; the third step is to print out the record-list for
<tt>a=eks</tt>, then the record-list for <tt>a=pan</tt>, then the record-list
for <tt>a=wye</tt>.  The group-by operation omits the middle sort and just puts
like records together, for those times when a sort isn&rsquo;t desired. In
particular, the ordering of group-by fields for group-by is the order in which
they were encountered in the data stream, which in some cases may be more interesting
to you.

<!-- ================================================================ -->
<h2>group-like</h2>

POKI_RUN_COMMAND{{mlr group-like --help}}HERE

<p/> This groups together records having the same schema (i.e. same ordered list of field names)
which is useful for making sense of time-ordered output as described in
POKI_PUT_LINK_FOR_PAGE(record-heterogeneity.html)HERE &mdash; in particular, in
preparation for CSV or pretty-print output.

<table><tr><td>
POKI_RUN_COMMAND{{mlr cat data/het.dkvp}}HERE
</td><td>
POKI_RUN_COMMAND{{mlr --opprint group-like data/het.dkvp}}HERE
</td></tr></table>

<!-- ================================================================ -->
<h2>having-fields</h2>

POKI_RUN_COMMAND{{mlr having-fields --help}}HERE

<p/> Similar to <a href="#group-like"><tt>group-like</tt></a>, this retains records with specified schema.

<table><tr><td>
POKI_RUN_COMMAND{{mlr cat data/het.dkvp}}HERE
</td></tr><tr><td>
POKI_RUN_COMMAND{{mlr having-fields --at-least resource data/het.dkvp}}HERE
</td></tr><tr><td>
POKI_RUN_COMMAND{{mlr having-fields --which-are resource,ok,loadsec data/het.dkvp}}HERE
</td></tr></table>

<!-- ================================================================ -->
<h2>head</h2>

POKI_RUN_COMMAND{{mlr head --help}}HERE

Note that <tt>head</tt> is distinct from <a href="#top"><tt>top</tt></a>
&mdash; <tt>head</tt> shows fields which appear first in the data stream;
<tt>top</tt> shows fields which are numerically largest (or smallest).

<table><tr><td>
POKI_RUN_COMMAND{{mlr --opprint head -n 4 data/medium}}HERE
</td><td>
POKI_RUN_COMMAND{{mlr --opprint head -n 1 -g b data/medium}}HERE
</td></tr></table>

<!-- ================================================================ -->
<h2>histogram</h2>

POKI_RUN_COMMAND{{mlr histogram --help}}HERE

This is just a histogram; there&rsquo;s not too much to say here. A note about
binning, by example: Suppose you use <tt>--lo 0.0 --hi 1.0 --nbins 10 -f
x</tt>.  The input numbers less than 0 or greater than 1 aren&rsquo;t counted
in any bin.  Input numbers equal to 1 are counted in the last bin. That is, bin
0 has <tt>0.0 &le; x &lt; 0.1</tt>, bin 1 has <tt>0.1 &le; x &lt; 0.2</tt>,
etc., but bin 9 has <tt>0.9 &le; x &le; 1.0</tt>.

POKI_RUN_COMMAND{{mlr --opprint put '$x2=$x**2;$x3=$x2*$x' then histogram -f x,x2,x3 --lo 0 --hi 1 --nbins 10 data/medium}}HERE

<!-- ================================================================ -->
<h2>join</h2>

POKI_RUN_COMMAND{{mlr join --help}}HERE

Examples:

<p/>Join larger table with IDs with smaller ID-to-name lookup table, showing only paired records:

<table><tr><td>
POKI_RUN_COMMAND{{mlr --icsvlite --opprint cat data/join-left-example.csv}}HERE
</td></tr><tr><td>
POKI_RUN_COMMAND{{mlr --icsvlite --opprint cat data/join-right-example.csv}}HERE
</td></tr><tr><td>
POKI_RUN_COMMAND{{mlr --icsvlite --opprint join -u -j id -r idcode -f data/join-left-example.csv data/join-right-example.csv}}HERE
</td></tr></table>

<p/>Same, but with sorting the input first:
<table><tr><td>
POKI_RUN_COMMAND{{mlr --icsvlite --opprint sort -f idcode then join -j id -r idcode -f data/join-left-example.csv data/join-right-example.csv}}HERE
</td></tr></table>

<p/>Same, but showing only unpaired records:
<table><tr><td>
POKI_RUN_COMMAND{{mlr --icsvlite --opprint join --np --ul --ur -u -j id -r idcode -f data/join-left-example.csv data/join-right-example.csv}}HERE
</td></tr></table>

<p/>Use prefixing options to disambiguate between otherwise identical non-join field names:
<table><tr><td>
POKI_RUN_COMMAND{{mlr --csvlite --opprint cat data/self-join.csv data/self-join.csv}}HERE
</td></tr><tr><td>
POKI_RUN_COMMAND{{mlr --csvlite --opprint join -j a --lp left_ --rp right_ -f data/self-join.csv data/self-join.csv}}HERE
</td></tr></table>

<p/>Use zero join columns:
<table><tr><td>
POKI_RUN_COMMAND{{mlr --csvlite --opprint join -j "" --lp left_ --rp right_ -f data/self-join.csv data/self-join.csv}}HERE
</td></tr></table>

<!-- ================================================================ -->
<h2>label</h2>

POKI_RUN_COMMAND{{mlr label --help}}HERE

See also <a href="#rename"><tt>rename</tt></a>.

<p/>Example: Files such as <tt>/etc/passwd</tt>, <tt>/etc/group</tt>, and so on
have implicit field names which are found in section-5 manpages. These field names may be made explicit as follows:

POKI_INCLUDE_ESCAPED(data/label-example.txt)HERE

<p/>Likewise, if you have CSV/CSV-lite input data which has somehow been bereft of its header line, you can re-add a header line using <tt>--implicit-csv-header</tt> and <tt>label</tt>:

POKI_RUN_COMMAND{{cat data/headerless.csv}}HERE
POKI_RUN_COMMAND{{mlr --csv --rs lf --implicit-csv-header cat data/headerless.csv}}HERE
POKI_RUN_COMMAND{{mlr --csv --rs lf --implicit-csv-header label name,age,status data/headerless.csv}}HERE
POKI_RUN_COMMAND{{mlr --icsv --rs lf --implicit-csv-header --opprint label name,age,status data/headerless.csv}}HERE

<!-- ================================================================ -->
<h2>merge-fields</h2>

POKI_RUN_COMMAND{{mlr merge-fields --help}}HERE

<p/>This is like <tt>mlr stats1</tt> but all accumulation is done across fields
within each given record: horizontal rather than vertical statistics, if you
will.

<p/>Examples:

POKI_RUN_COMMAND{{mlr --csvlite --opprint cat data/inout.csv}}HERE
POKI_RUN_COMMAND{{mlr --csvlite --opprint merge-fields -a min,max,sum -c _in,_out data/inout.csv}}HERE
POKI_RUN_COMMAND{{mlr --csvlite --opprint merge-fields -k -a sum -c _in,_out data/inout.csv}}HERE

<!-- ================================================================ -->
<h2>put</h2>

POKI_RUN_COMMAND{{mlr put --help}}HERE

<h3>Field names for put</h3>

<p/>Field names must be specified using a <tt>$</tt> in <a
href="#filter"><tt>filter</tt></a> and <tt>put</tt> expressions, even though
they don&rsquo;t appear in the data stream. For integer-indexed data, this
looks like <tt>awk</tt>&rsquo;s <tt>$1,$2,$3</tt>.  Likewise, enclose string
literals in double quotes in <tt>put</tt> expressions even though they
don&rsquo;t appear in file data. In particular, <tt>mlr put '$x="abc"'</tt>
creates the field <tt>x=abc</tt> and <tt>mlr filter '$x=="abc"'</tt> passes
through the field <tt>x</tt> if it has the value <tt>abc</tt>.  If field names
have special characters such as <tt>.</tt> then you can use braces, e.g.
<tt>'${field.name}'</tt>.

<h3>Built-in variables for put</h3>

<p/>Miller supports the following five built-in variables for <a
href="#filter"><tt>filter</tt></a> and <tt>put</tt>, all <tt>awk</tt>-inspired:
<tt>NF</tt>, <tt>NR</tt>, <tt>FNR</tt>, <tt>FILENUM</tt>, and
<tt>FILENAME</tt>, as well as the mathematical constants <tt>PI</tt> and
<tt>E</tt>.

<h3>Expression formatting for put</h3>

<p/>Multiple expressions may be given, separated by semicolons, and each may refer to the ones before:

POKI_RUN_COMMAND{{ruby -e '10.times{|i|puts "i=#{i}"}' | mlr --opprint put '$j = $i + 1; $k = $i +$j'}}HERE

POKI_RUN_COMMAND{{mlr --opprint put '$nf=NF; $nr=NR; $fnr=FNR; $filenum=FILENUM; $filename=FILENAME' data/small data/small2}}HERE

Newlines within the expression are ignored, which can help increase legibility of complex expressions:

POKI_INCLUDE_ESCAPED(data/put-multiline-example.txt)HERE

<h3>Gating assignments by boolean expressions for put</h3>

<p/>Boolean expressions may be used as gates for subsequent semicolon-separated
expressions within the same <tt>put</tt>. This can be used to allow assignments
to be done only when appropriate &mdash; e.g. for math-function domain
restrictions, regex-matching, and so on:

POKI_RUN_COMMAND{{mlr cat data/put-gating-example-1.dkvp}}HERE
POKI_RUN_COMMAND{{mlr put 'gate $x > 0.0; $y = log10($x); $z = sqrt($y)' data/put-gating-example-1.dkvp}}HERE
POKI_RUN_COMMAND{{mlr cat data/put-gating-example-2.dkvp}}HERE
POKI_RUN_COMMAND{{mlr put 'gate $a =~ "([a-z]+)_([0-9]+)"; $b = "left_\1"; $c = "right_\2"' data/put-gating-example-2.dkvp}}HERE

<p/>This produces heteregenous output which Miller, of course, has no problems
with (see POKI_PUT_LINK_FOR_PAGE(record-heterogeneity.html)HERE).  But
if homogeneous output is desired, the <tt>gate</tt> can be omitted.  This
causes <tt>put</tt> to evaluate the boolean expression (along with any side
effects, namely, regex-captures <tt>\1</tt>, <tt>\2</tt>, etc.) but
doesn&rsquo;t use it as a criterion for whether subsequent assignments should
be executed. Instead, subsequent assignments are done unconditionally:

POKI_RUN_COMMAND{{mlr put '$x > 0.0; $y = log10($x); $z = sqrt($y)' data/put-gating-example-1.dkvp}}HERE
POKI_RUN_COMMAND{{mlr put '$a =~ "([a-z]+)_([0-9]+)"; $b = "left_\1"; $c = "right_\2"' data/put-gating-example-2.dkvp}}HERE

<!-- ================================================================ -->
<h2>regularize</h2>

POKI_RUN_COMMAND{{mlr regularize --help}}HERE

<p/>This exists since hash-map software in various languages and tools
encountered in the wild does not always print similar rows with fields in the
same order: <tt>mlr regularize</tt> helps clean that up.

<p/>See also <a href="#reorder"><tt>reorder</tt></a>.

<!-- ================================================================ -->
<h2>rename</h2>

POKI_RUN_COMMAND{{mlr rename --help}}HERE

<table><tr><td>
POKI_RUN_COMMAND{{mlr --opprint cat data/small}}HERE
</td><td>
POKI_RUN_COMMAND{{mlr --opprint rename i,INDEX,b,COLUMN2 data/small}}HERE
</td></tr></table>

<p/>As discussed in POKI_PUT_LINK_FOR_PAGE(performance.html)HERE, <tt>sed</tt>
is significantly faster than Miller at doing this. However, Miller is
format-aware, so it knows to do renames only within specified field keys and
not any others, nor in field values which may happen to contain the same
pattern. Example:

<table><tr><td>
POKI_RUN_COMMAND{{sed 's/y/COLUMN5/g' data/small}}HERE
</td></tr><tr><td>
POKI_RUN_COMMAND{{mlr rename y,COLUMN5 data/small}}HERE
</td></tr></table>

See also <a href="#label"><tt>label</tt></a>.

<!-- ================================================================ -->
<h2>reorder</h2>

POKI_RUN_COMMAND{{mlr reorder --help}}HERE

This pivots specified field names to the start or end of the record &mdash; for
example when you have highly multi-column data and you want to bring a field or
two to the front of line where you can give a quick visual scan.

<table><tr><td>
POKI_RUN_COMMAND{{mlr --opprint cat data/small}}HERE
</td></tr><tr><td>
POKI_RUN_COMMAND{{mlr --opprint reorder -f i,b data/small}}HERE
</td><td>
POKI_RUN_COMMAND{{mlr --opprint reorder -e -f i,b data/small}}HERE
</td></tr></table>

<!-- ================================================================ -->
<h2>reshape</h2>

POKI_RUN_COMMAND{{mlr reshape --help}}HERE

<!-- ================================================================ -->
<h2>sample</h2>

POKI_RUN_COMMAND{{mlr sample --help}}HERE

<p/>This is reservoir-sampling: select <i>k</i> items from <i>n</i> with
uniform probability and no repeats in the sample. (If <i>n</i> is less than
<i>k</i>, then of course only <i>n</i> samples are produced.) With <tt>-g
{field names}</tt>, produce a <i>k</i>-sample for each distinct value of the
specified field names.

POKI_INCLUDE_ESCAPED(data/sample-example.txt)HERE

<p/>Note that no output is produced until all inputs are in. Another way to do
sampling, which works in the streaming case, is <tt>mlr filter 'urand() &amp;
0.001'</tt> where you tune the 0.001 to meet your needs.

<!-- ================================================================ -->
<h2>sec2gmt</h2>

POKI_RUN_COMMAND{{mlr sec2gmt -h}}HERE

<!-- ================================================================ -->
<h2>sort</h2>

POKI_RUN_COMMAND{{mlr sort --help}}HERE

<p/>Example:
POKI_RUN_COMMAND{{mlr --opprint sort -f a -nr x data/small}}HERE

<p/>Here&rsquo;s an example filtering log data: suppose multiple threads (labeled here by color) are all logging progress counts to a single log file. The log file is (by nature) chronological, so the progress of various threads is interleaved:

POKI_RUN_COMMAND{{head -n 10 data/multicountdown.dat}}HERE

<p/> We can group these by thread by sorting on the thread ID (here,
<tt>color</tt>). Since Miller&rsquo;s sort is stable, this means that
timestamps within each thread&rsquo;s log data are still chronological:

POKI_RUN_COMMAND{{head -n 20 data/multicountdown.dat | mlr --opprint sort -f color}}HERE

<p/>Any records not having all specified sort keys will appear at the end of the output, in the order they
were encountered, regardless of the specified sort order:

POKI_RUN_COMMAND{{mlr sort -n  x data/sort-missing.dkvp}}HERE
POKI_RUN_COMMAND{{mlr sort -nr x data/sort-missing.dkvp}}HERE

<!-- ================================================================ -->
<h2>stats1</h2>

POKI_RUN_COMMAND{{mlr stats1 --help}}HERE

These are simple univariate statistics on one or more number-valued fields
(<tt>count</tt> and <tt>mode</tt> apply to non-numeric fields as well),
optionally categorized by one or more other fields.

<table><tr><td>
POKI_RUN_COMMAND{{mlr --oxtab stats1 -a count,sum,min,p10,p50,mean,p90,max -f x,y data/medium}}HERE
</td></tr><tr><td>
POKI_RUN_COMMAND{{mlr --opprint stats1 -a mean -f x,y -g b then sort -f b data/medium}}HERE
</td></tr><tr><td>
POKI_RUN_COMMAND{{mlr --opprint stats1 -a p50,p99 -f u,v -g color then put '$ur=$u_p99/$u_p50;$vr=$v_p99/$v_p50' data/colored-shapes.dkvp}}HERE
</td></tr><tr><td>
POKI_RUN_COMMAND{{mlr --opprint count-distinct -f shape then sort -nr count data/colored-shapes.dkvp}}HERE
</td></tr><tr><td>
POKI_RUN_COMMAND{{mlr --opprint stats1 -a mode -f color -g shape data/colored-shapes.dkvp}}HERE
</td></tr></table>

<!-- ================================================================ -->
<h2>stats2</h2>

POKI_RUN_COMMAND{{mlr stats2 --help}}HERE

These are simple bivariate statistics on one or more pairs of number-valued
fields, optionally categorized by one or more fields.

<table><tr><td>
POKI_RUN_COMMAND{{mlr --oxtab put '$x2=$x*$x; $xy=$x*$y; $y2=$y**2' then stats2 -a cov,corr -f x,y,y,y,x2,xy,x2,y2 data/medium}}HERE
</td></tr><tr><td>
POKI_RUN_COMMAND{{mlr --opprint put '$x2=$x*$x; $xy=$x*$y; $y2=$y**2' then stats2 -a linreg-ols,r2 -f x,y,y,y,xy,y2 -g a data/medium}}HERE
</td></tr></table>

<p/>Here&rsquo;s an example simple line-fit. The <tt>x</tt> and <tt>y</tt>
fields of the <tt>data/medium</tt> dataset are just independent uniformly
distributed on the unit interval. Here we remove half the data and fit a line to it.

POKI_INCLUDE_ESCAPED(data/linreg-example.txt)HERE

<p/>I use <a href="https://github.com/johnkerl/pgr"><tt>pgr</tt></a> for
plotting; here&rsquo;s a screenshot.

<center>
<img src="data/linreg-example.jpg"/>
</center>

<p/> (Thanks Drew Kunas for a good conversation about PCA!)

<p/> Here&rsquo;s an example estimating time-to-completion for a set of jobs.
Input data comes from a log file, with number of work units left to do in the
<tt>count</tt> field and accumulated seconds in the <tt>upsec</tt> field,
labeled by the <tt>color</tt> field:

POKI_RUN_COMMAND{{head -n 10 data/multicountdown.dat}}HERE

We can do a linear regression on count remaining as a function of time: with <tt>c = m*u+b</tt> we want to find the
time when the count goes to zero, i.e. <tt>u=-b/m</tt>.

POKI_RUN_COMMAND{{mlr --oxtab stats2 -a linreg-pca -f upsec,count -g color then put '$donesec = -$upsec_count_pca_b/$upsec_count_pca_m' data/multicountdown.dat}}HERE

<!-- ================================================================ -->
<h2>step</h2>

POKI_RUN_COMMAND{{mlr step --help}}HERE

Most Miller commands are record-at-a-time, with the exception of <tt>stats1</tt>,
<tt>stats2</tt>, and <tt>histogram</tt> which compute aggregate output. The
<tt>step</tt> command is intermediate: it allows the option of adding fields
which are functions of fields from previous records. Rsum is short for <i>running sum</i>.

<table><tr><td>
POKI_RUN_COMMAND{{mlr --opprint step -a delta,rsum,counter -f x data/medium | head -15}}HERE
</td></tr><tr><td>
POKI_RUN_COMMAND{{mlr --opprint step -a delta,rsum,counter -f x -g a data/medium | head -15}}HERE
</td></tr><tr><td>
POKI_RUN_COMMAND{{mlr --opprint step -a ewma -f x -d 0.1,0.9 ../doc/data/medium | head -15}}HERE
</td></tr><tr><td>
POKI_RUN_COMMAND{{mlr --opprint step -a ewma -f x -d 0.1,0.9 -o smooth,rough ../doc/data/medium | head -15}}HERE
</td></tr></table>

Example deriving uptime-delta from system uptime:

POKI_INCLUDE_ESCAPED(data/ping-delta-example.txt)HERE

<!-- ================================================================ -->
<h2>tac</h2>

POKI_RUN_COMMAND{{mlr tac --help}}HERE

<p/>Prints the records in the input stream in reverse order. Note: this
requires Miller to retain all input records in memory before any output records
are produced.

<table><tr><td>
POKI_RUN_COMMAND{{mlr --icsv --opprint cat a.csv}}HERE
</td><td>
POKI_RUN_COMMAND{{mlr --icsv --opprint cat b.csv}}HERE
</td><td>
POKI_RUN_COMMAND{{mlr --icsv --opprint tac a.csv b.csv}}HERE
</td></tr></table>
<table><tr><td>
POKI_RUN_COMMAND{{mlr --icsv --opprint put '$filename=FILENAME' then tac a.csv b.csv}}HERE
</td></tr></table>

<!-- ================================================================ -->
<h2>tail</h2>

POKI_RUN_COMMAND{{mlr tail --help}}HERE

<p/> Prints the last <i>n</i> records in the input stream, optionally by category.

<table><tr><td>
POKI_RUN_COMMAND{{mlr --opprint tail -n 4 data/colored-shapes.dkvp}}HERE
</td></tr><tr><td>
POKI_RUN_COMMAND{{mlr --opprint tail -n 1 -g shape data/colored-shapes.dkvp}}HERE
</td></tr></table>

<!-- ================================================================ -->
<h2>top</h2>

POKI_RUN_COMMAND{{mlr top --help}}HERE

Note that <tt>top</tt> is distinct from <a href="#head"><tt>head</tt></a>
&mdash; <tt>head</tt> shows fields which appear first in the data stream;
<tt>top</tt> shows fields which are numerically largest (or smallest).

<table><tr><td>
POKI_RUN_COMMAND{{mlr --opprint top -n 4 -f x data/medium}}HERE
</td><td>
POKI_RUN_COMMAND{{mlr --opprint top -n 2 -f x -g a then sort -f a data/medium}}HERE
</td></tr></table>

<!-- ================================================================ -->
<h2>uniq</h2>

POKI_RUN_COMMAND{{mlr uniq --help}}HERE

<table><tr><td>
POKI_RUN_COMMAND{{wc -l data/colored-shapes.dkvp}}HERE
</td></tr><tr><td>
POKI_RUN_COMMAND{{mlr uniq -g color,shape data/colored-shapes.dkvp}}HERE
</td></tr><tr><td>
POKI_RUN_COMMAND{{mlr --opprint uniq -g color,shape -c then sort -f color,shape data/colored-shapes.dkvp}}HERE
</td></tr><tr><td>
POKI_RUN_COMMAND{{mlr --opprint uniq -n -g color,shape data/colored-shapes.dkvp}}HERE
</td></tr></table>

<!-- ================================================================ -->
<h1>then-chaining</h1>

<p/>
In accord with the
<a href="http://en.wikipedia.org/wiki/Unix_philosophy">Unix philosophy</a>, you can pipe data into or out of
Miller. For example:

POKI_CARDIFY(mlr cut --complement -f os_version *.dat | mlr sort -f hostname,uptime)HERE

<p/>
You can, if you like, instead simply chain commands together using the
<tt>then</tt> keyword:

POKI_CARDIFY(mlr cut --complement -f os_version then sort -f hostname,uptime *.dat)HERE

Here&rsquo;s a performance comparison:

POKI_INCLUDE_ESCAPED(data/then-chaining-performance.txt)HERE

There are two reasons to use then-chaining: one is for performance, although I
don&rsquo;t expect this to be a win in all cases.  Using then-chaining avoids
redundant string-parsing and string-formatting at each pipeline step: instead
input records are parsed once, they are fed through each pipeline stage in
memory, and then output records are formatted once. On the other hand, Miller
is single-threaded, while modern systems are usually multi-processor, and when
streaming-data programs operate through pipes, each one can use a CPU.  Rest
assured you get the same results either way.

<p/>The other reason to use then-chaining is for simplicity: you don&rsquo;t
have re-type formatting flags (e.g. <tt>--csv --rs lf --fs tab</tt>) at every
pipeline stage.

<!-- ================================================================ -->
<h1>Functions for filter and put</h1>

POKI_RUN_COMMAND{{mlr --help-all-functions}}HERE

<!-- ================================================================ -->
<h1>Operator precedence</h1>

<p/>Operators are listed in order of decreasing precedence, highest first.

<p/>
<div class="pokipanel">
<pre>
Operators              Associativity
---------              -------------
()                     left to right
**                     right to left
! ~ unary+ unary- &amp;    right to left
binary* / // %         left to right
binary+ binary- .      left to right
&lt;&lt; &gt;&gt;                  left to right
&amp;                      left to right
^                      left to right
|                      left to right
&lt; &lt;= &gt; &gt;=              left to right
== != =~ !=~           left to right
&amp;&amp;                     left to right
^^                     left to right
||                     left to right
? :                    right to left
=                      N/A for Miller (there is no $a=$b=$c)
</pre>
</div>
<p/>

<!-- ================================================================ -->
<h1>Operator and function semantics</h1>

<ul>

<li/> Functions are in general pass-throughs straight to the system-standard C
library.

<li/> The <tt>min</tt> and <tt>max</tt> functions are different from other
multi-argument functions which return null if any of their inputs are null: for
<tt>min</tt> and <tt>max</tt>, by contrast, if one argument is null, the other
is returned.

<li/> Symmetrically with respect to the bitwise OR, XOR, and AND operators
<tt>|</tt>, <tt>^</tt>, <tt>&amp;</tt>, Miller has logical operators
<tt>||</tt>, <tt>^^</tt>, <tt>&amp;&amp;</tt>: the logical XOR not existing in
C.

<li/> The exponentiation operator <tt>**</tt> is familiar from many languages.

<li/> The regex-match and regex-not-match operators <tt>=~</tt> and
<tt>!=~</tt> are similar to those in Ruby and Perl.

</ul>

<!-- ================================================================ -->
<h1>Arithmetic</h1>

<h2>Input scanning</h2>

<p/>Numbers in Miller are double-precision float or 64-bit signed integers.
Anything scannable as int, e.g <tt>123</tt> or <tt>0xabcd</tt>, is treated as
an integer; otherwise, input scannable as float (<tt>4.56</tt> or <tt>8e9</tt>)
is treated as float; everything else is a string.

<p/>If you want all numbers to be treated as floats, then you may use
<tt>float()</tt> in your filter/put expressions (e.g. replacing <tt>$c = $a *
$b</tt> with <tt>$c = float($a) * float($b)</tt>) &mdash; or, more simply, use
<tt>mlr filter -F</tt> and <tt>mlr put -F</tt> which forces all numeric input,
whether from expression literals or field values, to float. Likewise <tt>mlr
stats1 -F</tt> and <tt>mlr step -F</tt> force integerable accumulators (such as
<tt>count</tt>) to be done in floating-point.

<h2>Conversion by math routines</h2>

<p/>For most math functions, integers are cast to float on input, and produce
float output: e.g. <tt>exp(0) = 1.0</tt> rather than <tt>1</tt>.  The
following, however, produce integer output if their inputs are integers:
<tt>+</tt> <tt>-</tt> <tt>*</tt> <tt>/</tt> <tt>//</tt> <tt>%</tt> <tt>abs</tt>
<tt>ceil</tt> <tt>floor</tt> <tt>max</tt> <tt>min</tt> <tt>round</tt>
<tt>roundm</tt> <tt>sgn</tt>. As well, <tt>stats1 -a min</tt>, <tt>stats1 -a
max</tt>, <tt>stats1 -a sum</tt>, <tt>step -a delta</tt>, and <tt>step -a
rsum</tt> produce integer output if their inputs are integers.

<h2>Conversion by arithmetic operators</h2>

<p/>The sum, difference, and product of integers is again integer, except for
when that would overflow a 64-bit integer at which point Miller converts the
result to float.

<p/>The short of it is that Miller does this transparently for you so you
needn&rsquo;t think about it.

<p/>Implementation details of this, for the interested: integer adds and
subtracts overflow by at most one bit so it suffices to check sign-changes.
Thus, Miller allows you to add and subtract arbitrary 64-bit signed integers,
converting only to float precisely when the result is less than -2<sup>63</sup>
or greater than 2<sup>63</sup>-1.  Multiplies, on the other hand, can overflow
by a word size and a sign-change technique does not suffice to detect overflow.
Instead Miller tests whether the floating-point product exceeds the
representable integer range. Now, 64-bit integers have 64-bit precision while
IEEE-doubles have only 52-bit mantissas &mdash; so, there are 53 bits including
implicit leading one.  The following experiment explicitly demonstrates the
resolution at this range:

<div class=pokipanel>
<pre>
   64-bit integer     64-bit integer     Casted to double           Back to 64-bit
       in hex           in decimal                                    integer
0x7ffffffffffff9ff 9223372036854774271 9223372036854773760.000000 0x7ffffffffffff800
0x7ffffffffffffa00 9223372036854774272 9223372036854773760.000000 0x7ffffffffffff800
0x7ffffffffffffbff 9223372036854774783 9223372036854774784.000000 0x7ffffffffffffc00
0x7ffffffffffffc00 9223372036854774784 9223372036854774784.000000 0x7ffffffffffffc00
0x7ffffffffffffdff 9223372036854775295 9223372036854774784.000000 0x7ffffffffffffc00
0x7ffffffffffffe00 9223372036854775296 9223372036854775808.000000 0x8000000000000000
0x7ffffffffffffffe 9223372036854775806 9223372036854775808.000000 0x8000000000000000
0x7fffffffffffffff 9223372036854775807 9223372036854775808.000000 0x8000000000000000
</pre>
</div>

<p/>That is, one cannot check an integer product to see if it is precisely
greater than 2<sup>63</sup>-1 or less than -2<sup>63</sup> using either integer
arithmetic (it may have already overflowed) or using double-precision (due to
granularity).  Instead Miller checks for overflow in 64-bit integer
multiplication by seeing whether the absolute value of the double-precision
product exceeds the largest representable IEEE double less than 2<sup>63</sup>,
which we see from the listing above is 9223372036854774784. (An alternative
would be to do all integer multiplies using handcrafted multi-word 128-bit
arithmetic.  This approach is not taken.)

<h2>Pythonic division</h2>

<p/>Division and remainder are
<a href="http://python-history.blogspot.com/2010/08/why-pythons-integer-division-floors.html">
pythonic</a>:
  <ul>
  <li/> Quotient of integers is floating-point: <tt>7/2</tt> is <tt>3.5</tt>.
  <li/> Integer division is done with <tt>//</tt>: <tt>7/2</tt> is <tt>3</tt>.
    This rounds toward the negative.
  <li/> Remainders are non-negative.
  </ul>

<!-- ================================================================ -->
<h1>Regular expressions</h1>

<p/>Miller lets you use regular expressions (of type POSIX.2) in the following contexts:

<ul>

<li/> In <tt>mlr filter</tt> with <tt>=~</tt> or <tt>!=~</tt>, e.g. <tt>mlr
filter '$url =~ "http.*com"'</tt>

<li/> In <tt>mlr put</tt> with <tt>sub</tt> or <tt>gsub</tt>, e.g. <tt>mlr put
'$url = sub($url, "http.*com", "")'</tt>

<li/> In <tt>mlr having-fields</tt>, e.g. <tt>mlr having-fields
--any-matching '^sda[0-9]'</tt>

<li/> In <tt>mlr cut</tt>, e.g. <tt>mlr cut -r -f '^status$,^sda[0-9]'</tt>

<li/> In <tt>mlr rename</tt>, e.g. <tt>mlr rename -r '^(sda[0-9]).*$,dev/\1'</tt>

<li/> In <tt>mlr grep</tt>, e.g. <tt>mlr --csv grep 00188555487 myfiles*.csv</tt>

</ul>

<p/>Points demonstrated by the above examples:

<ul>

<li/> There are no implicit start-of-string or end-of-string anchors; please
use <tt>^</tt> and/or <tt>$</tt> explicitly.

<li/> Miller regexes are wrapped with double quotes rather than slashes.

<li/> The <tt>i</tt> after the ending double quote indicates a case-insensitive
regex.

<li/> Capture groups are wrapped with <tt>(...)</tt> rather than
<tt>\(...\)</tt>; use <tt>\(</tt> and <tt>\)</tt> to match against parentheses.

</ul>

<p/>For <tt>filter</tt> and <tt>put</tt>, if the regular expression is a string
literal (the normal case), it is precompiled at process start and reused
thereafter, which is efficient. If the regular expression is a more complex
expression, including string concatenation using <tt>.</tt>, or a column name
(in which case you can take regular expressions from input data!), then regexes
are compiled on each record which works but is less efficient. As well, in this
case there is no way to specify case-insensitive matching.

<p/>Example:

POKI_RUN_COMMAND{{cat data/regex-in-data.dat}}HERE
POKI_RUN_COMMAND{{mlr filter '$name =~ $regex' data/regex-in-data.dat}}HERE

<h2>Regex captures</h2>

<p/>Regex captures of the form <tt>\1</tt>, <tt>\2</tt>, etc. are supported as follows:
<ul>

<li/> Captures have in-function context for <tt>sub</tt> and <tt>gsub</tt>.
For example, the first <tt>\1,\2</tt> pair belong to the first <tt>sub</tt> and
the second <tt>\1,\2</tt> pair belong to the second <tt>sub</tt>:

<p/>
<div class=pokipanel>
<pre>
mlr put '$b = sub($a, "(..)_(...)", "\2-\1"); $c = sub($a, "(..)_(.)(..)", ":\1:\2:\3")'
</pre>
</div>

<li/> Captures endure for the entirety of a <tt>put</tt> for the <tt>=~</tt>
and <tt>!=~</tt> operators. For example, here the <tt>\1,\2</tt> are set by the
<tt>=~</tt> operator and are used by both subsequent assignment statements:

<p/>
<div class=pokipanel>
<pre>
mlr put '$a =~ "(..)_(....); $b = "left_\1"; $c = "right_\2"'
</pre>
</div>

<li/>The captures are not retained across multiple puts. For example, here the
<tt>\1,\2</tt> won&rsquo;t be expanded from the regex capture:

<p/>
<div class=pokipanel>
<pre>
mlr put '$a =~ "(..)_(....)' then {... something else ...} then put '$b = "left_\1"; $c = "right_\2"'
</pre>
</div>

<li/> Captures are ignored in <tt>filter</tt> for the <tt>=~</tt> and
<tt>!=~</tt> operators. For example, there is no mechanism provided to refer to
the first <tt>(..)</tt> as <tt>\1</tt> or to the second <tt>(....)</tt> as
<tt>\2</tt> in the following filter statement:

<p/>
<div class=pokipanel>
<pre>
mlr filter '$a =~ "(..)_(....)'
</pre>
</div>

<li/> Up to nine matches are supported: <tt>\1</tt> through <tt>\9</tt>, while
<tt>\0</tt> is the entire match string; <tt>\15</tt> is treated as <tt>\1</tt>
followed by an unrelated <tt>5</tt>.
</ul>