miller/doc/mlr.1.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE refentry PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd">
<?asciidoc-toc?>
<?asciidoc-numbered?>
<refentry lang="en">
<refentryinfo>
    <title>MILLER(1)</title>
</refentryinfo>
<refmeta>
<refentrytitle>miller</refentrytitle>
<manvolnum>1</manvolnum>
<refmiscinfo class="source">&#160;</refmiscinfo>
<refmiscinfo class="manual">&#160;</refmiscinfo>
</refmeta>
<refnamediv>
    <refname>miller</refname>
    <refpurpose>sed, awk, cut, join, and sort for name-indexed data such as CSV</refpurpose>
</refnamediv>
<refsynopsisdiv id="_synopsis">
<simpara></simpara>
</refsynopsisdiv>
<refsect1 id="_description">
<title>DESCRIPTION</title>
<simpara>With Miller you get to use named fields without needing to count positional
indices. This is something the Unix toolkit always could have done, and
arguably always should have done. It operates on key-value-pair data while the
familiar Unix tools operate on integer-indexed fields: if the natural data
structure for the latter is the array, then Miller’s natural data structure is
the insertion-ordered hash map. This encompasses a variety of data formats,
including but not limited to the familiar CSV. (Miller can handle
positionally-indexed data as a special case.)</simpara>
</refsect1>
<refsect1 id="_examples">
<title>EXAMPLES</title>
<literallayout class="monospaced">% mlr --csv cut -f hostname,uptime mydata.csv
% mlr --csv sort -f hostname,uptime mydata.csv
% mlr --csv put '$z = $x + 2.7*$y' mydata.csv
% mlr --csv filter '$status != "down"' mydata.csv</literallayout>
</refsect1>
<refsect1 id="_options">
<title>OPTIONS</title>
<simpara>In the following option flags, the version with "i" designates the input
stream, "o" the output stream, and the version without prefix sets the option
for both input and output stream. For example: <literal>--irs</literal> sets the input record
separator, <literal>--ors</literal> the output record separator, and <literal>--rs</literal> sets both the input
and output separator to the given value.</simpara>
<refsect2 id="_separator">
<title>SEPARATOR</title>
<variablelist>
<varlistentry>
<term>
--rs, --irs, --ors
</term>
<listitem>
<simpara>
    Record separators, defaulting to newline
</simpara>
</listitem>
</varlistentry>
<varlistentry>
<term>
--fs, --ifs, --ofs, --repifs
</term>
<listitem>
<simpara>
    Field  separators, defaulting to ","
</simpara>
</listitem>
</varlistentry>
<varlistentry>
<term>
--ps, --ips, --ops
</term>
<listitem>
<simpara>
    Pair   separators, defaulting to "="
</simpara>
</listitem>
</varlistentry>
</variablelist>
</refsect2>
<refsect2 id="_data_format">
<title>DATA-FORMAT</title>
<variablelist>
<varlistentry>
<term>
--dkvp, --idkvp, --odkvp
</term>
<listitem>
<simpara>
    Delimited key-value pairs, e.g "a=1,b=2" (default)
</simpara>
</listitem>
</varlistentry>
<varlistentry>
<term>
--nidx, --inidx, --onidx
</term>
<listitem>
<simpara>
    Implicitly-integer-indexed fields (Unix-toolkit style)
</simpara>
</listitem>
</varlistentry>
<varlistentry>
<term>
--csv, --icsv, --ocsv
</term>
<listitem>
<simpara>
    Comma-separated value (or tab-separated with --fs tab, etc.)
</simpara>
</listitem>
</varlistentry>
<varlistentry>
<term>
--pprint, --ipprint, --opprint, --right
</term>
<listitem>
<simpara>
    Pretty-printed tabular (produces no output until all input is in)
</simpara>
</listitem>
</varlistentry>
<varlistentry>
<term>
--pprint, --ipprint, --opprint, --right
</term>
<listitem>
<simpara>
    Pretty-printed tabular (produces no output until all input is in)
</simpara>
</listitem>
</varlistentry>
</variablelist>
<simpara><literal>-p</literal> is a keystroke-saver for <literal>--nidx --fs space --repifs</literal></simpara>
</refsect2>
<refsect2 id="_numerical_format">
<title>NUMERICAL FORMAT</title>
<variablelist>
<varlistentry>
<term>
</term>
<listitem>
<simpara>
    Sets the numerical format given a printf-style format string.
</simpara>
</listitem>
</varlistentry>
</variablelist>
</refsect2>
<refsect2 id="_other">
<title>OTHER</title>
<variablelist>
<varlistentry>
<term>
</term>
<listitem>
<simpara>
    Seeds the random number generator used for put/filter <literal>urand()</literal> with a
    number n of the form 12345678 or 0xcafefeed.
</simpara>
</listitem>
</varlistentry>
</variablelist>
</refsect2>
<refsect2 id="_verbs">
<title>VERBS</title>
<refsect3 id="_cat">
<title>cat</title>
<simpara>Usage: <literal>mlr cat</literal></simpara>
<simpara>Passes input records directly to output. Most useful for format conversion.</simpara>
</refsect3>
<refsect3 id="_check">
<title>check</title>
<simpara>Usage: <literal>mlr check</literal></simpara>
<simpara>Consumes records without printing any output. Useful for doing a
well-formatted check on input data.</simpara>
</refsect3>
<refsect3 id="_count_distinct">
<title>count-distinct</title>
<simpara>Usage: <literal>mlr count-distinct [options]</literal></simpara>
<simpara>Prints number of records having distinct values for specified field names.
Same as <literal>uniq -c</literal>.</simpara>
<variablelist>
<varlistentry>
<term>
-f {a,b,c}
</term>
<listitem>
<simpara>
    Field names for distinct count.
</simpara>
</listitem>
</varlistentry>
</variablelist>
</refsect3>
<refsect3 id="_cut">
<title>cut</title>
<simpara>Usage: <literal>mlr cut [options]</literal></simpara>
<simpara>Passes through input records with specified fields included/excluded.</simpara>
<variablelist>
<varlistentry>
<term>
-f {a,b,c}
</term>
<listitem>
<simpara>
    Field names to include for cut.
</simpara>
</listitem>
</varlistentry>
<varlistentry>
<term>
-o
</term>
<listitem>
<simpara>
    Retain fields in the order specified here in the argument list.
    Default is to retain them in the order found in the input data.
</simpara>
</listitem>
</varlistentry>
<varlistentry>
<term>
-x|--complement
</term>
<listitem>
<simpara>
    Exclude, rather that include, field names specified by -f.
</simpara>
</listitem>
</varlistentry>
</variablelist>
</refsect3>
<refsect3 id="_filter">
<title>filter</title>
<simpara></simpara>
<simpara>prints the AST (abstract syntax tree) for the expression, which gives full
transparency on the precedence and associativity rules of Miller&#8217;s grammar.
Please use a dollar sign for field names and double-quotes for string
literals.  Miller built-in variables are <literal>NF</literal>, <literal>NR</literal>, <literal>FNR</literal>, <literal>FILENUM</literal>,
<literal>FILENAME</literal>, <literal>PI</literal>, <literal>E</literal>.</simpara>
<simpara>Examples:</simpara>
<literallayout class="monospaced">mlr filter 'log10($count) &gt; 4.0'
mlr filter 'FNR == 2          (second record in each file)'
mlr filter 'urand() &lt; 0.001'  (subsampling)
mlr filter '$color != "blue" &amp;&amp; $value &gt; 4.2'
mlr filter '($x&lt;.5 &amp;&amp; $y&lt;.5) || ($x&gt;.5 &amp;&amp; $y&gt;.5)'</literallayout>
<simpara>Please see <ulink url="http://johnkerl.org/miller/doc/reference.html">http://johnkerl.org/miller/doc/reference.html</ulink> for more information including function list.</simpara>
</refsect3>
<refsect3 id="_group_by">
<title>group-by</title>
<simpara>Usage: <literal>mlr group-by {comma-separated field names}</literal></simpara>
<simpara>Outputs records in batches having identical values at specified field names.</simpara>
</refsect3>
<refsect3 id="_group_like">
<title>group-like</title>
<simpara>Usage: <literal>mlr group-like</literal></simpara>
<simpara>Outputs records in batches having identical field names.</simpara>
</refsect3>
<refsect3 id="_having_fields">
<title>having-fields</title>
<simpara>Usage: <literal>mlr having-fields [options]</literal></simpara>
<simpara>Conditionally passes through records depending on each record&#8217;s field names.</simpara>
<simpara>Options:</simpara>
<simpara>--at-least {a,b,c}
--which-are {a,b,c}
--at-most {a,b,c}</simpara>
</refsect3>
<refsect3 id="_head">
<title>head</title>
<simpara>Usage: <literal>mlr head [options]</literal></simpara>
<simpara>Passes through the first n records, optionally by category.</simpara>
<simpara>Options:</simpara>
<variablelist>
<varlistentry>
<term>
</term>
<listitem>
<simpara>
    Head count to print; default 10
</simpara>
</listitem>
</varlistentry>
<varlistentry>
<term>
-g {a,b,c}
</term>
<listitem>
<simpara>
    Optional group-by-field names for head counts
</simpara>
</listitem>
</varlistentry>
</variablelist>
</refsect3>
<refsect3 id="_histogram">
<title>histogram</title>
<simpara>Usage: <literal>mlr histogram [options]</literal></simpara>
<simpara>Just a histogram. Input values &lt; lo or &gt; hi are not counted.</simpara>
<simpara>Options:</simpara>
<variablelist>
<varlistentry>
<term>
-f {a,b,c}
</term>
<listitem>
<simpara>
    Value-field names for histogram counts
</simpara>
</listitem>
</varlistentry>
<varlistentry>
<term>
</term>
<listitem>
<simpara>
    Histogram low value
</simpara>
</listitem>
</varlistentry>
<varlistentry>
<term>
</term>
<listitem>
<simpara>
    Histogram high value
</simpara>
</listitem>
</varlistentry>
<varlistentry>
<term>
</term>
<listitem>
<simpara>
    Number of histogram bins
</simpara>
</listitem>
</varlistentry>
</variablelist>
</refsect3>
<refsect3 id="_join">
<title>join</title>
<simpara>Usage: <literal>mlr join [options]</literal></simpara>
<simpara>Joins records from specified left file name with records from all file names
at the end of the Miller argument list.  Functionality is essentially the same
as the system "join" command, but for record streams.</simpara>
<simpara>Options:</simpara>
<literallayout class="monospaced">-f {left file name}
-j {a,b,c}   Comma-separated join-field names for output
-l {a,b,c}   Comma-separated join-field names for left input file; defaults to -j values if omitted.
-r {a,b,c}   Comma-separated join-field names for right input file(s); defaults to -j values if omitted.
--lp {text}  Additional prefix for non-join output field names from the left file
--rp {text}  Additional prefix for non-join output field names from the right file(s)
--np         Do not emit paired records
--ul         Emit unpaired records from the left file
--ur         Emit unpaired records from the right file(s)
-u           Enable unsorted input. In this case, the entire left file will be loaded into memory.
             Without -u, records must be sorted lexically by their join-field names, else not all
             records will be paired.</literallayout>
<simpara>File-format options default to those for the right file names on the Miller
argument list, but may be overridden for the left file as follows. Please see
the main "mlr --help" for more information on syntax for these arguments.</simpara>
<literallayout class="monospaced">-i {one of csv,dkvp,nidx,pprint,xtab}
--irs {record-separator character}
--ifs {field-separator character}
--ips {pair-separator character}
--repifs
--repips
--use-mmap
--no-mmap</literallayout>
<simpara>Please see <ulink url="http://johnkerl.org/miller/doc/reference.html">http://johnkerl.org/miller/doc/reference.html</ulink> for more information
including examples.</simpara>
</refsect3>
<refsect3 id="_label">
<title>label</title>
<simpara>Usage: <literal>mlr label {new1,new2,new3,&#8230;}</literal></simpara>
<simpara>Given n comma-separated names, renames the first n fields of each record to
have the respective name. (Fields past the nth are left with their original
names.) Particularly useful with --inidx, to give useful names to otherwise
integer-indexed fields.</simpara>
</refsect3>
<refsect3 id="_put">
<title>put</title>
<simpara></simpara>
<simpara>Adds/updates specified field(s).</simpara>
<simpara>With <literal>-v</literal>, first prints the AST (abstract syntax tree) for the expression,
which gives full transparency on the precedence and associativity rules of
Miller&#8217;s grammar.  Please use a dollar sign for field names and double-quotes
for string literals.  Miller built-in variables are <literal>NF</literal>, <literal>NR</literal>, <literal>FNR</literal>,
<literal>FILENUM</literal>, <literal>FILENAME</literal>, <literal>PI</literal>, <literal>E</literal>.  Multiple assignments may be separated with
a semicolon.</simpara>
<simpara>Examples:</simpara>
<literallayout class="monospaced">mlr put '$y = log10($x); $z = sqrt($y)'
mlr put '$filename = FILENAME'
mlr put '$colored_shape = $color . "_" . $shape'
mlr put '$y = cos($theta); $z = atan2($y, $x)'</literallayout>
<simpara>Please see <ulink url="http://johnkerl.org/miller/doc/reference.html">http://johnkerl.org/miller/doc/reference.html</ulink> for more information
including function list.</simpara>
</refsect3>
<refsect3 id="_regularize">
<title>regularize</title>
<simpara>Usage: <literal>mlr regularize</literal></simpara>
<simpara>For records seen earlier in the data stream with same field names in a different order,
outputs them with field names in the previously encountered order.</simpara>
<simpara>Example:</simpara>
<simpara>input records          a=1,c=2,b=3, then e=4,d=5, then c=7,a=6,b=8
output as              a=1,c=2,b=3, then e=4,d=5, then a=6,c=7,b=8</simpara>
</refsect3>
<refsect3 id="_rename">
<title>rename</title>
<simpara>Usage: <literal>mlr rename {old1,new1,old2,new2,&#8230;}</literal></simpara>
<simpara>Renames specified fields.</simpara>
</refsect3>
<refsect3 id="_reorder">
<title>reorder</title>
<simpara>Usage: <literal>mlr reorder [options]</literal></simpara>
<simpara>Options:</simpara>
<variablelist>
<varlistentry>
<term>
-f {a,b,c}
</term>
<listitem>
<simpara>
    Field names to reorder.
</simpara>
</listitem>
</varlistentry>
<varlistentry>
<term>
-e
</term>
<listitem>
<simpara>
    Put specified field names at record end: default is to put at record start.
</simpara>
</listitem>
</varlistentry>
</variablelist>
<simpara>Examples:</simpara>
<literallayout class="monospaced">mlr reorder    -f a,b sends input record "d=4,b=2,a=1,c=3" to "a=1,b=2,d=4,c=3".
mlr reorder -e -f a,b sends input record "d=4,b=2,a=1,c=3" to "d=4,c=3,a=1,b=2".</literallayout>
</refsect3>
<refsect3 id="_sort">
<title>sort</title>
<simpara></simpara>
<simpara>Sorts records primarily by the first specified field, secondarily by the
second field, and so on.</simpara>
<simpara>Flags:</simpara>
<variablelist>
<varlistentry>
<term>
-f  {comma-separated field names}
</term>
<listitem>
<simpara>
    Lexical ascending
</simpara>
</listitem>
</varlistentry>
<varlistentry>
<term>
-n  {comma-separated field names}
</term>
<listitem>
<simpara>
    Numerical ascending; nulls sort last
</simpara>
</listitem>
</varlistentry>
<varlistentry>
<term>
-nf {comma-separated field names}
</term>
<listitem>
<simpara>
    Numerical ascending; nulls sort last
</simpara>
</listitem>
</varlistentry>
<varlistentry>
<term>
-r  {comma-separated field names}
</term>
<listitem>
<simpara>
    Lexical descending
</simpara>
</listitem>
</varlistentry>
<varlistentry>
<term>
-nr {comma-separated field names}
</term>
<listitem>
<simpara>
    Numerical descending; nulls sort first
</simpara>
</listitem>
</varlistentry>
</variablelist>
<simpara>Example:</simpara>
<literallayout class="monospaced">mlr sort -f a,b -nr x,y,z</literallayout>
<simpara>which is the same as:</simpara>
<literallayout class="monospaced">mlr sort -f a -f b -nr x -nr y -nr z</literallayout>
</refsect3>
<refsect3 id="_stats1">
<title>stats1</title>
<simpara>Usage: <literal>mlr stats1 [options]</literal></simpara>
<variablelist>
<varlistentry>
<term>
-a {sum,count,&#8230;}
</term>
<listitem>
<simpara>
    Names of accumulators: <literal>p10</literal>, <literal>p25.2</literal>, <literal>p50</literal>, <literal>p98</literal>, <literal>p100</literal>, etc. and/or
    one or more of: <literal>count</literal>, <literal>mode</literal>, <literal>sum</literal>, <literal>mean</literal>, <literal>stddev</literal>, <literal>var</literal>, <literal>meaneb</literal>,
    <literal>min</literal>, <literal>max</literal>.
</simpara>
</listitem>
</varlistentry>
<varlistentry>
<term>
-f {a,b,c}
</term>
<listitem>
<simpara>
    Value-field names on which to compute statistics
</simpara>
</listitem>
</varlistentry>
<varlistentry>
<term>
-g {d,e,f}
</term>
<listitem>
<simpara>
    Optional group-by-field names
</simpara>
</listitem>
</varlistentry>
</variablelist>
<simpara>Examples:</simpara>
<literallayout class="monospaced">mlr stats1 -a min,p10,p50,p90,max -f value -g size,shape
mlr stats1 -a count,mode -f size
mlr stats1 -a count,mode -f size -g shape</literallayout>
<simpara>Notes:</simpara>
<itemizedlist>
<listitem>
<simpara>
p50 is a synonym for median.
</simpara>
</listitem>
<listitem>
<simpara>
min and max output the same results as p0 and p100, respectively, but use less memory.
</simpara>
</listitem>
<listitem>
<simpara>
count and mode allow text input; the rest require numeric input. In particular, 1 and 1.0
  are distinct text for count and mode.
</simpara>
</listitem>
<listitem>
<simpara>
When there are mode ties, the first-encountered datum wins.
</simpara>
</listitem>
</itemizedlist>
</refsect3>
<refsect3 id="_stats2">
<title>stats2</title>
<simpara>Usage: <literal>mlr stats2 [options]</literal></simpara>
<variablelist>
<varlistentry>
<term>
-a {linreg-ols,corr,&#8230;}
</term>
<listitem>
<simpara>
    Names of accumulators: one or more of <literal>linreg-pca</literal>, <literal>linreg-ols</literal>, <literal>r2</literal>,
    <literal>corr</literal>, <literal>cov</literal>, <literal>covx</literal>. <literal>r2</literal> is a quality metric for <literal>linreg-ols</literal>;
    <literal>linrec-pca</literal> outputs its own quality metric.
</simpara>
</listitem>
</varlistentry>
<varlistentry>
<term>
-f {a,b,c,d}
</term>
<listitem>
<simpara>
    Value-field name-pairs on which to compute statistics. There must be an
    even number of names.
</simpara>
</listitem>
</varlistentry>
<varlistentry>
<term>
-g {e,f,g}
</term>
<listitem>
<simpara>
    Optional group-by-field names.
</simpara>
</listitem>
</varlistentry>
<varlistentry>
<term>
-v
</term>
<listitem>
<simpara>
    Print additional output for <literal>linreg-pca</literal>.
</simpara>
</listitem>
</varlistentry>
</variablelist>
<simpara>Examples:</simpara>
<literallayout class="monospaced">mlr stats2 -a linreg-pca -f x,y
mlr stats2 -a linreg-ols,r2 -f x,y -g size,shape
mlr stats2 -a corr -f x,y</literallayout>
</refsect3>
<refsect3 id="_step">
<title>step</title>
<simpara>Usage: <literal>mlr step [options]</literal></simpara>
<simpara>Computes values dependent on the previous record, optionally grouped by
category.</simpara>
<variablelist>
<varlistentry>
<term>
-a {delta,rsum,&#8230;}
</term>
<listitem>
<simpara>
    Names of steppers: one or more of delta rsum counter
</simpara>
</listitem>
</varlistentry>
<varlistentry>
<term>
-f {a,b,c}
</term>
<listitem>
<simpara>
    Value-field names on which to compute statistics
</simpara>
</listitem>
</varlistentry>
<varlistentry>
<term>
-g {d,e,f}
</term>
<listitem>
<simpara>
    Group-by-field names
</simpara>
</listitem>
</varlistentry>
</variablelist>
</refsect3>
<refsect3 id="_tac">
<title>tac</title>
<simpara>Usage: <literal>mlr tac</literal></simpara>
<simpara>Prints records in reverse order from the order in which they were encountered.</simpara>
</refsect3>
<refsect3 id="_tail">
<title>tail</title>
<simpara>Usage: <literal>mlr tail [options]</literal></simpara>
<simpara>Passes through the last n records, optionally by category.</simpara>
<variablelist>
<varlistentry>
<term>
</term>
<listitem>
<simpara>
    Tail count to print; default 10
</simpara>
</listitem>
</varlistentry>
<varlistentry>
<term>
-g {a,b,c}
</term>
<listitem>
<simpara>
    Optional group-by-field names for tail counts
</simpara>
</listitem>
</varlistentry>
</variablelist>
</refsect3>
<refsect3 id="_top">
<title>top</title>
<simpara>Usage: <literal>mlr top [options]</literal></simpara>
<simpara>Prints the n records with smallest/largest values at specified fields,
optionally by category.</simpara>
<variablelist>
<varlistentry>
<term>
-f {a,b,c}
</term>
<listitem>
<simpara>
    Value-field names for top counts
</simpara>
</listitem>
</varlistentry>
<varlistentry>
<term>
-g {d,e,f}
</term>
<listitem>
<simpara>
    Optional group-by-field names for top counts
</simpara>
</listitem>
</varlistentry>
<varlistentry>
<term>
</term>
<listitem>
<simpara>
    How many records to print per category; default 1
</simpara>
</listitem>
</varlistentry>
<varlistentry>
<term>
-a
</term>
<listitem>
<simpara>
    Print all fields for top-value records; default is to print only
    value and group-by fields.
</simpara>
</listitem>
</varlistentry>
<varlistentry>
<term>
--min
</term>
<listitem>
<simpara>
    Print top smallest values; default is top largest values
</simpara>
</listitem>
</varlistentry>
</variablelist>
</refsect3>
<refsect3 id="_uniq">
<title>uniq</title>
<simpara>Usage: <literal>mlr uniq [options]</literal></simpara>
<simpara>Prints distinct values for specified field names. With -c, same as
count-distinct.</simpara>
<variablelist>
<varlistentry>
<term>
-g {d,e,f}
</term>
<listitem>
<simpara>
    Group-by-field names for uniq counts
</simpara>
</listitem>
</varlistentry>
<varlistentry>
<term>
-c
</term>
<listitem>
<simpara>
    Show repeat counts in addition to unique values
</simpara>
</listitem>
</varlistentry>
</variablelist>
</refsect3>
</refsect2>
</refsect1>
<refsect1 id="_author">
<title>AUTHOR</title>
<simpara>miller is written by John Kerl &lt;<ulink url="mailto:kerl.john.r@gmail.com">kerl.john.r@gmail.com</ulink>&gt;.</simpara>
<simpara>This manual page has been composed from miller&#8217;s help output by Eric MSP Veith
&lt;<ulink url="mailto:eveith@veith-m.de">eveith@veith-m.de</ulink>&gt;.</simpara>
</refsect1>
<refsect1 id="_see_also">
<title>SEE ALSO</title>
<simpara>sed(1), awk(1), cut(1), join(1), sort(1), RFC 4180: Common Format and MIME
Type for Comma-Separated Values (CSV) Files, the miller website
<ulink url="http://johnkerl.org/miller/doc">http://johnkerl.org/miller/doc</ulink></simpara>
</refsect1>
</refentry>