miller/doc/content-for-index.html

<p/>Miller is like awk, sed, cut, join, and sort for <b>name-indexed data such as
CSV, TSV, and tabular JSON</b>. You get to work with your data using named
fields, without needing to count positional column indices.

<p/>This is something the Unix toolkit always could have done, and arguably
always should have done.  It operates on key-value-pair data while the familiar
Unix tools operate on integer-indexed fields: if the natural data structure for
the latter is the array, then Miller&rsquo;s natural data structure is the
insertion-ordered hash map.  This encompasses a <b>variety of data formats</b>,
including but not limited to the familiar CSV, TSV, and JSON.  (Miller can handle
<b>positionally-indexed data</b> as a special case.)

<p/> Features:
<ul>

<li/> Miller is <b>multi-purpose</b>: it&rsquo;s useful for <b>data
cleaning</b>, <b>data reduction</b>, <b>statistical reporting</b>,
<b>devops</b>, <b>system administration</b>, <b>log-file processing</b>,
<b>format conversion</b>, and <b>database-query post-processing</b>.

<li/> You can use Miller to snarf and munge <b>log-file data</b>, including
selecting out relevant substreams, then produce CSV format and load that into
all-in-memory/data-frame utilities for further statistical and/or graphical
processing.

<li/> Miller complements <b>data-analysis tools</b> such as <b>R</b>,
<b>pandas</b>, etc.: you can use Miller to <b>clean</b> and <b>prepare</b> your
data. While you can do <b>basic statistics</b> entirely in Miller, its
streaming-data feature and single-pass algorithms enable you to <b>reduce very
large data sets</b>.

<li/> Miller complements SQL <b>databases</b>: you can slice, dice, and
reformat data on the client side on its way into or out of a database. You can
also reap some of the benefits of databases for quick, setup-free one-off tasks
when you just need to query some data in disk files in a hurry.

<li/> Miller also goes beyond the classic Unix tools by stepping fully into our
modern, <b>no-SQL</b> world: its essential record-heterogeneity property allows
Miller to operate on data where records with different schema (field names) are
interleaved.

<li/> Miller is <b>streaming</b>: most operations need only a single record in
memory at a time, rather than ingesting all input before producing any output.
For those operations which require deeper retention (<tt>sort</tt>,
<tt>tac</tt>, <tt>stats1</tt>), Miller retains only as much data as needed.
This means that whenever functionally possible, you can operate on files which
are larger than your system&rsquo;s available RAM, and you can use Miller in
<b>tail -f</b> contexts.

<li/> Miller is <b>pipe-friendly</b> and interoperates with the Unix toolkit

<li/> Miller&rsquo;s I/O formats include <b>tabular pretty-printing</b>,
<b>positionally indexed</b> (Unix-toolkit style), CSV, JSON, and others

<li/> Miller does <b>conversion</b> between formats

<li/> Miller&rsquo;s <b>processing is format-aware</b>: e.g. CSV <tt>sort</tt>
and <tt>tac</tt> keep header lines first

<li/> Miller has high-throughput <b>performance</b> on par with the Unix toolkit

<li/> Not unlike <a href="http://stedolan.github.io/jq/">jq</a> (for JSON),
Miller is written in portable, modern C, with <b>zero runtime dependencies</b>.
You can download or compile a single binary, <tt>scp</tt> it to a faraway
machine, and expect it to work.

</ul>

<p>Releases and release notes:
<a href="https://github.com/johnkerl/miller/releases">https://github.com/johnkerl/miller/releases</a>.

<p/> Examples:

<div class="pokipanel">
<pre>
# Column select
% mlr --csv cut -f hostname,uptime mydata.csv
</pre>
</div>

<div class="pokipanel">
<pre>
# Add new columns as function of other columns
% mlr --nidx put '$sum = $7 < 0.0 ? 3.5 : $7 + 2.1*$8' *.dat
</pre>
</div>

<div class="pokipanel">
<pre>
# Row filter
% mlr --csv filter '$status != "down" && $upsec >= 10000' *.csv
</pre>
</div>

<div class="pokipanel">
<pre>
# Apply column labels and pretty-print
% grep -v '^#' /etc/group | mlr --ifs : --nidx --opprint label group,pass,gid,member then sort -f group
</pre>
</div>

<div class="pokipanel">
<pre>
# Join multiple data sources on key columns
% mlr join -j account_id -f accounts.dat then group-by account_name balances.dat
</pre>
</div>

<div class="pokipanel">
<pre>
# Multiple formats including JSON
% mlr --json put '$attr = sub($attr, "([0-9]+)_([0-9]+)_.*", "\1:\2")' data/*.json
</pre>
</div>

<div class="pokipanel">
<pre>
# Aggregate per-column statistics
% mlr stats1 -a min,mean,max,p10,p50,p90 -f flag,u,v data/*
</pre>
</div>

<div class="pokipanel">
<pre>
# Linear regression
% mlr stats2 -a linreg-pca -f u,v -g shape data/*
</pre>
</div>

<div class="pokipanel">
<pre>
# Aggregate custom per-column statistics
% mlr put -q '@sum[$a][$b] += $x; end {emit @sum, "a", "b"}' data/*
</pre>
</div>

<div class="pokipanel">
<pre>
# Iterate over data using DSL expressions
% mlr --from estimates.tbl put '
  for (k,v in $*) {
    if (is_numeric(v) && k =~ "^[t-z].*$") {
      $sum += v; $count += 1
    }
  }
  $mean = $sum / $count # no assignment if count unset
'
</pre>
</div>

<div class="pokipanel">
<pre>
# Run DSL expressions from a script file
% mlr --from infile.dat put -f analyze.mlr
</pre>
</div>

<div class="pokipanel">
<pre>
# Split/reduce output to multiple filenames
% mlr --from infile.dat put 'tee > "./taps/data-".$a."-".$b, $*'
</pre>
</div>

<div class="pokipanel">
<pre>
# Compressed I/O
% mlr --from infile.dat put 'tee | "gzip > ./taps/data-".$a."-".$b.".gz", $*'
</pre>
</div>

<div class="pokipanel">
<pre>
# Interoperate with other data-processing tools using standard pipes
% mlr --from infile.dat put -q '@v=$*; dump | "jq .[]"'
</pre>
</div>

<div class="pokipanel">
<pre>
# Tap/trace
% mlr --from infile.dat put  '(NR % 1000 == 0) { print > stderr, "Checkpoint ".NR}'
</pre>
</div>