mirror of
https://github.com/johnkerl/miller.git
synced 2026-07-21 10:29:02 +00:00
78 lines
3.6 KiB
HTML
78 lines
3.6 KiB
HTML
<center><table border=1 width="60%"><tr><td><center>
|
|
<b>
|
|
Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV,
|
|
TSV, and tabular JSON.
|
|
</b>
|
|
</center> </td></tr></table></center>
|
|
|
|
<p/> With Miller you get to use named fields without needing to count
|
|
positional indices. For example:
|
|
|
|
POKI_INCLUDE_ESCAPED(index-snippet.txt)HERE
|
|
|
|
<p/>This is something the Unix toolkit always could have done, and arguably
|
|
always should have done. It operates on key-value-pair data while the familiar
|
|
Unix tools operate on integer-indexed fields: if the natural data structure for
|
|
the latter is the array, then Miller’s natural data structure is the
|
|
insertion-ordered hash map. This encompasses a <b>variety of data formats</b>,
|
|
including but not limited to the familiar CSV, TSV, and JSON. (Miller can handle
|
|
<b>positionally-indexed data</b> as a special case.)
|
|
|
|
<p/> Features:
|
|
<ul>
|
|
|
|
<li/> Miller is <b>multi-purpose</b>: it’s useful for <b>data
|
|
cleaning</b>, <b>data reduction</b>, <b>statistical reporting</b>,
|
|
<b>devops</b>, <b>system administration</b>, <b>log-file processing</b>,
|
|
<b>format conversion</b>, and <b>database-query post-processing</b>.
|
|
|
|
<li/> You can use Miller to snarf and munge <b>log-file data</b>, including
|
|
selecting out relevant substreams, then produce CSV format and load that into
|
|
all-in-memory/data-frame utilities for further statistical and/or graphical
|
|
processing.
|
|
|
|
<li/> Miller complements <b>data-analysis tools</b> such as <b>R</b>,
|
|
<b>pandas</b>, etc.: you can use Miller to <b>clean</b> and <b>prepare</b> your
|
|
data. While you can do <b>basic statistics</b> entirely in Miller, its
|
|
streaming-data feature and single-pass algorithms enable you to <b>reduce very
|
|
large data sets</b>.
|
|
|
|
<li/> Miller complements SQL <b>databases</b>: you can slice, dice, and
|
|
reformat data on the client side on its way into or out of a database. You can
|
|
also reap some of the benefits of databases for quick, setup-free one-off tasks
|
|
when you just need to query some data in disk files in a hurry.
|
|
|
|
<li/> Miller also goes beyond the classic Unix tools by stepping fully into our
|
|
modern, <b>no-SQL</b> world: its essential record-heterogeneity property allows
|
|
Miller to operate on data where records with different schema (field names) are
|
|
interleaved.
|
|
|
|
<li/> Miller is <b>streaming</b>: most operations need only a single record in
|
|
memory at a time, rather than ingesting all input before producing any output.
|
|
For those operations which require deeper retention (<tt>sort</tt>,
|
|
<tt>tac</tt>, <tt>stats1</tt>), Miller retains only as much data as needed.
|
|
This means that whenever functionally possible, you can operate on files which
|
|
are larger than your system’s available RAM, and you can use Miller in
|
|
<b>tail -f</b> contexts.
|
|
|
|
<li/> Miller is <b>pipe-friendly</b> and interoperates with the Unix toolkit
|
|
|
|
<li/> Miller’s I/O formats include <b>tabular pretty-printing</b>,
|
|
<b>positionally indexed</b> (Unix-toolkit style), CSV, JSON, and others
|
|
|
|
<li/> Miller does <b>conversion</b> between formats
|
|
|
|
<li/> Miller’s <b>processing is format-aware</b>: e.g. CSV <tt>sort</tt>
|
|
and <tt>tac</tt> keep header lines first
|
|
|
|
<li/> Miller has high-throughput <b>performance</b> on par with the Unix toolkit
|
|
|
|
<li/> Not unlike <a href="http://stedolan.github.io/jq/">jq</a> (for JSON),
|
|
Miller is written in portable, modern C, with <b>zero runtime dependencies</b>.
|
|
You can download or compile a single binary, <tt>scp</tt> it to a faraway
|
|
machine, and expect it to work.
|
|
|
|
</ul>
|
|
|
|
<p>Releases and release notes:
|
|
<a href="https://github.com/johnkerl/miller/releases">https://github.com/johnkerl/miller/releases</a>.
|