mirror of
https://github.com/johnkerl/miller.git
synced 2026-07-21 02:23:20 +00:00
78 lines
No EOL
10 KiB
HTML
78 lines
No EOL
10 KiB
HTML
|
||
<!DOCTYPE html>
|
||
|
||
<html>
|
||
<head>
|
||
<meta charset="utf-8" />
|
||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||
<title>How original is Miller? — Miller 6.0.0-alpha documentation</title>
|
||
|
||
<link rel="stylesheet" href="_static/scrolls.css" type="text/css" />
|
||
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
|
||
<link rel="stylesheet" href="_static/print.css" type="text/css" />
|
||
|
||
<script id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script>
|
||
<script src="_static/jquery.js"></script>
|
||
<script src="_static/underscore.js"></script>
|
||
<script src="_static/doctools.js"></script>
|
||
<script src="_static/language_data.js"></script>
|
||
<script src="_static/theme_extras.js"></script>
|
||
<link rel="index" title="Index" href="genindex.html" />
|
||
<link rel="search" title="Search" href="search.html" />
|
||
<link rel="next" title="Performance" href="performance.html" />
|
||
<link rel="prev" title="Why call it Miller?" href="etymology.html" />
|
||
</head><body>
|
||
<div id="content">
|
||
<div class="header">
|
||
<h1 class="heading"><a href="index.html"
|
||
title="back to the documentation overview"><span>How original is Miller?</span></a></h1>
|
||
</div>
|
||
<div class="relnav" role="navigation" aria-label="related navigation">
|
||
<a href="etymology.html">« Why call it Miller?</a> |
|
||
<a href="#">How original is Miller?</a>
|
||
| <a href="performance.html">Performance »</a>
|
||
</div>
|
||
<div id="contentwrapper">
|
||
<div role="main">
|
||
|
||
<div class="section" id="how-original-is-miller">
|
||
<h1>How original is Miller?<a class="headerlink" href="#how-original-is-miller" title="Permalink to this headline">¶</a></h1>
|
||
<p>It isn’t. Miller is one of many, many participants in the online-analytical-processing culture. Other key participants include <code class="docutils literal notranslate"><span class="pre">awk</span></code>, SQL, spreadsheets, etc. etc. etc. Far from being an original concept, Miller explicitly strives to imitate several existing tools:</p>
|
||
<p><strong>The Unix toolkit</strong>: Intentional similarities as described in <a class="reference internal" href="feature-comparison.html"><span class="doc">Unix-toolkit context</span></a>.</p>
|
||
<p>Recipes abound for command-line data analysis using the Unix toolkit. Here are just a couple of my favorites:</p>
|
||
<ul class="simple">
|
||
<li><p><a class="reference external" href="http://en.wikibooks.org/wiki/Ad_Hoc_Data_Analysis_From_The_Unix_Command_Line">http://en.wikibooks.org/wiki/Ad_Hoc_Data_Analysis_From_The_Unix_Command_Line</a></p></li>
|
||
<li><p><a class="reference external" href="http://www.gregreda.com/2013/07/15/unix-commands-for-data-science">http://www.gregreda.com/2013/07/15/unix-commands-for-data-science</a></p></li>
|
||
<li><p><a class="reference external" href="https://github.com/dbohdan/structured-text-tools">https://github.com/dbohdan/structured-text-tools</a></p></li>
|
||
</ul>
|
||
<p><strong>RecordStream</strong>: Miller owes particular inspiration to <a class="reference external" href="https://github.com/benbernard/RecordStream">RecordStream</a>. The key difference is that RecordStream is a Perl-based tool for manipulating JSON (including requiring it to separately manipulate other formats such as CSV into and out of JSON), while Miller is fast Go which handles its formats natively. The similarities include the <code class="docutils literal notranslate"><span class="pre">sort</span></code>, <code class="docutils literal notranslate"><span class="pre">stats1</span></code> (analog of RecordStream’s <code class="docutils literal notranslate"><span class="pre">collate</span></code>), and <code class="docutils literal notranslate"><span class="pre">delta</span></code> operations, as well as <code class="docutils literal notranslate"><span class="pre">filter</span></code> and <code class="docutils literal notranslate"><span class="pre">put</span></code>, and pretty-print formatting.</p>
|
||
<p><strong>stats_m</strong>: A third source of lineage is my Python <a class="reference external" href="https://github.com/johnkerl/scripts-math/tree/master/stats">stats_m</a> module. This includes simple single-pass algorithms which form Miller’s <code class="docutils literal notranslate"><span class="pre">stats1</span></code> and <code class="docutils literal notranslate"><span class="pre">stats2</span></code> subcommands.</p>
|
||
<p><strong>SQL</strong>: Fourthly, Miller’s <code class="docutils literal notranslate"><span class="pre">group-by</span></code> command name is from SQL, as is the term <code class="docutils literal notranslate"><span class="pre">aggregate</span></code>.</p>
|
||
<p><strong>Added value</strong>: Miller’s added values include:</p>
|
||
<ul class="simple">
|
||
<li><p>Name-indexing, compared to the Unix toolkit’s positional indexing.</p></li>
|
||
<li><p>Raw speed, compared to <code class="docutils literal notranslate"><span class="pre">awk</span></code>, RecordStream, <code class="docutils literal notranslate"><span class="pre">stats_m</span></code>, or various other kinds of Python/Ruby/etc. scripts one can easily create.</p></li>
|
||
<li><p>Compact keystroking for many common tasks, with a decent amount of flexibility.</p></li>
|
||
<li><p>Ability to handle text files on the Unix pipe, without need for creating database tables, compared to SQL databases.</p></li>
|
||
<li><p>Various file formats, and on-the-fly format conversion.</p></li>
|
||
</ul>
|
||
<p><strong>jq</strong>: Miller does for name-indexed text what <a class="reference external" href="https://stedolan.github.io/jq/">jq</a> does for JSON. If you’re not already familiar with <code class="docutils literal notranslate"><span class="pre">jq</span></code>, please check it out!.</p>
|
||
<p><strong>What about similar tools?</strong></p>
|
||
<p>Here’s a comprehensive list: <a class="reference external" href="https://github.com/dbohdan/structured-text-tools">https://github.com/dbohdan/structured-text-tools</a>. Last I knew it doesn’t mention <a class="reference external" href="https://github.com/turicas/rows">rows</a> so here’s a plug for that as well. As it turns out, I learned about most of these after writing Miller.</p>
|
||
<p><strong>What about DOTADIW?</strong> One of the key points of the <a class="reference external" href="http://en.wikipedia.org/wiki/Unix_philosophy">Unix philosophy</a> is that a tool should do one thing and do it well. Hence <code class="docutils literal notranslate"><span class="pre">sort</span></code> and <code class="docutils literal notranslate"><span class="pre">cut</span></code> do just one thing. Why does Miller put <code class="docutils literal notranslate"><span class="pre">awk</span></code>-like processing, a few SQL-like operations, and statistical reduction all into one tool? This is a fair question. First note that many standard tools, such as <code class="docutils literal notranslate"><span class="pre">awk</span></code> and <code class="docutils literal notranslate"><span class="pre">perl</span></code>, do quite a few things – as does <code class="docutils literal notranslate"><span class="pre">jq</span></code>. But I could have pushed for putting format awareness and name-indexing options into <code class="docutils literal notranslate"><span class="pre">cut</span></code>, <code class="docutils literal notranslate"><span class="pre">awk</span></code>, and so on (so you could do <code class="docutils literal notranslate"><span class="pre">cut</span> <span class="pre">-f</span> <span class="pre">hostname,uptime</span></code> or <code class="docutils literal notranslate"><span class="pre">awk</span> <span class="pre">'{sum</span> <span class="pre">+=</span> <span class="pre">$x*$y}END{print</span> <span class="pre">sum}'</span></code>). Patching <code class="docutils literal notranslate"><span class="pre">cut</span></code>, <code class="docutils literal notranslate"><span class="pre">sort</span></code>, etc. on multiple operating systems is a non-starter in terms of uptake. Moreover, it makes sense for me to have Miller be a tool which collects together format-aware record-stream processing into one place, with good reuse of Miller-internal library code for its various features.</p>
|
||
<p><strong>Why not use Perl/Python/Ruby etc.?</strong> Maybe you should. With those tools you’ll get far more expressive power, and sufficiently quick turnaround time for small-to-medium-sized data. Using Miller you’ll get something less than a complete programming language, but which is fast, with moderate amounts of flexibility and much less keystroking.</p>
|
||
<p>When I was first developing Miller I made a survey of several languages. Using low-level implementation languages like C, Go, Rust, and Nim, I’d need to create my own domain-specific language (DSL) which would always be less featured than a full programming language, but I’d get better performance. Using high-level interpreted languages such as Perl/Python/Ruby I’d get the language’s <code class="docutils literal notranslate"><span class="pre">eval</span></code> for free and I wouldn’t need a DSL; Miller would have mainly been a set of format-specific I/O hooks. If I’d gotten good enough performance from the latter I’d have done it without question and Miller would be far more flexible. But high-level languages win the performance criteria by a landslide so we have Miller in Go with a custom DSL.</p>
|
||
<p><strong>No, really, why one more command-line data-manipulation tool?</strong> I wrote Miller because I was frustrated with tools like <code class="docutils literal notranslate"><span class="pre">grep</span></code>, <code class="docutils literal notranslate"><span class="pre">sed</span></code>, and so on being <em>line-aware</em> without being <em>format-aware</em>. The single most poignant example I can think of is seeing people grep data lines out of their CSV files and sadly losing their header lines. While some lighter-than-SQL processing is very nice to have, at core I wanted the format-awareness of <a class="reference external" href="https://github.com/benbernard/RecordStream">RecordStream</a> combined with the raw speed of the Unix toolkit. Miller does precisely that.</p>
|
||
</div>
|
||
|
||
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="footer" role="contentinfo">
|
||
© Copyright 2021, John Kerl.
|
||
Created using <a href="https://www.sphinx-doc.org/">Sphinx</a> 3.2.1.
|
||
</div>
|
||
</body>
|
||
</html> |