Mirrors/miller
1
0
Fork 0
mirror of https://github.com/johnkerl/miller.git synced 2026-07-17 16:38:54 +00:00
Code Issues Projects Releases Wiki Activity

doc iterate

Browse source
This commit is contained in:
John Kerl 2017-02-16 22:51:13 -05:00
parent d4c6c23e97
commit 45603c48e7
16 changed files with 987 additions and 45 deletions
Download patch file Download diff file Expand all files Collapse all files

169
doc/content-for-why.html Normal file
View file

@ -0,0 +1,169 @@
POKI_PUT_TOC_HERE
<p/> Someone asked me the other day about design, tradeoffs, thought process,
why I felt it necessary to build Miller, etc. Here are some answers.
<h1>Who is Miller for?</h1>
<p/> For background, I&rsquo;m a software engineer, with a heavy devops bent
and a non-trivial amount of data-engineering in my career.
<boldmaroon>Initially I wrote Miller mainly for myself:</boldmaroon> I&rsquo;m
coder-friendly (being a coder); I&rsquo;m Github-friendly; most of my data are
well-structured or easily structurable (TSV-formatted SQL-query output, CSV
files, log files, JSON data structures); I care about interoperability between
all the various formats Miller supports (I&rsquo;ve encountered them all); I do
all my work on Linux or OSX.
<p/> But now there&rsquo;s this neat little tool <boldmaroon>which seems to be
useful for people in various disciplines</boldmaroon>. I don&rsquo;t even know
entirely <i>who</i>. I can click through Github starrers and read a bit about
what they seem to do, but not everyone&rsquo;s <i>on</i> Github (or stars
things). I&rsquo;ve gotten a lot of feature requests through Github &mdash; but
only from people who are Github users. For sure, not everyone&rsquo;s on Linux
or OSX (I have a Windows port underway). Not everyone&rsquo;s a coder (it seems
like a lot of Miller&rsquo;s Github starrers are devops folks like myself, or
data-science-ish people, or biology/genomics folks.) A lot of people care 100%
about CSV. And so on.
<p/> So I wonder (please drop a note at <a
href="https://github.com/johnkerl/miller/issues">https://github.com/johnkerl/miller/issues</a>)
does Miller do what you need? Do you use it for all sorts of things, or just
one or two nice things? Are there things you wish it did but it doesn&rsquo;t?
Is it almost there, or just nowhere near what you want? Are there not enough
features or way too many? Are the docs too complicated; do you have a hard time
finding out how to do what you want? Should I think differently about what this
tool even <i>is</i> in the first place? Should I think differently about who
it&rsquo;s for?
<h1>What was Miller created to do?</h1>
<p/> First: there are tools like <tt>xsv</tt> which handles CSV marvelously and
<tt>jq</tt> which handles JSON marvelously, and so on &mdash; but I over the
years of my career in the software industry I&rsquo;ve found myself, and
others, doing a lot of ad-hoc things which really were fundamentally the same
<i>except</i> for format. So the number one thing about Miller is doing common
things while supporting <boldmaroon>multiple formats</boldmaroon>: (a) ingest a
list of records where a record is a list of key-value pairs (however
represented in the input files); (b) transform that stream of records; (c) emit
the transformed stream &mdash; either in the same format as input, or in a
different format.
<p/> Second thing, a lot like the first: just as I didn&rsquo;t want to build
something only for a single file format, I didn&rsquo;t want to build something
only for one problem domain. In my work doing software engineering, devops,
data engineering, etc. I saw a lot of commonalities and I wanted to
<boldmaroon>solve as many problems simultaneously as possible</boldmaroon>.
<p/> Third: it had to be <boldmaroon>streaming</boldmaroon>. As time goes by
and we (some of us, sometimes) have machines with tens or hundreds of GB of
RAM, it&rsquo;s maybe less important, but I&rsquo;m unhappy with tools which
ingest all data, then do stuff, then emit all data. One reason is to be able to
handle files bigger than available RAM. Another reason is to be able to handle
input which trickles in, e.g. you have some process emitting data now and then
and you can pipe it to Miller and it will emit transformed records one at a
time.
<p/> Fourth: it had to be <boldmaroon>fast</boldmaroon>. This precludes all
sorts of very nice things written in Ruby, for example. I love Ruby as a very
expressive language, and I have several very useful little utility scripts
written in Ruby. But a few years ago I ported over some of my old
tried-and-true C programs and the lines-of-code count was a <i>lot</i> lower
&mdash; it was great! Until I ran them on multi-GB files and realized they took
60x as long to complete. So I couldn&rsquo;t write Miller in Ruby, or in
languages like it. I was going to have to do something in a low-level language
in order to make it performant. I did simple experiments in several languages
&mdash; Ruby, Python, Lua, Rust, Go, D. In one I just read lines and printed
them back out &mdash; a line-oriented <tt>cat</tt>. In another I consumed input
lines like <tt>x=1,y=2,z=3</tt> one at a time, split them on commas and equals
signs to populate hash maps, transformed them (e.g. remove the <tt>y</tt>
field, and emitted them. Basically <tt>mlr cut -x -f y</tt> with DKVP format.
I didn&rsquo;t do anything fancy &mdash; just using each language&rsquo;s
<tt>getline</tt>, string-split, hashmap-put, etc. And nothing was as fast as
C, so I used C. (See also <a
href="whyc.html#C_vs._Go,_D,_Rust,_etc.;_C_is_fast">here</a>.)
<p/> Fifth thing: I wanted Miller to be <boldmaroon>pipe-friendly and
interoperate with other command-line tools</boldmaroon>. Since the basic
paradigm is ingest records, transform records, emit records &mdash; where the
input and output formats can be the same or different, and the transform can be
complex, or just pass-through &mdash; this means you can use it to transform
data, or re-format it, or both. So if you just want to do
data-cleaning/prep/formatting and do all the "real" work in R, you can. If you
just want a little glue script between other tools you can get that. And if you
want to do non-trivial data-reduction in Miller you can.
<p/> Sixth thing: Must have <boldmaroon>comprehensive documentation and
unit-test</boldmaroon>. Since Miller handles a lot of formats and solves a lot
of problems, there&rsquo;s a lot to test and a lot to keep working correctly as
I add features or optimize. And I wanted it to be able to explain itself
&mdash; not only through web docs like the one you&rsquo;re reading but also
through <tt>man mlr</tt> and <tt>mlr --help</tt>, <tt>mlr sort --help</tt>,
etc.
<p/> Seventh thing: <boldmaroon>Must have a domain-specific
language</boldmaroon> (DSL) <boldmaroon>but also must let you do common things
without it</boldmaroon>. All those little verbs Miller has to help you
<i>avoid</i> having to write for-loops are great. I use them for
keystroke-saving: <tt>mlr stats1 -a mean,stddev,min,max -f quantity</tt>, for
example, without you having to write for-loops or define accumulator variables.
But you also have to be able to break out of that and write arbitrary code when
you want to: <tt>mlr put '$distance = $rate * $time'</tt> or anything else you
can think up. In Perl/AWK/etc. it&rsquo;s all DSL. In xsv et al. it&rsquo;s
all verbs. In Miller I like having the combination.
<p/> Eighth thing: It&rsquo;s an <boldmaroon>awful lot of fun to
write</boldmaroon>. In my experience I didn&rsquo;t find any tools which do
multi-format, streaming, efficient, multi-purpose, with DSL and non-DSL, so I
wrote one. But I don&rsquo;t guarantee it&rsquo;s unique in the world. It fills
a niche in the world (people use it) but it also fills a niche in my life.
<h1>Tradeoffs</h1>
<p/> Miller is command-line-only by design. People who want a graphical user
interface won&rsquo;t find it here. This is in part (a) accommodating my
personal preferences, and in part (b) guided by my experience/belief that the
command line is very expressive. Steep learning curve, yes. I consider that
price worth paying.
<p/> Another tradeoff: supporting lists of records &mdash; each with only one
depth &mdash; keeps me supporting only what can be expressed in <i>all</i> of
those formats. E.g. in JSON you can have lists of lists of lists which Miller
just doesn&rsquo;t handle. So Miller can&rsquo;t (and won&rsquo;t) handle
arbitrary JSON because it only handles tabular data which can be expressed in a
variety of formats.
<p/> A third tradeoff is doing build-from-scratch in a low-level language.
It&rsquo;d be quicker to write (but slower to run) if written in a high-level
language. If Miller were written in Python, it would be implemented in
significantly fewer lines of code than its current C implementation. The DSL
would just be an <tt>eval</tt> of Python code. And it would run slower, but
maybe not enough slower to be a problem for most folks.
<p/> A fourth tradeoff is in the DSL (more visibly so in 5.0.0 but already in
pre-5.0.0): how much to make it dynamically typed &mdash; so you can just say
y=x+1 with a minimum number of keystrokes &mdash; vs. having it do a good job
of telling you when you&rsquo;ve made a typo. This is a common paradigm across
<i>all</i> languages. Some like Ruby you don&rsquo;t declare anything and
they&rsquo;re quick to code little stuff in but programs of even a few thousand
lines (which isn&rsquo;t large in the software world) become insanely
unmanageable. Then Java at the other extreme which is very typesafe but you
have to type in a lot of punctuation, angle brackets, datatypes, repetition,
etc. just to be able to get anything done. And some in the middle like Go which
are typesafe but with type inference which aim to do the best of both. In the
Miller (5.0.0) DSL you get y=x+1 by default but you can have things like int y
= x+1 etc. so the typesafety is opt-in. See also <a
href="reference-dsl.html#Type-checking">here</a> for more information on
type-checking.
<h1>Moving forward</h1>
<p/> I originally aimed Miller at people who already know what
<tt>sed</tt>/<tt>awk</tt>/<tt>cut</tt>/<tt>sort</tt>/<tt>join</tt> are and
wanted some options. But as time goes by I realize that tools like this can be
useful to folks who <i>don&rsquo;t</i> know what those things are; people who
aren&rsquo;t primarily coders; people who are scientists, or data scientists.
These days some journalists do data analysis. So moving forward in terms of
docs, I am working on having more cookbook, follow-by-example stuff in addition
to the existing language-reference kinds of stuff. And prioritizing a Windows
port &mdash; which is way overdue. And continuing to seek out input from people
who use Miller on where to go next.

23
doc/content-for-whyc.html
View file

@ -29,7 +29,28 @@ often. The D language (<a href="http://dlang.org">http://dolang.org</a>) is an
exciting and elegant successor to C++ (more about which below) &mdash; D has
many of Go&rsquo;s strengths, with a tighter stylistic similarity to C. And initial
experiments with Rust are intriguing. Yet with none of them could I obtain the
throughput I get in C: see the POKI_PUT_LINK_FOR_PAGE(performance.html)HERE page.
throughput I get in C.
<p/a>Specifically, I did simple experiments in several languages &mdash; Ruby,
Python, Lua, Rust, Go, D. In one I just read lines and printed them back out
&mdash; a line-oriented <tt>cat</tt>. In another I consumed input lines like
<tt>x=1,y=2,z=3</tt> one at a time, split them on commas and equals signs to
populate hash maps, transformed them (e.g. remove the <tt>y</tt> field), and
emitted them. Basically <tt>mlr cut -x -f y</tt> with DKVP format. I
didn&rsquo;t do anything fancy &mdash; just using each language&rsquo;s
<tt>getline</tt>, string-split, hashmap-put, etc. And nothing was as fast as
C, so I used C. Here are the experiments I kept (I failed to keep the
Lua code, for example):
<a href="../perf/catc.c.txt">C cat</a>,
<a href="../perf/catc0.c.txt">another C cat</a>,
<a href="../perf/catd.d.txt">D cat</a>,
<a href="../perf/catgo.go.txt">Go cat</a>,
<a href="../perf/catgo2.go.txt">another Go cat</a>,
<a href="../perf/catrust.rs.txt">Rust cat</a>,
<a href="../perf/nimcat.nim.txt">Nim cat</a>,
<a href="../perf/cutd.d.txt">D cut</a>,
<a href="../perf/cutgo.go.txt">Go cut</a>,
<a href="../perf/nimcut.nim.txt">Nim cut</a>.
<p/>One of Go&rsquo;s most powerful features is the ease with which it allows
quick-to-code, error-free concurrency. Yet Miller, like most high-volume

84
doc/cookbook.html
View file

@ -608,33 +608,33 @@ $ mlr --ofmt '%.9lf' --opprint seqgen --start 1 --stop 28 then put '
' then put '$seconds=systime()' then step -a delta -f seconds then cut -x -f seconds
i o fcount seconds_delta
1 1 1 0
2 2 3 0.000033140
3 3 5 0.000011921
2 2 3 0.000031948
3 3 5 0.000013113
4 5 9 0.000015974
5 8 15 0.000020027
6 13 25 0.000029087
7 21 41 0.000042915
8 34 67 0.000066996
9 55 109 0.000104904
10 89 177 0.000169039
11 144 287 0.000321150
12 233 465 0.000429869
13 377 753 0.000709057
14 610 1219 0.001135111
15 987 1973 0.001828909
16 1597 3193 0.002882004
17 2584 5167 0.004646063
18 4181 8361 0.010217905
19 6765 13529 0.014159918
20 10946 21891 0.019574165
21 17711 35421 0.031366825
22 28657 57313 0.050403118
23 46368 92735 0.089492083
24 75025 150049 0.139133930
25 121393 242785 0.208625078
26 196418 392835 0.348189831
27 317811 635621 0.540225029
28 514229 1028457 0.867680073
6 13 25 0.000029802
7 21 41 0.000044107
8 34 67 0.000068903
9 55 109 0.000108004
10 89 177 0.000172138
11 144 287 0.000277996
12 233 465 0.000442028
13 377 753 0.000705004
14 610 1219 0.001137972
15 987 1973 0.001832008
16 1597 3193 0.003048897
17 2584 5167 0.004843950
18 4181 8361 0.009150982
19 6765 13529 0.015188217
20 10946 21891 0.022678852
21 17711 35421 0.036461115
22 28657 57313 0.052274942
23 46368 92735 0.091028929
24 75025 150049 0.138079166
25 121393 242785 0.235754967
26 196418 392835 0.368572950
27 317811 635621 0.592602968
28 514229 1028457 0.926891088
</pre>
</div>
<p/>
@ -666,32 +666,32 @@ $ mlr --ofmt '%.9lf' --opprint seqgen --start 1 --stop 28 then put '
' then put '$seconds=systime()' then step -a delta -f seconds then cut -x -f seconds
i o fcount seconds_delta
1 1 1 0
2 2 3 0.000043869
3 3 3 0.000014067
4 5 3 0.000011921
2 2 3 0.000035048
3 3 3 0.000012875
4 5 3 0.000010967
5 8 3 0.000011206
6 13 3 0.000010967
7 21 3 0.000010014
8 34 3 0.000010967
8 34 3 0.000010014
9 55 3 0.000011921
10 89 3 0.000011921
11 144 3 0.000011206
12 233 3 0.000014782
13 377 3 0.000012159
10 89 3 0.000010014
11 144 3 0.000010014
12 233 3 0.000014067
13 377 3 0.000010967
14 610 3 0.000010014
15 987 3 0.000010967
16 1597 3 0.000010967
17 2584 3 0.000010014
18 4181 3 0.000010967
19 6765 3 0.000010967
15 987 3 0.000010014
16 1597 3 0.000010014
17 2584 3 0.000008821
18 4181 3 0.000010014
19 6765 3 0.000010014
20 10946 3 0.000010014
21 17711 3 0.000010967
21 17711 3 0.000010014
22 28657 3 0.000010014
23 46368 3 0.000013113
23 46368 3 0.000011921
24 75025 3 0.000010967
25 121393 3 0.000010967
25 121393 3 0.000010014
26 196418 3 0.000010014
27 317811 3 0.000010967
27 317811 3 0.000010014
28 514229 3 0.000010014
</pre>
</div>

326
doc/why.html Normal file
View file

@ -0,0 +1,326 @@
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html lang="en">
<!-- PAGE GENERATED FROM template.html and content-for-why.html BY poki. -->
<!-- PLEASE MAKE CHANGES THERE AND THEN RE-RUN poki. -->
<head>
<meta http-equiv="Content-type" content="text/html;charset=UTF-8"/>
<meta name="description" content="Miller documentation"/>
<meta name="viewport" content="width=device-width, initial-scale=1.0"/> <!-- mobile-friendly -->
<meta name="keywords"
content="John Kerl, Kerl, Miller, miller, mlr, OLAP, data analysis software, regression, correlation, variance, data tools, " />
<title> Why? </title>
<link rel="stylesheet" type="text/css" href="css/miller.css"/>
<link rel="stylesheet" type="text/css" href="css/poki-callbacks.css"/>
</head>
<!-- ================================================================ -->
<script type="text/javascript">
var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www.");
document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E"));
</script>
<script type="text/javascript">
try {
var pageTracker = _gat._getTracker("UA-15651652-1");
pageTracker._trackPageview();
} catch(err) {}
</script>
<script type="text/javascript">
function toggle(divName) {
var eleDiv = document.getElementById(divName);
if (eleDiv != null) {
if (eleDiv.style.display == "block") {
eleDiv.style.display = "none";
} else {
eleDiv.style.display = "block";
}
}
}
</script>
<!--
The background image is from a screenshot of a Google search for "data analysis
tools", lightened and sepia-toned. Over this was placed a Mac Terminal app with
very light-grey font and translucent background, in which a few statistical
Miller commands were run with pretty-print-tabular output format.
-->
<body background="pix/sepia-overlay.jpg">
<!-- ================================================================ -->
<table width="100%">
<tr>
<!-- navbar -->
<td width="15%">
<!--
<img src="pix/mlr.jpg" />
<img style="border-width:1px; color:black;" src="pix/mlr.jpg" />
-->
<div class="pokinav">
<center><titleinbody>Miller</titleinbody></center>
<!-- PAGE LIST GENERATED FROM template.html BY poki -->
<br/><b>Overview:</b>
<br/>&bull;&nbsp;<a href="index.html">About Miller</a>
<br/>&bull;&nbsp;<a href="10-min.html">Miller in 10 minutes</a>
<br/>&bull;&nbsp;<a href="file-formats.html">File formats</a>
<br/>&bull;&nbsp;<a href="feature-comparison.html">Miller features in the context of the Unix toolkit</a>
<br/>&bull;&nbsp;<a href="record-heterogeneity.html">Record-heterogeneity</a>
<br/>&bull;&nbsp;<a href="internationalization.html">Internationalization</a>
<br/><b>Using Miller:</b>
<br/>&bull;&nbsp;<a href="faq.html">FAQ</a>
<br/>&bull;&nbsp;<a href="data-examples.html">Data-diving examples</a>
<br/>&bull;&nbsp;<a href="cookbook.html">Cookbook</a>
<br/>&bull;&nbsp;<a href="manpage.html">Manpage</a>
<br/>&bull;&nbsp;<a href="reference.html">Reference</a>
<br/>&bull;&nbsp;<a href="reference-verbs.html">Reference: Verbs</a>
<br/>&bull;&nbsp;<a href="reference-dsl.html">Reference: DSL</a>
<br/>&bull;&nbsp;<a href="release-docs.html">Documents by release</a>
<br/>&bull;&nbsp;<a href="build.html">Installation, portability, dependencies, and testing</a>
<br/><b>Background:</b>
<br/>&bull;&nbsp;<a href="why.html"><b>Why?</b></a>
<br/>&bull;&nbsp;<a href="whyc.html">Why C?</a>
<br/>&bull;&nbsp;<a href="etymology.html">Why call it Miller?</a>
<br/>&bull;&nbsp;<a href="originality.html">How original is Miller?</a>
<br/>&bull;&nbsp;<a href="performance.html">Performance</a>
<br/><b>Repository:</b>
<br/>&bull;&nbsp;<a href="to-do.html">Things to do</a>
<br/>&bull;&nbsp;<a href="contact.html">Contact information</a>
<br/>&bull;&nbsp;<a href="https://github.com/johnkerl/miller">GitHub repo</a>
<br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/>
<br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/>
<br/> <br/> <br/> <br/> <br/> <br/>
</div>
</td>
<!-- page body -->
<td>
<!--
This is a visually gorgeous feature (here & in the CSS): it allows for
independent scroll of the nav and body panels. In particular the nav
stays on-screen as you scroll the body.
However, two problems:
(1) In Firefox & Chrome both I get janky end-of-body scrolls: there is
more content but I can't scroll down to it unless I repeatedly retry the
scrolldown. Which is weird.
(2) Worse, only the first page renders in PDF (again, Firefox & Chrome).
For now I'm disabling this separate-scroll feature. A frontender, I am
not ... maybe someday I'll find a config which gets *all* the features
I want; for now, it's a tradeoff.
-->
<!-- Implementation details: one bit is right here:
div style="overflow-y:scroll;height:1500px"
and the other bit is in css/poki-callbacks.css:
.pokinav {
display: inline-block;
background: #e8d9bc;
border: 1;
box-shadow: 0px 0px 3px 3px #C9C9C9;
margin: 10px;
padding-top: 10px;
padding-bottom: 10px;
padding-left: 10px;
padding-right: 10px;
overflow-y: scroll; < - - - - - - here
height: 1500px;
}
-->
<div>
<center> <titleinbody> Why? </titleinbody> </center>
<p/>
<!-- BODY COPIED FROM content-for-why.html BY poki -->
<div class="pokitoc">
<center><b>Contents:</b></center>
&bull;&nbsp;<a href="#Who_is_Miller_for?">Who is Miller for?</a><br/>
&bull;&nbsp;<a href="#What_was_Miller_created_to_do?">What was Miller created to do?</a><br/>
&bull;&nbsp;<a href="#Tradeoffs">Tradeoffs</a><br/>
&bull;&nbsp;<a href="#Moving_forward">Moving forward</a><br/>
</div>
<p/>
<p/> Someone asked me the other day about design, tradeoffs, thought process,
why I felt it necessary to build Miller, etc. Here are some answers.
<a id="Who_is_Miller_for?"/><h1>Who is Miller for?</h1>
<p/> For background, I&rsquo;m a software engineer, with a heavy devops bent
and a non-trivial amount of data-engineering in my career.
<boldmaroon>Initially I wrote Miller mainly for myself:</boldmaroon> I&rsquo;m
coder-friendly (being a coder); I&rsquo;m Github-friendly; most of my data are
well-structured or easily structurable (TSV-formatted SQL-query output, CSV
files, log files, JSON data structures); I care about interoperability between
all the various formats Miller supports (I&rsquo;ve encountered them all); I do
all my work on Linux or OSX.
<p/> But now there&rsquo;s this neat little tool <boldmaroon>which seems to be
useful for people in various disciplines</boldmaroon>. I don&rsquo;t even know
entirely <i>who</i>. I can click through Github starrers and read a bit about
what they seem to do, but not everyone&rsquo;s <i>on</i> Github (or stars
things). I&rsquo;ve gotten a lot of feature requests through Github &mdash; but
only from people who are Github users. For sure, not everyone&rsquo;s on Linux
or OSX (I have a Windows port underway). Not everyone&rsquo;s a coder (it seems
like a lot of Miller&rsquo;s Github starrers are devops folks like myself, or
data-science-ish people, or biology/genomics folks.) A lot of people care 100%
about CSV. And so on.
<p/> So I wonder (please drop a note at <a
href="https://github.com/johnkerl/miller/issues">https://github.com/johnkerl/miller/issues</a>)
does Miller do what you need? Do you use it for all sorts of things, or just
one or two nice things? Are there things you wish it did but it doesn&rsquo;t?
Is it almost there, or just nowhere near what you want? Are there not enough
features or way too many? Are the docs too complicated; do you have a hard time
finding out how to do what you want? Should I think differently about what this
tool even <i>is</i> in the first place? Should I think differently about who
it&rsquo;s for?
<a id="What_was_Miller_created_to_do?"/><h1>What was Miller created to do?</h1>
<p/> First: there are tools like <tt>xsv</tt> which handles CSV marvelously and
<tt>jq</tt> which handles JSON marvelously, and so on &mdash; but I over the
years of my career in the software industry I&rsquo;ve found myself, and
others, doing a lot of ad-hoc things which really were fundamentally the same
<i>except</i> for format. So the number one thing about Miller is doing common
things while supporting <boldmaroon>multiple formats</boldmaroon>: (a) ingest a
list of records where a record is a list of key-value pairs (however
represented in the input files); (b) transform that stream of records; (c) emit
the transformed stream &mdash; either in the same format as input, or in a
different format.
<p/> Second thing, a lot like the first: just as I didn&rsquo;t want to build
something only for a single file format, I didn&rsquo;t want to build something
only for one problem domain. In my work doing software engineering, devops,
data engineering, etc. I saw a lot of commonalities and I wanted to
<boldmaroon>solve as many problems simultaneously as possible</boldmaroon>.
<p/> Third: it had to be <boldmaroon>streaming</boldmaroon>. As time goes by
and we (some of us, sometimes) have machines with tens or hundreds of GB of
RAM, it&rsquo;s maybe less important, but I&rsquo;m unhappy with tools which
ingest all data, then do stuff, then emit all data. One reason is to be able to
handle files bigger than available RAM. Another reason is to be able to handle
input which trickles in, e.g. you have some process emitting data now and then
and you can pipe it to Miller and it will emit transformed records one at a
time.
<p/> Fourth: it had to be <boldmaroon>fast</boldmaroon>. This precludes all
sorts of very nice things written in Ruby, for example. I love Ruby as a very
expressive language, and I have several very useful little utility scripts
written in Ruby. But a few years ago I ported over some of my old
tried-and-true C programs and the lines-of-code count was a <i>lot</i> lower
&mdash; it was great! Until I ran them on multi-GB files and realized they took
60x as long to complete. So I couldn&rsquo;t write Miller in Ruby, or in
languages like it. I was going to have to do something in a low-level language
in order to make it performant. I did simple experiments in several languages
&mdash; Ruby, Python, Lua, Rust, Go, D. In one I just read lines and printed
them back out &mdash; a line-oriented <tt>cat</tt>. In another I consumed input
lines like <tt>x=1,y=2,z=3</tt> one at a time, split them on commas and equals
signs to populate hash maps, transformed them (e.g. remove the <tt>y</tt>
field, and emitted them. Basically <tt>mlr cut -x -f y</tt> with DKVP format.
I didn&rsquo;t do anything fancy &mdash; just using each language&rsquo;s
<tt>getline</tt>, string-split, hashmap-put, etc. And nothing was as fast as
C, so I used C. (See also <a
href="whyc.html#C_vs._Go,_D,_Rust,_etc.;_C_is_fast">here</a>.)
<p/> Fifth thing: I wanted Miller to be <boldmaroon>pipe-friendly and
interoperate with other command-line tools</boldmaroon>. Since the basic
paradigm is ingest records, transform records, emit records &mdash; where the
input and output formats can be the same or different, and the transform can be
complex, or just pass-through &mdash; this means you can use it to transform
data, or re-format it, or both. So if you just want to do
data-cleaning/prep/formatting and do all the "real" work in R, you can. If you
just want a little glue script between other tools you can get that. And if you
want to do non-trivial data-reduction in Miller you can.
<p/> Sixth thing: Must have <boldmaroon>comprehensive documentation and
unit-test</boldmaroon>. Since Miller handles a lot of formats and solves a lot
of problems, there&rsquo;s a lot to test and a lot to keep working correctly as
I add features or optimize. And I wanted it to be able to explain itself
&mdash; not only through web docs like the one you&rsquo;re reading but also
through <tt>man mlr</tt> and <tt>mlr --help</tt>, <tt>mlr sort --help</tt>,
etc.
<p/> Seventh thing: <boldmaroon>Must have a domain-specific
language</boldmaroon> (DSL) <boldmaroon>but also must let you do common things
without it</boldmaroon>. All those little verbs Miller has to help you
<i>avoid</i> having to write for-loops are great. I use them for
keystroke-saving: <tt>mlr stats1 -a mean,stddev,min,max -f quantity</tt>, for
example, without you having to write for-loops or define accumulator variables.
But you also have to be able to break out of that and write arbitrary code when
you want to: <tt>mlr put '$distance = $rate * $time'</tt> or anything else you
can think up. In Perl/AWK/etc. it&rsquo;s all DSL. In xsv et al. it&rsquo;s
all verbs. In Miller I like having the combination.
<p/> Eighth thing: It&rsquo;s an <boldmaroon>awful lot of fun to
write</boldmaroon>. In my experience I didn&rsquo;t find any tools which do
multi-format, streaming, efficient, multi-purpose, with DSL and non-DSL, so I
wrote one. But I don&rsquo;t guarantee it&rsquo;s unique in the world. It fills
a niche in the world (people use it) but it also fills a niche in my life.
<a id="Tradeoffs"/><h1>Tradeoffs</h1>
<p/> Miller is command-line-only by design. People who want a graphical user
interface won&rsquo;t find it here. This is in part (a) accommodating my
personal preferences, and in part (b) guided by my experience/belief that the
command line is very expressive. Steep learning curve, yes. I consider that
price worth paying.
<p/> Another tradeoff: supporting lists of records &mdash; each with only one
depth &mdash; keeps me supporting only what can be expressed in <i>all</i> of
those formats. E.g. in JSON you can have lists of lists of lists which Miller
just doesn&rsquo;t handle. So Miller can&rsquo;t (and won&rsquo;t) handle
arbitrary JSON because it only handles tabular data which can be expressed in a
variety of formats.
<p/> A third tradeoff is doing build-from-scratch in a low-level language.
It&rsquo;d be quicker to write (but slower to run) if written in a high-level
language. If Miller were written in Python, it would be implemented in
significantly fewer lines of code than its current C implementation. The DSL
would just be an <tt>eval</tt> of Python code. And it would run slower, but
maybe not enough slower to be a problem for most folks.
<p/> A fourth tradeoff is in the DSL (more visibly so in 5.0.0 but already in
pre-5.0.0): how much to make it dynamically typed &mdash; so you can just say
y=x+1 with a minimum number of keystrokes &mdash; vs. having it do a good job
of telling you when you&rsquo;ve made a typo. This is a common paradigm across
<i>all</i> languages. Some like Ruby you don&rsquo;t declare anything and
they&rsquo;re quick to code little stuff in but programs of even a few thousand
lines (which isn&rsquo;t large in the software world) become insanely
unmanageable. Then Java at the other extreme which is very typesafe but you
have to type in a lot of punctuation, angle brackets, datatypes, repetition,
etc. just to be able to get anything done. And some in the middle like Go which
are typesafe but with type inference which aim to do the best of both. In the
Miller (5.0.0) DSL you get y=x+1 by default but you can have things like int y
= x+1 etc. so the typesafety is opt-in. See also <a
href="reference-dsl.html#Type-checking">here</a> for more information on
type-checking.
<a id="Moving_forward"/><h1>Moving forward</h1>
<p/> I originally aimed Miller at people who already know what
<tt>sed</tt>/<tt>awk</tt>/<tt>cut</tt>/<tt>sort</tt>/<tt>join</tt> are and
wanted some options. But as time goes by I realize that tools like this can be
useful to folks who <i>don&rsquo;t</i> know what those things are; people who
aren&rsquo;t primarily coders; people who are scientists, or data scientists.
These days some journalists do data analysis. So moving forward in terms of
docs, I am working on having more cookbook, follow-by-example stuff in addition
to the existing language-reference kinds of stuff. And prioritizing a Windows
port &mdash; which is way overdue. And continuing to seek out input from people
who use Miller on where to go next.
</div>
</td>
</table>
</body>
</html>

23
doc/whyc.html
View file

@ -183,7 +183,28 @@ often. The D language (<a href="http://dlang.org">http://dolang.org</a>) is an
exciting and elegant successor to C++ (more about which below) &mdash; D has
many of Go&rsquo;s strengths, with a tighter stylistic similarity to C. And initial
experiments with Rust are intriguing. Yet with none of them could I obtain the
throughput I get in C: see the <a href="performance.html">Performance</a> page.
throughput I get in C.
<p/a>Specifically, I did simple experiments in several languages &mdash; Ruby,
Python, Lua, Rust, Go, D. In one I just read lines and printed them back out
&mdash; a line-oriented <tt>cat</tt>. In another I consumed input lines like
<tt>x=1,y=2,z=3</tt> one at a time, split them on commas and equals signs to
populate hash maps, transformed them (e.g. remove the <tt>y</tt> field), and
emitted them. Basically <tt>mlr cut -x -f y</tt> with DKVP format. I
didn&rsquo;t do anything fancy &mdash; just using each language&rsquo;s
<tt>getline</tt>, string-split, hashmap-put, etc. And nothing was as fast as
C, so I used C. Here are the experiments I kept (I failed to keep the
Lua code, for example):
<a href="../perf/catc.c.txt">C cat</a>,
<a href="../perf/catc0.c.txt">another C cat</a>,
<a href="../perf/catd.d.txt">D cat</a>,
<a href="../perf/catgo.go.txt">Go cat</a>,
<a href="../perf/catgo2.go.txt">another Go cat</a>,
<a href="../perf/catrust.rs.txt">Rust cat</a>,
<a href="../perf/nimcat.nim.txt">Nim cat</a>,
<a href="../perf/cutd.d.txt">D cut</a>,
<a href="../perf/cutgo.go.txt">Go cut</a>,
<a href="../perf/nimcut.nim.txt">Nim cut</a>.
<p/>One of Go&rsquo;s most powerful features is the ease with which it allows
quick-to-code, error-free concurrency. Yet Miller, like most high-volume

45
perf/catc.c.txt Normal file
View file

@ -0,0 +1,45 @@
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
// ----------------------------------------------------------------
static int do_stream(char* file_name) {
FILE* input_stream = stdin;
FILE* output_stream = stdout;
if (strcmp(file_name, "-")) {
input_stream = fopen(file_name, "r");
if (input_stream == NULL) {
perror(file_name);
return 0;
}
}
while (1) {
char* line = NULL;
size_t linecap = 0;
ssize_t linelen = getdelim(&line, &linecap, '\n', input_stream);
if (linelen <= 0) {
break;
}
fputs(line, output_stream);
free(line);
}
if (input_stream != stdin)
fclose(input_stream);
return 1;
}
// ================================================================
int main(int argc, char** argv) {
int ok = 1;
if (argc == 1) {
ok = ok && do_stream("-");
} else {
for (int argi = 1; argi < argc; argi++) {
ok = do_stream(argv[argi]);
}
}
return ok ? 0 : 1;
}

44
perf/catc0.c.txt Normal file
View file

@ -0,0 +1,44 @@
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#define MYBUFSIZ 8192
static char iobuf[MYBUFSIZ];
// ----------------------------------------------------------------
static int do_stream(char* file_name) {
FILE* input_stream = stdin;
FILE* output_stream = stdout;
if (strcmp(file_name, "-")) {
input_stream = fopen(file_name, "r");
if (input_stream == NULL) {
perror(file_name);
return 0;
}
}
while (1) {
char* line = fgets(iobuf, BUFSIZ, input_stream);
if (line == NULL)
break;
fputs(line, output_stream);
}
if (input_stream != stdin)
fclose(input_stream);
return 1;
}
// ================================================================
int main(int argc, char** argv) {
int ok = 1;
if (argc == 1) {
ok = ok && do_stream("-");
} else {
for (int argi = 1; argi < argc; argi++) {
ok = do_stream(argv[argi]);
}
}
return ok ? 0 : 1;
}

9
perf/catd.d.txt Normal file
View file

@ -0,0 +1,9 @@
// Reads $(D stdin) and writes it to $(D stdout).
import std.stdio;
void main()
{
string line;
while ((line = stdin.readln()) !is null)
write(line);
}

66
perf/catgo.go.txt Normal file
View file

@ -0,0 +1,66 @@
package main
import (
"bufio"
"io"
"log"
"os"
)
// ----------------------------------------------------------------
func main() {
args := os.Args[1:]
includeFields := []string {"a", "x"};
ok := true
if len(args) == 0 {
ok = handle("-", includeFields) && ok
} else {
for _, arg := range args {
ok = handle(arg, includeFields) && ok
}
}
if ok {
os.Exit(0)
} else {
os.Exit(1)
}
}
// ----------------------------------------------------------------
func handle(fileName string, includeFields []string) (ok bool) {
inputStream := os.Stdin
if fileName != "-" {
var err error
if inputStream, err = os.Open(fileName); err != nil {
log.Println(err)
return false
}
}
reader := bufio.NewReader(inputStream)
writer := bufio.NewWriter(os.Stdout)
eof := false
for !eof {
line, err := reader.ReadString('\n')
if err == io.EOF {
err = nil
eof = true
} else if err != nil {
log.Println(err)
if fileName != "-" {
inputStream.Close()
}
return false
} else {
writer.WriteString(line)
}
}
if fileName != "-" {
inputStream.Close()
}
writer.Flush()
return true
}

55
perf/catgo2.go.txt Normal file
View file

@ -0,0 +1,55 @@
package main
import (
"bufio"
"fmt"
"log"
"os"
)
// ----------------------------------------------------------------
func main() {
args := os.Args[1:]
includeFields := []string {"a", "x"};
ok := true
if len(args) == 0 {
ok = handle("-", includeFields) && ok
} else {
for _, arg := range args {
ok = handle(arg, includeFields) && ok
}
}
if ok {
os.Exit(0)
} else {
os.Exit(1)
}
}
// ----------------------------------------------------------------
func handle(fileName string, includeFields []string) (ok bool) {
inputStream := os.Stdin
if fileName != "-" {
var err error
if inputStream, err = os.Open(fileName); err != nil {
log.Println(err)
return false
}
}
scanner := bufio.NewScanner(inputStream)
for scanner.Scan() {
line := scanner.Text()
fmt.Println(line)
}
if err := scanner.Err(); err != nil {
fmt.Fprintln(os.Stderr, "reading standard input:", err)
}
if fileName != "-" {
inputStream.Close()
}
return true
}

16
perf/catrust.rs.txt Normal file
View file

@ -0,0 +1,16 @@
use std::io;
fn main() {
for line in io::stdin().lock().lines() {
print!("{}", line.unwrap());
}
}
//fn main() {
// let mut reader = io::stdin();
// let mut line;
// loop {
// line = reader.read_line();
// print!("{}\n", line);
// }
//}

40
perf/cutd.d.txt Normal file
View file

@ -0,0 +1,40 @@
// Reads $(D stdin) and writes it to $(D stdout).
// http://dlang.org/hash-map.html
import std.stdio;
import std.string;
import std.array;
void main() {
string[] includeFields = ["a", "x"];
string line;
while ((line = stdin.readln()) !is null) {
// Input string to hashmap.
string[string] oldmap;
string[] fields = split(line, ',');
foreach (field; fields) {
string[] kvps = split(field, '='); // really want splitN with max #parts = 2
oldmap[kvps[0]] = kvps[1];
}
// Hashmap-to-hashmap transform.
// Note: unordered hashmap here.
string[string] newmap;
foreach (includeField; includeFields) {
if (includeField in oldmap) {
newmap[includeField] = oldmap[includeField];
}
}
// Hashmap to output strings.
int i = 0;
foreach (key; newmap.keys) {
if (i > 0)
write(',');
write(key);
write('=');
write(newmap[key]);
i++;
}
write('\n');
}
}

1
perf/cutgo.go
View file

@ -100,7 +100,6 @@ func handle(fileName string, includeFields []string) (ok bool) {
// Write to output stream
//fmt.Println("")
writer.WriteString(out)
// xxx now == ?!? 0.720s
// delta 0.390s
// 56%

114
perf/cutgo.go.txt Normal file
View file

@ -0,0 +1,114 @@
package main
import (
"bufio"
"io"
"log"
"os"
"strings"
)
// ----------------------------------------------------------------
func main() {
args := os.Args[1:]
includeFields := []string {"a", "x"};
ok := true
if len(args) == 0 {
ok = handle("-", includeFields) && ok
} else {
for _, arg := range args {
ok = handle(arg, includeFields) && ok
}
}
if ok {
os.Exit(0)
} else {
os.Exit(1)
}
}
// ----------------------------------------------------------------
func handle(fileName string, includeFields []string) (ok bool) {
inputStream := os.Stdin
if fileName != "-" {
var err error
if inputStream, err = os.Open(fileName); err != nil {
log.Println(err)
return false
}
}
reader := bufio.NewReader(inputStream)
writer := bufio.NewWriter(os.Stdout)
eof := false
for !eof {
line, err := reader.ReadString('\n')
if err == io.EOF {
err = nil
eof = true
} else if err != nil {
log.Println(err)
if fileName != "-" {
inputStream.Close()
}
return false
} else {
// 0.030s
// Line to map
mymap := make(map[string]string)
fields := strings.Split(line, ",");
for _, field := range(fields) {
kvps := strings.SplitN(field, "=", 2)
mymap[kvps[0]] = kvps[1]
}
// 0.220s
// delta 0.190s
// 27%
// Map-to-map transform
newmap := make(map[string]string)
for _, includeField := range(includeFields) {
value, present := mymap[includeField]
if present {
newmap[includeField] = value
}
}
// 0.280s
// delta 0.060s
// 9%
// Map to string
outs := make([]string, len(newmap))
i := 0
for k, v := range(newmap) {
outs[i] = k + "=" + v
i++
}
// 0.320s
// delta 0.040s
// 6%
out := strings.Join(outs, ",")
// 0.330s
// delta 0.010s
// 2%
// Write to output stream
//fmt.Println("")
writer.WriteString(out)
// delta 0.390s
// 56%
}
}
if fileName != "-" {
inputStream.Close()
}
writer.Flush()
return true
}

2
perf/nimcat.nim.txt Normal file
View file

@ -0,0 +1,2 @@
for line in stdin.lines:
echo(line)

15
perf/nimcut.nim.txt Normal file
View file

@ -0,0 +1,15 @@
import strutils, tables
for line in stdin.lines:
#var map: OrderedTable[string,string]
var map = {"":""}.newOrderedTable
#var map = initTable[string, string]
#var map: OrderedTable[string, string]
#var map: newOrderedTable[string, string](16)
for word in line.split(","):
var pair = word.split("=")
#echo(pair[0])
#echo(pair[1])
#echo()
#map[pair[0]] = pair[1]
map.add(pair[0], pair[1])