mirror of
https://github.com/johnkerl/miller.git
synced 2026-07-18 00:45:47 +00:00
CSS neatens
This commit is contained in:
parent
973104ac76
commit
21067acde0
13 changed files with 140 additions and 140 deletions
|
|
@ -81,7 +81,7 @@ pageTracker._trackPageview();
|
|||
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="bodyToggler.toggle('body_section_toggle_csv_file_examples');" href="javascript:;">Toggle section visibility</button>
|
||||
<div id="body_section_toggle_csv_file_examples" style="display: block">
|
||||
|
||||
<p/><boldmaroon> Sample CSV data file: </boldmaroon>
|
||||
<p class="boldmaroon"> Sample CSV data file: </p>
|
||||
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
|
|
@ -102,7 +102,7 @@ purple,square,0,91,72.3735,8.2430
|
|||
</div>
|
||||
<p/>
|
||||
|
||||
<p/><boldmaroon> <code>mlr cat</code> is like cat ...</boldmaroon>
|
||||
<p class="boldmaroon"><code>mlr cat</code> is like cat ...</p>
|
||||
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
|
|
@ -123,7 +123,7 @@ purple,square,0,91,72.3735,8.2430
|
|||
</div>
|
||||
<p/>
|
||||
|
||||
<p/><boldmaroon>... but it can also do format conversion (here, to pretty-printed tabular format): </boldmaroon>
|
||||
<p class="boldmaroon">... but it can also do format conversion (here, to pretty-printed tabular format): </p>
|
||||
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
|
|
@ -144,8 +144,8 @@ purple square 0 91 72.3735 8.2430
|
|||
</div>
|
||||
<p/>
|
||||
|
||||
<p/><boldmaroon> <code>mlr head</code> and <code>mlr tail</code> count records rather than lines. The CSV
|
||||
header is included either way:</boldmaroon>
|
||||
<p class="boldmaroon"> <code>mlr head</code> and <code>mlr tail</code> count records rather than lines. The CSV
|
||||
header is included either way:</p>
|
||||
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
|
|
@ -172,8 +172,8 @@ purple,square,0,91,72.3735,8.2430
|
|||
</div>
|
||||
<p/>
|
||||
|
||||
<p/><boldmaroon> Sort primarily alphabetically on one field, then secondarily
|
||||
numerically descending on another field: </boldmaroon>
|
||||
<p class="boldmaroon"> Sort primarily alphabetically on one field, then secondarily
|
||||
numerically descending on another field: </p>
|
||||
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
|
|
@ -194,7 +194,7 @@ yellow triangle 1 11 43.6498 9.8870
|
|||
</div>
|
||||
<p/>
|
||||
|
||||
<p/><boldmaroon> Use <code>cut</code> to retain only specified fields, in input-data order:</boldmaroon>
|
||||
<p class="boldmaroon"> Use <code>cut</code> to retain only specified fields, in input-data order:</p>
|
||||
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
|
|
@ -215,7 +215,7 @@ square 0
|
|||
</div>
|
||||
<p/>
|
||||
|
||||
<p/><boldmaroon> Use <code>cut -o</code> to retain only specified fields, in your specified order:</boldmaroon>
|
||||
<p class="boldmaroon"> Use <code>cut -o</code> to retain only specified fields, in your specified order:</p>
|
||||
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
|
|
@ -236,7 +236,7 @@ flag shape
|
|||
</div>
|
||||
<p/>
|
||||
|
||||
<p/><boldmaroon> Use <code>cut -x</code> to omit specified fields:</boldmaroon>
|
||||
<p class="boldmaroon"> Use <code>cut -x</code> to omit specified fields:</p>
|
||||
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
|
|
@ -257,7 +257,7 @@ purple 91 72.3735 8.2430
|
|||
</div>
|
||||
<p/>
|
||||
|
||||
<p/><boldmaroon> Use <code>filter</code> to retain specified records:</boldmaroon>
|
||||
<p class="boldmaroon"> Use <code>filter</code> to retain specified records:</p>
|
||||
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
|
|
@ -282,7 +282,7 @@ red circle 1 16 13.8103 2.9010
|
|||
</div>
|
||||
<p/>
|
||||
|
||||
<p/><boldmaroon> Use <code>put</code> to add/replace fields which are computed from other fields:</boldmaroon>
|
||||
<p class="boldmaroon"> Use <code>put</code> to add/replace fields which are computed from other fields:</p>
|
||||
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
|
|
@ -303,10 +303,10 @@ purple square 0 91 72.3735 8.2430 8.779995 purple_square
|
|||
</div>
|
||||
<p/>
|
||||
|
||||
<p/><boldmaroon> Even though Miller’s main selling point is
|
||||
<p class="boldmaroon"> Even though Miller’s main selling point is
|
||||
name-indexing, sometimes you really want to refer to a field name by its
|
||||
positional index. Use <code>$[[3]]</code> to access the name of field 3 or
|
||||
<code>$[[[3]]]</code> to access the value of field 3:</boldmaroon>
|
||||
<code>$[[[3]]]</code> to access the value of field 3:</p>
|
||||
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
|
|
@ -345,7 +345,7 @@ purple square NEW 91 72.3735 8.2430
|
|||
</div>
|
||||
<p/>
|
||||
|
||||
<p/><boldmaroon> JSON output:</boldmaroon>
|
||||
<p class="boldmaroon"> JSON output:</p>
|
||||
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
|
|
@ -365,7 +365,7 @@ $ mlr --icsv --ojson put '$ratio = $quantity/$rate; $shape = toupper($shape)' ex
|
|||
</div>
|
||||
<p/>
|
||||
|
||||
<p/><boldmaroon> JSON output with vertical-formatting flags:</boldmaroon>
|
||||
<p class="boldmaroon"> JSON output with vertical-formatting flags:</p>
|
||||
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
|
|
@ -393,11 +393,11 @@ $ mlr --icsv --ojson --jvstack --jlistwrap tail -n 2 example.csv
|
|||
</div>
|
||||
<p/>
|
||||
|
||||
<p/><boldmaroon> Use <code>then</code> to pipe commands together. Also, the
|
||||
<p class="boldmaroon"> Use <code>then</code> to pipe commands together. Also, the
|
||||
<code>-g</code> option for many Miller commands is for group-by: here, <code>head -n
|
||||
1 -g shape</code> outputs the first record for each distinct value of the
|
||||
<code>shape</code> field. This means we’re finding the record with highest
|
||||
<code>index</code> field for each distinct <code>shape</code> field:</boldmaroon>
|
||||
<code>index</code> field for each distinct <code>shape</code> field:</p>
|
||||
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
|
|
@ -411,9 +411,9 @@ purple triangle 0 65 80.1405 5.8240
|
|||
</div>
|
||||
<p/>
|
||||
|
||||
<p/><boldmaroon> Statistics can be computed with or without group-by field(s). Also, the first of these two
|
||||
<p class="boldmaroon"> Statistics can be computed with or without group-by field(s). Also, the first of these two
|
||||
examples uses <code>--oxtab</code> output format which is a nice alternative to <code>--opprint</code> when you
|
||||
have lots of columns:</boldmaroon>
|
||||
have lots of columns:</p>
|
||||
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
|
|
@ -461,11 +461,11 @@ square purple 1 72.373500 72.373500 72.373500
|
|||
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="bodyToggler.togglebodyToggler.toggle('body_section_toggle_printing_to_files');" href="javascript:;">Toggle section visibility</button>
|
||||
<div id="body_section_toggle_printing_to_files" style="display: block">
|
||||
|
||||
<p/>Often we want to print output <boldmaroon>to the screen</boldmaroon>. Miller does this by default, as we’ve
|
||||
seen in the previous examples.
|
||||
<p>Often we want to print output <span class="boldmaroon">to the screen</span>. Miller does this by default, as we’ve
|
||||
seen in the previous examples.</p>
|
||||
|
||||
<p/> Sometimes we want to print output to another file: <boldmaroon>just use '>
|
||||
outputfilenamegoeshere'</boldmaroon> at the end of your command:
|
||||
<p/> Sometimes we want to print output to another file: <span class="boldmaroon">just use '>
|
||||
outputfilenamegoeshere'</span> at the end of your command:</p>
|
||||
|
||||
<table><tr><td>
|
||||
<div class="pokipanel"> <pre>
|
||||
|
|
@ -490,7 +490,7 @@ purple square 0 91 72.3735 8.2430
|
|||
</pre> </div>
|
||||
</td></tr></table>
|
||||
|
||||
<p/> Other times we just want our files to be changed in-place: <boldmaroon>just use 'mlr -I'</boldmaroon>.
|
||||
<p/> Other times we just want our files to be changed in-place: <span class="boldmaroon">just use 'mlr -I'</span>.</p>
|
||||
|
||||
<table><tr><td>
|
||||
<div class="pokipanel"> <pre>
|
||||
|
|
@ -538,8 +538,8 @@ mlr -I --csv cut -x -f unwanted_column_name *.csv
|
|||
|
||||
<p/> If you like, you can first copy off your original data somewhere else, before doing in-place operations.
|
||||
|
||||
<p/><boldmaroon> Lastly, using <code>tee</code> within <code>put</code>, you can split your input data into separate files
|
||||
per one or more field names:</boldmaroon>
|
||||
<p class="boldmaroon"> Lastly, using <code>tee</code> within <code>put</code>, you can split your input data into separate files
|
||||
per one or more field names:</p>
|
||||
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
|
|
@ -616,7 +616,7 @@ shape=square,flag=0,index=36
|
|||
</pre>
|
||||
</div>
|
||||
|
||||
<p/>Data written this way are called <boldmaroon>DKVP</boldmaroon>, for <i>delimited key-value pairs</i>.
|
||||
<p>Data written this way are called <span class="boldmaroon">DKVP</span>, for <i>delimited key-value pairs</i>.</p>
|
||||
|
||||
<p/>We’ve also already seen other ways to write the same data:
|
||||
|
||||
|
|
@ -640,8 +640,8 @@ shape=square,flag=0,index=36 flag 1 "flag": 0,
|
|||
</pre>
|
||||
</div>
|
||||
|
||||
<p/><boldmaroon>Anything we can do with CSV input data, we can do with any
|
||||
other format input data.</boldmaroon> And you can read from one format, do any
|
||||
<p class="boldmaroon">Anything we can do with CSV input data, we can do with any
|
||||
other format input data.</p> And you can read from one format, do any
|
||||
record-processing, and output to the same format as the input, or to a
|
||||
different output format.
|
||||
|
||||
|
|
|
|||
|
|
@ -353,7 +353,7 @@ other 21st-century compilers)
|
|||
|
||||
<a id="Optional_external_dependencies"/><h2>Optional external dependencies</h2>
|
||||
|
||||
This documentation pageset is built using <boldmaroon>Poki</boldmaroon>:
|
||||
This documentation pageset is built using <span class="boldmaroon">Poki</span>:
|
||||
<a href="http://johnkerl.org/poki/doc">docs here</a>,
|
||||
<a href="https://github.com/johnkerl/poki">source code here</a>.
|
||||
Note that <a href="http://johnkerl.org/miller/doc/index.html">http://johnkerl.org/miller/doc/index.html</a>
|
||||
|
|
|
|||
|
|
@ -8,77 +8,77 @@ POKI_PUT_TOC_HERE
|
|||
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="bodyToggler.toggle('body_section_toggle_csv_file_examples');" href="javascript:;">Toggle section visibility</button>
|
||||
<div id="body_section_toggle_csv_file_examples" style="display: block">
|
||||
|
||||
<p/><boldmaroon> Sample CSV data file: </boldmaroon>
|
||||
<p class="boldmaroon"> Sample CSV data file: </p>
|
||||
|
||||
POKI_RUN_COMMAND{{cat example.csv}}HERE
|
||||
|
||||
<p/><boldmaroon> <code>mlr cat</code> is like cat ...</boldmaroon>
|
||||
<p class="boldmaroon"><code>mlr cat</code> is like cat ...</p>
|
||||
|
||||
POKI_RUN_COMMAND{{mlr --csv cat example.csv}}HERE
|
||||
|
||||
<p/><boldmaroon>... but it can also do format conversion (here, to pretty-printed tabular format): </boldmaroon>
|
||||
<p class="boldmaroon">... but it can also do format conversion (here, to pretty-printed tabular format): </p>
|
||||
|
||||
POKI_RUN_COMMAND{{mlr --icsv --opprint cat example.csv}}HERE
|
||||
|
||||
<p/><boldmaroon> <code>mlr head</code> and <code>mlr tail</code> count records rather than lines. The CSV
|
||||
header is included either way:</boldmaroon>
|
||||
<p class="boldmaroon"> <code>mlr head</code> and <code>mlr tail</code> count records rather than lines. The CSV
|
||||
header is included either way:</p>
|
||||
|
||||
POKI_RUN_COMMAND{{mlr --csv head -n 4 example.csv}}HERE
|
||||
POKI_RUN_COMMAND{{mlr --csv tail -n 4 example.csv}}HERE
|
||||
|
||||
<p/><boldmaroon> Sort primarily alphabetically on one field, then secondarily
|
||||
numerically descending on another field: </boldmaroon>
|
||||
<p class="boldmaroon"> Sort primarily alphabetically on one field, then secondarily
|
||||
numerically descending on another field: </p>
|
||||
|
||||
POKI_RUN_COMMAND{{mlr --icsv --opprint sort -f shape -nr index example.csv}}HERE
|
||||
|
||||
<p/><boldmaroon> Use <code>cut</code> to retain only specified fields, in input-data order:</boldmaroon>
|
||||
<p class="boldmaroon"> Use <code>cut</code> to retain only specified fields, in input-data order:</p>
|
||||
|
||||
POKI_RUN_COMMAND{{mlr --icsv --opprint cut -f flag,shape example.csv}}HERE
|
||||
|
||||
<p/><boldmaroon> Use <code>cut -o</code> to retain only specified fields, in your specified order:</boldmaroon>
|
||||
<p class="boldmaroon"> Use <code>cut -o</code> to retain only specified fields, in your specified order:</p>
|
||||
|
||||
POKI_RUN_COMMAND{{mlr --icsv --opprint cut -o -f flag,shape example.csv}}HERE
|
||||
|
||||
<p/><boldmaroon> Use <code>cut -x</code> to omit specified fields:</boldmaroon>
|
||||
<p class="boldmaroon"> Use <code>cut -x</code> to omit specified fields:</p>
|
||||
|
||||
POKI_RUN_COMMAND{{mlr --icsv --opprint cut -x -f flag,shape example.csv}}HERE
|
||||
|
||||
<p/><boldmaroon> Use <code>filter</code> to retain specified records:</boldmaroon>
|
||||
<p class="boldmaroon"> Use <code>filter</code> to retain specified records:</p>
|
||||
|
||||
POKI_RUN_COMMAND{{mlr --icsv --opprint filter '$color == "red"' example.csv}}HERE
|
||||
POKI_RUN_COMMAND{{mlr --icsv --opprint filter '$color == "red" && $flag == 1' example.csv}}HERE
|
||||
|
||||
<p/><boldmaroon> Use <code>put</code> to add/replace fields which are computed from other fields:</boldmaroon>
|
||||
<p class="boldmaroon"> Use <code>put</code> to add/replace fields which are computed from other fields:</p>
|
||||
|
||||
POKI_RUN_COMMAND{{mlr --icsv --opprint put '$ratio = $quantity / $rate; $color_shape = $color . "_" . $shape' example.csv}}HERE
|
||||
|
||||
<p/><boldmaroon> Even though Miller’s main selling point is
|
||||
<p class="boldmaroon"> Even though Miller’s main selling point is
|
||||
name-indexing, sometimes you really want to refer to a field name by its
|
||||
positional index. Use <code>$[[3]]</code> to access the name of field 3 or
|
||||
<code>$[[[3]]]</code> to access the value of field 3:</boldmaroon>
|
||||
<code>$[[[3]]]</code> to access the value of field 3:</p>
|
||||
|
||||
POKI_RUN_COMMAND{{mlr --icsv --opprint put '$[[3]] = "NEW"' example.csv}}HERE
|
||||
POKI_RUN_COMMAND{{mlr --icsv --opprint put '$[[[3]]] = "NEW"' example.csv}}HERE
|
||||
|
||||
<p/><boldmaroon> JSON output:</boldmaroon>
|
||||
<p class="boldmaroon"> JSON output:</p>
|
||||
|
||||
POKI_RUN_COMMAND{{mlr --icsv --ojson put '$ratio = $quantity/$rate; $shape = toupper($shape)' example.csv}}HERE
|
||||
|
||||
<p/><boldmaroon> JSON output with vertical-formatting flags:</boldmaroon>
|
||||
<p class="boldmaroon"> JSON output with vertical-formatting flags:</p>
|
||||
|
||||
POKI_RUN_COMMAND{{mlr --icsv --ojson --jvstack --jlistwrap tail -n 2 example.csv}}HERE
|
||||
|
||||
<p/><boldmaroon> Use <code>then</code> to pipe commands together. Also, the
|
||||
<p class="boldmaroon"> Use <code>then</code> to pipe commands together. Also, the
|
||||
<code>-g</code> option for many Miller commands is for group-by: here, <code>head -n
|
||||
1 -g shape</code> outputs the first record for each distinct value of the
|
||||
<code>shape</code> field. This means we’re finding the record with highest
|
||||
<code>index</code> field for each distinct <code>shape</code> field:</boldmaroon>
|
||||
<code>index</code> field for each distinct <code>shape</code> field:</p>
|
||||
|
||||
POKI_RUN_COMMAND{{mlr --icsv --opprint sort -f shape -nr index then head -n 1 -g shape example.csv}}HERE
|
||||
|
||||
<p/><boldmaroon> Statistics can be computed with or without group-by field(s). Also, the first of these two
|
||||
<p class="boldmaroon"> Statistics can be computed with or without group-by field(s). Also, the first of these two
|
||||
examples uses <code>--oxtab</code> output format which is a nice alternative to <code>--opprint</code> when you
|
||||
have lots of columns:</boldmaroon>
|
||||
have lots of columns:</p>
|
||||
|
||||
POKI_RUN_COMMAND{{mlr --icsv --oxtab --from example.csv stats1 -a p0,p10,p25,p50,p75,p90,p99,p100 -f rate}}HERE
|
||||
POKI_RUN_COMMAND{{mlr --icsv --opprint --from example.csv stats1 -a count,min,mean,max -f quantity -g shape}}HERE
|
||||
|
|
@ -89,11 +89,11 @@ POKI_RUN_COMMAND{{mlr --icsv --opprint --from example.csv stats1 -a count,min,me
|
|||
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="bodyToggler.togglebodyToggler.toggle('body_section_toggle_printing_to_files');" href="javascript:;">Toggle section visibility</button>
|
||||
<div id="body_section_toggle_printing_to_files" style="display: block">
|
||||
|
||||
<p/>Often we want to print output <boldmaroon>to the screen</boldmaroon>. Miller does this by default, as we’ve
|
||||
seen in the previous examples.
|
||||
<p>Often we want to print output <span class="boldmaroon">to the screen</span>. Miller does this by default, as we’ve
|
||||
seen in the previous examples.</p>
|
||||
|
||||
<p/> Sometimes we want to print output to another file: <boldmaroon>just use '>
|
||||
outputfilenamegoeshere'</boldmaroon> at the end of your command:
|
||||
<p/> Sometimes we want to print output to another file: <span class="boldmaroon">just use '>
|
||||
outputfilenamegoeshere'</span> at the end of your command:</p>
|
||||
|
||||
<table><tr><td>
|
||||
<div class="pokipanel"> <pre>
|
||||
|
|
@ -118,7 +118,7 @@ purple square 0 91 72.3735 8.2430
|
|||
</pre> </div>
|
||||
</td></tr></table>
|
||||
|
||||
<p/> Other times we just want our files to be changed in-place: <boldmaroon>just use 'mlr -I'</boldmaroon>.
|
||||
<p/> Other times we just want our files to be changed in-place: <span class="boldmaroon">just use 'mlr -I'</span>.</p>
|
||||
|
||||
<table><tr><td>
|
||||
<div class="pokipanel"> <pre>
|
||||
|
|
@ -166,8 +166,8 @@ mlr -I --csv cut -x -f unwanted_column_name *.csv
|
|||
|
||||
<p/> If you like, you can first copy off your original data somewhere else, before doing in-place operations.
|
||||
|
||||
<p/><boldmaroon> Lastly, using <code>tee</code> within <code>put</code>, you can split your input data into separate files
|
||||
per one or more field names:</boldmaroon>
|
||||
<p class="boldmaroon"> Lastly, using <code>tee</code> within <code>put</code>, you can split your input data into separate files
|
||||
per one or more field names:</p>
|
||||
|
||||
POKI_RUN_COMMAND{{mlr --csv --from example.csv put -q 'tee > $shape.".csv", $*'}}HERE
|
||||
<table><tr><td>
|
||||
|
|
@ -207,7 +207,7 @@ shape=square,flag=0,index=36
|
|||
</pre>
|
||||
</div>
|
||||
|
||||
<p/>Data written this way are called <boldmaroon>DKVP</boldmaroon>, for <i>delimited key-value pairs</i>.
|
||||
<p>Data written this way are called <span class="boldmaroon">DKVP</span>, for <i>delimited key-value pairs</i>.</p>
|
||||
|
||||
<p/>We’ve also already seen other ways to write the same data:
|
||||
|
||||
|
|
@ -231,8 +231,8 @@ shape=square,flag=0,index=36 flag 1 "flag": 0,
|
|||
</pre>
|
||||
</div>
|
||||
|
||||
<p/><boldmaroon>Anything we can do with CSV input data, we can do with any
|
||||
other format input data.</boldmaroon> And you can read from one format, do any
|
||||
<p class="boldmaroon">Anything we can do with CSV input data, we can do with any
|
||||
other format input data.</p> And you can read from one format, do any
|
||||
record-processing, and output to the same format as the input, or to a
|
||||
different output format.
|
||||
|
||||
|
|
|
|||
|
|
@ -241,7 +241,7 @@ other 21st-century compilers)
|
|||
|
||||
<h2>Optional external dependencies</h2>
|
||||
|
||||
This documentation pageset is built using <boldmaroon>Poki</boldmaroon>:
|
||||
This documentation pageset is built using <span class="boldmaroon">Poki</span>:
|
||||
<a href="http://johnkerl.org/poki/doc">docs here</a>,
|
||||
<a href="https://github.com/johnkerl/poki">source code here</a>.
|
||||
Note that <a href="http://johnkerl.org/miller/doc/index.html">http://johnkerl.org/miller/doc/index.html</a>
|
||||
|
|
|
|||
|
|
@ -373,14 +373,14 @@ POKI_RUN_COMMAND{{mlr --csv join --ul -j id -f data/color-codes.csv then unspars
|
|||
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="bodyToggler.toggle('body_section_toggle_xml_or_json');" href="javascript:;">Toggle section visibility</button>
|
||||
<div id="body_section_toggle_xml_or_json" style="display: block">
|
||||
|
||||
<p/>Miller handles <boldmaroon>tabular data</boldmaroon>, which is a list of
|
||||
<p/>Miller handles <span class="boldmaroon">tabular data</span>, which is a list of
|
||||
records each having fields which are key-value pairs. Miller also doesn’t
|
||||
require that each record have the same field names (see also <a
|
||||
href="record-heterogeneity.html">here</a>). Regardless, tabular data is a
|
||||
<boldmaroon>non-recursive data structure</boldmaroon>.
|
||||
<span class="boldmaroon">non-recursive data structure</span>.
|
||||
|
||||
<p/> XML, JSON, etc. are, by contrast, all <boldmaroon>recursive</boldmaroon>
|
||||
or <boldmaroon>nested</boldmaroon> data structures. For example, in JSON
|
||||
<p/> XML, JSON, etc. are, by contrast, all <span class="boldmaroon">recursive</span>
|
||||
or <span class="boldmaroon">nested</span> data structures. For example, in JSON
|
||||
you can represent a hash map whose values are lists of lists.
|
||||
|
||||
<p/>Now, you can put tabular data into these formats — since list-of-key-value-pairs
|
||||
|
|
|
|||
|
|
@ -179,7 +179,7 @@ POKI_RUN_COMMAND{{mlr --nidx --fs ' ' --repifs cut -f 2,3 data/mydata.txt}}HERE
|
|||
|
||||
<p/>JSON is a format which supports arbitrarily deep nesting of
|
||||
“objects” (hashmaps) and “arrays” (lists), while Miller
|
||||
is a tool for handling <boldmaroon>tabular data</boldmaroon> only. This means
|
||||
is a tool for handling <span class="boldmaroon">tabular data</span> only. This means
|
||||
Miller cannot (and should not) handle arbitrary JSON. (Check out <a
|
||||
href="http://stedolan.github.io/jq/">jq</a>.)
|
||||
|
||||
|
|
@ -187,8 +187,8 @@ href="http://stedolan.github.io/jq/">jq</a>.)
|
|||
|
||||
<h2>Single-level JSON objects</h2>
|
||||
|
||||
An <boldmaroon>array of single-level objects</boldmaroon> is, quite simply,
|
||||
<boldmaroon>a table:</boldmaroon>
|
||||
An <span class="boldmaroon">array of single-level objects</span> is, quite simply,
|
||||
<span class="boldmaroon">a table:</span>
|
||||
|
||||
POKI_RUN_COMMAND{{mlr --json head -n 2 then cut -f color,shape data/json-example-1.json}}HERE
|
||||
POKI_RUN_COMMAND{{mlr --json --jvstack head -n 2 then cut -f color,u,v data/json-example-1.json}}HERE
|
||||
|
|
@ -196,7 +196,7 @@ POKI_RUN_COMMAND{{mlr --ijson --opprint stats1 -a mean,stddev,count -f u -g shap
|
|||
|
||||
<h2>Nested JSON objects</h2>
|
||||
|
||||
<p/>Additionally, Miller can <boldmaroon>tabularize nested objects by concatentating keys:</boldmaroon>
|
||||
<p/>Additionally, Miller can <span class="boldmaroon">tabularize nested objects by concatentating keys:</span>
|
||||
|
||||
POKI_RUN_COMMAND{{mlr --json --jvstack head -n 2 data/json-example-2.json}}HERE
|
||||
POKI_RUN_COMMAND{{mlr --ijson --opprint head -n 4 data/json-example-2.json}}HERE
|
||||
|
|
@ -358,15 +358,15 @@ POKI_RUN_COMMAND{{mlr --usage-format-conversion-keystroke-saver-options}}HERE
|
|||
<div id="body_section_toggle_autodetect_line_endings" style="display: block">
|
||||
|
||||
<p/> Default line endings (<code>--irs</code> and <code>--ors</code>) are <code>'auto'</code>
|
||||
which means <boldmaroon>autodetect from the input file format</boldmaroon>, as
|
||||
which means <span class="boldmaroon">autodetect from the input file format</span>, as
|
||||
long as the input file(s) have lines ending in either LF (also known as
|
||||
linefeed, <code>'\n'</code>, <code>0x0a</code>, Unix-style) or CRLF (also known as
|
||||
carriage-return/linefeed pairs, <code>'\r\n'</code>, <code>0x0d 0x0a</code>, Windows
|
||||
style).
|
||||
|
||||
<p/> <boldmaroon>If both IRS and ORS are auto (which is the default) then LF input will
|
||||
<p/> <span class="boldmaroon">If both IRS and ORS are auto (which is the default) then LF input will
|
||||
lead to LF output and CRLF input will lead to CRLF output, regardless of the
|
||||
platform you’re running on</boldmaroon>.
|
||||
platform you’re running on</span>.
|
||||
|
||||
<p/> The line-ending autodetector triggers on the first line ending detected in
|
||||
the input stream. E.g. if you specify a CRLF-terminated file on the command
|
||||
|
|
|
|||
|
|
@ -6,7 +6,7 @@ online-analytical-processing culture. Other key participants include
|
|||
concept, Miller explicitly strives to imitate several existing tools:
|
||||
|
||||
<p/>
|
||||
<boldmaroon>Unix toolkit</boldmaroon>: Intentional similarities as described in
|
||||
<span class="boldmaroon">Unix toolkit</span>: Intentional similarities as described in
|
||||
POKI_PUT_LINK_FOR_PAGE(feature-comparison.html)HERE.
|
||||
|
||||
<p/>Recipes abound for command-line data analysis using the Unix toolkit. Here are just a couple of my favorites:
|
||||
|
|
@ -16,7 +16,7 @@ POKI_PUT_LINK_FOR_PAGE(feature-comparison.html)HERE.
|
|||
<li/> <a href="https://github.com/dbohdan/structured-text-tools">https://github.com/dbohdan/structured-text-tools</a>
|
||||
</ul>
|
||||
|
||||
<p/> <boldmaroon>RecordStream</boldmaroon>: Miller owes particular inspiration
|
||||
<p/> <span class="boldmaroon">RecordStream</span>: Miller owes particular inspiration
|
||||
to <a href="https://github.com/benbernard/RecordStream">RecordStream</a>. The
|
||||
key difference is that RecordStream is a Perl-based tool for manipulating JSON
|
||||
(including requiring it to separately manipulate other formats such as CSV into
|
||||
|
|
@ -25,15 +25,15 @@ The similarities include the <code>sort</code>, <code>stats1</code> (analog of
|
|||
RecordStream’s <code>collate</code>), and <code>delta</code> operations, as well
|
||||
as <code>filter</code> and <code>put</code>, and pretty-print formatting.
|
||||
|
||||
<p/> <boldmaroon>stats_m</boldmaroon>: A third source of lineage is my Python
|
||||
<p/> <span class="boldmaroon">stats_m</span>: A third source of lineage is my Python
|
||||
<a href="https://github.com/johnkerl/scripts-math/tree/master/stats">stats_m</a>
|
||||
module. This includes simple single-pass algorithms which form Miller’s
|
||||
<code>stats1</code> and <code>stats2</code> subcommands.
|
||||
|
||||
<p/> <boldmaroon>SQL</boldmaroon>: Fourthly, Miller’s <code>group-by</code> command
|
||||
<p/> <span class="boldmaroon">SQL</span>: Fourthly, Miller’s <code>group-by</code> command
|
||||
name is from SQL, as is the term <code>aggregate</code>.
|
||||
|
||||
<p/> <boldmaroon>Added value</boldmaroon>:
|
||||
<p/> <span class="boldmaroon">Added value</span>:
|
||||
Miller’s added values include:
|
||||
<ul>
|
||||
<li> Name-indexing, compared to the Unix toolkit’s positional indexing.
|
||||
|
|
@ -43,17 +43,17 @@ Miller’s added values include:
|
|||
<li> Various file formats, and on-the-fly format conversion.
|
||||
</ul>
|
||||
|
||||
<p/><boldmaroon>jq</boldmaroon>: Miller does for name-indexed text what
|
||||
<p/><span class="boldmaroon">jq</span>: Miller does for name-indexed text what
|
||||
<a href="http://stedolan.github.io/jq/">jq</a> does for JSON. If you’re
|
||||
not already familiar with <code>jq</code>, please <a href="http://stedolan.github.io/jq/">check it out!</a>.
|
||||
|
||||
<p/><boldmaroon>What about similar tools?</boldmaroon>
|
||||
<p/><span class="boldmaroon">What about similar tools?</span>
|
||||
Here’s a comprehensive list:
|
||||
<a href="https://github.com/dbohdan/structured-text-tools">https://github.com/dbohdan/structured-text-tools</a>.
|
||||
It doesn’t mention <a href="https://github.com/turicas/rows">rows</a> so here’s a plug for that as well.
|
||||
As it turns out, I learned about most of these after writing Miller.
|
||||
|
||||
<p/><boldmaroon>What about DOTADIW?</boldmaroon> One of the key points of the
|
||||
<p/><span class="boldmaroon">What about DOTADIW?</span> One of the key points of the
|
||||
<a href="http://en.wikipedia.org/wiki/Unix_philosophy">Unix philosophy</a> is
|
||||
that a tool should do one thing and do it well. Hence <code>sort</code> and
|
||||
<code>cut</code> do just one thing. Why does Miller put <code>awk</code>-like
|
||||
|
|
@ -70,7 +70,7 @@ be a tool which collects together format-aware record-stream processing into
|
|||
one place, with good reuse of Miller-internal library code for its various
|
||||
features.
|
||||
|
||||
<p/><boldmaroon>Why not use Perl/Python/Ruby etc.?</boldmaroon> Maybe you
|
||||
<p/><span class="boldmaroon">Why not use Perl/Python/Ruby etc.?</span> Maybe you
|
||||
should. With those tools you’ll get far more expressive power, and
|
||||
sufficiently quick turnaround time for small-to-medium-sized data. Using
|
||||
Miller you’ll get something less than a complete programming language,
|
||||
|
|
@ -88,8 +88,8 @@ If I’d gotten good enough performance from the latter I’d have done
|
|||
it without question and Miller would be far more flexible. But C won the
|
||||
performance criteria by a landslide so we have Miller in C with a custom DSL.
|
||||
|
||||
<p/> <boldmaroon>No, really, why one more command-line data-manipulation
|
||||
tool?</boldmaroon> I wrote Miller because I was frustrated with tools like
|
||||
<p/> <span class="boldmaroon">No, really, why one more command-line data-manipulation
|
||||
tool?</span> I wrote Miller because I was frustrated with tools like
|
||||
<code>grep</code>, <code>sed</code>, and so on being <i>line-aware</i> without being
|
||||
<i>format-aware</i>. The single most poignant example I can think of is seeing
|
||||
people grep data lines out of their CSV files and sadly losing their header
|
||||
|
|
|
|||
|
|
@ -13,15 +13,15 @@ why I felt it necessary to build Miller, etc. Here are some answers.
|
|||
|
||||
<p/> For background, I’m a software engineer, with a heavy devops bent
|
||||
and a non-trivial amount of data-engineering in my career.
|
||||
<boldmaroon>Initially I wrote Miller mainly for myself:</boldmaroon> I’m
|
||||
<span class="boldmaroon">Initially I wrote Miller mainly for myself:</span> I’m
|
||||
coder-friendly (being a coder); I’m Github-friendly; most of my data are
|
||||
well-structured or easily structurable (TSV-formatted SQL-query output, CSV
|
||||
files, log files, JSON data structures); I care about interoperability between
|
||||
all the various formats Miller supports (I’ve encountered them all); I do
|
||||
all my work on Linux or OSX.
|
||||
|
||||
<p/> But now there’s this neat little tool <boldmaroon>which seems to be
|
||||
useful for people in various disciplines</boldmaroon>. I don’t even know
|
||||
<p/> But now there’s this neat little tool <span class="boldmaroon">which seems to be
|
||||
useful for people in various disciplines</span>. I don’t even know
|
||||
entirely <i>who</i>. I can click through Github starrers and read a bit about
|
||||
what they seem to do, but not everyone’s <i>on</i> Github (or stars
|
||||
things). I’ve gotten a lot of feature requests through Github — but
|
||||
|
|
@ -50,7 +50,7 @@ it’s for?
|
|||
years of my career in the software industry I’ve found myself, and
|
||||
others, doing a lot of ad-hoc things which really were fundamentally the same
|
||||
<i>except</i> for format. So the number one thing about Miller is doing common
|
||||
things while supporting <boldmaroon>multiple formats</boldmaroon>: (a) ingest a
|
||||
things while supporting <span class="boldmaroon">multiple formats</span>: (a) ingest a
|
||||
list of records where a record is a list of key-value pairs (however
|
||||
represented in the input files); (b) transform that stream of records; (c) emit
|
||||
the transformed stream — either in the same format as input, or in a
|
||||
|
|
@ -60,9 +60,9 @@ different format.
|
|||
something only for a single file format, I didn’t want to build something
|
||||
only for one problem domain. In my work doing software engineering, devops,
|
||||
data engineering, etc. I saw a lot of commonalities and I wanted to
|
||||
<boldmaroon>solve as many problems simultaneously as possible</boldmaroon>.
|
||||
<span class="boldmaroon">solve as many problems simultaneously as possible</span>.
|
||||
|
||||
<p/> Third: it had to be <boldmaroon>streaming</boldmaroon>. As time goes by
|
||||
<p/> Third: it had to be <span class="boldmaroon">streaming</span>. As time goes by
|
||||
and we (some of us, sometimes) have machines with tens or hundreds of GB of
|
||||
RAM, it’s maybe less important, but I’m unhappy with tools which
|
||||
ingest all data, then do stuff, then emit all data. One reason is to be able to
|
||||
|
|
@ -71,7 +71,7 @@ input which trickles in, e.g. you have some process emitting data now and then
|
|||
and you can pipe it to Miller and it will emit transformed records one at a
|
||||
time.
|
||||
|
||||
<p/> Fourth: it had to be <boldmaroon>fast</boldmaroon>. This precludes all
|
||||
<p/> Fourth: it had to be <span class="boldmaroon">fast</span>. This precludes all
|
||||
sorts of very nice things written in Ruby, for example. I love Ruby as a very
|
||||
expressive language, and I have several very useful little utility scripts
|
||||
written in Ruby. But a few years ago I ported over some of my old
|
||||
|
|
@ -83,8 +83,8 @@ in order to make it performant. I did simple experiments in several languages,
|
|||
and nothing was as fast as C, so I used C: see also <a
|
||||
href="whyc.html#C_vs._Go,_D,_Rust,_etc.;_C_is_fast">here</a>.
|
||||
|
||||
<p/> Fifth thing: I wanted Miller to be <boldmaroon>pipe-friendly and
|
||||
interoperate with other command-line tools</boldmaroon>. Since the basic
|
||||
<p/> Fifth thing: I wanted Miller to be <span class="boldmaroon">pipe-friendly and
|
||||
interoperate with other command-line tools</span>. Since the basic
|
||||
paradigm is ingest records, transform records, emit records — where the
|
||||
input and output formats can be the same or different, and the transform can be
|
||||
complex, or just pass-through — this means you can use it to transform
|
||||
|
|
@ -93,17 +93,17 @@ data-cleaning/prep/formatting and do all the "real" work in R, you can. If you
|
|||
just want a little glue script between other tools you can get that. And if you
|
||||
want to do non-trivial data-reduction in Miller you can.
|
||||
|
||||
<p/> Sixth thing: Must have <boldmaroon>comprehensive documentation and
|
||||
unit-test</boldmaroon>. Since Miller handles a lot of formats and solves a lot
|
||||
<p/> Sixth thing: Must have <span class="boldmaroon">comprehensive documentation and
|
||||
unit-test</span>. Since Miller handles a lot of formats and solves a lot
|
||||
of problems, there’s a lot to test and a lot to keep working correctly as
|
||||
I add features or optimize. And I wanted it to be able to explain itself
|
||||
— not only through web docs like the one you’re reading but also
|
||||
through <code>man mlr</code> and <code>mlr --help</code>, <code>mlr sort --help</code>,
|
||||
etc.
|
||||
|
||||
<p/> Seventh thing: <boldmaroon>Must have a domain-specific
|
||||
language</boldmaroon> (DSL) <boldmaroon>but also must let you do common things
|
||||
without it</boldmaroon>. All those little verbs Miller has to help you
|
||||
<p/> Seventh thing: <span class="boldmaroon">Must have a domain-specific
|
||||
language</span> (DSL) <span class="boldmaroon">but also must let you do common things
|
||||
without it</span>. All those little verbs Miller has to help you
|
||||
<i>avoid</i> having to write for-loops are great. I use them for
|
||||
keystroke-saving: <code>mlr stats1 -a mean,stddev,min,max -f quantity</code>, for
|
||||
example, without you having to write for-loops or define accumulator variables.
|
||||
|
|
@ -112,8 +112,8 @@ you want to: <code>mlr put '$distance = $rate * $time'</code> or anything else y
|
|||
can think up. In Perl/AWK/etc. it’s all DSL. In xsv et al. it’s
|
||||
all verbs. In Miller I like having the combination.
|
||||
|
||||
<p/> Eighth thing: It’s an <boldmaroon>awful lot of fun to
|
||||
write</boldmaroon>. In my experience I didn’t find any tools which do
|
||||
<p/> Eighth thing: It’s an <span class="boldmaroon">awful lot of fun to
|
||||
write</span>. In my experience I didn’t find any tools which do
|
||||
multi-format, streaming, efficient, multi-purpose, with DSL and non-DSL, so I
|
||||
wrote one. But I don’t guarantee it’s unique in the world. It fills
|
||||
a niche in the world (people use it) but it also fills a niche in my life.
|
||||
|
|
|
|||
|
|
@ -14,10 +14,10 @@ body {
|
|||
background-color: white;
|
||||
}
|
||||
|
||||
maroon {
|
||||
.maroon {
|
||||
color: maroon;
|
||||
}
|
||||
boldmaroon {
|
||||
.boldmaroon {
|
||||
font-weight: bold;
|
||||
color: maroon;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -995,14 +995,14 @@ id,code,color
|
|||
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="bodyToggler.toggle('body_section_toggle_xml_or_json');" href="javascript:;">Toggle section visibility</button>
|
||||
<div id="body_section_toggle_xml_or_json" style="display: block">
|
||||
|
||||
<p/>Miller handles <boldmaroon>tabular data</boldmaroon>, which is a list of
|
||||
<p/>Miller handles <span class="boldmaroon">tabular data</span>, which is a list of
|
||||
records each having fields which are key-value pairs. Miller also doesn’t
|
||||
require that each record have the same field names (see also <a
|
||||
href="record-heterogeneity.html">here</a>). Regardless, tabular data is a
|
||||
<boldmaroon>non-recursive data structure</boldmaroon>.
|
||||
<span class="boldmaroon">non-recursive data structure</span>.
|
||||
|
||||
<p/> XML, JSON, etc. are, by contrast, all <boldmaroon>recursive</boldmaroon>
|
||||
or <boldmaroon>nested</boldmaroon> data structures. For example, in JSON
|
||||
<p/> XML, JSON, etc. are, by contrast, all <span class="boldmaroon">recursive</span>
|
||||
or <span class="boldmaroon">nested</span> data structures. For example, in JSON
|
||||
you can represent a hash map whose values are lists of lists.
|
||||
|
||||
<p/>Now, you can put tabular data into these formats — since list-of-key-value-pairs
|
||||
|
|
|
|||
|
|
@ -420,7 +420,7 @@ light
|
|||
|
||||
<p/>JSON is a format which supports arbitrarily deep nesting of
|
||||
“objects” (hashmaps) and “arrays” (lists), while Miller
|
||||
is a tool for handling <boldmaroon>tabular data</boldmaroon> only. This means
|
||||
is a tool for handling <span class="boldmaroon">tabular data</span> only. This means
|
||||
Miller cannot (and should not) handle arbitrary JSON. (Check out <a
|
||||
href="http://stedolan.github.io/jq/">jq</a>.)
|
||||
|
||||
|
|
@ -428,8 +428,8 @@ href="http://stedolan.github.io/jq/">jq</a>.)
|
|||
|
||||
<a id="Single-level_JSON_objects"/><h2>Single-level JSON objects</h2>
|
||||
|
||||
An <boldmaroon>array of single-level objects</boldmaroon> is, quite simply,
|
||||
<boldmaroon>a table:</boldmaroon>
|
||||
An <span class="boldmaroon">array of single-level objects</span> is, quite simply,
|
||||
<span class="boldmaroon">a table:</span>
|
||||
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
|
|
@ -471,7 +471,7 @@ circle 0.366013 0.209094 3
|
|||
|
||||
<a id="Nested_JSON_objects"/><h2>Nested JSON objects</h2>
|
||||
|
||||
<p/>Additionally, Miller can <boldmaroon>tabularize nested objects by concatentating keys:</boldmaroon>
|
||||
<p/>Additionally, Miller can <span class="boldmaroon">tabularize nested objects by concatentating keys:</span>
|
||||
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
|
|
@ -878,15 +878,15 @@ output only.
|
|||
<div id="body_section_toggle_autodetect_line_endings" style="display: block">
|
||||
|
||||
<p/> Default line endings (<code>--irs</code> and <code>--ors</code>) are <code>'auto'</code>
|
||||
which means <boldmaroon>autodetect from the input file format</boldmaroon>, as
|
||||
which means <span class="boldmaroon">autodetect from the input file format</span>, as
|
||||
long as the input file(s) have lines ending in either LF (also known as
|
||||
linefeed, <code>'\n'</code>, <code>0x0a</code>, Unix-style) or CRLF (also known as
|
||||
carriage-return/linefeed pairs, <code>'\r\n'</code>, <code>0x0d 0x0a</code>, Windows
|
||||
style).
|
||||
|
||||
<p/> <boldmaroon>If both IRS and ORS are auto (which is the default) then LF input will
|
||||
<p/> <span class="boldmaroon">If both IRS and ORS are auto (which is the default) then LF input will
|
||||
lead to LF output and CRLF input will lead to CRLF output, regardless of the
|
||||
platform you’re running on</boldmaroon>.
|
||||
platform you’re running on</span>.
|
||||
|
||||
<p/> The line-ending autodetector triggers on the first line ending detected in
|
||||
the input stream. E.g. if you specify a CRLF-terminated file on the command
|
||||
|
|
|
|||
|
|
@ -72,7 +72,7 @@ online-analytical-processing culture. Other key participants include
|
|||
concept, Miller explicitly strives to imitate several existing tools:
|
||||
|
||||
<p/>
|
||||
<boldmaroon>Unix toolkit</boldmaroon>: Intentional similarities as described in
|
||||
<span class="boldmaroon">Unix toolkit</span>: Intentional similarities as described in
|
||||
<a href="feature-comparison.html">Unix-toolkit context</a>.
|
||||
|
||||
<p/>Recipes abound for command-line data analysis using the Unix toolkit. Here are just a couple of my favorites:
|
||||
|
|
@ -82,7 +82,7 @@ concept, Miller explicitly strives to imitate several existing tools:
|
|||
<li/> <a href="https://github.com/dbohdan/structured-text-tools">https://github.com/dbohdan/structured-text-tools</a>
|
||||
</ul>
|
||||
|
||||
<p/> <boldmaroon>RecordStream</boldmaroon>: Miller owes particular inspiration
|
||||
<p/> <span class="boldmaroon">RecordStream</span>: Miller owes particular inspiration
|
||||
to <a href="https://github.com/benbernard/RecordStream">RecordStream</a>. The
|
||||
key difference is that RecordStream is a Perl-based tool for manipulating JSON
|
||||
(including requiring it to separately manipulate other formats such as CSV into
|
||||
|
|
@ -91,15 +91,15 @@ The similarities include the <code>sort</code>, <code>stats1</code> (analog of
|
|||
RecordStream’s <code>collate</code>), and <code>delta</code> operations, as well
|
||||
as <code>filter</code> and <code>put</code>, and pretty-print formatting.
|
||||
|
||||
<p/> <boldmaroon>stats_m</boldmaroon>: A third source of lineage is my Python
|
||||
<p/> <span class="boldmaroon">stats_m</span>: A third source of lineage is my Python
|
||||
<a href="https://github.com/johnkerl/scripts-math/tree/master/stats">stats_m</a>
|
||||
module. This includes simple single-pass algorithms which form Miller’s
|
||||
<code>stats1</code> and <code>stats2</code> subcommands.
|
||||
|
||||
<p/> <boldmaroon>SQL</boldmaroon>: Fourthly, Miller’s <code>group-by</code> command
|
||||
<p/> <span class="boldmaroon">SQL</span>: Fourthly, Miller’s <code>group-by</code> command
|
||||
name is from SQL, as is the term <code>aggregate</code>.
|
||||
|
||||
<p/> <boldmaroon>Added value</boldmaroon>:
|
||||
<p/> <span class="boldmaroon">Added value</span>:
|
||||
Miller’s added values include:
|
||||
<ul>
|
||||
<li> Name-indexing, compared to the Unix toolkit’s positional indexing.
|
||||
|
|
@ -109,17 +109,17 @@ Miller’s added values include:
|
|||
<li> Various file formats, and on-the-fly format conversion.
|
||||
</ul>
|
||||
|
||||
<p/><boldmaroon>jq</boldmaroon>: Miller does for name-indexed text what
|
||||
<p/><span class="boldmaroon">jq</span>: Miller does for name-indexed text what
|
||||
<a href="http://stedolan.github.io/jq/">jq</a> does for JSON. If you’re
|
||||
not already familiar with <code>jq</code>, please <a href="http://stedolan.github.io/jq/">check it out!</a>.
|
||||
|
||||
<p/><boldmaroon>What about similar tools?</boldmaroon>
|
||||
<p/><span class="boldmaroon">What about similar tools?</span>
|
||||
Here’s a comprehensive list:
|
||||
<a href="https://github.com/dbohdan/structured-text-tools">https://github.com/dbohdan/structured-text-tools</a>.
|
||||
It doesn’t mention <a href="https://github.com/turicas/rows">rows</a> so here’s a plug for that as well.
|
||||
As it turns out, I learned about most of these after writing Miller.
|
||||
|
||||
<p/><boldmaroon>What about DOTADIW?</boldmaroon> One of the key points of the
|
||||
<p/><span class="boldmaroon">What about DOTADIW?</span> One of the key points of the
|
||||
<a href="http://en.wikipedia.org/wiki/Unix_philosophy">Unix philosophy</a> is
|
||||
that a tool should do one thing and do it well. Hence <code>sort</code> and
|
||||
<code>cut</code> do just one thing. Why does Miller put <code>awk</code>-like
|
||||
|
|
@ -136,7 +136,7 @@ be a tool which collects together format-aware record-stream processing into
|
|||
one place, with good reuse of Miller-internal library code for its various
|
||||
features.
|
||||
|
||||
<p/><boldmaroon>Why not use Perl/Python/Ruby etc.?</boldmaroon> Maybe you
|
||||
<p/><span class="boldmaroon">Why not use Perl/Python/Ruby etc.?</span> Maybe you
|
||||
should. With those tools you’ll get far more expressive power, and
|
||||
sufficiently quick turnaround time for small-to-medium-sized data. Using
|
||||
Miller you’ll get something less than a complete programming language,
|
||||
|
|
@ -154,8 +154,8 @@ If I’d gotten good enough performance from the latter I’d have done
|
|||
it without question and Miller would be far more flexible. But C won the
|
||||
performance criteria by a landslide so we have Miller in C with a custom DSL.
|
||||
|
||||
<p/> <boldmaroon>No, really, why one more command-line data-manipulation
|
||||
tool?</boldmaroon> I wrote Miller because I was frustrated with tools like
|
||||
<p/> <span class="boldmaroon">No, really, why one more command-line data-manipulation
|
||||
tool?</span> I wrote Miller because I was frustrated with tools like
|
||||
<code>grep</code>, <code>sed</code>, and so on being <i>line-aware</i> without being
|
||||
<i>format-aware</i>. The single most poignant example I can think of is seeing
|
||||
people grep data lines out of their CSV files and sadly losing their header
|
||||
|
|
|
|||
32
doc/why.html
32
doc/why.html
|
|
@ -84,15 +84,15 @@ why I felt it necessary to build Miller, etc. Here are some answers.
|
|||
|
||||
<p/> For background, I’m a software engineer, with a heavy devops bent
|
||||
and a non-trivial amount of data-engineering in my career.
|
||||
<boldmaroon>Initially I wrote Miller mainly for myself:</boldmaroon> I’m
|
||||
<span class="boldmaroon">Initially I wrote Miller mainly for myself:</span> I’m
|
||||
coder-friendly (being a coder); I’m Github-friendly; most of my data are
|
||||
well-structured or easily structurable (TSV-formatted SQL-query output, CSV
|
||||
files, log files, JSON data structures); I care about interoperability between
|
||||
all the various formats Miller supports (I’ve encountered them all); I do
|
||||
all my work on Linux or OSX.
|
||||
|
||||
<p/> But now there’s this neat little tool <boldmaroon>which seems to be
|
||||
useful for people in various disciplines</boldmaroon>. I don’t even know
|
||||
<p/> But now there’s this neat little tool <span class="boldmaroon">which seems to be
|
||||
useful for people in various disciplines</span>. I don’t even know
|
||||
entirely <i>who</i>. I can click through Github starrers and read a bit about
|
||||
what they seem to do, but not everyone’s <i>on</i> Github (or stars
|
||||
things). I’ve gotten a lot of feature requests through Github — but
|
||||
|
|
@ -121,7 +121,7 @@ it’s for?
|
|||
years of my career in the software industry I’ve found myself, and
|
||||
others, doing a lot of ad-hoc things which really were fundamentally the same
|
||||
<i>except</i> for format. So the number one thing about Miller is doing common
|
||||
things while supporting <boldmaroon>multiple formats</boldmaroon>: (a) ingest a
|
||||
things while supporting <span class="boldmaroon">multiple formats</span>: (a) ingest a
|
||||
list of records where a record is a list of key-value pairs (however
|
||||
represented in the input files); (b) transform that stream of records; (c) emit
|
||||
the transformed stream — either in the same format as input, or in a
|
||||
|
|
@ -131,9 +131,9 @@ different format.
|
|||
something only for a single file format, I didn’t want to build something
|
||||
only for one problem domain. In my work doing software engineering, devops,
|
||||
data engineering, etc. I saw a lot of commonalities and I wanted to
|
||||
<boldmaroon>solve as many problems simultaneously as possible</boldmaroon>.
|
||||
<span class="boldmaroon">solve as many problems simultaneously as possible</span>.
|
||||
|
||||
<p/> Third: it had to be <boldmaroon>streaming</boldmaroon>. As time goes by
|
||||
<p/> Third: it had to be <span class="boldmaroon">streaming</span>. As time goes by
|
||||
and we (some of us, sometimes) have machines with tens or hundreds of GB of
|
||||
RAM, it’s maybe less important, but I’m unhappy with tools which
|
||||
ingest all data, then do stuff, then emit all data. One reason is to be able to
|
||||
|
|
@ -142,7 +142,7 @@ input which trickles in, e.g. you have some process emitting data now and then
|
|||
and you can pipe it to Miller and it will emit transformed records one at a
|
||||
time.
|
||||
|
||||
<p/> Fourth: it had to be <boldmaroon>fast</boldmaroon>. This precludes all
|
||||
<p/> Fourth: it had to be <span class="boldmaroon">fast</span>. This precludes all
|
||||
sorts of very nice things written in Ruby, for example. I love Ruby as a very
|
||||
expressive language, and I have several very useful little utility scripts
|
||||
written in Ruby. But a few years ago I ported over some of my old
|
||||
|
|
@ -154,8 +154,8 @@ in order to make it performant. I did simple experiments in several languages,
|
|||
and nothing was as fast as C, so I used C: see also <a
|
||||
href="whyc.html#C_vs._Go,_D,_Rust,_etc.;_C_is_fast">here</a>.
|
||||
|
||||
<p/> Fifth thing: I wanted Miller to be <boldmaroon>pipe-friendly and
|
||||
interoperate with other command-line tools</boldmaroon>. Since the basic
|
||||
<p/> Fifth thing: I wanted Miller to be <span class="boldmaroon">pipe-friendly and
|
||||
interoperate with other command-line tools</span>. Since the basic
|
||||
paradigm is ingest records, transform records, emit records — where the
|
||||
input and output formats can be the same or different, and the transform can be
|
||||
complex, or just pass-through — this means you can use it to transform
|
||||
|
|
@ -164,17 +164,17 @@ data-cleaning/prep/formatting and do all the "real" work in R, you can. If you
|
|||
just want a little glue script between other tools you can get that. And if you
|
||||
want to do non-trivial data-reduction in Miller you can.
|
||||
|
||||
<p/> Sixth thing: Must have <boldmaroon>comprehensive documentation and
|
||||
unit-test</boldmaroon>. Since Miller handles a lot of formats and solves a lot
|
||||
<p/> Sixth thing: Must have <span class="boldmaroon">comprehensive documentation and
|
||||
unit-test</span>. Since Miller handles a lot of formats and solves a lot
|
||||
of problems, there’s a lot to test and a lot to keep working correctly as
|
||||
I add features or optimize. And I wanted it to be able to explain itself
|
||||
— not only through web docs like the one you’re reading but also
|
||||
through <code>man mlr</code> and <code>mlr --help</code>, <code>mlr sort --help</code>,
|
||||
etc.
|
||||
|
||||
<p/> Seventh thing: <boldmaroon>Must have a domain-specific
|
||||
language</boldmaroon> (DSL) <boldmaroon>but also must let you do common things
|
||||
without it</boldmaroon>. All those little verbs Miller has to help you
|
||||
<p/> Seventh thing: <span class="boldmaroon">Must have a domain-specific
|
||||
language</span> (DSL) <span class="boldmaroon">but also must let you do common things
|
||||
without it</span>. All those little verbs Miller has to help you
|
||||
<i>avoid</i> having to write for-loops are great. I use them for
|
||||
keystroke-saving: <code>mlr stats1 -a mean,stddev,min,max -f quantity</code>, for
|
||||
example, without you having to write for-loops or define accumulator variables.
|
||||
|
|
@ -183,8 +183,8 @@ you want to: <code>mlr put '$distance = $rate * $time'</code> or anything else y
|
|||
can think up. In Perl/AWK/etc. it’s all DSL. In xsv et al. it’s
|
||||
all verbs. In Miller I like having the combination.
|
||||
|
||||
<p/> Eighth thing: It’s an <boldmaroon>awful lot of fun to
|
||||
write</boldmaroon>. In my experience I didn’t find any tools which do
|
||||
<p/> Eighth thing: It’s an <span class="boldmaroon">awful lot of fun to
|
||||
write</span>. In my experience I didn’t find any tools which do
|
||||
multi-format, streaming, efficient, multi-purpose, with DSL and non-DSL, so I
|
||||
wrote one. But I don’t guarantee it’s unique in the world. It fills
|
||||
a niche in the world (people use it) but it also fills a niche in my life.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue