Mirrors/miller
1
0
Fork 0
mirror of https://github.com/johnkerl/miller.git synced 2026-08-02 12:32:21 +00:00
Code Issues Projects Releases Wiki Activity

10-minute neatens

Browse source
This commit is contained in:
John Kerl 2020-08-06 11:42:55 -04:00
parent 5ebd1b2b3a
commit 7bcaf2e7b0
2 changed files with 125 additions and 65 deletions
Download patch file Download diff file Expand all files Collapse all files

117
doc/10-min.html
View file

@ -64,6 +64,8 @@ pageTracker._trackPageview();
<div class="pokitoc">
<center><titleinbody>10-minute intro</titleinbody></center>
&bull;&nbsp;<a href="#CSV-file_examples">CSV-file examples</a><br/>
&bull;&nbsp;<a href="#JSON-file_examples">JSON-file examples</a><br/>
&bull;&nbsp;<a href="#Sorts_and_stats">Sorts and stats</a><br/>
&bull;&nbsp;<a href="#Choices_for_printing_to_files">Choices for printing to files</a><br/>
&bull;&nbsp;<a href="#Other-format_examples">Other-format examples</a><br/>
&bull;&nbsp;<a href="#SQL-output_examples">SQL-output examples</a><br/>
@ -81,7 +83,7 @@ pageTracker._trackPageview();
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="bodyToggler.toggle('body_section_toggle_csv_file_examples');" href="javascript:;">Toggle section visibility</button>
<div id="body_section_toggle_csv_file_examples" style="display: block">
<p class="boldmaroon"> Sample CSV data file: </p>
<p class="boldmaroon"> Suppose you have this CSV data file: </p>
<p/>
<div class="pokipanel">
@ -102,7 +104,8 @@ purple,square,0,91,72.3735,8.2430
</div>
<p/>
<p class="boldmaroon"><code>mlr cat</code> is like cat ...</p>
<p class="boldmaroon"><code>mlr cat</code> is like cat -- it passes the data
through unmodified ...</p>
<p/>
<div class="pokipanel">
@ -123,7 +126,7 @@ purple,square,0,91,72.3735,8.2430
</div>
<p/>
<p class="boldmaroon">... but it can also do format conversion (here, to pretty-printed tabular format): </p>
<p class="boldmaroon">... but it can also do format conversion (here, you can pretty-print in tabular format): </p>
<p/>
<div class="pokipanel">
@ -144,8 +147,9 @@ purple square 0 91 72.3735 8.2430
</div>
<p/>
<p class="boldmaroon"> <code>mlr head</code> and <code>mlr tail</code> count records rather than lines. The CSV
header is included either way:</p>
<p class="boldmaroon"> <code>mlr head</code> and <code>mlr tail</code> count
records rather than lines. Whethere you&rsquo;re getting the first few records
or the last few, the CSV header is included either way:</p>
<p/>
<div class="pokipanel">
@ -172,8 +176,8 @@ purple,square,0,91,72.3735,8.2430
</div>
<p/>
<p class="boldmaroon"> Sort primarily alphabetically on one field, then secondarily
numerically descending on another field: </p>
<p class="boldmaroon"> You can sort primarily alphabetically on one field, then
secondarily numerically descending on another field: </p>
<p/>
<div class="pokipanel">
@ -194,7 +198,8 @@ yellow triangle 1 11 43.6498 9.8870
</div>
<p/>
<p class="boldmaroon"> Use <code>cut</code> to retain only specified fields, in input-data order:</p>
<p class="boldmaroon"> You can use <code>cut</code> to retain only specified fields, in the
same order they appeared in the input data:</p>
<p/>
<div class="pokipanel">
@ -215,7 +220,8 @@ square 0
</div>
<p/>
<p class="boldmaroon"> Use <code>cut -o</code> to retain only specified fields, in your specified order:</p>
<p class="boldmaroon"> You can also use <code>cut -o</code> to retain only
specified fields in your preferred order:</p>
<p/>
<div class="pokipanel">
@ -236,7 +242,8 @@ flag shape
</div>
<p/>
<p class="boldmaroon"> Use <code>cut -x</code> to omit specified fields:</p>
<p class="boldmaroon"> You can use <code>cut -x</code> to omit fields you
don&rsquo;t care about:</p>
<p/>
<div class="pokipanel">
@ -257,7 +264,7 @@ purple 91 72.3735 8.2430
</div>
<p/>
<p class="boldmaroon"> Use <code>filter</code> to retain specified records:</p>
<p class="boldmaroon"> You can use <code>filter</code> to keep only records you care about:</p>
<p/>
<div class="pokipanel">
@ -282,7 +289,8 @@ red circle 1 16 13.8103 2.9010
</div>
<p/>
<p class="boldmaroon"> Use <code>put</code> to add/replace fields which are computed from other fields:</p>
<p class="boldmaroon"> You can use <code>put</code> to create new fields which
are computed from other fields:</p>
<p/>
<div class="pokipanel">
@ -345,7 +353,12 @@ purple square NEW 91 72.3735 8.2430
</div>
<p/>
<p class="boldmaroon"> JSON output:</p>
<a id="JSON-file_examples"/><h1>JSON-file examples</h1>
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="bodyToggler.toggle('body_section_toggle_json_file_examples');" href="javascript:;">Toggle section visibility</button>
<div id="body_section_toggle_json_file_examples" style="display: block">
<p class="boldmaroon"> OK, CSV and pretty-print are fine. But Miller can also convert between
a few other formats -- let&rsquo;s take a look at JSON output:</p>
<p/>
<div class="pokipanel">
@ -365,13 +378,12 @@ $ mlr --icsv --ojson put '$ratio = $quantity/$rate; $shape = toupper($shape)' ex
</div>
<p/>
<p class="boldmaroon"> JSON output with vertical-formatting flags:</p>
<p class="boldmaroon"> Or, JSON output with vertical-formatting flags:</p>
<p/>
<div class="pokipanel">
<pre>
$ mlr --icsv --ojson --jvstack --jlistwrap tail -n 2 example.csv
[
$ mlr --icsv --ojsonx tail -n 2 example.csv
{
"color": "yellow",
"shape": "circle",
@ -380,7 +392,7 @@ $ mlr --icsv --ojson --jvstack --jlistwrap tail -n 2 example.csv
"quantity": 63.5058,
"rate": 8.3350
}
,{
{
"color": "purple",
"shape": "square",
"flag": 0,
@ -388,16 +400,37 @@ $ mlr --icsv --ojson --jvstack --jlistwrap tail -n 2 example.csv
"quantity": 72.3735,
"rate": 8.2430
}
]
</pre>
</div>
<p/>
<p class="boldmaroon"> Use <code>then</code> to pipe commands together. Also, the
<code>-g</code> option for many Miller commands is for group-by: here, <code>head -n
1 -g shape</code> outputs the first record for each distinct value of the
<code>shape</code> field. This means we&rsquo;re finding the record with highest
<code>index</code> field for each distinct <code>shape</code> field:</p>
<a id="Sorts_and_stats"/><h1>Sorts and stats</h1>
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="bodyToggler.toggle('body_section_toggle_sort_and_stats_examples');" href="javascript:;">Toggle section visibility</button>
<div id="body_section_toggle_sort_and_stats_examples" style="display: block">
<p class="boldmaroon"> Now suppose you want to sort the data on a given column,
<i>and then</i> take the top few in that ordering. You can use Miller&rsquo;s
<code>then</code> feature to pipe commands together. </p>
<p class="boldmaroon"> Here are the records with the top three <code>index</code> values: </p>
<p/>
<div class="pokipanel">
<pre>
$ mlr --icsv --opprint sort -f shape -nr index then head -n 3 example.csv
color shape flag index quantity rate
yellow circle 1 87 63.5058 8.3350
yellow circle 1 73 63.9785 4.2370
red circle 1 16 13.8103 2.9010
</pre>
</div>
<p/>
<p class="boldmaroon"> Lots of Miller commands take a <code>-g</code> option
for group-by: here, <code>head -n 1 -g shape</code> outputs the first record
for each distinct value of the <code>shape</code> field. This means we&rsquo;re
finding the record with highest <code>index</code> field for each distinct
<code>shape</code> field:</p>
<p/>
<div class="pokipanel">
@ -411,25 +444,8 @@ purple triangle 0 65 80.1405 5.8240
</div>
<p/>
<p class="boldmaroon"> Statistics can be computed with or without group-by field(s). Also, the first of these two
examples uses <code>--oxtab</code> output format which is a nice alternative to <code>--opprint</code> when you
have lots of columns:</p>
<p class="boldmaroon"> Statistics can be computed with or without group-by field(s)</p>
<p/>
<div class="pokipanel">
<pre>
$ mlr --icsv --oxtab --from example.csv stats1 -a p0,p10,p25,p50,p75,p90,p99,p100 -f rate
rate_p0 0.013000
rate_p10 2.901000
rate_p25 4.237000
rate_p50 8.243000
rate_p75 8.591000
rate_p90 9.887000
rate_p99 9.887000
rate_p100 9.887000
</pre>
</div>
<p/>
<p/>
<div class="pokipanel">
<pre>
@ -456,6 +472,25 @@ square purple 1 72.373500 72.373500 72.373500
</div>
<p/>
<p class="boldmaroon"> If your output has a lot of columns, you can use XTAB
format to line things up vertically for you instead: </p>
<p/>
<div class="pokipanel">
<pre>
$ mlr --icsv --oxtab --from example.csv stats1 -a p0,p10,p25,p50,p75,p90,p99,p100 -f rate
rate_p0 0.013000
rate_p10 2.901000
rate_p25 4.237000
rate_p50 8.243000
rate_p75 8.591000
rate_p90 9.887000
rate_p99 9.887000
rate_p100 9.887000
</pre>
</div>
<p/>
</div>
<a id="Choices_for_printing_to_files"/><h1>Choices for printing to files</h1>
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="bodyToggler.togglebodyToggler.toggle('body_section_toggle_printing_to_files');" href="javascript:;">Toggle section visibility</button>

73
doc/content-for-10-min.html
View file

@ -8,47 +8,53 @@ POKI_PUT_TOC_HERE
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="bodyToggler.toggle('body_section_toggle_csv_file_examples');" href="javascript:;">Toggle section visibility</button>
<div id="body_section_toggle_csv_file_examples" style="display: block">
<p class="boldmaroon"> Sample CSV data file: </p>
<p class="boldmaroon"> Suppose you have this CSV data file: </p>
POKI_RUN_COMMAND{{cat example.csv}}HERE
<p class="boldmaroon"><code>mlr cat</code> is like cat ...</p>
<p class="boldmaroon"><code>mlr cat</code> is like cat -- it passes the data
through unmodified ...</p>
POKI_RUN_COMMAND{{mlr --csv cat example.csv}}HERE
<p class="boldmaroon">... but it can also do format conversion (here, to pretty-printed tabular format): </p>
<p class="boldmaroon">... but it can also do format conversion (here, you can pretty-print in tabular format): </p>
POKI_RUN_COMMAND{{mlr --icsv --opprint cat example.csv}}HERE
<p class="boldmaroon"> <code>mlr head</code> and <code>mlr tail</code> count records rather than lines. The CSV
header is included either way:</p>
<p class="boldmaroon"> <code>mlr head</code> and <code>mlr tail</code> count
records rather than lines. Whethere you&rsquo;re getting the first few records
or the last few, the CSV header is included either way:</p>
POKI_RUN_COMMAND{{mlr --csv head -n 4 example.csv}}HERE
POKI_RUN_COMMAND{{mlr --csv tail -n 4 example.csv}}HERE
<p class="boldmaroon"> Sort primarily alphabetically on one field, then secondarily
numerically descending on another field: </p>
<p class="boldmaroon"> You can sort primarily alphabetically on one field, then
secondarily numerically descending on another field: </p>
POKI_RUN_COMMAND{{mlr --icsv --opprint sort -f shape -nr index example.csv}}HERE
<p class="boldmaroon"> Use <code>cut</code> to retain only specified fields, in input-data order:</p>
<p class="boldmaroon"> You can use <code>cut</code> to retain only specified fields, in the
same order they appeared in the input data:</p>
POKI_RUN_COMMAND{{mlr --icsv --opprint cut -f flag,shape example.csv}}HERE
<p class="boldmaroon"> Use <code>cut -o</code> to retain only specified fields, in your specified order:</p>
<p class="boldmaroon"> You can also use <code>cut -o</code> to retain only
specified fields in your preferred order:</p>
POKI_RUN_COMMAND{{mlr --icsv --opprint cut -o -f flag,shape example.csv}}HERE
<p class="boldmaroon"> Use <code>cut -x</code> to omit specified fields:</p>
<p class="boldmaroon"> You can use <code>cut -x</code> to omit fields you
don&rsquo;t care about:</p>
POKI_RUN_COMMAND{{mlr --icsv --opprint cut -x -f flag,shape example.csv}}HERE
<p class="boldmaroon"> Use <code>filter</code> to retain specified records:</p>
<p class="boldmaroon"> You can use <code>filter</code> to keep only records you care about:</p>
POKI_RUN_COMMAND{{mlr --icsv --opprint filter '$color == "red"' example.csv}}HERE
POKI_RUN_COMMAND{{mlr --icsv --opprint filter '$color == "red" && $flag == 1' example.csv}}HERE
<p class="boldmaroon"> Use <code>put</code> to add/replace fields which are computed from other fields:</p>
<p class="boldmaroon"> You can use <code>put</code> to create new fields which
are computed from other fields:</p>
POKI_RUN_COMMAND{{mlr --icsv --opprint put '$ratio = $quantity / $rate; $color_shape = $color . "_" . $shape' example.csv}}HERE
@ -60,30 +66,49 @@ positional index. Use <code>$[[3]]</code> to access the name of field 3 or
POKI_RUN_COMMAND{{mlr --icsv --opprint put '$[[3]] = "NEW"' example.csv}}HERE
POKI_RUN_COMMAND{{mlr --icsv --opprint put '$[[[3]]] = "NEW"' example.csv}}HERE
<p class="boldmaroon"> JSON output:</p>
<h1>JSON-file examples</h1>
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="bodyToggler.toggle('body_section_toggle_json_file_examples');" href="javascript:;">Toggle section visibility</button>
<div id="body_section_toggle_json_file_examples" style="display: block">
<p class="boldmaroon"> OK, CSV and pretty-print are fine. But Miller can also convert between
a few other formats -- let&rsquo;s take a look at JSON output:</p>
POKI_RUN_COMMAND{{mlr --icsv --ojson put '$ratio = $quantity/$rate; $shape = toupper($shape)' example.csv}}HERE
<p class="boldmaroon"> JSON output with vertical-formatting flags:</p>
<p class="boldmaroon"> Or, JSON output with vertical-formatting flags:</p>
POKI_RUN_COMMAND{{mlr --icsv --ojson --jvstack --jlistwrap tail -n 2 example.csv}}HERE
POKI_RUN_COMMAND{{mlr --icsv --ojsonx tail -n 2 example.csv}}HERE
<p class="boldmaroon"> Use <code>then</code> to pipe commands together. Also, the
<code>-g</code> option for many Miller commands is for group-by: here, <code>head -n
1 -g shape</code> outputs the first record for each distinct value of the
<code>shape</code> field. This means we&rsquo;re finding the record with highest
<code>index</code> field for each distinct <code>shape</code> field:</p>
<h1>Sorts and stats</h1>
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="bodyToggler.toggle('body_section_toggle_sort_and_stats_examples');" href="javascript:;">Toggle section visibility</button>
<div id="body_section_toggle_sort_and_stats_examples" style="display: block">
<p class="boldmaroon"> Now suppose you want to sort the data on a given column,
<i>and then</i> take the top few in that ordering. You can use Miller&rsquo;s
<code>then</code> feature to pipe commands together. </p>
<p class="boldmaroon"> Here are the records with the top three <code>index</code> values: </p>
POKI_RUN_COMMAND{{mlr --icsv --opprint sort -f shape -nr index then head -n 3 example.csv}}HERE
<p class="boldmaroon"> Lots of Miller commands take a <code>-g</code> option
for group-by: here, <code>head -n 1 -g shape</code> outputs the first record
for each distinct value of the <code>shape</code> field. This means we&rsquo;re
finding the record with highest <code>index</code> field for each distinct
<code>shape</code> field:</p>
POKI_RUN_COMMAND{{mlr --icsv --opprint sort -f shape -nr index then head -n 1 -g shape example.csv}}HERE
<p class="boldmaroon"> Statistics can be computed with or without group-by field(s). Also, the first of these two
examples uses <code>--oxtab</code> output format which is a nice alternative to <code>--opprint</code> when you
have lots of columns:</p>
<p class="boldmaroon"> Statistics can be computed with or without group-by field(s)</p>
POKI_RUN_COMMAND{{mlr --icsv --oxtab --from example.csv stats1 -a p0,p10,p25,p50,p75,p90,p99,p100 -f rate}}HERE
POKI_RUN_COMMAND{{mlr --icsv --opprint --from example.csv stats1 -a count,min,mean,max -f quantity -g shape}}HERE
POKI_RUN_COMMAND{{mlr --icsv --opprint --from example.csv stats1 -a count,min,mean,max -f quantity -g shape,color}}HERE
<p class="boldmaroon"> If your output has a lot of columns, you can use XTAB
format to line things up vertically for you instead: </p>
POKI_RUN_COMMAND{{mlr --icsv --oxtab --from example.csv stats1 -a p0,p10,p25,p50,p75,p90,p99,p100 -f rate}}HERE
</div>
<h1>Choices for printing to files</h1>
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="bodyToggler.togglebodyToggler.toggle('body_section_toggle_printing_to_files');" href="javascript:;">Toggle section visibility</button>