Mirrors/miller
1
0
Fork 0
mirror of https://github.com/johnkerl/miller.git synced 2026-08-03 13:03:00 +00:00
Code Issues Projects Releases Wiki Activity
miller/doc/content-for-10-min.html
John Kerl 7bcaf2e7b0 10-minute neatens
2020-08-06 11:42:55 -04:00

565 lines
20 KiB
HTML
Raw Blame History

POKI_PUT_TOC_HERE
<p/>
<button style="font-weight:bold;color:maroon;border:0" onclick="bodyToggler.expandAll();" href="javascript:;">Expand all sections</button>
<button style="font-weight:bold;color:maroon;border:0" onclick="bodyToggler.collapseAll();" href="javascript:;">Collapse all sections</button>
<h1>CSV-file examples</h1>
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="bodyToggler.toggle('body_section_toggle_csv_file_examples');" href="javascript:;">Toggle section visibility</button>
<div id="body_section_toggle_csv_file_examples" style="display: block">
<p class="boldmaroon"> Suppose you have this CSV data file: </p>
POKI_RUN_COMMAND{{cat example.csv}}HERE
<p class="boldmaroon"><code>mlr cat</code> is like cat -- it passes the data
through unmodified ...</p>
POKI_RUN_COMMAND{{mlr --csv cat example.csv}}HERE
<p class="boldmaroon">... but it can also do format conversion (here, you can pretty-print in tabular format): </p>
POKI_RUN_COMMAND{{mlr --icsv --opprint cat example.csv}}HERE
<p class="boldmaroon"> <code>mlr head</code> and <code>mlr tail</code> count
records rather than lines. Whethere you&rsquo;re getting the first few records
or the last few, the CSV header is included either way:</p>
POKI_RUN_COMMAND{{mlr --csv head -n 4 example.csv}}HERE
POKI_RUN_COMMAND{{mlr --csv tail -n 4 example.csv}}HERE
<p class="boldmaroon"> You can sort primarily alphabetically on one field, then
secondarily numerically descending on another field: </p>
POKI_RUN_COMMAND{{mlr --icsv --opprint sort -f shape -nr index example.csv}}HERE
<p class="boldmaroon"> You can use <code>cut</code> to retain only specified fields, in the
same order they appeared in the input data:</p>
POKI_RUN_COMMAND{{mlr --icsv --opprint cut -f flag,shape example.csv}}HERE
<p class="boldmaroon"> You can also use <code>cut -o</code> to retain only
specified fields in your preferred order:</p>
POKI_RUN_COMMAND{{mlr --icsv --opprint cut -o -f flag,shape example.csv}}HERE
<p class="boldmaroon"> You can use <code>cut -x</code> to omit fields you
don&rsquo;t care about:</p>
POKI_RUN_COMMAND{{mlr --icsv --opprint cut -x -f flag,shape example.csv}}HERE
<p class="boldmaroon"> You can use <code>filter</code> to keep only records you care about:</p>
POKI_RUN_COMMAND{{mlr --icsv --opprint filter '$color == "red"' example.csv}}HERE
POKI_RUN_COMMAND{{mlr --icsv --opprint filter '$color == "red" && $flag == 1' example.csv}}HERE
<p class="boldmaroon"> You can use <code>put</code> to create new fields which
are computed from other fields:</p>
POKI_RUN_COMMAND{{mlr --icsv --opprint put '$ratio = $quantity / $rate; $color_shape = $color . "_" . $shape' example.csv}}HERE
<p class="boldmaroon"> Even though Miller&rsquo;s main selling point is
name-indexing, sometimes you really want to refer to a field name by its
positional index. Use <code>$[[3]]</code> to access the name of field 3 or
<code>$[[[3]]]</code> to access the value of field 3:</p>
POKI_RUN_COMMAND{{mlr --icsv --opprint put '$[[3]] = "NEW"' example.csv}}HERE
POKI_RUN_COMMAND{{mlr --icsv --opprint put '$[[[3]]] = "NEW"' example.csv}}HERE
<h1>JSON-file examples</h1>
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="bodyToggler.toggle('body_section_toggle_json_file_examples');" href="javascript:;">Toggle section visibility</button>
<div id="body_section_toggle_json_file_examples" style="display: block">
<p class="boldmaroon"> OK, CSV and pretty-print are fine. But Miller can also convert between
a few other formats -- let&rsquo;s take a look at JSON output:</p>
POKI_RUN_COMMAND{{mlr --icsv --ojson put '$ratio = $quantity/$rate; $shape = toupper($shape)' example.csv}}HERE
<p class="boldmaroon"> Or, JSON output with vertical-formatting flags:</p>
POKI_RUN_COMMAND{{mlr --icsv --ojsonx tail -n 2 example.csv}}HERE
<h1>Sorts and stats</h1>
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="bodyToggler.toggle('body_section_toggle_sort_and_stats_examples');" href="javascript:;">Toggle section visibility</button>
<div id="body_section_toggle_sort_and_stats_examples" style="display: block">
<p class="boldmaroon"> Now suppose you want to sort the data on a given column,
<i>and then</i> take the top few in that ordering. You can use Miller&rsquo;s
<code>then</code> feature to pipe commands together. </p>
<p class="boldmaroon"> Here are the records with the top three <code>index</code> values: </p>
POKI_RUN_COMMAND{{mlr --icsv --opprint sort -f shape -nr index then head -n 3 example.csv}}HERE
<p class="boldmaroon"> Lots of Miller commands take a <code>-g</code> option
for group-by: here, <code>head -n 1 -g shape</code> outputs the first record
for each distinct value of the <code>shape</code> field. This means we&rsquo;re
finding the record with highest <code>index</code> field for each distinct
<code>shape</code> field:</p>
POKI_RUN_COMMAND{{mlr --icsv --opprint sort -f shape -nr index then head -n 1 -g shape example.csv}}HERE
<p class="boldmaroon"> Statistics can be computed with or without group-by field(s)</p>
POKI_RUN_COMMAND{{mlr --icsv --opprint --from example.csv stats1 -a count,min,mean,max -f quantity -g shape}}HERE
POKI_RUN_COMMAND{{mlr --icsv --opprint --from example.csv stats1 -a count,min,mean,max -f quantity -g shape,color}}HERE
<p class="boldmaroon"> If your output has a lot of columns, you can use XTAB
format to line things up vertically for you instead: </p>
POKI_RUN_COMMAND{{mlr --icsv --oxtab --from example.csv stats1 -a p0,p10,p25,p50,p75,p90,p99,p100 -f rate}}HERE
</div>
<h1>Choices for printing to files</h1>
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="bodyToggler.togglebodyToggler.toggle('body_section_toggle_printing_to_files');" href="javascript:;">Toggle section visibility</button>
<div id="body_section_toggle_printing_to_files" style="display: block">
<p>Often we want to print output <span class="boldmaroon">to the screen</span>. Miller does this by default, as we&rsquo;ve
seen in the previous examples.</p>
<p/> Sometimes we want to print output to another file: <span class="boldmaroon">just use '>
outputfilenamegoeshere'</span> at the end of your command:</p>
<table><tr><td>
<div class="pokipanel"> <pre>
% mlr --icsv --opprint cat example.csv &gt; newfile.csv
# Output goes to the new file;
# nothing is printed to the screen.
</pre> </div>
</td><td>
<div class="pokipanel"> <pre>
% cat newfile.csv
color shape flag index quantity rate
yellow triangle 1 11 43.6498 9.8870
red square 1 15 79.2778 0.0130
red circle 1 16 13.8103 2.9010
red square 0 48 77.5542 7.4670
purple triangle 0 51 81.2290 8.5910
red square 0 64 77.1991 9.5310
purple triangle 0 65 80.1405 5.8240
yellow circle 1 73 63.9785 4.2370
yellow circle 1 87 63.5058 8.3350
purple square 0 91 72.3735 8.2430
</pre> </div>
</td></tr></table>
<p/> Other times we just want our files to be changed in-place: <span class="boldmaroon">just use 'mlr -I'</span>.</p>
<table><tr><td>
<div class="pokipanel"> <pre>
% cp example.csv newfile.txt
% cat newfile.txt
color,shape,flag,index,quantity,rate
yellow,triangle,1,11,43.6498,9.8870
red,square,1,15,79.2778,0.0130
red,circle,1,16,13.8103,2.9010
red,square,0,48,77.5542,7.4670
purple,triangle,0,51,81.2290,8.5910
red,square,0,64,77.1991,9.5310
purple,triangle,0,65,80.1405,5.8240
yellow,circle,1,73,63.9785,4.2370
yellow,circle,1,87,63.5058,8.3350
purple,square,0,91,72.3735,8.2430
</pre> </div>
</td><td>
<div class="pokipanel"> <pre>
% mlr -I --icsv --opprint cat newfile.txt
% cat newfile.txt
color shape flag index quantity rate
yellow triangle 1 11 43.6498 9.8870
red square 1 15 79.2778 0.0130
red circle 1 16 13.8103 2.9010
red square 0 48 77.5542 7.4670
purple triangle 0 51 81.2290 8.5910
red square 0 64 77.1991 9.5310
purple triangle 0 65 80.1405 5.8240
yellow circle 1 73 63.9785 4.2370
yellow circle 1 87 63.5058 8.3350
purple square 0 91 72.3735 8.2430
</pre> </div>
</td></tr></table>
<p/> Also using <code>mlr -I</code> you can bulk-operate on lots of files: e.g.
<table><tr><td>
<div class="pokipanel"> <pre>
mlr -I --csv cut -x -f unwanted_column_name *.csv
</pre> </div>
</td></tr></table>
<p/> If you like, you can first copy off your original data somewhere else, before doing in-place operations.
<p class="boldmaroon"> Lastly, using <code>tee</code> within <code>put</code>, you can split your input data into separate files
per one or more field names:</p>
POKI_RUN_COMMAND{{mlr --csv --from example.csv put -q 'tee > $shape.".csv", $*'}}HERE
<table><tr><td>
POKI_RUN_COMMAND{{cat circle.csv}}HERE
</td><td>
POKI_RUN_COMMAND{{cat square.csv}}HERE
</td><td>
POKI_RUN_COMMAND{{cat triangle.csv}}HERE
</td></tr></table>
</div>
<h1>Other-format examples</h1>
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="bodyToggler.toggle('body_section_toggle_other_format_examples');" href="javascript:;">Toggle section visibility</button>
<div id="body_section_toggle_other_format_examples" style="display: block">
<p/>What&rsquo;s a CSV file, really? It&rsquo;s an array of rows, or
<i>records</i>, each being a list of key-value pairs, or <i>fields</i>: for CSV
it so happens that all the keys are shared in the header line and the values
vary data line by data line.
<p/>For example, if you have
<div class="pokipanel">
<pre>
shape,flag,index
circle,1,24
square,0,36
</pre>
</div>
<p/>then that&rsquo;s a way of saying
<div class="pokipanel">
<pre>
shape=circle,flag=1,index=24
shape=square,flag=0,index=36
</pre>
</div>
<p>Data written this way are called <span class="boldmaroon">DKVP</span>, for <i>delimited key-value pairs</i>.</p>
<p/>We&rsquo;ve also already seen other ways to write the same data:
<div class="pokipanel">
<pre>
CSV PPRINT JSON
shape,flag,index shape flag index [
circle,1,24 circle 1 24 {
square,0,36 square 0 36 "shape": "circle",
"flag": 1,
"index": 24
},
DKVP XTAB {
shape=circle,flag=1,index=24 shape circle "shape": "square",
shape=square,flag=0,index=36 flag 1 "flag": 0,
index 24 "index": 36
}
shape square ]
flag 0
index 36
</pre>
</div>
<p class="boldmaroon">Anything we can do with CSV input data, we can do with any
other format input data.</p> And you can read from one format, do any
record-processing, and output to the same format as the input, or to a
different output format.
</div>
<h1>SQL-output examples</h1>
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="bodyToggler.toggle('body_section_toggle_sql_output_examples');" href="javascript:;">Toggle section visibility</button>
<div id="body_section_toggle_sql_output_examples" style="display: block">
<p/>I like to produce SQL-query output with header-column and tab delimiter:
this is CSV but with a tab instead of a comma, also known as TSV. Then I
post-process with <code>mlr --tsv</code> or <code>mlr --tsvlite</code>. This
means I can do some (or all, or none) of my data processing within SQL queries,
and some (or none, or all) of my data processing using Miller &mdash; whichever
is most convenient for my needs at the moment.
<p/>For example, using default output formatting in <code>mysql</code> we get
formatting like Miller&rsquo;s <code>--opprint --barred</code>:
<div class="pokipanel">
<pre>
$ mysql --database=mydb -e 'show columns in mytable'
+------------------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+------------------+--------------+------+-----+---------+-------+
| id | bigint(20) | NO | MUL | NULL | |
| category | varchar(256) | NO | | NULL | |
| is_permanent | tinyint(1) | NO | | NULL | |
| assigned_to | bigint(20) | YES | | NULL | |
| last_update_time | int(11) | YES | | NULL | |
+------------------+--------------+------+-----+---------+-------+
</pre>
</div>
<p/>Using <code>mysql</code>&rsquo;s <code>-B</code> we get TSV output:
<div class="pokipanel">
<pre>
$ mysql --database=mydb -B -e 'show columns in mytable' | mlr --itsvlite --opprint cat
Field Type Null Key Default Extra
id bigint(20) NO MUL NULL -
category varchar(256) NO - NULL -
is_permanent tinyint(1) NO - NULL -
assigned_to bigint(20) YES - NULL -
last_update_time int(11) YES - NULL -
</pre>
</div>
<p/>Since Miller handles TSV output, we can do as much or as little processing
as we want in the SQL query, then send the rest on to Miller. This includes
outputting as JSON, doing further selects/joins in Miller, doing stats, etc.
etc.
<div class="pokipanel">
<pre>
$ mysql --database=mydb -B -e 'show columns in mytable' | mlr --itsvlite --ojson --jlistwrap --jvstack cat
[
{
"Field": "id",
"Type": "bigint(20)",
"Null": "NO",
"Key": "MUL",
"Default": "NULL",
"Extra": ""
},
{
"Field": "category",
"Type": "varchar(256)",
"Null": "NO",
"Key": "",
"Default": "NULL",
"Extra": ""
},
{
"Field": "is_permanent",
"Type": "tinyint(1)",
"Null": "NO",
"Key": "",
"Default": "NULL",
"Extra": ""
},
{
"Field": "assigned_to",
"Type": "bigint(20)",
"Null": "YES",
"Key": "",
"Default": "NULL",
"Extra": ""
},
{
"Field": "last_update_time",
"Type": "int(11)",
"Null": "YES",
"Key": "",
"Default": "NULL",
"Extra": ""
}
]
</pre>
</div>
<div class="pokipanel">
<pre>
$ mysql --database=mydb -B -e 'select * from mytable' > query.tsv
$ mlr --from query.tsv --t2p stats1 -a count -f id -g category,assigned_to
category assigned_to id_count
special 10000978 207
special 10003924 385
special 10009872 168
standard 10000978 524
standard 10003924 392
standard 10009872 108
...
</pre>
</div>
<p/>Again, all the examples in the CSV section apply here &mdash; just change the input-format
flags.
</div>
<h1>SQL-input examples</h1>
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="bodyToggler.toggle('body_section_toggle_sql_input_examples');" href="javascript:;">Toggle section visibility</button>
<div id="body_section_toggle_sql_input_examples" style="display: block">
<p/> One use of NIDX (value-only, no keys) format is for loading up SQL tables.
<p/> Create and load SQL table:
<div class="pokipanel"><pre>
mysql> CREATE TABLE abixy(
a VARCHAR(32),
b VARCHAR(32),
i BIGINT(10),
x DOUBLE,
y DOUBLE
);
Query OK, 0 rows affected (0.01 sec)
bash$ mlr --onidx --fs comma cat data/medium > medium.nidx
mysql> LOAD DATA LOCAL INFILE 'medium.nidx' REPLACE INTO TABLE abixy FIELDS TERMINATED BY ',' ;
Query OK, 10000 rows affected (0.07 sec)
Records: 10000 Deleted: 0 Skipped: 0 Warnings: 0
mysql> SELECT COUNT(*) AS count FROM abixy;
+-------+
| count |
+-------+
| 10000 |
+-------+
1 row in set (0.00 sec)
mysql> SELECT * FROM abixy LIMIT 10;
+------+------+------+---------------------+---------------------+
| a | b | i | x | y |
+------+------+------+---------------------+---------------------+
| pan | pan | 1 | 0.3467901443380824 | 0.7268028627434533 |
| eks | pan | 2 | 0.7586799647899636 | 0.5221511083334797 |
| wye | wye | 3 | 0.20460330576630303 | 0.33831852551664776 |
| eks | wye | 4 | 0.38139939387114097 | 0.13418874328430463 |
| wye | pan | 5 | 0.5732889198020006 | 0.8636244699032729 |
| zee | pan | 6 | 0.5271261600918548 | 0.49322128674835697 |
| eks | zee | 7 | 0.6117840605678454 | 0.1878849191181694 |
| zee | wye | 8 | 0.5985540091064224 | 0.976181385699006 |
| hat | wye | 9 | 0.03144187646093577 | 0.7495507603507059 |
| pan | wye | 10 | 0.5026260055412137 | 0.9526183602969864 |
+------+------+------+---------------------+---------------------+
</pre></div>
<p/> Aggregate counts within SQL:
<div class="pokipanel"><pre>
mysql> SELECT a, b, COUNT(*) AS count FROM abixy GROUP BY a, b ORDER BY COUNT DESC;
+------+------+-------+
| a | b | count |
+------+------+-------+
| zee | wye | 455 |
| pan | eks | 429 |
| pan | pan | 427 |
| wye | hat | 426 |
| hat | wye | 423 |
| pan | hat | 417 |
| eks | hat | 417 |
| pan | zee | 413 |
| eks | eks | 413 |
| zee | hat | 409 |
| eks | wye | 407 |
| zee | zee | 403 |
| pan | wye | 395 |
| wye | pan | 392 |
| zee | eks | 391 |
| zee | pan | 389 |
| hat | eks | 389 |
| wye | eks | 386 |
| wye | zee | 385 |
| hat | zee | 385 |
| hat | hat | 381 |
| wye | wye | 377 |
| eks | pan | 371 |
| hat | pan | 363 |
| eks | zee | 357 |
+------+------+-------+
25 rows in set (0.01 sec)
</pre></div>
<p/> Aggregate counts within Miller:
<div class="pokipanel"><pre>
$ mlr --opprint uniq -c -g a,b then sort -nr count data/medium
a b count
zee wye 455
pan eks 429
pan pan 427
wye hat 426
hat wye 423
pan hat 417
eks hat 417
eks eks 413
pan zee 413
zee hat 409
eks wye 407
zee zee 403
pan wye 395
hat pan 363
eks zee 357
</pre></div>
<p/> Pipe SQL output to aggregate counts within Miller:
<div class="pokipanel"><pre>
$ mysql -D miller -B -e 'select * from abixy' | mlr --itsv --opprint uniq -c -g a,b then sort -nr count
a b count
zee wye 455
pan eks 429
pan pan 427
wye hat 426
hat wye 423
pan hat 417
eks hat 417
eks eks 413
pan zee 413
zee hat 409
eks wye 407
zee zee 403
pan wye 395
wye pan 392
zee eks 391
zee pan 389
hat eks 389
wye eks 386
hat zee 385
wye zee 385
hat hat 381
wye wye 377
eks pan 371
hat pan 363
eks zee 357
</pre></div>
</div>
<h1>Log-processing examples</h1>
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="bodyToggler.toggle('body_section_toggle_log_processing_examples');" href="javascript:;">Toggle section visibility</button>
<div id="body_section_toggle_log_processing_examples" style="display: block">
<p/>Another of my favorite use-cases for Miller is doing ad-hoc processing of
log-file data. Here&rsquo;s where DKVP format really shines: one, since the
field names and field values are present on every line, every line stands on
its own. That means you can <code>grep</code> or what have you. Also it means not
every line needs to have the same list of field names (&ldquo;schema &rdquo;).
<p/>Again, all the examples in the CSV section apply here &mdash; just change
the input-format flags. But there&rsquo;s more you can do when not all the
records have the same shape.
<p/>Writing a program &mdash; in any language whatsoever &mdash; you can have
it print out log lines as it goes along, with items for various events jumbled
together. After the program has finished running you can sort it all out,
filter it, analyze it, and learn from it.
<p/> Suppose your program has printed something like this:
POKI_RUN_COMMAND{{cat log.txt}}HERE
<p/> Each print statement simply contains local information: the current
timestamp, whether a particular cache was hit or not, etc. Then using either
the system <code>grep</code> command, or Miller&rsquo;s <code>having-fields</code>, or
<code>is_present</code>, we can pick out the parts we want and analyze them:
POKI_INCLUDE_AND_RUN_ESCAPED(10-1.sh)HERE
POKI_INCLUDE_AND_RUN_ESCAPED(10-2.sh)HERE
<p/>Alternatively, we can simply group the similar data for a better look:
POKI_RUN_COMMAND{{mlr --opprint group-like log.txt}}HERE
POKI_RUN_COMMAND{{mlr --opprint group-like then sec2gmt time log.txt}}HERE
</div>
<h1>More</h1>
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="bodyToggler.toggle('body_section_toggle_more');" href="javascript:;">Toggle section visibility</button>
<div id="body_section_toggle_more" style="display: block">
<p/>Please see the <a href="reference.html">reference</a> for complete
information, as well as the <a href="faq.html">FAQ</a> and the <a
href="cookbook.html">cookbook</a> for more tips.
</div>
Reference in a new issue View git blame Copy permalink