Mirrors/miller
1
0
Fork 0
mirror of https://github.com/johnkerl/miller.git synced 2026-07-22 07:30:43 +00:00
Code Issues Projects Releases Wiki Activity
miller/doc/content-for-10-min.html
John Kerl 18cd483b77 10-minutes SQL content
2016-09-25 20:56:23 -04:00

287 lines
10 KiB
HTML
Raw Blame History

POKI_PUT_TOC_HERE
<h1>CSV-file examples</h1>
<p/><boldmaroon> Sample CSV data file: </boldmaroon>
POKI_RUN_COMMAND{{cat example.csv}}HERE
<p/><boldmaroon> <tt>mlr cat</tt> is like cat ...</boldmaroon>
POKI_RUN_COMMAND{{mlr --csv --rs lf cat example.csv}}HERE
<p/><boldmaroon>... but it can also do format conversion (here, to pretty-printed tabular format): </boldmaroon>
POKI_RUN_COMMAND{{mlr --icsv --irs lf --opprint cat example.csv}}HERE
<p/><boldmaroon> <tt>mlr head</tt> and <tt>mlr tail</tt> count records. The CSV
header is included either way:</boldmaroon>
POKI_RUN_COMMAND{{mlr --csv --rs lf head -n 4 example.csv}}HERE
POKI_RUN_COMMAND{{mlr --csv --rs lf tail -n 4 example.csv}}HERE
<p/><boldmaroon> Sort primarily alphabetically on one field, then secondarily
numerically descending on another field: </boldmaroon>
POKI_RUN_COMMAND{{mlr --icsv --irs lf --opprint sort -f shape -nr index example.csv}}HERE
<p/><boldmaroon> Use <tt>cut</tt> to retain only specified fields, in input-data order:</boldmaroon>
POKI_RUN_COMMAND{{mlr --icsv --irs lf --opprint cut -f flag,shape example.csv}}HERE
<p/><boldmaroon> Use <tt>cut -o</tt> to retain only specified fields, in your specified order:</boldmaroon>
POKI_RUN_COMMAND{{mlr --icsv --irs lf --opprint cut -o -f flag,shape example.csv}}HERE
<p/><boldmaroon> Use <tt>cut -x</tt> to omit specified fields:</boldmaroon>
POKI_RUN_COMMAND{{mlr --icsv --irs lf --opprint cut -x -f flag,shape example.csv}}HERE
<p/><boldmaroon> Use <tt>filter</tt> to retain specified records:</boldmaroon>
POKI_RUN_COMMAND{{mlr --icsv --irs lf --opprint filter '$color == "red"' example.csv}}HERE
POKI_RUN_COMMAND{{mlr --icsv --irs lf --opprint filter '$color == "red" && $flag == 1' example.csv}}HERE
<p/><boldmaroon> Use <tt>put</tt> to add/replace fields which are computed from other fields:</boldmaroon>
POKI_RUN_COMMAND{{mlr --icsv --irs lf --opprint put '$ratio = $quantity / $rate; $color_shape = $color . "_" . $shape' example.csv}}HERE
<p/><boldmaroon> JSON output:</boldmaroon>
POKI_RUN_COMMAND{{mlr --icsv --irs lf --ojson put '$ratio = $quantity/$rate; $shape = toupper($shape)' example.csv}}HERE
POKI_RUN_COMMAND{{mlr --icsv --irs lf --ojson --jvstack --jlistwrap tail -n 2 example.csv}}HERE
<p/><boldmaroon> Use <tt>then</tt> to pipe commands together. Also, the
<tt>-g</tt> option for many Miller commands is for group-by: here,
<tt>head -n 1 -g shape</tt> outputs the first record for each distinct
value of the <tt>shape</tt> field.</boldmaroon>
POKI_RUN_COMMAND{{mlr --icsv --irs lf --opprint sort -f shape -nr index then head -n 1 -g shape example.csv}}HERE
<p/><boldmaroon> Statistics can be computed with or without group-by field(s). Also, the first of these two
examples uses <tt>--oxtab</tt> output format which is a nice alternative to <tt>--opprint</tt> when you
have lots of columns:</boldmaroon>
POKI_RUN_COMMAND{{mlr --icsv --irs lf --oxtab --from example.csv stats1 -a p0,p10,p25,p50,p75,p90,p99,p100 -f rate}}HERE
POKI_RUN_COMMAND{{mlr --icsv --irs lf --opprint --from example.csv stats1 -a count,min,mean,max -f quantity -g shape}}HERE
POKI_RUN_COMMAND{{mlr --icsv --irs lf --opprint --from example.csv stats1 -a count,min,mean,max -f quantity -g shape,color}}HERE
<p/><boldmaroon> Using <tt>tee</tt> within <tt>put</tt>, you can split your input data into separate files
per one or more field names:</boldmaroon>
POKI_RUN_COMMAND{{mlr --csv --rs lf --from example.csv put -q 'tee > $shape.".csv", $*'}}HERE
<table><tr><td>
POKI_RUN_COMMAND{{cat circle.csv}}HERE
</td><td>
POKI_RUN_COMMAND{{cat square.csv}}HERE
</td><td>
POKI_RUN_COMMAND{{cat triangle.csv}}HERE
</td></tr></table>
<h1>Other-format examples</h1>
<p/>What&rsquo;s a CSV file, really? It&rsquo;s an array of rows, or
<i>records</i>, each being a list of key-value pairs, or <i>fields</i>: for CSV
it so happens that all the keys are shared in the header line and the values
vary data line by data line.
<p/>For example, if you have
<div class="pokipanel">
<pre>
shape,flag,index
circle,1,24
square,0,36
</pre>
</div>
<p/>then that&rsquo;s a way of saying
<div class="pokipanel">
<pre>
shape=circle,flag=1,index=24
shape=square,flag=0,index=36
</pre>
</div>
<p/>Data written this way are called <boldmaroon>DKVP</boldmaroon>, for <i>delimited key-value pairs</i>.
<p/>We&rsquo;ve also already seen other ways to write the same data:
<div class="pokipanel">
<pre>
CSV PPRINT JSON
shape,flag,index shape flag index [
circle,1,24 circle 1 24 {
square,0,36 square 0 36 "shape": "circle",
"flag": 1,
"index": 24
},
DKVP XTAB {
shape=circle,flag=1,index=24 shape circle "shape": "square",
shape=square,flag=0,index=36 flag 1 "flag": 0,
index 24 "index": 36
}
shape square ]
flag 0
index 36
</pre>
</div>
<p/><boldmaroon>Anything we can do with CSV input data, we can do with any
other format input data.</boldmaroon> And you can read from one format, do any
record-processing, and output to the same format as the input, or to a
different output format.
<h1>SQL-output examples</h1>
<p/>I like to produce SQL-query output with header-column and tab delimiter:
this is CSV but with a tab instead of a comma, also known as TSV. Then I
post-process with <tt>mlr --tsv --rs lf</tt> or <tt>mlr --tsvlite</tt>. This
means I can do some (or all, or none) of my data processing within SQL queries,
and some (or none, or all) of my data processing using Miller &mdash; whichever
is most convenient for my needs at the moment.
<p/>For example, using default output formatting in <tt>mysql</tt> we get
formatting like Miller&rsquo;s <tt>--opprint --barred</tt>:
<div class="pokipanel">
<pre>
$ mysql --database=mydb -e 'show columns in mytable'
+------------------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+------------------+--------------+------+-----+---------+-------+
| id | bigint(20) | NO | MUL | NULL | |
| category | varchar(256) | NO | | NULL | |
| is_permanent | tinyint(1) | NO | | NULL | |
| assigned_to | bigint(20) | YES | | NULL | |
| last_update_time | int(11) | YES | | NULL | |
+------------------+--------------+------+-----+---------+-------+
</pre>
</div>
<p/>Using <tt>mysql</tt>&rsquo;s <tt>-B</tt> we get TSV output:
<div class="pokipanel">
<pre>
$ mysql --database=mydb -B -e 'show columns in mytable' | mlr --itsvlite --opprint cat
Field Type Null Key Default Extra
id bigint(20) NO MUL NULL -
category varchar(256) NO - NULL -
is_permanent tinyint(1) NO - NULL -
assigned_to bigint(20) YES - NULL -
last_update_time int(11) YES - NULL -
</pre>
</div>
<p/>Since Miller handles TSV output, we can do as much or as little processing
as we want in the SQL query, then send the rest on to Miller. This includes
outputting as JSON, doing further selects/joins in Miller, doing stats, etc.
etc.
<div class="pokipanel">
<pre>
$ mysql --database=mydb -B -e 'show columns in mytable' | mlr --itsvlite --ojson --jlistwrap --jvstack cat
[
{
"Field": "id",
"Type": "bigint(20)",
"Null": "NO",
"Key": "MUL",
"Default": "NULL",
"Extra": ""
},
{
"Field": "category",
"Type": "varchar(256)",
"Null": "NO",
"Key": "",
"Default": "NULL",
"Extra": ""
},
{
"Field": "is_permanent",
"Type": "tinyint(1)",
"Null": "NO",
"Key": "",
"Default": "NULL",
"Extra": ""
},
{
"Field": "assigned_to",
"Type": "bigint(20)",
"Null": "YES",
"Key": "",
"Default": "NULL",
"Extra": ""
},
{
"Field": "last_update_time",
"Type": "int(11)",
"Null": "YES",
"Key": "",
"Default": "NULL",
"Extra": ""
}
]
</pre>
</div>
<div class="pokipanel">
<pre>
$ mysql --database=mydb -B -e 'select * from mytable' > query.tsv
$ mlr --from query.tsv --t2p stats1 -a count -f id -g category,assigned_to
category assigned_to id_count
special 10000978 207
special 10003924 385
special 10009872 168
standard 10000978 524
standard 10003924 392
standard 10009872 108
...
</pre>
</div>
<p/>Again, all the examples in the CSV section apply here &mdash; just change the input-format
flags.
<h1>Log-processing examples</h1>
<p/>Another of my favorite use-cases for Miller is doing ad-hoc processing of
log-file data. Here&rsquo;s where DKVP format really shines: one, since the
field names and field values are present on every line, every line stands on
its own. That means you can <tt>grep</tt> or what have you. Also it means not
every line needs to have the same list of field names (&ldquo;schema &rdquo;).
<p/>Again, all the examples in the CSV section apply here &mdash; just change
the input-format flags. But there&rsquo;s more you can do when not all the
records have the same shape.
<p/>Writing a program &mdash; in any language whatsoever &mdash; you can have
it print out log lines as it goes along, with items for various events jumbled
together. After the program has finished running you can sort it all out,
filter it, analyze it, and learn from it.
<p/> Suppose your program has printed something like this:
POKI_RUN_COMMAND{{cat log.txt}}HERE
<p/> Each print statement simply contains local information: the current
timestamp, whether a particular cache was hit or not, etc. Then using either
the system <tt>grep</tt> command, or Miller&rsquo;s <tt>having-fields</tt>, or
<tt>ispresent</tt>, we can pick out the parts we want and analyze them:
POKI_INCLUDE_AND_RUN_ESCAPED(10-1.sh)HERE
POKI_INCLUDE_AND_RUN_ESCAPED(10-2.sh)HERE
<p/>Alternatively, we can simply group the similar data for a better look:
POKI_RUN_COMMAND{{mlr --opprint group-like log.txt}}HERE
POKI_RUN_COMMAND{{mlr --opprint group-like then sec2gmt time log.txt}}HERE
<h1>More</h1>
<p/>Please see the <a href="reference.html">reference</a> for complete
information, as well as the <a href="faq.html">FAQ</a> and the <a
href="cookbook.html">cookbook</a> for more tips.
Reference in a new issue View git blame Copy permalink