mirror of
https://github.com/johnkerl/miller.git
synced 2026-07-22 07:30:43 +00:00
375 lines
11 KiB
HTML
375 lines
11 KiB
HTML
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
|
|
<html lang="en">
|
|
|
|
<!-- PAGE GENERATED FROM template.html and content-for-file-formats.html BY poki. -->
|
|
<!-- PLEASE MAKE CHANGES THERE AND THEN RE-RUN poki. -->
|
|
<head>
|
|
<meta http-equiv="Content-type" content="text/html;charset=UTF-8"/>
|
|
<meta name="description" content="Miller documentation"/>
|
|
<meta name="viewport" content="width=device-width, initial-scale=1.0"/> <!-- mobile-friendly -->
|
|
<meta name="keywords"
|
|
content="John Kerl, Kerl, Miller, miller, mlr, OLAP, data analysis software, regression, correlation, variance, data tools, " />
|
|
|
|
<title> File formats </title>
|
|
<link rel="stylesheet" type="text/css" href="css/miller.css"/>
|
|
<link rel="stylesheet" type="text/css" href="css/poki-callbacks.css"/>
|
|
</head>
|
|
|
|
<!-- ================================================================ -->
|
|
<script type="text/javascript">
|
|
var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www.");
|
|
document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E"));
|
|
</script>
|
|
<script type="text/javascript">
|
|
try {
|
|
var pageTracker = _gat._getTracker("UA-15651652-1");
|
|
pageTracker._trackPageview();
|
|
} catch(err) {}
|
|
</script>
|
|
|
|
<script type="text/javascript">
|
|
function toggle(divName) {
|
|
var eleDiv = document.getElementById(divName);
|
|
if (eleDiv != null) {
|
|
if (eleDiv.style.display == "block") {
|
|
eleDiv.style.display = "none";
|
|
} else {
|
|
eleDiv.style.display = "block";
|
|
}
|
|
}
|
|
}
|
|
</script>
|
|
|
|
<!--
|
|
The background image is from a screenshot of a Google search for "data analysis
|
|
tools", lightened and sepia-toned. Over this was placed a Mac Terminal app with
|
|
very light-grey font and translucent background, in which a few statistical
|
|
Miller commands were run with pretty-print-tabular output format.
|
|
-->
|
|
<body background="pix/sepia-overlay.jpg">
|
|
|
|
<!-- ================================================================ -->
|
|
<table width="100%">
|
|
<tr>
|
|
|
|
<!-- navbar -->
|
|
<td width="15%">
|
|
<!--
|
|
<img src="pix/mlr.jpg" />
|
|
<img style="border-width:1px; color:black;" src="pix/mlr.jpg" />
|
|
-->
|
|
|
|
<div class="pokinav">
|
|
<center><titleinbody>Miller</titleinbody></center>
|
|
|
|
<!-- PAGE LIST GENERATED FROM template.html BY poki -->
|
|
<br/>User info:
|
|
<br/>• <a href="index.html">About Miller</a>
|
|
<br/>• <a href="file-formats.html"><b>File formats</b></a>
|
|
<br/>• <a href="feature-comparison.html">Miller features in the context of the Unix toolkit</a>
|
|
<br/>• <a href="record-heterogeneity.html">Record-heterogeneity</a>
|
|
<br/>• <a href="performance.html">Performance</a>
|
|
<br/>• <a href="etymology.html">Why call it Miller?</a>
|
|
<br/>• <a href="originality.html">How original is Miller?</a>
|
|
<br/>• <a href="reference.html">Reference</a>
|
|
<br/>• <a href="data-examples.html">Data examples</a>
|
|
<br/>• <a href="internationalization.html">Internationalization</a>
|
|
<br/>• <a href="to-do.html">Things to do</a>
|
|
<br/>Developer info:
|
|
<br/>• <a href="build.html">Compiling, portability, dependencies, and testing</a>
|
|
<br/>• <a href="whyc.html">Why C?</a>
|
|
<br/>• <a href="contact.html">Contact information</a>
|
|
<br/>• <a href="https://github.com/johnkerl/miller">GitHub repo</a>
|
|
<br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/>
|
|
<br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/>
|
|
<br/> <br/> <br/> <br/> <br/> <br/>
|
|
</div>
|
|
</td>
|
|
|
|
<!-- page body -->
|
|
<td>
|
|
<div style="overflow-y:scroll;height:1500px">
|
|
<center> <titleinbody> File formats </titleinbody> </center>
|
|
<p/>
|
|
|
|
<!-- BODY COPIED FROM content-for-file-formats.html BY poki -->
|
|
<div class="pokitoc">
|
|
<center><b>Contents:</b></center>
|
|
• <a href="#CSV/TSV/etc.">CSV/TSV/etc.</a><br/>
|
|
• <a href="#Pretty-printed">Pretty-printed</a><br/>
|
|
• <a href="#Key-value_pairs">Key-value pairs</a><br/>
|
|
• <a href="#Index-numbered_(toolkit_style)">Index-numbered (toolkit style)</a><br/>
|
|
• <a href="#Vertical_tabular">Vertical tabular</a><br/>
|
|
</div>
|
|
<p/>
|
|
|
|
<a id="CSV/TSV/etc."/><h1>CSV/TSV/etc.</h1>
|
|
When <tt>mlr</tt> is invoked with the <tt>--csv</tt> or <tt>--csvlite</tt> option,
|
|
key names are found on the first record and values are taken from subsequent
|
|
records. This includes the case of CSV-formatted files. See
|
|
<a href="record-heterogeneity.html">Record-heterogeneity</a> for how Miller handles
|
|
changes of field names within a single data stream.
|
|
|
|
<p/> Miller has record separator <tt>RS</tt> and field separator <tt>FS</tt>,
|
|
just as <tt>awk</tt> does. For TSV, use <tt>--fs tab</tt>; to convert TSV to
|
|
CSV, use <tt>--ifs tab --ofs comma</tt>, etc. (See also
|
|
<a href="reference.html">Reference</a>.)
|
|
|
|
<p/>Miller’s <tt>--csv</tt> flag supports RFC-4180 CSV (<a href="https://tools.ietf.org/html/rfc4180">
|
|
https://tools.ietf.org/html/rfc4180</a>). This includes CRLF line-terminators by default, regardless
|
|
of platform. You can use <tt>mlr --csv --rs lf</tt> for native Un*x (LF-terminated) CSV files.
|
|
|
|
<a id="Pretty-printed"/><h1>Pretty-printed</h1>
|
|
Miller’s pretty-print format is like CSV, but column-aligned. For example, compare
|
|
|
|
<table><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --ocsv cat data/small
|
|
a,b,i,x,y
|
|
pan,pan,1,0.3467901443380824,0.7268028627434533
|
|
eks,pan,2,0.7586799647899636,0.5221511083334797
|
|
wye,wye,3,0.20460330576630303,0.33831852551664776
|
|
eks,wye,4,0.38139939387114097,0.13418874328430463
|
|
wye,pan,5,0.5732889198020006,0.8636244699032729
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td>
|
|
<td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint cat data/small
|
|
a b i x y
|
|
pan pan 1 0.3467901443380824 0.7268028627434533
|
|
eks pan 2 0.7586799647899636 0.5221511083334797
|
|
wye wye 3 0.20460330576630303 0.33831852551664776
|
|
eks wye 4 0.38139939387114097 0.13418874328430463
|
|
wye pan 5 0.5732889198020006 0.8636244699032729
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr></table>
|
|
|
|
Note that while Miller is a line-at-a-time processor and retains input lines in
|
|
memory only where necessary (e.g. for sort), pretty-print output requires it to
|
|
accumulate all input lines (so that it can compute maximum column widths)
|
|
before producing any output. This has two consequences: (a) pretty-print output
|
|
won’t work on <tt>tail -f</tt> contexts, where Miller will be waiting for
|
|
an end-of-file marker which never arrives; (b) pretty-print output for large
|
|
files is constrained by available machine memory.
|
|
|
|
<p/> See <a href="record-heterogeneity.html">Record-heterogeneity</a> for how Miller
|
|
handles changes of field names within a single data stream.
|
|
|
|
<a id="Key-value_pairs"/><h1>Key-value pairs</h1>
|
|
Miller’s default file format is DKVP, for <b>delimited key-value pairs</b>. Example:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr cat data/small
|
|
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
|
|
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
|
|
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
|
|
a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463
|
|
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
Such data are easy to generate, e.g. in Ruby with
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
puts "host=#{hostname},seconds=#{t2-t1},message=#{msg}"
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
puts mymap.collect{|k,v| "#{k}=#{v}"}.join(',')
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
or <tt>print</tt> statements in various languages, e.g.
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
echo "type=3,user=$USER,date=$date\n";
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
logger.log("type=3,user=$USER,date=$date\n");
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/> As discussed in <a href="record-heterogeneity.html">Record-heterogeneity</a>, Miller handles
|
|
changes of field names within the same data stream. But using DKVP format this is particularly
|
|
natural. One of my favorite use-cases for Miller is in application/server logs, where I log all sorts
|
|
of lines such as
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
resource=/path/to/file,loadsec=0.45,ok=true
|
|
record_count=100, resource=/path/to/file
|
|
resource=/some/other/path,loadsec=0.97,ok=false
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
etc. and I just log them as needed. Then later, I can use <tt>grep</tt>, <tt>mlr --opprint group-like</tt>, etc.
|
|
to analyze my logs.
|
|
|
|
<p/>See <a href="reference.html">Reference</a> regarding how to specify separators other than
|
|
the default equals-sign and comma.
|
|
|
|
<a id="Index-numbered_(toolkit_style)"/><h1>Index-numbered (toolkit style)</h1>
|
|
|
|
With <tt>--inidx --ifs ' ' --repifs</tt>, Miller splits lines on whitespace and
|
|
assigns integer field names starting with 1. This recapitulates Unix-toolkit
|
|
behavior.
|
|
|
|
<p/> Example with index-numbered output:
|
|
|
|
<table><tr> <td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ cat data/small
|
|
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
|
|
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
|
|
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
|
|
a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463
|
|
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td> <td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --onidx --ofs ' ' cat data/small
|
|
pan pan 1 0.3467901443380824 0.7268028627434533
|
|
eks pan 2 0.7586799647899636 0.5221511083334797
|
|
wye wye 3 0.20460330576630303 0.33831852551664776
|
|
eks wye 4 0.38139939387114097 0.13418874328430463
|
|
wye pan 5 0.5732889198020006 0.8636244699032729
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td> </tr></table>
|
|
|
|
<p/> Example with index-numbered input:
|
|
|
|
<table><tr> <td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ cat data/mydata.txt
|
|
oh say can you see
|
|
by the dawn's
|
|
early light
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td> <td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --inidx --ifs ' ' --odkvp cat data/mydata.txt
|
|
1=oh,2=say,3=can,4=you,5=see
|
|
1=by,2=the,3=dawn's
|
|
1=early,2=light
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td> </tr></table>
|
|
|
|
<p/> Example with index-numbered input and output:
|
|
|
|
<table><tr> <td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ cat data/mydata.txt
|
|
oh say can you see
|
|
by the dawn's
|
|
early light
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td> <td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --nidx --fs ' ' --repifs cut -f 2,3 data/mydata.txt
|
|
say can
|
|
the dawn's
|
|
light
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td> </tr></table>
|
|
|
|
<a id="Vertical_tabular"/><h1>Vertical tabular</h1>
|
|
|
|
<p/>This is perhaps most useful for looking a very wide and/or multi-column
|
|
data which causes line-wraps on the screen (but see also <a
|
|
href="https://github.com/twosigma/ngrid">https://github.com/twosigma/ngrid</a>
|
|
for an entirely different, very powerful option). Namely:
|
|
|
|
<table><tr> <td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ grep -v '^#' /etc/passwd | head -n 6 | mlr --nidx --fs : --opprint cat
|
|
1 2 3 4 5 6 7
|
|
nobody * -2 -2 Unprivileged User /var/empty /usr/bin/false
|
|
root * 0 0 System Administrator /var/root /bin/sh
|
|
daemon * 1 1 System Services /var/root /usr/bin/false
|
|
_uucp * 4 4 Unix to Unix Copy Protocol /var/spool/uucp /usr/sbin/uucico
|
|
_taskgated * 13 13 Task Gate Daemon /var/empty /usr/bin/false
|
|
_networkd * 24 24 Network Services /var/networkd /usr/bin/false
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr> <tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ grep -v '^#' /etc/passwd | head -n 2 | mlr --nidx --fs : --oxtab cat
|
|
1 nobody
|
|
2 *
|
|
3 -2
|
|
4 -2
|
|
5 Unprivileged User
|
|
6 /var/empty
|
|
7 /usr/bin/false
|
|
|
|
1 root
|
|
2 *
|
|
3 0
|
|
4 0
|
|
5 System Administrator
|
|
6 /var/root
|
|
7 /bin/sh
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td> </tr></table>
|
|
|
|
</div>
|
|
</td>
|
|
|
|
</table>
|
|
</body>
|
|
</html>
|