miller/doc/content-for-file-formats.html

POKI_PUT_TOC_HERE

<h1>Examples</h1>

POKI_RUN_COMMAND{{mlr --usage-data-format-examples}}HERE

<h1>DKVP: Key-value pairs</h1>
Miller&rsquo;s default file format is DKVP, for <b>delimited key-value pairs</b>. Example:

POKI_RUN_COMMAND{{mlr cat data/small}}HERE

Such data are easy to generate, e.g. in Ruby with
POKI_CARDIFY(puts "host=#{hostname},seconds=#{t2-t1},message=#{msg}")HERE
POKI_CARDIFY{{puts mymap.collect{|k,v| "#{k}=#{v}"}.join(',')}}HERE
or <tt>print</tt> statements in various languages, e.g.
POKI_CARDIFY(echo "type=3,user=$USER,date=$date\n";)HERE
POKI_CARDIFY{{logger.log("type=3,user=$USER,date=$date\n");}}HERE

<p/>Fields lacking an IPS will have positional index (starting at 1) used as
the key, as in NIDX format. For example, <tt>dish=7,egg=8,flint</tt> is parsed
as <tt>"dish" =&gt; "7", "egg" =&gt; "8", "3" =&gt; "flint"</tt> and
<tt>dish,egg,flint</tt> is parsed as <tt>"1" =&gt; "dish", "2" =&gt; "egg", "3"
=&gt; "flint"</tt>.

<p/> As discussed in POKI_PUT_LINK_FOR_PAGE(record-heterogeneity.html)HERE,
Miller handles changes of field names within the same data stream. But using
DKVP format this is particularly natural. One of my favorite use-cases for
Miller is in application/server logs, where I log all sorts of lines such as

<p/>
<div class="pokipanel">
<pre>
resource=/path/to/file,loadsec=0.45,ok=true
record_count=100, resource=/path/to/file
resource=/some/other/path,loadsec=0.97,ok=false
</pre>
</div>
<p/>

etc. and I just log them as needed. Then later, I can use <tt>grep</tt>, <tt>mlr --opprint group-like</tt>, etc.
to analyze my logs.

<p/>See POKI_PUT_LINK_FOR_PAGE(reference.html)HERE regarding how to specify separators other than
the default equals-sign and comma.

<h1>NIDX: Index-numbered (toolkit style)</h1>

With <tt>--inidx --ifs ' ' --repifs</tt>, Miller splits lines on whitespace and
assigns integer field names starting with 1. This recapitulates Unix-toolkit
behavior.

<p/> Example with index-numbered output:

<table><tr> <td>
POKI_RUN_COMMAND{{cat data/small}}HERE
</td> <td>
POKI_RUN_COMMAND{{mlr --onidx --ofs ' ' cat data/small}}HERE
</td> </tr></table>

<p/> Example with index-numbered input:

<table><tr> <td>
POKI_RUN_COMMAND{{cat data/mydata.txt}}HERE
</td> <td>
POKI_RUN_COMMAND{{mlr --inidx --ifs ' ' --odkvp cat data/mydata.txt}}HERE
</td> </tr></table>

<p/> Example with index-numbered input and output:

<table><tr> <td>
POKI_RUN_COMMAND{{cat data/mydata.txt}}HERE
</td> <td>
POKI_RUN_COMMAND{{mlr --nidx --fs ' ' --repifs cut -f 2,3 data/mydata.txt}}HERE
</td> </tr></table>

<h1>CSV/TSV/etc.</h1>
When <tt>mlr</tt> is invoked with the <tt>--csv</tt> or <tt>--csvlite</tt> option,
key names are found on the first record and values are taken from subsequent
records.  This includes the case of CSV-formatted files.  See
POKI_PUT_LINK_FOR_PAGE(record-heterogeneity.html)HERE for how Miller handles
changes of field names within a single data stream.

<p/> Miller has record separator <tt>RS</tt> and field separator <tt>FS</tt>,
just as <tt>awk</tt> does.  For TSV, use <tt>--fs tab</tt>; to convert TSV to
CSV, use <tt>--ifs tab --ofs comma</tt>, etc.  (See also
POKI_PUT_LINK_FOR_PAGE(reference.html)HERE.)

<p/> The following are synonymous pairs:
<ul>
<li/> <tt>--tsv</tt> and <tt>--csv --fs tab</tt>
<li/> <tt>--itsv</tt> and <tt>--icsv --ifs tab</tt>
<li/> <tt>--otsv</tt> and <tt>--ocsv --ofs tab</tt>
<li/> <tt>--tsvlite</tt> and <tt>--csvlite --fs tab</tt>
<li/> <tt>--itsvlite</tt> and <tt>--icsvlite --ifs tab</tt>
<li/> <tt>--otsvlite</tt> and <tt>--ocsvlite --ofs tab</tt>
</ul>

<p/>Miller&rsquo;s <tt>--csv</tt> flag supports RFC-4180 CSV (<a href="https://tools.ietf.org/html/rfc4180">
https://tools.ietf.org/html/rfc4180</a>). This includes CRLF line-terminators by default, regardless
of platform.

<p/>
<b>
Please use <tt>mlr --csv --rs lf</tt> for native Un*x (linefeed-terminated) CSV files.
</b>

<p/>Instead of specifying <tt>--rs lf</tt> on each invocation, you can instead
have <tt>MLR_CSV_DEFAULT_RS=lf</tt> in your shell environment: e.g. put
<tt>export MLR_CSV_DEFAULT_RS=lf</tt> in your <tt>~/.bashrc</tt> or
<tt>~/.zshrc</tt>, or <tt>setenv MLR_CSV_DEFAULT_RS lf</tt> in your
<tt>~/.cshrc</tt>, as a one-time setup step.

<p/>The RFC says, somewhat briefly, that &ldquo;there may be a header
line&rdquo;. Miller&rsquo;s <tt>--implicit-csv-header</tt> option allows you to
read CSV data which lacks a header line, applying column labels <tt>1</tt>,
<tt>2</tt>, <tt>3</tt>, etc. for you. You may also use Miller&rsquo;s
<tt>label</tt> to replace those numerical column names with labels of your
choosing.

<p/>Here are the differences between CSV and CSV-lite:

<ul>

<li/>CSV supports <a href="https://tools.ietf.org/html/rfc4180">
RFC-4180</a>)-style double-quoting, including the ability to have commas and/or
CR-LF line-endings contained within an input field; CSV-lite does not.

<li/>Default record separator for CSV is CR-LF; default record separator for
CSV-lite is LF.

<li/>CSV does not allow heterogeneous data; CSV-lite does (see also <a
href="record-heterogeneity.html">here</a>).

<li/>The CSV-lite input-reading code is more efficient than the CSV
input-reader.

</ul>

<p/>Here are things they have in common:

<ul>

<li/>The ability to specify record/field separators other than the default,
e.g. CR-LF vs. LF, or tab instead of comma for TSV, and so on.

<li/>The <tt>--implicit-csv-header</tt> flag for input and the
<tt>--headerless-csv-output</tt> flag for output.

</ul>

<h1>Tabular JSON</h1>

<p/>JSON is a format which supports arbitrarily deep nesting of
&ldquo;objects&rdquo; (hashmaps) and &ldquo;arrays&rdquo; (lists), while Miller
is a tool for handling <boldmaroon>tabular data</boldmaroon> only. This means
Miller cannot (and should not) handle arbitrary JSON. (Check out <a
href="http://stedolan.github.io/jq/">jq</a>.)

<p/>But if you have tabular data represented in JSON then Miller can handle that for you.

<h2>Single-level JSON objects</h2>

An <boldmaroon>array of single-level objects</boldmaroon> is, quite simply,
<boldmaroon>a table:</boldmaroon>

POKI_RUN_COMMAND{{mlr --json head -n 2 data/json-example-1.json}}HERE
POKI_RUN_COMMAND{{mlr --json --jvstack head -n 2 then cut -f color,u,v data/json-example-1.json}}HERE
POKI_RUN_COMMAND{{mlr --ijson --opprint stats1 -a mean,stddev,count -f u -g shape data/json-example-1.json}}HERE

<h2>Nested JSON objects</h2>

<p/>Additionally, Miller can <boldmaroon>tabularize nested objects by concatentating keys:</boldmaroon>

POKI_RUN_COMMAND{{mlr --json --jvstack head -n 2 data/json-example-2.json}}HERE
POKI_RUN_COMMAND{{mlr --ijson --opprint head -n 4 data/json-example-2.json}}HERE

<p/>Note in particular that as far as Miller&rsquo;s <tt>put</tt> and <tt>filter</tt>, as well as other
I/O formats, are concerned, these are simply field names with colons in them:

POKI_RUN_COMMAND{{mlr --json --jvstack head -n 1 then put '${values:uv} = ${values:u} * ${values:v}' data/json-example-2.json}}HERE

<h2>Formatting JSON options</h2>

<p/>JSON isn&rsquo;t a parameterized format, so <tt>RS</tt>, <tt>FS</tt>,
<tt>PS</tt> aren&rsquo;t specifiable. Nonetheless, you can do the following:

<ul>

<li/>Use <tt>--jvstack</tt> to pretty-print JSON objects with multi-line
(vertically stacked) spacing. By defaulty, each Miller record (JSON object) is
one per line.

<li/>Use <tt>--jlistwrap</tt> to print the sequence of JSON objects wrapped in
an outermost <tt>[</tt> and <tt>]</tt>. By default, these aren&rsquo;t printed.

<li/>Use <tt>--jquoteall</tt> to double-quote all object values. By default,
integers, floating-point numbers, and booleans <tt>true</tt> and <tt>false</tt>
are not double-quoted when they appear as JSON-object keys.

<li/>Use <tt>--jflatsep yourstringhere</tt> to specify the string used for
key concatenation: this defaults to a single colon.

</ul>

<p/>Again, please see <a href="http://stedolan.github.io/jq/">jq</a> for a
truly powerful, JSON-specific tool.

<h2>JSON non-streaming</h2>

<p/>The JSON parser does not return until all input is parsed: in particular
this means that, unlike for other file formats, Miller does not (at present)
handle JSON files in <tt>tail -f</tt> contexts.

<h1>PPRINT: Pretty-printed tabular</h1>
Miller&rsquo;s pretty-print format is like CSV, but column-aligned.  For example, compare

<table><tr><td>
POKI_RUN_COMMAND{{mlr --ocsv cat data/small}}HERE
</td>
<td>
POKI_RUN_COMMAND{{mlr --opprint cat data/small}}HERE
</td></tr></table>

Note that while Miller is a line-at-a-time processor and retains input lines in
memory only where necessary (e.g. for sort), pretty-print output requires it to
accumulate all input lines (so that it can compute maximum column widths)
before producing any output. This has two consequences: (a) pretty-print output
won&rsquo;t work on <tt>tail -f</tt> contexts, where Miller will be waiting for
an end-of-file marker which never arrives; (b) pretty-print output for large
files is constrained by available machine memory.

<p/> See POKI_PUT_LINK_FOR_PAGE(record-heterogeneity.html)HERE for how Miller
handles changes of field names within a single data stream.

<h1>XTAB: Vertical tabular</h1>

<p/>This is perhaps most useful for looking a very wide and/or multi-column
data which causes line-wraps on the screen (but see also <a
href="https://github.com/twosigma/ngrid">https://github.com/twosigma/ngrid</a>
for an entirely different, very powerful option). Namely:

<table><tr> <td>
POKI_INCLUDE_ESCAPED(data/system-file-opprint-example.txt)HERE
</td></tr> <tr><td>
POKI_INCLUDE_ESCAPED(data/system-file-oxtab-example.txt)HERE
</td></tr> <tr><td>
POKI_INCLUDE_ESCAPED(data/system-file-ojson-example.txt)HERE
</td> </tr></table>

<h1>Markdown tabular</h1>

<p/>Markdown format looks like this:

POKI_RUN_COMMAND{{mlr --omd cat data/small}}HERE

which renders like this when dropped into various web tools (e.g. github comments):

<p/>
<img src="pix/omd.png"/>

<p/> As of Miller 4.3.0, markdown format is supported only for output, not input.