miller/doc/content-for-faq.html

POKI_PUT_TOC_HERE

<h1>Number one FAQ</h1>

<b>
Please use <tt>mlr --csv --rs lf</tt> for native Un*x (linefeed-terminated) CSV files.
</b>

<p/>Instead of specifying <tt>--rs lf</tt> on each invocation, you can instead
have <tt>MLR_CSV_DEFAULT_RS=lf</tt> in your shell environment: e.g. put
<tt>export MLR_CSV_DEFAULT_RS=lf</tt> in your <tt>~/.bashrc</tt> or
<tt>~/.zshrc</tt>, or <tt>setenv MLR_CSV_DEFAULT_RS lf</tt> in your
<tt>~/.cshrc</tt>, as a one-time setup step.

<h1>No output at all</h1>

<p/>Check the line-terminators of the data, e.g. with the command-line
<tt>file</tt> program. Example: for CSV, Miller&rsquo;s default line terminator
is CR/LF (carriage return followed by linefeed, following
<a href="https://tools.ietf.org/html/rfc4180">RFC4180</a>). Yet if your CSV has
*nix-standard LF line endings, Miller will keep reading the file looking for a
CR/LF which never appears. Solution in this case: tell Miller the input has LF line-terminator, e.g. <b>mlr --csv --rs
lf {remaining arguments ...}</b>.

<p/>Also try <tt>od -xcv</tt> and/or <tt>cat -e</tt> on your file to check for non-printable characters.

<h1>Fields not selected</h1>

<p/>Check the field-separators of the data, e.g. with the command-line
<tt>head</tt> program. Example: for CSV, Miller&rsquo;s default record
separator is comma; if your data is tab-delimited, e.g. <tt>aTABbTABc</tt>,
then Miller won&rsquo;t find three fields named <tt>a</tt>, <tt>b</tt>, and
<tt>c</tt> but rather just one named <tt>aTABbTABc</tt>.  Solution in this
case: <tt>mlr --fs tab {remaining arguments ...}</tt>.

<p/>Also try <tt>od -xcv</tt> and/or <tt>cat -e</tt> on your file to check for non-printable characters.

<h1>Diagnosing delimiter specifications</h1>

POKI_INCLUDE_ESCAPED(data/delimiter-examples.txt)HERE

<h1>Error-output in certain string cases</h1>

<p/> <tt>mlr put '$y = string($x)' then put '$z = $y . $y'</tt> gives
<tt>(error)</tt> on numeric data such as <tt>x=123</tt>, while <tt>mlr put
'$z=string($x).string($x)'</tt> and <tt>mlr put '$y = string($x); $z = $y .
$y'</tt> do not.  This is because in the first case <tt>y</tt> is computed and
stored as a string, then re-parsed as an integer, for which
string-concatenation is an invalid operator. In the second case, casts are done
independently; in the third case, both assignments are within the same
<tt>put</tt> statement, where type information is maintained for the duration
of all assignments in the <tt>put</tt>.

<h1>How do I examine then-chaining?</h1>

<p/>Then-chaining found in Miller is intended to function the same as Unix
pipes, but with less keystroking. You can print your data one pipeline step at
a time, to see what intermediate output at one step becomes the input to the
next step.

<p/>First, look at the input data:

POKI_RUN_COMMAND{{cat data/then-example.csv}}HERE

Next, run the first step of your command, omitting anything from the first <tt>then</tt> onward:

POKI_RUN_COMMAND{{mlr --icsv --rs lf --opprint count-distinct -f Status,Payment_Type data/then-example.csv}}HERE

After that, run it with the next <tt>then</tt> step included:

POKI_RUN_COMMAND{{mlr --icsv --rs lf --opprint count-distinct -f Status,Payment_Type then sort -nr count data/then-example.csv}}HERE

Now if you use <tt>then</tt> to include another verb after that, the columns
<tt>Status</tt>, <tt>Payment_Type</tt>, and <tt>count</tt> will be the input to
that verb.

<p/>Note, by the way, that you&rsquo;ll get the same results using pipes:
POKI_RUN_COMMAND{{mlr --csv --rs lf count-distinct -f Status,Payment_Type data/then-example.csv | mlr --icsv --rs lf --opprint sort -nr count}}HERE

<h1>I assigned $9 and it&rsquo;s not 9th</h1>

<p/> Miller records are ordered lists of key-value pairs. For NIDX format, DKVP
format when keys are missing, or CSV/CSV-lite format with
<tt>--implicit-csv-header</tt>, Miller will sequentially assign keys of the
form <tt>1</tt>, <tt>2</tt>, etc. But these are not integer array indices:
they&rsquo;re just field names taken from the initial field ordering in the
input data.

POKI_RUN_COMMAND{{echo x,y,z | mlr --dkvp cat}}HERE
POKI_RUN_COMMAND{{echo x,y,z | mlr --dkvp put '$6="a";$4="b";$55="cde"'}}HERE
POKI_RUN_COMMAND{{echo x,y,z | mlr --nidx cat}}HERE
POKI_RUN_COMMAND{{echo x,y,z | mlr --csv --rs lf --implicit-csv-header cat}}HERE
POKI_RUN_COMMAND{{echo x,y,z | mlr --dkvp rename 2,999}}HERE
POKI_RUN_COMMAND{{echo x,y,z | mlr --dkvp rename 2,newname}}HERE
POKI_RUN_COMMAND{{echo x,y,z | mlr --csv --rs lf --implicit-csv-header reorder -f 3,1,2}}HERE

<h1>Why doesn&rsquo;t mlr cut put fields in the order I want?</h1>

<p/>Example: columns <tt>x,i,a</tt> were requested but they appear here in the order <tt>a,i,x</tt>:

POKI_RUN_COMMAND{{cat data/small}}HERE
POKI_RUN_COMMAND{{mlr cut -f x,i,a data/small}}HERE

<p/>The issue is that Miller&rsquo;s <tt>cut</tt>, by default, outputs cut fields in the order they
appear in the input data. This design decision was made intentionally to parallel the *nix system <tt>cut</tt>
command, which has the same semantics.

<p/>The solution is to use the <tt>-o</tt> option:

POKI_RUN_COMMAND{{mlr cut -o -f x,i,a data/small}}HERE

<h1>NR is not consecutive after then-chaining</h1>

<p/> Given this input data:
POKI_RUN_COMMAND{{cat data/small}}HERE

why don&rsquo;t I see <tt>NR=1</tt> and <tt>NR=2</tt> here??

POKI_RUN_COMMAND{{mlr filter '$x > 0.5' then put '$NR = NR' data/small}}HERE

<p/>The reason is that <tt>NR</tt> is computed for the original input records and isn&rsquo;t dynamically
updated. By contrast, <tt>NF</tt> is dynamically updated: it&rsquo;s the number of fields in the
current record, and if you add/remove a field, the value of <tt>NF</tt> will change:

POKI_RUN_COMMAND{{echo x=1,y=2,z=3 | mlr put '$nf1 = NF; $u = 4; $nf2 = NF; unset $x,$y,$z; $nf3 = NF'}}HERE

<p/><tt>NR</tt>, by contrast (and <tt>FNR</tt> as well), retains the value from the original input stream,
and records may be dropped by a <tt>filter</tt> within a <tt>then</tt>-chain. To recover consecutive record
numbers, you can use out-of-stream variables as follows:

POKI_INCLUDE_AND_RUN_ESCAPED(data/dynamic-nr.sh)HERE

<h1>Why am I not seeing all possible joins occur?</h1>

<p/>For example, the right file here has nine records, and the left file should
add in the <tt>hostname</tt> column &mdash; so the join output should also have
9 records:

POKI_RUN_COMMAND{{mlr --icsvlite --opprint cat data/join-u-left.csv}}HERE
POKI_RUN_COMMAND{{mlr --icsvlite --opprint cat data/join-u-right.csv}}HERE
POKI_RUN_COMMAND{{mlr --icsvlite --opprint join -j ipaddr -f data/join-u-left.csv data/join-u-right.csv}}HERE

<p/>The issue is that Miller&rsquo;s <tt>join</tt>, by default, takes input
sorted (lexically ascending) by the sort keys on both the left and right files.
This design decision was made intentionally to parallel the *nix system
<tt>join</tt> command, which has the same semantics. The benefit of this
default is that the joiner program can stream through the left and right files,
needing to load neither entirely into memory. The drawback, of course, is that
is requires sorted input.

<p/>The solution (besides pre-sorting the input files on the join keys) is to
simply use <b>mlr join -u</b>. This loads the left file entirely into memory
(while the right file is still streamed one line at a time) and does all
possible joins without requiring sorted input:

POKI_RUN_COMMAND{{mlr --icsvlite --opprint join -u -j ipaddr -f data/join-u-left.csv data/join-u-right.csv}}HERE

<p/>General advice is to make sure the left-file is relatively small, e.g.
containing name-to-number mappings, while saving large amounts of data for the
right file.

<h1>What about XML or JSON file formats?</h1>

<p/>Miller handles <boldmaroon>tabular data</boldmaroon>, which is a list of
records each having fields which are key-value pairs. Miller also doesn&rsquo;t
require that each record have the same field names (see also <a
href="record-heterogeneity.html">here</a>). Regardless, tabular data is a
<boldmaroon>non-recursive data structure</boldmaroon>.

<p/> XML, JSON, etc. are, by contrast, all <boldmaroon>recursive</boldmaroon>
or <boldmaroon>nested</boldmaroon> data structures. For example, in JSON
you can represent a hash map whose values are lists of lists.

<p/>Now, you can put tabular data into these formats &mdash; since list-of-key-value-pairs
is one of the things representable in XML or JSON. Example:

<p/>
<div class="pokipanel">
<pre>
# DKVP
x=1,y=2
z=3

# XML
&lt;table&gt;
  &lt;record&gt;
    &lt;field&gt;
      &lt;key&gt; x &lt;/key&gt; &lt;value&gt; 1 &lt;/value&gt;
    &lt;/field&gt;
    &lt;field&gt;
      &lt;key&gt; y &lt;/key&gt; &lt;value&gt; 2 &lt;/value&gt;
    &lt;/field&gt;
  &lt;/record&gt;
    &lt;field&gt;
      &lt;key&gt; z &lt;/key&gt; &lt;value&gt; 3 &lt;/value&gt;
    &lt;/field&gt;
  &lt;record&gt;
  &lt;/record&gt;
&lt;/table&gt;

# JSON
[{"x":1,"y":2},{"z":3}]
</pre>
</div>

<p/>However, a tool like Miller which handles non-recursive data is never going
to be able to handle full XML/JSON semantics &mdash; only a small subset.  If
tabular data represented in XML/JSON/etc are sufficiently well-structured, it
may be easy to grep/sed out the data into a simpler text form &mdash; this is a
general text-processing problem.

<p/>Miller does support tabular data represented in JSON: please see
POKI_PUT_LINK_FOR_PAGE(file-formats.html)HERE.  See als <a
href="http://stedolan.github.io/jq/">jq</a> for a truly powerful, JSON-specific
tool.

<p/>For XML, my suggestion is to use a tool like
<a href="http://ff-extractor.sourceforge.net/">ff-extractor</a> to do format
conversion.