miller/doc/content-for-faq.html

POKI_PUT_TOC_HERE

<p/>
<button style="font-weight:bold;color:maroon;border:0" onclick="vis_expand_all_body_sections();" href="javascript:;">Expand all sections</button>
<button style="font-weight:bold;color:maroon;border:0" onclick="vis_collapse_all_body_sections();" href="javascript:;">Collapse all sections</button>

<h1>No output at all</h1>
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="vis_toggle_by_name('body_section_toggle_no_output_at_all');" href="javascript:;">Toggle section visibility</button>
<div id="body_section_toggle_no_output_at_all" style="display: block">

<p/>Try <code>od -xcv</code> and/or <code>cat -e</code> on your file to check for non-printable characters.

<p/>If you&rsquo;re using Miller version less than 5.0.0 (try
<code>mlr --version</code> on your system to find out), when the
line-ending-autodetect feature was introduced, please see
<a href="http://johnkerl.org/miller-releases/miller-4.5.0/doc/index.html">here</a>.

</div>
<h1>Fields not selected</h1>
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="vis_toggle_by_name('body_section_toggle_fields_not_selected');" href="javascript:;">Toggle section visibility</button>
<div id="body_section_toggle_fields_not_selected" style="display: block">

<p/>Check the field-separators of the data, e.g. with the command-line
<code>head</code> program. Example: for CSV, Miller&rsquo;s default record
separator is comma; if your data is tab-delimited, e.g. <code>aTABbTABc</code>,
then Miller won&rsquo;t find three fields named <code>a</code>, <code>b</code>, and
<code>c</code> but rather just one named <code>aTABbTABc</code>.  Solution in this
case: <code>mlr --fs tab {remaining arguments ...}</code>.

<p/>Also try <code>od -xcv</code> and/or <code>cat -e</code> on your file to check for non-printable characters.

</div>
<h1>Diagnosing delimiter specifications</h1>
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="vis_toggle_by_name('body_section_toggle_diagnosing_delimiter_specifications');" href="javascript:;">Toggle section visibility</button>
<div id="body_section_toggle_diagnosing_delimiter_specifications" style="display: block">

POKI_INCLUDE_ESCAPED(data/delimiter-examples.txt)HERE

</div>
<h1>How do I suppress numeric conversion?</h1>
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="vis_toggle_by_name('body_section_toggle_examine_then_chaining');" href="javascript:;">Toggle section visibility</button>
<div id="body_section_toggle_suppress_numeric_conversion" style="display: block">

<p/><b>TL;DR use put -S</b>.

<p/> Within <code>mlr put</code> and <code>mlr filter</code>, the default behavior for
scanning input records is to parse them as integer, if possible, then as float,
if possible, else leave them as string:

POKI_RUN_COMMAND{{cat data/scan-example-1.tbl}}HERE

POKI_RUN_COMMAND{{mlr --pprint put '$copy = $value; $type = typeof($value)' data/scan-example-1.tbl}}HERE

<p/>The numeric-conversion rule is simple:

<ul>
<li/> Try to scan as integer (<code>"1"</code> should be int);
<li/> if that doesn&rsquo;t succeed, try to scan as float (<code>"1.0"</code> should be float);
<li/> if that doesn&rsquo;t succeed, leave the value as a string (<code>"1x"</code> is string).
</ul>

<p/>This is a sensible default: you should be able to put <code>'$z = $x +
$y'</code> without having to write <code>'$z = int($x) + float($y)'</code>.  Also
note that default output format for floating-point numbers created by
<code>put</code> (and other verbs such as <code>stats1</code>) is six decimal places;
you can override this using <code>mlr --ofmt</code>.  Also note that Miller uses
your system&rsquo;s C library functions whenever possible: e.g. <code>sscanf</code>
for converting strings to integer or floating-point.

<p/>But now suppose you have data like these:

POKI_RUN_COMMAND{{cat data/scan-example-2.tbl}}HERE

POKI_RUN_COMMAND{{mlr --pprint put '$copy = $value; $type = typeof($value)' data/scan-example-2.tbl}}HERE

<p/> The same conversion rules as above are being used. Namely:

<ul>
<li/> By default field values are inferred to int, else float, else string;

<li/> leading zeroes indicate octal for integers (<code>sscanf</code> semantics);

<li/> since <code>0008</code> doesn't scan as integer (leading 0 requests octal but 8
isn't a valid octal digit), the float scan is tried next and it succeeds;

<li/> default floating-point output format is 6 decimal places (override with <code>mlr --ofmt</code>).
</ul>

<p/> Taken individually the rules make sense; taken collectively they produce a mishmash of types here.

<p/>The solution is to <b>use the -S flag</b> for <code>mlr put</code> and/or <code>mlr filter</code>.
Then all field values are left as string. You can type-coerce on demand using syntax like
<code>'$z = int($x) + float($y)'</code>. (See also the
<a href="reference-verbs.html#put">put documentation</a>; see also
<a href="https://github.com/johnkerl/miller/issues/150">https://github.com/johnkerl/miller/issues/150</a>.)

POKI_RUN_COMMAND{{mlr --pprint put -S '$copy = $value; $type = typeof($value)' data/scan-example-2.tbl}}HERE

</div>
<h1>How do I examine then-chaining?</h1>
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="vis_toggle_by_name('body_section_toggle_examine_then_chaining');" href="javascript:;">Toggle section visibility</button>
<div id="body_section_toggle_examine_then_chaining" style="display: block">

<p/>Then-chaining found in Miller is intended to function the same as Unix
pipes, but with less keystroking. You can print your data one pipeline step at
a time, to see what intermediate output at one step becomes the input to the
next step.

<p/>First, look at the input data:

POKI_RUN_COMMAND{{cat data/then-example.csv}}HERE

Next, run the first step of your command, omitting anything from the first <code>then</code> onward:

POKI_RUN_COMMAND{{mlr --icsv --opprint count-distinct -f Status,Payment_Type data/then-example.csv}}HERE

After that, run it with the next <code>then</code> step included:

POKI_RUN_COMMAND{{mlr --icsv --opprint count-distinct -f Status,Payment_Type then sort -nr count data/then-example.csv}}HERE

Now if you use <code>then</code> to include another verb after that, the columns
<code>Status</code>, <code>Payment_Type</code>, and <code>count</code> will be the input to
that verb.

<p/>Note, by the way, that you&rsquo;ll get the same results using pipes:
POKI_RUN_COMMAND{{mlr --csv count-distinct -f Status,Payment_Type data/then-example.csv | mlr --icsv --opprint sort -nr count}}HERE

</div>
<h1>I assigned $9 and it&rsquo;s not 9th</h1>
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="vis_toggle_by_name('body_section_toggle_9_not_9th');" href="javascript:;">Toggle section visibility</button>
<div id="body_section_toggle_9_not_9th" style="display: block">

<p/> Miller records are ordered lists of key-value pairs. For NIDX format, DKVP
format when keys are missing, or CSV/CSV-lite format with
<code>--implicit-csv-header</code>, Miller will sequentially assign keys of the
form <code>1</code>, <code>2</code>, etc. But these are not integer array indices:
they&rsquo;re just field names taken from the initial field ordering in the
input data.

POKI_RUN_COMMAND{{echo x,y,z | mlr --dkvp cat}}HERE
POKI_RUN_COMMAND{{echo x,y,z | mlr --dkvp put '$6="a";$4="b";$55="cde"'}}HERE
POKI_RUN_COMMAND{{echo x,y,z | mlr --nidx cat}}HERE
POKI_RUN_COMMAND{{echo x,y,z | mlr --csv --implicit-csv-header cat}}HERE
POKI_RUN_COMMAND{{echo x,y,z | mlr --dkvp rename 2,999}}HERE
POKI_RUN_COMMAND{{echo x,y,z | mlr --dkvp rename 2,newname}}HERE
POKI_RUN_COMMAND{{echo x,y,z | mlr --csv --implicit-csv-header reorder -f 3,1,2}}HERE

</div>
<h1>How can I filter by date?</h1>
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="vis_toggle_by_name('body_section_toggle_date_filtering');" href="javascript:;">Toggle section visibility</button>
<div id="body_section_toggle_date_filtering" style="display: block">

<p/> Given input like

POKI_RUN_COMMAND{{cat dates.csv}}HERE

we can use <code>strptime</code> to parse the date field into seconds-since-epoch
and then do numeric comparisons.  Simply match your input dataset&rsquo;s
date-formatting to the <a href="reference-dsl.html#strptime">strptime</a>
format-string.  For example:

POKI_RUN_COMMAND{{mlr --csv filter 'strptime($date, "%Y-%m-%d") > strptime("2018-03-03", "%Y-%m-%d")' dates.csv}}HERE

<p/>Caveat: localtime-handling in timezones with DST is still a work in progress; see
<a href="https://github.com/johnkerl/miller/issues/170">https://github.com/johnkerl/miller/issues/170</a>.
See also <a href="https://github.com/johnkerl/miller/issues/208">https://github.com/johnkerl/miller/issues/208</a>
&mdash; thanks @aborruso!

</div>
<h1>How can I handle commas-as-data in various formats?</h1>
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="vis_toggle_by_name('body_section_toggle_comma_handling');" href="javascript:;">Toggle section visibility</button>
<div id="body_section_toggle_comma_handling" style="display: block">

<p/> <a href="file-formats.html#CSV/TSV/ASV/USV/etc.">CSV</a> handles this well and by design:

POKI_RUN_COMMAND{{cat commas.csv}}HERE

<p/> Likewise <a href="file-formats.html#Tabular_JSON">JSON</a>:

POKI_RUN_COMMAND{{mlr --icsv --ojson cat commas.csv}}HERE

<p/> For Miller&rsquo;s <a href="file-formats.html#XTAB:_Vertical_tabular">XTAB</a>
there is no escaping for carriage returns, but commas work fine:

POKI_RUN_COMMAND{{mlr --icsv --oxtab cat commas.csv}}HERE

<p/> But for <a href="file-formats.html#DKVP:_Key-value_pairs">DKVP</a>
and
<a href="file-formats.html#NIDX:_Index-numbered_(toolkit_style)">NIDX</a>, commas
are the default field separator. And &mdash; as of Miller 5.4.0 anyway &mdash;
there is no CSV-style double-quote-handling like there is for CSV. So commas within the data
look like delimiters:

POKI_RUN_COMMAND{{mlr --icsv --odkvp cat commas.csv}}HERE

<p/> One solution is to use a different delimiter, such as a pipe character:

POKI_RUN_COMMAND{{mlr --icsv --odkvp --ofs pipe cat commas.csv}}HERE

<p/> To be extra-sure to avoid data/delimiter clashes, you can also use control
characters as delimiters &mdash; here, control-A:

POKI_RUN_COMMAND{{mlr --icsv --odkvp --ofs '\001'  cat commas.csv | cat -v}}HERE

</div>
<h1>How can I handle field names with special symbols in them?</h1>
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="vis_toggle_by_name('body_section_toggle_field_names_with_special_symbols');" href="javascript:;">Toggle section visibility</button>
<div id="body_section_toggle_field_names_with_special_symbols" style="display: block">

<p/>Simply surround the field names with curly braces:

POKI_RUN_COMMAND{{echo 'x.a=3,y:b=4,z/c=5' | mlr put '${product.all} = ${x.a} * ${y:b} * ${z/c}'}}HERE

</div>
<h1>How to escape '?' in regexes?</h1>
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="vis_toggle_by_name('body_section_toggle_escape_in_regex');" href="javascript:;">Toggle section visibility</button>
<div id="body_section_toggle_escape_in_regex" style="display: block">

<p/> One way is to use square brackets; an alternative is to use simple
string-substitution rather than a regular expression.

POKI_RUN_COMMAND{{cat data/question.dat}}HERE
POKI_RUN_COMMAND{{mlr --oxtab put '$c = gsub($a, "[?]"," ...")' data/question.dat}}HERE
POKI_RUN_COMMAND{{mlr --oxtab put '$c = ssub($a, "?"," ...")' data/question.dat}}HERE

<p/> The <code>ssub</code> function exists precisely for this reason: so you don&rsquo;t have to escape anything.

</div>
<h1>How can I put single-quotes into strings?</h1>
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="vis_toggle_by_name('body_section_toggle_single_quotes_in_strings');" href="javascript:;">Toggle section visibility</button>
<div id="body_section_toggle_single_quotes_in_strings" style="display: block">

<p/> This is a little tricky due to the shell&rsquo;s handling of quotes. For simplicity, let&rsquo;s first put
an update script into a file:

POKI_INCLUDE_ESCAPED(data/single-quote-example.mlr)HERE

POKI_RUN_COMMAND{{echo a=bcd | mlr put -f data/single-quote-example.mlr}}HERE

<p/>So, it&rsquo;s simple: Miller&rsquo;s DSL uses double quotes for strings,
and you can put single quotes (or backslash-escaped double-quotes) inside
strings, no problem.

<p/> Without putting the update expression in a file, it&rsquo;s messier:

POKI_RUN_COMMAND{{echo a=bcd | mlr put '$a="It'\''s OK, I said, '\''for now'\''."'}}HERE

<p/> The idea is that the outermost single-quotes are to protect the
<code>put</code> expression from the shell, and the double quotes within them are
for Miller. To get a single quote in the middle there, you need to actually put it <i>outside</i> the single-quoting
for the shell. The pieces are

<ul>
<li/> <code>$a="It</code>
<li/> <code>\'</code>
<li/> <code>s OK, I said,</code>
<li/> <code>\'</code>
<li/> <code>for now</code>
<li/> <code>\'</code>
<li/> <code>.</code>
</ul>

all concatenated together.

</div>
<h1>Why doesn&rsquo;t mlr cut put fields in the order I want?</h1>
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="vis_toggle_by_name('body_section_toggle_cut_out_of_order');" href="javascript:;">Toggle section visibility</button>
<div id="body_section_toggle_cut_out_of_order" style="display: block">

<p/>Example: columns <code>x,i,a</code> were requested but they appear here in the order <code>a,i,x</code>:

POKI_RUN_COMMAND{{cat data/small}}HERE
POKI_RUN_COMMAND{{mlr cut -f x,i,a data/small}}HERE

<p/>The issue is that Miller&rsquo;s <code>cut</code>, by default, outputs cut fields in the order they
appear in the input data. This design decision was made intentionally to parallel the *nix system <code>cut</code>
command, which has the same semantics.

<p/>The solution is to use the <code>-o</code> option:

POKI_RUN_COMMAND{{mlr cut -o -f x,i,a data/small}}HERE

</div>
<h1>NR is not consecutive after then-chaining</h1>
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="vis_toggle_by_name('body_section_toggle_NR_not_consecutive_after_then');" href="javascript:;">Toggle section visibility</button>
<div id="body_section_toggle_NR_not_consecutive_after_then" style="display: block">

<p/> Given this input data:
POKI_RUN_COMMAND{{cat data/small}}HERE

why don&rsquo;t I see <code>NR=1</code> and <code>NR=2</code> here??

POKI_RUN_COMMAND{{mlr filter '$x > 0.5' then put '$NR = NR' data/small}}HERE

<p/>The reason is that <code>NR</code> is computed for the original input records and isn&rsquo;t dynamically
updated. By contrast, <code>NF</code> is dynamically updated: it&rsquo;s the number of fields in the
current record, and if you add/remove a field, the value of <code>NF</code> will change:

POKI_RUN_COMMAND{{echo x=1,y=2,z=3 | mlr put '$nf1 = NF; $u = 4; $nf2 = NF; unset $x,$y,$z; $nf3 = NF'}}HERE

<p/><code>NR</code>, by contrast (and <code>FNR</code> as well), retains the value from the original input stream,
and records may be dropped by a <code>filter</code> within a <code>then</code>-chain. To recover consecutive record
numbers, you can use out-of-stream variables as follows:

POKI_INCLUDE_AND_RUN_ESCAPED(data/dynamic-nr.sh)HERE

<p/>Or, simply use <code>mlr cat -n</code>:

POKI_RUN_COMMAND{{mlr filter '$x > 0.5' then cat -n data/small}}HERE

</div>
<h1>Why am I not seeing all possible joins occur?</h1>
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="vis_toggle_by_name('body_section_toggle_not_all_possible_joins');" href="javascript:;">Toggle section visibility</button>
<div id="body_section_toggle_not_all_possible_joins" style="display: block">

<p/><b>This section describes behavior before Miller 5.1.0. As of 5.1.0, <code>-u</code> is the default.</b>

<p/>For example, the right file here has nine records, and the left file should
add in the <code>hostname</code> column &mdash; so the join output should also have
9 records:

POKI_RUN_COMMAND{{mlr --icsvlite --opprint cat data/join-u-left.csv}}HERE
POKI_RUN_COMMAND{{mlr --icsvlite --opprint cat data/join-u-right.csv}}HERE
POKI_RUN_COMMAND{{mlr --icsvlite --opprint join -s -j ipaddr -f data/join-u-left.csv data/join-u-right.csv}}HERE

<p/>The issue is that Miller&rsquo;s <code>join</code>, by default (before 5.1.0),
took input sorted (lexically ascending) by the sort keys on both the left and
right files.  This design decision was made intentionally to parallel the *nix
system <code>join</code> command, which has the same semantics. The benefit of this
default is that the joiner program can stream through the left and right files,
needing to load neither entirely into memory. The drawback, of course, is that
is requires sorted input.

<p/>The solution (besides pre-sorting the input files on the join keys) is to
simply use <b>mlr join -u</b> (which is now the default). This loads the left
file entirely into memory (while the right file is still streamed one line at a
time) and does all possible joins without requiring sorted input:

POKI_RUN_COMMAND{{mlr --icsvlite --opprint join -u -j ipaddr -f data/join-u-left.csv data/join-u-right.csv}}HERE

<p/>General advice is to make sure the left-file is relatively small, e.g.
containing name-to-number mappings, while saving large amounts of data for the
right file.

</div>
<h1>How to rectangularize after joins with unpaired?</h1>
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="vis_toggle_by_name('body_section_toggle_rectangularize_after_join');" href="javascript:;">Toggle section visibility</button>
<div id="body_section_toggle_rectangularize_after_join" style="display: block">

<p/> Suppose you have the following two data files:

POKI_INCLUDE_ESCAPED(data/color-codes.csv)HERE
POKI_INCLUDE_ESCAPED(data/color-names.csv)HERE

<p/> Joining on color the results are as expected:

POKI_RUN_COMMAND{{mlr --csv join -j id -f data/color-codes.csv data/color-names.csv}}HERE

<p/> However, if we ask for left-unpaireds, since there&rsquo;s no
<code>color</code> column, we get a row not having the same column names as the
other:

POKI_RUN_COMMAND{{mlr --csv join --ul -j id -f data/color-codes.csv data/color-names.csv}}HERE

<p/> To fix this, we can use <b>unsparsify</b>:

POKI_RUN_COMMAND{{mlr --csv join --ul -j id -f data/color-codes.csv then unsparsify --fill-with "" data/color-names.csv}}HERE

<p/> Thanks to @aborruso for the tip!

</div>
<h1>What about XML or JSON file formats?</h1>
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="vis_toggle_by_name('body_section_toggle_xml_or_json');" href="javascript:;">Toggle section visibility</button>
<div id="body_section_toggle_xml_or_json" style="display: block">

<p/>Miller handles <boldmaroon>tabular data</boldmaroon>, which is a list of
records each having fields which are key-value pairs. Miller also doesn&rsquo;t
require that each record have the same field names (see also <a
href="record-heterogeneity.html">here</a>). Regardless, tabular data is a
<boldmaroon>non-recursive data structure</boldmaroon>.

<p/> XML, JSON, etc. are, by contrast, all <boldmaroon>recursive</boldmaroon>
or <boldmaroon>nested</boldmaroon> data structures. For example, in JSON
you can represent a hash map whose values are lists of lists.

<p/>Now, you can put tabular data into these formats &mdash; since list-of-key-value-pairs
is one of the things representable in XML or JSON. Example:

<p/>
<div class="pokipanel">
<pre>
# DKVP
x=1,y=2
z=3

# XML
&lt;table&gt;
  &lt;record&gt;
    &lt;field&gt;
      &lt;key&gt; x &lt;/key&gt; &lt;value&gt; 1 &lt;/value&gt;
    &lt;/field&gt;
    &lt;field&gt;
      &lt;key&gt; y &lt;/key&gt; &lt;value&gt; 2 &lt;/value&gt;
    &lt;/field&gt;
  &lt;/record&gt;
    &lt;field&gt;
      &lt;key&gt; z &lt;/key&gt; &lt;value&gt; 3 &lt;/value&gt;
    &lt;/field&gt;
  &lt;record&gt;
  &lt;/record&gt;
&lt;/table&gt;

# JSON
[{"x":1,"y":2},{"z":3}]
</pre>
</div>

<p/>However, a tool like Miller which handles non-recursive data is never going
to be able to handle full XML/JSON semantics &mdash; only a small subset.  If
tabular data represented in XML/JSON/etc are sufficiently well-structured, it
may be easy to grep/sed out the data into a simpler text form &mdash; this is a
general text-processing problem.

<p/>Miller does support tabular data represented in JSON: please see
POKI_PUT_LINK_FOR_PAGE(file-formats.html)HERE.  See also <a
href="http://stedolan.github.io/jq/">jq</a> for a truly powerful, JSON-specific
tool.

<p/>For XML, my suggestion is to use a tool like
<a href="http://ff-extractor.sourceforge.net/">ff-extractor</a> to do format
conversion.

</div>