miller/doc/content-for-cookbook.html

<p/>
<center>
<boldmaroon>Common patterns</boldmaroon>
</center>

POKI_PUT_TOC_HERE

<p/>
<button style="font-weight:bold;color:maroon;border:0" onclick="expand_all();" href="javascript:;">Expand all sections</button>
<button style="font-weight:bold;color:maroon;border:0" onclick="collapse_all();" href="javascript:;">Collapse all sections</button>

<!-- ================================================================ -->
<h1>Headerless CSV on input or output</h1>
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="toggle_by_name('section_toggle_headerless_csv');" href="javascript:;">Toggle section visibility</button>
<div id="section_toggle_headerless_csv" style="display: block">

<p/>Sometimes we get CSV files which lack a header. For example:

POKI_RUN_COMMAND{{cat data/headerless.csv}}HERE

<p/> You can use Miller to add a header. The <tt>--implicit-csv-header</tt> applies positionally indexed labels:

POKI_RUN_COMMAND{{mlr --csv --implicit-csv-header cat data/headerless.csv}}HERE
POKI_RUN_COMMAND{{mlr --icsv --implicit-csv-header --opprint cat data/headerless.csv}}HERE

<p/> Following that, you can rename the positionally indexed labels to names with meaning for your context.
For example:

POKI_RUN_COMMAND{{mlr --csv --implicit-csv-header label name,age,status data/headerless.csv}}HERE
POKI_RUN_COMMAND{{mlr --icsv --implicit-csv-header --opprint label name,age,status data/headerless.csv}}HERE

<p/> Likewise, if you need to produce CSV which is lacking its header, you can pipe Miller&rsquo;s output
to the system command <tt>sed 1d</tt>, or you can use Miller&rsquo;s <tt>--headerless-csv-output</tt> option:

POKI_RUN_COMMAND{{head -5 data/colored-shapes.dkvp | mlr --ocsv cat}}HERE
POKI_RUN_COMMAND{{head -5 data/colored-shapes.dkvp | mlr --ocsv --headerless-csv-output cat}}HERE

<!-- ================================================================ -->
</div>
<h1>Bulk rename of fields</h1>
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="toggle_by_name('section_toggle_bulk_rename');" href="javascript:;">Toggle section visibility</button>
<div id="section_toggle_bulk_rename" style="display: block">

<p/>Suppose you want to replace spaces with underscores in your column names:

POKI_RUN_COMMAND{{cat data/spaces.csv}}HERE

<p/>The simplest way is to use <tt>mlr rename</tt> with <tt>-g</tt> (for global
replace, not just first occurrence of space within each field) and <tt>-r</tt>
for pattern-matching (rather than explicit single-column renames):

POKI_RUN_COMMAND{{mlr --csv rename -g -r ' ,_'  data/spaces.csv}}HERE
POKI_RUN_COMMAND{{mlr --csv --opprint rename -g -r ' ,_'  data/spaces.csv}}HERE

<p/>You can also do this with a for-loop but it puts the modified fields after the unmodified fields:

POKI_RUN_COMMAND{{cat data/bulk-rename-for-loop.mlr}}HERE

POKI_RUN_COMMAND{{mlr --icsv --opprint put -f data/bulk-rename-for-loop.mlr data/spaces.csv}}HERE

<!-- ================================================================ -->
</div>
<h1>Numbering and renumbering records</h1>
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="toggle_by_name('section_toggle_renumbering_records');" href="javascript:;">Toggle section visibility</button>
<div id="section_toggle_renumbering_records" style="display: block">

<p/> The <tt>awk</tt>-like built-in variable <tt>NR</tt> is incremented for each input record:

POKI_RUN_COMMAND{{cat data/small}}HERE
POKI_RUN_COMMAND{{mlr put '$nr = NR' data/small}}HERE

<p/> However, this is the record number within the original input stream
&mdash; not after any filtering you may have done:

POKI_RUN_COMMAND{{mlr filter '$a == "wye"' then put '$nr = NR' data/small}}HERE

<p/> There are two good options here. One is to use the <tt>cat</tt> verb with <tt>-n</tt>:

POKI_RUN_COMMAND{{mlr filter '$a == "wye"' then cat -n data/small}}HERE

<p/> The other is to keep your own counter within the <tt>put</tt> DSL:

POKI_RUN_COMMAND{{mlr filter '$a == "wye"' then put 'begin {@n = 1} $n = @n; @n += 1' data/small}}HERE

<p/> The difference is a matter of taste (although <tt>mlr cat -n</tt> puts the counter first).

<!-- ================================================================ -->
</div>
<h1>Data-cleaning examples</h1>
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="toggle_by_name('section_toggle_data_cleaning_examples');" href="javascript:;">Toggle section visibility</button>
<div id="section_toggle_data_cleaning_examples" style="display: block">

<p/> Here are some ways to use the type-checking options as described in
the POKI_PUT_LINK_FOR_PAGE(reference-dsl.html#Type-test_and_type-assertion_expressions)HERE.
Suppose you have the following data file, with inconsistent typing for boolean.
(Also imagine that, for the sake of discussion, we have a million-line file
rather than a four-line file, so we can&rsquo;t see it all at once and some
automation is called for.)

POKI_RUN_COMMAND{{cat data/het-bool.csv}}HERE

<p/> One option is to coerce everything to boolean, or integer:

POKI_RUN_COMMAND{{mlr --icsv --opprint put '$reachable = boolean($reachable)' data/het-bool.csv}}HERE
POKI_RUN_COMMAND{{mlr --icsv --opprint put '$reachable = int(boolean($reachable))' data/het-bool.csv}}HERE

<p/> A second option is to flag badly formatted data within the output stream:

POKI_RUN_COMMAND{{mlr --icsv --opprint put '$format_ok = is_string($reachable)' data/het-bool.csv}}HERE

<p/> Or perhaps to flag badly formatted data outside the output stream:

POKI_RUN_COMMAND{{mlr --icsv --opprint put 'if (!is_string($reachable)) {eprint "Malformed at NR=".NR} ' data/het-bool.csv}}HERE

<p/> A third way is to abort the process on first instance of bad data:

POKI_RUN_COMMAND_TOLERATING_ERROR{{mlr --csv put '$reachable = asserting_string($reachable)' data/het-bool.csv}}HERE

</div>
<!-- ================================================================ -->
<h1>Finding missing dates</h1>
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="toggle_by_name('section_toggle_finding_missing_dates');" href="javascript:;">Toggle section visibility</button>
<div id="section_toggle_finding_missing_dates" style="display: block">

<p/>Suppose you have some date-stamped data which may (or may not) be missing entries for one or more dates:

POKI_RUN_COMMAND{{head -n 10 data/miss-date.csv}}HERE
POKI_RUN_COMMAND{{wc -l data/miss-date.csv}}HERE

<p/>Since there are 1372 lines in the data file, some automation is called for.
To find the missing dates, you can convert the dates to seconds since the epoch
using <tt>strptime</tt>, then compute adjacent differences (the <tt>cat -n</tt>
simply inserts record-counters):

POKI_INCLUDE_AND_RUN_ESCAPED(data/miss-date-1.sh)HERE

<p/>Then, filter for adjacent difference not being 86400 (the number of seconds in a day):

POKI_INCLUDE_AND_RUN_ESCAPED(data/miss-date-2.sh)HERE

<p/> Given this, it&rsquo;s now easy to see where the gaps are:

POKI_RUN_COMMAND{{mlr cat -n then filter '$n >= 770 && $n <= 780' data/miss-date.csv}}HERE
POKI_RUN_COMMAND{{mlr cat -n then filter '$n >= 1115 && $n <= 1125' data/miss-date.csv}}HERE

</div>
<!-- ================================================================ -->
<h1>Two-pass algorithms</h1>
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="toggle_by_name('section_toggle_two_pass_algorithms');" href="javascript:;">Toggle section visibility</button>
<div id="section_toggle_two_pass_algorithms" style="display: block">

<p/>Miller is a streaming record processor; commands are performed once per
record. This makes Miller particularly suitable for single-pass algorithms,
allowing many of its verbs to process files that are (much) larger than the
amount of RAM present in your system. (Of course, Miller verbs such as
<tt>sort</tt>, </tt>tac</tt>, etc. all must ingest and retain all input records
before emitting any output records.) You can also use out-of-stream variables
to perform multi-pass computations, at the price of retaining all input records
in memory.

<h2>Two-pass algorithms: computation of percentages</h2>

<p/> For example, mapping numeric values down a column to the percentage
between their min and max values is two-pass: on the first pass you find the
min and max values, then on the second, map each record&rsquo;s value to a
percentage.

POKI_INCLUDE_AND_RUN_ESCAPED(data/two-pass-percentage.sh)HERE

<h2>Two-pass algorithms: line-number ratios</h2>

<p/>Similarly, finding the total record count requires first reading through
all the data:

POKI_INCLUDE_AND_RUN_ESCAPED(data/two-pass-record-numbers.sh)HERE

<h2>Two-pass algorithms: records having max value</h2>

<p/>The idea is to retain records having the largest value of <tt>n</tt> in the
following data:

POKI_RUN_COMMAND{{mlr --itsv --opprint cat data/maxrows.tsv}}HERE

<p/>Of course, the largest value of <tt>n</tt> isn&rsquo;t known until after
all data have been read. Using an out-of-stream variable we can retain all
records as they are read, then filter them at the end:

POKI_RUN_COMMAND{{cat data/maxrows.mlr}}HERE

POKI_RUN_COMMAND{{mlr --itsv --opprint put -q -f data/maxrows.mlr data/maxrows.tsv}}HERE

<!-- ================================================================ -->
</div>
<h1>Rectangularizing data</h1>
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="toggle_by_name('section_toggle_rectangularizing_data');" href="javascript:;">Toggle section visibility</button>
<div id="section_toggle_rectangularizing_data" style="display: block">

<p/>Suppose you have a method (in whatever language) which is printing things of the form

POKI_INCLUDE_ESCAPED(data/rect-outer.txt)HERE

and then calls another method which prints things of the form

POKI_INCLUDE_ESCAPED(data/rect-middle.txt)HERE

and then, perhaps, that second method calls a third method which prints things of the form

POKI_INCLUDE_ESCAPED(data/rect-inner.txt)HERE

with the result that your program&rsquo;s output is

POKI_INCLUDE_ESCAPED(data/rect.txt)HERE

The idea here is that middles starting with a 1 belong to the outer value of 1,
and so on.  (For example, the outer values might be account IDs, the middle
values might be invoice IDs, and the inner values might be invoice line-items.)
If you want all the middle and inner lines to have the context of which outers
they belong to, you can modify your software to pass all those through your
methods. Alternatively, don&rsquo;t refactor your code just to handle some
ad-hoc log-data formatting &mdash; instead, use the following to rectangularize
the data.  The idea is to use an out-of-stream variable to accumulate fields
across records. Clear that variable when you see an outer ID; accumulate
fields; emit output when you see the inner IDs.

POKI_INCLUDE_AND_RUN_ESCAPED(data/rect.sh)HERE

<!-- ================================================================ -->
</div>
<h1>Regularizing ragged CSV</h1>
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="toggle_by_name('section_toggle_rectangularizing_ragged_csv');" href="javascript:;">Toggle section visibility</button>
<div id="section_toggle_rectangularizing_ragged_csv" style="display: block">

<p/>Miller handles compliant CSV: in particular, it&rsquo;s an error if the
number of data fields in a given data line don&rsquo;t match the number of
header lines. But in the event that you have a CSV file in which some lines
have less than the full number of fields, you can use Miller to pad them out.
The trick is to use NIDX format, for which each line stands on its own without
respect to a header line.

POKI_RUN_COMMAND{{cat data/ragged.csv}}HERE
POKI_INCLUDE_AND_RUN_ESCAPED(data/ragged-csv.sh)HERE

or, more simply,

POKI_INCLUDE_AND_RUN_ESCAPED(data/ragged-csv-2.sh)HERE

</div>
<!-- ================================================================ -->
<h1>Parsing log-file output</h1>
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="toggle_by_name('section_toggle_parsing_log_file_output');" href="javascript:;">Toggle section visibility</button>
<div id="section_toggle_parsing_log_file_output" style="display: block">

<p/>This, of course, depends highly on what&rsquo;s in your log files. But, as
an example, suppose you have log-file lines such as

POKI_CARDIFY(2015-10-08 08:29:09,445 INFO com.company.path.to.ClassName @ [sometext] various/sorts/of data {& punctuation} hits=1 status=0 time=2.378)HERE

I prefer to pre-filter with <tt>grep</tt> and/or <tt>sed</tt> to extract the structured text, then hand that to Miller. Example:

POKI_CARDIFY(grep 'various sorts' *.log | sed 's/.*} //' | mlr --fs space --repifs --oxtab stats1 -a min,p10,p50,p90,max -f time -g status)HERE

</div>
<!-- ================================================================ -->
<h1>Memoization with out-of-stream variables</h1>
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="toggle_by_name('section_toggle_memoization_with_oosvars');" href="javascript:;">Toggle section visibility</button>
<div id="section_toggle_memoization_with_oosvars" style="display: block">

<p/> The recursive function for the Fibonacci sequence is famous for its computational complexity.
Namely, using
<i>f</i>(0)=1,
<i>f</i>(1)=1,
<i>f</i>(<i>n</i>)=<i>f</i>(<i>n</i>-1)+<i>f</i>(<i>n</i>-2) for <i>n</i>&ge;2,
the evaluation tree branches left as well as right at each non-trivial level, resulting in millions
or more paths to the root 0/1 nodes for larger <i>n</i>. This program

POKI_INCLUDE_ESCAPED(data/fibo-uncached.sh)HERE

<p/> produces output like this:

<p/>
<div class="pokipanel"><pre>
i  o      fcount  seconds_delta
1  1      1       0
2  2      3       0.000039101
3  3      5       0.000015974
4  5      9       0.000019073
5  8      15      0.000026941
6  13     25      0.000036955
7  21     41      0.000056028
8  34     67      0.000086069
9  55     109     0.000134945
10 89     177     0.000217915
11 144    287     0.000355959
12 233    465     0.000506163
13 377    753     0.000811815
14 610    1219    0.001297235
15 987    1973    0.001960993
16 1597   3193    0.003417969
17 2584   5167    0.006215811
18 4181   8361    0.008294106
19 6765   13529   0.012095928
20 10946  21891   0.019592047
21 17711  35421   0.031193972
22 28657  57313   0.057254076
23 46368  92735   0.080307961
24 75025  150049  0.129482031
25 121393 242785  0.213325977
26 196418 392835  0.334423065
27 317811 635621  0.605969906
28 514229 1028457 0.971235037
</pre></div>

<p/> Note that the time it takes to evaluate the function is blowing up exponentially as the input argument
increases. Using <tt>@</tt>-variables, which persist across records, we can cache and reuse the results
of previous computations:

POKI_INCLUDE_ESCAPED(data/fibo-cached.sh)HERE

<p/> with output like this:
<p/>
<div class="pokipanel"><pre>
i  o      fcount seconds_delta
1  1      1      0
2  2      3      0.000053883
3  3      3      0.000035048
4  5      3      0.000045061
5  8      3      0.000014067
6  13     3      0.000028849
7  21     3      0.000028133
8  34     3      0.000027895
9  55     3      0.000014067
10 89     3      0.000015020
11 144    3      0.000012875
12 233    3      0.000033140
13 377    3      0.000014067
14 610    3      0.000012875
15 987    3      0.000029087
16 1597   3      0.000013828
17 2584   3      0.000013113
18 4181   3      0.000012875
19 6765   3      0.000013113
20 10946  3      0.000012875
21 17711  3      0.000013113
22 28657  3      0.000013113
23 46368  3      0.000015974
24 75025  3      0.000012875
25 121393 3      0.000013113
26 196418 3      0.000012875
27 317811 3      0.000013113
28 514229 3      0.000012875
</pre></div>

</div>