Often we want to print output to the screen. Miller does this by default, as we’ve
seen in the previous examples.
@@ -652,7 +661,10 @@ purple,triangle,0,65,80.1405,5.8240
+
Other-format examples
+
+
What’s a CSV file, really? It’s an array of rows, or
records, each being a list of key-value pairs, or fields: for CSV
@@ -707,7 +719,10 @@ other format input data. And you can read from one format, do any
record-processing, and output to the same format as the input, or to a
different output format.
+
SQL-output examples
+
+
I like to produce SQL-query output with header-column and tab delimiter:
this is CSV but with a tab instead of a comma, also known as TSV. Then I
@@ -820,7 +835,10 @@ standard 10009872 108
Again, all the examples in the CSV section apply here — just change the input-format
flags.
+
Log-processing examples
+
+
Another of my favorite use-cases for Miller is doing ad-hoc processing of
log-file data. Here’s where DKVP format really shines: one, since the
@@ -1038,11 +1056,16 @@ time batch_size num_filtered
+
More
+
+
Please see the reference for complete
information, as well as the FAQ and the cookbook for more tips.
+
+
Often we want to print output to the screen. Miller does this by default, as we’ve
seen in the previous examples.
@@ -157,7 +166,10 @@ POKI_RUN_COMMAND{{cat square.csv}}HERE
POKI_RUN_COMMAND{{cat triangle.csv}}HERE
+
Other-format examples
+
+
What’s a CSV file, really? It’s an array of rows, or
records, each being a list of key-value pairs, or fields: for CSV
@@ -212,7 +224,10 @@ other format input data. And you can read from one format, do any
record-processing, and output to the same format as the input, or to a
different output format.
+
SQL-output examples
+
+
I like to produce SQL-query output with header-column and tab delimiter:
this is CSV but with a tab instead of a comma, also known as TSV. Then I
@@ -325,7 +340,10 @@ standard 10009872 108
Again, all the examples in the CSV section apply here — just change the input-format
flags.
+
Log-processing examples
+
+
Another of my favorite use-cases for Miller is doing ad-hoc processing of
log-file data. Here’s where DKVP format really shines: one, since the
@@ -359,8 +377,13 @@ POKI_INCLUDE_AND_RUN_ESCAPED(10-2.sh)HERE
POKI_RUN_COMMAND{{mlr --opprint group-like log.txt}}HERE
POKI_RUN_COMMAND{{mlr --opprint group-like then sec2gmt time log.txt}}HERE
+
More
+
+
Please see the reference for complete
information, as well as the FAQ and the cookbook for more tips.
+
+
diff --git a/doc/content-for-cookbook.html b/doc/content-for-cookbook.html
index a4f0f3972..e1938f77d 100644
--- a/doc/content-for-cookbook.html
+++ b/doc/content-for-cookbook.html
@@ -1,11 +1,17 @@
POKI_PUT_TOC_HERE
+
+
+
+
Disclaimer: This page is about how to do some corner-case things in
ways you mightn’t have thought of. For an intro, please also see
Miller in 10 minutes.
Parsing log-file output
+
+
This, of course, depends highly on what’s in your log files. But, as
an example, suppose you have log-file lines such as
@@ -17,7 +23,10 @@ I prefer to pre-filter with grep and/or sed to extract the str
POKI_CARDIFY(grep 'various sorts' *.log | sed 's/.*} //' | mlr --fs space --repifs --oxtab stats1 -a min,p10,p50,p90,max -f time -g status)HERE
+
Rectangularizing data
+
+
Suppose you have a method (in whatever language) which is printing things of the form
@@ -48,7 +57,10 @@ output when you see the inner IDs.
POKI_INCLUDE_AND_RUN_ESCAPED(data/rect.sh)HERE
+
Sometimes we get CSV files which lack a header. For example:
@@ -85,7 +100,10 @@ POKI_RUN_COMMAND{{head -5 data/colored-shapes.dkvp | mlr --ocsv cat}}HERE
POKI_RUN_COMMAND{{head -5 data/colored-shapes.dkvp | mlr --ocsv --headerless-csv-output cat}}HERE
+
Regularizing ragged CSV
+
+
Miller handles compliant CSV: in particular, it’s an error if the
number of data fields in a given data line don’t match the number of
@@ -102,7 +120,10 @@ or, more simply,
POKI_INCLUDE_AND_RUN_ESCAPED(data/ragged-csv-2.sh)HERE
+
Program timing
+
+
This admittedly artificial example demonstrates using Miller time and stats
functions to introspectly acquire some information about Miller’s own
@@ -112,7 +133,10 @@ timestamps.
POKI_INCLUDE_ESCAPED(data/timing-example.txt)HERE
+
Memoization with out-of-stream variables
+
+
The recursive function for the Fibonacci sequence is famous for its computational complexity.
Namely, using
@@ -131,7 +155,10 @@ of previous computations:
POKI_INCLUDE_AND_RUN_ESCAPED(data/fibo-cached.sh)HERE
+
Finding missing dates
+
+
Suppose you have some date-stamped data which may (or may not) be missing entries for one or more dates:
@@ -153,7 +180,10 @@ POKI_RUN_COMMAND{{mlr cat -n then filter '$n >= 770 && $n <= 780' data/miss-date
POKI_RUN_COMMAND{{mlr cat -n then filter '$n >= 1115 && $n <= 1125' data/miss-date.csv}}HERE
+
Generating random variables
+
+
Here can chain together a few simple building blocks:
@@ -177,7 +207,10 @@ The output is as follows:
POKI_RUN_COMMAND{{sh expo-sample.sh}}HERE
+
Two-pass algorithms
+
+
Miller is a streaming record processor; commands are performed once per
record. This makes Miller particularly suitable for single-pass algorithms,
@@ -220,7 +253,10 @@ POKI_RUN_COMMAND{{cat data/maxrows.mlr}}HERE
POKI_RUN_COMMAND{{mlr --itsv --opprint put -q -f data/maxrows.mlr data/maxrows.tsv}}HERE
+
Filtering paragraphs of text
+
+
The idea is to use a record separator which is a pair of newlines. Then, if
you want each paragraph to be a record with a single value, use a
@@ -235,12 +271,18 @@ POKI_RUN_COMMAND{{mlr --from paragraphs.txt --nidx --rs '\n\n' --fs '\001' filte
POKI_RUN_COMMAND{{mlr --from paragraphs.txt --nidx --rs '\n\n' --fs '\n' cut -f 1,3}}HERE
+
Doing arithmetic on fields with currency symbols
+
+
POKI_INCLUDE_ESCAPED(data/dollar-sign.txt)HERE
+
Using out-of-stream variables
+
+
One of Miller’s strengths is its compact notation: for example, given input of the form
@@ -322,3 +364,5 @@ POKI_RUN_COMMAND{{mlr --opprint put '$x_delta = is_present(@last[$a]) ? $x - @la
POKI_INCLUDE_AND_RUN_ESCAPED(verb-example-ewma.sh)HERE
POKI_INCLUDE_AND_RUN_ESCAPED(oosvar-example-ewma.sh)HERE
+
+
diff --git a/doc/cookbook.html b/doc/cookbook.html
index c257865d5..87f3fa40f 100644
--- a/doc/cookbook.html
+++ b/doc/cookbook.html
@@ -211,12 +211,18 @@ Miller commands were run with pretty-print-tabular output format.
+
+
+
+
Disclaimer: This page is about how to do some corner-case things in
ways you mightn’t have thought of. For an intro, please also see
Miller in 10 minutes.
Parsing log-file output
+
+
This, of course, depends highly on what’s in your log files. But, as
an example, suppose you have log-file lines such as
@@ -240,7 +246,10 @@ grep 'various sorts' *.log | sed 's/.*} //' | mlr --fs space --repifs --oxtab st
+
Rectangularizing data
+
+
Suppose you have a method (in whatever language) which is printing things of the form
@@ -347,7 +356,10 @@ outer=3,middle=31,inner1=313,inner2=314
+
Bulk rename of field names
+
+
@@ -413,7 +425,10 @@ def a_b_c g_h_i
+
Headerless CSV on input or output
+
+
Sometimes we get CSV files which lack a header. For example:
@@ -514,7 +529,10 @@ purple,triangle,0,51,0.4355354501763202,0.8591292672156728,0.8122903963006748,5.
+
Regularizing ragged CSV
+
+
Miller handles compliant CSV: in particular, it’s an error if the
number of data fields in a given data line don’t match the number of
@@ -576,7 +594,10 @@ a,b,c
+
Program timing
+
+
This admittedly artificial example demonstrates using Miller time and stats
functions to introspectly acquire some information about Miller’s own
@@ -618,7 +639,10 @@ t_delta_max 5.388259888e-05
+
Suppose you have some date-stamped data which may (or may not) be missing entries for one or more dates:
@@ -849,7 +876,10 @@ n=1125,1=2015-04-05,2=182528
+
Generating random variables
+
+
Here can chain together a few simple building blocks:
@@ -963,7 +993,10 @@ bin_lo bin_hi u_count s_count
+
Two-pass algorithms
+
+
Miller is a streaming record processor; commands are performed once per
record. This makes Miller particularly suitable for single-pass algorithms,
@@ -1131,7 +1164,10 @@ green purple 5 0.203577
+
Filtering paragraphs of text
+
+
The idea is to use a record separator which is a pair of newlines. Then, if
you want each paragraph to be a record with a single value, use a
@@ -1211,7 +1247,10 @@ falls mainly on the plain. The rain in Spain falls mainly on the plain. The
+
Doing arithmetic on fields with currency symbols
+
+
@@ -1253,7 +1292,10 @@ Amount_sum
+
Using out-of-stream variables
+
+
One of Miller’s strengths is its compact notation: for example, given input of the form
@@ -1650,6 +1692,8 @@ wye pan 5 0.5732889198020006 0.8636244699032729 0.391064