From 8ef4cf1046f97bc2e7fb9914ea164a060105cca7 Mon Sep 17 00:00:00 2001 From: John Kerl Date: Fri, 17 Feb 2017 16:05:44 -0500 Subject: [PATCH] more section-visibility JS toggles --- c/todo.txt | 1 + doc/10-min.html | 23 ++++++ doc/content-for-10-min.html | 23 ++++++ doc/content-for-cookbook.html | 44 +++++++++++ doc/cookbook.html | 134 ++++++++++++++++++++++------------ 5 files changed, 180 insertions(+), 45 deletions(-) diff --git a/c/todo.txt b/c/todo.txt index 1b43398ba..40cd8613d 100644 --- a/c/todo.txt +++ b/c/todo.txt @@ -335,6 +335,7 @@ DOC: * 10-min * inverse-transform-sampling example at cookbook. * mlr -F and -K. Note -l -L -f -F -k -K. +* section toggles * go/d/rust cmps @ getline * ruby/python cmps @ mand diff --git a/doc/10-min.html b/doc/10-min.html index 881c85cad..2c87b0867 100644 --- a/doc/10-min.html +++ b/doc/10-min.html @@ -193,7 +193,13 @@ Miller commands were run with pretty-print-tabular output format.

+

+ + +

CSV-file examples

+ +

Sample CSV data file: @@ -524,7 +530,10 @@ square purple 1 72.373500 72.373500 72.373500

+

Choices for printing to files

+ +

Often we want to print output to the screen. Miller does this by default, as we’ve seen in the previous examples. @@ -652,7 +661,10 @@ purple,triangle,0,65,80.1405,5.8240

+

Other-format examples

+ +

What’s a CSV file, really? It’s an array of rows, or records, each being a list of key-value pairs, or fields: for CSV @@ -707,7 +719,10 @@ other format input data. And you can read from one format, do any record-processing, and output to the same format as the input, or to a different output format. +

SQL-output examples

+ +

I like to produce SQL-query output with header-column and tab delimiter: this is CSV but with a tab instead of a comma, also known as TSV. Then I @@ -820,7 +835,10 @@ standard 10009872 108

Again, all the examples in the CSV section apply here — just change the input-format flags. +

Log-processing examples

+ +

Another of my favorite use-cases for Miller is doing ad-hoc processing of log-file data. Here’s where DKVP format really shines: one, since the @@ -1038,11 +1056,16 @@ time batch_size num_filtered

+

More

+ +

Please see the reference for complete information, as well as the FAQ and the cookbook for more tips. + +

diff --git a/doc/content-for-10-min.html b/doc/content-for-10-min.html index bf9349058..1353bda51 100644 --- a/doc/content-for-10-min.html +++ b/doc/content-for-10-min.html @@ -1,6 +1,12 @@ POKI_PUT_TOC_HERE +

+ + +

CSV-file examples

+ +

Sample CSV data file: @@ -66,7 +72,10 @@ POKI_RUN_COMMAND{{mlr --icsv --oxtab --from example.csv stats1 -a p0,p10,p25,p50 POKI_RUN_COMMAND{{mlr --icsv --opprint --from example.csv stats1 -a count,min,mean,max -f quantity -g shape}}HERE POKI_RUN_COMMAND{{mlr --icsv --opprint --from example.csv stats1 -a count,min,mean,max -f quantity -g shape,color}}HERE +

Choices for printing to files

+ +

Often we want to print output to the screen. Miller does this by default, as we’ve seen in the previous examples. @@ -157,7 +166,10 @@ POKI_RUN_COMMAND{{cat square.csv}}HERE POKI_RUN_COMMAND{{cat triangle.csv}}HERE +

Other-format examples

+ +

What’s a CSV file, really? It’s an array of rows, or records, each being a list of key-value pairs, or fields: for CSV @@ -212,7 +224,10 @@ other format input data. And you can read from one format, do any record-processing, and output to the same format as the input, or to a different output format. +

SQL-output examples

+ +

I like to produce SQL-query output with header-column and tab delimiter: this is CSV but with a tab instead of a comma, also known as TSV. Then I @@ -325,7 +340,10 @@ standard 10009872 108

Again, all the examples in the CSV section apply here — just change the input-format flags. +

Log-processing examples

+ +

Another of my favorite use-cases for Miller is doing ad-hoc processing of log-file data. Here’s where DKVP format really shines: one, since the @@ -359,8 +377,13 @@ POKI_INCLUDE_AND_RUN_ESCAPED(10-2.sh)HERE POKI_RUN_COMMAND{{mlr --opprint group-like log.txt}}HERE POKI_RUN_COMMAND{{mlr --opprint group-like then sec2gmt time log.txt}}HERE +

More

+ +

Please see the reference for complete information, as well as the FAQ and the cookbook for more tips. + +

diff --git a/doc/content-for-cookbook.html b/doc/content-for-cookbook.html index a4f0f3972..e1938f77d 100644 --- a/doc/content-for-cookbook.html +++ b/doc/content-for-cookbook.html @@ -1,11 +1,17 @@ POKI_PUT_TOC_HERE +

+ + +

Disclaimer: This page is about how to do some corner-case things in ways you mightn’t have thought of. For an intro, please also see Miller in 10 minutes.

Parsing log-file output

+ +

This, of course, depends highly on what’s in your log files. But, as an example, suppose you have log-file lines such as @@ -17,7 +23,10 @@ I prefer to pre-filter with grep and/or sed to extract the str POKI_CARDIFY(grep 'various sorts' *.log | sed 's/.*} //' | mlr --fs space --repifs --oxtab stats1 -a min,p10,p50,p90,max -f time -g status)HERE +

Rectangularizing data

+ +

Suppose you have a method (in whatever language) which is printing things of the form @@ -48,7 +57,10 @@ output when you see the inner IDs. POKI_INCLUDE_AND_RUN_ESCAPED(data/rect.sh)HERE +

Bulk rename of field names

+ +
POKI_RUN_COMMAND{{cat data/spaces.csv}}HERE POKI_RUN_COMMAND{{mlr --csv rename -r -g ' ,_' data/spaces.csv}}HERE @@ -61,7 +73,10 @@ POKI_RUN_COMMAND{{cat data/bulk-rename-for-loop.mlr}}HERE POKI_RUN_COMMAND{{mlr --icsv --opprint put -f data/bulk-rename-for-loop.mlr data/spaces.csv}}HERE +

Headerless CSV on input or output

+ +

Sometimes we get CSV files which lack a header. For example: @@ -85,7 +100,10 @@ POKI_RUN_COMMAND{{head -5 data/colored-shapes.dkvp | mlr --ocsv cat}}HERE POKI_RUN_COMMAND{{head -5 data/colored-shapes.dkvp | mlr --ocsv --headerless-csv-output cat}}HERE +

Regularizing ragged CSV

+ +

Miller handles compliant CSV: in particular, it’s an error if the number of data fields in a given data line don’t match the number of @@ -102,7 +120,10 @@ or, more simply, POKI_INCLUDE_AND_RUN_ESCAPED(data/ragged-csv-2.sh)HERE +

Program timing

+ +
This admittedly artificial example demonstrates using Miller time and stats functions to introspectly acquire some information about Miller’s own @@ -112,7 +133,10 @@ timestamps. POKI_INCLUDE_ESCAPED(data/timing-example.txt)HERE +

Memoization with out-of-stream variables

+ +

The recursive function for the Fibonacci sequence is famous for its computational complexity. Namely, using @@ -131,7 +155,10 @@ of previous computations: POKI_INCLUDE_AND_RUN_ESCAPED(data/fibo-cached.sh)HERE +

Finding missing dates

+ +

Suppose you have some date-stamped data which may (or may not) be missing entries for one or more dates: @@ -153,7 +180,10 @@ POKI_RUN_COMMAND{{mlr cat -n then filter '$n >= 770 && $n <= 780' data/miss-date POKI_RUN_COMMAND{{mlr cat -n then filter '$n >= 1115 && $n <= 1125' data/miss-date.csv}}HERE +

Generating random variables

+ +

Here can chain together a few simple building blocks: @@ -177,7 +207,10 @@ The output is as follows: POKI_RUN_COMMAND{{sh expo-sample.sh}}HERE +

Two-pass algorithms

+ +

Miller is a streaming record processor; commands are performed once per record. This makes Miller particularly suitable for single-pass algorithms, @@ -220,7 +253,10 @@ POKI_RUN_COMMAND{{cat data/maxrows.mlr}}HERE POKI_RUN_COMMAND{{mlr --itsv --opprint put -q -f data/maxrows.mlr data/maxrows.tsv}}HERE +

Filtering paragraphs of text

+ +

The idea is to use a record separator which is a pair of newlines. Then, if you want each paragraph to be a record with a single value, use a @@ -235,12 +271,18 @@ POKI_RUN_COMMAND{{mlr --from paragraphs.txt --nidx --rs '\n\n' --fs '\001' filte POKI_RUN_COMMAND{{mlr --from paragraphs.txt --nidx --rs '\n\n' --fs '\n' cut -f 1,3}}HERE +

Doing arithmetic on fields with currency symbols

+ +
POKI_INCLUDE_ESCAPED(data/dollar-sign.txt)HERE +

Using out-of-stream variables

+ +

One of Miller’s strengths is its compact notation: for example, given input of the form @@ -322,3 +364,5 @@ POKI_RUN_COMMAND{{mlr --opprint put '$x_delta = is_present(@last[$a]) ? $x - @la POKI_INCLUDE_AND_RUN_ESCAPED(verb-example-ewma.sh)HERE POKI_INCLUDE_AND_RUN_ESCAPED(oosvar-example-ewma.sh)HERE + +

diff --git a/doc/cookbook.html b/doc/cookbook.html index c257865d5..87f3fa40f 100644 --- a/doc/cookbook.html +++ b/doc/cookbook.html @@ -211,12 +211,18 @@ Miller commands were run with pretty-print-tabular output format.

+

+ + +

Disclaimer: This page is about how to do some corner-case things in ways you mightn’t have thought of. For an intro, please also see Miller in 10 minutes.

Parsing log-file output

+ +

This, of course, depends highly on what’s in your log files. But, as an example, suppose you have log-file lines such as @@ -240,7 +246,10 @@ grep 'various sorts' *.log | sed 's/.*} //' | mlr --fs space --repifs --oxtab st

+

Rectangularizing data

+ +

Suppose you have a method (in whatever language) which is printing things of the form @@ -347,7 +356,10 @@ outer=3,middle=31,inner1=313,inner2=314

+

Bulk rename of field names

+ +

@@ -413,7 +425,10 @@ def a_b_c g_h_i

+

Headerless CSV on input or output

+ +

Sometimes we get CSV files which lack a header. For example: @@ -514,7 +529,10 @@ purple,triangle,0,51,0.4355354501763202,0.8591292672156728,0.8122903963006748,5.

+

Regularizing ragged CSV

+ +

Miller handles compliant CSV: in particular, it’s an error if the number of data fields in a given data line don’t match the number of @@ -576,7 +594,10 @@ a,b,c

+

Program timing

+ +
This admittedly artificial example demonstrates using Miller time and stats functions to introspectly acquire some information about Miller’s own @@ -618,7 +639,10 @@ t_delta_max 5.388259888e-05

+

Memoization with out-of-stream variables

+ +

The recursive function for the Fibonacci sequence is famous for its computational complexity. Namely, using @@ -649,32 +673,32 @@ $ mlr --ofmt '%.9lf' --opprint seqgen --start 1 --stop 28 then put ' i o fcount seconds_delta 1 1 1 0 2 2 3 0.000031948 -3 3 5 0.000012159 -4 5 9 0.000015020 -5 8 15 0.000020981 -6 13 25 0.000027895 -7 21 41 0.000044107 -8 34 67 0.000065804 -9 55 109 0.000105143 -10 89 177 0.000167847 -11 144 287 0.000271082 -12 233 465 0.000427961 -13 377 753 0.000688076 -14 610 1219 0.001102924 -15 987 1973 0.001820087 -16 1597 3193 0.002871990 -17 2584 5167 0.004658937 -18 4181 8361 0.007560968 -19 6765 13529 0.012096167 -20 10946 21891 0.019388914 -21 17711 35421 0.031304121 -22 28657 57313 0.050275803 -23 46368 92735 0.080559015 -24 75025 150049 0.136618137 -25 121393 242785 0.208474874 -26 196418 392835 0.335786104 -27 317811 635621 0.553169966 -28 514229 1028457 0.884209871 +3 3 5 0.000013113 +4 5 9 0.000014782 +5 8 15 0.000022173 +6 13 25 0.000028849 +7 21 41 0.000045061 +8 34 67 0.000067949 +9 55 109 0.000108004 +10 89 177 0.000173092 +11 144 287 0.000277996 +12 233 465 0.000439882 +13 377 753 0.000705004 +14 610 1219 0.001131058 +15 987 1973 0.001841068 +16 1597 3193 0.003005981 +17 2584 5167 0.004643917 +18 4181 8361 0.007503033 +19 6765 13529 0.012021065 +20 10946 21891 0.019377947 +21 17711 35421 0.031213999 +22 28657 57313 0.050358057 +23 46368 92735 0.080504894 +24 75025 150049 0.131229162 +25 121393 242785 0.218618870 +26 196418 392835 0.372801065 +27 317811 635621 0.605077982 +28 514229 1028457 0.981553078

@@ -706,39 +730,42 @@ $ mlr --ofmt '%.9lf' --opprint seqgen --start 1 --stop 28 then put ' ' then put '$seconds=systime()' then step -a delta -f seconds then cut -x -f seconds i o fcount seconds_delta 1 1 1 0 -2 2 3 0.000035048 +2 2 3 0.000036001 3 3 3 0.000013113 -4 5 3 0.000010967 +4 5 3 0.000011921 5 8 3 0.000010967 6 13 3 0.000010014 -7 21 3 0.000010014 -8 34 3 0.000010014 -9 55 3 0.000011921 -10 89 3 0.000010014 -11 144 3 0.000010014 -12 233 3 0.000014067 -13 377 3 0.000010014 +7 21 3 0.000010967 +8 34 3 0.000010967 +9 55 3 0.000012159 +10 89 3 0.000009775 +11 144 3 0.000011206 +12 233 3 0.000013828 +13 377 3 0.000010967 14 610 3 0.000010014 -15 987 3 0.000010014 -16 1597 3 0.000009775 -17 2584 3 0.000011206 -18 4181 3 0.000008821 -19 6765 3 0.000010014 -20 10946 3 0.000010014 +15 987 3 0.000011206 +16 1597 3 0.000010014 +17 2584 3 0.000009775 +18 4181 3 0.000010014 +19 6765 3 0.000011206 +20 10946 3 0.000009775 21 17711 3 0.000010014 22 28657 3 0.000010014 -23 46368 3 0.000011921 +23 46368 3 0.000013113 24 75025 3 0.000010014 -25 121393 3 0.000011206 -26 196418 3 0.000009775 -27 317811 3 0.000010014 +25 121393 3 0.000010967 +26 196418 3 0.000010014 +27 317811 3 0.000010967 28 514229 3 0.000010014

+

Finding missing dates

+ +

Suppose you have some date-stamped data which may (or may not) be missing entries for one or more dates: @@ -849,7 +876,10 @@ n=1125,1=2015-04-05,2=182528

+

Generating random variables

+ +

Here can chain together a few simple building blocks: @@ -963,7 +993,10 @@ bin_lo bin_hi u_count s_count

+

Two-pass algorithms

+ +

Miller is a streaming record processor; commands are performed once per record. This makes Miller particularly suitable for single-pass algorithms, @@ -1131,7 +1164,10 @@ green purple 5 0.203577

+

Filtering paragraphs of text

+ +

The idea is to use a record separator which is a pair of newlines. Then, if you want each paragraph to be a record with a single value, use a @@ -1211,7 +1247,10 @@ falls mainly on the plain. The rain in Spain falls mainly on the plain. The

+

Doing arithmetic on fields with currency symbols

+ +

@@ -1253,7 +1292,10 @@ Amount_sum

+

Using out-of-stream variables

+ +

One of Miller’s strengths is its compact notation: for example, given input of the form @@ -1650,6 +1692,8 @@ wye pan 5 0.5732889198020006 0.8636244699032729 0.391064

+ +