diff --git a/docs6/10min.rst b/docs6/10min.rst index 887f06f1e..2e931fa61 100644 --- a/docs6/10min.rst +++ b/docs6/10min.rst @@ -524,6 +524,8 @@ How to specify these to Miller: You can read more about this at the :doc:`file-formats` page. +.. _10min-choices-for-printing-to-files: + Choices for printing to files ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/docs6/10min.rst.in b/docs6/10min.rst.in index b909cf7ca..9f05256da 100644 --- a/docs6/10min.rst.in +++ b/docs6/10min.rst.in @@ -266,6 +266,8 @@ How to specify these to Miller: You can read more about this at the :doc:`file-formats` page. +.. _10min-choices-for-printing-to-files: + Choices for printing to files ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/docs6/cookbook.rst.in b/docs6/cookbook.rst.in deleted file mode 100644 index cb7baaa89..000000000 --- a/docs6/cookbook.rst.in +++ /dev/null @@ -1,264 +0,0 @@ -Cookbook part 1: common patterns -================================================================ - -Data-cleaning examples ----------------------------------------------------------------- - -Here are some ways to use the type-checking options as described in :ref:`reference-dsl-type-tests-and-assertions` Suppose you have the following data file, with inconsistent typing for boolean. (Also imagine that, for the sake of discussion, we have a million-line file rather than a four-line file, so we can't see it all at once and some automation is called for.) - -GENRST_RUN_COMMAND -cat data/het-bool.csv -GENRST_EOF - -One option is to coerce everything to boolean, or integer: - -GENRST_RUN_COMMAND -mlr --icsv --opprint put '$reachable = boolean($reachable)' data/het-bool.csv -GENRST_EOF - -GENRST_RUN_COMMAND -mlr --icsv --opprint put '$reachable = int(boolean($reachable))' data/het-bool.csv -GENRST_EOF - -A second option is to flag badly formatted data within the output stream: - -GENRST_RUN_COMMAND -mlr --icsv --opprint put '$format_ok = is_string($reachable)' data/het-bool.csv -GENRST_EOF - -Or perhaps to flag badly formatted data outside the output stream: - -GENRST_RUN_COMMAND -mlr --icsv --opprint put ' - if (!is_string($reachable)) {eprint "Malformed at NR=".NR} -' data/het-bool.csv -GENRST_EOF - -A third way is to abort the process on first instance of bad data: - -GENRST_RUN_COMMAND_TOLERATING_ERROR -mlr --csv put '$reachable = asserting_string($reachable)' data/het-bool.csv -GENRST_EOF - -Showing differences between successive queries ----------------------------------------------------------------- - -Suppose you have a database query which you run at one point in time, producing the output on the left, then again later producing the output on the right: - -GENRST_RUN_COMMAND -cat data/previous_counters.csv -GENRST_EOF - -GENRST_RUN_COMMAND -cat data/current_counters.csv -GENRST_EOF - -And, suppose you want to compute the differences in the counters between adjacent keys. Since the color names aren't all in the same order, nor are they all present on both sides, we can't just paste the two files side-by-side and do some column-four-minus-column-two arithmetic. - -First, rename counter columns to make them distinct: - -GENRST_RUN_COMMAND -mlr --csv rename count,previous_count data/previous_counters.csv > data/prevtemp.csv -GENRST_EOF - -GENRST_RUN_COMMAND -cat data/prevtemp.csv -GENRST_EOF - -GENRST_RUN_COMMAND -mlr --csv rename count,current_count data/current_counters.csv > data/currtemp.csv -GENRST_EOF - -GENRST_RUN_COMMAND -cat data/currtemp.csv -GENRST_EOF - -Then, join on the key field(s), and use unsparsify to zero-fill counters absent on one side but present on the other. Use ``--ul`` and ``--ur`` to emit unpaired records (namely, purple on the left and yellow on the right): - -GENRST_INCLUDE_AND_RUN_ESCAPED(data/previous-to-current.sh) - -Two-pass algorithms ----------------------------------------------------------------- - -Miller is a streaming record processor; commands are performed once per record. This makes Miller particularly suitable for single-pass algorithms, allowing many of its verbs to process files that are (much) larger than the amount of RAM present in your system. (Of course, Miller verbs such as ``sort``, ``tac``, etc. all must ingest and retain all input records before emitting any output records.) You can also use out-of-stream variables to perform multi-pass computations, at the price of retaining all input records in memory. - -Two-pass algorithms: computation of percentages -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -For example, mapping numeric values down a column to the percentage between their min and max values is two-pass: on the first pass you find the min and max values, then on the second, map each record's value to a percentage. - -GENRST_INCLUDE_AND_RUN_ESCAPED(data/two-pass-percentage.sh) - -Two-pass algorithms: line-number ratios -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Similarly, finding the total record count requires first reading through all the data: - -GENRST_INCLUDE_AND_RUN_ESCAPED(data/two-pass-record-numbers.sh) - -Two-pass algorithms: records having max value -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The idea is to retain records having the largest value of ``n`` in the following data: - -GENRST_RUN_COMMAND -mlr --itsv --opprint cat data/maxrows.tsv -GENRST_EOF - -Of course, the largest value of ``n`` isn't known until after all data have been read. Using an out-of-stream variable we can retain all records as they are read, then filter them at the end: - -GENRST_RUN_COMMAND -cat data/maxrows.mlr -GENRST_EOF - -GENRST_RUN_COMMAND -mlr --itsv --opprint put -q -f data/maxrows.mlr data/maxrows.tsv -GENRST_EOF - -Feature-counting ----------------------------------------------------------------- - -Suppose you have some heterogeneous data like this: - -GENRST_INCLUDE_ESCAPED(data/features.json) - -A reasonable question to ask is, how many occurrences of each field are there? And, what percentage of total row count has each of them? Since the denominator of the percentage is not known until the end, this is a two-pass algorithm: - -GENRST_INCLUDE_ESCAPED(data/feature-count.mlr) - -Then - -GENRST_RUN_COMMAND -mlr --json put -q -f data/feature-count.mlr data/features.json -GENRST_EOF - -GENRST_RUN_COMMAND -mlr --ijson --opprint put -q -f data/feature-count.mlr data/features.json -GENRST_EOF - -Unsparsing ----------------------------------------------------------------- - -The previous section discussed how to fill out missing data fields within CSV with full header line -- so the list of all field names is present within the header line. Next, let's look at a related problem: we have data where each record has various key names but we want to produce rectangular output having the union of all key names. - -For example, suppose you have JSON input like this: - -GENRST_RUN_COMMAND -cat data/sparse.json -GENRST_EOF - -There are field names ``a``, ``b``, ``v``, ``u``, ``x``, ``w`` in the data -- but not all in every record. Since we don't know the names of all the keys until we've read them all, this needs to be a two-pass algorithm. On the first pass, remember all the unique key names and all the records; on the second pass, loop through the records filling in absent values, then producing output. Use ``put -q`` since we don't want to produce per-record output, only emitting output in the ``end`` block: - -GENRST_RUN_COMMAND -cat data/unsparsify.mlr -GENRST_EOF - -GENRST_RUN_COMMAND -mlr --json put -q -f data/unsparsify.mlr data/sparse.json -GENRST_EOF - -GENRST_RUN_COMMAND -mlr --ijson --ocsv put -q -f data/unsparsify.mlr data/sparse.json -GENRST_EOF - -GENRST_RUN_COMMAND -mlr --ijson --opprint put -q -f data/unsparsify.mlr data/sparse.json -GENRST_EOF - -There is a keystroke-saving verb for this: :ref:`mlr unsparsify `. - -Parsing log-file output ----------------------------------------------------------------- - -This, of course, depends highly on what's in your log files. But, as an example, suppose you have log-file lines such as - -GENRST_CARDIFY -2015-10-08 08:29:09,445 INFO com.company.path.to.ClassName @ [sometext] various/sorts/of data {& punctuation} hits=1 status=0 time=2.378 -GENRST_EOF - -I prefer to pre-filter with ``grep`` and/or ``sed`` to extract the structured text, then hand that to Miller. Example: - -GENRST_SHOW_COMMAND -grep 'various sorts' *.log | sed 's/.*} //' | mlr --fs space --repifs --oxtab stats1 -a min,p10,p50,p90,max -f time -g status -GENRST_EOF - -.. _cookbook-memoization-with-oosvars: - -Memoization with out-of-stream variables ----------------------------------------------------------------- - -The recursive function for the Fibonacci sequence is famous for its computational complexity. Namely, using f(0)=1, f(1)=1, f(n)=f(n-1)+f(n-2) for n>=2, the evaluation tree branches left as well as right at each non-trivial level, resulting in millions or more paths to the root 0/1 nodes for larger n. This program - -GENRST_INCLUDE_ESCAPED(data/fibo-uncached.sh) - -produces output like this: - -GENRST_CARDIFY -i o fcount seconds_delta -1 1 1 0 -2 2 3 0.000039101 -3 3 5 0.000015974 -4 5 9 0.000019073 -5 8 15 0.000026941 -6 13 25 0.000036955 -7 21 41 0.000056028 -8 34 67 0.000086069 -9 55 109 0.000134945 -10 89 177 0.000217915 -11 144 287 0.000355959 -12 233 465 0.000506163 -13 377 753 0.000811815 -14 610 1219 0.001297235 -15 987 1973 0.001960993 -16 1597 3193 0.003417969 -17 2584 5167 0.006215811 -18 4181 8361 0.008294106 -19 6765 13529 0.012095928 -20 10946 21891 0.019592047 -21 17711 35421 0.031193972 -22 28657 57313 0.057254076 -23 46368 92735 0.080307961 -24 75025 150049 0.129482031 -25 121393 242785 0.213325977 -26 196418 392835 0.334423065 -27 317811 635621 0.605969906 -28 514229 1028457 0.971235037 -GENRST_EOF - -Note that the time it takes to evaluate the function is blowing up exponentially as the input argument increases. Using ``@``-variables, which persist across records, we can cache and reuse the results of previous computations: - -GENRST_INCLUDE_ESCAPED(data/fibo-cached.sh) - -with output like this: - -GENRST_CARDIFY -i o fcount seconds_delta -1 1 1 0 -2 2 3 0.000053883 -3 3 3 0.000035048 -4 5 3 0.000045061 -5 8 3 0.000014067 -6 13 3 0.000028849 -7 21 3 0.000028133 -8 34 3 0.000027895 -9 55 3 0.000014067 -10 89 3 0.000015020 -11 144 3 0.000012875 -12 233 3 0.000033140 -13 377 3 0.000014067 -14 610 3 0.000012875 -15 987 3 0.000029087 -16 1597 3 0.000013828 -17 2584 3 0.000013113 -18 4181 3 0.000012875 -19 6765 3 0.000013113 -20 10946 3 0.000012875 -21 17711 3 0.000013113 -22 28657 3 0.000013113 -23 46368 3 0.000015974 -24 75025 3 0.000012875 -25 121393 3 0.000013113 -26 196418 3 0.000012875 -27 317811 3 0.000013113 -28 514229 3 0.000012875 -GENRST_EOF diff --git a/docs6/cookbook2.rst b/docs6/cookbook2.rst deleted file mode 100644 index 560001aa6..000000000 --- a/docs6/cookbook2.rst +++ /dev/null @@ -1,529 +0,0 @@ -.. - PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE. - -Cookbook part 2: Random things, and some math -================================================================ - -Randomly selecting words from a list ----------------------------------------------------------------- - -Given this `word list <./data/english-words.txt>`_, first take a look to see what the first few lines look like: - -.. code-block:: none - :emphasize-lines: 1-1 - - head data/english-words.txt - a - aa - aal - aalii - aam - aardvark - aardwolf - aba - abac - abaca - -Then the following will randomly sample ten words with four to eight characters in them: - -.. code-block:: none - :emphasize-lines: 1-1 - - mlr --from data/english-words.txt --nidx filter -S 'n=strlen($1);4<=n&&n<=8' then sample -k 10 - thionine - birchman - mildewy - avigate - addedly - abaze - askant - aiming - insulant - coinmate - -Randomly generating jabberwocky words ----------------------------------------------------------------- - -These are simple *n*-grams as `described here `_. Some common functions are `located here `_. Then here are scripts for `1-grams `_ `2-grams `_ `3-grams `_ `4-grams `_, and `5-grams `_. - -The idea is that words from the input file are consumed, then taken apart and pasted back together in ways which imitate the letter-to-letter transitions found in the word list -- giving us automatically generated words in the same vein as *bromance* and *spork*: - -.. code-block:: none - :emphasize-lines: 1-1 - - mlr --nidx --from ./ngrams/gsl-2000.txt put -q -f ./ngrams/ngfuncs.mlr -f ./ngrams/ng5.mlr - beard - plastinguish - politicially - noise - loan - country - controductionary - suppery - lose - lessors - dollar - judge - rottendence - lessenger - diffendant - suggestional - -Program timing ----------------------------------------------------------------- - -This admittedly artificial example demonstrates using Miller time and stats functions to introspectively acquire some information about Miller's own runtime. The ``delta`` function computes the difference between successive timestamps. - -.. code-block:: none - - $ ruby -e '10000.times{|i|puts "i=#{i+1}"}' > lines.txt - - $ head -n 5 lines.txt - i=1 - i=2 - i=3 - i=4 - i=5 - - mlr --ofmt '%.9le' --opprint put '$t=systime()' then step -a delta -f t lines.txt | head -n 7 - i t t_delta - 1 1430603027.018016 1.430603027e+09 - 2 1430603027.018043 2.694129944e-05 - 3 1430603027.018048 5.006790161e-06 - 4 1430603027.018052 4.053115845e-06 - 5 1430603027.018055 2.861022949e-06 - 6 1430603027.018058 3.099441528e-06 - - mlr --ofmt '%.9le' --oxtab \ - put '$t=systime()' then \ - step -a delta -f t then \ - filter '$i>1' then \ - stats1 -a min,mean,max -f t_delta \ - lines.txt - t_delta_min 2.861022949e-06 - t_delta_mean 4.077508505e-06 - t_delta_max 5.388259888e-05 - -Computing interquartile ranges ----------------------------------------------------------------- - -For one or more specified field names, simply compute p25 and p75, then write the IQR as the difference of p75 and p25: - -.. code-block:: none - :emphasize-lines: 1-3 - - mlr --oxtab stats1 -f x -a p25,p75 \ - then put '$x_iqr = $x_p75 - $x_p25' \ - data/medium - x_p25 0.24667037823231752 - x_p75 0.7481860062358446 - x_iqr 0.5015156280035271 - -For wildcarded field names, first compute p25 and p75, then loop over field names with ``p25`` in them: - -.. code-block:: none - :emphasize-lines: 1-7 - - mlr --oxtab stats1 --fr '[i-z]' -a p25,p75 \ - then put 'for (k,v in $*) { - if (k =~ "(.*)_p25") { - $["\1_iqr"] = $["\1_p75"] - $["\1_p25"] - } - }' \ - data/medium - -Computing weighted means ----------------------------------------------------------------- - -This might be more elegantly implemented as an option within the ``stats1`` verb. Meanwhile, it's expressible within the DSL: - -.. code-block:: none - :emphasize-lines: 1-24 - - mlr --from data/medium put -q ' - # Using the y field for weighting in this example - weight = $y; - - # Using the a field for weighted aggregation in this example - @sumwx[$a] += weight * $i; - @sumw[$a] += weight; - - @sumx[$a] += $i; - @sumn[$a] += 1; - - end { - map wmean = {}; - map mean = {}; - for (a in @sumwx) { - wmean[a] = @sumwx[a] / @sumw[a] - } - for (a in @sumx) { - mean[a] = @sumx[a] / @sumn[a] - } - #emit wmean, "a"; - #emit mean, "a"; - emit (wmean, mean), "a"; - }' - a=pan,wmean=4979.563722208067,mean=5028.259010091302 - a=eks,wmean=4890.3815931472145,mean=4956.2900763358775 - a=wye,wmean=4946.987746229947,mean=4920.001017293998 - a=zee,wmean=5164.719684856538,mean=5123.092330239375 - a=hat,wmean=4925.533162478552,mean=4967.743946419371 - -Generating random numbers from various distributions ----------------------------------------------------------------- - -Here we can chain together a few simple building blocks: - -.. code-block:: none - :emphasize-lines: 1-1 - - cat expo-sample.sh - # Generate 100,000 pairs of independent and identically distributed - # exponentially distributed random variables with the same rate parameter - # (namely, 2.5). Then compute histograms of one of them, along with - # histograms for their sum and their product. - # - # See also https://en.wikipedia.org/wiki/Exponential_distribution - # - # Here I'm using a specified random-number seed so this example always - # produces the same output for this web document: in everyday practice we - # wouldn't do that. - - mlr -n \ - --seed 0 \ - --opprint \ - seqgen --stop 100000 \ - then put ' - # https://en.wikipedia.org/wiki/Inverse_transform_sampling - func expo_sample(lambda) { - return -log(1-urand())/lambda - } - $u = expo_sample(2.5); - $v = expo_sample(2.5); - $s = $u + $v; - $p = $u * $v; - ' \ - then histogram -f u,s,p --lo 0 --hi 2 --nbins 50 \ - then bar -f u_count,s_count,p_count --auto -w 20 - -Namely: - -* Set the Miller random-number seed so this webdoc looks the same every time I regenerate it. -* Use pretty-printed tabular output. -* Use pretty-printed tabular output. -* Use ``seqgen`` to produce 100,000 records ``i=0``, ``i=1``, etc. -* Send those to a ``put`` step which defines an inverse-transform-sampling function and calls it twice, then computes the sum and product of samples. -* Send those to a histogram, and from there to a bar-plotter. This is just for visualization; you could just as well output CSV and send that off to your own plotting tool, etc. - -The output is as follows: - -.. code-block:: none - :emphasize-lines: 1-1 - - sh expo-sample.sh - bin_lo bin_hi u_count s_count p_count - 0 0.04 [64]*******************#[9554] [326]#...................[3703] [19]*******************#[39809] - 0.04 0.08 [64]*****************...[9554] [326]*****...............[3703] [19]*******.............[39809] - 0.08 0.12 [64]****************....[9554] [326]*********...........[3703] [19]****................[39809] - 0.12 0.16 [64]**************......[9554] [326]************........[3703] [19]***.................[39809] - 0.16 0.2 [64]*************.......[9554] [326]**************......[3703] [19]**..................[39809] - 0.2 0.24 [64]************........[9554] [326]*****************...[3703] [19]*...................[39809] - 0.24 0.28 [64]**********..........[9554] [326]******************..[3703] [19]*...................[39809] - 0.28 0.32 [64]*********...........[9554] [326]******************..[3703] [19]*...................[39809] - 0.32 0.36 [64]********............[9554] [326]*******************.[3703] [19]#...................[39809] - 0.36 0.4 [64]*******.............[9554] [326]*******************#[3703] [19]#...................[39809] - 0.4 0.44 [64]*******.............[9554] [326]*******************.[3703] [19]#...................[39809] - 0.44 0.48 [64]******..............[9554] [326]*******************.[3703] [19]#...................[39809] - 0.48 0.52 [64]*****...............[9554] [326]******************..[3703] [19]#...................[39809] - 0.52 0.56 [64]*****...............[9554] [326]******************..[3703] [19]#...................[39809] - 0.56 0.6 [64]****................[9554] [326]*****************...[3703] [19]#...................[39809] - 0.6 0.64 [64]****................[9554] [326]******************..[3703] [19]#...................[39809] - 0.64 0.68 [64]***.................[9554] [326]****************....[3703] [19]#...................[39809] - 0.68 0.72 [64]***.................[9554] [326]****************....[3703] [19]#...................[39809] - 0.72 0.76 [64]***.................[9554] [326]***************.....[3703] [19]#...................[39809] - 0.76 0.8 [64]**..................[9554] [326]**************......[3703] [19]#...................[39809] - 0.8 0.84 [64]**..................[9554] [326]*************.......[3703] [19]#...................[39809] - 0.84 0.88 [64]**..................[9554] [326]************........[3703] [19]#...................[39809] - 0.88 0.92 [64]**..................[9554] [326]************........[3703] [19]#...................[39809] - 0.92 0.96 [64]*...................[9554] [326]***********.........[3703] [19]#...................[39809] - 0.96 1 [64]*...................[9554] [326]**********..........[3703] [19]#...................[39809] - 1 1.04 [64]*...................[9554] [326]*********...........[3703] [19]#...................[39809] - 1.04 1.08 [64]*...................[9554] [326]********............[3703] [19]#...................[39809] - 1.08 1.12 [64]*...................[9554] [326]********............[3703] [19]#...................[39809] - 1.12 1.16 [64]*...................[9554] [326]********............[3703] [19]#...................[39809] - 1.16 1.2 [64]*...................[9554] [326]*******.............[3703] [19]#...................[39809] - 1.2 1.24 [64]#...................[9554] [326]******..............[3703] [19]#...................[39809] - 1.24 1.28 [64]#...................[9554] [326]*****...............[3703] [19]#...................[39809] - 1.28 1.32 [64]#...................[9554] [326]*****...............[3703] [19]#...................[39809] - 1.32 1.36 [64]#...................[9554] [326]****................[3703] [19]#...................[39809] - 1.36 1.4 [64]#...................[9554] [326]****................[3703] [19]#...................[39809] - 1.4 1.44 [64]#...................[9554] [326]****................[3703] [19]#...................[39809] - 1.44 1.48 [64]#...................[9554] [326]***.................[3703] [19]#...................[39809] - 1.48 1.52 [64]#...................[9554] [326]***.................[3703] [19]#...................[39809] - 1.52 1.56 [64]#...................[9554] [326]***.................[3703] [19]#...................[39809] - 1.56 1.6 [64]#...................[9554] [326]**..................[3703] [19]#...................[39809] - 1.6 1.64 [64]#...................[9554] [326]**..................[3703] [19]#...................[39809] - 1.64 1.68 [64]#...................[9554] [326]**..................[3703] [19]#...................[39809] - 1.68 1.72 [64]#...................[9554] [326]*...................[3703] [19]#...................[39809] - 1.72 1.76 [64]#...................[9554] [326]*...................[3703] [19]#...................[39809] - 1.76 1.8 [64]#...................[9554] [326]*...................[3703] [19]#...................[39809] - 1.8 1.84 [64]#...................[9554] [326]#...................[3703] [19]#...................[39809] - 1.84 1.88 [64]#...................[9554] [326]#...................[3703] [19]#...................[39809] - 1.88 1.92 [64]#...................[9554] [326]#...................[3703] [19]#...................[39809] - 1.92 1.96 [64]#...................[9554] [326]#...................[3703] [19]#...................[39809] - 1.96 2 [64]#...................[9554] [326]#...................[3703] [19]#...................[39809] - -Sieve of Eratosthenes ----------------------------------------------------------------- - -The `Sieve of Eratosthenes `_ is a standard introductory programming topic. The idea is to find all primes up to some *N* by making a list of the numbers 1 to *N*, then striking out all multiples of 2 except 2 itself, all multiples of 3 except 3 itself, all multiples of 4 except 4 itself, and so on. Whatever survives that without getting marked is a prime. This is easy enough in Miller. Notice that here all the work is in ``begin`` and ``end`` statements; there is no file input (so we use ``mlr -n`` to keep Miller from waiting for input data). - -.. code-block:: none - :emphasize-lines: 1-1 - - cat programs/sieve.mlr - # ================================================================ - # Sieve of Eratosthenes: simple example of Miller DSL as programming language. - # ================================================================ - - # Put this in a begin-block so we can do either - # mlr -n put -q -f name-of-this-file.mlr - # or - # mlr -n put -q -f name-of-this-file.mlr -e '@n = 200' - # i.e. 100 is the default upper limit, and another can be specified using -e. - begin { - @n = 100; - } - - end { - for (int i = 0; i <= @n; i += 1) { - @s[i] = true; - } - @s[0] = false; # 0 is neither prime nor composite - @s[1] = false; # 1 is neither prime nor composite - # Strike out multiples - for (int i = 2; i <= @n; i += 1) { - for (int j = i+i; j <= @n; j += i) { - @s[j] = false; - } - } - # Print survivors - for (int i = 0; i <= @n; i += 1) { - if (@s[i]) { - print i; - } - } - } - -.. code-block:: none - :emphasize-lines: 1-1 - - mlr -n put -f programs/sieve.mlr - 2 - 3 - 5 - 7 - 11 - 13 - 17 - 19 - 23 - 29 - 31 - 37 - 41 - 43 - 47 - 53 - 59 - 61 - 67 - 71 - 73 - 79 - 83 - 89 - 97 - -Mandelbrot-set generator ----------------------------------------------------------------- - -The `Mandelbrot set `_ is also easily expressed. This isn't an important case of data-processing in the vein for which Miller was designed, but it is an example of Miller as a general-purpose programming language -- a test case for the expressiveness of the language. - -The (approximate) computation of points in the complex plane which are and aren't members is just a few lines of complex arithmetic (see the Wikipedia article); how to render them is another task. Using graphics libraries you can create PNG or JPEG files, but another fun way to do this is by printing various characters to the screen: - -.. code-block:: none - :emphasize-lines: 1-1 - - cat programs/mand.mlr - # Mandelbrot set generator: simple example of Miller DSL as programming language. - begin { - # Set defaults - @rcorn = -2.0; - @icorn = -2.0; - @side = 4.0; - @iheight = 50; - @iwidth = 100; - @maxits = 100; - @levelstep = 5; - @chars = "@X*o-."; # Palette of characters to print to the screen. - @verbose = false; - @do_julia = false; - @jr = 0.0; # Real part of Julia point, if any - @ji = 0.0; # Imaginary part of Julia point, if any - } - - # Here, we can override defaults from an input file (if any). In Miller's - # put/filter DSL, absent-null right-hand sides result in no assignment so we - # can simply put @rcorn = $rcorn: if there is a field in the input like - # 'rcorn = -1.847' we'll read and use it, else we'll keep the default. - @rcorn = $rcorn; - @icorn = $icorn; - @side = $side; - @iheight = $iheight; - @iwidth = $iwidth; - @maxits = $maxits; - @levelstep = $levelstep; - @chars = $chars; - @verbose = $verbose; - @do_julia = $do_julia; - @jr = $jr; - @ji = $ji; - - end { - if (@verbose) { - print "RCORN = ".@rcorn; - print "ICORN = ".@icorn; - print "SIDE = ".@side; - print "IHEIGHT = ".@iheight; - print "IWIDTH = ".@iwidth; - print "MAXITS = ".@maxits; - print "LEVELSTEP = ".@levelstep; - print "CHARS = ".@chars; - } - - # Iterate over a matrix of rows and columns, printing one character for each cell. - for (int ii = @iheight-1; ii >= 0; ii -= 1) { - num pi = @icorn + (ii/@iheight) * @side; - for (int ir = 0; ir < @iwidth; ir += 1) { - num pr = @rcorn + (ir/@iwidth) * @side; - printn get_point_plot(pr, pi, @maxits, @do_julia, @jr, @ji); - } - print; - } - } - - # This is a function to approximate membership in the Mandelbrot set (or Julia - # set for a given Julia point if do_julia == true) for a given point in the - # complex plane. - func get_point_plot(pr, pi, maxits, do_julia, jr, ji) { - num zr = 0.0; - num zi = 0.0; - num cr = 0.0; - num ci = 0.0; - - if (!do_julia) { - zr = 0.0; - zi = 0.0; - cr = pr; - ci = pi; - } else { - zr = pr; - zi = pi; - cr = jr; - ci = ji; - } - - int iti = 0; - bool escaped = false; - num zt = 0; - for (iti = 0; iti < maxits; iti += 1) { - num mag = zr*zr + zi+zi; - if (mag > 4.0) { - escaped = true; - break; - } - # z := z^2 + c - zt = zr*zr - zi*zi + cr; - zi = 2*zr*zi + ci; - zr = zt; - } - if (!escaped) { - return "."; - } else { - # The // operator is Miller's (pythonic) integer-division operator - int level = (iti // @levelstep) % strlen(@chars); - return substr(@chars, level, level); - } - } - -At standard resolution this makes a nice little ASCII plot: - -.. code-block:: none - :emphasize-lines: 1-1 - - mlr -n put -f ./programs/mand.mlr - @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXX.XXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXooXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXX**o..*XXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXX*-....-oXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXX@XXXXXXXXXX*......o*XXXXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXXX**oo*-.-........oo.XXXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXXXXXX....................X..o-XXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@@@@@@@@@XXXXXXXXXXXXXXX*oo......................oXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@@@@@@@XXX*XXXXXXXXXXXX**o........................*X*X@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@@@@XXXXXXooo***o*.*XX**X..........................o-XX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@@XXXXXXXX*-.......-***.............................oXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@XXXXXXXX*@..........Xo............................*XX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@XXXX@XXXXXXXX*o@oX...........@...........................oXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - .........................................................o*XXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@XXXXXXXXX*-.oX...........@...........................oXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@XXXXXXXXXX**@..........*o............................*XXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@XXXXXXXXXXXXX-........***.............................oXXXXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@XXXXXXXXXXXXoo****o*.XX***@..........................o-XXXXXXXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@@@@@XXXXX*XXXX*XXXXXXX**-........................***XXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@@@@@@@@@@@XXXXXXXXXXXXX*o*.....................@o*XXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXXXXX*....................*..o-XX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXX*ooo*-.o........oo.X*XXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXX**@.....*XXXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXX*o....-o*XXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXXXo*o..*XXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXXXXXX*o*XXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXXXXX@XXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXX@@XXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ - -But using a very small font size (as small as my Mac will let me go), and by choosing the coordinates to zoom in on a particular part of the complex plane, we can get a nice little picture: - -.. code-block:: none - - #!/bin/bash - # Get the number of rows and columns from the terminal window dimensions - iheight=$(stty size | mlr --nidx --fs space cut -f 1) - iwidth=$(stty size | mlr --nidx --fs space cut -f 2) - echo "rcorn=-1.755350,icorn=+0.014230,side=0.000020,maxits=10000,iheight=$iheight,iwidth=$iwidth" \ - | mlr put -f programs/mand.mlr - -.. image:: pix/mand.png diff --git a/docs6/cookbook2.rst.in b/docs6/cookbook2.rst.in deleted file mode 100644 index 8917ff748..000000000 --- a/docs6/cookbook2.rst.in +++ /dev/null @@ -1,156 +0,0 @@ -Cookbook part 2: Random things, and some math -================================================================ - -Randomly selecting words from a list ----------------------------------------------------------------- - -Given this `word list <./data/english-words.txt>`_, first take a look to see what the first few lines look like: - -GENRST_CARDIFY_HIGHLIGHT_ONE -head data/english-words.txt -a -aa -aal -aalii -aam -aardvark -aardwolf -aba -abac -abaca -GENRST_EOF - -Then the following will randomly sample ten words with four to eight characters in them: - -GENRST_CARDIFY_HIGHLIGHT_ONE -mlr --from data/english-words.txt --nidx filter -S 'n=strlen($1);4<=n&&n<=8' then sample -k 10 -thionine -birchman -mildewy -avigate -addedly -abaze -askant -aiming -insulant -coinmate -GENRST_EOF - -Randomly generating jabberwocky words ----------------------------------------------------------------- - -These are simple *n*-grams as `described here `_. Some common functions are `located here `_. Then here are scripts for `1-grams `_ `2-grams `_ `3-grams `_ `4-grams `_, and `5-grams `_. - -The idea is that words from the input file are consumed, then taken apart and pasted back together in ways which imitate the letter-to-letter transitions found in the word list -- giving us automatically generated words in the same vein as *bromance* and *spork*: - -GENRST_CARDIFY_HIGHLIGHT_ONE -mlr --nidx --from ./ngrams/gsl-2000.txt put -q -f ./ngrams/ngfuncs.mlr -f ./ngrams/ng5.mlr -beard -plastinguish -politicially -noise -loan -country -controductionary -suppery -lose -lessors -dollar -judge -rottendence -lessenger -diffendant -suggestional -GENRST_EOF - -Program timing ----------------------------------------------------------------- - -This admittedly artificial example demonstrates using Miller time and stats functions to introspectively acquire some information about Miller's own runtime. The ``delta`` function computes the difference between successive timestamps. - -GENRST_INCLUDE_ESCAPED(data/timing-example.txt) - -Computing interquartile ranges ----------------------------------------------------------------- - -For one or more specified field names, simply compute p25 and p75, then write the IQR as the difference of p75 and p25: - -GENRST_INCLUDE_AND_RUN_ESCAPED(data/iqr1.sh) - -For wildcarded field names, first compute p25 and p75, then loop over field names with ``p25`` in them: - -GENRST_INCLUDE_AND_RUN_ESCAPED(data/iqrn.sh) - -Computing weighted means ----------------------------------------------------------------- - -This might be more elegantly implemented as an option within the ``stats1`` verb. Meanwhile, it's expressible within the DSL: - -GENRST_INCLUDE_AND_RUN_ESCAPED(data/weighted-mean.sh) - -Generating random numbers from various distributions ----------------------------------------------------------------- - -Here we can chain together a few simple building blocks: - -GENRST_RUN_COMMAND -cat expo-sample.sh -GENRST_EOF - -Namely: - -* Set the Miller random-number seed so this webdoc looks the same every time I regenerate it. -* Use pretty-printed tabular output. -* Use pretty-printed tabular output. -* Use ``seqgen`` to produce 100,000 records ``i=0``, ``i=1``, etc. -* Send those to a ``put`` step which defines an inverse-transform-sampling function and calls it twice, then computes the sum and product of samples. -* Send those to a histogram, and from there to a bar-plotter. This is just for visualization; you could just as well output CSV and send that off to your own plotting tool, etc. - -The output is as follows: - -GENRST_RUN_COMMAND -sh expo-sample.sh -GENRST_EOF - -Sieve of Eratosthenes ----------------------------------------------------------------- - -The `Sieve of Eratosthenes `_ is a standard introductory programming topic. The idea is to find all primes up to some *N* by making a list of the numbers 1 to *N*, then striking out all multiples of 2 except 2 itself, all multiples of 3 except 3 itself, all multiples of 4 except 4 itself, and so on. Whatever survives that without getting marked is a prime. This is easy enough in Miller. Notice that here all the work is in ``begin`` and ``end`` statements; there is no file input (so we use ``mlr -n`` to keep Miller from waiting for input data). - -GENRST_RUN_COMMAND -cat programs/sieve.mlr -GENRST_EOF - -GENRST_RUN_COMMAND -mlr -n put -f programs/sieve.mlr -GENRST_EOF - -Mandelbrot-set generator ----------------------------------------------------------------- - -The `Mandelbrot set `_ is also easily expressed. This isn't an important case of data-processing in the vein for which Miller was designed, but it is an example of Miller as a general-purpose programming language -- a test case for the expressiveness of the language. - -The (approximate) computation of points in the complex plane which are and aren't members is just a few lines of complex arithmetic (see the Wikipedia article); how to render them is another task. Using graphics libraries you can create PNG or JPEG files, but another fun way to do this is by printing various characters to the screen: - -GENRST_RUN_COMMAND -cat programs/mand.mlr -GENRST_EOF - -At standard resolution this makes a nice little ASCII plot: - -GENRST_RUN_COMMAND -mlr -n put -f ./programs/mand.mlr -GENRST_EOF - -But using a very small font size (as small as my Mac will let me go), and by choosing the coordinates to zoom in on a particular part of the complex plane, we can get a nice little picture: - -GENRST_CARDIFY -#!/bin/bash -# Get the number of rows and columns from the terminal window dimensions -iheight=$(stty size | mlr --nidx --fs space cut -f 1) -iwidth=$(stty size | mlr --nidx --fs space cut -f 2) -echo "rcorn=-1.755350,icorn=+0.014230,side=0.000020,maxits=10000,iheight=$iheight,iwidth=$iwidth" \ -| mlr put -f programs/mand.mlr -GENRST_EOF - -.. image:: pix/mand.png diff --git a/docs6/cookbook3.rst b/docs6/cookbook3.rst deleted file mode 100644 index f3fd137ef..000000000 --- a/docs6/cookbook3.rst +++ /dev/null @@ -1,353 +0,0 @@ -.. - PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE. - -Cookbook part 3: Stats with and without out-of-stream variables -================================================================ - -Overview ----------------------------------------------------------------- - -One of Miller's strengths is its compact notation: for example, given input of the form - -.. code-block:: none - :emphasize-lines: 1-1 - - head -n 5 ../data/medium - a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533 - a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797 - a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776 - a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463 - a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729 - -you can simply do - -.. code-block:: none - :emphasize-lines: 1-1 - - mlr --oxtab stats1 -a sum -f x ../data/medium - x_sum 4986.019681679581 - -or - -.. code-block:: none - :emphasize-lines: 1-1 - - mlr --opprint stats1 -a sum -f x -g b ../data/medium - b x_sum - pan 965.7636699425815 - wye 1023.5484702619565 - zee 979.7420161495838 - eks 1016.7728571314786 - hat 1000.192668193983 - -rather than the more tedious - -.. code-block:: none - :emphasize-lines: 1-6 - - mlr --oxtab put -q ' - @x_sum += $x; - end { - emit @x_sum - } - ' data/medium - x_sum 4986.019681679581 - -or - -.. code-block:: none - :emphasize-lines: 1-6 - - mlr --opprint put -q ' - @x_sum[$b] += $x; - end { - emit @x_sum, "b" - } - ' data/medium - b x_sum - pan 965.7636699425815 - wye 1023.5484702619565 - zee 979.7420161495838 - eks 1016.7728571314786 - hat 1000.192668193983 - -The former (``mlr stats1`` et al.) has the advantages of being easier to type, being less error-prone to type, and running faster. - -Nonetheless, out-of-stream variables (which I whimsically call *oosvars*), begin/end blocks, and emit statements give you the ability to implement logic -- if you wish to do so -- which isn't present in other Miller verbs. (If you find yourself often using the same out-of-stream-variable logic over and over, please file a request at https://github.com/johnkerl/miller/issues to get it implemented directly in Go as a Miller verb of its own.) - -The following examples compute some things using oosvars which are already computable using Miller verbs, by way of providing food for thought. - -Mean without/with oosvars ----------------------------------------------------------------- - -.. code-block:: none - :emphasize-lines: 1-1 - - mlr --opprint stats1 -a mean -f x data/medium - x_mean - 0.49860196816795804 - -.. code-block:: none - :emphasize-lines: 1-8 - - mlr --opprint put -q ' - @x_sum += $x; - @x_count += 1; - end { - @x_mean = @x_sum / @x_count; - emit @x_mean - } - ' data/medium - x_mean - 0.49860196816795804 - -Keyed mean without/with oosvars ----------------------------------------------------------------- - -.. code-block:: none - :emphasize-lines: 1-1 - - mlr --opprint stats1 -a mean -f x -g a,b data/medium - a b x_mean - pan pan 0.5133141190437597 - eks pan 0.48507555383425127 - wye wye 0.49150092785839306 - eks wye 0.4838950517724162 - wye pan 0.4996119901034838 - zee pan 0.5198298297816007 - eks zee 0.49546320772681596 - zee wye 0.5142667998230479 - hat wye 0.49381326184632596 - pan wye 0.5023618498923658 - zee eks 0.4883932942792647 - hat zee 0.5099985721987774 - hat eks 0.48587864619953547 - wye hat 0.4977304763723314 - pan eks 0.5036718595143479 - eks eks 0.5227992666570941 - hat hat 0.47993053101017374 - hat pan 0.4643355557376876 - zee zee 0.5127559183726382 - pan hat 0.492140950155604 - pan zee 0.4966041598627583 - zee hat 0.46772617655014515 - wye zee 0.5059066170573692 - eks hat 0.5006790659966355 - wye eks 0.5306035254809106 - -.. code-block:: none - :emphasize-lines: 1-10 - - mlr --opprint put -q ' - @x_sum[$a][$b] += $x; - @x_count[$a][$b] += 1; - end{ - for ((a, b), v in @x_sum) { - @x_mean[a][b] = @x_sum[a][b] / @x_count[a][b]; - } - emit @x_mean, "a", "b" - } - ' data/medium - a b x_mean - pan pan 0.5133141190437597 - pan wye 0.5023618498923658 - pan eks 0.5036718595143479 - pan hat 0.492140950155604 - pan zee 0.4966041598627583 - eks pan 0.48507555383425127 - eks wye 0.4838950517724162 - eks zee 0.49546320772681596 - eks eks 0.5227992666570941 - eks hat 0.5006790659966355 - wye wye 0.49150092785839306 - wye pan 0.4996119901034838 - wye hat 0.4977304763723314 - wye zee 0.5059066170573692 - wye eks 0.5306035254809106 - zee pan 0.5198298297816007 - zee wye 0.5142667998230479 - zee eks 0.4883932942792647 - zee zee 0.5127559183726382 - zee hat 0.46772617655014515 - hat wye 0.49381326184632596 - hat zee 0.5099985721987774 - hat eks 0.48587864619953547 - hat hat 0.47993053101017374 - hat pan 0.4643355557376876 - -Variance and standard deviation without/with oosvars ----------------------------------------------------------------- - -.. code-block:: none - :emphasize-lines: 1-1 - - mlr --oxtab stats1 -a count,sum,mean,var,stddev -f x data/medium - x_count 10000 - x_sum 4986.019681679581 - x_mean 0.49860196816795804 - x_var 0.08426974433144456 - x_stddev 0.2902925151144007 - -.. code-block:: none - :emphasize-lines: 1-1 - - cat variance.mlr - @n += 1; - @sumx += $x; - @sumx2 += $x**2; - end { - @mean = @sumx / @n; - @var = (@sumx2 - @mean * (2 * @sumx - @n * @mean)) / (@n - 1); - @stddev = sqrt(@var); - emitf @n, @sumx, @sumx2, @mean, @var, @stddev - } - -.. code-block:: none - :emphasize-lines: 1-1 - - mlr --oxtab put -q -f variance.mlr data/medium - n 10000 - sumx 4986.019681679581 - sumx2 3328.652400179729 - mean 0.49860196816795804 - var 0.08426974433144456 - stddev 0.2902925151144007 - -You can also do this keyed, of course, imitating the keyed-mean example above. - -Min/max without/with oosvars ----------------------------------------------------------------- - -.. code-block:: none - :emphasize-lines: 1-1 - - mlr --oxtab stats1 -a min,max -f x data/medium - x_min 4.509679127584487e-05 - x_max 0.999952670371898 - -.. code-block:: none - :emphasize-lines: 1-5 - - mlr --oxtab put -q ' - @x_min = min(@x_min, $x); - @x_max = max(@x_max, $x); - end{emitf @x_min, @x_max} - ' data/medium - x_min 4.509679127584487e-05 - x_max 0.999952670371898 - -Keyed min/max without/with oosvars ----------------------------------------------------------------- - -.. code-block:: none - :emphasize-lines: 1-1 - - mlr --opprint stats1 -a min,max -f x -g a data/medium - a x_min x_max - pan 0.00020390740306253097 0.9994029107062516 - eks 0.0006917972627396018 0.9988110946859143 - wye 0.0001874794831505655 0.9998228522652893 - zee 0.0005486114815762555 0.9994904324789629 - hat 4.509679127584487e-05 0.999952670371898 - -.. code-block:: none - :emphasize-lines: 1-7 - - mlr --opprint --from data/medium put -q ' - @min[$a] = min(@min[$a], $x); - @max[$a] = max(@max[$a], $x); - end{ - emit (@min, @max), "a"; - } - ' - a min max - pan 0.00020390740306253097 0.9994029107062516 - eks 0.0006917972627396018 0.9988110946859143 - wye 0.0001874794831505655 0.9998228522652893 - zee 0.0005486114815762555 0.9994904324789629 - hat 4.509679127584487e-05 0.999952670371898 - -Delta without/with oosvars ----------------------------------------------------------------- - -.. code-block:: none - :emphasize-lines: 1-1 - - mlr --opprint step -a delta -f x data/small - a b i x y x_delta - pan pan 1 0.3467901443380824 0.7268028627434533 0 - eks pan 2 0.7586799647899636 0.5221511083334797 0.41188982045188116 - wye wye 3 0.20460330576630303 0.33831852551664776 -0.5540766590236605 - eks wye 4 0.38139939387114097 0.13418874328430463 0.17679608810483793 - wye pan 5 0.5732889198020006 0.8636244699032729 0.19188952593085962 - -.. code-block:: none - :emphasize-lines: 1-4 - - mlr --opprint put ' - $x_delta = is_present(@last) ? $x - @last : 0; - @last = $x - ' data/small - a b i x y x_delta - pan pan 1 0.3467901443380824 0.7268028627434533 0 - eks pan 2 0.7586799647899636 0.5221511083334797 0.41188982045188116 - wye wye 3 0.20460330576630303 0.33831852551664776 -0.5540766590236605 - eks wye 4 0.38139939387114097 0.13418874328430463 0.17679608810483793 - wye pan 5 0.5732889198020006 0.8636244699032729 0.19188952593085962 - -Keyed delta without/with oosvars ----------------------------------------------------------------- - -.. code-block:: none - :emphasize-lines: 1-1 - - mlr --opprint step -a delta -f x -g a data/small - a b i x y x_delta - pan pan 1 0.3467901443380824 0.7268028627434533 0 - eks pan 2 0.7586799647899636 0.5221511083334797 0 - wye wye 3 0.20460330576630303 0.33831852551664776 0 - eks wye 4 0.38139939387114097 0.13418874328430463 -0.3772805709188226 - wye pan 5 0.5732889198020006 0.8636244699032729 0.36868561403569755 - -.. code-block:: none - :emphasize-lines: 1-4 - - mlr --opprint put ' - $x_delta = is_present(@last[$a]) ? $x - @last[$a] : 0; - @last[$a]=$x - ' data/small - a b i x y x_delta - pan pan 1 0.3467901443380824 0.7268028627434533 0 - eks pan 2 0.7586799647899636 0.5221511083334797 0 - wye wye 3 0.20460330576630303 0.33831852551664776 0 - eks wye 4 0.38139939387114097 0.13418874328430463 -0.3772805709188226 - wye pan 5 0.5732889198020006 0.8636244699032729 0.36868561403569755 - -Exponentially weighted moving averages without/with oosvars ----------------------------------------------------------------- - -.. code-block:: none - :emphasize-lines: 1-1 - - mlr --opprint step -a ewma -d 0.1 -f x data/small - a b i x y x_ewma_0.1 - pan pan 1 0.3467901443380824 0.7268028627434533 0.3467901443380824 - eks pan 2 0.7586799647899636 0.5221511083334797 0.3879791263832706 - wye wye 3 0.20460330576630303 0.33831852551664776 0.36964154432157387 - eks wye 4 0.38139939387114097 0.13418874328430463 0.37081732927653055 - wye pan 5 0.5732889198020006 0.8636244699032729 0.3910644883290776 - -.. code-block:: none - :emphasize-lines: 1-5 - - mlr --opprint put ' - begin{ @a=0.1 }; - $e = NR==1 ? $x : @a * $x + (1 - @a) * @e; - @e=$e - ' data/small - a b i x y e - pan pan 1 0.3467901443380824 0.7268028627434533 0.3467901443380824 - eks pan 2 0.7586799647899636 0.5221511083334797 0.3879791263832706 - wye wye 3 0.20460330576630303 0.33831852551664776 0.36964154432157387 - eks wye 4 0.38139939387114097 0.13418874328430463 0.37081732927653055 - wye pan 5 0.5732889198020006 0.8636244699032729 0.3910644883290776 diff --git a/docs6/cookbook3.rst.in b/docs6/cookbook3.rst.in deleted file mode 100644 index 399188775..000000000 --- a/docs6/cookbook3.rst.in +++ /dev/null @@ -1,131 +0,0 @@ -Cookbook part 3: Stats with and without out-of-stream variables -================================================================ - -Overview ----------------------------------------------------------------- - -One of Miller's strengths is its compact notation: for example, given input of the form - -GENRST_RUN_COMMAND -head -n 5 ../data/medium -GENRST_EOF - -you can simply do - -GENRST_RUN_COMMAND -mlr --oxtab stats1 -a sum -f x ../data/medium -GENRST_EOF - -or - -GENRST_RUN_COMMAND -mlr --opprint stats1 -a sum -f x -g b ../data/medium -GENRST_EOF - -rather than the more tedious - -GENRST_INCLUDE_AND_RUN_ESCAPED(oosvar-example-sum.sh) - -or - -GENRST_INCLUDE_AND_RUN_ESCAPED(oosvar-example-sum-grouped.sh) - -The former (``mlr stats1`` et al.) has the advantages of being easier to type, being less error-prone to type, and running faster. - -Nonetheless, out-of-stream variables (which I whimsically call *oosvars*), begin/end blocks, and emit statements give you the ability to implement logic -- if you wish to do so -- which isn't present in other Miller verbs. (If you find yourself often using the same out-of-stream-variable logic over and over, please file a request at https://github.com/johnkerl/miller/issues to get it implemented directly in Go as a Miller verb of its own.) - -The following examples compute some things using oosvars which are already computable using Miller verbs, by way of providing food for thought. - -Mean without/with oosvars ----------------------------------------------------------------- - -GENRST_RUN_COMMAND -mlr --opprint stats1 -a mean -f x data/medium -GENRST_EOF - -GENRST_INCLUDE_AND_RUN_ESCAPED(data/mean-with-oosvars.sh) - -Keyed mean without/with oosvars ----------------------------------------------------------------- - -GENRST_RUN_COMMAND -mlr --opprint stats1 -a mean -f x -g a,b data/medium -GENRST_EOF - -GENRST_INCLUDE_AND_RUN_ESCAPED(data/keyed-mean-with-oosvars.sh) - -Variance and standard deviation without/with oosvars ----------------------------------------------------------------- - -GENRST_RUN_COMMAND -mlr --oxtab stats1 -a count,sum,mean,var,stddev -f x data/medium -GENRST_EOF - -GENRST_RUN_COMMAND -cat variance.mlr -GENRST_EOF - -GENRST_RUN_COMMAND -mlr --oxtab put -q -f variance.mlr data/medium -GENRST_EOF - -You can also do this keyed, of course, imitating the keyed-mean example above. - -Min/max without/with oosvars ----------------------------------------------------------------- - -GENRST_RUN_COMMAND -mlr --oxtab stats1 -a min,max -f x data/medium -GENRST_EOF - -GENRST_RUN_COMMAND -mlr --oxtab put -q ' - @x_min = min(@x_min, $x); - @x_max = max(@x_max, $x); - end{emitf @x_min, @x_max} -' data/medium -GENRST_EOF - -Keyed min/max without/with oosvars ----------------------------------------------------------------- - -GENRST_RUN_COMMAND -mlr --opprint stats1 -a min,max -f x -g a data/medium -GENRST_EOF - -GENRST_INCLUDE_AND_RUN_ESCAPED(data/keyed-min-max-with-oosvars.sh) - -Delta without/with oosvars ----------------------------------------------------------------- - -GENRST_RUN_COMMAND -mlr --opprint step -a delta -f x data/small -GENRST_EOF - -GENRST_RUN_COMMAND -mlr --opprint put ' - $x_delta = is_present(@last) ? $x - @last : 0; - @last = $x -' data/small -GENRST_EOF - -Keyed delta without/with oosvars ----------------------------------------------------------------- - -GENRST_RUN_COMMAND -mlr --opprint step -a delta -f x -g a data/small -GENRST_EOF - -GENRST_RUN_COMMAND -mlr --opprint put ' - $x_delta = is_present(@last[$a]) ? $x - @last[$a] : 0; - @last[$a]=$x -' data/small -GENRST_EOF - -Exponentially weighted moving averages without/with oosvars ----------------------------------------------------------------- - -GENRST_INCLUDE_AND_RUN_ESCAPED(verb-example-ewma.sh) - -GENRST_INCLUDE_AND_RUN_ESCAPED(oosvar-example-ewma.sh) diff --git a/docs6/data-cleaning-examples.rst b/docs6/data-cleaning-examples.rst new file mode 100644 index 000000000..4e4638bdc --- /dev/null +++ b/docs6/data-cleaning-examples.rst @@ -0,0 +1,77 @@ +.. + PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE. + +Data-cleaning examples +================================================================ + +Here are some ways to use the type-checking options as described in :ref:`reference-dsl-type-tests-and-assertions` Suppose you have the following data file, with inconsistent typing for boolean. (Also imagine that, for the sake of discussion, we have a million-line file rather than a four-line file, so we can't see it all at once and some automation is called for.) + +.. code-block:: none + :emphasize-lines: 1-1 + + cat data/het-bool.csv + name,reachable + barney,false + betty,true + fred,true + wilma,1 + +One option is to coerce everything to boolean, or integer: + +.. code-block:: none + :emphasize-lines: 1-1 + + mlr --icsv --opprint put '$reachable = boolean($reachable)' data/het-bool.csv + name reachable + barney false + betty true + fred true + wilma true + +.. code-block:: none + :emphasize-lines: 1-1 + + mlr --icsv --opprint put '$reachable = int(boolean($reachable))' data/het-bool.csv + name reachable + barney 0 + betty 1 + fred 1 + wilma 1 + +A second option is to flag badly formatted data within the output stream: + +.. code-block:: none + :emphasize-lines: 1-1 + + mlr --icsv --opprint put '$format_ok = is_string($reachable)' data/het-bool.csv + name reachable format_ok + barney false false + betty true false + fred true false + wilma 1 false + +Or perhaps to flag badly formatted data outside the output stream: + +.. code-block:: none + :emphasize-lines: 1-3 + + mlr --icsv --opprint put ' + if (!is_string($reachable)) {eprint "Malformed at NR=".NR} + ' data/het-bool.csv + Malformed at NR=1 + Malformed at NR=2 + Malformed at NR=3 + Malformed at NR=4 + name reachable + barney false + betty true + fred true + wilma 1 + +A third way is to abort the process on first instance of bad data: + +.. code-block:: none + :emphasize-lines: 1-1 + + mlr --csv put '$reachable = asserting_string($reachable)' data/het-bool.csv + Miller: is_string type-assertion failed at NR=1 FNR=1 FILENAME=data/het-bool.csv diff --git a/docs6/data-cleaning-examples.rst.in b/docs6/data-cleaning-examples.rst.in new file mode 100644 index 000000000..a1e55f9a0 --- /dev/null +++ b/docs6/data-cleaning-examples.rst.in @@ -0,0 +1,38 @@ +Data-cleaning examples +================================================================ + +Here are some ways to use the type-checking options as described in :ref:`reference-dsl-type-tests-and-assertions` Suppose you have the following data file, with inconsistent typing for boolean. (Also imagine that, for the sake of discussion, we have a million-line file rather than a four-line file, so we can't see it all at once and some automation is called for.) + +GENRST_RUN_COMMAND +cat data/het-bool.csv +GENRST_EOF + +One option is to coerce everything to boolean, or integer: + +GENRST_RUN_COMMAND +mlr --icsv --opprint put '$reachable = boolean($reachable)' data/het-bool.csv +GENRST_EOF + +GENRST_RUN_COMMAND +mlr --icsv --opprint put '$reachable = int(boolean($reachable))' data/het-bool.csv +GENRST_EOF + +A second option is to flag badly formatted data within the output stream: + +GENRST_RUN_COMMAND +mlr --icsv --opprint put '$format_ok = is_string($reachable)' data/het-bool.csv +GENRST_EOF + +Or perhaps to flag badly formatted data outside the output stream: + +GENRST_RUN_COMMAND +mlr --icsv --opprint put ' + if (!is_string($reachable)) {eprint "Malformed at NR=".NR} +' data/het-bool.csv +GENRST_EOF + +A third way is to abort the process on first instance of bad data: + +GENRST_RUN_COMMAND_TOLERATING_ERROR +mlr --csv put '$reachable = asserting_string($reachable)' data/het-bool.csv +GENRST_EOF diff --git a/docs6/data/delimiter-examples.txt b/docs6/data/delimiter-examples.txt deleted file mode 100644 index 27d7c7233..000000000 --- a/docs6/data/delimiter-examples.txt +++ /dev/null @@ -1,57 +0,0 @@ -# Use the `file` command to see if there are CR/LF terminators (in this case, -# there are not): -$ file data/colours.csv -data/colours.csv: UTF-8 Unicode text - -# Look at the file to find names of fields -$ cat data/colours.csv -KEY;DE;EN;ES;FI;FR;IT;NL;PL;RO;TR -masterdata_colourcode_1;Weiß;White;Blanco;Valkoinen;Blanc;Bianco;Wit;Biały;Alb;Beyaz -masterdata_colourcode_2;Schwarz;Black;Negro;Musta;Noir;Nero;Zwart;Czarny;Negru;Siyah - -# Extract a few fields: -$ mlr --csv cut -f KEY,PL,RO data/colours.csv -(only blank lines appear) - -# Use XTAB output format to get a sharper picture of where records/fields -# are being split: -$ mlr --icsv --oxtab cat data/colours.csv -KEY;DE;EN;ES;FI;FR;IT;NL;PL;RO;TR masterdata_colourcode_1;Weiß;White;Blanco;Valkoinen;Blanc;Bianco;Wit;Biały;Alb;Beyaz - -KEY;DE;EN;ES;FI;FR;IT;NL;PL;RO;TR masterdata_colourcode_2;Schwarz;Black;Negro;Musta;Noir;Nero;Zwart;Czarny;Negru;Siyah - -# Using XTAB output format makes it clearer that KEY;DE;...;RO;TR is being -# treated as a single field name in the CSV header, and likewise each -# subsequent line is being treated as a single field value. This is because -# the default field separator is a comma but we have semicolons here. -# Use XTAB again with different field separator (--fs semicolon): - mlr --icsv --ifs semicolon --oxtab cat data/colours.csv -KEY masterdata_colourcode_1 -DE Weiß -EN White -ES Blanco -FI Valkoinen -FR Blanc -IT Bianco -NL Wit -PL Biały -RO Alb -TR Beyaz - -KEY masterdata_colourcode_2 -DE Schwarz -EN Black -ES Negro -FI Musta -FR Noir -IT Nero -NL Zwart -PL Czarny -RO Negru -TR Siyah - -# Using the new field-separator, retry the cut: - mlr --csv --fs semicolon cut -f KEY,PL,RO data/colours.csv -KEY;PL;RO -masterdata_colourcode_1;Biały;Alb -masterdata_colourcode_2;Czarny;Negru diff --git a/docs6/cookbook4.rst b/docs6/dkvp-examples.rst similarity index 99% rename from docs6/cookbook4.rst rename to docs6/dkvp-examples.rst index 40e0ac446..840e84d07 100644 --- a/docs6/cookbook4.rst +++ b/docs6/dkvp-examples.rst @@ -1,7 +1,7 @@ .. PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE. -Cookbook4: to be filed +DKVP I/O examples ====================== DKVP I/O in Python diff --git a/docs6/cookbook4.rst.in b/docs6/dkvp-examples.rst.in similarity index 97% rename from docs6/cookbook4.rst.in rename to docs6/dkvp-examples.rst.in index 73ccf4c65..29777205f 100644 --- a/docs6/cookbook4.rst.in +++ b/docs6/dkvp-examples.rst.in @@ -1,4 +1,4 @@ -Cookbook4: to be filed +DKVP I/O examples ====================== DKVP I/O in Python diff --git a/docs6/index.rst b/docs6/index.rst index 78d16e12f..edfcfb03f 100644 --- a/docs6/index.rst +++ b/docs6/index.rst @@ -11,7 +11,6 @@ Getting started 10min keystroke-savers programming-language - repl miller-on-windows community @@ -28,6 +27,7 @@ Miller in more detail internationalization output-colorization customization + repl new-in-miller-6 contributing @@ -37,29 +37,24 @@ FAQs and recipes .. toctree:: :maxdepth: 1 - shapes-of-data csv-with-and-without-headers + shapes-of-data operating-on-all-fields special-symbols-and-formatting dates-and-times then-chaining joins shell-commands - -To be refactored ----------------------------------------------------------------- - -.. toctree:: - :maxdepth: 1 - - quick-examples data-diving-examples - cookbook - cookbook2 - cookbook3 - cookbook4 - sql-examples log-processing-examples + sql-examples + data-cleaning-examples + statistics-examples + randomizing-examples + two-pass-algorithms + dkvp-examples + programming-examples + misc-examples Background ---------------------------------------------------------------- diff --git a/docs6/joins.rst b/docs6/joins.rst index c42334cda..842f68908 100644 --- a/docs6/joins.rst +++ b/docs6/joins.rst @@ -178,5 +178,3 @@ We can run the input file through multiple ``join`` commands in a ``then``-chain 20 idle Carol mix 10 idle Bob knead 30 occupied Alice clean - -.. _cookbook-data-cleaning-examples: diff --git a/docs6/joins.rst.in b/docs6/joins.rst.in index 34476b803..7cf89861a 100644 --- a/docs6/joins.rst.in +++ b/docs6/joins.rst.in @@ -87,5 +87,3 @@ mlr --icsv --opprint join -f multi-join/name-lookup.csv -j id \ then join -f multi-join/status-lookup.csv -j id \ multi-join/input.csv GENRST_EOF - -.. _cookbook-data-cleaning-examples: diff --git a/docs6/log-processing-examples.rst b/docs6/log-processing-examples.rst index 17615175b..99c7866a9 100644 --- a/docs6/log-processing-examples.rst +++ b/docs6/log-processing-examples.rst @@ -2,10 +2,13 @@ PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE. Log-processing examples ----------------------------------------------------------------- +================================================================ Another of my favorite use-cases for Miller is doing ad-hoc processing of log-file data. Here's where DKVP format really shines: one, since the field names and field values are present on every line, every line stands on its own. That means you can ``grep`` or what have you. Also it means not every line needs to have the same list of field names ("schema"). +Generating and aggregating log-file output +---------------------------------------------------------------- + Again, all the examples in the CSV section apply here -- just change the input-format flags. But there's more you can do when not all the records have the same shape. Writing a program -- in any language whatsoever -- you can have it print out log lines as it goes along, with items for various events jumbled together. After the program has finished running you can sort it all out, filter it, analyze it, and learn from it. @@ -194,3 +197,21 @@ Alternatively, we can simply group the similar data for a better look: 2016-09-02T12:35:20Z 100 554 2016-09-02T12:35:36Z 100 612 2016-09-02T12:35:42Z 100 728 + +Parsing log-file output +---------------------------------------------------------------- + +This, of course, depends highly on what's in your log files. But, as an example, suppose you have log-file lines such as + +.. code-block:: none + + 2015-10-08 08:29:09,445 INFO com.company.path.to.ClassName @ [sometext] various/sorts/of data {& punctuation} hits=1 status=0 time=2.378 + +I prefer to pre-filter with ``grep`` and/or ``sed`` to extract the structured text, then hand that to Miller. Example: + +.. code-block:: none + :emphasize-lines: 1-3 + + grep 'various sorts' *.log \ + | sed 's/.*} //' \ + | mlr --fs space --repifs --oxtab stats1 -a min,p10,p50,p90,max -f time -g status diff --git a/docs6/log-processing-examples.rst.in b/docs6/log-processing-examples.rst.in index 1170656f7..e52ce3f31 100644 --- a/docs6/log-processing-examples.rst.in +++ b/docs6/log-processing-examples.rst.in @@ -1,8 +1,11 @@ Log-processing examples ----------------------------------------------------------------- +================================================================ Another of my favorite use-cases for Miller is doing ad-hoc processing of log-file data. Here's where DKVP format really shines: one, since the field names and field values are present on every line, every line stands on its own. That means you can ``grep`` or what have you. Also it means not every line needs to have the same list of field names ("schema"). +Generating and aggregating log-file output +---------------------------------------------------------------- + Again, all the examples in the CSV section apply here -- just change the input-format flags. But there's more you can do when not all the records have the same shape. Writing a program -- in any language whatsoever -- you can have it print out log lines as it goes along, with items for various events jumbled together. After the program has finished running you can sort it all out, filter it, analyze it, and learn from it. @@ -28,3 +31,20 @@ GENRST_EOF GENRST_RUN_COMMAND mlr --opprint group-like then sec2gmt time log.txt GENRST_EOF + +Parsing log-file output +---------------------------------------------------------------- + +This, of course, depends highly on what's in your log files. But, as an example, suppose you have log-file lines such as + +GENRST_CARDIFY +2015-10-08 08:29:09,445 INFO com.company.path.to.ClassName @ [sometext] various/sorts/of data {& punctuation} hits=1 status=0 time=2.378 +GENRST_EOF + +I prefer to pre-filter with ``grep`` and/or ``sed`` to extract the structured text, then hand that to Miller. Example: + +GENRST_SHOW_COMMAND +grep 'various sorts' *.log \ + | sed 's/.*} //' \ + | mlr --fs space --repifs --oxtab stats1 -a min,p10,p50,p90,max -f time -g status +GENRST_EOF diff --git a/docs6/misc-examples.rst b/docs6/misc-examples.rst new file mode 100644 index 000000000..d5cb55ba3 --- /dev/null +++ b/docs6/misc-examples.rst @@ -0,0 +1,344 @@ +.. + PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE. + +Miscellaneous examples +================================================================ + +Column select: + +.. code-block:: none + :emphasize-lines: 1-1 + + mlr --csv cut -f hostname,uptime mydata.csv + +Add new columns as function of other columns: + +.. code-block:: none + :emphasize-lines: 1-1 + + mlr --nidx put '$sum = $7 < 0.0 ? 3.5 : $7 + 2.1*$8' *.dat + +Row filter: + +.. code-block:: none + :emphasize-lines: 1-1 + + mlr --csv filter '$status != "down" && $upsec >= 10000' *.csv + +Apply column labels and pretty-print: + +.. code-block:: none + :emphasize-lines: 1-1 + + grep -v '^#' /etc/group | mlr --ifs : --nidx --opprint label group,pass,gid,member then sort -f group + +Join multiple data sources on key columns: + +.. code-block:: none + :emphasize-lines: 1-1 + + mlr join -j account_id -f accounts.dat then group-by account_name balances.dat + +Mulltiple formats including JSON: + +.. code-block:: none + :emphasize-lines: 1-1 + + mlr --json put '$attr = sub($attr, "([0-9]+)_([0-9]+)_.*", "\1:\2")' data/*.json + +Aggregate per-column statistics: + +.. code-block:: none + :emphasize-lines: 1-1 + + mlr stats1 -a min,mean,max,p10,p50,p90 -f flag,u,v data/* + +Linear regression: + +.. code-block:: none + :emphasize-lines: 1-1 + + mlr stats2 -a linreg-pca -f u,v -g shape data/* + +Aggregate custom per-column statistics: + +.. code-block:: none + :emphasize-lines: 1-1 + + mlr put -q '@sum[$a][$b] += $x; end {emit @sum, "a", "b"}' data/* + +Iterate over data using DSL expressions: + +.. code-block:: none + :emphasize-lines: 1-1 + + mlr --from estimates.tbl put ' + for (k,v in $*) { + if (is_numeric(v) && k =~ "^[t-z].*$") { + $sum += v; $count += 1 + } + } + $mean = $sum / $count # no assignment if count unset + ' + +Run DSL expressions from a script file: + +.. code-block:: none + :emphasize-lines: 1-1 + + mlr --from infile.dat put -f analyze.mlr + +Split/reduce output to multiple filenames: + +.. code-block:: none + :emphasize-lines: 1-1 + + mlr --from infile.dat put 'tee > "./taps/data-".$a."-".$b, $*' + +Compressed I/O: + +.. code-block:: none + :emphasize-lines: 1-1 + + mlr --from infile.dat put 'tee | "gzip > ./taps/data-".$a."-".$b.".gz", $*' + +Interoperate with other data-processing tools using standard pipes: + +.. code-block:: none + :emphasize-lines: 1-1 + + mlr --from infile.dat put -q '@v=$*; dump | "jq .[]"' + +Tap/trace: + +.. code-block:: none + :emphasize-lines: 1-1 + + mlr --from infile.dat put '(NR % 1000 == 0) { print > stderr, "Checkpoint ".NR}' + +Program timing +---------------------------------------------------------------- + +This admittedly artificial example demonstrates using Miller time and stats functions to introspectively acquire some information about Miller's own runtime. The ``delta`` function computes the difference between successive timestamps. + +.. code-block:: none + + $ ruby -e '10000.times{|i|puts "i=#{i+1}"}' > lines.txt + + $ head -n 5 lines.txt + i=1 + i=2 + i=3 + i=4 + i=5 + + mlr --ofmt '%.9le' --opprint put '$t=systime()' then step -a delta -f t lines.txt | head -n 7 + i t t_delta + 1 1430603027.018016 1.430603027e+09 + 2 1430603027.018043 2.694129944e-05 + 3 1430603027.018048 5.006790161e-06 + 4 1430603027.018052 4.053115845e-06 + 5 1430603027.018055 2.861022949e-06 + 6 1430603027.018058 3.099441528e-06 + + mlr --ofmt '%.9le' --oxtab \ + put '$t=systime()' then \ + step -a delta -f t then \ + filter '$i>1' then \ + stats1 -a min,mean,max -f t_delta \ + lines.txt + t_delta_min 2.861022949e-06 + t_delta_mean 4.077508505e-06 + t_delta_max 5.388259888e-05 + +Showing differences between successive queries +---------------------------------------------------------------- + +Suppose you have a database query which you run at one point in time, producing the output on the left, then again later producing the output on the right: + +.. code-block:: none + :emphasize-lines: 1-1 + + cat data/previous_counters.csv + color,count + red,3472 + blue,6838 + orange,694 + purple,12 + +.. code-block:: none + :emphasize-lines: 1-1 + + cat data/current_counters.csv + color,count + red,3467 + orange,670 + yellow,27 + blue,6944 + +And, suppose you want to compute the differences in the counters between adjacent keys. Since the color names aren't all in the same order, nor are they all present on both sides, we can't just paste the two files side-by-side and do some column-four-minus-column-two arithmetic. + +First, rename counter columns to make them distinct: + +.. code-block:: none + :emphasize-lines: 1-1 + + mlr --csv rename count,previous_count data/previous_counters.csv > data/prevtemp.csv + +.. code-block:: none + :emphasize-lines: 1-1 + + cat data/prevtemp.csv + color,previous_count + red,3472 + blue,6838 + orange,694 + purple,12 + +.. code-block:: none + :emphasize-lines: 1-1 + + mlr --csv rename count,current_count data/current_counters.csv > data/currtemp.csv + +.. code-block:: none + :emphasize-lines: 1-1 + + cat data/currtemp.csv + color,current_count + red,3467 + orange,670 + yellow,27 + blue,6944 + +Then, join on the key field(s), and use unsparsify to zero-fill counters absent on one side but present on the other. Use ``--ul`` and ``--ur`` to emit unpaired records (namely, purple on the left and yellow on the right): + +.. code-block:: none + :emphasize-lines: 1-5 + + mlr --icsv --opprint \ + join -j color --ul --ur -f data/prevtemp.csv \ + then unsparsify --fill-with 0 \ + then put '$count_delta = $current_count - $previous_count' \ + data/currtemp.csv + color previous_count current_count count_delta + red 3472 3467 -5 + orange 694 670 -24 + yellow 0 27 (error) + blue 6838 6944 106 + purple 12 0 (error) + +.. _cookbook-memoization-with-oosvars: + +Memoization with out-of-stream variables +---------------------------------------------------------------- + +The recursive function for the Fibonacci sequence is famous for its computational complexity. Namely, using f(0)=1, f(1)=1, f(n)=f(n-1)+f(n-2) for n>=2, the evaluation tree branches left as well as right at each non-trivial level, resulting in millions or more paths to the root 0/1 nodes for larger n. This program + +.. code-block:: none + + mlr --ofmt '%.9lf' --opprint seqgen --start 1 --stop 28 then put ' + func f(n) { + @fcount += 1; # count number of calls to the function + if (n < 2) { + return 1 + } else { + return f(n-1) + f(n-2) # recurse + } + } + + @fcount = 0; + $o = f($i); + $fcount = @fcount; + + ' then put '$seconds=systime()' then step -a delta -f seconds then cut -x -f seconds + +produces output like this: + +.. code-block:: none + + i o fcount seconds_delta + 1 1 1 0 + 2 2 3 0.000039101 + 3 3 5 0.000015974 + 4 5 9 0.000019073 + 5 8 15 0.000026941 + 6 13 25 0.000036955 + 7 21 41 0.000056028 + 8 34 67 0.000086069 + 9 55 109 0.000134945 + 10 89 177 0.000217915 + 11 144 287 0.000355959 + 12 233 465 0.000506163 + 13 377 753 0.000811815 + 14 610 1219 0.001297235 + 15 987 1973 0.001960993 + 16 1597 3193 0.003417969 + 17 2584 5167 0.006215811 + 18 4181 8361 0.008294106 + 19 6765 13529 0.012095928 + 20 10946 21891 0.019592047 + 21 17711 35421 0.031193972 + 22 28657 57313 0.057254076 + 23 46368 92735 0.080307961 + 24 75025 150049 0.129482031 + 25 121393 242785 0.213325977 + 26 196418 392835 0.334423065 + 27 317811 635621 0.605969906 + 28 514229 1028457 0.971235037 + +Note that the time it takes to evaluate the function is blowing up exponentially as the input argument increases. Using ``@``-variables, which persist across records, we can cache and reuse the results of previous computations: + +.. code-block:: none + + mlr --ofmt '%.9lf' --opprint seqgen --start 1 --stop 28 then put ' + func f(n) { + @fcount += 1; # count number of calls to the function + if (is_present(@fcache[n])) { # cache hit + return @fcache[n] + } else { # cache miss + num rv = 1; + if (n >= 2) { + rv = f(n-1) + f(n-2) # recurse + } + @fcache[n] = rv; + return rv + } + } + @fcount = 0; + $o = f($i); + $fcount = @fcount; + ' then put '$seconds=systime()' then step -a delta -f seconds then cut -x -f seconds + +with output like this: + +.. code-block:: none + + i o fcount seconds_delta + 1 1 1 0 + 2 2 3 0.000053883 + 3 3 3 0.000035048 + 4 5 3 0.000045061 + 5 8 3 0.000014067 + 6 13 3 0.000028849 + 7 21 3 0.000028133 + 8 34 3 0.000027895 + 9 55 3 0.000014067 + 10 89 3 0.000015020 + 11 144 3 0.000012875 + 12 233 3 0.000033140 + 13 377 3 0.000014067 + 14 610 3 0.000012875 + 15 987 3 0.000029087 + 16 1597 3 0.000013828 + 17 2584 3 0.000013113 + 18 4181 3 0.000012875 + 19 6765 3 0.000013113 + 20 10946 3 0.000012875 + 21 17711 3 0.000013113 + 22 28657 3 0.000013113 + 23 46368 3 0.000015974 + 24 75025 3 0.000012875 + 25 121393 3 0.000013113 + 26 196418 3 0.000012875 + 27 317811 3 0.000013113 + 28 514229 3 0.000012875 diff --git a/docs6/misc-examples.rst.in b/docs6/misc-examples.rst.in new file mode 100644 index 000000000..f6d087643 --- /dev/null +++ b/docs6/misc-examples.rst.in @@ -0,0 +1,224 @@ +Miscellaneous examples +================================================================ + +Column select: + +GENRST_SHOW_COMMAND +mlr --csv cut -f hostname,uptime mydata.csv +GENRST_EOF + +Add new columns as function of other columns: + +GENRST_CARDIFY_HIGHLIGHT_ONE +mlr --nidx put '$sum = $7 < 0.0 ? 3.5 : $7 + 2.1*$8' *.dat +GENRST_EOF + +Row filter: + +GENRST_CARDIFY_HIGHLIGHT_ONE +mlr --csv filter '$status != "down" && $upsec >= 10000' *.csv +GENRST_EOF + +Apply column labels and pretty-print: + +GENRST_CARDIFY_HIGHLIGHT_ONE +grep -v '^#' /etc/group | mlr --ifs : --nidx --opprint label group,pass,gid,member then sort -f group +GENRST_EOF + +Join multiple data sources on key columns: + +GENRST_CARDIFY_HIGHLIGHT_ONE +mlr join -j account_id -f accounts.dat then group-by account_name balances.dat +GENRST_EOF + +Mulltiple formats including JSON: + +GENRST_CARDIFY_HIGHLIGHT_ONE +mlr --json put '$attr = sub($attr, "([0-9]+)_([0-9]+)_.*", "\1:\2")' data/*.json +GENRST_EOF + +Aggregate per-column statistics: + +GENRST_CARDIFY_HIGHLIGHT_ONE +mlr stats1 -a min,mean,max,p10,p50,p90 -f flag,u,v data/* +GENRST_EOF + +Linear regression: + +GENRST_CARDIFY_HIGHLIGHT_ONE +mlr stats2 -a linreg-pca -f u,v -g shape data/* +GENRST_EOF + +Aggregate custom per-column statistics: + +GENRST_CARDIFY_HIGHLIGHT_ONE +mlr put -q '@sum[$a][$b] += $x; end {emit @sum, "a", "b"}' data/* +GENRST_EOF + +Iterate over data using DSL expressions: + +GENRST_CARDIFY_HIGHLIGHT_ONE +mlr --from estimates.tbl put ' + for (k,v in $*) { + if (is_numeric(v) && k =~ "^[t-z].*$") { + $sum += v; $count += 1 + } + } + $mean = $sum / $count # no assignment if count unset +' +GENRST_EOF + +Run DSL expressions from a script file: + +GENRST_CARDIFY_HIGHLIGHT_ONE +mlr --from infile.dat put -f analyze.mlr +GENRST_EOF + +Split/reduce output to multiple filenames: + +GENRST_CARDIFY_HIGHLIGHT_ONE +mlr --from infile.dat put 'tee > "./taps/data-".$a."-".$b, $*' +GENRST_EOF + +Compressed I/O: + +GENRST_CARDIFY_HIGHLIGHT_ONE +mlr --from infile.dat put 'tee | "gzip > ./taps/data-".$a."-".$b.".gz", $*' +GENRST_EOF + +Interoperate with other data-processing tools using standard pipes: + +GENRST_CARDIFY_HIGHLIGHT_ONE +mlr --from infile.dat put -q '@v=$*; dump | "jq .[]"' +GENRST_EOF + +Tap/trace: + +GENRST_CARDIFY_HIGHLIGHT_ONE +mlr --from infile.dat put '(NR % 1000 == 0) { print > stderr, "Checkpoint ".NR}' +GENRST_EOF + +Program timing +---------------------------------------------------------------- + +This admittedly artificial example demonstrates using Miller time and stats functions to introspectively acquire some information about Miller's own runtime. The ``delta`` function computes the difference between successive timestamps. + +GENRST_INCLUDE_ESCAPED(data/timing-example.txt) + +Showing differences between successive queries +---------------------------------------------------------------- + +Suppose you have a database query which you run at one point in time, producing the output on the left, then again later producing the output on the right: + +GENRST_RUN_COMMAND +cat data/previous_counters.csv +GENRST_EOF + +GENRST_RUN_COMMAND +cat data/current_counters.csv +GENRST_EOF + +And, suppose you want to compute the differences in the counters between adjacent keys. Since the color names aren't all in the same order, nor are they all present on both sides, we can't just paste the two files side-by-side and do some column-four-minus-column-two arithmetic. + +First, rename counter columns to make them distinct: + +GENRST_RUN_COMMAND +mlr --csv rename count,previous_count data/previous_counters.csv > data/prevtemp.csv +GENRST_EOF + +GENRST_RUN_COMMAND +cat data/prevtemp.csv +GENRST_EOF + +GENRST_RUN_COMMAND +mlr --csv rename count,current_count data/current_counters.csv > data/currtemp.csv +GENRST_EOF + +GENRST_RUN_COMMAND +cat data/currtemp.csv +GENRST_EOF + +Then, join on the key field(s), and use unsparsify to zero-fill counters absent on one side but present on the other. Use ``--ul`` and ``--ur`` to emit unpaired records (namely, purple on the left and yellow on the right): + +GENRST_INCLUDE_AND_RUN_ESCAPED(data/previous-to-current.sh) + +.. _cookbook-memoization-with-oosvars: + +Memoization with out-of-stream variables +---------------------------------------------------------------- + +The recursive function for the Fibonacci sequence is famous for its computational complexity. Namely, using f(0)=1, f(1)=1, f(n)=f(n-1)+f(n-2) for n>=2, the evaluation tree branches left as well as right at each non-trivial level, resulting in millions or more paths to the root 0/1 nodes for larger n. This program + +GENRST_INCLUDE_ESCAPED(data/fibo-uncached.sh) + +produces output like this: + +GENRST_CARDIFY +i o fcount seconds_delta +1 1 1 0 +2 2 3 0.000039101 +3 3 5 0.000015974 +4 5 9 0.000019073 +5 8 15 0.000026941 +6 13 25 0.000036955 +7 21 41 0.000056028 +8 34 67 0.000086069 +9 55 109 0.000134945 +10 89 177 0.000217915 +11 144 287 0.000355959 +12 233 465 0.000506163 +13 377 753 0.000811815 +14 610 1219 0.001297235 +15 987 1973 0.001960993 +16 1597 3193 0.003417969 +17 2584 5167 0.006215811 +18 4181 8361 0.008294106 +19 6765 13529 0.012095928 +20 10946 21891 0.019592047 +21 17711 35421 0.031193972 +22 28657 57313 0.057254076 +23 46368 92735 0.080307961 +24 75025 150049 0.129482031 +25 121393 242785 0.213325977 +26 196418 392835 0.334423065 +27 317811 635621 0.605969906 +28 514229 1028457 0.971235037 +GENRST_EOF + +Note that the time it takes to evaluate the function is blowing up exponentially as the input argument increases. Using ``@``-variables, which persist across records, we can cache and reuse the results of previous computations: + +GENRST_INCLUDE_ESCAPED(data/fibo-cached.sh) + +with output like this: + +GENRST_CARDIFY +i o fcount seconds_delta +1 1 1 0 +2 2 3 0.000053883 +3 3 3 0.000035048 +4 5 3 0.000045061 +5 8 3 0.000014067 +6 13 3 0.000028849 +7 21 3 0.000028133 +8 34 3 0.000027895 +9 55 3 0.000014067 +10 89 3 0.000015020 +11 144 3 0.000012875 +12 233 3 0.000033140 +13 377 3 0.000014067 +14 610 3 0.000012875 +15 987 3 0.000029087 +16 1597 3 0.000013828 +17 2584 3 0.000013113 +18 4181 3 0.000012875 +19 6765 3 0.000013113 +20 10946 3 0.000012875 +21 17711 3 0.000013113 +22 28657 3 0.000013113 +23 46368 3 0.000015974 +24 75025 3 0.000012875 +25 121393 3 0.000013113 +26 196418 3 0.000012875 +27 317811 3 0.000013113 +28 514229 3 0.000012875 +GENRST_EOF diff --git a/docs6/programming-examples.rst b/docs6/programming-examples.rst new file mode 100644 index 000000000..f05421833 --- /dev/null +++ b/docs6/programming-examples.rst @@ -0,0 +1,261 @@ +.. + PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE. + +Programming-language examples +================================================================ + +Here are a few things focusing on Miller's DSL as a programming language per se, outside of its normal use for streaming record-processing. + +Sieve of Eratosthenes +---------------------------------------------------------------- + +The `Sieve of Eratosthenes `_ is a standard introductory programming topic. The idea is to find all primes up to some *N* by making a list of the numbers 1 to *N*, then striking out all multiples of 2 except 2 itself, all multiples of 3 except 3 itself, all multiples of 4 except 4 itself, and so on. Whatever survives that without getting marked is a prime. This is easy enough in Miller. Notice that here all the work is in ``begin`` and ``end`` statements; there is no file input (so we use ``mlr -n`` to keep Miller from waiting for input data). + +.. code-block:: none + :emphasize-lines: 1-1 + + cat programs/sieve.mlr + # ================================================================ + # Sieve of Eratosthenes: simple example of Miller DSL as programming language. + # ================================================================ + + # Put this in a begin-block so we can do either + # mlr -n put -q -f name-of-this-file.mlr + # or + # mlr -n put -q -f name-of-this-file.mlr -e '@n = 200' + # i.e. 100 is the default upper limit, and another can be specified using -e. + begin { + @n = 100; + } + + end { + for (int i = 0; i <= @n; i += 1) { + @s[i] = true; + } + @s[0] = false; # 0 is neither prime nor composite + @s[1] = false; # 1 is neither prime nor composite + # Strike out multiples + for (int i = 2; i <= @n; i += 1) { + for (int j = i+i; j <= @n; j += i) { + @s[j] = false; + } + } + # Print survivors + for (int i = 0; i <= @n; i += 1) { + if (@s[i]) { + print i; + } + } + } + +.. code-block:: none + :emphasize-lines: 1-1 + + mlr -n put -f programs/sieve.mlr + 2 + 3 + 5 + 7 + 11 + 13 + 17 + 19 + 23 + 29 + 31 + 37 + 41 + 43 + 47 + 53 + 59 + 61 + 67 + 71 + 73 + 79 + 83 + 89 + 97 + +Mandelbrot-set generator +---------------------------------------------------------------- + +The `Mandelbrot set `_ is also easily expressed. This isn't an important case of data-processing in the vein for which Miller was designed, but it is an example of Miller as a general-purpose programming language -- a test case for the expressiveness of the language. + +The (approximate) computation of points in the complex plane which are and aren't members is just a few lines of complex arithmetic (see the Wikipedia article); how to render them is another task. Using graphics libraries you can create PNG or JPEG files, but another fun way to do this is by printing various characters to the screen: + +.. code-block:: none + :emphasize-lines: 1-1 + + cat programs/mand.mlr + # Mandelbrot set generator: simple example of Miller DSL as programming language. + begin { + # Set defaults + @rcorn = -2.0; + @icorn = -2.0; + @side = 4.0; + @iheight = 50; + @iwidth = 100; + @maxits = 100; + @levelstep = 5; + @chars = "@X*o-."; # Palette of characters to print to the screen. + @verbose = false; + @do_julia = false; + @jr = 0.0; # Real part of Julia point, if any + @ji = 0.0; # Imaginary part of Julia point, if any + } + + # Here, we can override defaults from an input file (if any). In Miller's + # put/filter DSL, absent-null right-hand sides result in no assignment so we + # can simply put @rcorn = $rcorn: if there is a field in the input like + # 'rcorn = -1.847' we'll read and use it, else we'll keep the default. + @rcorn = $rcorn; + @icorn = $icorn; + @side = $side; + @iheight = $iheight; + @iwidth = $iwidth; + @maxits = $maxits; + @levelstep = $levelstep; + @chars = $chars; + @verbose = $verbose; + @do_julia = $do_julia; + @jr = $jr; + @ji = $ji; + + end { + if (@verbose) { + print "RCORN = ".@rcorn; + print "ICORN = ".@icorn; + print "SIDE = ".@side; + print "IHEIGHT = ".@iheight; + print "IWIDTH = ".@iwidth; + print "MAXITS = ".@maxits; + print "LEVELSTEP = ".@levelstep; + print "CHARS = ".@chars; + } + + # Iterate over a matrix of rows and columns, printing one character for each cell. + for (int ii = @iheight-1; ii >= 0; ii -= 1) { + num pi = @icorn + (ii/@iheight) * @side; + for (int ir = 0; ir < @iwidth; ir += 1) { + num pr = @rcorn + (ir/@iwidth) * @side; + printn get_point_plot(pr, pi, @maxits, @do_julia, @jr, @ji); + } + print; + } + } + + # This is a function to approximate membership in the Mandelbrot set (or Julia + # set for a given Julia point if do_julia == true) for a given point in the + # complex plane. + func get_point_plot(pr, pi, maxits, do_julia, jr, ji) { + num zr = 0.0; + num zi = 0.0; + num cr = 0.0; + num ci = 0.0; + + if (!do_julia) { + zr = 0.0; + zi = 0.0; + cr = pr; + ci = pi; + } else { + zr = pr; + zi = pi; + cr = jr; + ci = ji; + } + + int iti = 0; + bool escaped = false; + num zt = 0; + for (iti = 0; iti < maxits; iti += 1) { + num mag = zr*zr + zi+zi; + if (mag > 4.0) { + escaped = true; + break; + } + # z := z^2 + c + zt = zr*zr - zi*zi + cr; + zi = 2*zr*zi + ci; + zr = zt; + } + if (!escaped) { + return "."; + } else { + # The // operator is Miller's (pythonic) integer-division operator + int level = (iti // @levelstep) % strlen(@chars); + return substr(@chars, level, level); + } + } + +At standard resolution this makes a nice little ASCII plot: + +.. code-block:: none + :emphasize-lines: 1-1 + + mlr -n put -f ./programs/mand.mlr + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXX.XXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXooXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXX**o..*XXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXX*-....-oXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXX@XXXXXXXXXX*......o*XXXXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXXX**oo*-.-........oo.XXXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXXXXXX....................X..o-XXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@@@@@@@@@XXXXXXXXXXXXXXX*oo......................oXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@@@@@@@XXX*XXXXXXXXXXXX**o........................*X*X@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@@@@XXXXXXooo***o*.*XX**X..........................o-XX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@@XXXXXXXX*-.......-***.............................oXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@XXXXXXXX*@..........Xo............................*XX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@XXXX@XXXXXXXX*o@oX...........@...........................oXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + .........................................................o*XXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@XXXXXXXXX*-.oX...........@...........................oXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@XXXXXXXXXX**@..........*o............................*XXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@XXXXXXXXXXXXX-........***.............................oXXXXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@XXXXXXXXXXXXoo****o*.XX***@..........................o-XXXXXXXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@@@@@XXXXX*XXXX*XXXXXXX**-........................***XXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@@@@@@@@@@@XXXXXXXXXXXXX*o*.....................@o*XXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXXXXX*....................*..o-XX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXX*ooo*-.o........oo.X*XXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXX**@.....*XXXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXX*o....-o*XXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXXXo*o..*XXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXXXXXX*o*XXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXXXXX@XXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXX@@XXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ + +But using a very small font size (as small as my Mac will let me go), and by choosing the coordinates to zoom in on a particular part of the complex plane, we can get a nice little picture: + +.. code-block:: none + + #!/bin/bash + # Get the number of rows and columns from the terminal window dimensions + iheight=$(stty size | mlr --nidx --fs space cut -f 1) + iwidth=$(stty size | mlr --nidx --fs space cut -f 2) + echo "rcorn=-1.755350,icorn=+0.014230,side=0.000020,maxits=10000,iheight=$iheight,iwidth=$iwidth" \ + | mlr put -f programs/mand.mlr + +.. image:: pix/mand.png diff --git a/docs6/programming-examples.rst.in b/docs6/programming-examples.rst.in new file mode 100644 index 000000000..b990cf146 --- /dev/null +++ b/docs6/programming-examples.rst.in @@ -0,0 +1,47 @@ +Programming-language examples +================================================================ + +Here are a few things focusing on Miller's DSL as a programming language per se, outside of its normal use for streaming record-processing. + +Sieve of Eratosthenes +---------------------------------------------------------------- + +The `Sieve of Eratosthenes `_ is a standard introductory programming topic. The idea is to find all primes up to some *N* by making a list of the numbers 1 to *N*, then striking out all multiples of 2 except 2 itself, all multiples of 3 except 3 itself, all multiples of 4 except 4 itself, and so on. Whatever survives that without getting marked is a prime. This is easy enough in Miller. Notice that here all the work is in ``begin`` and ``end`` statements; there is no file input (so we use ``mlr -n`` to keep Miller from waiting for input data). + +GENRST_RUN_COMMAND +cat programs/sieve.mlr +GENRST_EOF + +GENRST_RUN_COMMAND +mlr -n put -f programs/sieve.mlr +GENRST_EOF + +Mandelbrot-set generator +---------------------------------------------------------------- + +The `Mandelbrot set `_ is also easily expressed. This isn't an important case of data-processing in the vein for which Miller was designed, but it is an example of Miller as a general-purpose programming language -- a test case for the expressiveness of the language. + +The (approximate) computation of points in the complex plane which are and aren't members is just a few lines of complex arithmetic (see the Wikipedia article); how to render them is another task. Using graphics libraries you can create PNG or JPEG files, but another fun way to do this is by printing various characters to the screen: + +GENRST_RUN_COMMAND +cat programs/mand.mlr +GENRST_EOF + +At standard resolution this makes a nice little ASCII plot: + +GENRST_RUN_COMMAND +mlr -n put -f ./programs/mand.mlr +GENRST_EOF + +But using a very small font size (as small as my Mac will let me go), and by choosing the coordinates to zoom in on a particular part of the complex plane, we can get a nice little picture: + +GENRST_CARDIFY +#!/bin/bash +# Get the number of rows and columns from the terminal window dimensions +iheight=$(stty size | mlr --nidx --fs space cut -f 1) +iwidth=$(stty size | mlr --nidx --fs space cut -f 2) +echo "rcorn=-1.755350,icorn=+0.014230,side=0.000020,maxits=10000,iheight=$iheight,iwidth=$iwidth" \ +| mlr put -f programs/mand.mlr +GENRST_EOF + +.. image:: pix/mand.png diff --git a/docs6/quick-examples.rst b/docs6/quick-examples.rst deleted file mode 100644 index 3c0857e19..000000000 --- a/docs6/quick-examples.rst +++ /dev/null @@ -1,117 +0,0 @@ -.. - PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE. - -Quick examples -================================================================ - -Column select: - -.. code-block:: none - :emphasize-lines: 1-1 - - mlr --csv cut -f hostname,uptime mydata.csv - -Add new columns as function of other columns: - -.. code-block:: none - :emphasize-lines: 1-1 - - mlr --nidx put '$sum = $7 < 0.0 ? 3.5 : $7 + 2.1*$8' *.dat - -Row filter: - -.. code-block:: none - :emphasize-lines: 1-1 - - mlr --csv filter '$status != "down" && $upsec >= 10000' *.csv - -Apply column labels and pretty-print: - -.. code-block:: none - :emphasize-lines: 1-1 - - grep -v '^#' /etc/group | mlr --ifs : --nidx --opprint label group,pass,gid,member then sort -f group - -Join multiple data sources on key columns: - -.. code-block:: none - :emphasize-lines: 1-1 - - mlr join -j account_id -f accounts.dat then group-by account_name balances.dat - -Mulltiple formats including JSON: - -.. code-block:: none - :emphasize-lines: 1-1 - - mlr --json put '$attr = sub($attr, "([0-9]+)_([0-9]+)_.*", "\1:\2")' data/*.json - -Aggregate per-column statistics: - -.. code-block:: none - :emphasize-lines: 1-1 - - mlr stats1 -a min,mean,max,p10,p50,p90 -f flag,u,v data/* - -Linear regression: - -.. code-block:: none - :emphasize-lines: 1-1 - - mlr stats2 -a linreg-pca -f u,v -g shape data/* - -Aggregate custom per-column statistics: - -.. code-block:: none - :emphasize-lines: 1-1 - - mlr put -q '@sum[$a][$b] += $x; end {emit @sum, "a", "b"}' data/* - -Iterate over data using DSL expressions: - -.. code-block:: none - :emphasize-lines: 1-1 - - mlr --from estimates.tbl put ' - for (k,v in $*) { - if (is_numeric(v) && k =~ "^[t-z].*$") { - $sum += v; $count += 1 - } - } - $mean = $sum / $count # no assignment if count unset - ' - -Run DSL expressions from a script file: - -.. code-block:: none - :emphasize-lines: 1-1 - - mlr --from infile.dat put -f analyze.mlr - -Split/reduce output to multiple filenames: - -.. code-block:: none - :emphasize-lines: 1-1 - - mlr --from infile.dat put 'tee > "./taps/data-".$a."-".$b, $*' - -Compressed I/O: - -.. code-block:: none - :emphasize-lines: 1-1 - - mlr --from infile.dat put 'tee | "gzip > ./taps/data-".$a."-".$b.".gz", $*' - -Interoperate with other data-processing tools using standard pipes: - -.. code-block:: none - :emphasize-lines: 1-1 - - mlr --from infile.dat put -q '@v=$*; dump | "jq .[]"' - -Tap/trace: - -.. code-block:: none - :emphasize-lines: 1-1 - - mlr --from infile.dat put '(NR % 1000 == 0) { print > stderr, "Checkpoint ".NR}' diff --git a/docs6/quick-examples.rst.in b/docs6/quick-examples.rst.in deleted file mode 100644 index c93b416fb..000000000 --- a/docs6/quick-examples.rst.in +++ /dev/null @@ -1,99 +0,0 @@ -Quick examples -================================================================ - -Column select: - -GENRST_SHOW_COMMAND -mlr --csv cut -f hostname,uptime mydata.csv -GENRST_EOF - -Add new columns as function of other columns: - -GENRST_CARDIFY_HIGHLIGHT_ONE -mlr --nidx put '$sum = $7 < 0.0 ? 3.5 : $7 + 2.1*$8' *.dat -GENRST_EOF - -Row filter: - -GENRST_CARDIFY_HIGHLIGHT_ONE -mlr --csv filter '$status != "down" && $upsec >= 10000' *.csv -GENRST_EOF - -Apply column labels and pretty-print: - -GENRST_CARDIFY_HIGHLIGHT_ONE -grep -v '^#' /etc/group | mlr --ifs : --nidx --opprint label group,pass,gid,member then sort -f group -GENRST_EOF - -Join multiple data sources on key columns: - -GENRST_CARDIFY_HIGHLIGHT_ONE -mlr join -j account_id -f accounts.dat then group-by account_name balances.dat -GENRST_EOF - -Mulltiple formats including JSON: - -GENRST_CARDIFY_HIGHLIGHT_ONE -mlr --json put '$attr = sub($attr, "([0-9]+)_([0-9]+)_.*", "\1:\2")' data/*.json -GENRST_EOF - -Aggregate per-column statistics: - -GENRST_CARDIFY_HIGHLIGHT_ONE -mlr stats1 -a min,mean,max,p10,p50,p90 -f flag,u,v data/* -GENRST_EOF - -Linear regression: - -GENRST_CARDIFY_HIGHLIGHT_ONE -mlr stats2 -a linreg-pca -f u,v -g shape data/* -GENRST_EOF - -Aggregate custom per-column statistics: - -GENRST_CARDIFY_HIGHLIGHT_ONE -mlr put -q '@sum[$a][$b] += $x; end {emit @sum, "a", "b"}' data/* -GENRST_EOF - -Iterate over data using DSL expressions: - -GENRST_CARDIFY_HIGHLIGHT_ONE -mlr --from estimates.tbl put ' - for (k,v in $*) { - if (is_numeric(v) && k =~ "^[t-z].*$") { - $sum += v; $count += 1 - } - } - $mean = $sum / $count # no assignment if count unset -' -GENRST_EOF - -Run DSL expressions from a script file: - -GENRST_CARDIFY_HIGHLIGHT_ONE -mlr --from infile.dat put -f analyze.mlr -GENRST_EOF - -Split/reduce output to multiple filenames: - -GENRST_CARDIFY_HIGHLIGHT_ONE -mlr --from infile.dat put 'tee > "./taps/data-".$a."-".$b, $*' -GENRST_EOF - -Compressed I/O: - -GENRST_CARDIFY_HIGHLIGHT_ONE -mlr --from infile.dat put 'tee | "gzip > ./taps/data-".$a."-".$b.".gz", $*' -GENRST_EOF - -Interoperate with other data-processing tools using standard pipes: - -GENRST_CARDIFY_HIGHLIGHT_ONE -mlr --from infile.dat put -q '@v=$*; dump | "jq .[]"' -GENRST_EOF - -Tap/trace: - -GENRST_CARDIFY_HIGHLIGHT_ONE -mlr --from infile.dat put '(NR % 1000 == 0) { print > stderr, "Checkpoint ".NR}' -GENRST_EOF diff --git a/docs6/randomizing-examples.rst b/docs6/randomizing-examples.rst new file mode 100644 index 000000000..7a056738a --- /dev/null +++ b/docs6/randomizing-examples.rst @@ -0,0 +1,174 @@ +.. + PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE. + +Randomizing examples +================================================================ + +Generating random numbers from various distributions +---------------------------------------------------------------- + +Here we can chain together a few simple building blocks: + +.. code-block:: none + :emphasize-lines: 1-1 + + cat expo-sample.sh + # Generate 100,000 pairs of independent and identically distributed + # exponentially distributed random variables with the same rate parameter + # (namely, 2.5). Then compute histograms of one of them, along with + # histograms for their sum and their product. + # + # See also https://en.wikipedia.org/wiki/Exponential_distribution + # + # Here I'm using a specified random-number seed so this example always + # produces the same output for this web document: in everyday practice we + # wouldn't do that. + + mlr -n \ + --seed 0 \ + --opprint \ + seqgen --stop 100000 \ + then put ' + # https://en.wikipedia.org/wiki/Inverse_transform_sampling + func expo_sample(lambda) { + return -log(1-urand())/lambda + } + $u = expo_sample(2.5); + $v = expo_sample(2.5); + $s = $u + $v; + $p = $u * $v; + ' \ + then histogram -f u,s,p --lo 0 --hi 2 --nbins 50 \ + then bar -f u_count,s_count,p_count --auto -w 20 + +Namely: + +* Set the Miller random-number seed so this webdoc looks the same every time I regenerate it. +* Use pretty-printed tabular output. +* Use pretty-printed tabular output. +* Use ``seqgen`` to produce 100,000 records ``i=0``, ``i=1``, etc. +* Send those to a ``put`` step which defines an inverse-transform-sampling function and calls it twice, then computes the sum and product of samples. +* Send those to a histogram, and from there to a bar-plotter. This is just for visualization; you could just as well output CSV and send that off to your own plotting tool, etc. + +The output is as follows: + +.. code-block:: none + :emphasize-lines: 1-1 + + sh expo-sample.sh + bin_lo bin_hi u_count s_count p_count + 0 0.04 [64]*******************#[9554] [326]#...................[3703] [19]*******************#[39809] + 0.04 0.08 [64]*****************...[9554] [326]*****...............[3703] [19]*******.............[39809] + 0.08 0.12 [64]****************....[9554] [326]*********...........[3703] [19]****................[39809] + 0.12 0.16 [64]**************......[9554] [326]************........[3703] [19]***.................[39809] + 0.16 0.2 [64]*************.......[9554] [326]**************......[3703] [19]**..................[39809] + 0.2 0.24 [64]************........[9554] [326]*****************...[3703] [19]*...................[39809] + 0.24 0.28 [64]**********..........[9554] [326]******************..[3703] [19]*...................[39809] + 0.28 0.32 [64]*********...........[9554] [326]******************..[3703] [19]*...................[39809] + 0.32 0.36 [64]********............[9554] [326]*******************.[3703] [19]#...................[39809] + 0.36 0.4 [64]*******.............[9554] [326]*******************#[3703] [19]#...................[39809] + 0.4 0.44 [64]*******.............[9554] [326]*******************.[3703] [19]#...................[39809] + 0.44 0.48 [64]******..............[9554] [326]*******************.[3703] [19]#...................[39809] + 0.48 0.52 [64]*****...............[9554] [326]******************..[3703] [19]#...................[39809] + 0.52 0.56 [64]*****...............[9554] [326]******************..[3703] [19]#...................[39809] + 0.56 0.6 [64]****................[9554] [326]*****************...[3703] [19]#...................[39809] + 0.6 0.64 [64]****................[9554] [326]******************..[3703] [19]#...................[39809] + 0.64 0.68 [64]***.................[9554] [326]****************....[3703] [19]#...................[39809] + 0.68 0.72 [64]***.................[9554] [326]****************....[3703] [19]#...................[39809] + 0.72 0.76 [64]***.................[9554] [326]***************.....[3703] [19]#...................[39809] + 0.76 0.8 [64]**..................[9554] [326]**************......[3703] [19]#...................[39809] + 0.8 0.84 [64]**..................[9554] [326]*************.......[3703] [19]#...................[39809] + 0.84 0.88 [64]**..................[9554] [326]************........[3703] [19]#...................[39809] + 0.88 0.92 [64]**..................[9554] [326]************........[3703] [19]#...................[39809] + 0.92 0.96 [64]*...................[9554] [326]***********.........[3703] [19]#...................[39809] + 0.96 1 [64]*...................[9554] [326]**********..........[3703] [19]#...................[39809] + 1 1.04 [64]*...................[9554] [326]*********...........[3703] [19]#...................[39809] + 1.04 1.08 [64]*...................[9554] [326]********............[3703] [19]#...................[39809] + 1.08 1.12 [64]*...................[9554] [326]********............[3703] [19]#...................[39809] + 1.12 1.16 [64]*...................[9554] [326]********............[3703] [19]#...................[39809] + 1.16 1.2 [64]*...................[9554] [326]*******.............[3703] [19]#...................[39809] + 1.2 1.24 [64]#...................[9554] [326]******..............[3703] [19]#...................[39809] + 1.24 1.28 [64]#...................[9554] [326]*****...............[3703] [19]#...................[39809] + 1.28 1.32 [64]#...................[9554] [326]*****...............[3703] [19]#...................[39809] + 1.32 1.36 [64]#...................[9554] [326]****................[3703] [19]#...................[39809] + 1.36 1.4 [64]#...................[9554] [326]****................[3703] [19]#...................[39809] + 1.4 1.44 [64]#...................[9554] [326]****................[3703] [19]#...................[39809] + 1.44 1.48 [64]#...................[9554] [326]***.................[3703] [19]#...................[39809] + 1.48 1.52 [64]#...................[9554] [326]***.................[3703] [19]#...................[39809] + 1.52 1.56 [64]#...................[9554] [326]***.................[3703] [19]#...................[39809] + 1.56 1.6 [64]#...................[9554] [326]**..................[3703] [19]#...................[39809] + 1.6 1.64 [64]#...................[9554] [326]**..................[3703] [19]#...................[39809] + 1.64 1.68 [64]#...................[9554] [326]**..................[3703] [19]#...................[39809] + 1.68 1.72 [64]#...................[9554] [326]*...................[3703] [19]#...................[39809] + 1.72 1.76 [64]#...................[9554] [326]*...................[3703] [19]#...................[39809] + 1.76 1.8 [64]#...................[9554] [326]*...................[3703] [19]#...................[39809] + 1.8 1.84 [64]#...................[9554] [326]#...................[3703] [19]#...................[39809] + 1.84 1.88 [64]#...................[9554] [326]#...................[3703] [19]#...................[39809] + 1.88 1.92 [64]#...................[9554] [326]#...................[3703] [19]#...................[39809] + 1.92 1.96 [64]#...................[9554] [326]#...................[3703] [19]#...................[39809] + 1.96 2 [64]#...................[9554] [326]#...................[3703] [19]#...................[39809] + +Randomly selecting words from a list +---------------------------------------------------------------- + +Given this `word list <./data/english-words.txt>`_, first take a look to see what the first few lines look like: + +.. code-block:: none + :emphasize-lines: 1-1 + + head data/english-words.txt + a + aa + aal + aalii + aam + aardvark + aardwolf + aba + abac + abaca + +Then the following will randomly sample ten words with four to eight characters in them: + +.. code-block:: none + :emphasize-lines: 1-1 + + mlr --from data/english-words.txt --nidx filter -S 'n=strlen($1);4<=n&&n<=8' then sample -k 10 + thionine + birchman + mildewy + avigate + addedly + abaze + askant + aiming + insulant + coinmate + +Randomly generating jabberwocky words +---------------------------------------------------------------- + +These are simple *n*-grams as `described here `_. Some common functions are `located here `_. Then here are scripts for `1-grams `_ `2-grams `_ `3-grams `_ `4-grams `_, and `5-grams `_. + +The idea is that words from the input file are consumed, then taken apart and pasted back together in ways which imitate the letter-to-letter transitions found in the word list -- giving us automatically generated words in the same vein as *bromance* and *spork*: + +.. code-block:: none + :emphasize-lines: 1-1 + + mlr --nidx --from ./ngrams/gsl-2000.txt put -q -f ./ngrams/ngfuncs.mlr -f ./ngrams/ng5.mlr + beard + plastinguish + politicially + noise + loan + country + controductionary + suppery + lose + lessors + dollar + judge + rottendence + lessenger + diffendant + suggestional diff --git a/docs6/randomizing-examples.rst.in b/docs6/randomizing-examples.rst.in new file mode 100644 index 000000000..c275be5a9 --- /dev/null +++ b/docs6/randomizing-examples.rst.in @@ -0,0 +1,88 @@ +Randomizing examples +================================================================ + +Generating random numbers from various distributions +---------------------------------------------------------------- + +Here we can chain together a few simple building blocks: + +GENRST_RUN_COMMAND +cat expo-sample.sh +GENRST_EOF + +Namely: + +* Set the Miller random-number seed so this webdoc looks the same every time I regenerate it. +* Use pretty-printed tabular output. +* Use pretty-printed tabular output. +* Use ``seqgen`` to produce 100,000 records ``i=0``, ``i=1``, etc. +* Send those to a ``put`` step which defines an inverse-transform-sampling function and calls it twice, then computes the sum and product of samples. +* Send those to a histogram, and from there to a bar-plotter. This is just for visualization; you could just as well output CSV and send that off to your own plotting tool, etc. + +The output is as follows: + +GENRST_RUN_COMMAND +sh expo-sample.sh +GENRST_EOF + +Randomly selecting words from a list +---------------------------------------------------------------- + +Given this `word list <./data/english-words.txt>`_, first take a look to see what the first few lines look like: + +GENRST_CARDIFY_HIGHLIGHT_ONE +head data/english-words.txt +a +aa +aal +aalii +aam +aardvark +aardwolf +aba +abac +abaca +GENRST_EOF + +Then the following will randomly sample ten words with four to eight characters in them: + +GENRST_CARDIFY_HIGHLIGHT_ONE +mlr --from data/english-words.txt --nidx filter -S 'n=strlen($1);4<=n&&n<=8' then sample -k 10 +thionine +birchman +mildewy +avigate +addedly +abaze +askant +aiming +insulant +coinmate +GENRST_EOF + +Randomly generating jabberwocky words +---------------------------------------------------------------- + +These are simple *n*-grams as `described here `_. Some common functions are `located here `_. Then here are scripts for `1-grams `_ `2-grams `_ `3-grams `_ `4-grams `_, and `5-grams `_. + +The idea is that words from the input file are consumed, then taken apart and pasted back together in ways which imitate the letter-to-letter transitions found in the word list -- giving us automatically generated words in the same vein as *bromance* and *spork*: + +GENRST_CARDIFY_HIGHLIGHT_ONE +mlr --nidx --from ./ngrams/gsl-2000.txt put -q -f ./ngrams/ngfuncs.mlr -f ./ngrams/ng5.mlr +beard +plastinguish +politicially +noise +loan +country +controductionary +suppery +lose +lessors +dollar +judge +rottendence +lessenger +diffendant +suggestional +GENRST_EOF diff --git a/docs6/reference-dsl-variables.rst b/docs6/reference-dsl-variables.rst index be5c8e36e..1903103dd 100644 --- a/docs6/reference-dsl-variables.rst +++ b/docs6/reference-dsl-variables.rst @@ -656,7 +656,7 @@ The following ``is...`` functions take a value and return a boolean indicating w asserting_present asserting_string -Please see :ref:`cookbook-data-cleaning-examples` for examples of how to use these. +See :doc:`data-cleaning-examples` for examples of how to use these. Type-declarations for local variables, function parameter, and function return values ............................................................................................... diff --git a/docs6/reference-dsl-variables.rst.in b/docs6/reference-dsl-variables.rst.in index b9ca5a5b3..fefacbe7b 100644 --- a/docs6/reference-dsl-variables.rst.in +++ b/docs6/reference-dsl-variables.rst.in @@ -262,7 +262,7 @@ GENRST_RUN_COMMAND mlr -f | grep ^assert GENRST_EOF -Please see :ref:`cookbook-data-cleaning-examples` for examples of how to use these. +See :doc:`data-cleaning-examples` for examples of how to use these. Type-declarations for local variables, function parameter, and function return values ............................................................................................... diff --git a/docs6/shapes-of-data.rst b/docs6/shapes-of-data.rst index cdaed6387..fe4fe60a1 100644 --- a/docs6/shapes-of-data.rst +++ b/docs6/shapes-of-data.rst @@ -21,36 +21,48 @@ Also try ``od -xcv`` and/or ``cat -e`` on your file to check for non-printable c Diagnosing delimiter specifications ---------------------------------------------------------------- -.. code-block:: none +Use the ``file`` command to see if there are CR/LF terminators (in this case, # there are not): - # Use the `file` command to see if there are CR/LF terminators (in this case, - # there are not): - $ file data/colours.csv +.. code-block:: none + :emphasize-lines: 1-1 + + file data/colours.csv data/colours.csv: UTF-8 Unicode text - - # Look at the file to find names of fields - $ cat data/colours.csv + +Look at the file to find names of fields + +.. code-block:: none + :emphasize-lines: 1-1 + + cat data/colours.csv KEY;DE;EN;ES;FI;FR;IT;NL;PL;RO;TR masterdata_colourcode_1;Weiß;White;Blanco;Valkoinen;Blanc;Bianco;Wit;Biały;Alb;Beyaz masterdata_colourcode_2;Schwarz;Black;Negro;Musta;Noir;Nero;Zwart;Czarny;Negru;Siyah - - # Extract a few fields: - $ mlr --csv cut -f KEY,PL,RO data/colours.csv + +Extract a few fields: + +.. code-block:: none + :emphasize-lines: 1-1 + + mlr --csv cut -f KEY,PL,RO data/colours.csv (only blank lines appear) - - # Use XTAB output format to get a sharper picture of where records/fields - # are being split: - $ mlr --icsv --oxtab cat data/colours.csv + +Use XTAB output format to get a sharper picture of where records/fields are being split: + +.. code-block:: none + :emphasize-lines: 1-1 + + mlr --icsv --oxtab cat data/colours.csv KEY;DE;EN;ES;FI;FR;IT;NL;PL;RO;TR masterdata_colourcode_1;Weiß;White;Blanco;Valkoinen;Blanc;Bianco;Wit;Biały;Alb;Beyaz KEY;DE;EN;ES;FI;FR;IT;NL;PL;RO;TR masterdata_colourcode_2;Schwarz;Black;Negro;Musta;Noir;Nero;Zwart;Czarny;Negru;Siyah - - # Using XTAB output format makes it clearer that KEY;DE;...;RO;TR is being - # treated as a single field name in the CSV header, and likewise each - # subsequent line is being treated as a single field value. This is because - # the default field separator is a comma but we have semicolons here. - # Use XTAB again with different field separator (--fs semicolon): - mlr --icsv --ifs semicolon --oxtab cat data/colours.csv + +Using XTAB output format makes it clearer that ``KEY;DE;...;RO;TR`` is being treated as a single field name in the CSV header, and likewise each subsequent line is being treated as a single field value. This is because the default field separator is a comma but we have semicolons here. Use XTAB again with different field separator (``--fs semicolon``): + +.. code-block:: none + :emphasize-lines: 1-1 + + mlr --icsv --ifs semicolon --oxtab cat data/colours.csv KEY masterdata_colourcode_1 DE Weiß EN White @@ -74,9 +86,13 @@ Diagnosing delimiter specifications PL Czarny RO Negru TR Siyah - - # Using the new field-separator, retry the cut: - mlr --csv --fs semicolon cut -f KEY,PL,RO data/colours.csv + +Using the new field-separator, retry the cut: + +.. code-block:: none + :emphasize-lines: 1-1 + + mlr --csv --fs semicolon cut -f KEY,PL,RO data/colours.csv KEY;PL;RO masterdata_colourcode_1;Biały;Alb masterdata_colourcode_2;Czarny;Negru @@ -84,7 +100,7 @@ Diagnosing delimiter specifications I assigned $9 and it's not 9th ---------------------------------------------------------------- -Miller records are ordered lists of key-value pairs. For NIDX format, DKVP format when keys are missing, or CSV/CSV-lite format with ``--implicit-csv-header``, Miller will sequentially assign keys of the form ``1``, ``2``, etc. But these are not integer array indices: they're just field names taken from the initial field ordering in the input data. +Miller records are ordered lists of key-value pairs. For NIDX format, DKVP format when keys are missing, or CSV/CSV-lite format with ``--implicit-csv-header``, Miller will sequentially assign keys of the form ``1``, ``2``, etc. But these are not integer array indices: they're just field names taken from the initial field ordering in the input data, when it is originally read from the input file(s). .. code-block:: none :emphasize-lines: 1-1 diff --git a/docs6/shapes-of-data.rst.in b/docs6/shapes-of-data.rst.in index a667ae93b..9ff56607e 100644 --- a/docs6/shapes-of-data.rst.in +++ b/docs6/shapes-of-data.rst.in @@ -18,12 +18,80 @@ Also try ``od -xcv`` and/or ``cat -e`` on your file to check for non-printable c Diagnosing delimiter specifications ---------------------------------------------------------------- -GENRST_INCLUDE_ESCAPED(data/delimiter-examples.txt) +Use the ``file`` command to see if there are CR/LF terminators (in this case, # there are not): + +GENRST_CARDIFY_HIGHLIGHT_ONE +file data/colours.csv +data/colours.csv: UTF-8 Unicode text +GENRST_EOF + +Look at the file to find names of fields + +GENRST_CARDIFY_HIGHLIGHT_ONE +cat data/colours.csv +KEY;DE;EN;ES;FI;FR;IT;NL;PL;RO;TR +masterdata_colourcode_1;Weiß;White;Blanco;Valkoinen;Blanc;Bianco;Wit;Biały;Alb;Beyaz +masterdata_colourcode_2;Schwarz;Black;Negro;Musta;Noir;Nero;Zwart;Czarny;Negru;Siyah +GENRST_EOF + +Extract a few fields: + +GENRST_CARDIFY_HIGHLIGHT_ONE +mlr --csv cut -f KEY,PL,RO data/colours.csv +(only blank lines appear) +GENRST_EOF + +Use XTAB output format to get a sharper picture of where records/fields are being split: + +GENRST_CARDIFY_HIGHLIGHT_ONE +mlr --icsv --oxtab cat data/colours.csv +KEY;DE;EN;ES;FI;FR;IT;NL;PL;RO;TR masterdata_colourcode_1;Weiß;White;Blanco;Valkoinen;Blanc;Bianco;Wit;Biały;Alb;Beyaz + +KEY;DE;EN;ES;FI;FR;IT;NL;PL;RO;TR masterdata_colourcode_2;Schwarz;Black;Negro;Musta;Noir;Nero;Zwart;Czarny;Negru;Siyah +GENRST_EOF + +Using XTAB output format makes it clearer that ``KEY;DE;...;RO;TR`` is being treated as a single field name in the CSV header, and likewise each subsequent line is being treated as a single field value. This is because the default field separator is a comma but we have semicolons here. Use XTAB again with different field separator (``--fs semicolon``): + +GENRST_CARDIFY_HIGHLIGHT_ONE +mlr --icsv --ifs semicolon --oxtab cat data/colours.csv +KEY masterdata_colourcode_1 +DE Weiß +EN White +ES Blanco +FI Valkoinen +FR Blanc +IT Bianco +NL Wit +PL Biały +RO Alb +TR Beyaz + +KEY masterdata_colourcode_2 +DE Schwarz +EN Black +ES Negro +FI Musta +FR Noir +IT Nero +NL Zwart +PL Czarny +RO Negru +TR Siyah +GENRST_EOF + +Using the new field-separator, retry the cut: + +GENRST_CARDIFY_HIGHLIGHT_ONE +mlr --csv --fs semicolon cut -f KEY,PL,RO data/colours.csv +KEY;PL;RO +masterdata_colourcode_1;Biały;Alb +masterdata_colourcode_2;Czarny;Negru +GENRST_EOF I assigned $9 and it's not 9th ---------------------------------------------------------------- -Miller records are ordered lists of key-value pairs. For NIDX format, DKVP format when keys are missing, or CSV/CSV-lite format with ``--implicit-csv-header``, Miller will sequentially assign keys of the form ``1``, ``2``, etc. But these are not integer array indices: they're just field names taken from the initial field ordering in the input data. +Miller records are ordered lists of key-value pairs. For NIDX format, DKVP format when keys are missing, or CSV/CSV-lite format with ``--implicit-csv-header``, Miller will sequentially assign keys of the form ``1``, ``2``, etc. But these are not integer array indices: they're just field names taken from the initial field ordering in the input data, when it is originally read from the input file(s). GENRST_RUN_COMMAND echo x,y,z | mlr --dkvp cat diff --git a/docs6/special-symbols-and-formatting.rst b/docs6/special-symbols-and-formatting.rst index 8cbb28433..0bddbd3b5 100644 --- a/docs6/special-symbols-and-formatting.rst +++ b/docs6/special-symbols-and-formatting.rst @@ -144,104 +144,3 @@ One way is to use square brackets; an alternative is to use simple string-substi c is it ... The ``ssub`` function exists precisely for this reason: so you don't have to escape anything. - -How do I suppress numeric conversion? ----------------------------------------------------------------- - -**TL;DR use put -S**. - -**TODO: probably remove this for Miller 6** - -Within ``mlr put`` and ``mlr filter``, the default behavior for scanning input records is to parse them as integer, if possible, then as float, if possible, else leave them as string: - -.. code-block:: none - :emphasize-lines: 1-1 - - cat data/scan-example-1.tbl - value - 1 - 2.0 - 3x - hello - -.. code-block:: none - :emphasize-lines: 1-1 - - mlr --pprint put '$copy = $value; $type = typeof($value)' data/scan-example-1.tbl - value copy type - 1 1 int - 2.0 2.0 float - 3x 3x string - hello hello string - -The numeric-conversion rule is simple: - -* Try to scan as integer (``"1"`` should be int); -* If that doesn't succeed, try to scan as float (``"1.0"`` should be float); -* If that doesn't succeed, leave the value as a string (``"1x"`` is string). - -This is a sensible default: you should be able to put ``'$z = $x + $y'`` without having to write ``'$z = int($x) + float($y)'``. Also note that default output format for floating-point numbers created by ``put`` (and other verbs such as ``stats1``) is six decimal places; you can override this using ``mlr --ofmt``. Also note that Miller uses your system's Go library functions whenever possible: e.g. ``sscanf`` for converting strings to integer or floating-point. - -But now suppose you have data like these: - -.. code-block:: none - :emphasize-lines: 1-1 - - cat data/scan-example-2.tbl - value - 0001 - 0002 - 0005 - 0005WA - 0006 - 0007 - 0007WA - 0008 - 0009 - 0010 - -.. code-block:: none - :emphasize-lines: 1-1 - - mlr --pprint put '$copy = $value; $type = typeof($value)' data/scan-example-2.tbl - value copy type - 0001 0001 int - 0002 0002 int - 0005 0005 int - 0005WA 0005WA string - 0006 0006 int - 0007 0007 int - 0007WA 0007WA string - 0008 0008 float - 0009 0009 float - 0010 0010 int - -The same conversion rules as above are being used. Namely: - -* By default field values are inferred to int, else float, else string; - -* leading zeroes indicate octal for integers (``sscanf`` semantics); - -* since ``0008`` doesn't scan as integer (leading 0 requests octal but 8 isn't a valid octal digit), the float scan is tried next and it succeeds; - -* default floating-point output format is 6 decimal places (override with ``mlr --ofmt``). - -Taken individually the rules make sense; taken collectively they produce a mishmash of types here. - -The solution is to **use the -S flag** for ``mlr put`` and/or ``mlr filter``. Then all field values are left as string. You can type-coerce on demand using syntax like ``'$z = int($x) + float($y)'``. (See also :doc:`reference-dsl`; see also https://github.com/johnkerl/miller/issues/150.) - -.. code-block:: none - :emphasize-lines: 1-1 - - mlr --pprint put -S '$copy = $value; $type = typeof($value)' data/scan-example-2.tbl - value copy type - 0001 0001 int - 0002 0002 int - 0005 0005 int - 0005WA 0005WA string - 0006 0006 int - 0007 0007 int - 0007WA 0007WA string - 0008 0008 float - 0009 0009 float - 0010 0010 int diff --git a/docs6/special-symbols-and-formatting.rst.in b/docs6/special-symbols-and-formatting.rst.in index 315ec033f..f18a2fe16 100644 --- a/docs6/special-symbols-and-formatting.rst.in +++ b/docs6/special-symbols-and-formatting.rst.in @@ -95,56 +95,3 @@ mlr --oxtab put '$c = ssub($a, "?"," ...")' data/question.dat GENRST_EOF The ``ssub`` function exists precisely for this reason: so you don't have to escape anything. - -How do I suppress numeric conversion? ----------------------------------------------------------------- - -**TL;DR use put -S**. - -**TODO: probably remove this for Miller 6** - -Within ``mlr put`` and ``mlr filter``, the default behavior for scanning input records is to parse them as integer, if possible, then as float, if possible, else leave them as string: - -GENRST_RUN_COMMAND -cat data/scan-example-1.tbl -GENRST_EOF - -GENRST_RUN_COMMAND -mlr --pprint put '$copy = $value; $type = typeof($value)' data/scan-example-1.tbl -GENRST_EOF - -The numeric-conversion rule is simple: - -* Try to scan as integer (``"1"`` should be int); -* If that doesn't succeed, try to scan as float (``"1.0"`` should be float); -* If that doesn't succeed, leave the value as a string (``"1x"`` is string). - -This is a sensible default: you should be able to put ``'$z = $x + $y'`` without having to write ``'$z = int($x) + float($y)'``. Also note that default output format for floating-point numbers created by ``put`` (and other verbs such as ``stats1``) is six decimal places; you can override this using ``mlr --ofmt``. Also note that Miller uses your system's Go library functions whenever possible: e.g. ``sscanf`` for converting strings to integer or floating-point. - -But now suppose you have data like these: - -GENRST_RUN_COMMAND -cat data/scan-example-2.tbl -GENRST_EOF - -GENRST_RUN_COMMAND -mlr --pprint put '$copy = $value; $type = typeof($value)' data/scan-example-2.tbl -GENRST_EOF - -The same conversion rules as above are being used. Namely: - -* By default field values are inferred to int, else float, else string; - -* leading zeroes indicate octal for integers (``sscanf`` semantics); - -* since ``0008`` doesn't scan as integer (leading 0 requests octal but 8 isn't a valid octal digit), the float scan is tried next and it succeeds; - -* default floating-point output format is 6 decimal places (override with ``mlr --ofmt``). - -Taken individually the rules make sense; taken collectively they produce a mishmash of types here. - -The solution is to **use the -S flag** for ``mlr put`` and/or ``mlr filter``. Then all field values are left as string. You can type-coerce on demand using syntax like ``'$z = int($x) + float($y)'``. (See also :doc:`reference-dsl`; see also https://github.com/johnkerl/miller/issues/150.) - -GENRST_RUN_COMMAND -mlr --pprint put -S '$copy = $value; $type = typeof($value)' data/scan-example-2.tbl -GENRST_EOF diff --git a/docs6/statistics-examples.rst b/docs6/statistics-examples.rst new file mode 100644 index 000000000..eab281d39 --- /dev/null +++ b/docs6/statistics-examples.rst @@ -0,0 +1,71 @@ +.. + PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE. + +Statistics examples +==================== + +Computing interquartile ranges +---------------------------------------------------------------- + +For one or more specified field names, simply compute p25 and p75, then write the IQR as the difference of p75 and p25: + +.. code-block:: none + :emphasize-lines: 1-3 + + mlr --oxtab stats1 -f x -a p25,p75 \ + then put '$x_iqr = $x_p75 - $x_p25' \ + data/medium + x_p25 0.24667037823231752 + x_p75 0.7481860062358446 + x_iqr 0.5015156280035271 + +For wildcarded field names, first compute p25 and p75, then loop over field names with ``p25`` in them: + +.. code-block:: none + :emphasize-lines: 1-7 + + mlr --oxtab stats1 --fr '[i-z]' -a p25,p75 \ + then put 'for (k,v in $*) { + if (k =~ "(.*)_p25") { + $["\1_iqr"] = $["\1_p75"] - $["\1_p25"] + } + }' \ + data/medium + +Computing weighted means +---------------------------------------------------------------- + +This might be more elegantly implemented as an option within the ``stats1`` verb. Meanwhile, it's expressible within the DSL: + +.. code-block:: none + :emphasize-lines: 1-24 + + mlr --from data/medium put -q ' + # Using the y field for weighting in this example + weight = $y; + + # Using the a field for weighted aggregation in this example + @sumwx[$a] += weight * $i; + @sumw[$a] += weight; + + @sumx[$a] += $i; + @sumn[$a] += 1; + + end { + map wmean = {}; + map mean = {}; + for (a in @sumwx) { + wmean[a] = @sumwx[a] / @sumw[a] + } + for (a in @sumx) { + mean[a] = @sumx[a] / @sumn[a] + } + #emit wmean, "a"; + #emit mean, "a"; + emit (wmean, mean), "a"; + }' + a=pan,wmean=4979.563722208067,mean=5028.259010091302 + a=eks,wmean=4890.3815931472145,mean=4956.2900763358775 + a=wye,wmean=4946.987746229947,mean=4920.001017293998 + a=zee,wmean=5164.719684856538,mean=5123.092330239375 + a=hat,wmean=4925.533162478552,mean=4967.743946419371 diff --git a/docs6/statistics-examples.rst.in b/docs6/statistics-examples.rst.in new file mode 100644 index 000000000..877e01b14 --- /dev/null +++ b/docs6/statistics-examples.rst.in @@ -0,0 +1,20 @@ +Statistics examples +==================== + +Computing interquartile ranges +---------------------------------------------------------------- + +For one or more specified field names, simply compute p25 and p75, then write the IQR as the difference of p75 and p25: + +GENRST_INCLUDE_AND_RUN_ESCAPED(data/iqr1.sh) + +For wildcarded field names, first compute p25 and p75, then loop over field names with ``p25`` in them: + +GENRST_INCLUDE_AND_RUN_ESCAPED(data/iqrn.sh) + +Computing weighted means +---------------------------------------------------------------- + +This might be more elegantly implemented as an option within the ``stats1`` verb. Meanwhile, it's expressible within the DSL: + +GENRST_INCLUDE_AND_RUN_ESCAPED(data/weighted-mean.sh) diff --git a/docs6/cookbook.rst b/docs6/two-pass-algorithms.rst similarity index 53% rename from docs6/cookbook.rst rename to docs6/two-pass-algorithms.rst index cd6d83463..67f41e1f2 100644 --- a/docs6/cookbook.rst +++ b/docs6/two-pass-algorithms.rst @@ -1,167 +1,86 @@ .. PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE. -Cookbook part 1: common patterns +Two-pass algorithms ================================================================ -Data-cleaning examples ----------------------------------------------------------------- - -Here are some ways to use the type-checking options as described in :ref:`reference-dsl-type-tests-and-assertions` Suppose you have the following data file, with inconsistent typing for boolean. (Also imagine that, for the sake of discussion, we have a million-line file rather than a four-line file, so we can't see it all at once and some automation is called for.) - -.. code-block:: none - :emphasize-lines: 1-1 - - cat data/het-bool.csv - name,reachable - barney,false - betty,true - fred,true - wilma,1 - -One option is to coerce everything to boolean, or integer: - -.. code-block:: none - :emphasize-lines: 1-1 - - mlr --icsv --opprint put '$reachable = boolean($reachable)' data/het-bool.csv - name reachable - barney false - betty true - fred true - wilma true - -.. code-block:: none - :emphasize-lines: 1-1 - - mlr --icsv --opprint put '$reachable = int(boolean($reachable))' data/het-bool.csv - name reachable - barney 0 - betty 1 - fred 1 - wilma 1 - -A second option is to flag badly formatted data within the output stream: - -.. code-block:: none - :emphasize-lines: 1-1 - - mlr --icsv --opprint put '$format_ok = is_string($reachable)' data/het-bool.csv - name reachable format_ok - barney false false - betty true false - fred true false - wilma 1 false - -Or perhaps to flag badly formatted data outside the output stream: - -.. code-block:: none - :emphasize-lines: 1-3 - - mlr --icsv --opprint put ' - if (!is_string($reachable)) {eprint "Malformed at NR=".NR} - ' data/het-bool.csv - Malformed at NR=1 - Malformed at NR=2 - Malformed at NR=3 - Malformed at NR=4 - name reachable - barney false - betty true - fred true - wilma 1 - -A third way is to abort the process on first instance of bad data: - -.. code-block:: none - :emphasize-lines: 1-1 - - mlr --csv put '$reachable = asserting_string($reachable)' data/het-bool.csv - Miller: is_string type-assertion failed at NR=1 FNR=1 FILENAME=data/het-bool.csv - -Showing differences between successive queries ----------------------------------------------------------------- - -Suppose you have a database query which you run at one point in time, producing the output on the left, then again later producing the output on the right: - -.. code-block:: none - :emphasize-lines: 1-1 - - cat data/previous_counters.csv - color,count - red,3472 - blue,6838 - orange,694 - purple,12 - -.. code-block:: none - :emphasize-lines: 1-1 - - cat data/current_counters.csv - color,count - red,3467 - orange,670 - yellow,27 - blue,6944 - -And, suppose you want to compute the differences in the counters between adjacent keys. Since the color names aren't all in the same order, nor are they all present on both sides, we can't just paste the two files side-by-side and do some column-four-minus-column-two arithmetic. - -First, rename counter columns to make them distinct: - -.. code-block:: none - :emphasize-lines: 1-1 - - mlr --csv rename count,previous_count data/previous_counters.csv > data/prevtemp.csv - -.. code-block:: none - :emphasize-lines: 1-1 - - cat data/prevtemp.csv - color,previous_count - red,3472 - blue,6838 - orange,694 - purple,12 - -.. code-block:: none - :emphasize-lines: 1-1 - - mlr --csv rename count,current_count data/current_counters.csv > data/currtemp.csv - -.. code-block:: none - :emphasize-lines: 1-1 - - cat data/currtemp.csv - color,current_count - red,3467 - orange,670 - yellow,27 - blue,6944 - -Then, join on the key field(s), and use unsparsify to zero-fill counters absent on one side but present on the other. Use ``--ul`` and ``--ur`` to emit unpaired records (namely, purple on the left and yellow on the right): - -.. code-block:: none - :emphasize-lines: 1-5 - - mlr --icsv --opprint \ - join -j color --ul --ur -f data/prevtemp.csv \ - then unsparsify --fill-with 0 \ - then put '$count_delta = $current_count - $previous_count' \ - data/currtemp.csv - color previous_count current_count count_delta - red 3472 3467 -5 - orange 694 670 -24 - yellow 0 27 (error) - blue 6838 6944 106 - purple 12 0 (error) - -Two-pass algorithms +Overview ---------------------------------------------------------------- Miller is a streaming record processor; commands are performed once per record. This makes Miller particularly suitable for single-pass algorithms, allowing many of its verbs to process files that are (much) larger than the amount of RAM present in your system. (Of course, Miller verbs such as ``sort``, ``tac``, etc. all must ingest and retain all input records before emitting any output records.) You can also use out-of-stream variables to perform multi-pass computations, at the price of retaining all input records in memory. -Two-pass algorithms: computation of percentages -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +One of Miller's strengths is its compact notation: for example, given input of the form + +.. code-block:: none + :emphasize-lines: 1-1 + + head -n 5 ../data/medium + a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533 + a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797 + a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776 + a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463 + a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729 + +you can simply do + +.. code-block:: none + :emphasize-lines: 1-1 + + mlr --oxtab stats1 -a sum -f x ../data/medium + x_sum 4986.019681679581 + +or + +.. code-block:: none + :emphasize-lines: 1-1 + + mlr --opprint stats1 -a sum -f x -g b ../data/medium + b x_sum + pan 965.7636699425815 + wye 1023.5484702619565 + zee 979.7420161495838 + eks 1016.7728571314786 + hat 1000.192668193983 + +rather than the more tedious + +.. code-block:: none + :emphasize-lines: 1-6 + + mlr --oxtab put -q ' + @x_sum += $x; + end { + emit @x_sum + } + ' data/medium + x_sum 4986.019681679581 + +or + +.. code-block:: none + :emphasize-lines: 1-6 + + mlr --opprint put -q ' + @x_sum[$b] += $x; + end { + emit @x_sum, "b" + } + ' data/medium + b x_sum + pan 965.7636699425815 + wye 1023.5484702619565 + zee 979.7420161495838 + eks 1016.7728571314786 + hat 1000.192668193983 + +The former (``mlr stats1`` et al.) has the advantages of being easier to type, being less error-prone to type, and running faster. + +Nonetheless, out-of-stream variables (which I whimsically call *oosvars*), begin/end blocks, and emit statements give you the ability to implement logic -- if you wish to do so -- which isn't present in other Miller verbs. (If you find yourself often using the same out-of-stream-variable logic over and over, please file a request at https://github.com/johnkerl/miller/issues to get it implemented directly in Go as a Miller verb of its own.) + +The following examples compute some things using oosvars which are already computable using Miller verbs, by way of providing food for thought. + +Computation of percentages +---------------------------------------------------------------- For example, mapping numeric values down a column to the percentage between their min and max values is two-pass: on the first pass you find the min and max values, then on the second, map each record's value to a percentage. @@ -191,8 +110,8 @@ For example, mapping numeric values down a column to the percentage between thei 4 0.38139939387114097 31.90823602213647 5 0.5732889198020006 66.54054236562845 -Two-pass algorithms: line-number ratios -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Line-number ratios +---------------------------------------------------------------- Similarly, finding the total record count requires first reading through all the data: @@ -217,8 +136,8 @@ Similarly, finding the total record count requires first reading through all the 4 5 (error) eks wye 4 0.38139939387114097 0.13418874328430463 5 5 (error) wye pan 5 0.5732889198020006 0.8636244699032729 -Two-pass algorithms: records having max value -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Records having max value +---------------------------------------------------------------- The idea is to retain records having the largest value of ``n`` in the following data: @@ -523,134 +442,277 @@ There are field names ``a``, ``b``, ``v``, ``u``, ``x``, ``w`` in the data -- bu There is a keystroke-saving verb for this: :ref:`mlr unsparsify `. -Parsing log-file output +Mean without/with oosvars ---------------------------------------------------------------- -This, of course, depends highly on what's in your log files. But, as an example, suppose you have log-file lines such as - -.. code-block:: none - - 2015-10-08 08:29:09,445 INFO com.company.path.to.ClassName @ [sometext] various/sorts/of data {& punctuation} hits=1 status=0 time=2.378 - -I prefer to pre-filter with ``grep`` and/or ``sed`` to extract the structured text, then hand that to Miller. Example: - .. code-block:: none :emphasize-lines: 1-1 - grep 'various sorts' *.log | sed 's/.*} //' | mlr --fs space --repifs --oxtab stats1 -a min,p10,p50,p90,max -f time -g status + mlr --opprint stats1 -a mean -f x data/medium + x_mean + 0.49860196816795804 -.. _cookbook-memoization-with-oosvars: +.. code-block:: none + :emphasize-lines: 1-8 -Memoization with out-of-stream variables + mlr --opprint put -q ' + @x_sum += $x; + @x_count += 1; + end { + @x_mean = @x_sum / @x_count; + emit @x_mean + } + ' data/medium + x_mean + 0.49860196816795804 + +Keyed mean without/with oosvars ---------------------------------------------------------------- -The recursive function for the Fibonacci sequence is famous for its computational complexity. Namely, using f(0)=1, f(1)=1, f(n)=f(n-1)+f(n-2) for n>=2, the evaluation tree branches left as well as right at each non-trivial level, resulting in millions or more paths to the root 0/1 nodes for larger n. This program +.. code-block:: none + :emphasize-lines: 1-1 + + mlr --opprint stats1 -a mean -f x -g a,b data/medium + a b x_mean + pan pan 0.5133141190437597 + eks pan 0.48507555383425127 + wye wye 0.49150092785839306 + eks wye 0.4838950517724162 + wye pan 0.4996119901034838 + zee pan 0.5198298297816007 + eks zee 0.49546320772681596 + zee wye 0.5142667998230479 + hat wye 0.49381326184632596 + pan wye 0.5023618498923658 + zee eks 0.4883932942792647 + hat zee 0.5099985721987774 + hat eks 0.48587864619953547 + wye hat 0.4977304763723314 + pan eks 0.5036718595143479 + eks eks 0.5227992666570941 + hat hat 0.47993053101017374 + hat pan 0.4643355557376876 + zee zee 0.5127559183726382 + pan hat 0.492140950155604 + pan zee 0.4966041598627583 + zee hat 0.46772617655014515 + wye zee 0.5059066170573692 + eks hat 0.5006790659966355 + wye eks 0.5306035254809106 .. code-block:: none + :emphasize-lines: 1-10 - mlr --ofmt '%.9lf' --opprint seqgen --start 1 --stop 28 then put ' - func f(n) { - @fcount += 1; # count number of calls to the function - if (n < 2) { - return 1 - } else { - return f(n-1) + f(n-2) # recurse - } - } - - @fcount = 0; - $o = f($i); - $fcount = @fcount; - - ' then put '$seconds=systime()' then step -a delta -f seconds then cut -x -f seconds - -produces output like this: - -.. code-block:: none - - i o fcount seconds_delta - 1 1 1 0 - 2 2 3 0.000039101 - 3 3 5 0.000015974 - 4 5 9 0.000019073 - 5 8 15 0.000026941 - 6 13 25 0.000036955 - 7 21 41 0.000056028 - 8 34 67 0.000086069 - 9 55 109 0.000134945 - 10 89 177 0.000217915 - 11 144 287 0.000355959 - 12 233 465 0.000506163 - 13 377 753 0.000811815 - 14 610 1219 0.001297235 - 15 987 1973 0.001960993 - 16 1597 3193 0.003417969 - 17 2584 5167 0.006215811 - 18 4181 8361 0.008294106 - 19 6765 13529 0.012095928 - 20 10946 21891 0.019592047 - 21 17711 35421 0.031193972 - 22 28657 57313 0.057254076 - 23 46368 92735 0.080307961 - 24 75025 150049 0.129482031 - 25 121393 242785 0.213325977 - 26 196418 392835 0.334423065 - 27 317811 635621 0.605969906 - 28 514229 1028457 0.971235037 - -Note that the time it takes to evaluate the function is blowing up exponentially as the input argument increases. Using ``@``-variables, which persist across records, we can cache and reuse the results of previous computations: - -.. code-block:: none - - mlr --ofmt '%.9lf' --opprint seqgen --start 1 --stop 28 then put ' - func f(n) { - @fcount += 1; # count number of calls to the function - if (is_present(@fcache[n])) { # cache hit - return @fcache[n] - } else { # cache miss - num rv = 1; - if (n >= 2) { - rv = f(n-1) + f(n-2) # recurse - } - @fcache[n] = rv; - return rv + mlr --opprint put -q ' + @x_sum[$a][$b] += $x; + @x_count[$a][$b] += 1; + end{ + for ((a, b), v in @x_sum) { + @x_mean[a][b] = @x_sum[a][b] / @x_count[a][b]; } + emit @x_mean, "a", "b" } - @fcount = 0; - $o = f($i); - $fcount = @fcount; - ' then put '$seconds=systime()' then step -a delta -f seconds then cut -x -f seconds + ' data/medium + a b x_mean + pan pan 0.5133141190437597 + pan wye 0.5023618498923658 + pan eks 0.5036718595143479 + pan hat 0.492140950155604 + pan zee 0.4966041598627583 + eks pan 0.48507555383425127 + eks wye 0.4838950517724162 + eks zee 0.49546320772681596 + eks eks 0.5227992666570941 + eks hat 0.5006790659966355 + wye wye 0.49150092785839306 + wye pan 0.4996119901034838 + wye hat 0.4977304763723314 + wye zee 0.5059066170573692 + wye eks 0.5306035254809106 + zee pan 0.5198298297816007 + zee wye 0.5142667998230479 + zee eks 0.4883932942792647 + zee zee 0.5127559183726382 + zee hat 0.46772617655014515 + hat wye 0.49381326184632596 + hat zee 0.5099985721987774 + hat eks 0.48587864619953547 + hat hat 0.47993053101017374 + hat pan 0.4643355557376876 -with output like this: +Variance and standard deviation without/with oosvars +---------------------------------------------------------------- .. code-block:: none + :emphasize-lines: 1-1 - i o fcount seconds_delta - 1 1 1 0 - 2 2 3 0.000053883 - 3 3 3 0.000035048 - 4 5 3 0.000045061 - 5 8 3 0.000014067 - 6 13 3 0.000028849 - 7 21 3 0.000028133 - 8 34 3 0.000027895 - 9 55 3 0.000014067 - 10 89 3 0.000015020 - 11 144 3 0.000012875 - 12 233 3 0.000033140 - 13 377 3 0.000014067 - 14 610 3 0.000012875 - 15 987 3 0.000029087 - 16 1597 3 0.000013828 - 17 2584 3 0.000013113 - 18 4181 3 0.000012875 - 19 6765 3 0.000013113 - 20 10946 3 0.000012875 - 21 17711 3 0.000013113 - 22 28657 3 0.000013113 - 23 46368 3 0.000015974 - 24 75025 3 0.000012875 - 25 121393 3 0.000013113 - 26 196418 3 0.000012875 - 27 317811 3 0.000013113 - 28 514229 3 0.000012875 + mlr --oxtab stats1 -a count,sum,mean,var,stddev -f x data/medium + x_count 10000 + x_sum 4986.019681679581 + x_mean 0.49860196816795804 + x_var 0.08426974433144456 + x_stddev 0.2902925151144007 + +.. code-block:: none + :emphasize-lines: 1-1 + + cat variance.mlr + @n += 1; + @sumx += $x; + @sumx2 += $x**2; + end { + @mean = @sumx / @n; + @var = (@sumx2 - @mean * (2 * @sumx - @n * @mean)) / (@n - 1); + @stddev = sqrt(@var); + emitf @n, @sumx, @sumx2, @mean, @var, @stddev + } + +.. code-block:: none + :emphasize-lines: 1-1 + + mlr --oxtab put -q -f variance.mlr data/medium + n 10000 + sumx 4986.019681679581 + sumx2 3328.652400179729 + mean 0.49860196816795804 + var 0.08426974433144456 + stddev 0.2902925151144007 + +You can also do this keyed, of course, imitating the keyed-mean example above. + +Min/max without/with oosvars +---------------------------------------------------------------- + +.. code-block:: none + :emphasize-lines: 1-1 + + mlr --oxtab stats1 -a min,max -f x data/medium + x_min 4.509679127584487e-05 + x_max 0.999952670371898 + +.. code-block:: none + :emphasize-lines: 1-5 + + mlr --oxtab put -q ' + @x_min = min(@x_min, $x); + @x_max = max(@x_max, $x); + end{emitf @x_min, @x_max} + ' data/medium + x_min 4.509679127584487e-05 + x_max 0.999952670371898 + +Keyed min/max without/with oosvars +---------------------------------------------------------------- + +.. code-block:: none + :emphasize-lines: 1-1 + + mlr --opprint stats1 -a min,max -f x -g a data/medium + a x_min x_max + pan 0.00020390740306253097 0.9994029107062516 + eks 0.0006917972627396018 0.9988110946859143 + wye 0.0001874794831505655 0.9998228522652893 + zee 0.0005486114815762555 0.9994904324789629 + hat 4.509679127584487e-05 0.999952670371898 + +.. code-block:: none + :emphasize-lines: 1-7 + + mlr --opprint --from data/medium put -q ' + @min[$a] = min(@min[$a], $x); + @max[$a] = max(@max[$a], $x); + end{ + emit (@min, @max), "a"; + } + ' + a min max + pan 0.00020390740306253097 0.9994029107062516 + eks 0.0006917972627396018 0.9988110946859143 + wye 0.0001874794831505655 0.9998228522652893 + zee 0.0005486114815762555 0.9994904324789629 + hat 4.509679127584487e-05 0.999952670371898 + +Delta without/with oosvars +---------------------------------------------------------------- + +.. code-block:: none + :emphasize-lines: 1-1 + + mlr --opprint step -a delta -f x data/small + a b i x y x_delta + pan pan 1 0.3467901443380824 0.7268028627434533 0 + eks pan 2 0.7586799647899636 0.5221511083334797 0.41188982045188116 + wye wye 3 0.20460330576630303 0.33831852551664776 -0.5540766590236605 + eks wye 4 0.38139939387114097 0.13418874328430463 0.17679608810483793 + wye pan 5 0.5732889198020006 0.8636244699032729 0.19188952593085962 + +.. code-block:: none + :emphasize-lines: 1-4 + + mlr --opprint put ' + $x_delta = is_present(@last) ? $x - @last : 0; + @last = $x + ' data/small + a b i x y x_delta + pan pan 1 0.3467901443380824 0.7268028627434533 0 + eks pan 2 0.7586799647899636 0.5221511083334797 0.41188982045188116 + wye wye 3 0.20460330576630303 0.33831852551664776 -0.5540766590236605 + eks wye 4 0.38139939387114097 0.13418874328430463 0.17679608810483793 + wye pan 5 0.5732889198020006 0.8636244699032729 0.19188952593085962 + +Keyed delta without/with oosvars +---------------------------------------------------------------- + +.. code-block:: none + :emphasize-lines: 1-1 + + mlr --opprint step -a delta -f x -g a data/small + a b i x y x_delta + pan pan 1 0.3467901443380824 0.7268028627434533 0 + eks pan 2 0.7586799647899636 0.5221511083334797 0 + wye wye 3 0.20460330576630303 0.33831852551664776 0 + eks wye 4 0.38139939387114097 0.13418874328430463 -0.3772805709188226 + wye pan 5 0.5732889198020006 0.8636244699032729 0.36868561403569755 + +.. code-block:: none + :emphasize-lines: 1-4 + + mlr --opprint put ' + $x_delta = is_present(@last[$a]) ? $x - @last[$a] : 0; + @last[$a]=$x + ' data/small + a b i x y x_delta + pan pan 1 0.3467901443380824 0.7268028627434533 0 + eks pan 2 0.7586799647899636 0.5221511083334797 0 + wye wye 3 0.20460330576630303 0.33831852551664776 0 + eks wye 4 0.38139939387114097 0.13418874328430463 -0.3772805709188226 + wye pan 5 0.5732889198020006 0.8636244699032729 0.36868561403569755 + +Exponentially weighted moving averages without/with oosvars +---------------------------------------------------------------- + +.. code-block:: none + :emphasize-lines: 1-1 + + mlr --opprint step -a ewma -d 0.1 -f x data/small + a b i x y x_ewma_0.1 + pan pan 1 0.3467901443380824 0.7268028627434533 0.3467901443380824 + eks pan 2 0.7586799647899636 0.5221511083334797 0.3879791263832706 + wye wye 3 0.20460330576630303 0.33831852551664776 0.36964154432157387 + eks wye 4 0.38139939387114097 0.13418874328430463 0.37081732927653055 + wye pan 5 0.5732889198020006 0.8636244699032729 0.3910644883290776 + +.. code-block:: none + :emphasize-lines: 1-5 + + mlr --opprint put ' + begin{ @a=0.1 }; + $e = NR==1 ? $x : @a * $x + (1 - @a) * @e; + @e=$e + ' data/small + a b i x y e + pan pan 1 0.3467901443380824 0.7268028627434533 0.3467901443380824 + eks pan 2 0.7586799647899636 0.5221511083334797 0.3879791263832706 + wye wye 3 0.20460330576630303 0.33831852551664776 0.36964154432157387 + eks wye 4 0.38139939387114097 0.13418874328430463 0.37081732927653055 + wye pan 5 0.5732889198020006 0.8636244699032729 0.3910644883290776 diff --git a/docs6/two-pass-algorithms.rst.in b/docs6/two-pass-algorithms.rst.in new file mode 100644 index 000000000..4f30361e7 --- /dev/null +++ b/docs6/two-pass-algorithms.rst.in @@ -0,0 +1,218 @@ +Two-pass algorithms +================================================================ + +Overview +---------------------------------------------------------------- + +Miller is a streaming record processor; commands are performed once per record. This makes Miller particularly suitable for single-pass algorithms, allowing many of its verbs to process files that are (much) larger than the amount of RAM present in your system. (Of course, Miller verbs such as ``sort``, ``tac``, etc. all must ingest and retain all input records before emitting any output records.) You can also use out-of-stream variables to perform multi-pass computations, at the price of retaining all input records in memory. + +One of Miller's strengths is its compact notation: for example, given input of the form + +GENRST_RUN_COMMAND +head -n 5 ../data/medium +GENRST_EOF + +you can simply do + +GENRST_RUN_COMMAND +mlr --oxtab stats1 -a sum -f x ../data/medium +GENRST_EOF + +or + +GENRST_RUN_COMMAND +mlr --opprint stats1 -a sum -f x -g b ../data/medium +GENRST_EOF + +rather than the more tedious + +GENRST_INCLUDE_AND_RUN_ESCAPED(oosvar-example-sum.sh) + +or + +GENRST_INCLUDE_AND_RUN_ESCAPED(oosvar-example-sum-grouped.sh) + +The former (``mlr stats1`` et al.) has the advantages of being easier to type, being less error-prone to type, and running faster. + +Nonetheless, out-of-stream variables (which I whimsically call *oosvars*), begin/end blocks, and emit statements give you the ability to implement logic -- if you wish to do so -- which isn't present in other Miller verbs. (If you find yourself often using the same out-of-stream-variable logic over and over, please file a request at https://github.com/johnkerl/miller/issues to get it implemented directly in Go as a Miller verb of its own.) + +The following examples compute some things using oosvars which are already computable using Miller verbs, by way of providing food for thought. + +Computation of percentages +---------------------------------------------------------------- + +For example, mapping numeric values down a column to the percentage between their min and max values is two-pass: on the first pass you find the min and max values, then on the second, map each record's value to a percentage. + +GENRST_INCLUDE_AND_RUN_ESCAPED(data/two-pass-percentage.sh) + +Line-number ratios +---------------------------------------------------------------- + +Similarly, finding the total record count requires first reading through all the data: + +GENRST_INCLUDE_AND_RUN_ESCAPED(data/two-pass-record-numbers.sh) + +Records having max value +---------------------------------------------------------------- + +The idea is to retain records having the largest value of ``n`` in the following data: + +GENRST_RUN_COMMAND +mlr --itsv --opprint cat data/maxrows.tsv +GENRST_EOF + +Of course, the largest value of ``n`` isn't known until after all data have been read. Using an out-of-stream variable we can retain all records as they are read, then filter them at the end: + +GENRST_RUN_COMMAND +cat data/maxrows.mlr +GENRST_EOF + +GENRST_RUN_COMMAND +mlr --itsv --opprint put -q -f data/maxrows.mlr data/maxrows.tsv +GENRST_EOF + +Feature-counting +---------------------------------------------------------------- + +Suppose you have some heterogeneous data like this: + +GENRST_INCLUDE_ESCAPED(data/features.json) + +A reasonable question to ask is, how many occurrences of each field are there? And, what percentage of total row count has each of them? Since the denominator of the percentage is not known until the end, this is a two-pass algorithm: + +GENRST_INCLUDE_ESCAPED(data/feature-count.mlr) + +Then + +GENRST_RUN_COMMAND +mlr --json put -q -f data/feature-count.mlr data/features.json +GENRST_EOF + +GENRST_RUN_COMMAND +mlr --ijson --opprint put -q -f data/feature-count.mlr data/features.json +GENRST_EOF + +Unsparsing +---------------------------------------------------------------- + +The previous section discussed how to fill out missing data fields within CSV with full header line -- so the list of all field names is present within the header line. Next, let's look at a related problem: we have data where each record has various key names but we want to produce rectangular output having the union of all key names. + +For example, suppose you have JSON input like this: + +GENRST_RUN_COMMAND +cat data/sparse.json +GENRST_EOF + +There are field names ``a``, ``b``, ``v``, ``u``, ``x``, ``w`` in the data -- but not all in every record. Since we don't know the names of all the keys until we've read them all, this needs to be a two-pass algorithm. On the first pass, remember all the unique key names and all the records; on the second pass, loop through the records filling in absent values, then producing output. Use ``put -q`` since we don't want to produce per-record output, only emitting output in the ``end`` block: + +GENRST_RUN_COMMAND +cat data/unsparsify.mlr +GENRST_EOF + +GENRST_RUN_COMMAND +mlr --json put -q -f data/unsparsify.mlr data/sparse.json +GENRST_EOF + +GENRST_RUN_COMMAND +mlr --ijson --ocsv put -q -f data/unsparsify.mlr data/sparse.json +GENRST_EOF + +GENRST_RUN_COMMAND +mlr --ijson --opprint put -q -f data/unsparsify.mlr data/sparse.json +GENRST_EOF + +There is a keystroke-saving verb for this: :ref:`mlr unsparsify `. + +Mean without/with oosvars +---------------------------------------------------------------- + +GENRST_RUN_COMMAND +mlr --opprint stats1 -a mean -f x data/medium +GENRST_EOF + +GENRST_INCLUDE_AND_RUN_ESCAPED(data/mean-with-oosvars.sh) + +Keyed mean without/with oosvars +---------------------------------------------------------------- + +GENRST_RUN_COMMAND +mlr --opprint stats1 -a mean -f x -g a,b data/medium +GENRST_EOF + +GENRST_INCLUDE_AND_RUN_ESCAPED(data/keyed-mean-with-oosvars.sh) + +Variance and standard deviation without/with oosvars +---------------------------------------------------------------- + +GENRST_RUN_COMMAND +mlr --oxtab stats1 -a count,sum,mean,var,stddev -f x data/medium +GENRST_EOF + +GENRST_RUN_COMMAND +cat variance.mlr +GENRST_EOF + +GENRST_RUN_COMMAND +mlr --oxtab put -q -f variance.mlr data/medium +GENRST_EOF + +You can also do this keyed, of course, imitating the keyed-mean example above. + +Min/max without/with oosvars +---------------------------------------------------------------- + +GENRST_RUN_COMMAND +mlr --oxtab stats1 -a min,max -f x data/medium +GENRST_EOF + +GENRST_RUN_COMMAND +mlr --oxtab put -q ' + @x_min = min(@x_min, $x); + @x_max = max(@x_max, $x); + end{emitf @x_min, @x_max} +' data/medium +GENRST_EOF + +Keyed min/max without/with oosvars +---------------------------------------------------------------- + +GENRST_RUN_COMMAND +mlr --opprint stats1 -a min,max -f x -g a data/medium +GENRST_EOF + +GENRST_INCLUDE_AND_RUN_ESCAPED(data/keyed-min-max-with-oosvars.sh) + +Delta without/with oosvars +---------------------------------------------------------------- + +GENRST_RUN_COMMAND +mlr --opprint step -a delta -f x data/small +GENRST_EOF + +GENRST_RUN_COMMAND +mlr --opprint put ' + $x_delta = is_present(@last) ? $x - @last : 0; + @last = $x +' data/small +GENRST_EOF + +Keyed delta without/with oosvars +---------------------------------------------------------------- + +GENRST_RUN_COMMAND +mlr --opprint step -a delta -f x -g a data/small +GENRST_EOF + +GENRST_RUN_COMMAND +mlr --opprint put ' + $x_delta = is_present(@last[$a]) ? $x - @last[$a] : 0; + @last[$a]=$x +' data/small +GENRST_EOF + +Exponentially weighted moving averages without/with oosvars +---------------------------------------------------------------- + +GENRST_INCLUDE_AND_RUN_ESCAPED(verb-example-ewma.sh) + +GENRST_INCLUDE_AND_RUN_ESCAPED(oosvar-example-ewma.sh) diff --git a/go/todo.txt b/go/todo.txt index c68e7bd01..c37cacc93 100644 --- a/go/todo.txt +++ b/go/todo.txt @@ -7,6 +7,8 @@ TOP OF LIST: * survey email followups * template verb +! find a way to get timestamps out as ints not scinot + * windows ./mlr hack * windows &&/|| debug @@ -19,6 +21,9 @@ doclink etc: ---------------------------------------------------------------- DOC6 +* shell-commands.html + while-read example issue + * memory page: o open with "out of memory" -- what next? o streaming vs non-streaming