Mirrors/miller
1
0
Fork 0
mirror of https://github.com/johnkerl/miller.git synced 2026-07-29 02:30:12 +00:00
Code Issues Projects Releases Wiki Activity

Ongoing rework of FAQ/recipe "middle" doc sections (#592)

Browse source
This commit is contained in:
John Kerl 2021-07-01 04:28:42 +00:00 • committed by GitHub
parent 6e36f70d56
commit dd38445e89
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
36 changed files with 2073 additions and 2184 deletions
Download patch file Download diff file Expand all files Collapse all files

2
docs6/10min.rst
View file

@ -524,6 +524,8 @@ How to specify these to Miller:
You can read more about this at the :doc:`file-formats` page.
.. _10min-choices-for-printing-to-files:
Choices for printing to files
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

2
docs6/10min.rst.in
View file

@ -266,6 +266,8 @@ How to specify these to Miller:
You can read more about this at the :doc:`file-formats` page.
.. _10min-choices-for-printing-to-files:
Choices for printing to files
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

264
docs6/cookbook.rst.in
View file

@ -1,264 +0,0 @@
Cookbook part 1: common patterns
================================================================
Data-cleaning examples
----------------------------------------------------------------
Here are some ways to use the type-checking options as described in :ref:`reference-dsl-type-tests-and-assertions` Suppose you have the following data file, with inconsistent typing for boolean. (Also imagine that, for the sake of discussion, we have a million-line file rather than a four-line file, so we can't see it all at once and some automation is called for.)
GENRST_RUN_COMMAND
cat data/het-bool.csv
GENRST_EOF
One option is to coerce everything to boolean, or integer:
GENRST_RUN_COMMAND
mlr --icsv --opprint put '$reachable = boolean($reachable)' data/het-bool.csv
GENRST_EOF
GENRST_RUN_COMMAND
mlr --icsv --opprint put '$reachable = int(boolean($reachable))' data/het-bool.csv
GENRST_EOF
A second option is to flag badly formatted data within the output stream:
GENRST_RUN_COMMAND
mlr --icsv --opprint put '$format_ok = is_string($reachable)' data/het-bool.csv
GENRST_EOF
Or perhaps to flag badly formatted data outside the output stream:
GENRST_RUN_COMMAND
mlr --icsv --opprint put '
if (!is_string($reachable)) {eprint "Malformed at NR=".NR}
' data/het-bool.csv
GENRST_EOF
A third way is to abort the process on first instance of bad data:
GENRST_RUN_COMMAND_TOLERATING_ERROR
mlr --csv put '$reachable = asserting_string($reachable)' data/het-bool.csv
GENRST_EOF
Showing differences between successive queries
----------------------------------------------------------------
Suppose you have a database query which you run at one point in time, producing the output on the left, then again later producing the output on the right:
GENRST_RUN_COMMAND
cat data/previous_counters.csv
GENRST_EOF
GENRST_RUN_COMMAND
cat data/current_counters.csv
GENRST_EOF
And, suppose you want to compute the differences in the counters between adjacent keys. Since the color names aren't all in the same order, nor are they all present on both sides, we can't just paste the two files side-by-side and do some column-four-minus-column-two arithmetic.
First, rename counter columns to make them distinct:
GENRST_RUN_COMMAND
mlr --csv rename count,previous_count data/previous_counters.csv > data/prevtemp.csv
GENRST_EOF
GENRST_RUN_COMMAND
cat data/prevtemp.csv
GENRST_EOF
GENRST_RUN_COMMAND
mlr --csv rename count,current_count data/current_counters.csv > data/currtemp.csv
GENRST_EOF
GENRST_RUN_COMMAND
cat data/currtemp.csv
GENRST_EOF
Then, join on the key field(s), and use unsparsify to zero-fill counters absent on one side but present on the other. Use ``--ul`` and ``--ur`` to emit unpaired records (namely, purple on the left and yellow on the right):
GENRST_INCLUDE_AND_RUN_ESCAPED(data/previous-to-current.sh)
Two-pass algorithms
----------------------------------------------------------------
Miller is a streaming record processor; commands are performed once per record. This makes Miller particularly suitable for single-pass algorithms, allowing many of its verbs to process files that are (much) larger than the amount of RAM present in your system. (Of course, Miller verbs such as ``sort``, ``tac``, etc. all must ingest and retain all input records before emitting any output records.) You can also use out-of-stream variables to perform multi-pass computations, at the price of retaining all input records in memory.
Two-pass algorithms: computation of percentages
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
For example, mapping numeric values down a column to the percentage between their min and max values is two-pass: on the first pass you find the min and max values, then on the second, map each record's value to a percentage.
GENRST_INCLUDE_AND_RUN_ESCAPED(data/two-pass-percentage.sh)
Two-pass algorithms: line-number ratios
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Similarly, finding the total record count requires first reading through all the data:
GENRST_INCLUDE_AND_RUN_ESCAPED(data/two-pass-record-numbers.sh)
Two-pass algorithms: records having max value
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The idea is to retain records having the largest value of ``n`` in the following data:
GENRST_RUN_COMMAND
mlr --itsv --opprint cat data/maxrows.tsv
GENRST_EOF
Of course, the largest value of ``n`` isn't known until after all data have been read. Using an out-of-stream variable we can retain all records as they are read, then filter them at the end:
GENRST_RUN_COMMAND
cat data/maxrows.mlr
GENRST_EOF
GENRST_RUN_COMMAND
mlr --itsv --opprint put -q -f data/maxrows.mlr data/maxrows.tsv
GENRST_EOF
Feature-counting
----------------------------------------------------------------
Suppose you have some heterogeneous data like this:
GENRST_INCLUDE_ESCAPED(data/features.json)
A reasonable question to ask is, how many occurrences of each field are there? And, what percentage of total row count has each of them? Since the denominator of the percentage is not known until the end, this is a two-pass algorithm:
GENRST_INCLUDE_ESCAPED(data/feature-count.mlr)
Then
GENRST_RUN_COMMAND
mlr --json put -q -f data/feature-count.mlr data/features.json
GENRST_EOF
GENRST_RUN_COMMAND
mlr --ijson --opprint put -q -f data/feature-count.mlr data/features.json
GENRST_EOF
Unsparsing
----------------------------------------------------------------
The previous section discussed how to fill out missing data fields within CSV with full header line -- so the list of all field names is present within the header line. Next, let's look at a related problem: we have data where each record has various key names but we want to produce rectangular output having the union of all key names.
For example, suppose you have JSON input like this:
GENRST_RUN_COMMAND
cat data/sparse.json
GENRST_EOF
There are field names ``a``, ``b``, ``v``, ``u``, ``x``, ``w`` in the data -- but not all in every record. Since we don't know the names of all the keys until we've read them all, this needs to be a two-pass algorithm. On the first pass, remember all the unique key names and all the records; on the second pass, loop through the records filling in absent values, then producing output. Use ``put -q`` since we don't want to produce per-record output, only emitting output in the ``end`` block:
GENRST_RUN_COMMAND
cat data/unsparsify.mlr
GENRST_EOF
GENRST_RUN_COMMAND
mlr --json put -q -f data/unsparsify.mlr data/sparse.json
GENRST_EOF
GENRST_RUN_COMMAND
mlr --ijson --ocsv put -q -f data/unsparsify.mlr data/sparse.json
GENRST_EOF
GENRST_RUN_COMMAND
mlr --ijson --opprint put -q -f data/unsparsify.mlr data/sparse.json
GENRST_EOF
There is a keystroke-saving verb for this: :ref:`mlr unsparsify <reference-verbs-unsparsify>`.
Parsing log-file output
----------------------------------------------------------------
This, of course, depends highly on what's in your log files. But, as an example, suppose you have log-file lines such as
GENRST_CARDIFY
2015-10-08 08:29:09,445 INFO com.company.path.to.ClassName @ [sometext] various/sorts/of data {& punctuation} hits=1 status=0 time=2.378
GENRST_EOF
I prefer to pre-filter with ``grep`` and/or ``sed`` to extract the structured text, then hand that to Miller. Example:
GENRST_SHOW_COMMAND
grep 'various sorts' *.log | sed 's/.*} //' | mlr --fs space --repifs --oxtab stats1 -a min,p10,p50,p90,max -f time -g status
GENRST_EOF
.. _cookbook-memoization-with-oosvars:
Memoization with out-of-stream variables
----------------------------------------------------------------
The recursive function for the Fibonacci sequence is famous for its computational complexity. Namely, using f(0)=1, f(1)=1, f(n)=f(n-1)+f(n-2) for n>=2, the evaluation tree branches left as well as right at each non-trivial level, resulting in millions or more paths to the root 0/1 nodes for larger n. This program
GENRST_INCLUDE_ESCAPED(data/fibo-uncached.sh)
produces output like this:
GENRST_CARDIFY
i o fcount seconds_delta
1 1 1 0
2 2 3 0.000039101
3 3 5 0.000015974
4 5 9 0.000019073
5 8 15 0.000026941
6 13 25 0.000036955
7 21 41 0.000056028
8 34 67 0.000086069
9 55 109 0.000134945
10 89 177 0.000217915
11 144 287 0.000355959
12 233 465 0.000506163
13 377 753 0.000811815
14 610 1219 0.001297235
15 987 1973 0.001960993
16 1597 3193 0.003417969
17 2584 5167 0.006215811
18 4181 8361 0.008294106
19 6765 13529 0.012095928
20 10946 21891 0.019592047
21 17711 35421 0.031193972
22 28657 57313 0.057254076
23 46368 92735 0.080307961
24 75025 150049 0.129482031
25 121393 242785 0.213325977
26 196418 392835 0.334423065
27 317811 635621 0.605969906
28 514229 1028457 0.971235037
GENRST_EOF
Note that the time it takes to evaluate the function is blowing up exponentially as the input argument increases. Using ``@``-variables, which persist across records, we can cache and reuse the results of previous computations:
GENRST_INCLUDE_ESCAPED(data/fibo-cached.sh)
with output like this:
GENRST_CARDIFY
i o fcount seconds_delta
1 1 1 0
2 2 3 0.000053883
3 3 3 0.000035048
4 5 3 0.000045061
5 8 3 0.000014067
6 13 3 0.000028849
7 21 3 0.000028133
8 34 3 0.000027895
9 55 3 0.000014067
10 89 3 0.000015020
11 144 3 0.000012875
12 233 3 0.000033140
13 377 3 0.000014067
14 610 3 0.000012875
15 987 3 0.000029087
16 1597 3 0.000013828
17 2584 3 0.000013113
18 4181 3 0.000012875
19 6765 3 0.000013113
20 10946 3 0.000012875
21 17711 3 0.000013113
22 28657 3 0.000013113
23 46368 3 0.000015974
24 75025 3 0.000012875
25 121393 3 0.000013113
26 196418 3 0.000012875
27 317811 3 0.000013113
28 514229 3 0.000012875
GENRST_EOF

529
docs6/cookbook2.rst
View file

@ -1,529 +0,0 @@
..
PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE.
Cookbook part 2: Random things, and some math
================================================================
Randomly selecting words from a list
----------------------------------------------------------------
Given this `word list <./data/english-words.txt>`_, first take a look to see what the first few lines look like:
.. code-block:: none
:emphasize-lines: 1-1
head data/english-words.txt
a
aa
aal
aalii
aam
aardvark
aardwolf
aba
abac
abaca
Then the following will randomly sample ten words with four to eight characters in them:
.. code-block:: none
:emphasize-lines: 1-1
mlr --from data/english-words.txt --nidx filter -S 'n=strlen($1);4<=n&&n<=8' then sample -k 10
thionine
birchman
mildewy
avigate
addedly
abaze
askant
aiming
insulant
coinmate
Randomly generating jabberwocky words
----------------------------------------------------------------
These are simple *n*-grams as `described here <http://johnkerl.org/randspell/randspell-slides-ts.pdf>`_. Some common functions are `located here <https://github.com/johnkerl/miller/blob/master/docs/ngrams/ngfuncs.mlr.txt>`_. Then here are scripts for `1-grams <https://github.com/johnkerl/miller/blob/master/docs/ngrams/ng1.mlr.txt>`_ `2-grams <https://github.com/johnkerl/miller/blob/master/docs/ngrams/ng2.mlr.txt>`_ `3-grams <https://github.com/johnkerl/miller/blob/master/docs/ngrams/ng3.mlr.txt>`_ `4-grams <https://github.com/johnkerl/miller/blob/master/docs/ngrams/ng4.mlr.txt>`_, and `5-grams <https://github.com/johnkerl/miller/blob/master/docs/ngrams/ng5.mlr.txt>`_.
The idea is that words from the input file are consumed, then taken apart and pasted back together in ways which imitate the letter-to-letter transitions found in the word list -- giving us automatically generated words in the same vein as *bromance* and *spork*:
.. code-block:: none
:emphasize-lines: 1-1
mlr --nidx --from ./ngrams/gsl-2000.txt put -q -f ./ngrams/ngfuncs.mlr -f ./ngrams/ng5.mlr
beard
plastinguish
politicially
noise
loan
country
controductionary
suppery
lose
lessors
dollar
judge
rottendence
lessenger
diffendant
suggestional
Program timing
----------------------------------------------------------------
This admittedly artificial example demonstrates using Miller time and stats functions to introspectively acquire some information about Miller's own runtime. The ``delta`` function computes the difference between successive timestamps.
.. code-block:: none
$ ruby -e '10000.times{|i|puts "i=#{i+1}"}' > lines.txt
$ head -n 5 lines.txt
i=1
i=2
i=3
i=4
i=5
mlr --ofmt '%.9le' --opprint put '$t=systime()' then step -a delta -f t lines.txt | head -n 7
i t t_delta
1 1430603027.018016 1.430603027e+09
2 1430603027.018043 2.694129944e-05
3 1430603027.018048 5.006790161e-06
4 1430603027.018052 4.053115845e-06
5 1430603027.018055 2.861022949e-06
6 1430603027.018058 3.099441528e-06
mlr --ofmt '%.9le' --oxtab \
put '$t=systime()' then \
step -a delta -f t then \
filter '$i>1' then \
stats1 -a min,mean,max -f t_delta \
lines.txt
t_delta_min 2.861022949e-06
t_delta_mean 4.077508505e-06
t_delta_max 5.388259888e-05
Computing interquartile ranges
----------------------------------------------------------------
For one or more specified field names, simply compute p25 and p75, then write the IQR as the difference of p75 and p25:
.. code-block:: none
:emphasize-lines: 1-3
mlr --oxtab stats1 -f x -a p25,p75 \
then put '$x_iqr = $x_p75 - $x_p25' \
data/medium
x_p25 0.24667037823231752
x_p75 0.7481860062358446
x_iqr 0.5015156280035271
For wildcarded field names, first compute p25 and p75, then loop over field names with ``p25`` in them:
.. code-block:: none
:emphasize-lines: 1-7
mlr --oxtab stats1 --fr '[i-z]' -a p25,p75 \
then put 'for (k,v in $*) {
if (k =~ "(.*)_p25") {
$["\1_iqr"] = $["\1_p75"] - $["\1_p25"]
}
}' \
data/medium
Computing weighted means
----------------------------------------------------------------
This might be more elegantly implemented as an option within the ``stats1`` verb. Meanwhile, it's expressible within the DSL:
.. code-block:: none
:emphasize-lines: 1-24
mlr --from data/medium put -q '
# Using the y field for weighting in this example
weight = $y;
# Using the a field for weighted aggregation in this example
@sumwx[$a] += weight * $i;
@sumw[$a] += weight;
@sumx[$a] += $i;
@sumn[$a] += 1;
end {
map wmean = {};
map mean = {};
for (a in @sumwx) {
wmean[a] = @sumwx[a] / @sumw[a]
}
for (a in @sumx) {
mean[a] = @sumx[a] / @sumn[a]
}
#emit wmean, "a";
#emit mean, "a";
emit (wmean, mean), "a";
}'
a=pan,wmean=4979.563722208067,mean=5028.259010091302
a=eks,wmean=4890.3815931472145,mean=4956.2900763358775
a=wye,wmean=4946.987746229947,mean=4920.001017293998
a=zee,wmean=5164.719684856538,mean=5123.092330239375
a=hat,wmean=4925.533162478552,mean=4967.743946419371
Generating random numbers from various distributions
----------------------------------------------------------------
Here we can chain together a few simple building blocks:
.. code-block:: none
:emphasize-lines: 1-1
cat expo-sample.sh
# Generate 100,000 pairs of independent and identically distributed
# exponentially distributed random variables with the same rate parameter
# (namely, 2.5). Then compute histograms of one of them, along with
# histograms for their sum and their product.
#
# See also https://en.wikipedia.org/wiki/Exponential_distribution
#
# Here I'm using a specified random-number seed so this example always
# produces the same output for this web document: in everyday practice we
# wouldn't do that.
mlr -n \
--seed 0 \
--opprint \
seqgen --stop 100000 \
then put '
# https://en.wikipedia.org/wiki/Inverse_transform_sampling
func expo_sample(lambda) {
return -log(1-urand())/lambda
}
$u = expo_sample(2.5);
$v = expo_sample(2.5);
$s = $u + $v;
$p = $u * $v;
' \
then histogram -f u,s,p --lo 0 --hi 2 --nbins 50 \
then bar -f u_count,s_count,p_count --auto -w 20
Namely:
* Set the Miller random-number seed so this webdoc looks the same every time I regenerate it.
* Use pretty-printed tabular output.
* Use pretty-printed tabular output.
* Use ``seqgen`` to produce 100,000 records ``i=0``, ``i=1``, etc.
* Send those to a ``put`` step which defines an inverse-transform-sampling function and calls it twice, then computes the sum and product of samples.
* Send those to a histogram, and from there to a bar-plotter. This is just for visualization; you could just as well output CSV and send that off to your own plotting tool, etc.
The output is as follows:
.. code-block:: none
:emphasize-lines: 1-1
sh expo-sample.sh
bin_lo bin_hi u_count s_count p_count
0 0.04 [64]*******************#[9554] [326]#...................[3703] [19]*******************#[39809]
0.04 0.08 [64]*****************...[9554] [326]*****...............[3703] [19]*******.............[39809]
0.08 0.12 [64]****************....[9554] [326]*********...........[3703] [19]****................[39809]
0.12 0.16 [64]**************......[9554] [326]************........[3703] [19]***.................[39809]
0.16 0.2 [64]*************.......[9554] [326]**************......[3703] [19]**..................[39809]
0.2 0.24 [64]************........[9554] [326]*****************...[3703] [19]*...................[39809]
0.24 0.28 [64]**********..........[9554] [326]******************..[3703] [19]*...................[39809]
0.28 0.32 [64]*********...........[9554] [326]******************..[3703] [19]*...................[39809]
0.32 0.36 [64]********............[9554] [326]*******************.[3703] [19]#...................[39809]
0.36 0.4 [64]*******.............[9554] [326]*******************#[3703] [19]#...................[39809]
0.4 0.44 [64]*******.............[9554] [326]*******************.[3703] [19]#...................[39809]
0.44 0.48 [64]******..............[9554] [326]*******************.[3703] [19]#...................[39809]
0.48 0.52 [64]*****...............[9554] [326]******************..[3703] [19]#...................[39809]
0.52 0.56 [64]*****...............[9554] [326]******************..[3703] [19]#...................[39809]
0.56 0.6 [64]****................[9554] [326]*****************...[3703] [19]#...................[39809]
0.6 0.64 [64]****................[9554] [326]******************..[3703] [19]#...................[39809]
0.64 0.68 [64]***.................[9554] [326]****************....[3703] [19]#...................[39809]
0.68 0.72 [64]***.................[9554] [326]****************....[3703] [19]#...................[39809]
0.72 0.76 [64]***.................[9554] [326]***************.....[3703] [19]#...................[39809]
0.76 0.8 [64]**..................[9554] [326]**************......[3703] [19]#...................[39809]
0.8 0.84 [64]**..................[9554] [326]*************.......[3703] [19]#...................[39809]
0.84 0.88 [64]**..................[9554] [326]************........[3703] [19]#...................[39809]
0.88 0.92 [64]**..................[9554] [326]************........[3703] [19]#...................[39809]
0.92 0.96 [64]*...................[9554] [326]***********.........[3703] [19]#...................[39809]
0.96 1 [64]*...................[9554] [326]**********..........[3703] [19]#...................[39809]
1 1.04 [64]*...................[9554] [326]*********...........[3703] [19]#...................[39809]
1.04 1.08 [64]*...................[9554] [326]********............[3703] [19]#...................[39809]
1.08 1.12 [64]*...................[9554] [326]********............[3703] [19]#...................[39809]
1.12 1.16 [64]*...................[9554] [326]********............[3703] [19]#...................[39809]
1.16 1.2 [64]*...................[9554] [326]*******.............[3703] [19]#...................[39809]
1.2 1.24 [64]#...................[9554] [326]******..............[3703] [19]#...................[39809]
1.24 1.28 [64]#...................[9554] [326]*****...............[3703] [19]#...................[39809]
1.28 1.32 [64]#...................[9554] [326]*****...............[3703] [19]#...................[39809]
1.32 1.36 [64]#...................[9554] [326]****................[3703] [19]#...................[39809]
1.36 1.4 [64]#...................[9554] [326]****................[3703] [19]#...................[39809]
1.4 1.44 [64]#...................[9554] [326]****................[3703] [19]#...................[39809]
1.44 1.48 [64]#...................[9554] [326]***.................[3703] [19]#...................[39809]
1.48 1.52 [64]#...................[9554] [326]***.................[3703] [19]#...................[39809]
1.52 1.56 [64]#...................[9554] [326]***.................[3703] [19]#...................[39809]
1.56 1.6 [64]#...................[9554] [326]**..................[3703] [19]#...................[39809]
1.6 1.64 [64]#...................[9554] [326]**..................[3703] [19]#...................[39809]
1.64 1.68 [64]#...................[9554] [326]**..................[3703] [19]#...................[39809]
1.68 1.72 [64]#...................[9554] [326]*...................[3703] [19]#...................[39809]
1.72 1.76 [64]#...................[9554] [326]*...................[3703] [19]#...................[39809]
1.76 1.8 [64]#...................[9554] [326]*...................[3703] [19]#...................[39809]
1.8 1.84 [64]#...................[9554] [326]#...................[3703] [19]#...................[39809]
1.84 1.88 [64]#...................[9554] [326]#...................[3703] [19]#...................[39809]
1.88 1.92 [64]#...................[9554] [326]#...................[3703] [19]#...................[39809]
1.92 1.96 [64]#...................[9554] [326]#...................[3703] [19]#...................[39809]
1.96 2 [64]#...................[9554] [326]#...................[3703] [19]#...................[39809]
Sieve of Eratosthenes
----------------------------------------------------------------
The `Sieve of Eratosthenes <http://en.wikipedia.org/wiki/Sieve_of_Eratosthenes>`_ is a standard introductory programming topic. The idea is to find all primes up to some *N* by making a list of the numbers 1 to *N*, then striking out all multiples of 2 except 2 itself, all multiples of 3 except 3 itself, all multiples of 4 except 4 itself, and so on. Whatever survives that without getting marked is a prime. This is easy enough in Miller. Notice that here all the work is in ``begin`` and ``end`` statements; there is no file input (so we use ``mlr -n`` to keep Miller from waiting for input data).
.. code-block:: none
:emphasize-lines: 1-1
cat programs/sieve.mlr
# ================================================================
# Sieve of Eratosthenes: simple example of Miller DSL as programming language.
# ================================================================
# Put this in a begin-block so we can do either
# mlr -n put -q -f name-of-this-file.mlr
# or
# mlr -n put -q -f name-of-this-file.mlr -e '@n = 200'
# i.e. 100 is the default upper limit, and another can be specified using -e.
begin {
@n = 100;
}
end {
for (int i = 0; i <= @n; i += 1) {
@s[i] = true;
}
@s[0] = false; # 0 is neither prime nor composite
@s[1] = false; # 1 is neither prime nor composite
# Strike out multiples
for (int i = 2; i <= @n; i += 1) {
for (int j = i+i; j <= @n; j += i) {
@s[j] = false;
}
}
# Print survivors
for (int i = 0; i <= @n; i += 1) {
if (@s[i]) {
print i;
}
}
}
.. code-block:: none
:emphasize-lines: 1-1
mlr -n put -f programs/sieve.mlr
2
3
5
7
11
13
17
19
23
29
31
37
41
43
47
53
59
61
67
71
73
79
83
89
97
Mandelbrot-set generator
----------------------------------------------------------------
The `Mandelbrot set <http://en.wikipedia.org/wiki/Mandelbrot_set>`_ is also easily expressed. This isn't an important case of data-processing in the vein for which Miller was designed, but it is an example of Miller as a general-purpose programming language -- a test case for the expressiveness of the language.
The (approximate) computation of points in the complex plane which are and aren't members is just a few lines of complex arithmetic (see the Wikipedia article); how to render them is another task. Using graphics libraries you can create PNG or JPEG files, but another fun way to do this is by printing various characters to the screen:
.. code-block:: none
:emphasize-lines: 1-1
cat programs/mand.mlr
# Mandelbrot set generator: simple example of Miller DSL as programming language.
begin {
# Set defaults
@rcorn = -2.0;
@icorn = -2.0;
@side = 4.0;
@iheight = 50;
@iwidth = 100;
@maxits = 100;
@levelstep = 5;
@chars = "@X*o-."; # Palette of characters to print to the screen.
@verbose = false;
@do_julia = false;
@jr = 0.0; # Real part of Julia point, if any
@ji = 0.0; # Imaginary part of Julia point, if any
}
# Here, we can override defaults from an input file (if any). In Miller's
# put/filter DSL, absent-null right-hand sides result in no assignment so we
# can simply put @rcorn = $rcorn: if there is a field in the input like
# 'rcorn = -1.847' we'll read and use it, else we'll keep the default.
@rcorn = $rcorn;
@icorn = $icorn;
@side = $side;
@iheight = $iheight;
@iwidth = $iwidth;
@maxits = $maxits;
@levelstep = $levelstep;
@chars = $chars;
@verbose = $verbose;
@do_julia = $do_julia;
@jr = $jr;
@ji = $ji;
end {
if (@verbose) {
print "RCORN = ".@rcorn;
print "ICORN = ".@icorn;
print "SIDE = ".@side;
print "IHEIGHT = ".@iheight;
print "IWIDTH = ".@iwidth;
print "MAXITS = ".@maxits;
print "LEVELSTEP = ".@levelstep;
print "CHARS = ".@chars;
}
# Iterate over a matrix of rows and columns, printing one character for each cell.
for (int ii = @iheight-1; ii >= 0; ii -= 1) {
num pi = @icorn + (ii/@iheight) * @side;
for (int ir = 0; ir < @iwidth; ir += 1) {
num pr = @rcorn + (ir/@iwidth) * @side;
printn get_point_plot(pr, pi, @maxits, @do_julia, @jr, @ji);
}
print;
}
}
# This is a function to approximate membership in the Mandelbrot set (or Julia
# set for a given Julia point if do_julia == true) for a given point in the
# complex plane.
func get_point_plot(pr, pi, maxits, do_julia, jr, ji) {
num zr = 0.0;
num zi = 0.0;
num cr = 0.0;
num ci = 0.0;
if (!do_julia) {
zr = 0.0;
zi = 0.0;
cr = pr;
ci = pi;
} else {
zr = pr;
zi = pi;
cr = jr;
ci = ji;
}
int iti = 0;
bool escaped = false;
num zt = 0;
for (iti = 0; iti < maxits; iti += 1) {
num mag = zr*zr + zi+zi;
if (mag > 4.0) {
escaped = true;
break;
}
# z := z^2 + c
zt = zr*zr - zi*zi + cr;
zi = 2*zr*zi + ci;
zr = zt;
}
if (!escaped) {
return ".";
} else {
# The // operator is Miller's (pythonic) integer-division operator
int level = (iti // @levelstep) % strlen(@chars);
return substr(@chars, level, level);
}
}
At standard resolution this makes a nice little ASCII plot:
.. code-block:: none
:emphasize-lines: 1-1
mlr -n put -f ./programs/mand.mlr
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXX.XXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXooXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXX**o..*XXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXX*-....-oXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXX@XXXXXXXXXX*......o*XXXXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXXX**oo*-.-........oo.XXXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXXXXXX....................X..o-XXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@XXXXXXXXXXXXXXX*oo......................oXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@XXX*XXXXXXXXXXXX**o........................*X*X@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@XXXXXXooo***o*.*XX**X..........................o-XX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@XXXXXXXX*-.......-***.............................oXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@XXXXXXXX*@..........Xo............................*XX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@XXXX@XXXXXXXX*o@oX...........@...........................oXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
.........................................................o*XXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@XXXXXXXXX*-.oX...........@...........................oXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@XXXXXXXXXX**@..........*o............................*XXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@XXXXXXXXXXXXX-........***.............................oXXXXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@XXXXXXXXXXXXoo****o*.XX***@..........................o-XXXXXXXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@XXXXX*XXXX*XXXXXXX**-........................***XXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@XXXXXXXXXXXXX*o*.....................@o*XXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXXXXX*....................*..o-XX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXX*ooo*-.o........oo.X*XXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXX**@.....*XXXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXX*o....-o*XXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXXXo*o..*XXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXXXXXX*o*XXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXXXXX@XXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXX@@XXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
But using a very small font size (as small as my Mac will let me go), and by choosing the coordinates to zoom in on a particular part of the complex plane, we can get a nice little picture:
.. code-block:: none
#!/bin/bash
# Get the number of rows and columns from the terminal window dimensions
iheight=$(stty size | mlr --nidx --fs space cut -f 1)
iwidth=$(stty size | mlr --nidx --fs space cut -f 2)
echo "rcorn=-1.755350,icorn=+0.014230,side=0.000020,maxits=10000,iheight=$iheight,iwidth=$iwidth" \
| mlr put -f programs/mand.mlr
.. image:: pix/mand.png

156
docs6/cookbook2.rst.in
View file

@ -1,156 +0,0 @@
Cookbook part 2: Random things, and some math
================================================================
Randomly selecting words from a list
----------------------------------------------------------------
Given this `word list <./data/english-words.txt>`_, first take a look to see what the first few lines look like:
GENRST_CARDIFY_HIGHLIGHT_ONE
head data/english-words.txt
a
aa
aal
aalii
aam
aardvark
aardwolf
aba
abac
abaca
GENRST_EOF
Then the following will randomly sample ten words with four to eight characters in them:
GENRST_CARDIFY_HIGHLIGHT_ONE
mlr --from data/english-words.txt --nidx filter -S 'n=strlen($1);4<=n&&n<=8' then sample -k 10
thionine
birchman
mildewy
avigate
addedly
abaze
askant
aiming
insulant
coinmate
GENRST_EOF
Randomly generating jabberwocky words
----------------------------------------------------------------
These are simple *n*-grams as `described here <http://johnkerl.org/randspell/randspell-slides-ts.pdf>`_. Some common functions are `located here <https://github.com/johnkerl/miller/blob/master/docs/ngrams/ngfuncs.mlr.txt>`_. Then here are scripts for `1-grams <https://github.com/johnkerl/miller/blob/master/docs/ngrams/ng1.mlr.txt>`_ `2-grams <https://github.com/johnkerl/miller/blob/master/docs/ngrams/ng2.mlr.txt>`_ `3-grams <https://github.com/johnkerl/miller/blob/master/docs/ngrams/ng3.mlr.txt>`_ `4-grams <https://github.com/johnkerl/miller/blob/master/docs/ngrams/ng4.mlr.txt>`_, and `5-grams <https://github.com/johnkerl/miller/blob/master/docs/ngrams/ng5.mlr.txt>`_.
The idea is that words from the input file are consumed, then taken apart and pasted back together in ways which imitate the letter-to-letter transitions found in the word list -- giving us automatically generated words in the same vein as *bromance* and *spork*:
GENRST_CARDIFY_HIGHLIGHT_ONE
mlr --nidx --from ./ngrams/gsl-2000.txt put -q -f ./ngrams/ngfuncs.mlr -f ./ngrams/ng5.mlr
beard
plastinguish
politicially
noise
loan
country
controductionary
suppery
lose
lessors
dollar
judge
rottendence
lessenger
diffendant
suggestional
GENRST_EOF
Program timing
----------------------------------------------------------------
This admittedly artificial example demonstrates using Miller time and stats functions to introspectively acquire some information about Miller's own runtime. The ``delta`` function computes the difference between successive timestamps.
GENRST_INCLUDE_ESCAPED(data/timing-example.txt)
Computing interquartile ranges
----------------------------------------------------------------
For one or more specified field names, simply compute p25 and p75, then write the IQR as the difference of p75 and p25:
GENRST_INCLUDE_AND_RUN_ESCAPED(data/iqr1.sh)
For wildcarded field names, first compute p25 and p75, then loop over field names with ``p25`` in them:
GENRST_INCLUDE_AND_RUN_ESCAPED(data/iqrn.sh)
Computing weighted means
----------------------------------------------------------------
This might be more elegantly implemented as an option within the ``stats1`` verb. Meanwhile, it's expressible within the DSL:
GENRST_INCLUDE_AND_RUN_ESCAPED(data/weighted-mean.sh)
Generating random numbers from various distributions
----------------------------------------------------------------
Here we can chain together a few simple building blocks:
GENRST_RUN_COMMAND
cat expo-sample.sh
GENRST_EOF
Namely:
* Set the Miller random-number seed so this webdoc looks the same every time I regenerate it.
* Use pretty-printed tabular output.
* Use pretty-printed tabular output.
* Use ``seqgen`` to produce 100,000 records ``i=0``, ``i=1``, etc.
* Send those to a ``put`` step which defines an inverse-transform-sampling function and calls it twice, then computes the sum and product of samples.
* Send those to a histogram, and from there to a bar-plotter. This is just for visualization; you could just as well output CSV and send that off to your own plotting tool, etc.
The output is as follows:
GENRST_RUN_COMMAND
sh expo-sample.sh
GENRST_EOF
Sieve of Eratosthenes
----------------------------------------------------------------
The `Sieve of Eratosthenes <http://en.wikipedia.org/wiki/Sieve_of_Eratosthenes>`_ is a standard introductory programming topic. The idea is to find all primes up to some *N* by making a list of the numbers 1 to *N*, then striking out all multiples of 2 except 2 itself, all multiples of 3 except 3 itself, all multiples of 4 except 4 itself, and so on. Whatever survives that without getting marked is a prime. This is easy enough in Miller. Notice that here all the work is in ``begin`` and ``end`` statements; there is no file input (so we use ``mlr -n`` to keep Miller from waiting for input data).
GENRST_RUN_COMMAND
cat programs/sieve.mlr
GENRST_EOF
GENRST_RUN_COMMAND
mlr -n put -f programs/sieve.mlr
GENRST_EOF
Mandelbrot-set generator
----------------------------------------------------------------
The `Mandelbrot set <http://en.wikipedia.org/wiki/Mandelbrot_set>`_ is also easily expressed. This isn't an important case of data-processing in the vein for which Miller was designed, but it is an example of Miller as a general-purpose programming language -- a test case for the expressiveness of the language.
The (approximate) computation of points in the complex plane which are and aren't members is just a few lines of complex arithmetic (see the Wikipedia article); how to render them is another task. Using graphics libraries you can create PNG or JPEG files, but another fun way to do this is by printing various characters to the screen:
GENRST_RUN_COMMAND
cat programs/mand.mlr
GENRST_EOF
At standard resolution this makes a nice little ASCII plot:
GENRST_RUN_COMMAND
mlr -n put -f ./programs/mand.mlr
GENRST_EOF
But using a very small font size (as small as my Mac will let me go), and by choosing the coordinates to zoom in on a particular part of the complex plane, we can get a nice little picture:
GENRST_CARDIFY
#!/bin/bash
# Get the number of rows and columns from the terminal window dimensions
iheight=$(stty size | mlr --nidx --fs space cut -f 1)
iwidth=$(stty size | mlr --nidx --fs space cut -f 2)
echo "rcorn=-1.755350,icorn=+0.014230,side=0.000020,maxits=10000,iheight=$iheight,iwidth=$iwidth" \
| mlr put -f programs/mand.mlr
GENRST_EOF
.. image:: pix/mand.png

353
docs6/cookbook3.rst
View file

@ -1,353 +0,0 @@
..
PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE.
Cookbook part 3: Stats with and without out-of-stream variables
================================================================
Overview
----------------------------------------------------------------
One of Miller's strengths is its compact notation: for example, given input of the form
.. code-block:: none
:emphasize-lines: 1-1
head -n 5 ../data/medium
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
you can simply do
.. code-block:: none
:emphasize-lines: 1-1
mlr --oxtab stats1 -a sum -f x ../data/medium
x_sum 4986.019681679581
or
.. code-block:: none
:emphasize-lines: 1-1
mlr --opprint stats1 -a sum -f x -g b ../data/medium
b x_sum
pan 965.7636699425815
wye 1023.5484702619565
zee 979.7420161495838
eks 1016.7728571314786
hat 1000.192668193983
rather than the more tedious
.. code-block:: none
:emphasize-lines: 1-6
mlr --oxtab put -q '
@x_sum += $x;
end {
emit @x_sum
}
' data/medium
x_sum 4986.019681679581
or
.. code-block:: none
:emphasize-lines: 1-6
mlr --opprint put -q '
@x_sum[$b] += $x;
end {
emit @x_sum, "b"
}
' data/medium
b x_sum
pan 965.7636699425815
wye 1023.5484702619565
zee 979.7420161495838
eks 1016.7728571314786
hat 1000.192668193983
The former (``mlr stats1`` et al.) has the advantages of being easier to type, being less error-prone to type, and running faster.
Nonetheless, out-of-stream variables (which I whimsically call *oosvars*), begin/end blocks, and emit statements give you the ability to implement logic -- if you wish to do so -- which isn't present in other Miller verbs. (If you find yourself often using the same out-of-stream-variable logic over and over, please file a request at https://github.com/johnkerl/miller/issues to get it implemented directly in Go as a Miller verb of its own.)
The following examples compute some things using oosvars which are already computable using Miller verbs, by way of providing food for thought.
Mean without/with oosvars
----------------------------------------------------------------
.. code-block:: none
:emphasize-lines: 1-1
mlr --opprint stats1 -a mean -f x data/medium
x_mean
0.49860196816795804
.. code-block:: none
:emphasize-lines: 1-8
mlr --opprint put -q '
@x_sum += $x;
@x_count += 1;
end {
@x_mean = @x_sum / @x_count;
emit @x_mean
}
' data/medium
x_mean
0.49860196816795804
Keyed mean without/with oosvars
----------------------------------------------------------------
.. code-block:: none
:emphasize-lines: 1-1
mlr --opprint stats1 -a mean -f x -g a,b data/medium
a b x_mean
pan pan 0.5133141190437597
eks pan 0.48507555383425127
wye wye 0.49150092785839306
eks wye 0.4838950517724162
wye pan 0.4996119901034838
zee pan 0.5198298297816007
eks zee 0.49546320772681596
zee wye 0.5142667998230479
hat wye 0.49381326184632596
pan wye 0.5023618498923658
zee eks 0.4883932942792647
hat zee 0.5099985721987774
hat eks 0.48587864619953547
wye hat 0.4977304763723314
pan eks 0.5036718595143479
eks eks 0.5227992666570941
hat hat 0.47993053101017374
hat pan 0.4643355557376876
zee zee 0.5127559183726382
pan hat 0.492140950155604
pan zee 0.4966041598627583
zee hat 0.46772617655014515
wye zee 0.5059066170573692
eks hat 0.5006790659966355
wye eks 0.5306035254809106
.. code-block:: none
:emphasize-lines: 1-10
mlr --opprint put -q '
@x_sum[$a][$b] += $x;
@x_count[$a][$b] += 1;
end{
for ((a, b), v in @x_sum) {
@x_mean[a][b] = @x_sum[a][b] / @x_count[a][b];
}
emit @x_mean, "a", "b"
}
' data/medium
a b x_mean
pan pan 0.5133141190437597
pan wye 0.5023618498923658
pan eks 0.5036718595143479
pan hat 0.492140950155604
pan zee 0.4966041598627583
eks pan 0.48507555383425127
eks wye 0.4838950517724162
eks zee 0.49546320772681596
eks eks 0.5227992666570941
eks hat 0.5006790659966355
wye wye 0.49150092785839306
wye pan 0.4996119901034838
wye hat 0.4977304763723314
wye zee 0.5059066170573692
wye eks 0.5306035254809106
zee pan 0.5198298297816007
zee wye 0.5142667998230479
zee eks 0.4883932942792647
zee zee 0.5127559183726382
zee hat 0.46772617655014515
hat wye 0.49381326184632596
hat zee 0.5099985721987774
hat eks 0.48587864619953547
hat hat 0.47993053101017374
hat pan 0.4643355557376876
Variance and standard deviation without/with oosvars
----------------------------------------------------------------
.. code-block:: none
:emphasize-lines: 1-1
mlr --oxtab stats1 -a count,sum,mean,var,stddev -f x data/medium
x_count 10000
x_sum 4986.019681679581
x_mean 0.49860196816795804
x_var 0.08426974433144456
x_stddev 0.2902925151144007
.. code-block:: none
:emphasize-lines: 1-1
cat variance.mlr
@n += 1;
@sumx += $x;
@sumx2 += $x**2;
end {
@mean = @sumx / @n;
@var = (@sumx2 - @mean * (2 * @sumx - @n * @mean)) / (@n - 1);
@stddev = sqrt(@var);
emitf @n, @sumx, @sumx2, @mean, @var, @stddev
}
.. code-block:: none
:emphasize-lines: 1-1
mlr --oxtab put -q -f variance.mlr data/medium
n 10000
sumx 4986.019681679581
sumx2 3328.652400179729
mean 0.49860196816795804
var 0.08426974433144456
stddev 0.2902925151144007
You can also do this keyed, of course, imitating the keyed-mean example above.
Min/max without/with oosvars
----------------------------------------------------------------
.. code-block:: none
:emphasize-lines: 1-1
mlr --oxtab stats1 -a min,max -f x data/medium
x_min 4.509679127584487e-05
x_max 0.999952670371898
.. code-block:: none
:emphasize-lines: 1-5
mlr --oxtab put -q '
@x_min = min(@x_min, $x);
@x_max = max(@x_max, $x);
end{emitf @x_min, @x_max}
' data/medium
x_min 4.509679127584487e-05
x_max 0.999952670371898
Keyed min/max without/with oosvars
----------------------------------------------------------------
.. code-block:: none
:emphasize-lines: 1-1
mlr --opprint stats1 -a min,max -f x -g a data/medium
a x_min x_max
pan 0.00020390740306253097 0.9994029107062516
eks 0.0006917972627396018 0.9988110946859143
wye 0.0001874794831505655 0.9998228522652893
zee 0.0005486114815762555 0.9994904324789629
hat 4.509679127584487e-05 0.999952670371898
.. code-block:: none
:emphasize-lines: 1-7
mlr --opprint --from data/medium put -q '
@min[$a] = min(@min[$a], $x);
@max[$a] = max(@max[$a], $x);
end{
emit (@min, @max), "a";
}
'
a min max
pan 0.00020390740306253097 0.9994029107062516
eks 0.0006917972627396018 0.9988110946859143
wye 0.0001874794831505655 0.9998228522652893
zee 0.0005486114815762555 0.9994904324789629
hat 4.509679127584487e-05 0.999952670371898
Delta without/with oosvars
----------------------------------------------------------------
.. code-block:: none
:emphasize-lines: 1-1
mlr --opprint step -a delta -f x data/small
a b i x y x_delta
pan pan 1 0.3467901443380824 0.7268028627434533 0
eks pan 2 0.7586799647899636 0.5221511083334797 0.41188982045188116
wye wye 3 0.20460330576630303 0.33831852551664776 -0.5540766590236605
eks wye 4 0.38139939387114097 0.13418874328430463 0.17679608810483793
wye pan 5 0.5732889198020006 0.8636244699032729 0.19188952593085962
.. code-block:: none
:emphasize-lines: 1-4
mlr --opprint put '
$x_delta = is_present(@last) ? $x - @last : 0;
@last = $x
' data/small
a b i x y x_delta
pan pan 1 0.3467901443380824 0.7268028627434533 0
eks pan 2 0.7586799647899636 0.5221511083334797 0.41188982045188116
wye wye 3 0.20460330576630303 0.33831852551664776 -0.5540766590236605
eks wye 4 0.38139939387114097 0.13418874328430463 0.17679608810483793
wye pan 5 0.5732889198020006 0.8636244699032729 0.19188952593085962
Keyed delta without/with oosvars
----------------------------------------------------------------
.. code-block:: none
:emphasize-lines: 1-1
mlr --opprint step -a delta -f x -g a data/small
a b i x y x_delta
pan pan 1 0.3467901443380824 0.7268028627434533 0
eks pan 2 0.7586799647899636 0.5221511083334797 0
wye wye 3 0.20460330576630303 0.33831852551664776 0
eks wye 4 0.38139939387114097 0.13418874328430463 -0.3772805709188226
wye pan 5 0.5732889198020006 0.8636244699032729 0.36868561403569755
.. code-block:: none
:emphasize-lines: 1-4
mlr --opprint put '
$x_delta = is_present(@last[$a]) ? $x - @last[$a] : 0;
@last[$a]=$x
' data/small
a b i x y x_delta
pan pan 1 0.3467901443380824 0.7268028627434533 0
eks pan 2 0.7586799647899636 0.5221511083334797 0
wye wye 3 0.20460330576630303 0.33831852551664776 0
eks wye 4 0.38139939387114097 0.13418874328430463 -0.3772805709188226
wye pan 5 0.5732889198020006 0.8636244699032729 0.36868561403569755
Exponentially weighted moving averages without/with oosvars
----------------------------------------------------------------
.. code-block:: none
:emphasize-lines: 1-1
mlr --opprint step -a ewma -d 0.1 -f x data/small
a b i x y x_ewma_0.1
pan pan 1 0.3467901443380824 0.7268028627434533 0.3467901443380824
eks pan 2 0.7586799647899636 0.5221511083334797 0.3879791263832706
wye wye 3 0.20460330576630303 0.33831852551664776 0.36964154432157387
eks wye 4 0.38139939387114097 0.13418874328430463 0.37081732927653055
wye pan 5 0.5732889198020006 0.8636244699032729 0.3910644883290776
.. code-block:: none
:emphasize-lines: 1-5
mlr --opprint put '
begin{ @a=0.1 };
$e = NR==1 ? $x : @a * $x + (1 - @a) * @e;
@e=$e
' data/small
a b i x y e
pan pan 1 0.3467901443380824 0.7268028627434533 0.3467901443380824
eks pan 2 0.7586799647899636 0.5221511083334797 0.3879791263832706
wye wye 3 0.20460330576630303 0.33831852551664776 0.36964154432157387
eks wye 4 0.38139939387114097 0.13418874328430463 0.37081732927653055
wye pan 5 0.5732889198020006 0.8636244699032729 0.3910644883290776

131
docs6/cookbook3.rst.in
View file

@ -1,131 +0,0 @@
Cookbook part 3: Stats with and without out-of-stream variables
================================================================
Overview
----------------------------------------------------------------
One of Miller's strengths is its compact notation: for example, given input of the form
GENRST_RUN_COMMAND
head -n 5 ../data/medium
GENRST_EOF
you can simply do
GENRST_RUN_COMMAND
mlr --oxtab stats1 -a sum -f x ../data/medium
GENRST_EOF
or
GENRST_RUN_COMMAND
mlr --opprint stats1 -a sum -f x -g b ../data/medium
GENRST_EOF
rather than the more tedious
GENRST_INCLUDE_AND_RUN_ESCAPED(oosvar-example-sum.sh)
or
GENRST_INCLUDE_AND_RUN_ESCAPED(oosvar-example-sum-grouped.sh)
The former (``mlr stats1`` et al.) has the advantages of being easier to type, being less error-prone to type, and running faster.
Nonetheless, out-of-stream variables (which I whimsically call *oosvars*), begin/end blocks, and emit statements give you the ability to implement logic -- if you wish to do so -- which isn't present in other Miller verbs. (If you find yourself often using the same out-of-stream-variable logic over and over, please file a request at https://github.com/johnkerl/miller/issues to get it implemented directly in Go as a Miller verb of its own.)
The following examples compute some things using oosvars which are already computable using Miller verbs, by way of providing food for thought.
Mean without/with oosvars
----------------------------------------------------------------
GENRST_RUN_COMMAND
mlr --opprint stats1 -a mean -f x data/medium
GENRST_EOF
GENRST_INCLUDE_AND_RUN_ESCAPED(data/mean-with-oosvars.sh)
Keyed mean without/with oosvars
----------------------------------------------------------------
GENRST_RUN_COMMAND
mlr --opprint stats1 -a mean -f x -g a,b data/medium
GENRST_EOF
GENRST_INCLUDE_AND_RUN_ESCAPED(data/keyed-mean-with-oosvars.sh)
Variance and standard deviation without/with oosvars
----------------------------------------------------------------
GENRST_RUN_COMMAND
mlr --oxtab stats1 -a count,sum,mean,var,stddev -f x data/medium
GENRST_EOF
GENRST_RUN_COMMAND
cat variance.mlr
GENRST_EOF
GENRST_RUN_COMMAND
mlr --oxtab put -q -f variance.mlr data/medium
GENRST_EOF
You can also do this keyed, of course, imitating the keyed-mean example above.
Min/max without/with oosvars
----------------------------------------------------------------
GENRST_RUN_COMMAND
mlr --oxtab stats1 -a min,max -f x data/medium
GENRST_EOF
GENRST_RUN_COMMAND
mlr --oxtab put -q '
@x_min = min(@x_min, $x);
@x_max = max(@x_max, $x);
end{emitf @x_min, @x_max}
' data/medium
GENRST_EOF
Keyed min/max without/with oosvars
----------------------------------------------------------------
GENRST_RUN_COMMAND
mlr --opprint stats1 -a min,max -f x -g a data/medium
GENRST_EOF
GENRST_INCLUDE_AND_RUN_ESCAPED(data/keyed-min-max-with-oosvars.sh)
Delta without/with oosvars
----------------------------------------------------------------
GENRST_RUN_COMMAND
mlr --opprint step -a delta -f x data/small
GENRST_EOF
GENRST_RUN_COMMAND
mlr --opprint put '
$x_delta = is_present(@last) ? $x - @last : 0;
@last = $x
' data/small
GENRST_EOF
Keyed delta without/with oosvars
----------------------------------------------------------------
GENRST_RUN_COMMAND
mlr --opprint step -a delta -f x -g a data/small
GENRST_EOF
GENRST_RUN_COMMAND
mlr --opprint put '
$x_delta = is_present(@last[$a]) ? $x - @last[$a] : 0;
@last[$a]=$x
' data/small
GENRST_EOF
Exponentially weighted moving averages without/with oosvars
----------------------------------------------------------------
GENRST_INCLUDE_AND_RUN_ESCAPED(verb-example-ewma.sh)
GENRST_INCLUDE_AND_RUN_ESCAPED(oosvar-example-ewma.sh)

77
docs6/data-cleaning-examples.rst Normal file
View file

@ -0,0 +1,77 @@
..
PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE.
Data-cleaning examples
================================================================
Here are some ways to use the type-checking options as described in :ref:`reference-dsl-type-tests-and-assertions` Suppose you have the following data file, with inconsistent typing for boolean. (Also imagine that, for the sake of discussion, we have a million-line file rather than a four-line file, so we can't see it all at once and some automation is called for.)
.. code-block:: none
:emphasize-lines: 1-1
cat data/het-bool.csv
name,reachable
barney,false
betty,true
fred,true
wilma,1
One option is to coerce everything to boolean, or integer:
.. code-block:: none
:emphasize-lines: 1-1
mlr --icsv --opprint put '$reachable = boolean($reachable)' data/het-bool.csv
name reachable
barney false
betty true
fred true
wilma true
.. code-block:: none
:emphasize-lines: 1-1
mlr --icsv --opprint put '$reachable = int(boolean($reachable))' data/het-bool.csv
name reachable
barney 0
betty 1
fred 1
wilma 1
A second option is to flag badly formatted data within the output stream:
.. code-block:: none
:emphasize-lines: 1-1
mlr --icsv --opprint put '$format_ok = is_string($reachable)' data/het-bool.csv
name reachable format_ok
barney false false
betty true false
fred true false
wilma 1 false
Or perhaps to flag badly formatted data outside the output stream:
.. code-block:: none
:emphasize-lines: 1-3
mlr --icsv --opprint put '
if (!is_string($reachable)) {eprint "Malformed at NR=".NR}
' data/het-bool.csv
Malformed at NR=1
Malformed at NR=2
Malformed at NR=3
Malformed at NR=4
name reachable
barney false
betty true
fred true
wilma 1
A third way is to abort the process on first instance of bad data:
.. code-block:: none
:emphasize-lines: 1-1
mlr --csv put '$reachable = asserting_string($reachable)' data/het-bool.csv
Miller: is_string type-assertion failed at NR=1 FNR=1 FILENAME=data/het-bool.csv

38
docs6/data-cleaning-examples.rst.in Normal file
View file

@ -0,0 +1,38 @@
Data-cleaning examples
================================================================
Here are some ways to use the type-checking options as described in :ref:`reference-dsl-type-tests-and-assertions` Suppose you have the following data file, with inconsistent typing for boolean. (Also imagine that, for the sake of discussion, we have a million-line file rather than a four-line file, so we can't see it all at once and some automation is called for.)
GENRST_RUN_COMMAND
cat data/het-bool.csv
GENRST_EOF
One option is to coerce everything to boolean, or integer:
GENRST_RUN_COMMAND
mlr --icsv --opprint put '$reachable = boolean($reachable)' data/het-bool.csv
GENRST_EOF
GENRST_RUN_COMMAND
mlr --icsv --opprint put '$reachable = int(boolean($reachable))' data/het-bool.csv
GENRST_EOF
A second option is to flag badly formatted data within the output stream:
GENRST_RUN_COMMAND
mlr --icsv --opprint put '$format_ok = is_string($reachable)' data/het-bool.csv
GENRST_EOF
Or perhaps to flag badly formatted data outside the output stream:
GENRST_RUN_COMMAND
mlr --icsv --opprint put '
if (!is_string($reachable)) {eprint "Malformed at NR=".NR}
' data/het-bool.csv
GENRST_EOF
A third way is to abort the process on first instance of bad data:
GENRST_RUN_COMMAND_TOLERATING_ERROR
mlr --csv put '$reachable = asserting_string($reachable)' data/het-bool.csv
GENRST_EOF

57
docs6/data/delimiter-examples.txt
View file

@ -1,57 +0,0 @@
# Use the `file` command to see if there are CR/LF terminators (in this case,
# there are not):
$ file data/colours.csv
data/colours.csv: UTF-8 Unicode text
# Look at the file to find names of fields
$ cat data/colours.csv
KEY;DE;EN;ES;FI;FR;IT;NL;PL;RO;TR
masterdata_colourcode_1;Weiß;White;Blanco;Valkoinen;Blanc;Bianco;Wit;Biały;Alb;Beyaz
masterdata_colourcode_2;Schwarz;Black;Negro;Musta;Noir;Nero;Zwart;Czarny;Negru;Siyah
# Extract a few fields:
$ mlr --csv cut -f KEY,PL,RO data/colours.csv
(only blank lines appear)
# Use XTAB output format to get a sharper picture of where records/fields
# are being split:
$ mlr --icsv --oxtab cat data/colours.csv
KEY;DE;EN;ES;FI;FR;IT;NL;PL;RO;TR masterdata_colourcode_1;Weiß;White;Blanco;Valkoinen;Blanc;Bianco;Wit;Biały;Alb;Beyaz
KEY;DE;EN;ES;FI;FR;IT;NL;PL;RO;TR masterdata_colourcode_2;Schwarz;Black;Negro;Musta;Noir;Nero;Zwart;Czarny;Negru;Siyah
# Using XTAB output format makes it clearer that KEY;DE;...;RO;TR is being
# treated as a single field name in the CSV header, and likewise each
# subsequent line is being treated as a single field value. This is because
# the default field separator is a comma but we have semicolons here.
# Use XTAB again with different field separator (--fs semicolon):
mlr --icsv --ifs semicolon --oxtab cat data/colours.csv
KEY masterdata_colourcode_1
DE Weiß
EN White
ES Blanco
FI Valkoinen
FR Blanc
IT Bianco
NL Wit
PL Biały
RO Alb
TR Beyaz
KEY masterdata_colourcode_2
DE Schwarz
EN Black
ES Negro
FI Musta
FR Noir
IT Nero
NL Zwart
PL Czarny
RO Negru
TR Siyah
# Using the new field-separator, retry the cut:
mlr --csv --fs semicolon cut -f KEY,PL,RO data/colours.csv
KEY;PL;RO
masterdata_colourcode_1;Biały;Alb
masterdata_colourcode_2;Czarny;Negru

2
docs6/cookbook4.rst → docs6/dkvp-examples.rst
View file

@ -1,7 +1,7 @@
..
PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE.
Cookbook4: to be filed
DKVP I/O examples
======================
DKVP I/O in Python

2
docs6/cookbook4.rst.in → docs6/dkvp-examples.rst.in
View file

@ -1,4 +1,4 @@
Cookbook4: to be filed
DKVP I/O examples
======================
DKVP I/O in Python

25
docs6/index.rst
View file

@ -11,7 +11,6 @@ Getting started
10min
keystroke-savers
programming-language
repl
miller-on-windows
community
@ -28,6 +27,7 @@ Miller in more detail
internationalization
output-colorization
customization
repl
new-in-miller-6
contributing
@ -37,29 +37,24 @@ FAQs and recipes
.. toctree::
:maxdepth: 1
shapes-of-data
csv-with-and-without-headers
shapes-of-data
operating-on-all-fields
special-symbols-and-formatting
dates-and-times
then-chaining
joins
shell-commands
To be refactored
----------------------------------------------------------------
.. toctree::
:maxdepth: 1
quick-examples
data-diving-examples
cookbook
cookbook2
cookbook3
cookbook4
sql-examples
log-processing-examples
sql-examples
data-cleaning-examples
statistics-examples
randomizing-examples
two-pass-algorithms
dkvp-examples
programming-examples
misc-examples
Background
----------------------------------------------------------------

2
docs6/joins.rst
View file

@ -178,5 +178,3 @@ We can run the input file through multiple ``join`` commands in a ``then``-chain
20 idle Carol mix
10 idle Bob knead
30 occupied Alice clean
.. _cookbook-data-cleaning-examples:

2
docs6/joins.rst.in
View file

@ -87,5 +87,3 @@ mlr --icsv --opprint join -f multi-join/name-lookup.csv -j id \
then join -f multi-join/status-lookup.csv -j id \
multi-join/input.csv
GENRST_EOF
.. _cookbook-data-cleaning-examples:

23
docs6/log-processing-examples.rst
View file

@ -2,10 +2,13 @@
PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE.
Log-processing examples
----------------------------------------------------------------
================================================================
Another of my favorite use-cases for Miller is doing ad-hoc processing of log-file data. Here's where DKVP format really shines: one, since the field names and field values are present on every line, every line stands on its own. That means you can ``grep`` or what have you. Also it means not every line needs to have the same list of field names ("schema").
Generating and aggregating log-file output
----------------------------------------------------------------
Again, all the examples in the CSV section apply here -- just change the input-format flags. But there's more you can do when not all the records have the same shape.
Writing a program -- in any language whatsoever -- you can have it print out log lines as it goes along, with items for various events jumbled together. After the program has finished running you can sort it all out, filter it, analyze it, and learn from it.
@ -194,3 +197,21 @@ Alternatively, we can simply group the similar data for a better look:
2016-09-02T12:35:20Z 100 554
2016-09-02T12:35:36Z 100 612
2016-09-02T12:35:42Z 100 728
Parsing log-file output
----------------------------------------------------------------
This, of course, depends highly on what's in your log files. But, as an example, suppose you have log-file lines such as
.. code-block:: none
2015-10-08 08:29:09,445 INFO com.company.path.to.ClassName @ [sometext] various/sorts/of data {& punctuation} hits=1 status=0 time=2.378
I prefer to pre-filter with ``grep`` and/or ``sed`` to extract the structured text, then hand that to Miller. Example:
.. code-block:: none
:emphasize-lines: 1-3
grep 'various sorts' *.log \
| sed 's/.*} //' \
| mlr --fs space --repifs --oxtab stats1 -a min,p10,p50,p90,max -f time -g status

22
docs6/log-processing-examples.rst.in
View file

@ -1,8 +1,11 @@
Log-processing examples
----------------------------------------------------------------
================================================================
Another of my favorite use-cases for Miller is doing ad-hoc processing of log-file data. Here's where DKVP format really shines: one, since the field names and field values are present on every line, every line stands on its own. That means you can ``grep`` or what have you. Also it means not every line needs to have the same list of field names ("schema").
Generating and aggregating log-file output
----------------------------------------------------------------
Again, all the examples in the CSV section apply here -- just change the input-format flags. But there's more you can do when not all the records have the same shape.
Writing a program -- in any language whatsoever -- you can have it print out log lines as it goes along, with items for various events jumbled together. After the program has finished running you can sort it all out, filter it, analyze it, and learn from it.
@ -28,3 +31,20 @@ GENRST_EOF
GENRST_RUN_COMMAND
mlr --opprint group-like then sec2gmt time log.txt
GENRST_EOF
Parsing log-file output
----------------------------------------------------------------
This, of course, depends highly on what's in your log files. But, as an example, suppose you have log-file lines such as
GENRST_CARDIFY
2015-10-08 08:29:09,445 INFO com.company.path.to.ClassName @ [sometext] various/sorts/of data {& punctuation} hits=1 status=0 time=2.378
GENRST_EOF
I prefer to pre-filter with ``grep`` and/or ``sed`` to extract the structured text, then hand that to Miller. Example:
GENRST_SHOW_COMMAND
grep 'various sorts' *.log \
| sed 's/.*} //' \
| mlr --fs space --repifs --oxtab stats1 -a min,p10,p50,p90,max -f time -g status
GENRST_EOF

344
docs6/misc-examples.rst Normal file
View file

@ -0,0 +1,344 @@
..
PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE.
Miscellaneous examples
================================================================
Column select:
.. code-block:: none
:emphasize-lines: 1-1
mlr --csv cut -f hostname,uptime mydata.csv
Add new columns as function of other columns:
.. code-block:: none
:emphasize-lines: 1-1
mlr --nidx put '$sum = $7 < 0.0 ? 3.5 : $7 + 2.1*$8' *.dat
Row filter:
.. code-block:: none
:emphasize-lines: 1-1
mlr --csv filter '$status != "down" && $upsec >= 10000' *.csv
Apply column labels and pretty-print:
.. code-block:: none
:emphasize-lines: 1-1
grep -v '^#' /etc/group | mlr --ifs : --nidx --opprint label group,pass,gid,member then sort -f group
Join multiple data sources on key columns:
.. code-block:: none
:emphasize-lines: 1-1
mlr join -j account_id -f accounts.dat then group-by account_name balances.dat
Mulltiple formats including JSON:
.. code-block:: none
:emphasize-lines: 1-1
mlr --json put '$attr = sub($attr, "([0-9]+)_([0-9]+)_.*", "\1:\2")' data/*.json
Aggregate per-column statistics:
.. code-block:: none
:emphasize-lines: 1-1
mlr stats1 -a min,mean,max,p10,p50,p90 -f flag,u,v data/*
Linear regression:
.. code-block:: none
:emphasize-lines: 1-1
mlr stats2 -a linreg-pca -f u,v -g shape data/*
Aggregate custom per-column statistics:
.. code-block:: none
:emphasize-lines: 1-1
mlr put -q '@sum[$a][$b] += $x; end {emit @sum, "a", "b"}' data/*
Iterate over data using DSL expressions:
.. code-block:: none
:emphasize-lines: 1-1
mlr --from estimates.tbl put '
for (k,v in $*) {
if (is_numeric(v) && k =~ "^[t-z].*$") {
$sum += v; $count += 1
}
}
$mean = $sum / $count # no assignment if count unset
'
Run DSL expressions from a script file:
.. code-block:: none
:emphasize-lines: 1-1
mlr --from infile.dat put -f analyze.mlr
Split/reduce output to multiple filenames:
.. code-block:: none
:emphasize-lines: 1-1
mlr --from infile.dat put 'tee > "./taps/data-".$a."-".$b, $*'
Compressed I/O:
.. code-block:: none
:emphasize-lines: 1-1
mlr --from infile.dat put 'tee | "gzip > ./taps/data-".$a."-".$b.".gz", $*'
Interoperate with other data-processing tools using standard pipes:
.. code-block:: none
:emphasize-lines: 1-1
mlr --from infile.dat put -q '@v=$*; dump | "jq .[]"'
Tap/trace:
.. code-block:: none
:emphasize-lines: 1-1
mlr --from infile.dat put '(NR % 1000 == 0) { print > stderr, "Checkpoint ".NR}'
Program timing
----------------------------------------------------------------
This admittedly artificial example demonstrates using Miller time and stats functions to introspectively acquire some information about Miller's own runtime. The ``delta`` function computes the difference between successive timestamps.
.. code-block:: none
$ ruby -e '10000.times{|i|puts "i=#{i+1}"}' > lines.txt
$ head -n 5 lines.txt
i=1
i=2
i=3
i=4
i=5
mlr --ofmt '%.9le' --opprint put '$t=systime()' then step -a delta -f t lines.txt | head -n 7
i t t_delta
1 1430603027.018016 1.430603027e+09
2 1430603027.018043 2.694129944e-05
3 1430603027.018048 5.006790161e-06
4 1430603027.018052 4.053115845e-06
5 1430603027.018055 2.861022949e-06
6 1430603027.018058 3.099441528e-06
mlr --ofmt '%.9le' --oxtab \
put '$t=systime()' then \
step -a delta -f t then \
filter '$i>1' then \
stats1 -a min,mean,max -f t_delta \
lines.txt
t_delta_min 2.861022949e-06
t_delta_mean 4.077508505e-06
t_delta_max 5.388259888e-05
Showing differences between successive queries
----------------------------------------------------------------
Suppose you have a database query which you run at one point in time, producing the output on the left, then again later producing the output on the right:
.. code-block:: none
:emphasize-lines: 1-1
cat data/previous_counters.csv
color,count
red,3472
blue,6838
orange,694
purple,12
.. code-block:: none
:emphasize-lines: 1-1
cat data/current_counters.csv
color,count
red,3467
orange,670
yellow,27
blue,6944
And, suppose you want to compute the differences in the counters between adjacent keys. Since the color names aren't all in the same order, nor are they all present on both sides, we can't just paste the two files side-by-side and do some column-four-minus-column-two arithmetic.
First, rename counter columns to make them distinct:
.. code-block:: none
:emphasize-lines: 1-1
mlr --csv rename count,previous_count data/previous_counters.csv > data/prevtemp.csv
.. code-block:: none
:emphasize-lines: 1-1
cat data/prevtemp.csv
color,previous_count
red,3472
blue,6838
orange,694
purple,12
.. code-block:: none
:emphasize-lines: 1-1
mlr --csv rename count,current_count data/current_counters.csv > data/currtemp.csv
.. code-block:: none
:emphasize-lines: 1-1
cat data/currtemp.csv
color,current_count
red,3467
orange,670
yellow,27
blue,6944
Then, join on the key field(s), and use unsparsify to zero-fill counters absent on one side but present on the other. Use ``--ul`` and ``--ur`` to emit unpaired records (namely, purple on the left and yellow on the right):
.. code-block:: none
:emphasize-lines: 1-5
mlr --icsv --opprint \
join -j color --ul --ur -f data/prevtemp.csv \
then unsparsify --fill-with 0 \
then put '$count_delta = $current_count - $previous_count' \
data/currtemp.csv
color previous_count current_count count_delta
red 3472 3467 -5
orange 694 670 -24
yellow 0 27 (error)
blue 6838 6944 106
purple 12 0 (error)
.. _cookbook-memoization-with-oosvars:
Memoization with out-of-stream variables
----------------------------------------------------------------
The recursive function for the Fibonacci sequence is famous for its computational complexity. Namely, using f(0)=1, f(1)=1, f(n)=f(n-1)+f(n-2) for n>=2, the evaluation tree branches left as well as right at each non-trivial level, resulting in millions or more paths to the root 0/1 nodes for larger n. This program
.. code-block:: none
mlr --ofmt '%.9lf' --opprint seqgen --start 1 --stop 28 then put '
func f(n) {
@fcount += 1; # count number of calls to the function
if (n < 2) {
return 1
} else {
return f(n-1) + f(n-2) # recurse
}
}
@fcount = 0;
$o = f($i);
$fcount = @fcount;
' then put '$seconds=systime()' then step -a delta -f seconds then cut -x -f seconds
produces output like this:
.. code-block:: none
i o fcount seconds_delta
1 1 1 0
2 2 3 0.000039101
3 3 5 0.000015974
4 5 9 0.000019073
5 8 15 0.000026941
6 13 25 0.000036955
7 21 41 0.000056028
8 34 67 0.000086069
9 55 109 0.000134945
10 89 177 0.000217915
11 144 287 0.000355959
12 233 465 0.000506163
13 377 753 0.000811815
14 610 1219 0.001297235
15 987 1973 0.001960993
16 1597 3193 0.003417969
17 2584 5167 0.006215811
18 4181 8361 0.008294106
19 6765 13529 0.012095928
20 10946 21891 0.019592047
21 17711 35421 0.031193972
22 28657 57313 0.057254076
23 46368 92735 0.080307961
24 75025 150049 0.129482031
25 121393 242785 0.213325977
26 196418 392835 0.334423065
27 317811 635621 0.605969906
28 514229 1028457 0.971235037
Note that the time it takes to evaluate the function is blowing up exponentially as the input argument increases. Using ``@``-variables, which persist across records, we can cache and reuse the results of previous computations:
.. code-block:: none
mlr --ofmt '%.9lf' --opprint seqgen --start 1 --stop 28 then put '
func f(n) {
@fcount += 1; # count number of calls to the function
if (is_present(@fcache[n])) { # cache hit
return @fcache[n]
} else { # cache miss
num rv = 1;
if (n >= 2) {
rv = f(n-1) + f(n-2) # recurse
}
@fcache[n] = rv;
return rv
}
}
@fcount = 0;
$o = f($i);
$fcount = @fcount;
' then put '$seconds=systime()' then step -a delta -f seconds then cut -x -f seconds
with output like this:
.. code-block:: none
i o fcount seconds_delta
1 1 1 0
2 2 3 0.000053883
3 3 3 0.000035048
4 5 3 0.000045061
5 8 3 0.000014067
6 13 3 0.000028849
7 21 3 0.000028133
8 34 3 0.000027895
9 55 3 0.000014067
10 89 3 0.000015020
11 144 3 0.000012875
12 233 3 0.000033140
13 377 3 0.000014067
14 610 3 0.000012875
15 987 3 0.000029087
16 1597 3 0.000013828
17 2584 3 0.000013113
18 4181 3 0.000012875
19 6765 3 0.000013113
20 10946 3 0.000012875
21 17711 3 0.000013113
22 28657 3 0.000013113
23 46368 3 0.000015974
24 75025 3 0.000012875
25 121393 3 0.000013113
26 196418 3 0.000012875
27 317811 3 0.000013113
28 514229 3 0.000012875

224
docs6/misc-examples.rst.in Normal file
View file

@ -0,0 +1,224 @@
Miscellaneous examples
================================================================
Column select:
GENRST_SHOW_COMMAND
mlr --csv cut -f hostname,uptime mydata.csv
GENRST_EOF
Add new columns as function of other columns:
GENRST_CARDIFY_HIGHLIGHT_ONE
mlr --nidx put '$sum = $7 < 0.0 ? 3.5 : $7 + 2.1*$8' *.dat
GENRST_EOF
Row filter:
GENRST_CARDIFY_HIGHLIGHT_ONE
mlr --csv filter '$status != "down" && $upsec >= 10000' *.csv
GENRST_EOF
Apply column labels and pretty-print:
GENRST_CARDIFY_HIGHLIGHT_ONE
grep -v '^#' /etc/group | mlr --ifs : --nidx --opprint label group,pass,gid,member then sort -f group
GENRST_EOF
Join multiple data sources on key columns:
GENRST_CARDIFY_HIGHLIGHT_ONE
mlr join -j account_id -f accounts.dat then group-by account_name balances.dat
GENRST_EOF
Mulltiple formats including JSON:
GENRST_CARDIFY_HIGHLIGHT_ONE
mlr --json put '$attr = sub($attr, "([0-9]+)_([0-9]+)_.*", "\1:\2")' data/*.json
GENRST_EOF
Aggregate per-column statistics:
GENRST_CARDIFY_HIGHLIGHT_ONE
mlr stats1 -a min,mean,max,p10,p50,p90 -f flag,u,v data/*
GENRST_EOF
Linear regression:
GENRST_CARDIFY_HIGHLIGHT_ONE
mlr stats2 -a linreg-pca -f u,v -g shape data/*
GENRST_EOF
Aggregate custom per-column statistics:
GENRST_CARDIFY_HIGHLIGHT_ONE
mlr put -q '@sum[$a][$b] += $x; end {emit @sum, "a", "b"}' data/*
GENRST_EOF
Iterate over data using DSL expressions:
GENRST_CARDIFY_HIGHLIGHT_ONE
mlr --from estimates.tbl put '
for (k,v in $*) {
if (is_numeric(v) && k =~ "^[t-z].*$") {
$sum += v; $count += 1
}
}
$mean = $sum / $count # no assignment if count unset
'
GENRST_EOF
Run DSL expressions from a script file:
GENRST_CARDIFY_HIGHLIGHT_ONE
mlr --from infile.dat put -f analyze.mlr
GENRST_EOF
Split/reduce output to multiple filenames:
GENRST_CARDIFY_HIGHLIGHT_ONE
mlr --from infile.dat put 'tee > "./taps/data-".$a."-".$b, $*'
GENRST_EOF
Compressed I/O:
GENRST_CARDIFY_HIGHLIGHT_ONE
mlr --from infile.dat put 'tee | "gzip > ./taps/data-".$a."-".$b.".gz", $*'
GENRST_EOF
Interoperate with other data-processing tools using standard pipes:
GENRST_CARDIFY_HIGHLIGHT_ONE
mlr --from infile.dat put -q '@v=$*; dump | "jq .[]"'
GENRST_EOF
Tap/trace:
GENRST_CARDIFY_HIGHLIGHT_ONE
mlr --from infile.dat put '(NR % 1000 == 0) { print > stderr, "Checkpoint ".NR}'
GENRST_EOF
Program timing
----------------------------------------------------------------
This admittedly artificial example demonstrates using Miller time and stats functions to introspectively acquire some information about Miller's own runtime. The ``delta`` function computes the difference between successive timestamps.
GENRST_INCLUDE_ESCAPED(data/timing-example.txt)
Showing differences between successive queries
----------------------------------------------------------------
Suppose you have a database query which you run at one point in time, producing the output on the left, then again later producing the output on the right:
GENRST_RUN_COMMAND
cat data/previous_counters.csv
GENRST_EOF
GENRST_RUN_COMMAND
cat data/current_counters.csv
GENRST_EOF
And, suppose you want to compute the differences in the counters between adjacent keys. Since the color names aren't all in the same order, nor are they all present on both sides, we can't just paste the two files side-by-side and do some column-four-minus-column-two arithmetic.
First, rename counter columns to make them distinct:
GENRST_RUN_COMMAND
mlr --csv rename count,previous_count data/previous_counters.csv > data/prevtemp.csv
GENRST_EOF
GENRST_RUN_COMMAND
cat data/prevtemp.csv
GENRST_EOF
GENRST_RUN_COMMAND
mlr --csv rename count,current_count data/current_counters.csv > data/currtemp.csv
GENRST_EOF
GENRST_RUN_COMMAND
cat data/currtemp.csv
GENRST_EOF
Then, join on the key field(s), and use unsparsify to zero-fill counters absent on one side but present on the other. Use ``--ul`` and ``--ur`` to emit unpaired records (namely, purple on the left and yellow on the right):
GENRST_INCLUDE_AND_RUN_ESCAPED(data/previous-to-current.sh)
.. _cookbook-memoization-with-oosvars:
Memoization with out-of-stream variables
----------------------------------------------------------------
The recursive function for the Fibonacci sequence is famous for its computational complexity. Namely, using f(0)=1, f(1)=1, f(n)=f(n-1)+f(n-2) for n>=2, the evaluation tree branches left as well as right at each non-trivial level, resulting in millions or more paths to the root 0/1 nodes for larger n. This program
GENRST_INCLUDE_ESCAPED(data/fibo-uncached.sh)
produces output like this:
GENRST_CARDIFY
i o fcount seconds_delta
1 1 1 0
2 2 3 0.000039101
3 3 5 0.000015974
4 5 9 0.000019073
5 8 15 0.000026941
6 13 25 0.000036955
7 21 41 0.000056028
8 34 67 0.000086069
9 55 109 0.000134945
10 89 177 0.000217915
11 144 287 0.000355959
12 233 465 0.000506163
13 377 753 0.000811815
14 610 1219 0.001297235
15 987 1973 0.001960993
16 1597 3193 0.003417969
17 2584 5167 0.006215811
18 4181 8361 0.008294106
19 6765 13529 0.012095928
20 10946 21891 0.019592047
21 17711 35421 0.031193972
22 28657 57313 0.057254076
23 46368 92735 0.080307961
24 75025 150049 0.129482031
25 121393 242785 0.213325977
26 196418 392835 0.334423065
27 317811 635621 0.605969906
28 514229 1028457 0.971235037
GENRST_EOF
Note that the time it takes to evaluate the function is blowing up exponentially as the input argument increases. Using ``@``-variables, which persist across records, we can cache and reuse the results of previous computations:
GENRST_INCLUDE_ESCAPED(data/fibo-cached.sh)
with output like this:
GENRST_CARDIFY
i o fcount seconds_delta
1 1 1 0
2 2 3 0.000053883
3 3 3 0.000035048
4 5 3 0.000045061
5 8 3 0.000014067
6 13 3 0.000028849
7 21 3 0.000028133
8 34 3 0.000027895
9 55 3 0.000014067
10 89 3 0.000015020
11 144 3 0.000012875
12 233 3 0.000033140
13 377 3 0.000014067
14 610 3 0.000012875
15 987 3 0.000029087
16 1597 3 0.000013828
17 2584 3 0.000013113
18 4181 3 0.000012875
19 6765 3 0.000013113
20 10946 3 0.000012875
21 17711 3 0.000013113
22 28657 3 0.000013113
23 46368 3 0.000015974
24 75025 3 0.000012875
25 121393 3 0.000013113
26 196418 3 0.000012875
27 317811 3 0.000013113
28 514229 3 0.000012875
GENRST_EOF

261
docs6/programming-examples.rst Normal file
View file

@ -0,0 +1,261 @@
..
PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE.
Programming-language examples
================================================================
Here are a few things focusing on Miller's DSL as a programming language per se, outside of its normal use for streaming record-processing.
Sieve of Eratosthenes
----------------------------------------------------------------
The `Sieve of Eratosthenes <http://en.wikipedia.org/wiki/Sieve_of_Eratosthenes>`_ is a standard introductory programming topic. The idea is to find all primes up to some *N* by making a list of the numbers 1 to *N*, then striking out all multiples of 2 except 2 itself, all multiples of 3 except 3 itself, all multiples of 4 except 4 itself, and so on. Whatever survives that without getting marked is a prime. This is easy enough in Miller. Notice that here all the work is in ``begin`` and ``end`` statements; there is no file input (so we use ``mlr -n`` to keep Miller from waiting for input data).
.. code-block:: none
:emphasize-lines: 1-1
cat programs/sieve.mlr
# ================================================================
# Sieve of Eratosthenes: simple example of Miller DSL as programming language.
# ================================================================
# Put this in a begin-block so we can do either
# mlr -n put -q -f name-of-this-file.mlr
# or
# mlr -n put -q -f name-of-this-file.mlr -e '@n = 200'
# i.e. 100 is the default upper limit, and another can be specified using -e.
begin {
@n = 100;
}
end {
for (int i = 0; i <= @n; i += 1) {
@s[i] = true;
}
@s[0] = false; # 0 is neither prime nor composite
@s[1] = false; # 1 is neither prime nor composite
# Strike out multiples
for (int i = 2; i <= @n; i += 1) {
for (int j = i+i; j <= @n; j += i) {
@s[j] = false;
}
}
# Print survivors
for (int i = 0; i <= @n; i += 1) {
if (@s[i]) {
print i;
}
}
}
.. code-block:: none
:emphasize-lines: 1-1
mlr -n put -f programs/sieve.mlr
2
3
5
7
11
13
17
19
23
29
31
37
41
43
47
53
59
61
67
71
73
79
83
89
97
Mandelbrot-set generator
----------------------------------------------------------------
The `Mandelbrot set <http://en.wikipedia.org/wiki/Mandelbrot_set>`_ is also easily expressed. This isn't an important case of data-processing in the vein for which Miller was designed, but it is an example of Miller as a general-purpose programming language -- a test case for the expressiveness of the language.
The (approximate) computation of points in the complex plane which are and aren't members is just a few lines of complex arithmetic (see the Wikipedia article); how to render them is another task. Using graphics libraries you can create PNG or JPEG files, but another fun way to do this is by printing various characters to the screen:
.. code-block:: none
:emphasize-lines: 1-1
cat programs/mand.mlr
# Mandelbrot set generator: simple example of Miller DSL as programming language.
begin {
# Set defaults
@rcorn = -2.0;
@icorn = -2.0;
@side = 4.0;
@iheight = 50;
@iwidth = 100;
@maxits = 100;
@levelstep = 5;
@chars = "@X*o-."; # Palette of characters to print to the screen.
@verbose = false;
@do_julia = false;
@jr = 0.0; # Real part of Julia point, if any
@ji = 0.0; # Imaginary part of Julia point, if any
}
# Here, we can override defaults from an input file (if any). In Miller's
# put/filter DSL, absent-null right-hand sides result in no assignment so we
# can simply put @rcorn = $rcorn: if there is a field in the input like
# 'rcorn = -1.847' we'll read and use it, else we'll keep the default.
@rcorn = $rcorn;
@icorn = $icorn;
@side = $side;
@iheight = $iheight;
@iwidth = $iwidth;
@maxits = $maxits;
@levelstep = $levelstep;
@chars = $chars;
@verbose = $verbose;
@do_julia = $do_julia;
@jr = $jr;
@ji = $ji;
end {
if (@verbose) {
print "RCORN = ".@rcorn;
print "ICORN = ".@icorn;
print "SIDE = ".@side;
print "IHEIGHT = ".@iheight;
print "IWIDTH = ".@iwidth;
print "MAXITS = ".@maxits;
print "LEVELSTEP = ".@levelstep;
print "CHARS = ".@chars;
}
# Iterate over a matrix of rows and columns, printing one character for each cell.
for (int ii = @iheight-1; ii >= 0; ii -= 1) {
num pi = @icorn + (ii/@iheight) * @side;
for (int ir = 0; ir < @iwidth; ir += 1) {
num pr = @rcorn + (ir/@iwidth) * @side;
printn get_point_plot(pr, pi, @maxits, @do_julia, @jr, @ji);
}
print;
}
}
# This is a function to approximate membership in the Mandelbrot set (or Julia
# set for a given Julia point if do_julia == true) for a given point in the
# complex plane.
func get_point_plot(pr, pi, maxits, do_julia, jr, ji) {
num zr = 0.0;
num zi = 0.0;
num cr = 0.0;
num ci = 0.0;
if (!do_julia) {
zr = 0.0;
zi = 0.0;
cr = pr;
ci = pi;
} else {
zr = pr;
zi = pi;
cr = jr;
ci = ji;
}
int iti = 0;
bool escaped = false;
num zt = 0;
for (iti = 0; iti < maxits; iti += 1) {
num mag = zr*zr + zi+zi;
if (mag > 4.0) {
escaped = true;
break;
}
# z := z^2 + c
zt = zr*zr - zi*zi + cr;
zi = 2*zr*zi + ci;
zr = zt;
}
if (!escaped) {
return ".";
} else {
# The // operator is Miller's (pythonic) integer-division operator
int level = (iti // @levelstep) % strlen(@chars);
return substr(@chars, level, level);
}
}
At standard resolution this makes a nice little ASCII plot:
.. code-block:: none
:emphasize-lines: 1-1
mlr -n put -f ./programs/mand.mlr
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXX.XXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXooXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXX**o..*XXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXX*-....-oXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXX@XXXXXXXXXX*......o*XXXXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXXX**oo*-.-........oo.XXXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXXXXXX....................X..o-XXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@XXXXXXXXXXXXXXX*oo......................oXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@XXX*XXXXXXXXXXXX**o........................*X*X@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@XXXXXXooo***o*.*XX**X..........................o-XX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@XXXXXXXX*-.......-***.............................oXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@XXXXXXXX*@..........Xo............................*XX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@XXXX@XXXXXXXX*o@oX...........@...........................oXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
.........................................................o*XXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@XXXXXXXXX*-.oX...........@...........................oXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@XXXXXXXXXX**@..........*o............................*XXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@XXXXXXXXXXXXX-........***.............................oXXXXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@XXXXXXXXXXXXoo****o*.XX***@..........................o-XXXXXXXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@XXXXX*XXXX*XXXXXXX**-........................***XXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@XXXXXXXXXXXXX*o*.....................@o*XXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXXXXX*....................*..o-XX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXX*ooo*-.o........oo.X*XXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXX**@.....*XXXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXX*o....-o*XXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXXXo*o..*XXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXXXXXX*o*XXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXXXXX@XXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXX@@XXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
But using a very small font size (as small as my Mac will let me go), and by choosing the coordinates to zoom in on a particular part of the complex plane, we can get a nice little picture:
.. code-block:: none
#!/bin/bash
# Get the number of rows and columns from the terminal window dimensions
iheight=$(stty size | mlr --nidx --fs space cut -f 1)
iwidth=$(stty size | mlr --nidx --fs space cut -f 2)
echo "rcorn=-1.755350,icorn=+0.014230,side=0.000020,maxits=10000,iheight=$iheight,iwidth=$iwidth" \
| mlr put -f programs/mand.mlr
.. image:: pix/mand.png

47
docs6/programming-examples.rst.in Normal file
View file

@ -0,0 +1,47 @@
Programming-language examples
================================================================
Here are a few things focusing on Miller's DSL as a programming language per se, outside of its normal use for streaming record-processing.
Sieve of Eratosthenes
----------------------------------------------------------------
The `Sieve of Eratosthenes <http://en.wikipedia.org/wiki/Sieve_of_Eratosthenes>`_ is a standard introductory programming topic. The idea is to find all primes up to some *N* by making a list of the numbers 1 to *N*, then striking out all multiples of 2 except 2 itself, all multiples of 3 except 3 itself, all multiples of 4 except 4 itself, and so on. Whatever survives that without getting marked is a prime. This is easy enough in Miller. Notice that here all the work is in ``begin`` and ``end`` statements; there is no file input (so we use ``mlr -n`` to keep Miller from waiting for input data).
GENRST_RUN_COMMAND
cat programs/sieve.mlr
GENRST_EOF
GENRST_RUN_COMMAND
mlr -n put -f programs/sieve.mlr
GENRST_EOF
Mandelbrot-set generator
----------------------------------------------------------------
The `Mandelbrot set <http://en.wikipedia.org/wiki/Mandelbrot_set>`_ is also easily expressed. This isn't an important case of data-processing in the vein for which Miller was designed, but it is an example of Miller as a general-purpose programming language -- a test case for the expressiveness of the language.
The (approximate) computation of points in the complex plane which are and aren't members is just a few lines of complex arithmetic (see the Wikipedia article); how to render them is another task. Using graphics libraries you can create PNG or JPEG files, but another fun way to do this is by printing various characters to the screen:
GENRST_RUN_COMMAND
cat programs/mand.mlr
GENRST_EOF
At standard resolution this makes a nice little ASCII plot:
GENRST_RUN_COMMAND
mlr -n put -f ./programs/mand.mlr
GENRST_EOF
But using a very small font size (as small as my Mac will let me go), and by choosing the coordinates to zoom in on a particular part of the complex plane, we can get a nice little picture:
GENRST_CARDIFY
#!/bin/bash
# Get the number of rows and columns from the terminal window dimensions
iheight=$(stty size | mlr --nidx --fs space cut -f 1)
iwidth=$(stty size | mlr --nidx --fs space cut -f 2)
echo "rcorn=-1.755350,icorn=+0.014230,side=0.000020,maxits=10000,iheight=$iheight,iwidth=$iwidth" \
| mlr put -f programs/mand.mlr
GENRST_EOF
.. image:: pix/mand.png

117
docs6/quick-examples.rst
View file

@ -1,117 +0,0 @@
..
PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE.
Quick examples
================================================================
Column select:
.. code-block:: none
:emphasize-lines: 1-1
mlr --csv cut -f hostname,uptime mydata.csv
Add new columns as function of other columns:
.. code-block:: none
:emphasize-lines: 1-1
mlr --nidx put '$sum = $7 < 0.0 ? 3.5 : $7 + 2.1*$8' *.dat
Row filter:
.. code-block:: none
:emphasize-lines: 1-1
mlr --csv filter '$status != "down" && $upsec >= 10000' *.csv
Apply column labels and pretty-print:
.. code-block:: none
:emphasize-lines: 1-1
grep -v '^#' /etc/group | mlr --ifs : --nidx --opprint label group,pass,gid,member then sort -f group
Join multiple data sources on key columns:
.. code-block:: none
:emphasize-lines: 1-1
mlr join -j account_id -f accounts.dat then group-by account_name balances.dat
Mulltiple formats including JSON:
.. code-block:: none
:emphasize-lines: 1-1
mlr --json put '$attr = sub($attr, "([0-9]+)_([0-9]+)_.*", "\1:\2")' data/*.json
Aggregate per-column statistics:
.. code-block:: none
:emphasize-lines: 1-1
mlr stats1 -a min,mean,max,p10,p50,p90 -f flag,u,v data/*
Linear regression:
.. code-block:: none
:emphasize-lines: 1-1
mlr stats2 -a linreg-pca -f u,v -g shape data/*
Aggregate custom per-column statistics:
.. code-block:: none
:emphasize-lines: 1-1
mlr put -q '@sum[$a][$b] += $x; end {emit @sum, "a", "b"}' data/*
Iterate over data using DSL expressions:
.. code-block:: none
:emphasize-lines: 1-1
mlr --from estimates.tbl put '
for (k,v in $*) {
if (is_numeric(v) && k =~ "^[t-z].*$") {
$sum += v; $count += 1
}
}
$mean = $sum / $count # no assignment if count unset
'
Run DSL expressions from a script file:
.. code-block:: none
:emphasize-lines: 1-1
mlr --from infile.dat put -f analyze.mlr
Split/reduce output to multiple filenames:
.. code-block:: none
:emphasize-lines: 1-1
mlr --from infile.dat put 'tee > "./taps/data-".$a."-".$b, $*'
Compressed I/O:
.. code-block:: none
:emphasize-lines: 1-1
mlr --from infile.dat put 'tee | "gzip > ./taps/data-".$a."-".$b.".gz", $*'
Interoperate with other data-processing tools using standard pipes:
.. code-block:: none
:emphasize-lines: 1-1
mlr --from infile.dat put -q '@v=$*; dump | "jq .[]"'
Tap/trace:
.. code-block:: none
:emphasize-lines: 1-1
mlr --from infile.dat put '(NR % 1000 == 0) { print > stderr, "Checkpoint ".NR}'

99
docs6/quick-examples.rst.in
View file

@ -1,99 +0,0 @@
Quick examples
================================================================
Column select:
GENRST_SHOW_COMMAND
mlr --csv cut -f hostname,uptime mydata.csv
GENRST_EOF
Add new columns as function of other columns:
GENRST_CARDIFY_HIGHLIGHT_ONE
mlr --nidx put '$sum = $7 < 0.0 ? 3.5 : $7 + 2.1*$8' *.dat
GENRST_EOF
Row filter:
GENRST_CARDIFY_HIGHLIGHT_ONE
mlr --csv filter '$status != "down" && $upsec >= 10000' *.csv
GENRST_EOF
Apply column labels and pretty-print:
GENRST_CARDIFY_HIGHLIGHT_ONE
grep -v '^#' /etc/group | mlr --ifs : --nidx --opprint label group,pass,gid,member then sort -f group
GENRST_EOF
Join multiple data sources on key columns:
GENRST_CARDIFY_HIGHLIGHT_ONE
mlr join -j account_id -f accounts.dat then group-by account_name balances.dat
GENRST_EOF
Mulltiple formats including JSON:
GENRST_CARDIFY_HIGHLIGHT_ONE
mlr --json put '$attr = sub($attr, "([0-9]+)_([0-9]+)_.*", "\1:\2")' data/*.json
GENRST_EOF
Aggregate per-column statistics:
GENRST_CARDIFY_HIGHLIGHT_ONE
mlr stats1 -a min,mean,max,p10,p50,p90 -f flag,u,v data/*
GENRST_EOF
Linear regression:
GENRST_CARDIFY_HIGHLIGHT_ONE
mlr stats2 -a linreg-pca -f u,v -g shape data/*
GENRST_EOF
Aggregate custom per-column statistics:
GENRST_CARDIFY_HIGHLIGHT_ONE
mlr put -q '@sum[$a][$b] += $x; end {emit @sum, "a", "b"}' data/*
GENRST_EOF
Iterate over data using DSL expressions:
GENRST_CARDIFY_HIGHLIGHT_ONE
mlr --from estimates.tbl put '
for (k,v in $*) {
if (is_numeric(v) && k =~ "^[t-z].*$") {
$sum += v; $count += 1
}
}
$mean = $sum / $count # no assignment if count unset
'
GENRST_EOF
Run DSL expressions from a script file:
GENRST_CARDIFY_HIGHLIGHT_ONE
mlr --from infile.dat put -f analyze.mlr
GENRST_EOF
Split/reduce output to multiple filenames:
GENRST_CARDIFY_HIGHLIGHT_ONE
mlr --from infile.dat put 'tee > "./taps/data-".$a."-".$b, $*'
GENRST_EOF
Compressed I/O:
GENRST_CARDIFY_HIGHLIGHT_ONE
mlr --from infile.dat put 'tee | "gzip > ./taps/data-".$a."-".$b.".gz", $*'
GENRST_EOF
Interoperate with other data-processing tools using standard pipes:
GENRST_CARDIFY_HIGHLIGHT_ONE
mlr --from infile.dat put -q '@v=$*; dump | "jq .[]"'
GENRST_EOF
Tap/trace:
GENRST_CARDIFY_HIGHLIGHT_ONE
mlr --from infile.dat put '(NR % 1000 == 0) { print > stderr, "Checkpoint ".NR}'
GENRST_EOF

174
docs6/randomizing-examples.rst Normal file
View file

@ -0,0 +1,174 @@
..
PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE.
Randomizing examples
================================================================
Generating random numbers from various distributions
----------------------------------------------------------------
Here we can chain together a few simple building blocks:
.. code-block:: none
:emphasize-lines: 1-1
cat expo-sample.sh
# Generate 100,000 pairs of independent and identically distributed
# exponentially distributed random variables with the same rate parameter
# (namely, 2.5). Then compute histograms of one of them, along with
# histograms for their sum and their product.
#
# See also https://en.wikipedia.org/wiki/Exponential_distribution
#
# Here I'm using a specified random-number seed so this example always
# produces the same output for this web document: in everyday practice we
# wouldn't do that.
mlr -n \
--seed 0 \
--opprint \
seqgen --stop 100000 \
then put '
# https://en.wikipedia.org/wiki/Inverse_transform_sampling
func expo_sample(lambda) {
return -log(1-urand())/lambda
}
$u = expo_sample(2.5);
$v = expo_sample(2.5);
$s = $u + $v;
$p = $u * $v;
' \
then histogram -f u,s,p --lo 0 --hi 2 --nbins 50 \
then bar -f u_count,s_count,p_count --auto -w 20
Namely:
* Set the Miller random-number seed so this webdoc looks the same every time I regenerate it.
* Use pretty-printed tabular output.
* Use pretty-printed tabular output.
* Use ``seqgen`` to produce 100,000 records ``i=0``, ``i=1``, etc.
* Send those to a ``put`` step which defines an inverse-transform-sampling function and calls it twice, then computes the sum and product of samples.
* Send those to a histogram, and from there to a bar-plotter. This is just for visualization; you could just as well output CSV and send that off to your own plotting tool, etc.
The output is as follows:
.. code-block:: none
:emphasize-lines: 1-1
sh expo-sample.sh
bin_lo bin_hi u_count s_count p_count
0 0.04 [64]*******************#[9554] [326]#...................[3703] [19]*******************#[39809]
0.04 0.08 [64]*****************...[9554] [326]*****...............[3703] [19]*******.............[39809]
0.08 0.12 [64]****************....[9554] [326]*********...........[3703] [19]****................[39809]
0.12 0.16 [64]**************......[9554] [326]************........[3703] [19]***.................[39809]
0.16 0.2 [64]*************.......[9554] [326]**************......[3703] [19]**..................[39809]
0.2 0.24 [64]************........[9554] [326]*****************...[3703] [19]*...................[39809]
0.24 0.28 [64]**********..........[9554] [326]******************..[3703] [19]*...................[39809]
0.28 0.32 [64]*********...........[9554] [326]******************..[3703] [19]*...................[39809]
0.32 0.36 [64]********............[9554] [326]*******************.[3703] [19]#...................[39809]
0.36 0.4 [64]*******.............[9554] [326]*******************#[3703] [19]#...................[39809]
0.4 0.44 [64]*******.............[9554] [326]*******************.[3703] [19]#...................[39809]
0.44 0.48 [64]******..............[9554] [326]*******************.[3703] [19]#...................[39809]
0.48 0.52 [64]*****...............[9554] [326]******************..[3703] [19]#...................[39809]
0.52 0.56 [64]*****...............[9554] [326]******************..[3703] [19]#...................[39809]
0.56 0.6 [64]****................[9554] [326]*****************...[3703] [19]#...................[39809]
0.6 0.64 [64]****................[9554] [326]******************..[3703] [19]#...................[39809]
0.64 0.68 [64]***.................[9554] [326]****************....[3703] [19]#...................[39809]
0.68 0.72 [64]***.................[9554] [326]****************....[3703] [19]#...................[39809]
0.72 0.76 [64]***.................[9554] [326]***************.....[3703] [19]#...................[39809]
0.76 0.8 [64]**..................[9554] [326]**************......[3703] [19]#...................[39809]
0.8 0.84 [64]**..................[9554] [326]*************.......[3703] [19]#...................[39809]
0.84 0.88 [64]**..................[9554] [326]************........[3703] [19]#...................[39809]
0.88 0.92 [64]**..................[9554] [326]************........[3703] [19]#...................[39809]
0.92 0.96 [64]*...................[9554] [326]***********.........[3703] [19]#...................[39809]
0.96 1 [64]*...................[9554] [326]**********..........[3703] [19]#...................[39809]
1 1.04 [64]*...................[9554] [326]*********...........[3703] [19]#...................[39809]
1.04 1.08 [64]*...................[9554] [326]********............[3703] [19]#...................[39809]
1.08 1.12 [64]*...................[9554] [326]********............[3703] [19]#...................[39809]
1.12 1.16 [64]*...................[9554] [326]********............[3703] [19]#...................[39809]
1.16 1.2 [64]*...................[9554] [326]*******.............[3703] [19]#...................[39809]
1.2 1.24 [64]#...................[9554] [326]******..............[3703] [19]#...................[39809]
1.24 1.28 [64]#...................[9554] [326]*****...............[3703] [19]#...................[39809]
1.28 1.32 [64]#...................[9554] [326]*****...............[3703] [19]#...................[39809]
1.32 1.36 [64]#...................[9554] [326]****................[3703] [19]#...................[39809]
1.36 1.4 [64]#...................[9554] [326]****................[3703] [19]#...................[39809]
1.4 1.44 [64]#...................[9554] [326]****................[3703] [19]#...................[39809]
1.44 1.48 [64]#...................[9554] [326]***.................[3703] [19]#...................[39809]
1.48 1.52 [64]#...................[9554] [326]***.................[3703] [19]#...................[39809]
1.52 1.56 [64]#...................[9554] [326]***.................[3703] [19]#...................[39809]
1.56 1.6 [64]#...................[9554] [326]**..................[3703] [19]#...................[39809]
1.6 1.64 [64]#...................[9554] [326]**..................[3703] [19]#...................[39809]
1.64 1.68 [64]#...................[9554] [326]**..................[3703] [19]#...................[39809]
1.68 1.72 [64]#...................[9554] [326]*...................[3703] [19]#...................[39809]
1.72 1.76 [64]#...................[9554] [326]*...................[3703] [19]#...................[39809]
1.76 1.8 [64]#...................[9554] [326]*...................[3703] [19]#...................[39809]
1.8 1.84 [64]#...................[9554] [326]#...................[3703] [19]#...................[39809]
1.84 1.88 [64]#...................[9554] [326]#...................[3703] [19]#...................[39809]
1.88 1.92 [64]#...................[9554] [326]#...................[3703] [19]#...................[39809]
1.92 1.96 [64]#...................[9554] [326]#...................[3703] [19]#...................[39809]
1.96 2 [64]#...................[9554] [326]#...................[3703] [19]#...................[39809]
Randomly selecting words from a list
----------------------------------------------------------------
Given this `word list <./data/english-words.txt>`_, first take a look to see what the first few lines look like:
.. code-block:: none
:emphasize-lines: 1-1
head data/english-words.txt
a
aa
aal
aalii
aam
aardvark
aardwolf
aba
abac
abaca
Then the following will randomly sample ten words with four to eight characters in them:
.. code-block:: none
:emphasize-lines: 1-1
mlr --from data/english-words.txt --nidx filter -S 'n=strlen($1);4<=n&&n<=8' then sample -k 10
thionine
birchman
mildewy
avigate
addedly
abaze
askant
aiming
insulant
coinmate
Randomly generating jabberwocky words
----------------------------------------------------------------
These are simple *n*-grams as `described here <http://johnkerl.org/randspell/randspell-slides-ts.pdf>`_. Some common functions are `located here <https://github.com/johnkerl/miller/blob/master/docs/ngrams/ngfuncs.mlr.txt>`_. Then here are scripts for `1-grams <https://github.com/johnkerl/miller/blob/master/docs/ngrams/ng1.mlr.txt>`_ `2-grams <https://github.com/johnkerl/miller/blob/master/docs/ngrams/ng2.mlr.txt>`_ `3-grams <https://github.com/johnkerl/miller/blob/master/docs/ngrams/ng3.mlr.txt>`_ `4-grams <https://github.com/johnkerl/miller/blob/master/docs/ngrams/ng4.mlr.txt>`_, and `5-grams <https://github.com/johnkerl/miller/blob/master/docs/ngrams/ng5.mlr.txt>`_.
The idea is that words from the input file are consumed, then taken apart and pasted back together in ways which imitate the letter-to-letter transitions found in the word list -- giving us automatically generated words in the same vein as *bromance* and *spork*:
.. code-block:: none
:emphasize-lines: 1-1
mlr --nidx --from ./ngrams/gsl-2000.txt put -q -f ./ngrams/ngfuncs.mlr -f ./ngrams/ng5.mlr
beard
plastinguish
politicially
noise
loan
country
controductionary
suppery
lose
lessors
dollar
judge
rottendence
lessenger
diffendant
suggestional

88
docs6/randomizing-examples.rst.in Normal file
View file

@ -0,0 +1,88 @@
Randomizing examples
================================================================
Generating random numbers from various distributions
----------------------------------------------------------------
Here we can chain together a few simple building blocks:
GENRST_RUN_COMMAND
cat expo-sample.sh
GENRST_EOF
Namely:
* Set the Miller random-number seed so this webdoc looks the same every time I regenerate it.
* Use pretty-printed tabular output.
* Use pretty-printed tabular output.
* Use ``seqgen`` to produce 100,000 records ``i=0``, ``i=1``, etc.
* Send those to a ``put`` step which defines an inverse-transform-sampling function and calls it twice, then computes the sum and product of samples.
* Send those to a histogram, and from there to a bar-plotter. This is just for visualization; you could just as well output CSV and send that off to your own plotting tool, etc.
The output is as follows:
GENRST_RUN_COMMAND
sh expo-sample.sh
GENRST_EOF
Randomly selecting words from a list
----------------------------------------------------------------
Given this `word list <./data/english-words.txt>`_, first take a look to see what the first few lines look like:
GENRST_CARDIFY_HIGHLIGHT_ONE
head data/english-words.txt
a
aa
aal
aalii
aam
aardvark
aardwolf
aba
abac
abaca
GENRST_EOF
Then the following will randomly sample ten words with four to eight characters in them:
GENRST_CARDIFY_HIGHLIGHT_ONE
mlr --from data/english-words.txt --nidx filter -S 'n=strlen($1);4<=n&&n<=8' then sample -k 10
thionine
birchman
mildewy
avigate
addedly
abaze
askant
aiming
insulant
coinmate
GENRST_EOF
Randomly generating jabberwocky words
----------------------------------------------------------------
These are simple *n*-grams as `described here <http://johnkerl.org/randspell/randspell-slides-ts.pdf>`_. Some common functions are `located here <https://github.com/johnkerl/miller/blob/master/docs/ngrams/ngfuncs.mlr.txt>`_. Then here are scripts for `1-grams <https://github.com/johnkerl/miller/blob/master/docs/ngrams/ng1.mlr.txt>`_ `2-grams <https://github.com/johnkerl/miller/blob/master/docs/ngrams/ng2.mlr.txt>`_ `3-grams <https://github.com/johnkerl/miller/blob/master/docs/ngrams/ng3.mlr.txt>`_ `4-grams <https://github.com/johnkerl/miller/blob/master/docs/ngrams/ng4.mlr.txt>`_, and `5-grams <https://github.com/johnkerl/miller/blob/master/docs/ngrams/ng5.mlr.txt>`_.
The idea is that words from the input file are consumed, then taken apart and pasted back together in ways which imitate the letter-to-letter transitions found in the word list -- giving us automatically generated words in the same vein as *bromance* and *spork*:
GENRST_CARDIFY_HIGHLIGHT_ONE
mlr --nidx --from ./ngrams/gsl-2000.txt put -q -f ./ngrams/ngfuncs.mlr -f ./ngrams/ng5.mlr
beard
plastinguish
politicially
noise
loan
country
controductionary
suppery
lose
lessors
dollar
judge
rottendence
lessenger
diffendant
suggestional
GENRST_EOF

2
docs6/reference-dsl-variables.rst
View file

@ -656,7 +656,7 @@ The following ``is...`` functions take a value and return a boolean indicating w
asserting_present
asserting_string
Please see :ref:`cookbook-data-cleaning-examples` for examples of how to use these.
See :doc:`data-cleaning-examples` for examples of how to use these.
Type-declarations for local variables, function parameter, and function return values
...............................................................................................

2
docs6/reference-dsl-variables.rst.in
View file

@ -262,7 +262,7 @@ GENRST_RUN_COMMAND
mlr -f | grep ^assert
GENRST_EOF
Please see :ref:`cookbook-data-cleaning-examples` for examples of how to use these.
See :doc:`data-cleaning-examples` for examples of how to use these.
Type-declarations for local variables, function parameter, and function return values
...............................................................................................

66
docs6/shapes-of-data.rst
View file

@ -21,36 +21,48 @@ Also try ``od -xcv`` and/or ``cat -e`` on your file to check for non-printable c
Diagnosing delimiter specifications
----------------------------------------------------------------
.. code-block:: none
Use the ``file`` command to see if there are CR/LF terminators (in this case, # there are not):
# Use the `file` command to see if there are CR/LF terminators (in this case,
# there are not):
$ file data/colours.csv
.. code-block:: none
:emphasize-lines: 1-1
file data/colours.csv
data/colours.csv: UTF-8 Unicode text
# Look at the file to find names of fields
$ cat data/colours.csv
Look at the file to find names of fields
.. code-block:: none
:emphasize-lines: 1-1
cat data/colours.csv
KEY;DE;EN;ES;FI;FR;IT;NL;PL;RO;TR
masterdata_colourcode_1;Weiß;White;Blanco;Valkoinen;Blanc;Bianco;Wit;Biały;Alb;Beyaz
masterdata_colourcode_2;Schwarz;Black;Negro;Musta;Noir;Nero;Zwart;Czarny;Negru;Siyah
# Extract a few fields:
$ mlr --csv cut -f KEY,PL,RO data/colours.csv
Extract a few fields:
.. code-block:: none
:emphasize-lines: 1-1
mlr --csv cut -f KEY,PL,RO data/colours.csv
(only blank lines appear)
# Use XTAB output format to get a sharper picture of where records/fields
# are being split:
$ mlr --icsv --oxtab cat data/colours.csv
Use XTAB output format to get a sharper picture of where records/fields are being split:
.. code-block:: none
:emphasize-lines: 1-1
mlr --icsv --oxtab cat data/colours.csv
KEY;DE;EN;ES;FI;FR;IT;NL;PL;RO;TR masterdata_colourcode_1;Weiß;White;Blanco;Valkoinen;Blanc;Bianco;Wit;Biały;Alb;Beyaz
KEY;DE;EN;ES;FI;FR;IT;NL;PL;RO;TR masterdata_colourcode_2;Schwarz;Black;Negro;Musta;Noir;Nero;Zwart;Czarny;Negru;Siyah
# Using XTAB output format makes it clearer that KEY;DE;...;RO;TR is being
# treated as a single field name in the CSV header, and likewise each
# subsequent line is being treated as a single field value. This is because
# the default field separator is a comma but we have semicolons here.
# Use XTAB again with different field separator (--fs semicolon):
mlr --icsv --ifs semicolon --oxtab cat data/colours.csv
Using XTAB output format makes it clearer that ``KEY;DE;...;RO;TR`` is being treated as a single field name in the CSV header, and likewise each subsequent line is being treated as a single field value. This is because the default field separator is a comma but we have semicolons here. Use XTAB again with different field separator (``--fs semicolon``):
.. code-block:: none
:emphasize-lines: 1-1
mlr --icsv --ifs semicolon --oxtab cat data/colours.csv
KEY masterdata_colourcode_1
DE Weiß
EN White
@ -74,9 +86,13 @@ Diagnosing delimiter specifications
PL Czarny
RO Negru
TR Siyah
# Using the new field-separator, retry the cut:
mlr --csv --fs semicolon cut -f KEY,PL,RO data/colours.csv
Using the new field-separator, retry the cut:
.. code-block:: none
:emphasize-lines: 1-1
mlr --csv --fs semicolon cut -f KEY,PL,RO data/colours.csv
KEY;PL;RO
masterdata_colourcode_1;Biały;Alb
masterdata_colourcode_2;Czarny;Negru
@ -84,7 +100,7 @@ Diagnosing delimiter specifications
I assigned $9 and it's not 9th
----------------------------------------------------------------
Miller records are ordered lists of key-value pairs. For NIDX format, DKVP format when keys are missing, or CSV/CSV-lite format with ``--implicit-csv-header``, Miller will sequentially assign keys of the form ``1``, ``2``, etc. But these are not integer array indices: they're just field names taken from the initial field ordering in the input data.
Miller records are ordered lists of key-value pairs. For NIDX format, DKVP format when keys are missing, or CSV/CSV-lite format with ``--implicit-csv-header``, Miller will sequentially assign keys of the form ``1``, ``2``, etc. But these are not integer array indices: they're just field names taken from the initial field ordering in the input data, when it is originally read from the input file(s).
.. code-block:: none
:emphasize-lines: 1-1

72
docs6/shapes-of-data.rst.in
View file

@ -18,12 +18,80 @@ Also try ``od -xcv`` and/or ``cat -e`` on your file to check for non-printable c
Diagnosing delimiter specifications
----------------------------------------------------------------
GENRST_INCLUDE_ESCAPED(data/delimiter-examples.txt)
Use the ``file`` command to see if there are CR/LF terminators (in this case, # there are not):
GENRST_CARDIFY_HIGHLIGHT_ONE
file data/colours.csv
data/colours.csv: UTF-8 Unicode text
GENRST_EOF
Look at the file to find names of fields
GENRST_CARDIFY_HIGHLIGHT_ONE
cat data/colours.csv
KEY;DE;EN;ES;FI;FR;IT;NL;PL;RO;TR
masterdata_colourcode_1;Weiß;White;Blanco;Valkoinen;Blanc;Bianco;Wit;Biały;Alb;Beyaz
masterdata_colourcode_2;Schwarz;Black;Negro;Musta;Noir;Nero;Zwart;Czarny;Negru;Siyah
GENRST_EOF
Extract a few fields:
GENRST_CARDIFY_HIGHLIGHT_ONE
mlr --csv cut -f KEY,PL,RO data/colours.csv
(only blank lines appear)
GENRST_EOF
Use XTAB output format to get a sharper picture of where records/fields are being split:
GENRST_CARDIFY_HIGHLIGHT_ONE
mlr --icsv --oxtab cat data/colours.csv
KEY;DE;EN;ES;FI;FR;IT;NL;PL;RO;TR masterdata_colourcode_1;Weiß;White;Blanco;Valkoinen;Blanc;Bianco;Wit;Biały;Alb;Beyaz
KEY;DE;EN;ES;FI;FR;IT;NL;PL;RO;TR masterdata_colourcode_2;Schwarz;Black;Negro;Musta;Noir;Nero;Zwart;Czarny;Negru;Siyah
GENRST_EOF
Using XTAB output format makes it clearer that ``KEY;DE;...;RO;TR`` is being treated as a single field name in the CSV header, and likewise each subsequent line is being treated as a single field value. This is because the default field separator is a comma but we have semicolons here. Use XTAB again with different field separator (``--fs semicolon``):
GENRST_CARDIFY_HIGHLIGHT_ONE
mlr --icsv --ifs semicolon --oxtab cat data/colours.csv
KEY masterdata_colourcode_1
DE Weiß
EN White
ES Blanco
FI Valkoinen
FR Blanc
IT Bianco
NL Wit
PL Biały
RO Alb
TR Beyaz
KEY masterdata_colourcode_2
DE Schwarz
EN Black
ES Negro
FI Musta
FR Noir
IT Nero
NL Zwart
PL Czarny
RO Negru
TR Siyah
GENRST_EOF
Using the new field-separator, retry the cut:
GENRST_CARDIFY_HIGHLIGHT_ONE
mlr --csv --fs semicolon cut -f KEY,PL,RO data/colours.csv
KEY;PL;RO
masterdata_colourcode_1;Biały;Alb
masterdata_colourcode_2;Czarny;Negru
GENRST_EOF
I assigned $9 and it's not 9th
----------------------------------------------------------------
Miller records are ordered lists of key-value pairs. For NIDX format, DKVP format when keys are missing, or CSV/CSV-lite format with ``--implicit-csv-header``, Miller will sequentially assign keys of the form ``1``, ``2``, etc. But these are not integer array indices: they're just field names taken from the initial field ordering in the input data.
Miller records are ordered lists of key-value pairs. For NIDX format, DKVP format when keys are missing, or CSV/CSV-lite format with ``--implicit-csv-header``, Miller will sequentially assign keys of the form ``1``, ``2``, etc. But these are not integer array indices: they're just field names taken from the initial field ordering in the input data, when it is originally read from the input file(s).
GENRST_RUN_COMMAND
echo x,y,z | mlr --dkvp cat

101
docs6/special-symbols-and-formatting.rst
View file

@ -144,104 +144,3 @@ One way is to use square brackets; an alternative is to use simple string-substi
c is it ...
The ``ssub`` function exists precisely for this reason: so you don't have to escape anything.
How do I suppress numeric conversion?
----------------------------------------------------------------
**TL;DR use put -S**.
**TODO: probably remove this for Miller 6**
Within ``mlr put`` and ``mlr filter``, the default behavior for scanning input records is to parse them as integer, if possible, then as float, if possible, else leave them as string:
.. code-block:: none
:emphasize-lines: 1-1
cat data/scan-example-1.tbl
value
1
2.0
3x
hello
.. code-block:: none
:emphasize-lines: 1-1
mlr --pprint put '$copy = $value; $type = typeof($value)' data/scan-example-1.tbl
value copy type
1 1 int
2.0 2.0 float
3x 3x string
hello hello string
The numeric-conversion rule is simple:
* Try to scan as integer (``"1"`` should be int);
* If that doesn't succeed, try to scan as float (``"1.0"`` should be float);
* If that doesn't succeed, leave the value as a string (``"1x"`` is string).
This is a sensible default: you should be able to put ``'$z = $x + $y'`` without having to write ``'$z = int($x) + float($y)'``. Also note that default output format for floating-point numbers created by ``put`` (and other verbs such as ``stats1``) is six decimal places; you can override this using ``mlr --ofmt``. Also note that Miller uses your system's Go library functions whenever possible: e.g. ``sscanf`` for converting strings to integer or floating-point.
But now suppose you have data like these:
.. code-block:: none
:emphasize-lines: 1-1
cat data/scan-example-2.tbl
value
0001
0002
0005
0005WA
0006
0007
0007WA
0008
0009
0010
.. code-block:: none
:emphasize-lines: 1-1
mlr --pprint put '$copy = $value; $type = typeof($value)' data/scan-example-2.tbl
value copy type
0001 0001 int
0002 0002 int
0005 0005 int
0005WA 0005WA string
0006 0006 int
0007 0007 int
0007WA 0007WA string
0008 0008 float
0009 0009 float
0010 0010 int
The same conversion rules as above are being used. Namely:
* By default field values are inferred to int, else float, else string;
* leading zeroes indicate octal for integers (``sscanf`` semantics);
* since ``0008`` doesn't scan as integer (leading 0 requests octal but 8 isn't a valid octal digit), the float scan is tried next and it succeeds;
* default floating-point output format is 6 decimal places (override with ``mlr --ofmt``).
Taken individually the rules make sense; taken collectively they produce a mishmash of types here.
The solution is to **use the -S flag** for ``mlr put`` and/or ``mlr filter``. Then all field values are left as string. You can type-coerce on demand using syntax like ``'$z = int($x) + float($y)'``. (See also :doc:`reference-dsl`; see also https://github.com/johnkerl/miller/issues/150.)
.. code-block:: none
:emphasize-lines: 1-1
mlr --pprint put -S '$copy = $value; $type = typeof($value)' data/scan-example-2.tbl
value copy type
0001 0001 int
0002 0002 int
0005 0005 int
0005WA 0005WA string
0006 0006 int
0007 0007 int
0007WA 0007WA string
0008 0008 float
0009 0009 float
0010 0010 int

53
docs6/special-symbols-and-formatting.rst.in
View file

@ -95,56 +95,3 @@ mlr --oxtab put '$c = ssub($a, "?"," ...")' data/question.dat
GENRST_EOF
The ``ssub`` function exists precisely for this reason: so you don't have to escape anything.
How do I suppress numeric conversion?
----------------------------------------------------------------
**TL;DR use put -S**.
**TODO: probably remove this for Miller 6**
Within ``mlr put`` and ``mlr filter``, the default behavior for scanning input records is to parse them as integer, if possible, then as float, if possible, else leave them as string:
GENRST_RUN_COMMAND
cat data/scan-example-1.tbl
GENRST_EOF
GENRST_RUN_COMMAND
mlr --pprint put '$copy = $value; $type = typeof($value)' data/scan-example-1.tbl
GENRST_EOF
The numeric-conversion rule is simple:
* Try to scan as integer (``"1"`` should be int);
* If that doesn't succeed, try to scan as float (``"1.0"`` should be float);
* If that doesn't succeed, leave the value as a string (``"1x"`` is string).
This is a sensible default: you should be able to put ``'$z = $x + $y'`` without having to write ``'$z = int($x) + float($y)'``. Also note that default output format for floating-point numbers created by ``put`` (and other verbs such as ``stats1``) is six decimal places; you can override this using ``mlr --ofmt``. Also note that Miller uses your system's Go library functions whenever possible: e.g. ``sscanf`` for converting strings to integer or floating-point.
But now suppose you have data like these:
GENRST_RUN_COMMAND
cat data/scan-example-2.tbl
GENRST_EOF
GENRST_RUN_COMMAND
mlr --pprint put '$copy = $value; $type = typeof($value)' data/scan-example-2.tbl
GENRST_EOF
The same conversion rules as above are being used. Namely:
* By default field values are inferred to int, else float, else string;
* leading zeroes indicate octal for integers (``sscanf`` semantics);
* since ``0008`` doesn't scan as integer (leading 0 requests octal but 8 isn't a valid octal digit), the float scan is tried next and it succeeds;
* default floating-point output format is 6 decimal places (override with ``mlr --ofmt``).
Taken individually the rules make sense; taken collectively they produce a mishmash of types here.
The solution is to **use the -S flag** for ``mlr put`` and/or ``mlr filter``. Then all field values are left as string. You can type-coerce on demand using syntax like ``'$z = int($x) + float($y)'``. (See also :doc:`reference-dsl`; see also https://github.com/johnkerl/miller/issues/150.)
GENRST_RUN_COMMAND
mlr --pprint put -S '$copy = $value; $type = typeof($value)' data/scan-example-2.tbl
GENRST_EOF

71
docs6/statistics-examples.rst Normal file
View file

@ -0,0 +1,71 @@
..
PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE.
Statistics examples
====================
Computing interquartile ranges
----------------------------------------------------------------
For one or more specified field names, simply compute p25 and p75, then write the IQR as the difference of p75 and p25:
.. code-block:: none
:emphasize-lines: 1-3
mlr --oxtab stats1 -f x -a p25,p75 \
then put '$x_iqr = $x_p75 - $x_p25' \
data/medium
x_p25 0.24667037823231752
x_p75 0.7481860062358446
x_iqr 0.5015156280035271
For wildcarded field names, first compute p25 and p75, then loop over field names with ``p25`` in them:
.. code-block:: none
:emphasize-lines: 1-7
mlr --oxtab stats1 --fr '[i-z]' -a p25,p75 \
then put 'for (k,v in $*) {
if (k =~ "(.*)_p25") {
$["\1_iqr"] = $["\1_p75"] - $["\1_p25"]
}
}' \
data/medium
Computing weighted means
----------------------------------------------------------------
This might be more elegantly implemented as an option within the ``stats1`` verb. Meanwhile, it's expressible within the DSL:
.. code-block:: none
:emphasize-lines: 1-24
mlr --from data/medium put -q '
# Using the y field for weighting in this example
weight = $y;
# Using the a field for weighted aggregation in this example
@sumwx[$a] += weight * $i;
@sumw[$a] += weight;
@sumx[$a] += $i;
@sumn[$a] += 1;
end {
map wmean = {};
map mean = {};
for (a in @sumwx) {
wmean[a] = @sumwx[a] / @sumw[a]
}
for (a in @sumx) {
mean[a] = @sumx[a] / @sumn[a]
}
#emit wmean, "a";
#emit mean, "a";
emit (wmean, mean), "a";
}'
a=pan,wmean=4979.563722208067,mean=5028.259010091302
a=eks,wmean=4890.3815931472145,mean=4956.2900763358775
a=wye,wmean=4946.987746229947,mean=4920.001017293998
a=zee,wmean=5164.719684856538,mean=5123.092330239375
a=hat,wmean=4925.533162478552,mean=4967.743946419371

20
docs6/statistics-examples.rst.in Normal file
View file

@ -0,0 +1,20 @@
Statistics examples
====================
Computing interquartile ranges
----------------------------------------------------------------
For one or more specified field names, simply compute p25 and p75, then write the IQR as the difference of p75 and p25:
GENRST_INCLUDE_AND_RUN_ESCAPED(data/iqr1.sh)
For wildcarded field names, first compute p25 and p75, then loop over field names with ``p25`` in them:
GENRST_INCLUDE_AND_RUN_ESCAPED(data/iqrn.sh)
Computing weighted means
----------------------------------------------------------------
This might be more elegantly implemented as an option within the ``stats1`` verb. Meanwhile, it's expressible within the DSL:
GENRST_INCLUDE_AND_RUN_ESCAPED(data/weighted-mean.sh)

606
docs6/cookbook.rst → docs6/two-pass-algorithms.rst
View file

@ -1,167 +1,86 @@
..
PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE.
Cookbook part 1: common patterns
Two-pass algorithms
================================================================
Data-cleaning examples
----------------------------------------------------------------
Here are some ways to use the type-checking options as described in :ref:`reference-dsl-type-tests-and-assertions` Suppose you have the following data file, with inconsistent typing for boolean. (Also imagine that, for the sake of discussion, we have a million-line file rather than a four-line file, so we can't see it all at once and some automation is called for.)
.. code-block:: none
:emphasize-lines: 1-1
cat data/het-bool.csv
name,reachable
barney,false
betty,true
fred,true
wilma,1
One option is to coerce everything to boolean, or integer:
.. code-block:: none
:emphasize-lines: 1-1
mlr --icsv --opprint put '$reachable = boolean($reachable)' data/het-bool.csv
name reachable
barney false
betty true
fred true
wilma true
.. code-block:: none
:emphasize-lines: 1-1
mlr --icsv --opprint put '$reachable = int(boolean($reachable))' data/het-bool.csv
name reachable
barney 0
betty 1
fred 1
wilma 1
A second option is to flag badly formatted data within the output stream:
.. code-block:: none
:emphasize-lines: 1-1
mlr --icsv --opprint put '$format_ok = is_string($reachable)' data/het-bool.csv
name reachable format_ok
barney false false
betty true false
fred true false
wilma 1 false
Or perhaps to flag badly formatted data outside the output stream:
.. code-block:: none
:emphasize-lines: 1-3
mlr --icsv --opprint put '
if (!is_string($reachable)) {eprint "Malformed at NR=".NR}
' data/het-bool.csv
Malformed at NR=1
Malformed at NR=2
Malformed at NR=3
Malformed at NR=4
name reachable
barney false
betty true
fred true
wilma 1
A third way is to abort the process on first instance of bad data:
.. code-block:: none
:emphasize-lines: 1-1
mlr --csv put '$reachable = asserting_string($reachable)' data/het-bool.csv
Miller: is_string type-assertion failed at NR=1 FNR=1 FILENAME=data/het-bool.csv
Showing differences between successive queries
----------------------------------------------------------------
Suppose you have a database query which you run at one point in time, producing the output on the left, then again later producing the output on the right:
.. code-block:: none
:emphasize-lines: 1-1
cat data/previous_counters.csv
color,count
red,3472
blue,6838
orange,694
purple,12
.. code-block:: none
:emphasize-lines: 1-1
cat data/current_counters.csv
color,count
red,3467
orange,670
yellow,27
blue,6944
And, suppose you want to compute the differences in the counters between adjacent keys. Since the color names aren't all in the same order, nor are they all present on both sides, we can't just paste the two files side-by-side and do some column-four-minus-column-two arithmetic.
First, rename counter columns to make them distinct:
.. code-block:: none
:emphasize-lines: 1-1
mlr --csv rename count,previous_count data/previous_counters.csv > data/prevtemp.csv
.. code-block:: none
:emphasize-lines: 1-1
cat data/prevtemp.csv
color,previous_count
red,3472
blue,6838
orange,694
purple,12
.. code-block:: none
:emphasize-lines: 1-1
mlr --csv rename count,current_count data/current_counters.csv > data/currtemp.csv
.. code-block:: none
:emphasize-lines: 1-1
cat data/currtemp.csv
color,current_count
red,3467
orange,670
yellow,27
blue,6944
Then, join on the key field(s), and use unsparsify to zero-fill counters absent on one side but present on the other. Use ``--ul`` and ``--ur`` to emit unpaired records (namely, purple on the left and yellow on the right):
.. code-block:: none
:emphasize-lines: 1-5
mlr --icsv --opprint \
join -j color --ul --ur -f data/prevtemp.csv \
then unsparsify --fill-with 0 \
then put '$count_delta = $current_count - $previous_count' \
data/currtemp.csv
color previous_count current_count count_delta
red 3472 3467 -5
orange 694 670 -24
yellow 0 27 (error)
blue 6838 6944 106
purple 12 0 (error)
Two-pass algorithms
Overview
----------------------------------------------------------------
Miller is a streaming record processor; commands are performed once per record. This makes Miller particularly suitable for single-pass algorithms, allowing many of its verbs to process files that are (much) larger than the amount of RAM present in your system. (Of course, Miller verbs such as ``sort``, ``tac``, etc. all must ingest and retain all input records before emitting any output records.) You can also use out-of-stream variables to perform multi-pass computations, at the price of retaining all input records in memory.
Two-pass algorithms: computation of percentages
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
One of Miller's strengths is its compact notation: for example, given input of the form
.. code-block:: none
:emphasize-lines: 1-1
head -n 5 ../data/medium
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
you can simply do
.. code-block:: none
:emphasize-lines: 1-1
mlr --oxtab stats1 -a sum -f x ../data/medium
x_sum 4986.019681679581
or
.. code-block:: none
:emphasize-lines: 1-1
mlr --opprint stats1 -a sum -f x -g b ../data/medium
b x_sum
pan 965.7636699425815
wye 1023.5484702619565
zee 979.7420161495838
eks 1016.7728571314786
hat 1000.192668193983
rather than the more tedious
.. code-block:: none
:emphasize-lines: 1-6
mlr --oxtab put -q '
@x_sum += $x;
end {
emit @x_sum
}
' data/medium
x_sum 4986.019681679581
or
.. code-block:: none
:emphasize-lines: 1-6
mlr --opprint put -q '
@x_sum[$b] += $x;
end {
emit @x_sum, "b"
}
' data/medium
b x_sum
pan 965.7636699425815
wye 1023.5484702619565
zee 979.7420161495838
eks 1016.7728571314786
hat 1000.192668193983
The former (``mlr stats1`` et al.) has the advantages of being easier to type, being less error-prone to type, and running faster.
Nonetheless, out-of-stream variables (which I whimsically call *oosvars*), begin/end blocks, and emit statements give you the ability to implement logic -- if you wish to do so -- which isn't present in other Miller verbs. (If you find yourself often using the same out-of-stream-variable logic over and over, please file a request at https://github.com/johnkerl/miller/issues to get it implemented directly in Go as a Miller verb of its own.)
The following examples compute some things using oosvars which are already computable using Miller verbs, by way of providing food for thought.
Computation of percentages
----------------------------------------------------------------
For example, mapping numeric values down a column to the percentage between their min and max values is two-pass: on the first pass you find the min and max values, then on the second, map each record's value to a percentage.
@ -191,8 +110,8 @@ For example, mapping numeric values down a column to the percentage between thei
4 0.38139939387114097 31.90823602213647
5 0.5732889198020006 66.54054236562845
Two-pass algorithms: line-number ratios
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Line-number ratios
----------------------------------------------------------------
Similarly, finding the total record count requires first reading through all the data:
@ -217,8 +136,8 @@ Similarly, finding the total record count requires first reading through all the
4 5 (error) eks wye 4 0.38139939387114097 0.13418874328430463
5 5 (error) wye pan 5 0.5732889198020006 0.8636244699032729
Two-pass algorithms: records having max value
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Records having max value
----------------------------------------------------------------
The idea is to retain records having the largest value of ``n`` in the following data:
@ -523,134 +442,277 @@ There are field names ``a``, ``b``, ``v``, ``u``, ``x``, ``w`` in the data -- bu
There is a keystroke-saving verb for this: :ref:`mlr unsparsify <reference-verbs-unsparsify>`.
Parsing log-file output
Mean without/with oosvars
----------------------------------------------------------------
This, of course, depends highly on what's in your log files. But, as an example, suppose you have log-file lines such as
.. code-block:: none
2015-10-08 08:29:09,445 INFO com.company.path.to.ClassName @ [sometext] various/sorts/of data {& punctuation} hits=1 status=0 time=2.378
I prefer to pre-filter with ``grep`` and/or ``sed`` to extract the structured text, then hand that to Miller. Example:
.. code-block:: none
:emphasize-lines: 1-1
grep 'various sorts' *.log | sed 's/.*} //' | mlr --fs space --repifs --oxtab stats1 -a min,p10,p50,p90,max -f time -g status
mlr --opprint stats1 -a mean -f x data/medium
x_mean
0.49860196816795804
.. _cookbook-memoization-with-oosvars:
.. code-block:: none
:emphasize-lines: 1-8
Memoization with out-of-stream variables
mlr --opprint put -q '
@x_sum += $x;
@x_count += 1;
end {
@x_mean = @x_sum / @x_count;
emit @x_mean
}
' data/medium
x_mean
0.49860196816795804
Keyed mean without/with oosvars
----------------------------------------------------------------
The recursive function for the Fibonacci sequence is famous for its computational complexity. Namely, using f(0)=1, f(1)=1, f(n)=f(n-1)+f(n-2) for n>=2, the evaluation tree branches left as well as right at each non-trivial level, resulting in millions or more paths to the root 0/1 nodes for larger n. This program
.. code-block:: none
:emphasize-lines: 1-1
mlr --opprint stats1 -a mean -f x -g a,b data/medium
a b x_mean
pan pan 0.5133141190437597
eks pan 0.48507555383425127
wye wye 0.49150092785839306
eks wye 0.4838950517724162
wye pan 0.4996119901034838
zee pan 0.5198298297816007
eks zee 0.49546320772681596
zee wye 0.5142667998230479
hat wye 0.49381326184632596
pan wye 0.5023618498923658
zee eks 0.4883932942792647
hat zee 0.5099985721987774
hat eks 0.48587864619953547
wye hat 0.4977304763723314
pan eks 0.5036718595143479
eks eks 0.5227992666570941
hat hat 0.47993053101017374
hat pan 0.4643355557376876
zee zee 0.5127559183726382
pan hat 0.492140950155604
pan zee 0.4966041598627583
zee hat 0.46772617655014515
wye zee 0.5059066170573692
eks hat 0.5006790659966355
wye eks 0.5306035254809106
.. code-block:: none
:emphasize-lines: 1-10
mlr --ofmt '%.9lf' --opprint seqgen --start 1 --stop 28 then put '
func f(n) {
@fcount += 1; # count number of calls to the function
if (n < 2) {
return 1
} else {
return f(n-1) + f(n-2) # recurse
}
}
@fcount = 0;
$o = f($i);
$fcount = @fcount;
' then put '$seconds=systime()' then step -a delta -f seconds then cut -x -f seconds
produces output like this:
.. code-block:: none
i o fcount seconds_delta
1 1 1 0
2 2 3 0.000039101
3 3 5 0.000015974
4 5 9 0.000019073
5 8 15 0.000026941
6 13 25 0.000036955
7 21 41 0.000056028
8 34 67 0.000086069
9 55 109 0.000134945
10 89 177 0.000217915
11 144 287 0.000355959
12 233 465 0.000506163
13 377 753 0.000811815
14 610 1219 0.001297235
15 987 1973 0.001960993
16 1597 3193 0.003417969
17 2584 5167 0.006215811
18 4181 8361 0.008294106
19 6765 13529 0.012095928
20 10946 21891 0.019592047
21 17711 35421 0.031193972
22 28657 57313 0.057254076
23 46368 92735 0.080307961
24 75025 150049 0.129482031
25 121393 242785 0.213325977
26 196418 392835 0.334423065
27 317811 635621 0.605969906
28 514229 1028457 0.971235037
Note that the time it takes to evaluate the function is blowing up exponentially as the input argument increases. Using ``@``-variables, which persist across records, we can cache and reuse the results of previous computations:
.. code-block:: none
mlr --ofmt '%.9lf' --opprint seqgen --start 1 --stop 28 then put '
func f(n) {
@fcount += 1; # count number of calls to the function
if (is_present(@fcache[n])) { # cache hit
return @fcache[n]
} else { # cache miss
num rv = 1;
if (n >= 2) {
rv = f(n-1) + f(n-2) # recurse
}
@fcache[n] = rv;
return rv
mlr --opprint put -q '
@x_sum[$a][$b] += $x;
@x_count[$a][$b] += 1;
end{
for ((a, b), v in @x_sum) {
@x_mean[a][b] = @x_sum[a][b] / @x_count[a][b];
}
emit @x_mean, "a", "b"
}
@fcount = 0;
$o = f($i);
$fcount = @fcount;
' then put '$seconds=systime()' then step -a delta -f seconds then cut -x -f seconds
' data/medium
a b x_mean
pan pan 0.5133141190437597
pan wye 0.5023618498923658
pan eks 0.5036718595143479
pan hat 0.492140950155604
pan zee 0.4966041598627583
eks pan 0.48507555383425127
eks wye 0.4838950517724162
eks zee 0.49546320772681596
eks eks 0.5227992666570941
eks hat 0.5006790659966355
wye wye 0.49150092785839306
wye pan 0.4996119901034838
wye hat 0.4977304763723314
wye zee 0.5059066170573692
wye eks 0.5306035254809106
zee pan 0.5198298297816007
zee wye 0.5142667998230479
zee eks 0.4883932942792647
zee zee 0.5127559183726382
zee hat 0.46772617655014515
hat wye 0.49381326184632596
hat zee 0.5099985721987774
hat eks 0.48587864619953547
hat hat 0.47993053101017374
hat pan 0.4643355557376876
with output like this:
Variance and standard deviation without/with oosvars
----------------------------------------------------------------
.. code-block:: none
:emphasize-lines: 1-1
i o fcount seconds_delta
1 1 1 0
2 2 3 0.000053883
3 3 3 0.000035048
4 5 3 0.000045061
5 8 3 0.000014067
6 13 3 0.000028849
7 21 3 0.000028133
8 34 3 0.000027895
9 55 3 0.000014067
10 89 3 0.000015020
11 144 3 0.000012875
12 233 3 0.000033140
13 377 3 0.000014067
14 610 3 0.000012875
15 987 3 0.000029087
16 1597 3 0.000013828
17 2584 3 0.000013113
18 4181 3 0.000012875
19 6765 3 0.000013113
20 10946 3 0.000012875
21 17711 3 0.000013113
22 28657 3 0.000013113
23 46368 3 0.000015974
24 75025 3 0.000012875
25 121393 3 0.000013113
26 196418 3 0.000012875
27 317811 3 0.000013113
28 514229 3 0.000012875
mlr --oxtab stats1 -a count,sum,mean,var,stddev -f x data/medium
x_count 10000
x_sum 4986.019681679581
x_mean 0.49860196816795804
x_var 0.08426974433144456
x_stddev 0.2902925151144007
.. code-block:: none
:emphasize-lines: 1-1
cat variance.mlr
@n += 1;
@sumx += $x;
@sumx2 += $x**2;
end {
@mean = @sumx / @n;
@var = (@sumx2 - @mean * (2 * @sumx - @n * @mean)) / (@n - 1);
@stddev = sqrt(@var);
emitf @n, @sumx, @sumx2, @mean, @var, @stddev
}
.. code-block:: none
:emphasize-lines: 1-1
mlr --oxtab put -q -f variance.mlr data/medium
n 10000
sumx 4986.019681679581
sumx2 3328.652400179729
mean 0.49860196816795804
var 0.08426974433144456
stddev 0.2902925151144007
You can also do this keyed, of course, imitating the keyed-mean example above.
Min/max without/with oosvars
----------------------------------------------------------------
.. code-block:: none
:emphasize-lines: 1-1
mlr --oxtab stats1 -a min,max -f x data/medium
x_min 4.509679127584487e-05
x_max 0.999952670371898
.. code-block:: none
:emphasize-lines: 1-5
mlr --oxtab put -q '
@x_min = min(@x_min, $x);
@x_max = max(@x_max, $x);
end{emitf @x_min, @x_max}
' data/medium
x_min 4.509679127584487e-05
x_max 0.999952670371898
Keyed min/max without/with oosvars
----------------------------------------------------------------
.. code-block:: none
:emphasize-lines: 1-1
mlr --opprint stats1 -a min,max -f x -g a data/medium
a x_min x_max
pan 0.00020390740306253097 0.9994029107062516
eks 0.0006917972627396018 0.9988110946859143
wye 0.0001874794831505655 0.9998228522652893
zee 0.0005486114815762555 0.9994904324789629
hat 4.509679127584487e-05 0.999952670371898
.. code-block:: none
:emphasize-lines: 1-7
mlr --opprint --from data/medium put -q '
@min[$a] = min(@min[$a], $x);
@max[$a] = max(@max[$a], $x);
end{
emit (@min, @max), "a";
}
'
a min max
pan 0.00020390740306253097 0.9994029107062516
eks 0.0006917972627396018 0.9988110946859143
wye 0.0001874794831505655 0.9998228522652893
zee 0.0005486114815762555 0.9994904324789629
hat 4.509679127584487e-05 0.999952670371898
Delta without/with oosvars
----------------------------------------------------------------
.. code-block:: none
:emphasize-lines: 1-1
mlr --opprint step -a delta -f x data/small
a b i x y x_delta
pan pan 1 0.3467901443380824 0.7268028627434533 0
eks pan 2 0.7586799647899636 0.5221511083334797 0.41188982045188116
wye wye 3 0.20460330576630303 0.33831852551664776 -0.5540766590236605
eks wye 4 0.38139939387114097 0.13418874328430463 0.17679608810483793
wye pan 5 0.5732889198020006 0.8636244699032729 0.19188952593085962
.. code-block:: none
:emphasize-lines: 1-4
mlr --opprint put '
$x_delta = is_present(@last) ? $x - @last : 0;
@last = $x
' data/small
a b i x y x_delta
pan pan 1 0.3467901443380824 0.7268028627434533 0
eks pan 2 0.7586799647899636 0.5221511083334797 0.41188982045188116
wye wye 3 0.20460330576630303 0.33831852551664776 -0.5540766590236605
eks wye 4 0.38139939387114097 0.13418874328430463 0.17679608810483793
wye pan 5 0.5732889198020006 0.8636244699032729 0.19188952593085962
Keyed delta without/with oosvars
----------------------------------------------------------------
.. code-block:: none
:emphasize-lines: 1-1
mlr --opprint step -a delta -f x -g a data/small
a b i x y x_delta
pan pan 1 0.3467901443380824 0.7268028627434533 0
eks pan 2 0.7586799647899636 0.5221511083334797 0
wye wye 3 0.20460330576630303 0.33831852551664776 0
eks wye 4 0.38139939387114097 0.13418874328430463 -0.3772805709188226
wye pan 5 0.5732889198020006 0.8636244699032729 0.36868561403569755
.. code-block:: none
:emphasize-lines: 1-4
mlr --opprint put '
$x_delta = is_present(@last[$a]) ? $x - @last[$a] : 0;
@last[$a]=$x
' data/small
a b i x y x_delta
pan pan 1 0.3467901443380824 0.7268028627434533 0
eks pan 2 0.7586799647899636 0.5221511083334797 0
wye wye 3 0.20460330576630303 0.33831852551664776 0
eks wye 4 0.38139939387114097 0.13418874328430463 -0.3772805709188226
wye pan 5 0.5732889198020006 0.8636244699032729 0.36868561403569755
Exponentially weighted moving averages without/with oosvars
----------------------------------------------------------------
.. code-block:: none
:emphasize-lines: 1-1
mlr --opprint step -a ewma -d 0.1 -f x data/small
a b i x y x_ewma_0.1
pan pan 1 0.3467901443380824 0.7268028627434533 0.3467901443380824
eks pan 2 0.7586799647899636 0.5221511083334797 0.3879791263832706
wye wye 3 0.20460330576630303 0.33831852551664776 0.36964154432157387
eks wye 4 0.38139939387114097 0.13418874328430463 0.37081732927653055
wye pan 5 0.5732889198020006 0.8636244699032729 0.3910644883290776
.. code-block:: none
:emphasize-lines: 1-5
mlr --opprint put '
begin{ @a=0.1 };
$e = NR==1 ? $x : @a * $x + (1 - @a) * @e;
@e=$e
' data/small
a b i x y e
pan pan 1 0.3467901443380824 0.7268028627434533 0.3467901443380824
eks pan 2 0.7586799647899636 0.5221511083334797 0.3879791263832706
wye wye 3 0.20460330576630303 0.33831852551664776 0.36964154432157387
eks wye 4 0.38139939387114097 0.13418874328430463 0.37081732927653055
wye pan 5 0.5732889198020006 0.8636244699032729 0.3910644883290776

218
docs6/two-pass-algorithms.rst.in Normal file
View file

@ -0,0 +1,218 @@
Two-pass algorithms
================================================================
Overview
----------------------------------------------------------------
Miller is a streaming record processor; commands are performed once per record. This makes Miller particularly suitable for single-pass algorithms, allowing many of its verbs to process files that are (much) larger than the amount of RAM present in your system. (Of course, Miller verbs such as ``sort``, ``tac``, etc. all must ingest and retain all input records before emitting any output records.) You can also use out-of-stream variables to perform multi-pass computations, at the price of retaining all input records in memory.
One of Miller's strengths is its compact notation: for example, given input of the form
GENRST_RUN_COMMAND
head -n 5 ../data/medium
GENRST_EOF
you can simply do
GENRST_RUN_COMMAND
mlr --oxtab stats1 -a sum -f x ../data/medium
GENRST_EOF
or
GENRST_RUN_COMMAND
mlr --opprint stats1 -a sum -f x -g b ../data/medium
GENRST_EOF
rather than the more tedious
GENRST_INCLUDE_AND_RUN_ESCAPED(oosvar-example-sum.sh)
or
GENRST_INCLUDE_AND_RUN_ESCAPED(oosvar-example-sum-grouped.sh)
The former (``mlr stats1`` et al.) has the advantages of being easier to type, being less error-prone to type, and running faster.
Nonetheless, out-of-stream variables (which I whimsically call *oosvars*), begin/end blocks, and emit statements give you the ability to implement logic -- if you wish to do so -- which isn't present in other Miller verbs. (If you find yourself often using the same out-of-stream-variable logic over and over, please file a request at https://github.com/johnkerl/miller/issues to get it implemented directly in Go as a Miller verb of its own.)
The following examples compute some things using oosvars which are already computable using Miller verbs, by way of providing food for thought.
Computation of percentages
----------------------------------------------------------------
For example, mapping numeric values down a column to the percentage between their min and max values is two-pass: on the first pass you find the min and max values, then on the second, map each record's value to a percentage.
GENRST_INCLUDE_AND_RUN_ESCAPED(data/two-pass-percentage.sh)
Line-number ratios
----------------------------------------------------------------
Similarly, finding the total record count requires first reading through all the data:
GENRST_INCLUDE_AND_RUN_ESCAPED(data/two-pass-record-numbers.sh)
Records having max value
----------------------------------------------------------------
The idea is to retain records having the largest value of ``n`` in the following data:
GENRST_RUN_COMMAND
mlr --itsv --opprint cat data/maxrows.tsv
GENRST_EOF
Of course, the largest value of ``n`` isn't known until after all data have been read. Using an out-of-stream variable we can retain all records as they are read, then filter them at the end:
GENRST_RUN_COMMAND
cat data/maxrows.mlr
GENRST_EOF
GENRST_RUN_COMMAND
mlr --itsv --opprint put -q -f data/maxrows.mlr data/maxrows.tsv
GENRST_EOF
Feature-counting
----------------------------------------------------------------
Suppose you have some heterogeneous data like this:
GENRST_INCLUDE_ESCAPED(data/features.json)
A reasonable question to ask is, how many occurrences of each field are there? And, what percentage of total row count has each of them? Since the denominator of the percentage is not known until the end, this is a two-pass algorithm:
GENRST_INCLUDE_ESCAPED(data/feature-count.mlr)
Then
GENRST_RUN_COMMAND
mlr --json put -q -f data/feature-count.mlr data/features.json
GENRST_EOF
GENRST_RUN_COMMAND
mlr --ijson --opprint put -q -f data/feature-count.mlr data/features.json
GENRST_EOF
Unsparsing
----------------------------------------------------------------
The previous section discussed how to fill out missing data fields within CSV with full header line -- so the list of all field names is present within the header line. Next, let's look at a related problem: we have data where each record has various key names but we want to produce rectangular output having the union of all key names.
For example, suppose you have JSON input like this:
GENRST_RUN_COMMAND
cat data/sparse.json
GENRST_EOF
There are field names ``a``, ``b``, ``v``, ``u``, ``x``, ``w`` in the data -- but not all in every record. Since we don't know the names of all the keys until we've read them all, this needs to be a two-pass algorithm. On the first pass, remember all the unique key names and all the records; on the second pass, loop through the records filling in absent values, then producing output. Use ``put -q`` since we don't want to produce per-record output, only emitting output in the ``end`` block:
GENRST_RUN_COMMAND
cat data/unsparsify.mlr
GENRST_EOF
GENRST_RUN_COMMAND
mlr --json put -q -f data/unsparsify.mlr data/sparse.json
GENRST_EOF
GENRST_RUN_COMMAND
mlr --ijson --ocsv put -q -f data/unsparsify.mlr data/sparse.json
GENRST_EOF
GENRST_RUN_COMMAND
mlr --ijson --opprint put -q -f data/unsparsify.mlr data/sparse.json
GENRST_EOF
There is a keystroke-saving verb for this: :ref:`mlr unsparsify <reference-verbs-unsparsify>`.
Mean without/with oosvars
----------------------------------------------------------------
GENRST_RUN_COMMAND
mlr --opprint stats1 -a mean -f x data/medium
GENRST_EOF
GENRST_INCLUDE_AND_RUN_ESCAPED(data/mean-with-oosvars.sh)
Keyed mean without/with oosvars
----------------------------------------------------------------
GENRST_RUN_COMMAND
mlr --opprint stats1 -a mean -f x -g a,b data/medium
GENRST_EOF
GENRST_INCLUDE_AND_RUN_ESCAPED(data/keyed-mean-with-oosvars.sh)
Variance and standard deviation without/with oosvars
----------------------------------------------------------------
GENRST_RUN_COMMAND
mlr --oxtab stats1 -a count,sum,mean,var,stddev -f x data/medium
GENRST_EOF
GENRST_RUN_COMMAND
cat variance.mlr
GENRST_EOF
GENRST_RUN_COMMAND
mlr --oxtab put -q -f variance.mlr data/medium
GENRST_EOF
You can also do this keyed, of course, imitating the keyed-mean example above.
Min/max without/with oosvars
----------------------------------------------------------------
GENRST_RUN_COMMAND
mlr --oxtab stats1 -a min,max -f x data/medium
GENRST_EOF
GENRST_RUN_COMMAND
mlr --oxtab put -q '
@x_min = min(@x_min, $x);
@x_max = max(@x_max, $x);
end{emitf @x_min, @x_max}
' data/medium
GENRST_EOF
Keyed min/max without/with oosvars
----------------------------------------------------------------
GENRST_RUN_COMMAND
mlr --opprint stats1 -a min,max -f x -g a data/medium
GENRST_EOF
GENRST_INCLUDE_AND_RUN_ESCAPED(data/keyed-min-max-with-oosvars.sh)
Delta without/with oosvars
----------------------------------------------------------------
GENRST_RUN_COMMAND
mlr --opprint step -a delta -f x data/small
GENRST_EOF
GENRST_RUN_COMMAND
mlr --opprint put '
$x_delta = is_present(@last) ? $x - @last : 0;
@last = $x
' data/small
GENRST_EOF
Keyed delta without/with oosvars
----------------------------------------------------------------
GENRST_RUN_COMMAND
mlr --opprint step -a delta -f x -g a data/small
GENRST_EOF
GENRST_RUN_COMMAND
mlr --opprint put '
$x_delta = is_present(@last[$a]) ? $x - @last[$a] : 0;
@last[$a]=$x
' data/small
GENRST_EOF
Exponentially weighted moving averages without/with oosvars
----------------------------------------------------------------
GENRST_INCLUDE_AND_RUN_ESCAPED(verb-example-ewma.sh)
GENRST_INCLUDE_AND_RUN_ESCAPED(oosvar-example-ewma.sh)

5
go/todo.txt
View file

@ -7,6 +7,8 @@ TOP OF LIST:
* survey email followups
* template verb
! find a way to get timestamps out as ints not scinot
* windows ./mlr hack
* windows &&/|| debug
@ -19,6 +21,9 @@ doclink etc:
----------------------------------------------------------------
DOC6
* shell-commands.html
while-read example issue
* memory page:
o open with "out of memory" -- what next?
o streaming vs non-streaming