Mirrors/miller
1
0
Fork 0
mirror of https://github.com/johnkerl/miller.git synced 2026-07-26 09:23:55 +00:00
Code Issues Projects Releases Wiki Activity

Merge pull request #535 from johnkerl/docs6

Browse source 
Start port of docs to Miller 6
This commit is contained in:
John Kerl 2021-05-25 02:46:30 +00:00 • committed by GitHub
commit da16e95100
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
287 changed files with 274500 additions and 80 deletions
Download patch file Download diff file Expand all files Collapse all files

1
.gitignore vendored
View file

@ -82,6 +82,7 @@ push2
data/.gitignore
docs/_build
docs6/_build
c/mlr.static
miller-*.src.rpm

3
docs6/.vimrc Normal file
View file

@ -0,0 +1,3 @@
map \f :w<C-m>:!clear;make html<C-m>
map \d :w<C-m>:!clear;make clean html<C-m>
map \z :w<C-m>:%!h2rfoo<C-m>

2
docs6/10-1.sh Executable file
View file

@ -0,0 +1,2 @@
grep op=cache log.txt \
| mlr --idkvp --opprint stats1 -a mean -f hit -g type then sort -f type

5
docs6/10-2.sh Executable file
View file

@ -0,0 +1,5 @@
mlr --from log.txt --opprint \
filter 'is_present($batch_size)' \
then step -a delta -f time,num_filtered \
then sec2gmt time

396
docs6/10min.rst Normal file
View file

@ -0,0 +1,396 @@
..
PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE.
Miller in 10 minutes
====================
CSV-file examples
^^^^^^^^^^^^^^^^^
Suppose you have this CSV data file::
$ cat example.csv
color,shape,flag,index,quantity,rate
yellow,triangle,1,11,43.6498,9.8870
red,square,1,15,79.2778,0.0130
red,circle,1,16,13.8103,2.9010
red,square,0,48,77.5542,7.4670
purple,triangle,0,51,81.2290,8.5910
red,square,0,64,77.1991,9.5310
purple,triangle,0,65,80.1405,5.8240
yellow,circle,1,73,63.9785,4.2370
yellow,circle,1,87,63.5058,8.3350
purple,square,0,91,72.3735,8.2430
``mlr cat`` is like cat -- it passes the data through unmodified::
$ mlr --csv cat example.csv
color,shape,flag,index,quantity,rate
yellow,triangle,1,11,43.6498,9.8870
red,square,1,15,79.2778,0.0130
red,circle,1,16,13.8103,2.9010
red,square,0,48,77.5542,7.4670
purple,triangle,0,51,81.2290,8.5910
red,square,0,64,77.1991,9.5310
purple,triangle,0,65,80.1405,5.8240
yellow,circle,1,73,63.9785,4.2370
yellow,circle,1,87,63.5058,8.3350
purple,square,0,91,72.3735,8.2430
but it can also do format conversion (here, you can pretty-print in tabular format)::
$ mlr --icsv --opprint cat example.csv
color shape flag index quantity rate
yellow triangle 1 11 43.6498 9.8870
red square 1 15 79.2778 0.0130
red circle 1 16 13.8103 2.9010
red square 0 48 77.5542 7.4670
purple triangle 0 51 81.2290 8.5910
red square 0 64 77.1991 9.5310
purple triangle 0 65 80.1405 5.8240
yellow circle 1 73 63.9785 4.2370
yellow circle 1 87 63.5058 8.3350
purple square 0 91 72.3735 8.2430
``mlr head`` and ``mlr tail`` count records rather than lines. Whether you're getting the first few records or the last few, the CSV header is included either way::
$ mlr --csv head -n 4 example.csv
color,shape,flag,index,quantity,rate
yellow,triangle,1,11,43.6498,9.8870
red,square,1,15,79.2778,0.0130
red,circle,1,16,13.8103,2.9010
red,square,0,48,77.5542,7.4670
::
$ mlr --csv tail -n 4 example.csv
color,shape,flag,index,quantity,rate
purple,triangle,0,65,80.1405,5.8240
yellow,circle,1,73,63.9785,4.2370
yellow,circle,1,87,63.5058,8.3350
purple,square,0,91,72.3735,8.2430
You can sort primarily alphabetically on one field, then secondarily numerically descending on another field::
$ mlr --icsv --opprint sort -f shape -nr index example.csv
color shape flag index quantity rate
yellow circle 1 87 63.5058 8.3350
yellow circle 1 73 63.9785 4.2370
red circle 1 16 13.8103 2.9010
purple square 0 91 72.3735 8.2430
red square 0 64 77.1991 9.5310
red square 0 48 77.5542 7.4670
red square 1 15 79.2778 0.0130
purple triangle 0 65 80.1405 5.8240
purple triangle 0 51 81.2290 8.5910
yellow triangle 1 11 43.6498 9.8870
You can use ``cut`` to retain only specified fields, in the same order they appeared in the input data::
$ mlr --icsv --opprint cut -f flag,shape example.csv
shape flag
triangle 1
square 1
circle 1
square 0
triangle 0
square 0
triangle 0
circle 1
circle 1
square 0
You can also use ``cut -o`` to retain only specified fields in your preferred order::
$ mlr --icsv --opprint cut -o -f flag,shape example.csv
flag shape
1 triangle
1 square
1 circle
0 square
0 triangle
0 square
0 triangle
1 circle
1 circle
0 square
You can use ``cut -x`` to omit fields you don't care about::
$ mlr --icsv --opprint cut -x -f flag,shape example.csv
color index quantity rate
yellow 11 43.6498 9.8870
red 15 79.2778 0.0130
red 16 13.8103 2.9010
red 48 77.5542 7.4670
purple 51 81.2290 8.5910
red 64 77.1991 9.5310
purple 65 80.1405 5.8240
yellow 73 63.9785 4.2370
yellow 87 63.5058 8.3350
purple 91 72.3735 8.2430
You can use ``filter`` to keep only records you care about::
$ mlr --icsv --opprint filter '$color == "red"' example.csv
color shape flag index quantity rate
red square 1 15 79.2778 0.0130
red circle 1 16 13.8103 2.9010
red square 0 48 77.5542 7.4670
red square 0 64 77.1991 9.5310
::
$ mlr --icsv --opprint filter '$color == "red" && $flag == 1' example.csv
color shape flag index quantity rate
red square 1 15 79.2778 0.0130
red circle 1 16 13.8103 2.9010
You can use ``put`` to create new fields which are computed from other fields::
$ mlr --icsv --opprint put '$ratio = $quantity / $rate; $color_shape = $color . "_" . $shape' example.csv
color shape flag index quantity rate ratio color_shape
yellow triangle 1 11 43.6498 9.8870 4.414868 yellow_triangle
red square 1 15 79.2778 0.0130 6098.292308 red_square
red circle 1 16 13.8103 2.9010 4.760531 red_circle
red square 0 48 77.5542 7.4670 10.386260 red_square
purple triangle 0 51 81.2290 8.5910 9.455127 purple_triangle
red square 0 64 77.1991 9.5310 8.099790 red_square
purple triangle 0 65 80.1405 5.8240 13.760388 purple_triangle
yellow circle 1 73 63.9785 4.2370 15.099953 yellow_circle
yellow circle 1 87 63.5058 8.3350 7.619172 yellow_circle
purple square 0 91 72.3735 8.2430 8.779995 purple_square
Even though Miller's main selling point is name-indexing, sometimes you really want to refer to a field name by its positional index. Use ``$[[3]]`` to access the name of field 3 or ``$[[[3]]]`` to access the value of field 3::
$ mlr --icsv --opprint put '$[[3]] = "NEW"' example.csv
color shape NEW index quantity rate
yellow triangle 1 11 43.6498 9.8870
red square 1 15 79.2778 0.0130
red circle 1 16 13.8103 2.9010
red square 0 48 77.5542 7.4670
purple triangle 0 51 81.2290 8.5910
red square 0 64 77.1991 9.5310
purple triangle 0 65 80.1405 5.8240
yellow circle 1 73 63.9785 4.2370
yellow circle 1 87 63.5058 8.3350
purple square 0 91 72.3735 8.2430
::
$ mlr --icsv --opprint put '$[[[3]]] = "NEW"' example.csv
color shape flag index quantity rate
yellow triangle NEW 11 43.6498 9.8870
red square NEW 15 79.2778 0.0130
red circle NEW 16 13.8103 2.9010
red square NEW 48 77.5542 7.4670
purple triangle NEW 51 81.2290 8.5910
red square NEW 64 77.1991 9.5310
purple triangle NEW 65 80.1405 5.8240
yellow circle NEW 73 63.9785 4.2370
yellow circle NEW 87 63.5058 8.3350
purple square NEW 91 72.3735 8.2430
JSON-file examples
^^^^^^^^^^^^^^^^^^
OK, CSV and pretty-print are fine. But Miller can also convert between a few other formats -- let's take a look at JSON output::
$ mlr --icsv --ojson put '$ratio = $quantity/$rate; $shape = toupper($shape)' example.csv
{ "color": "yellow", "shape": "TRIANGLE", "flag": 1, "index": 11, "quantity": 43.6498, "rate": 9.8870, "ratio": 4.414868 }
{ "color": "red", "shape": "SQUARE", "flag": 1, "index": 15, "quantity": 79.2778, "rate": 0.0130, "ratio": 6098.292308 }
{ "color": "red", "shape": "CIRCLE", "flag": 1, "index": 16, "quantity": 13.8103, "rate": 2.9010, "ratio": 4.760531 }
{ "color": "red", "shape": "SQUARE", "flag": 0, "index": 48, "quantity": 77.5542, "rate": 7.4670, "ratio": 10.386260 }
{ "color": "purple", "shape": "TRIANGLE", "flag": 0, "index": 51, "quantity": 81.2290, "rate": 8.5910, "ratio": 9.455127 }
{ "color": "red", "shape": "SQUARE", "flag": 0, "index": 64, "quantity": 77.1991, "rate": 9.5310, "ratio": 8.099790 }
{ "color": "purple", "shape": "TRIANGLE", "flag": 0, "index": 65, "quantity": 80.1405, "rate": 5.8240, "ratio": 13.760388 }
{ "color": "yellow", "shape": "CIRCLE", "flag": 1, "index": 73, "quantity": 63.9785, "rate": 4.2370, "ratio": 15.099953 }
{ "color": "yellow", "shape": "CIRCLE", "flag": 1, "index": 87, "quantity": 63.5058, "rate": 8.3350, "ratio": 7.619172 }
{ "color": "purple", "shape": "SQUARE", "flag": 0, "index": 91, "quantity": 72.3735, "rate": 8.2430, "ratio": 8.779995 }
Sorts and stats
^^^^^^^^^^^^^^^
Now suppose you want to sort the data on a given column, *and then* take the top few in that ordering. You can use Miller's ``then`` feature to pipe commands together.
Here are the records with the top three ``index`` values::
$ mlr --icsv --opprint sort -f shape -nr index then head -n 3 example.csv
color shape flag index quantity rate
yellow circle 1 87 63.5058 8.3350
yellow circle 1 73 63.9785 4.2370
red circle 1 16 13.8103 2.9010
Lots of Miller commands take a ``-g`` option for group-by: here, ``head -n 1 -g shape`` outputs the first record for each distinct value of the ``shape`` field. This means we're finding the record with highest ``index`` field for each distinct ``shape`` field::
$ mlr --icsv --opprint sort -f shape -nr index then head -n 1 -g shape example.csv
color shape flag index quantity rate
yellow circle 1 87 63.5058 8.3350
purple square 0 91 72.3735 8.2430
purple triangle 0 65 80.1405 5.8240
Statistics can be computed with or without group-by field(s)::
$ mlr --icsv --opprint --from example.csv stats1 -a count,min,mean,max -f quantity -g shape
shape quantity_count quantity_min quantity_mean quantity_max
triangle 3 43.649800 68.339767 81.229000
square 4 72.373500 76.601150 79.277800
circle 3 13.810300 47.098200 63.978500
::
$ mlr --icsv --opprint --from example.csv stats1 -a count,min,mean,max -f quantity -g shape,color
shape color quantity_count quantity_min quantity_mean quantity_max
triangle yellow 1 43.649800 43.649800 43.649800
square red 3 77.199100 78.010367 79.277800
circle red 1 13.810300 13.810300 13.810300
triangle purple 2 80.140500 80.684750 81.229000
circle yellow 2 63.505800 63.742150 63.978500
square purple 1 72.373500 72.373500 72.373500
If your output has a lot of columns, you can use XTAB format to line things up vertically for you instead::
$ mlr --icsv --oxtab --from example.csv stats1 -a p0,p10,p25,p50,p75,p90,p99,p100 -f rate
rate_p0 0.013000
rate_p10 2.901000
rate_p25 4.237000
rate_p50 8.243000
rate_p75 8.591000
rate_p90 9.887000
rate_p99 9.887000
rate_p100 9.887000
.. _10min-choices-for-printing-to-files:
Choices for printing to files
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Often we want to print output to the screen. Miller does this by default, as we've seen in the previous examples.
Sometimes we want to print output to another file: just use **> outputfilenamegoeshere** at the end of your command:
::
% mlr --icsv --opprint cat example.csv > newfile.csv
# Output goes to the new file;
# nothing is printed to the screen.
::
% cat newfile.csv
color shape flag index quantity rate
yellow triangle 1 11 43.6498 9.8870
red square 1 15 79.2778 0.0130
red circle 1 16 13.8103 2.9010
red square 0 48 77.5542 7.4670
purple triangle 0 51 81.2290 8.5910
red square 0 64 77.1991 9.5310
purple triangle 0 65 80.1405 5.8240
yellow circle 1 73 63.9785 4.2370
yellow circle 1 87 63.5058 8.3350
purple square 0 91 72.3735 8.2430
Other times we just want our files to be **changed in-place**: just use **mlr -I**::
% cp example.csv newfile.txt
% cat newfile.txt
color,shape,flag,index,quantity,rate
yellow,triangle,1,11,43.6498,9.8870
red,square,1,15,79.2778,0.0130
red,circle,1,16,13.8103,2.9010
red,square,0,48,77.5542,7.4670
purple,triangle,0,51,81.2290,8.5910
red,square,0,64,77.1991,9.5310
purple,triangle,0,65,80.1405,5.8240
yellow,circle,1,73,63.9785,4.2370
yellow,circle,1,87,63.5058,8.3350
purple,square,0,91,72.3735,8.2430
% mlr -I --icsv --opprint cat newfile.txt
% cat newfile.txt
color shape flag index quantity rate
yellow triangle 1 11 43.6498 9.8870
red square 1 15 79.2778 0.0130
red circle 1 16 13.8103 2.9010
red square 0 48 77.5542 7.4670
purple triangle 0 51 81.2290 8.5910
red square 0 64 77.1991 9.5310
purple triangle 0 65 80.1405 5.8240
yellow circle 1 73 63.9785 4.2370
yellow circle 1 87 63.5058 8.3350
purple square 0 91 72.3735 8.2430
Also using ``mlr -I`` you can bulk-operate on lots of files: e.g.::
mlr -I --csv cut -x -f unwanted_column_name *.csv
If you like, you can first copy off your original data somewhere else, before doing in-place operations.
Lastly, using ``tee`` within ``put``, you can split your input data into separate files per one or more field names::
$ mlr --csv --from example.csv put -q 'tee > $shape.".csv", $*'
::
$ cat circle.csv
color,shape,flag,index,quantity,rate
red,circle,1,16,13.8103,2.9010
yellow,circle,1,73,63.9785,4.2370
yellow,circle,1,87,63.5058,8.3350
::
$ cat square.csv
color,shape,flag,index,quantity,rate
red,square,1,15,79.2778,0.0130
red,square,0,48,77.5542,7.4670
red,square,0,64,77.1991,9.5310
purple,square,0,91,72.3735,8.2430
::
$ cat triangle.csv
color,shape,flag,index,quantity,rate
yellow,triangle,1,11,43.6498,9.8870
purple,triangle,0,51,81.2290,8.5910
purple,triangle,0,65,80.1405,5.8240
Other-format examples
^^^^^^^^^^^^^^^^^^^^^
What's a CSV file, really? It's an array of rows, or *records*, each being a list of key-value pairs, or *fields*: for CSV it so happens that all the keys are shared in the header line and the values vary data line by data line.
For example, if you have::
shape,flag,index
circle,1,24
square,0,36
then that's a way of saying::
shape=circle,flag=1,index=24
shape=square,flag=0,index=36
Data written this way are called **DKVP**, for *delimited key-value pairs*.
We've also already seen other ways to write the same data::
CSV PPRINT JSON
shape,flag,index shape flag index [
circle,1,24 circle 1 24 {
square,0,36 square 0 36 "shape": "circle",
"flag": 1,
"index": 24
},
DKVP XTAB {
shape=circle,flag=1,index=24 shape circle "shape": "square",
shape=square,flag=0,index=36 flag 1 "flag": 0,
index 24 "index": 36
}
shape square ]
flag 0
index 36
Anything we can do with CSV input data, we can do with any other format input data. And you can read from one format, do any record-processing, and output to the same format as the input, or to a different output format.

215
docs6/10min.rst.in Normal file
View file

@ -0,0 +1,215 @@
Miller in 10 minutes
====================
CSV-file examples
^^^^^^^^^^^^^^^^^
Suppose you have this CSV data file::
POKI_RUN_COMMAND{{cat example.csv}}HERE
``mlr cat`` is like cat -- it passes the data through unmodified::
POKI_RUN_COMMAND{{mlr --csv cat example.csv}}HERE
but it can also do format conversion (here, you can pretty-print in tabular format)::
POKI_RUN_COMMAND{{mlr --icsv --opprint cat example.csv}}HERE
``mlr head`` and ``mlr tail`` count records rather than lines. Whether you're getting the first few records or the last few, the CSV header is included either way::
POKI_RUN_COMMAND{{mlr --csv head -n 4 example.csv}}HERE
::
POKI_RUN_COMMAND{{mlr --csv tail -n 4 example.csv}}HERE
You can sort primarily alphabetically on one field, then secondarily numerically descending on another field::
POKI_RUN_COMMAND{{mlr --icsv --opprint sort -f shape -nr index example.csv}}HERE
You can use ``cut`` to retain only specified fields, in the same order they appeared in the input data::
POKI_RUN_COMMAND{{mlr --icsv --opprint cut -f flag,shape example.csv}}HERE
You can also use ``cut -o`` to retain only specified fields in your preferred order::
POKI_RUN_COMMAND{{mlr --icsv --opprint cut -o -f flag,shape example.csv}}HERE
You can use ``cut -x`` to omit fields you don't care about::
POKI_RUN_COMMAND{{mlr --icsv --opprint cut -x -f flag,shape example.csv}}HERE
You can use ``filter`` to keep only records you care about::
POKI_RUN_COMMAND{{mlr --icsv --opprint filter '$color == "red"' example.csv}}HERE
::
POKI_RUN_COMMAND{{mlr --icsv --opprint filter '$color == "red" && $flag == 1' example.csv}}HERE
You can use ``put`` to create new fields which are computed from other fields::
POKI_RUN_COMMAND{{mlr --icsv --opprint put '$ratio = $quantity / $rate; $color_shape = $color . "_" . $shape' example.csv}}HERE
Even though Miller's main selling point is name-indexing, sometimes you really want to refer to a field name by its positional index. Use ``$[[3]]`` to access the name of field 3 or ``$[[[3]]]`` to access the value of field 3::
POKI_RUN_COMMAND{{mlr --icsv --opprint put '$[[3]] = "NEW"' example.csv}}HERE
::
POKI_RUN_COMMAND{{mlr --icsv --opprint put '$[[[3]]] = "NEW"' example.csv}}HERE
JSON-file examples
^^^^^^^^^^^^^^^^^^
OK, CSV and pretty-print are fine. But Miller can also convert between a few other formats -- let's take a look at JSON output::
POKI_RUN_COMMAND{{mlr --icsv --ojson put '$ratio = $quantity/$rate; $shape = toupper($shape)' example.csv}}HERE
Sorts and stats
^^^^^^^^^^^^^^^
Now suppose you want to sort the data on a given column, *and then* take the top few in that ordering. You can use Miller's ``then`` feature to pipe commands together.
Here are the records with the top three ``index`` values::
POKI_RUN_COMMAND{{mlr --icsv --opprint sort -f shape -nr index then head -n 3 example.csv}}HERE
Lots of Miller commands take a ``-g`` option for group-by: here, ``head -n 1 -g shape`` outputs the first record for each distinct value of the ``shape`` field. This means we're finding the record with highest ``index`` field for each distinct ``shape`` field::
POKI_RUN_COMMAND{{mlr --icsv --opprint sort -f shape -nr index then head -n 1 -g shape example.csv}}HERE
Statistics can be computed with or without group-by field(s)::
POKI_RUN_COMMAND{{mlr --icsv --opprint --from example.csv stats1 -a count,min,mean,max -f quantity -g shape}}HERE
::
POKI_RUN_COMMAND{{mlr --icsv --opprint --from example.csv stats1 -a count,min,mean,max -f quantity -g shape,color}}HERE
If your output has a lot of columns, you can use XTAB format to line things up vertically for you instead::
POKI_RUN_COMMAND{{mlr --icsv --oxtab --from example.csv stats1 -a p0,p10,p25,p50,p75,p90,p99,p100 -f rate}}HERE
.. _10min-choices-for-printing-to-files:
Choices for printing to files
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Often we want to print output to the screen. Miller does this by default, as we've seen in the previous examples.
Sometimes we want to print output to another file: just use **> outputfilenamegoeshere** at the end of your command:
::
% mlr --icsv --opprint cat example.csv > newfile.csv
# Output goes to the new file;
# nothing is printed to the screen.
::
% cat newfile.csv
color shape flag index quantity rate
yellow triangle 1 11 43.6498 9.8870
red square 1 15 79.2778 0.0130
red circle 1 16 13.8103 2.9010
red square 0 48 77.5542 7.4670
purple triangle 0 51 81.2290 8.5910
red square 0 64 77.1991 9.5310
purple triangle 0 65 80.1405 5.8240
yellow circle 1 73 63.9785 4.2370
yellow circle 1 87 63.5058 8.3350
purple square 0 91 72.3735 8.2430
Other times we just want our files to be **changed in-place**: just use **mlr -I**::
% cp example.csv newfile.txt
% cat newfile.txt
color,shape,flag,index,quantity,rate
yellow,triangle,1,11,43.6498,9.8870
red,square,1,15,79.2778,0.0130
red,circle,1,16,13.8103,2.9010
red,square,0,48,77.5542,7.4670
purple,triangle,0,51,81.2290,8.5910
red,square,0,64,77.1991,9.5310
purple,triangle,0,65,80.1405,5.8240
yellow,circle,1,73,63.9785,4.2370
yellow,circle,1,87,63.5058,8.3350
purple,square,0,91,72.3735,8.2430
% mlr -I --icsv --opprint cat newfile.txt
% cat newfile.txt
color shape flag index quantity rate
yellow triangle 1 11 43.6498 9.8870
red square 1 15 79.2778 0.0130
red circle 1 16 13.8103 2.9010
red square 0 48 77.5542 7.4670
purple triangle 0 51 81.2290 8.5910
red square 0 64 77.1991 9.5310
purple triangle 0 65 80.1405 5.8240
yellow circle 1 73 63.9785 4.2370
yellow circle 1 87 63.5058 8.3350
purple square 0 91 72.3735 8.2430
Also using ``mlr -I`` you can bulk-operate on lots of files: e.g.::
mlr -I --csv cut -x -f unwanted_column_name *.csv
If you like, you can first copy off your original data somewhere else, before doing in-place operations.
Lastly, using ``tee`` within ``put``, you can split your input data into separate files per one or more field names::
POKI_RUN_COMMAND{{mlr --csv --from example.csv put -q 'tee > $shape.".csv", $*'}}HERE
::
POKI_RUN_COMMAND{{cat circle.csv}}HERE
::
POKI_RUN_COMMAND{{cat square.csv}}HERE
::
POKI_RUN_COMMAND{{cat triangle.csv}}HERE
Other-format examples
^^^^^^^^^^^^^^^^^^^^^
What's a CSV file, really? It's an array of rows, or *records*, each being a list of key-value pairs, or *fields*: for CSV it so happens that all the keys are shared in the header line and the values vary data line by data line.
For example, if you have::
shape,flag,index
circle,1,24
square,0,36
then that's a way of saying::
shape=circle,flag=1,index=24
shape=square,flag=0,index=36
Data written this way are called **DKVP**, for *delimited key-value pairs*.
We've also already seen other ways to write the same data::
CSV PPRINT JSON
shape,flag,index shape flag index [
circle,1,24 circle 1 24 {
square,0,36 square 0 36 "shape": "circle",
"flag": 1,
"index": 24
},
DKVP XTAB {
shape=circle,flag=1,index=24 shape circle "shape": "square",
shape=square,flag=0,index=36 flag 1 "flag": 0,
index 24 "index": 36
}
shape square ]
flag 0
index 36
Anything we can do with CSV input data, we can do with any other format input data. And you can read from one format, do any record-processing, and output to the same format as the input, or to a different output format.

28
docs6/Makefile Normal file
View file

@ -0,0 +1,28 @@
# Minimal makefile for Sphinx documentation
#
# Note: run this after make in the ../c directory and make in the ../man directory
# since ../c/mlr is used to autogenerate ../man/manpage.txt which is used in this directory.
# See also https://miller.readthedocs.io/en/latest/build.html#creating-a-new-release-for-developers
# You can set these variables from the command line, and also
# from the environment for the first two.
SPHINXOPTS ?=
SPHINXBUILD ?= sphinx-build
SOURCEDIR = .
BUILDDIR = _build
# Respective MANPATH entries would include /usr/local/share/man or $HOME/man.
INSTALLDIR=/usr/local/share/man/man1
INSTALLHOME=$(HOME)/man/man1
# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
.PHONY: help Makefile
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
./genrst
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

40
docs6/README.md Normal file
View file

@ -0,0 +1,40 @@
# Miller Sphinx docs
## Why use Sphinx
* Connects to https://miller.readthedocs.io so people can get their docmods onto the web instead of the self-hosted https://johnkerl.org/miller/doc. Thanks to @pabloab for the great advice!
* More standard look and feel -- lots of people use readthedocs for other things so this should feel familiar
* We get a Search feature for free
## Contributing
* You need `pip install sphinx` (or `pip3 install sphinx`)
* The docs include lots of live code examples which will be invoked using `mlr` which must be somewhere in your `$PATH`
* Clone https://github.com/johnkerl/miller and cd into `docs/` within your clone
* Editing loop:
* Edit `*.rst.in`
* Run `make html`
* Either `open _build/html/index.html` (MacOS) or point your browser to `file:///path/to/your/clone/of/miller/docs/_build/html/index.html`
* Submitting:
* `git add` your modified files, `git commit`, `git push`, and submit a PR at https://github.com/johnkerl/miller
* A nice markup reference: https://www.sphinx-doc.org/en/1.8/usage/restructuredtext/basics.html
## Notes
* CSS:
* I used the Sphinx Classic theme which I like a lot except the colors -- it's a blue scheme and Miller has never been blue.
* Files are in `docs/_static/*.css` where I marked my mods with `/* CHANGE ME */`.
* If you modify the CSS you must run `make clean html` (not just `make html`) then reload in your browser.
* Live code:
* I didn't find a way to include non-Python live-code examples within Sphinx so I adapted the pre-Sphinx Miller-doc strategy which is to have a generator script read a template file (here, `foo.rst.in`), run the marked lines, and generate the output file (`foo.rst`).
* Edit the `*.rst.in` files, not `*.rst` directly.
* Within the `*.rst.in` files are lines like `POKI_RUN_COMMAND`. These will be run, and their output included, by `make html` which calls the `genrst` script for you.
* readthedocs:
* https://readthedocs.org/
* https://readthedocs.org/projects/miller/
* https://readthedocs.org/projects/miller/builds/
* https://miller.readthedocs.io/en/latest/
## To do
* Let's all discuss if/how we want the v2 docs to be structured better than the v1 docs.

860
docs6/_static/basic.css Normal file
View file

@ -0,0 +1,860 @@
/*
* basic.css
* ~~~~~~~~~
*
* Sphinx stylesheet -- basic theme.
*
* :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
* :license: BSD, see LICENSE for details.
*
*/
/* -- main layout ----------------------------------------------------------- */
div.clearer {
clear: both;
}
div.section::after {
display: block;
content: '';
clear: left;
}
/* -- relbar ---------------------------------------------------------------- */
div.related {
width: 100%;
font-size: 90%;
}
div.related h3 {
display: none;
}
div.related ul {
margin: 0;
padding: 0 0 0 10px;
list-style: none;
}
div.related li {
display: inline;
}
/* CHANGE ME */
div.body li {
margin:10px 0 0 0;
}
div.related li.right {
float: right;
margin-right: 5px;
}
/* -- sidebar --------------------------------------------------------------- */
div.sphinxsidebarwrapper {
padding: 10px 5px 0 10px;
}
div.sphinxsidebar {
float: left;
width: 230px;
margin-left: -100%;
font-size: 90%;
word-wrap: break-word;
overflow-wrap : break-word;
}
div.sphinxsidebar ul {
list-style: none;
}
div.sphinxsidebar ul ul,
div.sphinxsidebar ul.want-points {
margin-left: 20px;
list-style: square;
}
div.sphinxsidebar ul ul {
margin-top: 0;
margin-bottom: 0;
}
div.sphinxsidebar form {
margin-top: 10px;
}
div.sphinxsidebar input {
border: 1px solid #98dbcc;
font-family: sans-serif;
font-size: 1em;
}
div.sphinxsidebar #searchbox form.search {
overflow: hidden;
}
div.sphinxsidebar #searchbox input[type="text"] {
float: left;
width: 80%;
padding: 0.25em;
box-sizing: border-box;
}
div.sphinxsidebar #searchbox input[type="submit"] {
float: left;
width: 20%;
border-left: none;
padding: 0.25em;
box-sizing: border-box;
}
img {
border: 0;
max-width: 100%;
}
/* -- search page ----------------------------------------------------------- */
ul.search {
margin: 10px 0 0 20px;
padding: 0;
}
ul.search li {
padding: 5px 0 5px 20px;
background-image: url(file.png);
background-repeat: no-repeat;
background-position: 0 7px;
}
ul.search li a {
font-weight: bold;
}
ul.search li div.context {
color: #888;
margin: 2px 0 0 30px;
text-align: left;
}
ul.keywordmatches li.goodmatch a {
font-weight: bold;
}
/* -- index page ------------------------------------------------------------ */
table.contentstable {
width: 90%;
margin-left: auto;
margin-right: auto;
}
table.contentstable p.biglink {
line-height: 150%;
}
a.biglink {
font-size: 1.3em;
}
span.linkdescr {
font-style: italic;
padding-top: 5px;
font-size: 90%;
}
/* -- general index --------------------------------------------------------- */
table.indextable {
width: 100%;
}
table.indextable td {
text-align: left;
vertical-align: top;
}
table.indextable ul {
margin-top: 0;
margin-bottom: 0;
list-style-type: none;
}
table.indextable > tbody > tr > td > ul {
padding-left: 0em;
}
table.indextable tr.pcap {
height: 10px;
}
table.indextable tr.cap {
margin-top: 10px;
background-color: #f2f2f2;
}
img.toggler {
margin-right: 3px;
margin-top: 3px;
cursor: pointer;
}
div.modindex-jumpbox {
border-top: 1px solid #ddd;
border-bottom: 1px solid #ddd;
margin: 1em 0 1em 0;
padding: 0.4em;
}
div.genindex-jumpbox {
border-top: 1px solid #ddd;
border-bottom: 1px solid #ddd;
margin: 1em 0 1em 0;
padding: 0.4em;
}
/* -- domain module index --------------------------------------------------- */
table.modindextable td {
padding: 2px;
border-collapse: collapse;
}
/* -- general body styles --------------------------------------------------- */
div.body {
min-width: 450px;
max-width: 800px;
}
div.body p, div.body dd, div.body li, div.body blockquote {
-moz-hyphens: auto;
-ms-hyphens: auto;
-webkit-hyphens: auto;
hyphens: auto;
}
a.headerlink {
visibility: hidden;
}
a.brackets:before,
span.brackets > a:before{
content: "[";
}
a.brackets:after,
span.brackets > a:after {
content: "]";
}
h1:hover > a.headerlink,
h2:hover > a.headerlink,
h3:hover > a.headerlink,
h4:hover > a.headerlink,
h5:hover > a.headerlink,
h6:hover > a.headerlink,
dt:hover > a.headerlink,
caption:hover > a.headerlink,
p.caption:hover > a.headerlink,
div.code-block-caption:hover > a.headerlink {
visibility: visible;
}
div.body p.caption {
text-align: inherit;
}
div.body td {
text-align: left;
}
.first {
margin-top: 0 !important;
}
p.rubric {
margin-top: 30px;
font-weight: bold;
}
img.align-left, .figure.align-left, object.align-left {
clear: left;
float: left;
margin-right: 1em;
}
img.align-right, .figure.align-right, object.align-right {
clear: right;
float: right;
margin-left: 1em;
}
img.align-center, .figure.align-center, object.align-center {
display: block;
margin-left: auto;
margin-right: auto;
}
img.align-default, .figure.align-default {
display: block;
margin-left: auto;
margin-right: auto;
}
.align-left {
text-align: left;
}
.align-center {
text-align: center;
}
.align-default {
text-align: center;
}
.align-right {
text-align: right;
}
/* -- sidebars -------------------------------------------------------------- */
div.sidebar {
margin: 0 0 0.5em 1em;
border: 1px solid #ddb;
padding: 7px;
background-color: #ffe;
width: 40%;
float: right;
clear: right;
overflow-x: auto;
}
p.sidebar-title {
font-weight: bold;
}
div.admonition, div.topic, blockquote {
clear: left;
}
/* -- topics ---------------------------------------------------------------- */
div.topic {
border: 1px solid #ccc;
padding: 7px;
margin: 10px 0 10px 0;
}
p.topic-title {
font-size: 1.1em;
font-weight: bold;
margin-top: 10px;
}
/* -- admonitions ----------------------------------------------------------- */
div.admonition {
margin-top: 10px;
margin-bottom: 10px;
padding: 7px;
}
div.admonition dt {
font-weight: bold;
}
p.admonition-title {
margin: 0px 10px 5px 0px;
font-weight: bold;
}
div.body p.centered {
text-align: center;
margin-top: 25px;
}
/* -- content of sidebars/topics/admonitions -------------------------------- */
div.sidebar > :last-child,
div.topic > :last-child,
div.admonition > :last-child {
margin-bottom: 0;
}
div.sidebar::after,
div.topic::after,
div.admonition::after,
blockquote::after {
display: block;
content: '';
clear: both;
}
/* -- tables ---------------------------------------------------------------- */
table.docutils {
margin-top: 10px;
margin-bottom: 10px;
border: 0;
border-collapse: collapse;
}
table.align-center {
margin-left: auto;
margin-right: auto;
}
table.align-default {
margin-left: auto;
margin-right: auto;
}
table caption span.caption-number {
font-style: italic;
}
table caption span.caption-text {
}
table.docutils td, table.docutils th {
padding: 1px 8px 1px 5px;
border-top: 0;
border-left: 0;
border-right: 0;
border-bottom: 1px solid #aaa;
}
table.footnote td, table.footnote th {
border: 0 !important;
}
th {
text-align: left;
padding-right: 5px;
}
table.citation {
border-left: solid 1px gray;
margin-left: 1px;
}
table.citation td {
border-bottom: none;
}
th > :first-child,
td > :first-child {
margin-top: 0px;
}
th > :last-child,
td > :last-child {
margin-bottom: 0px;
}
/* -- figures --------------------------------------------------------------- */
div.figure {
margin: 0.5em;
padding: 0.5em;
}
div.figure p.caption {
padding: 0.3em;
}
div.figure p.caption span.caption-number {
font-style: italic;
}
div.figure p.caption span.caption-text {
}
/* -- field list styles ----------------------------------------------------- */
table.field-list td, table.field-list th {
border: 0 !important;
}
.field-list ul {
margin: 0;
padding-left: 1em;
}
.field-list p {
margin: 0;
}
.field-name {
-moz-hyphens: manual;
-ms-hyphens: manual;
-webkit-hyphens: manual;
hyphens: manual;
}
/* -- hlist styles ---------------------------------------------------------- */
table.hlist {
margin: 1em 0;
}
table.hlist td {
vertical-align: top;
}
/* -- other body styles ----------------------------------------------------- */
ol.arabic {
list-style: decimal;
}
ol.loweralpha {
list-style: lower-alpha;
}
ol.upperalpha {
list-style: upper-alpha;
}
ol.lowerroman {
list-style: lower-roman;
}
ol.upperroman {
list-style: upper-roman;
}
:not(li) > ol > li:first-child > :first-child,
:not(li) > ul > li:first-child > :first-child {
margin-top: 0px;
}
:not(li) > ol > li:last-child > :last-child,
:not(li) > ul > li:last-child > :last-child {
margin-bottom: 0px;
}
ol.simple ol p,
ol.simple ul p,
ul.simple ol p,
ul.simple ul p {
margin-top: 0;
}
ol.simple > li:not(:first-child) > p,
ul.simple > li:not(:first-child) > p {
margin-top: 0;
}
ol.simple p,
ul.simple p {
margin-bottom: 0;
}
dl.footnote > dt,
dl.citation > dt {
float: left;
margin-right: 0.5em;
}
dl.footnote > dd,
dl.citation > dd {
margin-bottom: 0em;
}
dl.footnote > dd:after,
dl.citation > dd:after {
content: "";
clear: both;
}
dl.field-list {
display: grid;
grid-template-columns: fit-content(30%) auto;
}
dl.field-list > dt {
font-weight: bold;
word-break: break-word;
padding-left: 0.5em;
padding-right: 5px;
}
dl.field-list > dt:after {
content: ":";
}
dl.field-list > dd {
padding-left: 0.5em;
margin-top: 0em;
margin-left: 0em;
margin-bottom: 0em;
}
dl {
margin-bottom: 15px;
}
dd > :first-child {
margin-top: 0px;
}
dd ul, dd table {
margin-bottom: 10px;
}
dd {
margin-top: 3px;
margin-bottom: 10px;
margin-left: 30px;
}
dl > dd:last-child,
dl > dd:last-child > :last-child {
margin-bottom: 0;
}
dt:target, span.highlighted {
background-color: #0be54e;
}
rect.highlighted {
fill: #fbe54e;
}
dl.glossary dt {
font-weight: bold;
font-size: 1.1em;
}
.optional {
font-size: 1.3em;
}
.sig-paren {
font-size: larger;
}
.versionmodified {
font-style: italic;
}
.system-message {
background-color: #fda;
padding: 5px;
border: 3px solid red;
}
.footnote:target {
background-color: #ffa;
}
.line-block {
display: block;
margin-top: 1em;
margin-bottom: 1em;
}
.line-block .line-block {
margin-top: 0;
margin-bottom: 0;
margin-left: 1.5em;
}
.guilabel, .menuselection {
font-family: sans-serif;
}
.accelerator {
text-decoration: underline;
}
.classifier {
font-style: oblique;
}
.classifier:before {
font-style: normal;
margin: 0.5em;
content: ":";
}
abbr, acronym {
border-bottom: dotted 1px;
cursor: help;
}
/* -- code displays --------------------------------------------------------- */
pre {
overflow: auto;
overflow-y: hidden; /* fixes display issues on Chrome browsers */
}
pre, div[class*="highlight-"] {
clear: both;
}
span.pre {
-moz-hyphens: none;
-ms-hyphens: none;
-webkit-hyphens: none;
hyphens: none;
}
div[class*="highlight-"] {
margin: 1em 0;
}
td.linenos pre {
border: 0;
background-color: transparent;
color: #aaa;
}
table.highlighttable {
display: block;
}
table.highlighttable tbody {
display: block;
}
table.highlighttable tr {
display: flex;
}
table.highlighttable td {
margin: 0;
padding: 0;
}
table.highlighttable td.linenos {
padding-right: 0.5em;
}
table.highlighttable td.code {
flex: 1;
overflow: hidden;
}
.highlight .hll {
display: block;
}
div.highlight pre,
table.highlighttable pre {
margin: 0;
}
div.code-block-caption + div {
margin-top: 0;
}
div.code-block-caption {
margin-top: 1em;
padding: 2px 5px;
font-size: small;
}
div.code-block-caption code {
background-color: transparent;
}
table.highlighttable td.linenos,
div.doctest > div.highlight span.gp { /* gp: Generic.Prompt */
user-select: none;
}
div.code-block-caption span.caption-number {
padding: 0.1em 0.3em;
font-style: italic;
}
div.code-block-caption span.caption-text {
}
div.literal-block-wrapper {
margin: 1em 0;
}
code.descname {
background-color: transparent;
font-weight: bold;
font-size: 1.2em;
}
code.descclassname {
background-color: transparent;
}
code.xref, a code {
background-color: transparent;
font-weight: bold;
}
h1 code, h2 code, h3 code, h4 code, h5 code, h6 code {
background-color: transparent;
}
.viewcode-link {
float: right;
}
.viewcode-back {
float: right;
font-family: sans-serif;
}
div.viewcode-block:target {
margin: -1px -10px;
padding: 0 10px;
}
/* -- math display ---------------------------------------------------------- */
img.math {
vertical-align: middle;
}
div.body div.math p {
text-align: center;
}
span.eqno {
float: right;
}
span.eqno a.headerlink {
position: absolute;
z-index: 1;
}
div.math:hover a.headerlink {
visibility: visible;
}
/* -- printout stylesheet --------------------------------------------------- */
@media print {
div.document,
div.documentwrapper,
div.bodywrapper {
margin: 0 !important;
width: 100%;
}
div.sphinxsidebar,
div.related,
div.footer,
#top-link {
display: none;
}
}

271
docs6/_static/classic.css Normal file
View file

@ -0,0 +1,271 @@
/*
* classic.css_t
* ~~~~~~~~~~~~~
*
* Sphinx stylesheet -- classic theme.
*
* :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
* :license: BSD, see LICENSE for details.
*
*/
@import url("basic.css");
/* -- page layout ----------------------------------------------------------- */
html {
/* CSS hack for macOS's scrollbar (see #1125) */
background-color: #FFFFFF;
}
body {
font-family: sans-serif;
font-size: 100%;
background-color: #808080;
color: #000;
margin: 0;
padding: 0;
}
div.document {
/* CHANGE ME */
background-color: #c0c0c0;
}
div.documentwrapper {
float: left;
width: 100%;
}
div.bodywrapper {
margin: 0 0 0 230px;
}
div.body {
background-color: #ffffff;
color: #000000;
padding: 0 20px 30px 20px;
}
div.footer {
color: #ffffff;
width: 100%;
padding: 9px 0 9px 0;
text-align: center;
font-size: 75%;
}
div.footer a {
color: #ffffff;
text-decoration: underline;
}
div.related {
/* CHANGE ME */
background-color: #808080;
line-height: 30px;
color: #000000;
}
div.related a {
/* CHANGE ME */
color: #000000;
}
div.sphinxsidebar {
}
div.sphinxsidebar h3 {
font-family: 'Trebuchet MS', sans-serif;
color: #000000;
font-size: 1.4em;
font-weight: normal;
margin: 0;
padding: 0;
}
div.sphinxsidebar h3 a {
color: #000000;
}
div.sphinxsidebar h4 {
font-family: 'Trebuchet MS', sans-serif;
color: #000000;
font-size: 1.3em;
font-weight: normal;
margin: 5px 0 0 0;
padding: 0;
}
div.sphinxsidebar p {
color: #000000;
}
div.sphinxsidebar p.topless {
margin: 5px 10px 10px 10px;
}
div.sphinxsidebar ul {
margin: 10px;
padding: 0;
color: #ffffff;
}
div.sphinxsidebar a {
/* CHANGE ME */
color: #404040;
}
div.sphinxsidebar input {
border: 1px solid #98dbcc;
font-family: sans-serif;
font-size: 1em;
}
/* -- hyperlink styles ------------------------------------------------------ */
a {
color: #800000;
text-decoration: none;
}
a:visited {
color: #800000;
text-decoration: none;
}
a:hover {
text-decoration: underline;
}
/* -- body styles ----------------------------------------------------------- */
div.body h1,
div.body h2,
div.body h3,
div.body h4,
div.body h5,
div.body h6 {
font-family: 'Trebuchet MS', sans-serif;
background-color: #f2f2f2;
font-weight: normal;
/* CHANGE ME */
color: #800000;
border-bottom: 1px solid #ccc;
margin: 20px -20px 10px -20px;
padding: 3px 0 3px 10px;
}
div.body h1 { margin-top: 0; font-size: 200%; }
div.body h2 { font-size: 160%; }
div.body h3 { font-size: 140%; }
div.body h4 { font-size: 120%; }
div.body h5 { font-size: 110%; }
div.body h6 { font-size: 100%; }
a.headerlink {
color: #c60f0f;
font-size: 0.8em;
padding: 0 4px 0 4px;
text-decoration: none;
}
a.headerlink:hover {
background-color: #c60f0f;
color: white;
}
div.body p, div.body dd, div.body li, div.body blockquote {
text-align: justify;
line-height: 130%;
}
div.admonition p.admonition-title + p {
display: inline;
}
div.admonition p {
margin-bottom: 5px;
}
div.admonition pre {
margin-bottom: 5px;
}
div.admonition ul, div.admonition ol {
margin-bottom: 5px;
}
div.note {
background-color: #eee;
border: 1px solid #ccc;
}
div.seealso {
background-color: #ffc;
border: 1px solid #ff6;
}
div.topic {
background-color: #eee;
}
div.warning {
background-color: #ffe4e4;
border: 1px solid #f66;
}
p.admonition-title {
display: inline;
}
p.admonition-title:after {
content: ":";
}
pre {
padding: 5px;
background-color: unset;
color: unset;
line-height: 120%;
border: 1px solid #ac9;
border-left: none;
border-right: none;
}
code {
background-color: #ecf0f3;
padding: 0 1px 0 1px;
font-size: 0.95em;
}
th, dl.field-list > dt {
background-color: #ede;
}
.warning code {
background: #efc2c2;
}
.note code {
background: #d6d6d6;
}
.viewcode-back {
font-family: sans-serif;
}
div.viewcode-block:target {
background-color: #f4debf;
border-top: 1px solid #ac9;
border-bottom: 1px solid #ac9;
}
div.code-block-caption {
color: #efefef;
background-color: #1c4e63;
}

76
docs6/_static/pygments.css Normal file
View file

@ -0,0 +1,76 @@
pre { line-height: 125%; margin: 0; }
td.linenos pre { color: #000000; background-color: #f0f0f0; padding: 0 5px 0 5px; }
span.linenos { color: #000000; background-color: #f0f0f0; padding: 0 5px 0 5px; }
td.linenos pre.special { color: #000000; background-color: #ffffc0; padding: 0 5px 0 5px; }
span.linenos.special { color: #000000; background-color: #ffffc0; padding: 0 5px 0 5px; }
.highlight .hll { background-color: #ffffcc }
/*.highlight { background: #eeffcc; }*/
/* CHANGE ME */
.highlight { background: #eae2cb; }
.highlight .c { color: #408090; font-style: italic } /* Comment */
.highlight .err { border: 1px solid #FF0000 } /* Error */
.highlight .k { color: #007020; font-weight: bold } /* Keyword */
.highlight .o { color: #666666 } /* Operator */
.highlight .ch { color: #408090; font-style: italic } /* Comment.Hashbang */
.highlight .cm { color: #408090; font-style: italic } /* Comment.Multiline */
.highlight .cp { color: #007020 } /* Comment.Preproc */
.highlight .cpf { color: #408090; font-style: italic } /* Comment.PreprocFile */
.highlight .c1 { color: #408090; font-style: italic } /* Comment.Single */
.highlight .cs { color: #408090; background-color: #fff0f0 } /* Comment.Special */
.highlight .gd { color: #A00000 } /* Generic.Deleted */
.highlight .ge { font-style: italic } /* Generic.Emph */
.highlight .gr { color: #FF0000 } /* Generic.Error */
.highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */
.highlight .gi { color: #00A000 } /* Generic.Inserted */
.highlight .go { color: #333333 } /* Generic.Output */
.highlight .gp { color: #c65d09; font-weight: bold } /* Generic.Prompt */
.highlight .gs { font-weight: bold } /* Generic.Strong */
.highlight .gu { color: #800080; font-weight: bold } /* Generic.Subheading */
.highlight .gt { color: #0044DD } /* Generic.Traceback */
.highlight .kc { color: #007020; font-weight: bold } /* Keyword.Constant */
.highlight .kd { color: #007020; font-weight: bold } /* Keyword.Declaration */
.highlight .kn { color: #007020; font-weight: bold } /* Keyword.Namespace */
.highlight .kp { color: #007020 } /* Keyword.Pseudo */
.highlight .kr { color: #007020; font-weight: bold } /* Keyword.Reserved */
.highlight .kt { color: #902000 } /* Keyword.Type */
.highlight .m { color: #208050 } /* Literal.Number */
.highlight .s { color: #4070a0 } /* Literal.String */
.highlight .na { color: #4070a0 } /* Name.Attribute */
.highlight .nb { color: #007020 } /* Name.Builtin */
.highlight .nc { color: #0e84b5; font-weight: bold } /* Name.Class */
.highlight .no { color: #60add5 } /* Name.Constant */
.highlight .nd { color: #555555; font-weight: bold } /* Name.Decorator */
.highlight .ni { color: #d55537; font-weight: bold } /* Name.Entity */
.highlight .ne { color: #007020 } /* Name.Exception */
.highlight .nf { color: #06287e } /* Name.Function */
.highlight .nl { color: #002070; font-weight: bold } /* Name.Label */
.highlight .nn { color: #0e84b5; font-weight: bold } /* Name.Namespace */
.highlight .nt { color: #062873; font-weight: bold } /* Name.Tag */
.highlight .nv { color: #bb60d5 } /* Name.Variable */
.highlight .ow { color: #007020; font-weight: bold } /* Operator.Word */
.highlight .w { color: #bbbbbb } /* Text.Whitespace */
.highlight .mb { color: #208050 } /* Literal.Number.Bin */
.highlight .mf { color: #208050 } /* Literal.Number.Float */
.highlight .mh { color: #208050 } /* Literal.Number.Hex */
.highlight .mi { color: #208050 } /* Literal.Number.Integer */
.highlight .mo { color: #208050 } /* Literal.Number.Oct */
.highlight .sa { color: #4070a0 } /* Literal.String.Affix */
.highlight .sb { color: #4070a0 } /* Literal.String.Backtick */
.highlight .sc { color: #4070a0 } /* Literal.String.Char */
.highlight .dl { color: #4070a0 } /* Literal.String.Delimiter */
.highlight .sd { color: #4070a0; font-style: italic } /* Literal.String.Doc */
.highlight .s2 { color: #4070a0 } /* Literal.String.Double */
.highlight .se { color: #4070a0; font-weight: bold } /* Literal.String.Escape */
.highlight .sh { color: #4070a0 } /* Literal.String.Heredoc */
.highlight .si { color: #70a0d0; font-style: italic } /* Literal.String.Interpol */
.highlight .sx { color: #c65d09 } /* Literal.String.Other */
.highlight .sr { color: #235388 } /* Literal.String.Regex */
.highlight .s1 { color: #4070a0 } /* Literal.String.Single */
.highlight .ss { color: #517918 } /* Literal.String.Symbol */
.highlight .bp { color: #007020 } /* Name.Builtin.Pseudo */
.highlight .fm { color: #06287e } /* Name.Function.Magic */
.highlight .vc { color: #bb60d5 } /* Name.Variable.Class */
.highlight .vg { color: #bb60d5 } /* Name.Variable.Global */
.highlight .vi { color: #bb60d5 } /* Name.Variable.Instance */
.highlight .vm { color: #bb60d5 } /* Name.Variable.Magic */
.highlight .il { color: #208050 } /* Literal.Number.Integer.Long */

131
docs6/build.rst Normal file
View file

@ -0,0 +1,131 @@
..
PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE.
Building from source
================================================================
Please also see :doc:`install` for information about pre-built executables.
Miller license
----------------------------------------------------------------
Two-clause BSD license https://github.com/johnkerl/miller/blob/master/LICENSE.txt.
From release tarball
----------------------------------------------------------------
* Obtain ``mlr-i.j.k.tar.gz`` from https://github.com/johnkerl/miller/tags, replacing ``i.j.k`` with the desired release, e.g. ``2.2.1``.
* ``tar zxvf mlr-i.j.k.tar.gz``
* ``cd mlr-i.j.k``
* Install the following packages using your system's package manager (``apt-get``, ``yum install``, etc.): **flex**
* Various configuration options of your choice, e.g.
* ``./configure``
* ``./configure --prefix=/usr/local``
* ``./configure --prefix=$HOME/pkgs``
* ``./configure CC=clang``
* ``./configure --disable-shared`` (to make a statically linked executable)
* ``./configure 'CFLAGS=-Wall -std=gnu99 -O3'``
* etc.
* ``make`` creates the ``c/mlr`` executable
* ``make check``
* ``make install`` copies the ``c/mlr`` executable to your prefix's ``bin`` subdirectory.
From git clone
----------------------------------------------------------------
* ``git clone https://github.com/johnkerl/miller``
* ``cd miller/go``
* ``./build``
In case of problems
----------------------------------------------------------------
If you have any build errors, feel free to contact me at mailto:kerl.john.r+miller@gmail.com -- or, better, open an issue with "New Issue" at https://github.com/johnkerl/miller/issues.
Dependencies
----------------------------------------------------------------
Required external dependencies
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
These are necessary to produce the ``mlr`` executable.
* Go version 1.16 or higher
Optional external dependencies
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This documentation pageset is built using Sphinx. Please see `./README.md` for details.
Creating a new release: for developers
----------------------------------------------------------------
At present I'm the primary developer so this is just my checklist for making new releases.
In this example I am using version 3.4.0; of course that will change for subsequent revisions.
* Update version found in ``mlr --version`` and ``man mlr``:
* Edit ``configure.ac``, ``c/mlrvers.h``, ``miller.spec``, and ``docs/conf.py`` from ``3.3.2-dev`` to ``3.4.0``.
* Do a fresh ``autoreconf -fiv`` and commit the output. (Preferably on a Linux host, rather than MacOS, to reduce needless diffs in autogen build files.)
* ``make -C c -f Makefile.no-autoconfig installhome && make -C man -f Makefile.no-autoconfig installhome && make -C docs -f Makefile.no-autoconfig html``
* The ordering is important: the first build creates ``mlr``; the second runs ``mlr`` to create ``manpage.txt``; the third includes ``manpage.txt`` into one of its outputs.
* Commit and push.
* Create the release tarball and SRPM:
* On buildbox: ``./configure && make distcheck``
* On buildbox: make SRPM as in https://github.com/johnkerl/miller/blob/master/README-RPM.md
* On all buildboxes: ``cd c`` and ``make -f Makefile.no-autoconfig mlr.static``. Then copy ``mlr.static`` to ``../mlr.{arch}``. (This may require as prerequisite ``sudo yum install glibc-static`` or the like.)
* For static binaries, please do ``ldd mlr.static`` and make sure it says ``not a dynamic executable``.
* Then ``mv mlr.static ../mlr.linux_x86_64``
* Pull back release tarball ``mlr-3.4.0.tar.gz`` and SRPM ``miller-3.4.0-1.el6.src.rpm`` from buildbox, and ``mlr.{arch}`` binaries from whatever buildboxes.
* Download ``mlr.exe`` and ``msys-2.0.dll`` from https://ci.appveyor.com/project/johnkerl/miller/build/artifacts.
* Create the Github release tag:
* Don't forget the ``v`` in ``v3.4.0``
* Write the release notes
* Attach the release tarball, SRPM, and binaries. Double-check assets were successfully uploaded.
* Publish the release
* Check the release-specific docs:
* Look at https://miller.readthedocs.io for new-version docs, after a few minutes' propagation time.
* Notify:
* Submit ``brew`` pull request; notify any other distros which don't appear to have autoupdated since the previous release (notes below)
* Similarly for ``macports``: https://github.com/macports/macports-ports/blob/master/textproc/miller/Portfile.
* Social-media updates.
::
git remote add upstream https://github.com/Homebrew/homebrew-core # one-time setup only
git fetch upstream
git rebase upstream/master
git checkout -b miller-3.4.0
shasum -a 256 /path/to/mlr-3.4.0.tar.gz
edit Formula/miller.rb
# Test the URL from the line like
# url "https://github.com/johnkerl/miller/releases/download/v3.4.0/mlr-3.4.0.tar.gz"
# in a browser for typos
# A '@BrewTestBot Test this please' comment within the homebrew-core pull request will restart the homebrew travis build
git add Formula/miller.rb
git commit -m 'miller 3.4.0'
git push -u origin miller-3.4.0
(submit the pull request)
* Afterwork:
* Edit ``configure.ac`` and ``c/mlrvers.h`` to change version from ``3.4.0`` to ``3.4.0-dev``.
* ``make -C c -f Makefile.no-autoconfig installhome && make -C doc -f Makefile.no-autoconfig all installhome``
* Commit and push.
Misc. development notes
----------------------------------------------------------------
I use terminal width 120 and tabwidth 4.

128
docs6/build.rst.in Normal file
View file

@ -0,0 +1,128 @@
Building from source
================================================================
Please also see :doc:`install` for information about pre-built executables.
Miller license
----------------------------------------------------------------
Two-clause BSD license https://github.com/johnkerl/miller/blob/master/LICENSE.txt.
From release tarball
----------------------------------------------------------------
* Obtain ``mlr-i.j.k.tar.gz`` from https://github.com/johnkerl/miller/tags, replacing ``i.j.k`` with the desired release, e.g. ``2.2.1``.
* ``tar zxvf mlr-i.j.k.tar.gz``
* ``cd mlr-i.j.k``
* Install the following packages using your system's package manager (``apt-get``, ``yum install``, etc.): **flex**
* Various configuration options of your choice, e.g.
* ``./configure``
* ``./configure --prefix=/usr/local``
* ``./configure --prefix=$HOME/pkgs``
* ``./configure CC=clang``
* ``./configure --disable-shared`` (to make a statically linked executable)
* ``./configure 'CFLAGS=-Wall -std=gnu99 -O3'``
* etc.
* ``make`` creates the ``c/mlr`` executable
* ``make check``
* ``make install`` copies the ``c/mlr`` executable to your prefix's ``bin`` subdirectory.
From git clone
----------------------------------------------------------------
* ``git clone https://github.com/johnkerl/miller``
* ``cd miller/go``
* ``./build``
In case of problems
----------------------------------------------------------------
If you have any build errors, feel free to contact me at mailto:kerl.john.r+miller@gmail.com -- or, better, open an issue with "New Issue" at https://github.com/johnkerl/miller/issues.
Dependencies
----------------------------------------------------------------
Required external dependencies
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
These are necessary to produce the ``mlr`` executable.
* Go version 1.16 or higher
Optional external dependencies
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This documentation pageset is built using Sphinx. Please see `./README.md` for details.
Creating a new release: for developers
----------------------------------------------------------------
At present I'm the primary developer so this is just my checklist for making new releases.
In this example I am using version 3.4.0; of course that will change for subsequent revisions.
* Update version found in ``mlr --version`` and ``man mlr``:
* Edit ``configure.ac``, ``c/mlrvers.h``, ``miller.spec``, and ``docs/conf.py`` from ``3.3.2-dev`` to ``3.4.0``.
* Do a fresh ``autoreconf -fiv`` and commit the output. (Preferably on a Linux host, rather than MacOS, to reduce needless diffs in autogen build files.)
* ``make -C c -f Makefile.no-autoconfig installhome && make -C man -f Makefile.no-autoconfig installhome && make -C docs -f Makefile.no-autoconfig html``
* The ordering is important: the first build creates ``mlr``; the second runs ``mlr`` to create ``manpage.txt``; the third includes ``manpage.txt`` into one of its outputs.
* Commit and push.
* Create the release tarball and SRPM:
* On buildbox: ``./configure && make distcheck``
* On buildbox: make SRPM as in https://github.com/johnkerl/miller/blob/master/README-RPM.md
* On all buildboxes: ``cd c`` and ``make -f Makefile.no-autoconfig mlr.static``. Then copy ``mlr.static`` to ``../mlr.{arch}``. (This may require as prerequisite ``sudo yum install glibc-static`` or the like.)
* For static binaries, please do ``ldd mlr.static`` and make sure it says ``not a dynamic executable``.
* Then ``mv mlr.static ../mlr.linux_x86_64``
* Pull back release tarball ``mlr-3.4.0.tar.gz`` and SRPM ``miller-3.4.0-1.el6.src.rpm`` from buildbox, and ``mlr.{arch}`` binaries from whatever buildboxes.
* Download ``mlr.exe`` and ``msys-2.0.dll`` from https://ci.appveyor.com/project/johnkerl/miller/build/artifacts.
* Create the Github release tag:
* Don't forget the ``v`` in ``v3.4.0``
* Write the release notes
* Attach the release tarball, SRPM, and binaries. Double-check assets were successfully uploaded.
* Publish the release
* Check the release-specific docs:
* Look at https://miller.readthedocs.io for new-version docs, after a few minutes' propagation time.
* Notify:
* Submit ``brew`` pull request; notify any other distros which don't appear to have autoupdated since the previous release (notes below)
* Similarly for ``macports``: https://github.com/macports/macports-ports/blob/master/textproc/miller/Portfile.
* Social-media updates.
::
git remote add upstream https://github.com/Homebrew/homebrew-core # one-time setup only
git fetch upstream
git rebase upstream/master
git checkout -b miller-3.4.0
shasum -a 256 /path/to/mlr-3.4.0.tar.gz
edit Formula/miller.rb
# Test the URL from the line like
# url "https://github.com/johnkerl/miller/releases/download/v3.4.0/mlr-3.4.0.tar.gz"
# in a browser for typos
# A '@BrewTestBot Test this please' comment within the homebrew-core pull request will restart the homebrew travis build
git add Formula/miller.rb
git commit -m 'miller 3.4.0'
git push -u origin miller-3.4.0
(submit the pull request)
* Afterwork:
* Edit ``configure.ac`` and ``c/mlrvers.h`` to change version from ``3.4.0`` to ``3.4.0-dev``.
* ``make -C c -f Makefile.no-autoconfig installhome && make -C doc -f Makefile.no-autoconfig all installhome``
* Commit and push.
Misc. development notes
----------------------------------------------------------------
I use terminal width 120 and tabwidth 4.

4
docs6/circle.csv Normal file
View file

@ -0,0 +1,4 @@
color,shape,flag,index,quantity,rate
red,circle,1,16,13.8103,2.9010
yellow,circle,1,73,63.9785,4.2370
yellow,circle,1,87,63.5058,8.3350
1 color shape flag index quantity rate
2 red circle 1 16 13.8103 2.9010
3 yellow circle 1 73 63.9785 4.2370
4 yellow circle 1 87 63.5058 8.3350

3
docs6/commas.csv Normal file
View file

@ -0,0 +1,3 @@
Name,Role
"Xiao, Lin",administrator
"Khavari, Darius",tester
1 Name Role
2 Xiao, Lin administrator
3 Khavari, Darius tester

57
docs6/conf.py Normal file
View file

@ -0,0 +1,57 @@
# Configuration file for the Sphinx documentation builder.
#
# This file only contains a selection of the most common options. For a full
# list see the documentation:
# https://www.sphinx-doc.org/en/master/usage/configuration.html
# -- Path setup --------------------------------------------------------------
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
# import os
# import sys
# sys.path.insert(0, os.path.abspath('.'))
# -- Project information -----------------------------------------------------
project = 'Miller'
copyright = '2020, John Kerl'
author = 'John Kerl'
# The full version, including alpha/beta/rc tags
release = '5.10.2'
# -- General configuration ---------------------------------------------------
master_doc = 'index'
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
]
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
# -- Options for HTML output -------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
#html_theme = 'alabaster'
html_theme = 'classic'
#html_theme = 'sphinxdoc'
#html_theme = 'nature'
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']

11
docs6/contact.rst Normal file
View file

@ -0,0 +1,11 @@
..
PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE.
Contact
================================================================
Bug reports, feature requests, etc.: https://github.com/johnkerl/miller/issues
For issues involving this documentation site please also use https://github.com/johnkerl/miller/issues
Other correspondence: mailto:kerl.john.r+miller@gmail.com

8
docs6/contact.rst.in Normal file
View file

@ -0,0 +1,8 @@
Contact
================================================================
Bug reports, feature requests, etc.: https://github.com/johnkerl/miller/issues
For issues involving this documentation site please also use https://github.com/johnkerl/miller/issues
Other correspondence: mailto:kerl.john.r+miller@gmail.com

1143
docs6/cookbook.rst Normal file
View file

File diff suppressed because it is too large Load diff

563
docs6/cookbook.rst.in Normal file
View file

@ -0,0 +1,563 @@
Cookbook part 1: common patterns
================================================================
Headerless CSV on input or output
----------------------------------------------------------------
Sometimes we get CSV files which lack a header. For example:
::
POKI_RUN_COMMAND{{cat data/headerless.csv}}HERE
You can use Miller to add a header. The ``--implicit-csv-header`` applies positionally indexed labels:
::
POKI_RUN_COMMAND{{mlr --csv --implicit-csv-header cat data/headerless.csv}}HERE
Following that, you can rename the positionally indexed labels to names with meaning for your context. For example:
::
POKI_RUN_COMMAND{{mlr --csv --implicit-csv-header label name,age,status data/headerless.csv}}HERE
Likewise, if you need to produce CSV which is lacking its header, you can pipe Miller's output to the system command ``sed 1d``, or you can use Miller's ``--headerless-csv-output`` option:
::
POKI_RUN_COMMAND{{head -5 data/colored-shapes.dkvp | mlr --ocsv cat}}HERE
::
POKI_RUN_COMMAND{{head -5 data/colored-shapes.dkvp | mlr --ocsv --headerless-csv-output cat}}HERE
Lastly, often we say "CSV" or "TSV" when we have positionally indexed data in columns which are separated by commas or tabs, respectively. In this case it's perhaps simpler to **just use NIDX format** which was designed for this purpose. (See also :doc:`file-formats`.) For example:
::
POKI_RUN_COMMAND{{mlr --inidx --ifs comma --oxtab cut -f 1,3 data/headerless.csv}}HERE
Doing multiple joins
----------------------------------------------------------------
Suppose we have the following data:
::
POKI_RUN_COMMAND{{cat multi-join/input.csv}}HERE
And we want to augment the ``id`` column with lookups from the following data files:
::
POKI_RUN_COMMAND{{cat multi-join/name-lookup.csv}}HERE
::
POKI_RUN_COMMAND{{cat multi-join/status-lookup.csv}}HERE
We can run the input file through multiple ``join`` commands in a ``then``-chain:
::
POKI_RUN_COMMAND{{mlr --icsv --opprint join -f multi-join/name-lookup.csv -j id then join -f multi-join/status-lookup.csv -j id multi-join/input.csv}}HERE
Bulk rename of fields
----------------------------------------------------------------
Suppose you want to replace spaces with underscores in your column names:
::
POKI_RUN_COMMAND{{cat data/spaces.csv}}HERE
The simplest way is to use ``mlr rename`` with ``-g`` (for global replace, not just first occurrence of space within each field) and ``-r`` for pattern-matching (rather than explicit single-column renames):
::
POKI_RUN_COMMAND{{mlr --csv rename -g -r ' ,_' data/spaces.csv}}HERE
::
POKI_RUN_COMMAND{{mlr --csv --opprint rename -g -r ' ,_' data/spaces.csv}}HERE
You can also do this with a for-loop:
::
POKI_RUN_COMMAND{{cat data/bulk-rename-for-loop.mlr}}HERE
::
POKI_RUN_COMMAND{{mlr --icsv --opprint put -f data/bulk-rename-for-loop.mlr data/spaces.csv}}HERE
Search-and-replace over all fields
----------------------------------------------------------------
How to do ``$name = gsub($name, "old", "new")`` for all fields?
::
POKI_RUN_COMMAND{{cat data/sar.csv}}HERE
::
POKI_RUN_COMMAND{{cat data/sar.mlr}}HERE
::
POKI_RUN_COMMAND{{mlr --csv put -f data/sar.mlr data/sar.csv}}HERE
Full field renames and reassigns
----------------------------------------------------------------
Using Miller 5.0.0's map literals and assigning to ``$*``, you can fully generalize :ref:`mlr rename <reference-verbs-rename>`, :ref:`mlr reorder <reference-verbs-reorder>`, etc.
::
POKI_RUN_COMMAND{{cat data/small}}HERE
::
POKI_INCLUDE_AND_RUN_ESCAPED(data/full-reorg.sh)HERE
Numbering and renumbering records
----------------------------------------------------------------
The ``awk``-like built-in variable ``NR`` is incremented for each input record:
::
POKI_RUN_COMMAND{{cat data/small}}HERE
::
POKI_RUN_COMMAND{{mlr put '$nr = NR' data/small}}HERE
However, this is the record number within the original input stream -- not after any filtering you may have done:
::
POKI_RUN_COMMAND{{mlr filter '$a == "wye"' then put '$nr = NR' data/small}}HERE
There are two good options here. One is to use the ``cat`` verb with ``-n``:
::
POKI_RUN_COMMAND{{mlr filter '$a == "wye"' then cat -n data/small}}HERE
The other is to keep your own counter within the ``put`` DSL:
::
POKI_RUN_COMMAND{{mlr filter '$a == "wye"' then put 'begin {@n = 1} $n = @n; @n += 1' data/small}}HERE
The difference is a matter of taste (although ``mlr cat -n`` puts the counter first).
Options for dealing with duplicate rows
----------------------------------------------------------------
If your data has records appearing multiple times, you can use :ref:`mlr uniq <reference-verbs-uniq>` to show and/or count the unique records.
If you want to look at partial uniqueness -- for example, show only the first record for each unique combination of the ``account_id`` and ``account_status`` fields -- you might use ``mlr head -n 1 -g account_id,account_status``. Please also see :ref:`mlr head <reference-verbs-head>`.
.. _cookbook-data-cleaning-examples:
Data-cleaning examples
----------------------------------------------------------------
Here are some ways to use the type-checking options as described in :ref:`reference-dsl-type-tests-and-assertions` Suppose you have the following data file, with inconsistent typing for boolean. (Also imagine that, for the sake of discussion, we have a million-line file rather than a four-line file, so we can't see it all at once and some automation is called for.)
::
POKI_RUN_COMMAND{{cat data/het-bool.csv}}HERE
One option is to coerce everything to boolean, or integer:
::
POKI_RUN_COMMAND{{mlr --icsv --opprint put '$reachable = boolean($reachable)' data/het-bool.csv}}HERE
::
POKI_RUN_COMMAND{{mlr --icsv --opprint put '$reachable = int(boolean($reachable))' data/het-bool.csv}}HERE
A second option is to flag badly formatted data within the output stream:
::
POKI_RUN_COMMAND{{mlr --icsv --opprint put '$format_ok = is_string($reachable)' data/het-bool.csv}}HERE
Or perhaps to flag badly formatted data outside the output stream:
::
POKI_RUN_COMMAND{{mlr --icsv --opprint put 'if (!is_string($reachable)) {eprint "Malformed at NR=".NR} ' data/het-bool.csv}}HERE
A third way is to abort the process on first instance of bad data:
::
POKI_RUN_COMMAND_TOLERATING_ERROR{{mlr --csv put '$reachable = asserting_string($reachable)' data/het-bool.csv}}HERE
Splitting nested fields
----------------------------------------------------------------
Suppose you have a TSV file like this:
::
POKI_INCLUDE_ESCAPED(data/nested.tsv)HERE
The simplest option is to use :ref:`mlr nest <reference-verbs-nest>`:
::
POKI_RUN_COMMAND{{mlr --tsv nest --explode --values --across-records -f b --nested-fs : data/nested.tsv}}HERE
::
POKI_RUN_COMMAND{{mlr --tsv nest --explode --values --across-fields -f b --nested-fs : data/nested.tsv}}HERE
While ``mlr nest`` is simplest, let's also take a look at a few ways to do this using the ``put`` DSL.
One option to split out the colon-delimited values in the ``b`` column is to use ``splitnv`` to create an integer-indexed map and loop over it, adding new fields to the current record:
::
POKI_RUN_COMMAND{{mlr --from data/nested.tsv --itsv --oxtab put 'o=splitnv($b, ":"); for (k,v in o) {$["p".k]=v}'}}HERE
while another is to loop over the same map from ``splitnv`` and use it (with ``put -q`` to suppress printing the original record) to produce multiple records:
::
POKI_RUN_COMMAND{{mlr --from data/nested.tsv --itsv --oxtab put -q 'o=splitnv($b, ":"); for (k,v in o) {x=mapsum($*, {"b":v}); emit x}'}}HERE
::
POKI_RUN_COMMAND{{mlr --from data/nested.tsv --tsv put -q 'o=splitnv($b, ":"); for (k,v in o) {x=mapsum($*, {"b":v}); emit x}'}}HERE
Showing differences between successive queries
----------------------------------------------------------------
Suppose you have a database query which you run at one point in time, producing the output on the left, then again later producing the output on the right:
::
POKI_RUN_COMMAND{{cat data/previous_counters.csv}}HERE
::
POKI_RUN_COMMAND{{cat data/current_counters.csv}}HERE
And, suppose you want to compute the differences in the counters between adjacent keys. Since the color names aren't all in the same order, nor are they all present on both sides, we can't just paste the two files side-by-side and do some column-four-minus-column-two arithmetic.
First, rename counter columns to make them distinct:
::
POKI_RUN_COMMAND{{mlr --csv rename count,previous_count data/previous_counters.csv > data/prevtemp.csv}}HERE
::
POKI_RUN_COMMAND{{cat data/prevtemp.csv}}HERE
::
POKI_RUN_COMMAND{{mlr --csv rename count,current_count data/current_counters.csv > data/currtemp.csv}}HERE
::
POKI_RUN_COMMAND{{cat data/currtemp.csv}}HERE
Then, join on the key field(s), and use unsparsify to zero-fill counters absent on one side but present on the other. Use ``--ul`` and ``--ur`` to emit unpaired records (namely, purple on the left and yellow on the right):
::
POKI_INCLUDE_AND_RUN_ESCAPED(data/previous-to-current.sh)HERE
Finding missing dates
----------------------------------------------------------------
Suppose you have some date-stamped data which may (or may not) be missing entries for one or more dates:
::
POKI_RUN_COMMAND{{head -n 10 data/miss-date.csv}}HERE
::
POKI_RUN_COMMAND{{wc -l data/miss-date.csv}}HERE
Since there are 1372 lines in the data file, some automation is called for. To find the missing dates, you can convert the dates to seconds since the epoch using ``strptime``, then compute adjacent differences (the ``cat -n`` simply inserts record-counters):
::
POKI_INCLUDE_AND_RUN_ESCAPED(data/miss-date-1.sh)HERE
Then, filter for adjacent difference not being 86400 (the number of seconds in a day):
::
POKI_INCLUDE_AND_RUN_ESCAPED(data/miss-date-2.sh)HERE
Given this, it's now easy to see where the gaps are:
::
POKI_RUN_COMMAND{{mlr cat -n then filter '$n >= 770 && $n <= 780' data/miss-date.csv}}HERE
::
POKI_RUN_COMMAND{{mlr cat -n then filter '$n >= 1115 && $n <= 1125' data/miss-date.csv}}HERE
Two-pass algorithms
----------------------------------------------------------------
Miller is a streaming record processor; commands are performed once per record. This makes Miller particularly suitable for single-pass algorithms, allowing many of its verbs to process files that are (much) larger than the amount of RAM present in your system. (Of course, Miller verbs such as ``sort``, ``tac``, etc. all must ingest and retain all input records before emitting any output records.) You can also use out-of-stream variables to perform multi-pass computations, at the price of retaining all input records in memory.
Two-pass algorithms: computation of percentages
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
For example, mapping numeric values down a column to the percentage between their min and max values is two-pass: on the first pass you find the min and max values, then on the second, map each record's value to a percentage.
::
POKI_INCLUDE_AND_RUN_ESCAPED(data/two-pass-percentage.sh)HERE
Two-pass algorithms: line-number ratios
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Similarly, finding the total record count requires first reading through all the data:
::
POKI_INCLUDE_AND_RUN_ESCAPED(data/two-pass-record-numbers.sh)HERE
Two-pass algorithms: records having max value
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The idea is to retain records having the largest value of ``n`` in the following data:
::
POKI_RUN_COMMAND{{mlr --itsv --opprint cat data/maxrows.tsv}}HERE
Of course, the largest value of ``n`` isn't known until after all data have been read. Using an out-of-stream variable we can retain all records as they are read, then filter them at the end:
::
POKI_RUN_COMMAND{{cat data/maxrows.mlr}}HERE
::
POKI_RUN_COMMAND{{mlr --itsv --opprint put -q -f data/maxrows.mlr data/maxrows.tsv}}HERE
Rectangularizing data
----------------------------------------------------------------
Suppose you have a method (in whatever language) which is printing things of the form
::
POKI_INCLUDE_ESCAPED(data/rect-outer.txt)HERE
and then calls another method which prints things of the form
::
POKI_INCLUDE_ESCAPED(data/rect-middle.txt)HERE
and then, perhaps, that second method calls a third method which prints things of the form
::
POKI_INCLUDE_ESCAPED(data/rect-inner.txt)HERE
with the result that your program's output is
::
POKI_INCLUDE_ESCAPED(data/rect.txt)HERE
The idea here is that middles starting with a 1 belong to the outer value of 1, and so on. (For example, the outer values might be account IDs, the middle values might be invoice IDs, and the inner values might be invoice line-items.) If you want all the middle and inner lines to have the context of which outers they belong to, you can modify your software to pass all those through your methods. Alternatively, don't refactor your code just to handle some ad-hoc log-data formatting -- instead, use the following to rectangularize the data. The idea is to use an out-of-stream variable to accumulate fields across records. Clear that variable when you see an outer ID; accumulate fields; emit output when you see the inner IDs.
::
POKI_INCLUDE_AND_RUN_ESCAPED(data/rect.sh)HERE
Regularizing ragged CSV
----------------------------------------------------------------
Miller handles compliant CSV: in particular, it's an error if the number of data fields in a given data line don't match the number of header lines. But in the event that you have a CSV file in which some lines have less than the full number of fields, you can use Miller to pad them out. The trick is to use NIDX format, for which each line stands on its own without respect to a header line.
::
POKI_RUN_COMMAND{{cat data/ragged.csv}}HERE
::
POKI_INCLUDE_AND_RUN_ESCAPED(data/ragged-csv.sh)HERE
or, more simply,
::
POKI_INCLUDE_AND_RUN_ESCAPED(data/ragged-csv-2.sh)HERE
Feature-counting
----------------------------------------------------------------
Suppose you have some heterogeneous data like this:
::
POKI_INCLUDE_ESCAPED(data/features.json)HERE
A reasonable question to ask is, how many occurrences of each field are there? And, what percentage of total row count has each of them? Since the denominator of the percentage is not known until the end, this is a two-pass algorithm:
::
POKI_INCLUDE_ESCAPED(data/feature-count.mlr)HERE
Then
::
POKI_RUN_COMMAND{{mlr --json put -q -f data/feature-count.mlr data/features.json}}HERE
::
POKI_RUN_COMMAND{{mlr --ijson --opprint put -q -f data/feature-count.mlr data/features.json}}HERE
Unsparsing
----------------------------------------------------------------
The previous section discussed how to fill out missing data fields within CSV with full header line -- so the list of all field names is present within the header line. Next, let's look at a related problem: we have data where each record has various key names but we want to produce rectangular output having the union of all key names.
For example, suppose you have JSON input like this:
::
POKI_RUN_COMMAND{{cat data/sparse.json}}HERE
There are field names ``a``, ``b``, ``v``, ``u``, ``x``, ``w`` in the data -- but not all in every record. Since we don't know the names of all the keys until we've read them all, this needs to be a two-pass algorithm. On the first pass, remember all the unique key names and all the records; on the second pass, loop through the records filling in absent values, then producing output. Use ``put -q`` since we don't want to produce per-record output, only emitting output in the ``end`` block:
::
POKI_RUN_COMMAND{{cat data/unsparsify.mlr}}HERE
::
POKI_RUN_COMMAND{{mlr --json put -q -f data/unsparsify.mlr data/sparse.json}}HERE
::
POKI_RUN_COMMAND{{mlr --ijson --ocsv put -q -f data/unsparsify.mlr data/sparse.json}}HERE
::
POKI_RUN_COMMAND{{mlr --ijson --opprint put -q -f data/unsparsify.mlr data/sparse.json}}HERE
There is a keystroke-saving verb for this: :ref:`mlr unsparsify <reference-verbs-unsparsify>`.
Parsing log-file output
----------------------------------------------------------------
This, of course, depends highly on what's in your log files. But, as an example, suppose you have log-file lines such as
::
2015-10-08 08:29:09,445 INFO com.company.path.to.ClassName @ [sometext] various/sorts/of data {& punctuation} hits=1 status=0 time=2.378
I prefer to pre-filter with ``grep`` and/or ``sed`` to extract the structured text, then hand that to Miller. Example:
::
grep 'various sorts' *.log | sed 's/.*} //' | mlr --fs space --repifs --oxtab stats1 -a min,p10,p50,p90,max -f time -g status
.. _cookbook-memoization-with-oosvars:
Memoization with out-of-stream variables
----------------------------------------------------------------
The recursive function for the Fibonacci sequence is famous for its computational complexity. Namely, using *f*(0)=1, *f*(1)=1, *f*(*n*)=*f*(*n*-1)+*f*(*n*-2) for *n*&ge;2, the evaluation tree branches left as well as right at each non-trivial level, resulting in millions or more paths to the root 0/1 nodes for larger *n*. This program
::
POKI_INCLUDE_ESCAPED(data/fibo-uncached.sh)HERE
produces output like this:
::
i o fcount seconds_delta
1 1 1 0
2 2 3 0.000039101
3 3 5 0.000015974
4 5 9 0.000019073
5 8 15 0.000026941
6 13 25 0.000036955
7 21 41 0.000056028
8 34 67 0.000086069
9 55 109 0.000134945
10 89 177 0.000217915
11 144 287 0.000355959
12 233 465 0.000506163
13 377 753 0.000811815
14 610 1219 0.001297235
15 987 1973 0.001960993
16 1597 3193 0.003417969
17 2584 5167 0.006215811
18 4181 8361 0.008294106
19 6765 13529 0.012095928
20 10946 21891 0.019592047
21 17711 35421 0.031193972
22 28657 57313 0.057254076
23 46368 92735 0.080307961
24 75025 150049 0.129482031
25 121393 242785 0.213325977
26 196418 392835 0.334423065
27 317811 635621 0.605969906
28 514229 1028457 0.971235037
Note that the time it takes to evaluate the function is blowing up exponentially as the input argument increases. Using ``@``-variables, which persist across records, we can cache and reuse the results of previous computations:
::
POKI_INCLUDE_ESCAPED(data/fibo-cached.sh)HERE
with output like this:
::
i o fcount seconds_delta
1 1 1 0
2 2 3 0.000053883
3 3 3 0.000035048
4 5 3 0.000045061
5 8 3 0.000014067
6 13 3 0.000028849
7 21 3 0.000028133
8 34 3 0.000027895
9 55 3 0.000014067
10 89 3 0.000015020
11 144 3 0.000012875
12 233 3 0.000033140
13 377 3 0.000014067
14 610 3 0.000012875
15 987 3 0.000029087
16 1597 3 0.000013828
17 2584 3 0.000013113
18 4181 3 0.000012875
19 6765 3 0.000013113
20 10946 3 0.000012875
21 17711 3 0.000013113
22 28657 3 0.000013113
23 46368 3 0.000015974
24 75025 3 0.000012875
25 121393 3 0.000013113
26 196418 3 0.000012875
27 317811 3 0.000013113
28 514229 3 0.000012875

526
docs6/cookbook2.rst Normal file
View file

@ -0,0 +1,526 @@
..
PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE.
Cookbook part 2: Random things, and some math
================================================================
Randomly selecting words from a list
----------------------------------------------------------------
Given this `word list <https://github.com/johnkerl/miller/blob/master/docs/data/english-words.txt>`_, first take a look to see what the first few lines look like:
::
$ head data/english-words.txt
a
aa
aal
aalii
aam
aardvark
aardwolf
aba
abac
abaca
Then the following will randomly sample ten words with four to eight characters in them:
::
$ mlr --from data/english-words.txt --nidx filter -S 'n=strlen($1);4<=n&&n<=8' then sample -k 10
thionine
birchman
mildewy
avigate
addedly
abaze
askant
aiming
insulant
coinmate
Randomly generating jabberwocky words
----------------------------------------------------------------
These are simple *n*-grams as `described here <http://johnkerl.org/randspell/randspell-slides-ts.pdf>`_. Some common functions are `located here <https://github.com/johnkerl/miller/blob/master/docs/ngrams/ngfuncs.mlr.txt>`_. Then here are scripts for `1-grams <https://github.com/johnkerl/miller/blob/master/docs/ngrams/ng1.mlr.txt>`_ `2-grams <https://github.com/johnkerl/miller/blob/master/docs/ngrams/ng2.mlr.txt>`_ `3-grams <https://github.com/johnkerl/miller/blob/master/docs/ngrams/ng3.mlr.txt>`_ `4-grams <https://github.com/johnkerl/miller/blob/master/docs/ngrams/ng4.mlr.txt>`_, and `5-grams <https://github.com/johnkerl/miller/blob/master/docs/ngrams/ng5.mlr.txt>`_.
The idea is that words from the input file are consumed, then taken apart and pasted back together in ways which imitate the letter-to-letter transitions found in the word list -- giving us automatically generated words in the same vein as *bromance* and *spork*:
::
$ mlr --nidx --from ./ngrams/gsl-2000.txt put -q -f ./ngrams/ngfuncs.mlr -f ./ngrams/ng5.mlr
beard
plastinguish
politicially
noise
loan
country
controductionary
suppery
lose
lessors
dollar
judge
rottendence
lessenger
diffendant
suggestional
Program timing
----------------------------------------------------------------
This admittedly artificial example demonstrates using Miller time and stats functions to introspectively acquire some information about Miller's own runtime. The ``delta`` function computes the difference between successive timestamps.
::
$ ruby -e '10000.times{|i|puts "i=#{i+1}"}' > lines.txt
$ head -n 5 lines.txt
i=1
i=2
i=3
i=4
i=5
mlr --ofmt '%.9le' --opprint put '$t=systime()' then step -a delta -f t lines.txt | head -n 7
i t t_delta
1 1430603027.018016 1.430603027e+09
2 1430603027.018043 2.694129944e-05
3 1430603027.018048 5.006790161e-06
4 1430603027.018052 4.053115845e-06
5 1430603027.018055 2.861022949e-06
6 1430603027.018058 3.099441528e-06
mlr --ofmt '%.9le' --oxtab \
put '$t=systime()' then \
step -a delta -f t then \
filter '$i>1' then \
stats1 -a min,mean,max -f t_delta \
lines.txt
t_delta_min 2.861022949e-06
t_delta_mean 4.077508505e-06
t_delta_max 5.388259888e-05
Computing interquartile ranges
----------------------------------------------------------------
For one or more specified field names, simply compute p25 and p75, then write the IQR as the difference of p75 and p25:
::
$ mlr --oxtab stats1 -f x -a p25,p75 \
then put '$x_iqr = $x_p75 - $x_p25' \
data/medium
x_p25 0.246670
x_p75 0.748186
x_iqr 0.501516
For wildcarded field names, first compute p25 and p75, then loop over field names with ``p25`` in them:
::
$ mlr --oxtab stats1 --fr '[i-z]' -a p25,p75 \
then put 'for (k,v in $*) {
if (k =~ "(.*)_p25") {
$["\1_iqr"] = $["\1_p75"] - $["\1_p25"]
}
}' \
data/medium
i_p25 2501
i_p75 7501
x_p25 0.246670
x_p75 0.748186
y_p25 0.252137
y_p75 0.764003
i_iqr 5000
x_iqr 0.501516
y_iqr 0.511866
Computing weighted means
----------------------------------------------------------------
This might be more elegantly implemented as an option within the ``stats1`` verb. Meanwhile, it's expressible within the DSL:
::
$ mlr --from data/medium put -q '
# Using the y field for weighting in this example
weight = $y;
# Using the a field for weighted aggregation in this example
@sumwx[$a] += weight * $i;
@sumw[$a] += weight;
@sumx[$a] += $i;
@sumn[$a] += 1;
end {
map wmean = {};
map mean = {};
for (a in @sumwx) {
wmean[a] = @sumwx[a] / @sumw[a]
}
for (a in @sumx) {
mean[a] = @sumx[a] / @sumn[a]
}
#emit wmean, "a";
#emit mean, "a";
emit (wmean, mean), "a";
}'
a=pan,wmean=4979.563722,mean=5028.259010
a=eks,wmean=4890.381593,mean=4956.290076
a=wye,wmean=4946.987746,mean=4920.001017
a=zee,wmean=5164.719685,mean=5123.092330
a=hat,wmean=4925.533162,mean=4967.743946
Generating random numbers from various distributions
----------------------------------------------------------------
Here we can chain together a few simple building blocks:
::
$ cat expo-sample.sh
# Generate 100,000 pairs of independent and identically distributed
# exponentially distributed random variables with the same rate parameter
# (namely, 2.5). Then compute histograms of one of them, along with
# histograms for their sum and their product.
#
# See also https://en.wikipedia.org/wiki/Exponential_distribution
#
# Here I'm using a specified random-number seed so this example always
# produces the same output for this web document: in everyday practice we
# wouldn't do that.
mlr -n \
--seed 0 \
--opprint \
seqgen --stop 100000 \
then put '
# https://en.wikipedia.org/wiki/Inverse_transform_sampling
func expo_sample(lambda) {
return -log(1-urand())/lambda
}
$u = expo_sample(2.5);
$v = expo_sample(2.5);
$s = $u + $v;
$p = $u * $v;
' \
then histogram -f u,s,p --lo 0 --hi 2 --nbins 50 \
then bar -f u_count,s_count,p_count --auto -w 20
Namely:
* Set the Miller random-number seed so this webdoc looks the same every time I regenerate it.
* Use pretty-printed tabular output.
* Use pretty-printed tabular output.
* Use ``seqgen`` to produce 100,000 records ``i=0``, ``i=1``, etc.
* Send those to a ``put`` step which defines an inverse-transform-sampling function and calls it twice, then computes the sum and product of samples.
* Send those to a histogram, and from there to a bar-plotter. This is just for visualization; you could just as well output CSV and send that off to your own plotting tool, etc.
The output is as follows:
::
$ sh expo-sample.sh
bin_lo bin_hi u_count s_count p_count
0.000000 0.040000 [78]*******************#[9497] [353]#...................[3732] [20]*******************#[39755]
0.040000 0.080000 [78]******************..[9497] [353]*****...............[3732] [20]*******.............[39755]
0.080000 0.120000 [78]****************....[9497] [353]*********...........[3732] [20]****................[39755]
0.120000 0.160000 [78]**************......[9497] [353]************........[3732] [20]***.................[39755]
0.160000 0.200000 [78]*************.......[9497] [353]**************......[3732] [20]**..................[39755]
0.200000 0.240000 [78]************........[9497] [353]****************....[3732] [20]*...................[39755]
0.240000 0.280000 [78]**********..........[9497] [353]******************..[3732] [20]*...................[39755]
0.280000 0.320000 [78]**********..........[9497] [353]******************..[3732] [20]*...................[39755]
0.320000 0.360000 [78]*********...........[9497] [353]*******************.[3732] [20]#...................[39755]
0.360000 0.400000 [78]********............[9497] [353]*******************.[3732] [20]#...................[39755]
0.400000 0.440000 [78]*******.............[9497] [353]*******************#[3732] [20]#...................[39755]
0.440000 0.480000 [78]******..............[9497] [353]******************..[3732] [20]#...................[39755]
0.480000 0.520000 [78]*****...............[9497] [353]******************..[3732] [20]#...................[39755]
0.520000 0.560000 [78]*****...............[9497] [353]******************..[3732] [20]#...................[39755]
0.560000 0.600000 [78]****................[9497] [353]*****************...[3732] [20]#...................[39755]
0.600000 0.640000 [78]****................[9497] [353]*****************...[3732] [20]#...................[39755]
0.640000 0.680000 [78]****................[9497] [353]****************....[3732] [20]#...................[39755]
0.680000 0.720000 [78]***.................[9497] [353]****************....[3732] [20]#...................[39755]
0.720000 0.760000 [78]***.................[9497] [353]**************......[3732] [20]#...................[39755]
0.760000 0.800000 [78]**..................[9497] [353]**************......[3732] [20]#...................[39755]
0.800000 0.840000 [78]**..................[9497] [353]*************.......[3732] [20]#...................[39755]
0.840000 0.880000 [78]**..................[9497] [353]************........[3732] [20]#...................[39755]
0.880000 0.920000 [78]**..................[9497] [353]***********.........[3732] [20]#...................[39755]
0.920000 0.960000 [78]*...................[9497] [353]***********.........[3732] [20]#...................[39755]
0.960000 1.000000 [78]*...................[9497] [353]**********..........[3732] [20]#...................[39755]
1.000000 1.040000 [78]*...................[9497] [353]*********...........[3732] [20]#...................[39755]
1.040000 1.080000 [78]*...................[9497] [353]*********...........[3732] [20]#...................[39755]
1.080000 1.120000 [78]*...................[9497] [353]********............[3732] [20]#...................[39755]
1.120000 1.160000 [78]*...................[9497] [353]********............[3732] [20]#...................[39755]
1.160000 1.200000 [78]#...................[9497] [353]*******.............[3732] [20]#...................[39755]
1.200000 1.240000 [78]#...................[9497] [353]******..............[3732] [20]#...................[39755]
1.240000 1.280000 [78]#...................[9497] [353]*****...............[3732] [20]#...................[39755]
1.280000 1.320000 [78]#...................[9497] [353]*****...............[3732] [20]#...................[39755]
1.320000 1.360000 [78]#...................[9497] [353]*****...............[3732] [20]#...................[39755]
1.360000 1.400000 [78]#...................[9497] [353]****................[3732] [20]#...................[39755]
1.400000 1.440000 [78]#...................[9497] [353]****................[3732] [20]#...................[39755]
1.440000 1.480000 [78]#...................[9497] [353]***.................[3732] [20]#...................[39755]
1.480000 1.520000 [78]#...................[9497] [353]***.................[3732] [20]#...................[39755]
1.520000 1.560000 [78]#...................[9497] [353]***.................[3732] [20]#...................[39755]
1.560000 1.600000 [78]#...................[9497] [353]**..................[3732] [20]#...................[39755]
1.600000 1.640000 [78]#...................[9497] [353]**..................[3732] [20]#...................[39755]
1.640000 1.680000 [78]#...................[9497] [353]*...................[3732] [20]#...................[39755]
1.680000 1.720000 [78]#...................[9497] [353]*...................[3732] [20]#...................[39755]
1.720000 1.760000 [78]#...................[9497] [353]*...................[3732] [20]#...................[39755]
1.760000 1.800000 [78]#...................[9497] [353]*...................[3732] [20]#...................[39755]
1.800000 1.840000 [78]#...................[9497] [353]#...................[3732] [20]#...................[39755]
1.840000 1.880000 [78]#...................[9497] [353]#...................[3732] [20]#...................[39755]
1.880000 1.920000 [78]#...................[9497] [353]#...................[3732] [20]#...................[39755]
1.920000 1.960000 [78]#...................[9497] [353]#...................[3732] [20]#...................[39755]
1.960000 2.000000 [78]#...................[9497] [353]#...................[3732] [20]#...................[39755]
Sieve of Eratosthenes
----------------------------------------------------------------
The `Sieve of Eratosthenes <http://en.wikipedia.org/wiki/Sieve_of_Eratosthenes>`_ is a standard introductory programming topic. The idea is to find all primes up to some *N* by making a list of the numbers 1 to *N*, then striking out all multiples of 2 except 2 itself, all multiples of 3 except 3 itself, all multiples of 4 except 4 itself, and so on. Whatever survives that without getting marked is a prime. This is easy enough in Miller. Notice that here all the work is in ``begin`` and ``end`` statements; there is no file input (so we use ``mlr -n`` to keep Miller from waiting for input data).
::
$ cat programs/sieve.mlr
# ================================================================
# Sieve of Eratosthenes: simple example of Miller DSL as programming language.
# ================================================================
# Put this in a begin-block so we can do either
# mlr -n put -q -f name-of-this-file.mlr
# or
# mlr -n put -q -f name-of-this-file.mlr -e '@n = 200'
# i.e. 100 is the default upper limit, and another can be specified using -e.
begin {
@n = 100;
}
end {
for (int i = 0; i <= @n; i += 1) {
@s[i] = true;
}
@s[0] = false; # 0 is neither prime nor composite
@s[1] = false; # 1 is neither prime nor composite
# Strike out multiples
for (int i = 2; i <= @n; i += 1) {
for (int j = i+i; j <= @n; j += i) {
@s[j] = false;
}
}
# Print survivors
for (int i = 0; i <= @n; i += 1) {
if (@s[i]) {
print i;
}
}
}
::
$ mlr -n put -f programs/sieve.mlr
2
3
5
7
11
13
17
19
23
29
31
37
41
43
47
53
59
61
67
71
73
79
83
89
97
Mandelbrot-set generator
----------------------------------------------------------------
The `Mandelbrot set <http://en.wikipedia.org/wiki/Mandelbrot_set>`_ is also easily expressed. This isn't an important case of data-processing in the vein for which Miller was designed, but it is an example of Miller as a general-purpose programming language -- a test case for the expressiveness of the language.
The (approximate) computation of points in the complex plane which are and aren't members is just a few lines of complex arithmetic (see the Wikipedia article); how to render them is another task. Using graphics libraries you can create PNG or JPEG files, but another fun way to do this is by printing various characters to the screen:
::
$ cat programs/mand.mlr
# Mandelbrot set generator: simple example of Miller DSL as programming language.
begin {
# Set defaults
@rcorn = -2.0;
@icorn = -2.0;
@side = 4.0;
@iheight = 50;
@iwidth = 100;
@maxits = 100;
@levelstep = 5;
@chars = "@X*o-."; # Palette of characters to print to the screen.
@verbose = false;
@do_julia = false;
@jr = 0.0; # Real part of Julia point, if any
@ji = 0.0; # Imaginary part of Julia point, if any
}
# Here, we can override defaults from an input file (if any). In Miller's
# put/filter DSL, absent-null right-hand sides result in no assignment so we
# can simply put @rcorn = $rcorn: if there is a field in the input like
# 'rcorn = -1.847' we'll read and use it, else we'll keep the default.
@rcorn = $rcorn;
@icorn = $icorn;
@side = $side;
@iheight = $iheight;
@iwidth = $iwidth;
@maxits = $maxits;
@levelstep = $levelstep;
@chars = $chars;
@verbose = $verbose;
@do_julia = $do_julia;
@jr = $jr;
@ji = $ji;
end {
if (@verbose) {
print "RCORN = ".@rcorn;
print "ICORN = ".@icorn;
print "SIDE = ".@side;
print "IHEIGHT = ".@iheight;
print "IWIDTH = ".@iwidth;
print "MAXITS = ".@maxits;
print "LEVELSTEP = ".@levelstep;
print "CHARS = ".@chars;
}
# Iterate over a matrix of rows and columns, printing one character for each cell.
for (int ii = @iheight-1; ii >= 0; ii -= 1) {
num pi = @icorn + (ii/@iheight) * @side;
for (int ir = 0; ir < @iwidth; ir += 1) {
num pr = @rcorn + (ir/@iwidth) * @side;
printn get_point_plot(pr, pi, @maxits, @do_julia, @jr, @ji);
}
print;
}
}
# This is a function to approximate membership in the Mandelbrot set (or Julia
# set for a given Julia point if do_julia == true) for a given point in the
# complex plane.
func get_point_plot(pr, pi, maxits, do_julia, jr, ji) {
num zr = 0.0;
num zi = 0.0;
num cr = 0.0;
num ci = 0.0;
if (!do_julia) {
zr = 0.0;
zi = 0.0;
cr = pr;
ci = pi;
} else {
zr = pr;
zi = pi;
cr = jr;
ci = ji;
}
int iti = 0;
bool escaped = false;
num zt = 0;
for (iti = 0; iti < maxits; iti += 1) {
num mag = zr*zr + zi+zi;
if (mag > 4.0) {
escaped = true;
break;
}
# z := z^2 + c
zt = zr*zr - zi*zi + cr;
zi = 2*zr*zi + ci;
zr = zt;
}
if (!escaped) {
return ".";
} else {
# The // operator is Miller's (pythonic) integer-division operator
int level = (iti // @levelstep) % strlen(@chars);
return substr(@chars, level, level);
}
}
At standard resolution this makes a nice little ASCII plot:
::
$ mlr -n put -f ./programs/mand.mlr
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXX.XXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXooXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXX**o..*XXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXX*-....-oXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXX@XXXXXXXXXX*......o*XXXXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXXX**oo*-.-........oo.XXXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXXXXXX....................X..o-XXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@XXXXXXXXXXXXXXX*oo......................oXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@XXX*XXXXXXXXXXXX**o........................*X*X@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@XXXXXXooo***o*.*XX**X..........................o-XX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@XXXXXXXX*-.......-***.............................oXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@XXXXXXXX*@..........Xo............................*XX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@XXXX@XXXXXXXX*o@oX...........@...........................oXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
.........................................................o*XXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@XXXXXXXXX*-.oX...........@...........................oXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@XXXXXXXXXX**@..........*o............................*XXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@XXXXXXXXXXXXX-........***.............................oXXXXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@XXXXXXXXXXXXoo****o*.XX***@..........................o-XXXXXXXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@XXXXX*XXXX*XXXXXXX**-........................***XXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@XXXXXXXXXXXXX*o*.....................@o*XXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXXXXX*....................*..o-XX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXX*ooo*-.o........oo.X*XXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXX**@.....*XXXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXX*o....-o*XXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXXXo*o..*XXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXXXXXX*o*XXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXXXXX@XXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXX@@XXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
But using a very small font size (as small as my Mac will let me go), and by choosing the coordinates to zoom in on a particular part of the complex plane, we can get a nice little picture:
::
#!/bin/bash
# Get the number of rows and columns from the terminal window dimensions
iheight=$(stty size | mlr --nidx --fs space cut -f 1)
iwidth=$(stty size | mlr --nidx --fs space cut -f 2)
echo "rcorn=-1.755350,icorn=+0.014230,side=0.000020,maxits=10000,iheight=$iheight,iwidth=$iwidth" \
| mlr put -f programs/mand.mlr
.. image:: pix/mand.png

164
docs6/cookbook2.rst.in Normal file
View file

@ -0,0 +1,164 @@
Cookbook part 2: Random things, and some math
================================================================
Randomly selecting words from a list
----------------------------------------------------------------
Given this `word list <https://github.com/johnkerl/miller/blob/master/docs/data/english-words.txt>`_, first take a look to see what the first few lines look like:
::
$ head data/english-words.txt
a
aa
aal
aalii
aam
aardvark
aardwolf
aba
abac
abaca
Then the following will randomly sample ten words with four to eight characters in them:
::
$ mlr --from data/english-words.txt --nidx filter -S 'n=strlen($1);4<=n&&n<=8' then sample -k 10
thionine
birchman
mildewy
avigate
addedly
abaze
askant
aiming
insulant
coinmate
Randomly generating jabberwocky words
----------------------------------------------------------------
These are simple *n*-grams as `described here <http://johnkerl.org/randspell/randspell-slides-ts.pdf>`_. Some common functions are `located here <https://github.com/johnkerl/miller/blob/master/docs/ngrams/ngfuncs.mlr.txt>`_. Then here are scripts for `1-grams <https://github.com/johnkerl/miller/blob/master/docs/ngrams/ng1.mlr.txt>`_ `2-grams <https://github.com/johnkerl/miller/blob/master/docs/ngrams/ng2.mlr.txt>`_ `3-grams <https://github.com/johnkerl/miller/blob/master/docs/ngrams/ng3.mlr.txt>`_ `4-grams <https://github.com/johnkerl/miller/blob/master/docs/ngrams/ng4.mlr.txt>`_, and `5-grams <https://github.com/johnkerl/miller/blob/master/docs/ngrams/ng5.mlr.txt>`_.
The idea is that words from the input file are consumed, then taken apart and pasted back together in ways which imitate the letter-to-letter transitions found in the word list -- giving us automatically generated words in the same vein as *bromance* and *spork*:
::
$ mlr --nidx --from ./ngrams/gsl-2000.txt put -q -f ./ngrams/ngfuncs.mlr -f ./ngrams/ng5.mlr
beard
plastinguish
politicially
noise
loan
country
controductionary
suppery
lose
lessors
dollar
judge
rottendence
lessenger
diffendant
suggestional
Program timing
----------------------------------------------------------------
This admittedly artificial example demonstrates using Miller time and stats functions to introspectively acquire some information about Miller's own runtime. The ``delta`` function computes the difference between successive timestamps.
::
POKI_INCLUDE_ESCAPED(data/timing-example.txt)HERE
Computing interquartile ranges
----------------------------------------------------------------
For one or more specified field names, simply compute p25 and p75, then write the IQR as the difference of p75 and p25:
::
POKI_INCLUDE_AND_RUN_ESCAPED(data/iqr1.sh)HERE
For wildcarded field names, first compute p25 and p75, then loop over field names with ``p25`` in them:
::
POKI_INCLUDE_AND_RUN_ESCAPED(data/iqrn.sh)HERE
Computing weighted means
----------------------------------------------------------------
This might be more elegantly implemented as an option within the ``stats1`` verb. Meanwhile, it's expressible within the DSL:
::
POKI_INCLUDE_AND_RUN_ESCAPED(data/weighted-mean.sh)HERE
Generating random numbers from various distributions
----------------------------------------------------------------
Here we can chain together a few simple building blocks:
::
POKI_RUN_COMMAND{{cat expo-sample.sh}}HERE
Namely:
* Set the Miller random-number seed so this webdoc looks the same every time I regenerate it.
* Use pretty-printed tabular output.
* Use pretty-printed tabular output.
* Use ``seqgen`` to produce 100,000 records ``i=0``, ``i=1``, etc.
* Send those to a ``put`` step which defines an inverse-transform-sampling function and calls it twice, then computes the sum and product of samples.
* Send those to a histogram, and from there to a bar-plotter. This is just for visualization; you could just as well output CSV and send that off to your own plotting tool, etc.
The output is as follows:
::
POKI_RUN_COMMAND{{sh expo-sample.sh}}HERE
Sieve of Eratosthenes
----------------------------------------------------------------
The `Sieve of Eratosthenes <http://en.wikipedia.org/wiki/Sieve_of_Eratosthenes>`_ is a standard introductory programming topic. The idea is to find all primes up to some *N* by making a list of the numbers 1 to *N*, then striking out all multiples of 2 except 2 itself, all multiples of 3 except 3 itself, all multiples of 4 except 4 itself, and so on. Whatever survives that without getting marked is a prime. This is easy enough in Miller. Notice that here all the work is in ``begin`` and ``end`` statements; there is no file input (so we use ``mlr -n`` to keep Miller from waiting for input data).
::
POKI_RUN_COMMAND{{cat programs/sieve.mlr}}HERE
::
POKI_RUN_COMMAND{{mlr -n put -f programs/sieve.mlr}}HERE
Mandelbrot-set generator
----------------------------------------------------------------
The `Mandelbrot set <http://en.wikipedia.org/wiki/Mandelbrot_set>`_ is also easily expressed. This isn't an important case of data-processing in the vein for which Miller was designed, but it is an example of Miller as a general-purpose programming language -- a test case for the expressiveness of the language.
The (approximate) computation of points in the complex plane which are and aren't members is just a few lines of complex arithmetic (see the Wikipedia article); how to render them is another task. Using graphics libraries you can create PNG or JPEG files, but another fun way to do this is by printing various characters to the screen:
::
POKI_RUN_COMMAND{{cat programs/mand.mlr}}HERE
At standard resolution this makes a nice little ASCII plot:
::
POKI_RUN_COMMAND{{mlr -n put -f ./programs/mand.mlr}}HERE
But using a very small font size (as small as my Mac will let me go), and by choosing the coordinates to zoom in on a particular part of the complex plane, we can get a nice little picture:
::
#!/bin/bash
# Get the number of rows and columns from the terminal window dimensions
iheight=$(stty size | mlr --nidx --fs space cut -f 1)
iwidth=$(stty size | mlr --nidx --fs space cut -f 2)
echo "rcorn=-1.755350,icorn=+0.014230,side=0.000020,maxits=10000,iheight=$iheight,iwidth=$iwidth" \
| mlr put -f programs/mand.mlr
.. image:: pix/mand.png

321
docs6/cookbook3.rst Normal file
View file

@ -0,0 +1,321 @@
..
PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE.
Cookbook part 3: Stats with and without out-of-stream variables
================================================================
Overview
----------------------------------------------------------------
One of Miller's strengths is its compact notation: for example, given input of the form
::
$ head -n 5 ../data/medium
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
you can simply do
::
$ mlr --oxtab stats1 -a sum -f x ../data/medium
x_sum 4986.019682
or
::
$ mlr --opprint stats1 -a sum -f x -g b ../data/medium
b x_sum
pan 965.763670
wye 1023.548470
zee 979.742016
eks 1016.772857
hat 1000.192668
rather than the more tedious
::
$ mlr --oxtab put -q '
@x_sum += $x;
end {
emit @x_sum
}
' data/medium
x_sum 4986.019682
or
::
$ mlr --opprint put -q '
@x_sum[$b] += $x;
end {
emit @x_sum, "b"
}
' data/medium
b x_sum
pan 965.763670
wye 1023.548470
zee 979.742016
eks 1016.772857
hat 1000.192668
The former (``mlr stats1`` et al.) has the advantages of being easier to type, being less error-prone to type, and running faster.
Nonetheless, out-of-stream variables (which I whimsically call *oosvars*), begin/end blocks, and emit statements give you the ability to implement logic -- if you wish to do so -- which isn't present in other Miller verbs. (If you find yourself often using the same out-of-stream-variable logic over and over, please file a request at https://github.com/johnkerl/miller/issues to get it implemented directly in Go as a Miller verb of its own.)
The following examples compute some things using oosvars which are already computable using Miller verbs, by way of providing food for thought.
Mean without/with oosvars
----------------------------------------------------------------
::
$ mlr --opprint stats1 -a mean -f x data/medium
x_mean
0.498602
::
$ mlr --opprint put -q '
@x_sum += $x;
@x_count += 1;
end {
@x_mean = @x_sum / @x_count;
emit @x_mean
}
' data/medium
x_mean
0.498602
Keyed mean without/with oosvars
----------------------------------------------------------------
::
$ mlr --opprint stats1 -a mean -f x -g a,b data/medium
a b x_mean
pan pan 0.513314
eks pan 0.485076
wye wye 0.491501
eks wye 0.483895
wye pan 0.499612
zee pan 0.519830
eks zee 0.495463
zee wye 0.514267
hat wye 0.493813
pan wye 0.502362
zee eks 0.488393
hat zee 0.509999
hat eks 0.485879
wye hat 0.497730
pan eks 0.503672
eks eks 0.522799
hat hat 0.479931
hat pan 0.464336
zee zee 0.512756
pan hat 0.492141
pan zee 0.496604
zee hat 0.467726
wye zee 0.505907
eks hat 0.500679
wye eks 0.530604
::
$ mlr --opprint put -q '
@x_sum[$a][$b] += $x;
@x_count[$a][$b] += 1;
end{
for ((a, b), v in @x_sum) {
@x_mean[a][b] = @x_sum[a][b] / @x_count[a][b];
}
emit @x_mean, "a", "b"
}
' data/medium
a b x_mean
pan pan 0.513314
pan wye 0.502362
pan eks 0.503672
pan hat 0.492141
pan zee 0.496604
eks pan 0.485076
eks wye 0.483895
eks zee 0.495463
eks eks 0.522799
eks hat 0.500679
wye wye 0.491501
wye pan 0.499612
wye hat 0.497730
wye zee 0.505907
wye eks 0.530604
zee pan 0.519830
zee wye 0.514267
zee eks 0.488393
zee zee 0.512756
zee hat 0.467726
hat wye 0.493813
hat zee 0.509999
hat eks 0.485879
hat hat 0.479931
hat pan 0.464336
Variance and standard deviation without/with oosvars
----------------------------------------------------------------
::
$ mlr --oxtab stats1 -a count,sum,mean,var,stddev -f x data/medium
x_count 10000
x_sum 4986.019682
x_mean 0.498602
x_var 0.084270
x_stddev 0.290293
::
$ cat variance.mlr
@n += 1;
@sumx += $x;
@sumx2 += $x**2;
end {
@mean = @sumx / @n;
@var = (@sumx2 - @mean * (2 * @sumx - @n * @mean)) / (@n - 1);
@stddev = sqrt(@var);
emitf @n, @sumx, @sumx2, @mean, @var, @stddev
}
::
$ mlr --oxtab put -q -f variance.mlr data/medium
n 10000
sumx 4986.019682
sumx2 3328.652400
mean 0.498602
var 0.084270
stddev 0.290293
You can also do this keyed, of course, imitating the keyed-mean example above.
Min/max without/with oosvars
----------------------------------------------------------------
::
$ mlr --oxtab stats1 -a min,max -f x data/medium
x_min 0.000045
x_max 0.999953
::
$ mlr --oxtab put -q '@x_min = min(@x_min, $x); @x_max = max(@x_max, $x); end{emitf @x_min, @x_max}' data/medium
x_min 0.000045
x_max 0.999953
Keyed min/max without/with oosvars
----------------------------------------------------------------
::
$ mlr --opprint stats1 -a min,max -f x -g a data/medium
a x_min x_max
pan 0.000204 0.999403
eks 0.000692 0.998811
wye 0.000187 0.999823
zee 0.000549 0.999490
hat 0.000045 0.999953
::
$ mlr --opprint --from data/medium put -q '
@min[$a] = min(@min[$a], $x);
@max[$a] = max(@max[$a], $x);
end{
emit (@min, @max), "a";
}
'
a min max
pan 0.000204 0.999403
eks 0.000692 0.998811
wye 0.000187 0.999823
zee 0.000549 0.999490
hat 0.000045 0.999953
Delta without/with oosvars
----------------------------------------------------------------
::
$ mlr --opprint step -a delta -f x data/small
a b i x y x_delta
pan pan 1 0.3467901443380824 0.7268028627434533 0
eks pan 2 0.7586799647899636 0.5221511083334797 0.411890
wye wye 3 0.20460330576630303 0.33831852551664776 -0.554077
eks wye 4 0.38139939387114097 0.13418874328430463 0.176796
wye pan 5 0.5732889198020006 0.8636244699032729 0.191890
::
$ mlr --opprint put '$x_delta = is_present(@last) ? $x - @last : 0; @last = $x' data/small
a b i x y x_delta
pan pan 1 0.3467901443380824 0.7268028627434533 0
eks pan 2 0.7586799647899636 0.5221511083334797 0.411890
wye wye 3 0.20460330576630303 0.33831852551664776 -0.554077
eks wye 4 0.38139939387114097 0.13418874328430463 0.176796
wye pan 5 0.5732889198020006 0.8636244699032729 0.191890
Keyed delta without/with oosvars
----------------------------------------------------------------
::
$ mlr --opprint step -a delta -f x -g a data/small
a b i x y x_delta
pan pan 1 0.3467901443380824 0.7268028627434533 0
eks pan 2 0.7586799647899636 0.5221511083334797 0
wye wye 3 0.20460330576630303 0.33831852551664776 0
eks wye 4 0.38139939387114097 0.13418874328430463 -0.377281
wye pan 5 0.5732889198020006 0.8636244699032729 0.368686
::
$ mlr --opprint put '$x_delta = is_present(@last[$a]) ? $x - @last[$a] : 0; @last[$a]=$x' data/small
a b i x y x_delta
pan pan 1 0.3467901443380824 0.7268028627434533 0
eks pan 2 0.7586799647899636 0.5221511083334797 0
wye wye 3 0.20460330576630303 0.33831852551664776 0
eks wye 4 0.38139939387114097 0.13418874328430463 -0.377281
wye pan 5 0.5732889198020006 0.8636244699032729 0.368686
Exponentially weighted moving averages without/with oosvars
----------------------------------------------------------------
::
$ mlr --opprint step -a ewma -d 0.1 -f x data/small
a b i x y x_ewma_0.1
pan pan 1 0.3467901443380824 0.7268028627434533 0.346790
eks pan 2 0.7586799647899636 0.5221511083334797 0.387979
wye wye 3 0.20460330576630303 0.33831852551664776 0.369642
eks wye 4 0.38139939387114097 0.13418874328430463 0.370817
wye pan 5 0.5732889198020006 0.8636244699032729 0.391064
::
$ mlr --opprint put '
begin{ @a=0.1 };
$e = NR==1 ? $x : @a * $x + (1 - @a) * @e;
@e=$e
' data/small
a b i x y e
pan pan 1 0.3467901443380824 0.7268028627434533 0.346790
eks pan 2 0.7586799647899636 0.5221511083334797 0.387979
wye wye 3 0.20460330576630303 0.33831852551664776 0.369642
eks wye 4 0.38139939387114097 0.13418874328430463 0.370817
wye pan 5 0.5732889198020006 0.8636244699032729 0.391064

135
docs6/cookbook3.rst.in Normal file
View file

@ -0,0 +1,135 @@
Cookbook part 3: Stats with and without out-of-stream variables
================================================================
Overview
----------------------------------------------------------------
One of Miller's strengths is its compact notation: for example, given input of the form
::
POKI_RUN_COMMAND{{head -n 5 ../data/medium}}HERE
you can simply do
::
POKI_RUN_COMMAND{{mlr --oxtab stats1 -a sum -f x ../data/medium}}HERE
or
::
POKI_RUN_COMMAND{{mlr --opprint stats1 -a sum -f x -g b ../data/medium}}HERE
rather than the more tedious
::
POKI_INCLUDE_AND_RUN_ESCAPED(oosvar-example-sum.sh)HERE
or
::
POKI_INCLUDE_AND_RUN_ESCAPED(oosvar-example-sum-grouped.sh)HERE
The former (``mlr stats1`` et al.) has the advantages of being easier to type, being less error-prone to type, and running faster.
Nonetheless, out-of-stream variables (which I whimsically call *oosvars*), begin/end blocks, and emit statements give you the ability to implement logic -- if you wish to do so -- which isn't present in other Miller verbs. (If you find yourself often using the same out-of-stream-variable logic over and over, please file a request at https://github.com/johnkerl/miller/issues to get it implemented directly in Go as a Miller verb of its own.)
The following examples compute some things using oosvars which are already computable using Miller verbs, by way of providing food for thought.
Mean without/with oosvars
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr --opprint stats1 -a mean -f x data/medium}}HERE
::
POKI_INCLUDE_AND_RUN_ESCAPED(data/mean-with-oosvars.sh)HERE
Keyed mean without/with oosvars
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr --opprint stats1 -a mean -f x -g a,b data/medium}}HERE
::
POKI_INCLUDE_AND_RUN_ESCAPED(data/keyed-mean-with-oosvars.sh)HERE
Variance and standard deviation without/with oosvars
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr --oxtab stats1 -a count,sum,mean,var,stddev -f x data/medium}}HERE
::
POKI_RUN_COMMAND{{cat variance.mlr}}HERE
::
POKI_RUN_COMMAND{{mlr --oxtab put -q -f variance.mlr data/medium}}HERE
You can also do this keyed, of course, imitating the keyed-mean example above.
Min/max without/with oosvars
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr --oxtab stats1 -a min,max -f x data/medium}}HERE
::
POKI_RUN_COMMAND{{mlr --oxtab put -q '@x_min = min(@x_min, $x); @x_max = max(@x_max, $x); end{emitf @x_min, @x_max}' data/medium}}HERE
Keyed min/max without/with oosvars
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr --opprint stats1 -a min,max -f x -g a data/medium}}HERE
::
POKI_INCLUDE_AND_RUN_ESCAPED(data/keyed-min-max-with-oosvars.sh)HERE
Delta without/with oosvars
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr --opprint step -a delta -f x data/small}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint put '$x_delta = is_present(@last) ? $x - @last : 0; @last = $x' data/small}}HERE
Keyed delta without/with oosvars
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr --opprint step -a delta -f x -g a data/small}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint put '$x_delta = is_present(@last[$a]) ? $x - @last[$a] : 0; @last[$a]=$x' data/small}}HERE
Exponentially weighted moving averages without/with oosvars
----------------------------------------------------------------
::
POKI_INCLUDE_AND_RUN_ESCAPED(verb-example-ewma.sh)HERE
::
POKI_INCLUDE_AND_RUN_ESCAPED(oosvar-example-ewma.sh)HERE

BIN
docs6/coverart/cover-cat.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 102 KiB

BIN
docs6/coverart/cover-combined.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 507 KiB

BIN
docs6/coverart/cover-filter-to-json.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 105 KiB

BIN
docs6/coverart/cover-sort-to-pprint.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 111 KiB

BIN
docs6/coverart/coverart.key Executable file
View file

Binary file not shown.

94
docs6/customization.rst Normal file
View file

@ -0,0 +1,94 @@
..
PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE.
Customization: .mlrrc
================================================================
How to use .mlrrc
----------------------------------------------------------------
Suppose you always use CSV files. Then instead of always having to type ``--csv`` as in
::
mlr --csv cut -x -f extra mydata.csv
::
mlr --csv sort -n id mydata.csv
and so on, you can instead put the following into your ``$HOME/.mlrrc``:
::
--csv
Then you can just type things like
::
mlr cut -x -f extra mydata.csv
::
mlr sort -n id mydata.csv
and the ``--csv`` part will automatically be understood. (If you do want to process, say, a JSON file then ``mlr --json ...`` at the command line will override the default from your ``.mlrrc``.)
What you can put in your .mlrrc
----------------------------------------------------------------
* You can include any command-line flags, except the "terminal" ones such as ``--help``.
* The ``--prepipe``, ``--load``, and ``--mload`` flags aren't allowed in ``.mlrrc`` as they control code execution, and could result in your scripts running things you don't expect if you receive data from someone with a ``.mlrrc`` in it.
* The formatting rule is you need to put one flag beginning with ``--`` per line: for example, ``--csv`` on one line and ``--nr-progress-mod 1000`` on a separate line.
* Since every line starts with a ``--`` option, you can leave off the initial ``--`` if you want. For example, ``ojson`` is the same as ``--ojson``, and ``nr-progress-mod 1000`` is the same as ``--nr-progress-mod 1000``.
* Comments are from a ``#`` to the end of the line.
* Empty lines are ignored -- including lines which are empty after comments are removed.
Here is an example ``.mlrrc file``:
::
# These are my preferred default settings for Miller
# Input and output formats are CSV by default (unless otherwise specified
# on the mlr command line):
csv
# If a data line has fewer fields than the header line, instead of erroring
# (which is the default), just insert empty values for the missing ones:
allow-ragged-csv-input
# These are no-ops for CSV, but when I do use JSON output, I want these
# pretty-printing options to be used:
jvstack
jlistwrap
# Use "@", rather than "#", for comments within data files:
skip-comments-with @
Where to put your .mlrrc
----------------------------------------------------------------
* If the environment variable ``MLRRC`` is set:
* If its value is ``__none__`` then no ``.mlrrc`` files are processed. (This is nice for things like regression testing.)
* Otherwise, its value (as a filename) is loaded and processed. If there are syntax errors, they abort ``mlr`` with a usage message (as if you had mistyped something on the command line). If the file can't be loaded at all, though, it is silently skipped.
* Any ``.mlrrc`` in your home directory or current directory is ignored whenever ``MLRRC`` is set in the environment.
* Example line in your shell's rc file: ``export MLRRC=/path/to/my/mlrrc``
* Otherwise:
* If ``$HOME/.mlrrc`` exists, it's processed as above.
* If ``./.mlrrc`` exists, it's then also processed as above.
* The idea is you can have all your settings in your ``$HOME/.mlrrc``, then override maybe one or two for your current directory if you like.

75
docs6/customization.rst.in Normal file
View file

@ -0,0 +1,75 @@
Customization: .mlrrc
================================================================
How to use .mlrrc
----------------------------------------------------------------
Suppose you always use CSV files. Then instead of always having to type ``--csv`` as in
::
mlr --csv cut -x -f extra mydata.csv
::
mlr --csv sort -n id mydata.csv
and so on, you can instead put the following into your ``$HOME/.mlrrc``:
::
--csv
Then you can just type things like
::
mlr cut -x -f extra mydata.csv
::
mlr sort -n id mydata.csv
and the ``--csv`` part will automatically be understood. (If you do want to process, say, a JSON file then ``mlr --json ...`` at the command line will override the default from your ``.mlrrc``.)
What you can put in your .mlrrc
----------------------------------------------------------------
* You can include any command-line flags, except the "terminal" ones such as ``--help``.
* The ``--prepipe``, ``--load``, and ``--mload`` flags aren't allowed in ``.mlrrc`` as they control code execution, and could result in your scripts running things you don't expect if you receive data from someone with a ``.mlrrc`` in it.
* The formatting rule is you need to put one flag beginning with ``--`` per line: for example, ``--csv`` on one line and ``--nr-progress-mod 1000`` on a separate line.
* Since every line starts with a ``--`` option, you can leave off the initial ``--`` if you want. For example, ``ojson`` is the same as ``--ojson``, and ``nr-progress-mod 1000`` is the same as ``--nr-progress-mod 1000``.
* Comments are from a ``#`` to the end of the line.
* Empty lines are ignored -- including lines which are empty after comments are removed.
Here is an example ``.mlrrc file``:
::
POKI_INCLUDE_ESCAPED(sample_mlrrc)HERE
Where to put your .mlrrc
----------------------------------------------------------------
* If the environment variable ``MLRRC`` is set:
* If its value is ``__none__`` then no ``.mlrrc`` files are processed. (This is nice for things like regression testing.)
* Otherwise, its value (as a filename) is loaded and processed. If there are syntax errors, they abort ``mlr`` with a usage message (as if you had mistyped something on the command line). If the file can't be loaded at all, though, it is silently skipped.
* Any ``.mlrrc`` in your home directory or current directory is ignored whenever ``MLRRC`` is set in the environment.
* Example line in your shell's rc file: ``export MLRRC=/path/to/my/mlrrc``
* Otherwise:
* If ``$HOME/.mlrrc`` exists, it's processed as above.
* If ``./.mlrrc`` exists, it's then also processed as above.
* The idea is you can have all your settings in your ``$HOME/.mlrrc``, then override maybe one or two for your current directory if you like.

194
docs6/data-examples.rst Normal file
View file

@ -0,0 +1,194 @@
..
PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE.
Data-diving examples
================================================================
flins data
----------------------------------------------------------------
The `flins.csv <data/flins.csv>`_ file is some sample data obtained from https://support.spatialkey.com/spatialkey-sample-csv-data.
Vertical-tabular format is good for a quick look at CSV data layout -- seeing what columns you have to work with:
::
$ head -n 2 data/flins.csv | mlr --icsv --oxtab cat
county Seminole
tiv_2011 22890.55
tiv_2012 20848.71
line Residential
A few simple queries:
::
$ mlr --from data/flins.csv --icsv --opprint count-distinct -f county | head
county count
Seminole 1
Miami Dade 2
Palm Beach 1
Highlands 2
Duval 1
St. Johns 1
$ mlr --from data/flins.csv --icsv --opprint count-distinct -f construction,line
Categorization of total insured value:
::
$ mlr --from data/flins.csv --icsv --opprint stats1 -a min,mean,max -f tiv_2012
tiv_2012_min tiv_2012_mean tiv_2012_max
19757.910000 1061531.463750 2785551.630000
$ mlr --from data/flins.csv --icsv --opprint stats1 -a min,mean,max -f tiv_2012 -g construction,line
$ mlr --from data/flins.csv --icsv --oxtab stats1 -a p0,p10,p50,p90,p95,p99,p100 -f hu_site_deductible
hu_site_deductible_p0
hu_site_deductible_p10
hu_site_deductible_p50
hu_site_deductible_p90
hu_site_deductible_p95
hu_site_deductible_p99
hu_site_deductible_p100
$ mlr --from data/flins.csv --icsv --opprint stats1 -a p95,p99,p100 -f hu_site_deductible -g county then sort -f county | head
county hu_site_deductible_p95 hu_site_deductible_p99 hu_site_deductible_p100
Duval - - -
Highlands - - -
Miami Dade - - -
Palm Beach - - -
Seminole - - -
St. Johns - - -
$ mlr --from data/flins.csv --icsv --oxtab stats2 -a corr,linreg-ols,r2 -f tiv_2011,tiv_2012
tiv_2011_tiv_2012_corr 0.935363
tiv_2011_tiv_2012_ols_m 1.089091
tiv_2011_tiv_2012_ols_b 103095.523356
tiv_2011_tiv_2012_ols_n 8
tiv_2011_tiv_2012_r2 0.874904
$ mlr --from data/flins.csv --icsv --opprint stats2 -a corr,linreg-ols,r2 -f tiv_2011,tiv_2012 -g county
county tiv_2011_tiv_2012_corr tiv_2011_tiv_2012_ols_m tiv_2011_tiv_2012_ols_b tiv_2011_tiv_2012_ols_n tiv_2011_tiv_2012_r2
Seminole - - - 1 -
Miami Dade 1.000000 0.930643 -2311.154328 2 1.000000
Palm Beach - - - 1 -
Highlands 1.000000 1.055693 -4529.793939 2 1.000000
Duval - - - 1 -
St. Johns - - - 1 -
Color/shape data
----------------------------------------------------------------
The `colored-shapes.dkvp <https://github.com/johnkerl/miller/blob/master/docs/data/colored-shapes.dkvp>`_ file is some sample data produced by the `mkdat2 <https://github.com/johnkerl/miller/blob/master/doc/datagen/mkdat2>`_ script. The idea is:
* Produce some data with known distributions and correlations, and verify that Miller recovers those properties empirically.
* Each record is labeled with one of a few colors and one of a few shapes.
* The ``flag`` field is 0 or 1, with probability dependent on color
* The ``u`` field is plain uniform on the unit interval.
* The ``v`` field is the same, except tightly correlated with ``u`` for red circles.
* The ``w`` field is autocorrelated for each color/shape pair.
* The ``x`` field is boring Gaussian with mean 5 and standard deviation about 1.2, with no dependence on color or shape.
Peek at the data:
::
$ wc -l data/colored-shapes.dkvp
10078 data/colored-shapes.dkvp
$ head -n 6 data/colored-shapes.dkvp | mlr --opprint cat
color shape flag i u v w x
yellow triangle 1 11 0.6321695890307647 0.9887207810889004 0.4364983936735774 5.7981881667050565
red square 1 15 0.21966833570651523 0.001257332190235938 0.7927778364718627 2.944117399716207
red circle 1 16 0.20901671281497636 0.29005231936593445 0.13810280912907674 5.065034003400998
red square 0 48 0.9562743938458542 0.7467203085342884 0.7755423050923582 7.117831369597269
purple triangle 0 51 0.4355354501763202 0.8591292672156728 0.8122903963006748 5.753094629505863
red square 0 64 0.2015510269821953 0.9531098083420033 0.7719912015786777 5.612050466474166
Look at uncategorized stats (using `creach <https://github.com/johnkerl/scripts/blob/master/fundam/creach>`_ for spacing).
Here it looks reasonable that ``u`` is unit-uniform; something's up with ``v`` but we can't yet see what:
::
$ mlr --oxtab stats1 -a min,mean,max -f flag,u,v data/colored-shapes.dkvp | creach 3
flag_min 0
flag_mean 0.398889
flag_max 1
u_min 0.000044
u_mean 0.498326
u_max 0.999969
v_min -0.092709
v_mean 0.497787
v_max 1.072500
The histogram shows the different distribution of 0/1 flags:
::
$ mlr --opprint histogram -f flag,u,v --lo -0.1 --hi 1.1 --nbins 12 data/colored-shapes.dkvp
bin_lo bin_hi flag_count u_count v_count
-0.100000 0.000000 6058 0 36
0.000000 0.100000 0 1062 988
0.100000 0.200000 0 985 1003
0.200000 0.300000 0 1024 1014
0.300000 0.400000 0 1002 991
0.400000 0.500000 0 989 1041
0.500000 0.600000 0 1001 1016
0.600000 0.700000 0 972 962
0.700000 0.800000 0 1035 1070
0.800000 0.900000 0 995 993
0.900000 1.000000 4020 1013 939
1.000000 1.100000 0 0 25
Look at univariate stats by color and shape. In particular, color-dependent flag probabilities pop out, aligning with their original Bernoulli probablities from the data-generator script:
::
$ mlr --opprint stats1 -a min,mean,max -f flag,u,v -g color then sort -f color data/colored-shapes.dkvp
color flag_min flag_mean flag_max u_min u_mean u_max v_min v_mean v_max
blue 0 0.584354 1 0.000044 0.517717 0.999969 0.001489 0.491056 0.999576
green 0 0.209197 1 0.000488 0.504861 0.999936 0.000501 0.499085 0.999676
orange 0 0.521452 1 0.001235 0.490532 0.998885 0.002449 0.487764 0.998475
purple 0 0.090193 1 0.000266 0.494005 0.999647 0.000364 0.497051 0.999975
red 0 0.303167 1 0.000671 0.492560 0.999882 -0.092709 0.496535 1.072500
yellow 0 0.892427 1 0.001300 0.497129 0.999923 0.000711 0.510627 0.999919
$ mlr --opprint stats1 -a min,mean,max -f flag,u,v -g shape then sort -f shape data/colored-shapes.dkvp
shape flag_min flag_mean flag_max u_min u_mean u_max v_min v_mean v_max
circle 0 0.399846 1 0.000044 0.498555 0.999923 -0.092709 0.495524 1.072500
square 0 0.396112 1 0.000188 0.499385 0.999969 0.000089 0.496538 0.999975
triangle 0 0.401542 1 0.000881 0.496859 0.999661 0.000717 0.501050 0.999995
Look at bivariate stats by color and shape. In particular, ``u,v`` pairwise correlation for red circles pops out:
::
$ mlr --opprint --right stats2 -a corr -f u,v,w,x data/colored-shapes.dkvp
u_v_corr w_x_corr
0.133418 -0.011320
$ mlr --opprint --right stats2 -a corr -f u,v,w,x -g color,shape then sort -nr u_v_corr data/colored-shapes.dkvp
color shape u_v_corr w_x_corr
red circle 0.980798 -0.018565
orange square 0.176858 -0.071044
green circle 0.057644 0.011795
red square 0.055745 -0.000680
yellow triangle 0.044573 0.024605
yellow square 0.043792 -0.044623
purple circle 0.035874 0.134112
blue square 0.032412 -0.053508
blue triangle 0.015356 -0.000608
orange circle 0.010519 -0.162795
red triangle 0.008098 0.012486
purple triangle 0.005155 -0.045058
purple square -0.025680 0.057694
green square -0.025776 -0.003265
orange triangle -0.030457 -0.131870
yellow circle -0.064773 0.073695
blue circle -0.102348 -0.030529
green triangle -0.109018 -0.048488

88
docs6/data-examples.rst.in Normal file
View file

@ -0,0 +1,88 @@
Data-diving examples
================================================================
flins data
----------------------------------------------------------------
The `flins.csv <data/flins.csv>`_ file is some sample data obtained from https://support.spatialkey.com/spatialkey-sample-csv-data.
Vertical-tabular format is good for a quick look at CSV data layout -- seeing what columns you have to work with:
::
POKI_RUN_COMMAND{{head -n 2 data/flins.csv | mlr --icsv --oxtab cat}}HERE
A few simple queries:
::
POKI_RUN_COMMAND{{mlr --from data/flins.csv --icsv --opprint count-distinct -f county | head}}HERE
POKI_RUN_COMMAND{{mlr --from data/flins.csv --icsv --opprint count-distinct -f construction,line}}HERE
Categorization of total insured value:
::
POKI_RUN_COMMAND{{mlr --from data/flins.csv --icsv --opprint stats1 -a min,mean,max -f tiv_2012}}HERE
POKI_RUN_COMMAND{{mlr --from data/flins.csv --icsv --opprint stats1 -a min,mean,max -f tiv_2012 -g construction,line}}HERE
POKI_RUN_COMMAND{{mlr --from data/flins.csv --icsv --oxtab stats1 -a p0,p10,p50,p90,p95,p99,p100 -f hu_site_deductible}}HERE
POKI_RUN_COMMAND{{mlr --from data/flins.csv --icsv --opprint stats1 -a p95,p99,p100 -f hu_site_deductible -g county then sort -f county | head}}HERE
POKI_RUN_COMMAND{{mlr --from data/flins.csv --icsv --oxtab stats2 -a corr,linreg-ols,r2 -f tiv_2011,tiv_2012}}HERE
POKI_RUN_COMMAND{{mlr --from data/flins.csv --icsv --opprint stats2 -a corr,linreg-ols,r2 -f tiv_2011,tiv_2012 -g county}}HERE
Color/shape data
----------------------------------------------------------------
The `colored-shapes.dkvp <https://github.com/johnkerl/miller/blob/master/docs/data/colored-shapes.dkvp>`_ file is some sample data produced by the `mkdat2 <https://github.com/johnkerl/miller/blob/master/doc/datagen/mkdat2>`_ script. The idea is:
* Produce some data with known distributions and correlations, and verify that Miller recovers those properties empirically.
* Each record is labeled with one of a few colors and one of a few shapes.
* The ``flag`` field is 0 or 1, with probability dependent on color
* The ``u`` field is plain uniform on the unit interval.
* The ``v`` field is the same, except tightly correlated with ``u`` for red circles.
* The ``w`` field is autocorrelated for each color/shape pair.
* The ``x`` field is boring Gaussian with mean 5 and standard deviation about 1.2, with no dependence on color or shape.
Peek at the data:
::
POKI_RUN_COMMAND{{wc -l data/colored-shapes.dkvp}}HERE
POKI_RUN_COMMAND{{head -n 6 data/colored-shapes.dkvp | mlr --opprint cat}}HERE
Look at uncategorized stats (using `creach <https://github.com/johnkerl/scripts/blob/master/fundam/creach>`_ for spacing).
Here it looks reasonable that ``u`` is unit-uniform; something's up with ``v`` but we can't yet see what:
::
POKI_RUN_COMMAND{{mlr --oxtab stats1 -a min,mean,max -f flag,u,v data/colored-shapes.dkvp | creach 3}}HERE
The histogram shows the different distribution of 0/1 flags:
::
POKI_RUN_COMMAND{{mlr --opprint histogram -f flag,u,v --lo -0.1 --hi 1.1 --nbins 12 data/colored-shapes.dkvp}}HERE
Look at univariate stats by color and shape. In particular, color-dependent flag probabilities pop out, aligning with their original Bernoulli probablities from the data-generator script:
::
POKI_RUN_COMMAND{{mlr --opprint stats1 -a min,mean,max -f flag,u,v -g color then sort -f color data/colored-shapes.dkvp}}HERE
POKI_RUN_COMMAND{{mlr --opprint stats1 -a min,mean,max -f flag,u,v -g shape then sort -f shape data/colored-shapes.dkvp}}HERE
Look at bivariate stats by color and shape. In particular, ``u,v`` pairwise correlation for red circles pops out:
::
POKI_RUN_COMMAND{{mlr --opprint --right stats2 -a corr -f u,v,w,x data/colored-shapes.dkvp}}HERE
POKI_RUN_COMMAND{{mlr --opprint --right stats2 -a corr -f u,v,w,x -g color,shape then sort -nr u_v_corr data/colored-shapes.dkvp}}HERE

319
docs6/data-sharing.rst Normal file
View file

@ -0,0 +1,319 @@
..
PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE.
Mixing with other languages
================================================================
As discussed in the section on :doc:`file-formats`, Miller supports several different file formats. Different tools are good at different things, so it's important to be able to move data into and out of other languages. **CSV** and **JSON** are well-known, of course; here are some examples using **DKVP** format, with **Ruby** and **Python**. Last, we show how to use arbitrary **shell commands** to extend functionality beyond Miller's domain-specific language.
DKVP I/O in Python
----------------------------------------------------------------
Here are the I/O routines:
::
#!/usr/bin/env python
# ================================================================
# Example of DKVP I/O using Python.
#
# Key point: Use Miller for what it's good at; pass data into/out of tools in
# other languages to do what they're good at.
#
# bash$ python -i dkvp_io.py
#
# # READ
# >>> map = dkvpline2map('x=1,y=2', '=', ',')
# >>> map
# OrderedDict([('x', '1'), ('y', '2')])
#
# # MODIFY
# >>> map['z'] = map['x'] + map['y']
# >>> map
# OrderedDict([('x', '1'), ('y', '2'), ('z', 3)])
#
# # WRITE
# >>> line = map2dkvpline(map, '=', ',')
# >>> line
# 'x=1,y=2,z=3'
#
# ================================================================
import re
import collections
# ----------------------------------------------------------------
# ips and ifs (input pair separator and input field separator) are nominally '=' and ','.
def dkvpline2map(line, ips, ifs):
pairs = re.split(ifs, line)
map = collections.OrderedDict()
for pair in pairs:
key, value = re.split(ips, pair, 1)
# Type inference:
try:
value = int(value)
except:
try:
value = float(value)
except:
pass
map[key] = value
return map
# ----------------------------------------------------------------
# ops and ofs (output pair separator and output field separator) are nominally '=' and ','.
def map2dkvpline(map , ops, ofs):
line = ''
pairs = []
for key in map:
pairs.append(str(key) + ops + str(map[key]))
return str.join(ofs, pairs)
And here is an example using them:
::
$ cat polyglot-dkvp-io/example.py
#!/usr/bin/env python
import sys
import re
import copy
import dkvp_io
while True:
# Read the original record:
line = sys.stdin.readline().strip()
if line == '':
break
map = dkvp_io.dkvpline2map(line, '=', ',')
# Drop a field:
map.pop('x')
# Compute some new fields:
map['ab'] = map['a'] + map['b']
map['iy'] = map['i'] + map['y']
# Add new fields which show type of each already-existing field:
omap = copy.copy(map) # since otherwise the for-loop will modify what it loops over
keys = omap.keys()
for key in keys:
# Convert "<type 'int'>" to just "int", etc.:
type_string = str(map[key].__class__)
type_string = re.sub("<type '", "", type_string) # python2
type_string = re.sub("<class '", "", type_string) # python3
type_string = re.sub("'>", "", type_string)
map['t'+key] = type_string
# Write the modified record:
print(dkvp_io.map2dkvpline(map, '=', ','))
Run as-is:
::
$ python polyglot-dkvp-io/example.py < data/small
a=pan,b=pan,i=1,y=0.7268028627434533,ab=panpan,iy=1.7268028627434533,ta=str,tb=str,ti=int,ty=float,tab=str,tiy=float
a=eks,b=pan,i=2,y=0.5221511083334797,ab=ekspan,iy=2.5221511083334796,ta=str,tb=str,ti=int,ty=float,tab=str,tiy=float
a=wye,b=wye,i=3,y=0.33831852551664776,ab=wyewye,iy=3.3383185255166477,ta=str,tb=str,ti=int,ty=float,tab=str,tiy=float
a=eks,b=wye,i=4,y=0.13418874328430463,ab=ekswye,iy=4.134188743284304,ta=str,tb=str,ti=int,ty=float,tab=str,tiy=float
a=wye,b=pan,i=5,y=0.8636244699032729,ab=wyepan,iy=5.863624469903273,ta=str,tb=str,ti=int,ty=float,tab=str,tiy=float
Run as-is, then pipe to Miller for pretty-printing:
::
$ python polyglot-dkvp-io/example.py < data/small | mlr --opprint cat
a b i y ab iy ta tb ti ty tab tiy
pan pan 1 0.7268028627434533 panpan 1.7268028627434533 str str int float str float
eks pan 2 0.5221511083334797 ekspan 2.5221511083334796 str str int float str float
wye wye 3 0.33831852551664776 wyewye 3.3383185255166477 str str int float str float
eks wye 4 0.13418874328430463 ekswye 4.134188743284304 str str int float str float
wye pan 5 0.8636244699032729 wyepan 5.863624469903273 str str int float str float
DKVP I/O in Ruby
----------------------------------------------------------------
Here are the I/O routines:
::
#!/usr/bin/env ruby
# ================================================================
# Example of DKVP I/O using Ruby.
#
# Key point: Use Miller for what it's good at; pass data into/out of tools in
# other languages to do what they're good at.
#
# bash$ irb -I. -r dkvp_io.rb
#
# # READ
# irb(main):001:0> map = dkvpline2map('x=1,y=2', '=', ',')
# => {"x"=>"1", "y"=>"2"}
#
# # MODIFY
# irb(main):001:0> map['z'] = map['x'] + map['y']
# => 3
#
# # WRITE
# irb(main):002:0> line = map2dkvpline(map, '=', ',')
# => "x=1,y=2,z=3"
#
# ================================================================
# ----------------------------------------------------------------
# ips and ifs (input pair separator and input field separator) are nominally '=' and ','.
def dkvpline2map(line, ips, ifs)
map = {}
line.split(ifs).each do |pair|
(k, v) = pair.split(ips, 2)
# Type inference:
begin
v = Integer(v)
rescue ArgumentError
begin
v = Float(v)
rescue ArgumentError
# Leave as string
end
end
map[k] = v
end
map
end
# ----------------------------------------------------------------
# ops and ofs (output pair separator and output field separator) are nominally '=' and ','.
def map2dkvpline(map, ops, ofs)
map.collect{|k,v| k.to_s + ops + v.to_s}.join(ofs)
end
And here is an example using them:
::
$ cat polyglot-dkvp-io/example.rb
#!/usr/bin/env ruby
require 'dkvp_io'
ARGF.each do |line|
# Read the original record:
map = dkvpline2map(line.chomp, '=', ',')
# Drop a field:
map.delete('x')
# Compute some new fields:
map['ab'] = map['a'] + map['b']
map['iy'] = map['i'] + map['y']
# Add new fields which show type of each already-existing field:
keys = map.keys
keys.each do |key|
map['t'+key] = map[key].class
end
# Write the modified record:
puts map2dkvpline(map, '=', ',')
end
Run as-is:
::
$ ruby -I./polyglot-dkvp-io polyglot-dkvp-io/example.rb data/small
a=pan,b=pan,i=1,y=0.7268028627434533,ab=panpan,iy=1.7268028627434533,ta=String,tb=String,ti=Integer,ty=Float,tab=String,tiy=Float
a=eks,b=pan,i=2,y=0.5221511083334797,ab=ekspan,iy=2.5221511083334796,ta=String,tb=String,ti=Integer,ty=Float,tab=String,tiy=Float
a=wye,b=wye,i=3,y=0.33831852551664776,ab=wyewye,iy=3.3383185255166477,ta=String,tb=String,ti=Integer,ty=Float,tab=String,tiy=Float
a=eks,b=wye,i=4,y=0.13418874328430463,ab=ekswye,iy=4.134188743284304,ta=String,tb=String,ti=Integer,ty=Float,tab=String,tiy=Float
a=wye,b=pan,i=5,y=0.8636244699032729,ab=wyepan,iy=5.863624469903273,ta=String,tb=String,ti=Integer,ty=Float,tab=String,tiy=Float
Run as-is, then pipe to Miller for pretty-printing:
::
$ ruby -I./polyglot-dkvp-io polyglot-dkvp-io/example.rb data/small | mlr --opprint cat
a b i y ab iy ta tb ti ty tab tiy
pan pan 1 0.7268028627434533 panpan 1.7268028627434533 String String Integer Float String Float
eks pan 2 0.5221511083334797 ekspan 2.5221511083334796 String String Integer Float String Float
wye wye 3 0.33831852551664776 wyewye 3.3383185255166477 String String Integer Float String Float
eks wye 4 0.13418874328430463 ekswye 4.134188743284304 String String Integer Float String Float
wye pan 5 0.8636244699032729 wyepan 5.863624469903273 String String Integer Float String Float
SQL-output examples
----------------------------------------------------------------
Please see :ref:`sql-output-examples`.
SQL-input examples
----------------------------------------------------------------
Please see :ref:`sql-input-examples`.
Running shell commands
----------------------------------------------------------------
The :ref:`reference-dsl-system` DSL function allows you to run a specific shell command and put its output -- minus the final newline -- into a record field. The command itself is any string, either a literal string, or a concatenation of strings, perhaps including other field values or what have you.
::
$ mlr --opprint put '$o = system("echo hello world")' data/small
a b i x y o
pan pan 1 0.3467901443380824 0.7268028627434533 hello world
eks pan 2 0.7586799647899636 0.5221511083334797 hello world
wye wye 3 0.20460330576630303 0.33831852551664776 hello world
eks wye 4 0.38139939387114097 0.13418874328430463 hello world
wye pan 5 0.5732889198020006 0.8636244699032729 hello world
::
$ mlr --opprint put '$o = system("echo {" . NR . "}")' data/small
a b i x y o
pan pan 1 0.3467901443380824 0.7268028627434533 {1}
eks pan 2 0.7586799647899636 0.5221511083334797 {2}
wye wye 3 0.20460330576630303 0.33831852551664776 {3}
eks wye 4 0.38139939387114097 0.13418874328430463 {4}
wye pan 5 0.5732889198020006 0.8636244699032729 {5}
::
$ mlr --opprint put '$o = system("echo -n ".$a."| sha1sum")' data/small
a b i x y o
pan pan 1 0.3467901443380824 0.7268028627434533 f29c748220331c273ef16d5115f6ecd799947f13 -
eks pan 2 0.7586799647899636 0.5221511083334797 456d988ecb3bf1b75f057fc6e9fe70db464e9388 -
wye wye 3 0.20460330576630303 0.33831852551664776 eab0de043d67f441c7fd1e335f0ca38708e6ebf7 -
eks wye 4 0.38139939387114097 0.13418874328430463 456d988ecb3bf1b75f057fc6e9fe70db464e9388 -
wye pan 5 0.5732889198020006 0.8636244699032729 eab0de043d67f441c7fd1e335f0ca38708e6ebf7 -
Note that running a subprocess on every record takes a non-trivial amount of time. Comparing asking the system ``date`` command for the current time in nanoseconds versus computing it in process:
..
hard-coded, not live-code, since %N doesn't exist on all platforms
::
$ mlr --opprint put '$t=system("date +%s.%N")' then step -a delta -f t data/small
a b i x y t t_delta
pan pan 1 0.3467901443380824 0.7268028627434533 1568774318.513903817 0
eks pan 2 0.7586799647899636 0.5221511083334797 1568774318.514722876 0.000819
wye wye 3 0.20460330576630303 0.33831852551664776 1568774318.515618046 0.000895
eks wye 4 0.38139939387114097 0.13418874328430463 1568774318.516547441 0.000929
wye pan 5 0.5732889198020006 0.8636244699032729 1568774318.517518828 0.000971
::
$ mlr --opprint put '$t=systime()' then step -a delta -f t data/small
a b i x y t t_delta
pan pan 1 0.3467901443380824 0.7268028627434533 1568774318.518699 0
eks pan 2 0.7586799647899636 0.5221511083334797 1568774318.518717 0.000018
wye wye 3 0.20460330576630303 0.33831852551664776 1568774318.518723 0.000006
eks wye 4 0.38139939387114097 0.13418874328430463 1568774318.518727 0.000004
wye pan 5 0.5732889198020006 0.8636244699032729 1568774318.518730 0.000003

110
docs6/data-sharing.rst.in Normal file
View file

@ -0,0 +1,110 @@
Mixing with other languages
================================================================
As discussed in the section on :doc:`file-formats`, Miller supports several different file formats. Different tools are good at different things, so it's important to be able to move data into and out of other languages. **CSV** and **JSON** are well-known, of course; here are some examples using **DKVP** format, with **Ruby** and **Python**. Last, we show how to use arbitrary **shell commands** to extend functionality beyond Miller's domain-specific language.
DKVP I/O in Python
----------------------------------------------------------------
Here are the I/O routines:
::
POKI_INCLUDE_ESCAPED(polyglot-dkvp-io/dkvp_io.py)HERE
And here is an example using them:
::
POKI_RUN_COMMAND{{cat polyglot-dkvp-io/example.py}}HERE
Run as-is:
::
POKI_RUN_COMMAND{{python polyglot-dkvp-io/example.py < data/small}}HERE
Run as-is, then pipe to Miller for pretty-printing:
::
POKI_RUN_COMMAND{{python polyglot-dkvp-io/example.py < data/small | mlr --opprint cat}}HERE
DKVP I/O in Ruby
----------------------------------------------------------------
Here are the I/O routines:
::
POKI_INCLUDE_ESCAPED(polyglot-dkvp-io/dkvp_io.rb)HERE
And here is an example using them:
::
POKI_RUN_COMMAND{{cat polyglot-dkvp-io/example.rb}}HERE
Run as-is:
::
POKI_RUN_COMMAND{{ruby -I./polyglot-dkvp-io polyglot-dkvp-io/example.rb data/small}}HERE
Run as-is, then pipe to Miller for pretty-printing:
::
POKI_RUN_COMMAND{{ruby -I./polyglot-dkvp-io polyglot-dkvp-io/example.rb data/small | mlr --opprint cat}}HERE
SQL-output examples
----------------------------------------------------------------
Please see :ref:`sql-output-examples`.
SQL-input examples
----------------------------------------------------------------
Please see :ref:`sql-input-examples`.
Running shell commands
----------------------------------------------------------------
The :ref:`reference-dsl-system` DSL function allows you to run a specific shell command and put its output -- minus the final newline -- into a record field. The command itself is any string, either a literal string, or a concatenation of strings, perhaps including other field values or what have you.
::
POKI_RUN_COMMAND{{mlr --opprint put '$o = system("echo hello world")' data/small}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint put '$o = system("echo {" . NR . "}")' data/small}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint put '$o = system("echo -n ".$a."| sha1sum")' data/small}}HERE
Note that running a subprocess on every record takes a non-trivial amount of time. Comparing asking the system ``date`` command for the current time in nanoseconds versus computing it in process:
..
hard-coded, not live-code, since %N doesn't exist on all platforms
::
$ mlr --opprint put '$t=system("date +%s.%N")' then step -a delta -f t data/small
a b i x y t t_delta
pan pan 1 0.3467901443380824 0.7268028627434533 1568774318.513903817 0
eks pan 2 0.7586799647899636 0.5221511083334797 1568774318.514722876 0.000819
wye wye 3 0.20460330576630303 0.33831852551664776 1568774318.515618046 0.000895
eks wye 4 0.38139939387114097 0.13418874328430463 1568774318.516547441 0.000929
wye pan 5 0.5732889198020006 0.8636244699032729 1568774318.517518828 0.000971
::
$ mlr --opprint put '$t=systime()' then step -a delta -f t data/small
a b i x y t t_delta
pan pan 1 0.3467901443380824 0.7268028627434533 1568774318.518699 0
eks pan 2 0.7586799647899636 0.5221511083334797 1568774318.518717 0.000018
wye wye 3 0.20460330576630303 0.33831852551664776 1568774318.518723 0.000006
eks wye 4 0.38139939387114097 0.13418874328430463 1568774318.518727 0.000004
wye pan 5 0.5732889198020006 0.8636244699032729 1568774318.518730 0.000003

3
docs6/data/a.csv Normal file
View file

@ -0,0 +1,3 @@
a,b,c
1,2,3
4,5,6
1 a b c
2 1 2 3
3 4 5 6

2
docs6/data/a.dkvp Normal file
View file

@ -0,0 +1,2 @@
a=1,b=2,c=3
a=4,b=5,c=6

2
docs6/data/b.csv Normal file
View file

@ -0,0 +1,2 @@
a,b,c
7,8,9
1 a b c
2 7 8 9

1
docs6/data/b.dkvp Normal file
View file

@ -0,0 +1 @@
a=7,b=8,c=9

5
docs6/data/begin-end-example-1.sh Normal file
View file

@ -0,0 +1,5 @@
mlr put '
begin { @sum = 0 };
@x_sum += $x;
end { emit @x_sum }
' ../data/small

4
docs6/data/begin-end-example-2.sh Normal file
View file

@ -0,0 +1,4 @@
mlr put '
@x_sum += $x;
end { emit @x_sum }
' ../data/small

4
docs6/data/begin-end-example-3.sh Normal file
View file

@ -0,0 +1,4 @@
mlr put -q '
@x_sum += $x;
end { emit @x_sum }
' ../data/small

8
docs6/data/begin-end-example-4.sh Normal file
View file

@ -0,0 +1,8 @@
mlr put -q '
@x_count += 1;
@x_sum += $x;
end {
emit @x_count;
emit @x_sum;
}
' ../data/small

1
docs6/data/begin-end-example-5.sh Normal file
View file

@ -0,0 +1 @@
mlr stats1 -a count,sum -f x ../data/small

8
docs6/data/begin-end-example-6.sh Normal file
View file

@ -0,0 +1,8 @@
mlr put -q '
@x_count[$a] += 1;
@x_sum[$a] += $x;
end {
emit @x_count, "a";
emit @x_sum, "a";
}
' ../data/small

7
docs6/data/begin-end-example-6a.sh Normal file
View file

@ -0,0 +1,7 @@
mlr --from data/medium put -q '
@x_count[$a][$b] += 1;
@x_sum[$a][$b] += $x;
end {
emit (@x_count, @x_sum), "a", "b";
}
'

1
docs6/data/begin-end-example-7.sh Normal file
View file

@ -0,0 +1 @@
mlr stats1 -a count,sum -f x -g a ../data/small

0
docs6/data/begin-end-example-7b.sh Normal file
View file

15
docs6/data/begin-end-example-8.sh Normal file
View file

@ -0,0 +1,15 @@
mlr put '
begin {
@num_total = 0;
@num_positive = 0;
};
@num_total += 1;
$x > 0.0 {
@num_positive += 1;
$y = log10($x); $z = sqrt($y)
};
end {
emitf @num_total, @num_positive
}
' data/put-gating-example-1.dkvp

5
docs6/data/budget.csv Normal file
View file

@ -0,0 +1,5 @@
# Asana -- here are the budget figures you asked for!
type,quantity
purple,456.78
green,678.12
orange,123.45
1 # Asana -- here are the budget figures you asked for!
2 type,quantity
3 purple,456.78
4 green,678.12
5 orange,123.45

5
docs6/data/bulk-rename-for-loop.mlr Normal file
View file

@ -0,0 +1,5 @@
map newrec = {};
for (oldk, v in $*) {
newrec[gsub(oldk, " ", "_")] = v;
}
$* = newrec

4
docs6/data/clean-whitespace.csv Normal file
View file

@ -0,0 +1,4 @@
Name , Preference
Ann Simons, blue
Bob Wang , red
Carol Vee, yellow
1 Name Preference
2 Ann Simons blue
3 Bob Wang red
4 Carol Vee yellow

4
docs6/data/color-codes.csv Normal file
View file

@ -0,0 +1,4 @@
id,code
3,0000ff
2,00ff00
4,ff0000
1 id code
2 3 0000ff
3 2 00ff00
4 4 ff0000

3
docs6/data/color-names.csv Normal file
View file

@ -0,0 +1,3 @@
id,color
4,red
2,green
1 id color
2 4 red
3 2 green

10078
docs6/data/colored-shapes.dkvp Normal file
View file

File diff suppressed because it is too large Load diff

3
docs6/data/colours.csv Normal file
View file

@ -0,0 +1,3 @@
KEY;DE;EN;ES;FI;FR;IT;NL;PL;RO;TR
masterdata_colourcode_1;Weiß;White;Blanco;Valkoinen;Blanc;Bianco;Wit;Biały;Alb;Beyaz
masterdata_colourcode_2;Schwarz;Black;Negro;Musta;Noir;Nero;Zwart;Czarny;Negru;Siyah
1 KEY DE EN ES FI FR IT NL PL RO TR
2 masterdata_colourcode_1 Weiß White Blanco Valkoinen Blanc Bianco Wit Biały Alb Beyaz
3 masterdata_colourcode_2 Schwarz Black Negro Musta Noir Nero Zwart Czarny Negru Siyah

5
docs6/data/current_counters.csv Normal file
View file

@ -0,0 +1,5 @@
color,count
red,3467
orange,670
yellow,27
blue,6944
1 color count
2 red 3467
3 orange 670
4 yellow 27
5 blue 6944

5
docs6/data/currtemp.csv Normal file
View file

@ -0,0 +1,5 @@
color,current_count
red,3467
orange,670
yellow,27
blue,6944
1 color current_count
2 red 3467
3 orange 670
4 yellow 27
5 blue 6944

57
docs6/data/delimiter-examples.txt Normal file
View file

@ -0,0 +1,57 @@
# Use the `file` command to see if there are CR/LF terminators (in this case,
# there are not):
$ file data/colours.csv
data/colours.csv: UTF-8 Unicode text
# Look at the file to find names of fields
$ cat data/colours.csv
KEY;DE;EN;ES;FI;FR;IT;NL;PL;RO;TR
masterdata_colourcode_1;Weiß;White;Blanco;Valkoinen;Blanc;Bianco;Wit;Biały;Alb;Beyaz
masterdata_colourcode_2;Schwarz;Black;Negro;Musta;Noir;Nero;Zwart;Czarny;Negru;Siyah
# Extract a few fields:
$ mlr --csv cut -f KEY,PL,RO data/colours.csv
(only blank lines appear)
# Use XTAB output format to get a sharper picture of where records/fields
# are being split:
$ mlr --icsv --oxtab cat data/colours.csv
KEY;DE;EN;ES;FI;FR;IT;NL;PL;RO;TR masterdata_colourcode_1;Weiß;White;Blanco;Valkoinen;Blanc;Bianco;Wit;Biały;Alb;Beyaz
KEY;DE;EN;ES;FI;FR;IT;NL;PL;RO;TR masterdata_colourcode_2;Schwarz;Black;Negro;Musta;Noir;Nero;Zwart;Czarny;Negru;Siyah
# Using XTAB output format makes it clearer that KEY;DE;...;RO;TR is being
# treated as a single field name in the CSV header, and likewise each
# subsequent line is being treated as a single field value. This is because
# the default field separator is a comma but we have semicolons here.
# Use XTAB again with different field separator (--fs semicolon):
mlr --icsv --ifs semicolon --oxtab cat data/colours.csv
KEY masterdata_colourcode_1
DE Weiß
EN White
ES Blanco
FI Valkoinen
FR Blanc
IT Bianco
NL Wit
PL Biały
RO Alb
TR Beyaz
KEY masterdata_colourcode_2
DE Schwarz
EN Black
ES Negro
FI Musta
FR Noir
IT Nero
NL Zwart
PL Czarny
RO Negru
TR Siyah
# Using the new field-separator, retry the cut:
mlr --csv --fs semicolon cut -f KEY,PL,RO data/colours.csv
KEY;PL;RO
masterdata_colourcode_1;Biały;Alb
masterdata_colourcode_2;Czarny;Negru

32
docs6/data/dollar-sign.txt Normal file
View file

@ -0,0 +1,32 @@
$ cat sample.csv
EventOccurred,EventType,Description,Status,PaymentType,NameonAccount,TransactionNumber,Amount
10/1/2015,Charged Back,Reason: Authorization Revoked By Customer,Disputed,Checking,John,1,$230.36
10/1/2015,Charged Back,Reason: Authorization Revoked By Customer,Disputed,Checking,Fred,2,$32.25
10/1/2015,Charged Back,Reason: Customer Advises Not Authorized,Disputed,Checking,Bob,3,$39.02
10/1/2015,Charged Back,Reason: Authorization Revoked By Customer,Disputed,Checking,Alice,4,$57.54
10/1/2015,Charged Back,Reason: Authorization Revoked By Customer,Disputed,Checking,Jungle,5,$230.36
10/1/2015,Charged Back,Reason: Payment Stopped,Disputed,Checking,Joe,6,$281.96
10/2/2015,Charged Back,Reason: Customer Advises Not Authorized,Disputed,Checking,Joseph,7,$188.19
10/2/2015,Charged Back,Reason: Customer Advises Not Authorized,Disputed,Checking,Joseph,8,$188.19
10/2/2015,Charged Back,Reason: Payment Stopped,Disputed,Checking,Anthony,9,$250.00
$ mlr --icsv --opprint cat sample.csv
EventOccurred EventType Description Status PaymentType NameonAccount TransactionNumber Amount
10/1/2015 Charged Back Reason: Authorization Revoked By Customer Disputed Checking John 1 $230.36
10/1/2015 Charged Back Reason: Authorization Revoked By Customer Disputed Checking Fred 2 $32.25
10/1/2015 Charged Back Reason: Customer Advises Not Authorized Disputed Checking Bob 3 $39.02
10/1/2015 Charged Back Reason: Authorization Revoked By Customer Disputed Checking Alice 4 $57.54
10/1/2015 Charged Back Reason: Authorization Revoked By Customer Disputed Checking Jungle 5 $230.36
10/1/2015 Charged Back Reason: Payment Stopped Disputed Checking Joe 6 $281.96
10/2/2015 Charged Back Reason: Customer Advises Not Authorized Disputed Checking Joseph 7 $188.19
10/2/2015 Charged Back Reason: Customer Advises Not Authorized Disputed Checking Joseph 8 $188.19
10/2/2015 Charged Back Reason: Payment Stopped Disputed Checking Anthony 9 $250.00
$ mlr --csv put '$Amount = sub(string($Amount), "\$", "")' then stats1 -a sum -f Amount sample.csv
Amount_sum
1497.870000
$ mlr --csv --ofmt '%.2lf' put '$Amount = sub(string($Amount), "\$", "")' then stats1 -a sum -f Amount sample.csv
Amount_sum
1497.87

11
docs6/data/dynamic-nr.sh Normal file
View file

@ -0,0 +1,11 @@
mlr --opprint --from data/small put '
begin{ @nr1 = 0 }
@nr1 += 1;
$nr1 = @nr1
' \
then filter '$x>0.5' \
then put '
begin{ @nr2 = 0 }
@nr2 += 1;
$nr2 = @nr2
'

10
docs6/data/emit-lashed.sh Normal file
View file

@ -0,0 +1,10 @@
mlr --from data/medium --opprint put -q '
@x_count[$a][$b] += 1;
@x_sum[$a][$b] += $x;
end {
for ((a, b), _ in @x_count) {
@x_mean[a][b] = @x_sum[a][b] / @x_count[a][b]
}
emit (@x_sum, @x_count, @x_mean), "a", "b"
}
'

210687
docs6/data/english-words.txt Normal file
View file

File diff suppressed because it is too large Load diff

14
docs6/data/factorial-example.sh Normal file
View file

@ -0,0 +1,14 @@
mlr --opprint --from data/small put '
func f(n) {
if (is_numeric(n)) {
if (n > 0) {
return n * f(n-1);
} else {
return 1;
}
}
# implicitly return absent-null if non-numeric
}
$ox = f($x + NR);
$oi = f($i);
'

1
docs6/data/fe-example-1.sh Normal file
View file

@ -0,0 +1 @@
mlr --from data/small put '$xy = sqrt($x**2 + $y**2)'

1
docs6/data/fe-example-2.sh Normal file
View file

@ -0,0 +1 @@
mlr --from data/small put 'func f(a, b) { return sqrt(a**2 + b**2) } $xy = f($x, $y)'

4
docs6/data/fe-example-3.mlr Normal file
View file

@ -0,0 +1,4 @@
func f(a, b) {
return sqrt(a**2 + b**2)
}
$xy = f($x, $y)

3
docs6/data/fe-example-4.mlr Normal file
View file

@ -0,0 +1,3 @@
func f(a, b) {
return sqrt(a**2 + b**2)
}

13
docs6/data/feature-count.mlr Normal file
View file

@ -0,0 +1,13 @@
for (key in $*) {
@key_counts[key] += 1;
}
@record_count += 1;
end {
for (key in @key_counts) {
@key_fraction[key] = @key_counts[key] / @record_count
}
emit @record_count;
emit @key_counts, "key";
emit @key_fraction,"key"
}

12
docs6/data/features.json Normal file
View file

@ -0,0 +1,12 @@
{ "qoh": 29874, "rate": 1.68, "latency": 0.02 }
{ "name": "alice", "uid": 572 }
{ "qoh": 1227, "rate": 1.01, "latency": 0.07 }
{ "qoh": 13458, "rate": 1.72, "latency": 0.04 }
{ "qoh": 56782, "rate": 1.64 }
{ "qoh": 23512, "rate": 1.71, "latency": 0.03 }
{ "qoh": 9876, "rate": 1.89, "latency": 0.08 }
{ "name": "bill", "uid": 684 }
{ "name": "chuck", "uid2": 908 }
{ "name": "dottie", "uid": 440 }
{ "qoh": 0, "rate": 0.40, "latency": 0.01 }
{ "qoh": 5438, "rate": 1.56, "latency": 0.17 }

18
docs6/data/fibo-cached.sh Normal file
View file

@ -0,0 +1,18 @@
mlr --ofmt '%.9lf' --opprint seqgen --start 1 --stop 28 then put '
func f(n) {
@fcount += 1; # count number of calls to the function
if (is_present(@fcache[n])) { # cache hit
return @fcache[n]
} else { # cache miss
num rv = 1;
if (n >= 2) {
rv = f(n-1) + f(n-2) # recurse
}
@fcache[n] = rv;
return rv
}
}
@fcount = 0;
$o = f($i);
$fcount = @fcount;
' then put '$seconds=systime()' then step -a delta -f seconds then cut -x -f seconds

16
docs6/data/fibo-uncached.sh Normal file
View file

@ -0,0 +1,16 @@
mlr --ofmt '%.9lf' --opprint seqgen --start 1 --stop 28 then put '
func f(n) {
@fcount += 1; # count number of calls to the function
if (n < 2) {
return 1
} else {
return f(n-1) + f(n-2) # recurse
}
}
@fcount = 0;
$o = f($i);
$fcount = @fcount;
' then put '$seconds=systime()' then step -a delta -f seconds then cut -x -f seconds

4
docs6/data/fill-down.csv Normal file
View file

@ -0,0 +1,4 @@
a,b,c
1,,3
4,5,6
7,,9
1 a b c
2 1 3
3 4 5 6
4 7 9

5
docs6/data/filter-multiline-example.txt Normal file
View file

@ -0,0 +1,5 @@
mlr --opprint filter '
($x > 0.5 && $y < 0.5)
||
($x < 0.5 && $y > 0.5)
' then stats2 -a corr -f x,y data/medium

9
docs6/data/flins.csv Normal file
View file

@ -0,0 +1,9 @@
county,tiv_2011,tiv_2012,line
Seminole,22890.55,20848.71,Residential
Miami Dade,1158674.85,1076001.08,Residential
Palm Beach,1174081.5,1856589.17,Residential
Miami Dade,2850980.31,2650932.72,Commercial
Highlands,23006.41,19757.91,Residential
Highlands,49155.16,47362.96,Residential
Duval,1731888.18,2785551.63,Residential
St. Johns,29589.12,35207.53,Residential
1 county tiv_2011 tiv_2012 line
2 Seminole 22890.55 20848.71 Residential
3 Miami Dade 1158674.85 1076001.08 Residential
4 Palm Beach 1174081.5 1856589.17 Residential
5 Miami Dade 2850980.31 2650932.72 Commercial
6 Highlands 23006.41 19757.91 Residential
7 Highlands 49155.16 47362.96 Residential
8 Duval 1731888.18 2785551.63 Residential
9 St. Johns 29589.12 35207.53 Residential

8
docs6/data/flins.json Normal file
View file

@ -0,0 +1,8 @@
{ "county": "Seminole", "tiv_2011": 22890.55, "tiv_2012": 20848.71, "line": "Residential" }
{ "county": "Miami Dade", "tiv_2011": 1158674.85, "tiv_2012": 1076001.08, "line": "Residential" }
{ "county": "Palm Beach", "tiv_2011": 1174081.5, "tiv_2012": 1856589.17, "line": "Residential" }
{ "county": "Miami Dade", "tiv_2011": 2850980.31, "tiv_2012": 2650932.72, "line": "Commercial" }
{ "county": "Highlands", "tiv_2011": 23006.41, "tiv_2012": 19757.91, "line": "Residential" }
{ "county": "Highlands", "tiv_2011": 49155.16, "tiv_2012": 47362.96, "line": "Residential" }
{ "county": "Duval", "tiv_2011": 1731888.18, "tiv_2012": 2785551.63, "line": "Residential" }
{ "county": "St. Johns", "tiv_2011": 29589.12, "tiv_2012": 35207.53, "line": "Residential" }

8
docs6/data/for-oosvar-example-0a.txt Normal file
View file

@ -0,0 +1,8 @@
# Parentheses are optional for single key:
for (k1, v in @a["b"]["c"]) { ... }
for ((k1), v in @a["b"]["c"]) { ... }
# Parentheses are required for multiple keys:
for ((k1, k2), v in @a["b"]["c"]) { ... } # Loop over subhashmap of a variable
for ((k1, k2, k3), v in @a["b"]["c"]) { ... } # Ditto
for ((k1, k2, k3), v in @a { ... } # Loop over variable starting from basename
for ((k1, k2, k3), v in @* { ... } # Loop over all variables (k1 is bound to basename)

10
docs6/data/for-oosvar-example-0b.sh Normal file
View file

@ -0,0 +1,10 @@
mlr -n put --jknquoteint -q '
begin {
@myvar = {
1: 2,
3: { 4 : 5 },
6: { 7: { 8: 9 } }
}
}
end { dump }
'

16
docs6/data/for-oosvar-example-0c.sh Normal file
View file

@ -0,0 +1,16 @@
mlr -n put --jknquoteint -q '
begin {
@myvar = {
1: 2,
3: { 4 : 5 },
6: { 7: { 8: 9 } }
}
}
end {
for (k, v in @myvar) {
print
"key=" . k .
",valuetype=" . typeof(v);
}
}
'

17
docs6/data/for-oosvar-example-0d.sh Normal file
View file

@ -0,0 +1,17 @@
mlr -n put --jknquoteint -q '
begin {
@myvar = {
1: 2,
3: { 4 : 5 },
6: { 7: { 8: 9 } }
}
}
end {
for ((k1, k2), v in @myvar) {
print
"key1=" . k1 .
",key2=" . k2 .
",valuetype=" . typeof(v);
}
}
'

17
docs6/data/for-oosvar-example-0e.sh Normal file
View file

@ -0,0 +1,17 @@
mlr -n put --jknquoteint -q '
begin {
@myvar = {
1: 2,
3: { 4 : 5 },
6: { 7: { 8: 9 } }
}
}
end {
for ((k1, k2), v in @myvar[6]) {
print
"key1=" . k1 .
",key2=" . k2 .
",valuetype=" . typeof(v);
}
}
'

11
docs6/data/for-srec-example-1.sh Normal file
View file

@ -0,0 +1,11 @@
mlr --pprint --from data/for-srec-example.tbl put '
$sum1 = $f1 + $f2 + $f3;
$sum2 = 0;
$sum3 = 0;
for (key, value in $*) {
if (key =~ "^f[0-9]+") {
$sum2 += value;
$sum3 += $[key];
}
}
'

10
docs6/data/for-srec-example-2.sh Normal file
View file

@ -0,0 +1,10 @@
mlr --from data/small --opprint put '
$sum1 = 0;
$sum2 = 0;
for (k,v in $*) {
if (is_numeric(v)) {
$sum1 +=v;
$sum2 += $[k];
}
}
'

9
docs6/data/for-srec-example-3.sh Normal file
View file

@ -0,0 +1,9 @@
mlr --from data/small --opprint put '
sum = 0;
for (k,v in $*) {
if (is_numeric(v)) {
sum += $[k];
}
}
$sum = sum
'

4
docs6/data/for-srec-example.tbl Normal file
View file

@ -0,0 +1,4 @@
label1 label2 f1 f2 f3
blue green 100 240 350
red green 120 11 195
yellow blue 140 0 240

12
docs6/data/fraction-example.csv Normal file
View file

@ -0,0 +1,12 @@
u=female,v=red,n=2458
u=female,v=green,n=192
u=female,v=blue,n=337
u=female,v=purple,n=468
u=female,v=yellow,n=3
u=female,v=orange,n=17
u=male,v=red,n=143
u=male,v=green,n=227
u=male,v=blue,n=2034
u=male,v=purple,n=12
u=male,v=yellow,n=1192
u=male,v=orange,n=448
1 u=female v=red n=2458
2 u=female v=green n=192
3 u=female v=blue n=337
4 u=female v=purple n=468
5 u=female v=yellow n=3
6 u=female v=orange n=17
7 u=male v=red n=143
8 u=male v=green n=227
9 u=male v=blue n=2034
10 u=male v=purple n=12
11 u=male v=yellow n=1192
12 u=male v=orange n=448

15
docs6/data/full-reorg.sh Normal file
View file

@ -0,0 +1,15 @@
mlr put '
begin {
@i_cumu = 0;
}
@i_cumu += $i;
$* = {
"z": $x + y,
"KEYFIELD": $a,
"i": @i_cumu,
"b": $b,
"y": $x,
"x": $y,
};
' data/small

4
docs6/data/headerless.csv Normal file
View file

@ -0,0 +1,4 @@
John,23,present
Fred,34,present
Alice,56,missing
Carol,45,present
1 John 23 present
2 Fred 34 present
3 Alice 56 missing
4 Carol 45 present

5
docs6/data/het-bool.csv Normal file
View file

@ -0,0 +1,5 @@
name,reachable
barney,false
betty,true
fred,true
wilma,1
1 name reachable
2 barney false
3 betty true
4 fred true
5 wilma 1

14
docs6/data/het.csv Normal file
View file

@ -0,0 +1,14 @@
resource,loadsec,ok
/path/to/file,0.45,true
record_count,resource
100,/path/to/file
resource,loadsec,ok
/path/to/second/file,0.32,true
record_count,resource
150,/path/to/second/file
resource,loadsec,ok
/some/other/path,0.97,false
1 resource,loadsec,ok
2 /path/to/file,0.45,true
3 record_count,resource
4 100,/path/to/file
5 resource,loadsec,ok
6 /path/to/second/file,0.32,true
7 record_count,resource
8 150,/path/to/second/file
9 resource,loadsec,ok
10 /some/other/path,0.97,false

5
docs6/data/het.dkvp Normal file
View file

@ -0,0 +1,5 @@
resource=/path/to/file,loadsec=0.45,ok=true
record_count=100,resource=/path/to/file
resource=/path/to/second/file,loadsec=0.32,ok=true
record_count=150,resource=/path/to/second/file
resource=/some/other/path,loadsec=0.97,ok=false

11
docs6/data/if-chain.sh Normal file
View file

@ -0,0 +1,11 @@
mlr put '
if (NR == 2) {
...
} elif (NR ==4) {
...
} elif (NR ==6) {
...
} else {
...
}
'

4
docs6/data/inout.csv Normal file
View file

@ -0,0 +1,4 @@
a_in,a_out,b_in,b_out
436,490,446,195
526,320,963,780
220,888,705,831
1 a_in a_out b_in b_out
2 436 490 446 195
3 526 320 963 780
4 220 888 705 831

3
docs6/data/iqr1.sh Normal file
View file

@ -0,0 +1,3 @@
mlr --oxtab stats1 -f x -a p25,p75 \
then put '$x_iqr = $x_p75 - $x_p25' \
data/medium

7
docs6/data/iqrn.sh Normal file
View file

@ -0,0 +1,7 @@
mlr --oxtab stats1 --fr '[i-z]' -a p25,p75 \
then put 'for (k,v in $*) {
if (k =~ "(.*)_p25") {
$["\1_iqr"] = $["\1_p75"] - $["\1_p25"]
}
}' \
data/medium

6
docs6/data/join-left-example.csv Normal file
View file

@ -0,0 +1,6 @@
id,name
100,alice
200,bob
300,carol
400,david
500,edgar
1 id name
2 100 alice
3 200 bob
4 300 carol
5 400 david
6 500 edgar

21
docs6/data/join-right-example.csv Normal file
View file

@ -0,0 +1,21 @@
status,idcode
present,400
present,100
missing,200
present,100
present,200
missing,100
missing,200
present,300
missing,600
present,400
present,400
present,300
present,100
missing,400
present,200
present,200
present,200
present,200
present,400
present,300
1 status idcode
2 present 400
3 present 100
4 missing 200
5 present 100
6 present 200
7 missing 100
8 missing 200
9 present 300
10 missing 600
11 present 400
12 present 400
13 present 300
14 present 100
15 missing 400
16 present 200
17 present 200
18 present 200
19 present 200
20 present 400
21 present 300

4
docs6/data/join-u-left.csv Normal file
View file

@ -0,0 +1,4 @@
hostname,ipaddr
nadir.east.our.org,10.3.1.18
zenith.west.our.org,10.3.1.27
apoapsis.east.our.org,10.4.5.94
1 hostname ipaddr
2 nadir.east.our.org 10.3.1.18
3 zenith.west.our.org 10.3.1.27
4 apoapsis.east.our.org 10.4.5.94

10
docs6/data/join-u-right.csv Normal file
View file

@ -0,0 +1,10 @@
ipaddr,timestamp,bytes
10.3.1.27,1448762579,4568
10.3.1.18,1448762578,8729
10.4.5.94,1448762579,17445
10.3.1.27,1448762589,12
10.3.1.18,1448762588,44558
10.4.5.94,1448762589,8899
10.3.1.27,1448762599,0
10.3.1.18,1448762598,73425
10.4.5.94,1448762599,12200
1 ipaddr timestamp bytes
2 10.3.1.27 1448762579 4568
3 10.3.1.18 1448762578 8729
4 10.4.5.94 1448762579 17445
5 10.3.1.27 1448762589 12
6 10.3.1.18 1448762588 44558
7 10.4.5.94 1448762589 8899
8 10.3.1.27 1448762599 0
9 10.3.1.18 1448762598 73425
10 10.4.5.94 1448762599 12200

100
docs6/data/json-example-1.json Normal file
View file

@ -0,0 +1,100 @@
{
"color": "yellow",
"shape": "triangle",
"flag": 1,
"i": 11,
"u": 0.6321695890307647,
"v": 0.9887207810889004,
"w": 0.4364983936735774,
"x": 5.7981881667050565
}
{
"color": "red",
"shape": "square",
"flag": 1,
"i": 15,
"u": 0.21966833570651523,
"v": 0.001257332190235938,
"w": 0.7927778364718627,
"x": 2.944117399716207
}
{
"color": "red",
"shape": "circle",
"flag": 1,
"i": 16,
"u": 0.20901671281497636,
"v": 0.29005231936593445,
"w": 0.13810280912907674,
"x": 5.065034003400998
}
{
"color": "red",
"shape": "square",
"flag": 0,
"i": 48,
"u": 0.9562743938458542,
"v": 0.7467203085342884,
"w": 0.7755423050923582,
"x": 7.117831369597269
}
{
"color": "purple",
"shape": "triangle",
"flag": 0,
"i": 51,
"u": 0.4355354501763202,
"v": 0.8591292672156728,
"w": 0.8122903963006748,
"x": 5.753094629505863
}
{
"color": "red",
"shape": "square",
"flag": 0,
"i": 64,
"u": 0.2015510269821953,
"v": 0.9531098083420033,
"w": 0.7719912015786777,
"x": 5.612050466474166
}
{
"color": "purple",
"shape": "triangle",
"flag": 0,
"i": 65,
"u": 0.6842806710360729,
"v": 0.5823723856331258,
"w": 0.8014053396013747,
"x": 5.805148213865135
}
{
"color": "yellow",
"shape": "circle",
"flag": 1,
"i": 73,
"u": 0.6033649768090676,
"v": 0.42370791211283076,
"w": 0.639785141788745,
"x": 7.006414410739997
}
{
"color": "yellow",
"shape": "circle",
"flag": 1,
"i": 87,
"u": 0.2856563669907619,
"v": 0.8335161523929382,
"w": 0.6350579406858395,
"x": 6.350035616385983
}
{
"color": "purple",
"shape": "square",
"flag": 0,
"i": 91,
"u": 0.25992639068499135,
"v": 0.824321938346312,
"w": 0.7237347131411271,
"x": 6.854221024776646
}

Some files were not shown because too many files have changed in this diff Show more