Mirrors/miller
1
0
Fork 0
mirror of https://github.com/johnkerl/miller.git synced 2026-08-03 13:03:00 +00:00
Code Issues Projects Releases Wiki Activity

Remove leading "$ " from code examples, now that highlighting is in place

Browse source
This commit is contained in:
John Kerl 2021-06-20 22:56:42 -04:00
parent 26894014db
commit d972b52639
41 changed files with 661 additions and 664 deletions
Download patch file Download diff file Expand all files Collapse all files

82
docs6/10min.rst
View file

@ -20,7 +20,7 @@ As a first check, you should be able to run ``mlr --version`` at your system's c
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --version
mlr --version
Miller v6.0.0-dev
As a second check, given (`example.csv <./example.csv>`_) you should be able to do
@ -28,7 +28,7 @@ As a second check, given (`example.csv <./example.csv>`_) you should be able to
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --csv cat example.csv
mlr --csv cat example.csv
color,shape,flag,index,quantity,rate
yellow,triangle,true,11,43.6498,9.8870
red,square,true,15,79.2778,0.0130
@ -44,7 +44,7 @@ As a second check, given (`example.csv <./example.csv>`_) you should be able to
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --icsv --opprint cat example.csv
mlr --icsv --opprint cat example.csv
color shape flag index quantity rate
yellow triangle true 11 43.6498 9.8870
red square true 15 79.2778 0.0130
@ -69,7 +69,7 @@ Let's take a quick look at some of the most useful Miller verbs -- file-format-a
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --csv cat example.csv
mlr --csv cat example.csv
color,shape,flag,index,quantity,rate
yellow,triangle,true,11,43.6498,9.8870
red,square,true,15,79.2778,0.0130
@ -87,7 +87,7 @@ But ``mlr cat`` can also do format conversion -- for example, you can pretty-pri
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --icsv --opprint cat example.csv
mlr --icsv --opprint cat example.csv
color shape flag index quantity rate
yellow triangle true 11 43.6498 9.8870
red square true 15 79.2778 0.0130
@ -105,7 +105,7 @@ But ``mlr cat`` can also do format conversion -- for example, you can pretty-pri
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --csv head -n 4 example.csv
mlr --csv head -n 4 example.csv
color,shape,flag,index,quantity,rate
yellow,triangle,true,11,43.6498,9.8870
red,square,true,15,79.2778,0.0130
@ -115,7 +115,7 @@ But ``mlr cat`` can also do format conversion -- for example, you can pretty-pri
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --csv tail -n 4 example.csv
mlr --csv tail -n 4 example.csv
color,shape,flag,index,quantity,rate
purple,triangle,false,65,80.1405,5.8240
yellow,circle,true,73,63.9785,4.2370
@ -125,7 +125,7 @@ But ``mlr cat`` can also do format conversion -- for example, you can pretty-pri
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --icsv --ojson tail -n 2 example.csv
mlr --icsv --ojson tail -n 2 example.csv
{
"color": "yellow",
"shape": "circle",
@ -148,7 +148,7 @@ You can sort on a single field:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --icsv --opprint sort -f shape example.csv
mlr --icsv --opprint sort -f shape example.csv
color shape flag index quantity rate
red circle true 16 13.8103 2.9010
yellow circle true 73 63.9785 4.2370
@ -166,7 +166,7 @@ Or, you can sort primarily alphabetically on one field, then secondarily numeric
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --icsv --opprint sort -f shape -nr index example.csv
mlr --icsv --opprint sort -f shape -nr index example.csv
color shape flag index quantity rate
yellow circle true 87 63.5058 8.3350
yellow circle true 73 63.9785 4.2370
@ -184,7 +184,7 @@ If there are fields you don't want to see in your data, you can use ``cut`` to k
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --icsv --opprint cut -f flag,shape example.csv
mlr --icsv --opprint cut -f flag,shape example.csv
shape flag
triangle true
square true
@ -202,7 +202,7 @@ You can also use ``cut -o`` to keep specified fields, but in your preferred orde
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --icsv --opprint cut -o -f flag,shape example.csv
mlr --icsv --opprint cut -o -f flag,shape example.csv
flag shape
true triangle
true square
@ -220,7 +220,7 @@ You can use ``cut -x`` to omit fields you don't care about:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --icsv --opprint cut -x -f flag,shape example.csv
mlr --icsv --opprint cut -x -f flag,shape example.csv
color index quantity rate
yellow 11 43.6498 9.8870
red 15 79.2778 0.0130
@ -238,7 +238,7 @@ You can use ``filter`` to keep only records you care about:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --icsv --opprint filter '$color == "red"' example.csv
mlr --icsv --opprint filter '$color == "red"' example.csv
color shape flag index quantity rate
red square true 15 79.2778 0.0130
red circle true 16 13.8103 2.9010
@ -248,7 +248,7 @@ You can use ``filter`` to keep only records you care about:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --icsv --opprint filter '$color == "red" && $flag == true' example.csv
mlr --icsv --opprint filter '$color == "red" && $flag == true' example.csv
color shape flag index quantity rate
red square true 15 79.2778 0.0130
red circle true 16 13.8103 2.9010
@ -258,7 +258,7 @@ You can use ``put`` to create new fields which are computed from other fields:
.. code-block:: none
:emphasize-lines: 1-4
$ mlr --icsv --opprint put '
mlr --icsv --opprint put '
$ratio = $quantity / $rate;
$color_shape = $color . "_" . $shape
' example.csv
@ -279,7 +279,7 @@ Even though Miller's main selling point is name-indexing, sometimes you really w
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --icsv --opprint put '$[[3]] = "NEW"' example.csv
mlr --icsv --opprint put '$[[3]] = "NEW"' example.csv
color shape NEW index quantity rate
yellow triangle true 11 43.6498 9.8870
red square true 15 79.2778 0.0130
@ -295,7 +295,7 @@ Even though Miller's main selling point is name-indexing, sometimes you really w
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --icsv --opprint put '$[[[3]]] = "NEW"' example.csv
mlr --icsv --opprint put '$[[[3]]] = "NEW"' example.csv
color shape flag index quantity rate
yellow triangle NEW 11 43.6498 9.8870
red square NEW 15 79.2778 0.0130
@ -318,7 +318,7 @@ Miller takes all the files from the command line as an input stream. But it's fo
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/a.csv
cat data/a.csv
a,b,c
1,2,3
4,5,6
@ -326,14 +326,14 @@ Miller takes all the files from the command line as an input stream. But it's fo
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/b.csv
cat data/b.csv
a,b,c
7,8,9
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/a.csv data/b.csv
cat data/a.csv data/b.csv
a,b,c
1,2,3
4,5,6
@ -345,7 +345,7 @@ However, ``mlr cat`` will not:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --csv cat data/a.csv data/b.csv
mlr --csv cat data/a.csv data/b.csv
a,b,c
1,2,3
4,5,6
@ -359,7 +359,7 @@ Often we want to chain queries together -- for example, sorting by a field and t
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --csv sort -nr index example.csv | mlr --icsv --opprint head -n 3
mlr --csv sort -nr index example.csv | mlr --icsv --opprint head -n 3
color shape flag index quantity rate
purple square false 91 72.3735 8.2430
yellow circle true 87 63.5058 8.3350
@ -370,7 +370,7 @@ This works fine -- but Miller also lets you chain verbs together using the word
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --icsv --opprint sort -nr index then head -n 3 example.csv
mlr --icsv --opprint sort -nr index then head -n 3 example.csv
color shape flag index quantity rate
purple square false 91 72.3735 8.2430
yellow circle true 87 63.5058 8.3350
@ -381,7 +381,7 @@ As another convenience, you can put the filename first using ``--from``. When yo
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --icsv --opprint --from example.csv sort -nr index then head -n 3
mlr --icsv --opprint --from example.csv sort -nr index then head -n 3
color shape flag index quantity rate
purple square false 91 72.3735 8.2430
yellow circle true 87 63.5058 8.3350
@ -390,7 +390,7 @@ As another convenience, you can put the filename first using ``--from``. When yo
.. code-block:: none
:emphasize-lines: 1-4
$ mlr --icsv --opprint --from example.csv \
mlr --icsv --opprint --from example.csv \
sort -nr index \
then head -n 3 \
then cut -f shape,quantity
@ -409,7 +409,7 @@ Here are the records with the top three ``index`` values:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --icsv --opprint sort -nr index then head -n 3 example.csv
mlr --icsv --opprint sort -nr index then head -n 3 example.csv
color shape flag index quantity rate
purple square false 91 72.3735 8.2430
yellow circle true 87 63.5058 8.3350
@ -420,7 +420,7 @@ Lots of Miller commands take a ``-g`` option for group-by: here, ``head -n 1 -g
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --icsv --opprint sort -f shape -nr index then head -n 1 -g shape example.csv
mlr --icsv --opprint sort -f shape -nr index then head -n 1 -g shape example.csv
color shape flag index quantity rate
yellow circle true 87 63.5058 8.3350
purple square false 91 72.3735 8.2430
@ -431,7 +431,7 @@ Statistics can be computed with or without group-by field(s):
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --icsv --opprint --from example.csv stats1 -a count,min,mean,max -f quantity -g shape
mlr --icsv --opprint --from example.csv stats1 -a count,min,mean,max -f quantity -g shape
shape quantity_count quantity_min quantity_mean quantity_max
triangle 3 43.6498 68.33976666666666 81.229
square 4 72.3735 76.60114999999999 79.2778
@ -440,7 +440,7 @@ Statistics can be computed with or without group-by field(s):
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --icsv --opprint --from example.csv stats1 -a count,min,mean,max -f quantity -g shape,color
mlr --icsv --opprint --from example.csv stats1 -a count,min,mean,max -f quantity -g shape,color
shape color quantity_count quantity_min quantity_mean quantity_max
triangle yellow 1 43.6498 43.6498 43.6498
square red 3 77.1991 78.01036666666666 79.2778
@ -454,7 +454,7 @@ If your output has a lot of columns, you can use XTAB format to line things up v
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --icsv --oxtab --from example.csv stats1 -a p0,p10,p25,p50,p75,p90,p99,p100 -f rate
mlr --icsv --oxtab --from example.csv stats1 -a p0,p10,p25,p50,p75,p90,p99,p100 -f rate
rate_p0 0.0130
rate_p10 2.9010
rate_p25 4.2370
@ -534,14 +534,14 @@ Sometimes, though, we want to print output to another file. Just use **> outputf
.. code-block:: none
:emphasize-lines: 1,1
% mlr --icsv --opprint cat example.csv > newfile.csv
mlr --icsv --opprint cat example.csv > newfile.csv
# Output goes to the new file;
# nothing is printed to the screen.
.. code-block:: none
:emphasize-lines: 1,1
% cat newfile.csv
cat newfile.csv
color shape flag index quantity rate
yellow triangle true 11 43.6498 9.8870
red square true 15 79.2778 0.0130
@ -559,12 +559,12 @@ Other times we just want our files to be **changed in-place**: just use **mlr -I
.. code-block:: none
:emphasize-lines: 1,1
% cp example.csv newfile.txt
cp example.csv newfile.txt
.. code-block:: none
:emphasize-lines: 1,1
% cat newfile.txt
cat newfile.txt
color,shape,flag,index,quantity,rate
yellow,triangle,true,11,43.6498,9.8870
red,square,true,15,79.2778,0.0130
@ -580,12 +580,12 @@ Other times we just want our files to be **changed in-place**: just use **mlr -I
.. code-block:: none
:emphasize-lines: 1,1
% mlr -I --csv sort -f shape newfile.txt
mlr -I --csv sort -f shape newfile.txt
.. code-block:: none
:emphasize-lines: 1,1
% cat newfile.txt
cat newfile.txt
color,shape,flag,index,quantity,rate
red,circle,true,16,13.8103,2.9010
yellow,circle,true,73,63.9785,4.2370
@ -612,12 +612,12 @@ Lastly, using ``tee`` within ``put``, you can split your input data into separat
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --csv --from example.csv put -q 'tee > $shape.".csv", $*'
mlr --csv --from example.csv put -q 'tee > $shape.".csv", $*'
.. code-block:: none
:emphasize-lines: 1-1
$ cat circle.csv
cat circle.csv
color,shape,flag,index,quantity,rate
red,circle,true,16,13.8103,2.9010
yellow,circle,true,73,63.9785,4.2370
@ -626,7 +626,7 @@ Lastly, using ``tee`` within ``put``, you can split your input data into separat
.. code-block:: none
:emphasize-lines: 1-1
$ cat square.csv
cat square.csv
color,shape,flag,index,quantity,rate
red,square,true,15,79.2778,0.0130
red,square,false,48,77.5542,7.4670
@ -636,7 +636,7 @@ Lastly, using ``tee`` within ``put``, you can split your input data into separat
.. code-block:: none
:emphasize-lines: 1-1
$ cat triangle.csv
cat triangle.csv
color,shape,flag,index,quantity,rate
yellow,triangle,true,11,43.6498,9.8870
purple,triangle,false,51,81.2290,8.5910

12
docs6/10min.rst.in
View file

@ -276,14 +276,14 @@ Sometimes, though, we want to print output to another file. Just use **> outputf
.. code-block:: none
:emphasize-lines: 1,1
% mlr --icsv --opprint cat example.csv > newfile.csv
mlr --icsv --opprint cat example.csv > newfile.csv
# Output goes to the new file;
# nothing is printed to the screen.
.. code-block:: none
:emphasize-lines: 1,1
% cat newfile.csv
cat newfile.csv
color shape flag index quantity rate
yellow triangle true 11 43.6498 9.8870
red square true 15 79.2778 0.0130
@ -301,12 +301,12 @@ Other times we just want our files to be **changed in-place**: just use **mlr -I
.. code-block:: none
:emphasize-lines: 1,1
% cp example.csv newfile.txt
cp example.csv newfile.txt
.. code-block:: none
:emphasize-lines: 1,1
% cat newfile.txt
cat newfile.txt
color,shape,flag,index,quantity,rate
yellow,triangle,true,11,43.6498,9.8870
red,square,true,15,79.2778,0.0130
@ -322,12 +322,12 @@ Other times we just want our files to be **changed in-place**: just use **mlr -I
.. code-block:: none
:emphasize-lines: 1,1
% mlr -I --csv sort -f shape newfile.txt
mlr -I --csv sort -f shape newfile.txt
.. code-block:: none
:emphasize-lines: 1,1
% cat newfile.txt
cat newfile.txt
color,shape,flag,index,quantity,rate
red,circle,true,16,13.8103,2.9010
yellow,circle,true,73,63.9785,4.2370

50
docs6/cookbook.rst
View file

@ -12,7 +12,7 @@ Here are some ways to use the type-checking options as described in :ref:`refere
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/het-bool.csv
cat data/het-bool.csv
name,reachable
barney,false
betty,true
@ -24,7 +24,7 @@ One option is to coerce everything to boolean, or integer:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --icsv --opprint put '$reachable = boolean($reachable)' data/het-bool.csv
mlr --icsv --opprint put '$reachable = boolean($reachable)' data/het-bool.csv
name reachable
barney false
betty true
@ -34,7 +34,7 @@ One option is to coerce everything to boolean, or integer:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --icsv --opprint put '$reachable = int(boolean($reachable))' data/het-bool.csv
mlr --icsv --opprint put '$reachable = int(boolean($reachable))' data/het-bool.csv
name reachable
barney 0
betty 1
@ -46,7 +46,7 @@ A second option is to flag badly formatted data within the output stream:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --icsv --opprint put '$format_ok = is_string($reachable)' data/het-bool.csv
mlr --icsv --opprint put '$format_ok = is_string($reachable)' data/het-bool.csv
name reachable format_ok
barney false false
betty true false
@ -58,7 +58,7 @@ Or perhaps to flag badly formatted data outside the output stream:
.. code-block:: none
:emphasize-lines: 1-3
$ mlr --icsv --opprint put '
mlr --icsv --opprint put '
if (!is_string($reachable)) {eprint "Malformed at NR=".NR}
' data/het-bool.csv
Malformed at NR=1
@ -76,7 +76,7 @@ A third way is to abort the process on first instance of bad data:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --csv put '$reachable = asserting_string($reachable)' data/het-bool.csv
mlr --csv put '$reachable = asserting_string($reachable)' data/het-bool.csv
Miller: is_string type-assertion failed at NR=1 FNR=1 FILENAME=data/het-bool.csv
Showing differences between successive queries
@ -87,7 +87,7 @@ Suppose you have a database query which you run at one point in time, producing
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/previous_counters.csv
cat data/previous_counters.csv
color,count
red,3472
blue,6838
@ -97,7 +97,7 @@ Suppose you have a database query which you run at one point in time, producing
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/current_counters.csv
cat data/current_counters.csv
color,count
red,3467
orange,670
@ -111,12 +111,12 @@ First, rename counter columns to make them distinct:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --csv rename count,previous_count data/previous_counters.csv > data/prevtemp.csv
mlr --csv rename count,previous_count data/previous_counters.csv > data/prevtemp.csv
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/prevtemp.csv
cat data/prevtemp.csv
color,previous_count
red,3472
blue,6838
@ -126,12 +126,12 @@ First, rename counter columns to make them distinct:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --csv rename count,current_count data/current_counters.csv > data/currtemp.csv
mlr --csv rename count,current_count data/current_counters.csv > data/currtemp.csv
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/currtemp.csv
cat data/currtemp.csv
color,current_count
red,3467
orange,670
@ -143,7 +143,7 @@ Then, join on the key field(s), and use unsparsify to zero-fill counters absent
.. code-block:: none
:emphasize-lines: 1-5
$ mlr --icsv --opprint \
mlr --icsv --opprint \
join -j color --ul --ur -f data/prevtemp.csv \
then unsparsify --fill-with 0 \
then put '$count_delta = $current_count - $previous_count' \
@ -168,7 +168,7 @@ For example, mapping numeric values down a column to the percentage between thei
.. code-block:: none
:emphasize-lines: 1-16
$ mlr --from data/small --opprint put -q '
mlr --from data/small --opprint put -q '
# These are executed once per record, which is the first pass.
# The key is to use NR to index an out-of-stream variable to
# retain all the x-field values.
@ -199,7 +199,7 @@ Similarly, finding the total record count requires first reading through all the
.. code-block:: none
:emphasize-lines: 1-11
$ mlr --opprint --from data/small put -q '
mlr --opprint --from data/small put -q '
@records[NR] = $*;
end {
for((I,k),v in @records) {
@ -225,7 +225,7 @@ The idea is to retain records having the largest value of ``n`` in the following
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --itsv --opprint cat data/maxrows.tsv
mlr --itsv --opprint cat data/maxrows.tsv
a b n score
purple red 5 0.743231
blue purple 2 0.093710
@ -264,7 +264,7 @@ Of course, the largest value of ``n`` isn't known until after all data have been
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/maxrows.mlr
cat data/maxrows.mlr
# Retain all records
@records[NR] = $*;
# Track max value of n
@ -284,7 +284,7 @@ Of course, the largest value of ``n`` isn't known until after all data have been
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --itsv --opprint put -q -f data/maxrows.mlr data/maxrows.tsv
mlr --itsv --opprint put -q -f data/maxrows.mlr data/maxrows.tsv
a b n score
purple red 5 0.743231
purple red 5 0.389055
@ -339,7 +339,7 @@ Then
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --json put -q -f data/feature-count.mlr data/features.json
mlr --json put -q -f data/feature-count.mlr data/features.json
{
"record_count": 12
}
@ -395,7 +395,7 @@ Then
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --ijson --opprint put -q -f data/feature-count.mlr data/features.json
mlr --ijson --opprint put -q -f data/feature-count.mlr data/features.json
record_count
12
@ -425,7 +425,7 @@ For example, suppose you have JSON input like this:
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/sparse.json
cat data/sparse.json
{"a":1,"b":2,"v":3}
{"u":1,"b":2}
{"a":1,"v":2,"x":3}
@ -436,7 +436,7 @@ There are field names ``a``, ``b``, ``v``, ``u``, ``x``, ``w`` in the data -- bu
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/unsparsify.mlr
cat data/unsparsify.mlr
# First pass:
# Remember all unique key names:
for (k in $*) {
@ -467,7 +467,7 @@ There are field names ``a``, ``b``, ``v``, ``u``, ``x``, ``w`` in the data -- bu
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --json put -q -f data/unsparsify.mlr data/sparse.json
mlr --json put -q -f data/unsparsify.mlr data/sparse.json
{
"a": 1,
"b": 2,
@ -504,7 +504,7 @@ There are field names ``a``, ``b``, ``v``, ``u``, ``x``, ``w`` in the data -- bu
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --ijson --ocsv put -q -f data/unsparsify.mlr data/sparse.json
mlr --ijson --ocsv put -q -f data/unsparsify.mlr data/sparse.json
a,b,v,u,x,w
1,2,3,,,
,2,,1,,
@ -514,7 +514,7 @@ There are field names ``a``, ``b``, ``v``, ``u``, ``x``, ``w`` in the data -- bu
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --ijson --opprint put -q -f data/unsparsify.mlr data/sparse.json
mlr --ijson --opprint put -q -f data/unsparsify.mlr data/sparse.json
a b v u x w
1 2 3 - - -
- 2 - 1 - -

18
docs6/cookbook2.rst
View file

@ -112,7 +112,7 @@ For one or more specified field names, simply compute p25 and p75, then write th
.. code-block:: none
:emphasize-lines: 1-3
$ mlr --oxtab stats1 -f x -a p25,p75 \
mlr --oxtab stats1 -f x -a p25,p75 \
then put '$x_iqr = $x_p75 - $x_p25' \
data/medium
x_p25 0.24667037823231752
@ -124,7 +124,7 @@ For wildcarded field names, first compute p25 and p75, then loop over field name
.. code-block:: none
:emphasize-lines: 1-7
$ mlr --oxtab stats1 --fr '[i-z]' -a p25,p75 \
mlr --oxtab stats1 --fr '[i-z]' -a p25,p75 \
then put 'for (k,v in $*) {
if (k =~ "(.*)_p25") {
$["\1_iqr"] = $["\1_p75"] - $["\1_p25"]
@ -140,7 +140,7 @@ This might be more elegantly implemented as an option within the ``stats1`` verb
.. code-block:: none
:emphasize-lines: 1-24
$ mlr --from data/medium put -q '
mlr --from data/medium put -q '
# Using the y field for weighting in this example
weight = $y;
@ -178,7 +178,7 @@ Here we can chain together a few simple building blocks:
.. code-block:: none
:emphasize-lines: 1-1
$ cat expo-sample.sh
cat expo-sample.sh
# Generate 100,000 pairs of independent and identically distributed
# exponentially distributed random variables with the same rate parameter
# (namely, 2.5). Then compute histograms of one of them, along with
@ -221,7 +221,7 @@ The output is as follows:
.. code-block:: none
:emphasize-lines: 1-1
$ sh expo-sample.sh
sh expo-sample.sh
bin_lo bin_hi u_count s_count p_count
0 0.04 [64]*******************#[9554] [326]#...................[3703] [19]*******************#[39809]
0.04 0.08 [64]*****************...[9554] [326]*****...............[3703] [19]*******.............[39809]
@ -282,7 +282,7 @@ The `Sieve of Eratosthenes <http://en.wikipedia.org/wiki/Sieve_of_Eratosthenes>`
.. code-block:: none
:emphasize-lines: 1-1
$ cat programs/sieve.mlr
cat programs/sieve.mlr
# ================================================================
# Sieve of Eratosthenes: simple example of Miller DSL as programming language.
# ================================================================
@ -319,7 +319,7 @@ The `Sieve of Eratosthenes <http://en.wikipedia.org/wiki/Sieve_of_Eratosthenes>`
.. code-block:: none
:emphasize-lines: 1-1
$ mlr -n put -f programs/sieve.mlr
mlr -n put -f programs/sieve.mlr
2
3
5
@ -356,7 +356,7 @@ The (approximate) computation of points in the complex plane which are and aren'
.. code-block:: none
:emphasize-lines: 1-1
$ cat programs/mand.mlr
cat programs/mand.mlr
# Mandelbrot set generator: simple example of Miller DSL as programming language.
begin {
# Set defaults
@ -463,7 +463,7 @@ At standard resolution this makes a nice little ASCII plot:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr -n put -f ./programs/mand.mlr
mlr -n put -f ./programs/mand.mlr
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

44
docs6/cookbook3.rst
View file

@ -12,7 +12,7 @@ One of Miller's strengths is its compact notation: for example, given input of t
.. code-block:: none
:emphasize-lines: 1-1
$ head -n 5 ../data/medium
head -n 5 ../data/medium
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
@ -24,7 +24,7 @@ you can simply do
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --oxtab stats1 -a sum -f x ../data/medium
mlr --oxtab stats1 -a sum -f x ../data/medium
x_sum 4986.019681679581
or
@ -32,7 +32,7 @@ or
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --opprint stats1 -a sum -f x -g b ../data/medium
mlr --opprint stats1 -a sum -f x -g b ../data/medium
b x_sum
pan 965.7636699425815
wye 1023.5484702619565
@ -45,7 +45,7 @@ rather than the more tedious
.. code-block:: none
:emphasize-lines: 1-6
$ mlr --oxtab put -q '
mlr --oxtab put -q '
@x_sum += $x;
end {
emit @x_sum
@ -58,7 +58,7 @@ or
.. code-block:: none
:emphasize-lines: 1-6
$ mlr --opprint put -q '
mlr --opprint put -q '
@x_sum[$b] += $x;
end {
emit @x_sum, "b"
@ -83,14 +83,14 @@ Mean without/with oosvars
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --opprint stats1 -a mean -f x data/medium
mlr --opprint stats1 -a mean -f x data/medium
x_mean
0.49860196816795804
.. code-block:: none
:emphasize-lines: 1-8
$ mlr --opprint put -q '
mlr --opprint put -q '
@x_sum += $x;
@x_count += 1;
end {
@ -107,7 +107,7 @@ Keyed mean without/with oosvars
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --opprint stats1 -a mean -f x -g a,b data/medium
mlr --opprint stats1 -a mean -f x -g a,b data/medium
a b x_mean
pan pan 0.5133141190437597
eks pan 0.48507555383425127
@ -138,7 +138,7 @@ Keyed mean without/with oosvars
.. code-block:: none
:emphasize-lines: 1-10
$ mlr --opprint put -q '
mlr --opprint put -q '
@x_sum[$a][$b] += $x;
@x_count[$a][$b] += 1;
end{
@ -181,7 +181,7 @@ Variance and standard deviation without/with oosvars
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --oxtab stats1 -a count,sum,mean,var,stddev -f x data/medium
mlr --oxtab stats1 -a count,sum,mean,var,stddev -f x data/medium
x_count 10000
x_sum 4986.019681679581
x_mean 0.49860196816795804
@ -191,7 +191,7 @@ Variance and standard deviation without/with oosvars
.. code-block:: none
:emphasize-lines: 1-1
$ cat variance.mlr
cat variance.mlr
@n += 1;
@sumx += $x;
@sumx2 += $x**2;
@ -205,7 +205,7 @@ Variance and standard deviation without/with oosvars
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --oxtab put -q -f variance.mlr data/medium
mlr --oxtab put -q -f variance.mlr data/medium
n 10000
sumx 4986.019681679581
sumx2 3328.652400179729
@ -221,14 +221,14 @@ Min/max without/with oosvars
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --oxtab stats1 -a min,max -f x data/medium
mlr --oxtab stats1 -a min,max -f x data/medium
x_min 4.509679127584487e-05
x_max 0.999952670371898
.. code-block:: none
:emphasize-lines: 1-5
$ mlr --oxtab put -q '
mlr --oxtab put -q '
@x_min = min(@x_min, $x);
@x_max = max(@x_max, $x);
end{emitf @x_min, @x_max}
@ -242,7 +242,7 @@ Keyed min/max without/with oosvars
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --opprint stats1 -a min,max -f x -g a data/medium
mlr --opprint stats1 -a min,max -f x -g a data/medium
a x_min x_max
pan 0.00020390740306253097 0.9994029107062516
eks 0.0006917972627396018 0.9988110946859143
@ -253,7 +253,7 @@ Keyed min/max without/with oosvars
.. code-block:: none
:emphasize-lines: 1-7
$ mlr --opprint --from data/medium put -q '
mlr --opprint --from data/medium put -q '
@min[$a] = min(@min[$a], $x);
@max[$a] = max(@max[$a], $x);
end{
@ -273,7 +273,7 @@ Delta without/with oosvars
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --opprint step -a delta -f x data/small
mlr --opprint step -a delta -f x data/small
a b i x y x_delta
pan pan 1 0.3467901443380824 0.7268028627434533 0
eks pan 2 0.7586799647899636 0.5221511083334797 0.41188982045188116
@ -284,7 +284,7 @@ Delta without/with oosvars
.. code-block:: none
:emphasize-lines: 1-4
$ mlr --opprint put '
mlr --opprint put '
$x_delta = is_present(@last) ? $x - @last : 0;
@last = $x
' data/small
@ -301,7 +301,7 @@ Keyed delta without/with oosvars
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --opprint step -a delta -f x -g a data/small
mlr --opprint step -a delta -f x -g a data/small
a b i x y x_delta
pan pan 1 0.3467901443380824 0.7268028627434533 0
eks pan 2 0.7586799647899636 0.5221511083334797 0
@ -312,7 +312,7 @@ Keyed delta without/with oosvars
.. code-block:: none
:emphasize-lines: 1-4
$ mlr --opprint put '
mlr --opprint put '
$x_delta = is_present(@last[$a]) ? $x - @last[$a] : 0;
@last[$a]=$x
' data/small
@ -329,7 +329,7 @@ Exponentially weighted moving averages without/with oosvars
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --opprint step -a ewma -d 0.1 -f x data/small
mlr --opprint step -a ewma -d 0.1 -f x data/small
a b i x y x_ewma_0.1
pan pan 1 0.3467901443380824 0.7268028627434533 0.3467901443380824
eks pan 2 0.7586799647899636 0.5221511083334797 0.3879791263832706
@ -340,7 +340,7 @@ Exponentially weighted moving averages without/with oosvars
.. code-block:: none
:emphasize-lines: 1-5
$ mlr --opprint put '
mlr --opprint put '
begin{ @a=0.1 };
$e = NR==1 ? $x : @a * $x + (1 - @a) * @e;
@e=$e

12
docs6/cookbook4.rst
View file

@ -75,7 +75,7 @@ And here is an example using them:
.. code-block:: none
:emphasize-lines: 1-1
$ cat polyglot-dkvp-io/example.py
cat polyglot-dkvp-io/example.py
#!/usr/bin/env python
import sys
@ -116,7 +116,7 @@ Run as-is:
.. code-block:: none
:emphasize-lines: 1-1
$ python polyglot-dkvp-io/example.py < data/small
python polyglot-dkvp-io/example.py < data/small
a=pan,b=pan,i=1,y=0.7268028627434533,ab=panpan,iy=1.7268028627434533,ta=str,tb=str,ti=int,ty=float,tab=str,tiy=float
a=eks,b=pan,i=2,y=0.5221511083334797,ab=ekspan,iy=2.5221511083334796,ta=str,tb=str,ti=int,ty=float,tab=str,tiy=float
a=wye,b=wye,i=3,y=0.33831852551664776,ab=wyewye,iy=3.3383185255166477,ta=str,tb=str,ti=int,ty=float,tab=str,tiy=float
@ -128,7 +128,7 @@ Run as-is, then pipe to Miller for pretty-printing:
.. code-block:: none
:emphasize-lines: 1-1
$ python polyglot-dkvp-io/example.py < data/small | mlr --opprint cat
python polyglot-dkvp-io/example.py < data/small | mlr --opprint cat
a b i y ab iy ta tb ti ty tab tiy
pan pan 1 0.7268028627434533 panpan 1.7268028627434533 str str int float str float
eks pan 2 0.5221511083334797 ekspan 2.5221511083334796 str str int float str float
@ -201,7 +201,7 @@ And here is an example using them:
.. code-block:: none
:emphasize-lines: 1-1
$ cat polyglot-dkvp-io/example.rb
cat polyglot-dkvp-io/example.rb
#!/usr/bin/env ruby
require 'dkvp_io'
@ -232,7 +232,7 @@ Run as-is:
.. code-block:: none
:emphasize-lines: 1-1
$ ruby -I./polyglot-dkvp-io polyglot-dkvp-io/example.rb data/small
ruby -I./polyglot-dkvp-io polyglot-dkvp-io/example.rb data/small
a=pan,b=pan,i=1,y=0.7268028627434533,ab=panpan,iy=1.7268028627434533,ta=String,tb=String,ti=Integer,ty=Float,tab=String,tiy=Float
a=eks,b=pan,i=2,y=0.5221511083334797,ab=ekspan,iy=2.5221511083334796,ta=String,tb=String,ti=Integer,ty=Float,tab=String,tiy=Float
a=wye,b=wye,i=3,y=0.33831852551664776,ab=wyewye,iy=3.3383185255166477,ta=String,tb=String,ti=Integer,ty=Float,tab=String,tiy=Float
@ -244,7 +244,7 @@ Run as-is, then pipe to Miller for pretty-printing:
.. code-block:: none
:emphasize-lines: 1-1
$ ruby -I./polyglot-dkvp-io polyglot-dkvp-io/example.rb data/small | mlr --opprint cat
ruby -I./polyglot-dkvp-io polyglot-dkvp-io/example.rb data/small | mlr --opprint cat
a b i y ab iy ta tb ti ty tab tiy
pan pan 1 0.7268028627434533 panpan 1.7268028627434533 String String Integer Float String Float
eks pan 2 0.5221511083334797 ekspan 2.5221511083334796 String String Integer Float String Float

18
docs6/csv-with-and-without-headers.rst
View file

@ -12,7 +12,7 @@ Sometimes we get CSV files which lack a header. For example (`data/headerless.cs
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/headerless.csv
cat data/headerless.csv
John,23,present
Fred,34,present
Alice,56,missing
@ -23,7 +23,7 @@ You can use Miller to add a header. The ``--implicit-csv-header`` applies positi
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --csv --implicit-csv-header cat data/headerless.csv
mlr --csv --implicit-csv-header cat data/headerless.csv
1,2,3
John,23,present
Fred,34,present
@ -35,7 +35,7 @@ Following that, you can rename the positionally indexed labels to names with mea
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --csv --implicit-csv-header label name,age,status data/headerless.csv
mlr --csv --implicit-csv-header label name,age,status data/headerless.csv
name,age,status
John,23,present
Fred,34,present
@ -47,7 +47,7 @@ Likewise, if you need to produce CSV which is lacking its header, you can pipe M
.. code-block:: none
:emphasize-lines: 1-1
$ head -5 data/colored-shapes.dkvp | mlr --ocsv cat
head -5 data/colored-shapes.dkvp | mlr --ocsv cat
color,shape,flag,i,u,v,w,x
yellow,triangle,1,11,0.6321695890307647,0.9887207810889004,0.4364983936735774,5.7981881667050565
red,square,1,15,0.21966833570651523,0.001257332190235938,0.7927778364718627,2.944117399716207
@ -58,7 +58,7 @@ Likewise, if you need to produce CSV which is lacking its header, you can pipe M
.. code-block:: none
:emphasize-lines: 1-1
$ head -5 data/colored-shapes.dkvp | mlr --ocsv --headerless-csv-output cat
head -5 data/colored-shapes.dkvp | mlr --ocsv --headerless-csv-output cat
yellow,triangle,1,11,0.6321695890307647,0.9887207810889004,0.4364983936735774,5.7981881667050565
red,square,1,15,0.21966833570651523,0.001257332190235938,0.7927778364718627,2.944117399716207
red,circle,1,16,0.20901671281497636,0.29005231936593445,0.13810280912907674,5.065034003400998
@ -70,7 +70,7 @@ Lastly, often we say "CSV" or "TSV" when we have positionally indexed data in co
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --inidx --ifs comma --oxtab cut -f 1,3 data/headerless.csv
mlr --inidx --ifs comma --oxtab cut -f 1,3 data/headerless.csv
1 John
3 present
@ -91,7 +91,7 @@ Miller handles compliant CSV: in particular, it's an error if the number of data
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/ragged.csv
cat data/ragged.csv
a,b,c
1,2,3
4,5
@ -100,7 +100,7 @@ Miller handles compliant CSV: in particular, it's an error if the number of data
.. code-block:: none
:emphasize-lines: 1-8
$ mlr --from data/ragged.csv --fs comma --nidx put '
mlr --from data/ragged.csv --fs comma --nidx put '
@maxnf = max(@maxnf, NF);
@nf = NF;
while(@nf < @maxnf) {
@ -118,7 +118,7 @@ or, more simply,
.. code-block:: none
:emphasize-lines: 1-6
$ mlr --from data/ragged.csv --fs comma --nidx put '
mlr --from data/ragged.csv --fs comma --nidx put '
@maxnf = max(@maxnf, NF);
while(NF < @maxnf) {
$[NF+1] = "";

34
docs6/data-diving-examples.rst
View file

@ -14,7 +14,7 @@ Vertical-tabular format is good for a quick look at CSV data layout -- seeing wh
.. code-block:: none
:emphasize-lines: 1-1
$ head -n 2 data/flins.csv | mlr --icsv --oxtab cat
head -n 2 data/flins.csv | mlr --icsv --oxtab cat
county Seminole
tiv_2011 22890.55
tiv_2012 20848.71
@ -25,7 +25,7 @@ A few simple queries:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --from data/flins.csv --icsv --opprint count-distinct -f county | head
mlr --from data/flins.csv --icsv --opprint count-distinct -f county | head
county count
Seminole 1
Miami Dade 2
@ -37,33 +37,33 @@ A few simple queries:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --from data/flins.csv --icsv --opprint count-distinct -f construction,line
mlr --from data/flins.csv --icsv --opprint count-distinct -f construction,line
Categorization of total insured value:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --from data/flins.csv --icsv --opprint stats1 -a min,mean,max -f tiv_2012
mlr --from data/flins.csv --icsv --opprint stats1 -a min,mean,max -f tiv_2012
tiv_2012_min tiv_2012_mean tiv_2012_max
19757.91 1.0615314637499999e+06 2.78555163e+06
.. code-block:: none
:emphasize-lines: 1-2
$ mlr --from data/flins.csv --icsv --opprint \
mlr --from data/flins.csv --icsv --opprint \
stats1 -a min,mean,max -f tiv_2012 -g construction,line
.. code-block:: none
:emphasize-lines: 1-2
$ mlr --from data/flins.csv --icsv --oxtab \
mlr --from data/flins.csv --icsv --oxtab \
stats1 -a p0,p10,p50,p90,p95,p99,p100 -f hu_site_deductible
.. code-block:: none
:emphasize-lines: 1-3
$ mlr --from data/flins.csv --icsv --opprint \
mlr --from data/flins.csv --icsv --opprint \
stats1 -a p95,p99,p100 -f hu_site_deductible -g county \
then sort -f county | head
county
@ -77,7 +77,7 @@ Categorization of total insured value:
.. code-block:: none
:emphasize-lines: 1-2
$ mlr --from data/flins.csv --icsv --oxtab \
mlr --from data/flins.csv --icsv --oxtab \
stats2 -a corr,linreg-ols,r2 -f tiv_2011,tiv_2012
tiv_2011_tiv_2012_corr 0.9353629581411828
tiv_2011_tiv_2012_ols_m 1.0890905877734807
@ -88,7 +88,7 @@ Categorization of total insured value:
.. code-block:: none
:emphasize-lines: 1-2
$ mlr --from data/flins.csv --icsv --opprint \
mlr --from data/flins.csv --icsv --opprint \
stats2 -a corr,linreg-ols,r2 -f tiv_2011,tiv_2012 -g county
county tiv_2011_tiv_2012_corr tiv_2011_tiv_2012_ols_m tiv_2011_tiv_2012_ols_b tiv_2011_tiv_2012_ols_n tiv_2011_tiv_2012_r2
Seminole - - - 1 -
@ -116,13 +116,13 @@ Peek at the data:
.. code-block:: none
:emphasize-lines: 1-1
$ wc -l data/colored-shapes.dkvp
wc -l data/colored-shapes.dkvp
10078 data/colored-shapes.dkvp
.. code-block:: none
:emphasize-lines: 1-1
$ head -n 6 data/colored-shapes.dkvp | mlr --opprint cat
head -n 6 data/colored-shapes.dkvp | mlr --opprint cat
color shape flag i u v w x
yellow triangle 1 11 0.6321695890307647 0.9887207810889004 0.4364983936735774 5.7981881667050565
red square 1 15 0.21966833570651523 0.001257332190235938 0.7927778364718627 2.944117399716207
@ -138,7 +138,7 @@ Here it looks reasonable that ``u`` is unit-uniform; something's up with ``v`` b
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --oxtab stats1 -a min,mean,max -f flag,u,v data/colored-shapes.dkvp | creach 3
mlr --oxtab stats1 -a min,mean,max -f flag,u,v data/colored-shapes.dkvp | creach 3
flag_min 0
flag_mean 0.39888866838658465
flag_max 1
@ -156,7 +156,7 @@ The histogram shows the different distribution of 0/1 flags:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --opprint histogram -f flag,u,v --lo -0.1 --hi 1.1 --nbins 12 data/colored-shapes.dkvp
mlr --opprint histogram -f flag,u,v --lo -0.1 --hi 1.1 --nbins 12 data/colored-shapes.dkvp
bin_lo bin_hi flag_count u_count v_count
-0.010000000000000002 0.09000000000000002 6058 0 36
0.09000000000000002 0.19000000000000003 0 1062 988
@ -176,7 +176,7 @@ Look at univariate stats by color and shape. In particular, color-dependent flag
.. code-block:: none
:emphasize-lines: 1-3
$ mlr --opprint stats1 -a min,mean,max -f flag,u,v -g color \
mlr --opprint stats1 -a min,mean,max -f flag,u,v -g color \
then sort -f color \
data/colored-shapes.dkvp
color flag_min flag_mean flag_max u_min u_mean u_max v_min v_mean v_max
@ -190,7 +190,7 @@ Look at univariate stats by color and shape. In particular, color-dependent flag
.. code-block:: none
:emphasize-lines: 1-3
$ mlr --opprint stats1 -a min,mean,max -f flag,u,v -g shape \
mlr --opprint stats1 -a min,mean,max -f flag,u,v -g shape \
then sort -f shape \
data/colored-shapes.dkvp
shape flag_min flag_mean flag_max u_min u_mean u_max v_min v_mean v_max
@ -203,14 +203,14 @@ Look at bivariate stats by color and shape. In particular, ``u,v`` pairwise corr
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --opprint --right stats2 -a corr -f u,v,w,x data/colored-shapes.dkvp
mlr --opprint --right stats2 -a corr -f u,v,w,x data/colored-shapes.dkvp
u_v_corr w_x_corr
0.13341803768384553 -0.011319938208638764
.. code-block:: none
:emphasize-lines: 1-3
$ mlr --opprint --right \
mlr --opprint --right \
stats2 -a corr -f u,v,w,x -g color,shape then sort -nr u_v_corr \
data/colored-shapes.dkvp
color shape u_v_corr w_x_corr

16
docs6/dates-and-times.rst
View file

@ -12,7 +12,7 @@ Given input like
.. code-block:: none
:emphasize-lines: 1-1
$ cat dates.csv
cat dates.csv
date,event
2018-02-03,initialization
2018-03-07,discovery
@ -23,7 +23,7 @@ we can use ``strptime`` to parse the date field into seconds-since-epoch and the
.. code-block:: none
:emphasize-lines: 1-3
$ mlr --csv filter '
mlr --csv filter '
strptime($date, "%Y-%m-%d") > strptime("2018-03-03", "%Y-%m-%d")
' dates.csv
date,event
@ -39,7 +39,7 @@ Suppose you have some date-stamped data which may (or may not) be missing entrie
.. code-block:: none
:emphasize-lines: 1-1
$ head -n 10 data/miss-date.csv
head -n 10 data/miss-date.csv
date,qoh
2012-03-05,10055
2012-03-06,10486
@ -54,7 +54,7 @@ Suppose you have some date-stamped data which may (or may not) be missing entrie
.. code-block:: none
:emphasize-lines: 1-1
$ wc -l data/miss-date.csv
wc -l data/miss-date.csv
1372 data/miss-date.csv
Since there are 1372 lines in the data file, some automation is called for. To find the missing dates, you can convert the dates to seconds since the epoch using ``strptime``, then compute adjacent differences (the ``cat -n`` simply inserts record-counters):
@ -62,7 +62,7 @@ Since there are 1372 lines in the data file, some automation is called for. To f
.. code-block:: none
:emphasize-lines: 1-5
$ mlr --from data/miss-date.csv --icsv \
mlr --from data/miss-date.csv --icsv \
cat -n \
then put '$datestamp = strptime($date, "%Y-%m-%d")' \
then step -a delta -f datestamp \
@ -83,7 +83,7 @@ Then, filter for adjacent difference not being 86400 (the number of seconds in a
.. code-block:: none
:emphasize-lines: 1-5
$ mlr --from data/miss-date.csv --icsv \
mlr --from data/miss-date.csv --icsv \
cat -n \
then put '$datestamp = strptime($date, "%Y-%m-%d")' \
then step -a delta -f datestamp \
@ -96,7 +96,7 @@ Given this, it's now easy to see where the gaps are:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr cat -n then filter '$n >= 770 && $n <= 780' data/miss-date.csv
mlr cat -n then filter '$n >= 770 && $n <= 780' data/miss-date.csv
n=770,1=2014-04-12,2=129435
n=771,1=2014-04-13,2=129868
n=772,1=2014-04-14,2=129797
@ -112,7 +112,7 @@ Given this, it's now easy to see where the gaps are:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr cat -n then filter '$n >= 1115 && $n <= 1125' data/miss-date.csv
mlr cat -n then filter '$n >= 1115 && $n <= 1125' data/miss-date.csv
n=1115,1=2015-03-25,2=181006
n=1116,1=2015-03-26,2=180995
n=1117,1=2015-03-27,2=181043

8
docs6/feature-comparison.rst
View file

@ -14,7 +14,7 @@ Miller respects CSV headers. If you do ``mlr --csv cat *.csv`` then the header l
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/a.csv
cat data/a.csv
a,b,c
1,2,3
4,5,6
@ -22,14 +22,14 @@ Miller respects CSV headers. If you do ``mlr --csv cat *.csv`` then the header l
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/b.csv
cat data/b.csv
a,b,c
7,8,9
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --csv cat data/a.csv data/b.csv
mlr --csv cat data/a.csv data/b.csv
a,b,c
1,2,3
4,5,6
@ -38,7 +38,7 @@ Miller respects CSV headers. If you do ``mlr --csv cat *.csv`` then the header l
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --csv sort -nr b data/a.csv data/b.csv
mlr --csv sort -nr b data/a.csv data/b.csv
a,b,c
7,8,9
4,5,6

52
docs6/file-formats.rst
View file

@ -14,7 +14,7 @@ Examples
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --usage-data-format-examples
mlr --usage-data-format-examples
CSV/CSV-lite: comma-separated values with separate header line
TSV: same but with tabs in places of commas
+---------------------+
@ -124,7 +124,7 @@ Miller's default file format is DKVP, for **delimited key-value pairs**. Example
.. code-block:: none
:emphasize-lines: 1-1
$ mlr cat data/small
mlr cat data/small
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
@ -180,7 +180,7 @@ Example with index-numbered output:
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/small
cat data/small
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
@ -190,7 +190,7 @@ Example with index-numbered output:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --onidx --ofs ' ' cat data/small
mlr --onidx --ofs ' ' cat data/small
pan pan 1 0.3467901443380824 0.7268028627434533
eks pan 2 0.7586799647899636 0.5221511083334797
wye wye 3 0.20460330576630303 0.33831852551664776
@ -202,7 +202,7 @@ Example with index-numbered input:
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/mydata.txt
cat data/mydata.txt
oh say can you see
by the dawn's
early light
@ -210,7 +210,7 @@ Example with index-numbered input:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --inidx --ifs ' ' --odkvp cat data/mydata.txt
mlr --inidx --ifs ' ' --odkvp cat data/mydata.txt
1=oh,2=say,3=can,4=you,5=see
1=by,2=the,3=dawn's
1=early,2=light
@ -220,7 +220,7 @@ Example with index-numbered input and output:
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/mydata.txt
cat data/mydata.txt
oh say can you see
by the dawn's
early light
@ -228,7 +228,7 @@ Example with index-numbered input and output:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --nidx --fs ' ' --repifs cut -f 2,3 data/mydata.txt
mlr --nidx --fs ' ' --repifs cut -f 2,3 data/mydata.txt
say can
the dawn's
light
@ -252,7 +252,7 @@ An **array of single-level objects** is, quite simply, **a table**:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --json head -n 2 then cut -f color,shape data/json-example-1.json
mlr --json head -n 2 then cut -f color,shape data/json-example-1.json
{
"color": "yellow",
"shape": "triangle"
@ -265,7 +265,7 @@ An **array of single-level objects** is, quite simply, **a table**:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --json --jvstack head -n 2 then cut -f color,u,v data/json-example-1.json
mlr --json --jvstack head -n 2 then cut -f color,u,v data/json-example-1.json
{
"color": "yellow",
"u": 0.6321695890307647,
@ -280,7 +280,7 @@ An **array of single-level objects** is, quite simply, **a table**:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --ijson --opprint stats1 -a mean,stddev,count -f u -g shape data/json-example-1.json
mlr --ijson --opprint stats1 -a mean,stddev,count -f u -g shape data/json-example-1.json
shape u_mean u_stddev u_count
triangle 0.5839952367477192 0.13118354465618046 3
square 0.409355036804889 0.3654281755508655 4
@ -294,7 +294,7 @@ Additionally, Miller can **tabularize nested objects by concatentating keys**:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --json --jvstack head -n 2 data/json-example-2.json
mlr --json --jvstack head -n 2 data/json-example-2.json
{
"flag": 1,
"i": 11,
@ -327,7 +327,7 @@ Additionally, Miller can **tabularize nested objects by concatentating keys**:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --ijson --opprint head -n 4 data/json-example-2.json
mlr --ijson --opprint head -n 4 data/json-example-2.json
flag i attributes.color attributes.shape values.u values.v values.w values.x
1 11 yellow triangle 0.632170 0.988721 0.436498 5.798188
1 15 red square 0.219668 0.001257 0.792778 2.944117
@ -339,7 +339,7 @@ Note in particular that as far as Miller's ``put`` and ``filter``, as well as ot
.. code-block:: none
:emphasize-lines: 1-3
$ mlr --json --jvstack head -n 1 \
mlr --json --jvstack head -n 1 \
then put '${values:uv} = ${values:u} * ${values:v}' \
data/json-example-2.json
{
@ -367,7 +367,7 @@ Suppose we have arrays like this in our input data:
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/json-example-3.json
cat data/json-example-3.json
{
"label": "orange",
"values": [12.2, 13.8, 17.2]
@ -382,7 +382,7 @@ Then integer indices (starting from 0 and counting up) are used as map keys:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --ijson --oxtab cat data/json-example-3.json
mlr --ijson --oxtab cat data/json-example-3.json
label orange
values.1 12.2
values.2 13.8
@ -397,7 +397,7 @@ When the data are written back out as JSON, field names are re-expanded as above
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --json --jvstack cat data/json-example-3.json
mlr --json --jvstack cat data/json-example-3.json
{
"label": "orange",
"values": [12.2, 13.8, 17.2]
@ -440,7 +440,7 @@ Miller's pretty-print format is like CSV, but column-aligned. For example, comp
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --ocsv cat data/small
mlr --ocsv cat data/small
a,b,i,x,y
pan,pan,1,0.3467901443380824,0.7268028627434533
eks,pan,2,0.7586799647899636,0.5221511083334797
@ -451,7 +451,7 @@ Miller's pretty-print format is like CSV, but column-aligned. For example, comp
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --opprint cat data/small
mlr --opprint cat data/small
a b i x y
pan pan 1 0.3467901443380824 0.7268028627434533
eks pan 2 0.7586799647899636 0.5221511083334797
@ -468,7 +468,7 @@ For output only (this isn't supported in the input-scanner as of 5.0.0) you can
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --opprint --barred cat data/small
mlr --opprint --barred cat data/small
+-----+-----+---+---------------------+---------------------+
| a | b | i | x | y |
+-----+-----+---+---------------------+---------------------+
@ -550,7 +550,7 @@ Markdown format looks like this:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --omd cat data/small
mlr --omd cat data/small
| a | b | i | x | y |
| --- | --- | --- | --- | --- |
| pan | pan | 1 | 0.3467901443380824 | 0.7268028627434533 |
@ -573,7 +573,7 @@ While you can do format conversion using ``mlr --icsv --ojson cat myfile.csv``,
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --usage-format-conversion-keystroke-saver-options
mlr --usage-format-conversion-keystroke-saver-options
As keystroke-savers for format-conversion you may use the following:
--c2t --c2d --c2n --c2j --c2x --c2p --c2m
--t2c --t2d --t2n --t2j --t2x --t2p --t2m
@ -609,7 +609,7 @@ You can include comments within your data files, and either have them ignored, o
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --usage-comments-in-data
mlr --usage-comments-in-data
--skip-comments Ignore commented lines (prefixed by "#")
within the input.
--skip-comments-with {string} Ignore commented lines within input, with
@ -632,7 +632,7 @@ Examples:
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/budget.csv
cat data/budget.csv
# Asana -- here are the budget figures you asked for!
type,quantity
purple,456.78
@ -642,7 +642,7 @@ Examples:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --skip-comments --icsv --opprint sort -nr quantity data/budget.csv
mlr --skip-comments --icsv --opprint sort -nr quantity data/budget.csv
type quantity
green 678.12
purple 456.78
@ -651,7 +651,7 @@ Examples:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --pass-comments --icsv --opprint sort -nr quantity data/budget.csv
mlr --pass-comments --icsv --opprint sort -nr quantity data/budget.csv
# Asana -- here are the budget figures you asked for!
type quantity
green 678.12

9
docs6/genrst-filter
View file

@ -92,16 +92,14 @@ def run_command(cmd_lines, output_handle)
if status != 0
raise "\"#{cmd}\" exited with non-zero code #{status}."
end
cmd_line_1 = cmd_lines.shift
write_card(['$ '+cmd_line_1] + cmd_lines, cmd_output.split(/\n/), output_handle)
write_card(cmd_lines, cmd_output.split(/\n/), output_handle)
end
# ----------------------------------------------------------------
def run_command_tolerating_error(cmd_lines, output_handle)
cmd = cmd_lines.join("\n")
cmd_output = `#{cmd} 2>&1`
cmd_line_1 = cmd_lines.shift
write_card(['$ '+cmd_line_1] + cmd_lines, cmd_output.split(/\n/), output_handle)
write_card(cmd_lines, cmd_output.split(/\n/), output_handle)
end
# ----------------------------------------------------------------
@ -112,8 +110,7 @@ end
# ----------------------------------------------------------------
def show_command(cmd_lines, output_handle)
# The command can be multi-line
cmd_line_1 = cmd_lines.shift
write_card(['$ '+cmd_line_1] + cmd_lines, [], output_handle)
write_card(cmd_lines, [], output_handle)
end
# ----------------------------------------------------------------

22
docs6/joins.rst
View file

@ -14,7 +14,7 @@ For example, the right file here has nine records, and the left file should add
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --icsvlite --opprint cat data/join-u-left.csv
mlr --icsvlite --opprint cat data/join-u-left.csv
hostname ipaddr
nadir.east.our.org 10.3.1.18
zenith.west.our.org 10.3.1.27
@ -23,7 +23,7 @@ For example, the right file here has nine records, and the left file should add
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --icsvlite --opprint cat data/join-u-right.csv
mlr --icsvlite --opprint cat data/join-u-right.csv
ipaddr timestamp bytes
10.3.1.27 1448762579 4568
10.3.1.18 1448762578 8729
@ -38,7 +38,7 @@ For example, the right file here has nine records, and the left file should add
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --icsvlite --opprint join -s -j ipaddr -f data/join-u-left.csv data/join-u-right.csv
mlr --icsvlite --opprint join -s -j ipaddr -f data/join-u-left.csv data/join-u-right.csv
ipaddr hostname timestamp bytes
10.3.1.27 zenith.west.our.org 1448762579 4568
10.4.5.94 apoapsis.east.our.org 1448762579 17445
@ -52,7 +52,7 @@ The solution (besides pre-sorting the input files on the join keys) is to simply
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --icsvlite --opprint join -u -j ipaddr -f data/join-u-left.csv data/join-u-right.csv
mlr --icsvlite --opprint join -u -j ipaddr -f data/join-u-left.csv data/join-u-right.csv
ipaddr hostname timestamp bytes
10.3.1.27 zenith.west.our.org 1448762579 4568
10.3.1.18 nadir.east.our.org 1448762578 8729
@ -89,7 +89,7 @@ Joining on color the results are as expected:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --csv join -j id -f data/color-codes.csv data/color-names.csv
mlr --csv join -j id -f data/color-codes.csv data/color-names.csv
id,code,color
4,ff0000,red
2,00ff00,green
@ -99,7 +99,7 @@ However, if we ask for left-unpaireds, since there's no ``color`` column, we get
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --csv join --ul -j id -f data/color-codes.csv data/color-names.csv
mlr --csv join --ul -j id -f data/color-codes.csv data/color-names.csv
id,code,color
4,ff0000,red
2,00ff00,green
@ -112,7 +112,7 @@ To fix this, we can use **unsparsify**:
.. code-block:: none
:emphasize-lines: 1-3
$ mlr --csv join --ul -j id -f data/color-codes.csv \
mlr --csv join --ul -j id -f data/color-codes.csv \
then unsparsify --fill-with "" \
data/color-names.csv
id,code,color
@ -130,7 +130,7 @@ Suppose we have the following data:
.. code-block:: none
:emphasize-lines: 1-1
$ cat multi-join/input.csv
cat multi-join/input.csv
id,task
10,chop
20,puree
@ -146,7 +146,7 @@ And we want to augment the ``id`` column with lookups from the following data fi
.. code-block:: none
:emphasize-lines: 1-1
$ cat multi-join/name-lookup.csv
cat multi-join/name-lookup.csv
id,name
30,Alice
10,Bob
@ -155,7 +155,7 @@ And we want to augment the ``id`` column with lookups from the following data fi
.. code-block:: none
:emphasize-lines: 1-1
$ cat multi-join/status-lookup.csv
cat multi-join/status-lookup.csv
id,status
30,occupied
10,idle
@ -166,7 +166,7 @@ We can run the input file through multiple ``join`` commands in a ``then``-chain
.. code-block:: none
:emphasize-lines: 1-3
$ mlr --icsv --opprint join -f multi-join/name-lookup.csv -j id \
mlr --icsv --opprint join -f multi-join/name-lookup.csv -j id \
then join -f multi-join/status-lookup.csv -j id \
multi-join/input.csv
id status name task

10
docs6/keystroke-savers.rst
View file

@ -12,7 +12,7 @@ In our examples so far we've often made use of ``mlr --icsv --opprint`` or ``mlr
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --c2p head -n 2 example.csv
mlr --c2p head -n 2 example.csv
color shape flag index quantity rate
yellow triangle true 11 43.6498 9.8870
red square true 15 79.2778 0.0130
@ -20,7 +20,7 @@ In our examples so far we've often made use of ``mlr --icsv --opprint`` or ``mlr
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --c2j head -n 2 example.csv
mlr --c2j head -n 2 example.csv
{
"color": "yellow",
"shape": "triangle",
@ -48,7 +48,7 @@ Already we saw that you can put the filename first using ``--from``. When you're
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --c2p --from example.csv sort -nr index then head -n 3
mlr --c2p --from example.csv sort -nr index then head -n 3
color shape flag index quantity rate
purple square false 91 72.3735 8.2430
yellow circle true 87 63.5058 8.3350
@ -57,7 +57,7 @@ Already we saw that you can put the filename first using ``--from``. When you're
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --c2p --from example.csv sort -nr index then head -n 3 then cut -f shape,quantity
mlr --c2p --from example.csv sort -nr index then head -n 3 then cut -f shape,quantity
shape quantity
square 72.3735
circle 63.5058
@ -68,7 +68,7 @@ If there's more than one input file, you can use ``--mfrom``, then however many
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --c2p --mfrom data/*.csv -- sort -n index
mlr --c2p --mfrom data/*.csv -- sort -n index
.mlrrc file
^^^^^^^^^^^

10
docs6/log-processing-examples.rst
View file

@ -15,7 +15,7 @@ Suppose your program has printed something like this (`log.txt <./log.txt>`_):
.. code-block:: none
:emphasize-lines: 1-1
$ cat log.txt
cat log.txt
op=enter,time=1472819681
op=cache,type=A9,hit=0
op=cache,type=A4,hit=1
@ -65,7 +65,7 @@ Each print statement simply contains local information: the current timestamp, w
.. code-block:: none
:emphasize-lines: 1-2
$ grep op=cache log.txt \
grep op=cache log.txt \
| mlr --idkvp --opprint stats1 -a mean -f hit -g type then sort -f type
type hit_mean
A1 0.8571428571428571
@ -75,7 +75,7 @@ Each print statement simply contains local information: the current timestamp, w
.. code-block:: none
:emphasize-lines: 1-4
$ mlr --from log.txt --opprint \
mlr --from log.txt --opprint \
filter 'is_present($batch_size)' \
then step -a delta -f time,num_filtered \
then sec2gmt time
@ -92,7 +92,7 @@ Alternatively, we can simply group the similar data for a better look:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --opprint group-like log.txt
mlr --opprint group-like log.txt
op time
enter 1472819681
@ -145,7 +145,7 @@ Alternatively, we can simply group the similar data for a better look:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --opprint group-like then sec2gmt time log.txt
mlr --opprint group-like then sec2gmt time log.txt
op time
enter 2016-09-02T12:34:41Z

4
docs6/new-in-miller-6.rst
View file

@ -89,7 +89,7 @@ For example (see https://github.com/johnkerl/miller/issues/178) you can now do
.. code-block:: none
:emphasize-lines: 1-1
$ echo '{ "a": "0123" }' | mlr --json cat
echo '{ "a": "0123" }' | mlr --json cat
{
"a": "0123"
}
@ -97,7 +97,7 @@ For example (see https://github.com/johnkerl/miller/issues/178) you can now do
.. code-block:: none
:emphasize-lines: 1-1
$ echo '{ "x": 1.230, "y": 1.230000000 }' | mlr --json cat
echo '{ "x": 1.230, "y": 1.230000000 }' | mlr --json cat
{
"x": 1.230,
"y": 1.230000000

20
docs6/operating-on-all-fields.rst
View file

@ -12,7 +12,7 @@ Suppose you want to replace spaces with underscores in your column names:
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/spaces.csv
cat data/spaces.csv
a b c,def,g h i
123,4567,890
2468,1357,3579
@ -23,7 +23,7 @@ The simplest way is to use ``mlr rename`` with ``-g`` (for global replace, not j
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --csv rename -g -r ' ,_' data/spaces.csv
mlr --csv rename -g -r ' ,_' data/spaces.csv
a_b_c,def,g_h_i
123,4567,890
2468,1357,3579
@ -32,7 +32,7 @@ The simplest way is to use ``mlr rename`` with ``-g`` (for global replace, not j
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --csv --opprint rename -g -r ' ,_' data/spaces.csv
mlr --csv --opprint rename -g -r ' ,_' data/spaces.csv
a_b_c def g_h_i
123 4567 890
2468 1357 3579
@ -43,7 +43,7 @@ You can also do this with a for-loop:
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/bulk-rename-for-loop.mlr
cat data/bulk-rename-for-loop.mlr
map newrec = {};
for (oldk, v in $*) {
newrec[gsub(oldk, " ", "_")] = v;
@ -53,7 +53,7 @@ You can also do this with a for-loop:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --icsv --opprint put -f data/bulk-rename-for-loop.mlr data/spaces.csv
mlr --icsv --opprint put -f data/bulk-rename-for-loop.mlr data/spaces.csv
a_b_c def g_h_i
123 4567 890
2468 1357 3579
@ -67,7 +67,7 @@ How to do ``$name = gsub($name, "old", "new")`` for all fields?
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/sar.csv
cat data/sar.csv
a,b,c
the quick,brown fox,jumped
over,the,lazy dogs
@ -75,7 +75,7 @@ How to do ``$name = gsub($name, "old", "new")`` for all fields?
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/sar.mlr
cat data/sar.mlr
for (k in $*) {
$[k] = gsub($[k], "e", "X");
}
@ -83,7 +83,7 @@ How to do ``$name = gsub($name, "old", "new")`` for all fields?
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --csv put -f data/sar.mlr data/sar.csv
mlr --csv put -f data/sar.mlr data/sar.csv
a,b,c
thX quick,brown fox,jumpXd
ovXr,thX,lazy dogs
@ -96,7 +96,7 @@ Using Miller 5.0.0's map literals and assigning to ``$*``, you can fully general
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/small
cat data/small
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
@ -106,7 +106,7 @@ Using Miller 5.0.0's map literals and assigning to ``$*``, you can fully general
.. code-block:: none
:emphasize-lines: 1-15
$ mlr put '
mlr put '
begin {
@i_cumu = 0;
}

40
docs6/programming-language.rst
View file

@ -18,7 +18,7 @@ Let's keep using the sample `example.csv <./example.csv>`_. When we type
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --c2p put '$cost = $quantity * $rate' example.csv
mlr --c2p put '$cost = $quantity * $rate' example.csv
color shape flag index quantity rate cost
yellow triangle true 11 43.6498 9.8870 431.5655726
red square true 15 79.2778 0.0130 1.0306114
@ -46,7 +46,7 @@ You can use more than one statement, separating them with semicolons, and option
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --c2p put '$cost = $quantity * $rate; $index = $index * 100' example.csv
mlr --c2p put '$cost = $quantity * $rate; $index = $index * 100' example.csv
color shape flag index quantity rate cost
yellow triangle true 1100 43.6498 9.8870 431.5655726
red square true 1500 79.2778 0.0130 1.0306114
@ -62,7 +62,7 @@ You can use more than one statement, separating them with semicolons, and option
.. code-block:: none
:emphasize-lines: 1-4
$ mlr --c2p put '
mlr --c2p put '
$cost = $quantity * $rate;
$index *= 100
' example.csv
@ -83,14 +83,14 @@ One of Miller's key features is the ability to express data-transformation right
.. code-block:: none
:emphasize-lines: 1-1
$ cat dsl-example.mlr
cat dsl-example.mlr
$cost = $quantity * $rate;
$index *= 100
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --c2p put -f dsl-example.mlr example.csv
mlr --c2p put -f dsl-example.mlr example.csv
color shape flag index quantity rate cost
yellow triangle true 1100 43.6498 9.8870 431.5655726
red square true 1500 79.2778 0.0130 1.0306114
@ -117,7 +117,7 @@ To make ``begin`` and ``end`` statements useful, we need somewhere to put things
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --c2p --from example.csv put 'begin { @sum = 0 } @sum += $quantity; end {emit @sum}'
mlr --c2p --from example.csv put 'begin { @sum = 0 } @sum += $quantity; end {emit @sum}'
color shape flag index quantity rate
yellow triangle true 11 43.6498 9.8870
red square true 15 79.2778 0.0130
@ -138,14 +138,14 @@ If you want the end-block output to be the only output, and not include the inpu
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --c2p --from example.csv put -q 'begin { @sum = 0 } @sum += $quantity; end {emit @sum}'
mlr --c2p --from example.csv put -q 'begin { @sum = 0 } @sum += $quantity; end {emit @sum}'
sum
652.7185
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --c2j --from example.csv put -q 'begin { @sum = 0 } @sum += $quantity; end {emit @sum}'
mlr --c2j --from example.csv put -q 'begin { @sum = 0 } @sum += $quantity; end {emit @sum}'
{
"sum": 652.7185
}
@ -153,7 +153,7 @@ If you want the end-block output to be the only output, and not include the inpu
.. code-block:: none
:emphasize-lines: 1-6
$ mlr --c2j --from example.csv put -q '
mlr --c2j --from example.csv put -q '
begin { @count = 0; @sum = 0 }
@count += 1;
@sum += $quantity;
@ -180,7 +180,7 @@ Also inspired by `AWK <https://en.wikipedia.org/wiki/AWK>`_, the Miller DSL has
.. code-block:: none
:emphasize-lines: 1-1
$ cat context-example.mlr
cat context-example.mlr
$nf = NF;
$nr = NR;
$fnr = FNR;
@ -191,7 +191,7 @@ Also inspired by `AWK <https://en.wikipedia.org/wiki/AWK>`_, the Miller DSL has
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --c2p put -f context-example.mlr data/a.csv data/b.csv
mlr --c2p put -f context-example.mlr data/a.csv data/b.csv
a b c nf nr fnr filename filenum newnf
1 2 3 3 1 1 data/a.csv 1 8
4 5 6 3 2 2 data/a.csv 1 8
@ -205,7 +205,7 @@ You can define your own functions:
.. code-block:: none
:emphasize-lines: 1-1
$ cat factorial-example.mlr
cat factorial-example.mlr
func factorial(n) {
if (n <= 1) {
return n
@ -217,7 +217,7 @@ You can define your own functions:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --c2p --from example.csv put -f factorial-example.mlr -e '$fact = factorial(NR)'
mlr --c2p --from example.csv put -f factorial-example.mlr -e '$fact = factorial(NR)'
color shape flag index quantity rate fact
yellow triangle true 11 43.6498 9.8870 1
red square true 15 79.2778 0.0130 2
@ -244,7 +244,7 @@ Suppose you want to only compute sums conditionally -- you can use an ``if`` sta
.. code-block:: none
:emphasize-lines: 1-1
$ cat if-example.mlr
cat if-example.mlr
begin {
@count_of_red = 0;
@sum_of_red = 0
@ -262,7 +262,7 @@ Suppose you want to only compute sums conditionally -- you can use an ``if`` sta
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --c2p --from example.csv put -q -f if-example.mlr
mlr --c2p --from example.csv put -q -f if-example.mlr
count_of_red sum_of_red
4 247.84139999999996
@ -278,7 +278,7 @@ current record as a hashmap:
.. code-block:: none
:emphasize-lines: 1-1
$ cat for-example.mlr
cat for-example.mlr
for (k, v in $*) {
print "KEY IS ". k . " VALUE IS ". v;
}
@ -287,7 +287,7 @@ current record as a hashmap:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --csv cat data/a.csv
mlr --csv cat data/a.csv
a,b,c
1,2,3
4,5,6
@ -295,7 +295,7 @@ current record as a hashmap:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --csv --from data/a.csv put -qf for-example.mlr
mlr --csv --from data/a.csv put -qf for-example.mlr
KEY IS a VALUE IS 1
KEY IS b VALUE IS 2
KEY IS c VALUE IS 3
@ -337,7 +337,7 @@ For example, you can sum up all the ``$a`` values across records without having
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --json cat absent-example.json
mlr --json cat absent-example.json
{
"a": 1,
"b": 2
@ -353,7 +353,7 @@ For example, you can sum up all the ``$a`` values across records without having
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --json put '@sum_of_a += $a; end {emit @sum_of_a}' absent-example.json
mlr --json put '@sum_of_a += $a; end {emit @sum_of_a}' absent-example.json
{
"a": 1,
"b": 2

30
docs6/quick-examples.rst.in
View file

@ -6,70 +6,70 @@ Column select:
.. code-block:: none
:emphasize-lines: 1,1
% mlr --csv cut -f hostname,uptime mydata.csv
mlr --csv cut -f hostname,uptime mydata.csv
Add new columns as function of other columns:
.. code-block:: none
:emphasize-lines: 1,1
% mlr --nidx put '$sum = $7 < 0.0 ? 3.5 : $7 + 2.1*$8' *.dat
mlr --nidx put '$sum = $7 < 0.0 ? 3.5 : $7 + 2.1*$8' *.dat
Row filter:
.. code-block:: none
:emphasize-lines: 1,1
% mlr --csv filter '$status != "down" && $upsec >= 10000' *.csv
mlr --csv filter '$status != "down" && $upsec >= 10000' *.csv
Apply column labels and pretty-print:
.. code-block:: none
:emphasize-lines: 1,1
% grep -v '^#' /etc/group | mlr --ifs : --nidx --opprint label group,pass,gid,member then sort -f group
grep -v '^#' /etc/group | mlr --ifs : --nidx --opprint label group,pass,gid,member then sort -f group
Join multiple data sources on key columns:
.. code-block:: none
:emphasize-lines: 1,1
% mlr join -j account_id -f accounts.dat then group-by account_name balances.dat
mlr join -j account_id -f accounts.dat then group-by account_name balances.dat
Mulltiple formats including JSON:
.. code-block:: none
:emphasize-lines: 1,1
% mlr --json put '$attr = sub($attr, "([0-9]+)_([0-9]+)_.*", "\1:\2")' data/*.json
mlr --json put '$attr = sub($attr, "([0-9]+)_([0-9]+)_.*", "\1:\2")' data/*.json
Aggregate per-column statistics:
.. code-block:: none
:emphasize-lines: 1,1
% mlr stats1 -a min,mean,max,p10,p50,p90 -f flag,u,v data/*
mlr stats1 -a min,mean,max,p10,p50,p90 -f flag,u,v data/*
Linear regression:
.. code-block:: none
:emphasize-lines: 1,1
% mlr stats2 -a linreg-pca -f u,v -g shape data/*
mlr stats2 -a linreg-pca -f u,v -g shape data/*
Aggregate custom per-column statistics:
.. code-block:: none
:emphasize-lines: 1,1
% mlr put -q '@sum[$a][$b] += $x; end {emit @sum, "a", "b"}' data/*
mlr put -q '@sum[$a][$b] += $x; end {emit @sum, "a", "b"}' data/*
Iterate over data using DSL expressions:
.. code-block:: none
:emphasize-lines: 1,1
% mlr --from estimates.tbl put '
mlr --from estimates.tbl put '
for (k,v in $*) {
if (is_numeric(v) && k =~ "^[t-z].*$") {
$sum += v; $count += 1
@ -83,32 +83,32 @@ Run DSL expressions from a script file:
.. code-block:: none
:emphasize-lines: 1,1
% mlr --from infile.dat put -f analyze.mlr
mlr --from infile.dat put -f analyze.mlr
Split/reduce output to multiple filenames:
.. code-block:: none
:emphasize-lines: 1,1
% mlr --from infile.dat put 'tee > "./taps/data-".$a."-".$b, $*'
mlr --from infile.dat put 'tee > "./taps/data-".$a."-".$b, $*'
Compressed I/O:
.. code-block:: none
:emphasize-lines: 1,1
% mlr --from infile.dat put 'tee | "gzip > ./taps/data-".$a."-".$b.".gz", $*'
mlr --from infile.dat put 'tee | "gzip > ./taps/data-".$a."-".$b.".gz", $*'
Interoperate with other data-processing tools using standard pipes:
.. code-block:: none
:emphasize-lines: 1,1
% mlr --from infile.dat put -q '@v=$*; dump | "jq .[]"'
mlr --from infile.dat put -q '@v=$*; dump | "jq .[]"'
Tap/trace:
.. code-block:: none
:emphasize-lines: 1,1
% mlr --from infile.dat put '(NR % 1000 == 0) { print > stderr, "Checkpoint ".NR}'
mlr --from infile.dat put '(NR % 1000 == 0) { print > stderr, "Checkpoint ".NR}'

26
docs6/record-heterogeneity.rst
View file

@ -19,7 +19,7 @@ Miller simply prints a newline and a new header when there is a schema change. W
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/het.dkvp
cat data/het.dkvp
resource=/path/to/file,loadsec=0.45,ok=true
record_count=100,resource=/path/to/file
resource=/path/to/second/file,loadsec=0.32,ok=true
@ -29,7 +29,7 @@ Miller simply prints a newline and a new header when there is a schema change. W
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --ocsvlite cat data/het.dkvp
mlr --ocsvlite cat data/het.dkvp
resource,loadsec,ok
/path/to/file,0.45,true
@ -48,7 +48,7 @@ Miller simply prints a newline and a new header when there is a schema change. W
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --opprint cat data/het.dkvp
mlr --opprint cat data/het.dkvp
resource loadsec ok
/path/to/file 0.45 true
@ -69,7 +69,7 @@ Miller handles explicit header changes as just shown. If your CSV input contains
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/ragged.csv
cat data/ragged.csv
a,b,c
1,2,3
4,5
@ -78,7 +78,7 @@ Miller handles explicit header changes as just shown. If your CSV input contains
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --icsv --oxtab --allow-ragged-csv-input cat data/ragged.csv
mlr --icsv --oxtab --allow-ragged-csv-input cat data/ragged.csv
a 1
b 2
c 3
@ -97,7 +97,7 @@ You may also find Miller's ``group-like`` feature handy (see also :doc:`referenc
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --ocsvlite group-like data/het.dkvp
mlr --ocsvlite group-like data/het.dkvp
resource,loadsec,ok
/path/to/file,0.45,true
/path/to/second/file,0.32,true
@ -110,7 +110,7 @@ You may also find Miller's ``group-like`` feature handy (see also :doc:`referenc
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --opprint group-like data/het.dkvp
mlr --opprint group-like data/het.dkvp
resource loadsec ok
/path/to/file 0.45 true
/path/to/second/file 0.32 true
@ -128,7 +128,7 @@ For these formats, record-heterogeneity comes naturally:
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/het.dkvp
cat data/het.dkvp
resource=/path/to/file,loadsec=0.45,ok=true
record_count=100,resource=/path/to/file
resource=/path/to/second/file,loadsec=0.32,ok=true
@ -138,7 +138,7 @@ For these formats, record-heterogeneity comes naturally:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --onidx --ofs ' ' cat data/het.dkvp
mlr --onidx --ofs ' ' cat data/het.dkvp
/path/to/file 0.45 true
100 /path/to/file
/path/to/second/file 0.32 true
@ -148,7 +148,7 @@ For these formats, record-heterogeneity comes naturally:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --oxtab cat data/het.dkvp
mlr --oxtab cat data/het.dkvp
resource /path/to/file
loadsec 0.45
ok true
@ -170,7 +170,7 @@ For these formats, record-heterogeneity comes naturally:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --oxtab group-like data/het.dkvp
mlr --oxtab group-like data/het.dkvp
resource /path/to/file
loadsec 0.45
ok true
@ -197,7 +197,7 @@ Miller operates on specified fields and takes the rest along: for example, if yo
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/sort-het.dkvp
cat data/sort-het.dkvp
count=500,color=green
count=600
status=ok,count=250,hours=0.22
@ -209,7 +209,7 @@ Miller operates on specified fields and takes the rest along: for example, if yo
.. code-block:: none
:emphasize-lines: 1-1
$ mlr sort -n count data/sort-het.dkvp
mlr sort -n count data/sort-het.dkvp
count=100,color=green
status=ok,count=200,hours=3.4
status=ok,count=250,hours=0.22

2
docs6/reference-dsl-arrays.rst
View file

@ -9,7 +9,7 @@ TODO
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --json cat data/array-example.json
mlr --json cat data/array-example.json
{
"key": "ax04",
"samples": [45, 67, 22]

52
docs6/reference-dsl-control-structures.rst
View file

@ -12,7 +12,7 @@ These are reminiscent of ``awk`` syntax. They can be used to allow assignments
.. code-block:: none
:emphasize-lines: 1-1
$ mlr cat data/put-gating-example-1.dkvp
mlr cat data/put-gating-example-1.dkvp
x=-1
x=0
x=1
@ -22,7 +22,7 @@ These are reminiscent of ``awk`` syntax. They can be used to allow assignments
.. code-block:: none
:emphasize-lines: 1-1
$ mlr put '$x > 0.0 { $y = log10($x); $z = sqrt($y) }' data/put-gating-example-1.dkvp
mlr put '$x > 0.0 { $y = log10($x); $z = sqrt($y) }' data/put-gating-example-1.dkvp
x=-1
x=0
x=1,y=0,z=0
@ -32,7 +32,7 @@ These are reminiscent of ``awk`` syntax. They can be used to allow assignments
.. code-block:: none
:emphasize-lines: 1-1
$ mlr cat data/put-gating-example-2.dkvp
mlr cat data/put-gating-example-2.dkvp
a=abc_123
a=some other name
a=xyz_789
@ -40,7 +40,7 @@ These are reminiscent of ``awk`` syntax. They can be used to allow assignments
.. code-block:: none
:emphasize-lines: 1-5
$ mlr put '
mlr put '
$a =~ "([a-z]+)_([0-9]+)" {
$b = "left_\1"; $c = "right_\2"
}' \
@ -54,7 +54,7 @@ This produces heteregenous output which Miller, of course, has no problems with
.. code-block:: none
:emphasize-lines: 1-1
$ mlr put '$x > 0.0; $y = log10($x); $z = sqrt($y)' data/put-gating-example-1.dkvp
mlr put '$x > 0.0; $y = log10($x); $z = sqrt($y)' data/put-gating-example-1.dkvp
x=1,y=0,z=0
x=2,y=0.3010299956639812,z=0.5486620049392715
x=3,y=0.4771212547196624,z=0.6907396432228734
@ -62,7 +62,7 @@ This produces heteregenous output which Miller, of course, has no problems with
.. code-block:: none
:emphasize-lines: 1-5
$ mlr put '
mlr put '
$a =~ "([a-z]+)_([0-9]+)";
$b = "left_\1";
$c = "right_\2"
@ -107,7 +107,7 @@ Miller's ``while`` and ``do-while`` are unsurprising in comparison to various la
.. code-block:: none
:emphasize-lines: 1-6
$ echo x=1,y=2 | mlr put '
echo x=1,y=2 | mlr put '
while (NF < 10) {
$[NF+1] = ""
}
@ -118,7 +118,7 @@ Miller's ``while`` and ``do-while`` are unsurprising in comparison to various la
.. code-block:: none
:emphasize-lines: 1-9
$ echo x=1,y=2 | mlr put '
echo x=1,y=2 | mlr put '
do {
$[NF+1] = "";
if (NF == 5) {
@ -147,7 +147,7 @@ The ``key`` variable is always bound to the *key* of key-value pairs:
.. code-block:: none
:emphasize-lines: 1-8
$ mlr --from data/small put '
mlr --from data/small put '
print "NR = ".NR;
for (key in $*) {
value = $[key];
@ -194,7 +194,7 @@ The ``key`` variable is always bound to the *key* of key-value pairs:
.. code-block:: none
:emphasize-lines: 1-8
$ mlr -n put '
mlr -n put '
end {
o = {1:2, 3:{4:5}};
for (key in o) {
@ -215,7 +215,7 @@ Single-level keys may be gotten at using either ``for(k,v)`` or ``for((k),v)``;
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/for-srec-example.tbl
cat data/for-srec-example.tbl
label1 label2 f1 f2 f3
blue green 100 240 350
red green 120 11 195
@ -224,7 +224,7 @@ Single-level keys may be gotten at using either ``for(k,v)`` or ``for((k),v)``;
.. code-block:: none
:emphasize-lines: 1-11
$ mlr --pprint --from data/for-srec-example.tbl put '
mlr --pprint --from data/for-srec-example.tbl put '
$sum1 = $f1 + $f2 + $f3;
$sum2 = 0;
$sum3 = 0;
@ -243,7 +243,7 @@ Single-level keys may be gotten at using either ``for(k,v)`` or ``for((k),v)``;
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --from data/small --opprint put 'for (k,v in $*) { $[k."_type"] = typeof(v) }'
mlr --from data/small --opprint put 'for (k,v in $*) { $[k."_type"] = typeof(v) }'
a b i x y a_type b_type i_type x_type y_type
pan pan 1 0.3467901443380824 0.7268028627434533 string string int float float
eks pan 2 0.7586799647899636 0.5221511083334797 string string int float float
@ -258,7 +258,7 @@ Important note: to avoid inconsistent looping behavior in case you're setting ne
.. code-block:: none
:emphasize-lines: 1-10
$ mlr --from data/small --opprint put '
mlr --from data/small --opprint put '
$sum1 = 0;
$sum2 = 0;
for (k,v in $*) {
@ -280,7 +280,7 @@ It can be confusing to modify the stream record while iterating over a copy of i
.. code-block:: none
:emphasize-lines: 1-9
$ mlr --from data/small --opprint put '
mlr --from data/small --opprint put '
sum = 0;
for (k,v in $*) {
if (is_numeric(v)) {
@ -314,7 +314,7 @@ That's confusing in the abstract, so a concrete example is in order. Suppose the
.. code-block:: none
:emphasize-lines: 1-10
$ mlr -n put --jknquoteint -q '
mlr -n put --jknquoteint -q '
begin {
@myvar = {
1: 2,
@ -343,7 +343,7 @@ Then we can get at various values as follows:
.. code-block:: none
:emphasize-lines: 1-16
$ mlr -n put --jknquoteint -q '
mlr -n put --jknquoteint -q '
begin {
@myvar = {
1: 2,
@ -366,7 +366,7 @@ Then we can get at various values as follows:
.. code-block:: none
:emphasize-lines: 1-17
$ mlr -n put --jknquoteint -q '
mlr -n put --jknquoteint -q '
begin {
@myvar = {
1: 2,
@ -389,7 +389,7 @@ Then we can get at various values as follows:
.. code-block:: none
:emphasize-lines: 1-17
$ mlr -n put --jknquoteint -q '
mlr -n put --jknquoteint -q '
begin {
@myvar = {
1: 2,
@ -416,7 +416,7 @@ These are supported as follows:
.. code-block:: none
:emphasize-lines: 1-7
$ mlr --from data/small --opprint put '
mlr --from data/small --opprint put '
num suma = 0;
for (a = 1; a <= NR; a += 1) {
suma += a;
@ -433,7 +433,7 @@ These are supported as follows:
.. code-block:: none
:emphasize-lines: 1-10
$ mlr --from data/small --opprint put '
mlr --from data/small --opprint put '
num suma = 0;
num sumb = 0;
for (num a = 1, num b = 1; a <= NR; a += 1, b *= 2) {
@ -470,7 +470,7 @@ Miller supports an ``awk``-like ``begin/end`` syntax. The statements in the ``b
.. code-block:: none
:emphasize-lines: 1-5
$ mlr put '
mlr put '
begin { @sum = 0 };
@x_sum += $x;
end { emit @x_sum }
@ -492,7 +492,7 @@ Since uninitialized out-of-stream variables default to 0 for addition/substracti
.. code-block:: none
:emphasize-lines: 1-4
$ mlr put '
mlr put '
@x_sum += $x;
end { emit @x_sum }
' ../data/small
@ -513,7 +513,7 @@ The **put -q** option is a shorthand which suppresses printing of each output re
.. code-block:: none
:emphasize-lines: 1-4
$ mlr put -q '
mlr put -q '
@x_sum += $x;
end { emit @x_sum }
' ../data/small
@ -524,7 +524,7 @@ We can do similarly with multiple out-of-stream variables:
.. code-block:: none
:emphasize-lines: 1-8
$ mlr put -q '
mlr put -q '
@x_count += 1;
@x_sum += $x;
end {
@ -540,7 +540,7 @@ This is of course not much different than
.. code-block:: none
:emphasize-lines: 1-1
$ mlr stats1 -a count,sum -f x ../data/small
mlr stats1 -a count,sum -f x ../data/small
x_count=10,x_sum=4.536293840335763
Note that it's a syntax error for begin/end blocks to refer to field names (beginning with ``$``), since these execute outside the context of input records.

8
docs6/reference-dsl-filter-statements.rst
View file

@ -9,14 +9,14 @@ You can use ``filter`` within ``put``. In fact, the following two are synonymous
.. code-block:: none
:emphasize-lines: 1-1
$ mlr filter 'NR==2 || NR==3' data/small
mlr filter 'NR==2 || NR==3' data/small
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
.. code-block:: none
:emphasize-lines: 1-1
$ mlr put 'filter NR==2 || NR==3' data/small
mlr put 'filter NR==2 || NR==3' data/small
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
@ -25,7 +25,7 @@ The former, of course, is much easier to type. But the latter allows you to defi
.. code-block:: none
:emphasize-lines: 1-1
$ mlr put '@running_sum += $x; filter @running_sum > 1.3' data/small
mlr put '@running_sum += $x; filter @running_sum > 1.3' data/small
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
@ -33,6 +33,6 @@ The former, of course, is much easier to type. But the latter allows you to defi
.. code-block:: none
:emphasize-lines: 1-1
$ mlr put '$z = $x * $y; filter $z > 0.3' data/small
mlr put '$z = $x * $y; filter $z > 0.3' data/small
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797,z=0.3961455844854848
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729,z=0.4951063394654227

58
docs6/reference-dsl-output-statements.rst
View file

@ -79,13 +79,13 @@ Details:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --help-keyword print
mlr --help-keyword print
TODO: port mlr_dsl_keyword_usage
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --help-keyword dump
mlr --help-keyword dump
TODO: port mlr_dsl_keyword_usage
* ``mlr put`` sends the current record (possibly modified by the ``put`` expression) to the output record stream. Records are then input to the following verb in a ``then``-chain (if any), else printed to standard output (unless ``put -q``). The **tee** keyword *additionally* writes the output record to specified file(s) or pipe-to command, or immediately to ``stdout``/``stderr``.
@ -93,7 +93,7 @@ Details:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --help-keyword tee
mlr --help-keyword tee
TODO: port mlr_dsl_keyword_usage
* ``mlr put``'s ``emitf``, ``emitp``, and ``emit`` send out-of-stream variables to the output record stream. These are then input to the following verb in a ``then``-chain (if any), else printed to standard output. When redirected with ``>``, ``>>``, or ``|``, they *instead* write the out-of-stream variable(s) to specified file(s) or pipe-to command, or immediately to ``stdout``/``stderr``.
@ -101,19 +101,19 @@ Details:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --help-keyword emitf
mlr --help-keyword emitf
TODO: port mlr_dsl_keyword_usage
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --help-keyword emitp
mlr --help-keyword emitp
TODO: port mlr_dsl_keyword_usage
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --help-keyword emit
mlr --help-keyword emit
TODO: port mlr_dsl_keyword_usage
.. _reference-dsl-emit-statements:
@ -128,7 +128,7 @@ Use **emitf** to output several out-of-stream variables side-by-side in the same
.. code-block:: none
:emphasize-lines: 1-6
$ mlr put -q '
mlr put -q '
@count += 1;
@x_sum += $x;
@y_sum += $y;
@ -141,7 +141,7 @@ Use **emit** to output an out-of-stream variable. If it's non-indexed you'll get
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/small
cat data/small
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
@ -151,7 +151,7 @@ Use **emit** to output an out-of-stream variable. If it's non-indexed you'll get
.. code-block:: none
:emphasize-lines: 1-1
$ mlr put -q '@sum += $x; end { dump }' data/small
mlr put -q '@sum += $x; end { dump }' data/small
{
"sum": 2.264761728567491
}
@ -159,7 +159,7 @@ Use **emit** to output an out-of-stream variable. If it's non-indexed you'll get
.. code-block:: none
:emphasize-lines: 1-1
$ mlr put -q '@sum += $x; end { emit @sum }' data/small
mlr put -q '@sum += $x; end { emit @sum }' data/small
sum=2.264761728567491
If it's indexed then use as many names after ``emit`` as there are indices:
@ -167,7 +167,7 @@ If it's indexed then use as many names after ``emit`` as there are indices:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr put -q '@sum[$a] += $x; end { dump }' data/small
mlr put -q '@sum[$a] += $x; end { dump }' data/small
{
"sum": {
"pan": 0.3467901443380824,
@ -179,7 +179,7 @@ If it's indexed then use as many names after ``emit`` as there are indices:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr put -q '@sum[$a] += $x; end { emit @sum, "a" }' data/small
mlr put -q '@sum[$a] += $x; end { emit @sum, "a" }' data/small
a=pan,sum=0.3467901443380824
a=eks,sum=1.1400793586611044
a=wye,sum=0.7778922255683036
@ -187,7 +187,7 @@ If it's indexed then use as many names after ``emit`` as there are indices:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr put -q '@sum[$a][$b] += $x; end { dump }' data/small
mlr put -q '@sum[$a][$b] += $x; end { dump }' data/small
{
"sum": {
"pan": {
@ -207,7 +207,7 @@ If it's indexed then use as many names after ``emit`` as there are indices:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr put -q '@sum[$a][$b] += $x; end { emit @sum, "a", "b" }' data/small
mlr put -q '@sum[$a][$b] += $x; end { emit @sum, "a", "b" }' data/small
a=pan,b=pan,sum=0.3467901443380824
a=eks,b=pan,sum=0.7586799647899636
a=eks,b=wye,sum=0.38139939387114097
@ -217,7 +217,7 @@ If it's indexed then use as many names after ``emit`` as there are indices:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr put -q '@sum[$a][$b][$i] += $x; end { dump }' data/small
mlr put -q '@sum[$a][$b][$i] += $x; end { dump }' data/small
{
"sum": {
"pan": {
@ -247,7 +247,7 @@ If it's indexed then use as many names after ``emit`` as there are indices:
.. code-block:: none
:emphasize-lines: 1-4
$ mlr put -q '
mlr put -q '
@sum[$a][$b][$i] += $x;
end { emit @sum, "a", "b", "i" }
' data/small
@ -262,7 +262,7 @@ Now for **emitp**: if you have as many names following ``emit`` as there are lev
.. code-block:: none
:emphasize-lines: 1-1
$ mlr put -q '@sum[$a][$b] += $x; end { dump }' data/small
mlr put -q '@sum[$a][$b] += $x; end { dump }' data/small
{
"sum": {
"pan": {
@ -282,7 +282,7 @@ Now for **emitp**: if you have as many names following ``emit`` as there are lev
.. code-block:: none
:emphasize-lines: 1-1
$ mlr put -q '@sum[$a][$b] += $x; end { emit @sum, "a" }' data/small
mlr put -q '@sum[$a][$b] += $x; end { emit @sum, "a" }' data/small
a=pan,pan=0.3467901443380824
a=eks,pan=0.7586799647899636,wye=0.38139939387114097
a=wye,wye=0.20460330576630303,pan=0.5732889198020006
@ -290,13 +290,13 @@ Now for **emitp**: if you have as many names following ``emit`` as there are lev
.. code-block:: none
:emphasize-lines: 1-1
$ mlr put -q '@sum[$a][$b] += $x; end { emit @sum }' data/small
mlr put -q '@sum[$a][$b] += $x; end { emit @sum }' data/small
pan.pan=0.3467901443380824,eks.pan=0.7586799647899636,eks.wye=0.38139939387114097,wye.wye=0.20460330576630303,wye.pan=0.5732889198020006
.. code-block:: none
:emphasize-lines: 1-1
$ mlr put -q '@sum[$a][$b] += $x; end { emitp @sum, "a" }' data/small
mlr put -q '@sum[$a][$b] += $x; end { emitp @sum, "a" }' data/small
a=pan,sum.pan=0.3467901443380824
a=eks,sum.pan=0.7586799647899636,sum.wye=0.38139939387114097
a=wye,sum.wye=0.20460330576630303,sum.pan=0.5732889198020006
@ -304,13 +304,13 @@ Now for **emitp**: if you have as many names following ``emit`` as there are lev
.. code-block:: none
:emphasize-lines: 1-1
$ mlr put -q '@sum[$a][$b] += $x; end { emitp @sum }' data/small
mlr put -q '@sum[$a][$b] += $x; end { emitp @sum }' data/small
sum.pan.pan=0.3467901443380824,sum.eks.pan=0.7586799647899636,sum.eks.wye=0.38139939387114097,sum.wye.wye=0.20460330576630303,sum.wye.pan=0.5732889198020006
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --oxtab put -q '@sum[$a][$b] += $x; end { emitp @sum }' data/small
mlr --oxtab put -q '@sum[$a][$b] += $x; end { emitp @sum }' data/small
sum.pan.pan 0.3467901443380824
sum.eks.pan 0.7586799647899636
sum.eks.wye 0.38139939387114097
@ -323,7 +323,7 @@ keys for ``emitp`` (it defaults to a colon):
.. code-block:: none
:emphasize-lines: 1-1
$ mlr put -q --oflatsep / '@sum[$a][$b] += $x; end { emitp @sum, "a" }' data/small
mlr put -q --oflatsep / '@sum[$a][$b] += $x; end { emitp @sum, "a" }' data/small
a=pan,sum.pan=0.3467901443380824
a=eks,sum.pan=0.7586799647899636,sum.wye=0.38139939387114097
a=wye,sum.wye=0.20460330576630303,sum.pan=0.5732889198020006
@ -331,13 +331,13 @@ keys for ``emitp`` (it defaults to a colon):
.. code-block:: none
:emphasize-lines: 1-1
$ mlr put -q --oflatsep / '@sum[$a][$b] += $x; end { emitp @sum }' data/small
mlr put -q --oflatsep / '@sum[$a][$b] += $x; end { emitp @sum }' data/small
sum.pan.pan=0.3467901443380824,sum.eks.pan=0.7586799647899636,sum.eks.wye=0.38139939387114097,sum.wye.wye=0.20460330576630303,sum.wye.pan=0.5732889198020006
.. code-block:: none
:emphasize-lines: 1-4
$ mlr --oxtab put -q --oflatsep / '
mlr --oxtab put -q --oflatsep / '
@sum[$a][$b] += $x;
end { emitp @sum }
' data/small
@ -356,7 +356,7 @@ including their names in parentheses:
.. code-block:: none
:emphasize-lines: 1-10
$ mlr --from data/medium --opprint put -q '
mlr --from data/medium --opprint put -q '
@x_count[$a][$b] += 1;
@x_sum[$a][$b] += $x;
end {
@ -403,7 +403,7 @@ Use **emit all** (or ``emit @*`` which is synonymous) to output all out-of-strea
.. code-block:: none
:emphasize-lines: 1-5
$ mlr --from data/small --opprint put -q '
mlr --from data/small --opprint put -q '
@v[$a][$b]["sum"] += $x;
@v[$a][$b]["count"] += 1;
end{emit @*,"a","b"}
@ -420,7 +420,7 @@ Use **emit all** (or ``emit @*`` which is synonymous) to output all out-of-strea
.. code-block:: none
:emphasize-lines: 1-5
$ mlr --from data/small --opprint put -q '
mlr --from data/small --opprint put -q '
@sum[$a][$b] += $x;
@count[$a][$b] += 1;
end{emit @*,"a","b"}
@ -446,7 +446,7 @@ Use **emit all** (or ``emit @*`` which is synonymous) to output all out-of-strea
.. code-block:: none
:emphasize-lines: 1-5
$ mlr --from data/small --opprint put -q '
mlr --from data/small --opprint put -q '
@sum[$a][$b] += $x;
@count[$a][$b] += 1;
end{emit (@sum, @count),"a","b"}

24
docs6/reference-dsl-syntax.rst
View file

@ -12,7 +12,7 @@ Multiple expressions may be given, separated by semicolons, and each may refer t
.. code-block:: none
:emphasize-lines: 1-1
$ ruby -e '10.times{|i|puts "i=#{i}"}' | mlr --opprint put '$j = $i + 1; $k = $i +$j'
ruby -e '10.times{|i|puts "i=#{i}"}' | mlr --opprint put '$j = $i + 1; $k = $i +$j'
i j k
0 1 1
1 2 3
@ -30,7 +30,7 @@ Newlines within the expression are ignored, which can help increase legibility o
.. code-block:: none
:emphasize-lines: 1-7
$ mlr --opprint put '
mlr --opprint put '
$nf = NF;
$nr = NR;
$fnr = FNR;
@ -52,7 +52,7 @@ Newlines within the expression are ignored, which can help increase legibility o
.. code-block:: none
:emphasize-lines: 1-3
$ mlr --opprint filter '($x > 0.5 && $y < 0.5) || ($x < 0.5 && $y > 0.5)' \
mlr --opprint filter '($x > 0.5 && $y < 0.5) || ($x < 0.5 && $y > 0.5)' \
then stats2 -a corr -f x,y \
data/medium
x_y_corr
@ -68,7 +68,7 @@ The simplest way to enter expressions for ``put`` and ``filter`` is between sing
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --from data/small put '$xy = sqrt($x**2 + $y**2)'
mlr --from data/small put '$xy = sqrt($x**2 + $y**2)'
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533,xy=0.8052985815845617
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797,xy=0.9209978658539777
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776,xy=0.3953756915115773
@ -78,7 +78,7 @@ The simplest way to enter expressions for ``put`` and ``filter`` is between sing
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --from data/small put 'func f(a, b) { return sqrt(a**2 + b**2) } $xy = f($x, $y)'
mlr --from data/small put 'func f(a, b) { return sqrt(a**2 + b**2) } $xy = f($x, $y)'
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533,xy=0.8052985815845617
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797,xy=0.9209978658539777
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776,xy=0.3953756915115773
@ -91,7 +91,7 @@ You may, though, find it convenient to put expressions into files for reuse, and
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/fe-example-3.mlr
cat data/fe-example-3.mlr
func f(a, b) {
return sqrt(a**2 + b**2)
}
@ -100,7 +100,7 @@ You may, though, find it convenient to put expressions into files for reuse, and
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --from data/small put -f data/fe-example-3.mlr
mlr --from data/small put -f data/fe-example-3.mlr
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533,xy=0.8052985815845617
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797,xy=0.9209978658539777
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776,xy=0.3953756915115773
@ -112,7 +112,7 @@ If you have some of the logic in a file and you want to write the rest on the co
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/fe-example-4.mlr
cat data/fe-example-4.mlr
func f(a, b) {
return sqrt(a**2 + b**2)
}
@ -120,7 +120,7 @@ If you have some of the logic in a file and you want to write the rest on the co
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --from data/small put -f data/fe-example-4.mlr -e '$xy = f($x, $y)'
mlr --from data/small put -f data/fe-example-4.mlr -e '$xy = f($x, $y)'
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533,xy=0.8052985815845617
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797,xy=0.9209978658539777
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776,xy=0.3953756915115773
@ -150,13 +150,13 @@ Semicolons are optional after closing curly braces (which close conditionals and
.. code-block:: none
:emphasize-lines: 1-1
$ echo x=1,y=2 | mlr put 'while (NF < 10) { $[NF+1] = ""} $foo = "bar"'
echo x=1,y=2 | mlr put 'while (NF < 10) { $[NF+1] = ""} $foo = "bar"'
x=1,y=2,3=,4=,5=,6=,7=,8=,9=,10=,foo=bar
.. code-block:: none
:emphasize-lines: 1-1
$ echo x=1,y=2 | mlr put 'while (NF < 10) { $[NF+1] = ""}; $foo = "bar"'
echo x=1,y=2 | mlr put 'while (NF < 10) { $[NF+1] = ""}; $foo = "bar"'
x=1,y=2,3=,4=,5=,6=,7=,8=,9=,10=,foo=bar
Semicolons are required between statements even if those statements are on separate lines. **Newlines** are for your convenience but have no syntactic meaning: line endings do not terminate statements. For example, adjacent assignment statements must be separated by semicolons even if those statements are on separate lines:
@ -178,7 +178,7 @@ Semicolons are required between statements even if those statements are on separ
.. code-block:: none
:emphasize-lines: 1-17
$ mlr --csvlite --from data/a.csv put '
mlr --csvlite --from data/a.csv put '
func f(
num a,
num b,

8
docs6/reference-dsl-unset-statements.rst
View file

@ -9,7 +9,7 @@ You can clear a map key by assigning the empty string as its value: ``$x=""`` or
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/small
cat data/small
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
@ -19,7 +19,7 @@ You can clear a map key by assigning the empty string as its value: ``$x=""`` or
.. code-block:: none
:emphasize-lines: 1-1
$ mlr put 'unset $x, $a' data/small
mlr put 'unset $x, $a' data/small
b=pan,i=1,y=0.7268028627434533
b=pan,i=2,y=0.5221511083334797
b=wye,i=3,y=0.33831852551664776
@ -31,7 +31,7 @@ This can also be done, of course, using ``mlr cut -x``. You can also clear out-o
.. code-block:: none
:emphasize-lines: 1-1
$ mlr put -q '@sum[$a][$b] += $x; end { dump; unset @sum; dump }' data/small
mlr put -q '@sum[$a][$b] += $x; end { dump; unset @sum; dump }' data/small
{
"sum": {
"pan": {
@ -52,7 +52,7 @@ This can also be done, of course, using ``mlr cut -x``. You can also clear out-o
.. code-block:: none
:emphasize-lines: 1-1
$ mlr put -q '@sum[$a][$b] += $x; end { dump; unset @sum["eks"]; dump }' data/small
mlr put -q '@sum[$a][$b] += $x; end { dump; unset @sum["eks"]; dump }' data/small
{
"sum": {
"pan": {

4
docs6/reference-dsl-user-defined-functions.rst
View file

@ -14,7 +14,7 @@ Here's the obligatory example of a recursive function to compute the factorial f
.. code-block:: none
:emphasize-lines: 1-14
$ mlr --opprint --from data/small put '
mlr --opprint --from data/small put '
func f(n) {
if (is_numeric(n)) {
if (n > 0) {
@ -63,7 +63,7 @@ Example:
.. code-block:: none
:emphasize-lines: 1-17
$ mlr --opprint --from data/small put -q '
mlr --opprint --from data/small put -q '
begin {
@call_count = 0;
}

72
docs6/reference-dsl-variables.rst
View file

@ -26,7 +26,7 @@ Namely, Miller supports the following five built-in variables for :doc:`filter a
.. code-block:: none
:emphasize-lines: 1-1
$ mlr filter 'FNR == 2' data/small*
mlr filter 'FNR == 2' data/small*
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
1=pan,2=pan,3=1,4=0.3467901443380824,5=0.7268028627434533
a=wye,b=eks,i=10000,x=0.734806020620654365,y=0.884788571337605134
@ -34,7 +34,7 @@ Namely, Miller supports the following five built-in variables for :doc:`filter a
.. code-block:: none
:emphasize-lines: 1-1
$ mlr put '$fnr = FNR' data/small*
mlr put '$fnr = FNR' data/small*
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533,fnr=1
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797,fnr=2
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776,fnr=3
@ -59,7 +59,7 @@ Their **scope is global**: you can refer to them in any ``filter`` or ``put`` st
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --csv put '$nr = NR' data/a.csv
mlr --csv put '$nr = NR' data/a.csv
a,b,c,nr
1,2,3,1
4,5,6,2
@ -67,7 +67,7 @@ Their **scope is global**: you can refer to them in any ``filter`` or ``put`` st
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --csv repeat -n 3 then put '$nr = NR' data/a.csv
mlr --csv repeat -n 3 then put '$nr = NR' data/a.csv
a,b,c,nr
1,2,3,1
1,2,3,1
@ -92,12 +92,12 @@ You may also use a **computed field name** in square brackets, e.g.
.. code-block:: none
:emphasize-lines: 1-1
$ echo a=3,b=4 | mlr filter '$["x"] < 0.5'
echo a=3,b=4 | mlr filter '$["x"] < 0.5'
.. code-block:: none
:emphasize-lines: 1-1
$ echo s=green,t=blue,a=3,b=4 | mlr put '$[$s."_".$t] = $a * $b'
echo s=green,t=blue,a=3,b=4 | mlr put '$[$s."_".$t] = $a * $b'
s=green,t=blue,a=3,b=4,green_blue=12
Notes:
@ -122,7 +122,7 @@ Then using a computed field name, ``$[ $[[3]] ]`` is the value in the third fiel
.. code-block:: none
:emphasize-lines: 1-1
$ mlr cat data/small
mlr cat data/small
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
@ -132,7 +132,7 @@ Then using a computed field name, ``$[ $[[3]] ]`` is the value in the third fiel
.. code-block:: none
:emphasize-lines: 1-1
$ mlr put '$[[3]] = "NEW"' data/small
mlr put '$[[3]] = "NEW"' data/small
a=pan,b=pan,NEW=1,x=0.3467901443380824,y=0.7268028627434533
a=eks,b=pan,NEW=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,NEW=3,x=0.20460330576630303,y=0.33831852551664776
@ -142,7 +142,7 @@ Then using a computed field name, ``$[ $[[3]] ]`` is the value in the third fiel
.. code-block:: none
:emphasize-lines: 1-1
$ mlr put '$[[[3]]] = "NEW"' data/small
mlr put '$[[[3]]] = "NEW"' data/small
a=pan,b=pan,i=NEW,x=0.3467901443380824,y=0.7268028627434533
a=eks,b=pan,i=NEW,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,i=NEW,x=0.20460330576630303,y=0.33831852551664776
@ -152,7 +152,7 @@ Then using a computed field name, ``$[ $[[3]] ]`` is the value in the third fiel
.. code-block:: none
:emphasize-lines: 1-1
$ mlr put '$NEW = $[[NR]]' data/small
mlr put '$NEW = $[[NR]]' data/small
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533,NEW=a
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797,NEW=b
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776,NEW=i
@ -162,7 +162,7 @@ Then using a computed field name, ``$[ $[[3]] ]`` is the value in the third fiel
.. code-block:: none
:emphasize-lines: 1-1
$ mlr put '$NEW = $[[[NR]]]' data/small
mlr put '$NEW = $[[[NR]]]' data/small
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533,NEW=pan
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797,NEW=pan
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776,NEW=3
@ -172,7 +172,7 @@ Then using a computed field name, ``$[ $[[3]] ]`` is the value in the third fiel
.. code-block:: none
:emphasize-lines: 1-1
$ mlr put '$[[[NR]]] = "NEW"' data/small
mlr put '$[[[NR]]] = "NEW"' data/small
a=NEW,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
a=eks,b=NEW,i=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,i=NEW,x=0.20460330576630303,y=0.33831852551664776
@ -184,7 +184,7 @@ Right-hand side accesses to non-existent fields -- i.e. with index less than 1 o
.. code-block:: none
:emphasize-lines: 1-1
$ mlr put '$[[6]] = "NEW"' data/small
mlr put '$[[6]] = "NEW"' data/small
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
@ -194,7 +194,7 @@ Right-hand side accesses to non-existent fields -- i.e. with index less than 1 o
.. code-block:: none
:emphasize-lines: 1-1
$ mlr put '$[[[6]]] = "NEW"' data/small
mlr put '$[[[6]]] = "NEW"' data/small
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
@ -215,7 +215,7 @@ You may use a **computed key** in square brackets, e.g.
.. code-block:: none
:emphasize-lines: 1-1
$ echo s=green,t=blue,a=3,b=4 | mlr put -q '@[$s."_".$t] = $a * $b; emit all'
echo s=green,t=blue,a=3,b=4 | mlr put -q '@[$s."_".$t] = $a * $b; emit all'
green_blue=12
Out-of-stream variables are **scoped** to the ``put`` command in which they appear. In particular, if you have two or more ``put`` commands separated by ``then``, each put will have its own set of out-of-stream variables:
@ -223,14 +223,14 @@ Out-of-stream variables are **scoped** to the ``put`` command in which they appe
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/a.dkvp
cat data/a.dkvp
a=1,b=2,c=3
a=4,b=5,c=6
.. code-block:: none
:emphasize-lines: 1-3
$ mlr put '@sum += $a; end {emit @sum}' \
mlr put '@sum += $a; end {emit @sum}' \
then put 'is_present($a) {$a=10*$a; @sum += $a}; end {emit @sum}' \
data/a.dkvp
a=10,b=2,c=3
@ -250,7 +250,7 @@ Using an index on the ``@count`` and ``@sum`` variables, we get the benefit of t
.. code-block:: none
:emphasize-lines: 1-8
$ mlr put -q '
mlr put -q '
@x_count[$a] += 1;
@x_sum[$a] += $x;
end {
@ -272,7 +272,7 @@ Using an index on the ``@count`` and ``@sum`` variables, we get the benefit of t
.. code-block:: none
:emphasize-lines: 1-1
$ mlr stats1 -a count,sum -f x -g a ../data/small
mlr stats1 -a count,sum -f x -g a ../data/small
a=pan,x_count=2,x_sum=0.8494161498792961
a=eks,x_count=3,x_sum=1.75186341922895
a=wye,x_count=2,x_sum=0.7778922255683036
@ -284,7 +284,7 @@ Indices can be arbitrarily deep -- here there are two or more of them:
.. code-block:: none
:emphasize-lines: 1-7
$ mlr --from data/medium put -q '
mlr --from data/medium put -q '
@x_count[$a][$b] += 1;
@x_sum[$a][$b] += $x;
end {
@ -324,7 +324,7 @@ Begin/end blocks can be mixed with pattern/action blocks. For example:
.. code-block:: none
:emphasize-lines: 1-14
$ mlr put '
mlr put '
begin {
@num_total = 0;
@num_positive = 0;
@ -357,7 +357,7 @@ For example:
.. code-block:: none
:emphasize-lines: 1-16
$ # Here I'm using a specified random-number seed so this example always
# Here I'm using a specified random-number seed so this example always
# produces the same output for this web document: in everyday practice we
# would leave off the --seed 12345 part.
mlr --seed 12345 seqgen --start 1 --stop 10 then put '
@ -413,7 +413,7 @@ The following example demonstrates the scope rules:
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/scope-example.mlr
cat data/scope-example.mlr
func f(a) { # argument is local to the function
var b = 100; # local to the function
c = 100; # local to the function; does not overwrite outer c
@ -441,7 +441,7 @@ The following example demonstrates the scope rules:
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/scope-example.dat
cat data/scope-example.dat
n=1,x=123
n=2,x=456
n=3,x=789
@ -449,7 +449,7 @@ The following example demonstrates the scope rules:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --oxtab --from data/scope-example.dat put -f data/scope-example.mlr
mlr --oxtab --from data/scope-example.dat put -f data/scope-example.mlr
n 1
x 123
outer_a 10
@ -477,7 +477,7 @@ And this example demonstrates the type-declaration rules:
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/type-decl-example.mlr
cat data/type-decl-example.mlr
subr s(a, str b, int c) { # a is implicitly var (untyped).
# b is explicitly str.
# c is explicitly int.
@ -526,7 +526,7 @@ For example, the following swaps the input stream's ``a`` and ``i`` fields, modi
.. code-block:: none
:emphasize-lines: 1-7
$ mlr --opprint put '
mlr --opprint put '
$* = {
"a": $i,
"i": $a,
@ -545,7 +545,7 @@ Likewise, you can assign map literals to out-of-stream variables or local variab
.. code-block:: none
:emphasize-lines: 1-7
$ mlr --from data/small put '
mlr --from data/small put '
func f(map m): map {
m["x"] *= 200;
return m;
@ -563,7 +563,7 @@ Like out-of-stream and local variables, map literals can be multi-level:
.. code-block:: none
:emphasize-lines: 1-19
$ mlr --from data/small put -q '
mlr --from data/small put -q '
begin {
@o = {
"nrec": 0,
@ -611,7 +611,7 @@ The following ``is...`` functions take a value and return a boolean indicating w
.. code-block:: none
:emphasize-lines: 1-1
$ mlr -F | grep ^is
mlr -F | grep ^is
is_absent
is_array
is_bool
@ -635,7 +635,7 @@ The following ``is...`` functions take a value and return a boolean indicating w
.. code-block:: none
:emphasize-lines: 1-1
$ mlr -F | grep ^assert
mlr -F | grep ^assert
asserting_absent
asserting_array
asserting_bool
@ -749,7 +749,7 @@ Example recursive copy of out-of-stream variables:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --opprint put -q '@v["sum"] += $x; @v["count"] += 1; end{dump; @w = @v; dump}' data/small
mlr --opprint put -q '@v["sum"] += $x; @v["count"] += 1; end{dump; @w = @v; dump}' data/small
{
"v": {
"sum": 2.264761728567491,
@ -772,7 +772,7 @@ Example of out-of-stream variable assigned to full stream record, where the 2nd
.. code-block:: none
:emphasize-lines: 1-1
$ mlr put 'NR == 2 {@keep = $*}; NR == 4 {$* = @keep}' data/small
mlr put 'NR == 2 {@keep = $*}; NR == 4 {$* = @keep}' data/small
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
@ -784,7 +784,7 @@ Example of full stream record assigned to an out-of-stream variable, finding the
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/small
cat data/small
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
@ -794,7 +794,7 @@ Example of full stream record assigned to an out-of-stream variable, finding the
.. code-block:: none
:emphasize-lines: 1-4
$ mlr --opprint put -q '
mlr --opprint put -q '
is_null(@xmax) || $x > @xmax {@xmax=$x; @recmax=$*};
end {emit @recmax}
' data/small
@ -807,6 +807,6 @@ Keywords for filter and put
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --help-all-keywords
mlr --help-all-keywords
TODO: port mlr_dsl_list_all_keywords

10
docs6/reference-dsl.rst
View file

@ -14,7 +14,7 @@ Example:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr stats1 -a sum -f x -g a data/small
mlr stats1 -a sum -f x -g a data/small
a=pan,x_sum=0.3467901443380824
a=eks,x_sum=1.1400793586611044
a=wye,x_sum=0.7778922255683036
@ -30,7 +30,7 @@ Example:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr put -q '@x_sum[$a] += $x; end{emit @x_sum, "a"}' data/small
mlr put -q '@x_sum[$a] += $x; end{emit @x_sum, "a"}' data/small
a=pan,x_sum=0.3467901443380824
a=eks,x_sum=1.1400793586611044
a=wye,x_sum=0.7778922255683036
@ -48,7 +48,7 @@ The essential usages of ``mlr filter`` and ``mlr put`` are for record-selection
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/small
cat data/small
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
@ -60,7 +60,7 @@ you might retain only the records whose ``a`` field has value ``eks``:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr filter '$a == "eks"' data/small
mlr filter '$a == "eks"' data/small
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463
@ -69,7 +69,7 @@ or you might add a new field which is a function of existing fields:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr put '$ab = $a . "_" . $b ' data/small
mlr put '$ab = $a . "_" . $b ' data/small
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533,ab=pan_pan
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797,ab=eks_pan
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776,ab=wye_wye

20
docs6/reference-main-auxiliary-commands.rst
View file

@ -9,7 +9,7 @@ There are a few nearly-standalone programs which have nothing to do with the res
.. code-block:: none
:emphasize-lines: 1-1
$ mlr aux-list
mlr aux-list
Available subcommands:
aux-list
hex
@ -23,7 +23,7 @@ There are a few nearly-standalone programs which have nothing to do with the res
.. code-block:: none
:emphasize-lines: 1-1
$ mlr lecat --help
mlr lecat --help
Usage: mlr lecat [options] {zero or more file names}
Simple hex-dump.
If zero file names are supplied, standard input is read.
@ -34,7 +34,7 @@ There are a few nearly-standalone programs which have nothing to do with the res
.. code-block:: none
:emphasize-lines: 1-1
$ mlr termcvt --help
mlr termcvt --help
Usage: mlr termcvt [option] {zero or more file names}
Option (exactly one is required):
--cr2crlf
@ -51,7 +51,7 @@ There are a few nearly-standalone programs which have nothing to do with the res
.. code-block:: none
:emphasize-lines: 1-1
$ mlr hex --help
mlr hex --help
Usage: mlr hex [options] {zero or more file names}
Simple hex-dump.
If zero file names are supplied, standard input is read.
@ -62,7 +62,7 @@ There are a few nearly-standalone programs which have nothing to do with the res
.. code-block:: none
:emphasize-lines: 1-1
$ mlr unhex --help
mlr unhex --help
Usage: mlr unhex [options] {zero or more file names}
Simple hex-dump.
If zero file names are supplied, standard input is read.
@ -75,19 +75,19 @@ Examples:
.. code-block:: none
:emphasize-lines: 1-1
$ echo 'Hello, world!' | mlr lecat --mono
echo 'Hello, world!' | mlr lecat --mono
Hello, world![LF]
.. code-block:: none
:emphasize-lines: 1-1
$ echo 'Hello, world!' | mlr termcvt --lf2crlf | mlr lecat --mono
echo 'Hello, world!' | mlr termcvt --lf2crlf | mlr lecat --mono
Hello, world![CR][LF]
.. code-block:: none
:emphasize-lines: 1-1
$ mlr hex data/budget.csv
mlr hex data/budget.csv
00000000: 23 20 41 73 61 6e 61 20 2d 2d 20 68 65 72 65 20 |# Asana -- here |
00000010: 61 72 65 20 74 68 65 20 62 75 64 67 65 74 20 66 |are the budget f|
00000020: 69 67 75 72 65 73 20 79 6f 75 20 61 73 6b 65 64 |igures you asked|
@ -99,7 +99,7 @@ Examples:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr hex -r data/budget.csv
mlr hex -r data/budget.csv
23 20 41 73 61 6e 61 20 2d 2d 20 68 65 72 65 20
61 72 65 20 74 68 65 20 62 75 64 67 65 74 20 66
69 67 75 72 65 73 20 79 6f 75 20 61 73 6b 65 64
@ -111,7 +111,7 @@ Examples:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr hex -r data/budget.csv | sed 's/20/2a/g' | mlr unhex
mlr hex -r data/budget.csv | sed 's/20/2a/g' | mlr unhex
#*Asana*--*here*are*the*budget*figures*you*asked*for!
type,quantity
purple,456.78

10
docs6/reference-main-io-options.rst
View file

@ -24,7 +24,7 @@ These are as discussed in :doc:`file-formats`, with the exception of ``--right``
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --opprint cat data/small
mlr --opprint cat data/small
a b i x y
pan pan 1 0.3467901443380824 0.7268028627434533
eks pan 2 0.7586799647899636 0.5221511083334797
@ -35,7 +35,7 @@ These are as discussed in :doc:`file-formats`, with the exception of ``--right``
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --opprint --right cat data/small
mlr --opprint --right cat data/small
a b i x y
pan pan 1 0.3467901443380824 0.7268028627434533
eks pan 2 0.7586799647899636 0.5221511083334797
@ -138,13 +138,13 @@ To apply formatting to a single field, overriding the global ``ofmt``, use ``fmt
.. code-block:: none
:emphasize-lines: 1-1
$ echo 'x=3.1,y=4.3' | mlr put '$z=fmtnum($x*$y,"%08lf")'
echo 'x=3.1,y=4.3' | mlr put '$z=fmtnum($x*$y,"%08lf")'
x=3.1,y=4.3,z=%!l(float64=00013.33)f
.. code-block:: none
:emphasize-lines: 1-1
$ echo 'x=0xffff,y=0xff' | mlr put '$z=fmtnum(int($x*$y),"%08llx")'
echo 'x=0xffff,y=0xff' | mlr put '$z=fmtnum(int($x*$y),"%08llx")'
x=0xffff,y=0xff,z=%!l(int=16711425)lx
Input conversion from hexadecimal is done automatically on fields handled by ``mlr put`` and ``mlr filter`` as long as the field value begins with "0x". To apply output conversion to hexadecimal on a single column, you may use ``fmtnum``, or the keystroke-saving ``hexfmt`` function. Example:
@ -152,5 +152,5 @@ Input conversion from hexadecimal is done automatically on fields handled by ``m
.. code-block:: none
:emphasize-lines: 1-1
$ echo 'x=0xffff,y=0xff' | mlr put '$z=hexfmt($x*$y)'
echo 'x=0xffff,y=0xff' | mlr put '$z=hexfmt($x*$y)'
x=0xffff,y=0xff,z=0xfeff01

26
docs6/reference-main-null-data.rst
View file

@ -21,7 +21,7 @@ Rules for null-handling:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr cat data/sort-null.dat
mlr cat data/sort-null.dat
a=3,b=2
a=1,b=8
a=,b=4
@ -31,7 +31,7 @@ Rules for null-handling:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr sort -n a data/sort-null.dat
mlr sort -n a data/sort-null.dat
a=1,b=8
a=3,b=2
a=5,b=7
@ -41,7 +41,7 @@ Rules for null-handling:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr sort -nr a data/sort-null.dat
mlr sort -nr a data/sort-null.dat
a=,b=4
a=5,b=7
a=3,b=2
@ -53,19 +53,19 @@ Rules for null-handling:
.. code-block:: none
:emphasize-lines: 1-1
$ echo 'x=2,y=3' | mlr put '$a=$x+$y'
echo 'x=2,y=3' | mlr put '$a=$x+$y'
x=2,y=3,a=5
.. code-block:: none
:emphasize-lines: 1-1
$ echo 'x=,y=3' | mlr put '$a=$x+$y'
echo 'x=,y=3' | mlr put '$a=$x+$y'
x=,y=3,a=
.. code-block:: none
:emphasize-lines: 1-1
$ echo 'x=,y=3' | mlr put '$a=log($x);$b=log($y)'
echo 'x=,y=3' | mlr put '$a=log($x);$b=log($y)'
x=,y=3,a=,b=1.0986122886681096
with the exception that the ``min`` and ``max`` functions are special: if one argument is non-null, it wins:
@ -73,7 +73,7 @@ with the exception that the ``min`` and ``max`` functions are special: if one ar
.. code-block:: none
:emphasize-lines: 1-1
$ echo 'x=,y=3' | mlr put '$a=min($x,$y);$b=max($x,$y)'
echo 'x=,y=3' | mlr put '$a=min($x,$y);$b=max($x,$y)'
x=,y=3,a=3,b=
* Functions of *absent* variables (e.g. ``mlr put '$y = log10($nonesuch)'``) evaluate to absent, and arithmetic/bitwise/boolean operators with both operands being absent evaluate to absent. Arithmetic operators with one absent operand return the other operand. More specifically, absent values act like zero for addition/subtraction, and one for multiplication: Furthermore, **any expression which evaluates to absent is not stored in the left-hand side of an assignment statement**:
@ -81,13 +81,13 @@ with the exception that the ``min`` and ``max`` functions are special: if one ar
.. code-block:: none
:emphasize-lines: 1-1
$ echo 'x=2,y=3' | mlr put '$a=$u+$v; $b=$u+$y; $c=$x+$y'
echo 'x=2,y=3' | mlr put '$a=$u+$v; $b=$u+$y; $c=$x+$y'
x=2,y=3,b=3,c=5
.. code-block:: none
:emphasize-lines: 1-1
$ echo 'x=2,y=3' | mlr put '$a=min($x,$v);$b=max($u,$y);$c=min($u,$v)'
echo 'x=2,y=3' | mlr put '$a=min($x,$v);$b=max($u,$y);$c=min($u,$v)'
x=2,y=3,a=2,b=3
* Likewise, for assignment to maps, **absent-valued keys or values result in a skipped assignment**.
@ -107,7 +107,7 @@ Since absent plus absent is absent (and likewise for other operators), accumulat
.. code-block:: none
:emphasize-lines: 1-1
$ mlr cat data/het.dkvp
mlr cat data/het.dkvp
resource=/path/to/file,loadsec=0.45,ok=true
record_count=100,resource=/path/to/file
resource=/path/to/second/file,loadsec=0.32,ok=true
@ -117,7 +117,7 @@ Since absent plus absent is absent (and likewise for other operators), accumulat
.. code-block:: none
:emphasize-lines: 1-1
$ mlr put 'is_present($loadsec) { $loadmillis = $loadsec * 1000 }' data/het.dkvp
mlr put 'is_present($loadsec) { $loadmillis = $loadsec * 1000 }' data/het.dkvp
resource=/path/to/file,loadsec=0.45,ok=true,loadmillis=450
record_count=100,resource=/path/to/file
resource=/path/to/second/file,loadsec=0.32,ok=true,loadmillis=320
@ -127,7 +127,7 @@ Since absent plus absent is absent (and likewise for other operators), accumulat
.. code-block:: none
:emphasize-lines: 1-1
$ mlr put '$loadmillis = (is_present($loadsec) ? $loadsec : 0.0) * 1000' data/het.dkvp
mlr put '$loadmillis = (is_present($loadsec) ? $loadsec : 0.0) * 1000' data/het.dkvp
resource=/path/to/file,loadsec=0.45,ok=true,loadmillis=450
record_count=100,resource=/path/to/file,loadmillis=0
resource=/path/to/second/file,loadsec=0.32,ok=true,loadmillis=320
@ -139,5 +139,5 @@ If you're interested in a formal description of how empty and absent fields part
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --print-type-arithmetic-info
mlr --print-type-arithmetic-info
TODO: port printTypeArithmeticInfo

4
docs6/reference-main-online-help.rst
View file

@ -11,7 +11,7 @@ Examples:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --help
mlr --help
Usage: mlr [I/O options] {verb} [verb-dependent options ...] {zero or more file names}
COMMAND-LINE-SYNTAX EXAMPLES:
@ -585,7 +585,7 @@ Examples:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr sort --help
mlr sort --help
Usage: mlr sort {flags}
Sorts records primarily by the first specified field, secondarily by the second
field, and so on. (Any records not having all specified sort keys will appear

8
docs6/reference-main-overview.rst
View file

@ -19,7 +19,7 @@ For example, reading from a file:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --icsv --opprint head -n 2 then sort -f shape example.csv
mlr --icsv --opprint head -n 2 then sort -f shape example.csv
color shape flag index quantity rate
red square true 15 79.2778 0.0130
yellow triangle true 11 43.6498 9.8870
@ -29,7 +29,7 @@ Reading from standard input:
.. code-block:: none
:emphasize-lines: 1-1
$ cat example.csv | mlr --icsv --opprint head -n 2 then sort -f shape
cat example.csv | mlr --icsv --opprint head -n 2 then sort -f shape
color shape flag index quantity rate
red square true 15 79.2778 0.0130
yellow triangle true 11 43.6498 9.8870
@ -50,7 +50,7 @@ Example of using a verb for data processing:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr stats1 -a sum -f x -g a data/small
mlr stats1 -a sum -f x -g a data/small
a=pan,x_sum=0.3467901443380824
a=eks,x_sum=1.1400793586611044
a=wye,x_sum=0.7778922255683036
@ -66,7 +66,7 @@ Example of doing the same thing using a DSL expression:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr put -q '@x_sum[$a] += $x; end{emit @x_sum, "a"}' data/small
mlr put -q '@x_sum[$a] += $x; end{emit @x_sum, "a"}' data/small
a=pan,x_sum=0.3467901443380824
a=eks,x_sum=1.1400793586611044
a=wye,x_sum=0.7778922255683036

4
docs6/reference-main-regular-expressions.rst
View file

@ -35,7 +35,7 @@ Example:
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/regex-in-data.dat
cat data/regex-in-data.dat
name=jane,regex=^j.*e$
name=bill,regex=^b[ou]ll$
name=bull,regex=^b[ou]ll$
@ -43,7 +43,7 @@ Example:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr filter '$name =~ $regex' data/regex-in-data.dat
mlr filter '$name =~ $regex' data/regex-in-data.dat
name=jane,regex=^j.*e$
name=bull,regex=^b[ou]ll$

396
docs6/reference-verbs.rst
View file

File diff suppressed because it is too large Load diff

2
docs6/repl.rst
View file

@ -32,7 +32,7 @@ Using ``put`` and ``filter``, you can do the following as we've seen above:
.. code-block:: none
:emphasize-lines: 1-2
$ mlr --icsv --ojson --from example.csv head -n 2 \
mlr --icsv --ojson --from example.csv head -n 2 \
then put 'begin {print "HELLO"} $z = $x + $y; end {print "GOODBYE"}'
HELLO
{

42
docs6/shapes-of-data.rst
View file

@ -89,44 +89,44 @@ Miller records are ordered lists of key-value pairs. For NIDX format, DKVP forma
.. code-block:: none
:emphasize-lines: 1-1
$ echo x,y,z | mlr --dkvp cat
echo x,y,z | mlr --dkvp cat
1=x,2=y,3=z
.. code-block:: none
:emphasize-lines: 1-1
$ echo x,y,z | mlr --dkvp put '$6="a";$4="b";$55="cde"'
echo x,y,z | mlr --dkvp put '$6="a";$4="b";$55="cde"'
1=x,2=y,3=z,6=a,4=b,55=cde
.. code-block:: none
:emphasize-lines: 1-1
$ echo x,y,z | mlr --nidx cat
echo x,y,z | mlr --nidx cat
x,y,z
.. code-block:: none
:emphasize-lines: 1-1
$ echo x,y,z | mlr --csv --implicit-csv-header cat
echo x,y,z | mlr --csv --implicit-csv-header cat
1,2,3
x,y,z
.. code-block:: none
:emphasize-lines: 1-1
$ echo x,y,z | mlr --dkvp rename 2,999
echo x,y,z | mlr --dkvp rename 2,999
1=x,999=y,3=z
.. code-block:: none
:emphasize-lines: 1-1
$ echo x,y,z | mlr --dkvp rename 2,newname
echo x,y,z | mlr --dkvp rename 2,newname
1=x,newname=y,3=z
.. code-block:: none
:emphasize-lines: 1-1
$ echo x,y,z | mlr --csv --implicit-csv-header reorder -f 3,1,2
echo x,y,z | mlr --csv --implicit-csv-header reorder -f 3,1,2
3,1,2
z,x,y
@ -138,7 +138,7 @@ Example: columns ``x,i,a`` were requested but they appear here in the order ``a,
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/small
cat data/small
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
@ -148,7 +148,7 @@ Example: columns ``x,i,a`` were requested but they appear here in the order ``a,
.. code-block:: none
:emphasize-lines: 1-1
$ mlr cut -f x,i,a data/small
mlr cut -f x,i,a data/small
a=pan,i=1,x=0.3467901443380824
a=eks,i=2,x=0.7586799647899636
a=wye,i=3,x=0.20460330576630303
@ -162,7 +162,7 @@ The solution is to use the ``-o`` option:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr cut -o -f x,i,a data/small
mlr cut -o -f x,i,a data/small
x=0.3467901443380824,i=1,a=pan
x=0.7586799647899636,i=2,a=eks
x=0.20460330576630303,i=3,a=wye
@ -177,7 +177,7 @@ The ``awk``-like built-in variable ``NR`` is incremented for each input record:
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/small
cat data/small
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
@ -187,7 +187,7 @@ The ``awk``-like built-in variable ``NR`` is incremented for each input record:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr put '$nr = NR' data/small
mlr put '$nr = NR' data/small
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533,nr=1
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797,nr=2
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776,nr=3
@ -199,7 +199,7 @@ However, this is the record number within the original input stream -- not after
.. code-block:: none
:emphasize-lines: 1-1
$ mlr filter '$a == "wye"' then put '$nr = NR' data/small
mlr filter '$a == "wye"' then put '$nr = NR' data/small
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776,nr=3
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729,nr=5
@ -208,7 +208,7 @@ There are two good options here. One is to use the ``cat`` verb with ``-n``:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr filter '$a == "wye"' then cat -n data/small
mlr filter '$a == "wye"' then cat -n data/small
n=1,a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
n=2,a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
@ -217,7 +217,7 @@ The other is to keep your own counter within the ``put`` DSL:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr filter '$a == "wye"' then put 'begin {@n = 1} $n = @n; @n += 1' data/small
mlr filter '$a == "wye"' then put 'begin {@n = 1} $n = @n; @n += 1' data/small
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776,n=1
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729,n=2
@ -239,7 +239,7 @@ The simplest option is to use :ref:`mlr nest <reference-verbs-nest>`:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --tsv nest --explode --values --across-records -f b --nested-fs : data/nested.tsv
mlr --tsv nest --explode --values --across-records -f b --nested-fs : data/nested.tsv
a b
x z
s u
@ -249,7 +249,7 @@ The simplest option is to use :ref:`mlr nest <reference-verbs-nest>`:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --tsv nest --explode --values --across-fields -f b --nested-fs : data/nested.tsv
mlr --tsv nest --explode --values --across-fields -f b --nested-fs : data/nested.tsv
a b_1
x z
@ -263,7 +263,7 @@ One option to split out the colon-delimited values in the ``b`` column is to use
.. code-block:: none
:emphasize-lines: 1-6
$ mlr --from data/nested.tsv --itsv --oxtab put '
mlr --from data/nested.tsv --itsv --oxtab put '
o = splitnv($b, ":");
for (k,v in o) {
$["p".k]=v
@ -284,7 +284,7 @@ while another is to loop over the same map from ``splitnv`` and use it (with ``p
.. code-block:: none
:emphasize-lines: 1-7
$ mlr --from data/nested.tsv --itsv --oxtab put -q '
mlr --from data/nested.tsv --itsv --oxtab put -q '
o = splitnv($b, ":");
for (k,v in o) {
x = mapsum($*, {"b":v});
@ -306,7 +306,7 @@ while another is to loop over the same map from ``splitnv`` and use it (with ``p
.. code-block:: none
:emphasize-lines: 1-6
$ mlr --from data/nested.tsv --tsv put -q '
mlr --from data/nested.tsv --tsv put -q '
o = splitnv($b, ":");
for (k,v in o) {
x = mapsum($*, {"b":v}); emit x
@ -387,7 +387,7 @@ The idea here is that middles starting with a 1 belong to the outer value of 1,
.. code-block:: none
:emphasize-lines: 1-10
$ mlr --from data/rect.txt put -q '
mlr --from data/rect.txt put -q '
is_present($outer) {
unset @r
}

6
docs6/shell-commands.rst
View file

@ -11,7 +11,7 @@ The :ref:`reference-dsl-system` DSL function allows you to run a specific shell
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --opprint put '$o = system("echo hello world")' data/small
mlr --opprint put '$o = system("echo hello world")' data/small
a b i x y o
pan pan 1 0.3467901443380824 0.7268028627434533 hello world
eks pan 2 0.7586799647899636 0.5221511083334797 hello world
@ -22,7 +22,7 @@ The :ref:`reference-dsl-system` DSL function allows you to run a specific shell
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --opprint put '$o = system("echo {" . NR . "}")' data/small
mlr --opprint put '$o = system("echo {" . NR . "}")' data/small
a b i x y o
pan pan 1 0.3467901443380824 0.7268028627434533 {1}
eks pan 2 0.7586799647899636 0.5221511083334797 {2}
@ -33,7 +33,7 @@ The :ref:`reference-dsl-system` DSL function allows you to run a specific shell
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --opprint put '$o = system("echo -n ".$a."| sha1sum")' data/small
mlr --opprint put '$o = system("echo -n ".$a."| sha1sum")' data/small
a b i x y o
pan pan 1 0.3467901443380824 0.7268028627434533 f29c748220331c273ef16d5115f6ecd799947f13 -
eks pan 2 0.7586799647899636 0.5221511083334797 456d988ecb3bf1b75f057fc6e9fe70db464e9388 -

34
docs6/special-symbols-and-formatting.rst
View file

@ -12,7 +12,7 @@ How can I handle commas-as-data in various formats?
.. code-block:: none
:emphasize-lines: 1-1
$ cat commas.csv
cat commas.csv
Name,Role
"Xiao, Lin",administrator
"Khavari, Darius",tester
@ -22,7 +22,7 @@ Likewise :ref:`file-formats-json`:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --icsv --ojson cat commas.csv
mlr --icsv --ojson cat commas.csv
{
"Name": "Xiao, Lin",
"Role": "administrator"
@ -37,7 +37,7 @@ For Miller's :ref:`vertical-tabular format <file-formats-xtab>` there is no esca
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --icsv --oxtab cat commas.csv
mlr --icsv --oxtab cat commas.csv
Name Xiao, Lin
Role administrator
@ -49,7 +49,7 @@ But for :ref:`Key-value_pairs <file-formats-dkvp>` and :ref:`index-numbered <fil
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --icsv --odkvp cat commas.csv
mlr --icsv --odkvp cat commas.csv
Name=Xiao, Lin,Role=administrator
Name=Khavari, Darius,Role=tester
@ -58,7 +58,7 @@ One solution is to use a different delimiter, such as a pipe character:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --icsv --odkvp --ofs pipe cat commas.csv
mlr --icsv --odkvp --ofs pipe cat commas.csv
Name=Xiao, Lin|Role=administrator
Name=Khavari, Darius|Role=tester
@ -68,7 +68,7 @@ characters as delimiters -- here, control-A:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --icsv --odkvp --ofs '\001' cat commas.csv | cat -v
mlr --icsv --odkvp --ofs '\001' cat commas.csv | cat -v
Name=Xiao, Lin\001Role=administrator
Name=Khavari, Darius\001Role=tester
@ -80,7 +80,7 @@ Simply surround the field names with curly braces:
.. code-block:: none
:emphasize-lines: 1-1
$ echo 'x.a=3,y:b=4,z/c=5' | mlr put '${product.all} = ${x.a} * ${y:b} * ${z/c}'
echo 'x.a=3,y:b=4,z/c=5' | mlr put '${product.all} = ${x.a} * ${y:b} * ${z/c}'
x.a=3,y:b=4,z/c=5,product.all=60
How can I put single-quotes into strings?
@ -95,7 +95,7 @@ This is a little tricky due to the shell's handling of quotes. For simplicity, l
.. code-block:: none
:emphasize-lines: 1-1
$ echo a=bcd | mlr put -f data/single-quote-example.mlr
echo a=bcd | mlr put -f data/single-quote-example.mlr
a=It's OK, I said, then 'for now'.
So, it's simple: Miller's DSL uses double quotes for strings, and you can put single quotes (or backslash-escaped double-quotes) inside strings, no problem.
@ -105,7 +105,7 @@ Without putting the update expression in a file, it's messier:
.. code-block:: none
:emphasize-lines: 1-1
$ echo a=bcd | mlr put '$a="It'\''s OK, I said, '\''for now'\''."'
echo a=bcd | mlr put '$a="It'\''s OK, I said, '\''for now'\''."'
a=It's OK, I said, 'for now'.
The idea is that the outermost single-quotes are to protect the ``put`` expression from the shell, and the double quotes within them are for Miller. To get a single quote in the middle there, you need to actually put it *outside* the single-quoting for the shell. The pieces are the following, all concatenated together:
@ -126,19 +126,19 @@ One way is to use square brackets; an alternative is to use simple string-substi
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/question.dat
cat data/question.dat
a=is it?,b=it is!
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --oxtab put '$c = gsub($a, "[?]"," ...")' data/question.dat
mlr --oxtab put '$c = gsub($a, "[?]"," ...")' data/question.dat
a is it?
b it is!
c is it ...
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --oxtab put '$c = ssub($a, "?"," ...")' data/question.dat
mlr --oxtab put '$c = ssub($a, "?"," ...")' data/question.dat
a is it?
b it is!
c is it ...
@ -157,7 +157,7 @@ Within ``mlr put`` and ``mlr filter``, the default behavior for scanning input r
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/scan-example-1.tbl
cat data/scan-example-1.tbl
value
1
2.0
@ -167,7 +167,7 @@ Within ``mlr put`` and ``mlr filter``, the default behavior for scanning input r
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --pprint put '$copy = $value; $type = typeof($value)' data/scan-example-1.tbl
mlr --pprint put '$copy = $value; $type = typeof($value)' data/scan-example-1.tbl
value copy type
1 1 int
2.0 2.0 float
@ -187,7 +187,7 @@ But now suppose you have data like these:
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/scan-example-2.tbl
cat data/scan-example-2.tbl
value
0001
0002
@ -203,7 +203,7 @@ But now suppose you have data like these:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --pprint put '$copy = $value; $type = typeof($value)' data/scan-example-2.tbl
mlr --pprint put '$copy = $value; $type = typeof($value)' data/scan-example-2.tbl
value copy type
0001 0001 int
0002 0002 int
@ -233,7 +233,7 @@ The solution is to **use the -S flag** for ``mlr put`` and/or ``mlr filter``. Th
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --pprint put -S '$copy = $value; $type = typeof($value)' data/scan-example-2.tbl
mlr --pprint put -S '$copy = $value; $type = typeof($value)' data/scan-example-2.tbl
value copy type
0001 0001 int
0002 0002 int

18
docs6/then-chaining.rst
View file

@ -14,7 +14,7 @@ First, look at the input data:
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/then-example.csv
cat data/then-example.csv
Status,Payment_Type,Amount
paid,cash,10.00
pending,debit,20.00
@ -27,7 +27,7 @@ Next, run the first step of your command, omitting anything from the first ``the
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --icsv --opprint count-distinct -f Status,Payment_Type data/then-example.csv
mlr --icsv --opprint count-distinct -f Status,Payment_Type data/then-example.csv
Status Payment_Type count
paid cash 2
pending debit 1
@ -39,7 +39,7 @@ After that, run it with the next ``then`` step included:
.. code-block:: none
:emphasize-lines: 1-3
$ mlr --icsv --opprint count-distinct -f Status,Payment_Type \
mlr --icsv --opprint count-distinct -f Status,Payment_Type \
then sort -nr count \
data/then-example.csv
Status Payment_Type count
@ -55,7 +55,7 @@ Note, by the way, that you'll get the same results using pipes:
.. code-block:: none
:emphasize-lines: 1-2
$ mlr --csv count-distinct -f Status,Payment_Type data/then-example.csv \
mlr --csv count-distinct -f Status,Payment_Type data/then-example.csv \
| mlr --icsv --opprint sort -nr count
Status Payment_Type count
paid cash 2
@ -71,7 +71,7 @@ Given this input data:
.. code-block:: none
:emphasize-lines: 1-1
$ cat data/small
cat data/small
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
@ -83,7 +83,7 @@ why don't I see ``NR=1`` and ``NR=2`` here??
.. code-block:: none
:emphasize-lines: 1-1
$ mlr filter '$x > 0.5' then put '$NR = NR' data/small
mlr filter '$x > 0.5' then put '$NR = NR' data/small
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797,NR=2
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729,NR=5
@ -92,7 +92,7 @@ The reason is that ``NR`` is computed for the original input records and isn't d
.. code-block:: none
:emphasize-lines: 1-1
$ echo x=1,y=2,z=3 | mlr put '$nf1 = NF; $u = 4; $nf2 = NF; unset $x,$y,$z; $nf3 = NF'
echo x=1,y=2,z=3 | mlr put '$nf1 = NF; $u = 4; $nf2 = NF; unset $x,$y,$z; $nf3 = NF'
nf1=3,u=4,nf2=5,nf3=3
``NR``, by contrast (and ``FNR`` as well), retains the value from the original input stream, and records may be dropped by a ``filter`` within a ``then``-chain. To recover consecutive record numbers, you can use out-of-stream variables as follows:
@ -100,7 +100,7 @@ The reason is that ``NR`` is computed for the original input records and isn't d
.. code-block:: none
:emphasize-lines: 1-11
$ mlr --opprint --from data/small put '
mlr --opprint --from data/small put '
begin{ @nr1 = 0 }
@nr1 += 1;
$nr1 = @nr1
@ -120,6 +120,6 @@ Or, simply use ``mlr cat -n``:
.. code-block:: none
:emphasize-lines: 1-1
$ mlr filter '$x > 0.5' then cat -n data/small
mlr filter '$x > 0.5' then cat -n data/small
n=1,a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
n=2,a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729