Mirrors/miller
1
0
Fork 0
mirror of https://github.com/johnkerl/miller.git synced 2026-07-18 00:45:47 +00:00
Code Issues Projects Releases Wiki Activity

sphinx-neaten

Browse source
This commit is contained in:
John Kerl 2021-05-31 01:51:01 -04:00
parent 61808e9123
commit ab1731f7e7
72 changed files with 1627 additions and 1626 deletions
Download patch file Download diff file Expand all files Collapse all files

70
docs/10min.rst
View file

@ -9,7 +9,7 @@ CSV-file examples
Suppose you have this CSV data file:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat example.csv
@ -27,7 +27,7 @@ Suppose you have this CSV data file:
``mlr cat`` is like cat -- it passes the data through unmodified:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --csv cat example.csv
@ -45,7 +45,7 @@ Suppose you have this CSV data file:
but it can also do format conversion (here, you can pretty-print in tabular format):
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint cat example.csv
@ -63,7 +63,7 @@ but it can also do format conversion (here, you can pretty-print in tabular form
``mlr head`` and ``mlr tail`` count records rather than lines. Whethere you're getting the first few records or the last few, the CSV header is included either way:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --csv head -n 4 example.csv
@ -73,7 +73,7 @@ but it can also do format conversion (here, you can pretty-print in tabular form
red,circle,1,16,13.8103,2.9010
red,square,0,48,77.5542,7.4670
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --csv tail -n 4 example.csv
@ -85,7 +85,7 @@ but it can also do format conversion (here, you can pretty-print in tabular form
You can sort primarily alphabetically on one field, then secondarily numerically descending on another field:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint sort -f shape -nr index example.csv
@ -103,7 +103,7 @@ You can sort primarily alphabetically on one field, then secondarily numerically
You can use ``cut`` to retain only specified fields, in the same order they appeared in the input data:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint cut -f flag,shape example.csv
@ -121,7 +121,7 @@ You can use ``cut`` to retain only specified fields, in the same order they appe
You can also use ``cut -o`` to retain only specified fields in your preferred order:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint cut -o -f flag,shape example.csv
@ -139,7 +139,7 @@ You can also use ``cut -o`` to retain only specified fields in your preferred or
You can use ``cut -x`` to omit fields you don't care about:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint cut -x -f flag,shape example.csv
@ -157,7 +157,7 @@ You can use ``cut -x`` to omit fields you don't care about:
You can use ``filter`` to keep only records you care about:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint filter '$color == "red"' example.csv
@ -167,7 +167,7 @@ You can use ``filter`` to keep only records you care about:
red square 0 48 77.5542 7.4670
red square 0 64 77.1991 9.5310
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint filter '$color == "red" && $flag == 1' example.csv
@ -177,7 +177,7 @@ You can use ``filter`` to keep only records you care about:
You can use ``put`` to create new fields which are computed from other fields:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint put '$ratio = $quantity / $rate; $color_shape = $color . "_" . $shape' example.csv
@ -195,7 +195,7 @@ You can use ``put`` to create new fields which are computed from other fields:
Even though Miller's main selling point is name-indexing, sometimes you really want to refer to a field name by its positional index. Use ``$[[3]]`` to access the name of field 3 or ``$[[[3]]]`` to access the value of field 3:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint put '$[[3]] = "NEW"' example.csv
@ -211,7 +211,7 @@ Even though Miller's main selling point is name-indexing, sometimes you really w
yellow circle 1 87 63.5058 8.3350
purple square 0 91 72.3735 8.2430
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint put '$[[[3]]] = "NEW"' example.csv
@ -232,7 +232,7 @@ JSON-file examples
OK, CSV and pretty-print are fine. But Miller can also convert between a few other formats -- let's take a look at JSON output:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --ojson put '$ratio = $quantity/$rate; $shape = toupper($shape)' example.csv
@ -249,7 +249,7 @@ OK, CSV and pretty-print are fine. But Miller can also convert between a few oth
Or, JSON output with vertical-formatting flags:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --ojsonx tail -n 2 example.csv
@ -277,7 +277,7 @@ Now suppose you want to sort the data on a given column, *and then* take the top
Here are the records with the top three ``index`` values:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint sort -f shape -nr index then head -n 3 example.csv
@ -288,7 +288,7 @@ Here are the records with the top three ``index`` values:
Lots of Miller commands take a ``-g`` option for group-by: here, ``head -n 1 -g shape`` outputs the first record for each distinct value of the ``shape`` field. This means we're finding the record with highest ``index`` field for each distinct ``shape`` field:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint sort -f shape -nr index then head -n 1 -g shape example.csv
@ -299,7 +299,7 @@ Lots of Miller commands take a ``-g`` option for group-by: here, ``head -n 1 -g
Statistics can be computed with or without group-by field(s):
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint --from example.csv stats1 -a count,min,mean,max -f quantity -g shape
@ -308,7 +308,7 @@ Statistics can be computed with or without group-by field(s):
square 4 72.373500 76.601150 79.277800
circle 3 13.810300 47.098200 63.978500
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint --from example.csv stats1 -a count,min,mean,max -f quantity -g shape,color
@ -322,7 +322,7 @@ Statistics can be computed with or without group-by field(s):
If your output has a lot of columns, you can use XTAB format to line things up vertically for you instead:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --oxtab --from example.csv stats1 -a p0,p10,p25,p50,p75,p90,p99,p100 -f rate
@ -344,14 +344,14 @@ Often we want to print output to the screen. Miller does this by default, as we'
Sometimes we want to print output to another file: just use **> outputfilenamegoeshere** at the end of your command:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
% mlr --icsv --opprint cat example.csv > newfile.csv
# Output goes to the new file;
# nothing is printed to the screen.
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
% cat newfile.csv
@ -369,12 +369,12 @@ Sometimes we want to print output to another file: just use **> outputfilenamego
Other times we just want our files to be **changed in-place**: just use **mlr -I**:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
% cp example.csv newfile.txt
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
% cat newfile.txt
@ -390,12 +390,12 @@ Other times we just want our files to be **changed in-place**: just use **mlr -I
yellow,circle,1,87,63.5058,8.3350
purple,square,0,91,72.3735,8.2430
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
% mlr -I --icsv --opprint cat newfile.txt
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
% cat newfile.txt
@ -413,7 +413,7 @@ Other times we just want our files to be **changed in-place**: just use **mlr -I
Also using ``mlr -I`` you can bulk-operate on lots of files: e.g.:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
mlr -I --csv cut -x -f unwanted_column_name *.csv
@ -422,12 +422,12 @@ If you like, you can first copy off your original data somewhere else, before do
Lastly, using ``tee`` within ``put``, you can split your input data into separate files per one or more field names:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --csv --from example.csv put -q 'tee > $shape.".csv", $*'
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat circle.csv
@ -436,7 +436,7 @@ Lastly, using ``tee`` within ``put``, you can split your input data into separat
yellow,circle,1,73,63.9785,4.2370
yellow,circle,1,87,63.5058,8.3350
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat square.csv
@ -446,7 +446,7 @@ Lastly, using ``tee`` within ``put``, you can split your input data into separat
red,square,0,64,77.1991,9.5310
purple,square,0,91,72.3735,8.2430
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat triangle.csv
@ -462,7 +462,7 @@ What's a CSV file, really? It's an array of rows, or *records*, each being a lis
For example, if you have:
.. code-block:: bash
.. code-block:: none
shape,flag,index
circle,1,24
@ -470,7 +470,7 @@ For example, if you have:
then that's a way of saying:
.. code-block:: bash
.. code-block:: none
shape=circle,flag=1,index=24
shape=square,flag=0,index=36
@ -479,7 +479,7 @@ Data written this way are called **DKVP**, for *delimited key-value pairs*.
We've also already seen other ways to write the same data:
.. code-block:: bash
.. code-block:: none
CSV PPRINT JSON
shape,flag,index shape flag index [

20
docs/10min.rst.in
View file

@ -97,14 +97,14 @@ Often we want to print output to the screen. Miller does this by default, as we'
Sometimes we want to print output to another file: just use **> outputfilenamegoeshere** at the end of your command:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
% mlr --icsv --opprint cat example.csv > newfile.csv
# Output goes to the new file;
# nothing is printed to the screen.
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
% cat newfile.csv
@ -122,12 +122,12 @@ Sometimes we want to print output to another file: just use **> outputfilenamego
Other times we just want our files to be **changed in-place**: just use **mlr -I**:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
% cp example.csv newfile.txt
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
% cat newfile.txt
@ -143,12 +143,12 @@ Other times we just want our files to be **changed in-place**: just use **mlr -I
yellow,circle,1,87,63.5058,8.3350
purple,square,0,91,72.3735,8.2430
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
% mlr -I --icsv --opprint cat newfile.txt
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
% cat newfile.txt
@ -166,7 +166,7 @@ Other times we just want our files to be **changed in-place**: just use **mlr -I
Also using ``mlr -I`` you can bulk-operate on lots of files: e.g.:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
mlr -I --csv cut -x -f unwanted_column_name *.csv
@ -190,7 +190,7 @@ What's a CSV file, really? It's an array of rows, or *records*, each being a lis
For example, if you have:
.. code-block:: bash
.. code-block:: none
shape,flag,index
circle,1,24
@ -198,7 +198,7 @@ For example, if you have:
then that's a way of saying:
.. code-block:: bash
.. code-block:: none
shape=circle,flag=1,index=24
shape=square,flag=0,index=36
@ -207,7 +207,7 @@ Data written this way are called **DKVP**, for *delimited key-value pairs*.
We've also already seen other ways to write the same data:
.. code-block:: bash
.. code-block:: none
CSV PPRINT JSON
shape,flag,index shape flag index [

2
docs/Makefile
View file

@ -24,5 +24,5 @@ help:
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
#### temp ./genrst
./genrst
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

8
docs/build.rst
View file

@ -72,7 +72,7 @@ Miller has been built on Windows using MSYS2: http://www.msys2.org/. You can in
You will first need to install MSYS2: http://www.msys2.org/. Then, start an MSYS2 shell, e.g. (supposing you installed MSYS2 to ``C:\msys2\``) run ``C:\msys2\mingw64.exe``. Within the MSYS2 shell, you can run the following to install dependent packages:
.. code-block:: bash
.. code-block:: none
pacman -Syu
pacman -Su
@ -90,13 +90,13 @@ There is a unit-test false-negative issue involving the semantics of the ``mkste
Within MSYS2 you can run ``mlr``: simply copy it from the ``c`` subdirectory to your desired location somewhere within your MSYS2 ``$PATH``. To run ``mlr`` outside of MSYS2, just as with precompiled binaries as described above, you'll need ``msys-2.0.dll``. One way to do this is to augment your path:
.. code-block:: bash
.. code-block:: none
C:\> set PATH=%PATH%;\msys64\mingw64\bin
Another way to do it is to copy the Miller executable and the DLL to the same directory:
.. code-block:: bash
.. code-block:: none
C:\> mkdir \mbin
C:\> copy \msys64\mingw64\bin\msys-2.0.dll \mbin
@ -181,7 +181,7 @@ In this example I am using version 3.4.0; of course that will change for subsequ
* Similarly for ``macports``: https://github.com/macports/macports-ports/blob/master/textproc/miller/Portfile.
* Social-media updates.
.. code-block:: bash
.. code-block:: none
git remote add upstream https://github.com/Homebrew/homebrew-core # one-time setup only
git fetch upstream

8
docs/build.rst.in
View file

@ -69,7 +69,7 @@ Miller has been built on Windows using MSYS2: http://www.msys2.org/. You can in
You will first need to install MSYS2: http://www.msys2.org/. Then, start an MSYS2 shell, e.g. (supposing you installed MSYS2 to ``C:\msys2\``) run ``C:\msys2\mingw64.exe``. Within the MSYS2 shell, you can run the following to install dependent packages:
.. code-block:: bash
.. code-block:: none
pacman -Syu
pacman -Su
@ -87,13 +87,13 @@ There is a unit-test false-negative issue involving the semantics of the ``mkste
Within MSYS2 you can run ``mlr``: simply copy it from the ``c`` subdirectory to your desired location somewhere within your MSYS2 ``$PATH``. To run ``mlr`` outside of MSYS2, just as with precompiled binaries as described above, you'll need ``msys-2.0.dll``. One way to do this is to augment your path:
.. code-block:: bash
.. code-block:: none
C:\> set PATH=%PATH%;\msys64\mingw64\bin
Another way to do it is to copy the Miller executable and the DLL to the same directory:
.. code-block:: bash
.. code-block:: none
C:\> mkdir \mbin
C:\> copy \msys64\mingw64\bin\msys-2.0.dll \mbin
@ -178,7 +178,7 @@ In this example I am using version 3.4.0; of course that will change for subsequ
* Similarly for ``macports``: https://github.com/macports/macports-ports/blob/master/textproc/miller/Portfile.
* Social-media updates.
.. code-block:: bash
.. code-block:: none
git remote add upstream https://github.com/Homebrew/homebrew-core # one-time setup only
git fetch upstream

156
docs/cookbook.rst
View file

@ -9,7 +9,7 @@ Headerless CSV on input or output
Sometimes we get CSV files which lack a header. For example:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/headerless.csv
@ -20,7 +20,7 @@ Sometimes we get CSV files which lack a header. For example:
You can use Miller to add a header. The ``--implicit-csv-header`` applies positionally indexed labels:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --csv --implicit-csv-header cat data/headerless.csv
@ -32,7 +32,7 @@ You can use Miller to add a header. The ``--implicit-csv-header`` applies positi
Following that, you can rename the positionally indexed labels to names with meaning for your context. For example:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --csv --implicit-csv-header label name,age,status data/headerless.csv
@ -44,7 +44,7 @@ Following that, you can rename the positionally indexed labels to names with mea
Likewise, if you need to produce CSV which is lacking its header, you can pipe Miller's output to the system command ``sed 1d``, or you can use Miller's ``--headerless-csv-output`` option:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ head -5 data/colored-shapes.dkvp | mlr --ocsv cat
@ -55,7 +55,7 @@ Likewise, if you need to produce CSV which is lacking its header, you can pipe M
red,square,0,48,0.9562743938458542,0.7467203085342884,0.7755423050923582,7.117831369597269
purple,triangle,0,51,0.4355354501763202,0.8591292672156728,0.8122903963006748,5.753094629505863
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ head -5 data/colored-shapes.dkvp | mlr --ocsv --headerless-csv-output cat
@ -67,7 +67,7 @@ Likewise, if you need to produce CSV which is lacking its header, you can pipe M
Lastly, often we say "CSV" or "TSV" when we have positionally indexed data in columns which are separated by commas or tabs, respectively. In this case it's perhaps simpler to **just use NIDX format** which was designed for this purpose. (See also :doc:`file-formats`.) For example:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --inidx --ifs comma --oxtab cut -f 1,3 data/headerless.csv
@ -88,7 +88,7 @@ Doing multiple joins
Suppose we have the following data:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat multi-join/input.csv
@ -104,7 +104,7 @@ Suppose we have the following data:
And we want to augment the ``id`` column with lookups from the following data files:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat multi-join/name-lookup.csv
@ -113,7 +113,7 @@ And we want to augment the ``id`` column with lookups from the following data fi
10,Bob
20,Carol
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat multi-join/status-lookup.csv
@ -124,7 +124,7 @@ And we want to augment the ``id`` column with lookups from the following data fi
We can run the input file through multiple ``join`` commands in a ``then``-chain:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint join -f multi-join/name-lookup.csv -j id then join -f multi-join/status-lookup.csv -j id multi-join/input.csv
@ -143,7 +143,7 @@ Bulk rename of fields
Suppose you want to replace spaces with underscores in your column names:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/spaces.csv
@ -154,7 +154,7 @@ Suppose you want to replace spaces with underscores in your column names:
The simplest way is to use ``mlr rename`` with ``-g`` (for global replace, not just first occurrence of space within each field) and ``-r`` for pattern-matching (rather than explicit single-column renames):
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --csv rename -g -r ' ,_' data/spaces.csv
@ -163,7 +163,7 @@ The simplest way is to use ``mlr rename`` with ``-g`` (for global replace, not j
2468,1357,3579
9987,3312,4543
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --csv --opprint rename -g -r ' ,_' data/spaces.csv
@ -174,7 +174,7 @@ The simplest way is to use ``mlr rename`` with ``-g`` (for global replace, not j
You can also do this with a for-loop:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/bulk-rename-for-loop.mlr
@ -184,7 +184,7 @@ You can also do this with a for-loop:
}
$* = newrec
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint put -f data/bulk-rename-for-loop.mlr data/spaces.csv
@ -198,7 +198,7 @@ Search-and-replace over all fields
How to do ``$name = gsub($name, "old", "new")`` for all fields?
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/sar.csv
@ -206,7 +206,7 @@ How to do ``$name = gsub($name, "old", "new")`` for all fields?
the quick,brown fox,jumped
over,the,lazy dogs
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/sar.mlr
@ -214,7 +214,7 @@ How to do ``$name = gsub($name, "old", "new")`` for all fields?
$[k] = gsub($[k], "e", "X");
}
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --csv put -f data/sar.mlr data/sar.csv
@ -227,7 +227,7 @@ Full field renames and reassigns
Using Miller 5.0.0's map literals and assigning to ``$*``, you can fully generalize :ref:`mlr rename <reference-verbs-rename>`, :ref:`mlr reorder <reference-verbs-reorder>`, etc.
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/small
@ -237,7 +237,7 @@ Using Miller 5.0.0's map literals and assigning to ``$*``, you can fully general
a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr put '
@ -266,7 +266,7 @@ Numbering and renumbering records
The ``awk``-like built-in variable ``NR`` is incremented for each input record:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/small
@ -276,7 +276,7 @@ The ``awk``-like built-in variable ``NR`` is incremented for each input record:
a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr put '$nr = NR' data/small
@ -288,7 +288,7 @@ The ``awk``-like built-in variable ``NR`` is incremented for each input record:
However, this is the record number within the original input stream -- not after any filtering you may have done:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr filter '$a == "wye"' then put '$nr = NR' data/small
@ -297,7 +297,7 @@ However, this is the record number within the original input stream -- not after
There are two good options here. One is to use the ``cat`` verb with ``-n``:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr filter '$a == "wye"' then cat -n data/small
@ -306,7 +306,7 @@ There are two good options here. One is to use the ``cat`` verb with ``-n``:
The other is to keep your own counter within the ``put`` DSL:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr filter '$a == "wye"' then put 'begin {@n = 1} $n = @n; @n += 1' data/small
@ -329,7 +329,7 @@ Data-cleaning examples
Here are some ways to use the type-checking options as described in :ref:`reference-dsl-type-tests-and-assertions` Suppose you have the following data file, with inconsistent typing for boolean. (Also imagine that, for the sake of discussion, we have a million-line file rather than a four-line file, so we can't see it all at once and some automation is called for.)
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/het-bool.csv
@ -341,7 +341,7 @@ Here are some ways to use the type-checking options as described in :ref:`refere
One option is to coerce everything to boolean, or integer:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint put '$reachable = boolean($reachable)' data/het-bool.csv
@ -351,7 +351,7 @@ One option is to coerce everything to boolean, or integer:
fred true
wilma true
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint put '$reachable = int(boolean($reachable))' data/het-bool.csv
@ -363,7 +363,7 @@ One option is to coerce everything to boolean, or integer:
A second option is to flag badly formatted data within the output stream:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint put '$format_ok = is_string($reachable)' data/het-bool.csv
@ -375,7 +375,7 @@ A second option is to flag badly formatted data within the output stream:
Or perhaps to flag badly formatted data outside the output stream:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint put 'if (!is_string($reachable)) {eprint "Malformed at NR=".NR} ' data/het-bool.csv
@ -388,7 +388,7 @@ Or perhaps to flag badly formatted data outside the output stream:
A third way is to abort the process on first instance of bad data:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --csv put '$reachable = asserting_string($reachable)' data/het-bool.csv
@ -403,7 +403,7 @@ Splitting nested fields
Suppose you have a TSV file like this:
.. code-block:: bash
.. code-block:: none
a b
x z
@ -411,7 +411,7 @@ Suppose you have a TSV file like this:
The simplest option is to use :ref:`mlr nest <reference-verbs-nest>`:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --tsv nest --explode --values --across-records -f b --nested-fs : data/nested.tsv
@ -421,7 +421,7 @@ The simplest option is to use :ref:`mlr nest <reference-verbs-nest>`:
s v
s w
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --tsv nest --explode --values --across-fields -f b --nested-fs : data/nested.tsv
@ -435,7 +435,7 @@ While ``mlr nest`` is simplest, let's also take a look at a few ways to do this
One option to split out the colon-delimited values in the ``b`` column is to use ``splitnv`` to create an integer-indexed map and loop over it, adding new fields to the current record:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from data/nested.tsv --itsv --oxtab put 'o=splitnv($b, ":"); for (k,v in o) {$["p".k]=v}'
@ -451,7 +451,7 @@ One option to split out the colon-delimited values in the ``b`` column is to use
while another is to loop over the same map from ``splitnv`` and use it (with ``put -q`` to suppress printing the original record) to produce multiple records:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from data/nested.tsv --itsv --oxtab put -q 'o=splitnv($b, ":"); for (k,v in o) {emit mapsum($*, {"b":v})}'
@ -467,7 +467,7 @@ while another is to loop over the same map from ``splitnv`` and use it (with ``p
a s
b w
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from data/nested.tsv --tsv put -q 'o=splitnv($b, ":"); for (k,v in o) {emit mapsum($*, {"b":v})}'
@ -482,7 +482,7 @@ Showing differences between successive queries
Suppose you have a database query which you run at one point in time, producing the output on the left, then again later producing the output on the right:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/previous_counters.csv
@ -492,7 +492,7 @@ Suppose you have a database query which you run at one point in time, producing
orange,694
purple,12
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/current_counters.csv
@ -506,12 +506,12 @@ And, suppose you want to compute the differences in the counters between adjacen
First, rename counter columns to make them distinct:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --csv rename count,previous_count data/previous_counters.csv > data/prevtemp.csv
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/prevtemp.csv
@ -521,12 +521,12 @@ First, rename counter columns to make them distinct:
orange,694
purple,12
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --csv rename count,current_count data/current_counters.csv > data/currtemp.csv
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/currtemp.csv
@ -538,7 +538,7 @@ First, rename counter columns to make them distinct:
Then, join on the key field(s), and use unsparsify to zero-fill counters absent on one side but present on the other. Use ``--ul`` and ``--ur`` to emit unpaired records (namely, purple on the left and yellow on the right):
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint \
@ -558,7 +558,7 @@ Finding missing dates
Suppose you have some date-stamped data which may (or may not) be missing entries for one or more dates:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ head -n 10 data/miss-date.csv
@ -573,7 +573,7 @@ Suppose you have some date-stamped data which may (or may not) be missing entrie
2012-03-12,11043
2012-03-13,11177
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ wc -l data/miss-date.csv
@ -581,7 +581,7 @@ Suppose you have some date-stamped data which may (or may not) be missing entrie
Since there are 1372 lines in the data file, some automation is called for. To find the missing dates, you can convert the dates to seconds since the epoch using ``strptime``, then compute adjacent differences (the ``cat -n`` simply inserts record-counters):
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from data/miss-date.csv --icsv \
@ -602,7 +602,7 @@ Since there are 1372 lines in the data file, some automation is called for. To f
Then, filter for adjacent difference not being 86400 (the number of seconds in a day):
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from data/miss-date.csv --icsv \
@ -615,7 +615,7 @@ Then, filter for adjacent difference not being 86400 (the number of seconds in a
Given this, it's now easy to see where the gaps are:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr cat -n then filter '$n >= 770 && $n <= 780' data/miss-date.csv
@ -631,7 +631,7 @@ Given this, it's now easy to see where the gaps are:
n=779,1=2014-04-23,2=130849
n=780,1=2014-04-24,2=131026
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr cat -n then filter '$n >= 1115 && $n <= 1125' data/miss-date.csv
@ -657,7 +657,7 @@ Two-pass algorithms: computation of percentages
For example, mapping numeric values down a column to the percentage between their min and max values is two-pass: on the first pass you find the min and max values, then on the second, map each record's value to a percentage.
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from data/small --opprint put -q '
@ -688,7 +688,7 @@ Two-pass algorithms: line-number ratios
Similarly, finding the total record count requires first reading through all the data:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint --from data/small put -q '
@ -714,7 +714,7 @@ Two-pass algorithms: records having max value
The idea is to retain records having the largest value of ``n`` in the following data:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --itsv --opprint cat data/maxrows.tsv
@ -753,7 +753,7 @@ The idea is to retain records having the largest value of ``n`` in the following
Of course, the largest value of ``n`` isn't known until after all data have been read. Using an out-of-stream variable we can retain all records as they are read, then filter them at the end:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/maxrows.mlr
@ -773,7 +773,7 @@ Of course, the largest value of ``n`` isn't known until after all data have been
}
}
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --itsv --opprint put -q -f data/maxrows.mlr data/maxrows.tsv
@ -793,7 +793,7 @@ Rectangularizing data
Suppose you have a method (in whatever language) which is printing things of the form
.. code-block:: bash
.. code-block:: none
outer=1
outer=2
@ -801,7 +801,7 @@ Suppose you have a method (in whatever language) which is printing things of the
and then calls another method which prints things of the form
.. code-block:: bash
.. code-block:: none
middle=10
middle=11
@ -813,7 +813,7 @@ and then calls another method which prints things of the form
and then, perhaps, that second method calls a third method which prints things of the form
.. code-block:: bash
.. code-block:: none
inner1=100,inner2=101
inner1=120,inner2=121
@ -825,7 +825,7 @@ and then, perhaps, that second method calls a third method which prints things o
with the result that your program's output is
.. code-block:: bash
.. code-block:: none
outer=1
middle=10
@ -847,7 +847,7 @@ with the result that your program's output is
The idea here is that middles starting with a 1 belong to the outer value of 1, and so on. (For example, the outer values might be account IDs, the middle values might be invoice IDs, and the inner values might be invoice line-items.) If you want all the middle and inner lines to have the context of which outers they belong to, you can modify your software to pass all those through your methods. Alternatively, don't refactor your code just to handle some ad-hoc log-data formatting -- instead, use the following to rectangularize the data. The idea is to use an out-of-stream variable to accumulate fields across records. Clear that variable when you see an outer ID; accumulate fields; emit output when you see the inner IDs.
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from data/rect.txt put -q '
@ -873,7 +873,7 @@ Regularizing ragged CSV
Miller handles compliant CSV: in particular, it's an error if the number of data fields in a given data line don't match the number of header lines. But in the event that you have a CSV file in which some lines have less than the full number of fields, you can use Miller to pad them out. The trick is to use NIDX format, for which each line stands on its own without respect to a header line.
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/ragged.csv
@ -882,7 +882,7 @@ Miller handles compliant CSV: in particular, it's an error if the number of data
4,5
6,7,8,9
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from data/ragged.csv --fs comma --nidx put '
@ -900,7 +900,7 @@ Miller handles compliant CSV: in particular, it's an error if the number of data
or, more simply,
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from data/ragged.csv --fs comma --nidx put '
@ -919,7 +919,7 @@ Feature-counting
Suppose you have some heterogeneous data like this:
.. code-block:: bash
.. code-block:: none
{ "qoh": 29874, "rate": 1.68, "latency": 0.02 }
{ "name": "alice", "uid": 572 }
@ -936,7 +936,7 @@ Suppose you have some heterogeneous data like this:
A reasonable question to ask is, how many occurrences of each field are there? And, what percentage of total row count has each of them? Since the denominator of the percentage is not known until the end, this is a two-pass algorithm:
.. code-block:: bash
.. code-block:: none
for (key in $*) {
@key_counts[key] += 1;
@ -954,7 +954,7 @@ A reasonable question to ask is, how many occurrences of each field are there? A
Then
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --json put -q -f data/feature-count.mlr data/features.json
@ -972,7 +972,7 @@ Then
{ "key": "uid", "key_fraction": 0.250000 }
{ "key": "uid2", "key_fraction": 0.083333 }
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --ijson --opprint put -q -f data/feature-count.mlr data/features.json
@ -1002,7 +1002,7 @@ The previous section discussed how to fill out missing data fields within CSV wi
For example, suppose you have JSON input like this:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/sparse.json
@ -1013,7 +1013,7 @@ For example, suppose you have JSON input like this:
There are field names ``a``, ``b``, ``v``, ``u``, ``x``, ``w`` in the data -- but not all in every record. Since we don't know the names of all the keys until we've read them all, this needs to be a two-pass algorithm. On the first pass, remember all the unique key names and all the records; on the second pass, loop through the records filling in absent values, then producing output. Use ``put -q`` since we don't want to produce per-record output, only emitting output in the ``end`` block:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/unsparsify.mlr
@ -1044,7 +1044,7 @@ There are field names ``a``, ``b``, ``v``, ``u``, ``x``, ``w`` in the data -- bu
}
}
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --json put -q -f data/unsparsify.mlr data/sparse.json
@ -1053,7 +1053,7 @@ There are field names ``a``, ``b``, ``v``, ``u``, ``x``, ``w`` in the data -- bu
{ "a": 1, "b": "", "v": 2, "u": "", "x": 3, "w": "" }
{ "a": "", "b": "", "v": 1, "u": "", "x": "", "w": 2 }
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --ijson --ocsv put -q -f data/unsparsify.mlr data/sparse.json
@ -1063,7 +1063,7 @@ There are field names ``a``, ``b``, ``v``, ``u``, ``x``, ``w`` in the data -- bu
1,,2,,3,
,,1,,,2
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --ijson --opprint put -q -f data/unsparsify.mlr data/sparse.json
@ -1080,13 +1080,13 @@ Parsing log-file output
This, of course, depends highly on what's in your log files. But, as an example, suppose you have log-file lines such as
.. code-block:: bash
.. code-block:: none
2015-10-08 08:29:09,445 INFO com.company.path.to.ClassName @ [sometext] various/sorts/of data {& punctuation} hits=1 status=0 time=2.378
I prefer to pre-filter with ``grep`` and/or ``sed`` to extract the structured text, then hand that to Miller. Example:
.. code-block:: bash
.. code-block:: none
grep 'various sorts' *.log | sed 's/.*} //' | mlr --fs space --repifs --oxtab stats1 -a min,p10,p50,p90,max -f time -g status
@ -1097,7 +1097,7 @@ Memoization with out-of-stream variables
The recursive function for the Fibonacci sequence is famous for its computational complexity. Namely, using *f*(0)=1, *f*(1)=1, *f*(*n*)=*f*(*n*-1)+*f*(*n*-2) for *n*&ge;2, the evaluation tree branches left as well as right at each non-trivial level, resulting in millions or more paths to the root 0/1 nodes for larger *n*. This program
.. code-block:: bash
.. code-block:: none
mlr --ofmt '%.9lf' --opprint seqgen --start 1 --stop 28 then put '
func f(n) {
@ -1118,7 +1118,7 @@ The recursive function for the Fibonacci sequence is famous for its computationa
produces output like this:
.. code-block:: bash
.. code-block:: none
i o fcount seconds_delta
1 1 1 0
@ -1152,7 +1152,7 @@ produces output like this:
Note that the time it takes to evaluate the function is blowing up exponentially as the input argument increases. Using ``@``-variables, which persist across records, we can cache and reuse the results of previous computations:
.. code-block:: bash
.. code-block:: none
mlr --ofmt '%.9lf' --opprint seqgen --start 1 --stop 28 then put '
func f(n) {
@ -1175,7 +1175,7 @@ Note that the time it takes to evaluate the function is blowing up exponentially
with output like this:
.. code-block:: bash
.. code-block:: none
i o fcount seconds_delta
1 1 1 0

8
docs/cookbook.rst.in
View file

@ -323,13 +323,13 @@ Parsing log-file output
This, of course, depends highly on what's in your log files. But, as an example, suppose you have log-file lines such as
.. code-block:: bash
.. code-block:: none
2015-10-08 08:29:09,445 INFO com.company.path.to.ClassName @ [sometext] various/sorts/of data {& punctuation} hits=1 status=0 time=2.378
I prefer to pre-filter with ``grep`` and/or ``sed`` to extract the structured text, then hand that to Miller. Example:
.. code-block:: bash
.. code-block:: none
grep 'various sorts' *.log | sed 's/.*} //' | mlr --fs space --repifs --oxtab stats1 -a min,p10,p50,p90,max -f time -g status
@ -344,7 +344,7 @@ POKI_INCLUDE_ESCAPED(data/fibo-uncached.sh)HERE
produces output like this:
.. code-block:: bash
.. code-block:: none
i o fcount seconds_delta
1 1 1 0
@ -382,7 +382,7 @@ POKI_INCLUDE_ESCAPED(data/fibo-cached.sh)HERE
with output like this:
.. code-block:: bash
.. code-block:: none
i o fcount seconds_delta
1 1 1 0

28
docs/cookbook2.rst
View file

@ -9,7 +9,7 @@ Randomly selecting words from a list
Given this `word list <https://github.com/johnkerl/miller/blob/master/docs/data/english-words.txt>`_, first take a look to see what the first few lines look like:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ head data/english-words.txt
@ -26,7 +26,7 @@ Given this `word list <https://github.com/johnkerl/miller/blob/master/docs/data/
Then the following will randomly sample ten words with four to eight characters in them:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from data/english-words.txt --nidx filter -S 'n=strlen($1);4<=n&&n<=8' then sample -k 10
@ -48,7 +48,7 @@ These are simple *n*-grams as `described here <http://johnkerl.org/randspell/ran
The idea is that words from the input file are consumed, then taken apart and pasted back together in ways which imitate the letter-to-letter transitions found in the word list -- giving us automatically generated words in the same vein as *bromance* and *spork*:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --nidx --from ./ngrams/gsl-2000.txt put -q -f ./ngrams/ngfuncs.mlr -f ./ngrams/ng5.mlr
@ -74,7 +74,7 @@ Program timing
This admittedly artificial example demonstrates using Miller time and stats functions to introspectively acquire some information about Miller's own runtime. The ``delta`` function computes the difference between successive timestamps.
.. code-block:: bash
.. code-block:: none
$ ruby -e '10000.times{|i|puts "i=#{i+1}"}' > lines.txt
@ -109,7 +109,7 @@ Computing interquartile ranges
For one or more specified field names, simply compute p25 and p75, then write the IQR as the difference of p75 and p25:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --oxtab stats1 -f x -a p25,p75 \
@ -121,7 +121,7 @@ For one or more specified field names, simply compute p25 and p75, then write th
For wildcarded field names, first compute p25 and p75, then loop over field names with ``p25`` in them:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --oxtab stats1 --fr '[i-z]' -a p25,p75 \
@ -146,7 +146,7 @@ Computing weighted means
This might be more elegantly implemented as an option within the ``stats1`` verb. Meanwhile, it's expressible within the DSL:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from data/medium put -q '
@ -184,7 +184,7 @@ Generating random numbers from various distributions
Here we can chain together a few simple building blocks:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat expo-sample.sh
@ -227,7 +227,7 @@ Namely:
The output is as follows:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ sh expo-sample.sh
@ -288,7 +288,7 @@ Sieve of Eratosthenes
The `Sieve of Eratosthenes <http://en.wikipedia.org/wiki/Sieve_of_Eratosthenes>`_ is a standard introductory programming topic. The idea is to find all primes up to some *N* by making a list of the numbers 1 to *N*, then striking out all multiples of 2 except 2 itself, all multiples of 3 except 3 itself, all multiples of 4 except 4 itself, and so on. Whatever survives that without getting marked is a prime. This is easy enough in Miller. Notice that here all the work is in ``begin`` and ``end`` statements; there is no file input (so we use ``mlr -n`` to keep Miller from waiting for input data).
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat programs/sieve.mlr
@ -325,7 +325,7 @@ The `Sieve of Eratosthenes <http://en.wikipedia.org/wiki/Sieve_of_Eratosthenes>`
}
}
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr -n put -f programs/sieve.mlr
@ -362,7 +362,7 @@ The `Mandelbrot set <http://en.wikipedia.org/wiki/Mandelbrot_set>`_ is also easi
The (approximate) computation of points in the complex plane which are and aren't members is just a few lines of complex arithmetic (see the Wikipedia article); how to render them is another task. Using graphics libraries you can create PNG or JPEG files, but another fun way to do this is by printing various characters to the screen:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat programs/mand.mlr
@ -469,7 +469,7 @@ The (approximate) computation of points in the complex plane which are and aren'
At standard resolution this makes a nice little ASCII plot:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr -n put -f ./programs/mand.mlr
@ -526,7 +526,7 @@ At standard resolution this makes a nice little ASCII plot:
But using a very small font size (as small as my Mac will let me go), and by choosing the coordinates to zoom in on a particular part of the complex plane, we can get a nice little picture:
.. code-block:: bash
.. code-block:: none
#!/bin/bash
# Get the number of rows and columns from the terminal window dimensions

8
docs/cookbook2.rst.in
View file

@ -6,7 +6,7 @@ Randomly selecting words from a list
Given this `word list <https://github.com/johnkerl/miller/blob/master/docs/data/english-words.txt>`_, first take a look to see what the first few lines look like:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ head data/english-words.txt
@ -23,7 +23,7 @@ Given this `word list <https://github.com/johnkerl/miller/blob/master/docs/data/
Then the following will randomly sample ten words with four to eight characters in them:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from data/english-words.txt --nidx filter -S 'n=strlen($1);4<=n&&n<=8' then sample -k 10
@ -45,7 +45,7 @@ These are simple *n*-grams as `described here <http://johnkerl.org/randspell/ran
The idea is that words from the input file are consumed, then taken apart and pasted back together in ways which imitate the letter-to-letter transitions found in the word list -- giving us automatically generated words in the same vein as *bromance* and *spork*:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --nidx --from ./ngrams/gsl-2000.txt put -q -f ./ngrams/ngfuncs.mlr -f ./ngrams/ng5.mlr
@ -135,7 +135,7 @@ POKI_RUN_COMMAND{{mlr -n put -f ./programs/mand.mlr}}HERE
But using a very small font size (as small as my Mac will let me go), and by choosing the coordinates to zoom in on a particular part of the complex plane, we can get a nice little picture:
.. code-block:: bash
.. code-block:: none
#!/bin/bash
# Get the number of rows and columns from the terminal window dimensions

44
docs/cookbook3.rst
View file

@ -9,7 +9,7 @@ Overview
One of Miller's strengths is its compact notation: for example, given input of the form
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ head -n 5 ../data/medium
@ -21,7 +21,7 @@ One of Miller's strengths is its compact notation: for example, given input of t
you can simply do
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --oxtab stats1 -a sum -f x ../data/medium
@ -29,7 +29,7 @@ you can simply do
or
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint stats1 -a sum -f x -g b ../data/medium
@ -42,7 +42,7 @@ or
rather than the more tedious
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --oxtab put -q '
@ -55,7 +55,7 @@ rather than the more tedious
or
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint put -q '
@ -80,14 +80,14 @@ The following examples compute some things using oosvars which are already compu
Mean without/with oosvars
----------------------------------------------------------------
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint stats1 -a mean -f x data/medium
x_mean
0.498602
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint put -q '
@ -104,7 +104,7 @@ Mean without/with oosvars
Keyed mean without/with oosvars
----------------------------------------------------------------
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint stats1 -a mean -f x -g a,b data/medium
@ -135,7 +135,7 @@ Keyed mean without/with oosvars
eks hat 0.500679
wye eks 0.530604
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint put -q '
@ -178,7 +178,7 @@ Keyed mean without/with oosvars
Variance and standard deviation without/with oosvars
----------------------------------------------------------------
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --oxtab stats1 -a count,sum,mean,var,stddev -f x data/medium
@ -188,7 +188,7 @@ Variance and standard deviation without/with oosvars
x_var 0.084270
x_stddev 0.290293
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat variance.mlr
@ -202,7 +202,7 @@ Variance and standard deviation without/with oosvars
emitf @n, @sumx, @sumx2, @mean, @var, @stddev
}
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --oxtab put -q -f variance.mlr data/medium
@ -218,14 +218,14 @@ You can also do this keyed, of course, imitating the keyed-mean example above.
Min/max without/with oosvars
----------------------------------------------------------------
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --oxtab stats1 -a min,max -f x data/medium
x_min 0.000045
x_max 0.999953
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --oxtab put -q '@x_min = min(@x_min, $x); @x_max = max(@x_max, $x); end{emitf @x_min, @x_max}' data/medium
@ -235,7 +235,7 @@ Min/max without/with oosvars
Keyed min/max without/with oosvars
----------------------------------------------------------------
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint stats1 -a min,max -f x -g a data/medium
@ -246,7 +246,7 @@ Keyed min/max without/with oosvars
zee 0.000549 0.999490
hat 0.000045 0.999953
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint --from data/medium put -q '
@ -266,7 +266,7 @@ Keyed min/max without/with oosvars
Delta without/with oosvars
----------------------------------------------------------------
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint step -a delta -f x data/small
@ -277,7 +277,7 @@ Delta without/with oosvars
eks wye 4 0.38139939387114097 0.13418874328430463 0.176796
wye pan 5 0.5732889198020006 0.8636244699032729 0.191890
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint put '$x_delta = is_present(@last) ? $x - @last : 0; @last = $x' data/small
@ -291,7 +291,7 @@ Delta without/with oosvars
Keyed delta without/with oosvars
----------------------------------------------------------------
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint step -a delta -f x -g a data/small
@ -302,7 +302,7 @@ Keyed delta without/with oosvars
eks wye 4 0.38139939387114097 0.13418874328430463 -0.377281
wye pan 5 0.5732889198020006 0.8636244699032729 0.368686
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint put '$x_delta = is_present(@last[$a]) ? $x - @last[$a] : 0; @last[$a]=$x' data/small
@ -316,7 +316,7 @@ Keyed delta without/with oosvars
Exponentially weighted moving averages without/with oosvars
----------------------------------------------------------------
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint step -a ewma -d 0.1 -f x data/small
@ -327,7 +327,7 @@ Exponentially weighted moving averages without/with oosvars
eks wye 4 0.38139939387114097 0.13418874328430463 0.370817
wye pan 5 0.5732889198020006 0.8636244699032729 0.391064
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint put '

12
docs/customization.rst
View file

@ -9,30 +9,30 @@ How to use .mlrrc
Suppose you always use CSV files. Then instead of always having to type ``--csv`` as in
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
mlr --csv cut -x -f extra mydata.csv
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
mlr --csv sort -n id mydata.csv
and so on, you can instead put the following into your ``$HOME/.mlrrc``:
.. code-block:: bash
.. code-block:: none
--csv
Then you can just type things like
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
mlr cut -x -f extra mydata.csv
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
mlr sort -n id mydata.csv
@ -54,7 +54,7 @@ What you can put in your .mlrrc
Here is an example ``.mlrrc file``:
.. code-block:: bash
.. code-block:: none
# These are my preferred default settings for Miller

10
docs/customization.rst.in
View file

@ -6,30 +6,30 @@ How to use .mlrrc
Suppose you always use CSV files. Then instead of always having to type ``--csv`` as in
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
mlr --csv cut -x -f extra mydata.csv
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
mlr --csv sort -n id mydata.csv
and so on, you can instead put the following into your ``$HOME/.mlrrc``:
.. code-block:: bash
.. code-block:: none
--csv
Then you can just type things like
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
mlr cut -x -f extra mydata.csv
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
mlr sort -n id mydata.csv

34
docs/data-examples.rst
View file

@ -11,7 +11,7 @@ The `flins.csv <https://github.com/johnkerl/miller/blob/main/docs/data/flins.csv
Vertical-tabular format is good for a quick look at CSV data layout -- seeing what columns you have to work with:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ head -n 2 data/flins.csv | mlr --icsv --oxtab cat
@ -22,7 +22,7 @@ Vertical-tabular format is good for a quick look at CSV data layout -- seeing wh
A few simple queries:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from data/flins.csv --icsv --opprint count-distinct -f county | head
@ -34,26 +34,26 @@ A few simple queries:
Duval 1
St. Johns 1
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from data/flins.csv --icsv --opprint count-distinct -f construction,line
Categorization of total insured value:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from data/flins.csv --icsv --opprint stats1 -a min,mean,max -f tiv_2012
tiv_2012_min tiv_2012_mean tiv_2012_max
19757.910000 1061531.463750 2785551.630000
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from data/flins.csv --icsv --opprint stats1 -a min,mean,max -f tiv_2012 -g construction,line
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from data/flins.csv --icsv --oxtab stats1 -a p0,p10,p50,p90,p95,p99,p100 -f hu_site_deductible
@ -65,7 +65,7 @@ Categorization of total insured value:
hu_site_deductible_p99
hu_site_deductible_p100
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from data/flins.csv --icsv --opprint stats1 -a p95,p99,p100 -f hu_site_deductible -g county then sort -f county | head
@ -77,7 +77,7 @@ Categorization of total insured value:
Seminole - - -
St. Johns - - -
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from data/flins.csv --icsv --oxtab stats2 -a corr,linreg-ols,r2 -f tiv_2011,tiv_2012
@ -87,7 +87,7 @@ Categorization of total insured value:
tiv_2011_tiv_2012_ols_n 8
tiv_2011_tiv_2012_r2 0.874904
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from data/flins.csv --icsv --opprint stats2 -a corr,linreg-ols,r2 -f tiv_2011,tiv_2012 -g county
@ -114,13 +114,13 @@ The `colored-shapes.dkvp <https://github.com/johnkerl/miller/blob/master/docs/da
Peek at the data:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ wc -l data/colored-shapes.dkvp
10078 data/colored-shapes.dkvp
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ head -n 6 data/colored-shapes.dkvp | mlr --opprint cat
@ -136,7 +136,7 @@ Look at uncategorized stats (using `creach <https://github.com/johnkerl/scripts/
Here it looks reasonable that ``u`` is unit-uniform; something's up with ``v`` but we can't yet see what:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --oxtab stats1 -a min,mean,max -f flag,u,v data/colored-shapes.dkvp | creach 3
@ -154,7 +154,7 @@ Here it looks reasonable that ``u`` is unit-uniform; something's up with ``v`` b
The histogram shows the different distribution of 0/1 flags:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint histogram -f flag,u,v --lo -0.1 --hi 1.1 --nbins 12 data/colored-shapes.dkvp
@ -174,7 +174,7 @@ The histogram shows the different distribution of 0/1 flags:
Look at univariate stats by color and shape. In particular, color-dependent flag probabilities pop out, aligning with their original Bernoulli probablities from the data-generator script:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint stats1 -a min,mean,max -f flag,u,v -g color then sort -f color data/colored-shapes.dkvp
@ -186,7 +186,7 @@ Look at univariate stats by color and shape. In particular, color-dependent flag
red 0 0.303167 1 0.000671 0.492560 0.999882 -0.092709 0.496535 1.072500
yellow 0 0.892427 1 0.001300 0.497129 0.999923 0.000711 0.510627 0.999919
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint stats1 -a min,mean,max -f flag,u,v -g shape then sort -f shape data/colored-shapes.dkvp
@ -197,14 +197,14 @@ Look at univariate stats by color and shape. In particular, color-dependent flag
Look at bivariate stats by color and shape. In particular, ``u,v`` pairwise correlation for red circles pops out:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint --right stats2 -a corr -f u,v,w,x data/colored-shapes.dkvp
u_v_corr w_x_corr
0.133418 -0.011320
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint --right stats2 -a corr -f u,v,w,x -g color,shape then sort -nr u_v_corr data/colored-shapes.dkvp

26
docs/data-sharing.rst
View file

@ -11,7 +11,7 @@ DKVP I/O in Python
Here are the I/O routines:
.. code-block:: bash
.. code-block:: none
#!/usr/bin/env python
@ -74,7 +74,7 @@ Here are the I/O routines:
And here is an example using them:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat polyglot-dkvp-io/example.py
@ -115,7 +115,7 @@ And here is an example using them:
Run as-is:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ python polyglot-dkvp-io/example.py < data/small
@ -127,7 +127,7 @@ Run as-is:
Run as-is, then pipe to Miller for pretty-printing:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ python polyglot-dkvp-io/example.py < data/small | mlr --opprint cat
@ -143,7 +143,7 @@ DKVP I/O in Ruby
Here are the I/O routines:
.. code-block:: bash
.. code-block:: none
#!/usr/bin/env ruby
@ -200,7 +200,7 @@ Here are the I/O routines:
And here is an example using them:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat polyglot-dkvp-io/example.rb
@ -231,7 +231,7 @@ And here is an example using them:
Run as-is:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ ruby -I./polyglot-dkvp-io polyglot-dkvp-io/example.rb data/small
@ -243,7 +243,7 @@ Run as-is:
Run as-is, then pipe to Miller for pretty-printing:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ ruby -I./polyglot-dkvp-io polyglot-dkvp-io/example.rb data/small | mlr --opprint cat
@ -269,7 +269,7 @@ Running shell commands
The :ref:`reference-dsl-system` DSL function allows you to run a specific shell command and put its output -- minus the final newline -- into a record field. The command itself is any string, either a literal string, or a concatenation of strings, perhaps including other field values or what have you.
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint put '$o = system("echo hello world")' data/small
@ -280,7 +280,7 @@ The :ref:`reference-dsl-system` DSL function allows you to run a specific shell
eks wye 4 0.38139939387114097 0.13418874328430463 hello world
wye pan 5 0.5732889198020006 0.8636244699032729 hello world
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint put '$o = system("echo {" . NR . "}")' data/small
@ -291,7 +291,7 @@ The :ref:`reference-dsl-system` DSL function allows you to run a specific shell
eks wye 4 0.38139939387114097 0.13418874328430463 {4}
wye pan 5 0.5732889198020006 0.8636244699032729 {5}
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint put '$o = system("echo -n ".$a."| sha1sum")' data/small
@ -307,7 +307,7 @@ Note that running a subprocess on every record takes a non-trivial amount of tim
..
hard-coded, not live-code, since %N doesn't exist on all platforms
.. code-block:: bash
.. code-block:: none
$ mlr --opprint put '$t=system("date +%s.%N")' then step -a delta -f t data/small
a b i x y t t_delta
@ -317,7 +317,7 @@ Note that running a subprocess on every record takes a non-trivial amount of tim
eks wye 4 0.38139939387114097 0.13418874328430463 1568774318.516547441 0.000929
wye pan 5 0.5732889198020006 0.8636244699032729 1568774318.517518828 0.000971
.. code-block:: bash
.. code-block:: none
$ mlr --opprint put '$t=systime()' then step -a delta -f t data/small
a b i x y t t_delta

4
docs/data-sharing.rst.in
View file

@ -67,7 +67,7 @@ Note that running a subprocess on every record takes a non-trivial amount of tim
..
hard-coded, not live-code, since %N doesn't exist on all platforms
.. code-block:: bash
.. code-block:: none
$ mlr --opprint put '$t=system("date +%s.%N")' then step -a delta -f t data/small
a b i x y t t_delta
@ -77,7 +77,7 @@ Note that running a subprocess on every record takes a non-trivial amount of tim
eks wye 4 0.38139939387114097 0.13418874328430463 1568774318.516547441 0.000929
wye pan 5 0.5732889198020006 0.8636244699032729 1568774318.517518828 0.000971
.. code-block:: bash
.. code-block:: none
$ mlr --opprint put '$t=systime()' then step -a delta -f t data/small
a b i x y t t_delta

104
docs/faq.rst
View file

@ -21,7 +21,7 @@ Also try ``od -xcv`` and/or ``cat -e`` on your file to check for non-printable c
Diagnosing delimiter specifications
----------------------------------------------------------------
.. code-block:: bash
.. code-block:: none
# Use the `file` command to see if there are CR/LF terminators (in this case,
# there are not):
@ -88,7 +88,7 @@ How do I suppress numeric conversion?
Within ``mlr put`` and ``mlr filter``, the default behavior for scanning input records is to parse them as integer, if possible, then as float, if possible, else leave them as string:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/scan-example-1.tbl
@ -98,7 +98,7 @@ Within ``mlr put`` and ``mlr filter``, the default behavior for scanning input r
3x
hello
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --pprint put '$copy = $value; $type = typeof($value)' data/scan-example-1.tbl
@ -118,7 +118,7 @@ This is a sensible default: you should be able to put ``'$z = $x + $y'`` without
But now suppose you have data like these:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/scan-example-2.tbl
@ -134,7 +134,7 @@ But now suppose you have data like these:
0009
0010
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --pprint put '$copy = $value; $type = typeof($value)' data/scan-example-2.tbl
@ -164,7 +164,7 @@ Taken individually the rules make sense; taken collectively they produce a mishm
The solution is to **use the -S flag** for ``mlr put`` and/or ``mlr filter``. Then all field values are left as string. You can type-coerce on demand using syntax like ``'$z = int($x) + float($y)'``. (See also :doc:`reference-dsl`; see also https://github.com/johnkerl/miller/issues/150.)
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --pprint put -S '$copy = $value; $type = typeof($value)' data/scan-example-2.tbl
@ -187,7 +187,7 @@ Then-chaining found in Miller is intended to function the same as Unix pipes, bu
First, look at the input data:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/then-example.csv
@ -200,7 +200,7 @@ First, look at the input data:
Next, run the first step of your command, omitting anything from the first ``then`` onward:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint count-distinct -f Status,Payment_Type data/then-example.csv
@ -212,7 +212,7 @@ Next, run the first step of your command, omitting anything from the first ``the
After that, run it with the next ``then`` step included:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint count-distinct -f Status,Payment_Type then sort -nr count data/then-example.csv
@ -226,7 +226,7 @@ Now if you use ``then`` to include another verb after that, the columns ``Status
Note, by the way, that you'll get the same results using pipes:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --csv count-distinct -f Status,Payment_Type data/then-example.csv | mlr --icsv --opprint sort -nr count
@ -241,44 +241,44 @@ I assigned $9 and it's not 9th
Miller records are ordered lists of key-value pairs. For NIDX format, DKVP format when keys are missing, or CSV/CSV-lite format with ``--implicit-csv-header``, Miller will sequentially assign keys of the form ``1``, ``2``, etc. But these are not integer array indices: they're just field names taken from the initial field ordering in the input data.
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ echo x,y,z | mlr --dkvp cat
1=x,2=y,3=z
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ echo x,y,z | mlr --dkvp put '$6="a";$4="b";$55="cde"'
1=x,2=y,3=z,6=a,4=b,55=cde
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ echo x,y,z | mlr --nidx cat
x,y,z
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ echo x,y,z | mlr --csv --implicit-csv-header cat
1,2,3
x,y,z
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ echo x,y,z | mlr --dkvp rename 2,999
1=x,999=y,3=z
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ echo x,y,z | mlr --dkvp rename 2,newname
1=x,newname=y,3=z
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ echo x,y,z | mlr --csv --implicit-csv-header reorder -f 3,1,2
@ -290,7 +290,7 @@ How can I filter by date?
Given input like
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat dates.csv
@ -301,7 +301,7 @@ Given input like
we can use ``strptime`` to parse the date field into seconds-since-epoch and then do numeric comparisons. Simply match your input dataset's date-formatting to the :ref:`reference-dsl-strptime` format-string. For example:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --csv filter 'strptime($date, "%Y-%m-%d") > strptime("2018-03-03", "%Y-%m-%d")' dates.csv
@ -315,7 +315,7 @@ How can I handle commas-as-data in various formats?
:doc:`CSV <file-formats>` handles this well and by design:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat commas.csv
@ -325,7 +325,7 @@ How can I handle commas-as-data in various formats?
Likewise :ref:`file-formats-json`:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --ojson cat commas.csv
@ -334,7 +334,7 @@ Likewise :ref:`file-formats-json`:
For Miller's :ref:`vertical-tabular format <file-formats-xtab>` there is no escaping for carriage returns, but commas work fine:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --oxtab cat commas.csv
@ -346,7 +346,7 @@ For Miller's :ref:`vertical-tabular format <file-formats-xtab>` there is no esca
But for :ref:`Key-value_pairs <file-formats-dkvp>` and :ref:`index-numbered <file-formats-nidx>`, commas are the default field separator. And -- as of Miller 5.4.0 anyway -- there is no CSV-style double-quote-handling like there is for CSV. So commas within the data look like delimiters:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --odkvp cat commas.csv
@ -355,7 +355,7 @@ But for :ref:`Key-value_pairs <file-formats-dkvp>` and :ref:`index-numbered <fil
One solution is to use a different delimiter, such as a pipe character:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --odkvp --ofs pipe cat commas.csv
@ -365,7 +365,7 @@ One solution is to use a different delimiter, such as a pipe character:
To be extra-sure to avoid data/delimiter clashes, you can also use control
characters as delimiters -- here, control-A:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --odkvp --ofs '\001' cat commas.csv | cat -v
@ -377,7 +377,7 @@ How can I handle field names with special symbols in them?
Simply surround the field names with curly braces:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ echo 'x.a=3,y:b=4,z/c=5' | mlr put '${product.all} = ${x.a} * ${y:b} * ${z/c}'
@ -388,19 +388,19 @@ How to escape '?' in regexes?
One way is to use square brackets; an alternative is to use simple string-substitution rather than a regular expression.
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/question.dat
a=is it?,b=it is!
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --oxtab put '$c = gsub($a, "[?]"," ...")' data/question.dat
a is it?
b it is!
c is it ...
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --oxtab put '$c = ssub($a, "?"," ...")' data/question.dat
@ -415,11 +415,11 @@ How can I put single-quotes into strings?
This is a little tricky due to the shell's handling of quotes. For simplicity, let's first put an update script into a file:
.. code-block:: bash
.. code-block:: none
$a = "It's OK, I said, then 'for now'."
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ echo a=bcd | mlr put -f data/single-quote-example.mlr
@ -429,7 +429,7 @@ So, it's simple: Miller's DSL uses double quotes for strings, and you can put si
Without putting the update expression in a file, it's messier:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ echo a=bcd | mlr put '$a="It'\''s OK, I said, '\''for now'\''."'
@ -450,7 +450,7 @@ Why doesn't mlr cut put fields in the order I want?
Example: columns ``x,i,a`` were requested but they appear here in the order ``a,i,x``:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/small
@ -460,7 +460,7 @@ Example: columns ``x,i,a`` were requested but they appear here in the order ``a,
a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr cut -f x,i,a data/small
@ -474,7 +474,7 @@ The issue is that Miller's ``cut``, by default, outputs cut fields in the order
The solution is to use the ``-o`` option:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr cut -o -f x,i,a data/small
@ -489,7 +489,7 @@ NR is not consecutive after then-chaining
Given this input data:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/small
@ -501,7 +501,7 @@ Given this input data:
why don't I see ``NR=1`` and ``NR=2`` here??
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr filter '$x > 0.5' then put '$NR = NR' data/small
@ -510,7 +510,7 @@ why don't I see ``NR=1`` and ``NR=2`` here??
The reason is that ``NR`` is computed for the original input records and isn't dynamically updated. By contrast, ``NF`` is dynamically updated: it's the number of fields in the current record, and if you add/remove a field, the value of ``NF`` will change:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ echo x=1,y=2,z=3 | mlr put '$nf1 = NF; $u = 4; $nf2 = NF; unset $x,$y,$z; $nf3 = NF'
@ -518,7 +518,7 @@ The reason is that ``NR`` is computed for the original input records and isn't d
``NR``, by contrast (and ``FNR`` as well), retains the value from the original input stream, and records may be dropped by a ``filter`` within a ``then``-chain. To recover consecutive record numbers, you can use out-of-stream variables as follows:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint --from data/small put '
@ -538,7 +538,7 @@ The reason is that ``NR`` is computed for the original input records and isn't d
Or, simply use ``mlr cat -n``:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr filter '$x > 0.5' then cat -n data/small
@ -552,7 +552,7 @@ Why am I not seeing all possible joins occur?
For example, the right file here has nine records, and the left file should add in the ``hostname`` column -- so the join output should also have 9 records:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsvlite --opprint cat data/join-u-left.csv
@ -561,7 +561,7 @@ For example, the right file here has nine records, and the left file should add
zenith.west.our.org 10.3.1.27
apoapsis.east.our.org 10.4.5.94
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsvlite --opprint cat data/join-u-right.csv
@ -576,7 +576,7 @@ For example, the right file here has nine records, and the left file should add
10.3.1.18 1448762598 73425
10.4.5.94 1448762599 12200
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsvlite --opprint join -s -j ipaddr -f data/join-u-left.csv data/join-u-right.csv
@ -590,7 +590,7 @@ The issue is that Miller's ``join``, by default (before 5.1.0), took input sorte
The solution (besides pre-sorting the input files on the join keys) is to simply use **mlr join -u** (which is now the default). This loads the left file entirely into memory (while the right file is still streamed one line at a time) and does all possible joins without requiring sorted input:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsvlite --opprint join -u -j ipaddr -f data/join-u-left.csv data/join-u-right.csv
@ -612,14 +612,14 @@ How to rectangularize after joins with unpaired?
Suppose you have the following two data files:
.. code-block:: bash
.. code-block:: none
id,code
3,0000ff
2,00ff00
4,ff0000
.. code-block:: bash
.. code-block:: none
id,color
4,red
@ -627,7 +627,7 @@ Suppose you have the following two data files:
Joining on color the results are as expected:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --csv join -j id -f data/color-codes.csv data/color-names.csv
@ -637,7 +637,7 @@ Joining on color the results are as expected:
However, if we ask for left-unpaireds, since there's no ``color`` column, we get a row not having the same column names as the other:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --csv join --ul -j id -f data/color-codes.csv data/color-names.csv
@ -650,7 +650,7 @@ However, if we ask for left-unpaireds, since there's no ``color`` column, we get
To fix this, we can use **unsparsify**:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --csv join --ul -j id -f data/color-codes.csv then unsparsify --fill-with "" data/color-names.csv
@ -670,13 +670,13 @@ XML, JSON, etc. are, by contrast, all **recursive** or **nested** data structure
Now, you can put tabular data into these formats -- since list-of-key-value-pairs is one of the things representable in XML or JSON. Example:
.. code-block:: bash
.. code-block:: none
# DKVP
x=1,y=2
z=3
.. code-block:: bash
.. code-block:: none
# XML
<table>
@ -695,7 +695,7 @@ Now, you can put tabular data into these formats -- since list-of-key-value-pair
</record>
</table>
.. code-block:: bash
.. code-block:: none
# JSON
[{"x":1,"y":2},{"z":3}]

6
docs/faq.rst.in
View file

@ -278,13 +278,13 @@ XML, JSON, etc. are, by contrast, all **recursive** or **nested** data structure
Now, you can put tabular data into these formats -- since list-of-key-value-pairs is one of the things representable in XML or JSON. Example:
.. code-block:: bash
.. code-block:: none
# DKVP
x=1,y=2
z=3
.. code-block:: bash
.. code-block:: none
# XML
<table>
@ -303,7 +303,7 @@ Now, you can put tabular data into these formats -- since list-of-key-value-pair
</record>
</table>
.. code-block:: bash
.. code-block:: none
# JSON
[{"x":1,"y":2},{"z":3}]

8
docs/feature-comparison.rst
View file

@ -11,7 +11,7 @@ File-format awareness
Miller respects CSV headers. If you do ``mlr --csv cat *.csv`` then the header line is written once:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/a.csv
@ -19,14 +19,14 @@ Miller respects CSV headers. If you do ``mlr --csv cat *.csv`` then the header l
1,2,3
4,5,6
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/b.csv
a,b,c
7,8,9
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --csv cat data/a.csv data/b.csv
@ -35,7 +35,7 @@ Miller respects CSV headers. If you do ``mlr --csv cat *.csv`` then the header l
4,5,6
7,8,9
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --csv sort -nr b data/a.csv data/b.csv

68
docs/file-formats.rst
View file

@ -9,7 +9,7 @@ Miller handles name-indexed data using several formats: some you probably know b
Examples
----------------------------------------------------------------
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --usage-data-format-examples
@ -118,7 +118,7 @@ DKVP: Key-value pairs
Miller's default file format is DKVP, for **delimited key-value pairs**. Example:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr cat data/small
@ -130,21 +130,21 @@ Miller's default file format is DKVP, for **delimited key-value pairs**. Example
Such data are easy to generate, e.g. in Ruby with
.. code-block:: bash
.. code-block:: none
puts "host=#{hostname},seconds=#{t2-t1},message=#{msg}"
.. code-block:: bash
.. code-block:: none
puts mymap.collect{|k,v| "#{k}=#{v}"}.join(',')
or ``print`` statements in various languages, e.g.
.. code-block:: bash
.. code-block:: none
echo "type=3,user=$USER,date=$date\n";
.. code-block:: bash
.. code-block:: none
logger.log("type=3,user=$USER,date=$date\n");
@ -152,7 +152,7 @@ Fields lacking an IPS will have positional index (starting at 1) used as the key
As discussed in :doc:`record-heterogeneity`, Miller handles changes of field names within the same data stream. But using DKVP format this is particularly natural. One of my favorite use-cases for Miller is in application/server logs, where I log all sorts of lines such as
.. code-block:: bash
.. code-block:: none
resource=/path/to/file,loadsec=0.45,ok=true
record_count=100, resource=/path/to/file
@ -172,7 +172,7 @@ With ``--inidx --ifs ' ' --repifs``, Miller splits lines on whitespace and assig
Example with index-numbered output:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/small
@ -182,7 +182,7 @@ Example with index-numbered output:
a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --onidx --ofs ' ' cat data/small
@ -194,7 +194,7 @@ Example with index-numbered output:
Example with index-numbered input:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/mydata.txt
@ -202,7 +202,7 @@ Example with index-numbered input:
by the dawn's
early light
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --inidx --ifs ' ' --odkvp cat data/mydata.txt
@ -212,7 +212,7 @@ Example with index-numbered input:
Example with index-numbered input and output:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/mydata.txt
@ -220,7 +220,7 @@ Example with index-numbered input and output:
by the dawn's
early light
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --nidx --fs ' ' --repifs cut -f 2,3 data/mydata.txt
@ -242,14 +242,14 @@ Single-level JSON objects
An **array of single-level objects** is, quite simply, **a table**:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --json head -n 2 then cut -f color,shape data/json-example-1.json
{ "color": "yellow", "shape": "triangle" }
{ "color": "red", "shape": "square" }
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --json --jvstack head -n 2 then cut -f color,u,v data/json-example-1.json
@ -264,7 +264,7 @@ An **array of single-level objects** is, quite simply, **a table**:
"v": 0.001257332190235938
}
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --ijson --opprint stats1 -a mean,stddev,count -f u -g shape data/json-example-1.json
@ -278,7 +278,7 @@ Nested JSON objects
Additionally, Miller can **tabularize nested objects by concatentating keys**:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --json --jvstack head -n 2 data/json-example-2.json
@ -311,7 +311,7 @@ Additionally, Miller can **tabularize nested objects by concatentating keys**:
}
}
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --ijson --opprint head -n 4 data/json-example-2.json
@ -323,7 +323,7 @@ Additionally, Miller can **tabularize nested objects by concatentating keys**:
Note in particular that as far as Miller's ``put`` and ``filter``, as well as other I/O formats, are concerned, these are simply field names with colons in them:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --json --jvstack head -n 1 then put '${values:uv} = ${values:u} * ${values:v}' data/json-example-2.json
@ -350,7 +350,7 @@ Arrays aren't supported in Miller's ``put``/``filter`` DSL. By default, JSON arr
Suppose we have arrays like this in our input data:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/json-example-3.json
@ -365,7 +365,7 @@ Suppose we have arrays like this in our input data:
Then integer indices (starting from 0 and counting up) are used as map keys:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --ijson --oxtab cat data/json-example-3.json
@ -380,7 +380,7 @@ Then integer indices (starting from 0 and counting up) are used as map keys:
When the data are written back out as JSON, field names are re-expanded as above, but what were arrays on input are now maps on output:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --json --jvstack cat data/json-example-3.json
@ -437,7 +437,7 @@ PPRINT: Pretty-printed tabular
Miller's pretty-print format is like CSV, but column-aligned. For example, compare
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --ocsv cat data/small
@ -448,7 +448,7 @@ Miller's pretty-print format is like CSV, but column-aligned. For example, comp
eks,wye,4,0.38139939387114097,0.13418874328430463
wye,pan,5,0.5732889198020006,0.8636244699032729
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint cat data/small
@ -465,7 +465,7 @@ See :doc:`record-heterogeneity` for how Miller handles changes of field names wi
For output only (this isn't supported in the input-scanner as of 5.0.0) you can use ``--barred`` with pprint output format:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint --barred cat data/small
@ -487,7 +487,7 @@ XTAB: Vertical tabular
This is perhaps most useful for looking a very wide and/or multi-column data which causes line-wraps on the screen (but see also
`ngrid <https://github.com/twosigma/ngrid/>`_ for an entirely different, very powerful option). Namely:
.. code-block:: bash
.. code-block:: none
$ grep -v '^#' /etc/passwd | head -n 6 | mlr --nidx --fs : --opprint cat
1 2 3 4 5 6 7
@ -498,7 +498,7 @@ This is perhaps most useful for looking a very wide and/or multi-column data whi
_taskgated * 13 13 Task Gate Daemon /var/empty /usr/bin/false
_networkd * 24 24 Network Services /var/networkd /usr/bin/false
.. code-block:: bash
.. code-block:: none
$ grep -v '^#' /etc/passwd | head -n 2 | mlr --nidx --fs : --oxtab cat
1 nobody
@ -517,7 +517,7 @@ This is perhaps most useful for looking a very wide and/or multi-column data whi
6 /var/root
7 /bin/sh
.. code-block:: bash
.. code-block:: none
$ grep -v '^#' /etc/passwd | head -n 2 | \
mlr --nidx --fs : --ojson --jvstack --jlistwrap label name,password,uid,gid,gecos,home_dir,shell
@ -547,7 +547,7 @@ Markdown tabular
Markdown format looks like this:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --omd cat data/small
@ -570,7 +570,7 @@ Data-conversion keystroke-savers
While you can do format conversion using ``mlr --icsv --ojson cat myfile.csv``, there are also keystroke-savers for this purpose, such as ``mlr --c2j cat myfile.csv``. For a complete list:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --usage-format-conversion-keystroke-saver-options
@ -606,7 +606,7 @@ Comments in data
You can include comments within your data files, and either have them ignored, or passed directly through to the standard output as soon as they are encountered:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --usage-comments-in-data
@ -629,7 +629,7 @@ You can include comments within your data files, and either have them ignored, o
Examples:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/budget.csv
@ -639,7 +639,7 @@ Examples:
green,678.12
orange,123.45
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --skip-comments --icsv --opprint sort -nr quantity data/budget.csv
@ -648,7 +648,7 @@ Examples:
purple 456.78
orange 123.45
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --pass-comments --icsv --opprint sort -nr quantity data/budget.csv

10
docs/file-formats.rst.in
View file

@ -57,21 +57,21 @@ POKI_RUN_COMMAND{{mlr cat data/small}}HERE
Such data are easy to generate, e.g. in Ruby with
.. code-block:: bash
.. code-block:: none
puts "host=#{hostname},seconds=#{t2-t1},message=#{msg}"
.. code-block:: bash
.. code-block:: none
puts mymap.collect{|k,v| "#{k}=#{v}"}.join(',')
or ``print`` statements in various languages, e.g.
.. code-block:: bash
.. code-block:: none
echo "type=3,user=$USER,date=$date\n";
.. code-block:: bash
.. code-block:: none
logger.log("type=3,user=$USER,date=$date\n");
@ -79,7 +79,7 @@ Fields lacking an IPS will have positional index (starting at 1) used as the key
As discussed in :doc:`record-heterogeneity`, Miller handles changes of field names within the same data stream. But using DKVP format this is particularly natural. One of my favorite use-cases for Miller is in application/server logs, where I log all sorts of lines such as
.. code-block:: bash
.. code-block:: none
resource=/path/to/file,loadsec=0.45,ok=true
record_count=100, resource=/path/to/file

12
docs/install.rst
View file

@ -9,38 +9,38 @@ Prebuilt executables via package managers
`Homebrew <https://brew.sh/>`_ installation support for OSX is available via
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
brew update && brew install miller
...and also via `MacPorts <https://www.macports.org/>`_:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
sudo port selfupdate && sudo port install miller
You may already have the ``mlr`` executable available in your platform's package manager on NetBSD, Debian Linux, Ubuntu Xenial and upward, Arch Linux, or perhaps other distributions. For example, on various Linux distributions you might do one of the following:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
sudo apt-get install miller
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
sudo apt install miller
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
sudo yum install miller
On Windows, Miller is available via `Chocolatey <https://chocolatey.org/>`_:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
choco install miller

12
docs/install.rst.in
View file

@ -6,38 +6,38 @@ Prebuilt executables via package managers
`Homebrew <https://brew.sh/>`_ installation support for OSX is available via
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
brew update && brew install miller
...and also via `MacPorts <https://www.macports.org/>`_:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
sudo port selfupdate && sudo port install miller
You may already have the ``mlr`` executable available in your platform's package manager on NetBSD, Debian Linux, Ubuntu Xenial and upward, Arch Linux, or perhaps other distributions. For example, on various Linux distributions you might do one of the following:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
sudo apt-get install miller
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
sudo apt install miller
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
sudo yum install miller
On Windows, Miller is available via `Chocolatey <https://chocolatey.org/>`_:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
choco install miller

10
docs/log-processing-examples.rst
View file

@ -12,7 +12,7 @@ Writing a program -- in any language whatsoever -- you can have it print out log
Suppose your program has printed something like this:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat log.txt
@ -62,7 +62,7 @@ Suppose your program has printed something like this:
Each print statement simply contains local information: the current timestamp, whether a particular cache was hit or not, etc. Then using either the system ``grep`` command, or Miller's ``having-fields``, or ``is_present``, we can pick out the parts we want and analyze them:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ grep op=cache log.txt \
@ -72,7 +72,7 @@ Each print statement simply contains local information: the current timestamp, w
A4 0.714286
A9 0.090909
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from log.txt --opprint \
@ -89,7 +89,7 @@ Each print statement simply contains local information: the current timestamp, w
Alternatively, we can simply group the similar data for a better look:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint group-like log.txt
@ -142,7 +142,7 @@ Alternatively, we can simply group the similar data for a better look:
1472819736 100 612
1472819742 100 728
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint group-like then sec2gmt time log.txt

2
docs/manpage.rst
View file

@ -6,7 +6,7 @@ Manpage
This is simply a copy of what you should see on running **man mlr** at a command prompt, once Miller is installed on your system.
.. code-block:: bash
.. code-block:: none
MILLER(1) MILLER(1)

2
docs/poki
View file

@ -90,7 +90,7 @@ end
# ----------------------------------------------------------------
def write_card(highlight_first_line, content_lines, output_handle)
output_handle.puts('.. code-block:: bash')
output_handle.puts('.. code-block:: none')
if highlight_first_line
output_handle.puts(' :emphasize-lines: 1,1') # 'hll' in _static/*.css
end

30
docs/quick-examples.rst
View file

@ -6,70 +6,70 @@ Quick examples
Column select:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
% mlr --csv cut -f hostname,uptime mydata.csv
Add new columns as function of other columns:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
% mlr --nidx put '$sum = $7 < 0.0 ? 3.5 : $7 + 2.1*$8' *.dat
Row filter:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
% mlr --csv filter '$status != "down" && $upsec >= 10000' *.csv
Apply column labels and pretty-print:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
% grep -v '^#' /etc/group | mlr --ifs : --nidx --opprint label group,pass,gid,member then sort -f group
Join multiple data sources on key columns:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
% mlr join -j account_id -f accounts.dat then group-by account_name balances.dat
Multiple formats including JSON:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
% mlr --json put '$attr = sub($attr, "([0-9]+)_([0-9]+)_.*", "\1:\2")' data/*.json
Aggregate per-column statistics:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
% mlr stats1 -a min,mean,max,p10,p50,p90 -f flag,u,v data/*
Linear regression:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
% mlr stats2 -a linreg-pca -f u,v -g shape data/*
Aggregate custom per-column statistics:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
% mlr put -q '@sum[$a][$b] += $x; end {emit @sum, "a", "b"}' data/*
Iterate over data using DSL expressions:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
% mlr --from estimates.tbl put '
@ -83,35 +83,35 @@ Iterate over data using DSL expressions:
Run DSL expressions from a script file:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
% mlr --from infile.dat put -f analyze.mlr
Split/reduce output to multiple filenames:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
% mlr --from infile.dat put 'tee > "./taps/data-".$a."-".$b, $*'
Compressed I/O:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
% mlr --from infile.dat put 'tee | "gzip > ./taps/data-".$a."-".$b.".gz", $*'
Interoperate with other data-processing tools using standard pipes:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
% mlr --from infile.dat put -q '@v=$*; dump | "jq .[]"'
Tap/trace:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
% mlr --from infile.dat put '(NR % 1000 == 0) { print > stderr, "Checkpoint ".NR}'

30
docs/quick-examples.rst.in
View file

@ -3,70 +3,70 @@ Quick examples
Column select:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
% mlr --csv cut -f hostname,uptime mydata.csv
Add new columns as function of other columns:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
% mlr --nidx put '$sum = $7 < 0.0 ? 3.5 : $7 + 2.1*$8' *.dat
Row filter:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
% mlr --csv filter '$status != "down" && $upsec >= 10000' *.csv
Apply column labels and pretty-print:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
% grep -v '^#' /etc/group | mlr --ifs : --nidx --opprint label group,pass,gid,member then sort -f group
Join multiple data sources on key columns:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
% mlr join -j account_id -f accounts.dat then group-by account_name balances.dat
Multiple formats including JSON:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
% mlr --json put '$attr = sub($attr, "([0-9]+)_([0-9]+)_.*", "\1:\2")' data/*.json
Aggregate per-column statistics:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
% mlr stats1 -a min,mean,max,p10,p50,p90 -f flag,u,v data/*
Linear regression:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
% mlr stats2 -a linreg-pca -f u,v -g shape data/*
Aggregate custom per-column statistics:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
% mlr put -q '@sum[$a][$b] += $x; end {emit @sum, "a", "b"}' data/*
Iterate over data using DSL expressions:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
% mlr --from estimates.tbl put '
@ -80,35 +80,35 @@ Iterate over data using DSL expressions:
Run DSL expressions from a script file:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
% mlr --from infile.dat put -f analyze.mlr
Split/reduce output to multiple filenames:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
% mlr --from infile.dat put 'tee > "./taps/data-".$a."-".$b, $*'
Compressed I/O:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
% mlr --from infile.dat put 'tee | "gzip > ./taps/data-".$a."-".$b.".gz", $*'
Interoperate with other data-processing tools using standard pipes:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
% mlr --from infile.dat put -q '@v=$*; dump | "jq .[]"'
Tap/trace:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
% mlr --from infile.dat put '(NR % 1000 == 0) { print > stderr, "Checkpoint ".NR}'

26
docs/record-heterogeneity.rst
View file

@ -16,7 +16,7 @@ CSV and pretty-print
Miller simply prints a newline and a new header when there is a schema change. When there is no schema change, you get CSV per se as a special case. Likewise, Miller reads heterogeneous CSV or pretty-print input the same way. The difference between CSV and CSV-lite is that the former is RFC4180-compliant, while the latter readily handles heterogeneous data (which is non-compliant). For example:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/het.dkvp
@ -26,7 +26,7 @@ Miller simply prints a newline and a new header when there is a schema change. W
record_count=150,resource=/path/to/second/file
resource=/some/other/path,loadsec=0.97,ok=false
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --ocsvlite cat data/het.dkvp
@ -45,7 +45,7 @@ Miller simply prints a newline and a new header when there is a schema change. W
resource,loadsec,ok
/some/other/path,0.97,false
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint cat data/het.dkvp
@ -66,7 +66,7 @@ Miller simply prints a newline and a new header when there is a schema change. W
Miller handles explicit header changes as just shown. If your CSV input contains ragged data -- if there are implicit header changes -- you can use ``--allow-ragged-csv-input`` (or keystroke-saver ``--ragged``). For too-short data lines, values are filled with empty string; for too-long data lines, missing field names are replaced with positional indices (counting up from 1, not 0), as follows:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/ragged.csv
@ -75,7 +75,7 @@ Miller handles explicit header changes as just shown. If your CSV input contains
4,5
6,7,8,9
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --oxtab --allow-ragged-csv-input cat data/ragged.csv
@ -94,7 +94,7 @@ Miller handles explicit header changes as just shown. If your CSV input contains
You may also find Miller's ``group-like`` feature handy (see also :doc:`reference`):
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --ocsvlite group-like data/het.dkvp
@ -107,7 +107,7 @@ You may also find Miller's ``group-like`` feature handy (see also :doc:`referenc
100,/path/to/file
150,/path/to/second/file
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint group-like data/het.dkvp
@ -125,7 +125,7 @@ Key-value-pair, vertical-tabular, and index-numbered formats
For these formats, record-heterogeneity comes naturally:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/het.dkvp
@ -135,7 +135,7 @@ For these formats, record-heterogeneity comes naturally:
record_count=150,resource=/path/to/second/file
resource=/some/other/path,loadsec=0.97,ok=false
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --onidx --ofs ' ' cat data/het.dkvp
@ -145,7 +145,7 @@ For these formats, record-heterogeneity comes naturally:
150 /path/to/second/file
/some/other/path 0.97 false
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --oxtab cat data/het.dkvp
@ -167,7 +167,7 @@ For these formats, record-heterogeneity comes naturally:
loadsec 0.97
ok false
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --oxtab group-like data/het.dkvp
@ -194,7 +194,7 @@ For processing
Miller operates on specified fields and takes the rest along: for example, if you are sorting on the ``count`` field then all records in the input stream must have a ``count`` field but the other fields can vary, and moreover the sorted-on field name(s) don't need to be in the same position on each line:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/sort-het.dkvp
@ -206,7 +206,7 @@ Miller operates on specified fields and takes the rest along: for example, if yo
count=100,color=green
count=450
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr sort -n count data/sort-het.dkvp

266
docs/reference-dsl.rst
View file

File diff suppressed because it is too large Load diff

22
docs/reference-dsl.rst.in
View file

@ -121,17 +121,17 @@ POKI_INCLUDE_AND_RUN_ESCAPED(data/trailing-commas.sh)HERE
Bodies for all compound statements must be enclosed in **curly braces**, even if the body is a single statement:
.. code-block:: bash
.. code-block:: none
mlr put 'if ($x == 1) $y = 2' # Syntax error
.. code-block:: bash
.. code-block:: none
mlr put 'if ($x == 1) { $y = 2 }' # This is OK
Bodies for compound statements may be empty:
.. code-block:: bash
.. code-block:: none
mlr put 'if ($x == 1) { }' # This no-op is syntactically acceptable
@ -360,7 +360,7 @@ Local variables can be defined either untyped as in ``x = 1``, or typed as in ``
The reason for ``num`` is that ``int`` and ``float`` typedecls are very precise:
.. code-block:: bash
.. code-block:: none
float a = 0; # Runtime error since 0 is int not float
int b = 1.0; # Runtime error since 1.0 is float not int
@ -371,7 +371,7 @@ A suggestion is to use ``num`` for general use when you want numeric content, an
The ``var`` type declaration indicates no type restrictions, e.g. ``var x = 1`` has the same type restrictions on ``x`` as ``x = 1``. The difference is in intentional shadowing: if you have ``x = 1`` in outer scope and ``x = 2`` in inner scope (e.g. within a for-loop or an if-statement) then outer-scope ``x`` has value 2 after the second assignment. But if you have ``var x = 2`` in the inner scope, then you are declaring a variable scoped to the inner block.) For example:
.. code-block:: bash
.. code-block:: none
x = 1;
if (NR == 4) {
@ -379,7 +379,7 @@ The ``var`` type declaration indicates no type restrictions, e.g. ``var x = 1``
}
print x; # Value of x is now two
.. code-block:: bash
.. code-block:: none
x = 1;
if (NR == 4) {
@ -389,7 +389,7 @@ The ``var`` type declaration indicates no type restrictions, e.g. ``var x = 1``
Likewise function arguments can optionally be typed, with type enforced when the function is called:
.. code-block:: bash
.. code-block:: none
func f(map m, int i) {
...
@ -404,7 +404,7 @@ Likewise function arguments can optionally be typed, with type enforced when the
Thirdly, function return values can be type-checked at the point of ``return`` using ``:`` and a typedecl after the parameter list:
.. code-block:: bash
.. code-block:: none
func f(map m, int i): bool {
...
@ -463,7 +463,7 @@ Operator precedence
Operators are listed in order of decreasing precedence, highest first.
.. code-block:: bash
.. code-block:: none
Operators Associativity
--------- -------------
@ -524,11 +524,11 @@ If-statements
These are again reminiscent of ``awk``. Pattern-action blocks are a special case of ``if`` with no ``elif`` or ``else`` blocks, no ``if`` keyword, and parentheses optional around the boolean expression:
.. code-block:: bash
.. code-block:: none
mlr put 'NR == 4 {$foo = "bar"}'
.. code-block:: bash
.. code-block:: none
mlr put 'if (NR == 4) {$foo = "bar"}'

414
docs/reference-verbs.rst
View file

File diff suppressed because it is too large Load diff

8
docs/reference-verbs.rst.in
View file

@ -69,7 +69,7 @@ POKI_RUN_COMMAND{{mlr bootstrap --help}}HERE
The canonical use for bootstrap sampling is to put error bars on statistical quantities, such as mean. For example:
.. code-block:: bash
.. code-block:: none
$ mlr --opprint stats1 -a mean,count -f u -g color data/colored-shapes.dkvp
color u_mean u_count
@ -80,7 +80,7 @@ The canonical use for bootstrap sampling is to put error bars on statistical qua
blue 0.517717 1470
orange 0.490532 303
.. code-block:: bash
.. code-block:: none
$ mlr --opprint bootstrap then stats1 -a mean,count -f u -g color data/colored-shapes.dkvp
color u_mean u_count
@ -91,7 +91,7 @@ The canonical use for bootstrap sampling is to put error bars on statistical qua
blue 0.512529 1496
orange 0.521030 321
.. code-block:: bash
.. code-block:: none
$ mlr --opprint bootstrap then stats1 -a mean,count -f u -g color data/colored-shapes.dkvp
color u_mean u_count
@ -102,7 +102,7 @@ The canonical use for bootstrap sampling is to put error bars on statistical qua
green 0.496803 1075
purple 0.486337 1199
.. code-block:: bash
.. code-block:: none
$ mlr --opprint bootstrap then stats1 -a mean,count -f u -g color data/colored-shapes.dkvp
color u_mean u_count

96
docs/reference.rst
View file

@ -13,7 +13,7 @@ Command overview
Whereas the Unix toolkit is made of the separate executables ``cat``, ``tail``, ``cut``,
``sort``, etc., Miller has subcommands, or **verbs**, invoked as follows:
.. code-block:: bash
.. code-block:: none
mlr tac *.dat
mlr cut --complement -f os_version *.dat
@ -39,7 +39,7 @@ Formats
Options:
.. code-block:: bash
.. code-block:: none
--dkvp --idkvp --odkvp
--nidx --inidx --onidx
@ -51,7 +51,7 @@ Options:
These are as discussed in :doc:`file-formats`, with the exception of ``--right`` which makes pretty-printed output right-aligned:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint cat data/small
@ -62,7 +62,7 @@ These are as discussed in :doc:`file-formats`, with the exception of ``--right``
eks wye 4 0.38139939387114097 0.13418874328430463
wye pan 5 0.5732889198020006 0.8636244699032729
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint --right cat data/small
@ -97,14 +97,14 @@ Compression
Options:
.. code-block:: bash
.. code-block:: none
--prepipe {command}
The prepipe command is anything which reads from standard input and produces data acceptable to Miller. Nominally this allows you to use whichever decompression utilities you have installed on your system, on a per-file basis. If the command has flags, quote them: e.g. ``mlr --prepipe 'zcat -cf'``. Examples:
.. code-block:: bash
.. code-block:: none
# These two produce the same output:
$ gunzip < myfile1.csv.gz | mlr cut -f hostname,uptime
@ -113,14 +113,14 @@ The prepipe command is anything which reads from standard input and produces dat
$ mlr --prepipe gunzip cut -f hostname,uptime myfile1.csv.gz myfile2.csv.gz
$ mlr --prepipe gunzip --idkvp --oxtab cut -f hostname,uptime myfile1.dat.gz myfile2.dat.gz
.. code-block:: bash
.. code-block:: none
# Similar to the above, but with compressed output as well as input:
$ gunzip < myfile1.csv.gz | mlr cut -f hostname,uptime | gzip > outfile.csv.gz
$ mlr --prepipe gunzip cut -f hostname,uptime myfile1.csv.gz | gzip > outfile.csv.gz
$ mlr --prepipe gunzip cut -f hostname,uptime myfile1.csv.gz myfile2.csv.gz | gzip > outfile.csv.gz
.. code-block:: bash
.. code-block:: none
# Similar to the above, but with different compression tools for input and output:
$ gunzip < myfile1.csv.gz | mlr cut -f hostname,uptime | xz -z > outfile.csv.xz
@ -136,7 +136,7 @@ Miller has record separators ``IRS`` and ``ORS``, field separators ``IFS`` and `
Options:
.. code-block:: bash
.. code-block:: none
--rs --irs --ors
--fs --ifs --ofs --repifs
@ -157,7 +157,7 @@ Number formatting
The command-line option ``--ofmt {format string}`` is the global number format for commands which generate numeric output, e.g. ``stats1``, ``stats2``, ``histogram``, and ``step``, as well as ``mlr put``. Examples:
.. code-block:: bash
.. code-block:: none
--ofmt %.9le --ofmt %.6lf --ofmt %.0lf
@ -165,13 +165,13 @@ These are just C ``printf`` formats applied to double-precision numbers. Please
To apply formatting to a single field, overriding the global ``ofmt``, use ``fmtnum`` function within ``mlr put``. For example:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ echo 'x=3.1,y=4.3' | mlr put '$z=fmtnum($x*$y,"%08lf")'
x=3.1,y=4.3,z=13.330000
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ echo 'x=0xffff,y=0xff' | mlr put '$z=fmtnum(int($x*$y),"%08llx")'
@ -179,7 +179,7 @@ To apply formatting to a single field, overriding the global ``ofmt``, use ``fmt
Input conversion from hexadecimal is done automatically on fields handled by ``mlr put`` and ``mlr filter`` as long as the field value begins with "0x". To apply output conversion to hexadecimal on a single column, you may use ``fmtnum``, or the keystroke-saving ``hexfmt`` function. Example:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ echo 'x=0xffff,y=0xff' | mlr put '$z=hexfmt($x*$y)'
@ -200,13 +200,13 @@ then-chaining
In accord with the `Unix philosophy <http://en.wikipedia.org/wiki/Unix_philosophy>`_, you can pipe data into or out of Miller. For example:
.. code-block:: bash
.. code-block:: none
mlr cut --complement -f os_version *.dat | mlr sort -f hostname,uptime
You can, if you like, instead simply chain commands together using the ``then`` keyword:
.. code-block:: bash
.. code-block:: none
mlr cut --complement -f os_version then sort -f hostname,uptime *.dat
@ -214,7 +214,7 @@ You can, if you like, instead simply chain commands together using the ``then``
Here's a performance comparison:
.. code-block:: bash
.. code-block:: none
% cat piped.sh
mlr cut -x -f i,y data/big | mlr sort -n y > /dev/null
@ -242,7 +242,7 @@ Auxiliary commands
There are a few nearly-standalone programs which have nothing to do with the rest of Miller, do not participate in record streams, and do not deal with file formats. They might as well be little standalone executables but they're delivered within the main Miller executable for convenience.
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr aux-list
@ -255,7 +255,7 @@ There are a few nearly-standalone programs which have nothing to do with the res
netbsd-strptime
For more information, please invoke mlr {subcommand} --help
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr lecat --help
@ -266,7 +266,7 @@ There are a few nearly-standalone programs which have nothing to do with the res
--mono: don't try to colorize the output
-h or --help: print this message
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr termcvt --help
@ -283,7 +283,7 @@ There are a few nearly-standalone programs which have nothing to do with the res
Zero file names means read from standard input.
Output is always to standard output; files are not written in-place.
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr hex --help
@ -294,7 +294,7 @@ There are a few nearly-standalone programs which have nothing to do with the res
-r: print only raw hex without leading offset indicators or trailing ASCII dump.
-h or --help: print this message
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr unhex --help
@ -306,19 +306,19 @@ There are a few nearly-standalone programs which have nothing to do with the res
Examples:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ echo 'Hello, world!' | mlr lecat --mono
Hello, world![LF]
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ echo 'Hello, world!' | mlr termcvt --lf2crlf | mlr lecat --mono
Hello, world![CR][LF]
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr hex data/budget.csv
@ -330,7 +330,7 @@ Examples:
00000050: 38 0a 67 72 65 65 6e 2c 36 37 38 2e 31 32 0a 6f |8.green,678.12.o|
00000060: 72 61 6e 67 65 2c 31 32 33 2e 34 35 0a |range,123.45.|
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr hex -r data/budget.csv
@ -342,7 +342,7 @@ Examples:
38 0a 67 72 65 65 6e 2c 36 37 38 2e 31 32 0a 6f
72 61 6e 67 65 2c 31 32 33 2e 34 35 0a
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr hex -r data/budget.csv | sed 's/20/2a/g' | mlr unhex
@ -394,7 +394,7 @@ Rules for null-handling:
* Records with one or more empty sort-field values sort after records with all sort-field values present:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr cat data/sort-null.dat
@ -404,7 +404,7 @@ Rules for null-handling:
x=9,b=10
a=5,b=7
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr sort -n a data/sort-null.dat
@ -414,7 +414,7 @@ Rules for null-handling:
a=,b=4
x=9,b=10
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr sort -nr a data/sort-null.dat
@ -426,19 +426,19 @@ Rules for null-handling:
* Functions/operators which have one or more *empty* arguments produce empty output: e.g.
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ echo 'x=2,y=3' | mlr put '$a=$x+$y'
x=2,y=3,a=5
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ echo 'x=,y=3' | mlr put '$a=$x+$y'
x=,y=3,a=
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ echo 'x=,y=3' | mlr put '$a=log($x);$b=log($y)'
@ -446,7 +446,7 @@ Rules for null-handling:
with the exception that the ``min`` and ``max`` functions are special: if one argument is non-null, it wins:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ echo 'x=,y=3' | mlr put '$a=min($x,$y);$b=max($x,$y)'
@ -454,13 +454,13 @@ with the exception that the ``min`` and ``max`` functions are special: if one ar
* Functions of *absent* variables (e.g. ``mlr put '$y = log10($nonesuch)'``) evaluate to absent, and arithmetic/bitwise/boolean operators with both operands being absent evaluate to absent. Arithmetic operators with one absent operand return the other operand. More specifically, absent values act like zero for addition/subtraction, and one for multiplication: Furthermore, **any expression which evaluates to absent is not stored in the left-hand side of an assignment statement**:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ echo 'x=2,y=3' | mlr put '$a=$u+$v; $b=$u+$y; $c=$x+$y'
x=2,y=3,b=3,c=5
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ echo 'x=2,y=3' | mlr put '$a=min($x,$v);$b=max($u,$y);$c=min($u,$v)'
@ -480,7 +480,7 @@ The reasoning is as follows:
Since absent plus absent is absent (and likewise for other operators), accumulations such as ``@sum += $x`` work correctly on heterogenous data, as do within-record formulas if both operands are absent. If one operand is present, you may get behavior you don't desire. To work around this -- namely, to set an output field only for records which have all the inputs present -- you can use a pattern-action block with ``is_present``:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr cat data/het.dkvp
@ -490,7 +490,7 @@ Since absent plus absent is absent (and likewise for other operators), accumulat
record_count=150,resource=/path/to/second/file
resource=/some/other/path,loadsec=0.97,ok=false
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr put 'is_present($loadsec) { $loadmillis = $loadsec * 1000 }' data/het.dkvp
@ -500,7 +500,7 @@ Since absent plus absent is absent (and likewise for other operators), accumulat
record_count=150,resource=/path/to/second/file
resource=/some/other/path,loadsec=0.97,ok=false,loadmillis=970.000000
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr put '$loadmillis = (is_present($loadsec) ? $loadsec : 0.0) * 1000' data/het.dkvp
@ -512,7 +512,7 @@ Since absent plus absent is absent (and likewise for other operators), accumulat
If you're interested in a formal description of how empty and absent fields participate in arithmetic, here's a table for plus (other arithmetic/boolean/bitwise operators are similar):
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --print-type-arithmetic-info
@ -580,7 +580,7 @@ For ``filter`` and ``put``, if the regular expression is a string literal (the n
Example:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/regex-in-data.dat
@ -588,7 +588,7 @@ Example:
name=bill,regex=^b[ou]ll$
name=bull,regex=^b[ou]ll$
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr filter '$name =~ $regex' data/regex-in-data.dat
@ -602,25 +602,25 @@ Regex captures of the form ``\0`` through ``\9`` are supported as
* Captures have in-function context for ``sub`` and ``gsub``. For example, the first ``\1,\2`` pair belong to the first ``sub`` and the second ``\1,\2`` pair belong to the second ``sub``:
.. code-block:: bash
.. code-block:: none
mlr put '$b = sub($a, "(..)_(...)", "\2-\1"); $c = sub($a, "(..)_(.)(..)", ":\1:\2:\3")'
* Captures endure for the entirety of a ``put`` for the ``=~`` and ``!=~`` operators. For example, here the ``\1,\2`` are set by the ``=~`` operator and are used by both subsequent assignment statements:
.. code-block:: bash
.. code-block:: none
mlr put '$a =~ "(..)_(....); $b = "left_\1"; $c = "right_\2"'
* The captures are not retained across multiple puts. For example, here the ``\1,\2`` won't be expanded from the regex capture:
.. code-block:: bash
.. code-block:: none
mlr put '$a =~ "(..)_(....)' then {... something else ...} then put '$b = "left_\1"; $c = "right_\2"'
* Captures are ignored in ``filter`` for the ``=~`` and ``!=~`` operators. For example, there is no mechanism provided to refer to the first ``(..)`` as ``\1`` or to the second ``(....)`` as ``\2`` in the following filter statement:
.. code-block:: bash
.. code-block:: none
mlr filter '$a =~ "(..)_(....)'
@ -650,7 +650,7 @@ The short of it is that Miller does this transparently for you so you needn't th
Implementation details of this, for the interested: integer adds and subtracts overflow by at most one bit so it suffices to check sign-changes. Thus, Miller allows you to add and subtract arbitrary 64-bit signed integers, converting only to float precisely when the result is less than -2\ :sup:`63` or greater than 2\ :sup:`63`\ -1. Multiplies, on the other hand, can overflow by a word size and a sign-change technique does not suffice to detect overflow. Instead Miller tests whether the floating-point product exceeds the representable integer range. Now, 64-bit integers have 64-bit precision while IEEE-doubles have only 52-bit mantissas -- so, there are 53 bits including implicit leading one. The following experiment explicitly demonstrates the resolution at this range:
.. code-block:: bash
.. code-block:: none
64-bit integer 64-bit integer Casted to double Back to 64-bit
in hex in decimal integer
@ -679,7 +679,7 @@ On-line help
Examples:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --help
@ -1105,7 +1105,7 @@ Examples:
For more information please see http://johnkerl.org/miller/doc and/or
http://github.com/johnkerl/miller. This is Miller version v5.10.2-dev.
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr sort --help

28
docs/reference.rst.in
View file

@ -32,7 +32,7 @@ Formats
Options:
.. code-block:: bash
.. code-block:: none
--dkvp --idkvp --odkvp
--nidx --inidx --onidx
@ -72,14 +72,14 @@ Compression
Options:
.. code-block:: bash
.. code-block:: none
--prepipe {command}
The prepipe command is anything which reads from standard input and produces data acceptable to Miller. Nominally this allows you to use whichever decompression utilities you have installed on your system, on a per-file basis. If the command has flags, quote them: e.g. ``mlr --prepipe 'zcat -cf'``. Examples:
.. code-block:: bash
.. code-block:: none
# These two produce the same output:
$ gunzip < myfile1.csv.gz | mlr cut -f hostname,uptime
@ -88,14 +88,14 @@ The prepipe command is anything which reads from standard input and produces dat
$ mlr --prepipe gunzip cut -f hostname,uptime myfile1.csv.gz myfile2.csv.gz
$ mlr --prepipe gunzip --idkvp --oxtab cut -f hostname,uptime myfile1.dat.gz myfile2.dat.gz
.. code-block:: bash
.. code-block:: none
# Similar to the above, but with compressed output as well as input:
$ gunzip < myfile1.csv.gz | mlr cut -f hostname,uptime | gzip > outfile.csv.gz
$ mlr --prepipe gunzip cut -f hostname,uptime myfile1.csv.gz | gzip > outfile.csv.gz
$ mlr --prepipe gunzip cut -f hostname,uptime myfile1.csv.gz myfile2.csv.gz | gzip > outfile.csv.gz
.. code-block:: bash
.. code-block:: none
# Similar to the above, but with different compression tools for input and output:
$ gunzip < myfile1.csv.gz | mlr cut -f hostname,uptime | xz -z > outfile.csv.xz
@ -111,7 +111,7 @@ Miller has record separators ``IRS`` and ``ORS``, field separators ``IFS`` and `
Options:
.. code-block:: bash
.. code-block:: none
--rs --irs --ors
--fs --ifs --ofs --repifs
@ -132,7 +132,7 @@ Number formatting
The command-line option ``--ofmt {format string}`` is the global number format for commands which generate numeric output, e.g. ``stats1``, ``stats2``, ``histogram``, and ``step``, as well as ``mlr put``. Examples:
.. code-block:: bash
.. code-block:: none
--ofmt %.9le --ofmt %.6lf --ofmt %.0lf
@ -163,13 +163,13 @@ then-chaining
In accord with the `Unix philosophy <http://en.wikipedia.org/wiki/Unix_philosophy>`_, you can pipe data into or out of Miller. For example:
.. code-block:: bash
.. code-block:: none
mlr cut --complement -f os_version *.dat | mlr sort -f hostname,uptime
You can, if you like, instead simply chain commands together using the ``then`` keyword:
.. code-block:: bash
.. code-block:: none
mlr cut --complement -f os_version then sort -f hostname,uptime *.dat
@ -365,25 +365,25 @@ Regex captures of the form ``\0`` through ``\9`` are supported as
* Captures have in-function context for ``sub`` and ``gsub``. For example, the first ``\1,\2`` pair belong to the first ``sub`` and the second ``\1,\2`` pair belong to the second ``sub``:
.. code-block:: bash
.. code-block:: none
mlr put '$b = sub($a, "(..)_(...)", "\2-\1"); $c = sub($a, "(..)_(.)(..)", ":\1:\2:\3")'
* Captures endure for the entirety of a ``put`` for the ``=~`` and ``!=~`` operators. For example, here the ``\1,\2`` are set by the ``=~`` operator and are used by both subsequent assignment statements:
.. code-block:: bash
.. code-block:: none
mlr put '$a =~ "(..)_(....); $b = "left_\1"; $c = "right_\2"'
* The captures are not retained across multiple puts. For example, here the ``\1,\2`` won't be expanded from the regex capture:
.. code-block:: bash
.. code-block:: none
mlr put '$a =~ "(..)_(....)' then {... something else ...} then put '$b = "left_\1"; $c = "right_\2"'
* Captures are ignored in ``filter`` for the ``=~`` and ``!=~`` operators. For example, there is no mechanism provided to refer to the first ``(..)`` as ``\1`` or to the second ``(....)`` as ``\2`` in the following filter statement:
.. code-block:: bash
.. code-block:: none
mlr filter '$a =~ "(..)_(....)'
@ -413,7 +413,7 @@ The short of it is that Miller does this transparently for you so you needn't th
Implementation details of this, for the interested: integer adds and subtracts overflow by at most one bit so it suffices to check sign-changes. Thus, Miller allows you to add and subtract arbitrary 64-bit signed integers, converting only to float precisely when the result is less than -2\ :sup:`63` or greater than 2\ :sup:`63`\ -1. Multiplies, on the other hand, can overflow by a word size and a sign-change technique does not suffice to detect overflow. Instead Miller tests whether the floating-point product exceeds the representable integer range. Now, 64-bit integers have 64-bit precision while IEEE-doubles have only 52-bit mantissas -- so, there are 53 bits including implicit leading one. The following experiment explicitly demonstrates the resolution at this range:
.. code-block:: bash
.. code-block:: none
64-bit integer 64-bit integer Casted to double Back to 64-bit
in hex in decimal integer

26
docs/sql-examples.rst
View file

@ -13,7 +13,7 @@ I like to produce SQL-query output with header-column and tab delimiter: this is
For example, using default output formatting in ``mysql`` we get formatting like Miller's ``--opprint --barred``:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mysql --database=mydb -e 'show columns in mytable'
@ -29,7 +29,7 @@ For example, using default output formatting in ``mysql`` we get formatting like
Using ``mysql``'s ``-B`` we get TSV output:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mysql --database=mydb -B -e 'show columns in mytable' | mlr --itsvlite --opprint cat
@ -42,7 +42,7 @@ Using ``mysql``'s ``-B`` we get TSV output:
Since Miller handles TSV output, we can do as much or as little processing as we want in the SQL query, then send the rest on to Miller. This includes outputting as JSON, doing further selects/joins in Miller, doing stats, etc. etc.:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mysql --database=mydb -B -e 'show columns in mytable' | mlr --itsvlite --ojson --jlistwrap --jvstack cat
@ -89,12 +89,12 @@ Since Miller handles TSV output, we can do as much or as little processing as we
}
]
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mysql --database=mydb -B -e 'select * from mytable' > query.tsv
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from query.tsv --t2p stats1 -a count -f id -g category,assigned_to
@ -118,7 +118,7 @@ One use of NIDX (value-only, no keys) format is for loading up SQL tables.
Create and load SQL table:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
mysql> CREATE TABLE abixy(
@ -130,19 +130,19 @@ Create and load SQL table:
);
Query OK, 0 rows affected (0.01 sec)
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
bash$ mlr --onidx --fs comma cat data/medium > medium.nidx
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
mysql> LOAD DATA LOCAL INFILE 'medium.nidx' REPLACE INTO TABLE abixy FIELDS TERMINATED BY ',' ;
Query OK, 10000 rows affected (0.07 sec)
Records: 10000 Deleted: 0 Skipped: 0 Warnings: 0
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
mysql> SELECT COUNT(*) AS count FROM abixy;
@ -153,7 +153,7 @@ Create and load SQL table:
+-------+
1 row in set (0.00 sec)
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
mysql> SELECT * FROM abixy LIMIT 10;
@ -174,7 +174,7 @@ Create and load SQL table:
Aggregate counts within SQL:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
mysql> SELECT a, b, COUNT(*) AS count FROM abixy GROUP BY a, b ORDER BY COUNT DESC;
@ -211,7 +211,7 @@ Aggregate counts within SQL:
Aggregate counts within Miller:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint uniq -c -g a,b then sort -nr count data/medium
@ -234,7 +234,7 @@ Aggregate counts within Miller:
Pipe SQL output to aggregate counts within Miller:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mysql -D miller -B -e 'select * from abixy' | mlr --itsv --opprint uniq -c -g a,b then sort -nr count

26
docs/sql-examples.rst.in
View file

@ -10,7 +10,7 @@ I like to produce SQL-query output with header-column and tab delimiter: this is
For example, using default output formatting in ``mysql`` we get formatting like Miller's ``--opprint --barred``:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mysql --database=mydb -e 'show columns in mytable'
@ -26,7 +26,7 @@ For example, using default output formatting in ``mysql`` we get formatting like
Using ``mysql``'s ``-B`` we get TSV output:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mysql --database=mydb -B -e 'show columns in mytable' | mlr --itsvlite --opprint cat
@ -39,7 +39,7 @@ Using ``mysql``'s ``-B`` we get TSV output:
Since Miller handles TSV output, we can do as much or as little processing as we want in the SQL query, then send the rest on to Miller. This includes outputting as JSON, doing further selects/joins in Miller, doing stats, etc. etc.:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mysql --database=mydb -B -e 'show columns in mytable' | mlr --itsvlite --ojson --jlistwrap --jvstack cat
@ -86,12 +86,12 @@ Since Miller handles TSV output, we can do as much or as little processing as we
}
]
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mysql --database=mydb -B -e 'select * from mytable' > query.tsv
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from query.tsv --t2p stats1 -a count -f id -g category,assigned_to
@ -115,7 +115,7 @@ One use of NIDX (value-only, no keys) format is for loading up SQL tables.
Create and load SQL table:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
mysql> CREATE TABLE abixy(
@ -127,19 +127,19 @@ Create and load SQL table:
);
Query OK, 0 rows affected (0.01 sec)
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
bash$ mlr --onidx --fs comma cat data/medium > medium.nidx
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
mysql> LOAD DATA LOCAL INFILE 'medium.nidx' REPLACE INTO TABLE abixy FIELDS TERMINATED BY ',' ;
Query OK, 10000 rows affected (0.07 sec)
Records: 10000 Deleted: 0 Skipped: 0 Warnings: 0
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
mysql> SELECT COUNT(*) AS count FROM abixy;
@ -150,7 +150,7 @@ Create and load SQL table:
+-------+
1 row in set (0.00 sec)
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
mysql> SELECT * FROM abixy LIMIT 10;
@ -171,7 +171,7 @@ Create and load SQL table:
Aggregate counts within SQL:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
mysql> SELECT a, b, COUNT(*) AS count FROM abixy GROUP BY a, b ORDER BY COUNT DESC;
@ -208,7 +208,7 @@ Aggregate counts within SQL:
Aggregate counts within Miller:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint uniq -c -g a,b then sort -nr count data/medium
@ -231,7 +231,7 @@ Aggregate counts within Miller:
Pipe SQL output to aggregate counts within Miller:
.. code-block:: bash
.. code-block:: none
:emphasize-lines: 1,1
$ mysql -D miller -B -e 'select * from abixy' | mlr --itsv --opprint uniq -c -g a,b then sort -nr count

68
docs6/10min.rst
View file

@ -9,7 +9,7 @@ CSV-file examples
Suppose you have this CSV data file:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat example.csv
@ -27,7 +27,7 @@ Suppose you have this CSV data file:
``mlr cat`` is like cat -- it passes the data through unmodified:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --csv cat example.csv
@ -45,7 +45,7 @@ Suppose you have this CSV data file:
but it can also do format conversion (here, you can pretty-print in tabular format):
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint cat example.csv
@ -63,7 +63,7 @@ but it can also do format conversion (here, you can pretty-print in tabular form
``mlr head`` and ``mlr tail`` count records rather than lines. Whether you're getting the first few records or the last few, the CSV header is included either way:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --csv head -n 4 example.csv
@ -73,7 +73,7 @@ but it can also do format conversion (here, you can pretty-print in tabular form
red,circle,1,16,13.8103,2.9010
red,square,0,48,77.5542,7.4670
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --csv tail -n 4 example.csv
@ -85,7 +85,7 @@ but it can also do format conversion (here, you can pretty-print in tabular form
You can sort primarily alphabetically on one field, then secondarily numerically descending on another field:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint sort -f shape -nr index example.csv
@ -103,7 +103,7 @@ You can sort primarily alphabetically on one field, then secondarily numerically
You can use ``cut`` to retain only specified fields, in the same order they appeared in the input data:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint cut -f flag,shape example.csv
@ -121,7 +121,7 @@ You can use ``cut`` to retain only specified fields, in the same order they appe
You can also use ``cut -o`` to retain only specified fields in your preferred order:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint cut -o -f flag,shape example.csv
@ -139,7 +139,7 @@ You can also use ``cut -o`` to retain only specified fields in your preferred or
You can use ``cut -x`` to omit fields you don't care about:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint cut -x -f flag,shape example.csv
@ -157,7 +157,7 @@ You can use ``cut -x`` to omit fields you don't care about:
You can use ``filter`` to keep only records you care about:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint filter '$color == "red"' example.csv
@ -167,7 +167,7 @@ You can use ``filter`` to keep only records you care about:
red square 0 48 77.5542 7.4670
red square 0 64 77.1991 9.5310
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint filter '$color == "red" && $flag == 1' example.csv
@ -177,7 +177,7 @@ You can use ``filter`` to keep only records you care about:
You can use ``put`` to create new fields which are computed from other fields:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint put '$ratio = $quantity / $rate; $color_shape = $color . "_" . $shape' example.csv
@ -195,7 +195,7 @@ You can use ``put`` to create new fields which are computed from other fields:
Even though Miller's main selling point is name-indexing, sometimes you really want to refer to a field name by its positional index. Use ``$[[3]]`` to access the name of field 3 or ``$[[[3]]]`` to access the value of field 3:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint put '$[[3]] = "NEW"' example.csv
@ -211,7 +211,7 @@ Even though Miller's main selling point is name-indexing, sometimes you really w
yellow circle 1 87 63.5058 8.3350
purple square 0 91 72.3735 8.2430
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint put '$[[[3]]] = "NEW"' example.csv
@ -232,7 +232,7 @@ JSON-file examples
OK, CSV and pretty-print are fine. But Miller can also convert between a few other formats -- let's take a look at JSON output:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --ojson put '$ratio = $quantity/$rate; $shape = toupper($shape)' example.csv
@ -254,7 +254,7 @@ Now suppose you want to sort the data on a given column, *and then* take the top
Here are the records with the top three ``index`` values:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint sort -f shape -nr index then head -n 3 example.csv
@ -265,7 +265,7 @@ Here are the records with the top three ``index`` values:
Lots of Miller commands take a ``-g`` option for group-by: here, ``head -n 1 -g shape`` outputs the first record for each distinct value of the ``shape`` field. This means we're finding the record with highest ``index`` field for each distinct ``shape`` field:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint sort -f shape -nr index then head -n 1 -g shape example.csv
@ -276,7 +276,7 @@ Lots of Miller commands take a ``-g`` option for group-by: here, ``head -n 1 -g
Statistics can be computed with or without group-by field(s):
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint --from example.csv stats1 -a count,min,mean,max -f quantity -g shape
@ -285,7 +285,7 @@ Statistics can be computed with or without group-by field(s):
square 4 72.373500 76.601150 79.277800
circle 3 13.810300 47.098200 63.978500
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint --from example.csv stats1 -a count,min,mean,max -f quantity -g shape,color
@ -299,7 +299,7 @@ Statistics can be computed with or without group-by field(s):
If your output has a lot of columns, you can use XTAB format to line things up vertically for you instead:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --oxtab --from example.csv stats1 -a p0,p10,p25,p50,p75,p90,p99,p100 -f rate
@ -321,14 +321,14 @@ Often we want to print output to the screen. Miller does this by default, as we'
Sometimes we want to print output to another file: just use **> outputfilenamegoeshere** at the end of your command:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
% mlr --icsv --opprint cat example.csv > newfile.csv
# Output goes to the new file;
# nothing is printed to the screen.
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
% cat newfile.csv
@ -347,12 +347,12 @@ Sometimes we want to print output to another file: just use **> outputfilenamego
Other times we just want our files to be **changed in-place**: just use **mlr -I**:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
% cp example.csv newfile.txt
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
% cat newfile.txt
@ -368,12 +368,12 @@ Other times we just want our files to be **changed in-place**: just use **mlr -I
yellow,circle,1,87,63.5058,8.3350
purple,square,0,91,72.3735,8.2430
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
% mlr -I --icsv --opprint cat newfile.txt
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
% cat newfile.txt
@ -391,7 +391,7 @@ Other times we just want our files to be **changed in-place**: just use **mlr -I
Also using ``mlr -I`` you can bulk-operate on lots of files: e.g.:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
mlr -I --csv cut -x -f unwanted_column_name *.csv
@ -400,12 +400,12 @@ If you like, you can first copy off your original data somewhere else, before do
Lastly, using ``tee`` within ``put``, you can split your input data into separate files per one or more field names:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --csv --from example.csv put -q 'tee > $shape.".csv", $*'
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat circle.csv
@ -414,7 +414,7 @@ Lastly, using ``tee`` within ``put``, you can split your input data into separat
yellow,circle,1,73,63.9785,4.2370
yellow,circle,1,87,63.5058,8.3350
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat square.csv
@ -424,7 +424,7 @@ Lastly, using ``tee`` within ``put``, you can split your input data into separat
red,square,0,64,77.1991,9.5310
purple,square,0,91,72.3735,8.2430
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat triangle.csv
@ -440,7 +440,7 @@ What's a CSV file, really? It's an array of rows, or *records*, each being a lis
For example, if you have:
.. code-block::
.. code-block:: none
shape,flag,index
circle,1,24
@ -448,7 +448,7 @@ For example, if you have:
then that's a way of saying:
.. code-block::
.. code-block:: none
shape=circle,flag=1,index=24
shape=square,flag=0,index=36
@ -458,7 +458,7 @@ Data written this way are called **DKVP**, for *delimited key-value pairs*.
We've also already seen other ways to write the same data:
.. code-block::
.. code-block:: none
CSV PPRINT JSON
shape,flag,index shape flag index [

20
docs6/10min.rst.in
View file

@ -93,14 +93,14 @@ Often we want to print output to the screen. Miller does this by default, as we'
Sometimes we want to print output to another file: just use **> outputfilenamegoeshere** at the end of your command:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
% mlr --icsv --opprint cat example.csv > newfile.csv
# Output goes to the new file;
# nothing is printed to the screen.
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
% cat newfile.csv
@ -119,12 +119,12 @@ Sometimes we want to print output to another file: just use **> outputfilenamego
Other times we just want our files to be **changed in-place**: just use **mlr -I**:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
% cp example.csv newfile.txt
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
% cat newfile.txt
@ -140,12 +140,12 @@ Other times we just want our files to be **changed in-place**: just use **mlr -I
yellow,circle,1,87,63.5058,8.3350
purple,square,0,91,72.3735,8.2430
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
% mlr -I --icsv --opprint cat newfile.txt
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
% cat newfile.txt
@ -163,7 +163,7 @@ Other times we just want our files to be **changed in-place**: just use **mlr -I
Also using ``mlr -I`` you can bulk-operate on lots of files: e.g.:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
mlr -I --csv cut -x -f unwanted_column_name *.csv
@ -187,7 +187,7 @@ What's a CSV file, really? It's an array of rows, or *records*, each being a lis
For example, if you have:
.. code-block::
.. code-block:: none
shape,flag,index
circle,1,24
@ -195,7 +195,7 @@ For example, if you have:
then that's a way of saying:
.. code-block::
.. code-block:: none
shape=circle,flag=1,index=24
shape=square,flag=0,index=36
@ -205,7 +205,7 @@ Data written this way are called **DKVP**, for *delimited key-value pairs*.
We've also already seen other ways to write the same data:
.. code-block::
.. code-block:: none
CSV PPRINT JSON
shape,flag,index shape flag index [

2
docs6/build.rst
View file

@ -101,7 +101,7 @@ In this example I am using version 3.4.0; of course that will change for subsequ
* Similarly for ``macports``: https://github.com/macports/macports-ports/blob/master/textproc/miller/Portfile.
* Social-media updates.
.. code-block::
.. code-block:: none
git remote add upstream https://github.com/Homebrew/homebrew-core # one-time setup only
git fetch upstream

2
docs6/build.rst.in
View file

@ -98,7 +98,7 @@ In this example I am using version 3.4.0; of course that will change for subsequ
* Similarly for ``macports``: https://github.com/macports/macports-ports/blob/master/textproc/miller/Portfile.
* Social-media updates.
.. code-block::
.. code-block:: none
git remote add upstream https://github.com/Homebrew/homebrew-core # one-time setup only
git fetch upstream

156
docs6/cookbook.rst
View file

@ -9,7 +9,7 @@ Headerless CSV on input or output
Sometimes we get CSV files which lack a header. For example:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/headerless.csv
@ -20,7 +20,7 @@ Sometimes we get CSV files which lack a header. For example:
You can use Miller to add a header. The ``--implicit-csv-header`` applies positionally indexed labels:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --csv --implicit-csv-header cat data/headerless.csv
@ -32,7 +32,7 @@ You can use Miller to add a header. The ``--implicit-csv-header`` applies positi
Following that, you can rename the positionally indexed labels to names with meaning for your context. For example:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --csv --implicit-csv-header label name,age,status data/headerless.csv
@ -44,7 +44,7 @@ Following that, you can rename the positionally indexed labels to names with mea
Likewise, if you need to produce CSV which is lacking its header, you can pipe Miller's output to the system command ``sed 1d``, or you can use Miller's ``--headerless-csv-output`` option:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ head -5 data/colored-shapes.dkvp | mlr --ocsv cat
@ -55,7 +55,7 @@ Likewise, if you need to produce CSV which is lacking its header, you can pipe M
red,square,0,48,0.9562743938458542,0.7467203085342884,0.7755423050923582,7.117831369597269
purple,triangle,0,51,0.4355354501763202,0.8591292672156728,0.8122903963006748,5.753094629505863
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ head -5 data/colored-shapes.dkvp | mlr --ocsv --headerless-csv-output cat
@ -67,7 +67,7 @@ Likewise, if you need to produce CSV which is lacking its header, you can pipe M
Lastly, often we say "CSV" or "TSV" when we have positionally indexed data in columns which are separated by commas or tabs, respectively. In this case it's perhaps simpler to **just use NIDX format** which was designed for this purpose. (See also :doc:`file-formats`.) For example:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --inidx --ifs comma --oxtab cut -f 1,3 data/headerless.csv
@ -88,7 +88,7 @@ Doing multiple joins
Suppose we have the following data:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat multi-join/input.csv
@ -104,7 +104,7 @@ Suppose we have the following data:
And we want to augment the ``id`` column with lookups from the following data files:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat multi-join/name-lookup.csv
@ -113,7 +113,7 @@ And we want to augment the ``id`` column with lookups from the following data fi
10,Bob
20,Carol
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat multi-join/status-lookup.csv
@ -124,7 +124,7 @@ And we want to augment the ``id`` column with lookups from the following data fi
We can run the input file through multiple ``join`` commands in a ``then``-chain:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint join -f multi-join/name-lookup.csv -j id then join -f multi-join/status-lookup.csv -j id multi-join/input.csv
@ -143,7 +143,7 @@ Bulk rename of fields
Suppose you want to replace spaces with underscores in your column names:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/spaces.csv
@ -154,7 +154,7 @@ Suppose you want to replace spaces with underscores in your column names:
The simplest way is to use ``mlr rename`` with ``-g`` (for global replace, not just first occurrence of space within each field) and ``-r`` for pattern-matching (rather than explicit single-column renames):
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --csv rename -g -r ' ,_' data/spaces.csv
@ -163,7 +163,7 @@ The simplest way is to use ``mlr rename`` with ``-g`` (for global replace, not j
2468,1357,3579
9987,3312,4543
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --csv --opprint rename -g -r ' ,_' data/spaces.csv
@ -174,7 +174,7 @@ The simplest way is to use ``mlr rename`` with ``-g`` (for global replace, not j
You can also do this with a for-loop:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/bulk-rename-for-loop.mlr
@ -184,7 +184,7 @@ You can also do this with a for-loop:
}
$* = newrec
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint put -f data/bulk-rename-for-loop.mlr data/spaces.csv
@ -198,7 +198,7 @@ Search-and-replace over all fields
How to do ``$name = gsub($name, "old", "new")`` for all fields?
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/sar.csv
@ -206,7 +206,7 @@ How to do ``$name = gsub($name, "old", "new")`` for all fields?
the quick,brown fox,jumped
over,the,lazy dogs
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/sar.mlr
@ -214,7 +214,7 @@ How to do ``$name = gsub($name, "old", "new")`` for all fields?
$[k] = gsub($[k], "e", "X");
}
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --csv put -f data/sar.mlr data/sar.csv
@ -227,7 +227,7 @@ Full field renames and reassigns
Using Miller 5.0.0's map literals and assigning to ``$*``, you can fully generalize :ref:`mlr rename <reference-verbs-rename>`, :ref:`mlr reorder <reference-verbs-reorder>`, etc.
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/small
@ -237,7 +237,7 @@ Using Miller 5.0.0's map literals and assigning to ``$*``, you can fully general
a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr put '
@ -266,7 +266,7 @@ Numbering and renumbering records
The ``awk``-like built-in variable ``NR`` is incremented for each input record:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/small
@ -276,7 +276,7 @@ The ``awk``-like built-in variable ``NR`` is incremented for each input record:
a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr put '$nr = NR' data/small
@ -288,7 +288,7 @@ The ``awk``-like built-in variable ``NR`` is incremented for each input record:
However, this is the record number within the original input stream -- not after any filtering you may have done:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr filter '$a == "wye"' then put '$nr = NR' data/small
@ -297,7 +297,7 @@ However, this is the record number within the original input stream -- not after
There are two good options here. One is to use the ``cat`` verb with ``-n``:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr filter '$a == "wye"' then cat -n data/small
@ -306,7 +306,7 @@ There are two good options here. One is to use the ``cat`` verb with ``-n``:
The other is to keep your own counter within the ``put`` DSL:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr filter '$a == "wye"' then put 'begin {@n = 1} $n = @n; @n += 1' data/small
@ -329,7 +329,7 @@ Data-cleaning examples
Here are some ways to use the type-checking options as described in :ref:`reference-dsl-type-tests-and-assertions` Suppose you have the following data file, with inconsistent typing for boolean. (Also imagine that, for the sake of discussion, we have a million-line file rather than a four-line file, so we can't see it all at once and some automation is called for.)
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/het-bool.csv
@ -341,7 +341,7 @@ Here are some ways to use the type-checking options as described in :ref:`refere
One option is to coerce everything to boolean, or integer:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint put '$reachable = boolean($reachable)' data/het-bool.csv
@ -351,7 +351,7 @@ One option is to coerce everything to boolean, or integer:
fred true
wilma true
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint put '$reachable = int(boolean($reachable))' data/het-bool.csv
@ -363,7 +363,7 @@ One option is to coerce everything to boolean, or integer:
A second option is to flag badly formatted data within the output stream:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint put '$format_ok = is_string($reachable)' data/het-bool.csv
@ -375,7 +375,7 @@ A second option is to flag badly formatted data within the output stream:
Or perhaps to flag badly formatted data outside the output stream:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint put 'if (!is_string($reachable)) {eprint "Malformed at NR=".NR} ' data/het-bool.csv
@ -388,7 +388,7 @@ Or perhaps to flag badly formatted data outside the output stream:
A third way is to abort the process on first instance of bad data:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --csv put '$reachable = asserting_string($reachable)' data/het-bool.csv
@ -403,7 +403,7 @@ Splitting nested fields
Suppose you have a TSV file like this:
.. code-block::
.. code-block:: none
a b
x z
@ -411,7 +411,7 @@ Suppose you have a TSV file like this:
The simplest option is to use :ref:`mlr nest <reference-verbs-nest>`:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --tsv nest --explode --values --across-records -f b --nested-fs : data/nested.tsv
@ -421,7 +421,7 @@ The simplest option is to use :ref:`mlr nest <reference-verbs-nest>`:
s v
s w
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --tsv nest --explode --values --across-fields -f b --nested-fs : data/nested.tsv
@ -435,7 +435,7 @@ While ``mlr nest`` is simplest, let's also take a look at a few ways to do this
One option to split out the colon-delimited values in the ``b`` column is to use ``splitnv`` to create an integer-indexed map and loop over it, adding new fields to the current record:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from data/nested.tsv --itsv --oxtab put 'o=splitnv($b, ":"); for (k,v in o) {$["p".k]=v}'
@ -451,7 +451,7 @@ One option to split out the colon-delimited values in the ``b`` column is to use
while another is to loop over the same map from ``splitnv`` and use it (with ``put -q`` to suppress printing the original record) to produce multiple records:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from data/nested.tsv --itsv --oxtab put -q 'o=splitnv($b, ":"); for (k,v in o) {x=mapsum($*, {"b":v}); emit x}'
@ -467,7 +467,7 @@ while another is to loop over the same map from ``splitnv`` and use it (with ``p
a s
b w
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from data/nested.tsv --tsv put -q 'o=splitnv($b, ":"); for (k,v in o) {x=mapsum($*, {"b":v}); emit x}'
@ -482,7 +482,7 @@ Showing differences between successive queries
Suppose you have a database query which you run at one point in time, producing the output on the left, then again later producing the output on the right:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/previous_counters.csv
@ -492,7 +492,7 @@ Suppose you have a database query which you run at one point in time, producing
orange,694
purple,12
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/current_counters.csv
@ -506,12 +506,12 @@ And, suppose you want to compute the differences in the counters between adjacen
First, rename counter columns to make them distinct:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --csv rename count,previous_count data/previous_counters.csv > data/prevtemp.csv
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/prevtemp.csv
@ -521,12 +521,12 @@ First, rename counter columns to make them distinct:
orange,694
purple,12
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --csv rename count,current_count data/current_counters.csv > data/currtemp.csv
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/currtemp.csv
@ -538,7 +538,7 @@ First, rename counter columns to make them distinct:
Then, join on the key field(s), and use unsparsify to zero-fill counters absent on one side but present on the other. Use ``--ul`` and ``--ur`` to emit unpaired records (namely, purple on the left and yellow on the right):
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint \
@ -558,7 +558,7 @@ Finding missing dates
Suppose you have some date-stamped data which may (or may not) be missing entries for one or more dates:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ head -n 10 data/miss-date.csv
@ -573,7 +573,7 @@ Suppose you have some date-stamped data which may (or may not) be missing entrie
2012-03-12,11043
2012-03-13,11177
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ wc -l data/miss-date.csv
@ -581,7 +581,7 @@ Suppose you have some date-stamped data which may (or may not) be missing entrie
Since there are 1372 lines in the data file, some automation is called for. To find the missing dates, you can convert the dates to seconds since the epoch using ``strptime``, then compute adjacent differences (the ``cat -n`` simply inserts record-counters):
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from data/miss-date.csv --icsv \
@ -602,7 +602,7 @@ Since there are 1372 lines in the data file, some automation is called for. To f
Then, filter for adjacent difference not being 86400 (the number of seconds in a day):
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from data/miss-date.csv --icsv \
@ -615,7 +615,7 @@ Then, filter for adjacent difference not being 86400 (the number of seconds in a
Given this, it's now easy to see where the gaps are:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr cat -n then filter '$n >= 770 && $n <= 780' data/miss-date.csv
@ -631,7 +631,7 @@ Given this, it's now easy to see where the gaps are:
n=779,1=2014-04-23,2=130849
n=780,1=2014-04-24,2=131026
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr cat -n then filter '$n >= 1115 && $n <= 1125' data/miss-date.csv
@ -657,7 +657,7 @@ Two-pass algorithms: computation of percentages
For example, mapping numeric values down a column to the percentage between their min and max values is two-pass: on the first pass you find the min and max values, then on the second, map each record's value to a percentage.
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from data/small --opprint put -q '
@ -688,7 +688,7 @@ Two-pass algorithms: line-number ratios
Similarly, finding the total record count requires first reading through all the data:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint --from data/small put -q '
@ -714,7 +714,7 @@ Two-pass algorithms: records having max value
The idea is to retain records having the largest value of ``n`` in the following data:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --itsv --opprint cat data/maxrows.tsv
@ -753,7 +753,7 @@ The idea is to retain records having the largest value of ``n`` in the following
Of course, the largest value of ``n`` isn't known until after all data have been read. Using an out-of-stream variable we can retain all records as they are read, then filter them at the end:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/maxrows.mlr
@ -773,7 +773,7 @@ Of course, the largest value of ``n`` isn't known until after all data have been
}
}
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --itsv --opprint put -q -f data/maxrows.mlr data/maxrows.tsv
@ -793,7 +793,7 @@ Rectangularizing data
Suppose you have a method (in whatever language) which is printing things of the form
.. code-block::
.. code-block:: none
outer=1
outer=2
@ -801,7 +801,7 @@ Suppose you have a method (in whatever language) which is printing things of the
and then calls another method which prints things of the form
.. code-block::
.. code-block:: none
middle=10
middle=11
@ -813,7 +813,7 @@ and then calls another method which prints things of the form
and then, perhaps, that second method calls a third method which prints things of the form
.. code-block::
.. code-block:: none
inner1=100,inner2=101
inner1=120,inner2=121
@ -825,7 +825,7 @@ and then, perhaps, that second method calls a third method which prints things o
with the result that your program's output is
.. code-block::
.. code-block:: none
outer=1
middle=10
@ -847,7 +847,7 @@ with the result that your program's output is
The idea here is that middles starting with a 1 belong to the outer value of 1, and so on. (For example, the outer values might be account IDs, the middle values might be invoice IDs, and the inner values might be invoice line-items.) If you want all the middle and inner lines to have the context of which outers they belong to, you can modify your software to pass all those through your methods. Alternatively, don't refactor your code just to handle some ad-hoc log-data formatting -- instead, use the following to rectangularize the data. The idea is to use an out-of-stream variable to accumulate fields across records. Clear that variable when you see an outer ID; accumulate fields; emit output when you see the inner IDs.
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from data/rect.txt put -q '
@ -873,7 +873,7 @@ Regularizing ragged CSV
Miller handles compliant CSV: in particular, it's an error if the number of data fields in a given data line don't match the number of header lines. But in the event that you have a CSV file in which some lines have less than the full number of fields, you can use Miller to pad them out. The trick is to use NIDX format, for which each line stands on its own without respect to a header line.
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/ragged.csv
@ -882,7 +882,7 @@ Miller handles compliant CSV: in particular, it's an error if the number of data
4,5
6,7,8,9
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from data/ragged.csv --fs comma --nidx put '
@ -900,7 +900,7 @@ Miller handles compliant CSV: in particular, it's an error if the number of data
or, more simply,
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from data/ragged.csv --fs comma --nidx put '
@ -919,7 +919,7 @@ Feature-counting
Suppose you have some heterogeneous data like this:
.. code-block::
.. code-block:: none
{ "qoh": 29874, "rate": 1.68, "latency": 0.02 }
{ "name": "alice", "uid": 572 }
@ -936,7 +936,7 @@ Suppose you have some heterogeneous data like this:
A reasonable question to ask is, how many occurrences of each field are there? And, what percentage of total row count has each of them? Since the denominator of the percentage is not known until the end, this is a two-pass algorithm:
.. code-block::
.. code-block:: none
for (key in $*) {
@key_counts[key] += 1;
@ -954,7 +954,7 @@ A reasonable question to ask is, how many occurrences of each field are there? A
Then
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --json put -q -f data/feature-count.mlr data/features.json
@ -972,7 +972,7 @@ Then
{ "key": "uid", "key_fraction": 0.250000 }
{ "key": "uid2", "key_fraction": 0.083333 }
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --ijson --opprint put -q -f data/feature-count.mlr data/features.json
@ -1002,7 +1002,7 @@ The previous section discussed how to fill out missing data fields within CSV wi
For example, suppose you have JSON input like this:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/sparse.json
@ -1013,7 +1013,7 @@ For example, suppose you have JSON input like this:
There are field names ``a``, ``b``, ``v``, ``u``, ``x``, ``w`` in the data -- but not all in every record. Since we don't know the names of all the keys until we've read them all, this needs to be a two-pass algorithm. On the first pass, remember all the unique key names and all the records; on the second pass, loop through the records filling in absent values, then producing output. Use ``put -q`` since we don't want to produce per-record output, only emitting output in the ``end`` block:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/unsparsify.mlr
@ -1044,7 +1044,7 @@ There are field names ``a``, ``b``, ``v``, ``u``, ``x``, ``w`` in the data -- bu
}
}
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --json put -q -f data/unsparsify.mlr data/sparse.json
@ -1053,7 +1053,7 @@ There are field names ``a``, ``b``, ``v``, ``u``, ``x``, ``w`` in the data -- bu
{ "a": 1, "b": "", "v": 2, "u": "", "x": 3, "w": "" }
{ "a": "", "b": "", "v": 1, "u": "", "x": "", "w": 2 }
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --ijson --ocsv put -q -f data/unsparsify.mlr data/sparse.json
@ -1063,7 +1063,7 @@ There are field names ``a``, ``b``, ``v``, ``u``, ``x``, ``w`` in the data -- bu
1,,2,,3,
,,1,,,2
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --ijson --opprint put -q -f data/unsparsify.mlr data/sparse.json
@ -1080,13 +1080,13 @@ Parsing log-file output
This, of course, depends highly on what's in your log files. But, as an example, suppose you have log-file lines such as
.. code-block::
.. code-block:: none
2015-10-08 08:29:09,445 INFO com.company.path.to.ClassName @ [sometext] various/sorts/of data {& punctuation} hits=1 status=0 time=2.378
I prefer to pre-filter with ``grep`` and/or ``sed`` to extract the structured text, then hand that to Miller. Example:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
grep 'various sorts' *.log | sed 's/.*} //' | mlr --fs space --repifs --oxtab stats1 -a min,p10,p50,p90,max -f time -g status
@ -1098,7 +1098,7 @@ Memoization with out-of-stream variables
The recursive function for the Fibonacci sequence is famous for its computational complexity. Namely, using f(0)=1, f(1)=1, f(n)=f(n-1)+f(n-2) for n>=2, the evaluation tree branches left as well as right at each non-trivial level, resulting in millions or more paths to the root 0/1 nodes for larger n. This program
.. code-block::
.. code-block:: none
mlr --ofmt '%.9lf' --opprint seqgen --start 1 --stop 28 then put '
func f(n) {
@ -1119,7 +1119,7 @@ The recursive function for the Fibonacci sequence is famous for its computationa
produces output like this:
.. code-block::
.. code-block:: none
i o fcount seconds_delta
1 1 1 0
@ -1153,7 +1153,7 @@ produces output like this:
Note that the time it takes to evaluate the function is blowing up exponentially as the input argument increases. Using ``@``-variables, which persist across records, we can cache and reuse the results of previous computations:
.. code-block::
.. code-block:: none
mlr --ofmt '%.9lf' --opprint seqgen --start 1 --stop 28 then put '
func f(n) {
@ -1176,7 +1176,7 @@ Note that the time it takes to evaluate the function is blowing up exponentially
with output like this:
.. code-block::
.. code-block:: none
i o fcount seconds_delta
1 1 1 0

8
docs6/cookbook.rst.in
View file

@ -323,13 +323,13 @@ Parsing log-file output
This, of course, depends highly on what's in your log files. But, as an example, suppose you have log-file lines such as
.. code-block::
.. code-block:: none
2015-10-08 08:29:09,445 INFO com.company.path.to.ClassName @ [sometext] various/sorts/of data {& punctuation} hits=1 status=0 time=2.378
I prefer to pre-filter with ``grep`` and/or ``sed`` to extract the structured text, then hand that to Miller. Example:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
grep 'various sorts' *.log | sed 's/.*} //' | mlr --fs space --repifs --oxtab stats1 -a min,p10,p50,p90,max -f time -g status
@ -345,7 +345,7 @@ POKI_INCLUDE_ESCAPED(data/fibo-uncached.sh)HERE
produces output like this:
.. code-block::
.. code-block:: none
i o fcount seconds_delta
1 1 1 0
@ -383,7 +383,7 @@ POKI_INCLUDE_ESCAPED(data/fibo-cached.sh)HERE
with output like this:
.. code-block::
.. code-block:: none
i o fcount seconds_delta
1 1 1 0

28
docs6/cookbook2.rst
View file

@ -9,7 +9,7 @@ Randomly selecting words from a list
Given this `word list <https://github.com/johnkerl/miller/blob/master/docs/data/english-words.txt>`_, first take a look to see what the first few lines look like:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ head data/english-words.txt
@ -26,7 +26,7 @@ Given this `word list <https://github.com/johnkerl/miller/blob/master/docs/data/
Then the following will randomly sample ten words with four to eight characters in them:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from data/english-words.txt --nidx filter -S 'n=strlen($1);4<=n&&n<=8' then sample -k 10
@ -48,7 +48,7 @@ These are simple *n*-grams as `described here <http://johnkerl.org/randspell/ran
The idea is that words from the input file are consumed, then taken apart and pasted back together in ways which imitate the letter-to-letter transitions found in the word list -- giving us automatically generated words in the same vein as *bromance* and *spork*:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --nidx --from ./ngrams/gsl-2000.txt put -q -f ./ngrams/ngfuncs.mlr -f ./ngrams/ng5.mlr
@ -74,7 +74,7 @@ Program timing
This admittedly artificial example demonstrates using Miller time and stats functions to introspectively acquire some information about Miller's own runtime. The ``delta`` function computes the difference between successive timestamps.
.. code-block::
.. code-block:: none
$ ruby -e '10000.times{|i|puts "i=#{i+1}"}' > lines.txt
@ -109,7 +109,7 @@ Computing interquartile ranges
For one or more specified field names, simply compute p25 and p75, then write the IQR as the difference of p75 and p25:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --oxtab stats1 -f x -a p25,p75 \
@ -121,7 +121,7 @@ For one or more specified field names, simply compute p25 and p75, then write th
For wildcarded field names, first compute p25 and p75, then loop over field names with ``p25`` in them:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --oxtab stats1 --fr '[i-z]' -a p25,p75 \
@ -146,7 +146,7 @@ Computing weighted means
This might be more elegantly implemented as an option within the ``stats1`` verb. Meanwhile, it's expressible within the DSL:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from data/medium put -q '
@ -184,7 +184,7 @@ Generating random numbers from various distributions
Here we can chain together a few simple building blocks:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat expo-sample.sh
@ -227,7 +227,7 @@ Namely:
The output is as follows:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ sh expo-sample.sh
@ -288,7 +288,7 @@ Sieve of Eratosthenes
The `Sieve of Eratosthenes <http://en.wikipedia.org/wiki/Sieve_of_Eratosthenes>`_ is a standard introductory programming topic. The idea is to find all primes up to some *N* by making a list of the numbers 1 to *N*, then striking out all multiples of 2 except 2 itself, all multiples of 3 except 3 itself, all multiples of 4 except 4 itself, and so on. Whatever survives that without getting marked is a prime. This is easy enough in Miller. Notice that here all the work is in ``begin`` and ``end`` statements; there is no file input (so we use ``mlr -n`` to keep Miller from waiting for input data).
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat programs/sieve.mlr
@ -325,7 +325,7 @@ The `Sieve of Eratosthenes <http://en.wikipedia.org/wiki/Sieve_of_Eratosthenes>`
}
}
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr -n put -f programs/sieve.mlr
@ -362,7 +362,7 @@ The `Mandelbrot set <http://en.wikipedia.org/wiki/Mandelbrot_set>`_ is also easi
The (approximate) computation of points in the complex plane which are and aren't members is just a few lines of complex arithmetic (see the Wikipedia article); how to render them is another task. Using graphics libraries you can create PNG or JPEG files, but another fun way to do this is by printing various characters to the screen:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat programs/mand.mlr
@ -469,7 +469,7 @@ The (approximate) computation of points in the complex plane which are and aren'
At standard resolution this makes a nice little ASCII plot:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr -n put -f ./programs/mand.mlr
@ -526,7 +526,7 @@ At standard resolution this makes a nice little ASCII plot:
But using a very small font size (as small as my Mac will let me go), and by choosing the coordinates to zoom in on a particular part of the complex plane, we can get a nice little picture:
.. code-block::
.. code-block:: none
#!/bin/bash
# Get the number of rows and columns from the terminal window dimensions

8
docs6/cookbook2.rst.in
View file

@ -6,7 +6,7 @@ Randomly selecting words from a list
Given this `word list <https://github.com/johnkerl/miller/blob/master/docs/data/english-words.txt>`_, first take a look to see what the first few lines look like:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ head data/english-words.txt
@ -23,7 +23,7 @@ Given this `word list <https://github.com/johnkerl/miller/blob/master/docs/data/
Then the following will randomly sample ten words with four to eight characters in them:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from data/english-words.txt --nidx filter -S 'n=strlen($1);4<=n&&n<=8' then sample -k 10
@ -45,7 +45,7 @@ These are simple *n*-grams as `described here <http://johnkerl.org/randspell/ran
The idea is that words from the input file are consumed, then taken apart and pasted back together in ways which imitate the letter-to-letter transitions found in the word list -- giving us automatically generated words in the same vein as *bromance* and *spork*:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --nidx --from ./ngrams/gsl-2000.txt put -q -f ./ngrams/ngfuncs.mlr -f ./ngrams/ng5.mlr
@ -135,7 +135,7 @@ POKI_RUN_COMMAND{{mlr -n put -f ./programs/mand.mlr}}HERE
But using a very small font size (as small as my Mac will let me go), and by choosing the coordinates to zoom in on a particular part of the complex plane, we can get a nice little picture:
.. code-block::
.. code-block:: none
#!/bin/bash
# Get the number of rows and columns from the terminal window dimensions

44
docs6/cookbook3.rst
View file

@ -9,7 +9,7 @@ Overview
One of Miller's strengths is its compact notation: for example, given input of the form
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ head -n 5 ../data/medium
@ -21,7 +21,7 @@ One of Miller's strengths is its compact notation: for example, given input of t
you can simply do
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --oxtab stats1 -a sum -f x ../data/medium
@ -29,7 +29,7 @@ you can simply do
or
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint stats1 -a sum -f x -g b ../data/medium
@ -42,7 +42,7 @@ or
rather than the more tedious
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --oxtab put -q '
@ -55,7 +55,7 @@ rather than the more tedious
or
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint put -q '
@ -80,14 +80,14 @@ The following examples compute some things using oosvars which are already compu
Mean without/with oosvars
----------------------------------------------------------------
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint stats1 -a mean -f x data/medium
x_mean
0.498602
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint put -q '
@ -104,7 +104,7 @@ Mean without/with oosvars
Keyed mean without/with oosvars
----------------------------------------------------------------
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint stats1 -a mean -f x -g a,b data/medium
@ -135,7 +135,7 @@ Keyed mean without/with oosvars
eks hat 0.500679
wye eks 0.530604
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint put -q '
@ -178,7 +178,7 @@ Keyed mean without/with oosvars
Variance and standard deviation without/with oosvars
----------------------------------------------------------------
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --oxtab stats1 -a count,sum,mean,var,stddev -f x data/medium
@ -188,7 +188,7 @@ Variance and standard deviation without/with oosvars
x_var 0.084270
x_stddev 0.290293
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat variance.mlr
@ -202,7 +202,7 @@ Variance and standard deviation without/with oosvars
emitf @n, @sumx, @sumx2, @mean, @var, @stddev
}
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --oxtab put -q -f variance.mlr data/medium
@ -218,14 +218,14 @@ You can also do this keyed, of course, imitating the keyed-mean example above.
Min/max without/with oosvars
----------------------------------------------------------------
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --oxtab stats1 -a min,max -f x data/medium
x_min 0.000045
x_max 0.999953
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --oxtab put -q '@x_min = min(@x_min, $x); @x_max = max(@x_max, $x); end{emitf @x_min, @x_max}' data/medium
@ -235,7 +235,7 @@ Min/max without/with oosvars
Keyed min/max without/with oosvars
----------------------------------------------------------------
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint stats1 -a min,max -f x -g a data/medium
@ -246,7 +246,7 @@ Keyed min/max without/with oosvars
zee 0.000549 0.999490
hat 0.000045 0.999953
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint --from data/medium put -q '
@ -266,7 +266,7 @@ Keyed min/max without/with oosvars
Delta without/with oosvars
----------------------------------------------------------------
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint step -a delta -f x data/small
@ -277,7 +277,7 @@ Delta without/with oosvars
eks wye 4 0.38139939387114097 0.13418874328430463 0.176796
wye pan 5 0.5732889198020006 0.8636244699032729 0.191890
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint put '$x_delta = is_present(@last) ? $x - @last : 0; @last = $x' data/small
@ -291,7 +291,7 @@ Delta without/with oosvars
Keyed delta without/with oosvars
----------------------------------------------------------------
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint step -a delta -f x -g a data/small
@ -302,7 +302,7 @@ Keyed delta without/with oosvars
eks wye 4 0.38139939387114097 0.13418874328430463 -0.377281
wye pan 5 0.5732889198020006 0.8636244699032729 0.368686
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint put '$x_delta = is_present(@last[$a]) ? $x - @last[$a] : 0; @last[$a]=$x' data/small
@ -316,7 +316,7 @@ Keyed delta without/with oosvars
Exponentially weighted moving averages without/with oosvars
----------------------------------------------------------------
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint step -a ewma -d 0.1 -f x data/small
@ -327,7 +327,7 @@ Exponentially weighted moving averages without/with oosvars
eks wye 4 0.38139939387114097 0.13418874328430463 0.370817
wye pan 5 0.5732889198020006 0.8636244699032729 0.391064
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint put '

12
docs6/customization.rst
View file

@ -9,30 +9,30 @@ How to use .mlrrc
Suppose you always use CSV files. Then instead of always having to type ``--csv`` as in
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
mlr --csv cut -x -f extra mydata.csv
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
mlr --csv sort -n id mydata.csv
and so on, you can instead put the following into your ``$HOME/.mlrrc``:
.. code-block::
.. code-block:: none
--csv
Then you can just type things like
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
mlr cut -x -f extra mydata.csv
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
mlr sort -n id mydata.csv
@ -56,7 +56,7 @@ What you can put in your .mlrrc
Here is an example ``.mlrrc file``:
.. code-block::
.. code-block:: none
# These are my preferred default settings for Miller

10
docs6/customization.rst.in
View file

@ -6,30 +6,30 @@ How to use .mlrrc
Suppose you always use CSV files. Then instead of always having to type ``--csv`` as in
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
mlr --csv cut -x -f extra mydata.csv
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
mlr --csv sort -n id mydata.csv
and so on, you can instead put the following into your ``$HOME/.mlrrc``:
.. code-block::
.. code-block:: none
--csv
Then you can just type things like
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
mlr cut -x -f extra mydata.csv
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
mlr sort -n id mydata.csv

34
docs6/data-examples.rst
View file

@ -11,7 +11,7 @@ The `flins.csv <data/flins.csv>`_ file is some sample data obtained from https:/
Vertical-tabular format is good for a quick look at CSV data layout -- seeing what columns you have to work with:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ head -n 2 data/flins.csv | mlr --icsv --oxtab cat
@ -22,7 +22,7 @@ Vertical-tabular format is good for a quick look at CSV data layout -- seeing wh
A few simple queries:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from data/flins.csv --icsv --opprint count-distinct -f county | head
@ -34,26 +34,26 @@ A few simple queries:
Duval 1
St. Johns 1
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from data/flins.csv --icsv --opprint count-distinct -f construction,line
Categorization of total insured value:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from data/flins.csv --icsv --opprint stats1 -a min,mean,max -f tiv_2012
tiv_2012_min tiv_2012_mean tiv_2012_max
19757.910000 1061531.463750 2785551.630000
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from data/flins.csv --icsv --opprint stats1 -a min,mean,max -f tiv_2012 -g construction,line
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from data/flins.csv --icsv --oxtab stats1 -a p0,p10,p50,p90,p95,p99,p100 -f hu_site_deductible
@ -65,7 +65,7 @@ Categorization of total insured value:
hu_site_deductible_p99
hu_site_deductible_p100
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from data/flins.csv --icsv --opprint stats1 -a p95,p99,p100 -f hu_site_deductible -g county then sort -f county | head
@ -77,7 +77,7 @@ Categorization of total insured value:
Seminole - - -
St. Johns - - -
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from data/flins.csv --icsv --oxtab stats2 -a corr,linreg-ols,r2 -f tiv_2011,tiv_2012
@ -87,7 +87,7 @@ Categorization of total insured value:
tiv_2011_tiv_2012_ols_n 8
tiv_2011_tiv_2012_r2 0.874904
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from data/flins.csv --icsv --opprint stats2 -a corr,linreg-ols,r2 -f tiv_2011,tiv_2012 -g county
@ -114,13 +114,13 @@ The `colored-shapes.dkvp <https://github.com/johnkerl/miller/blob/master/docs/da
Peek at the data:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ wc -l data/colored-shapes.dkvp
10078 data/colored-shapes.dkvp
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ head -n 6 data/colored-shapes.dkvp | mlr --opprint cat
@ -136,7 +136,7 @@ Look at uncategorized stats (using `creach <https://github.com/johnkerl/scripts/
Here it looks reasonable that ``u`` is unit-uniform; something's up with ``v`` but we can't yet see what:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --oxtab stats1 -a min,mean,max -f flag,u,v data/colored-shapes.dkvp | creach 3
@ -154,7 +154,7 @@ Here it looks reasonable that ``u`` is unit-uniform; something's up with ``v`` b
The histogram shows the different distribution of 0/1 flags:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint histogram -f flag,u,v --lo -0.1 --hi 1.1 --nbins 12 data/colored-shapes.dkvp
@ -174,7 +174,7 @@ The histogram shows the different distribution of 0/1 flags:
Look at univariate stats by color and shape. In particular, color-dependent flag probabilities pop out, aligning with their original Bernoulli probablities from the data-generator script:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint stats1 -a min,mean,max -f flag,u,v -g color then sort -f color data/colored-shapes.dkvp
@ -186,7 +186,7 @@ Look at univariate stats by color and shape. In particular, color-dependent flag
red 0 0.303167 1 0.000671 0.492560 0.999882 -0.092709 0.496535 1.072500
yellow 0 0.892427 1 0.001300 0.497129 0.999923 0.000711 0.510627 0.999919
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint stats1 -a min,mean,max -f flag,u,v -g shape then sort -f shape data/colored-shapes.dkvp
@ -197,14 +197,14 @@ Look at univariate stats by color and shape. In particular, color-dependent flag
Look at bivariate stats by color and shape. In particular, ``u,v`` pairwise correlation for red circles pops out:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint --right stats2 -a corr -f u,v,w,x data/colored-shapes.dkvp
u_v_corr w_x_corr
0.133418 -0.011320
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint --right stats2 -a corr -f u,v,w,x -g color,shape then sort -nr u_v_corr data/colored-shapes.dkvp

26
docs6/data-sharing.rst
View file

@ -11,7 +11,7 @@ DKVP I/O in Python
Here are the I/O routines:
.. code-block::
.. code-block:: none
#!/usr/bin/env python
@ -74,7 +74,7 @@ Here are the I/O routines:
And here is an example using them:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat polyglot-dkvp-io/example.py
@ -115,7 +115,7 @@ And here is an example using them:
Run as-is:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ python polyglot-dkvp-io/example.py < data/small
@ -127,7 +127,7 @@ Run as-is:
Run as-is, then pipe to Miller for pretty-printing:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ python polyglot-dkvp-io/example.py < data/small | mlr --opprint cat
@ -143,7 +143,7 @@ DKVP I/O in Ruby
Here are the I/O routines:
.. code-block::
.. code-block:: none
#!/usr/bin/env ruby
@ -200,7 +200,7 @@ Here are the I/O routines:
And here is an example using them:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat polyglot-dkvp-io/example.rb
@ -231,7 +231,7 @@ And here is an example using them:
Run as-is:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ ruby -I./polyglot-dkvp-io polyglot-dkvp-io/example.rb data/small
@ -243,7 +243,7 @@ Run as-is:
Run as-is, then pipe to Miller for pretty-printing:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ ruby -I./polyglot-dkvp-io polyglot-dkvp-io/example.rb data/small | mlr --opprint cat
@ -269,7 +269,7 @@ Running shell commands
The :ref:`reference-dsl-system` DSL function allows you to run a specific shell command and put its output -- minus the final newline -- into a record field. The command itself is any string, either a literal string, or a concatenation of strings, perhaps including other field values or what have you.
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint put '$o = system("echo hello world")' data/small
@ -280,7 +280,7 @@ The :ref:`reference-dsl-system` DSL function allows you to run a specific shell
eks wye 4 0.38139939387114097 0.13418874328430463 hello world
wye pan 5 0.5732889198020006 0.8636244699032729 hello world
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint put '$o = system("echo {" . NR . "}")' data/small
@ -291,7 +291,7 @@ The :ref:`reference-dsl-system` DSL function allows you to run a specific shell
eks wye 4 0.38139939387114097 0.13418874328430463 {4}
wye pan 5 0.5732889198020006 0.8636244699032729 {5}
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint put '$o = system("echo -n ".$a."| sha1sum")' data/small
@ -307,7 +307,7 @@ Note that running a subprocess on every record takes a non-trivial amount of tim
..
hard-coded, not live-code, since %N doesn't exist on all platforms
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint put '$t=system("date +%s.%N")' then step -a delta -f t data/small
@ -318,7 +318,7 @@ Note that running a subprocess on every record takes a non-trivial amount of tim
eks wye 4 0.38139939387114097 0.13418874328430463 1568774318.516547441 0.000929
wye pan 5 0.5732889198020006 0.8636244699032729 1568774318.517518828 0.000971
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint put '$t=systime()' then step -a delta -f t data/small

4
docs6/data-sharing.rst.in
View file

@ -67,7 +67,7 @@ Note that running a subprocess on every record takes a non-trivial amount of tim
..
hard-coded, not live-code, since %N doesn't exist on all platforms
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint put '$t=system("date +%s.%N")' then step -a delta -f t data/small
@ -78,7 +78,7 @@ Note that running a subprocess on every record takes a non-trivial amount of tim
eks wye 4 0.38139939387114097 0.13418874328430463 1568774318.516547441 0.000929
wye pan 5 0.5732889198020006 0.8636244699032729 1568774318.517518828 0.000971
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint put '$t=systime()' then step -a delta -f t data/small

104
docs6/faq.rst
View file

@ -21,7 +21,7 @@ Also try ``od -xcv`` and/or ``cat -e`` on your file to check for non-printable c
Diagnosing delimiter specifications
----------------------------------------------------------------
.. code-block::
.. code-block:: none
# Use the `file` command to see if there are CR/LF terminators (in this case,
# there are not):
@ -88,7 +88,7 @@ How do I suppress numeric conversion?
Within ``mlr put`` and ``mlr filter``, the default behavior for scanning input records is to parse them as integer, if possible, then as float, if possible, else leave them as string:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/scan-example-1.tbl
@ -98,7 +98,7 @@ Within ``mlr put`` and ``mlr filter``, the default behavior for scanning input r
3x
hello
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --pprint put '$copy = $value; $type = typeof($value)' data/scan-example-1.tbl
@ -118,7 +118,7 @@ This is a sensible default: you should be able to put ``'$z = $x + $y'`` without
But now suppose you have data like these:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/scan-example-2.tbl
@ -134,7 +134,7 @@ But now suppose you have data like these:
0009
0010
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --pprint put '$copy = $value; $type = typeof($value)' data/scan-example-2.tbl
@ -164,7 +164,7 @@ Taken individually the rules make sense; taken collectively they produce a mishm
The solution is to **use the -S flag** for ``mlr put`` and/or ``mlr filter``. Then all field values are left as string. You can type-coerce on demand using syntax like ``'$z = int($x) + float($y)'``. (See also :doc:`reference-dsl`; see also https://github.com/johnkerl/miller/issues/150.)
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --pprint put -S '$copy = $value; $type = typeof($value)' data/scan-example-2.tbl
@ -187,7 +187,7 @@ Then-chaining found in Miller is intended to function the same as Unix pipes, bu
First, look at the input data:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/then-example.csv
@ -200,7 +200,7 @@ First, look at the input data:
Next, run the first step of your command, omitting anything from the first ``then`` onward:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint count-distinct -f Status,Payment_Type data/then-example.csv
@ -212,7 +212,7 @@ Next, run the first step of your command, omitting anything from the first ``the
After that, run it with the next ``then`` step included:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --opprint count-distinct -f Status,Payment_Type then sort -nr count data/then-example.csv
@ -226,7 +226,7 @@ Now if you use ``then`` to include another verb after that, the columns ``Status
Note, by the way, that you'll get the same results using pipes:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --csv count-distinct -f Status,Payment_Type data/then-example.csv | mlr --icsv --opprint sort -nr count
@ -241,44 +241,44 @@ I assigned $9 and it's not 9th
Miller records are ordered lists of key-value pairs. For NIDX format, DKVP format when keys are missing, or CSV/CSV-lite format with ``--implicit-csv-header``, Miller will sequentially assign keys of the form ``1``, ``2``, etc. But these are not integer array indices: they're just field names taken from the initial field ordering in the input data.
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ echo x,y,z | mlr --dkvp cat
1=x,2=y,3=z
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ echo x,y,z | mlr --dkvp put '$6="a";$4="b";$55="cde"'
1=x,2=y,3=z,6=a,4=b,55=cde
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ echo x,y,z | mlr --nidx cat
x,y,z
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ echo x,y,z | mlr --csv --implicit-csv-header cat
1,2,3
x,y,z
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ echo x,y,z | mlr --dkvp rename 2,999
1=x,999=y,3=z
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ echo x,y,z | mlr --dkvp rename 2,newname
1=x,newname=y,3=z
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ echo x,y,z | mlr --csv --implicit-csv-header reorder -f 3,1,2
@ -290,7 +290,7 @@ How can I filter by date?
Given input like
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat dates.csv
@ -301,7 +301,7 @@ Given input like
we can use ``strptime`` to parse the date field into seconds-since-epoch and then do numeric comparisons. Simply match your input dataset's date-formatting to the :ref:`reference-dsl-strptime` format-string. For example:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --csv filter 'strptime($date, "%Y-%m-%d") > strptime("2018-03-03", "%Y-%m-%d")' dates.csv
@ -315,7 +315,7 @@ How can I handle commas-as-data in various formats?
:doc:`CSV <file-formats>` handles this well and by design:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat commas.csv
@ -325,7 +325,7 @@ How can I handle commas-as-data in various formats?
Likewise :ref:`file-formats-json`:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --ojson cat commas.csv
@ -334,7 +334,7 @@ Likewise :ref:`file-formats-json`:
For Miller's :ref:`vertical-tabular format <file-formats-xtab>` there is no escaping for carriage returns, but commas work fine:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --oxtab cat commas.csv
@ -346,7 +346,7 @@ For Miller's :ref:`vertical-tabular format <file-formats-xtab>` there is no esca
But for :ref:`Key-value_pairs <file-formats-dkvp>` and :ref:`index-numbered <file-formats-nidx>`, commas are the default field separator. And -- as of Miller 5.4.0 anyway -- there is no CSV-style double-quote-handling like there is for CSV. So commas within the data look like delimiters:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --odkvp cat commas.csv
@ -355,7 +355,7 @@ But for :ref:`Key-value_pairs <file-formats-dkvp>` and :ref:`index-numbered <fil
One solution is to use a different delimiter, such as a pipe character:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --odkvp --ofs pipe cat commas.csv
@ -365,7 +365,7 @@ One solution is to use a different delimiter, such as a pipe character:
To be extra-sure to avoid data/delimiter clashes, you can also use control
characters as delimiters -- here, control-A:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --odkvp --ofs '\001' cat commas.csv | cat -v
@ -377,7 +377,7 @@ How can I handle field names with special symbols in them?
Simply surround the field names with curly braces:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ echo 'x.a=3,y:b=4,z/c=5' | mlr put '${product.all} = ${x.a} * ${y:b} * ${z/c}'
@ -388,19 +388,19 @@ How to escape '?' in regexes?
One way is to use square brackets; an alternative is to use simple string-substitution rather than a regular expression.
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/question.dat
a=is it?,b=it is!
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --oxtab put '$c = gsub($a, "[?]"," ...")' data/question.dat
a is it?
b it is!
c is it ...
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --oxtab put '$c = ssub($a, "?"," ...")' data/question.dat
@ -415,11 +415,11 @@ How can I put single-quotes into strings?
This is a little tricky due to the shell's handling of quotes. For simplicity, let's first put an update script into a file:
.. code-block::
.. code-block:: none
$a = "It's OK, I said, then 'for now'."
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ echo a=bcd | mlr put -f data/single-quote-example.mlr
@ -429,7 +429,7 @@ So, it's simple: Miller's DSL uses double quotes for strings, and you can put si
Without putting the update expression in a file, it's messier:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ echo a=bcd | mlr put '$a="It'\''s OK, I said, '\''for now'\''."'
@ -450,7 +450,7 @@ Why doesn't mlr cut put fields in the order I want?
Example: columns ``x,i,a`` were requested but they appear here in the order ``a,i,x``:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/small
@ -460,7 +460,7 @@ Example: columns ``x,i,a`` were requested but they appear here in the order ``a,
a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr cut -f x,i,a data/small
@ -474,7 +474,7 @@ The issue is that Miller's ``cut``, by default, outputs cut fields in the order
The solution is to use the ``-o`` option:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr cut -o -f x,i,a data/small
@ -489,7 +489,7 @@ NR is not consecutive after then-chaining
Given this input data:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/small
@ -501,7 +501,7 @@ Given this input data:
why don't I see ``NR=1`` and ``NR=2`` here??
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr filter '$x > 0.5' then put '$NR = NR' data/small
@ -510,7 +510,7 @@ why don't I see ``NR=1`` and ``NR=2`` here??
The reason is that ``NR`` is computed for the original input records and isn't dynamically updated. By contrast, ``NF`` is dynamically updated: it's the number of fields in the current record, and if you add/remove a field, the value of ``NF`` will change:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ echo x=1,y=2,z=3 | mlr put '$nf1 = NF; $u = 4; $nf2 = NF; unset $x,$y,$z; $nf3 = NF'
@ -518,7 +518,7 @@ The reason is that ``NR`` is computed for the original input records and isn't d
``NR``, by contrast (and ``FNR`` as well), retains the value from the original input stream, and records may be dropped by a ``filter`` within a ``then``-chain. To recover consecutive record numbers, you can use out-of-stream variables as follows:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint --from data/small put '
@ -538,7 +538,7 @@ The reason is that ``NR`` is computed for the original input records and isn't d
Or, simply use ``mlr cat -n``:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr filter '$x > 0.5' then cat -n data/small
@ -552,7 +552,7 @@ Why am I not seeing all possible joins occur?
For example, the right file here has nine records, and the left file should add in the ``hostname`` column -- so the join output should also have 9 records:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsvlite --opprint cat data/join-u-left.csv
@ -561,7 +561,7 @@ For example, the right file here has nine records, and the left file should add
zenith.west.our.org 10.3.1.27
apoapsis.east.our.org 10.4.5.94
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsvlite --opprint cat data/join-u-right.csv
@ -576,7 +576,7 @@ For example, the right file here has nine records, and the left file should add
10.3.1.18 1448762598 73425
10.4.5.94 1448762599 12200
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsvlite --opprint join -s -j ipaddr -f data/join-u-left.csv data/join-u-right.csv
@ -590,7 +590,7 @@ The issue is that Miller's ``join``, by default (before 5.1.0), took input sorte
The solution (besides pre-sorting the input files on the join keys) is to simply use **mlr join -u** (which is now the default). This loads the left file entirely into memory (while the right file is still streamed one line at a time) and does all possible joins without requiring sorted input:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsvlite --opprint join -u -j ipaddr -f data/join-u-left.csv data/join-u-right.csv
@ -612,14 +612,14 @@ How to rectangularize after joins with unpaired?
Suppose you have the following two data files:
.. code-block::
.. code-block:: none
id,code
3,0000ff
2,00ff00
4,ff0000
.. code-block::
.. code-block:: none
id,color
4,red
@ -627,7 +627,7 @@ Suppose you have the following two data files:
Joining on color the results are as expected:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --csv join -j id -f data/color-codes.csv data/color-names.csv
@ -637,7 +637,7 @@ Joining on color the results are as expected:
However, if we ask for left-unpaireds, since there's no ``color`` column, we get a row not having the same column names as the other:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --csv join --ul -j id -f data/color-codes.csv data/color-names.csv
@ -650,7 +650,7 @@ However, if we ask for left-unpaireds, since there's no ``color`` column, we get
To fix this, we can use **unsparsify**:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --csv join --ul -j id -f data/color-codes.csv then unsparsify --fill-with "" data/color-names.csv
@ -670,13 +670,13 @@ XML, JSON, etc. are, by contrast, all **recursive** or **nested** data structure
Now, you can put tabular data into these formats -- since list-of-key-value-pairs is one of the things representable in XML or JSON. Example:
.. code-block::
.. code-block:: none
# DKVP
x=1,y=2
z=3
.. code-block::
.. code-block:: none
# XML
<table>
@ -695,7 +695,7 @@ Now, you can put tabular data into these formats -- since list-of-key-value-pair
</record>
</table>
.. code-block::
.. code-block:: none
# JSON
[{"x":1,"y":2},{"z":3}]

6
docs6/faq.rst.in
View file

@ -278,13 +278,13 @@ XML, JSON, etc. are, by contrast, all **recursive** or **nested** data structure
Now, you can put tabular data into these formats -- since list-of-key-value-pairs is one of the things representable in XML or JSON. Example:
.. code-block::
.. code-block:: none
# DKVP
x=1,y=2
z=3
.. code-block::
.. code-block:: none
# XML
<table>
@ -303,7 +303,7 @@ Now, you can put tabular data into these formats -- since list-of-key-value-pair
</record>
</table>
.. code-block::
.. code-block:: none
# JSON
[{"x":1,"y":2},{"z":3}]

8
docs6/feature-comparison.rst
View file

@ -11,7 +11,7 @@ File-format awareness
Miller respects CSV headers. If you do ``mlr --csv cat *.csv`` then the header line is written once:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/a.csv
@ -19,14 +19,14 @@ Miller respects CSV headers. If you do ``mlr --csv cat *.csv`` then the header l
1,2,3
4,5,6
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/b.csv
a,b,c
7,8,9
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --csv cat data/a.csv data/b.csv
@ -35,7 +35,7 @@ Miller respects CSV headers. If you do ``mlr --csv cat *.csv`` then the header l
4,5,6
7,8,9
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --csv sort -nr b data/a.csv data/b.csv

68
docs6/file-formats.rst
View file

@ -9,7 +9,7 @@ Miller handles name-indexed data using several formats: some you probably know b
Examples
----------------------------------------------------------------
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --usage-data-format-examples
@ -118,7 +118,7 @@ DKVP: Key-value pairs
Miller's default file format is DKVP, for **delimited key-value pairs**. Example:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr cat data/small
@ -130,21 +130,21 @@ Miller's default file format is DKVP, for **delimited key-value pairs**. Example
Such data are easy to generate, e.g. in Ruby with
.. code-block::
.. code-block:: none
puts "host=#{hostname},seconds=#{t2-t1},message=#{msg}"
.. code-block::
.. code-block:: none
puts mymap.collect{|k,v| "#{k}=#{v}"}.join(',')
or ``print`` statements in various languages, e.g.
.. code-block::
.. code-block:: none
echo "type=3,user=$USER,date=$date\n";
.. code-block::
.. code-block:: none
logger.log("type=3,user=$USER,date=$date\n");
@ -152,7 +152,7 @@ Fields lacking an IPS will have positional index (starting at 1) used as the key
As discussed in :doc:`record-heterogeneity`, Miller handles changes of field names within the same data stream. But using DKVP format this is particularly natural. One of my favorite use-cases for Miller is in application/server logs, where I log all sorts of lines such as
.. code-block::
.. code-block:: none
resource=/path/to/file,loadsec=0.45,ok=true
record_count=100, resource=/path/to/file
@ -172,7 +172,7 @@ With ``--inidx --ifs ' ' --repifs``, Miller splits lines on whitespace and assig
Example with index-numbered output:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/small
@ -182,7 +182,7 @@ Example with index-numbered output:
a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --onidx --ofs ' ' cat data/small
@ -194,7 +194,7 @@ Example with index-numbered output:
Example with index-numbered input:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/mydata.txt
@ -202,7 +202,7 @@ Example with index-numbered input:
by the dawn's
early light
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --inidx --ifs ' ' --odkvp cat data/mydata.txt
@ -212,7 +212,7 @@ Example with index-numbered input:
Example with index-numbered input and output:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/mydata.txt
@ -220,7 +220,7 @@ Example with index-numbered input and output:
by the dawn's
early light
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --nidx --fs ' ' --repifs cut -f 2,3 data/mydata.txt
@ -242,14 +242,14 @@ Single-level JSON objects
An **array of single-level objects** is, quite simply, **a table**:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --json head -n 2 then cut -f color,shape data/json-example-1.json
{ "color": "yellow", "shape": "triangle" }
{ "color": "red", "shape": "square" }
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --json --jvstack head -n 2 then cut -f color,u,v data/json-example-1.json
@ -264,7 +264,7 @@ An **array of single-level objects** is, quite simply, **a table**:
"v": 0.001257332190235938
}
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --ijson --opprint stats1 -a mean,stddev,count -f u -g shape data/json-example-1.json
@ -278,7 +278,7 @@ Nested JSON objects
Additionally, Miller can **tabularize nested objects by concatentating keys**:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --json --jvstack head -n 2 data/json-example-2.json
@ -311,7 +311,7 @@ Additionally, Miller can **tabularize nested objects by concatentating keys**:
}
}
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --ijson --opprint head -n 4 data/json-example-2.json
@ -323,7 +323,7 @@ Additionally, Miller can **tabularize nested objects by concatentating keys**:
Note in particular that as far as Miller's ``put`` and ``filter``, as well as other I/O formats, are concerned, these are simply field names with colons in them:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --json --jvstack head -n 1 then put '${values:uv} = ${values:u} * ${values:v}' data/json-example-2.json
@ -350,7 +350,7 @@ Arrays aren't supported in Miller's ``put``/``filter`` DSL. By default, JSON arr
Suppose we have arrays like this in our input data:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/json-example-3.json
@ -365,7 +365,7 @@ Suppose we have arrays like this in our input data:
Then integer indices (starting from 0 and counting up) are used as map keys:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --ijson --oxtab cat data/json-example-3.json
@ -380,7 +380,7 @@ Then integer indices (starting from 0 and counting up) are used as map keys:
When the data are written back out as JSON, field names are re-expanded as above, but what were arrays on input are now maps on output:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --json --jvstack cat data/json-example-3.json
@ -437,7 +437,7 @@ PPRINT: Pretty-printed tabular
Miller's pretty-print format is like CSV, but column-aligned. For example, compare
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --ocsv cat data/small
@ -448,7 +448,7 @@ Miller's pretty-print format is like CSV, but column-aligned. For example, comp
eks,wye,4,0.38139939387114097,0.13418874328430463
wye,pan,5,0.5732889198020006,0.8636244699032729
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint cat data/small
@ -465,7 +465,7 @@ See :doc:`record-heterogeneity` for how Miller handles changes of field names wi
For output only (this isn't supported in the input-scanner as of 5.0.0) you can use ``--barred`` with pprint output format:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint --barred cat data/small
@ -487,7 +487,7 @@ XTAB: Vertical tabular
This is perhaps most useful for looking a very wide and/or multi-column data which causes line-wraps on the screen (but see also
`ngrid <https://github.com/twosigma/ngrid/>`_ for an entirely different, very powerful option). Namely:
.. code-block::
.. code-block:: none
$ grep -v '^#' /etc/passwd | head -n 6 | mlr --nidx --fs : --opprint cat
1 2 3 4 5 6 7
@ -498,7 +498,7 @@ This is perhaps most useful for looking a very wide and/or multi-column data whi
_taskgated * 13 13 Task Gate Daemon /var/empty /usr/bin/false
_networkd * 24 24 Network Services /var/networkd /usr/bin/false
.. code-block::
.. code-block:: none
$ grep -v '^#' /etc/passwd | head -n 2 | mlr --nidx --fs : --oxtab cat
1 nobody
@ -517,7 +517,7 @@ This is perhaps most useful for looking a very wide and/or multi-column data whi
6 /var/root
7 /bin/sh
.. code-block::
.. code-block:: none
$ grep -v '^#' /etc/passwd | head -n 2 | \
mlr --nidx --fs : --ojson --jvstack --jlistwrap label name,password,uid,gid,gecos,home_dir,shell
@ -547,7 +547,7 @@ Markdown tabular
Markdown format looks like this:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --omd cat data/small
@ -570,7 +570,7 @@ Data-conversion keystroke-savers
While you can do format conversion using ``mlr --icsv --ojson cat myfile.csv``, there are also keystroke-savers for this purpose, such as ``mlr --c2j cat myfile.csv``. For a complete list:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --usage-format-conversion-keystroke-saver-options
@ -606,7 +606,7 @@ Comments in data
You can include comments within your data files, and either have them ignored, or passed directly through to the standard output as soon as they are encountered:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --usage-comments-in-data
@ -629,7 +629,7 @@ You can include comments within your data files, and either have them ignored, o
Examples:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/budget.csv
@ -639,7 +639,7 @@ Examples:
green,678.12
orange,123.45
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --skip-comments --icsv --opprint sort -nr quantity data/budget.csv
@ -648,7 +648,7 @@ Examples:
purple 456.78
orange 123.45
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --pass-comments --icsv --opprint sort -nr quantity data/budget.csv

10
docs6/file-formats.rst.in
View file

@ -57,21 +57,21 @@ POKI_RUN_COMMAND{{mlr cat data/small}}HERE
Such data are easy to generate, e.g. in Ruby with
.. code-block::
.. code-block:: none
puts "host=#{hostname},seconds=#{t2-t1},message=#{msg}"
.. code-block::
.. code-block:: none
puts mymap.collect{|k,v| "#{k}=#{v}"}.join(',')
or ``print`` statements in various languages, e.g.
.. code-block::
.. code-block:: none
echo "type=3,user=$USER,date=$date\n";
.. code-block::
.. code-block:: none
logger.log("type=3,user=$USER,date=$date\n");
@ -79,7 +79,7 @@ Fields lacking an IPS will have positional index (starting at 1) used as the key
As discussed in :doc:`record-heterogeneity`, Miller handles changes of field names within the same data stream. But using DKVP format this is particularly natural. One of my favorite use-cases for Miller is in application/server logs, where I log all sorts of lines such as
.. code-block::
.. code-block:: none
resource=/path/to/file,loadsec=0.45,ok=true
record_count=100, resource=/path/to/file

12
docs6/install.rst
View file

@ -9,38 +9,38 @@ Prebuilt executables via package managers
`Homebrew <https://brew.sh/>`_ installation support for OSX is available via
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
brew update && brew install miller
...and also via `MacPorts <https://www.macports.org/>`_:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
sudo port selfupdate && sudo port install miller
You may already have the ``mlr`` executable available in your platform's package manager on NetBSD, Debian Linux, Ubuntu Xenial and upward, Arch Linux, or perhaps other distributions. For example, on various Linux distributions you might do one of the following:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
sudo apt-get install miller
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
sudo apt install miller
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
sudo yum install miller
On Windows, Miller is available via `Chocolatey <https://chocolatey.org/>`_:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
choco install miller

12
docs6/install.rst.in
View file

@ -6,38 +6,38 @@ Prebuilt executables via package managers
`Homebrew <https://brew.sh/>`_ installation support for OSX is available via
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
brew update && brew install miller
...and also via `MacPorts <https://www.macports.org/>`_:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
sudo port selfupdate && sudo port install miller
You may already have the ``mlr`` executable available in your platform's package manager on NetBSD, Debian Linux, Ubuntu Xenial and upward, Arch Linux, or perhaps other distributions. For example, on various Linux distributions you might do one of the following:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
sudo apt-get install miller
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
sudo apt install miller
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
sudo yum install miller
On Windows, Miller is available via `Chocolatey <https://chocolatey.org/>`_:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
choco install miller

10
docs6/log-processing-examples.rst
View file

@ -12,7 +12,7 @@ Writing a program -- in any language whatsoever -- you can have it print out log
Suppose your program has printed something like this:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat log.txt
@ -62,7 +62,7 @@ Suppose your program has printed something like this:
Each print statement simply contains local information: the current timestamp, whether a particular cache was hit or not, etc. Then using either the system ``grep`` command, or Miller's ``having-fields``, or ``is_present``, we can pick out the parts we want and analyze them:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ grep op=cache log.txt \
@ -72,7 +72,7 @@ Each print statement simply contains local information: the current timestamp, w
A4 0.714286
A9 0.090909
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --from log.txt --opprint \
@ -89,7 +89,7 @@ Each print statement simply contains local information: the current timestamp, w
Alternatively, we can simply group the similar data for a better look:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint group-like log.txt
@ -142,7 +142,7 @@ Alternatively, we can simply group the similar data for a better look:
1472819736 100 612
1472819742 100 728
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint group-like then sec2gmt time log.txt

2
docs6/manpage.rst
View file

@ -6,7 +6,7 @@ Manpage
This is simply a copy of what you should see on running **man mlr** at a command prompt, once Miller is installed on your system.
.. code-block::
.. code-block:: none
MILLER(1) MILLER(1)

2
docs6/poki
View file

@ -90,7 +90,7 @@ end
# ----------------------------------------------------------------
def write_card(highlight_first_line, content_lines, output_handle)
output_handle.puts('.. code-block::')
output_handle.puts('.. code-block:: none')
if highlight_first_line
output_handle.puts(' :emphasize-lines: 1,1') # 'hll' in _static/*.css
end

30
docs6/quick-examples.rst
View file

@ -8,70 +8,70 @@ Quick examples
Column select:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
% mlr --csv cut -f hostname,uptime mydata.csv
Add new columns as function of other columns:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
% mlr --nidx put '$sum = $7 < 0.0 ? 3.5 : $7 + 2.1*$8' *.dat
Row filter:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
% mlr --csv filter '$status != "down" && $upsec >= 10000' *.csv
Apply column labels and pretty-print:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
% grep -v '^#' /etc/group | mlr --ifs : --nidx --opprint label group,pass,gid,member then sort -f group
Join multiple data sources on key columns:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
% mlr join -j account_id -f accounts.dat then group-by account_name balances.dat
Mulltiple formats including JSON:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
% mlr --json put '$attr = sub($attr, "([0-9]+)_([0-9]+)_.*", "\1:\2")' data/*.json
Aggregate per-column statistics:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
% mlr stats1 -a min,mean,max,p10,p50,p90 -f flag,u,v data/*
Linear regression:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
% mlr stats2 -a linreg-pca -f u,v -g shape data/*
Aggregate custom per-column statistics:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
% mlr put -q '@sum[$a][$b] += $x; end {emit @sum, "a", "b"}' data/*
Iterate over data using DSL expressions:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
% mlr --from estimates.tbl put '
@ -85,35 +85,35 @@ Iterate over data using DSL expressions:
Run DSL expressions from a script file:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
% mlr --from infile.dat put -f analyze.mlr
Split/reduce output to multiple filenames:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
% mlr --from infile.dat put 'tee > "./taps/data-".$a."-".$b, $*'
Compressed I/O:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
% mlr --from infile.dat put 'tee | "gzip > ./taps/data-".$a."-".$b.".gz", $*'
Interoperate with other data-processing tools using standard pipes:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
% mlr --from infile.dat put -q '@v=$*; dump | "jq .[]"'
Tap/trace:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
% mlr --from infile.dat put '(NR % 1000 == 0) { print > stderr, "Checkpoint ".NR}'

30
docs6/quick-examples.rst.in
View file

@ -5,70 +5,70 @@ Quick examples
Column select:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
% mlr --csv cut -f hostname,uptime mydata.csv
Add new columns as function of other columns:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
% mlr --nidx put '$sum = $7 < 0.0 ? 3.5 : $7 + 2.1*$8' *.dat
Row filter:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
% mlr --csv filter '$status != "down" && $upsec >= 10000' *.csv
Apply column labels and pretty-print:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
% grep -v '^#' /etc/group | mlr --ifs : --nidx --opprint label group,pass,gid,member then sort -f group
Join multiple data sources on key columns:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
% mlr join -j account_id -f accounts.dat then group-by account_name balances.dat
Mulltiple formats including JSON:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
% mlr --json put '$attr = sub($attr, "([0-9]+)_([0-9]+)_.*", "\1:\2")' data/*.json
Aggregate per-column statistics:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
% mlr stats1 -a min,mean,max,p10,p50,p90 -f flag,u,v data/*
Linear regression:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
% mlr stats2 -a linreg-pca -f u,v -g shape data/*
Aggregate custom per-column statistics:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
% mlr put -q '@sum[$a][$b] += $x; end {emit @sum, "a", "b"}' data/*
Iterate over data using DSL expressions:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
% mlr --from estimates.tbl put '
@ -82,35 +82,35 @@ Iterate over data using DSL expressions:
Run DSL expressions from a script file:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
% mlr --from infile.dat put -f analyze.mlr
Split/reduce output to multiple filenames:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
% mlr --from infile.dat put 'tee > "./taps/data-".$a."-".$b, $*'
Compressed I/O:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
% mlr --from infile.dat put 'tee | "gzip > ./taps/data-".$a."-".$b.".gz", $*'
Interoperate with other data-processing tools using standard pipes:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
% mlr --from infile.dat put -q '@v=$*; dump | "jq .[]"'
Tap/trace:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
% mlr --from infile.dat put '(NR % 1000 == 0) { print > stderr, "Checkpoint ".NR}'

26
docs6/record-heterogeneity.rst
View file

@ -16,7 +16,7 @@ CSV and pretty-print
Miller simply prints a newline and a new header when there is a schema change. When there is no schema change, you get CSV per se as a special case. Likewise, Miller reads heterogeneous CSV or pretty-print input the same way. The difference between CSV and CSV-lite is that the former is RFC4180-compliant, while the latter readily handles heterogeneous data (which is non-compliant). For example:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/het.dkvp
@ -26,7 +26,7 @@ Miller simply prints a newline and a new header when there is a schema change. W
record_count=150,resource=/path/to/second/file
resource=/some/other/path,loadsec=0.97,ok=false
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --ocsvlite cat data/het.dkvp
@ -45,7 +45,7 @@ Miller simply prints a newline and a new header when there is a schema change. W
resource,loadsec,ok
/some/other/path,0.97,false
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint cat data/het.dkvp
@ -66,7 +66,7 @@ Miller simply prints a newline and a new header when there is a schema change. W
Miller handles explicit header changes as just shown. If your CSV input contains ragged data -- if there are implicit header changes -- you can use ``--allow-ragged-csv-input`` (or keystroke-saver ``--ragged``). For too-short data lines, values are filled with empty string; for too-long data lines, missing field names are replaced with positional indices (counting up from 1, not 0), as follows:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/ragged.csv
@ -75,7 +75,7 @@ Miller handles explicit header changes as just shown. If your CSV input contains
4,5
6,7,8,9
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --icsv --oxtab --allow-ragged-csv-input cat data/ragged.csv
@ -94,7 +94,7 @@ Miller handles explicit header changes as just shown. If your CSV input contains
You may also find Miller's ``group-like`` feature handy (see also :doc:`reference`):
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --ocsvlite group-like data/het.dkvp
@ -107,7 +107,7 @@ You may also find Miller's ``group-like`` feature handy (see also :doc:`referenc
100,/path/to/file
150,/path/to/second/file
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint group-like data/het.dkvp
@ -125,7 +125,7 @@ Key-value-pair, vertical-tabular, and index-numbered formats
For these formats, record-heterogeneity comes naturally:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/het.dkvp
@ -135,7 +135,7 @@ For these formats, record-heterogeneity comes naturally:
record_count=150,resource=/path/to/second/file
resource=/some/other/path,loadsec=0.97,ok=false
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --onidx --ofs ' ' cat data/het.dkvp
@ -145,7 +145,7 @@ For these formats, record-heterogeneity comes naturally:
150 /path/to/second/file
/some/other/path 0.97 false
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --oxtab cat data/het.dkvp
@ -167,7 +167,7 @@ For these formats, record-heterogeneity comes naturally:
loadsec 0.97
ok false
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --oxtab group-like data/het.dkvp
@ -194,7 +194,7 @@ For processing
Miller operates on specified fields and takes the rest along: for example, if you are sorting on the ``count`` field then all records in the input stream must have a ``count`` field but the other fields can vary, and moreover the sorted-on field name(s) don't need to be in the same position on each line:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/sort-het.dkvp
@ -206,7 +206,7 @@ Miller operates on specified fields and takes the rest along: for example, if yo
count=100,color=green
count=450
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr sort -n count data/sort-het.dkvp

266
docs6/reference-dsl.rst
View file

File diff suppressed because it is too large Load diff

22
docs6/reference-dsl.rst.in
View file

@ -121,19 +121,19 @@ POKI_INCLUDE_AND_RUN_ESCAPED(data/trailing-commas.sh)HERE
Bodies for all compound statements must be enclosed in **curly braces**, even if the body is a single statement:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
mlr put 'if ($x == 1) $y = 2' # Syntax error
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
mlr put 'if ($x == 1) { $y = 2 }' # This is OK
Bodies for compound statements may be empty:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
mlr put 'if ($x == 1) { }' # This no-op is syntactically acceptable
@ -363,7 +363,7 @@ Local variables can be defined either untyped as in ``x = 1``, or typed as in ``
The reason for ``num`` is that ``int`` and ``float`` typedecls are very precise:
.. code-block::
.. code-block:: none
float a = 0; # Runtime error since 0 is int not float
int b = 1.0; # Runtime error since 1.0 is float not int
@ -374,7 +374,7 @@ A suggestion is to use ``num`` for general use when you want numeric content, an
The ``var`` type declaration indicates no type restrictions, e.g. ``var x = 1`` has the same type restrictions on ``x`` as ``x = 1``. The difference is in intentional shadowing: if you have ``x = 1`` in outer scope and ``x = 2`` in inner scope (e.g. within a for-loop or an if-statement) then outer-scope ``x`` has value 2 after the second assignment. But if you have ``var x = 2`` in the inner scope, then you are declaring a variable scoped to the inner block.) For example:
.. code-block::
.. code-block:: none
x = 1;
if (NR == 4) {
@ -382,7 +382,7 @@ The ``var`` type declaration indicates no type restrictions, e.g. ``var x = 1``
}
print x; # Value of x is now two
.. code-block::
.. code-block:: none
x = 1;
if (NR == 4) {
@ -392,7 +392,7 @@ The ``var`` type declaration indicates no type restrictions, e.g. ``var x = 1``
Likewise function arguments can optionally be typed, with type enforced when the function is called:
.. code-block::
.. code-block:: none
func f(map m, int i) {
...
@ -407,7 +407,7 @@ Likewise function arguments can optionally be typed, with type enforced when the
Thirdly, function return values can be type-checked at the point of ``return`` using ``:`` and a typedecl after the parameter list:
.. code-block::
.. code-block:: none
func f(map m, int i): bool {
...
@ -466,7 +466,7 @@ Operator precedence
Operators are listed in order of decreasing precedence, highest first.
.. code-block::
.. code-block:: none
Operators Associativity
--------- -------------
@ -527,11 +527,11 @@ If-statements
These are again reminiscent of ``awk``. Pattern-action blocks are a special case of ``if`` with no ``elif`` or ``else`` blocks, no ``if`` keyword, and parentheses optional around the boolean expression:
.. code-block::
.. code-block:: none
mlr put 'NR == 4 {$foo = "bar"}'
.. code-block::
.. code-block:: none
mlr put 'if (NR == 4) {$foo = "bar"}'

414
docs6/reference-verbs.rst
View file

File diff suppressed because it is too large Load diff

8
docs6/reference-verbs.rst.in
View file

@ -69,7 +69,7 @@ POKI_RUN_COMMAND{{mlr bootstrap --help}}HERE
The canonical use for bootstrap sampling is to put error bars on statistical quantities, such as mean. For example:
.. code-block::
.. code-block:: none
$ mlr --opprint stats1 -a mean,count -f u -g color data/colored-shapes.dkvp
color u_mean u_count
@ -80,7 +80,7 @@ The canonical use for bootstrap sampling is to put error bars on statistical qua
blue 0.517717 1470
orange 0.490532 303
.. code-block::
.. code-block:: none
$ mlr --opprint bootstrap then stats1 -a mean,count -f u -g color data/colored-shapes.dkvp
color u_mean u_count
@ -91,7 +91,7 @@ The canonical use for bootstrap sampling is to put error bars on statistical qua
blue 0.512529 1496
orange 0.521030 321
.. code-block::
.. code-block:: none
$ mlr --opprint bootstrap then stats1 -a mean,count -f u -g color data/colored-shapes.dkvp
color u_mean u_count
@ -102,7 +102,7 @@ The canonical use for bootstrap sampling is to put error bars on statistical qua
green 0.496803 1075
purple 0.486337 1199
.. code-block::
.. code-block:: none
$ mlr --opprint bootstrap then stats1 -a mean,count -f u -g color data/colored-shapes.dkvp
color u_mean u_count

96
docs6/reference.rst
View file

@ -13,7 +13,7 @@ Command overview
Whereas the Unix toolkit is made of the separate executables ``cat``, ``tail``, ``cut``,
``sort``, etc., Miller has subcommands, or **verbs**, invoked as follows:
.. code-block::
.. code-block:: none
mlr tac *.dat
mlr cut --complement -f os_version *.dat
@ -39,7 +39,7 @@ Formats
Options:
.. code-block::
.. code-block:: none
--dkvp --idkvp --odkvp
--nidx --inidx --onidx
@ -51,7 +51,7 @@ Options:
These are as discussed in :doc:`file-formats`, with the exception of ``--right`` which makes pretty-printed output right-aligned:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint cat data/small
@ -62,7 +62,7 @@ These are as discussed in :doc:`file-formats`, with the exception of ``--right``
eks wye 4 0.38139939387114097 0.13418874328430463
wye pan 5 0.5732889198020006 0.8636244699032729
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint --right cat data/small
@ -97,14 +97,14 @@ Compression
Options:
.. code-block::
.. code-block:: none
--prepipe {command}
The prepipe command is anything which reads from standard input and produces data acceptable to Miller. Nominally this allows you to use whichever decompression utilities you have installed on your system, on a per-file basis. If the command has flags, quote them: e.g. ``mlr --prepipe 'zcat -cf'``. Examples:
.. code-block::
.. code-block:: none
# These two produce the same output:
$ gunzip < myfile1.csv.gz | mlr cut -f hostname,uptime
@ -113,14 +113,14 @@ The prepipe command is anything which reads from standard input and produces dat
$ mlr --prepipe gunzip cut -f hostname,uptime myfile1.csv.gz myfile2.csv.gz
$ mlr --prepipe gunzip --idkvp --oxtab cut -f hostname,uptime myfile1.dat.gz myfile2.dat.gz
.. code-block::
.. code-block:: none
# Similar to the above, but with compressed output as well as input:
$ gunzip < myfile1.csv.gz | mlr cut -f hostname,uptime | gzip > outfile.csv.gz
$ mlr --prepipe gunzip cut -f hostname,uptime myfile1.csv.gz | gzip > outfile.csv.gz
$ mlr --prepipe gunzip cut -f hostname,uptime myfile1.csv.gz myfile2.csv.gz | gzip > outfile.csv.gz
.. code-block::
.. code-block:: none
# Similar to the above, but with different compression tools for input and output:
$ gunzip < myfile1.csv.gz | mlr cut -f hostname,uptime | xz -z > outfile.csv.xz
@ -136,7 +136,7 @@ Miller has record separators ``IRS`` and ``ORS``, field separators ``IFS`` and `
Options:
.. code-block::
.. code-block:: none
--rs --irs --ors
--fs --ifs --ofs --repifs
@ -157,7 +157,7 @@ Number formatting
The command-line option ``--ofmt {format string}`` is the global number format for commands which generate numeric output, e.g. ``stats1``, ``stats2``, ``histogram``, and ``step``, as well as ``mlr put``. Examples:
.. code-block::
.. code-block:: none
--ofmt %.9le --ofmt %.6lf --ofmt %.0lf
@ -165,13 +165,13 @@ These are just familiar ``printf`` formats applied to double-precision numbers.
To apply formatting to a single field, overriding the global ``ofmt``, use ``fmtnum`` function within ``mlr put``. For example:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ echo 'x=3.1,y=4.3' | mlr put '$z=fmtnum($x*$y,"%08lf")'
x=3.1,y=4.3,z=13.330000
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ echo 'x=0xffff,y=0xff' | mlr put '$z=fmtnum(int($x*$y),"%08llx")'
@ -179,7 +179,7 @@ To apply formatting to a single field, overriding the global ``ofmt``, use ``fmt
Input conversion from hexadecimal is done automatically on fields handled by ``mlr put`` and ``mlr filter`` as long as the field value begins with "0x". To apply output conversion to hexadecimal on a single column, you may use ``fmtnum``, or the keystroke-saving ``hexfmt`` function. Example:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ echo 'x=0xffff,y=0xff' | mlr put '$z=hexfmt($x*$y)'
@ -200,14 +200,14 @@ then-chaining
In accord with the `Unix philosophy <http://en.wikipedia.org/wiki/Unix_philosophy>`_, you can pipe data into or out of Miller. For example:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
mlr cut --complement -f os_version *.dat | mlr sort -f hostname,uptime
You can, if you like, instead simply chain commands together using the ``then`` keyword:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
mlr cut --complement -f os_version then sort -f hostname,uptime *.dat
@ -216,7 +216,7 @@ You can, if you like, instead simply chain commands together using the ``then``
Here's a performance comparison:
.. code-block::
.. code-block:: none
% cat piped.sh
mlr cut -x -f i,y data/big | mlr sort -n y > /dev/null
@ -244,7 +244,7 @@ Auxiliary commands
There are a few nearly-standalone programs which have nothing to do with the rest of Miller, do not participate in record streams, and do not deal with file formats. They might as well be little standalone executables but they're delivered within the main Miller executable for convenience.
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr aux-list
@ -257,7 +257,7 @@ There are a few nearly-standalone programs which have nothing to do with the res
netbsd-strptime
For more information, please invoke mlr {subcommand} --help
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr lecat --help
@ -268,7 +268,7 @@ There are a few nearly-standalone programs which have nothing to do with the res
--mono: don't try to colorize the output
-h or --help: print this message
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr termcvt --help
@ -285,7 +285,7 @@ There are a few nearly-standalone programs which have nothing to do with the res
Zero file names means read from standard input.
Output is always to standard output; files are not written in-place.
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr hex --help
@ -296,7 +296,7 @@ There are a few nearly-standalone programs which have nothing to do with the res
-r: print only raw hex without leading offset indicators or trailing ASCII dump.
-h or --help: print this message
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr unhex --help
@ -308,19 +308,19 @@ There are a few nearly-standalone programs which have nothing to do with the res
Examples:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ echo 'Hello, world!' | mlr lecat --mono
Hello, world![LF]
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ echo 'Hello, world!' | mlr termcvt --lf2crlf | mlr lecat --mono
Hello, world![CR][LF]
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr hex data/budget.csv
@ -332,7 +332,7 @@ Examples:
00000050: 38 0a 67 72 65 65 6e 2c 36 37 38 2e 31 32 0a 6f |8.green,678.12.o|
00000060: 72 61 6e 67 65 2c 31 32 33 2e 34 35 0a |range,123.45.|
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr hex -r data/budget.csv
@ -344,7 +344,7 @@ Examples:
38 0a 67 72 65 65 6e 2c 36 37 38 2e 31 32 0a 6f
72 61 6e 67 65 2c 31 32 33 2e 34 35 0a
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr hex -r data/budget.csv | sed 's/20/2a/g' | mlr unhex
@ -396,7 +396,7 @@ Rules for null-handling:
* Records with one or more empty sort-field values sort after records with all sort-field values present:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr cat data/sort-null.dat
@ -406,7 +406,7 @@ Rules for null-handling:
x=9,b=10
a=5,b=7
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr sort -n a data/sort-null.dat
@ -416,7 +416,7 @@ Rules for null-handling:
a=,b=4
x=9,b=10
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr sort -nr a data/sort-null.dat
@ -428,19 +428,19 @@ Rules for null-handling:
* Functions/operators which have one or more *empty* arguments produce empty output: e.g.
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ echo 'x=2,y=3' | mlr put '$a=$x+$y'
x=2,y=3,a=5
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ echo 'x=,y=3' | mlr put '$a=$x+$y'
x=,y=3,a=
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ echo 'x=,y=3' | mlr put '$a=log($x);$b=log($y)'
@ -448,7 +448,7 @@ Rules for null-handling:
with the exception that the ``min`` and ``max`` functions are special: if one argument is non-null, it wins:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ echo 'x=,y=3' | mlr put '$a=min($x,$y);$b=max($x,$y)'
@ -456,13 +456,13 @@ with the exception that the ``min`` and ``max`` functions are special: if one ar
* Functions of *absent* variables (e.g. ``mlr put '$y = log10($nonesuch)'``) evaluate to absent, and arithmetic/bitwise/boolean operators with both operands being absent evaluate to absent. Arithmetic operators with one absent operand return the other operand. More specifically, absent values act like zero for addition/subtraction, and one for multiplication: Furthermore, **any expression which evaluates to absent is not stored in the left-hand side of an assignment statement**:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ echo 'x=2,y=3' | mlr put '$a=$u+$v; $b=$u+$y; $c=$x+$y'
x=2,y=3,b=3,c=5
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ echo 'x=2,y=3' | mlr put '$a=min($x,$v);$b=max($u,$y);$c=min($u,$v)'
@ -482,7 +482,7 @@ The reasoning is as follows:
Since absent plus absent is absent (and likewise for other operators), accumulations such as ``@sum += $x`` work correctly on heterogenous data, as do within-record formulas if both operands are absent. If one operand is present, you may get behavior you don't desire. To work around this -- namely, to set an output field only for records which have all the inputs present -- you can use a pattern-action block with ``is_present``:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr cat data/het.dkvp
@ -492,7 +492,7 @@ Since absent plus absent is absent (and likewise for other operators), accumulat
record_count=150,resource=/path/to/second/file
resource=/some/other/path,loadsec=0.97,ok=false
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr put 'is_present($loadsec) { $loadmillis = $loadsec * 1000 }' data/het.dkvp
@ -502,7 +502,7 @@ Since absent plus absent is absent (and likewise for other operators), accumulat
record_count=150,resource=/path/to/second/file
resource=/some/other/path,loadsec=0.97,ok=false,loadmillis=970.000000
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr put '$loadmillis = (is_present($loadsec) ? $loadsec : 0.0) * 1000' data/het.dkvp
@ -514,7 +514,7 @@ Since absent plus absent is absent (and likewise for other operators), accumulat
If you're interested in a formal description of how empty and absent fields participate in arithmetic, here's a table for plus (other arithmetic/boolean/bitwise operators are similar):
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --print-type-arithmetic-info
@ -582,7 +582,7 @@ For ``filter`` and ``put``, if the regular expression is a string literal (the n
Example:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ cat data/regex-in-data.dat
@ -590,7 +590,7 @@ Example:
name=bill,regex=^b[ou]ll$
name=bull,regex=^b[ou]ll$
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr filter '$name =~ $regex' data/regex-in-data.dat
@ -604,28 +604,28 @@ Regex captures of the form ``\0`` through ``\9`` are supported as
* Captures have in-function context for ``sub`` and ``gsub``. For example, the first ``\1,\2`` pair belong to the first ``sub`` and the second ``\1,\2`` pair belong to the second ``sub``:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
mlr put '$b = sub($a, "(..)_(...)", "\2-\1"); $c = sub($a, "(..)_(.)(..)", ":\1:\2:\3")'
* Captures endure for the entirety of a ``put`` for the ``=~`` and ``!=~`` operators. For example, here the ``\1,\2`` are set by the ``=~`` operator and are used by both subsequent assignment statements:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
mlr put '$a =~ "(..)_(....); $b = "left_\1"; $c = "right_\2"'
* The captures are not retained across multiple puts. For example, here the ``\1,\2`` won't be expanded from the regex capture:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
mlr put '$a =~ "(..)_(....)' then {... something else ...} then put '$b = "left_\1"; $c = "right_\2"'
* Captures are ignored in ``filter`` for the ``=~`` and ``!=~`` operators. For example, there is no mechanism provided to refer to the first ``(..)`` as ``\1`` or to the second ``(....)`` as ``\2`` in the following filter statement:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
mlr filter '$a =~ "(..)_(....)'
@ -656,7 +656,7 @@ The short of it is that Miller does this transparently for you so you needn't th
Implementation details of this, for the interested: integer adds and subtracts overflow by at most one bit so it suffices to check sign-changes. Thus, Miller allows you to add and subtract arbitrary 64-bit signed integers, converting only to float precisely when the result is less than -2\ :sup:`63` or greater than 2\ :sup:`63`\ -1. Multiplies, on the other hand, can overflow by a word size and a sign-change technique does not suffice to detect overflow. Instead Miller tests whether the floating-point product exceeds the representable integer range. Now, 64-bit integers have 64-bit precision while IEEE-doubles have only 52-bit mantissas -- so, there are 53 bits including implicit leading one. The following experiment explicitly demonstrates the resolution at this range:
.. code-block::
.. code-block:: none
64-bit integer 64-bit integer Casted to double Back to 64-bit
in hex in decimal integer
@ -685,7 +685,7 @@ On-line help
Examples:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --help
@ -1111,7 +1111,7 @@ Examples:
For more information please see http://johnkerl.org/miller/doc and/or
http://github.com/johnkerl/miller. This is Miller version v5.10.2-dev.
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr sort --help

28
docs6/reference.rst.in
View file

@ -32,7 +32,7 @@ Formats
Options:
.. code-block::
.. code-block:: none
--dkvp --idkvp --odkvp
--nidx --inidx --onidx
@ -72,14 +72,14 @@ Compression
Options:
.. code-block::
.. code-block:: none
--prepipe {command}
The prepipe command is anything which reads from standard input and produces data acceptable to Miller. Nominally this allows you to use whichever decompression utilities you have installed on your system, on a per-file basis. If the command has flags, quote them: e.g. ``mlr --prepipe 'zcat -cf'``. Examples:
.. code-block::
.. code-block:: none
# These two produce the same output:
$ gunzip < myfile1.csv.gz | mlr cut -f hostname,uptime
@ -88,14 +88,14 @@ The prepipe command is anything which reads from standard input and produces dat
$ mlr --prepipe gunzip cut -f hostname,uptime myfile1.csv.gz myfile2.csv.gz
$ mlr --prepipe gunzip --idkvp --oxtab cut -f hostname,uptime myfile1.dat.gz myfile2.dat.gz
.. code-block::
.. code-block:: none
# Similar to the above, but with compressed output as well as input:
$ gunzip < myfile1.csv.gz | mlr cut -f hostname,uptime | gzip > outfile.csv.gz
$ mlr --prepipe gunzip cut -f hostname,uptime myfile1.csv.gz | gzip > outfile.csv.gz
$ mlr --prepipe gunzip cut -f hostname,uptime myfile1.csv.gz myfile2.csv.gz | gzip > outfile.csv.gz
.. code-block::
.. code-block:: none
# Similar to the above, but with different compression tools for input and output:
$ gunzip < myfile1.csv.gz | mlr cut -f hostname,uptime | xz -z > outfile.csv.xz
@ -111,7 +111,7 @@ Miller has record separators ``IRS`` and ``ORS``, field separators ``IFS`` and `
Options:
.. code-block::
.. code-block:: none
--rs --irs --ors
--fs --ifs --ofs --repifs
@ -132,7 +132,7 @@ Number formatting
The command-line option ``--ofmt {format string}`` is the global number format for commands which generate numeric output, e.g. ``stats1``, ``stats2``, ``histogram``, and ``step``, as well as ``mlr put``. Examples:
.. code-block::
.. code-block:: none
--ofmt %.9le --ofmt %.6lf --ofmt %.0lf
@ -163,14 +163,14 @@ then-chaining
In accord with the `Unix philosophy <http://en.wikipedia.org/wiki/Unix_philosophy>`_, you can pipe data into or out of Miller. For example:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
mlr cut --complement -f os_version *.dat | mlr sort -f hostname,uptime
You can, if you like, instead simply chain commands together using the ``then`` keyword:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
mlr cut --complement -f os_version then sort -f hostname,uptime *.dat
@ -367,28 +367,28 @@ Regex captures of the form ``\0`` through ``\9`` are supported as
* Captures have in-function context for ``sub`` and ``gsub``. For example, the first ``\1,\2`` pair belong to the first ``sub`` and the second ``\1,\2`` pair belong to the second ``sub``:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
mlr put '$b = sub($a, "(..)_(...)", "\2-\1"); $c = sub($a, "(..)_(.)(..)", ":\1:\2:\3")'
* Captures endure for the entirety of a ``put`` for the ``=~`` and ``!=~`` operators. For example, here the ``\1,\2`` are set by the ``=~`` operator and are used by both subsequent assignment statements:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
mlr put '$a =~ "(..)_(....); $b = "left_\1"; $c = "right_\2"'
* The captures are not retained across multiple puts. For example, here the ``\1,\2`` won't be expanded from the regex capture:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
mlr put '$a =~ "(..)_(....)' then {... something else ...} then put '$b = "left_\1"; $c = "right_\2"'
* Captures are ignored in ``filter`` for the ``=~`` and ``!=~`` operators. For example, there is no mechanism provided to refer to the first ``(..)`` as ``\1`` or to the second ``(....)`` as ``\2`` in the following filter statement:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
mlr filter '$a =~ "(..)_(....)'
@ -419,7 +419,7 @@ The short of it is that Miller does this transparently for you so you needn't th
Implementation details of this, for the interested: integer adds and subtracts overflow by at most one bit so it suffices to check sign-changes. Thus, Miller allows you to add and subtract arbitrary 64-bit signed integers, converting only to float precisely when the result is less than -2\ :sup:`63` or greater than 2\ :sup:`63`\ -1. Multiplies, on the other hand, can overflow by a word size and a sign-change technique does not suffice to detect overflow. Instead Miller tests whether the floating-point product exceeds the representable integer range. Now, 64-bit integers have 64-bit precision while IEEE-doubles have only 52-bit mantissas -- so, there are 53 bits including implicit leading one. The following experiment explicitly demonstrates the resolution at this range:
.. code-block::
.. code-block:: none
64-bit integer 64-bit integer Casted to double Back to 64-bit
in hex in decimal integer

16
docs6/sql-examples.rst
View file

@ -13,7 +13,7 @@ I like to produce SQL-query output with header-column and tab delimiter: this is
For example, using default output formatting in ``mysql`` we get formatting like Miller's ``--opprint --barred``:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mysql --database=mydb -e 'show columns in mytable'
@ -29,7 +29,7 @@ For example, using default output formatting in ``mysql`` we get formatting like
Using ``mysql``'s ``-B`` we get TSV output:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mysql --database=mydb -B -e 'show columns in mytable' | mlr --itsvlite --opprint cat
@ -42,7 +42,7 @@ Using ``mysql``'s ``-B`` we get TSV output:
Since Miller handles TSV output, we can do as much or as little processing as we want in the SQL query, then send the rest on to Miller. This includes outputting as JSON, doing further selects/joins in Miller, doing stats, etc. etc.:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mysql --database=mydb -B -e 'show columns in mytable' | mlr --itsvlite --ojson --jlistwrap --jvstack cat
@ -89,7 +89,7 @@ Since Miller handles TSV output, we can do as much or as little processing as we
}
]
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mysql --database=mydb -B -e 'select * from mytable' > query.tsv
@ -115,7 +115,7 @@ One use of NIDX (value-only, no keys) format is for loading up SQL tables.
Create and load SQL table:
.. code-block::
.. code-block:: none
mysql> CREATE TABLE abixy(
a VARCHAR(32),
@ -158,7 +158,7 @@ Create and load SQL table:
Aggregate counts within SQL:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
mysql> SELECT a, b, COUNT(*) AS count FROM abixy GROUP BY a, b ORDER BY COUNT DESC;
@ -195,7 +195,7 @@ Aggregate counts within SQL:
Aggregate counts within Miller:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint uniq -c -g a,b then sort -nr count data/medium
@ -218,7 +218,7 @@ Aggregate counts within Miller:
Pipe SQL output to aggregate counts within Miller:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mysql -D miller -B -e 'select * from abixy' | mlr --itsv --opprint uniq -c -g a,b then sort -nr count

16
docs6/sql-examples.rst.in
View file

@ -10,7 +10,7 @@ I like to produce SQL-query output with header-column and tab delimiter: this is
For example, using default output formatting in ``mysql`` we get formatting like Miller's ``--opprint --barred``:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mysql --database=mydb -e 'show columns in mytable'
@ -26,7 +26,7 @@ For example, using default output formatting in ``mysql`` we get formatting like
Using ``mysql``'s ``-B`` we get TSV output:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mysql --database=mydb -B -e 'show columns in mytable' | mlr --itsvlite --opprint cat
@ -39,7 +39,7 @@ Using ``mysql``'s ``-B`` we get TSV output:
Since Miller handles TSV output, we can do as much or as little processing as we want in the SQL query, then send the rest on to Miller. This includes outputting as JSON, doing further selects/joins in Miller, doing stats, etc. etc.:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mysql --database=mydb -B -e 'show columns in mytable' | mlr --itsvlite --ojson --jlistwrap --jvstack cat
@ -86,7 +86,7 @@ Since Miller handles TSV output, we can do as much or as little processing as we
}
]
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mysql --database=mydb -B -e 'select * from mytable' > query.tsv
@ -112,7 +112,7 @@ One use of NIDX (value-only, no keys) format is for loading up SQL tables.
Create and load SQL table:
.. code-block::
.. code-block:: none
mysql> CREATE TABLE abixy(
a VARCHAR(32),
@ -155,7 +155,7 @@ Create and load SQL table:
Aggregate counts within SQL:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
mysql> SELECT a, b, COUNT(*) AS count FROM abixy GROUP BY a, b ORDER BY COUNT DESC;
@ -192,7 +192,7 @@ Aggregate counts within SQL:
Aggregate counts within Miller:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mlr --opprint uniq -c -g a,b then sort -nr count data/medium
@ -215,7 +215,7 @@ Aggregate counts within Miller:
Pipe SQL output to aggregate counts within Miller:
.. code-block::
.. code-block:: none
:emphasize-lines: 1,1
$ mysql -D miller -B -e 'select * from abixy' | mlr --itsv --opprint uniq -c -g a,b then sort -nr count

1
go/todo.txt
View file

@ -6,6 +6,7 @@ TOP OF LIST:
! genrst fixup !
* rst bold
https://stackoverflow.com/questions/26366552/bold-code-in-rst
syntax none ...
* prepipex