diff --git a/docs/10min.rst b/docs/10min.rst index 39bb96f1e..b24d3fc5e 100644 --- a/docs/10min.rst +++ b/docs/10min.rst @@ -9,7 +9,7 @@ CSV-file examples Suppose you have this CSV data file: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat example.csv @@ -27,7 +27,7 @@ Suppose you have this CSV data file: ``mlr cat`` is like cat -- it passes the data through unmodified: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv cat example.csv @@ -45,7 +45,7 @@ Suppose you have this CSV data file: but it can also do format conversion (here, you can pretty-print in tabular format): -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint cat example.csv @@ -63,7 +63,7 @@ but it can also do format conversion (here, you can pretty-print in tabular form ``mlr head`` and ``mlr tail`` count records rather than lines. Whethere you're getting the first few records or the last few, the CSV header is included either way: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv head -n 4 example.csv @@ -73,7 +73,7 @@ but it can also do format conversion (here, you can pretty-print in tabular form red,circle,1,16,13.8103,2.9010 red,square,0,48,77.5542,7.4670 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv tail -n 4 example.csv @@ -85,7 +85,7 @@ but it can also do format conversion (here, you can pretty-print in tabular form You can sort primarily alphabetically on one field, then secondarily numerically descending on another field: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint sort -f shape -nr index example.csv @@ -103,7 +103,7 @@ You can sort primarily alphabetically on one field, then secondarily numerically You can use ``cut`` to retain only specified fields, in the same order they appeared in the input data: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint cut -f flag,shape example.csv @@ -121,7 +121,7 @@ You can use ``cut`` to retain only specified fields, in the same order they appe You can also use ``cut -o`` to retain only specified fields in your preferred order: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint cut -o -f flag,shape example.csv @@ -139,7 +139,7 @@ You can also use ``cut -o`` to retain only specified fields in your preferred or You can use ``cut -x`` to omit fields you don't care about: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint cut -x -f flag,shape example.csv @@ -157,7 +157,7 @@ You can use ``cut -x`` to omit fields you don't care about: You can use ``filter`` to keep only records you care about: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint filter '$color == "red"' example.csv @@ -167,7 +167,7 @@ You can use ``filter`` to keep only records you care about: red square 0 48 77.5542 7.4670 red square 0 64 77.1991 9.5310 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint filter '$color == "red" && $flag == 1' example.csv @@ -177,7 +177,7 @@ You can use ``filter`` to keep only records you care about: You can use ``put`` to create new fields which are computed from other fields: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint put '$ratio = $quantity / $rate; $color_shape = $color . "_" . $shape' example.csv @@ -195,7 +195,7 @@ You can use ``put`` to create new fields which are computed from other fields: Even though Miller's main selling point is name-indexing, sometimes you really want to refer to a field name by its positional index. Use ``$[[3]]`` to access the name of field 3 or ``$[[[3]]]`` to access the value of field 3: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint put '$[[3]] = "NEW"' example.csv @@ -211,7 +211,7 @@ Even though Miller's main selling point is name-indexing, sometimes you really w yellow circle 1 87 63.5058 8.3350 purple square 0 91 72.3735 8.2430 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint put '$[[[3]]] = "NEW"' example.csv @@ -232,7 +232,7 @@ JSON-file examples OK, CSV and pretty-print are fine. But Miller can also convert between a few other formats -- let's take a look at JSON output: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --ojson put '$ratio = $quantity/$rate; $shape = toupper($shape)' example.csv @@ -249,7 +249,7 @@ OK, CSV and pretty-print are fine. But Miller can also convert between a few oth Or, JSON output with vertical-formatting flags: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --ojsonx tail -n 2 example.csv @@ -277,7 +277,7 @@ Now suppose you want to sort the data on a given column, *and then* take the top Here are the records with the top three ``index`` values: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint sort -f shape -nr index then head -n 3 example.csv @@ -288,7 +288,7 @@ Here are the records with the top three ``index`` values: Lots of Miller commands take a ``-g`` option for group-by: here, ``head -n 1 -g shape`` outputs the first record for each distinct value of the ``shape`` field. This means we're finding the record with highest ``index`` field for each distinct ``shape`` field: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint sort -f shape -nr index then head -n 1 -g shape example.csv @@ -299,7 +299,7 @@ Lots of Miller commands take a ``-g`` option for group-by: here, ``head -n 1 -g Statistics can be computed with or without group-by field(s): -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint --from example.csv stats1 -a count,min,mean,max -f quantity -g shape @@ -308,7 +308,7 @@ Statistics can be computed with or without group-by field(s): square 4 72.373500 76.601150 79.277800 circle 3 13.810300 47.098200 63.978500 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint --from example.csv stats1 -a count,min,mean,max -f quantity -g shape,color @@ -322,7 +322,7 @@ Statistics can be computed with or without group-by field(s): If your output has a lot of columns, you can use XTAB format to line things up vertically for you instead: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --oxtab --from example.csv stats1 -a p0,p10,p25,p50,p75,p90,p99,p100 -f rate @@ -344,14 +344,14 @@ Often we want to print output to the screen. Miller does this by default, as we' Sometimes we want to print output to another file: just use **> outputfilenamegoeshere** at the end of your command: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 % mlr --icsv --opprint cat example.csv > newfile.csv # Output goes to the new file; # nothing is printed to the screen. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 % cat newfile.csv @@ -369,12 +369,12 @@ Sometimes we want to print output to another file: just use **> outputfilenamego Other times we just want our files to be **changed in-place**: just use **mlr -I**: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 % cp example.csv newfile.txt -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 % cat newfile.txt @@ -390,12 +390,12 @@ Other times we just want our files to be **changed in-place**: just use **mlr -I yellow,circle,1,87,63.5058,8.3350 purple,square,0,91,72.3735,8.2430 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 % mlr -I --icsv --opprint cat newfile.txt -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 % cat newfile.txt @@ -413,7 +413,7 @@ Other times we just want our files to be **changed in-place**: just use **mlr -I Also using ``mlr -I`` you can bulk-operate on lots of files: e.g.: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 mlr -I --csv cut -x -f unwanted_column_name *.csv @@ -422,12 +422,12 @@ If you like, you can first copy off your original data somewhere else, before do Lastly, using ``tee`` within ``put``, you can split your input data into separate files per one or more field names: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv --from example.csv put -q 'tee > $shape.".csv", $*' -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat circle.csv @@ -436,7 +436,7 @@ Lastly, using ``tee`` within ``put``, you can split your input data into separat yellow,circle,1,73,63.9785,4.2370 yellow,circle,1,87,63.5058,8.3350 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat square.csv @@ -446,7 +446,7 @@ Lastly, using ``tee`` within ``put``, you can split your input data into separat red,square,0,64,77.1991,9.5310 purple,square,0,91,72.3735,8.2430 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat triangle.csv @@ -462,7 +462,7 @@ What's a CSV file, really? It's an array of rows, or *records*, each being a lis For example, if you have: -.. code-block:: bash +.. code-block:: none shape,flag,index circle,1,24 @@ -470,7 +470,7 @@ For example, if you have: then that's a way of saying: -.. code-block:: bash +.. code-block:: none shape=circle,flag=1,index=24 shape=square,flag=0,index=36 @@ -479,7 +479,7 @@ Data written this way are called **DKVP**, for *delimited key-value pairs*. We've also already seen other ways to write the same data: -.. code-block:: bash +.. code-block:: none CSV PPRINT JSON shape,flag,index shape flag index [ diff --git a/docs/10min.rst.in b/docs/10min.rst.in index 714b6bfea..4dc620ace 100644 --- a/docs/10min.rst.in +++ b/docs/10min.rst.in @@ -97,14 +97,14 @@ Often we want to print output to the screen. Miller does this by default, as we' Sometimes we want to print output to another file: just use **> outputfilenamegoeshere** at the end of your command: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 % mlr --icsv --opprint cat example.csv > newfile.csv # Output goes to the new file; # nothing is printed to the screen. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 % cat newfile.csv @@ -122,12 +122,12 @@ Sometimes we want to print output to another file: just use **> outputfilenamego Other times we just want our files to be **changed in-place**: just use **mlr -I**: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 % cp example.csv newfile.txt -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 % cat newfile.txt @@ -143,12 +143,12 @@ Other times we just want our files to be **changed in-place**: just use **mlr -I yellow,circle,1,87,63.5058,8.3350 purple,square,0,91,72.3735,8.2430 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 % mlr -I --icsv --opprint cat newfile.txt -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 % cat newfile.txt @@ -166,7 +166,7 @@ Other times we just want our files to be **changed in-place**: just use **mlr -I Also using ``mlr -I`` you can bulk-operate on lots of files: e.g.: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 mlr -I --csv cut -x -f unwanted_column_name *.csv @@ -190,7 +190,7 @@ What's a CSV file, really? It's an array of rows, or *records*, each being a lis For example, if you have: -.. code-block:: bash +.. code-block:: none shape,flag,index circle,1,24 @@ -198,7 +198,7 @@ For example, if you have: then that's a way of saying: -.. code-block:: bash +.. code-block:: none shape=circle,flag=1,index=24 shape=square,flag=0,index=36 @@ -207,7 +207,7 @@ Data written this way are called **DKVP**, for *delimited key-value pairs*. We've also already seen other ways to write the same data: -.. code-block:: bash +.. code-block:: none CSV PPRINT JSON shape,flag,index shape flag index [ diff --git a/docs/Makefile b/docs/Makefile index b345487bd..5641e5529 100644 --- a/docs/Makefile +++ b/docs/Makefile @@ -24,5 +24,5 @@ help: # Catch-all target: route all unknown targets to Sphinx using the new # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). %: Makefile - #### temp ./genrst + ./genrst @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) diff --git a/docs/build.rst b/docs/build.rst index 3df63b240..3ec1d9121 100644 --- a/docs/build.rst +++ b/docs/build.rst @@ -72,7 +72,7 @@ Miller has been built on Windows using MSYS2: http://www.msys2.org/. You can in You will first need to install MSYS2: http://www.msys2.org/. Then, start an MSYS2 shell, e.g. (supposing you installed MSYS2 to ``C:\msys2\``) run ``C:\msys2\mingw64.exe``. Within the MSYS2 shell, you can run the following to install dependent packages: -.. code-block:: bash +.. code-block:: none pacman -Syu pacman -Su @@ -90,13 +90,13 @@ There is a unit-test false-negative issue involving the semantics of the ``mkste Within MSYS2 you can run ``mlr``: simply copy it from the ``c`` subdirectory to your desired location somewhere within your MSYS2 ``$PATH``. To run ``mlr`` outside of MSYS2, just as with precompiled binaries as described above, you'll need ``msys-2.0.dll``. One way to do this is to augment your path: -.. code-block:: bash +.. code-block:: none C:\> set PATH=%PATH%;\msys64\mingw64\bin Another way to do it is to copy the Miller executable and the DLL to the same directory: -.. code-block:: bash +.. code-block:: none C:\> mkdir \mbin C:\> copy \msys64\mingw64\bin\msys-2.0.dll \mbin @@ -181,7 +181,7 @@ In this example I am using version 3.4.0; of course that will change for subsequ * Similarly for ``macports``: https://github.com/macports/macports-ports/blob/master/textproc/miller/Portfile. * Social-media updates. -.. code-block:: bash +.. code-block:: none git remote add upstream https://github.com/Homebrew/homebrew-core # one-time setup only git fetch upstream diff --git a/docs/build.rst.in b/docs/build.rst.in index e0508b162..acac15d2b 100644 --- a/docs/build.rst.in +++ b/docs/build.rst.in @@ -69,7 +69,7 @@ Miller has been built on Windows using MSYS2: http://www.msys2.org/. You can in You will first need to install MSYS2: http://www.msys2.org/. Then, start an MSYS2 shell, e.g. (supposing you installed MSYS2 to ``C:\msys2\``) run ``C:\msys2\mingw64.exe``. Within the MSYS2 shell, you can run the following to install dependent packages: -.. code-block:: bash +.. code-block:: none pacman -Syu pacman -Su @@ -87,13 +87,13 @@ There is a unit-test false-negative issue involving the semantics of the ``mkste Within MSYS2 you can run ``mlr``: simply copy it from the ``c`` subdirectory to your desired location somewhere within your MSYS2 ``$PATH``. To run ``mlr`` outside of MSYS2, just as with precompiled binaries as described above, you'll need ``msys-2.0.dll``. One way to do this is to augment your path: -.. code-block:: bash +.. code-block:: none C:\> set PATH=%PATH%;\msys64\mingw64\bin Another way to do it is to copy the Miller executable and the DLL to the same directory: -.. code-block:: bash +.. code-block:: none C:\> mkdir \mbin C:\> copy \msys64\mingw64\bin\msys-2.0.dll \mbin @@ -178,7 +178,7 @@ In this example I am using version 3.4.0; of course that will change for subsequ * Similarly for ``macports``: https://github.com/macports/macports-ports/blob/master/textproc/miller/Portfile. * Social-media updates. -.. code-block:: bash +.. code-block:: none git remote add upstream https://github.com/Homebrew/homebrew-core # one-time setup only git fetch upstream diff --git a/docs/cookbook.rst b/docs/cookbook.rst index e1fc82b38..9d145db48 100644 --- a/docs/cookbook.rst +++ b/docs/cookbook.rst @@ -9,7 +9,7 @@ Headerless CSV on input or output Sometimes we get CSV files which lack a header. For example: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/headerless.csv @@ -20,7 +20,7 @@ Sometimes we get CSV files which lack a header. For example: You can use Miller to add a header. The ``--implicit-csv-header`` applies positionally indexed labels: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv --implicit-csv-header cat data/headerless.csv @@ -32,7 +32,7 @@ You can use Miller to add a header. The ``--implicit-csv-header`` applies positi Following that, you can rename the positionally indexed labels to names with meaning for your context. For example: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv --implicit-csv-header label name,age,status data/headerless.csv @@ -44,7 +44,7 @@ Following that, you can rename the positionally indexed labels to names with mea Likewise, if you need to produce CSV which is lacking its header, you can pipe Miller's output to the system command ``sed 1d``, or you can use Miller's ``--headerless-csv-output`` option: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ head -5 data/colored-shapes.dkvp | mlr --ocsv cat @@ -55,7 +55,7 @@ Likewise, if you need to produce CSV which is lacking its header, you can pipe M red,square,0,48,0.9562743938458542,0.7467203085342884,0.7755423050923582,7.117831369597269 purple,triangle,0,51,0.4355354501763202,0.8591292672156728,0.8122903963006748,5.753094629505863 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ head -5 data/colored-shapes.dkvp | mlr --ocsv --headerless-csv-output cat @@ -67,7 +67,7 @@ Likewise, if you need to produce CSV which is lacking its header, you can pipe M Lastly, often we say "CSV" or "TSV" when we have positionally indexed data in columns which are separated by commas or tabs, respectively. In this case it's perhaps simpler to **just use NIDX format** which was designed for this purpose. (See also :doc:`file-formats`.) For example: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --inidx --ifs comma --oxtab cut -f 1,3 data/headerless.csv @@ -88,7 +88,7 @@ Doing multiple joins Suppose we have the following data: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat multi-join/input.csv @@ -104,7 +104,7 @@ Suppose we have the following data: And we want to augment the ``id`` column with lookups from the following data files: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat multi-join/name-lookup.csv @@ -113,7 +113,7 @@ And we want to augment the ``id`` column with lookups from the following data fi 10,Bob 20,Carol -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat multi-join/status-lookup.csv @@ -124,7 +124,7 @@ And we want to augment the ``id`` column with lookups from the following data fi We can run the input file through multiple ``join`` commands in a ``then``-chain: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint join -f multi-join/name-lookup.csv -j id then join -f multi-join/status-lookup.csv -j id multi-join/input.csv @@ -143,7 +143,7 @@ Bulk rename of fields Suppose you want to replace spaces with underscores in your column names: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/spaces.csv @@ -154,7 +154,7 @@ Suppose you want to replace spaces with underscores in your column names: The simplest way is to use ``mlr rename`` with ``-g`` (for global replace, not just first occurrence of space within each field) and ``-r`` for pattern-matching (rather than explicit single-column renames): -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv rename -g -r ' ,_' data/spaces.csv @@ -163,7 +163,7 @@ The simplest way is to use ``mlr rename`` with ``-g`` (for global replace, not j 2468,1357,3579 9987,3312,4543 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv --opprint rename -g -r ' ,_' data/spaces.csv @@ -174,7 +174,7 @@ The simplest way is to use ``mlr rename`` with ``-g`` (for global replace, not j You can also do this with a for-loop: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/bulk-rename-for-loop.mlr @@ -184,7 +184,7 @@ You can also do this with a for-loop: } $* = newrec -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint put -f data/bulk-rename-for-loop.mlr data/spaces.csv @@ -198,7 +198,7 @@ Search-and-replace over all fields How to do ``$name = gsub($name, "old", "new")`` for all fields? -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/sar.csv @@ -206,7 +206,7 @@ How to do ``$name = gsub($name, "old", "new")`` for all fields? the quick,brown fox,jumped over,the,lazy dogs -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/sar.mlr @@ -214,7 +214,7 @@ How to do ``$name = gsub($name, "old", "new")`` for all fields? $[k] = gsub($[k], "e", "X"); } -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv put -f data/sar.mlr data/sar.csv @@ -227,7 +227,7 @@ Full field renames and reassigns Using Miller 5.0.0's map literals and assigning to ``$*``, you can fully generalize :ref:`mlr rename `, :ref:`mlr reorder `, etc. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/small @@ -237,7 +237,7 @@ Using Miller 5.0.0's map literals and assigning to ``$*``, you can fully general a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463 a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put ' @@ -266,7 +266,7 @@ Numbering and renumbering records The ``awk``-like built-in variable ``NR`` is incremented for each input record: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/small @@ -276,7 +276,7 @@ The ``awk``-like built-in variable ``NR`` is incremented for each input record: a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463 a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put '$nr = NR' data/small @@ -288,7 +288,7 @@ The ``awk``-like built-in variable ``NR`` is incremented for each input record: However, this is the record number within the original input stream -- not after any filtering you may have done: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr filter '$a == "wye"' then put '$nr = NR' data/small @@ -297,7 +297,7 @@ However, this is the record number within the original input stream -- not after There are two good options here. One is to use the ``cat`` verb with ``-n``: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr filter '$a == "wye"' then cat -n data/small @@ -306,7 +306,7 @@ There are two good options here. One is to use the ``cat`` verb with ``-n``: The other is to keep your own counter within the ``put`` DSL: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr filter '$a == "wye"' then put 'begin {@n = 1} $n = @n; @n += 1' data/small @@ -329,7 +329,7 @@ Data-cleaning examples Here are some ways to use the type-checking options as described in :ref:`reference-dsl-type-tests-and-assertions` Suppose you have the following data file, with inconsistent typing for boolean. (Also imagine that, for the sake of discussion, we have a million-line file rather than a four-line file, so we can't see it all at once and some automation is called for.) -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/het-bool.csv @@ -341,7 +341,7 @@ Here are some ways to use the type-checking options as described in :ref:`refere One option is to coerce everything to boolean, or integer: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint put '$reachable = boolean($reachable)' data/het-bool.csv @@ -351,7 +351,7 @@ One option is to coerce everything to boolean, or integer: fred true wilma true -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint put '$reachable = int(boolean($reachable))' data/het-bool.csv @@ -363,7 +363,7 @@ One option is to coerce everything to boolean, or integer: A second option is to flag badly formatted data within the output stream: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint put '$format_ok = is_string($reachable)' data/het-bool.csv @@ -375,7 +375,7 @@ A second option is to flag badly formatted data within the output stream: Or perhaps to flag badly formatted data outside the output stream: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint put 'if (!is_string($reachable)) {eprint "Malformed at NR=".NR} ' data/het-bool.csv @@ -388,7 +388,7 @@ Or perhaps to flag badly formatted data outside the output stream: A third way is to abort the process on first instance of bad data: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv put '$reachable = asserting_string($reachable)' data/het-bool.csv @@ -403,7 +403,7 @@ Splitting nested fields Suppose you have a TSV file like this: -.. code-block:: bash +.. code-block:: none a b x z @@ -411,7 +411,7 @@ Suppose you have a TSV file like this: The simplest option is to use :ref:`mlr nest `: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --tsv nest --explode --values --across-records -f b --nested-fs : data/nested.tsv @@ -421,7 +421,7 @@ The simplest option is to use :ref:`mlr nest `: s v s w -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --tsv nest --explode --values --across-fields -f b --nested-fs : data/nested.tsv @@ -435,7 +435,7 @@ While ``mlr nest`` is simplest, let's also take a look at a few ways to do this One option to split out the colon-delimited values in the ``b`` column is to use ``splitnv`` to create an integer-indexed map and loop over it, adding new fields to the current record: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/nested.tsv --itsv --oxtab put 'o=splitnv($b, ":"); for (k,v in o) {$["p".k]=v}' @@ -451,7 +451,7 @@ One option to split out the colon-delimited values in the ``b`` column is to use while another is to loop over the same map from ``splitnv`` and use it (with ``put -q`` to suppress printing the original record) to produce multiple records: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/nested.tsv --itsv --oxtab put -q 'o=splitnv($b, ":"); for (k,v in o) {emit mapsum($*, {"b":v})}' @@ -467,7 +467,7 @@ while another is to loop over the same map from ``splitnv`` and use it (with ``p a s b w -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/nested.tsv --tsv put -q 'o=splitnv($b, ":"); for (k,v in o) {emit mapsum($*, {"b":v})}' @@ -482,7 +482,7 @@ Showing differences between successive queries Suppose you have a database query which you run at one point in time, producing the output on the left, then again later producing the output on the right: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/previous_counters.csv @@ -492,7 +492,7 @@ Suppose you have a database query which you run at one point in time, producing orange,694 purple,12 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/current_counters.csv @@ -506,12 +506,12 @@ And, suppose you want to compute the differences in the counters between adjacen First, rename counter columns to make them distinct: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv rename count,previous_count data/previous_counters.csv > data/prevtemp.csv -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/prevtemp.csv @@ -521,12 +521,12 @@ First, rename counter columns to make them distinct: orange,694 purple,12 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv rename count,current_count data/current_counters.csv > data/currtemp.csv -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/currtemp.csv @@ -538,7 +538,7 @@ First, rename counter columns to make them distinct: Then, join on the key field(s), and use unsparsify to zero-fill counters absent on one side but present on the other. Use ``--ul`` and ``--ur`` to emit unpaired records (namely, purple on the left and yellow on the right): -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint \ @@ -558,7 +558,7 @@ Finding missing dates Suppose you have some date-stamped data which may (or may not) be missing entries for one or more dates: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ head -n 10 data/miss-date.csv @@ -573,7 +573,7 @@ Suppose you have some date-stamped data which may (or may not) be missing entrie 2012-03-12,11043 2012-03-13,11177 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ wc -l data/miss-date.csv @@ -581,7 +581,7 @@ Suppose you have some date-stamped data which may (or may not) be missing entrie Since there are 1372 lines in the data file, some automation is called for. To find the missing dates, you can convert the dates to seconds since the epoch using ``strptime``, then compute adjacent differences (the ``cat -n`` simply inserts record-counters): -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/miss-date.csv --icsv \ @@ -602,7 +602,7 @@ Since there are 1372 lines in the data file, some automation is called for. To f Then, filter for adjacent difference not being 86400 (the number of seconds in a day): -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/miss-date.csv --icsv \ @@ -615,7 +615,7 @@ Then, filter for adjacent difference not being 86400 (the number of seconds in a Given this, it's now easy to see where the gaps are: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr cat -n then filter '$n >= 770 && $n <= 780' data/miss-date.csv @@ -631,7 +631,7 @@ Given this, it's now easy to see where the gaps are: n=779,1=2014-04-23,2=130849 n=780,1=2014-04-24,2=131026 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr cat -n then filter '$n >= 1115 && $n <= 1125' data/miss-date.csv @@ -657,7 +657,7 @@ Two-pass algorithms: computation of percentages For example, mapping numeric values down a column to the percentage between their min and max values is two-pass: on the first pass you find the min and max values, then on the second, map each record's value to a percentage. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/small --opprint put -q ' @@ -688,7 +688,7 @@ Two-pass algorithms: line-number ratios Similarly, finding the total record count requires first reading through all the data: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint --from data/small put -q ' @@ -714,7 +714,7 @@ Two-pass algorithms: records having max value The idea is to retain records having the largest value of ``n`` in the following data: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --itsv --opprint cat data/maxrows.tsv @@ -753,7 +753,7 @@ The idea is to retain records having the largest value of ``n`` in the following Of course, the largest value of ``n`` isn't known until after all data have been read. Using an out-of-stream variable we can retain all records as they are read, then filter them at the end: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/maxrows.mlr @@ -773,7 +773,7 @@ Of course, the largest value of ``n`` isn't known until after all data have been } } -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --itsv --opprint put -q -f data/maxrows.mlr data/maxrows.tsv @@ -793,7 +793,7 @@ Rectangularizing data Suppose you have a method (in whatever language) which is printing things of the form -.. code-block:: bash +.. code-block:: none outer=1 outer=2 @@ -801,7 +801,7 @@ Suppose you have a method (in whatever language) which is printing things of the and then calls another method which prints things of the form -.. code-block:: bash +.. code-block:: none middle=10 middle=11 @@ -813,7 +813,7 @@ and then calls another method which prints things of the form and then, perhaps, that second method calls a third method which prints things of the form -.. code-block:: bash +.. code-block:: none inner1=100,inner2=101 inner1=120,inner2=121 @@ -825,7 +825,7 @@ and then, perhaps, that second method calls a third method which prints things o with the result that your program's output is -.. code-block:: bash +.. code-block:: none outer=1 middle=10 @@ -847,7 +847,7 @@ with the result that your program's output is The idea here is that middles starting with a 1 belong to the outer value of 1, and so on. (For example, the outer values might be account IDs, the middle values might be invoice IDs, and the inner values might be invoice line-items.) If you want all the middle and inner lines to have the context of which outers they belong to, you can modify your software to pass all those through your methods. Alternatively, don't refactor your code just to handle some ad-hoc log-data formatting -- instead, use the following to rectangularize the data. The idea is to use an out-of-stream variable to accumulate fields across records. Clear that variable when you see an outer ID; accumulate fields; emit output when you see the inner IDs. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/rect.txt put -q ' @@ -873,7 +873,7 @@ Regularizing ragged CSV Miller handles compliant CSV: in particular, it's an error if the number of data fields in a given data line don't match the number of header lines. But in the event that you have a CSV file in which some lines have less than the full number of fields, you can use Miller to pad them out. The trick is to use NIDX format, for which each line stands on its own without respect to a header line. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/ragged.csv @@ -882,7 +882,7 @@ Miller handles compliant CSV: in particular, it's an error if the number of data 4,5 6,7,8,9 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/ragged.csv --fs comma --nidx put ' @@ -900,7 +900,7 @@ Miller handles compliant CSV: in particular, it's an error if the number of data or, more simply, -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/ragged.csv --fs comma --nidx put ' @@ -919,7 +919,7 @@ Feature-counting Suppose you have some heterogeneous data like this: -.. code-block:: bash +.. code-block:: none { "qoh": 29874, "rate": 1.68, "latency": 0.02 } { "name": "alice", "uid": 572 } @@ -936,7 +936,7 @@ Suppose you have some heterogeneous data like this: A reasonable question to ask is, how many occurrences of each field are there? And, what percentage of total row count has each of them? Since the denominator of the percentage is not known until the end, this is a two-pass algorithm: -.. code-block:: bash +.. code-block:: none for (key in $*) { @key_counts[key] += 1; @@ -954,7 +954,7 @@ A reasonable question to ask is, how many occurrences of each field are there? A Then -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --json put -q -f data/feature-count.mlr data/features.json @@ -972,7 +972,7 @@ Then { "key": "uid", "key_fraction": 0.250000 } { "key": "uid2", "key_fraction": 0.083333 } -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --ijson --opprint put -q -f data/feature-count.mlr data/features.json @@ -1002,7 +1002,7 @@ The previous section discussed how to fill out missing data fields within CSV wi For example, suppose you have JSON input like this: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/sparse.json @@ -1013,7 +1013,7 @@ For example, suppose you have JSON input like this: There are field names ``a``, ``b``, ``v``, ``u``, ``x``, ``w`` in the data -- but not all in every record. Since we don't know the names of all the keys until we've read them all, this needs to be a two-pass algorithm. On the first pass, remember all the unique key names and all the records; on the second pass, loop through the records filling in absent values, then producing output. Use ``put -q`` since we don't want to produce per-record output, only emitting output in the ``end`` block: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/unsparsify.mlr @@ -1044,7 +1044,7 @@ There are field names ``a``, ``b``, ``v``, ``u``, ``x``, ``w`` in the data -- bu } } -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --json put -q -f data/unsparsify.mlr data/sparse.json @@ -1053,7 +1053,7 @@ There are field names ``a``, ``b``, ``v``, ``u``, ``x``, ``w`` in the data -- bu { "a": 1, "b": "", "v": 2, "u": "", "x": 3, "w": "" } { "a": "", "b": "", "v": 1, "u": "", "x": "", "w": 2 } -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --ijson --ocsv put -q -f data/unsparsify.mlr data/sparse.json @@ -1063,7 +1063,7 @@ There are field names ``a``, ``b``, ``v``, ``u``, ``x``, ``w`` in the data -- bu 1,,2,,3, ,,1,,,2 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --ijson --opprint put -q -f data/unsparsify.mlr data/sparse.json @@ -1080,13 +1080,13 @@ Parsing log-file output This, of course, depends highly on what's in your log files. But, as an example, suppose you have log-file lines such as -.. code-block:: bash +.. code-block:: none 2015-10-08 08:29:09,445 INFO com.company.path.to.ClassName @ [sometext] various/sorts/of data {& punctuation} hits=1 status=0 time=2.378 I prefer to pre-filter with ``grep`` and/or ``sed`` to extract the structured text, then hand that to Miller. Example: -.. code-block:: bash +.. code-block:: none grep 'various sorts' *.log | sed 's/.*} //' | mlr --fs space --repifs --oxtab stats1 -a min,p10,p50,p90,max -f time -g status @@ -1097,7 +1097,7 @@ Memoization with out-of-stream variables The recursive function for the Fibonacci sequence is famous for its computational complexity. Namely, using *f*(0)=1, *f*(1)=1, *f*(*n*)=*f*(*n*-1)+*f*(*n*-2) for *n*≥2, the evaluation tree branches left as well as right at each non-trivial level, resulting in millions or more paths to the root 0/1 nodes for larger *n*. This program -.. code-block:: bash +.. code-block:: none mlr --ofmt '%.9lf' --opprint seqgen --start 1 --stop 28 then put ' func f(n) { @@ -1118,7 +1118,7 @@ The recursive function for the Fibonacci sequence is famous for its computationa produces output like this: -.. code-block:: bash +.. code-block:: none i o fcount seconds_delta 1 1 1 0 @@ -1152,7 +1152,7 @@ produces output like this: Note that the time it takes to evaluate the function is blowing up exponentially as the input argument increases. Using ``@``-variables, which persist across records, we can cache and reuse the results of previous computations: -.. code-block:: bash +.. code-block:: none mlr --ofmt '%.9lf' --opprint seqgen --start 1 --stop 28 then put ' func f(n) { @@ -1175,7 +1175,7 @@ Note that the time it takes to evaluate the function is blowing up exponentially with output like this: -.. code-block:: bash +.. code-block:: none i o fcount seconds_delta 1 1 1 0 diff --git a/docs/cookbook.rst.in b/docs/cookbook.rst.in index 533f00d9b..734e2f3d2 100644 --- a/docs/cookbook.rst.in +++ b/docs/cookbook.rst.in @@ -323,13 +323,13 @@ Parsing log-file output This, of course, depends highly on what's in your log files. But, as an example, suppose you have log-file lines such as -.. code-block:: bash +.. code-block:: none 2015-10-08 08:29:09,445 INFO com.company.path.to.ClassName @ [sometext] various/sorts/of data {& punctuation} hits=1 status=0 time=2.378 I prefer to pre-filter with ``grep`` and/or ``sed`` to extract the structured text, then hand that to Miller. Example: -.. code-block:: bash +.. code-block:: none grep 'various sorts' *.log | sed 's/.*} //' | mlr --fs space --repifs --oxtab stats1 -a min,p10,p50,p90,max -f time -g status @@ -344,7 +344,7 @@ POKI_INCLUDE_ESCAPED(data/fibo-uncached.sh)HERE produces output like this: -.. code-block:: bash +.. code-block:: none i o fcount seconds_delta 1 1 1 0 @@ -382,7 +382,7 @@ POKI_INCLUDE_ESCAPED(data/fibo-cached.sh)HERE with output like this: -.. code-block:: bash +.. code-block:: none i o fcount seconds_delta 1 1 1 0 diff --git a/docs/cookbook2.rst b/docs/cookbook2.rst index 1a88a0980..ee67e957b 100644 --- a/docs/cookbook2.rst +++ b/docs/cookbook2.rst @@ -9,7 +9,7 @@ Randomly selecting words from a list Given this `word list `_, first take a look to see what the first few lines look like: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ head data/english-words.txt @@ -26,7 +26,7 @@ Given this `word list lines.txt @@ -109,7 +109,7 @@ Computing interquartile ranges For one or more specified field names, simply compute p25 and p75, then write the IQR as the difference of p75 and p25: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --oxtab stats1 -f x -a p25,p75 \ @@ -121,7 +121,7 @@ For one or more specified field names, simply compute p25 and p75, then write th For wildcarded field names, first compute p25 and p75, then loop over field names with ``p25`` in them: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --oxtab stats1 --fr '[i-z]' -a p25,p75 \ @@ -146,7 +146,7 @@ Computing weighted means This might be more elegantly implemented as an option within the ``stats1`` verb. Meanwhile, it's expressible within the DSL: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/medium put -q ' @@ -184,7 +184,7 @@ Generating random numbers from various distributions Here we can chain together a few simple building blocks: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat expo-sample.sh @@ -227,7 +227,7 @@ Namely: The output is as follows: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ sh expo-sample.sh @@ -288,7 +288,7 @@ Sieve of Eratosthenes The `Sieve of Eratosthenes `_ is a standard introductory programming topic. The idea is to find all primes up to some *N* by making a list of the numbers 1 to *N*, then striking out all multiples of 2 except 2 itself, all multiples of 3 except 3 itself, all multiples of 4 except 4 itself, and so on. Whatever survives that without getting marked is a prime. This is easy enough in Miller. Notice that here all the work is in ``begin`` and ``end`` statements; there is no file input (so we use ``mlr -n`` to keep Miller from waiting for input data). -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat programs/sieve.mlr @@ -325,7 +325,7 @@ The `Sieve of Eratosthenes ` } } -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr -n put -f programs/sieve.mlr @@ -362,7 +362,7 @@ The `Mandelbrot set `_ is also easi The (approximate) computation of points in the complex plane which are and aren't members is just a few lines of complex arithmetic (see the Wikipedia article); how to render them is another task. Using graphics libraries you can create PNG or JPEG files, but another fun way to do this is by printing various characters to the screen: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat programs/mand.mlr @@ -469,7 +469,7 @@ The (approximate) computation of points in the complex plane which are and aren' At standard resolution this makes a nice little ASCII plot: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr -n put -f ./programs/mand.mlr @@ -526,7 +526,7 @@ At standard resolution this makes a nice little ASCII plot: But using a very small font size (as small as my Mac will let me go), and by choosing the coordinates to zoom in on a particular part of the complex plane, we can get a nice little picture: -.. code-block:: bash +.. code-block:: none #!/bin/bash # Get the number of rows and columns from the terminal window dimensions diff --git a/docs/cookbook2.rst.in b/docs/cookbook2.rst.in index f3fcfc412..b4df6520d 100644 --- a/docs/cookbook2.rst.in +++ b/docs/cookbook2.rst.in @@ -6,7 +6,7 @@ Randomly selecting words from a list Given this `word list `_, first take a look to see what the first few lines look like: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ head data/english-words.txt @@ -23,7 +23,7 @@ Given this `word list strptime("2018-03-03", "%Y-%m-%d")' dates.csv @@ -315,7 +315,7 @@ How can I handle commas-as-data in various formats? :doc:`CSV ` handles this well and by design: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat commas.csv @@ -325,7 +325,7 @@ How can I handle commas-as-data in various formats? Likewise :ref:`file-formats-json`: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --ojson cat commas.csv @@ -334,7 +334,7 @@ Likewise :ref:`file-formats-json`: For Miller's :ref:`vertical-tabular format ` there is no escaping for carriage returns, but commas work fine: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --oxtab cat commas.csv @@ -346,7 +346,7 @@ For Miller's :ref:`vertical-tabular format ` there is no esca But for :ref:`Key-value_pairs ` and :ref:`index-numbered `, commas are the default field separator. And -- as of Miller 5.4.0 anyway -- there is no CSV-style double-quote-handling like there is for CSV. So commas within the data look like delimiters: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --odkvp cat commas.csv @@ -355,7 +355,7 @@ But for :ref:`Key-value_pairs ` and :ref:`index-numbered 0.5' then put '$NR = NR' data/small @@ -510,7 +510,7 @@ why don't I see ``NR=1`` and ``NR=2`` here?? The reason is that ``NR`` is computed for the original input records and isn't dynamically updated. By contrast, ``NF`` is dynamically updated: it's the number of fields in the current record, and if you add/remove a field, the value of ``NF`` will change: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ echo x=1,y=2,z=3 | mlr put '$nf1 = NF; $u = 4; $nf2 = NF; unset $x,$y,$z; $nf3 = NF' @@ -518,7 +518,7 @@ The reason is that ``NR`` is computed for the original input records and isn't d ``NR``, by contrast (and ``FNR`` as well), retains the value from the original input stream, and records may be dropped by a ``filter`` within a ``then``-chain. To recover consecutive record numbers, you can use out-of-stream variables as follows: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint --from data/small put ' @@ -538,7 +538,7 @@ The reason is that ``NR`` is computed for the original input records and isn't d Or, simply use ``mlr cat -n``: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr filter '$x > 0.5' then cat -n data/small @@ -552,7 +552,7 @@ Why am I not seeing all possible joins occur? For example, the right file here has nine records, and the left file should add in the ``hostname`` column -- so the join output should also have 9 records: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsvlite --opprint cat data/join-u-left.csv @@ -561,7 +561,7 @@ For example, the right file here has nine records, and the left file should add zenith.west.our.org 10.3.1.27 apoapsis.east.our.org 10.4.5.94 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsvlite --opprint cat data/join-u-right.csv @@ -576,7 +576,7 @@ For example, the right file here has nine records, and the left file should add 10.3.1.18 1448762598 73425 10.4.5.94 1448762599 12200 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsvlite --opprint join -s -j ipaddr -f data/join-u-left.csv data/join-u-right.csv @@ -590,7 +590,7 @@ The issue is that Miller's ``join``, by default (before 5.1.0), took input sorte The solution (besides pre-sorting the input files on the join keys) is to simply use **mlr join -u** (which is now the default). This loads the left file entirely into memory (while the right file is still streamed one line at a time) and does all possible joins without requiring sorted input: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsvlite --opprint join -u -j ipaddr -f data/join-u-left.csv data/join-u-right.csv @@ -612,14 +612,14 @@ How to rectangularize after joins with unpaired? Suppose you have the following two data files: -.. code-block:: bash +.. code-block:: none id,code 3,0000ff 2,00ff00 4,ff0000 -.. code-block:: bash +.. code-block:: none id,color 4,red @@ -627,7 +627,7 @@ Suppose you have the following two data files: Joining on color the results are as expected: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv join -j id -f data/color-codes.csv data/color-names.csv @@ -637,7 +637,7 @@ Joining on color the results are as expected: However, if we ask for left-unpaireds, since there's no ``color`` column, we get a row not having the same column names as the other: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv join --ul -j id -f data/color-codes.csv data/color-names.csv @@ -650,7 +650,7 @@ However, if we ask for left-unpaireds, since there's no ``color`` column, we get To fix this, we can use **unsparsify**: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv join --ul -j id -f data/color-codes.csv then unsparsify --fill-with "" data/color-names.csv @@ -670,13 +670,13 @@ XML, JSON, etc. are, by contrast, all **recursive** or **nested** data structure Now, you can put tabular data into these formats -- since list-of-key-value-pairs is one of the things representable in XML or JSON. Example: -.. code-block:: bash +.. code-block:: none # DKVP x=1,y=2 z=3 -.. code-block:: bash +.. code-block:: none # XML @@ -695,7 +695,7 @@ Now, you can put tabular data into these formats -- since list-of-key-value-pair
-.. code-block:: bash +.. code-block:: none # JSON [{"x":1,"y":2},{"z":3}] diff --git a/docs/faq.rst.in b/docs/faq.rst.in index 4cf754bad..10238fe19 100644 --- a/docs/faq.rst.in +++ b/docs/faq.rst.in @@ -278,13 +278,13 @@ XML, JSON, etc. are, by contrast, all **recursive** or **nested** data structure Now, you can put tabular data into these formats -- since list-of-key-value-pairs is one of the things representable in XML or JSON. Example: -.. code-block:: bash +.. code-block:: none # DKVP x=1,y=2 z=3 -.. code-block:: bash +.. code-block:: none # XML @@ -303,7 +303,7 @@ Now, you can put tabular data into these formats -- since list-of-key-value-pair
-.. code-block:: bash +.. code-block:: none # JSON [{"x":1,"y":2},{"z":3}] diff --git a/docs/feature-comparison.rst b/docs/feature-comparison.rst index 6dc8b9d73..4d4871479 100644 --- a/docs/feature-comparison.rst +++ b/docs/feature-comparison.rst @@ -11,7 +11,7 @@ File-format awareness Miller respects CSV headers. If you do ``mlr --csv cat *.csv`` then the header line is written once: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/a.csv @@ -19,14 +19,14 @@ Miller respects CSV headers. If you do ``mlr --csv cat *.csv`` then the header l 1,2,3 4,5,6 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/b.csv a,b,c 7,8,9 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv cat data/a.csv data/b.csv @@ -35,7 +35,7 @@ Miller respects CSV headers. If you do ``mlr --csv cat *.csv`` then the header l 4,5,6 7,8,9 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv sort -nr b data/a.csv data/b.csv diff --git a/docs/file-formats.rst b/docs/file-formats.rst index e5d710bc5..ea551ef8a 100644 --- a/docs/file-formats.rst +++ b/docs/file-formats.rst @@ -9,7 +9,7 @@ Miller handles name-indexed data using several formats: some you probably know b Examples ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --usage-data-format-examples @@ -118,7 +118,7 @@ DKVP: Key-value pairs Miller's default file format is DKVP, for **delimited key-value pairs**. Example: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr cat data/small @@ -130,21 +130,21 @@ Miller's default file format is DKVP, for **delimited key-value pairs**. Example Such data are easy to generate, e.g. in Ruby with -.. code-block:: bash +.. code-block:: none puts "host=#{hostname},seconds=#{t2-t1},message=#{msg}" -.. code-block:: bash +.. code-block:: none puts mymap.collect{|k,v| "#{k}=#{v}"}.join(',') or ``print`` statements in various languages, e.g. -.. code-block:: bash +.. code-block:: none echo "type=3,user=$USER,date=$date\n"; -.. code-block:: bash +.. code-block:: none logger.log("type=3,user=$USER,date=$date\n"); @@ -152,7 +152,7 @@ Fields lacking an IPS will have positional index (starting at 1) used as the key As discussed in :doc:`record-heterogeneity`, Miller handles changes of field names within the same data stream. But using DKVP format this is particularly natural. One of my favorite use-cases for Miller is in application/server logs, where I log all sorts of lines such as -.. code-block:: bash +.. code-block:: none resource=/path/to/file,loadsec=0.45,ok=true record_count=100, resource=/path/to/file @@ -172,7 +172,7 @@ With ``--inidx --ifs ' ' --repifs``, Miller splits lines on whitespace and assig Example with index-numbered output: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/small @@ -182,7 +182,7 @@ Example with index-numbered output: a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463 a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --onidx --ofs ' ' cat data/small @@ -194,7 +194,7 @@ Example with index-numbered output: Example with index-numbered input: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/mydata.txt @@ -202,7 +202,7 @@ Example with index-numbered input: by the dawn's early light -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --inidx --ifs ' ' --odkvp cat data/mydata.txt @@ -212,7 +212,7 @@ Example with index-numbered input: Example with index-numbered input and output: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/mydata.txt @@ -220,7 +220,7 @@ Example with index-numbered input and output: by the dawn's early light -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --nidx --fs ' ' --repifs cut -f 2,3 data/mydata.txt @@ -242,14 +242,14 @@ Single-level JSON objects An **array of single-level objects** is, quite simply, **a table**: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --json head -n 2 then cut -f color,shape data/json-example-1.json { "color": "yellow", "shape": "triangle" } { "color": "red", "shape": "square" } -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --json --jvstack head -n 2 then cut -f color,u,v data/json-example-1.json @@ -264,7 +264,7 @@ An **array of single-level objects** is, quite simply, **a table**: "v": 0.001257332190235938 } -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --ijson --opprint stats1 -a mean,stddev,count -f u -g shape data/json-example-1.json @@ -278,7 +278,7 @@ Nested JSON objects Additionally, Miller can **tabularize nested objects by concatentating keys**: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --json --jvstack head -n 2 data/json-example-2.json @@ -311,7 +311,7 @@ Additionally, Miller can **tabularize nested objects by concatentating keys**: } } -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --ijson --opprint head -n 4 data/json-example-2.json @@ -323,7 +323,7 @@ Additionally, Miller can **tabularize nested objects by concatentating keys**: Note in particular that as far as Miller's ``put`` and ``filter``, as well as other I/O formats, are concerned, these are simply field names with colons in them: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --json --jvstack head -n 1 then put '${values:uv} = ${values:u} * ${values:v}' data/json-example-2.json @@ -350,7 +350,7 @@ Arrays aren't supported in Miller's ``put``/``filter`` DSL. By default, JSON arr Suppose we have arrays like this in our input data: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/json-example-3.json @@ -365,7 +365,7 @@ Suppose we have arrays like this in our input data: Then integer indices (starting from 0 and counting up) are used as map keys: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --ijson --oxtab cat data/json-example-3.json @@ -380,7 +380,7 @@ Then integer indices (starting from 0 and counting up) are used as map keys: When the data are written back out as JSON, field names are re-expanded as above, but what were arrays on input are now maps on output: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --json --jvstack cat data/json-example-3.json @@ -437,7 +437,7 @@ PPRINT: Pretty-printed tabular Miller's pretty-print format is like CSV, but column-aligned. For example, compare -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --ocsv cat data/small @@ -448,7 +448,7 @@ Miller's pretty-print format is like CSV, but column-aligned. For example, comp eks,wye,4,0.38139939387114097,0.13418874328430463 wye,pan,5,0.5732889198020006,0.8636244699032729 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint cat data/small @@ -465,7 +465,7 @@ See :doc:`record-heterogeneity` for how Miller handles changes of field names wi For output only (this isn't supported in the input-scanner as of 5.0.0) you can use ``--barred`` with pprint output format: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint --barred cat data/small @@ -487,7 +487,7 @@ XTAB: Vertical tabular This is perhaps most useful for looking a very wide and/or multi-column data which causes line-wraps on the screen (but see also `ngrid `_ for an entirely different, very powerful option). Namely: -.. code-block:: bash +.. code-block:: none $ grep -v '^#' /etc/passwd | head -n 6 | mlr --nidx --fs : --opprint cat 1 2 3 4 5 6 7 @@ -498,7 +498,7 @@ This is perhaps most useful for looking a very wide and/or multi-column data whi _taskgated * 13 13 Task Gate Daemon /var/empty /usr/bin/false _networkd * 24 24 Network Services /var/networkd /usr/bin/false -.. code-block:: bash +.. code-block:: none $ grep -v '^#' /etc/passwd | head -n 2 | mlr --nidx --fs : --oxtab cat 1 nobody @@ -517,7 +517,7 @@ This is perhaps most useful for looking a very wide and/or multi-column data whi 6 /var/root 7 /bin/sh -.. code-block:: bash +.. code-block:: none $ grep -v '^#' /etc/passwd | head -n 2 | \ mlr --nidx --fs : --ojson --jvstack --jlistwrap label name,password,uid,gid,gecos,home_dir,shell @@ -547,7 +547,7 @@ Markdown tabular Markdown format looks like this: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --omd cat data/small @@ -570,7 +570,7 @@ Data-conversion keystroke-savers While you can do format conversion using ``mlr --icsv --ojson cat myfile.csv``, there are also keystroke-savers for this purpose, such as ``mlr --c2j cat myfile.csv``. For a complete list: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --usage-format-conversion-keystroke-saver-options @@ -606,7 +606,7 @@ Comments in data You can include comments within your data files, and either have them ignored, or passed directly through to the standard output as soon as they are encountered: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --usage-comments-in-data @@ -629,7 +629,7 @@ You can include comments within your data files, and either have them ignored, o Examples: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/budget.csv @@ -639,7 +639,7 @@ Examples: green,678.12 orange,123.45 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --skip-comments --icsv --opprint sort -nr quantity data/budget.csv @@ -648,7 +648,7 @@ Examples: purple 456.78 orange 123.45 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --pass-comments --icsv --opprint sort -nr quantity data/budget.csv diff --git a/docs/file-formats.rst.in b/docs/file-formats.rst.in index acccbf63d..fb6c89652 100644 --- a/docs/file-formats.rst.in +++ b/docs/file-formats.rst.in @@ -57,21 +57,21 @@ POKI_RUN_COMMAND{{mlr cat data/small}}HERE Such data are easy to generate, e.g. in Ruby with -.. code-block:: bash +.. code-block:: none puts "host=#{hostname},seconds=#{t2-t1},message=#{msg}" -.. code-block:: bash +.. code-block:: none puts mymap.collect{|k,v| "#{k}=#{v}"}.join(',') or ``print`` statements in various languages, e.g. -.. code-block:: bash +.. code-block:: none echo "type=3,user=$USER,date=$date\n"; -.. code-block:: bash +.. code-block:: none logger.log("type=3,user=$USER,date=$date\n"); @@ -79,7 +79,7 @@ Fields lacking an IPS will have positional index (starting at 1) used as the key As discussed in :doc:`record-heterogeneity`, Miller handles changes of field names within the same data stream. But using DKVP format this is particularly natural. One of my favorite use-cases for Miller is in application/server logs, where I log all sorts of lines such as -.. code-block:: bash +.. code-block:: none resource=/path/to/file,loadsec=0.45,ok=true record_count=100, resource=/path/to/file diff --git a/docs/install.rst b/docs/install.rst index 73fac184e..13eacfda7 100644 --- a/docs/install.rst +++ b/docs/install.rst @@ -9,38 +9,38 @@ Prebuilt executables via package managers `Homebrew `_ installation support for OSX is available via -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 brew update && brew install miller ...and also via `MacPorts `_: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 sudo port selfupdate && sudo port install miller You may already have the ``mlr`` executable available in your platform's package manager on NetBSD, Debian Linux, Ubuntu Xenial and upward, Arch Linux, or perhaps other distributions. For example, on various Linux distributions you might do one of the following: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 sudo apt-get install miller -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 sudo apt install miller -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 sudo yum install miller On Windows, Miller is available via `Chocolatey `_: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 choco install miller diff --git a/docs/install.rst.in b/docs/install.rst.in index 19870111b..ccc87d82e 100644 --- a/docs/install.rst.in +++ b/docs/install.rst.in @@ -6,38 +6,38 @@ Prebuilt executables via package managers `Homebrew `_ installation support for OSX is available via -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 brew update && brew install miller ...and also via `MacPorts `_: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 sudo port selfupdate && sudo port install miller You may already have the ``mlr`` executable available in your platform's package manager on NetBSD, Debian Linux, Ubuntu Xenial and upward, Arch Linux, or perhaps other distributions. For example, on various Linux distributions you might do one of the following: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 sudo apt-get install miller -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 sudo apt install miller -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 sudo yum install miller On Windows, Miller is available via `Chocolatey `_: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 choco install miller diff --git a/docs/log-processing-examples.rst b/docs/log-processing-examples.rst index 180282a37..3550edabb 100644 --- a/docs/log-processing-examples.rst +++ b/docs/log-processing-examples.rst @@ -12,7 +12,7 @@ Writing a program -- in any language whatsoever -- you can have it print out log Suppose your program has printed something like this: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat log.txt @@ -62,7 +62,7 @@ Suppose your program has printed something like this: Each print statement simply contains local information: the current timestamp, whether a particular cache was hit or not, etc. Then using either the system ``grep`` command, or Miller's ``having-fields``, or ``is_present``, we can pick out the parts we want and analyze them: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ grep op=cache log.txt \ @@ -72,7 +72,7 @@ Each print statement simply contains local information: the current timestamp, w A4 0.714286 A9 0.090909 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from log.txt --opprint \ @@ -89,7 +89,7 @@ Each print statement simply contains local information: the current timestamp, w Alternatively, we can simply group the similar data for a better look: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint group-like log.txt @@ -142,7 +142,7 @@ Alternatively, we can simply group the similar data for a better look: 1472819736 100 612 1472819742 100 728 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint group-like then sec2gmt time log.txt diff --git a/docs/manpage.rst b/docs/manpage.rst index 3dc89a3fd..34d6fa074 100644 --- a/docs/manpage.rst +++ b/docs/manpage.rst @@ -6,7 +6,7 @@ Manpage This is simply a copy of what you should see on running **man mlr** at a command prompt, once Miller is installed on your system. -.. code-block:: bash +.. code-block:: none MILLER(1) MILLER(1) diff --git a/docs/poki b/docs/poki index e155c3fed..ba8626677 100755 --- a/docs/poki +++ b/docs/poki @@ -90,7 +90,7 @@ end # ---------------------------------------------------------------- def write_card(highlight_first_line, content_lines, output_handle) - output_handle.puts('.. code-block:: bash') + output_handle.puts('.. code-block:: none') if highlight_first_line output_handle.puts(' :emphasize-lines: 1,1') # 'hll' in _static/*.css end diff --git a/docs/quick-examples.rst b/docs/quick-examples.rst index b66ab711b..a1f4c1068 100644 --- a/docs/quick-examples.rst +++ b/docs/quick-examples.rst @@ -6,70 +6,70 @@ Quick examples Column select: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 % mlr --csv cut -f hostname,uptime mydata.csv Add new columns as function of other columns: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 % mlr --nidx put '$sum = $7 < 0.0 ? 3.5 : $7 + 2.1*$8' *.dat Row filter: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 % mlr --csv filter '$status != "down" && $upsec >= 10000' *.csv Apply column labels and pretty-print: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 % grep -v '^#' /etc/group | mlr --ifs : --nidx --opprint label group,pass,gid,member then sort -f group Join multiple data sources on key columns: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 % mlr join -j account_id -f accounts.dat then group-by account_name balances.dat Multiple formats including JSON: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 % mlr --json put '$attr = sub($attr, "([0-9]+)_([0-9]+)_.*", "\1:\2")' data/*.json Aggregate per-column statistics: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 % mlr stats1 -a min,mean,max,p10,p50,p90 -f flag,u,v data/* Linear regression: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 % mlr stats2 -a linreg-pca -f u,v -g shape data/* Aggregate custom per-column statistics: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 % mlr put -q '@sum[$a][$b] += $x; end {emit @sum, "a", "b"}' data/* Iterate over data using DSL expressions: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 % mlr --from estimates.tbl put ' @@ -83,35 +83,35 @@ Iterate over data using DSL expressions: Run DSL expressions from a script file: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 % mlr --from infile.dat put -f analyze.mlr Split/reduce output to multiple filenames: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 % mlr --from infile.dat put 'tee > "./taps/data-".$a."-".$b, $*' Compressed I/O: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 % mlr --from infile.dat put 'tee | "gzip > ./taps/data-".$a."-".$b.".gz", $*' Interoperate with other data-processing tools using standard pipes: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 % mlr --from infile.dat put -q '@v=$*; dump | "jq .[]"' Tap/trace: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 % mlr --from infile.dat put '(NR % 1000 == 0) { print > stderr, "Checkpoint ".NR}' diff --git a/docs/quick-examples.rst.in b/docs/quick-examples.rst.in index 93c23050d..4ae54d30d 100644 --- a/docs/quick-examples.rst.in +++ b/docs/quick-examples.rst.in @@ -3,70 +3,70 @@ Quick examples Column select: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 % mlr --csv cut -f hostname,uptime mydata.csv Add new columns as function of other columns: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 % mlr --nidx put '$sum = $7 < 0.0 ? 3.5 : $7 + 2.1*$8' *.dat Row filter: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 % mlr --csv filter '$status != "down" && $upsec >= 10000' *.csv Apply column labels and pretty-print: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 % grep -v '^#' /etc/group | mlr --ifs : --nidx --opprint label group,pass,gid,member then sort -f group Join multiple data sources on key columns: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 % mlr join -j account_id -f accounts.dat then group-by account_name balances.dat Multiple formats including JSON: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 % mlr --json put '$attr = sub($attr, "([0-9]+)_([0-9]+)_.*", "\1:\2")' data/*.json Aggregate per-column statistics: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 % mlr stats1 -a min,mean,max,p10,p50,p90 -f flag,u,v data/* Linear regression: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 % mlr stats2 -a linreg-pca -f u,v -g shape data/* Aggregate custom per-column statistics: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 % mlr put -q '@sum[$a][$b] += $x; end {emit @sum, "a", "b"}' data/* Iterate over data using DSL expressions: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 % mlr --from estimates.tbl put ' @@ -80,35 +80,35 @@ Iterate over data using DSL expressions: Run DSL expressions from a script file: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 % mlr --from infile.dat put -f analyze.mlr Split/reduce output to multiple filenames: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 % mlr --from infile.dat put 'tee > "./taps/data-".$a."-".$b, $*' Compressed I/O: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 % mlr --from infile.dat put 'tee | "gzip > ./taps/data-".$a."-".$b.".gz", $*' Interoperate with other data-processing tools using standard pipes: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 % mlr --from infile.dat put -q '@v=$*; dump | "jq .[]"' Tap/trace: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 % mlr --from infile.dat put '(NR % 1000 == 0) { print > stderr, "Checkpoint ".NR}' diff --git a/docs/record-heterogeneity.rst b/docs/record-heterogeneity.rst index b7bc72d19..757488b54 100644 --- a/docs/record-heterogeneity.rst +++ b/docs/record-heterogeneity.rst @@ -16,7 +16,7 @@ CSV and pretty-print Miller simply prints a newline and a new header when there is a schema change. When there is no schema change, you get CSV per se as a special case. Likewise, Miller reads heterogeneous CSV or pretty-print input the same way. The difference between CSV and CSV-lite is that the former is RFC4180-compliant, while the latter readily handles heterogeneous data (which is non-compliant). For example: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/het.dkvp @@ -26,7 +26,7 @@ Miller simply prints a newline and a new header when there is a schema change. W record_count=150,resource=/path/to/second/file resource=/some/other/path,loadsec=0.97,ok=false -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --ocsvlite cat data/het.dkvp @@ -45,7 +45,7 @@ Miller simply prints a newline and a new header when there is a schema change. W resource,loadsec,ok /some/other/path,0.97,false -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint cat data/het.dkvp @@ -66,7 +66,7 @@ Miller simply prints a newline and a new header when there is a schema change. W Miller handles explicit header changes as just shown. If your CSV input contains ragged data -- if there are implicit header changes -- you can use ``--allow-ragged-csv-input`` (or keystroke-saver ``--ragged``). For too-short data lines, values are filled with empty string; for too-long data lines, missing field names are replaced with positional indices (counting up from 1, not 0), as follows: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/ragged.csv @@ -75,7 +75,7 @@ Miller handles explicit header changes as just shown. If your CSV input contains 4,5 6,7,8,9 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --oxtab --allow-ragged-csv-input cat data/ragged.csv @@ -94,7 +94,7 @@ Miller handles explicit header changes as just shown. If your CSV input contains You may also find Miller's ``group-like`` feature handy (see also :doc:`reference`): -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --ocsvlite group-like data/het.dkvp @@ -107,7 +107,7 @@ You may also find Miller's ``group-like`` feature handy (see also :doc:`referenc 100,/path/to/file 150,/path/to/second/file -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint group-like data/het.dkvp @@ -125,7 +125,7 @@ Key-value-pair, vertical-tabular, and index-numbered formats For these formats, record-heterogeneity comes naturally: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/het.dkvp @@ -135,7 +135,7 @@ For these formats, record-heterogeneity comes naturally: record_count=150,resource=/path/to/second/file resource=/some/other/path,loadsec=0.97,ok=false -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --onidx --ofs ' ' cat data/het.dkvp @@ -145,7 +145,7 @@ For these formats, record-heterogeneity comes naturally: 150 /path/to/second/file /some/other/path 0.97 false -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --oxtab cat data/het.dkvp @@ -167,7 +167,7 @@ For these formats, record-heterogeneity comes naturally: loadsec 0.97 ok false -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --oxtab group-like data/het.dkvp @@ -194,7 +194,7 @@ For processing Miller operates on specified fields and takes the rest along: for example, if you are sorting on the ``count`` field then all records in the input stream must have a ``count`` field but the other fields can vary, and moreover the sorted-on field name(s) don't need to be in the same position on each line: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/sort-het.dkvp @@ -206,7 +206,7 @@ Miller operates on specified fields and takes the rest along: for example, if yo count=100,color=green count=450 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr sort -n count data/sort-het.dkvp diff --git a/docs/reference-dsl.rst b/docs/reference-dsl.rst index 9b33ba763..baf765c07 100644 --- a/docs/reference-dsl.rst +++ b/docs/reference-dsl.rst @@ -11,7 +11,7 @@ Here's comparison of verbs and ``put``/``filter`` DSL expressions: Example: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr stats1 -a sum -f x -g a data/small @@ -27,7 +27,7 @@ Example: Example: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q '@x_sum[$a] += $x; end{emit @x_sum, "a"}' data/small @@ -45,7 +45,7 @@ Please see :doc:`reference-verbs` for information on verbs other than ``put`` an The essential usages of ``mlr filter`` and ``mlr put`` are for record-selection and record-updating expressions, respectively. For example, given the following input data: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/small @@ -57,7 +57,7 @@ The essential usages of ``mlr filter`` and ``mlr put`` are for record-selection you might retain only the records whose ``a`` field has value ``eks``: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr filter '$a == "eks"' data/small @@ -66,7 +66,7 @@ you might retain only the records whose ``a`` field has value ``eks``: or you might add a new field which is a function of existing fields: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put '$ab = $a . "_" . $b ' data/small @@ -96,7 +96,7 @@ Expression formatting Multiple expressions may be given, separated by semicolons, and each may refer to the ones before: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ ruby -e '10.times{|i|puts "i=#{i}"}' | mlr --opprint put '$j = $i + 1; $k = $i +$j' @@ -114,7 +114,7 @@ Multiple expressions may be given, separated by semicolons, and each may refer t Newlines within the expression are ignored, which can help increase legibility of complex expressions: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint put ' @@ -136,7 +136,7 @@ Newlines within the expression are ignored, which can help increase legibility o hat wye 10002 0.321507044286237609 0.568893318795083758 5 9 4 2 data/small2 pan zee 10003 0.272054845593895200 0.425789896597056627 5 10 5 2 data/small2 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint filter '($x > 0.5 && $y < 0.5) || ($x < 0.5 && $y > 0.5)' then stats2 -a corr -f x,y data/medium @@ -150,7 +150,7 @@ Expressions from files The simplest way to enter expressions for ``put`` and ``filter`` is between single quotes on the command line, e.g. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/small put '$xy = sqrt($x**2 + $y**2)' @@ -160,7 +160,7 @@ The simplest way to enter expressions for ``put`` and ``filter`` is between sing a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463,xy=0.404317 a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729,xy=1.036584 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/small put 'func f(a, b) { return sqrt(a**2 + b**2) } $xy = f($x, $y)' @@ -173,7 +173,7 @@ The simplest way to enter expressions for ``put`` and ``filter`` is between sing You may, though, find it convenient to put expressions into files for reuse, and read them **using the -f option**. For example: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/fe-example-3.mlr @@ -182,7 +182,7 @@ You may, though, find it convenient to put expressions into files for reuse, and } $xy = f($x, $y) -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/small put -f data/fe-example-3.mlr @@ -194,7 +194,7 @@ You may, though, find it convenient to put expressions into files for reuse, and If you have some of the logic in a file and you want to write the rest on the command line, you can **use the -f and -e options together**: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/fe-example-4.mlr @@ -202,7 +202,7 @@ If you have some of the logic in a file and you want to write the rest on the co return sqrt(a**2 + b**2) } -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/small put -f data/fe-example-4.mlr -e '$xy = f($x, $y)' @@ -223,7 +223,7 @@ Semicolons, commas, newlines, and curly braces Miller uses **semicolons as statement separators**, not statement terminators. This means you can write: -.. code-block:: bash +.. code-block:: none mlr put 'x=1' mlr put 'x=1;$y=2' @@ -232,13 +232,13 @@ Miller uses **semicolons as statement separators**, not statement terminators. T Semicolons are optional after closing curly braces (which close conditionals and loops as discussed below). -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ echo x=1,y=2 | mlr put 'while (NF < 10) { $[NF+1] = ""} $foo = "bar"' x=1,y=2,3=,4=,5=,6=,7=,8=,9=,10=,foo=bar -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ echo x=1,y=2 | mlr put 'while (NF < 10) { $[NF+1] = ""}; $foo = "bar"' @@ -246,7 +246,7 @@ Semicolons are optional after closing curly braces (which close conditionals and Semicolons are required between statements even if those statements are on separate lines. **Newlines** are for your convenience but have no syntactic meaning: line endings do not terminate statements. For example, adjacent assignment statements must be separated by semicolons even if those statements are on separate lines: -.. code-block:: bash +.. code-block:: none mlr put ' $x = 1 @@ -260,7 +260,7 @@ Semicolons are required between statements even if those statements are on separ **Trailing commas** are allowed in function/subroutine definitions, function/subroutine callsites, and map literals. This is intended for (although not restricted to) the multi-line case: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csvlite --from data/a.csv put ' @@ -286,17 +286,17 @@ Semicolons are required between statements even if those statements are on separ Bodies for all compound statements must be enclosed in **curly braces**, even if the body is a single statement: -.. code-block:: bash +.. code-block:: none mlr put 'if ($x == 1) $y = 2' # Syntax error -.. code-block:: bash +.. code-block:: none mlr put 'if ($x == 1) { $y = 2 }' # This is OK Bodies for compound statements may be empty: -.. code-block:: bash +.. code-block:: none mlr put 'if ($x == 1) { }' # This no-op is syntactically acceptable @@ -322,7 +322,7 @@ These are written all in capital letters, such as ``NR``, ``NF``, ``FILENAME``, Namely, Miller supports the following five built-in variables for :doc:`filter and put `, all ``awk``-inspired: ``NF``, ``NR``, ``FNR``, ``FILENUM``, and ``FILENAME``, as well as the mathematical constants ``M_PI`` and ``M_E``. Lastly, the ``ENV`` hashmap allows read access to environment variables, e.g. ``ENV["HOME"]`` or ``ENV["foo_".$hostname]``. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr filter 'FNR == 2' data/small* @@ -330,7 +330,7 @@ Namely, Miller supports the following five built-in variables for :doc:`filter a 1=pan,2=pan,3=1,4=0.3467901443380824,5=0.7268028627434533 a=wye,b=eks,i=10000,x=0.734806020620654365,y=0.884788571337605134 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put '$fnr = FNR' data/small* @@ -355,7 +355,7 @@ Their values of ``NF``, ``NR``, ``FNR``, ``FILENUM``, and ``FILENAME`` change fr Their **scope is global**: you can refer to them in any ``filter`` or ``put`` statement. Their values are assigned by the input-record reader: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv put '$nr = NR' data/a.csv @@ -363,7 +363,7 @@ Their **scope is global**: you can refer to them in any ``filter`` or ``put`` st 1,2,3,1 4,5,6,2 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv repeat -n 3 then put '$nr = NR' data/a.csv @@ -388,12 +388,12 @@ If field names have **special characters** such as ``.`` then you can use braces You may also use a **computed field name** in square brackets, e.g. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ echo a=3,b=4 | mlr filter '$["x"] < 0.5' -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ echo s=green,t=blue,a=3,b=4 | mlr put '$[$s."_".$t] = $a * $b' @@ -418,7 +418,7 @@ Use ``$[[3]]`` to access the name of field 3. More generally, any expression ev Then using a computed field name, ``$[ $[[3]] ]`` is the value in the third field. This has the shorter equivalent notation ``$[[[3]]]``. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr cat data/small @@ -428,7 +428,7 @@ Then using a computed field name, ``$[ $[[3]] ]`` is the value in the third fiel a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463 a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put '$[[3]] = "NEW"' data/small @@ -438,7 +438,7 @@ Then using a computed field name, ``$[ $[[3]] ]`` is the value in the third fiel a=eks,b=wye,NEW=4,x=0.38139939387114097,y=0.13418874328430463 a=wye,b=pan,NEW=5,x=0.5732889198020006,y=0.8636244699032729 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put '$[[[3]]] = "NEW"' data/small @@ -448,7 +448,7 @@ Then using a computed field name, ``$[ $[[3]] ]`` is the value in the third fiel a=eks,b=wye,i=NEW,x=0.38139939387114097,y=0.13418874328430463 a=wye,b=pan,i=NEW,x=0.5732889198020006,y=0.8636244699032729 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put '$NEW = $[[NR]]' data/small @@ -458,7 +458,7 @@ Then using a computed field name, ``$[ $[[3]] ]`` is the value in the third fiel a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463,NEW=x a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729,NEW=y -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put '$NEW = $[[[NR]]]' data/small @@ -468,7 +468,7 @@ Then using a computed field name, ``$[ $[[3]] ]`` is the value in the third fiel a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463,NEW=0.381399 a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729,NEW=0.863624 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put '$[[[NR]]] = "NEW"' data/small @@ -480,7 +480,7 @@ Then using a computed field name, ``$[ $[[3]] ]`` is the value in the third fiel Right-hand side accesses to non-existent fields -- i.e. with index less than 1 or greater than ``NF`` -- return an absent value. Likewise, left-hand side accesses only refer to fields which already exist. For example, if a field has 5 records then assigning the name or value of the 6th (or 600th) field results in a no-op. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put '$[[6]] = "NEW"' data/small @@ -490,7 +490,7 @@ Right-hand side accesses to non-existent fields -- i.e. with index less than 1 o a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463 a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put '$[[[6]]] = "NEW"' data/small @@ -511,7 +511,7 @@ Just as for field names in stream records, if you want to define out-of-stream v You may use a **computed key** in square brackets, e.g. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ echo s=green,t=blue,a=3,b=4 | mlr put -q '@[$s."_".$t] = $a * $b; emit all' @@ -519,14 +519,14 @@ You may use a **computed key** in square brackets, e.g. Out-of-stream variables are **scoped** to the ``put`` command in which they appear. In particular, if you have two or more ``put`` commands separated by ``then``, each put will have its own set of out-of-stream variables: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/a.dkvp a=1,b=2,c=3 a=4,b=5,c=6 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put '@sum += $a; end {emit @sum}' then put 'is_present($a) {$a=10*$a; @sum += $a}; end {emit @sum}' data/a.dkvp @@ -544,7 +544,7 @@ Indexed out-of-stream variables Using an index on the ``@count`` and ``@sum`` variables, we get the benefit of the ``-g`` (group-by) option which ``mlr stats1`` and various other Miller commands have: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q ' @@ -566,7 +566,7 @@ Using an index on the ``@count`` and ``@sum`` variables, we get the benefit of t a=zee,x_sum=1.125680 a=hat,x_sum=0.031442 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr stats1 -a count,sum -f x -g a ../data/small @@ -578,7 +578,7 @@ Using an index on the ``@count`` and ``@sum`` variables, we get the benefit of t Indices can be arbitrarily deep -- here there are two or more of them: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/medium put -q ' @@ -618,7 +618,7 @@ The idea is that ``stats1``, and other Miller verbs, encapsulate frequently-used Begin/end blocks can be mixed with pattern/action blocks. For example: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put ' @@ -651,7 +651,7 @@ Local variables are similar to out-of-stream variables, except that their extent For example: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ # Here I'm using a specified random-number seed so this example always @@ -707,7 +707,7 @@ Things which are perhaps surprising compared to other languages: The following example demonstrates the scope rules: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/scope-example.mlr @@ -735,7 +735,7 @@ The following example demonstrates the scope rules: $outer_c = c; $outer_d = d; # there is no outer d defined so no assignment happens -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/scope-example.dat @@ -743,7 +743,7 @@ The following example demonstrates the scope rules: n=2,x=456 n=3,x=789 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --oxtab --from data/scope-example.dat put -f data/scope-example.mlr @@ -771,7 +771,7 @@ The following example demonstrates the scope rules: And this example demonstrates the type-declaration rules: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/type-decl-example.mlr @@ -820,7 +820,7 @@ Miller's ``put``/``filter`` DSL has four kinds of hashmaps. **Stream records** a For example, the following swaps the input stream's ``a`` and ``i`` fields, modifies ``y``, and drops the rest: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint put ' @@ -839,7 +839,7 @@ For example, the following swaps the input stream's ``a`` and ``i`` fields, modi Likewise, you can assign map literals to out-of-stream variables or local variables; pass them as arguments to user-defined functions, return them from functions, and so on: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/small put ' @@ -857,7 +857,7 @@ Likewise, you can assign map literals to out-of-stream variables or local variab Like out-of-stream and local variables, map literals can be multi-level: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/small put -q ' @@ -905,7 +905,7 @@ Type-test and type-assertion expressions The following ``is...`` functions take a value and return a boolean indicating whether the argument is of the indicated type. The ``assert_...`` functions return their argument if it is of the specified type, and cause a fatal error otherwise: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr -F | grep ^is @@ -926,7 +926,7 @@ The following ``is...`` functions take a value and return a boolean indicating w is_present is_string -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr -F | grep ^assert @@ -956,7 +956,7 @@ Local variables can be defined either untyped as in ``x = 1``, or typed as in `` The reason for ``num`` is that ``int`` and ``float`` typedecls are very precise: -.. code-block:: bash +.. code-block:: none float a = 0; # Runtime error since 0 is int not float int b = 1.0; # Runtime error since 1.0 is float not int @@ -967,7 +967,7 @@ A suggestion is to use ``num`` for general use when you want numeric content, an The ``var`` type declaration indicates no type restrictions, e.g. ``var x = 1`` has the same type restrictions on ``x`` as ``x = 1``. The difference is in intentional shadowing: if you have ``x = 1`` in outer scope and ``x = 2`` in inner scope (e.g. within a for-loop or an if-statement) then outer-scope ``x`` has value 2 after the second assignment. But if you have ``var x = 2`` in the inner scope, then you are declaring a variable scoped to the inner block.) For example: -.. code-block:: bash +.. code-block:: none x = 1; if (NR == 4) { @@ -975,7 +975,7 @@ The ``var`` type declaration indicates no type restrictions, e.g. ``var x = 1`` } print x; # Value of x is now two -.. code-block:: bash +.. code-block:: none x = 1; if (NR == 4) { @@ -985,7 +985,7 @@ The ``var`` type declaration indicates no type restrictions, e.g. ``var x = 1`` Likewise function arguments can optionally be typed, with type enforced when the function is called: -.. code-block:: bash +.. code-block:: none func f(map m, int i) { ... @@ -1000,7 +1000,7 @@ Likewise function arguments can optionally be typed, with type enforced when the Thirdly, function return values can be type-checked at the point of ``return`` using ``:`` and a typedecl after the parameter list: -.. code-block:: bash +.. code-block:: none func f(map m, int i): bool { ... @@ -1037,7 +1037,7 @@ There are three remaining kinds of variable assignment using out-of-stream varia Example recursive copy of out-of-stream variables: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint put -q '@v["sum"] += $x; @v["count"] += 1; end{dump; @w = @v; dump}' data/small @@ -1060,7 +1060,7 @@ Example recursive copy of out-of-stream variables: Example of out-of-stream variable assigned to full stream record, where the 2nd record is stashed, and the 4th record is overwritten with that: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put 'NR == 2 {@keep = $*}; NR == 4 {$* = @keep}' data/small @@ -1072,7 +1072,7 @@ Example of out-of-stream variable assigned to full stream record, where the 2nd Example of full stream record assigned to an out-of-stream variable, finding the record for which the ``x`` field has the largest value in the input stream: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/small @@ -1082,7 +1082,7 @@ Example of full stream record assigned to an out-of-stream variable, finding the a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463 a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint put -q 'is_null(@xmax) || $x > @xmax {@xmax=$x; @recmax=$*}; end {emit @recmax}' data/small @@ -1092,7 +1092,7 @@ Example of full stream record assigned to an out-of-stream variable, finding the Keywords for filter and put ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --help-all-keywords @@ -1395,7 +1395,7 @@ Operator precedence Operators are listed in order of decreasing precedence, highest first. -.. code-block:: bash +.. code-block:: none Operators Associativity --------- ------------- @@ -1437,7 +1437,7 @@ Pattern-action blocks These are reminiscent of ``awk`` syntax. They can be used to allow assignments to be done only when appropriate -- e.g. for math-function domain restrictions, regex-matching, and so on: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr cat data/put-gating-example-1.dkvp @@ -1447,7 +1447,7 @@ These are reminiscent of ``awk`` syntax. They can be used to allow assignments x=2 x=3 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put '$x > 0.0 { $y = log10($x); $z = sqrt($y) }' data/put-gating-example-1.dkvp @@ -1457,7 +1457,7 @@ These are reminiscent of ``awk`` syntax. They can be used to allow assignments x=2,y=0.301030,z=0.548662 x=3,y=0.477121,z=0.690740 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr cat data/put-gating-example-2.dkvp @@ -1465,7 +1465,7 @@ These are reminiscent of ``awk`` syntax. They can be used to allow assignments a=some other name a=xyz_789 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put '$a =~ "([a-z]+)_([0-9]+)" { $b = "left_\1"; $c = "right_\2" }' data/put-gating-example-2.dkvp @@ -1475,7 +1475,7 @@ These are reminiscent of ``awk`` syntax. They can be used to allow assignments This produces heteregenous output which Miller, of course, has no problems with (see :doc:`record-heterogeneity`). But if you want homogeneous output, the curly braces can be replaced with a semicolon between the expression and the body statements. This causes ``put`` to evaluate the boolean expression (along with any side effects, namely, regex-captures ``\1``, ``\2``, etc.) but doesn't use it as a criterion for whether subsequent assignments should be executed. Instead, subsequent assignments are done unconditionally: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put '$x > 0.0; $y = log10($x); $z = sqrt($y)' data/put-gating-example-1.dkvp @@ -1485,7 +1485,7 @@ This produces heteregenous output which Miller, of course, has no problems with x=2,y=0.301030,z=0.548662 x=3,y=0.477121,z=0.690740 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put '$a =~ "([a-z]+)_([0-9]+)"; $b = "left_\1"; $c = "right_\2"' data/put-gating-example-2.dkvp @@ -1498,17 +1498,17 @@ If-statements These are again reminiscent of ``awk``. Pattern-action blocks are a special case of ``if`` with no ``elif`` or ``else`` blocks, no ``if`` keyword, and parentheses optional around the boolean expression: -.. code-block:: bash +.. code-block:: none mlr put 'NR == 4 {$foo = "bar"}' -.. code-block:: bash +.. code-block:: none mlr put 'if (NR == 4) {$foo = "bar"}' Compound statements use ``elif`` (rather than ``elsif`` or ``else if``): -.. code-block:: bash +.. code-block:: none mlr put ' if (NR == 2) { @@ -1527,7 +1527,7 @@ While and do-while loops Miller's ``while`` and ``do-while`` are unsurprising in comparison to various languages, as are ``break`` and ``continue``: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ echo x=1,y=2 | mlr put ' @@ -1538,7 +1538,7 @@ Miller's ``while`` and ``do-while`` are unsurprising in comparison to various la ' x=1,y=2,3=,4=,5=,6=,7=,8=,9=,10=,foo=bar -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ echo x=1,y=2 | mlr put ' @@ -1567,7 +1567,7 @@ Key-only for-loops The ``key`` variable is always bound to the *key* of key-value pairs: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/small put ' @@ -1614,7 +1614,7 @@ The ``key`` variable is always bound to the *key* of key-value pairs: key:y value:0.863624 a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr -n put ' @@ -1635,7 +1635,7 @@ Key-value for-loops Single-level keys may be gotten at using either ``for(k,v)`` or ``for((k),v)``; multi-level keys may be gotten at using ``for((k1,k2,k3),v)`` and so on. The ``v`` variable will be bound to to a scalar value (a string or a number) if the map stops at that level, or to a map-valued variable if the map goes deeper. If the map isn't deep enough then the loop body won't be executed. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/for-srec-example.tbl @@ -1644,7 +1644,7 @@ Single-level keys may be gotten at using either ``for(k,v)`` or ``for((k),v)``; red green 120 11 195 yellow blue 140 0 240 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --pprint --from data/for-srec-example.tbl put ' @@ -1663,7 +1663,7 @@ Single-level keys may be gotten at using either ``for(k,v)`` or ``for((k),v)``; red green 120 11 195 326 326 326 yellow blue 140 0 240 380 380 380 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/small --opprint put 'for (k,v in $*) { $[k."_type"] = typeof(v) }' @@ -1678,7 +1678,7 @@ Note that the value of the current field in the for-loop can be gotten either us Important note: to avoid inconsistent looping behavior in case you're setting new fields (and/or unsetting existing ones) while looping over the record, **Miller makes a copy of the record before the loop: loop variables are bound from the copy and all other reads/writes involve the record itself**: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/small --opprint put ' @@ -1700,7 +1700,7 @@ Important note: to avoid inconsistent looping behavior in case you're setting ne It can be confusing to modify the stream record while iterating over a copy of it, so instead you might find it simpler to use a local variable in the loop and only update the stream record after the loop: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/small --opprint put ' @@ -1721,7 +1721,7 @@ It can be confusing to modify the stream record while iterating over a copy of i You can also start iterating on sub-hashmaps of an out-of-stream or local variable; you can loop over nested keys; you can loop over all out-of-stream variables. The bound variables are bound to a copy of the sub-hashmap as it was before the loop started. The sub-hashmap is specified by square-bracketed indices after ``in``, and additional deeper indices are bound to loop key-variables. The terminal values are bound to the loop value-variable whenever the keys are not too shallow. The value-variable may refer to a terminal (string, number) or it may be map-valued if the map goes deeper. Example indexing is as follows: -.. code-block:: bash +.. code-block:: none # Parentheses are optional for single key: for (k1, v in @a["b"]["c"]) { ... } @@ -1734,7 +1734,7 @@ You can also start iterating on sub-hashmaps of an out-of-stream or local variab That's confusing in the abstract, so a concrete example is in order. Suppose the out-of-stream variable ``@myvar`` is populated as follows: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr -n put --jknquoteint -q ' @@ -1763,7 +1763,7 @@ That's confusing in the abstract, so a concrete example is in order. Suppose the Then we can get at various values as follows: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr -n put --jknquoteint -q ' @@ -1786,7 +1786,7 @@ Then we can get at various values as follows: key=3,valuetype=map key=6,valuetype=map -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr -n put --jknquoteint -q ' @@ -1809,7 +1809,7 @@ Then we can get at various values as follows: key1=3,key2=4,valuetype=int key1=6,key2=7,valuetype=map -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr -n put --jknquoteint -q ' @@ -1836,7 +1836,7 @@ C-style triple-for loops These are supported as follows: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/small --opprint put ' @@ -1853,7 +1853,7 @@ These are supported as follows: eks wye 4 0.38139939387114097 0.13418874328430463 10 wye pan 5 0.5732889198020006 0.8636244699032729 15 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/small --opprint put ' @@ -1890,7 +1890,7 @@ Begin/end blocks Miller supports an ``awk``-like ``begin/end`` syntax. The statements in the ``begin`` block are executed before any input records are read; the statements in the ``end`` block are executed after the last input record is read. (If you want to execute some statement at the start of each file, not at the start of the first file as with ``begin``, you might use a pattern/action block of the form ``FNR == 1 { ... }``.) All statements outside of ``begin`` or ``end`` are, of course, executed on every input record. Semicolons separate statements inside or outside of begin/end blocks; semicolons are required between begin/end block bodies and any subsequent statement. For example: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put ' @@ -1912,7 +1912,7 @@ Miller supports an ``awk``-like ``begin/end`` syntax. The statements in the ``b Since uninitialized out-of-stream variables default to 0 for addition/substraction and 1 for multiplication when they appear on expression right-hand sides (not quite as in ``awk``, where they'd default to 0 either way), the above can be written more succinctly as -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put ' @@ -1933,7 +1933,7 @@ Since uninitialized out-of-stream variables default to 0 for addition/substracti The **put -q** option is a shorthand which suppresses printing of each output record, with only ``emit`` statements being output. So to get only summary outputs, one could write -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q ' @@ -1944,7 +1944,7 @@ The **put -q** option is a shorthand which suppresses printing of each output re We can do similarly with multiple out-of-stream variables: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q ' @@ -1960,7 +1960,7 @@ We can do similarly with multiple out-of-stream variables: This is of course not much different than -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr stats1 -a count,sum -f x ../data/small @@ -2043,7 +2043,7 @@ Details: * The ``print`` and ``dump`` keywords produce output immediately to standard output, or to specified file(s) or pipe-to command if present. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --help-keyword print @@ -2052,7 +2052,7 @@ Details: Example: mlr --from f.dat put -q 'for (k, v in $*) { print k . " => " . v }' Example: mlr --from f.dat put '(NR % 1000 == 0) { print > stderr, "Checkpoint ".NR}' -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --help-keyword dump @@ -2076,7 +2076,7 @@ Details: * ``mlr put`` sends the current record (possibly modified by the ``put`` expression) to the output record stream. Records are then input to the following verb in a ``then``-chain (if any), else printed to standard output (unless ``put -q``). The **tee** keyword *additionally* writes the output record to specified file(s) or pipe-to command, or immediately to ``stdout``/``stderr``. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --help-keyword tee @@ -2107,7 +2107,7 @@ Details: * ``mlr put``'s ``emitf``, ``emitp``, and ``emit`` send out-of-stream variables to the output record stream. These are then input to the following verb in a ``then``-chain (if any), else printed to standard output. When redirected with ``>``, ``>>``, or ``|``, they *instead* write the out-of-stream variable(s) to specified file(s) or pipe-to command, or immediately to ``stdout``/``stderr``. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --help-keyword emitf @@ -2138,7 +2138,7 @@ Details: Please see http://johnkerl.org/miller/doc for more information. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --help-keyword emitp @@ -2171,7 +2171,7 @@ Details: Please see http://johnkerl.org/miller/doc for more information. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --help-keyword emit @@ -2214,7 +2214,7 @@ There are three variants: ``emitf``, ``emit``, and ``emitp``. Keep in mind that Use **emitf** to output several out-of-stream variables side-by-side in the same output record. For ``emitf`` these mustn't have indexing using ``@name[...]``. Example: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q '@count += 1; @x_sum += $x; @y_sum += $y; end { emitf @count, @x_sum, @y_sum}' data/small @@ -2222,7 +2222,7 @@ Use **emitf** to output several out-of-stream variables side-by-side in the same Use **emit** to output an out-of-stream variable. If it's non-indexed you'll get a simple key-value pair: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/small @@ -2232,7 +2232,7 @@ Use **emit** to output an out-of-stream variable. If it's non-indexed you'll get a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463 a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q '@sum += $x; end { dump }' data/small @@ -2240,7 +2240,7 @@ Use **emit** to output an out-of-stream variable. If it's non-indexed you'll get "sum": 2.264762 } -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q '@sum += $x; end { emit @sum }' data/small @@ -2248,7 +2248,7 @@ Use **emit** to output an out-of-stream variable. If it's non-indexed you'll get If it's indexed then use as many names after ``emit`` as there are indices: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q '@sum[$a] += $x; end { dump }' data/small @@ -2260,7 +2260,7 @@ If it's indexed then use as many names after ``emit`` as there are indices: } } -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q '@sum[$a] += $x; end { emit @sum, "a" }' data/small @@ -2268,7 +2268,7 @@ If it's indexed then use as many names after ``emit`` as there are indices: a=eks,sum=1.140079 a=wye,sum=0.777892 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q '@sum[$a][$b] += $x; end { dump }' data/small @@ -2288,7 +2288,7 @@ If it's indexed then use as many names after ``emit`` as there are indices: } } -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q '@sum[$a][$b] += $x; end { emit @sum, "a", "b" }' data/small @@ -2298,7 +2298,7 @@ If it's indexed then use as many names after ``emit`` as there are indices: a=wye,b=wye,sum=0.204603 a=wye,b=pan,sum=0.573289 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q '@sum[$a][$b][$i] += $x; end { dump }' data/small @@ -2328,7 +2328,7 @@ If it's indexed then use as many names after ``emit`` as there are indices: } } -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q '@sum[$a][$b][$i] += $x; end { emit @sum, "a", "b", "i" }' data/small @@ -2340,7 +2340,7 @@ If it's indexed then use as many names after ``emit`` as there are indices: Now for **emitp**: if you have as many names following ``emit`` as there are levels in the out-of-stream variable's hashmap, then ``emit`` and ``emitp`` do the same thing. Where they differ is when you don't specify as many names as there are hashmap levels. In this case, Miller needs to flatten multiple map indices down to output-record keys: ``emitp`` includes full prefixing (hence the ``p`` in ``emitp``) while ``emit`` takes the deepest hashmap key as the output-record key: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q '@sum[$a][$b] += $x; end { dump }' data/small @@ -2360,7 +2360,7 @@ Now for **emitp**: if you have as many names following ``emit`` as there are lev } } -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q '@sum[$a][$b] += $x; end { emit @sum, "a" }' data/small @@ -2368,7 +2368,7 @@ Now for **emitp**: if you have as many names following ``emit`` as there are lev a=eks,pan=0.758680,wye=0.381399 a=wye,wye=0.204603,pan=0.573289 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q '@sum[$a][$b] += $x; end { emit @sum }' data/small @@ -2376,7 +2376,7 @@ Now for **emitp**: if you have as many names following ``emit`` as there are lev pan=0.758680,wye=0.381399 wye=0.204603,pan=0.573289 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q '@sum[$a][$b] += $x; end { emitp @sum, "a" }' data/small @@ -2384,13 +2384,13 @@ Now for **emitp**: if you have as many names following ``emit`` as there are lev a=eks,sum:pan=0.758680,sum:wye=0.381399 a=wye,sum:wye=0.204603,sum:pan=0.573289 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q '@sum[$a][$b] += $x; end { emitp @sum }' data/small sum:pan:pan=0.346790,sum:eks:pan=0.758680,sum:eks:wye=0.381399,sum:wye:wye=0.204603,sum:wye:pan=0.573289 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --oxtab put -q '@sum[$a][$b] += $x; end { emitp @sum }' data/small @@ -2403,7 +2403,7 @@ Now for **emitp**: if you have as many names following ``emit`` as there are lev Use **--oflatsep** to specify the character which joins multilevel keys for ``emitp`` (it defaults to a colon): -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q --oflatsep / '@sum[$a][$b] += $x; end { emitp @sum, "a" }' data/small @@ -2411,13 +2411,13 @@ keys for ``emitp`` (it defaults to a colon): a=eks,sum/pan=0.758680,sum/wye=0.381399 a=wye,sum/wye=0.204603,sum/pan=0.573289 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q --oflatsep / '@sum[$a][$b] += $x; end { emitp @sum }' data/small sum/pan/pan=0.346790,sum/eks/pan=0.758680,sum/eks/wye=0.381399,sum/wye/wye=0.204603,sum/wye/pan=0.573289 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --oxtab put -q --oflatsep / '@sum[$a][$b] += $x; end { emitp @sum }' data/small @@ -2433,7 +2433,7 @@ Multi-emit statements You can emit **multiple map-valued expressions side-by-side** by including their names in parentheses: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/medium --opprint put -q ' @@ -2480,7 +2480,7 @@ Emit-all statements Use **emit all** (or ``emit @*`` which is synonymous) to output all out-of-stream variables. You can use the following idiom to get various accumulators output side-by-side (reminiscent of ``mlr stats1``): -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/small --opprint put -q '@v[$a][$b]["sum"] += $x; @v[$a][$b]["count"] += 1; end{emit @*,"a","b"}' @@ -2491,7 +2491,7 @@ Use **emit all** (or ``emit @*`` which is synonymous) to output all out-of-strea wye wye 0.204603 1 wye pan 0.573289 1 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/small --opprint put -q '@sum[$a][$b] += $x; @count[$a][$b] += 1; end{emit @*,"a","b"}' @@ -2509,7 +2509,7 @@ Use **emit all** (or ``emit @*`` which is synonymous) to output all out-of-strea wye wye 1 wye pan 1 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/small --opprint put -q '@sum[$a][$b] += $x; @count[$a][$b] += 1; end{emit (@sum, @count),"a","b"}' @@ -2525,7 +2525,7 @@ Unset statements You can clear a map key by assigning the empty string as its value: ``$x=""`` or ``@x=""``. Using ``unset`` you can remove the key entirely. Examples: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/small @@ -2535,7 +2535,7 @@ You can clear a map key by assigning the empty string as its value: ``$x=""`` or a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463 a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put 'unset $x, $a' data/small @@ -2547,7 +2547,7 @@ You can clear a map key by assigning the empty string as its value: ``$x=""`` or This can also be done, of course, using ``mlr cut -x``. You can also clear out-of-stream or local variables, at the base name level, or at an indexed sublevel: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q '@sum[$a][$b] += $x; end { dump; unset @sum; dump }' data/small @@ -2569,7 +2569,7 @@ This can also be done, of course, using ``mlr cut -x``. You can also clear out-o { } -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q '@sum[$a][$b] += $x; end { dump; unset @sum["eks"]; dump }' data/small @@ -2607,14 +2607,14 @@ Filter statements You can use ``filter`` within ``put``. In fact, the following two are synonymous: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr filter 'NR==2 || NR==3' data/small a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797 a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put 'filter NR==2 || NR==3' data/small @@ -2623,7 +2623,7 @@ You can use ``filter`` within ``put``. In fact, the following two are synonymous The former, of course, is much easier to type. But the latter allows you to define more complex expressions for the filter, and/or do other things in addition to the filter: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put '@running_sum += $x; filter @running_sum > 1.3' data/small @@ -2631,7 +2631,7 @@ The former, of course, is much easier to type. But the latter allows you to defi a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463 a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put '$z = $x * $y; filter $z > 0.3' data/small @@ -4914,7 +4914,7 @@ User-defined functions Here's the obligatory example of a recursive function to compute the factorial function: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint --from data/small put ' @@ -4963,7 +4963,7 @@ User-defined subroutines Example: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint --from data/small put -q ' diff --git a/docs/reference-dsl.rst.in b/docs/reference-dsl.rst.in index 43700aa2e..5de7e0b82 100644 --- a/docs/reference-dsl.rst.in +++ b/docs/reference-dsl.rst.in @@ -121,17 +121,17 @@ POKI_INCLUDE_AND_RUN_ESCAPED(data/trailing-commas.sh)HERE Bodies for all compound statements must be enclosed in **curly braces**, even if the body is a single statement: -.. code-block:: bash +.. code-block:: none mlr put 'if ($x == 1) $y = 2' # Syntax error -.. code-block:: bash +.. code-block:: none mlr put 'if ($x == 1) { $y = 2 }' # This is OK Bodies for compound statements may be empty: -.. code-block:: bash +.. code-block:: none mlr put 'if ($x == 1) { }' # This no-op is syntactically acceptable @@ -360,7 +360,7 @@ Local variables can be defined either untyped as in ``x = 1``, or typed as in `` The reason for ``num`` is that ``int`` and ``float`` typedecls are very precise: -.. code-block:: bash +.. code-block:: none float a = 0; # Runtime error since 0 is int not float int b = 1.0; # Runtime error since 1.0 is float not int @@ -371,7 +371,7 @@ A suggestion is to use ``num`` for general use when you want numeric content, an The ``var`` type declaration indicates no type restrictions, e.g. ``var x = 1`` has the same type restrictions on ``x`` as ``x = 1``. The difference is in intentional shadowing: if you have ``x = 1`` in outer scope and ``x = 2`` in inner scope (e.g. within a for-loop or an if-statement) then outer-scope ``x`` has value 2 after the second assignment. But if you have ``var x = 2`` in the inner scope, then you are declaring a variable scoped to the inner block.) For example: -.. code-block:: bash +.. code-block:: none x = 1; if (NR == 4) { @@ -379,7 +379,7 @@ The ``var`` type declaration indicates no type restrictions, e.g. ``var x = 1`` } print x; # Value of x is now two -.. code-block:: bash +.. code-block:: none x = 1; if (NR == 4) { @@ -389,7 +389,7 @@ The ``var`` type declaration indicates no type restrictions, e.g. ``var x = 1`` Likewise function arguments can optionally be typed, with type enforced when the function is called: -.. code-block:: bash +.. code-block:: none func f(map m, int i) { ... @@ -404,7 +404,7 @@ Likewise function arguments can optionally be typed, with type enforced when the Thirdly, function return values can be type-checked at the point of ``return`` using ``:`` and a typedecl after the parameter list: -.. code-block:: bash +.. code-block:: none func f(map m, int i): bool { ... @@ -463,7 +463,7 @@ Operator precedence Operators are listed in order of decreasing precedence, highest first. -.. code-block:: bash +.. code-block:: none Operators Associativity --------- ------------- @@ -524,11 +524,11 @@ If-statements These are again reminiscent of ``awk``. Pattern-action blocks are a special case of ``if`` with no ``elif`` or ``else`` blocks, no ``if`` keyword, and parentheses optional around the boolean expression: -.. code-block:: bash +.. code-block:: none mlr put 'NR == 4 {$foo = "bar"}' -.. code-block:: bash +.. code-block:: none mlr put 'if (NR == 4) {$foo = "bar"}' diff --git a/docs/reference-verbs.rst b/docs/reference-verbs.rst index a165e4042..31f4cb183 100644 --- a/docs/reference-verbs.rst +++ b/docs/reference-verbs.rst @@ -15,7 +15,7 @@ Here's a comparison of verbs and ``put``/``filter`` DSL expressions: Example: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr stats1 -a sum -f x -g a data/small @@ -31,7 +31,7 @@ Example: Example: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q '@x_sum[$a] += $x; end{emit @x_sum, "a"}' data/small @@ -52,20 +52,20 @@ altkv Map list of values to alternating key/value pairs. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr altkv -h Usage: mlr altkv [no options] Given fields with values of the form a,b,c,d,e,f emits a=b,c=d,e=f pairs. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ echo 'a,b,c,d,e,f' | mlr altkv a=b,c=d,e=f -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ echo 'a,b,c,d,e,f,g' | mlr altkv @@ -78,7 +78,7 @@ bar Cheesy bar-charting. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr bar -h @@ -96,7 +96,7 @@ Cheesy bar-charting. --auto Automatically computes limits, ignoring --lo and --hi. Holds all records in memory before producing any output. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint cat data/small @@ -107,7 +107,7 @@ Cheesy bar-charting. eks wye 4 0.38139939387114097 0.13418874328430463 wye pan 5 0.5732889198020006 0.8636244699032729 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint bar --lo 0 --hi 1 -f x,y data/small @@ -118,7 +118,7 @@ Cheesy bar-charting. eks wye 4 ***************......................... *****................................... wye pan 5 **********************.................. **********************************...... -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint bar --lo 0.4 --hi 0.6 -f x,y data/small @@ -129,7 +129,7 @@ Cheesy bar-charting. eks wye 4 #....................................... #....................................... wye pan 5 **********************************...... ***************************************# -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint bar --auto -f x,y data/small @@ -145,7 +145,7 @@ Cheesy bar-charting. bootstrap ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr bootstrap --help @@ -158,7 +158,7 @@ bootstrap The canonical use for bootstrap sampling is to put error bars on statistical quantities, such as mean. For example: -.. code-block:: bash +.. code-block:: none $ mlr --opprint stats1 -a mean,count -f u -g color data/colored-shapes.dkvp color u_mean u_count @@ -169,7 +169,7 @@ The canonical use for bootstrap sampling is to put error bars on statistical qua blue 0.517717 1470 orange 0.490532 303 -.. code-block:: bash +.. code-block:: none $ mlr --opprint bootstrap then stats1 -a mean,count -f u -g color data/colored-shapes.dkvp color u_mean u_count @@ -180,7 +180,7 @@ The canonical use for bootstrap sampling is to put error bars on statistical qua blue 0.512529 1496 orange 0.521030 321 -.. code-block:: bash +.. code-block:: none $ mlr --opprint bootstrap then stats1 -a mean,count -f u -g color data/colored-shapes.dkvp color u_mean u_count @@ -191,7 +191,7 @@ The canonical use for bootstrap sampling is to put error bars on statistical qua green 0.496803 1075 purple 0.486337 1199 -.. code-block:: bash +.. code-block:: none $ mlr --opprint bootstrap then stats1 -a mean,count -f u -g color data/colored-shapes.dkvp color u_mean u_count @@ -209,7 +209,7 @@ cat Most useful for format conversions (see :doc:`file-formats`, and concatenating multiple same-schema CSV files to have the same header: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr cat -h @@ -222,7 +222,7 @@ Most useful for format conversions (see :doc:`file-formats`, and concatenating m -v Write a low-level record-structure dump to stderr. -N {name} Prepend field {name} to each record with record-counter starting at 1 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/a.csv @@ -230,14 +230,14 @@ Most useful for format conversions (see :doc:`file-formats`, and concatenating m 1,2,3 4,5,6 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/b.csv a,b,c 7,8,9 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv cat data/a.csv data/b.csv @@ -246,7 +246,7 @@ Most useful for format conversions (see :doc:`file-formats`, and concatenating m 4,5,6 7,8,9 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --oxtab cat data/a.csv data/b.csv @@ -262,7 +262,7 @@ Most useful for format conversions (see :doc:`file-formats`, and concatenating m b 8 c 9 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv cat -n data/a.csv data/b.csv @@ -271,7 +271,7 @@ Most useful for format conversions (see :doc:`file-formats`, and concatenating m 2,4,5,6 3,7,8,9 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint cat data/small @@ -282,7 +282,7 @@ Most useful for format conversions (see :doc:`file-formats`, and concatenating m eks wye 4 0.38139939387114097 0.13418874328430463 wye pan 5 0.5732889198020006 0.8636244699032729 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint cat -n -g a data/small @@ -298,7 +298,7 @@ Most useful for format conversions (see :doc:`file-formats`, and concatenating m check ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr check --help @@ -311,7 +311,7 @@ check clean-whitespace ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr clean-whitespace --help @@ -328,7 +328,7 @@ clean-whitespace It is an error to specify -k as well as -v -- to clean keys and values, leave off -k as well as -v. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --ojson cat data/clean-whitespace.csv @@ -336,7 +336,7 @@ clean-whitespace { " Name ": "Bob Wang ", " Preference ": " red " } { " Name ": " Carol Vee", " Preference ": " yellow" } -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --ojson clean-whitespace -k data/clean-whitespace.csv @@ -344,7 +344,7 @@ clean-whitespace { "Name": "Bob Wang ", "Preference": " red " } { "Name": " Carol Vee", "Preference": " yellow" } -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --ojson clean-whitespace -v data/clean-whitespace.csv @@ -352,7 +352,7 @@ clean-whitespace { " Name ": "Bob Wang", " Preference ": "red" } { " Name ": "Carol Vee", " Preference ": "yellow" } -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --ojson clean-whitespace data/clean-whitespace.csv @@ -373,7 +373,7 @@ Function links: count ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr count --help @@ -385,13 +385,13 @@ count -n Show only the number of distinct values. Not interesting without -g. -o {name} Field name for output count. Default "count". -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr count data/medium count=10000 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr count -g a data/medium @@ -401,13 +401,13 @@ count a=zee,count=2047 a=hat,count=1941 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr count -n -g a data/medium count=5 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr count -g b data/medium @@ -417,13 +417,13 @@ count b=eks,count=2008 b=hat,count=2050 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr count -n -g b data/medium count=5 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr count -g a,b data/medium @@ -458,7 +458,7 @@ count count-distinct ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr count-distinct --help @@ -477,7 +477,7 @@ count-distinct for distinct a field values and counts for distinct b field values separately. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr count-distinct -f a,b then sort -nr count data/medium @@ -507,7 +507,7 @@ count-distinct a=hat,b=pan,count=363 a=eks,b=zee,count=357 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr count-distinct -u -f a,b data/medium @@ -522,7 +522,7 @@ count-distinct field=b,value=eks,count=2008 field=b,value=hat,count=2050 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr count-distinct -f a,b -o someothername then sort -nr someothername data/medium @@ -552,7 +552,7 @@ count-distinct a=hat,b=pan,someothername=363 a=eks,b=zee,someothername=357 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr count-distinct -n -f a,b data/medium @@ -563,7 +563,7 @@ count-distinct count-similar ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr count-similar --help @@ -574,7 +574,7 @@ count-similar -g {d,e,f} Group-by-field names for counts. -o {name} Field name for output count. Default "count". -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint head -n 20 data/medium @@ -600,7 +600,7 @@ count-similar zee pan 19 0.43144132839222604 0.8442204830496998 eks wye 20 0.38245149780530685 0.4730652428100751 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint head -n 20 then count-similar -g a data/medium @@ -626,7 +626,7 @@ count-similar hat wye 9 0.03144187646093577 0.7495507603507059 2 hat zee 18 0.05727869223575699 0.13343527626645157 2 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint head -n 20 then count-similar -g a then sort -f a data/medium @@ -657,7 +657,7 @@ count-similar cut ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr cut --help @@ -678,7 +678,7 @@ cut mlr cut -r -f '^status$,"sda[0-9]"' mlr cut -r -f '^status$,"sda[0-9]"i' (this is case-insensitive) -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint cat data/small @@ -689,7 +689,7 @@ cut eks wye 4 0.38139939387114097 0.13418874328430463 wye pan 5 0.5732889198020006 0.8636244699032729 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint cut -f y,x,i data/small @@ -700,13 +700,13 @@ cut 4 0.38139939387114097 0.13418874328430463 5 0.5732889198020006 0.8636244699032729 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ echo 'a=1,b=2,c=3' | mlr cut -f b,c,a a=1,b=2,c=3 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ echo 'a=1,b=2,c=3' | mlr cut -o -f b,c,a @@ -717,7 +717,7 @@ cut decimate ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr decimate --help @@ -733,7 +733,7 @@ decimate fill-down ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr fill-down --help @@ -752,7 +752,7 @@ fill-down -f Field names for fill-down. -h|--help Show this message. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/fill-down.csv @@ -761,7 +761,7 @@ fill-down 4,5,6 7,,9 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv fill-down -f b data/fill-down.csv @@ -770,7 +770,7 @@ fill-down 4,5,6 7,5,9 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv fill-down -a -f b data/fill-down.csv @@ -784,7 +784,7 @@ fill-down filter ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr filter --help @@ -881,7 +881,7 @@ Please see :doc:`reference-dsl` for more information about the expression langua format-values ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr format-values --help @@ -915,7 +915,7 @@ format-values -n Coerce field values autodetected as int to float, and then apply the float format. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint format-values data/small @@ -926,7 +926,7 @@ format-values eks wye 4 0.381399 0.134189 wye pan 5 0.573289 0.863624 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint format-values -n data/small @@ -937,7 +937,7 @@ format-values eks wye 4.000000 0.381399 0.134189 wye pan 5.000000 0.573289 0.863624 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint format-values -i %08llx -f %.6le -s X%sX data/small @@ -948,7 +948,7 @@ format-values XeksX XwyeX 00000004 3.813994e-01 1.341887e-01 XwyeX XpanX 00000005 5.732889e-01 8.636245e-01 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint format-values -i %08llx -f %.6le -s X%sX -n data/small @@ -964,7 +964,7 @@ format-values fraction ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr fraction --help @@ -991,7 +991,7 @@ fraction For example, suppose you have the following CSV file: -.. code-block:: bash +.. code-block:: none u=female,v=red,n=2458 u=female,v=green,n=192 @@ -1008,7 +1008,7 @@ For example, suppose you have the following CSV file: Then we can see what each record's ``n`` contributes to the total ``n``: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint fraction -f n data/fraction-example.csv @@ -1028,7 +1028,7 @@ Then we can see what each record's ``n`` contributes to the total ``n``: Using ``-g`` we can split those out by gender, or by color: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint fraction -f n -g u data/fraction-example.csv @@ -1046,7 +1046,7 @@ Using ``-g`` we can split those out by gender, or by color: male yellow 1192 0.293886 male orange 448 0.110454 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint fraction -f n -g v data/fraction-example.csv @@ -1068,7 +1068,7 @@ We can see, for example, that 70.9% of females have red (on the left) while 94.5 To convert fractions to percents, you may use ``-p``: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint fraction -f n -p data/fraction-example.csv @@ -1088,7 +1088,7 @@ To convert fractions to percents, you may use ``-p``: Another often-used idiom is to convert from a point distribution to a cumulative distribution, also known as "running sums". Here, you can use ``-c``: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint fraction -f n -p -c data/fraction-example.csv @@ -1106,7 +1106,7 @@ Another often-used idiom is to convert from a point distribution to a cumulative male yellow 1192 94.051255 male orange 448 100 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint fraction -f n -g u -p -c data/fraction-example.csv @@ -1129,7 +1129,7 @@ Another often-used idiom is to convert from a point distribution to a cumulative grep ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr grep -h @@ -1155,7 +1155,7 @@ grep group-by ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr group-by --help @@ -1164,7 +1164,7 @@ group-by This is similar to ``sort`` but with less work. Namely, Miller's sort has three steps: read through the data and append linked lists of records, one for each unique combination of the key-field values; after all records are read, sort the key-field values; then print each record-list. The group-by operation simply omits the middle sort. An example should make this more clear. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint group-by a data/small @@ -1175,7 +1175,7 @@ This is similar to ``sort`` but with less work. Namely, Miller's sort has three wye wye 3 0.20460330576630303 0.33831852551664776 wye pan 5 0.5732889198020006 0.8636244699032729 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint sort -f a data/small @@ -1193,7 +1193,7 @@ In this example, since the sort is on field ``a``, the first step is to group to group-like ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr group-like --help @@ -1202,7 +1202,7 @@ group-like This groups together records having the same schema (i.e. same ordered list of field names) which is useful for making sense of time-ordered output as described in :doc:`record-heterogeneity` -- in particular, in preparation for CSV or pretty-print output. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr cat data/het.dkvp @@ -1212,7 +1212,7 @@ This groups together records having the same schema (i.e. same ordered list of f record_count=150,resource=/path/to/second/file resource=/some/other/path,loadsec=0.97,ok=false -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint group-like data/het.dkvp @@ -1230,7 +1230,7 @@ This groups together records having the same schema (i.e. same ordered list of f having-fields ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr having-fields --help @@ -1251,7 +1251,7 @@ having-fields Similar to :ref:`reference-verbs-group-like`, this retains records with specified schema. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr cat data/het.dkvp @@ -1261,7 +1261,7 @@ Similar to :ref:`reference-verbs-group-like`, this retains records with specifie record_count=150,resource=/path/to/second/file resource=/some/other/path,loadsec=0.97,ok=false -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr having-fields --at-least resource data/het.dkvp @@ -1271,7 +1271,7 @@ Similar to :ref:`reference-verbs-group-like`, this retains records with specifie record_count=150,resource=/path/to/second/file resource=/some/other/path,loadsec=0.97,ok=false -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr having-fields --which-are resource,ok,loadsec data/het.dkvp @@ -1284,7 +1284,7 @@ Similar to :ref:`reference-verbs-group-like`, this retains records with specifie head ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr head --help @@ -1297,7 +1297,7 @@ head Note that ``head`` is distinct from :ref:`reference-verbs-top` -- ``head`` shows fields which appear first in the data stream; ``top`` shows fields which are numerically largest (or smallest). -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint head -n 4 data/medium @@ -1307,7 +1307,7 @@ Note that ``head`` is distinct from :ref:`reference-verbs-top` -- ``head`` shows wye wye 3 0.20460330576630303 0.33831852551664776 eks wye 4 0.38139939387114097 0.13418874328430463 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint head -n 1 -g b data/medium @@ -1323,7 +1323,7 @@ Note that ``head`` is distinct from :ref:`reference-verbs-top` -- ``head`` shows histogram ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr histogram --help @@ -1339,7 +1339,7 @@ histogram This is just a histogram; there's not too much to say here. A note about binning, by example: Suppose you use ``--lo 0.0 --hi 1.0 --nbins 10 -f x``. The input numbers less than 0 or greater than 1 aren't counted in any bin. Input numbers equal to 1 are counted in the last bin. That is, bin 0 has ``0.0 ≤ x < 0.1``, bin 1 has ``0.1 ≤ x < 0.2``, etc., but bin 9 has ``0.9 ≤ x ≤ 1.0``. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint put '$x2=$x**2;$x3=$x2*$x' then histogram -f x,x2,x3 --lo 0 --hi 1 --nbins 10 data/medium @@ -1355,7 +1355,7 @@ This is just a histogram; there's not too much to say here. A note about binning 0.800000 0.900000 986 571 383 0.900000 1.000000 1013 507 341 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint put '$x2=$x**2;$x3=$x2*$x' then histogram -f x,x2,x3 --lo 0 --hi 1 --nbins 10 -o my_ data/medium @@ -1376,7 +1376,7 @@ This is just a histogram; there's not too much to say here. A note about binning join ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr join --help @@ -1431,7 +1431,7 @@ Examples: Join larger table with IDs with smaller ID-to-name lookup table, showing only paired records: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsvlite --opprint cat data/join-left-example.csv @@ -1442,7 +1442,7 @@ Join larger table with IDs with smaller ID-to-name lookup table, showing only pa 400 david 500 edgar -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsvlite --opprint cat data/join-right-example.csv @@ -1468,7 +1468,7 @@ Join larger table with IDs with smaller ID-to-name lookup table, showing only pa present 400 present 300 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsvlite --opprint join -u -j id -r idcode -f data/join-left-example.csv data/join-right-example.csv @@ -1495,7 +1495,7 @@ Join larger table with IDs with smaller ID-to-name lookup table, showing only pa Same, but with sorting the input first: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsvlite --opprint sort -f idcode then join -j id -r idcode -f data/join-left-example.csv data/join-right-example.csv @@ -1522,7 +1522,7 @@ Same, but with sorting the input first: Same, but showing only unpaired records: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsvlite --opprint join --np --ul --ur -u -j id -r idcode -f data/join-left-example.csv data/join-right-example.csv @@ -1534,7 +1534,7 @@ Same, but showing only unpaired records: Use prefixing options to disambiguate between otherwise identical non-join field names: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csvlite --opprint cat data/self-join.csv data/self-join.csv @@ -1544,7 +1544,7 @@ Use prefixing options to disambiguate between otherwise identical non-join field 1 2 3 1 4 5 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csvlite --opprint join -j a --lp left_ --rp right_ -f data/self-join.csv data/self-join.csv @@ -1556,7 +1556,7 @@ Use prefixing options to disambiguate between otherwise identical non-join field Use zero join columns: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csvlite --opprint join -j "" --lp left_ --rp right_ -f data/self-join.csv data/self-join.csv @@ -1571,7 +1571,7 @@ Use zero join columns: label ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr label --help @@ -1588,7 +1588,7 @@ See also :ref:`reference-verbs-rename`. Example: Files such as ``/etc/passwd``, ``/etc/group``, and so on have implicit field names which are found in section-5 manpages. These field names may be made explicit as follows: -.. code-block:: bash +.. code-block:: none % grep -v '^#' /etc/passwd | mlr --nidx --fs : --opprint label name,password,uid,gid,gecos,home_dir,shell | head name password uid gid gecos home_dir shell @@ -1604,7 +1604,7 @@ Example: Files such as ``/etc/passwd``, ``/etc/group``, and so on have implicit Likewise, if you have CSV/CSV-lite input data which has somehow been bereft of its header line, you can re-add a header line using ``--implicit-csv-header`` and ``label``: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/headerless.csv @@ -1613,7 +1613,7 @@ Likewise, if you have CSV/CSV-lite input data which has somehow been bereft of i Alice,56,missing Carol,45,present -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv --implicit-csv-header cat data/headerless.csv @@ -1623,7 +1623,7 @@ Likewise, if you have CSV/CSV-lite input data which has somehow been bereft of i Alice,56,missing Carol,45,present -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv --implicit-csv-header label name,age,status data/headerless.csv @@ -1633,7 +1633,7 @@ Likewise, if you have CSV/CSV-lite input data which has somehow been bereft of i Alice,56,missing Carol,45,present -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --implicit-csv-header --opprint label name,age,status data/headerless.csv @@ -1648,7 +1648,7 @@ Likewise, if you have CSV/CSV-lite input data which has somehow been bereft of i least-frequent ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr least-frequent -h @@ -1662,7 +1662,7 @@ least-frequent -o {name} Field name for output count. Default "count". See also "mlr most-frequent". -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint --from data/colored-shapes.dkvp least-frequent -f shape -n 5 @@ -1671,7 +1671,7 @@ least-frequent triangle 3372 square 4115 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint --from data/colored-shapes.dkvp least-frequent -f shape,color -n 5 @@ -1682,7 +1682,7 @@ least-frequent circle green 287 circle purple 289 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint --from data/colored-shapes.dkvp least-frequent -f shape,color -n 5 -o someothername @@ -1693,7 +1693,7 @@ least-frequent circle green 287 circle purple 289 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint --from data/colored-shapes.dkvp least-frequent -f shape,color -n 5 -b @@ -1711,7 +1711,7 @@ See also :ref:`reference-verbs-most-frequent`. merge-fields ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr merge-fields --help @@ -1764,7 +1764,7 @@ This is like ``mlr stats1`` but all accumulation is done across fields within ea Examples: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csvlite --opprint cat data/inout.csv @@ -1773,7 +1773,7 @@ Examples: 526 320 963 780 220 888 705 831 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csvlite --opprint merge-fields -a min,max,sum -c _in,_out data/inout.csv @@ -1782,7 +1782,7 @@ Examples: 320 526 846 780 963 1743 220 888 1108 705 831 1536 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csvlite --opprint merge-fields -k -a sum -c _in,_out data/inout.csv @@ -1796,7 +1796,7 @@ Examples: most-frequent ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr most-frequent -h @@ -1810,7 +1810,7 @@ most-frequent -o {name} Field name for output count. Default "count". See also "mlr least-frequent". -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint --from data/colored-shapes.dkvp most-frequent -f shape -n 5 @@ -1819,7 +1819,7 @@ most-frequent triangle 3372 circle 2591 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint --from data/colored-shapes.dkvp most-frequent -f shape,color -n 5 @@ -1830,7 +1830,7 @@ most-frequent square yellow 589 square blue 589 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint --from data/colored-shapes.dkvp most-frequent -f shape,color -n 5 -o someothername @@ -1841,7 +1841,7 @@ most-frequent square yellow 589 square blue 589 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint --from data/colored-shapes.dkvp most-frequent -f shape,color -n 5 -b @@ -1859,7 +1859,7 @@ See also :ref:`reference-verbs-least-frequent`. nest ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr nest -h @@ -1916,7 +1916,7 @@ nest nothing ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr nothing -h @@ -1929,7 +1929,7 @@ nothing put ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put --help @@ -2037,7 +2037,7 @@ Please see the :doc:`reference-dsl` for more information about the expression la regularize ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr regularize --help @@ -2057,14 +2057,14 @@ See also :ref:`reference-verbs-reorder`. remove-empty-columns ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr remove-empty-columns --help Usage: mlr remove-empty-columns Omits fields which are empty on every input row. Non-streaming. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/remove-empty-columns.csv @@ -2073,7 +2073,7 @@ remove-empty-columns 2,,4,,5 3,,5,,7 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv remove-empty-columns data/remove-empty-columns.csv @@ -2089,7 +2089,7 @@ Since this verb needs to read all records to see if any of them has a non-empty rename ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr rename --help @@ -2114,7 +2114,7 @@ rename mlr rename -r 'Date_([0-9]+).*,\1' Rename all such fields to be of the form 20151015 mlr rename -r '"name"i,Name' Rename "name", "Name", "NAME", etc. to "Name" -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint cat data/small @@ -2125,7 +2125,7 @@ rename eks wye 4 0.38139939387114097 0.13418874328430463 wye pan 5 0.5732889198020006 0.8636244699032729 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint rename i,INDEX,b,COLUMN2 data/small @@ -2138,7 +2138,7 @@ rename As discussed in :doc:`performance`, ``sed`` is significantly faster than Miller at doing this. However, Miller is format-aware, so it knows to do renames only within specified field keys and not any others, nor in field values which may happen to contain the same pattern. Example: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ sed 's/y/COLUMN5/g' data/small @@ -2148,7 +2148,7 @@ As discussed in :doc:`performance`, ``sed`` is significantly faster than Miller a=eks,b=wCOLUMN5e,i=4,x=0.38139939387114097,COLUMN5=0.13418874328430463 a=wCOLUMN5e,b=pan,i=5,x=0.5732889198020006,COLUMN5=0.8636244699032729 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr rename y,COLUMN5 data/small @@ -2165,7 +2165,7 @@ See also :ref:`reference-verbs-label`. reorder ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr reorder --help @@ -2187,7 +2187,7 @@ This pivots specified field names to the start or end of the record -- for example when you have highly multi-column data and you want to bring a field or two to the front of line where you can give a quick visual scan. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint cat data/small @@ -2198,7 +2198,7 @@ two to the front of line where you can give a quick visual scan. eks wye 4 0.38139939387114097 0.13418874328430463 wye pan 5 0.5732889198020006 0.8636244699032729 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint reorder -f i,b data/small @@ -2209,7 +2209,7 @@ two to the front of line where you can give a quick visual scan. 4 wye eks 0.38139939387114097 0.13418874328430463 5 pan wye 0.5732889198020006 0.8636244699032729 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint reorder -e -f i,b data/small @@ -2225,7 +2225,7 @@ two to the front of line where you can give a quick visual scan. repeat ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr repeat --help @@ -2258,7 +2258,7 @@ This is useful in at least two ways: one, as a data-generator as in the above example using ``urand()``; two, for reconstructing individual samples from data which has been count-aggregated: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/repeat-example.dat @@ -2266,7 +2266,7 @@ samples from data which has been count-aggregated: color=red,count=4 color=green,count=3 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr repeat -f count then cut -x -f count data/repeat-example.dat @@ -2292,7 +2292,7 @@ p10,p50,p90``, etc. reshape ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr reshape --help @@ -2357,7 +2357,7 @@ reshape sample ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr sample --help @@ -2373,7 +2373,7 @@ uniform probability and no repeats in the sample. (If *n* is less than {field names}``, produce a *k*-sample for each distinct value of the specified field names. -.. code-block:: bash +.. code-block:: none $ mlr --opprint sample -k 4 data/colored-shapes.dkvp color shape flag i u v w x @@ -2429,7 +2429,7 @@ sampling, which works in the streaming case, is ``mlr filter 'urand() & sec2gmt ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr sec2gmt -h @@ -2448,7 +2448,7 @@ sec2gmt sec2gmtdate ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr sec2gmtdate -h @@ -2465,7 +2465,7 @@ sec2gmtdate seqgen ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr seqgen -h @@ -2480,7 +2480,7 @@ seqgen stop, and step are all integers. Step may be negative. It may not be zero unless start == stop. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr seqgen --stop 10 @@ -2495,7 +2495,7 @@ seqgen i=9 i=10 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr seqgen --start 20 --stop 40 --step 4 @@ -2506,7 +2506,7 @@ seqgen i=36 i=40 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr seqgen --start 40 --stop 20 --step -4 @@ -2522,7 +2522,7 @@ seqgen shuffle ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr shuffle -h @@ -2536,7 +2536,7 @@ shuffle skip-trivial-records ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr skip-trivial-records -h @@ -2545,7 +2545,7 @@ skip-trivial-records * those with zero fields; * those for which all fields have empty value. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/trivial-records.csv @@ -2555,7 +2555,7 @@ skip-trivial-records ,, ,8,9 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv skip-trivial-records data/trivial-records.csv @@ -2569,7 +2569,7 @@ skip-trivial-records sort ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr sort --help @@ -2593,7 +2593,7 @@ sort Example: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint sort -f a -nr x data/small @@ -2606,7 +2606,7 @@ Example: Here's an example filtering log data: suppose multiple threads (labeled here by color) are all logging progress counts to a single log file. The log file is (by nature) chronological, so the progress of various threads is interleaved: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ head -n 10 data/multicountdown.dat @@ -2625,7 +2625,7 @@ We can group these by thread by sorting on the thread ID (here, ``color``). Since Miller's sort is stable, this means that timestamps within each thread's log data are still chronological: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ head -n 20 data/multicountdown.dat | mlr --opprint sort -f color @@ -2654,7 +2654,7 @@ timestamps within each thread's log data are still chronological: Any records not having all specified sort keys will appear at the end of the output, in the order they were encountered, regardless of the specified sort order: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr sort -n x data/sort-missing.dkvp @@ -2663,7 +2663,7 @@ were encountered, regardless of the specified sort order: x=4 a=3 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr sort -nr x data/sort-missing.dkvp @@ -2677,14 +2677,14 @@ were encountered, regardless of the specified sort order: sort-within-records ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr sort-within-records -h Usage: mlr sort-within-records [no options] Outputs records sorted lexically ascending by keys. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/sort-within-records.json @@ -2704,7 +2704,7 @@ sort-within-records "a": 9 } -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --ijson --opprint cat data/sort-within-records.json @@ -2717,7 +2717,7 @@ sort-within-records c b a 7 8 9 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --json sort-within-records data/sort-within-records.json @@ -2725,7 +2725,7 @@ sort-within-records { "a": 5, "b": 4, "c": 6 } { "a": 9, "b": 8, "c": 7 } -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --ijson --opprint sort-within-records data/sort-within-records.json @@ -2739,7 +2739,7 @@ sort-within-records stats1 ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr stats1 --help @@ -2799,7 +2799,7 @@ These are simple univariate statistics on one or more number-valued fields (``count`` and ``mode`` apply to non-numeric fields as well), optionally categorized by one or more other fields. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --oxtab stats1 -a count,sum,min,p10,p50,mean,p90,max -f x,y data/medium @@ -2820,7 +2820,7 @@ optionally categorized by one or more other fields. y_p90 0.905366 y_max 0.999965 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint stats1 -a mean -f x,y -g b then sort -f b data/medium @@ -2831,7 +2831,7 @@ optionally categorized by one or more other fields. wye 0.497593 0.504596 zee 0.504242 0.502997 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint stats1 -a p50,p99 -f u,v -g color then put '$ur=$u_p99/$u_p50;$vr=$v_p99/$v_p50' data/colored-shapes.dkvp @@ -2843,7 +2843,7 @@ optionally categorized by one or more other fields. blue 0.525226 0.992655 0.485170 0.993873 1.889958 2.048505 orange 0.483548 0.993635 0.480913 0.989102 2.054884 2.056717 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint count-distinct -f shape then sort -nr count data/colored-shapes.dkvp @@ -2852,7 +2852,7 @@ optionally categorized by one or more other fields. triangle 3372 circle 2591 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint stats1 -a mode -f color -g shape data/colored-shapes.dkvp @@ -2866,7 +2866,7 @@ optionally categorized by one or more other fields. stats2 ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr stats2 --help @@ -2900,7 +2900,7 @@ stats2 These are simple bivariate statistics on one or more pairs of number-valued fields, optionally categorized by one or more fields. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --oxtab put '$x2=$x*$x; $xy=$x*$y; $y2=$y**2' then stats2 -a cov,corr -f x,y,y,y,x2,xy,x2,y2 data/medium @@ -2913,7 +2913,7 @@ fields, optionally categorized by one or more fields. x2_y2_cov -0.000310 x2_y2_corr -0.003425 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint put '$x2=$x*$x; $xy=$x*$y; $y2=$y**2' then stats2 -a linreg-ols,r2 -f x,y,y,y,xy,y2 -g a data/medium @@ -2928,7 +2928,7 @@ Here's an example simple line-fit. The ``x`` and ``y`` fields of the ``data/medium`` dataset are just independent uniformly distributed on the unit interval. Here we remove half the data and fit a line to it. -.. code-block:: bash +.. code-block:: none # Prepare input data: @@ -2962,7 +2962,7 @@ I use `pgr `_ for plotting; here's a screenshot Here's an example estimating time-to-completion for a set of jobs. Input data comes from a log file, with number of work units left to do in the ``count`` field and accumulated seconds in the ``upsec`` field, labeled by the ``color`` field: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ head -n 10 data/multicountdown.dat @@ -2979,7 +2979,7 @@ Here's an example estimating time-to-completion for a set of jobs. Input data co We can do a linear regression on count remaining as a function of time: with ``c = m*u+b`` we want to find the time when the count goes to zero, i.e. ``u=-b/m``. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --oxtab stats2 -a linreg-pca -f upsec,count -g color then put '$donesec = -$upsec_count_pca_b/$upsec_count_pca_m' data/multicountdown.dat @@ -3016,7 +3016,7 @@ We can do a linear regression on count remaining as a function of time: with ``c step ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr step --help @@ -3058,7 +3058,7 @@ step Most Miller commands are record-at-a-time, with the exception of ``stats1``, ``stats2``, and ``histogram`` which compute aggregate output. The ``step`` command is intermediate: it allows the option of adding fields which are functions of fields from previous records. Rsum is short for *running sum*. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint step -a shift,delta,rsum,counter -f x data/medium | head -15 @@ -3078,7 +3078,7 @@ Most Miller commands are record-at-a-time, with the exception of ``stats1``, ``s eks pan 13 0.4915175580479536 0.7709126592971468 0.3676141320555616 0.123903 6.188474 13 eks zee 14 0.5207382318405251 0.34141681118811673 0.4915175580479536 0.029221 6.709213 14 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint step -a shift,delta,rsum,counter -f x -g a data/medium | head -15 @@ -3098,7 +3098,7 @@ Most Miller commands are record-at-a-time, with the exception of ``stats1``, ``s eks pan 13 0.4915175580479536 0.7709126592971468 0.6117840605678454 -0.120267 2.243381 4 eks zee 14 0.5207382318405251 0.34141681118811673 0.4915175580479536 0.029221 2.764119 5 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint step -a ewma -f x -d 0.1,0.9 data/medium | head -15 @@ -3119,7 +3119,7 @@ Most Miller commands are record-at-a-time, with the exception of ``stats1``, ``s eks zee 14 0.5207382318405251 0.34141681118811673 0.454003 0.516969 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint step -a ewma -f x -d 0.1,0.9 -o smooth,rough data/medium | head -15 @@ -3142,7 +3142,7 @@ Most Miller commands are record-at-a-time, with the exception of ``stats1``, ``s Example deriving uptime-delta from system uptime: -.. code-block:: bash +.. code-block:: none $ each 10 uptime | mlr -p step -a delta -f 11 ... @@ -3161,7 +3161,7 @@ Example deriving uptime-delta from system uptime: tac ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr tac --help @@ -3170,7 +3170,7 @@ tac Prints the records in the input stream in reverse order. Note: this requires Miller to retain all input records in memory before any output records are produced. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint cat data/a.csv @@ -3178,14 +3178,14 @@ Prints the records in the input stream in reverse order. Note: this requires Mil 1 2 3 4 5 6 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint cat data/b.csv a b c 7 8 9 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint tac data/a.csv data/b.csv @@ -3194,7 +3194,7 @@ Prints the records in the input stream in reverse order. Note: this requires Mil 4 5 6 1 2 3 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint put '$filename=FILENAME' then tac data/a.csv data/b.csv @@ -3208,7 +3208,7 @@ Prints the records in the input stream in reverse order. Note: this requires Mil tail ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr tail --help @@ -3219,7 +3219,7 @@ tail Prints the last *n* records in the input stream, optionally by category. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint tail -n 4 data/colored-shapes.dkvp @@ -3229,7 +3229,7 @@ Prints the last *n* records in the input stream, optionally by category. yellow triangle 0 99990 0.3839424618160777 0.55952913620132 0.5113763011485609 4.307973891915119 yellow circle 1 99994 0.764950884927175 0.25284227383991364 0.49969878539567425 5.013809741826425 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint tail -n 1 -g shape data/colored-shapes.dkvp @@ -3243,7 +3243,7 @@ Prints the last *n* records in the input stream, optionally by category. tee ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr tee --help @@ -3264,7 +3264,7 @@ tee top ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr top --help @@ -3283,7 +3283,7 @@ top Note that ``top`` is distinct from :ref:`reference-verbs-head` -- ``head`` shows fields which appear first in the data stream; ``top`` shows fields which are numerically largest (or smallest). -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint top -n 4 -f x data/medium @@ -3293,7 +3293,7 @@ Note that ``top`` is distinct from :ref:`reference-verbs-head` -- ``head`` shows 3 0.999733 4 0.999563 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint top -n 4 -f x -o someothername data/medium @@ -3303,7 +3303,7 @@ Note that ``top`` is distinct from :ref:`reference-verbs-head` -- ``head`` shows 3 0.999733 4 0.999563 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint top -n 2 -f x -g a then sort -f a data/medium @@ -3324,7 +3324,7 @@ Note that ``top`` is distinct from :ref:`reference-verbs-head` -- ``head`` shows uniq ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr uniq --help @@ -3344,13 +3344,13 @@ uniq There are two main ways to use ``mlr uniq``: the first way is with ``-g`` to specify group-by columns. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ wc -l data/colored-shapes.dkvp 10078 data/colored-shapes.dkvp -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr uniq -g color,shape data/colored-shapes.dkvp @@ -3373,7 +3373,7 @@ There are two main ways to use ``mlr uniq``: the first way is with ``-g`` to spe color=orange,shape=square color=orange,shape=circle -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint uniq -g color,shape -c then sort -f color,shape data/colored-shapes.dkvp @@ -3397,7 +3397,7 @@ There are two main ways to use ``mlr uniq``: the first way is with ``-g`` to spe yellow square 589 yellow triangle 468 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint uniq -g color,shape -c -o someothername then sort -nr someothername data/colored-shapes.dkvp @@ -3421,7 +3421,7 @@ There are two main ways to use ``mlr uniq``: the first way is with ``-g`` to spe orange triangle 107 orange circle 68 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint uniq -n -g color,shape data/colored-shapes.dkvp @@ -3430,7 +3430,7 @@ There are two main ways to use ``mlr uniq``: the first way is with ``-g`` to spe The second main way to use ``mlr uniq`` is without group-by columns, using ``-a`` instead: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/repeats.dkvp @@ -3492,13 +3492,13 @@ The second main way to use ``mlr uniq`` is without group-by columns, using ``-a` color=yellow,shape=circle,flag=1 color=purple,shape=square,flag=0 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ wc -l data/repeats.dkvp 57 data/repeats.dkvp -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint uniq -a data/repeats.dkvp @@ -3511,14 +3511,14 @@ The second main way to use ``mlr uniq`` is without group-by columns, using ``-a` red square 1 yellow triangle 1 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint uniq -a -n data/repeats.dkvp count 7 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint uniq -a -c data/repeats.dkvp @@ -3536,7 +3536,7 @@ The second main way to use ``mlr uniq`` is without group-by columns, using ``-a` unsparsify ---------------------------------------------------------------- -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr unsparsify --help @@ -3557,7 +3557,7 @@ unsparsify Examples: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/sparse.json @@ -3566,7 +3566,7 @@ Examples: {"a":1,"v":2,"x":3} {"v":1,"w":2} -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --json unsparsify data/sparse.json @@ -3575,7 +3575,7 @@ Examples: { "a": 1, "b": "", "v": 2, "u": "", "x": 3, "w": "" } { "a": "", "b": "", "v": 1, "u": "", "x": "", "w": 2 } -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --ijson --opprint unsparsify data/sparse.json @@ -3585,7 +3585,7 @@ Examples: 1 - 2 - 3 - - - 1 - - 2 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --ijson --opprint unsparsify --fill-with missing data/sparse.json @@ -3595,7 +3595,7 @@ Examples: 1 missing 2 missing 3 missing missing missing 1 missing missing 2 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --ijson --opprint unsparsify -f a,b,u data/sparse.json @@ -3611,7 +3611,7 @@ Examples: v w a b u 1 2 - - - -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --ijson --opprint unsparsify -f a,b,u,v,w,x then regularize data/sparse.json diff --git a/docs/reference-verbs.rst.in b/docs/reference-verbs.rst.in index 53e013408..58ed328d3 100644 --- a/docs/reference-verbs.rst.in +++ b/docs/reference-verbs.rst.in @@ -69,7 +69,7 @@ POKI_RUN_COMMAND{{mlr bootstrap --help}}HERE The canonical use for bootstrap sampling is to put error bars on statistical quantities, such as mean. For example: -.. code-block:: bash +.. code-block:: none $ mlr --opprint stats1 -a mean,count -f u -g color data/colored-shapes.dkvp color u_mean u_count @@ -80,7 +80,7 @@ The canonical use for bootstrap sampling is to put error bars on statistical qua blue 0.517717 1470 orange 0.490532 303 -.. code-block:: bash +.. code-block:: none $ mlr --opprint bootstrap then stats1 -a mean,count -f u -g color data/colored-shapes.dkvp color u_mean u_count @@ -91,7 +91,7 @@ The canonical use for bootstrap sampling is to put error bars on statistical qua blue 0.512529 1496 orange 0.521030 321 -.. code-block:: bash +.. code-block:: none $ mlr --opprint bootstrap then stats1 -a mean,count -f u -g color data/colored-shapes.dkvp color u_mean u_count @@ -102,7 +102,7 @@ The canonical use for bootstrap sampling is to put error bars on statistical qua green 0.496803 1075 purple 0.486337 1199 -.. code-block:: bash +.. code-block:: none $ mlr --opprint bootstrap then stats1 -a mean,count -f u -g color data/colored-shapes.dkvp color u_mean u_count diff --git a/docs/reference.rst b/docs/reference.rst index e750cb3c8..60de70995 100644 --- a/docs/reference.rst +++ b/docs/reference.rst @@ -13,7 +13,7 @@ Command overview Whereas the Unix toolkit is made of the separate executables ``cat``, ``tail``, ``cut``, ``sort``, etc., Miller has subcommands, or **verbs**, invoked as follows: -.. code-block:: bash +.. code-block:: none mlr tac *.dat mlr cut --complement -f os_version *.dat @@ -39,7 +39,7 @@ Formats Options: -.. code-block:: bash +.. code-block:: none --dkvp --idkvp --odkvp --nidx --inidx --onidx @@ -51,7 +51,7 @@ Options: These are as discussed in :doc:`file-formats`, with the exception of ``--right`` which makes pretty-printed output right-aligned: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint cat data/small @@ -62,7 +62,7 @@ These are as discussed in :doc:`file-formats`, with the exception of ``--right`` eks wye 4 0.38139939387114097 0.13418874328430463 wye pan 5 0.5732889198020006 0.8636244699032729 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint --right cat data/small @@ -97,14 +97,14 @@ Compression Options: -.. code-block:: bash +.. code-block:: none --prepipe {command} The prepipe command is anything which reads from standard input and produces data acceptable to Miller. Nominally this allows you to use whichever decompression utilities you have installed on your system, on a per-file basis. If the command has flags, quote them: e.g. ``mlr --prepipe 'zcat -cf'``. Examples: -.. code-block:: bash +.. code-block:: none # These two produce the same output: $ gunzip < myfile1.csv.gz | mlr cut -f hostname,uptime @@ -113,14 +113,14 @@ The prepipe command is anything which reads from standard input and produces dat $ mlr --prepipe gunzip cut -f hostname,uptime myfile1.csv.gz myfile2.csv.gz $ mlr --prepipe gunzip --idkvp --oxtab cut -f hostname,uptime myfile1.dat.gz myfile2.dat.gz -.. code-block:: bash +.. code-block:: none # Similar to the above, but with compressed output as well as input: $ gunzip < myfile1.csv.gz | mlr cut -f hostname,uptime | gzip > outfile.csv.gz $ mlr --prepipe gunzip cut -f hostname,uptime myfile1.csv.gz | gzip > outfile.csv.gz $ mlr --prepipe gunzip cut -f hostname,uptime myfile1.csv.gz myfile2.csv.gz | gzip > outfile.csv.gz -.. code-block:: bash +.. code-block:: none # Similar to the above, but with different compression tools for input and output: $ gunzip < myfile1.csv.gz | mlr cut -f hostname,uptime | xz -z > outfile.csv.xz @@ -136,7 +136,7 @@ Miller has record separators ``IRS`` and ``ORS``, field separators ``IFS`` and ` Options: -.. code-block:: bash +.. code-block:: none --rs --irs --ors --fs --ifs --ofs --repifs @@ -157,7 +157,7 @@ Number formatting The command-line option ``--ofmt {format string}`` is the global number format for commands which generate numeric output, e.g. ``stats1``, ``stats2``, ``histogram``, and ``step``, as well as ``mlr put``. Examples: -.. code-block:: bash +.. code-block:: none --ofmt %.9le --ofmt %.6lf --ofmt %.0lf @@ -165,13 +165,13 @@ These are just C ``printf`` formats applied to double-precision numbers. Please To apply formatting to a single field, overriding the global ``ofmt``, use ``fmtnum`` function within ``mlr put``. For example: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ echo 'x=3.1,y=4.3' | mlr put '$z=fmtnum($x*$y,"%08lf")' x=3.1,y=4.3,z=13.330000 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ echo 'x=0xffff,y=0xff' | mlr put '$z=fmtnum(int($x*$y),"%08llx")' @@ -179,7 +179,7 @@ To apply formatting to a single field, overriding the global ``ofmt``, use ``fmt Input conversion from hexadecimal is done automatically on fields handled by ``mlr put`` and ``mlr filter`` as long as the field value begins with "0x". To apply output conversion to hexadecimal on a single column, you may use ``fmtnum``, or the keystroke-saving ``hexfmt`` function. Example: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ echo 'x=0xffff,y=0xff' | mlr put '$z=hexfmt($x*$y)' @@ -200,13 +200,13 @@ then-chaining In accord with the `Unix philosophy `_, you can pipe data into or out of Miller. For example: -.. code-block:: bash +.. code-block:: none mlr cut --complement -f os_version *.dat | mlr sort -f hostname,uptime You can, if you like, instead simply chain commands together using the ``then`` keyword: -.. code-block:: bash +.. code-block:: none mlr cut --complement -f os_version then sort -f hostname,uptime *.dat @@ -214,7 +214,7 @@ You can, if you like, instead simply chain commands together using the ``then`` Here's a performance comparison: -.. code-block:: bash +.. code-block:: none % cat piped.sh mlr cut -x -f i,y data/big | mlr sort -n y > /dev/null @@ -242,7 +242,7 @@ Auxiliary commands There are a few nearly-standalone programs which have nothing to do with the rest of Miller, do not participate in record streams, and do not deal with file formats. They might as well be little standalone executables but they're delivered within the main Miller executable for convenience. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr aux-list @@ -255,7 +255,7 @@ There are a few nearly-standalone programs which have nothing to do with the res netbsd-strptime For more information, please invoke mlr {subcommand} --help -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr lecat --help @@ -266,7 +266,7 @@ There are a few nearly-standalone programs which have nothing to do with the res --mono: don't try to colorize the output -h or --help: print this message -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr termcvt --help @@ -283,7 +283,7 @@ There are a few nearly-standalone programs which have nothing to do with the res Zero file names means read from standard input. Output is always to standard output; files are not written in-place. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr hex --help @@ -294,7 +294,7 @@ There are a few nearly-standalone programs which have nothing to do with the res -r: print only raw hex without leading offset indicators or trailing ASCII dump. -h or --help: print this message -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr unhex --help @@ -306,19 +306,19 @@ There are a few nearly-standalone programs which have nothing to do with the res Examples: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ echo 'Hello, world!' | mlr lecat --mono Hello, world![LF] -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ echo 'Hello, world!' | mlr termcvt --lf2crlf | mlr lecat --mono Hello, world![CR][LF] -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr hex data/budget.csv @@ -330,7 +330,7 @@ Examples: 00000050: 38 0a 67 72 65 65 6e 2c 36 37 38 2e 31 32 0a 6f |8.green,678.12.o| 00000060: 72 61 6e 67 65 2c 31 32 33 2e 34 35 0a |range,123.45.| -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr hex -r data/budget.csv @@ -342,7 +342,7 @@ Examples: 38 0a 67 72 65 65 6e 2c 36 37 38 2e 31 32 0a 6f 72 61 6e 67 65 2c 31 32 33 2e 34 35 0a -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr hex -r data/budget.csv | sed 's/20/2a/g' | mlr unhex @@ -394,7 +394,7 @@ Rules for null-handling: * Records with one or more empty sort-field values sort after records with all sort-field values present: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr cat data/sort-null.dat @@ -404,7 +404,7 @@ Rules for null-handling: x=9,b=10 a=5,b=7 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr sort -n a data/sort-null.dat @@ -414,7 +414,7 @@ Rules for null-handling: a=,b=4 x=9,b=10 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr sort -nr a data/sort-null.dat @@ -426,19 +426,19 @@ Rules for null-handling: * Functions/operators which have one or more *empty* arguments produce empty output: e.g. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ echo 'x=2,y=3' | mlr put '$a=$x+$y' x=2,y=3,a=5 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ echo 'x=,y=3' | mlr put '$a=$x+$y' x=,y=3,a= -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ echo 'x=,y=3' | mlr put '$a=log($x);$b=log($y)' @@ -446,7 +446,7 @@ Rules for null-handling: with the exception that the ``min`` and ``max`` functions are special: if one argument is non-null, it wins: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ echo 'x=,y=3' | mlr put '$a=min($x,$y);$b=max($x,$y)' @@ -454,13 +454,13 @@ with the exception that the ``min`` and ``max`` functions are special: if one ar * Functions of *absent* variables (e.g. ``mlr put '$y = log10($nonesuch)'``) evaluate to absent, and arithmetic/bitwise/boolean operators with both operands being absent evaluate to absent. Arithmetic operators with one absent operand return the other operand. More specifically, absent values act like zero for addition/subtraction, and one for multiplication: Furthermore, **any expression which evaluates to absent is not stored in the left-hand side of an assignment statement**: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ echo 'x=2,y=3' | mlr put '$a=$u+$v; $b=$u+$y; $c=$x+$y' x=2,y=3,b=3,c=5 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ echo 'x=2,y=3' | mlr put '$a=min($x,$v);$b=max($u,$y);$c=min($u,$v)' @@ -480,7 +480,7 @@ The reasoning is as follows: Since absent plus absent is absent (and likewise for other operators), accumulations such as ``@sum += $x`` work correctly on heterogenous data, as do within-record formulas if both operands are absent. If one operand is present, you may get behavior you don't desire. To work around this -- namely, to set an output field only for records which have all the inputs present -- you can use a pattern-action block with ``is_present``: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr cat data/het.dkvp @@ -490,7 +490,7 @@ Since absent plus absent is absent (and likewise for other operators), accumulat record_count=150,resource=/path/to/second/file resource=/some/other/path,loadsec=0.97,ok=false -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put 'is_present($loadsec) { $loadmillis = $loadsec * 1000 }' data/het.dkvp @@ -500,7 +500,7 @@ Since absent plus absent is absent (and likewise for other operators), accumulat record_count=150,resource=/path/to/second/file resource=/some/other/path,loadsec=0.97,ok=false,loadmillis=970.000000 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr put '$loadmillis = (is_present($loadsec) ? $loadsec : 0.0) * 1000' data/het.dkvp @@ -512,7 +512,7 @@ Since absent plus absent is absent (and likewise for other operators), accumulat If you're interested in a formal description of how empty and absent fields participate in arithmetic, here's a table for plus (other arithmetic/boolean/bitwise operators are similar): -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --print-type-arithmetic-info @@ -580,7 +580,7 @@ For ``filter`` and ``put``, if the regular expression is a string literal (the n Example: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ cat data/regex-in-data.dat @@ -588,7 +588,7 @@ Example: name=bill,regex=^b[ou]ll$ name=bull,regex=^b[ou]ll$ -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr filter '$name =~ $regex' data/regex-in-data.dat @@ -602,25 +602,25 @@ Regex captures of the form ``\0`` through ``\9`` are supported as * Captures have in-function context for ``sub`` and ``gsub``. For example, the first ``\1,\2`` pair belong to the first ``sub`` and the second ``\1,\2`` pair belong to the second ``sub``: -.. code-block:: bash +.. code-block:: none mlr put '$b = sub($a, "(..)_(...)", "\2-\1"); $c = sub($a, "(..)_(.)(..)", ":\1:\2:\3")' * Captures endure for the entirety of a ``put`` for the ``=~`` and ``!=~`` operators. For example, here the ``\1,\2`` are set by the ``=~`` operator and are used by both subsequent assignment statements: -.. code-block:: bash +.. code-block:: none mlr put '$a =~ "(..)_(....); $b = "left_\1"; $c = "right_\2"' * The captures are not retained across multiple puts. For example, here the ``\1,\2`` won't be expanded from the regex capture: -.. code-block:: bash +.. code-block:: none mlr put '$a =~ "(..)_(....)' then {... something else ...} then put '$b = "left_\1"; $c = "right_\2"' * Captures are ignored in ``filter`` for the ``=~`` and ``!=~`` operators. For example, there is no mechanism provided to refer to the first ``(..)`` as ``\1`` or to the second ``(....)`` as ``\2`` in the following filter statement: -.. code-block:: bash +.. code-block:: none mlr filter '$a =~ "(..)_(....)' @@ -650,7 +650,7 @@ The short of it is that Miller does this transparently for you so you needn't th Implementation details of this, for the interested: integer adds and subtracts overflow by at most one bit so it suffices to check sign-changes. Thus, Miller allows you to add and subtract arbitrary 64-bit signed integers, converting only to float precisely when the result is less than -2\ :sup:`63` or greater than 2\ :sup:`63`\ -1. Multiplies, on the other hand, can overflow by a word size and a sign-change technique does not suffice to detect overflow. Instead Miller tests whether the floating-point product exceeds the representable integer range. Now, 64-bit integers have 64-bit precision while IEEE-doubles have only 52-bit mantissas -- so, there are 53 bits including implicit leading one. The following experiment explicitly demonstrates the resolution at this range: -.. code-block:: bash +.. code-block:: none 64-bit integer 64-bit integer Casted to double Back to 64-bit in hex in decimal integer @@ -679,7 +679,7 @@ On-line help Examples: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --help @@ -1105,7 +1105,7 @@ Examples: For more information please see http://johnkerl.org/miller/doc and/or http://github.com/johnkerl/miller. This is Miller version v5.10.2-dev. -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr sort --help diff --git a/docs/reference.rst.in b/docs/reference.rst.in index 00243d3b6..0dfc7aa71 100644 --- a/docs/reference.rst.in +++ b/docs/reference.rst.in @@ -32,7 +32,7 @@ Formats Options: -.. code-block:: bash +.. code-block:: none --dkvp --idkvp --odkvp --nidx --inidx --onidx @@ -72,14 +72,14 @@ Compression Options: -.. code-block:: bash +.. code-block:: none --prepipe {command} The prepipe command is anything which reads from standard input and produces data acceptable to Miller. Nominally this allows you to use whichever decompression utilities you have installed on your system, on a per-file basis. If the command has flags, quote them: e.g. ``mlr --prepipe 'zcat -cf'``. Examples: -.. code-block:: bash +.. code-block:: none # These two produce the same output: $ gunzip < myfile1.csv.gz | mlr cut -f hostname,uptime @@ -88,14 +88,14 @@ The prepipe command is anything which reads from standard input and produces dat $ mlr --prepipe gunzip cut -f hostname,uptime myfile1.csv.gz myfile2.csv.gz $ mlr --prepipe gunzip --idkvp --oxtab cut -f hostname,uptime myfile1.dat.gz myfile2.dat.gz -.. code-block:: bash +.. code-block:: none # Similar to the above, but with compressed output as well as input: $ gunzip < myfile1.csv.gz | mlr cut -f hostname,uptime | gzip > outfile.csv.gz $ mlr --prepipe gunzip cut -f hostname,uptime myfile1.csv.gz | gzip > outfile.csv.gz $ mlr --prepipe gunzip cut -f hostname,uptime myfile1.csv.gz myfile2.csv.gz | gzip > outfile.csv.gz -.. code-block:: bash +.. code-block:: none # Similar to the above, but with different compression tools for input and output: $ gunzip < myfile1.csv.gz | mlr cut -f hostname,uptime | xz -z > outfile.csv.xz @@ -111,7 +111,7 @@ Miller has record separators ``IRS`` and ``ORS``, field separators ``IFS`` and ` Options: -.. code-block:: bash +.. code-block:: none --rs --irs --ors --fs --ifs --ofs --repifs @@ -132,7 +132,7 @@ Number formatting The command-line option ``--ofmt {format string}`` is the global number format for commands which generate numeric output, e.g. ``stats1``, ``stats2``, ``histogram``, and ``step``, as well as ``mlr put``. Examples: -.. code-block:: bash +.. code-block:: none --ofmt %.9le --ofmt %.6lf --ofmt %.0lf @@ -163,13 +163,13 @@ then-chaining In accord with the `Unix philosophy `_, you can pipe data into or out of Miller. For example: -.. code-block:: bash +.. code-block:: none mlr cut --complement -f os_version *.dat | mlr sort -f hostname,uptime You can, if you like, instead simply chain commands together using the ``then`` keyword: -.. code-block:: bash +.. code-block:: none mlr cut --complement -f os_version then sort -f hostname,uptime *.dat @@ -365,25 +365,25 @@ Regex captures of the form ``\0`` through ``\9`` are supported as * Captures have in-function context for ``sub`` and ``gsub``. For example, the first ``\1,\2`` pair belong to the first ``sub`` and the second ``\1,\2`` pair belong to the second ``sub``: -.. code-block:: bash +.. code-block:: none mlr put '$b = sub($a, "(..)_(...)", "\2-\1"); $c = sub($a, "(..)_(.)(..)", ":\1:\2:\3")' * Captures endure for the entirety of a ``put`` for the ``=~`` and ``!=~`` operators. For example, here the ``\1,\2`` are set by the ``=~`` operator and are used by both subsequent assignment statements: -.. code-block:: bash +.. code-block:: none mlr put '$a =~ "(..)_(....); $b = "left_\1"; $c = "right_\2"' * The captures are not retained across multiple puts. For example, here the ``\1,\2`` won't be expanded from the regex capture: -.. code-block:: bash +.. code-block:: none mlr put '$a =~ "(..)_(....)' then {... something else ...} then put '$b = "left_\1"; $c = "right_\2"' * Captures are ignored in ``filter`` for the ``=~`` and ``!=~`` operators. For example, there is no mechanism provided to refer to the first ``(..)`` as ``\1`` or to the second ``(....)`` as ``\2`` in the following filter statement: -.. code-block:: bash +.. code-block:: none mlr filter '$a =~ "(..)_(....)' @@ -413,7 +413,7 @@ The short of it is that Miller does this transparently for you so you needn't th Implementation details of this, for the interested: integer adds and subtracts overflow by at most one bit so it suffices to check sign-changes. Thus, Miller allows you to add and subtract arbitrary 64-bit signed integers, converting only to float precisely when the result is less than -2\ :sup:`63` or greater than 2\ :sup:`63`\ -1. Multiplies, on the other hand, can overflow by a word size and a sign-change technique does not suffice to detect overflow. Instead Miller tests whether the floating-point product exceeds the representable integer range. Now, 64-bit integers have 64-bit precision while IEEE-doubles have only 52-bit mantissas -- so, there are 53 bits including implicit leading one. The following experiment explicitly demonstrates the resolution at this range: -.. code-block:: bash +.. code-block:: none 64-bit integer 64-bit integer Casted to double Back to 64-bit in hex in decimal integer diff --git a/docs/sql-examples.rst b/docs/sql-examples.rst index 8ce05164f..60c6adef8 100644 --- a/docs/sql-examples.rst +++ b/docs/sql-examples.rst @@ -13,7 +13,7 @@ I like to produce SQL-query output with header-column and tab delimiter: this is For example, using default output formatting in ``mysql`` we get formatting like Miller's ``--opprint --barred``: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mysql --database=mydb -e 'show columns in mytable' @@ -29,7 +29,7 @@ For example, using default output formatting in ``mysql`` we get formatting like Using ``mysql``'s ``-B`` we get TSV output: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mysql --database=mydb -B -e 'show columns in mytable' | mlr --itsvlite --opprint cat @@ -42,7 +42,7 @@ Using ``mysql``'s ``-B`` we get TSV output: Since Miller handles TSV output, we can do as much or as little processing as we want in the SQL query, then send the rest on to Miller. This includes outputting as JSON, doing further selects/joins in Miller, doing stats, etc. etc.: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mysql --database=mydb -B -e 'show columns in mytable' | mlr --itsvlite --ojson --jlistwrap --jvstack cat @@ -89,12 +89,12 @@ Since Miller handles TSV output, we can do as much or as little processing as we } ] -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mysql --database=mydb -B -e 'select * from mytable' > query.tsv -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from query.tsv --t2p stats1 -a count -f id -g category,assigned_to @@ -118,7 +118,7 @@ One use of NIDX (value-only, no keys) format is for loading up SQL tables. Create and load SQL table: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 mysql> CREATE TABLE abixy( @@ -130,19 +130,19 @@ Create and load SQL table: ); Query OK, 0 rows affected (0.01 sec) -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 bash$ mlr --onidx --fs comma cat data/medium > medium.nidx -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 mysql> LOAD DATA LOCAL INFILE 'medium.nidx' REPLACE INTO TABLE abixy FIELDS TERMINATED BY ',' ; Query OK, 10000 rows affected (0.07 sec) Records: 10000 Deleted: 0 Skipped: 0 Warnings: 0 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 mysql> SELECT COUNT(*) AS count FROM abixy; @@ -153,7 +153,7 @@ Create and load SQL table: +-------+ 1 row in set (0.00 sec) -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 mysql> SELECT * FROM abixy LIMIT 10; @@ -174,7 +174,7 @@ Create and load SQL table: Aggregate counts within SQL: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 mysql> SELECT a, b, COUNT(*) AS count FROM abixy GROUP BY a, b ORDER BY COUNT DESC; @@ -211,7 +211,7 @@ Aggregate counts within SQL: Aggregate counts within Miller: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint uniq -c -g a,b then sort -nr count data/medium @@ -234,7 +234,7 @@ Aggregate counts within Miller: Pipe SQL output to aggregate counts within Miller: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mysql -D miller -B -e 'select * from abixy' | mlr --itsv --opprint uniq -c -g a,b then sort -nr count diff --git a/docs/sql-examples.rst.in b/docs/sql-examples.rst.in index 0427ecb89..5487b60e9 100644 --- a/docs/sql-examples.rst.in +++ b/docs/sql-examples.rst.in @@ -10,7 +10,7 @@ I like to produce SQL-query output with header-column and tab delimiter: this is For example, using default output formatting in ``mysql`` we get formatting like Miller's ``--opprint --barred``: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mysql --database=mydb -e 'show columns in mytable' @@ -26,7 +26,7 @@ For example, using default output formatting in ``mysql`` we get formatting like Using ``mysql``'s ``-B`` we get TSV output: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mysql --database=mydb -B -e 'show columns in mytable' | mlr --itsvlite --opprint cat @@ -39,7 +39,7 @@ Using ``mysql``'s ``-B`` we get TSV output: Since Miller handles TSV output, we can do as much or as little processing as we want in the SQL query, then send the rest on to Miller. This includes outputting as JSON, doing further selects/joins in Miller, doing stats, etc. etc.: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mysql --database=mydb -B -e 'show columns in mytable' | mlr --itsvlite --ojson --jlistwrap --jvstack cat @@ -86,12 +86,12 @@ Since Miller handles TSV output, we can do as much or as little processing as we } ] -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mysql --database=mydb -B -e 'select * from mytable' > query.tsv -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from query.tsv --t2p stats1 -a count -f id -g category,assigned_to @@ -115,7 +115,7 @@ One use of NIDX (value-only, no keys) format is for loading up SQL tables. Create and load SQL table: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 mysql> CREATE TABLE abixy( @@ -127,19 +127,19 @@ Create and load SQL table: ); Query OK, 0 rows affected (0.01 sec) -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 bash$ mlr --onidx --fs comma cat data/medium > medium.nidx -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 mysql> LOAD DATA LOCAL INFILE 'medium.nidx' REPLACE INTO TABLE abixy FIELDS TERMINATED BY ',' ; Query OK, 10000 rows affected (0.07 sec) Records: 10000 Deleted: 0 Skipped: 0 Warnings: 0 -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 mysql> SELECT COUNT(*) AS count FROM abixy; @@ -150,7 +150,7 @@ Create and load SQL table: +-------+ 1 row in set (0.00 sec) -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 mysql> SELECT * FROM abixy LIMIT 10; @@ -171,7 +171,7 @@ Create and load SQL table: Aggregate counts within SQL: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 mysql> SELECT a, b, COUNT(*) AS count FROM abixy GROUP BY a, b ORDER BY COUNT DESC; @@ -208,7 +208,7 @@ Aggregate counts within SQL: Aggregate counts within Miller: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint uniq -c -g a,b then sort -nr count data/medium @@ -231,7 +231,7 @@ Aggregate counts within Miller: Pipe SQL output to aggregate counts within Miller: -.. code-block:: bash +.. code-block:: none :emphasize-lines: 1,1 $ mysql -D miller -B -e 'select * from abixy' | mlr --itsv --opprint uniq -c -g a,b then sort -nr count diff --git a/docs6/10min.rst b/docs6/10min.rst index 5c0c29424..01215b9d9 100644 --- a/docs6/10min.rst +++ b/docs6/10min.rst @@ -9,7 +9,7 @@ CSV-file examples Suppose you have this CSV data file: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat example.csv @@ -27,7 +27,7 @@ Suppose you have this CSV data file: ``mlr cat`` is like cat -- it passes the data through unmodified: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv cat example.csv @@ -45,7 +45,7 @@ Suppose you have this CSV data file: but it can also do format conversion (here, you can pretty-print in tabular format): -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint cat example.csv @@ -63,7 +63,7 @@ but it can also do format conversion (here, you can pretty-print in tabular form ``mlr head`` and ``mlr tail`` count records rather than lines. Whether you're getting the first few records or the last few, the CSV header is included either way: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv head -n 4 example.csv @@ -73,7 +73,7 @@ but it can also do format conversion (here, you can pretty-print in tabular form red,circle,1,16,13.8103,2.9010 red,square,0,48,77.5542,7.4670 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv tail -n 4 example.csv @@ -85,7 +85,7 @@ but it can also do format conversion (here, you can pretty-print in tabular form You can sort primarily alphabetically on one field, then secondarily numerically descending on another field: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint sort -f shape -nr index example.csv @@ -103,7 +103,7 @@ You can sort primarily alphabetically on one field, then secondarily numerically You can use ``cut`` to retain only specified fields, in the same order they appeared in the input data: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint cut -f flag,shape example.csv @@ -121,7 +121,7 @@ You can use ``cut`` to retain only specified fields, in the same order they appe You can also use ``cut -o`` to retain only specified fields in your preferred order: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint cut -o -f flag,shape example.csv @@ -139,7 +139,7 @@ You can also use ``cut -o`` to retain only specified fields in your preferred or You can use ``cut -x`` to omit fields you don't care about: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint cut -x -f flag,shape example.csv @@ -157,7 +157,7 @@ You can use ``cut -x`` to omit fields you don't care about: You can use ``filter`` to keep only records you care about: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint filter '$color == "red"' example.csv @@ -167,7 +167,7 @@ You can use ``filter`` to keep only records you care about: red square 0 48 77.5542 7.4670 red square 0 64 77.1991 9.5310 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint filter '$color == "red" && $flag == 1' example.csv @@ -177,7 +177,7 @@ You can use ``filter`` to keep only records you care about: You can use ``put`` to create new fields which are computed from other fields: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint put '$ratio = $quantity / $rate; $color_shape = $color . "_" . $shape' example.csv @@ -195,7 +195,7 @@ You can use ``put`` to create new fields which are computed from other fields: Even though Miller's main selling point is name-indexing, sometimes you really want to refer to a field name by its positional index. Use ``$[[3]]`` to access the name of field 3 or ``$[[[3]]]`` to access the value of field 3: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint put '$[[3]] = "NEW"' example.csv @@ -211,7 +211,7 @@ Even though Miller's main selling point is name-indexing, sometimes you really w yellow circle 1 87 63.5058 8.3350 purple square 0 91 72.3735 8.2430 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint put '$[[[3]]] = "NEW"' example.csv @@ -232,7 +232,7 @@ JSON-file examples OK, CSV and pretty-print are fine. But Miller can also convert between a few other formats -- let's take a look at JSON output: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --ojson put '$ratio = $quantity/$rate; $shape = toupper($shape)' example.csv @@ -254,7 +254,7 @@ Now suppose you want to sort the data on a given column, *and then* take the top Here are the records with the top three ``index`` values: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint sort -f shape -nr index then head -n 3 example.csv @@ -265,7 +265,7 @@ Here are the records with the top three ``index`` values: Lots of Miller commands take a ``-g`` option for group-by: here, ``head -n 1 -g shape`` outputs the first record for each distinct value of the ``shape`` field. This means we're finding the record with highest ``index`` field for each distinct ``shape`` field: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint sort -f shape -nr index then head -n 1 -g shape example.csv @@ -276,7 +276,7 @@ Lots of Miller commands take a ``-g`` option for group-by: here, ``head -n 1 -g Statistics can be computed with or without group-by field(s): -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint --from example.csv stats1 -a count,min,mean,max -f quantity -g shape @@ -285,7 +285,7 @@ Statistics can be computed with or without group-by field(s): square 4 72.373500 76.601150 79.277800 circle 3 13.810300 47.098200 63.978500 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint --from example.csv stats1 -a count,min,mean,max -f quantity -g shape,color @@ -299,7 +299,7 @@ Statistics can be computed with or without group-by field(s): If your output has a lot of columns, you can use XTAB format to line things up vertically for you instead: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --oxtab --from example.csv stats1 -a p0,p10,p25,p50,p75,p90,p99,p100 -f rate @@ -321,14 +321,14 @@ Often we want to print output to the screen. Miller does this by default, as we' Sometimes we want to print output to another file: just use **> outputfilenamegoeshere** at the end of your command: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 % mlr --icsv --opprint cat example.csv > newfile.csv # Output goes to the new file; # nothing is printed to the screen. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 % cat newfile.csv @@ -347,12 +347,12 @@ Sometimes we want to print output to another file: just use **> outputfilenamego Other times we just want our files to be **changed in-place**: just use **mlr -I**: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 % cp example.csv newfile.txt -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 % cat newfile.txt @@ -368,12 +368,12 @@ Other times we just want our files to be **changed in-place**: just use **mlr -I yellow,circle,1,87,63.5058,8.3350 purple,square,0,91,72.3735,8.2430 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 % mlr -I --icsv --opprint cat newfile.txt -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 % cat newfile.txt @@ -391,7 +391,7 @@ Other times we just want our files to be **changed in-place**: just use **mlr -I Also using ``mlr -I`` you can bulk-operate on lots of files: e.g.: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 mlr -I --csv cut -x -f unwanted_column_name *.csv @@ -400,12 +400,12 @@ If you like, you can first copy off your original data somewhere else, before do Lastly, using ``tee`` within ``put``, you can split your input data into separate files per one or more field names: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv --from example.csv put -q 'tee > $shape.".csv", $*' -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat circle.csv @@ -414,7 +414,7 @@ Lastly, using ``tee`` within ``put``, you can split your input data into separat yellow,circle,1,73,63.9785,4.2370 yellow,circle,1,87,63.5058,8.3350 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat square.csv @@ -424,7 +424,7 @@ Lastly, using ``tee`` within ``put``, you can split your input data into separat red,square,0,64,77.1991,9.5310 purple,square,0,91,72.3735,8.2430 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat triangle.csv @@ -440,7 +440,7 @@ What's a CSV file, really? It's an array of rows, or *records*, each being a lis For example, if you have: -.. code-block:: +.. code-block:: none shape,flag,index circle,1,24 @@ -448,7 +448,7 @@ For example, if you have: then that's a way of saying: -.. code-block:: +.. code-block:: none shape=circle,flag=1,index=24 shape=square,flag=0,index=36 @@ -458,7 +458,7 @@ Data written this way are called **DKVP**, for *delimited key-value pairs*. We've also already seen other ways to write the same data: -.. code-block:: +.. code-block:: none CSV PPRINT JSON shape,flag,index shape flag index [ diff --git a/docs6/10min.rst.in b/docs6/10min.rst.in index 00c01c0ff..b12c048f1 100644 --- a/docs6/10min.rst.in +++ b/docs6/10min.rst.in @@ -93,14 +93,14 @@ Often we want to print output to the screen. Miller does this by default, as we' Sometimes we want to print output to another file: just use **> outputfilenamegoeshere** at the end of your command: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 % mlr --icsv --opprint cat example.csv > newfile.csv # Output goes to the new file; # nothing is printed to the screen. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 % cat newfile.csv @@ -119,12 +119,12 @@ Sometimes we want to print output to another file: just use **> outputfilenamego Other times we just want our files to be **changed in-place**: just use **mlr -I**: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 % cp example.csv newfile.txt -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 % cat newfile.txt @@ -140,12 +140,12 @@ Other times we just want our files to be **changed in-place**: just use **mlr -I yellow,circle,1,87,63.5058,8.3350 purple,square,0,91,72.3735,8.2430 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 % mlr -I --icsv --opprint cat newfile.txt -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 % cat newfile.txt @@ -163,7 +163,7 @@ Other times we just want our files to be **changed in-place**: just use **mlr -I Also using ``mlr -I`` you can bulk-operate on lots of files: e.g.: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 mlr -I --csv cut -x -f unwanted_column_name *.csv @@ -187,7 +187,7 @@ What's a CSV file, really? It's an array of rows, or *records*, each being a lis For example, if you have: -.. code-block:: +.. code-block:: none shape,flag,index circle,1,24 @@ -195,7 +195,7 @@ For example, if you have: then that's a way of saying: -.. code-block:: +.. code-block:: none shape=circle,flag=1,index=24 shape=square,flag=0,index=36 @@ -205,7 +205,7 @@ Data written this way are called **DKVP**, for *delimited key-value pairs*. We've also already seen other ways to write the same data: -.. code-block:: +.. code-block:: none CSV PPRINT JSON shape,flag,index shape flag index [ diff --git a/docs6/build.rst b/docs6/build.rst index b549f6cce..474a84ffc 100644 --- a/docs6/build.rst +++ b/docs6/build.rst @@ -101,7 +101,7 @@ In this example I am using version 3.4.0; of course that will change for subsequ * Similarly for ``macports``: https://github.com/macports/macports-ports/blob/master/textproc/miller/Portfile. * Social-media updates. -.. code-block:: +.. code-block:: none git remote add upstream https://github.com/Homebrew/homebrew-core # one-time setup only git fetch upstream diff --git a/docs6/build.rst.in b/docs6/build.rst.in index c460a7331..13b80e733 100644 --- a/docs6/build.rst.in +++ b/docs6/build.rst.in @@ -98,7 +98,7 @@ In this example I am using version 3.4.0; of course that will change for subsequ * Similarly for ``macports``: https://github.com/macports/macports-ports/blob/master/textproc/miller/Portfile. * Social-media updates. -.. code-block:: +.. code-block:: none git remote add upstream https://github.com/Homebrew/homebrew-core # one-time setup only git fetch upstream diff --git a/docs6/cookbook.rst b/docs6/cookbook.rst index 49e659525..afb6510c3 100644 --- a/docs6/cookbook.rst +++ b/docs6/cookbook.rst @@ -9,7 +9,7 @@ Headerless CSV on input or output Sometimes we get CSV files which lack a header. For example: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/headerless.csv @@ -20,7 +20,7 @@ Sometimes we get CSV files which lack a header. For example: You can use Miller to add a header. The ``--implicit-csv-header`` applies positionally indexed labels: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv --implicit-csv-header cat data/headerless.csv @@ -32,7 +32,7 @@ You can use Miller to add a header. The ``--implicit-csv-header`` applies positi Following that, you can rename the positionally indexed labels to names with meaning for your context. For example: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv --implicit-csv-header label name,age,status data/headerless.csv @@ -44,7 +44,7 @@ Following that, you can rename the positionally indexed labels to names with mea Likewise, if you need to produce CSV which is lacking its header, you can pipe Miller's output to the system command ``sed 1d``, or you can use Miller's ``--headerless-csv-output`` option: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ head -5 data/colored-shapes.dkvp | mlr --ocsv cat @@ -55,7 +55,7 @@ Likewise, if you need to produce CSV which is lacking its header, you can pipe M red,square,0,48,0.9562743938458542,0.7467203085342884,0.7755423050923582,7.117831369597269 purple,triangle,0,51,0.4355354501763202,0.8591292672156728,0.8122903963006748,5.753094629505863 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ head -5 data/colored-shapes.dkvp | mlr --ocsv --headerless-csv-output cat @@ -67,7 +67,7 @@ Likewise, if you need to produce CSV which is lacking its header, you can pipe M Lastly, often we say "CSV" or "TSV" when we have positionally indexed data in columns which are separated by commas or tabs, respectively. In this case it's perhaps simpler to **just use NIDX format** which was designed for this purpose. (See also :doc:`file-formats`.) For example: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --inidx --ifs comma --oxtab cut -f 1,3 data/headerless.csv @@ -88,7 +88,7 @@ Doing multiple joins Suppose we have the following data: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat multi-join/input.csv @@ -104,7 +104,7 @@ Suppose we have the following data: And we want to augment the ``id`` column with lookups from the following data files: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat multi-join/name-lookup.csv @@ -113,7 +113,7 @@ And we want to augment the ``id`` column with lookups from the following data fi 10,Bob 20,Carol -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat multi-join/status-lookup.csv @@ -124,7 +124,7 @@ And we want to augment the ``id`` column with lookups from the following data fi We can run the input file through multiple ``join`` commands in a ``then``-chain: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint join -f multi-join/name-lookup.csv -j id then join -f multi-join/status-lookup.csv -j id multi-join/input.csv @@ -143,7 +143,7 @@ Bulk rename of fields Suppose you want to replace spaces with underscores in your column names: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/spaces.csv @@ -154,7 +154,7 @@ Suppose you want to replace spaces with underscores in your column names: The simplest way is to use ``mlr rename`` with ``-g`` (for global replace, not just first occurrence of space within each field) and ``-r`` for pattern-matching (rather than explicit single-column renames): -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv rename -g -r ' ,_' data/spaces.csv @@ -163,7 +163,7 @@ The simplest way is to use ``mlr rename`` with ``-g`` (for global replace, not j 2468,1357,3579 9987,3312,4543 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv --opprint rename -g -r ' ,_' data/spaces.csv @@ -174,7 +174,7 @@ The simplest way is to use ``mlr rename`` with ``-g`` (for global replace, not j You can also do this with a for-loop: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/bulk-rename-for-loop.mlr @@ -184,7 +184,7 @@ You can also do this with a for-loop: } $* = newrec -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint put -f data/bulk-rename-for-loop.mlr data/spaces.csv @@ -198,7 +198,7 @@ Search-and-replace over all fields How to do ``$name = gsub($name, "old", "new")`` for all fields? -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/sar.csv @@ -206,7 +206,7 @@ How to do ``$name = gsub($name, "old", "new")`` for all fields? the quick,brown fox,jumped over,the,lazy dogs -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/sar.mlr @@ -214,7 +214,7 @@ How to do ``$name = gsub($name, "old", "new")`` for all fields? $[k] = gsub($[k], "e", "X"); } -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv put -f data/sar.mlr data/sar.csv @@ -227,7 +227,7 @@ Full field renames and reassigns Using Miller 5.0.0's map literals and assigning to ``$*``, you can fully generalize :ref:`mlr rename `, :ref:`mlr reorder `, etc. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/small @@ -237,7 +237,7 @@ Using Miller 5.0.0's map literals and assigning to ``$*``, you can fully general a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463 a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put ' @@ -266,7 +266,7 @@ Numbering and renumbering records The ``awk``-like built-in variable ``NR`` is incremented for each input record: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/small @@ -276,7 +276,7 @@ The ``awk``-like built-in variable ``NR`` is incremented for each input record: a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463 a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put '$nr = NR' data/small @@ -288,7 +288,7 @@ The ``awk``-like built-in variable ``NR`` is incremented for each input record: However, this is the record number within the original input stream -- not after any filtering you may have done: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr filter '$a == "wye"' then put '$nr = NR' data/small @@ -297,7 +297,7 @@ However, this is the record number within the original input stream -- not after There are two good options here. One is to use the ``cat`` verb with ``-n``: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr filter '$a == "wye"' then cat -n data/small @@ -306,7 +306,7 @@ There are two good options here. One is to use the ``cat`` verb with ``-n``: The other is to keep your own counter within the ``put`` DSL: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr filter '$a == "wye"' then put 'begin {@n = 1} $n = @n; @n += 1' data/small @@ -329,7 +329,7 @@ Data-cleaning examples Here are some ways to use the type-checking options as described in :ref:`reference-dsl-type-tests-and-assertions` Suppose you have the following data file, with inconsistent typing for boolean. (Also imagine that, for the sake of discussion, we have a million-line file rather than a four-line file, so we can't see it all at once and some automation is called for.) -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/het-bool.csv @@ -341,7 +341,7 @@ Here are some ways to use the type-checking options as described in :ref:`refere One option is to coerce everything to boolean, or integer: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint put '$reachable = boolean($reachable)' data/het-bool.csv @@ -351,7 +351,7 @@ One option is to coerce everything to boolean, or integer: fred true wilma true -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint put '$reachable = int(boolean($reachable))' data/het-bool.csv @@ -363,7 +363,7 @@ One option is to coerce everything to boolean, or integer: A second option is to flag badly formatted data within the output stream: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint put '$format_ok = is_string($reachable)' data/het-bool.csv @@ -375,7 +375,7 @@ A second option is to flag badly formatted data within the output stream: Or perhaps to flag badly formatted data outside the output stream: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint put 'if (!is_string($reachable)) {eprint "Malformed at NR=".NR} ' data/het-bool.csv @@ -388,7 +388,7 @@ Or perhaps to flag badly formatted data outside the output stream: A third way is to abort the process on first instance of bad data: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv put '$reachable = asserting_string($reachable)' data/het-bool.csv @@ -403,7 +403,7 @@ Splitting nested fields Suppose you have a TSV file like this: -.. code-block:: +.. code-block:: none a b x z @@ -411,7 +411,7 @@ Suppose you have a TSV file like this: The simplest option is to use :ref:`mlr nest `: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --tsv nest --explode --values --across-records -f b --nested-fs : data/nested.tsv @@ -421,7 +421,7 @@ The simplest option is to use :ref:`mlr nest `: s v s w -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --tsv nest --explode --values --across-fields -f b --nested-fs : data/nested.tsv @@ -435,7 +435,7 @@ While ``mlr nest`` is simplest, let's also take a look at a few ways to do this One option to split out the colon-delimited values in the ``b`` column is to use ``splitnv`` to create an integer-indexed map and loop over it, adding new fields to the current record: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/nested.tsv --itsv --oxtab put 'o=splitnv($b, ":"); for (k,v in o) {$["p".k]=v}' @@ -451,7 +451,7 @@ One option to split out the colon-delimited values in the ``b`` column is to use while another is to loop over the same map from ``splitnv`` and use it (with ``put -q`` to suppress printing the original record) to produce multiple records: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/nested.tsv --itsv --oxtab put -q 'o=splitnv($b, ":"); for (k,v in o) {x=mapsum($*, {"b":v}); emit x}' @@ -467,7 +467,7 @@ while another is to loop over the same map from ``splitnv`` and use it (with ``p a s b w -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/nested.tsv --tsv put -q 'o=splitnv($b, ":"); for (k,v in o) {x=mapsum($*, {"b":v}); emit x}' @@ -482,7 +482,7 @@ Showing differences between successive queries Suppose you have a database query which you run at one point in time, producing the output on the left, then again later producing the output on the right: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/previous_counters.csv @@ -492,7 +492,7 @@ Suppose you have a database query which you run at one point in time, producing orange,694 purple,12 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/current_counters.csv @@ -506,12 +506,12 @@ And, suppose you want to compute the differences in the counters between adjacen First, rename counter columns to make them distinct: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv rename count,previous_count data/previous_counters.csv > data/prevtemp.csv -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/prevtemp.csv @@ -521,12 +521,12 @@ First, rename counter columns to make them distinct: orange,694 purple,12 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv rename count,current_count data/current_counters.csv > data/currtemp.csv -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/currtemp.csv @@ -538,7 +538,7 @@ First, rename counter columns to make them distinct: Then, join on the key field(s), and use unsparsify to zero-fill counters absent on one side but present on the other. Use ``--ul`` and ``--ur`` to emit unpaired records (namely, purple on the left and yellow on the right): -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint \ @@ -558,7 +558,7 @@ Finding missing dates Suppose you have some date-stamped data which may (or may not) be missing entries for one or more dates: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ head -n 10 data/miss-date.csv @@ -573,7 +573,7 @@ Suppose you have some date-stamped data which may (or may not) be missing entrie 2012-03-12,11043 2012-03-13,11177 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ wc -l data/miss-date.csv @@ -581,7 +581,7 @@ Suppose you have some date-stamped data which may (or may not) be missing entrie Since there are 1372 lines in the data file, some automation is called for. To find the missing dates, you can convert the dates to seconds since the epoch using ``strptime``, then compute adjacent differences (the ``cat -n`` simply inserts record-counters): -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/miss-date.csv --icsv \ @@ -602,7 +602,7 @@ Since there are 1372 lines in the data file, some automation is called for. To f Then, filter for adjacent difference not being 86400 (the number of seconds in a day): -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/miss-date.csv --icsv \ @@ -615,7 +615,7 @@ Then, filter for adjacent difference not being 86400 (the number of seconds in a Given this, it's now easy to see where the gaps are: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr cat -n then filter '$n >= 770 && $n <= 780' data/miss-date.csv @@ -631,7 +631,7 @@ Given this, it's now easy to see where the gaps are: n=779,1=2014-04-23,2=130849 n=780,1=2014-04-24,2=131026 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr cat -n then filter '$n >= 1115 && $n <= 1125' data/miss-date.csv @@ -657,7 +657,7 @@ Two-pass algorithms: computation of percentages For example, mapping numeric values down a column to the percentage between their min and max values is two-pass: on the first pass you find the min and max values, then on the second, map each record's value to a percentage. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/small --opprint put -q ' @@ -688,7 +688,7 @@ Two-pass algorithms: line-number ratios Similarly, finding the total record count requires first reading through all the data: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint --from data/small put -q ' @@ -714,7 +714,7 @@ Two-pass algorithms: records having max value The idea is to retain records having the largest value of ``n`` in the following data: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --itsv --opprint cat data/maxrows.tsv @@ -753,7 +753,7 @@ The idea is to retain records having the largest value of ``n`` in the following Of course, the largest value of ``n`` isn't known until after all data have been read. Using an out-of-stream variable we can retain all records as they are read, then filter them at the end: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/maxrows.mlr @@ -773,7 +773,7 @@ Of course, the largest value of ``n`` isn't known until after all data have been } } -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --itsv --opprint put -q -f data/maxrows.mlr data/maxrows.tsv @@ -793,7 +793,7 @@ Rectangularizing data Suppose you have a method (in whatever language) which is printing things of the form -.. code-block:: +.. code-block:: none outer=1 outer=2 @@ -801,7 +801,7 @@ Suppose you have a method (in whatever language) which is printing things of the and then calls another method which prints things of the form -.. code-block:: +.. code-block:: none middle=10 middle=11 @@ -813,7 +813,7 @@ and then calls another method which prints things of the form and then, perhaps, that second method calls a third method which prints things of the form -.. code-block:: +.. code-block:: none inner1=100,inner2=101 inner1=120,inner2=121 @@ -825,7 +825,7 @@ and then, perhaps, that second method calls a third method which prints things o with the result that your program's output is -.. code-block:: +.. code-block:: none outer=1 middle=10 @@ -847,7 +847,7 @@ with the result that your program's output is The idea here is that middles starting with a 1 belong to the outer value of 1, and so on. (For example, the outer values might be account IDs, the middle values might be invoice IDs, and the inner values might be invoice line-items.) If you want all the middle and inner lines to have the context of which outers they belong to, you can modify your software to pass all those through your methods. Alternatively, don't refactor your code just to handle some ad-hoc log-data formatting -- instead, use the following to rectangularize the data. The idea is to use an out-of-stream variable to accumulate fields across records. Clear that variable when you see an outer ID; accumulate fields; emit output when you see the inner IDs. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/rect.txt put -q ' @@ -873,7 +873,7 @@ Regularizing ragged CSV Miller handles compliant CSV: in particular, it's an error if the number of data fields in a given data line don't match the number of header lines. But in the event that you have a CSV file in which some lines have less than the full number of fields, you can use Miller to pad them out. The trick is to use NIDX format, for which each line stands on its own without respect to a header line. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/ragged.csv @@ -882,7 +882,7 @@ Miller handles compliant CSV: in particular, it's an error if the number of data 4,5 6,7,8,9 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/ragged.csv --fs comma --nidx put ' @@ -900,7 +900,7 @@ Miller handles compliant CSV: in particular, it's an error if the number of data or, more simply, -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/ragged.csv --fs comma --nidx put ' @@ -919,7 +919,7 @@ Feature-counting Suppose you have some heterogeneous data like this: -.. code-block:: +.. code-block:: none { "qoh": 29874, "rate": 1.68, "latency": 0.02 } { "name": "alice", "uid": 572 } @@ -936,7 +936,7 @@ Suppose you have some heterogeneous data like this: A reasonable question to ask is, how many occurrences of each field are there? And, what percentage of total row count has each of them? Since the denominator of the percentage is not known until the end, this is a two-pass algorithm: -.. code-block:: +.. code-block:: none for (key in $*) { @key_counts[key] += 1; @@ -954,7 +954,7 @@ A reasonable question to ask is, how many occurrences of each field are there? A Then -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --json put -q -f data/feature-count.mlr data/features.json @@ -972,7 +972,7 @@ Then { "key": "uid", "key_fraction": 0.250000 } { "key": "uid2", "key_fraction": 0.083333 } -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --ijson --opprint put -q -f data/feature-count.mlr data/features.json @@ -1002,7 +1002,7 @@ The previous section discussed how to fill out missing data fields within CSV wi For example, suppose you have JSON input like this: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/sparse.json @@ -1013,7 +1013,7 @@ For example, suppose you have JSON input like this: There are field names ``a``, ``b``, ``v``, ``u``, ``x``, ``w`` in the data -- but not all in every record. Since we don't know the names of all the keys until we've read them all, this needs to be a two-pass algorithm. On the first pass, remember all the unique key names and all the records; on the second pass, loop through the records filling in absent values, then producing output. Use ``put -q`` since we don't want to produce per-record output, only emitting output in the ``end`` block: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/unsparsify.mlr @@ -1044,7 +1044,7 @@ There are field names ``a``, ``b``, ``v``, ``u``, ``x``, ``w`` in the data -- bu } } -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --json put -q -f data/unsparsify.mlr data/sparse.json @@ -1053,7 +1053,7 @@ There are field names ``a``, ``b``, ``v``, ``u``, ``x``, ``w`` in the data -- bu { "a": 1, "b": "", "v": 2, "u": "", "x": 3, "w": "" } { "a": "", "b": "", "v": 1, "u": "", "x": "", "w": 2 } -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --ijson --ocsv put -q -f data/unsparsify.mlr data/sparse.json @@ -1063,7 +1063,7 @@ There are field names ``a``, ``b``, ``v``, ``u``, ``x``, ``w`` in the data -- bu 1,,2,,3, ,,1,,,2 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --ijson --opprint put -q -f data/unsparsify.mlr data/sparse.json @@ -1080,13 +1080,13 @@ Parsing log-file output This, of course, depends highly on what's in your log files. But, as an example, suppose you have log-file lines such as -.. code-block:: +.. code-block:: none 2015-10-08 08:29:09,445 INFO com.company.path.to.ClassName @ [sometext] various/sorts/of data {& punctuation} hits=1 status=0 time=2.378 I prefer to pre-filter with ``grep`` and/or ``sed`` to extract the structured text, then hand that to Miller. Example: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 grep 'various sorts' *.log | sed 's/.*} //' | mlr --fs space --repifs --oxtab stats1 -a min,p10,p50,p90,max -f time -g status @@ -1098,7 +1098,7 @@ Memoization with out-of-stream variables The recursive function for the Fibonacci sequence is famous for its computational complexity. Namely, using f(0)=1, f(1)=1, f(n)=f(n-1)+f(n-2) for n>=2, the evaluation tree branches left as well as right at each non-trivial level, resulting in millions or more paths to the root 0/1 nodes for larger n. This program -.. code-block:: +.. code-block:: none mlr --ofmt '%.9lf' --opprint seqgen --start 1 --stop 28 then put ' func f(n) { @@ -1119,7 +1119,7 @@ The recursive function for the Fibonacci sequence is famous for its computationa produces output like this: -.. code-block:: +.. code-block:: none i o fcount seconds_delta 1 1 1 0 @@ -1153,7 +1153,7 @@ produces output like this: Note that the time it takes to evaluate the function is blowing up exponentially as the input argument increases. Using ``@``-variables, which persist across records, we can cache and reuse the results of previous computations: -.. code-block:: +.. code-block:: none mlr --ofmt '%.9lf' --opprint seqgen --start 1 --stop 28 then put ' func f(n) { @@ -1176,7 +1176,7 @@ Note that the time it takes to evaluate the function is blowing up exponentially with output like this: -.. code-block:: +.. code-block:: none i o fcount seconds_delta 1 1 1 0 diff --git a/docs6/cookbook.rst.in b/docs6/cookbook.rst.in index 1e4f1dc60..03c6a95b3 100644 --- a/docs6/cookbook.rst.in +++ b/docs6/cookbook.rst.in @@ -323,13 +323,13 @@ Parsing log-file output This, of course, depends highly on what's in your log files. But, as an example, suppose you have log-file lines such as -.. code-block:: +.. code-block:: none 2015-10-08 08:29:09,445 INFO com.company.path.to.ClassName @ [sometext] various/sorts/of data {& punctuation} hits=1 status=0 time=2.378 I prefer to pre-filter with ``grep`` and/or ``sed`` to extract the structured text, then hand that to Miller. Example: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 grep 'various sorts' *.log | sed 's/.*} //' | mlr --fs space --repifs --oxtab stats1 -a min,p10,p50,p90,max -f time -g status @@ -345,7 +345,7 @@ POKI_INCLUDE_ESCAPED(data/fibo-uncached.sh)HERE produces output like this: -.. code-block:: +.. code-block:: none i o fcount seconds_delta 1 1 1 0 @@ -383,7 +383,7 @@ POKI_INCLUDE_ESCAPED(data/fibo-cached.sh)HERE with output like this: -.. code-block:: +.. code-block:: none i o fcount seconds_delta 1 1 1 0 diff --git a/docs6/cookbook2.rst b/docs6/cookbook2.rst index 1c6a3a8c3..c8f8c8e72 100644 --- a/docs6/cookbook2.rst +++ b/docs6/cookbook2.rst @@ -9,7 +9,7 @@ Randomly selecting words from a list Given this `word list `_, first take a look to see what the first few lines look like: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ head data/english-words.txt @@ -26,7 +26,7 @@ Given this `word list lines.txt @@ -109,7 +109,7 @@ Computing interquartile ranges For one or more specified field names, simply compute p25 and p75, then write the IQR as the difference of p75 and p25: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --oxtab stats1 -f x -a p25,p75 \ @@ -121,7 +121,7 @@ For one or more specified field names, simply compute p25 and p75, then write th For wildcarded field names, first compute p25 and p75, then loop over field names with ``p25`` in them: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --oxtab stats1 --fr '[i-z]' -a p25,p75 \ @@ -146,7 +146,7 @@ Computing weighted means This might be more elegantly implemented as an option within the ``stats1`` verb. Meanwhile, it's expressible within the DSL: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/medium put -q ' @@ -184,7 +184,7 @@ Generating random numbers from various distributions Here we can chain together a few simple building blocks: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat expo-sample.sh @@ -227,7 +227,7 @@ Namely: The output is as follows: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ sh expo-sample.sh @@ -288,7 +288,7 @@ Sieve of Eratosthenes The `Sieve of Eratosthenes `_ is a standard introductory programming topic. The idea is to find all primes up to some *N* by making a list of the numbers 1 to *N*, then striking out all multiples of 2 except 2 itself, all multiples of 3 except 3 itself, all multiples of 4 except 4 itself, and so on. Whatever survives that without getting marked is a prime. This is easy enough in Miller. Notice that here all the work is in ``begin`` and ``end`` statements; there is no file input (so we use ``mlr -n`` to keep Miller from waiting for input data). -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat programs/sieve.mlr @@ -325,7 +325,7 @@ The `Sieve of Eratosthenes ` } } -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr -n put -f programs/sieve.mlr @@ -362,7 +362,7 @@ The `Mandelbrot set `_ is also easi The (approximate) computation of points in the complex plane which are and aren't members is just a few lines of complex arithmetic (see the Wikipedia article); how to render them is another task. Using graphics libraries you can create PNG or JPEG files, but another fun way to do this is by printing various characters to the screen: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat programs/mand.mlr @@ -469,7 +469,7 @@ The (approximate) computation of points in the complex plane which are and aren' At standard resolution this makes a nice little ASCII plot: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr -n put -f ./programs/mand.mlr @@ -526,7 +526,7 @@ At standard resolution this makes a nice little ASCII plot: But using a very small font size (as small as my Mac will let me go), and by choosing the coordinates to zoom in on a particular part of the complex plane, we can get a nice little picture: -.. code-block:: +.. code-block:: none #!/bin/bash # Get the number of rows and columns from the terminal window dimensions diff --git a/docs6/cookbook2.rst.in b/docs6/cookbook2.rst.in index bda541487..b4df6520d 100644 --- a/docs6/cookbook2.rst.in +++ b/docs6/cookbook2.rst.in @@ -6,7 +6,7 @@ Randomly selecting words from a list Given this `word list `_, first take a look to see what the first few lines look like: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ head data/english-words.txt @@ -23,7 +23,7 @@ Given this `word list `_ file is some sample data obtained from https:/ Vertical-tabular format is good for a quick look at CSV data layout -- seeing what columns you have to work with: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ head -n 2 data/flins.csv | mlr --icsv --oxtab cat @@ -22,7 +22,7 @@ Vertical-tabular format is good for a quick look at CSV data layout -- seeing wh A few simple queries: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/flins.csv --icsv --opprint count-distinct -f county | head @@ -34,26 +34,26 @@ A few simple queries: Duval 1 St. Johns 1 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/flins.csv --icsv --opprint count-distinct -f construction,line Categorization of total insured value: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/flins.csv --icsv --opprint stats1 -a min,mean,max -f tiv_2012 tiv_2012_min tiv_2012_mean tiv_2012_max 19757.910000 1061531.463750 2785551.630000 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/flins.csv --icsv --opprint stats1 -a min,mean,max -f tiv_2012 -g construction,line -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/flins.csv --icsv --oxtab stats1 -a p0,p10,p50,p90,p95,p99,p100 -f hu_site_deductible @@ -65,7 +65,7 @@ Categorization of total insured value: hu_site_deductible_p99 hu_site_deductible_p100 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/flins.csv --icsv --opprint stats1 -a p95,p99,p100 -f hu_site_deductible -g county then sort -f county | head @@ -77,7 +77,7 @@ Categorization of total insured value: Seminole - - - St. Johns - - - -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/flins.csv --icsv --oxtab stats2 -a corr,linreg-ols,r2 -f tiv_2011,tiv_2012 @@ -87,7 +87,7 @@ Categorization of total insured value: tiv_2011_tiv_2012_ols_n 8 tiv_2011_tiv_2012_r2 0.874904 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/flins.csv --icsv --opprint stats2 -a corr,linreg-ols,r2 -f tiv_2011,tiv_2012 -g county @@ -114,13 +114,13 @@ The `colored-shapes.dkvp strptime("2018-03-03", "%Y-%m-%d")' dates.csv @@ -315,7 +315,7 @@ How can I handle commas-as-data in various formats? :doc:`CSV ` handles this well and by design: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat commas.csv @@ -325,7 +325,7 @@ How can I handle commas-as-data in various formats? Likewise :ref:`file-formats-json`: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --ojson cat commas.csv @@ -334,7 +334,7 @@ Likewise :ref:`file-formats-json`: For Miller's :ref:`vertical-tabular format ` there is no escaping for carriage returns, but commas work fine: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --oxtab cat commas.csv @@ -346,7 +346,7 @@ For Miller's :ref:`vertical-tabular format ` there is no esca But for :ref:`Key-value_pairs ` and :ref:`index-numbered `, commas are the default field separator. And -- as of Miller 5.4.0 anyway -- there is no CSV-style double-quote-handling like there is for CSV. So commas within the data look like delimiters: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --odkvp cat commas.csv @@ -355,7 +355,7 @@ But for :ref:`Key-value_pairs ` and :ref:`index-numbered 0.5' then put '$NR = NR' data/small @@ -510,7 +510,7 @@ why don't I see ``NR=1`` and ``NR=2`` here?? The reason is that ``NR`` is computed for the original input records and isn't dynamically updated. By contrast, ``NF`` is dynamically updated: it's the number of fields in the current record, and if you add/remove a field, the value of ``NF`` will change: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ echo x=1,y=2,z=3 | mlr put '$nf1 = NF; $u = 4; $nf2 = NF; unset $x,$y,$z; $nf3 = NF' @@ -518,7 +518,7 @@ The reason is that ``NR`` is computed for the original input records and isn't d ``NR``, by contrast (and ``FNR`` as well), retains the value from the original input stream, and records may be dropped by a ``filter`` within a ``then``-chain. To recover consecutive record numbers, you can use out-of-stream variables as follows: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint --from data/small put ' @@ -538,7 +538,7 @@ The reason is that ``NR`` is computed for the original input records and isn't d Or, simply use ``mlr cat -n``: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr filter '$x > 0.5' then cat -n data/small @@ -552,7 +552,7 @@ Why am I not seeing all possible joins occur? For example, the right file here has nine records, and the left file should add in the ``hostname`` column -- so the join output should also have 9 records: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsvlite --opprint cat data/join-u-left.csv @@ -561,7 +561,7 @@ For example, the right file here has nine records, and the left file should add zenith.west.our.org 10.3.1.27 apoapsis.east.our.org 10.4.5.94 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsvlite --opprint cat data/join-u-right.csv @@ -576,7 +576,7 @@ For example, the right file here has nine records, and the left file should add 10.3.1.18 1448762598 73425 10.4.5.94 1448762599 12200 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsvlite --opprint join -s -j ipaddr -f data/join-u-left.csv data/join-u-right.csv @@ -590,7 +590,7 @@ The issue is that Miller's ``join``, by default (before 5.1.0), took input sorte The solution (besides pre-sorting the input files on the join keys) is to simply use **mlr join -u** (which is now the default). This loads the left file entirely into memory (while the right file is still streamed one line at a time) and does all possible joins without requiring sorted input: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsvlite --opprint join -u -j ipaddr -f data/join-u-left.csv data/join-u-right.csv @@ -612,14 +612,14 @@ How to rectangularize after joins with unpaired? Suppose you have the following two data files: -.. code-block:: +.. code-block:: none id,code 3,0000ff 2,00ff00 4,ff0000 -.. code-block:: +.. code-block:: none id,color 4,red @@ -627,7 +627,7 @@ Suppose you have the following two data files: Joining on color the results are as expected: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv join -j id -f data/color-codes.csv data/color-names.csv @@ -637,7 +637,7 @@ Joining on color the results are as expected: However, if we ask for left-unpaireds, since there's no ``color`` column, we get a row not having the same column names as the other: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv join --ul -j id -f data/color-codes.csv data/color-names.csv @@ -650,7 +650,7 @@ However, if we ask for left-unpaireds, since there's no ``color`` column, we get To fix this, we can use **unsparsify**: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv join --ul -j id -f data/color-codes.csv then unsparsify --fill-with "" data/color-names.csv @@ -670,13 +670,13 @@ XML, JSON, etc. are, by contrast, all **recursive** or **nested** data structure Now, you can put tabular data into these formats -- since list-of-key-value-pairs is one of the things representable in XML or JSON. Example: -.. code-block:: +.. code-block:: none # DKVP x=1,y=2 z=3 -.. code-block:: +.. code-block:: none # XML @@ -695,7 +695,7 @@ Now, you can put tabular data into these formats -- since list-of-key-value-pair
-.. code-block:: +.. code-block:: none # JSON [{"x":1,"y":2},{"z":3}] diff --git a/docs6/faq.rst.in b/docs6/faq.rst.in index 96978808e..d1231f885 100644 --- a/docs6/faq.rst.in +++ b/docs6/faq.rst.in @@ -278,13 +278,13 @@ XML, JSON, etc. are, by contrast, all **recursive** or **nested** data structure Now, you can put tabular data into these formats -- since list-of-key-value-pairs is one of the things representable in XML or JSON. Example: -.. code-block:: +.. code-block:: none # DKVP x=1,y=2 z=3 -.. code-block:: +.. code-block:: none # XML @@ -303,7 +303,7 @@ Now, you can put tabular data into these formats -- since list-of-key-value-pair
-.. code-block:: +.. code-block:: none # JSON [{"x":1,"y":2},{"z":3}] diff --git a/docs6/feature-comparison.rst b/docs6/feature-comparison.rst index d2630ddeb..26922fcf7 100644 --- a/docs6/feature-comparison.rst +++ b/docs6/feature-comparison.rst @@ -11,7 +11,7 @@ File-format awareness Miller respects CSV headers. If you do ``mlr --csv cat *.csv`` then the header line is written once: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/a.csv @@ -19,14 +19,14 @@ Miller respects CSV headers. If you do ``mlr --csv cat *.csv`` then the header l 1,2,3 4,5,6 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/b.csv a,b,c 7,8,9 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv cat data/a.csv data/b.csv @@ -35,7 +35,7 @@ Miller respects CSV headers. If you do ``mlr --csv cat *.csv`` then the header l 4,5,6 7,8,9 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv sort -nr b data/a.csv data/b.csv diff --git a/docs6/file-formats.rst b/docs6/file-formats.rst index bbc902dac..ea551ef8a 100644 --- a/docs6/file-formats.rst +++ b/docs6/file-formats.rst @@ -9,7 +9,7 @@ Miller handles name-indexed data using several formats: some you probably know b Examples ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --usage-data-format-examples @@ -118,7 +118,7 @@ DKVP: Key-value pairs Miller's default file format is DKVP, for **delimited key-value pairs**. Example: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr cat data/small @@ -130,21 +130,21 @@ Miller's default file format is DKVP, for **delimited key-value pairs**. Example Such data are easy to generate, e.g. in Ruby with -.. code-block:: +.. code-block:: none puts "host=#{hostname},seconds=#{t2-t1},message=#{msg}" -.. code-block:: +.. code-block:: none puts mymap.collect{|k,v| "#{k}=#{v}"}.join(',') or ``print`` statements in various languages, e.g. -.. code-block:: +.. code-block:: none echo "type=3,user=$USER,date=$date\n"; -.. code-block:: +.. code-block:: none logger.log("type=3,user=$USER,date=$date\n"); @@ -152,7 +152,7 @@ Fields lacking an IPS will have positional index (starting at 1) used as the key As discussed in :doc:`record-heterogeneity`, Miller handles changes of field names within the same data stream. But using DKVP format this is particularly natural. One of my favorite use-cases for Miller is in application/server logs, where I log all sorts of lines such as -.. code-block:: +.. code-block:: none resource=/path/to/file,loadsec=0.45,ok=true record_count=100, resource=/path/to/file @@ -172,7 +172,7 @@ With ``--inidx --ifs ' ' --repifs``, Miller splits lines on whitespace and assig Example with index-numbered output: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/small @@ -182,7 +182,7 @@ Example with index-numbered output: a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463 a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --onidx --ofs ' ' cat data/small @@ -194,7 +194,7 @@ Example with index-numbered output: Example with index-numbered input: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/mydata.txt @@ -202,7 +202,7 @@ Example with index-numbered input: by the dawn's early light -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --inidx --ifs ' ' --odkvp cat data/mydata.txt @@ -212,7 +212,7 @@ Example with index-numbered input: Example with index-numbered input and output: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/mydata.txt @@ -220,7 +220,7 @@ Example with index-numbered input and output: by the dawn's early light -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --nidx --fs ' ' --repifs cut -f 2,3 data/mydata.txt @@ -242,14 +242,14 @@ Single-level JSON objects An **array of single-level objects** is, quite simply, **a table**: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --json head -n 2 then cut -f color,shape data/json-example-1.json { "color": "yellow", "shape": "triangle" } { "color": "red", "shape": "square" } -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --json --jvstack head -n 2 then cut -f color,u,v data/json-example-1.json @@ -264,7 +264,7 @@ An **array of single-level objects** is, quite simply, **a table**: "v": 0.001257332190235938 } -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --ijson --opprint stats1 -a mean,stddev,count -f u -g shape data/json-example-1.json @@ -278,7 +278,7 @@ Nested JSON objects Additionally, Miller can **tabularize nested objects by concatentating keys**: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --json --jvstack head -n 2 data/json-example-2.json @@ -311,7 +311,7 @@ Additionally, Miller can **tabularize nested objects by concatentating keys**: } } -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --ijson --opprint head -n 4 data/json-example-2.json @@ -323,7 +323,7 @@ Additionally, Miller can **tabularize nested objects by concatentating keys**: Note in particular that as far as Miller's ``put`` and ``filter``, as well as other I/O formats, are concerned, these are simply field names with colons in them: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --json --jvstack head -n 1 then put '${values:uv} = ${values:u} * ${values:v}' data/json-example-2.json @@ -350,7 +350,7 @@ Arrays aren't supported in Miller's ``put``/``filter`` DSL. By default, JSON arr Suppose we have arrays like this in our input data: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/json-example-3.json @@ -365,7 +365,7 @@ Suppose we have arrays like this in our input data: Then integer indices (starting from 0 and counting up) are used as map keys: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --ijson --oxtab cat data/json-example-3.json @@ -380,7 +380,7 @@ Then integer indices (starting from 0 and counting up) are used as map keys: When the data are written back out as JSON, field names are re-expanded as above, but what were arrays on input are now maps on output: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --json --jvstack cat data/json-example-3.json @@ -437,7 +437,7 @@ PPRINT: Pretty-printed tabular Miller's pretty-print format is like CSV, but column-aligned. For example, compare -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --ocsv cat data/small @@ -448,7 +448,7 @@ Miller's pretty-print format is like CSV, but column-aligned. For example, comp eks,wye,4,0.38139939387114097,0.13418874328430463 wye,pan,5,0.5732889198020006,0.8636244699032729 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint cat data/small @@ -465,7 +465,7 @@ See :doc:`record-heterogeneity` for how Miller handles changes of field names wi For output only (this isn't supported in the input-scanner as of 5.0.0) you can use ``--barred`` with pprint output format: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint --barred cat data/small @@ -487,7 +487,7 @@ XTAB: Vertical tabular This is perhaps most useful for looking a very wide and/or multi-column data which causes line-wraps on the screen (but see also `ngrid `_ for an entirely different, very powerful option). Namely: -.. code-block:: +.. code-block:: none $ grep -v '^#' /etc/passwd | head -n 6 | mlr --nidx --fs : --opprint cat 1 2 3 4 5 6 7 @@ -498,7 +498,7 @@ This is perhaps most useful for looking a very wide and/or multi-column data whi _taskgated * 13 13 Task Gate Daemon /var/empty /usr/bin/false _networkd * 24 24 Network Services /var/networkd /usr/bin/false -.. code-block:: +.. code-block:: none $ grep -v '^#' /etc/passwd | head -n 2 | mlr --nidx --fs : --oxtab cat 1 nobody @@ -517,7 +517,7 @@ This is perhaps most useful for looking a very wide and/or multi-column data whi 6 /var/root 7 /bin/sh -.. code-block:: +.. code-block:: none $ grep -v '^#' /etc/passwd | head -n 2 | \ mlr --nidx --fs : --ojson --jvstack --jlistwrap label name,password,uid,gid,gecos,home_dir,shell @@ -547,7 +547,7 @@ Markdown tabular Markdown format looks like this: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --omd cat data/small @@ -570,7 +570,7 @@ Data-conversion keystroke-savers While you can do format conversion using ``mlr --icsv --ojson cat myfile.csv``, there are also keystroke-savers for this purpose, such as ``mlr --c2j cat myfile.csv``. For a complete list: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --usage-format-conversion-keystroke-saver-options @@ -606,7 +606,7 @@ Comments in data You can include comments within your data files, and either have them ignored, or passed directly through to the standard output as soon as they are encountered: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --usage-comments-in-data @@ -629,7 +629,7 @@ You can include comments within your data files, and either have them ignored, o Examples: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/budget.csv @@ -639,7 +639,7 @@ Examples: green,678.12 orange,123.45 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --skip-comments --icsv --opprint sort -nr quantity data/budget.csv @@ -648,7 +648,7 @@ Examples: purple 456.78 orange 123.45 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --pass-comments --icsv --opprint sort -nr quantity data/budget.csv diff --git a/docs6/file-formats.rst.in b/docs6/file-formats.rst.in index 8f48d663a..fb6c89652 100644 --- a/docs6/file-formats.rst.in +++ b/docs6/file-formats.rst.in @@ -57,21 +57,21 @@ POKI_RUN_COMMAND{{mlr cat data/small}}HERE Such data are easy to generate, e.g. in Ruby with -.. code-block:: +.. code-block:: none puts "host=#{hostname},seconds=#{t2-t1},message=#{msg}" -.. code-block:: +.. code-block:: none puts mymap.collect{|k,v| "#{k}=#{v}"}.join(',') or ``print`` statements in various languages, e.g. -.. code-block:: +.. code-block:: none echo "type=3,user=$USER,date=$date\n"; -.. code-block:: +.. code-block:: none logger.log("type=3,user=$USER,date=$date\n"); @@ -79,7 +79,7 @@ Fields lacking an IPS will have positional index (starting at 1) used as the key As discussed in :doc:`record-heterogeneity`, Miller handles changes of field names within the same data stream. But using DKVP format this is particularly natural. One of my favorite use-cases for Miller is in application/server logs, where I log all sorts of lines such as -.. code-block:: +.. code-block:: none resource=/path/to/file,loadsec=0.45,ok=true record_count=100, resource=/path/to/file diff --git a/docs6/install.rst b/docs6/install.rst index 46cfe4ab6..f8544b2ec 100644 --- a/docs6/install.rst +++ b/docs6/install.rst @@ -9,38 +9,38 @@ Prebuilt executables via package managers `Homebrew `_ installation support for OSX is available via -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 brew update && brew install miller ...and also via `MacPorts `_: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 sudo port selfupdate && sudo port install miller You may already have the ``mlr`` executable available in your platform's package manager on NetBSD, Debian Linux, Ubuntu Xenial and upward, Arch Linux, or perhaps other distributions. For example, on various Linux distributions you might do one of the following: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 sudo apt-get install miller -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 sudo apt install miller -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 sudo yum install miller On Windows, Miller is available via `Chocolatey `_: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 choco install miller diff --git a/docs6/install.rst.in b/docs6/install.rst.in index 75da87972..350ad9db6 100644 --- a/docs6/install.rst.in +++ b/docs6/install.rst.in @@ -6,38 +6,38 @@ Prebuilt executables via package managers `Homebrew `_ installation support for OSX is available via -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 brew update && brew install miller ...and also via `MacPorts `_: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 sudo port selfupdate && sudo port install miller You may already have the ``mlr`` executable available in your platform's package manager on NetBSD, Debian Linux, Ubuntu Xenial and upward, Arch Linux, or perhaps other distributions. For example, on various Linux distributions you might do one of the following: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 sudo apt-get install miller -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 sudo apt install miller -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 sudo yum install miller On Windows, Miller is available via `Chocolatey `_: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 choco install miller diff --git a/docs6/log-processing-examples.rst b/docs6/log-processing-examples.rst index 9e6ecb930..3550edabb 100644 --- a/docs6/log-processing-examples.rst +++ b/docs6/log-processing-examples.rst @@ -12,7 +12,7 @@ Writing a program -- in any language whatsoever -- you can have it print out log Suppose your program has printed something like this: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat log.txt @@ -62,7 +62,7 @@ Suppose your program has printed something like this: Each print statement simply contains local information: the current timestamp, whether a particular cache was hit or not, etc. Then using either the system ``grep`` command, or Miller's ``having-fields``, or ``is_present``, we can pick out the parts we want and analyze them: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ grep op=cache log.txt \ @@ -72,7 +72,7 @@ Each print statement simply contains local information: the current timestamp, w A4 0.714286 A9 0.090909 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from log.txt --opprint \ @@ -89,7 +89,7 @@ Each print statement simply contains local information: the current timestamp, w Alternatively, we can simply group the similar data for a better look: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint group-like log.txt @@ -142,7 +142,7 @@ Alternatively, we can simply group the similar data for a better look: 1472819736 100 612 1472819742 100 728 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint group-like then sec2gmt time log.txt diff --git a/docs6/manpage.rst b/docs6/manpage.rst index 9d5acdcaf..34d6fa074 100644 --- a/docs6/manpage.rst +++ b/docs6/manpage.rst @@ -6,7 +6,7 @@ Manpage This is simply a copy of what you should see on running **man mlr** at a command prompt, once Miller is installed on your system. -.. code-block:: +.. code-block:: none MILLER(1) MILLER(1) diff --git a/docs6/poki b/docs6/poki index e61957f94..ba8626677 100755 --- a/docs6/poki +++ b/docs6/poki @@ -90,7 +90,7 @@ end # ---------------------------------------------------------------- def write_card(highlight_first_line, content_lines, output_handle) - output_handle.puts('.. code-block::') + output_handle.puts('.. code-block:: none') if highlight_first_line output_handle.puts(' :emphasize-lines: 1,1') # 'hll' in _static/*.css end diff --git a/docs6/quick-examples.rst b/docs6/quick-examples.rst index 1c890fc71..6718bf7f6 100644 --- a/docs6/quick-examples.rst +++ b/docs6/quick-examples.rst @@ -8,70 +8,70 @@ Quick examples Column select: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 % mlr --csv cut -f hostname,uptime mydata.csv Add new columns as function of other columns: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 % mlr --nidx put '$sum = $7 < 0.0 ? 3.5 : $7 + 2.1*$8' *.dat Row filter: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 % mlr --csv filter '$status != "down" && $upsec >= 10000' *.csv Apply column labels and pretty-print: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 % grep -v '^#' /etc/group | mlr --ifs : --nidx --opprint label group,pass,gid,member then sort -f group Join multiple data sources on key columns: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 % mlr join -j account_id -f accounts.dat then group-by account_name balances.dat Mulltiple formats including JSON: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 % mlr --json put '$attr = sub($attr, "([0-9]+)_([0-9]+)_.*", "\1:\2")' data/*.json Aggregate per-column statistics: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 % mlr stats1 -a min,mean,max,p10,p50,p90 -f flag,u,v data/* Linear regression: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 % mlr stats2 -a linreg-pca -f u,v -g shape data/* Aggregate custom per-column statistics: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 % mlr put -q '@sum[$a][$b] += $x; end {emit @sum, "a", "b"}' data/* Iterate over data using DSL expressions: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 % mlr --from estimates.tbl put ' @@ -85,35 +85,35 @@ Iterate over data using DSL expressions: Run DSL expressions from a script file: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 % mlr --from infile.dat put -f analyze.mlr Split/reduce output to multiple filenames: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 % mlr --from infile.dat put 'tee > "./taps/data-".$a."-".$b, $*' Compressed I/O: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 % mlr --from infile.dat put 'tee | "gzip > ./taps/data-".$a."-".$b.".gz", $*' Interoperate with other data-processing tools using standard pipes: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 % mlr --from infile.dat put -q '@v=$*; dump | "jq .[]"' Tap/trace: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 % mlr --from infile.dat put '(NR % 1000 == 0) { print > stderr, "Checkpoint ".NR}' diff --git a/docs6/quick-examples.rst.in b/docs6/quick-examples.rst.in index 97c11e536..bf9f93e93 100644 --- a/docs6/quick-examples.rst.in +++ b/docs6/quick-examples.rst.in @@ -5,70 +5,70 @@ Quick examples Column select: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 % mlr --csv cut -f hostname,uptime mydata.csv Add new columns as function of other columns: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 % mlr --nidx put '$sum = $7 < 0.0 ? 3.5 : $7 + 2.1*$8' *.dat Row filter: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 % mlr --csv filter '$status != "down" && $upsec >= 10000' *.csv Apply column labels and pretty-print: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 % grep -v '^#' /etc/group | mlr --ifs : --nidx --opprint label group,pass,gid,member then sort -f group Join multiple data sources on key columns: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 % mlr join -j account_id -f accounts.dat then group-by account_name balances.dat Mulltiple formats including JSON: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 % mlr --json put '$attr = sub($attr, "([0-9]+)_([0-9]+)_.*", "\1:\2")' data/*.json Aggregate per-column statistics: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 % mlr stats1 -a min,mean,max,p10,p50,p90 -f flag,u,v data/* Linear regression: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 % mlr stats2 -a linreg-pca -f u,v -g shape data/* Aggregate custom per-column statistics: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 % mlr put -q '@sum[$a][$b] += $x; end {emit @sum, "a", "b"}' data/* Iterate over data using DSL expressions: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 % mlr --from estimates.tbl put ' @@ -82,35 +82,35 @@ Iterate over data using DSL expressions: Run DSL expressions from a script file: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 % mlr --from infile.dat put -f analyze.mlr Split/reduce output to multiple filenames: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 % mlr --from infile.dat put 'tee > "./taps/data-".$a."-".$b, $*' Compressed I/O: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 % mlr --from infile.dat put 'tee | "gzip > ./taps/data-".$a."-".$b.".gz", $*' Interoperate with other data-processing tools using standard pipes: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 % mlr --from infile.dat put -q '@v=$*; dump | "jq .[]"' Tap/trace: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 % mlr --from infile.dat put '(NR % 1000 == 0) { print > stderr, "Checkpoint ".NR}' diff --git a/docs6/record-heterogeneity.rst b/docs6/record-heterogeneity.rst index 26d20a25c..757488b54 100644 --- a/docs6/record-heterogeneity.rst +++ b/docs6/record-heterogeneity.rst @@ -16,7 +16,7 @@ CSV and pretty-print Miller simply prints a newline and a new header when there is a schema change. When there is no schema change, you get CSV per se as a special case. Likewise, Miller reads heterogeneous CSV or pretty-print input the same way. The difference between CSV and CSV-lite is that the former is RFC4180-compliant, while the latter readily handles heterogeneous data (which is non-compliant). For example: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/het.dkvp @@ -26,7 +26,7 @@ Miller simply prints a newline and a new header when there is a schema change. W record_count=150,resource=/path/to/second/file resource=/some/other/path,loadsec=0.97,ok=false -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --ocsvlite cat data/het.dkvp @@ -45,7 +45,7 @@ Miller simply prints a newline and a new header when there is a schema change. W resource,loadsec,ok /some/other/path,0.97,false -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint cat data/het.dkvp @@ -66,7 +66,7 @@ Miller simply prints a newline and a new header when there is a schema change. W Miller handles explicit header changes as just shown. If your CSV input contains ragged data -- if there are implicit header changes -- you can use ``--allow-ragged-csv-input`` (or keystroke-saver ``--ragged``). For too-short data lines, values are filled with empty string; for too-long data lines, missing field names are replaced with positional indices (counting up from 1, not 0), as follows: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/ragged.csv @@ -75,7 +75,7 @@ Miller handles explicit header changes as just shown. If your CSV input contains 4,5 6,7,8,9 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --oxtab --allow-ragged-csv-input cat data/ragged.csv @@ -94,7 +94,7 @@ Miller handles explicit header changes as just shown. If your CSV input contains You may also find Miller's ``group-like`` feature handy (see also :doc:`reference`): -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --ocsvlite group-like data/het.dkvp @@ -107,7 +107,7 @@ You may also find Miller's ``group-like`` feature handy (see also :doc:`referenc 100,/path/to/file 150,/path/to/second/file -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint group-like data/het.dkvp @@ -125,7 +125,7 @@ Key-value-pair, vertical-tabular, and index-numbered formats For these formats, record-heterogeneity comes naturally: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/het.dkvp @@ -135,7 +135,7 @@ For these formats, record-heterogeneity comes naturally: record_count=150,resource=/path/to/second/file resource=/some/other/path,loadsec=0.97,ok=false -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --onidx --ofs ' ' cat data/het.dkvp @@ -145,7 +145,7 @@ For these formats, record-heterogeneity comes naturally: 150 /path/to/second/file /some/other/path 0.97 false -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --oxtab cat data/het.dkvp @@ -167,7 +167,7 @@ For these formats, record-heterogeneity comes naturally: loadsec 0.97 ok false -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --oxtab group-like data/het.dkvp @@ -194,7 +194,7 @@ For processing Miller operates on specified fields and takes the rest along: for example, if you are sorting on the ``count`` field then all records in the input stream must have a ``count`` field but the other fields can vary, and moreover the sorted-on field name(s) don't need to be in the same position on each line: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/sort-het.dkvp @@ -206,7 +206,7 @@ Miller operates on specified fields and takes the rest along: for example, if yo count=100,color=green count=450 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr sort -n count data/sort-het.dkvp diff --git a/docs6/reference-dsl.rst b/docs6/reference-dsl.rst index 8026097dd..5776b6ed2 100644 --- a/docs6/reference-dsl.rst +++ b/docs6/reference-dsl.rst @@ -11,7 +11,7 @@ Here's comparison of verbs and ``put``/``filter`` DSL expressions: Example: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr stats1 -a sum -f x -g a data/small @@ -27,7 +27,7 @@ Example: Example: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q '@x_sum[$a] += $x; end{emit @x_sum, "a"}' data/small @@ -45,7 +45,7 @@ Please see :doc:`reference-verbs` for information on verbs other than ``put`` an The essential usages of ``mlr filter`` and ``mlr put`` are for record-selection and record-updating expressions, respectively. For example, given the following input data: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/small @@ -57,7 +57,7 @@ The essential usages of ``mlr filter`` and ``mlr put`` are for record-selection you might retain only the records whose ``a`` field has value ``eks``: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr filter '$a == "eks"' data/small @@ -66,7 +66,7 @@ you might retain only the records whose ``a`` field has value ``eks``: or you might add a new field which is a function of existing fields: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put '$ab = $a . "_" . $b ' data/small @@ -96,7 +96,7 @@ Expression formatting Multiple expressions may be given, separated by semicolons, and each may refer to the ones before: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ ruby -e '10.times{|i|puts "i=#{i}"}' | mlr --opprint put '$j = $i + 1; $k = $i +$j' @@ -114,7 +114,7 @@ Multiple expressions may be given, separated by semicolons, and each may refer t Newlines within the expression are ignored, which can help increase legibility of complex expressions: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint put ' @@ -136,7 +136,7 @@ Newlines within the expression are ignored, which can help increase legibility o hat wye 10002 0.321507044286237609 0.568893318795083758 5 9 4 2 data/small2 pan zee 10003 0.272054845593895200 0.425789896597056627 5 10 5 2 data/small2 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint filter '($x > 0.5 && $y < 0.5) || ($x < 0.5 && $y > 0.5)' then stats2 -a corr -f x,y data/medium @@ -150,7 +150,7 @@ Expressions from files The simplest way to enter expressions for ``put`` and ``filter`` is between single quotes on the command line, e.g. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/small put '$xy = sqrt($x**2 + $y**2)' @@ -160,7 +160,7 @@ The simplest way to enter expressions for ``put`` and ``filter`` is between sing a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463,xy=0.404317 a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729,xy=1.036584 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/small put 'func f(a, b) { return sqrt(a**2 + b**2) } $xy = f($x, $y)' @@ -173,7 +173,7 @@ The simplest way to enter expressions for ``put`` and ``filter`` is between sing You may, though, find it convenient to put expressions into files for reuse, and read them **using the -f option**. For example: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/fe-example-3.mlr @@ -182,7 +182,7 @@ You may, though, find it convenient to put expressions into files for reuse, and } $xy = f($x, $y) -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/small put -f data/fe-example-3.mlr @@ -194,7 +194,7 @@ You may, though, find it convenient to put expressions into files for reuse, and If you have some of the logic in a file and you want to write the rest on the command line, you can **use the -f and -e options together**: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/fe-example-4.mlr @@ -202,7 +202,7 @@ If you have some of the logic in a file and you want to write the rest on the co return sqrt(a**2 + b**2) } -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/small put -f data/fe-example-4.mlr -e '$xy = f($x, $y)' @@ -223,7 +223,7 @@ Semicolons, commas, newlines, and curly braces Miller uses **semicolons as statement separators**, not statement terminators. This means you can write: -.. code-block:: +.. code-block:: none mlr put 'x=1' mlr put 'x=1;$y=2' @@ -232,13 +232,13 @@ Miller uses **semicolons as statement separators**, not statement terminators. T Semicolons are optional after closing curly braces (which close conditionals and loops as discussed below). -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ echo x=1,y=2 | mlr put 'while (NF < 10) { $[NF+1] = ""} $foo = "bar"' x=1,y=2,3=,4=,5=,6=,7=,8=,9=,10=,foo=bar -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ echo x=1,y=2 | mlr put 'while (NF < 10) { $[NF+1] = ""}; $foo = "bar"' @@ -246,7 +246,7 @@ Semicolons are optional after closing curly braces (which close conditionals and Semicolons are required between statements even if those statements are on separate lines. **Newlines** are for your convenience but have no syntactic meaning: line endings do not terminate statements. For example, adjacent assignment statements must be separated by semicolons even if those statements are on separate lines: -.. code-block:: +.. code-block:: none mlr put ' $x = 1 @@ -260,7 +260,7 @@ Semicolons are required between statements even if those statements are on separ **Trailing commas** are allowed in function/subroutine definitions, function/subroutine callsites, and map literals. This is intended for (although not restricted to) the multi-line case: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csvlite --from data/a.csv put ' @@ -286,19 +286,19 @@ Semicolons are required between statements even if those statements are on separ Bodies for all compound statements must be enclosed in **curly braces**, even if the body is a single statement: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 mlr put 'if ($x == 1) $y = 2' # Syntax error -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 mlr put 'if ($x == 1) { $y = 2 }' # This is OK Bodies for compound statements may be empty: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 mlr put 'if ($x == 1) { }' # This no-op is syntactically acceptable @@ -325,7 +325,7 @@ These are written all in capital letters, such as ``NR``, ``NF``, ``FILENAME``, Namely, Miller supports the following five built-in variables for :doc:`filter and put `, all ``awk``-inspired: ``NF``, ``NR``, ``FNR``, ``FILENUM``, and ``FILENAME``, as well as the mathematical constants ``M_PI`` and ``M_E``. Lastly, the ``ENV`` hashmap allows read access to environment variables, e.g. ``ENV["HOME"]`` or ``ENV["foo_".$hostname]``. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr filter 'FNR == 2' data/small* @@ -333,7 +333,7 @@ Namely, Miller supports the following five built-in variables for :doc:`filter a 1=pan,2=pan,3=1,4=0.3467901443380824,5=0.7268028627434533 a=wye,b=eks,i=10000,x=0.734806020620654365,y=0.884788571337605134 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put '$fnr = FNR' data/small* @@ -358,7 +358,7 @@ Their values of ``NF``, ``NR``, ``FNR``, ``FILENUM``, and ``FILENAME`` change fr Their **scope is global**: you can refer to them in any ``filter`` or ``put`` statement. Their values are assigned by the input-record reader: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv put '$nr = NR' data/a.csv @@ -366,7 +366,7 @@ Their **scope is global**: you can refer to them in any ``filter`` or ``put`` st 1,2,3,1 4,5,6,2 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv repeat -n 3 then put '$nr = NR' data/a.csv @@ -391,12 +391,12 @@ If field names have **special characters** such as ``.`` then you can use braces You may also use a **computed field name** in square brackets, e.g. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ echo a=3,b=4 | mlr filter '$["x"] < 0.5' -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ echo s=green,t=blue,a=3,b=4 | mlr put '$[$s."_".$t] = $a * $b' @@ -421,7 +421,7 @@ Use ``$[[3]]`` to access the name of field 3. More generally, any expression ev Then using a computed field name, ``$[ $[[3]] ]`` is the value in the third field. This has the shorter equivalent notation ``$[[[3]]]``. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr cat data/small @@ -431,7 +431,7 @@ Then using a computed field name, ``$[ $[[3]] ]`` is the value in the third fiel a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463 a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put '$[[3]] = "NEW"' data/small @@ -441,7 +441,7 @@ Then using a computed field name, ``$[ $[[3]] ]`` is the value in the third fiel a=eks,b=wye,NEW=4,x=0.38139939387114097,y=0.13418874328430463 a=wye,b=pan,NEW=5,x=0.5732889198020006,y=0.8636244699032729 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put '$[[[3]]] = "NEW"' data/small @@ -451,7 +451,7 @@ Then using a computed field name, ``$[ $[[3]] ]`` is the value in the third fiel a=eks,b=wye,i=NEW,x=0.38139939387114097,y=0.13418874328430463 a=wye,b=pan,i=NEW,x=0.5732889198020006,y=0.8636244699032729 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put '$NEW = $[[NR]]' data/small @@ -461,7 +461,7 @@ Then using a computed field name, ``$[ $[[3]] ]`` is the value in the third fiel a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463,NEW=x a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729,NEW=y -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put '$NEW = $[[[NR]]]' data/small @@ -471,7 +471,7 @@ Then using a computed field name, ``$[ $[[3]] ]`` is the value in the third fiel a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463,NEW=0.381399 a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729,NEW=0.863624 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put '$[[[NR]]] = "NEW"' data/small @@ -483,7 +483,7 @@ Then using a computed field name, ``$[ $[[3]] ]`` is the value in the third fiel Right-hand side accesses to non-existent fields -- i.e. with index less than 1 or greater than ``NF`` -- return an absent value. Likewise, left-hand side accesses only refer to fields which already exist. For example, if a field has 5 records then assigning the name or value of the 6th (or 600th) field results in a no-op. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put '$[[6]] = "NEW"' data/small @@ -493,7 +493,7 @@ Right-hand side accesses to non-existent fields -- i.e. with index less than 1 o a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463 a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put '$[[[6]]] = "NEW"' data/small @@ -514,7 +514,7 @@ Just as for field names in stream records, if you want to define out-of-stream v You may use a **computed key** in square brackets, e.g. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ echo s=green,t=blue,a=3,b=4 | mlr put -q '@[$s."_".$t] = $a * $b; emit all' @@ -522,14 +522,14 @@ You may use a **computed key** in square brackets, e.g. Out-of-stream variables are **scoped** to the ``put`` command in which they appear. In particular, if you have two or more ``put`` commands separated by ``then``, each put will have its own set of out-of-stream variables: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/a.dkvp a=1,b=2,c=3 a=4,b=5,c=6 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put '@sum += $a; end {emit @sum}' then put 'is_present($a) {$a=10*$a; @sum += $a}; end {emit @sum}' data/a.dkvp @@ -547,7 +547,7 @@ Indexed out-of-stream variables Using an index on the ``@count`` and ``@sum`` variables, we get the benefit of the ``-g`` (group-by) option which ``mlr stats1`` and various other Miller commands have: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q ' @@ -569,7 +569,7 @@ Using an index on the ``@count`` and ``@sum`` variables, we get the benefit of t a=zee,x_sum=1.125680 a=hat,x_sum=0.031442 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr stats1 -a count,sum -f x -g a ../data/small @@ -581,7 +581,7 @@ Using an index on the ``@count`` and ``@sum`` variables, we get the benefit of t Indices can be arbitrarily deep -- here there are two or more of them: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/medium put -q ' @@ -621,7 +621,7 @@ The idea is that ``stats1``, and other Miller verbs, encapsulate frequently-used Begin/end blocks can be mixed with pattern/action blocks. For example: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put ' @@ -654,7 +654,7 @@ Local variables are similar to out-of-stream variables, except that their extent For example: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ # Here I'm using a specified random-number seed so this example always @@ -710,7 +710,7 @@ Things which are perhaps surprising compared to other languages: The following example demonstrates the scope rules: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/scope-example.mlr @@ -738,7 +738,7 @@ The following example demonstrates the scope rules: $outer_c = c; $outer_d = d; # there is no outer d defined so no assignment happens -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/scope-example.dat @@ -746,7 +746,7 @@ The following example demonstrates the scope rules: n=2,x=456 n=3,x=789 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --oxtab --from data/scope-example.dat put -f data/scope-example.mlr @@ -774,7 +774,7 @@ The following example demonstrates the scope rules: And this example demonstrates the type-declaration rules: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/type-decl-example.mlr @@ -823,7 +823,7 @@ Miller's ``put``/``filter`` DSL has four kinds of hashmaps. **Stream records** a For example, the following swaps the input stream's ``a`` and ``i`` fields, modifies ``y``, and drops the rest: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint put ' @@ -842,7 +842,7 @@ For example, the following swaps the input stream's ``a`` and ``i`` fields, modi Likewise, you can assign map literals to out-of-stream variables or local variables; pass them as arguments to user-defined functions, return them from functions, and so on: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/small put ' @@ -860,7 +860,7 @@ Likewise, you can assign map literals to out-of-stream variables or local variab Like out-of-stream and local variables, map literals can be multi-level: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/small put -q ' @@ -908,7 +908,7 @@ Type-test and type-assertion expressions The following ``is...`` functions take a value and return a boolean indicating whether the argument is of the indicated type. The ``assert_...`` functions return their argument if it is of the specified type, and cause a fatal error otherwise: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr -F | grep ^is @@ -929,7 +929,7 @@ The following ``is...`` functions take a value and return a boolean indicating w is_present is_string -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr -F | grep ^assert @@ -959,7 +959,7 @@ Local variables can be defined either untyped as in ``x = 1``, or typed as in `` The reason for ``num`` is that ``int`` and ``float`` typedecls are very precise: -.. code-block:: +.. code-block:: none float a = 0; # Runtime error since 0 is int not float int b = 1.0; # Runtime error since 1.0 is float not int @@ -970,7 +970,7 @@ A suggestion is to use ``num`` for general use when you want numeric content, an The ``var`` type declaration indicates no type restrictions, e.g. ``var x = 1`` has the same type restrictions on ``x`` as ``x = 1``. The difference is in intentional shadowing: if you have ``x = 1`` in outer scope and ``x = 2`` in inner scope (e.g. within a for-loop or an if-statement) then outer-scope ``x`` has value 2 after the second assignment. But if you have ``var x = 2`` in the inner scope, then you are declaring a variable scoped to the inner block.) For example: -.. code-block:: +.. code-block:: none x = 1; if (NR == 4) { @@ -978,7 +978,7 @@ The ``var`` type declaration indicates no type restrictions, e.g. ``var x = 1`` } print x; # Value of x is now two -.. code-block:: +.. code-block:: none x = 1; if (NR == 4) { @@ -988,7 +988,7 @@ The ``var`` type declaration indicates no type restrictions, e.g. ``var x = 1`` Likewise function arguments can optionally be typed, with type enforced when the function is called: -.. code-block:: +.. code-block:: none func f(map m, int i) { ... @@ -1003,7 +1003,7 @@ Likewise function arguments can optionally be typed, with type enforced when the Thirdly, function return values can be type-checked at the point of ``return`` using ``:`` and a typedecl after the parameter list: -.. code-block:: +.. code-block:: none func f(map m, int i): bool { ... @@ -1040,7 +1040,7 @@ There are three remaining kinds of variable assignment using out-of-stream varia Example recursive copy of out-of-stream variables: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint put -q '@v["sum"] += $x; @v["count"] += 1; end{dump; @w = @v; dump}' data/small @@ -1063,7 +1063,7 @@ Example recursive copy of out-of-stream variables: Example of out-of-stream variable assigned to full stream record, where the 2nd record is stashed, and the 4th record is overwritten with that: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put 'NR == 2 {@keep = $*}; NR == 4 {$* = @keep}' data/small @@ -1075,7 +1075,7 @@ Example of out-of-stream variable assigned to full stream record, where the 2nd Example of full stream record assigned to an out-of-stream variable, finding the record for which the ``x`` field has the largest value in the input stream: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/small @@ -1085,7 +1085,7 @@ Example of full stream record assigned to an out-of-stream variable, finding the a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463 a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint put -q 'is_null(@xmax) || $x > @xmax {@xmax=$x; @recmax=$*}; end {emit @recmax}' data/small @@ -1095,7 +1095,7 @@ Example of full stream record assigned to an out-of-stream variable, finding the Keywords for filter and put ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --help-all-keywords @@ -1398,7 +1398,7 @@ Operator precedence Operators are listed in order of decreasing precedence, highest first. -.. code-block:: +.. code-block:: none Operators Associativity --------- ------------- @@ -1440,7 +1440,7 @@ Pattern-action blocks These are reminiscent of ``awk`` syntax. They can be used to allow assignments to be done only when appropriate -- e.g. for math-function domain restrictions, regex-matching, and so on: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr cat data/put-gating-example-1.dkvp @@ -1450,7 +1450,7 @@ These are reminiscent of ``awk`` syntax. They can be used to allow assignments x=2 x=3 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put '$x > 0.0 { $y = log10($x); $z = sqrt($y) }' data/put-gating-example-1.dkvp @@ -1460,7 +1460,7 @@ These are reminiscent of ``awk`` syntax. They can be used to allow assignments x=2,y=0.301030,z=0.548662 x=3,y=0.477121,z=0.690740 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr cat data/put-gating-example-2.dkvp @@ -1468,7 +1468,7 @@ These are reminiscent of ``awk`` syntax. They can be used to allow assignments a=some other name a=xyz_789 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put '$a =~ "([a-z]+)_([0-9]+)" { $b = "left_\1"; $c = "right_\2" }' data/put-gating-example-2.dkvp @@ -1478,7 +1478,7 @@ These are reminiscent of ``awk`` syntax. They can be used to allow assignments This produces heteregenous output which Miller, of course, has no problems with (see :doc:`record-heterogeneity`). But if you want homogeneous output, the curly braces can be replaced with a semicolon between the expression and the body statements. This causes ``put`` to evaluate the boolean expression (along with any side effects, namely, regex-captures ``\1``, ``\2``, etc.) but doesn't use it as a criterion for whether subsequent assignments should be executed. Instead, subsequent assignments are done unconditionally: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put '$x > 0.0; $y = log10($x); $z = sqrt($y)' data/put-gating-example-1.dkvp @@ -1488,7 +1488,7 @@ This produces heteregenous output which Miller, of course, has no problems with x=2,y=0.301030,z=0.548662 x=3,y=0.477121,z=0.690740 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put '$a =~ "([a-z]+)_([0-9]+)"; $b = "left_\1"; $c = "right_\2"' data/put-gating-example-2.dkvp @@ -1501,17 +1501,17 @@ If-statements These are again reminiscent of ``awk``. Pattern-action blocks are a special case of ``if`` with no ``elif`` or ``else`` blocks, no ``if`` keyword, and parentheses optional around the boolean expression: -.. code-block:: +.. code-block:: none mlr put 'NR == 4 {$foo = "bar"}' -.. code-block:: +.. code-block:: none mlr put 'if (NR == 4) {$foo = "bar"}' Compound statements use ``elif`` (rather than ``elsif`` or ``else if``): -.. code-block:: +.. code-block:: none mlr put ' if (NR == 2) { @@ -1530,7 +1530,7 @@ While and do-while loops Miller's ``while`` and ``do-while`` are unsurprising in comparison to various languages, as are ``break`` and ``continue``: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ echo x=1,y=2 | mlr put ' @@ -1541,7 +1541,7 @@ Miller's ``while`` and ``do-while`` are unsurprising in comparison to various la ' x=1,y=2,3=,4=,5=,6=,7=,8=,9=,10=,foo=bar -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ echo x=1,y=2 | mlr put ' @@ -1570,7 +1570,7 @@ Key-only for-loops The ``key`` variable is always bound to the *key* of key-value pairs: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/small put ' @@ -1617,7 +1617,7 @@ The ``key`` variable is always bound to the *key* of key-value pairs: key:y value:0.863624 a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr -n put ' @@ -1638,7 +1638,7 @@ Key-value for-loops Single-level keys may be gotten at using either ``for(k,v)`` or ``for((k),v)``; multi-level keys may be gotten at using ``for((k1,k2,k3),v)`` and so on. The ``v`` variable will be bound to to a scalar value (a string or a number) if the map stops at that level, or to a map-valued variable if the map goes deeper. If the map isn't deep enough then the loop body won't be executed. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/for-srec-example.tbl @@ -1647,7 +1647,7 @@ Single-level keys may be gotten at using either ``for(k,v)`` or ``for((k),v)``; red green 120 11 195 yellow blue 140 0 240 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --pprint --from data/for-srec-example.tbl put ' @@ -1666,7 +1666,7 @@ Single-level keys may be gotten at using either ``for(k,v)`` or ``for((k),v)``; red green 120 11 195 326 326 326 yellow blue 140 0 240 380 380 380 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/small --opprint put 'for (k,v in $*) { $[k."_type"] = typeof(v) }' @@ -1681,7 +1681,7 @@ Note that the value of the current field in the for-loop can be gotten either us Important note: to avoid inconsistent looping behavior in case you're setting new fields (and/or unsetting existing ones) while looping over the record, **Miller makes a copy of the record before the loop: loop variables are bound from the copy and all other reads/writes involve the record itself**: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/small --opprint put ' @@ -1703,7 +1703,7 @@ Important note: to avoid inconsistent looping behavior in case you're setting ne It can be confusing to modify the stream record while iterating over a copy of it, so instead you might find it simpler to use a local variable in the loop and only update the stream record after the loop: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/small --opprint put ' @@ -1724,7 +1724,7 @@ It can be confusing to modify the stream record while iterating over a copy of i You can also start iterating on sub-hashmaps of an out-of-stream or local variable; you can loop over nested keys; you can loop over all out-of-stream variables. The bound variables are bound to a copy of the sub-hashmap as it was before the loop started. The sub-hashmap is specified by square-bracketed indices after ``in``, and additional deeper indices are bound to loop key-variables. The terminal values are bound to the loop value-variable whenever the keys are not too shallow. The value-variable may refer to a terminal (string, number) or it may be map-valued if the map goes deeper. Example indexing is as follows: -.. code-block:: +.. code-block:: none # Parentheses are optional for single key: for (k1, v in @a["b"]["c"]) { ... } @@ -1737,7 +1737,7 @@ You can also start iterating on sub-hashmaps of an out-of-stream or local variab That's confusing in the abstract, so a concrete example is in order. Suppose the out-of-stream variable ``@myvar`` is populated as follows: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr -n put --jknquoteint -q ' @@ -1766,7 +1766,7 @@ That's confusing in the abstract, so a concrete example is in order. Suppose the Then we can get at various values as follows: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr -n put --jknquoteint -q ' @@ -1789,7 +1789,7 @@ Then we can get at various values as follows: key=3,valuetype=map key=6,valuetype=map -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr -n put --jknquoteint -q ' @@ -1812,7 +1812,7 @@ Then we can get at various values as follows: key1=3,key2=4,valuetype=int key1=6,key2=7,valuetype=map -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr -n put --jknquoteint -q ' @@ -1839,7 +1839,7 @@ C-style triple-for loops These are supported as follows: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/small --opprint put ' @@ -1856,7 +1856,7 @@ These are supported as follows: eks wye 4 0.38139939387114097 0.13418874328430463 10 wye pan 5 0.5732889198020006 0.8636244699032729 15 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/small --opprint put ' @@ -1893,7 +1893,7 @@ Begin/end blocks Miller supports an ``awk``-like ``begin/end`` syntax. The statements in the ``begin`` block are executed before any input records are read; the statements in the ``end`` block are executed after the last input record is read. (If you want to execute some statement at the start of each file, not at the start of the first file as with ``begin``, you might use a pattern/action block of the form ``FNR == 1 { ... }``.) All statements outside of ``begin`` or ``end`` are, of course, executed on every input record. Semicolons separate statements inside or outside of begin/end blocks; semicolons are required between begin/end block bodies and any subsequent statement. For example: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put ' @@ -1915,7 +1915,7 @@ Miller supports an ``awk``-like ``begin/end`` syntax. The statements in the ``b Since uninitialized out-of-stream variables default to 0 for addition/substraction and 1 for multiplication when they appear on expression right-hand sides (not quite as in ``awk``, where they'd default to 0 either way), the above can be written more succinctly as -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put ' @@ -1936,7 +1936,7 @@ Since uninitialized out-of-stream variables default to 0 for addition/substracti The **put -q** option is a shorthand which suppresses printing of each output record, with only ``emit`` statements being output. So to get only summary outputs, one could write -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q ' @@ -1947,7 +1947,7 @@ The **put -q** option is a shorthand which suppresses printing of each output re We can do similarly with multiple out-of-stream variables: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q ' @@ -1963,7 +1963,7 @@ We can do similarly with multiple out-of-stream variables: This is of course not much different than -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr stats1 -a count,sum -f x ../data/small @@ -2046,7 +2046,7 @@ Details: * The ``print`` and ``dump`` keywords produce output immediately to standard output, or to specified file(s) or pipe-to command if present. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --help-keyword print @@ -2055,7 +2055,7 @@ Details: Example: mlr --from f.dat put -q 'for (k, v in $*) { print k . " => " . v }' Example: mlr --from f.dat put '(NR % 1000 == 0) { print > stderr, "Checkpoint ".NR}' -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --help-keyword dump @@ -2079,7 +2079,7 @@ Details: * ``mlr put`` sends the current record (possibly modified by the ``put`` expression) to the output record stream. Records are then input to the following verb in a ``then``-chain (if any), else printed to standard output (unless ``put -q``). The **tee** keyword *additionally* writes the output record to specified file(s) or pipe-to command, or immediately to ``stdout``/``stderr``. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --help-keyword tee @@ -2110,7 +2110,7 @@ Details: * ``mlr put``'s ``emitf``, ``emitp``, and ``emit`` send out-of-stream variables to the output record stream. These are then input to the following verb in a ``then``-chain (if any), else printed to standard output. When redirected with ``>``, ``>>``, or ``|``, they *instead* write the out-of-stream variable(s) to specified file(s) or pipe-to command, or immediately to ``stdout``/``stderr``. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --help-keyword emitf @@ -2141,7 +2141,7 @@ Details: Please see http://johnkerl.org/miller/doc for more information. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --help-keyword emitp @@ -2174,7 +2174,7 @@ Details: Please see http://johnkerl.org/miller/doc for more information. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --help-keyword emit @@ -2217,7 +2217,7 @@ There are three variants: ``emitf``, ``emit``, and ``emitp``. Keep in mind that Use **emitf** to output several out-of-stream variables side-by-side in the same output record. For ``emitf`` these mustn't have indexing using ``@name[...]``. Example: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q '@count += 1; @x_sum += $x; @y_sum += $y; end { emitf @count, @x_sum, @y_sum}' data/small @@ -2225,7 +2225,7 @@ Use **emitf** to output several out-of-stream variables side-by-side in the same Use **emit** to output an out-of-stream variable. If it's non-indexed you'll get a simple key-value pair: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/small @@ -2235,7 +2235,7 @@ Use **emit** to output an out-of-stream variable. If it's non-indexed you'll get a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463 a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q '@sum += $x; end { dump }' data/small @@ -2243,7 +2243,7 @@ Use **emit** to output an out-of-stream variable. If it's non-indexed you'll get "sum": 2.264762 } -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q '@sum += $x; end { emit @sum }' data/small @@ -2251,7 +2251,7 @@ Use **emit** to output an out-of-stream variable. If it's non-indexed you'll get If it's indexed then use as many names after ``emit`` as there are indices: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q '@sum[$a] += $x; end { dump }' data/small @@ -2263,7 +2263,7 @@ If it's indexed then use as many names after ``emit`` as there are indices: } } -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q '@sum[$a] += $x; end { emit @sum, "a" }' data/small @@ -2271,7 +2271,7 @@ If it's indexed then use as many names after ``emit`` as there are indices: a=eks,sum=1.140079 a=wye,sum=0.777892 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q '@sum[$a][$b] += $x; end { dump }' data/small @@ -2291,7 +2291,7 @@ If it's indexed then use as many names after ``emit`` as there are indices: } } -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q '@sum[$a][$b] += $x; end { emit @sum, "a", "b" }' data/small @@ -2301,7 +2301,7 @@ If it's indexed then use as many names after ``emit`` as there are indices: a=wye,b=wye,sum=0.204603 a=wye,b=pan,sum=0.573289 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q '@sum[$a][$b][$i] += $x; end { dump }' data/small @@ -2331,7 +2331,7 @@ If it's indexed then use as many names after ``emit`` as there are indices: } } -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q '@sum[$a][$b][$i] += $x; end { emit @sum, "a", "b", "i" }' data/small @@ -2343,7 +2343,7 @@ If it's indexed then use as many names after ``emit`` as there are indices: Now for **emitp**: if you have as many names following ``emit`` as there are levels in the out-of-stream variable's hashmap, then ``emit`` and ``emitp`` do the same thing. Where they differ is when you don't specify as many names as there are hashmap levels. In this case, Miller needs to flatten multiple map indices down to output-record keys: ``emitp`` includes full prefixing (hence the ``p`` in ``emitp``) while ``emit`` takes the deepest hashmap key as the output-record key: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q '@sum[$a][$b] += $x; end { dump }' data/small @@ -2363,7 +2363,7 @@ Now for **emitp**: if you have as many names following ``emit`` as there are lev } } -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q '@sum[$a][$b] += $x; end { emit @sum, "a" }' data/small @@ -2371,7 +2371,7 @@ Now for **emitp**: if you have as many names following ``emit`` as there are lev a=eks,pan=0.758680,wye=0.381399 a=wye,wye=0.204603,pan=0.573289 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q '@sum[$a][$b] += $x; end { emit @sum }' data/small @@ -2379,7 +2379,7 @@ Now for **emitp**: if you have as many names following ``emit`` as there are lev pan=0.758680,wye=0.381399 wye=0.204603,pan=0.573289 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q '@sum[$a][$b] += $x; end { emitp @sum, "a" }' data/small @@ -2387,13 +2387,13 @@ Now for **emitp**: if you have as many names following ``emit`` as there are lev a=eks,sum:pan=0.758680,sum:wye=0.381399 a=wye,sum:wye=0.204603,sum:pan=0.573289 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q '@sum[$a][$b] += $x; end { emitp @sum }' data/small sum:pan:pan=0.346790,sum:eks:pan=0.758680,sum:eks:wye=0.381399,sum:wye:wye=0.204603,sum:wye:pan=0.573289 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --oxtab put -q '@sum[$a][$b] += $x; end { emitp @sum }' data/small @@ -2406,7 +2406,7 @@ Now for **emitp**: if you have as many names following ``emit`` as there are lev Use **--oflatsep** to specify the character which joins multilevel keys for ``emitp`` (it defaults to a colon): -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q --oflatsep / '@sum[$a][$b] += $x; end { emitp @sum, "a" }' data/small @@ -2414,13 +2414,13 @@ keys for ``emitp`` (it defaults to a colon): a=eks,sum/pan=0.758680,sum/wye=0.381399 a=wye,sum/wye=0.204603,sum/pan=0.573289 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q --oflatsep / '@sum[$a][$b] += $x; end { emitp @sum }' data/small sum/pan/pan=0.346790,sum/eks/pan=0.758680,sum/eks/wye=0.381399,sum/wye/wye=0.204603,sum/wye/pan=0.573289 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --oxtab put -q --oflatsep / '@sum[$a][$b] += $x; end { emitp @sum }' data/small @@ -2436,7 +2436,7 @@ Multi-emit statements You can emit **multiple map-valued expressions side-by-side** by including their names in parentheses: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/medium --opprint put -q ' @@ -2483,7 +2483,7 @@ Emit-all statements Use **emit all** (or ``emit @*`` which is synonymous) to output all out-of-stream variables. You can use the following idiom to get various accumulators output side-by-side (reminiscent of ``mlr stats1``): -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/small --opprint put -q '@v[$a][$b]["sum"] += $x; @v[$a][$b]["count"] += 1; end{emit @*,"a","b"}' @@ -2494,7 +2494,7 @@ Use **emit all** (or ``emit @*`` which is synonymous) to output all out-of-strea wye wye 0.204603 1 wye pan 0.573289 1 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/small --opprint put -q '@sum[$a][$b] += $x; @count[$a][$b] += 1; end{emit @*,"a","b"}' @@ -2512,7 +2512,7 @@ Use **emit all** (or ``emit @*`` which is synonymous) to output all out-of-strea wye wye 1 wye pan 1 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --from data/small --opprint put -q '@sum[$a][$b] += $x; @count[$a][$b] += 1; end{emit (@sum, @count),"a","b"}' @@ -2528,7 +2528,7 @@ Unset statements You can clear a map key by assigning the empty string as its value: ``$x=""`` or ``@x=""``. Using ``unset`` you can remove the key entirely. Examples: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/small @@ -2538,7 +2538,7 @@ You can clear a map key by assigning the empty string as its value: ``$x=""`` or a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463 a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put 'unset $x, $a' data/small @@ -2550,7 +2550,7 @@ You can clear a map key by assigning the empty string as its value: ``$x=""`` or This can also be done, of course, using ``mlr cut -x``. You can also clear out-of-stream or local variables, at the base name level, or at an indexed sublevel: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q '@sum[$a][$b] += $x; end { dump; unset @sum; dump }' data/small @@ -2572,7 +2572,7 @@ This can also be done, of course, using ``mlr cut -x``. You can also clear out-o { } -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q '@sum[$a][$b] += $x; end { dump; unset @sum["eks"]; dump }' data/small @@ -2610,14 +2610,14 @@ Filter statements You can use ``filter`` within ``put``. In fact, the following two are synonymous: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr filter 'NR==2 || NR==3' data/small a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797 a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put 'filter NR==2 || NR==3' data/small @@ -2626,7 +2626,7 @@ You can use ``filter`` within ``put``. In fact, the following two are synonymous The former, of course, is much easier to type. But the latter allows you to define more complex expressions for the filter, and/or do other things in addition to the filter: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put '@running_sum += $x; filter @running_sum > 1.3' data/small @@ -2634,7 +2634,7 @@ The former, of course, is much easier to type. But the latter allows you to defi a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463 a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put '$z = $x * $y; filter $z > 0.3' data/small @@ -4917,7 +4917,7 @@ User-defined functions Here's the obligatory example of a recursive function to compute the factorial function: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint --from data/small put ' @@ -4966,7 +4966,7 @@ User-defined subroutines Example: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint --from data/small put -q ' diff --git a/docs6/reference-dsl.rst.in b/docs6/reference-dsl.rst.in index def47e95b..261f54ac1 100644 --- a/docs6/reference-dsl.rst.in +++ b/docs6/reference-dsl.rst.in @@ -121,19 +121,19 @@ POKI_INCLUDE_AND_RUN_ESCAPED(data/trailing-commas.sh)HERE Bodies for all compound statements must be enclosed in **curly braces**, even if the body is a single statement: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 mlr put 'if ($x == 1) $y = 2' # Syntax error -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 mlr put 'if ($x == 1) { $y = 2 }' # This is OK Bodies for compound statements may be empty: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 mlr put 'if ($x == 1) { }' # This no-op is syntactically acceptable @@ -363,7 +363,7 @@ Local variables can be defined either untyped as in ``x = 1``, or typed as in `` The reason for ``num`` is that ``int`` and ``float`` typedecls are very precise: -.. code-block:: +.. code-block:: none float a = 0; # Runtime error since 0 is int not float int b = 1.0; # Runtime error since 1.0 is float not int @@ -374,7 +374,7 @@ A suggestion is to use ``num`` for general use when you want numeric content, an The ``var`` type declaration indicates no type restrictions, e.g. ``var x = 1`` has the same type restrictions on ``x`` as ``x = 1``. The difference is in intentional shadowing: if you have ``x = 1`` in outer scope and ``x = 2`` in inner scope (e.g. within a for-loop or an if-statement) then outer-scope ``x`` has value 2 after the second assignment. But if you have ``var x = 2`` in the inner scope, then you are declaring a variable scoped to the inner block.) For example: -.. code-block:: +.. code-block:: none x = 1; if (NR == 4) { @@ -382,7 +382,7 @@ The ``var`` type declaration indicates no type restrictions, e.g. ``var x = 1`` } print x; # Value of x is now two -.. code-block:: +.. code-block:: none x = 1; if (NR == 4) { @@ -392,7 +392,7 @@ The ``var`` type declaration indicates no type restrictions, e.g. ``var x = 1`` Likewise function arguments can optionally be typed, with type enforced when the function is called: -.. code-block:: +.. code-block:: none func f(map m, int i) { ... @@ -407,7 +407,7 @@ Likewise function arguments can optionally be typed, with type enforced when the Thirdly, function return values can be type-checked at the point of ``return`` using ``:`` and a typedecl after the parameter list: -.. code-block:: +.. code-block:: none func f(map m, int i): bool { ... @@ -466,7 +466,7 @@ Operator precedence Operators are listed in order of decreasing precedence, highest first. -.. code-block:: +.. code-block:: none Operators Associativity --------- ------------- @@ -527,11 +527,11 @@ If-statements These are again reminiscent of ``awk``. Pattern-action blocks are a special case of ``if`` with no ``elif`` or ``else`` blocks, no ``if`` keyword, and parentheses optional around the boolean expression: -.. code-block:: +.. code-block:: none mlr put 'NR == 4 {$foo = "bar"}' -.. code-block:: +.. code-block:: none mlr put 'if (NR == 4) {$foo = "bar"}' diff --git a/docs6/reference-verbs.rst b/docs6/reference-verbs.rst index a335b85fc..20f72fe29 100644 --- a/docs6/reference-verbs.rst +++ b/docs6/reference-verbs.rst @@ -15,7 +15,7 @@ Here's a comparison of verbs and ``put``/``filter`` DSL expressions: Example: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr stats1 -a sum -f x -g a data/small @@ -31,7 +31,7 @@ Example: Example: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put -q '@x_sum[$a] += $x; end{emit @x_sum, "a"}' data/small @@ -52,20 +52,20 @@ altkv Map list of values to alternating key/value pairs. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr altkv -h Usage: mlr altkv [no options] Given fields with values of the form a,b,c,d,e,f emits a=b,c=d,e=f pairs. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ echo 'a,b,c,d,e,f' | mlr altkv a=b,c=d,e=f -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ echo 'a,b,c,d,e,f,g' | mlr altkv @@ -78,7 +78,7 @@ bar Cheesy bar-charting. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr bar -h @@ -96,7 +96,7 @@ Cheesy bar-charting. --auto Automatically computes limits, ignoring --lo and --hi. Holds all records in memory before producing any output. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint cat data/small @@ -107,7 +107,7 @@ Cheesy bar-charting. eks wye 4 0.38139939387114097 0.13418874328430463 wye pan 5 0.5732889198020006 0.8636244699032729 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint bar --lo 0 --hi 1 -f x,y data/small @@ -118,7 +118,7 @@ Cheesy bar-charting. eks wye 4 ***************......................... *****................................... wye pan 5 **********************.................. **********************************...... -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint bar --lo 0.4 --hi 0.6 -f x,y data/small @@ -129,7 +129,7 @@ Cheesy bar-charting. eks wye 4 #....................................... #....................................... wye pan 5 **********************************...... ***************************************# -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint bar --auto -f x,y data/small @@ -145,7 +145,7 @@ Cheesy bar-charting. bootstrap ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr bootstrap --help @@ -158,7 +158,7 @@ bootstrap The canonical use for bootstrap sampling is to put error bars on statistical quantities, such as mean. For example: -.. code-block:: +.. code-block:: none $ mlr --opprint stats1 -a mean,count -f u -g color data/colored-shapes.dkvp color u_mean u_count @@ -169,7 +169,7 @@ The canonical use for bootstrap sampling is to put error bars on statistical qua blue 0.517717 1470 orange 0.490532 303 -.. code-block:: +.. code-block:: none $ mlr --opprint bootstrap then stats1 -a mean,count -f u -g color data/colored-shapes.dkvp color u_mean u_count @@ -180,7 +180,7 @@ The canonical use for bootstrap sampling is to put error bars on statistical qua blue 0.512529 1496 orange 0.521030 321 -.. code-block:: +.. code-block:: none $ mlr --opprint bootstrap then stats1 -a mean,count -f u -g color data/colored-shapes.dkvp color u_mean u_count @@ -191,7 +191,7 @@ The canonical use for bootstrap sampling is to put error bars on statistical qua green 0.496803 1075 purple 0.486337 1199 -.. code-block:: +.. code-block:: none $ mlr --opprint bootstrap then stats1 -a mean,count -f u -g color data/colored-shapes.dkvp color u_mean u_count @@ -209,7 +209,7 @@ cat Most useful for format conversions (see :doc:`file-formats`, and concatenating multiple same-schema CSV files to have the same header: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr cat -h @@ -222,7 +222,7 @@ Most useful for format conversions (see :doc:`file-formats`, and concatenating m -v Write a low-level record-structure dump to stderr. -N {name} Prepend field {name} to each record with record-counter starting at 1 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/a.csv @@ -230,14 +230,14 @@ Most useful for format conversions (see :doc:`file-formats`, and concatenating m 1,2,3 4,5,6 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/b.csv a,b,c 7,8,9 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv cat data/a.csv data/b.csv @@ -246,7 +246,7 @@ Most useful for format conversions (see :doc:`file-formats`, and concatenating m 4,5,6 7,8,9 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --oxtab cat data/a.csv data/b.csv @@ -262,7 +262,7 @@ Most useful for format conversions (see :doc:`file-formats`, and concatenating m b 8 c 9 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv cat -n data/a.csv data/b.csv @@ -271,7 +271,7 @@ Most useful for format conversions (see :doc:`file-formats`, and concatenating m 2,4,5,6 3,7,8,9 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint cat data/small @@ -282,7 +282,7 @@ Most useful for format conversions (see :doc:`file-formats`, and concatenating m eks wye 4 0.38139939387114097 0.13418874328430463 wye pan 5 0.5732889198020006 0.8636244699032729 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint cat -n -g a data/small @@ -298,7 +298,7 @@ Most useful for format conversions (see :doc:`file-formats`, and concatenating m check ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr check --help @@ -311,7 +311,7 @@ check clean-whitespace ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr clean-whitespace --help @@ -328,7 +328,7 @@ clean-whitespace It is an error to specify -k as well as -v -- to clean keys and values, leave off -k as well as -v. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --ojson cat data/clean-whitespace.csv @@ -336,7 +336,7 @@ clean-whitespace { " Name ": "Bob Wang ", " Preference ": " red " } { " Name ": " Carol Vee", " Preference ": " yellow" } -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --ojson clean-whitespace -k data/clean-whitespace.csv @@ -344,7 +344,7 @@ clean-whitespace { "Name": "Bob Wang ", "Preference": " red " } { "Name": " Carol Vee", "Preference": " yellow" } -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --ojson clean-whitespace -v data/clean-whitespace.csv @@ -352,7 +352,7 @@ clean-whitespace { " Name ": "Bob Wang", " Preference ": "red" } { " Name ": "Carol Vee", " Preference ": "yellow" } -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --ojson clean-whitespace data/clean-whitespace.csv @@ -373,7 +373,7 @@ Function links: count ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr count --help @@ -385,13 +385,13 @@ count -n Show only the number of distinct values. Not interesting without -g. -o {name} Field name for output count. Default "count". -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr count data/medium count=10000 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr count -g a data/medium @@ -401,13 +401,13 @@ count a=zee,count=2047 a=hat,count=1941 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr count -n -g a data/medium count=5 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr count -g b data/medium @@ -417,13 +417,13 @@ count b=eks,count=2008 b=hat,count=2050 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr count -n -g b data/medium count=5 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr count -g a,b data/medium @@ -458,7 +458,7 @@ count count-distinct ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr count-distinct --help @@ -477,7 +477,7 @@ count-distinct for distinct a field values and counts for distinct b field values separately. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr count-distinct -f a,b then sort -nr count data/medium @@ -507,7 +507,7 @@ count-distinct a=hat,b=pan,count=363 a=eks,b=zee,count=357 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr count-distinct -u -f a,b data/medium @@ -522,7 +522,7 @@ count-distinct field=b,value=eks,count=2008 field=b,value=hat,count=2050 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr count-distinct -f a,b -o someothername then sort -nr someothername data/medium @@ -552,7 +552,7 @@ count-distinct a=hat,b=pan,someothername=363 a=eks,b=zee,someothername=357 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr count-distinct -n -f a,b data/medium @@ -563,7 +563,7 @@ count-distinct count-similar ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr count-similar --help @@ -574,7 +574,7 @@ count-similar -g {d,e,f} Group-by-field names for counts. -o {name} Field name for output count. Default "count". -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint head -n 20 data/medium @@ -600,7 +600,7 @@ count-similar zee pan 19 0.43144132839222604 0.8442204830496998 eks wye 20 0.38245149780530685 0.4730652428100751 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint head -n 20 then count-similar -g a data/medium @@ -626,7 +626,7 @@ count-similar hat wye 9 0.03144187646093577 0.7495507603507059 2 hat zee 18 0.05727869223575699 0.13343527626645157 2 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint head -n 20 then count-similar -g a then sort -f a data/medium @@ -657,7 +657,7 @@ count-similar cut ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr cut --help @@ -678,7 +678,7 @@ cut mlr cut -r -f '^status$,"sda[0-9]"' mlr cut -r -f '^status$,"sda[0-9]"i' (this is case-insensitive) -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint cat data/small @@ -689,7 +689,7 @@ cut eks wye 4 0.38139939387114097 0.13418874328430463 wye pan 5 0.5732889198020006 0.8636244699032729 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint cut -f y,x,i data/small @@ -700,13 +700,13 @@ cut 4 0.38139939387114097 0.13418874328430463 5 0.5732889198020006 0.8636244699032729 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ echo 'a=1,b=2,c=3' | mlr cut -f b,c,a a=1,b=2,c=3 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ echo 'a=1,b=2,c=3' | mlr cut -o -f b,c,a @@ -717,7 +717,7 @@ cut decimate ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr decimate --help @@ -733,7 +733,7 @@ decimate fill-down ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr fill-down --help @@ -752,7 +752,7 @@ fill-down -f Field names for fill-down. -h|--help Show this message. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/fill-down.csv @@ -761,7 +761,7 @@ fill-down 4,5,6 7,,9 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv fill-down -f b data/fill-down.csv @@ -770,7 +770,7 @@ fill-down 4,5,6 7,5,9 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv fill-down -a -f b data/fill-down.csv @@ -784,7 +784,7 @@ fill-down filter ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr filter --help @@ -881,7 +881,7 @@ Please see :doc:`reference-dsl` for more information about the expression langua format-values ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr format-values --help @@ -915,7 +915,7 @@ format-values -n Coerce field values autodetected as int to float, and then apply the float format. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint format-values data/small @@ -926,7 +926,7 @@ format-values eks wye 4 0.381399 0.134189 wye pan 5 0.573289 0.863624 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint format-values -n data/small @@ -937,7 +937,7 @@ format-values eks wye 4.000000 0.381399 0.134189 wye pan 5.000000 0.573289 0.863624 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint format-values -i %08llx -f %.6le -s X%sX data/small @@ -948,7 +948,7 @@ format-values XeksX XwyeX 00000004 3.813994e-01 1.341887e-01 XwyeX XpanX 00000005 5.732889e-01 8.636245e-01 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint format-values -i %08llx -f %.6le -s X%sX -n data/small @@ -964,7 +964,7 @@ format-values fraction ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr fraction --help @@ -991,7 +991,7 @@ fraction For example, suppose you have the following CSV file: -.. code-block:: +.. code-block:: none u=female,v=red,n=2458 u=female,v=green,n=192 @@ -1008,7 +1008,7 @@ For example, suppose you have the following CSV file: Then we can see what each record's ``n`` contributes to the total ``n``: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint fraction -f n data/fraction-example.csv @@ -1028,7 +1028,7 @@ Then we can see what each record's ``n`` contributes to the total ``n``: Using ``-g`` we can split those out by gender, or by color: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint fraction -f n -g u data/fraction-example.csv @@ -1046,7 +1046,7 @@ Using ``-g`` we can split those out by gender, or by color: male yellow 1192 0.293886 male orange 448 0.110454 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint fraction -f n -g v data/fraction-example.csv @@ -1068,7 +1068,7 @@ We can see, for example, that 70.9% of females have red (on the left) while 94.5 To convert fractions to percents, you may use ``-p``: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint fraction -f n -p data/fraction-example.csv @@ -1088,7 +1088,7 @@ To convert fractions to percents, you may use ``-p``: Another often-used idiom is to convert from a point distribution to a cumulative distribution, also known as "running sums". Here, you can use ``-c``: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint fraction -f n -p -c data/fraction-example.csv @@ -1106,7 +1106,7 @@ Another often-used idiom is to convert from a point distribution to a cumulative male yellow 1192 94.051255 male orange 448 100 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint fraction -f n -g u -p -c data/fraction-example.csv @@ -1129,7 +1129,7 @@ Another often-used idiom is to convert from a point distribution to a cumulative grep ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr grep -h @@ -1155,7 +1155,7 @@ grep group-by ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr group-by --help @@ -1164,7 +1164,7 @@ group-by This is similar to ``sort`` but with less work. Namely, Miller's sort has three steps: read through the data and append linked lists of records, one for each unique combination of the key-field values; after all records are read, sort the key-field values; then print each record-list. The group-by operation simply omits the middle sort. An example should make this more clear. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint group-by a data/small @@ -1175,7 +1175,7 @@ This is similar to ``sort`` but with less work. Namely, Miller's sort has three wye wye 3 0.20460330576630303 0.33831852551664776 wye pan 5 0.5732889198020006 0.8636244699032729 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint sort -f a data/small @@ -1193,7 +1193,7 @@ In this example, since the sort is on field ``a``, the first step is to group to group-like ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr group-like --help @@ -1202,7 +1202,7 @@ group-like This groups together records having the same schema (i.e. same ordered list of field names) which is useful for making sense of time-ordered output as described in :doc:`record-heterogeneity` -- in particular, in preparation for CSV or pretty-print output. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr cat data/het.dkvp @@ -1212,7 +1212,7 @@ This groups together records having the same schema (i.e. same ordered list of f record_count=150,resource=/path/to/second/file resource=/some/other/path,loadsec=0.97,ok=false -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint group-like data/het.dkvp @@ -1230,7 +1230,7 @@ This groups together records having the same schema (i.e. same ordered list of f having-fields ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr having-fields --help @@ -1251,7 +1251,7 @@ having-fields Similar to :ref:`reference-verbs-group-like`, this retains records with specified schema. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr cat data/het.dkvp @@ -1261,7 +1261,7 @@ Similar to :ref:`reference-verbs-group-like`, this retains records with specifie record_count=150,resource=/path/to/second/file resource=/some/other/path,loadsec=0.97,ok=false -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr having-fields --at-least resource data/het.dkvp @@ -1271,7 +1271,7 @@ Similar to :ref:`reference-verbs-group-like`, this retains records with specifie record_count=150,resource=/path/to/second/file resource=/some/other/path,loadsec=0.97,ok=false -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr having-fields --which-are resource,ok,loadsec data/het.dkvp @@ -1284,7 +1284,7 @@ Similar to :ref:`reference-verbs-group-like`, this retains records with specifie head ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr head --help @@ -1297,7 +1297,7 @@ head Note that ``head`` is distinct from :ref:`reference-verbs-top` -- ``head`` shows fields which appear first in the data stream; ``top`` shows fields which are numerically largest (or smallest). -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint head -n 4 data/medium @@ -1307,7 +1307,7 @@ Note that ``head`` is distinct from :ref:`reference-verbs-top` -- ``head`` shows wye wye 3 0.20460330576630303 0.33831852551664776 eks wye 4 0.38139939387114097 0.13418874328430463 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint head -n 1 -g b data/medium @@ -1323,7 +1323,7 @@ Note that ``head`` is distinct from :ref:`reference-verbs-top` -- ``head`` shows histogram ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr histogram --help @@ -1339,7 +1339,7 @@ histogram This is just a histogram; there's not too much to say here. A note about binning, by example: Suppose you use ``--lo 0.0 --hi 1.0 --nbins 10 -f x``. The input numbers less than 0 or greater than 1 aren't counted in any bin. Input numbers equal to 1 are counted in the last bin. That is, bin 0 has ``0.0 ≤ x < 0.1``, bin 1 has ``0.1 ≤ x < 0.2``, etc., but bin 9 has ``0.9 ≤ x ≤ 1.0``. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint put '$x2=$x**2;$x3=$x2*$x' then histogram -f x,x2,x3 --lo 0 --hi 1 --nbins 10 data/medium @@ -1355,7 +1355,7 @@ This is just a histogram; there's not too much to say here. A note about binning 0.800000 0.900000 986 571 383 0.900000 1.000000 1013 507 341 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint put '$x2=$x**2;$x3=$x2*$x' then histogram -f x,x2,x3 --lo 0 --hi 1 --nbins 10 -o my_ data/medium @@ -1376,7 +1376,7 @@ This is just a histogram; there's not too much to say here. A note about binning join ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr join --help @@ -1431,7 +1431,7 @@ Examples: Join larger table with IDs with smaller ID-to-name lookup table, showing only paired records: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsvlite --opprint cat data/join-left-example.csv @@ -1442,7 +1442,7 @@ Join larger table with IDs with smaller ID-to-name lookup table, showing only pa 400 david 500 edgar -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsvlite --opprint cat data/join-right-example.csv @@ -1468,7 +1468,7 @@ Join larger table with IDs with smaller ID-to-name lookup table, showing only pa present 400 present 300 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsvlite --opprint join -u -j id -r idcode -f data/join-left-example.csv data/join-right-example.csv @@ -1495,7 +1495,7 @@ Join larger table with IDs with smaller ID-to-name lookup table, showing only pa Same, but with sorting the input first: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsvlite --opprint sort -f idcode then join -j id -r idcode -f data/join-left-example.csv data/join-right-example.csv @@ -1522,7 +1522,7 @@ Same, but with sorting the input first: Same, but showing only unpaired records: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsvlite --opprint join --np --ul --ur -u -j id -r idcode -f data/join-left-example.csv data/join-right-example.csv @@ -1534,7 +1534,7 @@ Same, but showing only unpaired records: Use prefixing options to disambiguate between otherwise identical non-join field names: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csvlite --opprint cat data/self-join.csv data/self-join.csv @@ -1544,7 +1544,7 @@ Use prefixing options to disambiguate between otherwise identical non-join field 1 2 3 1 4 5 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csvlite --opprint join -j a --lp left_ --rp right_ -f data/self-join.csv data/self-join.csv @@ -1556,7 +1556,7 @@ Use prefixing options to disambiguate between otherwise identical non-join field Use zero join columns: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csvlite --opprint join -j "" --lp left_ --rp right_ -f data/self-join.csv data/self-join.csv @@ -1571,7 +1571,7 @@ Use zero join columns: label ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr label --help @@ -1588,7 +1588,7 @@ See also :ref:`reference-verbs-rename`. Example: Files such as ``/etc/passwd``, ``/etc/group``, and so on have implicit field names which are found in section-5 manpages. These field names may be made explicit as follows: -.. code-block:: +.. code-block:: none % grep -v '^#' /etc/passwd | mlr --nidx --fs : --opprint label name,password,uid,gid,gecos,home_dir,shell | head name password uid gid gecos home_dir shell @@ -1604,7 +1604,7 @@ Example: Files such as ``/etc/passwd``, ``/etc/group``, and so on have implicit Likewise, if you have CSV/CSV-lite input data which has somehow been bereft of its header line, you can re-add a header line using ``--implicit-csv-header`` and ``label``: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/headerless.csv @@ -1613,7 +1613,7 @@ Likewise, if you have CSV/CSV-lite input data which has somehow been bereft of i Alice,56,missing Carol,45,present -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv --implicit-csv-header cat data/headerless.csv @@ -1623,7 +1623,7 @@ Likewise, if you have CSV/CSV-lite input data which has somehow been bereft of i Alice,56,missing Carol,45,present -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv --implicit-csv-header label name,age,status data/headerless.csv @@ -1633,7 +1633,7 @@ Likewise, if you have CSV/CSV-lite input data which has somehow been bereft of i Alice,56,missing Carol,45,present -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --implicit-csv-header --opprint label name,age,status data/headerless.csv @@ -1648,7 +1648,7 @@ Likewise, if you have CSV/CSV-lite input data which has somehow been bereft of i least-frequent ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr least-frequent -h @@ -1662,7 +1662,7 @@ least-frequent -o {name} Field name for output count. Default "count". See also "mlr most-frequent". -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint --from data/colored-shapes.dkvp least-frequent -f shape -n 5 @@ -1671,7 +1671,7 @@ least-frequent triangle 3372 square 4115 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint --from data/colored-shapes.dkvp least-frequent -f shape,color -n 5 @@ -1682,7 +1682,7 @@ least-frequent circle green 287 circle purple 289 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint --from data/colored-shapes.dkvp least-frequent -f shape,color -n 5 -o someothername @@ -1693,7 +1693,7 @@ least-frequent circle green 287 circle purple 289 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint --from data/colored-shapes.dkvp least-frequent -f shape,color -n 5 -b @@ -1711,7 +1711,7 @@ See also :ref:`reference-verbs-most-frequent`. merge-fields ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr merge-fields --help @@ -1764,7 +1764,7 @@ This is like ``mlr stats1`` but all accumulation is done across fields within ea Examples: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csvlite --opprint cat data/inout.csv @@ -1773,7 +1773,7 @@ Examples: 526 320 963 780 220 888 705 831 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csvlite --opprint merge-fields -a min,max,sum -c _in,_out data/inout.csv @@ -1782,7 +1782,7 @@ Examples: 320 526 846 780 963 1743 220 888 1108 705 831 1536 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csvlite --opprint merge-fields -k -a sum -c _in,_out data/inout.csv @@ -1796,7 +1796,7 @@ Examples: most-frequent ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr most-frequent -h @@ -1810,7 +1810,7 @@ most-frequent -o {name} Field name for output count. Default "count". See also "mlr least-frequent". -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint --from data/colored-shapes.dkvp most-frequent -f shape -n 5 @@ -1819,7 +1819,7 @@ most-frequent triangle 3372 circle 2591 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint --from data/colored-shapes.dkvp most-frequent -f shape,color -n 5 @@ -1830,7 +1830,7 @@ most-frequent square yellow 589 square blue 589 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint --from data/colored-shapes.dkvp most-frequent -f shape,color -n 5 -o someothername @@ -1841,7 +1841,7 @@ most-frequent square yellow 589 square blue 589 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint --from data/colored-shapes.dkvp most-frequent -f shape,color -n 5 -b @@ -1859,7 +1859,7 @@ See also :ref:`reference-verbs-least-frequent`. nest ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr nest -h @@ -1916,7 +1916,7 @@ nest nothing ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr nothing -h @@ -1929,7 +1929,7 @@ nothing put ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put --help @@ -2037,7 +2037,7 @@ Please see the :doc:`reference-dsl` for more information about the expression la regularize ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr regularize --help @@ -2057,14 +2057,14 @@ See also :ref:`reference-verbs-reorder`. remove-empty-columns ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr remove-empty-columns --help Usage: mlr remove-empty-columns Omits fields which are empty on every input row. Non-streaming. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/remove-empty-columns.csv @@ -2073,7 +2073,7 @@ remove-empty-columns 2,,4,,5 3,,5,,7 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv remove-empty-columns data/remove-empty-columns.csv @@ -2089,7 +2089,7 @@ Since this verb needs to read all records to see if any of them has a non-empty rename ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr rename --help @@ -2114,7 +2114,7 @@ rename mlr rename -r 'Date_([0-9]+).*,\1' Rename all such fields to be of the form 20151015 mlr rename -r '"name"i,Name' Rename "name", "Name", "NAME", etc. to "Name" -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint cat data/small @@ -2125,7 +2125,7 @@ rename eks wye 4 0.38139939387114097 0.13418874328430463 wye pan 5 0.5732889198020006 0.8636244699032729 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint rename i,INDEX,b,COLUMN2 data/small @@ -2138,7 +2138,7 @@ rename As discussed in :doc:`performance`, ``sed`` is significantly faster than Miller at doing this. However, Miller is format-aware, so it knows to do renames only within specified field keys and not any others, nor in field values which may happen to contain the same pattern. Example: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ sed 's/y/COLUMN5/g' data/small @@ -2148,7 +2148,7 @@ As discussed in :doc:`performance`, ``sed`` is significantly faster than Miller a=eks,b=wCOLUMN5e,i=4,x=0.38139939387114097,COLUMN5=0.13418874328430463 a=wCOLUMN5e,b=pan,i=5,x=0.5732889198020006,COLUMN5=0.8636244699032729 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr rename y,COLUMN5 data/small @@ -2165,7 +2165,7 @@ See also :ref:`reference-verbs-label`. reorder ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr reorder --help @@ -2187,7 +2187,7 @@ This pivots specified field names to the start or end of the record -- for example when you have highly multi-column data and you want to bring a field or two to the front of line where you can give a quick visual scan. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint cat data/small @@ -2198,7 +2198,7 @@ two to the front of line where you can give a quick visual scan. eks wye 4 0.38139939387114097 0.13418874328430463 wye pan 5 0.5732889198020006 0.8636244699032729 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint reorder -f i,b data/small @@ -2209,7 +2209,7 @@ two to the front of line where you can give a quick visual scan. 4 wye eks 0.38139939387114097 0.13418874328430463 5 pan wye 0.5732889198020006 0.8636244699032729 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint reorder -e -f i,b data/small @@ -2225,7 +2225,7 @@ two to the front of line where you can give a quick visual scan. repeat ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr repeat --help @@ -2258,7 +2258,7 @@ This is useful in at least two ways: one, as a data-generator as in the above example using ``urand()``; two, for reconstructing individual samples from data which has been count-aggregated: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/repeat-example.dat @@ -2266,7 +2266,7 @@ samples from data which has been count-aggregated: color=red,count=4 color=green,count=3 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr repeat -f count then cut -x -f count data/repeat-example.dat @@ -2292,7 +2292,7 @@ p10,p50,p90``, etc. reshape ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr reshape --help @@ -2357,7 +2357,7 @@ reshape sample ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr sample --help @@ -2373,7 +2373,7 @@ uniform probability and no repeats in the sample. (If *n* is less than {field names}``, produce a *k*-sample for each distinct value of the specified field names. -.. code-block:: +.. code-block:: none $ mlr --opprint sample -k 4 data/colored-shapes.dkvp color shape flag i u v w x @@ -2429,7 +2429,7 @@ sampling, which works in the streaming case, is ``mlr filter 'urand() & sec2gmt ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr sec2gmt -h @@ -2448,7 +2448,7 @@ sec2gmt sec2gmtdate ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr sec2gmtdate -h @@ -2465,7 +2465,7 @@ sec2gmtdate seqgen ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr seqgen -h @@ -2480,7 +2480,7 @@ seqgen stop, and step are all integers. Step may be negative. It may not be zero unless start == stop. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr seqgen --stop 10 @@ -2495,7 +2495,7 @@ seqgen i=9 i=10 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr seqgen --start 20 --stop 40 --step 4 @@ -2506,7 +2506,7 @@ seqgen i=36 i=40 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr seqgen --start 40 --stop 20 --step -4 @@ -2522,7 +2522,7 @@ seqgen shuffle ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr shuffle -h @@ -2536,7 +2536,7 @@ shuffle skip-trivial-records ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr skip-trivial-records -h @@ -2545,7 +2545,7 @@ skip-trivial-records * those with zero fields; * those for which all fields have empty value. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/trivial-records.csv @@ -2555,7 +2555,7 @@ skip-trivial-records ,, ,8,9 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --csv skip-trivial-records data/trivial-records.csv @@ -2569,7 +2569,7 @@ skip-trivial-records sort ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr sort --help @@ -2593,7 +2593,7 @@ sort Example: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint sort -f a -nr x data/small @@ -2606,7 +2606,7 @@ Example: Here's an example filtering log data: suppose multiple threads (labeled here by color) are all logging progress counts to a single log file. The log file is (by nature) chronological, so the progress of various threads is interleaved: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ head -n 10 data/multicountdown.dat @@ -2625,7 +2625,7 @@ We can group these by thread by sorting on the thread ID (here, ``color``). Since Miller's sort is stable, this means that timestamps within each thread's log data are still chronological: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ head -n 20 data/multicountdown.dat | mlr --opprint sort -f color @@ -2654,7 +2654,7 @@ timestamps within each thread's log data are still chronological: Any records not having all specified sort keys will appear at the end of the output, in the order they were encountered, regardless of the specified sort order: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr sort -n x data/sort-missing.dkvp @@ -2663,7 +2663,7 @@ were encountered, regardless of the specified sort order: x=4 a=3 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr sort -nr x data/sort-missing.dkvp @@ -2677,14 +2677,14 @@ were encountered, regardless of the specified sort order: sort-within-records ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr sort-within-records -h Usage: mlr sort-within-records [no options] Outputs records sorted lexically ascending by keys. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/sort-within-records.json @@ -2704,7 +2704,7 @@ sort-within-records "a": 9 } -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --ijson --opprint cat data/sort-within-records.json @@ -2717,7 +2717,7 @@ sort-within-records c b a 7 8 9 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --json sort-within-records data/sort-within-records.json @@ -2725,7 +2725,7 @@ sort-within-records { "a": 5, "b": 4, "c": 6 } { "a": 9, "b": 8, "c": 7 } -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --ijson --opprint sort-within-records data/sort-within-records.json @@ -2739,7 +2739,7 @@ sort-within-records stats1 ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr stats1 --help @@ -2799,7 +2799,7 @@ These are simple univariate statistics on one or more number-valued fields (``count`` and ``mode`` apply to non-numeric fields as well), optionally categorized by one or more other fields. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --oxtab stats1 -a count,sum,min,p10,p50,mean,p90,max -f x,y data/medium @@ -2820,7 +2820,7 @@ optionally categorized by one or more other fields. y_p90 0.905366 y_max 0.999965 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint stats1 -a mean -f x,y -g b then sort -f b data/medium @@ -2831,7 +2831,7 @@ optionally categorized by one or more other fields. wye 0.497593 0.504596 zee 0.504242 0.502997 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint stats1 -a p50,p99 -f u,v -g color then put '$ur=$u_p99/$u_p50;$vr=$v_p99/$v_p50' data/colored-shapes.dkvp @@ -2843,7 +2843,7 @@ optionally categorized by one or more other fields. blue 0.525226 0.992655 0.485170 0.993873 1.889958 2.048505 orange 0.483548 0.993635 0.480913 0.989102 2.054884 2.056717 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint count-distinct -f shape then sort -nr count data/colored-shapes.dkvp @@ -2852,7 +2852,7 @@ optionally categorized by one or more other fields. triangle 3372 circle 2591 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint stats1 -a mode -f color -g shape data/colored-shapes.dkvp @@ -2866,7 +2866,7 @@ optionally categorized by one or more other fields. stats2 ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr stats2 --help @@ -2900,7 +2900,7 @@ stats2 These are simple bivariate statistics on one or more pairs of number-valued fields, optionally categorized by one or more fields. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --oxtab put '$x2=$x*$x; $xy=$x*$y; $y2=$y**2' then stats2 -a cov,corr -f x,y,y,y,x2,xy,x2,y2 data/medium @@ -2913,7 +2913,7 @@ fields, optionally categorized by one or more fields. x2_y2_cov -0.000310 x2_y2_corr -0.003425 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint put '$x2=$x*$x; $xy=$x*$y; $y2=$y**2' then stats2 -a linreg-ols,r2 -f x,y,y,y,xy,y2 -g a data/medium @@ -2928,7 +2928,7 @@ Here's an example simple line-fit. The ``x`` and ``y`` fields of the ``data/medium`` dataset are just independent uniformly distributed on the unit interval. Here we remove half the data and fit a line to it. -.. code-block:: +.. code-block:: none # Prepare input data: @@ -2962,7 +2962,7 @@ I use `pgr `_ for plotting; here's a screenshot Here's an example estimating time-to-completion for a set of jobs. Input data comes from a log file, with number of work units left to do in the ``count`` field and accumulated seconds in the ``upsec`` field, labeled by the ``color`` field: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ head -n 10 data/multicountdown.dat @@ -2979,7 +2979,7 @@ Here's an example estimating time-to-completion for a set of jobs. Input data co We can do a linear regression on count remaining as a function of time: with ``c = m*u+b`` we want to find the time when the count goes to zero, i.e. ``u=-b/m``. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --oxtab stats2 -a linreg-pca -f upsec,count -g color then put '$donesec = -$upsec_count_pca_b/$upsec_count_pca_m' data/multicountdown.dat @@ -3016,7 +3016,7 @@ We can do a linear regression on count remaining as a function of time: with ``c step ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr step --help @@ -3058,7 +3058,7 @@ step Most Miller commands are record-at-a-time, with the exception of ``stats1``, ``stats2``, and ``histogram`` which compute aggregate output. The ``step`` command is intermediate: it allows the option of adding fields which are functions of fields from previous records. Rsum is short for *running sum*. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint step -a shift,delta,rsum,counter -f x data/medium | head -15 @@ -3078,7 +3078,7 @@ Most Miller commands are record-at-a-time, with the exception of ``stats1``, ``s eks pan 13 0.4915175580479536 0.7709126592971468 0.3676141320555616 0.123903 6.188474 13 eks zee 14 0.5207382318405251 0.34141681118811673 0.4915175580479536 0.029221 6.709213 14 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint step -a shift,delta,rsum,counter -f x -g a data/medium | head -15 @@ -3098,7 +3098,7 @@ Most Miller commands are record-at-a-time, with the exception of ``stats1``, ``s eks pan 13 0.4915175580479536 0.7709126592971468 0.6117840605678454 -0.120267 2.243381 4 eks zee 14 0.5207382318405251 0.34141681118811673 0.4915175580479536 0.029221 2.764119 5 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint step -a ewma -f x -d 0.1,0.9 data/medium | head -15 @@ -3118,7 +3118,7 @@ Most Miller commands are record-at-a-time, with the exception of ``stats1``, ``s eks pan 13 0.4915175580479536 0.7709126592971468 0.446588 0.483050 eks zee 14 0.5207382318405251 0.34141681118811673 0.454003 0.516969 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint step -a ewma -f x -d 0.1,0.9 -o smooth,rough data/medium | head -15 @@ -3141,7 +3141,7 @@ Most Miller commands are record-at-a-time, with the exception of ``stats1``, ``s Example deriving uptime-delta from system uptime: -.. code-block:: +.. code-block:: none $ each 10 uptime | mlr -p step -a delta -f 11 ... @@ -3160,7 +3160,7 @@ Example deriving uptime-delta from system uptime: tac ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr tac --help @@ -3169,7 +3169,7 @@ tac Prints the records in the input stream in reverse order. Note: this requires Miller to retain all input records in memory before any output records are produced. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint cat data/a.csv @@ -3177,14 +3177,14 @@ Prints the records in the input stream in reverse order. Note: this requires Mil 1 2 3 4 5 6 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint cat data/b.csv a b c 7 8 9 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint tac data/a.csv data/b.csv @@ -3193,7 +3193,7 @@ Prints the records in the input stream in reverse order. Note: this requires Mil 4 5 6 1 2 3 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --icsv --opprint put '$filename=FILENAME' then tac data/a.csv data/b.csv @@ -3207,7 +3207,7 @@ Prints the records in the input stream in reverse order. Note: this requires Mil tail ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr tail --help @@ -3218,7 +3218,7 @@ tail Prints the last *n* records in the input stream, optionally by category. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint tail -n 4 data/colored-shapes.dkvp @@ -3228,7 +3228,7 @@ Prints the last *n* records in the input stream, optionally by category. yellow triangle 0 99990 0.3839424618160777 0.55952913620132 0.5113763011485609 4.307973891915119 yellow circle 1 99994 0.764950884927175 0.25284227383991364 0.49969878539567425 5.013809741826425 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint tail -n 1 -g shape data/colored-shapes.dkvp @@ -3242,7 +3242,7 @@ Prints the last *n* records in the input stream, optionally by category. tee ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr tee --help @@ -3263,7 +3263,7 @@ tee top ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr top --help @@ -3282,7 +3282,7 @@ top Note that ``top`` is distinct from :ref:`reference-verbs-head` -- ``head`` shows fields which appear first in the data stream; ``top`` shows fields which are numerically largest (or smallest). -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint top -n 4 -f x data/medium @@ -3292,7 +3292,7 @@ Note that ``top`` is distinct from :ref:`reference-verbs-head` -- ``head`` shows 3 0.999733 4 0.999563 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint top -n 4 -f x -o someothername data/medium @@ -3302,7 +3302,7 @@ Note that ``top`` is distinct from :ref:`reference-verbs-head` -- ``head`` shows 3 0.999733 4 0.999563 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint top -n 2 -f x -g a then sort -f a data/medium @@ -3323,7 +3323,7 @@ Note that ``top`` is distinct from :ref:`reference-verbs-head` -- ``head`` shows uniq ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr uniq --help @@ -3343,13 +3343,13 @@ uniq There are two main ways to use ``mlr uniq``: the first way is with ``-g`` to specify group-by columns. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ wc -l data/colored-shapes.dkvp 10078 data/colored-shapes.dkvp -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr uniq -g color,shape data/colored-shapes.dkvp @@ -3372,7 +3372,7 @@ There are two main ways to use ``mlr uniq``: the first way is with ``-g`` to spe color=orange,shape=square color=orange,shape=circle -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint uniq -g color,shape -c then sort -f color,shape data/colored-shapes.dkvp @@ -3396,7 +3396,7 @@ There are two main ways to use ``mlr uniq``: the first way is with ``-g`` to spe yellow square 589 yellow triangle 468 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint uniq -g color,shape -c -o someothername then sort -nr someothername data/colored-shapes.dkvp @@ -3420,7 +3420,7 @@ There are two main ways to use ``mlr uniq``: the first way is with ``-g`` to spe orange triangle 107 orange circle 68 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint uniq -n -g color,shape data/colored-shapes.dkvp @@ -3429,7 +3429,7 @@ There are two main ways to use ``mlr uniq``: the first way is with ``-g`` to spe The second main way to use ``mlr uniq`` is without group-by columns, using ``-a`` instead: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/repeats.dkvp @@ -3491,13 +3491,13 @@ The second main way to use ``mlr uniq`` is without group-by columns, using ``-a` color=yellow,shape=circle,flag=1 color=purple,shape=square,flag=0 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ wc -l data/repeats.dkvp 57 data/repeats.dkvp -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint uniq -a data/repeats.dkvp @@ -3510,14 +3510,14 @@ The second main way to use ``mlr uniq`` is without group-by columns, using ``-a` red square 1 yellow triangle 1 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint uniq -a -n data/repeats.dkvp count 7 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint uniq -a -c data/repeats.dkvp @@ -3535,7 +3535,7 @@ The second main way to use ``mlr uniq`` is without group-by columns, using ``-a` unsparsify ---------------------------------------------------------------- -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr unsparsify --help @@ -3556,7 +3556,7 @@ unsparsify Examples: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/sparse.json @@ -3565,7 +3565,7 @@ Examples: {"a":1,"v":2,"x":3} {"v":1,"w":2} -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --json unsparsify data/sparse.json @@ -3574,7 +3574,7 @@ Examples: { "a": 1, "b": "", "v": 2, "u": "", "x": 3, "w": "" } { "a": "", "b": "", "v": 1, "u": "", "x": "", "w": 2 } -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --ijson --opprint unsparsify data/sparse.json @@ -3584,7 +3584,7 @@ Examples: 1 - 2 - 3 - - - 1 - - 2 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --ijson --opprint unsparsify --fill-with missing data/sparse.json @@ -3594,7 +3594,7 @@ Examples: 1 missing 2 missing 3 missing missing missing 1 missing missing 2 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --ijson --opprint unsparsify -f a,b,u data/sparse.json @@ -3610,7 +3610,7 @@ Examples: v w a b u 1 2 - - - -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --ijson --opprint unsparsify -f a,b,u,v,w,x then regularize data/sparse.json diff --git a/docs6/reference-verbs.rst.in b/docs6/reference-verbs.rst.in index b51b5e3ea..f77022937 100644 --- a/docs6/reference-verbs.rst.in +++ b/docs6/reference-verbs.rst.in @@ -69,7 +69,7 @@ POKI_RUN_COMMAND{{mlr bootstrap --help}}HERE The canonical use for bootstrap sampling is to put error bars on statistical quantities, such as mean. For example: -.. code-block:: +.. code-block:: none $ mlr --opprint stats1 -a mean,count -f u -g color data/colored-shapes.dkvp color u_mean u_count @@ -80,7 +80,7 @@ The canonical use for bootstrap sampling is to put error bars on statistical qua blue 0.517717 1470 orange 0.490532 303 -.. code-block:: +.. code-block:: none $ mlr --opprint bootstrap then stats1 -a mean,count -f u -g color data/colored-shapes.dkvp color u_mean u_count @@ -91,7 +91,7 @@ The canonical use for bootstrap sampling is to put error bars on statistical qua blue 0.512529 1496 orange 0.521030 321 -.. code-block:: +.. code-block:: none $ mlr --opprint bootstrap then stats1 -a mean,count -f u -g color data/colored-shapes.dkvp color u_mean u_count @@ -102,7 +102,7 @@ The canonical use for bootstrap sampling is to put error bars on statistical qua green 0.496803 1075 purple 0.486337 1199 -.. code-block:: +.. code-block:: none $ mlr --opprint bootstrap then stats1 -a mean,count -f u -g color data/colored-shapes.dkvp color u_mean u_count diff --git a/docs6/reference.rst b/docs6/reference.rst index 2ae477613..1a9c8ebff 100644 --- a/docs6/reference.rst +++ b/docs6/reference.rst @@ -13,7 +13,7 @@ Command overview Whereas the Unix toolkit is made of the separate executables ``cat``, ``tail``, ``cut``, ``sort``, etc., Miller has subcommands, or **verbs**, invoked as follows: -.. code-block:: +.. code-block:: none mlr tac *.dat mlr cut --complement -f os_version *.dat @@ -39,7 +39,7 @@ Formats Options: -.. code-block:: +.. code-block:: none --dkvp --idkvp --odkvp --nidx --inidx --onidx @@ -51,7 +51,7 @@ Options: These are as discussed in :doc:`file-formats`, with the exception of ``--right`` which makes pretty-printed output right-aligned: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint cat data/small @@ -62,7 +62,7 @@ These are as discussed in :doc:`file-formats`, with the exception of ``--right`` eks wye 4 0.38139939387114097 0.13418874328430463 wye pan 5 0.5732889198020006 0.8636244699032729 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint --right cat data/small @@ -97,14 +97,14 @@ Compression Options: -.. code-block:: +.. code-block:: none --prepipe {command} The prepipe command is anything which reads from standard input and produces data acceptable to Miller. Nominally this allows you to use whichever decompression utilities you have installed on your system, on a per-file basis. If the command has flags, quote them: e.g. ``mlr --prepipe 'zcat -cf'``. Examples: -.. code-block:: +.. code-block:: none # These two produce the same output: $ gunzip < myfile1.csv.gz | mlr cut -f hostname,uptime @@ -113,14 +113,14 @@ The prepipe command is anything which reads from standard input and produces dat $ mlr --prepipe gunzip cut -f hostname,uptime myfile1.csv.gz myfile2.csv.gz $ mlr --prepipe gunzip --idkvp --oxtab cut -f hostname,uptime myfile1.dat.gz myfile2.dat.gz -.. code-block:: +.. code-block:: none # Similar to the above, but with compressed output as well as input: $ gunzip < myfile1.csv.gz | mlr cut -f hostname,uptime | gzip > outfile.csv.gz $ mlr --prepipe gunzip cut -f hostname,uptime myfile1.csv.gz | gzip > outfile.csv.gz $ mlr --prepipe gunzip cut -f hostname,uptime myfile1.csv.gz myfile2.csv.gz | gzip > outfile.csv.gz -.. code-block:: +.. code-block:: none # Similar to the above, but with different compression tools for input and output: $ gunzip < myfile1.csv.gz | mlr cut -f hostname,uptime | xz -z > outfile.csv.xz @@ -136,7 +136,7 @@ Miller has record separators ``IRS`` and ``ORS``, field separators ``IFS`` and ` Options: -.. code-block:: +.. code-block:: none --rs --irs --ors --fs --ifs --ofs --repifs @@ -157,7 +157,7 @@ Number formatting The command-line option ``--ofmt {format string}`` is the global number format for commands which generate numeric output, e.g. ``stats1``, ``stats2``, ``histogram``, and ``step``, as well as ``mlr put``. Examples: -.. code-block:: +.. code-block:: none --ofmt %.9le --ofmt %.6lf --ofmt %.0lf @@ -165,13 +165,13 @@ These are just familiar ``printf`` formats applied to double-precision numbers. To apply formatting to a single field, overriding the global ``ofmt``, use ``fmtnum`` function within ``mlr put``. For example: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ echo 'x=3.1,y=4.3' | mlr put '$z=fmtnum($x*$y,"%08lf")' x=3.1,y=4.3,z=13.330000 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ echo 'x=0xffff,y=0xff' | mlr put '$z=fmtnum(int($x*$y),"%08llx")' @@ -179,7 +179,7 @@ To apply formatting to a single field, overriding the global ``ofmt``, use ``fmt Input conversion from hexadecimal is done automatically on fields handled by ``mlr put`` and ``mlr filter`` as long as the field value begins with "0x". To apply output conversion to hexadecimal on a single column, you may use ``fmtnum``, or the keystroke-saving ``hexfmt`` function. Example: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ echo 'x=0xffff,y=0xff' | mlr put '$z=hexfmt($x*$y)' @@ -200,14 +200,14 @@ then-chaining In accord with the `Unix philosophy `_, you can pipe data into or out of Miller. For example: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 mlr cut --complement -f os_version *.dat | mlr sort -f hostname,uptime You can, if you like, instead simply chain commands together using the ``then`` keyword: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 mlr cut --complement -f os_version then sort -f hostname,uptime *.dat @@ -216,7 +216,7 @@ You can, if you like, instead simply chain commands together using the ``then`` Here's a performance comparison: -.. code-block:: +.. code-block:: none % cat piped.sh mlr cut -x -f i,y data/big | mlr sort -n y > /dev/null @@ -244,7 +244,7 @@ Auxiliary commands There are a few nearly-standalone programs which have nothing to do with the rest of Miller, do not participate in record streams, and do not deal with file formats. They might as well be little standalone executables but they're delivered within the main Miller executable for convenience. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr aux-list @@ -257,7 +257,7 @@ There are a few nearly-standalone programs which have nothing to do with the res netbsd-strptime For more information, please invoke mlr {subcommand} --help -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr lecat --help @@ -268,7 +268,7 @@ There are a few nearly-standalone programs which have nothing to do with the res --mono: don't try to colorize the output -h or --help: print this message -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr termcvt --help @@ -285,7 +285,7 @@ There are a few nearly-standalone programs which have nothing to do with the res Zero file names means read from standard input. Output is always to standard output; files are not written in-place. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr hex --help @@ -296,7 +296,7 @@ There are a few nearly-standalone programs which have nothing to do with the res -r: print only raw hex without leading offset indicators or trailing ASCII dump. -h or --help: print this message -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr unhex --help @@ -308,19 +308,19 @@ There are a few nearly-standalone programs which have nothing to do with the res Examples: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ echo 'Hello, world!' | mlr lecat --mono Hello, world![LF] -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ echo 'Hello, world!' | mlr termcvt --lf2crlf | mlr lecat --mono Hello, world![CR][LF] -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr hex data/budget.csv @@ -332,7 +332,7 @@ Examples: 00000050: 38 0a 67 72 65 65 6e 2c 36 37 38 2e 31 32 0a 6f |8.green,678.12.o| 00000060: 72 61 6e 67 65 2c 31 32 33 2e 34 35 0a |range,123.45.| -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr hex -r data/budget.csv @@ -344,7 +344,7 @@ Examples: 38 0a 67 72 65 65 6e 2c 36 37 38 2e 31 32 0a 6f 72 61 6e 67 65 2c 31 32 33 2e 34 35 0a -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr hex -r data/budget.csv | sed 's/20/2a/g' | mlr unhex @@ -396,7 +396,7 @@ Rules for null-handling: * Records with one or more empty sort-field values sort after records with all sort-field values present: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr cat data/sort-null.dat @@ -406,7 +406,7 @@ Rules for null-handling: x=9,b=10 a=5,b=7 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr sort -n a data/sort-null.dat @@ -416,7 +416,7 @@ Rules for null-handling: a=,b=4 x=9,b=10 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr sort -nr a data/sort-null.dat @@ -428,19 +428,19 @@ Rules for null-handling: * Functions/operators which have one or more *empty* arguments produce empty output: e.g. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ echo 'x=2,y=3' | mlr put '$a=$x+$y' x=2,y=3,a=5 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ echo 'x=,y=3' | mlr put '$a=$x+$y' x=,y=3,a= -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ echo 'x=,y=3' | mlr put '$a=log($x);$b=log($y)' @@ -448,7 +448,7 @@ Rules for null-handling: with the exception that the ``min`` and ``max`` functions are special: if one argument is non-null, it wins: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ echo 'x=,y=3' | mlr put '$a=min($x,$y);$b=max($x,$y)' @@ -456,13 +456,13 @@ with the exception that the ``min`` and ``max`` functions are special: if one ar * Functions of *absent* variables (e.g. ``mlr put '$y = log10($nonesuch)'``) evaluate to absent, and arithmetic/bitwise/boolean operators with both operands being absent evaluate to absent. Arithmetic operators with one absent operand return the other operand. More specifically, absent values act like zero for addition/subtraction, and one for multiplication: Furthermore, **any expression which evaluates to absent is not stored in the left-hand side of an assignment statement**: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ echo 'x=2,y=3' | mlr put '$a=$u+$v; $b=$u+$y; $c=$x+$y' x=2,y=3,b=3,c=5 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ echo 'x=2,y=3' | mlr put '$a=min($x,$v);$b=max($u,$y);$c=min($u,$v)' @@ -482,7 +482,7 @@ The reasoning is as follows: Since absent plus absent is absent (and likewise for other operators), accumulations such as ``@sum += $x`` work correctly on heterogenous data, as do within-record formulas if both operands are absent. If one operand is present, you may get behavior you don't desire. To work around this -- namely, to set an output field only for records which have all the inputs present -- you can use a pattern-action block with ``is_present``: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr cat data/het.dkvp @@ -492,7 +492,7 @@ Since absent plus absent is absent (and likewise for other operators), accumulat record_count=150,resource=/path/to/second/file resource=/some/other/path,loadsec=0.97,ok=false -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put 'is_present($loadsec) { $loadmillis = $loadsec * 1000 }' data/het.dkvp @@ -502,7 +502,7 @@ Since absent plus absent is absent (and likewise for other operators), accumulat record_count=150,resource=/path/to/second/file resource=/some/other/path,loadsec=0.97,ok=false,loadmillis=970.000000 -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr put '$loadmillis = (is_present($loadsec) ? $loadsec : 0.0) * 1000' data/het.dkvp @@ -514,7 +514,7 @@ Since absent plus absent is absent (and likewise for other operators), accumulat If you're interested in a formal description of how empty and absent fields participate in arithmetic, here's a table for plus (other arithmetic/boolean/bitwise operators are similar): -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --print-type-arithmetic-info @@ -582,7 +582,7 @@ For ``filter`` and ``put``, if the regular expression is a string literal (the n Example: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ cat data/regex-in-data.dat @@ -590,7 +590,7 @@ Example: name=bill,regex=^b[ou]ll$ name=bull,regex=^b[ou]ll$ -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr filter '$name =~ $regex' data/regex-in-data.dat @@ -604,28 +604,28 @@ Regex captures of the form ``\0`` through ``\9`` are supported as * Captures have in-function context for ``sub`` and ``gsub``. For example, the first ``\1,\2`` pair belong to the first ``sub`` and the second ``\1,\2`` pair belong to the second ``sub``: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 mlr put '$b = sub($a, "(..)_(...)", "\2-\1"); $c = sub($a, "(..)_(.)(..)", ":\1:\2:\3")' * Captures endure for the entirety of a ``put`` for the ``=~`` and ``!=~`` operators. For example, here the ``\1,\2`` are set by the ``=~`` operator and are used by both subsequent assignment statements: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 mlr put '$a =~ "(..)_(....); $b = "left_\1"; $c = "right_\2"' * The captures are not retained across multiple puts. For example, here the ``\1,\2`` won't be expanded from the regex capture: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 mlr put '$a =~ "(..)_(....)' then {... something else ...} then put '$b = "left_\1"; $c = "right_\2"' * Captures are ignored in ``filter`` for the ``=~`` and ``!=~`` operators. For example, there is no mechanism provided to refer to the first ``(..)`` as ``\1`` or to the second ``(....)`` as ``\2`` in the following filter statement: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 mlr filter '$a =~ "(..)_(....)' @@ -656,7 +656,7 @@ The short of it is that Miller does this transparently for you so you needn't th Implementation details of this, for the interested: integer adds and subtracts overflow by at most one bit so it suffices to check sign-changes. Thus, Miller allows you to add and subtract arbitrary 64-bit signed integers, converting only to float precisely when the result is less than -2\ :sup:`63` or greater than 2\ :sup:`63`\ -1. Multiplies, on the other hand, can overflow by a word size and a sign-change technique does not suffice to detect overflow. Instead Miller tests whether the floating-point product exceeds the representable integer range. Now, 64-bit integers have 64-bit precision while IEEE-doubles have only 52-bit mantissas -- so, there are 53 bits including implicit leading one. The following experiment explicitly demonstrates the resolution at this range: -.. code-block:: +.. code-block:: none 64-bit integer 64-bit integer Casted to double Back to 64-bit in hex in decimal integer @@ -685,7 +685,7 @@ On-line help Examples: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --help @@ -1111,7 +1111,7 @@ Examples: For more information please see http://johnkerl.org/miller/doc and/or http://github.com/johnkerl/miller. This is Miller version v5.10.2-dev. -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr sort --help diff --git a/docs6/reference.rst.in b/docs6/reference.rst.in index 68d71c4fa..3b7e7f42e 100644 --- a/docs6/reference.rst.in +++ b/docs6/reference.rst.in @@ -32,7 +32,7 @@ Formats Options: -.. code-block:: +.. code-block:: none --dkvp --idkvp --odkvp --nidx --inidx --onidx @@ -72,14 +72,14 @@ Compression Options: -.. code-block:: +.. code-block:: none --prepipe {command} The prepipe command is anything which reads from standard input and produces data acceptable to Miller. Nominally this allows you to use whichever decompression utilities you have installed on your system, on a per-file basis. If the command has flags, quote them: e.g. ``mlr --prepipe 'zcat -cf'``. Examples: -.. code-block:: +.. code-block:: none # These two produce the same output: $ gunzip < myfile1.csv.gz | mlr cut -f hostname,uptime @@ -88,14 +88,14 @@ The prepipe command is anything which reads from standard input and produces dat $ mlr --prepipe gunzip cut -f hostname,uptime myfile1.csv.gz myfile2.csv.gz $ mlr --prepipe gunzip --idkvp --oxtab cut -f hostname,uptime myfile1.dat.gz myfile2.dat.gz -.. code-block:: +.. code-block:: none # Similar to the above, but with compressed output as well as input: $ gunzip < myfile1.csv.gz | mlr cut -f hostname,uptime | gzip > outfile.csv.gz $ mlr --prepipe gunzip cut -f hostname,uptime myfile1.csv.gz | gzip > outfile.csv.gz $ mlr --prepipe gunzip cut -f hostname,uptime myfile1.csv.gz myfile2.csv.gz | gzip > outfile.csv.gz -.. code-block:: +.. code-block:: none # Similar to the above, but with different compression tools for input and output: $ gunzip < myfile1.csv.gz | mlr cut -f hostname,uptime | xz -z > outfile.csv.xz @@ -111,7 +111,7 @@ Miller has record separators ``IRS`` and ``ORS``, field separators ``IFS`` and ` Options: -.. code-block:: +.. code-block:: none --rs --irs --ors --fs --ifs --ofs --repifs @@ -132,7 +132,7 @@ Number formatting The command-line option ``--ofmt {format string}`` is the global number format for commands which generate numeric output, e.g. ``stats1``, ``stats2``, ``histogram``, and ``step``, as well as ``mlr put``. Examples: -.. code-block:: +.. code-block:: none --ofmt %.9le --ofmt %.6lf --ofmt %.0lf @@ -163,14 +163,14 @@ then-chaining In accord with the `Unix philosophy `_, you can pipe data into or out of Miller. For example: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 mlr cut --complement -f os_version *.dat | mlr sort -f hostname,uptime You can, if you like, instead simply chain commands together using the ``then`` keyword: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 mlr cut --complement -f os_version then sort -f hostname,uptime *.dat @@ -367,28 +367,28 @@ Regex captures of the form ``\0`` through ``\9`` are supported as * Captures have in-function context for ``sub`` and ``gsub``. For example, the first ``\1,\2`` pair belong to the first ``sub`` and the second ``\1,\2`` pair belong to the second ``sub``: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 mlr put '$b = sub($a, "(..)_(...)", "\2-\1"); $c = sub($a, "(..)_(.)(..)", ":\1:\2:\3")' * Captures endure for the entirety of a ``put`` for the ``=~`` and ``!=~`` operators. For example, here the ``\1,\2`` are set by the ``=~`` operator and are used by both subsequent assignment statements: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 mlr put '$a =~ "(..)_(....); $b = "left_\1"; $c = "right_\2"' * The captures are not retained across multiple puts. For example, here the ``\1,\2`` won't be expanded from the regex capture: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 mlr put '$a =~ "(..)_(....)' then {... something else ...} then put '$b = "left_\1"; $c = "right_\2"' * Captures are ignored in ``filter`` for the ``=~`` and ``!=~`` operators. For example, there is no mechanism provided to refer to the first ``(..)`` as ``\1`` or to the second ``(....)`` as ``\2`` in the following filter statement: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 mlr filter '$a =~ "(..)_(....)' @@ -419,7 +419,7 @@ The short of it is that Miller does this transparently for you so you needn't th Implementation details of this, for the interested: integer adds and subtracts overflow by at most one bit so it suffices to check sign-changes. Thus, Miller allows you to add and subtract arbitrary 64-bit signed integers, converting only to float precisely when the result is less than -2\ :sup:`63` or greater than 2\ :sup:`63`\ -1. Multiplies, on the other hand, can overflow by a word size and a sign-change technique does not suffice to detect overflow. Instead Miller tests whether the floating-point product exceeds the representable integer range. Now, 64-bit integers have 64-bit precision while IEEE-doubles have only 52-bit mantissas -- so, there are 53 bits including implicit leading one. The following experiment explicitly demonstrates the resolution at this range: -.. code-block:: +.. code-block:: none 64-bit integer 64-bit integer Casted to double Back to 64-bit in hex in decimal integer diff --git a/docs6/sql-examples.rst b/docs6/sql-examples.rst index 96866c222..a830b9837 100644 --- a/docs6/sql-examples.rst +++ b/docs6/sql-examples.rst @@ -13,7 +13,7 @@ I like to produce SQL-query output with header-column and tab delimiter: this is For example, using default output formatting in ``mysql`` we get formatting like Miller's ``--opprint --barred``: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mysql --database=mydb -e 'show columns in mytable' @@ -29,7 +29,7 @@ For example, using default output formatting in ``mysql`` we get formatting like Using ``mysql``'s ``-B`` we get TSV output: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mysql --database=mydb -B -e 'show columns in mytable' | mlr --itsvlite --opprint cat @@ -42,7 +42,7 @@ Using ``mysql``'s ``-B`` we get TSV output: Since Miller handles TSV output, we can do as much or as little processing as we want in the SQL query, then send the rest on to Miller. This includes outputting as JSON, doing further selects/joins in Miller, doing stats, etc. etc.: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mysql --database=mydb -B -e 'show columns in mytable' | mlr --itsvlite --ojson --jlistwrap --jvstack cat @@ -89,7 +89,7 @@ Since Miller handles TSV output, we can do as much or as little processing as we } ] -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mysql --database=mydb -B -e 'select * from mytable' > query.tsv @@ -115,7 +115,7 @@ One use of NIDX (value-only, no keys) format is for loading up SQL tables. Create and load SQL table: -.. code-block:: +.. code-block:: none mysql> CREATE TABLE abixy( a VARCHAR(32), @@ -158,7 +158,7 @@ Create and load SQL table: Aggregate counts within SQL: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 mysql> SELECT a, b, COUNT(*) AS count FROM abixy GROUP BY a, b ORDER BY COUNT DESC; @@ -195,7 +195,7 @@ Aggregate counts within SQL: Aggregate counts within Miller: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint uniq -c -g a,b then sort -nr count data/medium @@ -218,7 +218,7 @@ Aggregate counts within Miller: Pipe SQL output to aggregate counts within Miller: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mysql -D miller -B -e 'select * from abixy' | mlr --itsv --opprint uniq -c -g a,b then sort -nr count diff --git a/docs6/sql-examples.rst.in b/docs6/sql-examples.rst.in index 865909ab8..51ef2d66c 100644 --- a/docs6/sql-examples.rst.in +++ b/docs6/sql-examples.rst.in @@ -10,7 +10,7 @@ I like to produce SQL-query output with header-column and tab delimiter: this is For example, using default output formatting in ``mysql`` we get formatting like Miller's ``--opprint --barred``: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mysql --database=mydb -e 'show columns in mytable' @@ -26,7 +26,7 @@ For example, using default output formatting in ``mysql`` we get formatting like Using ``mysql``'s ``-B`` we get TSV output: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mysql --database=mydb -B -e 'show columns in mytable' | mlr --itsvlite --opprint cat @@ -39,7 +39,7 @@ Using ``mysql``'s ``-B`` we get TSV output: Since Miller handles TSV output, we can do as much or as little processing as we want in the SQL query, then send the rest on to Miller. This includes outputting as JSON, doing further selects/joins in Miller, doing stats, etc. etc.: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mysql --database=mydb -B -e 'show columns in mytable' | mlr --itsvlite --ojson --jlistwrap --jvstack cat @@ -86,7 +86,7 @@ Since Miller handles TSV output, we can do as much or as little processing as we } ] -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mysql --database=mydb -B -e 'select * from mytable' > query.tsv @@ -112,7 +112,7 @@ One use of NIDX (value-only, no keys) format is for loading up SQL tables. Create and load SQL table: -.. code-block:: +.. code-block:: none mysql> CREATE TABLE abixy( a VARCHAR(32), @@ -155,7 +155,7 @@ Create and load SQL table: Aggregate counts within SQL: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 mysql> SELECT a, b, COUNT(*) AS count FROM abixy GROUP BY a, b ORDER BY COUNT DESC; @@ -192,7 +192,7 @@ Aggregate counts within SQL: Aggregate counts within Miller: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mlr --opprint uniq -c -g a,b then sort -nr count data/medium @@ -215,7 +215,7 @@ Aggregate counts within Miller: Pipe SQL output to aggregate counts within Miller: -.. code-block:: +.. code-block:: none :emphasize-lines: 1,1 $ mysql -D miller -B -e 'select * from abixy' | mlr --itsv --opprint uniq -c -g a,b then sort -nr count diff --git a/go/todo.txt b/go/todo.txt index e63007609..9b431b963 100644 --- a/go/todo.txt +++ b/go/todo.txt @@ -6,6 +6,7 @@ TOP OF LIST: ! genrst fixup ! * rst bold https://stackoverflow.com/questions/26366552/bold-code-in-rst + syntax none ... * prepipex