diff --git a/docs6/10min.rst b/docs6/10min.rst index 312b169f8..887f06f1e 100644 --- a/docs6/10min.rst +++ b/docs6/10min.rst @@ -20,7 +20,7 @@ As a first check, you should be able to run ``mlr --version`` at your system's c .. code-block:: none :emphasize-lines: 1-1 - $ mlr --version + mlr --version Miller v6.0.0-dev As a second check, given (`example.csv <./example.csv>`_) you should be able to do @@ -28,7 +28,7 @@ As a second check, given (`example.csv <./example.csv>`_) you should be able to .. code-block:: none :emphasize-lines: 1-1 - $ mlr --csv cat example.csv + mlr --csv cat example.csv color,shape,flag,index,quantity,rate yellow,triangle,true,11,43.6498,9.8870 red,square,true,15,79.2778,0.0130 @@ -44,7 +44,7 @@ As a second check, given (`example.csv <./example.csv>`_) you should be able to .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsv --opprint cat example.csv + mlr --icsv --opprint cat example.csv color shape flag index quantity rate yellow triangle true 11 43.6498 9.8870 red square true 15 79.2778 0.0130 @@ -69,7 +69,7 @@ Let's take a quick look at some of the most useful Miller verbs -- file-format-a .. code-block:: none :emphasize-lines: 1-1 - $ mlr --csv cat example.csv + mlr --csv cat example.csv color,shape,flag,index,quantity,rate yellow,triangle,true,11,43.6498,9.8870 red,square,true,15,79.2778,0.0130 @@ -87,7 +87,7 @@ But ``mlr cat`` can also do format conversion -- for example, you can pretty-pri .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsv --opprint cat example.csv + mlr --icsv --opprint cat example.csv color shape flag index quantity rate yellow triangle true 11 43.6498 9.8870 red square true 15 79.2778 0.0130 @@ -105,7 +105,7 @@ But ``mlr cat`` can also do format conversion -- for example, you can pretty-pri .. code-block:: none :emphasize-lines: 1-1 - $ mlr --csv head -n 4 example.csv + mlr --csv head -n 4 example.csv color,shape,flag,index,quantity,rate yellow,triangle,true,11,43.6498,9.8870 red,square,true,15,79.2778,0.0130 @@ -115,7 +115,7 @@ But ``mlr cat`` can also do format conversion -- for example, you can pretty-pri .. code-block:: none :emphasize-lines: 1-1 - $ mlr --csv tail -n 4 example.csv + mlr --csv tail -n 4 example.csv color,shape,flag,index,quantity,rate purple,triangle,false,65,80.1405,5.8240 yellow,circle,true,73,63.9785,4.2370 @@ -125,7 +125,7 @@ But ``mlr cat`` can also do format conversion -- for example, you can pretty-pri .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsv --ojson tail -n 2 example.csv + mlr --icsv --ojson tail -n 2 example.csv { "color": "yellow", "shape": "circle", @@ -148,7 +148,7 @@ You can sort on a single field: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsv --opprint sort -f shape example.csv + mlr --icsv --opprint sort -f shape example.csv color shape flag index quantity rate red circle true 16 13.8103 2.9010 yellow circle true 73 63.9785 4.2370 @@ -166,7 +166,7 @@ Or, you can sort primarily alphabetically on one field, then secondarily numeric .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsv --opprint sort -f shape -nr index example.csv + mlr --icsv --opprint sort -f shape -nr index example.csv color shape flag index quantity rate yellow circle true 87 63.5058 8.3350 yellow circle true 73 63.9785 4.2370 @@ -184,7 +184,7 @@ If there are fields you don't want to see in your data, you can use ``cut`` to k .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsv --opprint cut -f flag,shape example.csv + mlr --icsv --opprint cut -f flag,shape example.csv shape flag triangle true square true @@ -202,7 +202,7 @@ You can also use ``cut -o`` to keep specified fields, but in your preferred orde .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsv --opprint cut -o -f flag,shape example.csv + mlr --icsv --opprint cut -o -f flag,shape example.csv flag shape true triangle true square @@ -220,7 +220,7 @@ You can use ``cut -x`` to omit fields you don't care about: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsv --opprint cut -x -f flag,shape example.csv + mlr --icsv --opprint cut -x -f flag,shape example.csv color index quantity rate yellow 11 43.6498 9.8870 red 15 79.2778 0.0130 @@ -238,7 +238,7 @@ You can use ``filter`` to keep only records you care about: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsv --opprint filter '$color == "red"' example.csv + mlr --icsv --opprint filter '$color == "red"' example.csv color shape flag index quantity rate red square true 15 79.2778 0.0130 red circle true 16 13.8103 2.9010 @@ -248,7 +248,7 @@ You can use ``filter`` to keep only records you care about: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsv --opprint filter '$color == "red" && $flag == true' example.csv + mlr --icsv --opprint filter '$color == "red" && $flag == true' example.csv color shape flag index quantity rate red square true 15 79.2778 0.0130 red circle true 16 13.8103 2.9010 @@ -258,7 +258,7 @@ You can use ``put`` to create new fields which are computed from other fields: .. code-block:: none :emphasize-lines: 1-4 - $ mlr --icsv --opprint put ' + mlr --icsv --opprint put ' $ratio = $quantity / $rate; $color_shape = $color . "_" . $shape ' example.csv @@ -279,7 +279,7 @@ Even though Miller's main selling point is name-indexing, sometimes you really w .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsv --opprint put '$[[3]] = "NEW"' example.csv + mlr --icsv --opprint put '$[[3]] = "NEW"' example.csv color shape NEW index quantity rate yellow triangle true 11 43.6498 9.8870 red square true 15 79.2778 0.0130 @@ -295,7 +295,7 @@ Even though Miller's main selling point is name-indexing, sometimes you really w .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsv --opprint put '$[[[3]]] = "NEW"' example.csv + mlr --icsv --opprint put '$[[[3]]] = "NEW"' example.csv color shape flag index quantity rate yellow triangle NEW 11 43.6498 9.8870 red square NEW 15 79.2778 0.0130 @@ -318,7 +318,7 @@ Miller takes all the files from the command line as an input stream. But it's fo .. code-block:: none :emphasize-lines: 1-1 - $ cat data/a.csv + cat data/a.csv a,b,c 1,2,3 4,5,6 @@ -326,14 +326,14 @@ Miller takes all the files from the command line as an input stream. But it's fo .. code-block:: none :emphasize-lines: 1-1 - $ cat data/b.csv + cat data/b.csv a,b,c 7,8,9 .. code-block:: none :emphasize-lines: 1-1 - $ cat data/a.csv data/b.csv + cat data/a.csv data/b.csv a,b,c 1,2,3 4,5,6 @@ -345,7 +345,7 @@ However, ``mlr cat`` will not: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --csv cat data/a.csv data/b.csv + mlr --csv cat data/a.csv data/b.csv a,b,c 1,2,3 4,5,6 @@ -359,7 +359,7 @@ Often we want to chain queries together -- for example, sorting by a field and t .. code-block:: none :emphasize-lines: 1-1 - $ mlr --csv sort -nr index example.csv | mlr --icsv --opprint head -n 3 + mlr --csv sort -nr index example.csv | mlr --icsv --opprint head -n 3 color shape flag index quantity rate purple square false 91 72.3735 8.2430 yellow circle true 87 63.5058 8.3350 @@ -370,7 +370,7 @@ This works fine -- but Miller also lets you chain verbs together using the word .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsv --opprint sort -nr index then head -n 3 example.csv + mlr --icsv --opprint sort -nr index then head -n 3 example.csv color shape flag index quantity rate purple square false 91 72.3735 8.2430 yellow circle true 87 63.5058 8.3350 @@ -381,7 +381,7 @@ As another convenience, you can put the filename first using ``--from``. When yo .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsv --opprint --from example.csv sort -nr index then head -n 3 + mlr --icsv --opprint --from example.csv sort -nr index then head -n 3 color shape flag index quantity rate purple square false 91 72.3735 8.2430 yellow circle true 87 63.5058 8.3350 @@ -390,7 +390,7 @@ As another convenience, you can put the filename first using ``--from``. When yo .. code-block:: none :emphasize-lines: 1-4 - $ mlr --icsv --opprint --from example.csv \ + mlr --icsv --opprint --from example.csv \ sort -nr index \ then head -n 3 \ then cut -f shape,quantity @@ -409,7 +409,7 @@ Here are the records with the top three ``index`` values: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsv --opprint sort -nr index then head -n 3 example.csv + mlr --icsv --opprint sort -nr index then head -n 3 example.csv color shape flag index quantity rate purple square false 91 72.3735 8.2430 yellow circle true 87 63.5058 8.3350 @@ -420,7 +420,7 @@ Lots of Miller commands take a ``-g`` option for group-by: here, ``head -n 1 -g .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsv --opprint sort -f shape -nr index then head -n 1 -g shape example.csv + mlr --icsv --opprint sort -f shape -nr index then head -n 1 -g shape example.csv color shape flag index quantity rate yellow circle true 87 63.5058 8.3350 purple square false 91 72.3735 8.2430 @@ -431,7 +431,7 @@ Statistics can be computed with or without group-by field(s): .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsv --opprint --from example.csv stats1 -a count,min,mean,max -f quantity -g shape + mlr --icsv --opprint --from example.csv stats1 -a count,min,mean,max -f quantity -g shape shape quantity_count quantity_min quantity_mean quantity_max triangle 3 43.6498 68.33976666666666 81.229 square 4 72.3735 76.60114999999999 79.2778 @@ -440,7 +440,7 @@ Statistics can be computed with or without group-by field(s): .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsv --opprint --from example.csv stats1 -a count,min,mean,max -f quantity -g shape,color + mlr --icsv --opprint --from example.csv stats1 -a count,min,mean,max -f quantity -g shape,color shape color quantity_count quantity_min quantity_mean quantity_max triangle yellow 1 43.6498 43.6498 43.6498 square red 3 77.1991 78.01036666666666 79.2778 @@ -454,7 +454,7 @@ If your output has a lot of columns, you can use XTAB format to line things up v .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsv --oxtab --from example.csv stats1 -a p0,p10,p25,p50,p75,p90,p99,p100 -f rate + mlr --icsv --oxtab --from example.csv stats1 -a p0,p10,p25,p50,p75,p90,p99,p100 -f rate rate_p0 0.0130 rate_p10 2.9010 rate_p25 4.2370 @@ -534,14 +534,14 @@ Sometimes, though, we want to print output to another file. Just use **> outputf .. code-block:: none :emphasize-lines: 1,1 - % mlr --icsv --opprint cat example.csv > newfile.csv + mlr --icsv --opprint cat example.csv > newfile.csv # Output goes to the new file; # nothing is printed to the screen. .. code-block:: none :emphasize-lines: 1,1 - % cat newfile.csv + cat newfile.csv color shape flag index quantity rate yellow triangle true 11 43.6498 9.8870 red square true 15 79.2778 0.0130 @@ -559,12 +559,12 @@ Other times we just want our files to be **changed in-place**: just use **mlr -I .. code-block:: none :emphasize-lines: 1,1 - % cp example.csv newfile.txt + cp example.csv newfile.txt .. code-block:: none :emphasize-lines: 1,1 - % cat newfile.txt + cat newfile.txt color,shape,flag,index,quantity,rate yellow,triangle,true,11,43.6498,9.8870 red,square,true,15,79.2778,0.0130 @@ -580,12 +580,12 @@ Other times we just want our files to be **changed in-place**: just use **mlr -I .. code-block:: none :emphasize-lines: 1,1 - % mlr -I --csv sort -f shape newfile.txt + mlr -I --csv sort -f shape newfile.txt .. code-block:: none :emphasize-lines: 1,1 - % cat newfile.txt + cat newfile.txt color,shape,flag,index,quantity,rate red,circle,true,16,13.8103,2.9010 yellow,circle,true,73,63.9785,4.2370 @@ -612,12 +612,12 @@ Lastly, using ``tee`` within ``put``, you can split your input data into separat .. code-block:: none :emphasize-lines: 1-1 - $ mlr --csv --from example.csv put -q 'tee > $shape.".csv", $*' + mlr --csv --from example.csv put -q 'tee > $shape.".csv", $*' .. code-block:: none :emphasize-lines: 1-1 - $ cat circle.csv + cat circle.csv color,shape,flag,index,quantity,rate red,circle,true,16,13.8103,2.9010 yellow,circle,true,73,63.9785,4.2370 @@ -626,7 +626,7 @@ Lastly, using ``tee`` within ``put``, you can split your input data into separat .. code-block:: none :emphasize-lines: 1-1 - $ cat square.csv + cat square.csv color,shape,flag,index,quantity,rate red,square,true,15,79.2778,0.0130 red,square,false,48,77.5542,7.4670 @@ -636,7 +636,7 @@ Lastly, using ``tee`` within ``put``, you can split your input data into separat .. code-block:: none :emphasize-lines: 1-1 - $ cat triangle.csv + cat triangle.csv color,shape,flag,index,quantity,rate yellow,triangle,true,11,43.6498,9.8870 purple,triangle,false,51,81.2290,8.5910 diff --git a/docs6/10min.rst.in b/docs6/10min.rst.in index d29f6b169..2131d466c 100644 --- a/docs6/10min.rst.in +++ b/docs6/10min.rst.in @@ -276,14 +276,14 @@ Sometimes, though, we want to print output to another file. Just use **> outputf .. code-block:: none :emphasize-lines: 1,1 - % mlr --icsv --opprint cat example.csv > newfile.csv + mlr --icsv --opprint cat example.csv > newfile.csv # Output goes to the new file; # nothing is printed to the screen. .. code-block:: none :emphasize-lines: 1,1 - % cat newfile.csv + cat newfile.csv color shape flag index quantity rate yellow triangle true 11 43.6498 9.8870 red square true 15 79.2778 0.0130 @@ -301,12 +301,12 @@ Other times we just want our files to be **changed in-place**: just use **mlr -I .. code-block:: none :emphasize-lines: 1,1 - % cp example.csv newfile.txt + cp example.csv newfile.txt .. code-block:: none :emphasize-lines: 1,1 - % cat newfile.txt + cat newfile.txt color,shape,flag,index,quantity,rate yellow,triangle,true,11,43.6498,9.8870 red,square,true,15,79.2778,0.0130 @@ -322,12 +322,12 @@ Other times we just want our files to be **changed in-place**: just use **mlr -I .. code-block:: none :emphasize-lines: 1,1 - % mlr -I --csv sort -f shape newfile.txt + mlr -I --csv sort -f shape newfile.txt .. code-block:: none :emphasize-lines: 1,1 - % cat newfile.txt + cat newfile.txt color,shape,flag,index,quantity,rate red,circle,true,16,13.8103,2.9010 yellow,circle,true,73,63.9785,4.2370 diff --git a/docs6/cookbook.rst b/docs6/cookbook.rst index bed1bbc7d..c81bb6ee6 100644 --- a/docs6/cookbook.rst +++ b/docs6/cookbook.rst @@ -12,7 +12,7 @@ Here are some ways to use the type-checking options as described in :ref:`refere .. code-block:: none :emphasize-lines: 1-1 - $ cat data/het-bool.csv + cat data/het-bool.csv name,reachable barney,false betty,true @@ -24,7 +24,7 @@ One option is to coerce everything to boolean, or integer: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsv --opprint put '$reachable = boolean($reachable)' data/het-bool.csv + mlr --icsv --opprint put '$reachable = boolean($reachable)' data/het-bool.csv name reachable barney false betty true @@ -34,7 +34,7 @@ One option is to coerce everything to boolean, or integer: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsv --opprint put '$reachable = int(boolean($reachable))' data/het-bool.csv + mlr --icsv --opprint put '$reachable = int(boolean($reachable))' data/het-bool.csv name reachable barney 0 betty 1 @@ -46,7 +46,7 @@ A second option is to flag badly formatted data within the output stream: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsv --opprint put '$format_ok = is_string($reachable)' data/het-bool.csv + mlr --icsv --opprint put '$format_ok = is_string($reachable)' data/het-bool.csv name reachable format_ok barney false false betty true false @@ -58,7 +58,7 @@ Or perhaps to flag badly formatted data outside the output stream: .. code-block:: none :emphasize-lines: 1-3 - $ mlr --icsv --opprint put ' + mlr --icsv --opprint put ' if (!is_string($reachable)) {eprint "Malformed at NR=".NR} ' data/het-bool.csv Malformed at NR=1 @@ -76,7 +76,7 @@ A third way is to abort the process on first instance of bad data: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --csv put '$reachable = asserting_string($reachable)' data/het-bool.csv + mlr --csv put '$reachable = asserting_string($reachable)' data/het-bool.csv Miller: is_string type-assertion failed at NR=1 FNR=1 FILENAME=data/het-bool.csv Showing differences between successive queries @@ -87,7 +87,7 @@ Suppose you have a database query which you run at one point in time, producing .. code-block:: none :emphasize-lines: 1-1 - $ cat data/previous_counters.csv + cat data/previous_counters.csv color,count red,3472 blue,6838 @@ -97,7 +97,7 @@ Suppose you have a database query which you run at one point in time, producing .. code-block:: none :emphasize-lines: 1-1 - $ cat data/current_counters.csv + cat data/current_counters.csv color,count red,3467 orange,670 @@ -111,12 +111,12 @@ First, rename counter columns to make them distinct: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --csv rename count,previous_count data/previous_counters.csv > data/prevtemp.csv + mlr --csv rename count,previous_count data/previous_counters.csv > data/prevtemp.csv .. code-block:: none :emphasize-lines: 1-1 - $ cat data/prevtemp.csv + cat data/prevtemp.csv color,previous_count red,3472 blue,6838 @@ -126,12 +126,12 @@ First, rename counter columns to make them distinct: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --csv rename count,current_count data/current_counters.csv > data/currtemp.csv + mlr --csv rename count,current_count data/current_counters.csv > data/currtemp.csv .. code-block:: none :emphasize-lines: 1-1 - $ cat data/currtemp.csv + cat data/currtemp.csv color,current_count red,3467 orange,670 @@ -143,7 +143,7 @@ Then, join on the key field(s), and use unsparsify to zero-fill counters absent .. code-block:: none :emphasize-lines: 1-5 - $ mlr --icsv --opprint \ + mlr --icsv --opprint \ join -j color --ul --ur -f data/prevtemp.csv \ then unsparsify --fill-with 0 \ then put '$count_delta = $current_count - $previous_count' \ @@ -168,7 +168,7 @@ For example, mapping numeric values down a column to the percentage between thei .. code-block:: none :emphasize-lines: 1-16 - $ mlr --from data/small --opprint put -q ' + mlr --from data/small --opprint put -q ' # These are executed once per record, which is the first pass. # The key is to use NR to index an out-of-stream variable to # retain all the x-field values. @@ -199,7 +199,7 @@ Similarly, finding the total record count requires first reading through all the .. code-block:: none :emphasize-lines: 1-11 - $ mlr --opprint --from data/small put -q ' + mlr --opprint --from data/small put -q ' @records[NR] = $*; end { for((I,k),v in @records) { @@ -225,7 +225,7 @@ The idea is to retain records having the largest value of ``n`` in the following .. code-block:: none :emphasize-lines: 1-1 - $ mlr --itsv --opprint cat data/maxrows.tsv + mlr --itsv --opprint cat data/maxrows.tsv a b n score purple red 5 0.743231 blue purple 2 0.093710 @@ -264,7 +264,7 @@ Of course, the largest value of ``n`` isn't known until after all data have been .. code-block:: none :emphasize-lines: 1-1 - $ cat data/maxrows.mlr + cat data/maxrows.mlr # Retain all records @records[NR] = $*; # Track max value of n @@ -284,7 +284,7 @@ Of course, the largest value of ``n`` isn't known until after all data have been .. code-block:: none :emphasize-lines: 1-1 - $ mlr --itsv --opprint put -q -f data/maxrows.mlr data/maxrows.tsv + mlr --itsv --opprint put -q -f data/maxrows.mlr data/maxrows.tsv a b n score purple red 5 0.743231 purple red 5 0.389055 @@ -339,7 +339,7 @@ Then .. code-block:: none :emphasize-lines: 1-1 - $ mlr --json put -q -f data/feature-count.mlr data/features.json + mlr --json put -q -f data/feature-count.mlr data/features.json { "record_count": 12 } @@ -395,7 +395,7 @@ Then .. code-block:: none :emphasize-lines: 1-1 - $ mlr --ijson --opprint put -q -f data/feature-count.mlr data/features.json + mlr --ijson --opprint put -q -f data/feature-count.mlr data/features.json record_count 12 @@ -425,7 +425,7 @@ For example, suppose you have JSON input like this: .. code-block:: none :emphasize-lines: 1-1 - $ cat data/sparse.json + cat data/sparse.json {"a":1,"b":2,"v":3} {"u":1,"b":2} {"a":1,"v":2,"x":3} @@ -436,7 +436,7 @@ There are field names ``a``, ``b``, ``v``, ``u``, ``x``, ``w`` in the data -- bu .. code-block:: none :emphasize-lines: 1-1 - $ cat data/unsparsify.mlr + cat data/unsparsify.mlr # First pass: # Remember all unique key names: for (k in $*) { @@ -467,7 +467,7 @@ There are field names ``a``, ``b``, ``v``, ``u``, ``x``, ``w`` in the data -- bu .. code-block:: none :emphasize-lines: 1-1 - $ mlr --json put -q -f data/unsparsify.mlr data/sparse.json + mlr --json put -q -f data/unsparsify.mlr data/sparse.json { "a": 1, "b": 2, @@ -504,7 +504,7 @@ There are field names ``a``, ``b``, ``v``, ``u``, ``x``, ``w`` in the data -- bu .. code-block:: none :emphasize-lines: 1-1 - $ mlr --ijson --ocsv put -q -f data/unsparsify.mlr data/sparse.json + mlr --ijson --ocsv put -q -f data/unsparsify.mlr data/sparse.json a,b,v,u,x,w 1,2,3,,, ,2,,1,, @@ -514,7 +514,7 @@ There are field names ``a``, ``b``, ``v``, ``u``, ``x``, ``w`` in the data -- bu .. code-block:: none :emphasize-lines: 1-1 - $ mlr --ijson --opprint put -q -f data/unsparsify.mlr data/sparse.json + mlr --ijson --opprint put -q -f data/unsparsify.mlr data/sparse.json a b v u x w 1 2 3 - - - - 2 - 1 - - diff --git a/docs6/cookbook2.rst b/docs6/cookbook2.rst index 7b9fccf60..91510bcd7 100644 --- a/docs6/cookbook2.rst +++ b/docs6/cookbook2.rst @@ -112,7 +112,7 @@ For one or more specified field names, simply compute p25 and p75, then write th .. code-block:: none :emphasize-lines: 1-3 - $ mlr --oxtab stats1 -f x -a p25,p75 \ + mlr --oxtab stats1 -f x -a p25,p75 \ then put '$x_iqr = $x_p75 - $x_p25' \ data/medium x_p25 0.24667037823231752 @@ -124,7 +124,7 @@ For wildcarded field names, first compute p25 and p75, then loop over field name .. code-block:: none :emphasize-lines: 1-7 - $ mlr --oxtab stats1 --fr '[i-z]' -a p25,p75 \ + mlr --oxtab stats1 --fr '[i-z]' -a p25,p75 \ then put 'for (k,v in $*) { if (k =~ "(.*)_p25") { $["\1_iqr"] = $["\1_p75"] - $["\1_p25"] @@ -140,7 +140,7 @@ This might be more elegantly implemented as an option within the ``stats1`` verb .. code-block:: none :emphasize-lines: 1-24 - $ mlr --from data/medium put -q ' + mlr --from data/medium put -q ' # Using the y field for weighting in this example weight = $y; @@ -178,7 +178,7 @@ Here we can chain together a few simple building blocks: .. code-block:: none :emphasize-lines: 1-1 - $ cat expo-sample.sh + cat expo-sample.sh # Generate 100,000 pairs of independent and identically distributed # exponentially distributed random variables with the same rate parameter # (namely, 2.5). Then compute histograms of one of them, along with @@ -221,7 +221,7 @@ The output is as follows: .. code-block:: none :emphasize-lines: 1-1 - $ sh expo-sample.sh + sh expo-sample.sh bin_lo bin_hi u_count s_count p_count 0 0.04 [64]*******************#[9554] [326]#...................[3703] [19]*******************#[39809] 0.04 0.08 [64]*****************...[9554] [326]*****...............[3703] [19]*******.............[39809] @@ -282,7 +282,7 @@ The `Sieve of Eratosthenes ` .. code-block:: none :emphasize-lines: 1-1 - $ cat programs/sieve.mlr + cat programs/sieve.mlr # ================================================================ # Sieve of Eratosthenes: simple example of Miller DSL as programming language. # ================================================================ @@ -319,7 +319,7 @@ The `Sieve of Eratosthenes ` .. code-block:: none :emphasize-lines: 1-1 - $ mlr -n put -f programs/sieve.mlr + mlr -n put -f programs/sieve.mlr 2 3 5 @@ -356,7 +356,7 @@ The (approximate) computation of points in the complex plane which are and aren' .. code-block:: none :emphasize-lines: 1-1 - $ cat programs/mand.mlr + cat programs/mand.mlr # Mandelbrot set generator: simple example of Miller DSL as programming language. begin { # Set defaults @@ -463,7 +463,7 @@ At standard resolution this makes a nice little ASCII plot: .. code-block:: none :emphasize-lines: 1-1 - $ mlr -n put -f ./programs/mand.mlr + mlr -n put -f ./programs/mand.mlr @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ diff --git a/docs6/cookbook3.rst b/docs6/cookbook3.rst index c71ec98f2..f3fd137ef 100644 --- a/docs6/cookbook3.rst +++ b/docs6/cookbook3.rst @@ -12,7 +12,7 @@ One of Miller's strengths is its compact notation: for example, given input of t .. code-block:: none :emphasize-lines: 1-1 - $ head -n 5 ../data/medium + head -n 5 ../data/medium a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533 a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797 a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776 @@ -24,7 +24,7 @@ you can simply do .. code-block:: none :emphasize-lines: 1-1 - $ mlr --oxtab stats1 -a sum -f x ../data/medium + mlr --oxtab stats1 -a sum -f x ../data/medium x_sum 4986.019681679581 or @@ -32,7 +32,7 @@ or .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint stats1 -a sum -f x -g b ../data/medium + mlr --opprint stats1 -a sum -f x -g b ../data/medium b x_sum pan 965.7636699425815 wye 1023.5484702619565 @@ -45,7 +45,7 @@ rather than the more tedious .. code-block:: none :emphasize-lines: 1-6 - $ mlr --oxtab put -q ' + mlr --oxtab put -q ' @x_sum += $x; end { emit @x_sum @@ -58,7 +58,7 @@ or .. code-block:: none :emphasize-lines: 1-6 - $ mlr --opprint put -q ' + mlr --opprint put -q ' @x_sum[$b] += $x; end { emit @x_sum, "b" @@ -83,14 +83,14 @@ Mean without/with oosvars .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint stats1 -a mean -f x data/medium + mlr --opprint stats1 -a mean -f x data/medium x_mean 0.49860196816795804 .. code-block:: none :emphasize-lines: 1-8 - $ mlr --opprint put -q ' + mlr --opprint put -q ' @x_sum += $x; @x_count += 1; end { @@ -107,7 +107,7 @@ Keyed mean without/with oosvars .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint stats1 -a mean -f x -g a,b data/medium + mlr --opprint stats1 -a mean -f x -g a,b data/medium a b x_mean pan pan 0.5133141190437597 eks pan 0.48507555383425127 @@ -138,7 +138,7 @@ Keyed mean without/with oosvars .. code-block:: none :emphasize-lines: 1-10 - $ mlr --opprint put -q ' + mlr --opprint put -q ' @x_sum[$a][$b] += $x; @x_count[$a][$b] += 1; end{ @@ -181,7 +181,7 @@ Variance and standard deviation without/with oosvars .. code-block:: none :emphasize-lines: 1-1 - $ mlr --oxtab stats1 -a count,sum,mean,var,stddev -f x data/medium + mlr --oxtab stats1 -a count,sum,mean,var,stddev -f x data/medium x_count 10000 x_sum 4986.019681679581 x_mean 0.49860196816795804 @@ -191,7 +191,7 @@ Variance and standard deviation without/with oosvars .. code-block:: none :emphasize-lines: 1-1 - $ cat variance.mlr + cat variance.mlr @n += 1; @sumx += $x; @sumx2 += $x**2; @@ -205,7 +205,7 @@ Variance and standard deviation without/with oosvars .. code-block:: none :emphasize-lines: 1-1 - $ mlr --oxtab put -q -f variance.mlr data/medium + mlr --oxtab put -q -f variance.mlr data/medium n 10000 sumx 4986.019681679581 sumx2 3328.652400179729 @@ -221,14 +221,14 @@ Min/max without/with oosvars .. code-block:: none :emphasize-lines: 1-1 - $ mlr --oxtab stats1 -a min,max -f x data/medium + mlr --oxtab stats1 -a min,max -f x data/medium x_min 4.509679127584487e-05 x_max 0.999952670371898 .. code-block:: none :emphasize-lines: 1-5 - $ mlr --oxtab put -q ' + mlr --oxtab put -q ' @x_min = min(@x_min, $x); @x_max = max(@x_max, $x); end{emitf @x_min, @x_max} @@ -242,7 +242,7 @@ Keyed min/max without/with oosvars .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint stats1 -a min,max -f x -g a data/medium + mlr --opprint stats1 -a min,max -f x -g a data/medium a x_min x_max pan 0.00020390740306253097 0.9994029107062516 eks 0.0006917972627396018 0.9988110946859143 @@ -253,7 +253,7 @@ Keyed min/max without/with oosvars .. code-block:: none :emphasize-lines: 1-7 - $ mlr --opprint --from data/medium put -q ' + mlr --opprint --from data/medium put -q ' @min[$a] = min(@min[$a], $x); @max[$a] = max(@max[$a], $x); end{ @@ -273,7 +273,7 @@ Delta without/with oosvars .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint step -a delta -f x data/small + mlr --opprint step -a delta -f x data/small a b i x y x_delta pan pan 1 0.3467901443380824 0.7268028627434533 0 eks pan 2 0.7586799647899636 0.5221511083334797 0.41188982045188116 @@ -284,7 +284,7 @@ Delta without/with oosvars .. code-block:: none :emphasize-lines: 1-4 - $ mlr --opprint put ' + mlr --opprint put ' $x_delta = is_present(@last) ? $x - @last : 0; @last = $x ' data/small @@ -301,7 +301,7 @@ Keyed delta without/with oosvars .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint step -a delta -f x -g a data/small + mlr --opprint step -a delta -f x -g a data/small a b i x y x_delta pan pan 1 0.3467901443380824 0.7268028627434533 0 eks pan 2 0.7586799647899636 0.5221511083334797 0 @@ -312,7 +312,7 @@ Keyed delta without/with oosvars .. code-block:: none :emphasize-lines: 1-4 - $ mlr --opprint put ' + mlr --opprint put ' $x_delta = is_present(@last[$a]) ? $x - @last[$a] : 0; @last[$a]=$x ' data/small @@ -329,7 +329,7 @@ Exponentially weighted moving averages without/with oosvars .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint step -a ewma -d 0.1 -f x data/small + mlr --opprint step -a ewma -d 0.1 -f x data/small a b i x y x_ewma_0.1 pan pan 1 0.3467901443380824 0.7268028627434533 0.3467901443380824 eks pan 2 0.7586799647899636 0.5221511083334797 0.3879791263832706 @@ -340,7 +340,7 @@ Exponentially weighted moving averages without/with oosvars .. code-block:: none :emphasize-lines: 1-5 - $ mlr --opprint put ' + mlr --opprint put ' begin{ @a=0.1 }; $e = NR==1 ? $x : @a * $x + (1 - @a) * @e; @e=$e diff --git a/docs6/cookbook4.rst b/docs6/cookbook4.rst index e7d70d40a..40e0ac446 100644 --- a/docs6/cookbook4.rst +++ b/docs6/cookbook4.rst @@ -75,7 +75,7 @@ And here is an example using them: .. code-block:: none :emphasize-lines: 1-1 - $ cat polyglot-dkvp-io/example.py + cat polyglot-dkvp-io/example.py #!/usr/bin/env python import sys @@ -116,7 +116,7 @@ Run as-is: .. code-block:: none :emphasize-lines: 1-1 - $ python polyglot-dkvp-io/example.py < data/small + python polyglot-dkvp-io/example.py < data/small a=pan,b=pan,i=1,y=0.7268028627434533,ab=panpan,iy=1.7268028627434533,ta=str,tb=str,ti=int,ty=float,tab=str,tiy=float a=eks,b=pan,i=2,y=0.5221511083334797,ab=ekspan,iy=2.5221511083334796,ta=str,tb=str,ti=int,ty=float,tab=str,tiy=float a=wye,b=wye,i=3,y=0.33831852551664776,ab=wyewye,iy=3.3383185255166477,ta=str,tb=str,ti=int,ty=float,tab=str,tiy=float @@ -128,7 +128,7 @@ Run as-is, then pipe to Miller for pretty-printing: .. code-block:: none :emphasize-lines: 1-1 - $ python polyglot-dkvp-io/example.py < data/small | mlr --opprint cat + python polyglot-dkvp-io/example.py < data/small | mlr --opprint cat a b i y ab iy ta tb ti ty tab tiy pan pan 1 0.7268028627434533 panpan 1.7268028627434533 str str int float str float eks pan 2 0.5221511083334797 ekspan 2.5221511083334796 str str int float str float @@ -201,7 +201,7 @@ And here is an example using them: .. code-block:: none :emphasize-lines: 1-1 - $ cat polyglot-dkvp-io/example.rb + cat polyglot-dkvp-io/example.rb #!/usr/bin/env ruby require 'dkvp_io' @@ -232,7 +232,7 @@ Run as-is: .. code-block:: none :emphasize-lines: 1-1 - $ ruby -I./polyglot-dkvp-io polyglot-dkvp-io/example.rb data/small + ruby -I./polyglot-dkvp-io polyglot-dkvp-io/example.rb data/small a=pan,b=pan,i=1,y=0.7268028627434533,ab=panpan,iy=1.7268028627434533,ta=String,tb=String,ti=Integer,ty=Float,tab=String,tiy=Float a=eks,b=pan,i=2,y=0.5221511083334797,ab=ekspan,iy=2.5221511083334796,ta=String,tb=String,ti=Integer,ty=Float,tab=String,tiy=Float a=wye,b=wye,i=3,y=0.33831852551664776,ab=wyewye,iy=3.3383185255166477,ta=String,tb=String,ti=Integer,ty=Float,tab=String,tiy=Float @@ -244,7 +244,7 @@ Run as-is, then pipe to Miller for pretty-printing: .. code-block:: none :emphasize-lines: 1-1 - $ ruby -I./polyglot-dkvp-io polyglot-dkvp-io/example.rb data/small | mlr --opprint cat + ruby -I./polyglot-dkvp-io polyglot-dkvp-io/example.rb data/small | mlr --opprint cat a b i y ab iy ta tb ti ty tab tiy pan pan 1 0.7268028627434533 panpan 1.7268028627434533 String String Integer Float String Float eks pan 2 0.5221511083334797 ekspan 2.5221511083334796 String String Integer Float String Float diff --git a/docs6/csv-with-and-without-headers.rst b/docs6/csv-with-and-without-headers.rst index b88a1604b..45d24b36a 100644 --- a/docs6/csv-with-and-without-headers.rst +++ b/docs6/csv-with-and-without-headers.rst @@ -12,7 +12,7 @@ Sometimes we get CSV files which lack a header. For example (`data/headerless.cs .. code-block:: none :emphasize-lines: 1-1 - $ cat data/headerless.csv + cat data/headerless.csv John,23,present Fred,34,present Alice,56,missing @@ -23,7 +23,7 @@ You can use Miller to add a header. The ``--implicit-csv-header`` applies positi .. code-block:: none :emphasize-lines: 1-1 - $ mlr --csv --implicit-csv-header cat data/headerless.csv + mlr --csv --implicit-csv-header cat data/headerless.csv 1,2,3 John,23,present Fred,34,present @@ -35,7 +35,7 @@ Following that, you can rename the positionally indexed labels to names with mea .. code-block:: none :emphasize-lines: 1-1 - $ mlr --csv --implicit-csv-header label name,age,status data/headerless.csv + mlr --csv --implicit-csv-header label name,age,status data/headerless.csv name,age,status John,23,present Fred,34,present @@ -47,7 +47,7 @@ Likewise, if you need to produce CSV which is lacking its header, you can pipe M .. code-block:: none :emphasize-lines: 1-1 - $ head -5 data/colored-shapes.dkvp | mlr --ocsv cat + head -5 data/colored-shapes.dkvp | mlr --ocsv cat color,shape,flag,i,u,v,w,x yellow,triangle,1,11,0.6321695890307647,0.9887207810889004,0.4364983936735774,5.7981881667050565 red,square,1,15,0.21966833570651523,0.001257332190235938,0.7927778364718627,2.944117399716207 @@ -58,7 +58,7 @@ Likewise, if you need to produce CSV which is lacking its header, you can pipe M .. code-block:: none :emphasize-lines: 1-1 - $ head -5 data/colored-shapes.dkvp | mlr --ocsv --headerless-csv-output cat + head -5 data/colored-shapes.dkvp | mlr --ocsv --headerless-csv-output cat yellow,triangle,1,11,0.6321695890307647,0.9887207810889004,0.4364983936735774,5.7981881667050565 red,square,1,15,0.21966833570651523,0.001257332190235938,0.7927778364718627,2.944117399716207 red,circle,1,16,0.20901671281497636,0.29005231936593445,0.13810280912907674,5.065034003400998 @@ -70,7 +70,7 @@ Lastly, often we say "CSV" or "TSV" when we have positionally indexed data in co .. code-block:: none :emphasize-lines: 1-1 - $ mlr --inidx --ifs comma --oxtab cut -f 1,3 data/headerless.csv + mlr --inidx --ifs comma --oxtab cut -f 1,3 data/headerless.csv 1 John 3 present @@ -91,7 +91,7 @@ Miller handles compliant CSV: in particular, it's an error if the number of data .. code-block:: none :emphasize-lines: 1-1 - $ cat data/ragged.csv + cat data/ragged.csv a,b,c 1,2,3 4,5 @@ -100,7 +100,7 @@ Miller handles compliant CSV: in particular, it's an error if the number of data .. code-block:: none :emphasize-lines: 1-8 - $ mlr --from data/ragged.csv --fs comma --nidx put ' + mlr --from data/ragged.csv --fs comma --nidx put ' @maxnf = max(@maxnf, NF); @nf = NF; while(@nf < @maxnf) { @@ -118,7 +118,7 @@ or, more simply, .. code-block:: none :emphasize-lines: 1-6 - $ mlr --from data/ragged.csv --fs comma --nidx put ' + mlr --from data/ragged.csv --fs comma --nidx put ' @maxnf = max(@maxnf, NF); while(NF < @maxnf) { $[NF+1] = ""; diff --git a/docs6/data-diving-examples.rst b/docs6/data-diving-examples.rst index f52af3c91..e26805fcd 100644 --- a/docs6/data-diving-examples.rst +++ b/docs6/data-diving-examples.rst @@ -14,7 +14,7 @@ Vertical-tabular format is good for a quick look at CSV data layout -- seeing wh .. code-block:: none :emphasize-lines: 1-1 - $ head -n 2 data/flins.csv | mlr --icsv --oxtab cat + head -n 2 data/flins.csv | mlr --icsv --oxtab cat county Seminole tiv_2011 22890.55 tiv_2012 20848.71 @@ -25,7 +25,7 @@ A few simple queries: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --from data/flins.csv --icsv --opprint count-distinct -f county | head + mlr --from data/flins.csv --icsv --opprint count-distinct -f county | head county count Seminole 1 Miami Dade 2 @@ -37,33 +37,33 @@ A few simple queries: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --from data/flins.csv --icsv --opprint count-distinct -f construction,line + mlr --from data/flins.csv --icsv --opprint count-distinct -f construction,line Categorization of total insured value: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --from data/flins.csv --icsv --opprint stats1 -a min,mean,max -f tiv_2012 + mlr --from data/flins.csv --icsv --opprint stats1 -a min,mean,max -f tiv_2012 tiv_2012_min tiv_2012_mean tiv_2012_max 19757.91 1.0615314637499999e+06 2.78555163e+06 .. code-block:: none :emphasize-lines: 1-2 - $ mlr --from data/flins.csv --icsv --opprint \ + mlr --from data/flins.csv --icsv --opprint \ stats1 -a min,mean,max -f tiv_2012 -g construction,line .. code-block:: none :emphasize-lines: 1-2 - $ mlr --from data/flins.csv --icsv --oxtab \ + mlr --from data/flins.csv --icsv --oxtab \ stats1 -a p0,p10,p50,p90,p95,p99,p100 -f hu_site_deductible .. code-block:: none :emphasize-lines: 1-3 - $ mlr --from data/flins.csv --icsv --opprint \ + mlr --from data/flins.csv --icsv --opprint \ stats1 -a p95,p99,p100 -f hu_site_deductible -g county \ then sort -f county | head county @@ -77,7 +77,7 @@ Categorization of total insured value: .. code-block:: none :emphasize-lines: 1-2 - $ mlr --from data/flins.csv --icsv --oxtab \ + mlr --from data/flins.csv --icsv --oxtab \ stats2 -a corr,linreg-ols,r2 -f tiv_2011,tiv_2012 tiv_2011_tiv_2012_corr 0.9353629581411828 tiv_2011_tiv_2012_ols_m 1.0890905877734807 @@ -88,7 +88,7 @@ Categorization of total insured value: .. code-block:: none :emphasize-lines: 1-2 - $ mlr --from data/flins.csv --icsv --opprint \ + mlr --from data/flins.csv --icsv --opprint \ stats2 -a corr,linreg-ols,r2 -f tiv_2011,tiv_2012 -g county county tiv_2011_tiv_2012_corr tiv_2011_tiv_2012_ols_m tiv_2011_tiv_2012_ols_b tiv_2011_tiv_2012_ols_n tiv_2011_tiv_2012_r2 Seminole - - - 1 - @@ -116,13 +116,13 @@ Peek at the data: .. code-block:: none :emphasize-lines: 1-1 - $ wc -l data/colored-shapes.dkvp + wc -l data/colored-shapes.dkvp 10078 data/colored-shapes.dkvp .. code-block:: none :emphasize-lines: 1-1 - $ head -n 6 data/colored-shapes.dkvp | mlr --opprint cat + head -n 6 data/colored-shapes.dkvp | mlr --opprint cat color shape flag i u v w x yellow triangle 1 11 0.6321695890307647 0.9887207810889004 0.4364983936735774 5.7981881667050565 red square 1 15 0.21966833570651523 0.001257332190235938 0.7927778364718627 2.944117399716207 @@ -138,7 +138,7 @@ Here it looks reasonable that ``u`` is unit-uniform; something's up with ``v`` b .. code-block:: none :emphasize-lines: 1-1 - $ mlr --oxtab stats1 -a min,mean,max -f flag,u,v data/colored-shapes.dkvp | creach 3 + mlr --oxtab stats1 -a min,mean,max -f flag,u,v data/colored-shapes.dkvp | creach 3 flag_min 0 flag_mean 0.39888866838658465 flag_max 1 @@ -156,7 +156,7 @@ The histogram shows the different distribution of 0/1 flags: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint histogram -f flag,u,v --lo -0.1 --hi 1.1 --nbins 12 data/colored-shapes.dkvp + mlr --opprint histogram -f flag,u,v --lo -0.1 --hi 1.1 --nbins 12 data/colored-shapes.dkvp bin_lo bin_hi flag_count u_count v_count -0.010000000000000002 0.09000000000000002 6058 0 36 0.09000000000000002 0.19000000000000003 0 1062 988 @@ -176,7 +176,7 @@ Look at univariate stats by color and shape. In particular, color-dependent flag .. code-block:: none :emphasize-lines: 1-3 - $ mlr --opprint stats1 -a min,mean,max -f flag,u,v -g color \ + mlr --opprint stats1 -a min,mean,max -f flag,u,v -g color \ then sort -f color \ data/colored-shapes.dkvp color flag_min flag_mean flag_max u_min u_mean u_max v_min v_mean v_max @@ -190,7 +190,7 @@ Look at univariate stats by color and shape. In particular, color-dependent flag .. code-block:: none :emphasize-lines: 1-3 - $ mlr --opprint stats1 -a min,mean,max -f flag,u,v -g shape \ + mlr --opprint stats1 -a min,mean,max -f flag,u,v -g shape \ then sort -f shape \ data/colored-shapes.dkvp shape flag_min flag_mean flag_max u_min u_mean u_max v_min v_mean v_max @@ -203,14 +203,14 @@ Look at bivariate stats by color and shape. In particular, ``u,v`` pairwise corr .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint --right stats2 -a corr -f u,v,w,x data/colored-shapes.dkvp + mlr --opprint --right stats2 -a corr -f u,v,w,x data/colored-shapes.dkvp u_v_corr w_x_corr 0.13341803768384553 -0.011319938208638764 .. code-block:: none :emphasize-lines: 1-3 - $ mlr --opprint --right \ + mlr --opprint --right \ stats2 -a corr -f u,v,w,x -g color,shape then sort -nr u_v_corr \ data/colored-shapes.dkvp color shape u_v_corr w_x_corr diff --git a/docs6/dates-and-times.rst b/docs6/dates-and-times.rst index 7ef1942ef..94e877eb4 100644 --- a/docs6/dates-and-times.rst +++ b/docs6/dates-and-times.rst @@ -12,7 +12,7 @@ Given input like .. code-block:: none :emphasize-lines: 1-1 - $ cat dates.csv + cat dates.csv date,event 2018-02-03,initialization 2018-03-07,discovery @@ -23,7 +23,7 @@ we can use ``strptime`` to parse the date field into seconds-since-epoch and the .. code-block:: none :emphasize-lines: 1-3 - $ mlr --csv filter ' + mlr --csv filter ' strptime($date, "%Y-%m-%d") > strptime("2018-03-03", "%Y-%m-%d") ' dates.csv date,event @@ -39,7 +39,7 @@ Suppose you have some date-stamped data which may (or may not) be missing entrie .. code-block:: none :emphasize-lines: 1-1 - $ head -n 10 data/miss-date.csv + head -n 10 data/miss-date.csv date,qoh 2012-03-05,10055 2012-03-06,10486 @@ -54,7 +54,7 @@ Suppose you have some date-stamped data which may (or may not) be missing entrie .. code-block:: none :emphasize-lines: 1-1 - $ wc -l data/miss-date.csv + wc -l data/miss-date.csv 1372 data/miss-date.csv Since there are 1372 lines in the data file, some automation is called for. To find the missing dates, you can convert the dates to seconds since the epoch using ``strptime``, then compute adjacent differences (the ``cat -n`` simply inserts record-counters): @@ -62,7 +62,7 @@ Since there are 1372 lines in the data file, some automation is called for. To f .. code-block:: none :emphasize-lines: 1-5 - $ mlr --from data/miss-date.csv --icsv \ + mlr --from data/miss-date.csv --icsv \ cat -n \ then put '$datestamp = strptime($date, "%Y-%m-%d")' \ then step -a delta -f datestamp \ @@ -83,7 +83,7 @@ Then, filter for adjacent difference not being 86400 (the number of seconds in a .. code-block:: none :emphasize-lines: 1-5 - $ mlr --from data/miss-date.csv --icsv \ + mlr --from data/miss-date.csv --icsv \ cat -n \ then put '$datestamp = strptime($date, "%Y-%m-%d")' \ then step -a delta -f datestamp \ @@ -96,7 +96,7 @@ Given this, it's now easy to see where the gaps are: .. code-block:: none :emphasize-lines: 1-1 - $ mlr cat -n then filter '$n >= 770 && $n <= 780' data/miss-date.csv + mlr cat -n then filter '$n >= 770 && $n <= 780' data/miss-date.csv n=770,1=2014-04-12,2=129435 n=771,1=2014-04-13,2=129868 n=772,1=2014-04-14,2=129797 @@ -112,7 +112,7 @@ Given this, it's now easy to see where the gaps are: .. code-block:: none :emphasize-lines: 1-1 - $ mlr cat -n then filter '$n >= 1115 && $n <= 1125' data/miss-date.csv + mlr cat -n then filter '$n >= 1115 && $n <= 1125' data/miss-date.csv n=1115,1=2015-03-25,2=181006 n=1116,1=2015-03-26,2=180995 n=1117,1=2015-03-27,2=181043 diff --git a/docs6/feature-comparison.rst b/docs6/feature-comparison.rst index cef31cc50..2d7ed3d49 100644 --- a/docs6/feature-comparison.rst +++ b/docs6/feature-comparison.rst @@ -14,7 +14,7 @@ Miller respects CSV headers. If you do ``mlr --csv cat *.csv`` then the header l .. code-block:: none :emphasize-lines: 1-1 - $ cat data/a.csv + cat data/a.csv a,b,c 1,2,3 4,5,6 @@ -22,14 +22,14 @@ Miller respects CSV headers. If you do ``mlr --csv cat *.csv`` then the header l .. code-block:: none :emphasize-lines: 1-1 - $ cat data/b.csv + cat data/b.csv a,b,c 7,8,9 .. code-block:: none :emphasize-lines: 1-1 - $ mlr --csv cat data/a.csv data/b.csv + mlr --csv cat data/a.csv data/b.csv a,b,c 1,2,3 4,5,6 @@ -38,7 +38,7 @@ Miller respects CSV headers. If you do ``mlr --csv cat *.csv`` then the header l .. code-block:: none :emphasize-lines: 1-1 - $ mlr --csv sort -nr b data/a.csv data/b.csv + mlr --csv sort -nr b data/a.csv data/b.csv a,b,c 7,8,9 4,5,6 diff --git a/docs6/file-formats.rst b/docs6/file-formats.rst index be1165787..ff74aae03 100644 --- a/docs6/file-formats.rst +++ b/docs6/file-formats.rst @@ -14,7 +14,7 @@ Examples .. code-block:: none :emphasize-lines: 1-1 - $ mlr --usage-data-format-examples + mlr --usage-data-format-examples CSV/CSV-lite: comma-separated values with separate header line TSV: same but with tabs in places of commas +---------------------+ @@ -124,7 +124,7 @@ Miller's default file format is DKVP, for **delimited key-value pairs**. Example .. code-block:: none :emphasize-lines: 1-1 - $ mlr cat data/small + mlr cat data/small a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533 a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797 a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776 @@ -180,7 +180,7 @@ Example with index-numbered output: .. code-block:: none :emphasize-lines: 1-1 - $ cat data/small + cat data/small a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533 a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797 a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776 @@ -190,7 +190,7 @@ Example with index-numbered output: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --onidx --ofs ' ' cat data/small + mlr --onidx --ofs ' ' cat data/small pan pan 1 0.3467901443380824 0.7268028627434533 eks pan 2 0.7586799647899636 0.5221511083334797 wye wye 3 0.20460330576630303 0.33831852551664776 @@ -202,7 +202,7 @@ Example with index-numbered input: .. code-block:: none :emphasize-lines: 1-1 - $ cat data/mydata.txt + cat data/mydata.txt oh say can you see by the dawn's early light @@ -210,7 +210,7 @@ Example with index-numbered input: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --inidx --ifs ' ' --odkvp cat data/mydata.txt + mlr --inidx --ifs ' ' --odkvp cat data/mydata.txt 1=oh,2=say,3=can,4=you,5=see 1=by,2=the,3=dawn's 1=early,2=light @@ -220,7 +220,7 @@ Example with index-numbered input and output: .. code-block:: none :emphasize-lines: 1-1 - $ cat data/mydata.txt + cat data/mydata.txt oh say can you see by the dawn's early light @@ -228,7 +228,7 @@ Example with index-numbered input and output: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --nidx --fs ' ' --repifs cut -f 2,3 data/mydata.txt + mlr --nidx --fs ' ' --repifs cut -f 2,3 data/mydata.txt say can the dawn's light @@ -252,7 +252,7 @@ An **array of single-level objects** is, quite simply, **a table**: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --json head -n 2 then cut -f color,shape data/json-example-1.json + mlr --json head -n 2 then cut -f color,shape data/json-example-1.json { "color": "yellow", "shape": "triangle" @@ -265,7 +265,7 @@ An **array of single-level objects** is, quite simply, **a table**: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --json --jvstack head -n 2 then cut -f color,u,v data/json-example-1.json + mlr --json --jvstack head -n 2 then cut -f color,u,v data/json-example-1.json { "color": "yellow", "u": 0.6321695890307647, @@ -280,7 +280,7 @@ An **array of single-level objects** is, quite simply, **a table**: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --ijson --opprint stats1 -a mean,stddev,count -f u -g shape data/json-example-1.json + mlr --ijson --opprint stats1 -a mean,stddev,count -f u -g shape data/json-example-1.json shape u_mean u_stddev u_count triangle 0.5839952367477192 0.13118354465618046 3 square 0.409355036804889 0.3654281755508655 4 @@ -294,7 +294,7 @@ Additionally, Miller can **tabularize nested objects by concatentating keys**: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --json --jvstack head -n 2 data/json-example-2.json + mlr --json --jvstack head -n 2 data/json-example-2.json { "flag": 1, "i": 11, @@ -327,7 +327,7 @@ Additionally, Miller can **tabularize nested objects by concatentating keys**: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --ijson --opprint head -n 4 data/json-example-2.json + mlr --ijson --opprint head -n 4 data/json-example-2.json flag i attributes.color attributes.shape values.u values.v values.w values.x 1 11 yellow triangle 0.632170 0.988721 0.436498 5.798188 1 15 red square 0.219668 0.001257 0.792778 2.944117 @@ -339,7 +339,7 @@ Note in particular that as far as Miller's ``put`` and ``filter``, as well as ot .. code-block:: none :emphasize-lines: 1-3 - $ mlr --json --jvstack head -n 1 \ + mlr --json --jvstack head -n 1 \ then put '${values:uv} = ${values:u} * ${values:v}' \ data/json-example-2.json { @@ -367,7 +367,7 @@ Suppose we have arrays like this in our input data: .. code-block:: none :emphasize-lines: 1-1 - $ cat data/json-example-3.json + cat data/json-example-3.json { "label": "orange", "values": [12.2, 13.8, 17.2] @@ -382,7 +382,7 @@ Then integer indices (starting from 0 and counting up) are used as map keys: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --ijson --oxtab cat data/json-example-3.json + mlr --ijson --oxtab cat data/json-example-3.json label orange values.1 12.2 values.2 13.8 @@ -397,7 +397,7 @@ When the data are written back out as JSON, field names are re-expanded as above .. code-block:: none :emphasize-lines: 1-1 - $ mlr --json --jvstack cat data/json-example-3.json + mlr --json --jvstack cat data/json-example-3.json { "label": "orange", "values": [12.2, 13.8, 17.2] @@ -440,7 +440,7 @@ Miller's pretty-print format is like CSV, but column-aligned. For example, comp .. code-block:: none :emphasize-lines: 1-1 - $ mlr --ocsv cat data/small + mlr --ocsv cat data/small a,b,i,x,y pan,pan,1,0.3467901443380824,0.7268028627434533 eks,pan,2,0.7586799647899636,0.5221511083334797 @@ -451,7 +451,7 @@ Miller's pretty-print format is like CSV, but column-aligned. For example, comp .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint cat data/small + mlr --opprint cat data/small a b i x y pan pan 1 0.3467901443380824 0.7268028627434533 eks pan 2 0.7586799647899636 0.5221511083334797 @@ -468,7 +468,7 @@ For output only (this isn't supported in the input-scanner as of 5.0.0) you can .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint --barred cat data/small + mlr --opprint --barred cat data/small +-----+-----+---+---------------------+---------------------+ | a | b | i | x | y | +-----+-----+---+---------------------+---------------------+ @@ -550,7 +550,7 @@ Markdown format looks like this: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --omd cat data/small + mlr --omd cat data/small | a | b | i | x | y | | --- | --- | --- | --- | --- | | pan | pan | 1 | 0.3467901443380824 | 0.7268028627434533 | @@ -573,7 +573,7 @@ While you can do format conversion using ``mlr --icsv --ojson cat myfile.csv``, .. code-block:: none :emphasize-lines: 1-1 - $ mlr --usage-format-conversion-keystroke-saver-options + mlr --usage-format-conversion-keystroke-saver-options As keystroke-savers for format-conversion you may use the following: --c2t --c2d --c2n --c2j --c2x --c2p --c2m --t2c --t2d --t2n --t2j --t2x --t2p --t2m @@ -609,7 +609,7 @@ You can include comments within your data files, and either have them ignored, o .. code-block:: none :emphasize-lines: 1-1 - $ mlr --usage-comments-in-data + mlr --usage-comments-in-data --skip-comments Ignore commented lines (prefixed by "#") within the input. --skip-comments-with {string} Ignore commented lines within input, with @@ -632,7 +632,7 @@ Examples: .. code-block:: none :emphasize-lines: 1-1 - $ cat data/budget.csv + cat data/budget.csv # Asana -- here are the budget figures you asked for! type,quantity purple,456.78 @@ -642,7 +642,7 @@ Examples: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --skip-comments --icsv --opprint sort -nr quantity data/budget.csv + mlr --skip-comments --icsv --opprint sort -nr quantity data/budget.csv type quantity green 678.12 purple 456.78 @@ -651,7 +651,7 @@ Examples: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --pass-comments --icsv --opprint sort -nr quantity data/budget.csv + mlr --pass-comments --icsv --opprint sort -nr quantity data/budget.csv # Asana -- here are the budget figures you asked for! type quantity green 678.12 diff --git a/docs6/genrst-filter b/docs6/genrst-filter index 4c5cd1c0a..45e61cba8 100755 --- a/docs6/genrst-filter +++ b/docs6/genrst-filter @@ -92,16 +92,14 @@ def run_command(cmd_lines, output_handle) if status != 0 raise "\"#{cmd}\" exited with non-zero code #{status}." end - cmd_line_1 = cmd_lines.shift - write_card(['$ '+cmd_line_1] + cmd_lines, cmd_output.split(/\n/), output_handle) + write_card(cmd_lines, cmd_output.split(/\n/), output_handle) end # ---------------------------------------------------------------- def run_command_tolerating_error(cmd_lines, output_handle) cmd = cmd_lines.join("\n") cmd_output = `#{cmd} 2>&1` - cmd_line_1 = cmd_lines.shift - write_card(['$ '+cmd_line_1] + cmd_lines, cmd_output.split(/\n/), output_handle) + write_card(cmd_lines, cmd_output.split(/\n/), output_handle) end # ---------------------------------------------------------------- @@ -112,8 +110,7 @@ end # ---------------------------------------------------------------- def show_command(cmd_lines, output_handle) # The command can be multi-line - cmd_line_1 = cmd_lines.shift - write_card(['$ '+cmd_line_1] + cmd_lines, [], output_handle) + write_card(cmd_lines, [], output_handle) end # ---------------------------------------------------------------- diff --git a/docs6/joins.rst b/docs6/joins.rst index 61dfdb363..c42334cda 100644 --- a/docs6/joins.rst +++ b/docs6/joins.rst @@ -14,7 +14,7 @@ For example, the right file here has nine records, and the left file should add .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsvlite --opprint cat data/join-u-left.csv + mlr --icsvlite --opprint cat data/join-u-left.csv hostname ipaddr nadir.east.our.org 10.3.1.18 zenith.west.our.org 10.3.1.27 @@ -23,7 +23,7 @@ For example, the right file here has nine records, and the left file should add .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsvlite --opprint cat data/join-u-right.csv + mlr --icsvlite --opprint cat data/join-u-right.csv ipaddr timestamp bytes 10.3.1.27 1448762579 4568 10.3.1.18 1448762578 8729 @@ -38,7 +38,7 @@ For example, the right file here has nine records, and the left file should add .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsvlite --opprint join -s -j ipaddr -f data/join-u-left.csv data/join-u-right.csv + mlr --icsvlite --opprint join -s -j ipaddr -f data/join-u-left.csv data/join-u-right.csv ipaddr hostname timestamp bytes 10.3.1.27 zenith.west.our.org 1448762579 4568 10.4.5.94 apoapsis.east.our.org 1448762579 17445 @@ -52,7 +52,7 @@ The solution (besides pre-sorting the input files on the join keys) is to simply .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsvlite --opprint join -u -j ipaddr -f data/join-u-left.csv data/join-u-right.csv + mlr --icsvlite --opprint join -u -j ipaddr -f data/join-u-left.csv data/join-u-right.csv ipaddr hostname timestamp bytes 10.3.1.27 zenith.west.our.org 1448762579 4568 10.3.1.18 nadir.east.our.org 1448762578 8729 @@ -89,7 +89,7 @@ Joining on color the results are as expected: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --csv join -j id -f data/color-codes.csv data/color-names.csv + mlr --csv join -j id -f data/color-codes.csv data/color-names.csv id,code,color 4,ff0000,red 2,00ff00,green @@ -99,7 +99,7 @@ However, if we ask for left-unpaireds, since there's no ``color`` column, we get .. code-block:: none :emphasize-lines: 1-1 - $ mlr --csv join --ul -j id -f data/color-codes.csv data/color-names.csv + mlr --csv join --ul -j id -f data/color-codes.csv data/color-names.csv id,code,color 4,ff0000,red 2,00ff00,green @@ -112,7 +112,7 @@ To fix this, we can use **unsparsify**: .. code-block:: none :emphasize-lines: 1-3 - $ mlr --csv join --ul -j id -f data/color-codes.csv \ + mlr --csv join --ul -j id -f data/color-codes.csv \ then unsparsify --fill-with "" \ data/color-names.csv id,code,color @@ -130,7 +130,7 @@ Suppose we have the following data: .. code-block:: none :emphasize-lines: 1-1 - $ cat multi-join/input.csv + cat multi-join/input.csv id,task 10,chop 20,puree @@ -146,7 +146,7 @@ And we want to augment the ``id`` column with lookups from the following data fi .. code-block:: none :emphasize-lines: 1-1 - $ cat multi-join/name-lookup.csv + cat multi-join/name-lookup.csv id,name 30,Alice 10,Bob @@ -155,7 +155,7 @@ And we want to augment the ``id`` column with lookups from the following data fi .. code-block:: none :emphasize-lines: 1-1 - $ cat multi-join/status-lookup.csv + cat multi-join/status-lookup.csv id,status 30,occupied 10,idle @@ -166,7 +166,7 @@ We can run the input file through multiple ``join`` commands in a ``then``-chain .. code-block:: none :emphasize-lines: 1-3 - $ mlr --icsv --opprint join -f multi-join/name-lookup.csv -j id \ + mlr --icsv --opprint join -f multi-join/name-lookup.csv -j id \ then join -f multi-join/status-lookup.csv -j id \ multi-join/input.csv id status name task diff --git a/docs6/keystroke-savers.rst b/docs6/keystroke-savers.rst index 1c2a2294a..d32b4a1e9 100644 --- a/docs6/keystroke-savers.rst +++ b/docs6/keystroke-savers.rst @@ -12,7 +12,7 @@ In our examples so far we've often made use of ``mlr --icsv --opprint`` or ``mlr .. code-block:: none :emphasize-lines: 1-1 - $ mlr --c2p head -n 2 example.csv + mlr --c2p head -n 2 example.csv color shape flag index quantity rate yellow triangle true 11 43.6498 9.8870 red square true 15 79.2778 0.0130 @@ -20,7 +20,7 @@ In our examples so far we've often made use of ``mlr --icsv --opprint`` or ``mlr .. code-block:: none :emphasize-lines: 1-1 - $ mlr --c2j head -n 2 example.csv + mlr --c2j head -n 2 example.csv { "color": "yellow", "shape": "triangle", @@ -48,7 +48,7 @@ Already we saw that you can put the filename first using ``--from``. When you're .. code-block:: none :emphasize-lines: 1-1 - $ mlr --c2p --from example.csv sort -nr index then head -n 3 + mlr --c2p --from example.csv sort -nr index then head -n 3 color shape flag index quantity rate purple square false 91 72.3735 8.2430 yellow circle true 87 63.5058 8.3350 @@ -57,7 +57,7 @@ Already we saw that you can put the filename first using ``--from``. When you're .. code-block:: none :emphasize-lines: 1-1 - $ mlr --c2p --from example.csv sort -nr index then head -n 3 then cut -f shape,quantity + mlr --c2p --from example.csv sort -nr index then head -n 3 then cut -f shape,quantity shape quantity square 72.3735 circle 63.5058 @@ -68,7 +68,7 @@ If there's more than one input file, you can use ``--mfrom``, then however many .. code-block:: none :emphasize-lines: 1-1 - $ mlr --c2p --mfrom data/*.csv -- sort -n index + mlr --c2p --mfrom data/*.csv -- sort -n index .mlrrc file ^^^^^^^^^^^ diff --git a/docs6/log-processing-examples.rst b/docs6/log-processing-examples.rst index f2899034d..17615175b 100644 --- a/docs6/log-processing-examples.rst +++ b/docs6/log-processing-examples.rst @@ -15,7 +15,7 @@ Suppose your program has printed something like this (`log.txt <./log.txt>`_): .. code-block:: none :emphasize-lines: 1-1 - $ cat log.txt + cat log.txt op=enter,time=1472819681 op=cache,type=A9,hit=0 op=cache,type=A4,hit=1 @@ -65,7 +65,7 @@ Each print statement simply contains local information: the current timestamp, w .. code-block:: none :emphasize-lines: 1-2 - $ grep op=cache log.txt \ + grep op=cache log.txt \ | mlr --idkvp --opprint stats1 -a mean -f hit -g type then sort -f type type hit_mean A1 0.8571428571428571 @@ -75,7 +75,7 @@ Each print statement simply contains local information: the current timestamp, w .. code-block:: none :emphasize-lines: 1-4 - $ mlr --from log.txt --opprint \ + mlr --from log.txt --opprint \ filter 'is_present($batch_size)' \ then step -a delta -f time,num_filtered \ then sec2gmt time @@ -92,7 +92,7 @@ Alternatively, we can simply group the similar data for a better look: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint group-like log.txt + mlr --opprint group-like log.txt op time enter 1472819681 @@ -145,7 +145,7 @@ Alternatively, we can simply group the similar data for a better look: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint group-like then sec2gmt time log.txt + mlr --opprint group-like then sec2gmt time log.txt op time enter 2016-09-02T12:34:41Z diff --git a/docs6/new-in-miller-6.rst b/docs6/new-in-miller-6.rst index b74fb2c56..b9e9500c7 100644 --- a/docs6/new-in-miller-6.rst +++ b/docs6/new-in-miller-6.rst @@ -89,7 +89,7 @@ For example (see https://github.com/johnkerl/miller/issues/178) you can now do .. code-block:: none :emphasize-lines: 1-1 - $ echo '{ "a": "0123" }' | mlr --json cat + echo '{ "a": "0123" }' | mlr --json cat { "a": "0123" } @@ -97,7 +97,7 @@ For example (see https://github.com/johnkerl/miller/issues/178) you can now do .. code-block:: none :emphasize-lines: 1-1 - $ echo '{ "x": 1.230, "y": 1.230000000 }' | mlr --json cat + echo '{ "x": 1.230, "y": 1.230000000 }' | mlr --json cat { "x": 1.230, "y": 1.230000000 diff --git a/docs6/operating-on-all-fields.rst b/docs6/operating-on-all-fields.rst index 92ad1ee66..479a4579d 100644 --- a/docs6/operating-on-all-fields.rst +++ b/docs6/operating-on-all-fields.rst @@ -12,7 +12,7 @@ Suppose you want to replace spaces with underscores in your column names: .. code-block:: none :emphasize-lines: 1-1 - $ cat data/spaces.csv + cat data/spaces.csv a b c,def,g h i 123,4567,890 2468,1357,3579 @@ -23,7 +23,7 @@ The simplest way is to use ``mlr rename`` with ``-g`` (for global replace, not j .. code-block:: none :emphasize-lines: 1-1 - $ mlr --csv rename -g -r ' ,_' data/spaces.csv + mlr --csv rename -g -r ' ,_' data/spaces.csv a_b_c,def,g_h_i 123,4567,890 2468,1357,3579 @@ -32,7 +32,7 @@ The simplest way is to use ``mlr rename`` with ``-g`` (for global replace, not j .. code-block:: none :emphasize-lines: 1-1 - $ mlr --csv --opprint rename -g -r ' ,_' data/spaces.csv + mlr --csv --opprint rename -g -r ' ,_' data/spaces.csv a_b_c def g_h_i 123 4567 890 2468 1357 3579 @@ -43,7 +43,7 @@ You can also do this with a for-loop: .. code-block:: none :emphasize-lines: 1-1 - $ cat data/bulk-rename-for-loop.mlr + cat data/bulk-rename-for-loop.mlr map newrec = {}; for (oldk, v in $*) { newrec[gsub(oldk, " ", "_")] = v; @@ -53,7 +53,7 @@ You can also do this with a for-loop: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsv --opprint put -f data/bulk-rename-for-loop.mlr data/spaces.csv + mlr --icsv --opprint put -f data/bulk-rename-for-loop.mlr data/spaces.csv a_b_c def g_h_i 123 4567 890 2468 1357 3579 @@ -67,7 +67,7 @@ How to do ``$name = gsub($name, "old", "new")`` for all fields? .. code-block:: none :emphasize-lines: 1-1 - $ cat data/sar.csv + cat data/sar.csv a,b,c the quick,brown fox,jumped over,the,lazy dogs @@ -75,7 +75,7 @@ How to do ``$name = gsub($name, "old", "new")`` for all fields? .. code-block:: none :emphasize-lines: 1-1 - $ cat data/sar.mlr + cat data/sar.mlr for (k in $*) { $[k] = gsub($[k], "e", "X"); } @@ -83,7 +83,7 @@ How to do ``$name = gsub($name, "old", "new")`` for all fields? .. code-block:: none :emphasize-lines: 1-1 - $ mlr --csv put -f data/sar.mlr data/sar.csv + mlr --csv put -f data/sar.mlr data/sar.csv a,b,c thX quick,brown fox,jumpXd ovXr,thX,lazy dogs @@ -96,7 +96,7 @@ Using Miller 5.0.0's map literals and assigning to ``$*``, you can fully general .. code-block:: none :emphasize-lines: 1-1 - $ cat data/small + cat data/small a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533 a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797 a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776 @@ -106,7 +106,7 @@ Using Miller 5.0.0's map literals and assigning to ``$*``, you can fully general .. code-block:: none :emphasize-lines: 1-15 - $ mlr put ' + mlr put ' begin { @i_cumu = 0; } diff --git a/docs6/programming-language.rst b/docs6/programming-language.rst index 7c300ca0e..136831050 100644 --- a/docs6/programming-language.rst +++ b/docs6/programming-language.rst @@ -18,7 +18,7 @@ Let's keep using the sample `example.csv <./example.csv>`_. When we type .. code-block:: none :emphasize-lines: 1-1 - $ mlr --c2p put '$cost = $quantity * $rate' example.csv + mlr --c2p put '$cost = $quantity * $rate' example.csv color shape flag index quantity rate cost yellow triangle true 11 43.6498 9.8870 431.5655726 red square true 15 79.2778 0.0130 1.0306114 @@ -46,7 +46,7 @@ You can use more than one statement, separating them with semicolons, and option .. code-block:: none :emphasize-lines: 1-1 - $ mlr --c2p put '$cost = $quantity * $rate; $index = $index * 100' example.csv + mlr --c2p put '$cost = $quantity * $rate; $index = $index * 100' example.csv color shape flag index quantity rate cost yellow triangle true 1100 43.6498 9.8870 431.5655726 red square true 1500 79.2778 0.0130 1.0306114 @@ -62,7 +62,7 @@ You can use more than one statement, separating them with semicolons, and option .. code-block:: none :emphasize-lines: 1-4 - $ mlr --c2p put ' + mlr --c2p put ' $cost = $quantity * $rate; $index *= 100 ' example.csv @@ -83,14 +83,14 @@ One of Miller's key features is the ability to express data-transformation right .. code-block:: none :emphasize-lines: 1-1 - $ cat dsl-example.mlr + cat dsl-example.mlr $cost = $quantity * $rate; $index *= 100 .. code-block:: none :emphasize-lines: 1-1 - $ mlr --c2p put -f dsl-example.mlr example.csv + mlr --c2p put -f dsl-example.mlr example.csv color shape flag index quantity rate cost yellow triangle true 1100 43.6498 9.8870 431.5655726 red square true 1500 79.2778 0.0130 1.0306114 @@ -117,7 +117,7 @@ To make ``begin`` and ``end`` statements useful, we need somewhere to put things .. code-block:: none :emphasize-lines: 1-1 - $ mlr --c2p --from example.csv put 'begin { @sum = 0 } @sum += $quantity; end {emit @sum}' + mlr --c2p --from example.csv put 'begin { @sum = 0 } @sum += $quantity; end {emit @sum}' color shape flag index quantity rate yellow triangle true 11 43.6498 9.8870 red square true 15 79.2778 0.0130 @@ -138,14 +138,14 @@ If you want the end-block output to be the only output, and not include the inpu .. code-block:: none :emphasize-lines: 1-1 - $ mlr --c2p --from example.csv put -q 'begin { @sum = 0 } @sum += $quantity; end {emit @sum}' + mlr --c2p --from example.csv put -q 'begin { @sum = 0 } @sum += $quantity; end {emit @sum}' sum 652.7185 .. code-block:: none :emphasize-lines: 1-1 - $ mlr --c2j --from example.csv put -q 'begin { @sum = 0 } @sum += $quantity; end {emit @sum}' + mlr --c2j --from example.csv put -q 'begin { @sum = 0 } @sum += $quantity; end {emit @sum}' { "sum": 652.7185 } @@ -153,7 +153,7 @@ If you want the end-block output to be the only output, and not include the inpu .. code-block:: none :emphasize-lines: 1-6 - $ mlr --c2j --from example.csv put -q ' + mlr --c2j --from example.csv put -q ' begin { @count = 0; @sum = 0 } @count += 1; @sum += $quantity; @@ -180,7 +180,7 @@ Also inspired by `AWK `_, the Miller DSL has .. code-block:: none :emphasize-lines: 1-1 - $ cat context-example.mlr + cat context-example.mlr $nf = NF; $nr = NR; $fnr = FNR; @@ -191,7 +191,7 @@ Also inspired by `AWK `_, the Miller DSL has .. code-block:: none :emphasize-lines: 1-1 - $ mlr --c2p put -f context-example.mlr data/a.csv data/b.csv + mlr --c2p put -f context-example.mlr data/a.csv data/b.csv a b c nf nr fnr filename filenum newnf 1 2 3 3 1 1 data/a.csv 1 8 4 5 6 3 2 2 data/a.csv 1 8 @@ -205,7 +205,7 @@ You can define your own functions: .. code-block:: none :emphasize-lines: 1-1 - $ cat factorial-example.mlr + cat factorial-example.mlr func factorial(n) { if (n <= 1) { return n @@ -217,7 +217,7 @@ You can define your own functions: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --c2p --from example.csv put -f factorial-example.mlr -e '$fact = factorial(NR)' + mlr --c2p --from example.csv put -f factorial-example.mlr -e '$fact = factorial(NR)' color shape flag index quantity rate fact yellow triangle true 11 43.6498 9.8870 1 red square true 15 79.2778 0.0130 2 @@ -244,7 +244,7 @@ Suppose you want to only compute sums conditionally -- you can use an ``if`` sta .. code-block:: none :emphasize-lines: 1-1 - $ cat if-example.mlr + cat if-example.mlr begin { @count_of_red = 0; @sum_of_red = 0 @@ -262,7 +262,7 @@ Suppose you want to only compute sums conditionally -- you can use an ``if`` sta .. code-block:: none :emphasize-lines: 1-1 - $ mlr --c2p --from example.csv put -q -f if-example.mlr + mlr --c2p --from example.csv put -q -f if-example.mlr count_of_red sum_of_red 4 247.84139999999996 @@ -278,7 +278,7 @@ current record as a hashmap: .. code-block:: none :emphasize-lines: 1-1 - $ cat for-example.mlr + cat for-example.mlr for (k, v in $*) { print "KEY IS ". k . " VALUE IS ". v; } @@ -287,7 +287,7 @@ current record as a hashmap: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --csv cat data/a.csv + mlr --csv cat data/a.csv a,b,c 1,2,3 4,5,6 @@ -295,7 +295,7 @@ current record as a hashmap: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --csv --from data/a.csv put -qf for-example.mlr + mlr --csv --from data/a.csv put -qf for-example.mlr KEY IS a VALUE IS 1 KEY IS b VALUE IS 2 KEY IS c VALUE IS 3 @@ -337,7 +337,7 @@ For example, you can sum up all the ``$a`` values across records without having .. code-block:: none :emphasize-lines: 1-1 - $ mlr --json cat absent-example.json + mlr --json cat absent-example.json { "a": 1, "b": 2 @@ -353,7 +353,7 @@ For example, you can sum up all the ``$a`` values across records without having .. code-block:: none :emphasize-lines: 1-1 - $ mlr --json put '@sum_of_a += $a; end {emit @sum_of_a}' absent-example.json + mlr --json put '@sum_of_a += $a; end {emit @sum_of_a}' absent-example.json { "a": 1, "b": 2 diff --git a/docs6/quick-examples.rst.in b/docs6/quick-examples.rst.in index b58a97bf1..e10489b60 100644 --- a/docs6/quick-examples.rst.in +++ b/docs6/quick-examples.rst.in @@ -6,70 +6,70 @@ Column select: .. code-block:: none :emphasize-lines: 1,1 - % mlr --csv cut -f hostname,uptime mydata.csv + mlr --csv cut -f hostname,uptime mydata.csv Add new columns as function of other columns: .. code-block:: none :emphasize-lines: 1,1 - % mlr --nidx put '$sum = $7 < 0.0 ? 3.5 : $7 + 2.1*$8' *.dat + mlr --nidx put '$sum = $7 < 0.0 ? 3.5 : $7 + 2.1*$8' *.dat Row filter: .. code-block:: none :emphasize-lines: 1,1 - % mlr --csv filter '$status != "down" && $upsec >= 10000' *.csv + mlr --csv filter '$status != "down" && $upsec >= 10000' *.csv Apply column labels and pretty-print: .. code-block:: none :emphasize-lines: 1,1 - % grep -v '^#' /etc/group | mlr --ifs : --nidx --opprint label group,pass,gid,member then sort -f group + grep -v '^#' /etc/group | mlr --ifs : --nidx --opprint label group,pass,gid,member then sort -f group Join multiple data sources on key columns: .. code-block:: none :emphasize-lines: 1,1 - % mlr join -j account_id -f accounts.dat then group-by account_name balances.dat + mlr join -j account_id -f accounts.dat then group-by account_name balances.dat Mulltiple formats including JSON: .. code-block:: none :emphasize-lines: 1,1 - % mlr --json put '$attr = sub($attr, "([0-9]+)_([0-9]+)_.*", "\1:\2")' data/*.json + mlr --json put '$attr = sub($attr, "([0-9]+)_([0-9]+)_.*", "\1:\2")' data/*.json Aggregate per-column statistics: .. code-block:: none :emphasize-lines: 1,1 - % mlr stats1 -a min,mean,max,p10,p50,p90 -f flag,u,v data/* + mlr stats1 -a min,mean,max,p10,p50,p90 -f flag,u,v data/* Linear regression: .. code-block:: none :emphasize-lines: 1,1 - % mlr stats2 -a linreg-pca -f u,v -g shape data/* + mlr stats2 -a linreg-pca -f u,v -g shape data/* Aggregate custom per-column statistics: .. code-block:: none :emphasize-lines: 1,1 - % mlr put -q '@sum[$a][$b] += $x; end {emit @sum, "a", "b"}' data/* + mlr put -q '@sum[$a][$b] += $x; end {emit @sum, "a", "b"}' data/* Iterate over data using DSL expressions: .. code-block:: none :emphasize-lines: 1,1 - % mlr --from estimates.tbl put ' + mlr --from estimates.tbl put ' for (k,v in $*) { if (is_numeric(v) && k =~ "^[t-z].*$") { $sum += v; $count += 1 @@ -83,32 +83,32 @@ Run DSL expressions from a script file: .. code-block:: none :emphasize-lines: 1,1 - % mlr --from infile.dat put -f analyze.mlr + mlr --from infile.dat put -f analyze.mlr Split/reduce output to multiple filenames: .. code-block:: none :emphasize-lines: 1,1 - % mlr --from infile.dat put 'tee > "./taps/data-".$a."-".$b, $*' + mlr --from infile.dat put 'tee > "./taps/data-".$a."-".$b, $*' Compressed I/O: .. code-block:: none :emphasize-lines: 1,1 - % mlr --from infile.dat put 'tee | "gzip > ./taps/data-".$a."-".$b.".gz", $*' + mlr --from infile.dat put 'tee | "gzip > ./taps/data-".$a."-".$b.".gz", $*' Interoperate with other data-processing tools using standard pipes: .. code-block:: none :emphasize-lines: 1,1 - % mlr --from infile.dat put -q '@v=$*; dump | "jq .[]"' + mlr --from infile.dat put -q '@v=$*; dump | "jq .[]"' Tap/trace: .. code-block:: none :emphasize-lines: 1,1 - % mlr --from infile.dat put '(NR % 1000 == 0) { print > stderr, "Checkpoint ".NR}' + mlr --from infile.dat put '(NR % 1000 == 0) { print > stderr, "Checkpoint ".NR}' diff --git a/docs6/record-heterogeneity.rst b/docs6/record-heterogeneity.rst index cb4b66772..dc3da596d 100644 --- a/docs6/record-heterogeneity.rst +++ b/docs6/record-heterogeneity.rst @@ -19,7 +19,7 @@ Miller simply prints a newline and a new header when there is a schema change. W .. code-block:: none :emphasize-lines: 1-1 - $ cat data/het.dkvp + cat data/het.dkvp resource=/path/to/file,loadsec=0.45,ok=true record_count=100,resource=/path/to/file resource=/path/to/second/file,loadsec=0.32,ok=true @@ -29,7 +29,7 @@ Miller simply prints a newline and a new header when there is a schema change. W .. code-block:: none :emphasize-lines: 1-1 - $ mlr --ocsvlite cat data/het.dkvp + mlr --ocsvlite cat data/het.dkvp resource,loadsec,ok /path/to/file,0.45,true @@ -48,7 +48,7 @@ Miller simply prints a newline and a new header when there is a schema change. W .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint cat data/het.dkvp + mlr --opprint cat data/het.dkvp resource loadsec ok /path/to/file 0.45 true @@ -69,7 +69,7 @@ Miller handles explicit header changes as just shown. If your CSV input contains .. code-block:: none :emphasize-lines: 1-1 - $ cat data/ragged.csv + cat data/ragged.csv a,b,c 1,2,3 4,5 @@ -78,7 +78,7 @@ Miller handles explicit header changes as just shown. If your CSV input contains .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsv --oxtab --allow-ragged-csv-input cat data/ragged.csv + mlr --icsv --oxtab --allow-ragged-csv-input cat data/ragged.csv a 1 b 2 c 3 @@ -97,7 +97,7 @@ You may also find Miller's ``group-like`` feature handy (see also :doc:`referenc .. code-block:: none :emphasize-lines: 1-1 - $ mlr --ocsvlite group-like data/het.dkvp + mlr --ocsvlite group-like data/het.dkvp resource,loadsec,ok /path/to/file,0.45,true /path/to/second/file,0.32,true @@ -110,7 +110,7 @@ You may also find Miller's ``group-like`` feature handy (see also :doc:`referenc .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint group-like data/het.dkvp + mlr --opprint group-like data/het.dkvp resource loadsec ok /path/to/file 0.45 true /path/to/second/file 0.32 true @@ -128,7 +128,7 @@ For these formats, record-heterogeneity comes naturally: .. code-block:: none :emphasize-lines: 1-1 - $ cat data/het.dkvp + cat data/het.dkvp resource=/path/to/file,loadsec=0.45,ok=true record_count=100,resource=/path/to/file resource=/path/to/second/file,loadsec=0.32,ok=true @@ -138,7 +138,7 @@ For these formats, record-heterogeneity comes naturally: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --onidx --ofs ' ' cat data/het.dkvp + mlr --onidx --ofs ' ' cat data/het.dkvp /path/to/file 0.45 true 100 /path/to/file /path/to/second/file 0.32 true @@ -148,7 +148,7 @@ For these formats, record-heterogeneity comes naturally: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --oxtab cat data/het.dkvp + mlr --oxtab cat data/het.dkvp resource /path/to/file loadsec 0.45 ok true @@ -170,7 +170,7 @@ For these formats, record-heterogeneity comes naturally: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --oxtab group-like data/het.dkvp + mlr --oxtab group-like data/het.dkvp resource /path/to/file loadsec 0.45 ok true @@ -197,7 +197,7 @@ Miller operates on specified fields and takes the rest along: for example, if yo .. code-block:: none :emphasize-lines: 1-1 - $ cat data/sort-het.dkvp + cat data/sort-het.dkvp count=500,color=green count=600 status=ok,count=250,hours=0.22 @@ -209,7 +209,7 @@ Miller operates on specified fields and takes the rest along: for example, if yo .. code-block:: none :emphasize-lines: 1-1 - $ mlr sort -n count data/sort-het.dkvp + mlr sort -n count data/sort-het.dkvp count=100,color=green status=ok,count=200,hours=3.4 status=ok,count=250,hours=0.22 diff --git a/docs6/reference-dsl-arrays.rst b/docs6/reference-dsl-arrays.rst index d4ab4ff44..0a6bf720c 100644 --- a/docs6/reference-dsl-arrays.rst +++ b/docs6/reference-dsl-arrays.rst @@ -9,7 +9,7 @@ TODO .. code-block:: none :emphasize-lines: 1-1 - $ mlr --json cat data/array-example.json + mlr --json cat data/array-example.json { "key": "ax04", "samples": [45, 67, 22] diff --git a/docs6/reference-dsl-control-structures.rst b/docs6/reference-dsl-control-structures.rst index 2db3b5079..1508e5ae1 100644 --- a/docs6/reference-dsl-control-structures.rst +++ b/docs6/reference-dsl-control-structures.rst @@ -12,7 +12,7 @@ These are reminiscent of ``awk`` syntax. They can be used to allow assignments .. code-block:: none :emphasize-lines: 1-1 - $ mlr cat data/put-gating-example-1.dkvp + mlr cat data/put-gating-example-1.dkvp x=-1 x=0 x=1 @@ -22,7 +22,7 @@ These are reminiscent of ``awk`` syntax. They can be used to allow assignments .. code-block:: none :emphasize-lines: 1-1 - $ mlr put '$x > 0.0 { $y = log10($x); $z = sqrt($y) }' data/put-gating-example-1.dkvp + mlr put '$x > 0.0 { $y = log10($x); $z = sqrt($y) }' data/put-gating-example-1.dkvp x=-1 x=0 x=1,y=0,z=0 @@ -32,7 +32,7 @@ These are reminiscent of ``awk`` syntax. They can be used to allow assignments .. code-block:: none :emphasize-lines: 1-1 - $ mlr cat data/put-gating-example-2.dkvp + mlr cat data/put-gating-example-2.dkvp a=abc_123 a=some other name a=xyz_789 @@ -40,7 +40,7 @@ These are reminiscent of ``awk`` syntax. They can be used to allow assignments .. code-block:: none :emphasize-lines: 1-5 - $ mlr put ' + mlr put ' $a =~ "([a-z]+)_([0-9]+)" { $b = "left_\1"; $c = "right_\2" }' \ @@ -54,7 +54,7 @@ This produces heteregenous output which Miller, of course, has no problems with .. code-block:: none :emphasize-lines: 1-1 - $ mlr put '$x > 0.0; $y = log10($x); $z = sqrt($y)' data/put-gating-example-1.dkvp + mlr put '$x > 0.0; $y = log10($x); $z = sqrt($y)' data/put-gating-example-1.dkvp x=1,y=0,z=0 x=2,y=0.3010299956639812,z=0.5486620049392715 x=3,y=0.4771212547196624,z=0.6907396432228734 @@ -62,7 +62,7 @@ This produces heteregenous output which Miller, of course, has no problems with .. code-block:: none :emphasize-lines: 1-5 - $ mlr put ' + mlr put ' $a =~ "([a-z]+)_([0-9]+)"; $b = "left_\1"; $c = "right_\2" @@ -107,7 +107,7 @@ Miller's ``while`` and ``do-while`` are unsurprising in comparison to various la .. code-block:: none :emphasize-lines: 1-6 - $ echo x=1,y=2 | mlr put ' + echo x=1,y=2 | mlr put ' while (NF < 10) { $[NF+1] = "" } @@ -118,7 +118,7 @@ Miller's ``while`` and ``do-while`` are unsurprising in comparison to various la .. code-block:: none :emphasize-lines: 1-9 - $ echo x=1,y=2 | mlr put ' + echo x=1,y=2 | mlr put ' do { $[NF+1] = ""; if (NF == 5) { @@ -147,7 +147,7 @@ The ``key`` variable is always bound to the *key* of key-value pairs: .. code-block:: none :emphasize-lines: 1-8 - $ mlr --from data/small put ' + mlr --from data/small put ' print "NR = ".NR; for (key in $*) { value = $[key]; @@ -194,7 +194,7 @@ The ``key`` variable is always bound to the *key* of key-value pairs: .. code-block:: none :emphasize-lines: 1-8 - $ mlr -n put ' + mlr -n put ' end { o = {1:2, 3:{4:5}}; for (key in o) { @@ -215,7 +215,7 @@ Single-level keys may be gotten at using either ``for(k,v)`` or ``for((k),v)``; .. code-block:: none :emphasize-lines: 1-1 - $ cat data/for-srec-example.tbl + cat data/for-srec-example.tbl label1 label2 f1 f2 f3 blue green 100 240 350 red green 120 11 195 @@ -224,7 +224,7 @@ Single-level keys may be gotten at using either ``for(k,v)`` or ``for((k),v)``; .. code-block:: none :emphasize-lines: 1-11 - $ mlr --pprint --from data/for-srec-example.tbl put ' + mlr --pprint --from data/for-srec-example.tbl put ' $sum1 = $f1 + $f2 + $f3; $sum2 = 0; $sum3 = 0; @@ -243,7 +243,7 @@ Single-level keys may be gotten at using either ``for(k,v)`` or ``for((k),v)``; .. code-block:: none :emphasize-lines: 1-1 - $ mlr --from data/small --opprint put 'for (k,v in $*) { $[k."_type"] = typeof(v) }' + mlr --from data/small --opprint put 'for (k,v in $*) { $[k."_type"] = typeof(v) }' a b i x y a_type b_type i_type x_type y_type pan pan 1 0.3467901443380824 0.7268028627434533 string string int float float eks pan 2 0.7586799647899636 0.5221511083334797 string string int float float @@ -258,7 +258,7 @@ Important note: to avoid inconsistent looping behavior in case you're setting ne .. code-block:: none :emphasize-lines: 1-10 - $ mlr --from data/small --opprint put ' + mlr --from data/small --opprint put ' $sum1 = 0; $sum2 = 0; for (k,v in $*) { @@ -280,7 +280,7 @@ It can be confusing to modify the stream record while iterating over a copy of i .. code-block:: none :emphasize-lines: 1-9 - $ mlr --from data/small --opprint put ' + mlr --from data/small --opprint put ' sum = 0; for (k,v in $*) { if (is_numeric(v)) { @@ -314,7 +314,7 @@ That's confusing in the abstract, so a concrete example is in order. Suppose the .. code-block:: none :emphasize-lines: 1-10 - $ mlr -n put --jknquoteint -q ' + mlr -n put --jknquoteint -q ' begin { @myvar = { 1: 2, @@ -343,7 +343,7 @@ Then we can get at various values as follows: .. code-block:: none :emphasize-lines: 1-16 - $ mlr -n put --jknquoteint -q ' + mlr -n put --jknquoteint -q ' begin { @myvar = { 1: 2, @@ -366,7 +366,7 @@ Then we can get at various values as follows: .. code-block:: none :emphasize-lines: 1-17 - $ mlr -n put --jknquoteint -q ' + mlr -n put --jknquoteint -q ' begin { @myvar = { 1: 2, @@ -389,7 +389,7 @@ Then we can get at various values as follows: .. code-block:: none :emphasize-lines: 1-17 - $ mlr -n put --jknquoteint -q ' + mlr -n put --jknquoteint -q ' begin { @myvar = { 1: 2, @@ -416,7 +416,7 @@ These are supported as follows: .. code-block:: none :emphasize-lines: 1-7 - $ mlr --from data/small --opprint put ' + mlr --from data/small --opprint put ' num suma = 0; for (a = 1; a <= NR; a += 1) { suma += a; @@ -433,7 +433,7 @@ These are supported as follows: .. code-block:: none :emphasize-lines: 1-10 - $ mlr --from data/small --opprint put ' + mlr --from data/small --opprint put ' num suma = 0; num sumb = 0; for (num a = 1, num b = 1; a <= NR; a += 1, b *= 2) { @@ -470,7 +470,7 @@ Miller supports an ``awk``-like ``begin/end`` syntax. The statements in the ``b .. code-block:: none :emphasize-lines: 1-5 - $ mlr put ' + mlr put ' begin { @sum = 0 }; @x_sum += $x; end { emit @x_sum } @@ -492,7 +492,7 @@ Since uninitialized out-of-stream variables default to 0 for addition/substracti .. code-block:: none :emphasize-lines: 1-4 - $ mlr put ' + mlr put ' @x_sum += $x; end { emit @x_sum } ' ../data/small @@ -513,7 +513,7 @@ The **put -q** option is a shorthand which suppresses printing of each output re .. code-block:: none :emphasize-lines: 1-4 - $ mlr put -q ' + mlr put -q ' @x_sum += $x; end { emit @x_sum } ' ../data/small @@ -524,7 +524,7 @@ We can do similarly with multiple out-of-stream variables: .. code-block:: none :emphasize-lines: 1-8 - $ mlr put -q ' + mlr put -q ' @x_count += 1; @x_sum += $x; end { @@ -540,7 +540,7 @@ This is of course not much different than .. code-block:: none :emphasize-lines: 1-1 - $ mlr stats1 -a count,sum -f x ../data/small + mlr stats1 -a count,sum -f x ../data/small x_count=10,x_sum=4.536293840335763 Note that it's a syntax error for begin/end blocks to refer to field names (beginning with ``$``), since these execute outside the context of input records. diff --git a/docs6/reference-dsl-filter-statements.rst b/docs6/reference-dsl-filter-statements.rst index 12b40bc36..ae125047a 100644 --- a/docs6/reference-dsl-filter-statements.rst +++ b/docs6/reference-dsl-filter-statements.rst @@ -9,14 +9,14 @@ You can use ``filter`` within ``put``. In fact, the following two are synonymous .. code-block:: none :emphasize-lines: 1-1 - $ mlr filter 'NR==2 || NR==3' data/small + mlr filter 'NR==2 || NR==3' data/small a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797 a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776 .. code-block:: none :emphasize-lines: 1-1 - $ mlr put 'filter NR==2 || NR==3' data/small + mlr put 'filter NR==2 || NR==3' data/small a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797 a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776 @@ -25,7 +25,7 @@ The former, of course, is much easier to type. But the latter allows you to defi .. code-block:: none :emphasize-lines: 1-1 - $ mlr put '@running_sum += $x; filter @running_sum > 1.3' data/small + mlr put '@running_sum += $x; filter @running_sum > 1.3' data/small a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776 a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463 a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729 @@ -33,6 +33,6 @@ The former, of course, is much easier to type. But the latter allows you to defi .. code-block:: none :emphasize-lines: 1-1 - $ mlr put '$z = $x * $y; filter $z > 0.3' data/small + mlr put '$z = $x * $y; filter $z > 0.3' data/small a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797,z=0.3961455844854848 a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729,z=0.4951063394654227 diff --git a/docs6/reference-dsl-output-statements.rst b/docs6/reference-dsl-output-statements.rst index 57a2d8e2c..02341f102 100644 --- a/docs6/reference-dsl-output-statements.rst +++ b/docs6/reference-dsl-output-statements.rst @@ -79,13 +79,13 @@ Details: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --help-keyword print + mlr --help-keyword print TODO: port mlr_dsl_keyword_usage .. code-block:: none :emphasize-lines: 1-1 - $ mlr --help-keyword dump + mlr --help-keyword dump TODO: port mlr_dsl_keyword_usage * ``mlr put`` sends the current record (possibly modified by the ``put`` expression) to the output record stream. Records are then input to the following verb in a ``then``-chain (if any), else printed to standard output (unless ``put -q``). The **tee** keyword *additionally* writes the output record to specified file(s) or pipe-to command, or immediately to ``stdout``/``stderr``. @@ -93,7 +93,7 @@ Details: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --help-keyword tee + mlr --help-keyword tee TODO: port mlr_dsl_keyword_usage * ``mlr put``'s ``emitf``, ``emitp``, and ``emit`` send out-of-stream variables to the output record stream. These are then input to the following verb in a ``then``-chain (if any), else printed to standard output. When redirected with ``>``, ``>>``, or ``|``, they *instead* write the out-of-stream variable(s) to specified file(s) or pipe-to command, or immediately to ``stdout``/``stderr``. @@ -101,19 +101,19 @@ Details: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --help-keyword emitf + mlr --help-keyword emitf TODO: port mlr_dsl_keyword_usage .. code-block:: none :emphasize-lines: 1-1 - $ mlr --help-keyword emitp + mlr --help-keyword emitp TODO: port mlr_dsl_keyword_usage .. code-block:: none :emphasize-lines: 1-1 - $ mlr --help-keyword emit + mlr --help-keyword emit TODO: port mlr_dsl_keyword_usage .. _reference-dsl-emit-statements: @@ -128,7 +128,7 @@ Use **emitf** to output several out-of-stream variables side-by-side in the same .. code-block:: none :emphasize-lines: 1-6 - $ mlr put -q ' + mlr put -q ' @count += 1; @x_sum += $x; @y_sum += $y; @@ -141,7 +141,7 @@ Use **emit** to output an out-of-stream variable. If it's non-indexed you'll get .. code-block:: none :emphasize-lines: 1-1 - $ cat data/small + cat data/small a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533 a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797 a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776 @@ -151,7 +151,7 @@ Use **emit** to output an out-of-stream variable. If it's non-indexed you'll get .. code-block:: none :emphasize-lines: 1-1 - $ mlr put -q '@sum += $x; end { dump }' data/small + mlr put -q '@sum += $x; end { dump }' data/small { "sum": 2.264761728567491 } @@ -159,7 +159,7 @@ Use **emit** to output an out-of-stream variable. If it's non-indexed you'll get .. code-block:: none :emphasize-lines: 1-1 - $ mlr put -q '@sum += $x; end { emit @sum }' data/small + mlr put -q '@sum += $x; end { emit @sum }' data/small sum=2.264761728567491 If it's indexed then use as many names after ``emit`` as there are indices: @@ -167,7 +167,7 @@ If it's indexed then use as many names after ``emit`` as there are indices: .. code-block:: none :emphasize-lines: 1-1 - $ mlr put -q '@sum[$a] += $x; end { dump }' data/small + mlr put -q '@sum[$a] += $x; end { dump }' data/small { "sum": { "pan": 0.3467901443380824, @@ -179,7 +179,7 @@ If it's indexed then use as many names after ``emit`` as there are indices: .. code-block:: none :emphasize-lines: 1-1 - $ mlr put -q '@sum[$a] += $x; end { emit @sum, "a" }' data/small + mlr put -q '@sum[$a] += $x; end { emit @sum, "a" }' data/small a=pan,sum=0.3467901443380824 a=eks,sum=1.1400793586611044 a=wye,sum=0.7778922255683036 @@ -187,7 +187,7 @@ If it's indexed then use as many names after ``emit`` as there are indices: .. code-block:: none :emphasize-lines: 1-1 - $ mlr put -q '@sum[$a][$b] += $x; end { dump }' data/small + mlr put -q '@sum[$a][$b] += $x; end { dump }' data/small { "sum": { "pan": { @@ -207,7 +207,7 @@ If it's indexed then use as many names after ``emit`` as there are indices: .. code-block:: none :emphasize-lines: 1-1 - $ mlr put -q '@sum[$a][$b] += $x; end { emit @sum, "a", "b" }' data/small + mlr put -q '@sum[$a][$b] += $x; end { emit @sum, "a", "b" }' data/small a=pan,b=pan,sum=0.3467901443380824 a=eks,b=pan,sum=0.7586799647899636 a=eks,b=wye,sum=0.38139939387114097 @@ -217,7 +217,7 @@ If it's indexed then use as many names after ``emit`` as there are indices: .. code-block:: none :emphasize-lines: 1-1 - $ mlr put -q '@sum[$a][$b][$i] += $x; end { dump }' data/small + mlr put -q '@sum[$a][$b][$i] += $x; end { dump }' data/small { "sum": { "pan": { @@ -247,7 +247,7 @@ If it's indexed then use as many names after ``emit`` as there are indices: .. code-block:: none :emphasize-lines: 1-4 - $ mlr put -q ' + mlr put -q ' @sum[$a][$b][$i] += $x; end { emit @sum, "a", "b", "i" } ' data/small @@ -262,7 +262,7 @@ Now for **emitp**: if you have as many names following ``emit`` as there are lev .. code-block:: none :emphasize-lines: 1-1 - $ mlr put -q '@sum[$a][$b] += $x; end { dump }' data/small + mlr put -q '@sum[$a][$b] += $x; end { dump }' data/small { "sum": { "pan": { @@ -282,7 +282,7 @@ Now for **emitp**: if you have as many names following ``emit`` as there are lev .. code-block:: none :emphasize-lines: 1-1 - $ mlr put -q '@sum[$a][$b] += $x; end { emit @sum, "a" }' data/small + mlr put -q '@sum[$a][$b] += $x; end { emit @sum, "a" }' data/small a=pan,pan=0.3467901443380824 a=eks,pan=0.7586799647899636,wye=0.38139939387114097 a=wye,wye=0.20460330576630303,pan=0.5732889198020006 @@ -290,13 +290,13 @@ Now for **emitp**: if you have as many names following ``emit`` as there are lev .. code-block:: none :emphasize-lines: 1-1 - $ mlr put -q '@sum[$a][$b] += $x; end { emit @sum }' data/small + mlr put -q '@sum[$a][$b] += $x; end { emit @sum }' data/small pan.pan=0.3467901443380824,eks.pan=0.7586799647899636,eks.wye=0.38139939387114097,wye.wye=0.20460330576630303,wye.pan=0.5732889198020006 .. code-block:: none :emphasize-lines: 1-1 - $ mlr put -q '@sum[$a][$b] += $x; end { emitp @sum, "a" }' data/small + mlr put -q '@sum[$a][$b] += $x; end { emitp @sum, "a" }' data/small a=pan,sum.pan=0.3467901443380824 a=eks,sum.pan=0.7586799647899636,sum.wye=0.38139939387114097 a=wye,sum.wye=0.20460330576630303,sum.pan=0.5732889198020006 @@ -304,13 +304,13 @@ Now for **emitp**: if you have as many names following ``emit`` as there are lev .. code-block:: none :emphasize-lines: 1-1 - $ mlr put -q '@sum[$a][$b] += $x; end { emitp @sum }' data/small + mlr put -q '@sum[$a][$b] += $x; end { emitp @sum }' data/small sum.pan.pan=0.3467901443380824,sum.eks.pan=0.7586799647899636,sum.eks.wye=0.38139939387114097,sum.wye.wye=0.20460330576630303,sum.wye.pan=0.5732889198020006 .. code-block:: none :emphasize-lines: 1-1 - $ mlr --oxtab put -q '@sum[$a][$b] += $x; end { emitp @sum }' data/small + mlr --oxtab put -q '@sum[$a][$b] += $x; end { emitp @sum }' data/small sum.pan.pan 0.3467901443380824 sum.eks.pan 0.7586799647899636 sum.eks.wye 0.38139939387114097 @@ -323,7 +323,7 @@ keys for ``emitp`` (it defaults to a colon): .. code-block:: none :emphasize-lines: 1-1 - $ mlr put -q --oflatsep / '@sum[$a][$b] += $x; end { emitp @sum, "a" }' data/small + mlr put -q --oflatsep / '@sum[$a][$b] += $x; end { emitp @sum, "a" }' data/small a=pan,sum.pan=0.3467901443380824 a=eks,sum.pan=0.7586799647899636,sum.wye=0.38139939387114097 a=wye,sum.wye=0.20460330576630303,sum.pan=0.5732889198020006 @@ -331,13 +331,13 @@ keys for ``emitp`` (it defaults to a colon): .. code-block:: none :emphasize-lines: 1-1 - $ mlr put -q --oflatsep / '@sum[$a][$b] += $x; end { emitp @sum }' data/small + mlr put -q --oflatsep / '@sum[$a][$b] += $x; end { emitp @sum }' data/small sum.pan.pan=0.3467901443380824,sum.eks.pan=0.7586799647899636,sum.eks.wye=0.38139939387114097,sum.wye.wye=0.20460330576630303,sum.wye.pan=0.5732889198020006 .. code-block:: none :emphasize-lines: 1-4 - $ mlr --oxtab put -q --oflatsep / ' + mlr --oxtab put -q --oflatsep / ' @sum[$a][$b] += $x; end { emitp @sum } ' data/small @@ -356,7 +356,7 @@ including their names in parentheses: .. code-block:: none :emphasize-lines: 1-10 - $ mlr --from data/medium --opprint put -q ' + mlr --from data/medium --opprint put -q ' @x_count[$a][$b] += 1; @x_sum[$a][$b] += $x; end { @@ -403,7 +403,7 @@ Use **emit all** (or ``emit @*`` which is synonymous) to output all out-of-strea .. code-block:: none :emphasize-lines: 1-5 - $ mlr --from data/small --opprint put -q ' + mlr --from data/small --opprint put -q ' @v[$a][$b]["sum"] += $x; @v[$a][$b]["count"] += 1; end{emit @*,"a","b"} @@ -420,7 +420,7 @@ Use **emit all** (or ``emit @*`` which is synonymous) to output all out-of-strea .. code-block:: none :emphasize-lines: 1-5 - $ mlr --from data/small --opprint put -q ' + mlr --from data/small --opprint put -q ' @sum[$a][$b] += $x; @count[$a][$b] += 1; end{emit @*,"a","b"} @@ -446,7 +446,7 @@ Use **emit all** (or ``emit @*`` which is synonymous) to output all out-of-strea .. code-block:: none :emphasize-lines: 1-5 - $ mlr --from data/small --opprint put -q ' + mlr --from data/small --opprint put -q ' @sum[$a][$b] += $x; @count[$a][$b] += 1; end{emit (@sum, @count),"a","b"} diff --git a/docs6/reference-dsl-syntax.rst b/docs6/reference-dsl-syntax.rst index 454c73d07..44e4b5dd7 100644 --- a/docs6/reference-dsl-syntax.rst +++ b/docs6/reference-dsl-syntax.rst @@ -12,7 +12,7 @@ Multiple expressions may be given, separated by semicolons, and each may refer t .. code-block:: none :emphasize-lines: 1-1 - $ ruby -e '10.times{|i|puts "i=#{i}"}' | mlr --opprint put '$j = $i + 1; $k = $i +$j' + ruby -e '10.times{|i|puts "i=#{i}"}' | mlr --opprint put '$j = $i + 1; $k = $i +$j' i j k 0 1 1 1 2 3 @@ -30,7 +30,7 @@ Newlines within the expression are ignored, which can help increase legibility o .. code-block:: none :emphasize-lines: 1-7 - $ mlr --opprint put ' + mlr --opprint put ' $nf = NF; $nr = NR; $fnr = FNR; @@ -52,7 +52,7 @@ Newlines within the expression are ignored, which can help increase legibility o .. code-block:: none :emphasize-lines: 1-3 - $ mlr --opprint filter '($x > 0.5 && $y < 0.5) || ($x < 0.5 && $y > 0.5)' \ + mlr --opprint filter '($x > 0.5 && $y < 0.5) || ($x < 0.5 && $y > 0.5)' \ then stats2 -a corr -f x,y \ data/medium x_y_corr @@ -68,7 +68,7 @@ The simplest way to enter expressions for ``put`` and ``filter`` is between sing .. code-block:: none :emphasize-lines: 1-1 - $ mlr --from data/small put '$xy = sqrt($x**2 + $y**2)' + mlr --from data/small put '$xy = sqrt($x**2 + $y**2)' a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533,xy=0.8052985815845617 a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797,xy=0.9209978658539777 a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776,xy=0.3953756915115773 @@ -78,7 +78,7 @@ The simplest way to enter expressions for ``put`` and ``filter`` is between sing .. code-block:: none :emphasize-lines: 1-1 - $ mlr --from data/small put 'func f(a, b) { return sqrt(a**2 + b**2) } $xy = f($x, $y)' + mlr --from data/small put 'func f(a, b) { return sqrt(a**2 + b**2) } $xy = f($x, $y)' a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533,xy=0.8052985815845617 a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797,xy=0.9209978658539777 a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776,xy=0.3953756915115773 @@ -91,7 +91,7 @@ You may, though, find it convenient to put expressions into files for reuse, and .. code-block:: none :emphasize-lines: 1-1 - $ cat data/fe-example-3.mlr + cat data/fe-example-3.mlr func f(a, b) { return sqrt(a**2 + b**2) } @@ -100,7 +100,7 @@ You may, though, find it convenient to put expressions into files for reuse, and .. code-block:: none :emphasize-lines: 1-1 - $ mlr --from data/small put -f data/fe-example-3.mlr + mlr --from data/small put -f data/fe-example-3.mlr a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533,xy=0.8052985815845617 a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797,xy=0.9209978658539777 a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776,xy=0.3953756915115773 @@ -112,7 +112,7 @@ If you have some of the logic in a file and you want to write the rest on the co .. code-block:: none :emphasize-lines: 1-1 - $ cat data/fe-example-4.mlr + cat data/fe-example-4.mlr func f(a, b) { return sqrt(a**2 + b**2) } @@ -120,7 +120,7 @@ If you have some of the logic in a file and you want to write the rest on the co .. code-block:: none :emphasize-lines: 1-1 - $ mlr --from data/small put -f data/fe-example-4.mlr -e '$xy = f($x, $y)' + mlr --from data/small put -f data/fe-example-4.mlr -e '$xy = f($x, $y)' a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533,xy=0.8052985815845617 a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797,xy=0.9209978658539777 a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776,xy=0.3953756915115773 @@ -150,13 +150,13 @@ Semicolons are optional after closing curly braces (which close conditionals and .. code-block:: none :emphasize-lines: 1-1 - $ echo x=1,y=2 | mlr put 'while (NF < 10) { $[NF+1] = ""} $foo = "bar"' + echo x=1,y=2 | mlr put 'while (NF < 10) { $[NF+1] = ""} $foo = "bar"' x=1,y=2,3=,4=,5=,6=,7=,8=,9=,10=,foo=bar .. code-block:: none :emphasize-lines: 1-1 - $ echo x=1,y=2 | mlr put 'while (NF < 10) { $[NF+1] = ""}; $foo = "bar"' + echo x=1,y=2 | mlr put 'while (NF < 10) { $[NF+1] = ""}; $foo = "bar"' x=1,y=2,3=,4=,5=,6=,7=,8=,9=,10=,foo=bar Semicolons are required between statements even if those statements are on separate lines. **Newlines** are for your convenience but have no syntactic meaning: line endings do not terminate statements. For example, adjacent assignment statements must be separated by semicolons even if those statements are on separate lines: @@ -178,7 +178,7 @@ Semicolons are required between statements even if those statements are on separ .. code-block:: none :emphasize-lines: 1-17 - $ mlr --csvlite --from data/a.csv put ' + mlr --csvlite --from data/a.csv put ' func f( num a, num b, diff --git a/docs6/reference-dsl-unset-statements.rst b/docs6/reference-dsl-unset-statements.rst index 3e24fe437..726474b56 100644 --- a/docs6/reference-dsl-unset-statements.rst +++ b/docs6/reference-dsl-unset-statements.rst @@ -9,7 +9,7 @@ You can clear a map key by assigning the empty string as its value: ``$x=""`` or .. code-block:: none :emphasize-lines: 1-1 - $ cat data/small + cat data/small a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533 a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797 a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776 @@ -19,7 +19,7 @@ You can clear a map key by assigning the empty string as its value: ``$x=""`` or .. code-block:: none :emphasize-lines: 1-1 - $ mlr put 'unset $x, $a' data/small + mlr put 'unset $x, $a' data/small b=pan,i=1,y=0.7268028627434533 b=pan,i=2,y=0.5221511083334797 b=wye,i=3,y=0.33831852551664776 @@ -31,7 +31,7 @@ This can also be done, of course, using ``mlr cut -x``. You can also clear out-o .. code-block:: none :emphasize-lines: 1-1 - $ mlr put -q '@sum[$a][$b] += $x; end { dump; unset @sum; dump }' data/small + mlr put -q '@sum[$a][$b] += $x; end { dump; unset @sum; dump }' data/small { "sum": { "pan": { @@ -52,7 +52,7 @@ This can also be done, of course, using ``mlr cut -x``. You can also clear out-o .. code-block:: none :emphasize-lines: 1-1 - $ mlr put -q '@sum[$a][$b] += $x; end { dump; unset @sum["eks"]; dump }' data/small + mlr put -q '@sum[$a][$b] += $x; end { dump; unset @sum["eks"]; dump }' data/small { "sum": { "pan": { diff --git a/docs6/reference-dsl-user-defined-functions.rst b/docs6/reference-dsl-user-defined-functions.rst index e107d2e31..52c37de4a 100644 --- a/docs6/reference-dsl-user-defined-functions.rst +++ b/docs6/reference-dsl-user-defined-functions.rst @@ -14,7 +14,7 @@ Here's the obligatory example of a recursive function to compute the factorial f .. code-block:: none :emphasize-lines: 1-14 - $ mlr --opprint --from data/small put ' + mlr --opprint --from data/small put ' func f(n) { if (is_numeric(n)) { if (n > 0) { @@ -63,7 +63,7 @@ Example: .. code-block:: none :emphasize-lines: 1-17 - $ mlr --opprint --from data/small put -q ' + mlr --opprint --from data/small put -q ' begin { @call_count = 0; } diff --git a/docs6/reference-dsl-variables.rst b/docs6/reference-dsl-variables.rst index 53cc5b248..7c354042c 100644 --- a/docs6/reference-dsl-variables.rst +++ b/docs6/reference-dsl-variables.rst @@ -26,7 +26,7 @@ Namely, Miller supports the following five built-in variables for :doc:`filter a .. code-block:: none :emphasize-lines: 1-1 - $ mlr filter 'FNR == 2' data/small* + mlr filter 'FNR == 2' data/small* a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797 1=pan,2=pan,3=1,4=0.3467901443380824,5=0.7268028627434533 a=wye,b=eks,i=10000,x=0.734806020620654365,y=0.884788571337605134 @@ -34,7 +34,7 @@ Namely, Miller supports the following five built-in variables for :doc:`filter a .. code-block:: none :emphasize-lines: 1-1 - $ mlr put '$fnr = FNR' data/small* + mlr put '$fnr = FNR' data/small* a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533,fnr=1 a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797,fnr=2 a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776,fnr=3 @@ -59,7 +59,7 @@ Their **scope is global**: you can refer to them in any ``filter`` or ``put`` st .. code-block:: none :emphasize-lines: 1-1 - $ mlr --csv put '$nr = NR' data/a.csv + mlr --csv put '$nr = NR' data/a.csv a,b,c,nr 1,2,3,1 4,5,6,2 @@ -67,7 +67,7 @@ Their **scope is global**: you can refer to them in any ``filter`` or ``put`` st .. code-block:: none :emphasize-lines: 1-1 - $ mlr --csv repeat -n 3 then put '$nr = NR' data/a.csv + mlr --csv repeat -n 3 then put '$nr = NR' data/a.csv a,b,c,nr 1,2,3,1 1,2,3,1 @@ -92,12 +92,12 @@ You may also use a **computed field name** in square brackets, e.g. .. code-block:: none :emphasize-lines: 1-1 - $ echo a=3,b=4 | mlr filter '$["x"] < 0.5' + echo a=3,b=4 | mlr filter '$["x"] < 0.5' .. code-block:: none :emphasize-lines: 1-1 - $ echo s=green,t=blue,a=3,b=4 | mlr put '$[$s."_".$t] = $a * $b' + echo s=green,t=blue,a=3,b=4 | mlr put '$[$s."_".$t] = $a * $b' s=green,t=blue,a=3,b=4,green_blue=12 Notes: @@ -122,7 +122,7 @@ Then using a computed field name, ``$[ $[[3]] ]`` is the value in the third fiel .. code-block:: none :emphasize-lines: 1-1 - $ mlr cat data/small + mlr cat data/small a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533 a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797 a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776 @@ -132,7 +132,7 @@ Then using a computed field name, ``$[ $[[3]] ]`` is the value in the third fiel .. code-block:: none :emphasize-lines: 1-1 - $ mlr put '$[[3]] = "NEW"' data/small + mlr put '$[[3]] = "NEW"' data/small a=pan,b=pan,NEW=1,x=0.3467901443380824,y=0.7268028627434533 a=eks,b=pan,NEW=2,x=0.7586799647899636,y=0.5221511083334797 a=wye,b=wye,NEW=3,x=0.20460330576630303,y=0.33831852551664776 @@ -142,7 +142,7 @@ Then using a computed field name, ``$[ $[[3]] ]`` is the value in the third fiel .. code-block:: none :emphasize-lines: 1-1 - $ mlr put '$[[[3]]] = "NEW"' data/small + mlr put '$[[[3]]] = "NEW"' data/small a=pan,b=pan,i=NEW,x=0.3467901443380824,y=0.7268028627434533 a=eks,b=pan,i=NEW,x=0.7586799647899636,y=0.5221511083334797 a=wye,b=wye,i=NEW,x=0.20460330576630303,y=0.33831852551664776 @@ -152,7 +152,7 @@ Then using a computed field name, ``$[ $[[3]] ]`` is the value in the third fiel .. code-block:: none :emphasize-lines: 1-1 - $ mlr put '$NEW = $[[NR]]' data/small + mlr put '$NEW = $[[NR]]' data/small a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533,NEW=a a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797,NEW=b a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776,NEW=i @@ -162,7 +162,7 @@ Then using a computed field name, ``$[ $[[3]] ]`` is the value in the third fiel .. code-block:: none :emphasize-lines: 1-1 - $ mlr put '$NEW = $[[[NR]]]' data/small + mlr put '$NEW = $[[[NR]]]' data/small a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533,NEW=pan a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797,NEW=pan a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776,NEW=3 @@ -172,7 +172,7 @@ Then using a computed field name, ``$[ $[[3]] ]`` is the value in the third fiel .. code-block:: none :emphasize-lines: 1-1 - $ mlr put '$[[[NR]]] = "NEW"' data/small + mlr put '$[[[NR]]] = "NEW"' data/small a=NEW,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533 a=eks,b=NEW,i=2,x=0.7586799647899636,y=0.5221511083334797 a=wye,b=wye,i=NEW,x=0.20460330576630303,y=0.33831852551664776 @@ -184,7 +184,7 @@ Right-hand side accesses to non-existent fields -- i.e. with index less than 1 o .. code-block:: none :emphasize-lines: 1-1 - $ mlr put '$[[6]] = "NEW"' data/small + mlr put '$[[6]] = "NEW"' data/small a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533 a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797 a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776 @@ -194,7 +194,7 @@ Right-hand side accesses to non-existent fields -- i.e. with index less than 1 o .. code-block:: none :emphasize-lines: 1-1 - $ mlr put '$[[[6]]] = "NEW"' data/small + mlr put '$[[[6]]] = "NEW"' data/small a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533 a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797 a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776 @@ -215,7 +215,7 @@ You may use a **computed key** in square brackets, e.g. .. code-block:: none :emphasize-lines: 1-1 - $ echo s=green,t=blue,a=3,b=4 | mlr put -q '@[$s."_".$t] = $a * $b; emit all' + echo s=green,t=blue,a=3,b=4 | mlr put -q '@[$s."_".$t] = $a * $b; emit all' green_blue=12 Out-of-stream variables are **scoped** to the ``put`` command in which they appear. In particular, if you have two or more ``put`` commands separated by ``then``, each put will have its own set of out-of-stream variables: @@ -223,14 +223,14 @@ Out-of-stream variables are **scoped** to the ``put`` command in which they appe .. code-block:: none :emphasize-lines: 1-1 - $ cat data/a.dkvp + cat data/a.dkvp a=1,b=2,c=3 a=4,b=5,c=6 .. code-block:: none :emphasize-lines: 1-3 - $ mlr put '@sum += $a; end {emit @sum}' \ + mlr put '@sum += $a; end {emit @sum}' \ then put 'is_present($a) {$a=10*$a; @sum += $a}; end {emit @sum}' \ data/a.dkvp a=10,b=2,c=3 @@ -250,7 +250,7 @@ Using an index on the ``@count`` and ``@sum`` variables, we get the benefit of t .. code-block:: none :emphasize-lines: 1-8 - $ mlr put -q ' + mlr put -q ' @x_count[$a] += 1; @x_sum[$a] += $x; end { @@ -272,7 +272,7 @@ Using an index on the ``@count`` and ``@sum`` variables, we get the benefit of t .. code-block:: none :emphasize-lines: 1-1 - $ mlr stats1 -a count,sum -f x -g a ../data/small + mlr stats1 -a count,sum -f x -g a ../data/small a=pan,x_count=2,x_sum=0.8494161498792961 a=eks,x_count=3,x_sum=1.75186341922895 a=wye,x_count=2,x_sum=0.7778922255683036 @@ -284,7 +284,7 @@ Indices can be arbitrarily deep -- here there are two or more of them: .. code-block:: none :emphasize-lines: 1-7 - $ mlr --from data/medium put -q ' + mlr --from data/medium put -q ' @x_count[$a][$b] += 1; @x_sum[$a][$b] += $x; end { @@ -324,7 +324,7 @@ Begin/end blocks can be mixed with pattern/action blocks. For example: .. code-block:: none :emphasize-lines: 1-14 - $ mlr put ' + mlr put ' begin { @num_total = 0; @num_positive = 0; @@ -357,7 +357,7 @@ For example: .. code-block:: none :emphasize-lines: 1-16 - $ # Here I'm using a specified random-number seed so this example always + # Here I'm using a specified random-number seed so this example always # produces the same output for this web document: in everyday practice we # would leave off the --seed 12345 part. mlr --seed 12345 seqgen --start 1 --stop 10 then put ' @@ -413,7 +413,7 @@ The following example demonstrates the scope rules: .. code-block:: none :emphasize-lines: 1-1 - $ cat data/scope-example.mlr + cat data/scope-example.mlr func f(a) { # argument is local to the function var b = 100; # local to the function c = 100; # local to the function; does not overwrite outer c @@ -441,7 +441,7 @@ The following example demonstrates the scope rules: .. code-block:: none :emphasize-lines: 1-1 - $ cat data/scope-example.dat + cat data/scope-example.dat n=1,x=123 n=2,x=456 n=3,x=789 @@ -449,7 +449,7 @@ The following example demonstrates the scope rules: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --oxtab --from data/scope-example.dat put -f data/scope-example.mlr + mlr --oxtab --from data/scope-example.dat put -f data/scope-example.mlr n 1 x 123 outer_a 10 @@ -477,7 +477,7 @@ And this example demonstrates the type-declaration rules: .. code-block:: none :emphasize-lines: 1-1 - $ cat data/type-decl-example.mlr + cat data/type-decl-example.mlr subr s(a, str b, int c) { # a is implicitly var (untyped). # b is explicitly str. # c is explicitly int. @@ -526,7 +526,7 @@ For example, the following swaps the input stream's ``a`` and ``i`` fields, modi .. code-block:: none :emphasize-lines: 1-7 - $ mlr --opprint put ' + mlr --opprint put ' $* = { "a": $i, "i": $a, @@ -545,7 +545,7 @@ Likewise, you can assign map literals to out-of-stream variables or local variab .. code-block:: none :emphasize-lines: 1-7 - $ mlr --from data/small put ' + mlr --from data/small put ' func f(map m): map { m["x"] *= 200; return m; @@ -563,7 +563,7 @@ Like out-of-stream and local variables, map literals can be multi-level: .. code-block:: none :emphasize-lines: 1-19 - $ mlr --from data/small put -q ' + mlr --from data/small put -q ' begin { @o = { "nrec": 0, @@ -611,7 +611,7 @@ The following ``is...`` functions take a value and return a boolean indicating w .. code-block:: none :emphasize-lines: 1-1 - $ mlr -F | grep ^is + mlr -F | grep ^is is_absent is_array is_bool @@ -635,7 +635,7 @@ The following ``is...`` functions take a value and return a boolean indicating w .. code-block:: none :emphasize-lines: 1-1 - $ mlr -F | grep ^assert + mlr -F | grep ^assert asserting_absent asserting_array asserting_bool @@ -749,7 +749,7 @@ Example recursive copy of out-of-stream variables: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint put -q '@v["sum"] += $x; @v["count"] += 1; end{dump; @w = @v; dump}' data/small + mlr --opprint put -q '@v["sum"] += $x; @v["count"] += 1; end{dump; @w = @v; dump}' data/small { "v": { "sum": 2.264761728567491, @@ -772,7 +772,7 @@ Example of out-of-stream variable assigned to full stream record, where the 2nd .. code-block:: none :emphasize-lines: 1-1 - $ mlr put 'NR == 2 {@keep = $*}; NR == 4 {$* = @keep}' data/small + mlr put 'NR == 2 {@keep = $*}; NR == 4 {$* = @keep}' data/small a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533 a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797 a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776 @@ -784,7 +784,7 @@ Example of full stream record assigned to an out-of-stream variable, finding the .. code-block:: none :emphasize-lines: 1-1 - $ cat data/small + cat data/small a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533 a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797 a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776 @@ -794,7 +794,7 @@ Example of full stream record assigned to an out-of-stream variable, finding the .. code-block:: none :emphasize-lines: 1-4 - $ mlr --opprint put -q ' + mlr --opprint put -q ' is_null(@xmax) || $x > @xmax {@xmax=$x; @recmax=$*}; end {emit @recmax} ' data/small @@ -807,6 +807,6 @@ Keywords for filter and put .. code-block:: none :emphasize-lines: 1-1 - $ mlr --help-all-keywords + mlr --help-all-keywords TODO: port mlr_dsl_list_all_keywords diff --git a/docs6/reference-dsl.rst b/docs6/reference-dsl.rst index 89b5ed799..3343eb5f0 100644 --- a/docs6/reference-dsl.rst +++ b/docs6/reference-dsl.rst @@ -14,7 +14,7 @@ Example: .. code-block:: none :emphasize-lines: 1-1 - $ mlr stats1 -a sum -f x -g a data/small + mlr stats1 -a sum -f x -g a data/small a=pan,x_sum=0.3467901443380824 a=eks,x_sum=1.1400793586611044 a=wye,x_sum=0.7778922255683036 @@ -30,7 +30,7 @@ Example: .. code-block:: none :emphasize-lines: 1-1 - $ mlr put -q '@x_sum[$a] += $x; end{emit @x_sum, "a"}' data/small + mlr put -q '@x_sum[$a] += $x; end{emit @x_sum, "a"}' data/small a=pan,x_sum=0.3467901443380824 a=eks,x_sum=1.1400793586611044 a=wye,x_sum=0.7778922255683036 @@ -48,7 +48,7 @@ The essential usages of ``mlr filter`` and ``mlr put`` are for record-selection .. code-block:: none :emphasize-lines: 1-1 - $ cat data/small + cat data/small a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533 a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797 a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776 @@ -60,7 +60,7 @@ you might retain only the records whose ``a`` field has value ``eks``: .. code-block:: none :emphasize-lines: 1-1 - $ mlr filter '$a == "eks"' data/small + mlr filter '$a == "eks"' data/small a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797 a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463 @@ -69,7 +69,7 @@ or you might add a new field which is a function of existing fields: .. code-block:: none :emphasize-lines: 1-1 - $ mlr put '$ab = $a . "_" . $b ' data/small + mlr put '$ab = $a . "_" . $b ' data/small a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533,ab=pan_pan a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797,ab=eks_pan a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776,ab=wye_wye diff --git a/docs6/reference-main-auxiliary-commands.rst b/docs6/reference-main-auxiliary-commands.rst index a28262544..452578756 100644 --- a/docs6/reference-main-auxiliary-commands.rst +++ b/docs6/reference-main-auxiliary-commands.rst @@ -9,7 +9,7 @@ There are a few nearly-standalone programs which have nothing to do with the res .. code-block:: none :emphasize-lines: 1-1 - $ mlr aux-list + mlr aux-list Available subcommands: aux-list hex @@ -23,7 +23,7 @@ There are a few nearly-standalone programs which have nothing to do with the res .. code-block:: none :emphasize-lines: 1-1 - $ mlr lecat --help + mlr lecat --help Usage: mlr lecat [options] {zero or more file names} Simple hex-dump. If zero file names are supplied, standard input is read. @@ -34,7 +34,7 @@ There are a few nearly-standalone programs which have nothing to do with the res .. code-block:: none :emphasize-lines: 1-1 - $ mlr termcvt --help + mlr termcvt --help Usage: mlr termcvt [option] {zero or more file names} Option (exactly one is required): --cr2crlf @@ -51,7 +51,7 @@ There are a few nearly-standalone programs which have nothing to do with the res .. code-block:: none :emphasize-lines: 1-1 - $ mlr hex --help + mlr hex --help Usage: mlr hex [options] {zero or more file names} Simple hex-dump. If zero file names are supplied, standard input is read. @@ -62,7 +62,7 @@ There are a few nearly-standalone programs which have nothing to do with the res .. code-block:: none :emphasize-lines: 1-1 - $ mlr unhex --help + mlr unhex --help Usage: mlr unhex [options] {zero or more file names} Simple hex-dump. If zero file names are supplied, standard input is read. @@ -75,19 +75,19 @@ Examples: .. code-block:: none :emphasize-lines: 1-1 - $ echo 'Hello, world!' | mlr lecat --mono + echo 'Hello, world!' | mlr lecat --mono Hello, world![LF] .. code-block:: none :emphasize-lines: 1-1 - $ echo 'Hello, world!' | mlr termcvt --lf2crlf | mlr lecat --mono + echo 'Hello, world!' | mlr termcvt --lf2crlf | mlr lecat --mono Hello, world![CR][LF] .. code-block:: none :emphasize-lines: 1-1 - $ mlr hex data/budget.csv + mlr hex data/budget.csv 00000000: 23 20 41 73 61 6e 61 20 2d 2d 20 68 65 72 65 20 |# Asana -- here | 00000010: 61 72 65 20 74 68 65 20 62 75 64 67 65 74 20 66 |are the budget f| 00000020: 69 67 75 72 65 73 20 79 6f 75 20 61 73 6b 65 64 |igures you asked| @@ -99,7 +99,7 @@ Examples: .. code-block:: none :emphasize-lines: 1-1 - $ mlr hex -r data/budget.csv + mlr hex -r data/budget.csv 23 20 41 73 61 6e 61 20 2d 2d 20 68 65 72 65 20 61 72 65 20 74 68 65 20 62 75 64 67 65 74 20 66 69 67 75 72 65 73 20 79 6f 75 20 61 73 6b 65 64 @@ -111,7 +111,7 @@ Examples: .. code-block:: none :emphasize-lines: 1-1 - $ mlr hex -r data/budget.csv | sed 's/20/2a/g' | mlr unhex + mlr hex -r data/budget.csv | sed 's/20/2a/g' | mlr unhex #*Asana*--*here*are*the*budget*figures*you*asked*for! type,quantity purple,456.78 diff --git a/docs6/reference-main-io-options.rst b/docs6/reference-main-io-options.rst index 889eeddbf..c5d58d87b 100644 --- a/docs6/reference-main-io-options.rst +++ b/docs6/reference-main-io-options.rst @@ -24,7 +24,7 @@ These are as discussed in :doc:`file-formats`, with the exception of ``--right`` .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint cat data/small + mlr --opprint cat data/small a b i x y pan pan 1 0.3467901443380824 0.7268028627434533 eks pan 2 0.7586799647899636 0.5221511083334797 @@ -35,7 +35,7 @@ These are as discussed in :doc:`file-formats`, with the exception of ``--right`` .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint --right cat data/small + mlr --opprint --right cat data/small a b i x y pan pan 1 0.3467901443380824 0.7268028627434533 eks pan 2 0.7586799647899636 0.5221511083334797 @@ -138,13 +138,13 @@ To apply formatting to a single field, overriding the global ``ofmt``, use ``fmt .. code-block:: none :emphasize-lines: 1-1 - $ echo 'x=3.1,y=4.3' | mlr put '$z=fmtnum($x*$y,"%08lf")' + echo 'x=3.1,y=4.3' | mlr put '$z=fmtnum($x*$y,"%08lf")' x=3.1,y=4.3,z=%!l(float64=00013.33)f .. code-block:: none :emphasize-lines: 1-1 - $ echo 'x=0xffff,y=0xff' | mlr put '$z=fmtnum(int($x*$y),"%08llx")' + echo 'x=0xffff,y=0xff' | mlr put '$z=fmtnum(int($x*$y),"%08llx")' x=0xffff,y=0xff,z=%!l(int=16711425)lx Input conversion from hexadecimal is done automatically on fields handled by ``mlr put`` and ``mlr filter`` as long as the field value begins with "0x". To apply output conversion to hexadecimal on a single column, you may use ``fmtnum``, or the keystroke-saving ``hexfmt`` function. Example: @@ -152,5 +152,5 @@ Input conversion from hexadecimal is done automatically on fields handled by ``m .. code-block:: none :emphasize-lines: 1-1 - $ echo 'x=0xffff,y=0xff' | mlr put '$z=hexfmt($x*$y)' + echo 'x=0xffff,y=0xff' | mlr put '$z=hexfmt($x*$y)' x=0xffff,y=0xff,z=0xfeff01 diff --git a/docs6/reference-main-null-data.rst b/docs6/reference-main-null-data.rst index f33329556..223266668 100644 --- a/docs6/reference-main-null-data.rst +++ b/docs6/reference-main-null-data.rst @@ -21,7 +21,7 @@ Rules for null-handling: .. code-block:: none :emphasize-lines: 1-1 - $ mlr cat data/sort-null.dat + mlr cat data/sort-null.dat a=3,b=2 a=1,b=8 a=,b=4 @@ -31,7 +31,7 @@ Rules for null-handling: .. code-block:: none :emphasize-lines: 1-1 - $ mlr sort -n a data/sort-null.dat + mlr sort -n a data/sort-null.dat a=1,b=8 a=3,b=2 a=5,b=7 @@ -41,7 +41,7 @@ Rules for null-handling: .. code-block:: none :emphasize-lines: 1-1 - $ mlr sort -nr a data/sort-null.dat + mlr sort -nr a data/sort-null.dat a=,b=4 a=5,b=7 a=3,b=2 @@ -53,19 +53,19 @@ Rules for null-handling: .. code-block:: none :emphasize-lines: 1-1 - $ echo 'x=2,y=3' | mlr put '$a=$x+$y' + echo 'x=2,y=3' | mlr put '$a=$x+$y' x=2,y=3,a=5 .. code-block:: none :emphasize-lines: 1-1 - $ echo 'x=,y=3' | mlr put '$a=$x+$y' + echo 'x=,y=3' | mlr put '$a=$x+$y' x=,y=3,a= .. code-block:: none :emphasize-lines: 1-1 - $ echo 'x=,y=3' | mlr put '$a=log($x);$b=log($y)' + echo 'x=,y=3' | mlr put '$a=log($x);$b=log($y)' x=,y=3,a=,b=1.0986122886681096 with the exception that the ``min`` and ``max`` functions are special: if one argument is non-null, it wins: @@ -73,7 +73,7 @@ with the exception that the ``min`` and ``max`` functions are special: if one ar .. code-block:: none :emphasize-lines: 1-1 - $ echo 'x=,y=3' | mlr put '$a=min($x,$y);$b=max($x,$y)' + echo 'x=,y=3' | mlr put '$a=min($x,$y);$b=max($x,$y)' x=,y=3,a=3,b= * Functions of *absent* variables (e.g. ``mlr put '$y = log10($nonesuch)'``) evaluate to absent, and arithmetic/bitwise/boolean operators with both operands being absent evaluate to absent. Arithmetic operators with one absent operand return the other operand. More specifically, absent values act like zero for addition/subtraction, and one for multiplication: Furthermore, **any expression which evaluates to absent is not stored in the left-hand side of an assignment statement**: @@ -81,13 +81,13 @@ with the exception that the ``min`` and ``max`` functions are special: if one ar .. code-block:: none :emphasize-lines: 1-1 - $ echo 'x=2,y=3' | mlr put '$a=$u+$v; $b=$u+$y; $c=$x+$y' + echo 'x=2,y=3' | mlr put '$a=$u+$v; $b=$u+$y; $c=$x+$y' x=2,y=3,b=3,c=5 .. code-block:: none :emphasize-lines: 1-1 - $ echo 'x=2,y=3' | mlr put '$a=min($x,$v);$b=max($u,$y);$c=min($u,$v)' + echo 'x=2,y=3' | mlr put '$a=min($x,$v);$b=max($u,$y);$c=min($u,$v)' x=2,y=3,a=2,b=3 * Likewise, for assignment to maps, **absent-valued keys or values result in a skipped assignment**. @@ -107,7 +107,7 @@ Since absent plus absent is absent (and likewise for other operators), accumulat .. code-block:: none :emphasize-lines: 1-1 - $ mlr cat data/het.dkvp + mlr cat data/het.dkvp resource=/path/to/file,loadsec=0.45,ok=true record_count=100,resource=/path/to/file resource=/path/to/second/file,loadsec=0.32,ok=true @@ -117,7 +117,7 @@ Since absent plus absent is absent (and likewise for other operators), accumulat .. code-block:: none :emphasize-lines: 1-1 - $ mlr put 'is_present($loadsec) { $loadmillis = $loadsec * 1000 }' data/het.dkvp + mlr put 'is_present($loadsec) { $loadmillis = $loadsec * 1000 }' data/het.dkvp resource=/path/to/file,loadsec=0.45,ok=true,loadmillis=450 record_count=100,resource=/path/to/file resource=/path/to/second/file,loadsec=0.32,ok=true,loadmillis=320 @@ -127,7 +127,7 @@ Since absent plus absent is absent (and likewise for other operators), accumulat .. code-block:: none :emphasize-lines: 1-1 - $ mlr put '$loadmillis = (is_present($loadsec) ? $loadsec : 0.0) * 1000' data/het.dkvp + mlr put '$loadmillis = (is_present($loadsec) ? $loadsec : 0.0) * 1000' data/het.dkvp resource=/path/to/file,loadsec=0.45,ok=true,loadmillis=450 record_count=100,resource=/path/to/file,loadmillis=0 resource=/path/to/second/file,loadsec=0.32,ok=true,loadmillis=320 @@ -139,5 +139,5 @@ If you're interested in a formal description of how empty and absent fields part .. code-block:: none :emphasize-lines: 1-1 - $ mlr --print-type-arithmetic-info + mlr --print-type-arithmetic-info TODO: port printTypeArithmeticInfo diff --git a/docs6/reference-main-online-help.rst b/docs6/reference-main-online-help.rst index a80d76ed9..7fce44645 100644 --- a/docs6/reference-main-online-help.rst +++ b/docs6/reference-main-online-help.rst @@ -11,7 +11,7 @@ Examples: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --help + mlr --help Usage: mlr [I/O options] {verb} [verb-dependent options ...] {zero or more file names} COMMAND-LINE-SYNTAX EXAMPLES: @@ -585,7 +585,7 @@ Examples: .. code-block:: none :emphasize-lines: 1-1 - $ mlr sort --help + mlr sort --help Usage: mlr sort {flags} Sorts records primarily by the first specified field, secondarily by the second field, and so on. (Any records not having all specified sort keys will appear diff --git a/docs6/reference-main-overview.rst b/docs6/reference-main-overview.rst index 8ba62ff1d..4573d1cf5 100644 --- a/docs6/reference-main-overview.rst +++ b/docs6/reference-main-overview.rst @@ -19,7 +19,7 @@ For example, reading from a file: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsv --opprint head -n 2 then sort -f shape example.csv + mlr --icsv --opprint head -n 2 then sort -f shape example.csv color shape flag index quantity rate red square true 15 79.2778 0.0130 yellow triangle true 11 43.6498 9.8870 @@ -29,7 +29,7 @@ Reading from standard input: .. code-block:: none :emphasize-lines: 1-1 - $ cat example.csv | mlr --icsv --opprint head -n 2 then sort -f shape + cat example.csv | mlr --icsv --opprint head -n 2 then sort -f shape color shape flag index quantity rate red square true 15 79.2778 0.0130 yellow triangle true 11 43.6498 9.8870 @@ -50,7 +50,7 @@ Example of using a verb for data processing: .. code-block:: none :emphasize-lines: 1-1 - $ mlr stats1 -a sum -f x -g a data/small + mlr stats1 -a sum -f x -g a data/small a=pan,x_sum=0.3467901443380824 a=eks,x_sum=1.1400793586611044 a=wye,x_sum=0.7778922255683036 @@ -66,7 +66,7 @@ Example of doing the same thing using a DSL expression: .. code-block:: none :emphasize-lines: 1-1 - $ mlr put -q '@x_sum[$a] += $x; end{emit @x_sum, "a"}' data/small + mlr put -q '@x_sum[$a] += $x; end{emit @x_sum, "a"}' data/small a=pan,x_sum=0.3467901443380824 a=eks,x_sum=1.1400793586611044 a=wye,x_sum=0.7778922255683036 diff --git a/docs6/reference-main-regular-expressions.rst b/docs6/reference-main-regular-expressions.rst index a35360c51..a562c4b0a 100644 --- a/docs6/reference-main-regular-expressions.rst +++ b/docs6/reference-main-regular-expressions.rst @@ -35,7 +35,7 @@ Example: .. code-block:: none :emphasize-lines: 1-1 - $ cat data/regex-in-data.dat + cat data/regex-in-data.dat name=jane,regex=^j.*e$ name=bill,regex=^b[ou]ll$ name=bull,regex=^b[ou]ll$ @@ -43,7 +43,7 @@ Example: .. code-block:: none :emphasize-lines: 1-1 - $ mlr filter '$name =~ $regex' data/regex-in-data.dat + mlr filter '$name =~ $regex' data/regex-in-data.dat name=jane,regex=^j.*e$ name=bull,regex=^b[ou]ll$ diff --git a/docs6/reference-verbs.rst b/docs6/reference-verbs.rst index 45281a57a..a44f57b1a 100644 --- a/docs6/reference-verbs.rst +++ b/docs6/reference-verbs.rst @@ -38,7 +38,7 @@ Map list of values to alternating key/value pairs. .. code-block:: none :emphasize-lines: 1-1 - $ mlr altkv -h + mlr altkv -h Usage: mlr altkv [options] Given fields with values of the form a,b,c,d,e,f emits a=b,c=d,e=f pairs. Options: @@ -47,13 +47,13 @@ Map list of values to alternating key/value pairs. .. code-block:: none :emphasize-lines: 1-1 - $ echo 'a,b,c,d,e,f' | mlr altkv + echo 'a,b,c,d,e,f' | mlr altkv a=b,c=d,e=f .. code-block:: none :emphasize-lines: 1-1 - $ echo 'a,b,c,d,e,f,g' | mlr altkv + echo 'a,b,c,d,e,f,g' | mlr altkv a=b,c=d,e=f,4=g .. _reference-verbs-bar: @@ -66,7 +66,7 @@ Cheesy bar-charting. .. code-block:: none :emphasize-lines: 1-1 - $ mlr bar -h + mlr bar -h Usage: mlr bar [options] Replaces a numeric field with a number of asterisks, allowing for cheesy bar plots. These align best with --opprint or --oxtab output format. @@ -87,7 +87,7 @@ Cheesy bar-charting. .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint cat data/small + mlr --opprint cat data/small a b i x y pan pan 1 0.3467901443380824 0.7268028627434533 eks pan 2 0.7586799647899636 0.5221511083334797 @@ -98,7 +98,7 @@ Cheesy bar-charting. .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint bar --lo 0 --hi 1 -f x,y data/small + mlr --opprint bar --lo 0 --hi 1 -f x,y data/small a b i x y pan pan 1 *************........................... *****************************........... eks pan 2 ******************************.......... ********************.................... @@ -109,7 +109,7 @@ Cheesy bar-charting. .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint bar --lo 0.4 --hi 0.6 -f x,y data/small + mlr --opprint bar --lo 0.4 --hi 0.6 -f x,y data/small a b i x y pan pan 1 #....................................... ***************************************# eks pan 2 ***************************************# ************************................ @@ -120,7 +120,7 @@ Cheesy bar-charting. .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint bar --auto -f x,y data/small + mlr --opprint bar --auto -f x,y data/small a b i x y pan pan 1 [0.20460330576630303]**********..............................[0.7586799647899636] [0.13418874328430463]********************************........[0.8636244699032729] eks pan 2 [0.20460330576630303]***************************************#[0.7586799647899636] [0.13418874328430463]*********************...................[0.8636244699032729] @@ -136,7 +136,7 @@ bootstrap .. code-block:: none :emphasize-lines: 1-1 - $ mlr bootstrap --help + mlr bootstrap --help Usage: mlr bootstrap [options] Emits an n-sample, with replacement, of the input records. See also mlr sample and mlr shuffle. @@ -205,7 +205,7 @@ Most useful for format conversions (see :doc:`file-formats`, and concatenating m .. code-block:: none :emphasize-lines: 1-1 - $ mlr cat -h + mlr cat -h Usage: mlr cat [options] Passes input records directly to output. Most useful for format conversion. Options: @@ -217,7 +217,7 @@ Most useful for format conversions (see :doc:`file-formats`, and concatenating m .. code-block:: none :emphasize-lines: 1-1 - $ cat data/a.csv + cat data/a.csv a,b,c 1,2,3 4,5,6 @@ -225,14 +225,14 @@ Most useful for format conversions (see :doc:`file-formats`, and concatenating m .. code-block:: none :emphasize-lines: 1-1 - $ cat data/b.csv + cat data/b.csv a,b,c 7,8,9 .. code-block:: none :emphasize-lines: 1-1 - $ mlr --csv cat data/a.csv data/b.csv + mlr --csv cat data/a.csv data/b.csv a,b,c 1,2,3 4,5,6 @@ -241,7 +241,7 @@ Most useful for format conversions (see :doc:`file-formats`, and concatenating m .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsv --oxtab cat data/a.csv data/b.csv + mlr --icsv --oxtab cat data/a.csv data/b.csv a 1 b 2 c 3 @@ -257,7 +257,7 @@ Most useful for format conversions (see :doc:`file-formats`, and concatenating m .. code-block:: none :emphasize-lines: 1-1 - $ mlr --csv cat -n data/a.csv data/b.csv + mlr --csv cat -n data/a.csv data/b.csv n,a,b,c 1,1,2,3 2,4,5,6 @@ -266,7 +266,7 @@ Most useful for format conversions (see :doc:`file-formats`, and concatenating m .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint cat data/small + mlr --opprint cat data/small a b i x y pan pan 1 0.3467901443380824 0.7268028627434533 eks pan 2 0.7586799647899636 0.5221511083334797 @@ -277,7 +277,7 @@ Most useful for format conversions (see :doc:`file-formats`, and concatenating m .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint cat -n -g a data/small + mlr --opprint cat -n -g a data/small n a b i x y 1 pan pan 1 0.3467901443380824 0.7268028627434533 1 eks pan 2 0.7586799647899636 0.5221511083334797 @@ -293,7 +293,7 @@ check .. code-block:: none :emphasize-lines: 1-1 - $ mlr check --help + mlr check --help Usage: mlr check [options] Consumes records without printing any output. Useful for doing a well-formatted check on input data. @@ -308,7 +308,7 @@ clean-whitespace .. code-block:: none :emphasize-lines: 1-1 - $ mlr clean-whitespace --help + mlr clean-whitespace --help Usage: mlr clean-whitespace [options] For each record, for each field in the record, whitespace-cleans the keys and/or values. Whitespace-cleaning entails stripping leading and trailing whitespace, @@ -326,7 +326,7 @@ clean-whitespace .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsv --ojson cat data/clean-whitespace.csv + mlr --icsv --ojson cat data/clean-whitespace.csv { " Name ": " Ann Simons", " Preference ": " blue " @@ -343,7 +343,7 @@ clean-whitespace .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsv --ojson clean-whitespace -k data/clean-whitespace.csv + mlr --icsv --ojson clean-whitespace -k data/clean-whitespace.csv { "Name": " Ann Simons", "Preference": " blue " @@ -360,7 +360,7 @@ clean-whitespace .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsv --ojson clean-whitespace -v data/clean-whitespace.csv + mlr --icsv --ojson clean-whitespace -v data/clean-whitespace.csv { " Name ": "Ann Simons", " Preference ": "blue" @@ -377,7 +377,7 @@ clean-whitespace .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsv --ojson clean-whitespace data/clean-whitespace.csv + mlr --icsv --ojson clean-whitespace data/clean-whitespace.csv { "Name": "Ann Simons", "Preference": "blue" @@ -407,7 +407,7 @@ count .. code-block:: none :emphasize-lines: 1-1 - $ mlr count --help + mlr count --help Usage: mlr count [options] Prints number of records, optionally grouped by distinct values for specified field names. Options: @@ -419,13 +419,13 @@ count .. code-block:: none :emphasize-lines: 1-1 - $ mlr count data/medium + mlr count data/medium count=10000 .. code-block:: none :emphasize-lines: 1-1 - $ mlr count -g a data/medium + mlr count -g a data/medium a=pan,count=2081 a=eks,count=1965 a=wye,count=1966 @@ -435,13 +435,13 @@ count .. code-block:: none :emphasize-lines: 1-1 - $ mlr count -n -g a data/medium + mlr count -n -g a data/medium count=5 .. code-block:: none :emphasize-lines: 1-1 - $ mlr count -g b data/medium + mlr count -g b data/medium b=pan,count=1942 b=wye,count=2057 b=zee,count=1943 @@ -451,13 +451,13 @@ count .. code-block:: none :emphasize-lines: 1-1 - $ mlr count -n -g b data/medium + mlr count -n -g b data/medium count=5 .. code-block:: none :emphasize-lines: 1-1 - $ mlr count -g a,b data/medium + mlr count -g a,b data/medium a=pan,b=pan,count=427 a=eks,b=pan,count=371 a=wye,b=wye,count=377 @@ -492,7 +492,7 @@ count-distinct .. code-block:: none :emphasize-lines: 1-1 - $ mlr count-distinct --help + mlr count-distinct --help Usage: mlr count-distinct [options] Prints number of records having distinct values for specified field names. Same as uniq -c. @@ -511,7 +511,7 @@ count-distinct .. code-block:: none :emphasize-lines: 1-1 - $ mlr count-distinct -f a,b then sort -nr count data/medium + mlr count-distinct -f a,b then sort -nr count data/medium a=zee,b=wye,count=455 a=pan,b=eks,count=429 a=pan,b=pan,count=427 @@ -541,7 +541,7 @@ count-distinct .. code-block:: none :emphasize-lines: 1-1 - $ mlr count-distinct -u -f a,b data/medium + mlr count-distinct -u -f a,b data/medium field=a,value=pan,count=2081 field=a,value=eks,count=1965 field=a,value=wye,count=1966 @@ -556,7 +556,7 @@ count-distinct .. code-block:: none :emphasize-lines: 1-1 - $ mlr count-distinct -f a,b -o someothername then sort -nr someothername data/medium + mlr count-distinct -f a,b -o someothername then sort -nr someothername data/medium a=zee,b=wye,someothername=455 a=pan,b=eks,someothername=429 a=pan,b=pan,someothername=427 @@ -586,7 +586,7 @@ count-distinct .. code-block:: none :emphasize-lines: 1-1 - $ mlr count-distinct -n -f a,b data/medium + mlr count-distinct -n -f a,b data/medium count=25 .. _reference-verbs-count-similar: @@ -597,7 +597,7 @@ count-similar .. code-block:: none :emphasize-lines: 1-1 - $ mlr count-similar --help + mlr count-similar --help Usage: mlr count-similar [options] Ingests all records, then emits each record augmented by a count of the number of other records having the same group-by field values. @@ -609,7 +609,7 @@ count-similar .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint head -n 20 data/medium + mlr --opprint head -n 20 data/medium a b i x y pan pan 1 0.3467901443380824 0.7268028627434533 eks pan 2 0.7586799647899636 0.5221511083334797 @@ -635,7 +635,7 @@ count-similar .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint head -n 20 then count-similar -g a data/medium + mlr --opprint head -n 20 then count-similar -g a data/medium a b i x y count pan pan 1 0.3467901443380824 0.7268028627434533 4 pan wye 10 0.5026260055412137 0.9526183602969864 4 @@ -661,7 +661,7 @@ count-similar .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint head -n 20 then count-similar -g a then sort -f a data/medium + mlr --opprint head -n 20 then count-similar -g a then sort -f a data/medium a b i x y count eks pan 2 0.7586799647899636 0.5221511083334797 7 eks wye 4 0.38139939387114097 0.13418874328430463 7 @@ -692,7 +692,7 @@ cut .. code-block:: none :emphasize-lines: 1-1 - $ mlr cut --help + mlr cut --help Usage: mlr cut [options] Passes through input records with specified fields included/excluded. Options: @@ -715,7 +715,7 @@ cut .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint cat data/small + mlr --opprint cat data/small a b i x y pan pan 1 0.3467901443380824 0.7268028627434533 eks pan 2 0.7586799647899636 0.5221511083334797 @@ -726,7 +726,7 @@ cut .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint cut -f y,x,i data/small + mlr --opprint cut -f y,x,i data/small i x y 1 0.3467901443380824 0.7268028627434533 2 0.7586799647899636 0.5221511083334797 @@ -737,13 +737,13 @@ cut .. code-block:: none :emphasize-lines: 1-1 - $ echo 'a=1,b=2,c=3' | mlr cut -f b,c,a + echo 'a=1,b=2,c=3' | mlr cut -f b,c,a a=1,b=2,c=3 .. code-block:: none :emphasize-lines: 1-1 - $ echo 'a=1,b=2,c=3' | mlr cut -o -f b,c,a + echo 'a=1,b=2,c=3' | mlr cut -o -f b,c,a b=2,c=3,a=1 .. _reference-verbs-decimate: @@ -754,7 +754,7 @@ decimate .. code-block:: none :emphasize-lines: 1-1 - $ mlr decimate --help + mlr decimate --help Usage: mlr decimate [options] Passes through one of every n records, optionally by category. Options: @@ -772,7 +772,7 @@ fill-down .. code-block:: none :emphasize-lines: 1-1 - $ mlr fill-down --help + mlr fill-down --help Usage: mlr fill-down [options] If a given record has a missing value for a given field, fill that from the corresponding value from a previous record, if any. @@ -791,7 +791,7 @@ fill-down .. code-block:: none :emphasize-lines: 1-1 - $ cat data/fill-down.csv + cat data/fill-down.csv a,b,c 1,,3 4,5,6 @@ -800,7 +800,7 @@ fill-down .. code-block:: none :emphasize-lines: 1-1 - $ mlr --csv fill-down -f b data/fill-down.csv + mlr --csv fill-down -f b data/fill-down.csv a,b,c 1,,3 4,5,6 @@ -809,7 +809,7 @@ fill-down .. code-block:: none :emphasize-lines: 1-1 - $ mlr --csv fill-down -a -f b data/fill-down.csv + mlr --csv fill-down -a -f b data/fill-down.csv a,b,c 1,,3 4,5,6 @@ -823,7 +823,7 @@ filter .. code-block:: none :emphasize-lines: 1-1 - $ mlr filter --help + mlr filter --help Usage: mlr put [options] {DSL expression} Options: -f {file name} File containing a DSL expression. If the filename is a directory, @@ -890,7 +890,7 @@ format-values .. code-block:: none :emphasize-lines: 1-1 - $ mlr format-values --help + mlr format-values --help Usage: mlr format-values [options] Applies format strings to all field values, depending on autodetected type. * If a field value is detected to be integer, applies integer format. @@ -924,7 +924,7 @@ format-values .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint format-values data/small + mlr --opprint format-values data/small a b i x y pan pan 1 0.346790 0.726803 eks pan 2 0.758680 0.522151 @@ -935,7 +935,7 @@ format-values .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint format-values -n data/small + mlr --opprint format-values -n data/small a b i x y pan pan 1.000000 0.346790 0.726803 eks pan 2.000000 0.758680 0.522151 @@ -946,7 +946,7 @@ format-values .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint format-values -i %08llx -f %.6le -s X%sX data/small + mlr --opprint format-values -i %08llx -f %.6le -s X%sX data/small a b i x y XpanX XpanX %!l(int=00000001)lx %!l(float64=0.34679)e %!l(float64=0.726803)e XeksX XpanX %!l(int=00000002)lx %!l(float64=0.75868)e %!l(float64=0.522151)e @@ -957,7 +957,7 @@ format-values .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint format-values -i %08llx -f %.6le -s X%sX -n data/small + mlr --opprint format-values -i %08llx -f %.6le -s X%sX -n data/small a b i x y XpanX XpanX %!l(float64=1)e %!l(float64=0.34679)e %!l(float64=0.726803)e XeksX XpanX %!l(float64=2)e %!l(float64=0.75868)e %!l(float64=0.522151)e @@ -973,7 +973,7 @@ fraction .. code-block:: none :emphasize-lines: 1-1 - $ mlr fraction --help + mlr fraction --help Usage: mlr fraction [options] For each record's value in specified fields, computes the ratio of that value to the sum of values in that field over all input records. @@ -1017,7 +1017,7 @@ Then we can see what each record's ``n`` contributes to the total ``n``: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint fraction -f n data/fraction-example.csv + mlr --opprint fraction -f n data/fraction-example.csv u v n n_fraction female red 2458 0.32638427831629263 female green 192 0.025494622228123754 @@ -1037,7 +1037,7 @@ Using ``-g`` we can split those out by gender, or by color: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint fraction -f n -g u data/fraction-example.csv + mlr --opprint fraction -f n -g u data/fraction-example.csv u v n n_fraction female red 2458 0.7073381294964028 female green 192 0.05525179856115108 @@ -1055,7 +1055,7 @@ Using ``-g`` we can split those out by gender, or by color: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint fraction -f n -g v data/fraction-example.csv + mlr --opprint fraction -f n -g v data/fraction-example.csv u v n n_fraction female red 2458 0.9450211457131872 female green 192 0.45823389021479716 @@ -1077,7 +1077,7 @@ To convert fractions to percents, you may use ``-p``: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint fraction -f n -p data/fraction-example.csv + mlr --opprint fraction -f n -p data/fraction-example.csv u v n n_percent female red 2458 32.638427831629265 female green 192 2.5494622228123753 @@ -1097,7 +1097,7 @@ Another often-used idiom is to convert from a point distribution to a cumulative .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint fraction -f n -p -c data/fraction-example.csv + mlr --opprint fraction -f n -p -c data/fraction-example.csv u v n n_cumulative_percent female red 2458 32.638427831629265 female green 192 35.18789005444164 @@ -1115,7 +1115,7 @@ Another often-used idiom is to convert from a point distribution to a cumulative .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint fraction -f n -g u -p -c data/fraction-example.csv + mlr --opprint fraction -f n -g u -p -c data/fraction-example.csv u v n n_cumulative_percent female red 2458 70.73381294964028 female green 192 76.2589928057554 @@ -1138,7 +1138,7 @@ grep .. code-block:: none :emphasize-lines: 1-1 - $ mlr grep -h + mlr grep -h Usage: mlr grep [options] {regular expression} Passes through records which match the regular expression. Options: @@ -1165,7 +1165,7 @@ group-by .. code-block:: none :emphasize-lines: 1-1 - $ mlr group-by --help + mlr group-by --help Usage: mlr group-by [options] {comma-separated field names} Outputs records in batches having identical values at specified field names.Options: -h|--help Show this message. @@ -1175,7 +1175,7 @@ This is similar to ``sort`` but with less work. Namely, Miller's sort has three .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint group-by a data/small + mlr --opprint group-by a data/small a b i x y pan pan 1 0.3467901443380824 0.7268028627434533 eks pan 2 0.7586799647899636 0.5221511083334797 @@ -1186,7 +1186,7 @@ This is similar to ``sort`` but with less work. Namely, Miller's sort has three .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint sort -f a data/small + mlr --opprint sort -f a data/small a b i x y eks pan 2 0.7586799647899636 0.5221511083334797 eks wye 4 0.38139939387114097 0.13418874328430463 @@ -1204,7 +1204,7 @@ group-like .. code-block:: none :emphasize-lines: 1-1 - $ mlr group-like --help + mlr group-like --help Usage: mlr group-like [options] Outputs records in batches having identical field names.Options: -h|--help Show this message. @@ -1214,7 +1214,7 @@ This groups together records having the same schema (i.e. same ordered list of f .. code-block:: none :emphasize-lines: 1-1 - $ mlr cat data/het.dkvp + mlr cat data/het.dkvp resource=/path/to/file,loadsec=0.45,ok=true record_count=100,resource=/path/to/file resource=/path/to/second/file,loadsec=0.32,ok=true @@ -1224,7 +1224,7 @@ This groups together records having the same schema (i.e. same ordered list of f .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint group-like data/het.dkvp + mlr --opprint group-like data/het.dkvp resource loadsec ok /path/to/file 0.45 true /path/to/second/file 0.32 true @@ -1242,7 +1242,7 @@ having-fields .. code-block:: none :emphasize-lines: 1-1 - $ mlr having-fields --help + mlr having-fields --help Usage: mlr having-fields [options] Conditionally passes through records depending on each record's field names. Options: @@ -1263,7 +1263,7 @@ Similar to :ref:`reference-verbs-group-like`, this retains records with specifie .. code-block:: none :emphasize-lines: 1-1 - $ mlr cat data/het.dkvp + mlr cat data/het.dkvp resource=/path/to/file,loadsec=0.45,ok=true record_count=100,resource=/path/to/file resource=/path/to/second/file,loadsec=0.32,ok=true @@ -1273,7 +1273,7 @@ Similar to :ref:`reference-verbs-group-like`, this retains records with specifie .. code-block:: none :emphasize-lines: 1-1 - $ mlr having-fields --at-least resource data/het.dkvp + mlr having-fields --at-least resource data/het.dkvp resource=/path/to/file,loadsec=0.45,ok=true record_count=100,resource=/path/to/file resource=/path/to/second/file,loadsec=0.32,ok=true @@ -1283,7 +1283,7 @@ Similar to :ref:`reference-verbs-group-like`, this retains records with specifie .. code-block:: none :emphasize-lines: 1-1 - $ mlr having-fields --which-are resource,ok,loadsec data/het.dkvp + mlr having-fields --which-are resource,ok,loadsec data/het.dkvp resource=/path/to/file,loadsec=0.45,ok=true resource=/path/to/second/file,loadsec=0.32,ok=true resource=/some/other/path,loadsec=0.97,ok=false @@ -1296,7 +1296,7 @@ head .. code-block:: none :emphasize-lines: 1-1 - $ mlr head --help + mlr head --help Usage: mlr head [options] Passes through the first n records, optionally by category. Options: @@ -1309,7 +1309,7 @@ Note that ``head`` is distinct from :ref:`reference-verbs-top` -- ``head`` shows .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint head -n 4 data/medium + mlr --opprint head -n 4 data/medium a b i x y pan pan 1 0.3467901443380824 0.7268028627434533 eks pan 2 0.7586799647899636 0.5221511083334797 @@ -1319,7 +1319,7 @@ Note that ``head`` is distinct from :ref:`reference-verbs-top` -- ``head`` shows .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint head -n 1 -g b data/medium + mlr --opprint head -n 1 -g b data/medium a b i x y pan pan 1 0.3467901443380824 0.7268028627434533 wye wye 3 0.20460330576630303 0.33831852551664776 @@ -1335,7 +1335,7 @@ histogram .. code-block:: none :emphasize-lines: 1-1 - $ mlr histogram --help + mlr histogram --help Just a histogram. Input values < lo or > hi are not counted. Usage: mlr histogram [options] -f {a,b,c} Value-field names for histogram counts @@ -1352,7 +1352,7 @@ This is just a histogram; there's not too much to say here. A note about binning .. code-block:: none :emphasize-lines: 1-3 - $ mlr --opprint put '$x2=$x**2;$x3=$x2*$x' \ + mlr --opprint put '$x2=$x**2;$x3=$x2*$x' \ then histogram -f x,x2,x3 --lo 0 --hi 1 --nbins 10 \ data/medium bin_lo bin_hi x_count x2_count x3_count @@ -1370,7 +1370,7 @@ This is just a histogram; there's not too much to say here. A note about binning .. code-block:: none :emphasize-lines: 1-3 - $ mlr --opprint put '$x2=$x**2;$x3=$x2*$x' \ + mlr --opprint put '$x2=$x**2;$x3=$x2*$x' \ then histogram -f x,x2,x3 --lo 0 --hi 1 --nbins 10 -o my_ \ data/medium my_bin_lo my_bin_hi my_x_count my_x2_count my_x3_count @@ -1393,7 +1393,7 @@ join .. code-block:: none :emphasize-lines: 1-1 - $ mlr join --help + mlr join --help Usage: mlr sort {flags} Sorts records primarily by the first specified field, secondarily by the second field, and so on. (Any records not having all specified sort keys will appear @@ -1421,7 +1421,7 @@ Join larger table with IDs with smaller ID-to-name lookup table, showing only pa .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsvlite --opprint cat data/join-left-example.csv + mlr --icsvlite --opprint cat data/join-left-example.csv id name 100 alice 200 bob @@ -1432,7 +1432,7 @@ Join larger table with IDs with smaller ID-to-name lookup table, showing only pa .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsvlite --opprint cat data/join-right-example.csv + mlr --icsvlite --opprint cat data/join-right-example.csv status idcode present 400 present 100 @@ -1458,7 +1458,7 @@ Join larger table with IDs with smaller ID-to-name lookup table, showing only pa .. code-block:: none :emphasize-lines: 1-3 - $ mlr --icsvlite --opprint \ + mlr --icsvlite --opprint \ join -u -j id -r idcode -f data/join-left-example.csv \ data/join-right-example.csv id name status @@ -1487,7 +1487,7 @@ Same, but with sorting the input first: .. code-block:: none :emphasize-lines: 1-3 - $ mlr --icsvlite --opprint sort -f idcode \ + mlr --icsvlite --opprint sort -f idcode \ then join -j id -r idcode -f data/join-left-example.csv \ data/join-right-example.csv id name status @@ -1516,7 +1516,7 @@ Same, but showing only unpaired records: .. code-block:: none :emphasize-lines: 1-3 - $ mlr --icsvlite --opprint \ + mlr --icsvlite --opprint \ join --np --ul --ur -u -j id -r idcode -f data/join-left-example.csv \ data/join-right-example.csv status idcode @@ -1530,7 +1530,7 @@ Use prefixing options to disambiguate between otherwise identical non-join field .. code-block:: none :emphasize-lines: 1-1 - $ mlr --csvlite --opprint cat data/self-join.csv data/self-join.csv + mlr --csvlite --opprint cat data/self-join.csv data/self-join.csv a b c 1 2 3 1 4 5 @@ -1540,7 +1540,7 @@ Use prefixing options to disambiguate between otherwise identical non-join field .. code-block:: none :emphasize-lines: 1-1 - $ mlr --csvlite --opprint join -j a --lp left_ --rp right_ -f data/self-join.csv data/self-join.csv + mlr --csvlite --opprint join -j a --lp left_ --rp right_ -f data/self-join.csv data/self-join.csv a left_b left_c right_b right_c 1 2 3 2 3 1 4 5 2 3 @@ -1552,7 +1552,7 @@ Use zero join columns: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --csvlite --opprint join -j "" --lp left_ --rp right_ -f data/self-join.csv data/self-join.csv + mlr --csvlite --opprint join -j "" --lp left_ --rp right_ -f data/self-join.csv data/self-join.csv left_a left_b left_c right_a right_b right_c 1 2 3 1 2 3 1 4 5 1 2 3 @@ -1567,7 +1567,7 @@ label .. code-block:: none :emphasize-lines: 1-1 - $ mlr label --help + mlr label --help Usage: mlr label [options] {new1,new2,new3,...} Given n comma-separated names, renames the first n fields of each record to have the respective name. (Fields past the nth are left with their original @@ -1600,7 +1600,7 @@ Likewise, if you have CSV/CSV-lite input data which has somehow been bereft of i .. code-block:: none :emphasize-lines: 1-1 - $ cat data/headerless.csv + cat data/headerless.csv John,23,present Fred,34,present Alice,56,missing @@ -1609,7 +1609,7 @@ Likewise, if you have CSV/CSV-lite input data which has somehow been bereft of i .. code-block:: none :emphasize-lines: 1-1 - $ mlr --csv --implicit-csv-header cat data/headerless.csv + mlr --csv --implicit-csv-header cat data/headerless.csv 1,2,3 John,23,present Fred,34,present @@ -1619,7 +1619,7 @@ Likewise, if you have CSV/CSV-lite input data which has somehow been bereft of i .. code-block:: none :emphasize-lines: 1-1 - $ mlr --csv --implicit-csv-header label name,age,status data/headerless.csv + mlr --csv --implicit-csv-header label name,age,status data/headerless.csv name,age,status John,23,present Fred,34,present @@ -1629,7 +1629,7 @@ Likewise, if you have CSV/CSV-lite input data which has somehow been bereft of i .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsv --implicit-csv-header --opprint label name,age,status data/headerless.csv + mlr --icsv --implicit-csv-header --opprint label name,age,status data/headerless.csv name age status John 23 present Fred 34 present @@ -1644,7 +1644,7 @@ least-frequent .. code-block:: none :emphasize-lines: 1-1 - $ mlr least-frequent -h + mlr least-frequent -h Usage: mlr least-frequent [options] Shows the least frequently occurring distinct values for specified field names. The first entry is the statistical anti-mode; the remaining are runners-up. @@ -1658,7 +1658,7 @@ least-frequent .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint --from data/colored-shapes.dkvp least-frequent -f shape -n 5 + mlr --opprint --from data/colored-shapes.dkvp least-frequent -f shape -n 5 shape count circle 2591 triangle 3372 @@ -1667,7 +1667,7 @@ least-frequent .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint --from data/colored-shapes.dkvp least-frequent -f shape,color -n 5 + mlr --opprint --from data/colored-shapes.dkvp least-frequent -f shape,color -n 5 shape color count circle orange 68 triangle orange 107 @@ -1678,7 +1678,7 @@ least-frequent .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint --from data/colored-shapes.dkvp least-frequent -f shape,color -n 5 -o someothername + mlr --opprint --from data/colored-shapes.dkvp least-frequent -f shape,color -n 5 -o someothername shape color someothername circle orange 68 triangle orange 107 @@ -1689,7 +1689,7 @@ least-frequent .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint --from data/colored-shapes.dkvp least-frequent -f shape,color -n 5 -b + mlr --opprint --from data/colored-shapes.dkvp least-frequent -f shape,color -n 5 -b shape color circle orange triangle orange @@ -1707,7 +1707,7 @@ merge-fields .. code-block:: none :emphasize-lines: 1-1 - $ mlr merge-fields --help + mlr merge-fields --help Usage: mlr merge-fields [options] Computes univariate statistics for each input record, accumulated across specified fields. @@ -1759,7 +1759,7 @@ Examples: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --csvlite --opprint cat data/inout.csv + mlr --csvlite --opprint cat data/inout.csv a_in a_out b_in b_out 436 490 446 195 526 320 963 780 @@ -1768,7 +1768,7 @@ Examples: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --csvlite --opprint merge-fields -a min,max,sum -c _in,_out data/inout.csv + mlr --csvlite --opprint merge-fields -a min,max,sum -c _in,_out data/inout.csv a_min a_max a_sum b_min b_max b_sum 436 490 926 195 446 641 320 526 846 780 963 1743 @@ -1777,7 +1777,7 @@ Examples: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --csvlite --opprint merge-fields -k -a sum -c _in,_out data/inout.csv + mlr --csvlite --opprint merge-fields -k -a sum -c _in,_out data/inout.csv a_in a_out b_in b_out a_sum b_sum 436 490 446 195 926 641 526 320 963 780 846 1743 @@ -1791,7 +1791,7 @@ most-frequent .. code-block:: none :emphasize-lines: 1-1 - $ mlr most-frequent -h + mlr most-frequent -h Usage: mlr most-frequent [options] Shows the most frequently occurring distinct values for specified field names. The first entry is the statistical mode; the remaining are runners-up. @@ -1805,7 +1805,7 @@ most-frequent .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint --from data/colored-shapes.dkvp most-frequent -f shape -n 5 + mlr --opprint --from data/colored-shapes.dkvp most-frequent -f shape -n 5 shape count square 4115 triangle 3372 @@ -1814,29 +1814,29 @@ most-frequent .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint --from data/colored-shapes.dkvp most-frequent -f shape,color -n 5 + mlr --opprint --from data/colored-shapes.dkvp most-frequent -f shape,color -n 5 shape color count square red 1874 triangle red 1560 circle red 1207 - square blue 589 square yellow 589 + square blue 589 .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint --from data/colored-shapes.dkvp most-frequent -f shape,color -n 5 -o someothername + mlr --opprint --from data/colored-shapes.dkvp most-frequent -f shape,color -n 5 -o someothername shape color someothername square red 1874 triangle red 1560 circle red 1207 - square yellow 589 square blue 589 + square yellow 589 .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint --from data/colored-shapes.dkvp most-frequent -f shape,color -n 5 -b + mlr --opprint --from data/colored-shapes.dkvp most-frequent -f shape,color -n 5 -b shape color square red triangle red @@ -1854,7 +1854,7 @@ nest .. code-block:: none :emphasize-lines: 1-1 - $ mlr nest -h + mlr nest -h Usage: mlr nest [options] Explodes specified field values into separate fields/records, or reverses this. Options: @@ -1911,7 +1911,7 @@ nothing .. code-block:: none :emphasize-lines: 1-1 - $ mlr nothing -h + mlr nothing -h Usage: mlr nothing [options] Drops all input records. Useful for testing, or after tee/print/etc. have produced other output. @@ -1926,7 +1926,7 @@ put .. code-block:: none :emphasize-lines: 1-1 - $ mlr put --help + mlr put --help Usage: mlr put [options] {DSL expression} Options: -f {file name} File containing a DSL expression. If the filename is a directory, @@ -1993,7 +1993,7 @@ regularize .. code-block:: none :emphasize-lines: 1-1 - $ mlr regularize --help + mlr regularize --help Usage: mlr regularize [options] Outputs records sorted lexically ascending by keys.Options: -h|--help Show this message. @@ -2010,7 +2010,7 @@ remove-empty-columns .. code-block:: none :emphasize-lines: 1-1 - $ mlr remove-empty-columns --help + mlr remove-empty-columns --help Usage: mlr remove-empty-columns [options] Omits fields which are empty on every input row. Non-streaming. Options: @@ -2019,7 +2019,7 @@ remove-empty-columns .. code-block:: none :emphasize-lines: 1-1 - $ cat data/remove-empty-columns.csv + cat data/remove-empty-columns.csv a,b,c,d,e 1,,3,,5 2,,4,,5 @@ -2028,7 +2028,7 @@ remove-empty-columns .. code-block:: none :emphasize-lines: 1-1 - $ mlr --csv remove-empty-columns data/remove-empty-columns.csv + mlr --csv remove-empty-columns data/remove-empty-columns.csv a,c,e 1,3,5 2,4,5 @@ -2044,7 +2044,7 @@ rename .. code-block:: none :emphasize-lines: 1-1 - $ mlr rename --help + mlr rename --help Usage: mlr rename [options] {old1,new1,old2,new2,...} Renames specified fields. Options: @@ -2070,7 +2070,7 @@ rename .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint cat data/small + mlr --opprint cat data/small a b i x y pan pan 1 0.3467901443380824 0.7268028627434533 eks pan 2 0.7586799647899636 0.5221511083334797 @@ -2081,7 +2081,7 @@ rename .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint rename i,INDEX,b,COLUMN2 data/small + mlr --opprint rename i,INDEX,b,COLUMN2 data/small a COLUMN2 INDEX x y pan pan 1 0.3467901443380824 0.7268028627434533 eks pan 2 0.7586799647899636 0.5221511083334797 @@ -2094,7 +2094,7 @@ As discussed in :doc:`performance`, ``sed`` is significantly faster than Miller .. code-block:: none :emphasize-lines: 1-1 - $ sed 's/y/COLUMN5/g' data/small + sed 's/y/COLUMN5/g' data/small a=pan,b=pan,i=1,x=0.3467901443380824,COLUMN5=0.7268028627434533 a=eks,b=pan,i=2,x=0.7586799647899636,COLUMN5=0.5221511083334797 a=wCOLUMN5e,b=wCOLUMN5e,i=3,x=0.20460330576630303,COLUMN5=0.33831852551664776 @@ -2104,7 +2104,7 @@ As discussed in :doc:`performance`, ``sed`` is significantly faster than Miller .. code-block:: none :emphasize-lines: 1-1 - $ mlr rename y,COLUMN5 data/small + mlr rename y,COLUMN5 data/small a=pan,b=pan,i=1,x=0.3467901443380824,COLUMN5=0.7268028627434533 a=eks,b=pan,i=2,x=0.7586799647899636,COLUMN5=0.5221511083334797 a=wye,b=wye,i=3,x=0.20460330576630303,COLUMN5=0.33831852551664776 @@ -2121,7 +2121,7 @@ reorder .. code-block:: none :emphasize-lines: 1-1 - $ mlr reorder --help + mlr reorder --help Usage: mlr reorder [options] Moves specified names to start of record, or end of record. Options: @@ -2146,7 +2146,7 @@ two to the front of line where you can give a quick visual scan. .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint cat data/small + mlr --opprint cat data/small a b i x y pan pan 1 0.3467901443380824 0.7268028627434533 eks pan 2 0.7586799647899636 0.5221511083334797 @@ -2157,7 +2157,7 @@ two to the front of line where you can give a quick visual scan. .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint reorder -f i,b data/small + mlr --opprint reorder -f i,b data/small i b a x y 1 pan pan 0.3467901443380824 0.7268028627434533 2 pan eks 0.7586799647899636 0.5221511083334797 @@ -2168,7 +2168,7 @@ two to the front of line where you can give a quick visual scan. .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint reorder -e -f i,b data/small + mlr --opprint reorder -e -f i,b data/small a x y i b pan 0.3467901443380824 0.7268028627434533 1 pan eks 0.7586799647899636 0.5221511083334797 2 pan @@ -2184,7 +2184,7 @@ repeat .. code-block:: none :emphasize-lines: 1-1 - $ mlr repeat --help + mlr repeat --help Usage: mlr repeat [options] Copies input records to output records multiple times. Options must be exactly one of the following: @@ -2218,7 +2218,7 @@ samples from data which has been count-aggregated: .. code-block:: none :emphasize-lines: 1-1 - $ cat data/repeat-example.dat + cat data/repeat-example.dat color=blue,count=5 color=red,count=4 color=green,count=3 @@ -2226,7 +2226,7 @@ samples from data which has been count-aggregated: .. code-block:: none :emphasize-lines: 1-1 - $ mlr repeat -f count then cut -x -f count data/repeat-example.dat + mlr repeat -f count then cut -x -f count data/repeat-example.dat color=blue color=blue color=blue @@ -2252,7 +2252,7 @@ reshape .. code-block:: none :emphasize-lines: 1-1 - $ mlr reshape --help + mlr reshape --help Usage: mlr reshape [options] Wide-to-long options: -i {input field names} -o {key-field name,value-field name} @@ -2317,7 +2317,7 @@ sample .. code-block:: none :emphasize-lines: 1-1 - $ mlr sample --help + mlr sample --help Usage: mlr sample [options] Reservoir sampling (subsampling without replacement), optionally by category. See also mlr bootstrap and mlr shuffle. @@ -2391,7 +2391,7 @@ sec2gmt .. code-block:: none :emphasize-lines: 1-1 - $ mlr sec2gmt -h + mlr sec2gmt -h Usage: mlr sec2gmt [options] {comma-separated list of field names} Replaces a numeric field representing seconds since the epoch with the corresponding GMT timestamp; leaves non-numbers as-is. This is nothing @@ -2414,7 +2414,7 @@ sec2gmtdate .. code-block:: none :emphasize-lines: 1-1 - $ mlr sec2gmtdate -h + mlr sec2gmtdate -h Usage: ../c/mlr sec2gmtdate {comma-separated list of field names} Replaces a numeric field representing seconds since the epoch with the corresponding GMT year-month-day timestamp; leaves non-numbers as-is. @@ -2431,7 +2431,7 @@ seqgen .. code-block:: none :emphasize-lines: 1-1 - $ mlr seqgen -h + mlr seqgen -h Usage: mlr seqgen [options] Passes input records directly to output. Most useful for format conversion. Produces a sequence of counters. Discards the input record stream. Produces @@ -2450,7 +2450,7 @@ seqgen .. code-block:: none :emphasize-lines: 1-1 - $ mlr seqgen --stop 10 + mlr seqgen --stop 10 i=1 i=2 i=3 @@ -2465,7 +2465,7 @@ seqgen .. code-block:: none :emphasize-lines: 1-1 - $ mlr seqgen --start 20 --stop 40 --step 4 + mlr seqgen --start 20 --stop 40 --step 4 i=20 i=24 i=28 @@ -2476,7 +2476,7 @@ seqgen .. code-block:: none :emphasize-lines: 1-1 - $ mlr seqgen --start 40 --stop 20 --step -4 + mlr seqgen --start 40 --stop 20 --step -4 i=40 i=36 i=32 @@ -2492,7 +2492,7 @@ shuffle .. code-block:: none :emphasize-lines: 1-1 - $ mlr shuffle -h + mlr shuffle -h Usage: mlr shuffle [options] Outputs records randomly permuted. No output records are produced until all input records are read. See also mlr bootstrap and mlr sample. @@ -2507,7 +2507,7 @@ skip-trivial-records .. code-block:: none :emphasize-lines: 1-1 - $ mlr skip-trivial-records -h + mlr skip-trivial-records -h Usage: mlr skip-trivial-records [options] Passes through all records except those with zero fields, or those for which all fields have empty value. @@ -2517,7 +2517,7 @@ skip-trivial-records .. code-block:: none :emphasize-lines: 1-1 - $ cat data/trivial-records.csv + cat data/trivial-records.csv a,b,c 1,2,3 4,,6 @@ -2527,7 +2527,7 @@ skip-trivial-records .. code-block:: none :emphasize-lines: 1-1 - $ mlr --csv skip-trivial-records data/trivial-records.csv + mlr --csv skip-trivial-records data/trivial-records.csv a,b,c 1,2,3 4,,6 @@ -2541,7 +2541,7 @@ sort .. code-block:: none :emphasize-lines: 1-1 - $ mlr sort --help + mlr sort --help Usage: mlr sort {flags} Sorts records primarily by the first specified field, secondarily by the second field, and so on. (Any records not having all specified sort keys will appear @@ -2567,7 +2567,7 @@ Example: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint sort -f a -nr x data/small + mlr --opprint sort -f a -nr x data/small a b i x y eks pan 2 0.7586799647899636 0.5221511083334797 eks wye 4 0.38139939387114097 0.13418874328430463 @@ -2580,7 +2580,7 @@ Here's an example filtering log data: suppose multiple threads (labeled here by .. code-block:: none :emphasize-lines: 1-1 - $ head -n 10 data/multicountdown.dat + head -n 10 data/multicountdown.dat upsec=0.002,color=green,count=1203 upsec=0.083,color=red,count=3817 upsec=0.188,color=red,count=3801 @@ -2599,7 +2599,7 @@ timestamps within each thread's log data are still chronological: .. code-block:: none :emphasize-lines: 1-1 - $ head -n 20 data/multicountdown.dat | mlr --opprint sort -f color + head -n 20 data/multicountdown.dat | mlr --opprint sort -f color upsec color count 0.395 blue 2697 0.671 blue 2684 @@ -2628,7 +2628,7 @@ were encountered, regardless of the specified sort order: .. code-block:: none :emphasize-lines: 1-1 - $ mlr sort -n x data/sort-missing.dkvp + mlr sort -n x data/sort-missing.dkvp x=1 x=2 x=4 @@ -2637,7 +2637,7 @@ were encountered, regardless of the specified sort order: .. code-block:: none :emphasize-lines: 1-1 - $ mlr sort -nr x data/sort-missing.dkvp + mlr sort -nr x data/sort-missing.dkvp x=4 x=2 x=1 @@ -2651,7 +2651,7 @@ sort-within-records .. code-block:: none :emphasize-lines: 1-1 - $ mlr sort-within-records -h + mlr sort-within-records -h Usage: mlr sort-within-records [options] Outputs records sorted lexically ascending by keys. Options: @@ -2661,7 +2661,7 @@ sort-within-records .. code-block:: none :emphasize-lines: 1-1 - $ cat data/sort-within-records.json + cat data/sort-within-records.json { "a": 1, "b": 2, @@ -2681,7 +2681,7 @@ sort-within-records .. code-block:: none :emphasize-lines: 1-1 - $ mlr --ijson --opprint cat data/sort-within-records.json + mlr --ijson --opprint cat data/sort-within-records.json a b c 1 2 3 @@ -2694,7 +2694,7 @@ sort-within-records .. code-block:: none :emphasize-lines: 1-1 - $ mlr --json sort-within-records data/sort-within-records.json + mlr --json sort-within-records data/sort-within-records.json { "a": 1, "b": 2, @@ -2714,7 +2714,7 @@ sort-within-records .. code-block:: none :emphasize-lines: 1-1 - $ mlr --ijson --opprint sort-within-records data/sort-within-records.json + mlr --ijson --opprint sort-within-records data/sort-within-records.json a b c 1 2 3 5 4 6 @@ -2728,7 +2728,7 @@ stats1 .. code-block:: none :emphasize-lines: 1-1 - $ mlr stats1 --help + mlr stats1 --help Usage: mlr stats1 [options] Computes univariate statistics for one or more given fields, accumulated across the input record stream. @@ -2789,7 +2789,7 @@ optionally categorized by one or more other fields. .. code-block:: none :emphasize-lines: 1-1 - $ mlr --oxtab stats1 -a count,sum,min,p10,p50,mean,p90,max -f x,y data/medium + mlr --oxtab stats1 -a count,sum,min,p10,p50,mean,p90,max -f x,y data/medium x_count 10000 x_sum 4986.019681679581 x_min 4.509679127584487e-05 @@ -2810,7 +2810,7 @@ optionally categorized by one or more other fields. .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint stats1 -a mean -f x,y -g b then sort -f b data/medium + mlr --opprint stats1 -a mean -f x,y -g b then sort -f b data/medium b x_mean y_mean eks 0.5063609846272304 0.510292657158104 hat 0.4878988625336502 0.5131176341556505 @@ -2821,7 +2821,7 @@ optionally categorized by one or more other fields. .. code-block:: none :emphasize-lines: 1-3 - $ mlr --opprint stats1 -a p50,p99 -f u,v -g color \ + mlr --opprint stats1 -a p50,p99 -f u,v -g color \ then put '$ur=$u_p99/$u_p50;$vr=$v_p99/$v_p50' \ data/colored-shapes.dkvp color u_p50 u_p99 v_p50 v_p99 ur vr @@ -2835,7 +2835,7 @@ optionally categorized by one or more other fields. .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint count-distinct -f shape then sort -nr count data/colored-shapes.dkvp + mlr --opprint count-distinct -f shape then sort -nr count data/colored-shapes.dkvp shape count square 4115 triangle 3372 @@ -2844,7 +2844,7 @@ optionally categorized by one or more other fields. .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint stats1 -a mode -f color -g shape data/colored-shapes.dkvp + mlr --opprint stats1 -a mode -f color -g shape data/colored-shapes.dkvp shape color_mode triangle red square red @@ -2858,7 +2858,7 @@ stats2 .. code-block:: none :emphasize-lines: 1-1 - $ mlr stats2 --help + mlr stats2 --help Usage: mlr stats2 [options] Computes bivariate statistics for one or more given field-name pairs, accumulated across the input record stream. @@ -2892,7 +2892,7 @@ fields, optionally categorized by one or more fields. .. code-block:: none :emphasize-lines: 1-3 - $ mlr --oxtab put '$x2=$x*$x; $xy=$x*$y; $y2=$y**2' \ + mlr --oxtab put '$x2=$x*$x; $xy=$x*$y; $y2=$y**2' \ then stats2 -a cov,corr -f x,y,y,y,x2,xy,x2,y2 \ data/medium x_y_cov 4.2574820827444476e-05 @@ -2907,7 +2907,7 @@ fields, optionally categorized by one or more fields. .. code-block:: none :emphasize-lines: 1-3 - $ mlr --opprint put '$x2=$x*$x; $xy=$x*$y; $y2=$y**2' \ + mlr --opprint put '$x2=$x*$x; $xy=$x*$y; $y2=$y**2' \ then stats2 -a linreg-ols,r2 -f x,y,y,y,xy,y2 -g a \ data/medium a x_y_ols_m x_y_ols_b x_y_ols_n x_y_r2 y_y_ols_m y_y_ols_b y_y_ols_n y_y_r2 xy_y2_ols_m xy_y2_ols_b xy_y2_ols_n xy_y2_r2 @@ -2958,7 +2958,7 @@ Here's an example estimating time-to-completion for a set of jobs. Input data co .. code-block:: none :emphasize-lines: 1-1 - $ head -n 10 data/multicountdown.dat + head -n 10 data/multicountdown.dat upsec=0.002,color=green,count=1203 upsec=0.083,color=red,count=3817 upsec=0.188,color=red,count=3801 @@ -2975,7 +2975,7 @@ We can do a linear regression on count remaining as a function of time: with ``c .. code-block:: none :emphasize-lines: 1-3 - $ mlr --oxtab stats2 -a linreg-pca -f upsec,count -g color \ + mlr --oxtab stats2 -a linreg-pca -f upsec,count -g color \ then put '$donesec = -$upsec_count_pca_b/$upsec_count_pca_m' \ data/multicountdown.dat color green @@ -3014,7 +3014,7 @@ step .. code-block:: none :emphasize-lines: 1-1 - $ mlr step --help + mlr step --help Usage: mlr step [options] Computes values dependent on the previous record, optionally grouped by category. Options: @@ -3058,7 +3058,7 @@ Most Miller commands are record-at-a-time, with the exception of ``stats1``, ``s .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint step -a shift,delta,rsum,counter -f x data/medium | head -15 + mlr --opprint step -a shift,delta,rsum,counter -f x data/medium | head -15 a b i x y x_shift x_delta x_rsum x_counter pan pan 1 0.3467901443380824 0.7268028627434533 - 0 0.3467901443380824 1 eks pan 2 0.7586799647899636 0.5221511083334797 0.3467901443380824 0.41188982045188116 1.105470109128046 2 @@ -3078,7 +3078,7 @@ Most Miller commands are record-at-a-time, with the exception of ``stats1``, ``s .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint step -a shift,delta,rsum,counter -f x -g a data/medium | head -15 + mlr --opprint step -a shift,delta,rsum,counter -f x -g a data/medium | head -15 a b i x y x_shift x_delta x_rsum x_counter pan pan 1 0.3467901443380824 0.7268028627434533 - 0 0.3467901443380824 1 eks pan 2 0.7586799647899636 0.5221511083334797 - 0 0.7586799647899636 1 @@ -3098,7 +3098,7 @@ Most Miller commands are record-at-a-time, with the exception of ``stats1``, ``s .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint step -a ewma -f x -d 0.1,0.9 data/medium | head -15 + mlr --opprint step -a ewma -f x -d 0.1,0.9 data/medium | head -15 a b i x y x_ewma_0.1 x_ewma_0.9 pan pan 1 0.3467901443380824 0.7268028627434533 0.3467901443380824 0.3467901443380824 eks pan 2 0.7586799647899636 0.5221511083334797 0.3879791263832706 0.7174909827447755 @@ -3118,7 +3118,7 @@ Most Miller commands are record-at-a-time, with the exception of ``stats1``, ``s .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint step -a ewma -f x -d 0.1,0.9 -o smooth,rough data/medium | head -15 + mlr --opprint step -a ewma -f x -d 0.1,0.9 -o smooth,rough data/medium | head -15 a b i x y x_ewma_smooth x_ewma_rough pan pan 1 0.3467901443380824 0.7268028627434533 0.3467901443380824 0.3467901443380824 eks pan 2 0.7586799647899636 0.5221511083334797 0.3879791263832706 0.7174909827447755 @@ -3160,7 +3160,7 @@ tac .. code-block:: none :emphasize-lines: 1-1 - $ mlr tac --help + mlr tac --help Usage: mlr tac [options] Prints records in reverse order from the order in which they were encountered. Options: @@ -3171,7 +3171,7 @@ Prints the records in the input stream in reverse order. Note: this requires Mil .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsv --opprint cat data/a.csv + mlr --icsv --opprint cat data/a.csv a b c 1 2 3 4 5 6 @@ -3179,14 +3179,14 @@ Prints the records in the input stream in reverse order. Note: this requires Mil .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsv --opprint cat data/b.csv + mlr --icsv --opprint cat data/b.csv a b c 7 8 9 .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsv --opprint tac data/a.csv data/b.csv + mlr --icsv --opprint tac data/a.csv data/b.csv a b c 7 8 9 4 5 6 @@ -3195,7 +3195,7 @@ Prints the records in the input stream in reverse order. Note: this requires Mil .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsv --opprint put '$filename=FILENAME' then tac data/a.csv data/b.csv + mlr --icsv --opprint put '$filename=FILENAME' then tac data/a.csv data/b.csv a b c filename 7 8 9 data/b.csv 4 5 6 data/a.csv @@ -3209,7 +3209,7 @@ tail .. code-block:: none :emphasize-lines: 1-1 - $ mlr tail --help + mlr tail --help Usage: mlr tail [options] Passes through the last n records, optionally by category. Options: @@ -3222,7 +3222,7 @@ Prints the last *n* records in the input stream, optionally by category. .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint tail -n 4 data/colored-shapes.dkvp + mlr --opprint tail -n 4 data/colored-shapes.dkvp color shape flag i u v w x blue square 1 99974 0.6189062525431605 0.2637962404841453 0.5311465405784674 6.210738209085753 blue triangle 0 99976 0.008110504040268474 0.8267274952432482 0.4732962944898885 6.146956761817328 @@ -3232,7 +3232,7 @@ Prints the last *n* records in the input stream, optionally by category. .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint tail -n 1 -g shape data/colored-shapes.dkvp + mlr --opprint tail -n 1 -g shape data/colored-shapes.dkvp color shape flag i u v w x yellow triangle 0 99990 0.3839424618160777 0.55952913620132 0.5113763011485609 4.307973891915119 blue square 1 99974 0.6189062525431605 0.2637962404841453 0.5311465405784674 6.210738209085753 @@ -3246,7 +3246,7 @@ tee .. code-block:: none :emphasize-lines: 1-1 - $ mlr tee --help + mlr tee --help Usage: mlr tee [options] {filename} Options: -a Append to existing file, if any, rather than overwriting. @@ -3266,7 +3266,7 @@ top .. code-block:: none :emphasize-lines: 1-1 - $ mlr top --help + mlr top --help Usage: mlr top [options] -f {a,b,c} Value-field names for top counts. -g {d,e,f} Optional group-by-field names for top counts. @@ -3285,7 +3285,7 @@ Note that ``top`` is distinct from :ref:`reference-verbs-head` -- ``head`` shows .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint top -n 4 -f x data/medium + mlr --opprint top -n 4 -f x data/medium top_idx x_top 1 0.999952670371898 2 0.9998228522652893 @@ -3295,7 +3295,7 @@ Note that ``top`` is distinct from :ref:`reference-verbs-head` -- ``head`` shows .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint top -n 4 -f x -o someothername data/medium + mlr --opprint top -n 4 -f x -o someothername data/medium someothername x_top 1 0.999952670371898 2 0.9998228522652893 @@ -3305,7 +3305,7 @@ Note that ``top`` is distinct from :ref:`reference-verbs-head` -- ``head`` shows .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint top -n 2 -f x -g a then sort -f a data/medium + mlr --opprint top -n 2 -f x -g a then sort -f a data/medium a top_idx x_top eks 1 0.9988110946859143 eks 2 0.9985342548358704 @@ -3326,7 +3326,7 @@ uniq .. code-block:: none :emphasize-lines: 1-1 - $ mlr uniq --help + mlr uniq --help Usage: mlr uniq [options] Prints distinct values for specified field names. With -c, same as count-distinct. For uniq, -f is a synonym for -g. @@ -3346,13 +3346,13 @@ There are two main ways to use ``mlr uniq``: the first way is with ``-g`` to spe .. code-block:: none :emphasize-lines: 1-1 - $ wc -l data/colored-shapes.dkvp + wc -l data/colored-shapes.dkvp 10078 data/colored-shapes.dkvp .. code-block:: none :emphasize-lines: 1-1 - $ mlr uniq -g color,shape data/colored-shapes.dkvp + mlr uniq -g color,shape data/colored-shapes.dkvp color=yellow,shape=triangle color=red,shape=square color=red,shape=circle @@ -3375,7 +3375,7 @@ There are two main ways to use ``mlr uniq``: the first way is with ``-g`` to spe .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint uniq -g color,shape -c then sort -f color,shape data/colored-shapes.dkvp + mlr --opprint uniq -g color,shape -c then sort -f color,shape data/colored-shapes.dkvp color shape count blue circle 384 blue square 589 @@ -3399,7 +3399,7 @@ There are two main ways to use ``mlr uniq``: the first way is with ``-g`` to spe .. code-block:: none :emphasize-lines: 1-3 - $ mlr --opprint uniq -g color,shape -c -o someothername \ + mlr --opprint uniq -g color,shape -c -o someothername \ then sort -nr someothername \ data/colored-shapes.dkvp color shape someothername @@ -3425,7 +3425,7 @@ There are two main ways to use ``mlr uniq``: the first way is with ``-g`` to spe .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint uniq -n -g color,shape data/colored-shapes.dkvp + mlr --opprint uniq -n -g color,shape data/colored-shapes.dkvp count 18 @@ -3434,7 +3434,7 @@ The second main way to use ``mlr uniq`` is without group-by columns, using ``-a` .. code-block:: none :emphasize-lines: 1-1 - $ cat data/repeats.dkvp + cat data/repeats.dkvp color=red,shape=square,flag=0 color=purple,shape=triangle,flag=0 color=yellow,shape=circle,flag=1 @@ -3496,13 +3496,13 @@ The second main way to use ``mlr uniq`` is without group-by columns, using ``-a` .. code-block:: none :emphasize-lines: 1-1 - $ wc -l data/repeats.dkvp + wc -l data/repeats.dkvp 57 data/repeats.dkvp .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint uniq -a data/repeats.dkvp + mlr --opprint uniq -a data/repeats.dkvp color shape flag red square 0 purple triangle 0 @@ -3515,14 +3515,14 @@ The second main way to use ``mlr uniq`` is without group-by columns, using ``-a` .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint uniq -a -n data/repeats.dkvp + mlr --opprint uniq -a -n data/repeats.dkvp count 7 .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint uniq -a -c data/repeats.dkvp + mlr --opprint uniq -a -c data/repeats.dkvp count color shape flag 17 red square 0 11 purple triangle 0 @@ -3540,7 +3540,7 @@ unsparsify .. code-block:: none :emphasize-lines: 1-1 - $ mlr unsparsify --help + mlr unsparsify --help Usage: mlr unsparsify [options] Prints records with the union of field names over all input records. For field names absent in a given record but present in others, fills in @@ -3560,7 +3560,7 @@ Examples: .. code-block:: none :emphasize-lines: 1-1 - $ cat data/sparse.json + cat data/sparse.json {"a":1,"b":2,"v":3} {"u":1,"b":2} {"a":1,"v":2,"x":3} @@ -3569,7 +3569,7 @@ Examples: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --json unsparsify data/sparse.json + mlr --json unsparsify data/sparse.json { "a": 1, "b": 2, @@ -3606,7 +3606,7 @@ Examples: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --ijson --opprint unsparsify data/sparse.json + mlr --ijson --opprint unsparsify data/sparse.json a b v u x w 1 2 3 - - - - 2 - 1 - - @@ -3616,7 +3616,7 @@ Examples: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --ijson --opprint unsparsify --fill-with missing data/sparse.json + mlr --ijson --opprint unsparsify --fill-with missing data/sparse.json a b v u x w 1 2 3 missing missing missing missing 2 missing 1 missing missing @@ -3626,7 +3626,7 @@ Examples: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --ijson --opprint unsparsify -f a,b,u data/sparse.json + mlr --ijson --opprint unsparsify -f a,b,u data/sparse.json a b v u 1 2 3 - @@ -3642,7 +3642,7 @@ Examples: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --ijson --opprint unsparsify -f a,b,u,v,w,x then regularize data/sparse.json + mlr --ijson --opprint unsparsify -f a,b,u,v,w,x then regularize data/sparse.json a b v u w x 1 2 3 - - - - 2 - 1 - - diff --git a/docs6/repl.rst b/docs6/repl.rst index bf0fac4e5..83ec5e665 100644 --- a/docs6/repl.rst +++ b/docs6/repl.rst @@ -32,7 +32,7 @@ Using ``put`` and ``filter``, you can do the following as we've seen above: .. code-block:: none :emphasize-lines: 1-2 - $ mlr --icsv --ojson --from example.csv head -n 2 \ + mlr --icsv --ojson --from example.csv head -n 2 \ then put 'begin {print "HELLO"} $z = $x + $y; end {print "GOODBYE"}' HELLO { diff --git a/docs6/shapes-of-data.rst b/docs6/shapes-of-data.rst index bfbb53364..cdaed6387 100644 --- a/docs6/shapes-of-data.rst +++ b/docs6/shapes-of-data.rst @@ -89,44 +89,44 @@ Miller records are ordered lists of key-value pairs. For NIDX format, DKVP forma .. code-block:: none :emphasize-lines: 1-1 - $ echo x,y,z | mlr --dkvp cat + echo x,y,z | mlr --dkvp cat 1=x,2=y,3=z .. code-block:: none :emphasize-lines: 1-1 - $ echo x,y,z | mlr --dkvp put '$6="a";$4="b";$55="cde"' + echo x,y,z | mlr --dkvp put '$6="a";$4="b";$55="cde"' 1=x,2=y,3=z,6=a,4=b,55=cde .. code-block:: none :emphasize-lines: 1-1 - $ echo x,y,z | mlr --nidx cat + echo x,y,z | mlr --nidx cat x,y,z .. code-block:: none :emphasize-lines: 1-1 - $ echo x,y,z | mlr --csv --implicit-csv-header cat + echo x,y,z | mlr --csv --implicit-csv-header cat 1,2,3 x,y,z .. code-block:: none :emphasize-lines: 1-1 - $ echo x,y,z | mlr --dkvp rename 2,999 + echo x,y,z | mlr --dkvp rename 2,999 1=x,999=y,3=z .. code-block:: none :emphasize-lines: 1-1 - $ echo x,y,z | mlr --dkvp rename 2,newname + echo x,y,z | mlr --dkvp rename 2,newname 1=x,newname=y,3=z .. code-block:: none :emphasize-lines: 1-1 - $ echo x,y,z | mlr --csv --implicit-csv-header reorder -f 3,1,2 + echo x,y,z | mlr --csv --implicit-csv-header reorder -f 3,1,2 3,1,2 z,x,y @@ -138,7 +138,7 @@ Example: columns ``x,i,a`` were requested but they appear here in the order ``a, .. code-block:: none :emphasize-lines: 1-1 - $ cat data/small + cat data/small a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533 a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797 a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776 @@ -148,7 +148,7 @@ Example: columns ``x,i,a`` were requested but they appear here in the order ``a, .. code-block:: none :emphasize-lines: 1-1 - $ mlr cut -f x,i,a data/small + mlr cut -f x,i,a data/small a=pan,i=1,x=0.3467901443380824 a=eks,i=2,x=0.7586799647899636 a=wye,i=3,x=0.20460330576630303 @@ -162,7 +162,7 @@ The solution is to use the ``-o`` option: .. code-block:: none :emphasize-lines: 1-1 - $ mlr cut -o -f x,i,a data/small + mlr cut -o -f x,i,a data/small x=0.3467901443380824,i=1,a=pan x=0.7586799647899636,i=2,a=eks x=0.20460330576630303,i=3,a=wye @@ -177,7 +177,7 @@ The ``awk``-like built-in variable ``NR`` is incremented for each input record: .. code-block:: none :emphasize-lines: 1-1 - $ cat data/small + cat data/small a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533 a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797 a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776 @@ -187,7 +187,7 @@ The ``awk``-like built-in variable ``NR`` is incremented for each input record: .. code-block:: none :emphasize-lines: 1-1 - $ mlr put '$nr = NR' data/small + mlr put '$nr = NR' data/small a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533,nr=1 a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797,nr=2 a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776,nr=3 @@ -199,7 +199,7 @@ However, this is the record number within the original input stream -- not after .. code-block:: none :emphasize-lines: 1-1 - $ mlr filter '$a == "wye"' then put '$nr = NR' data/small + mlr filter '$a == "wye"' then put '$nr = NR' data/small a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776,nr=3 a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729,nr=5 @@ -208,7 +208,7 @@ There are two good options here. One is to use the ``cat`` verb with ``-n``: .. code-block:: none :emphasize-lines: 1-1 - $ mlr filter '$a == "wye"' then cat -n data/small + mlr filter '$a == "wye"' then cat -n data/small n=1,a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776 n=2,a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729 @@ -217,7 +217,7 @@ The other is to keep your own counter within the ``put`` DSL: .. code-block:: none :emphasize-lines: 1-1 - $ mlr filter '$a == "wye"' then put 'begin {@n = 1} $n = @n; @n += 1' data/small + mlr filter '$a == "wye"' then put 'begin {@n = 1} $n = @n; @n += 1' data/small a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776,n=1 a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729,n=2 @@ -239,7 +239,7 @@ The simplest option is to use :ref:`mlr nest `: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --tsv nest --explode --values --across-records -f b --nested-fs : data/nested.tsv + mlr --tsv nest --explode --values --across-records -f b --nested-fs : data/nested.tsv a b x z s u @@ -249,7 +249,7 @@ The simplest option is to use :ref:`mlr nest `: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --tsv nest --explode --values --across-fields -f b --nested-fs : data/nested.tsv + mlr --tsv nest --explode --values --across-fields -f b --nested-fs : data/nested.tsv a b_1 x z @@ -263,7 +263,7 @@ One option to split out the colon-delimited values in the ``b`` column is to use .. code-block:: none :emphasize-lines: 1-6 - $ mlr --from data/nested.tsv --itsv --oxtab put ' + mlr --from data/nested.tsv --itsv --oxtab put ' o = splitnv($b, ":"); for (k,v in o) { $["p".k]=v @@ -284,7 +284,7 @@ while another is to loop over the same map from ``splitnv`` and use it (with ``p .. code-block:: none :emphasize-lines: 1-7 - $ mlr --from data/nested.tsv --itsv --oxtab put -q ' + mlr --from data/nested.tsv --itsv --oxtab put -q ' o = splitnv($b, ":"); for (k,v in o) { x = mapsum($*, {"b":v}); @@ -306,7 +306,7 @@ while another is to loop over the same map from ``splitnv`` and use it (with ``p .. code-block:: none :emphasize-lines: 1-6 - $ mlr --from data/nested.tsv --tsv put -q ' + mlr --from data/nested.tsv --tsv put -q ' o = splitnv($b, ":"); for (k,v in o) { x = mapsum($*, {"b":v}); emit x @@ -387,7 +387,7 @@ The idea here is that middles starting with a 1 belong to the outer value of 1, .. code-block:: none :emphasize-lines: 1-10 - $ mlr --from data/rect.txt put -q ' + mlr --from data/rect.txt put -q ' is_present($outer) { unset @r } diff --git a/docs6/shell-commands.rst b/docs6/shell-commands.rst index 021a09bfe..c5dfc41dc 100644 --- a/docs6/shell-commands.rst +++ b/docs6/shell-commands.rst @@ -11,7 +11,7 @@ The :ref:`reference-dsl-system` DSL function allows you to run a specific shell .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint put '$o = system("echo hello world")' data/small + mlr --opprint put '$o = system("echo hello world")' data/small a b i x y o pan pan 1 0.3467901443380824 0.7268028627434533 hello world eks pan 2 0.7586799647899636 0.5221511083334797 hello world @@ -22,7 +22,7 @@ The :ref:`reference-dsl-system` DSL function allows you to run a specific shell .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint put '$o = system("echo {" . NR . "}")' data/small + mlr --opprint put '$o = system("echo {" . NR . "}")' data/small a b i x y o pan pan 1 0.3467901443380824 0.7268028627434533 {1} eks pan 2 0.7586799647899636 0.5221511083334797 {2} @@ -33,7 +33,7 @@ The :ref:`reference-dsl-system` DSL function allows you to run a specific shell .. code-block:: none :emphasize-lines: 1-1 - $ mlr --opprint put '$o = system("echo -n ".$a."| sha1sum")' data/small + mlr --opprint put '$o = system("echo -n ".$a."| sha1sum")' data/small a b i x y o pan pan 1 0.3467901443380824 0.7268028627434533 f29c748220331c273ef16d5115f6ecd799947f13 - eks pan 2 0.7586799647899636 0.5221511083334797 456d988ecb3bf1b75f057fc6e9fe70db464e9388 - diff --git a/docs6/special-symbols-and-formatting.rst b/docs6/special-symbols-and-formatting.rst index fcb0f797a..8cbb28433 100644 --- a/docs6/special-symbols-and-formatting.rst +++ b/docs6/special-symbols-and-formatting.rst @@ -12,7 +12,7 @@ How can I handle commas-as-data in various formats? .. code-block:: none :emphasize-lines: 1-1 - $ cat commas.csv + cat commas.csv Name,Role "Xiao, Lin",administrator "Khavari, Darius",tester @@ -22,7 +22,7 @@ Likewise :ref:`file-formats-json`: .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsv --ojson cat commas.csv + mlr --icsv --ojson cat commas.csv { "Name": "Xiao, Lin", "Role": "administrator" @@ -37,7 +37,7 @@ For Miller's :ref:`vertical-tabular format ` there is no esca .. code-block:: none :emphasize-lines: 1-1 - $ mlr --icsv --oxtab cat commas.csv + mlr --icsv --oxtab cat commas.csv Name Xiao, Lin Role administrator @@ -49,7 +49,7 @@ But for :ref:`Key-value_pairs ` and :ref:`index-numbered 0.5' then put '$NR = NR' data/small + mlr filter '$x > 0.5' then put '$NR = NR' data/small a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797,NR=2 a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729,NR=5 @@ -92,7 +92,7 @@ The reason is that ``NR`` is computed for the original input records and isn't d .. code-block:: none :emphasize-lines: 1-1 - $ echo x=1,y=2,z=3 | mlr put '$nf1 = NF; $u = 4; $nf2 = NF; unset $x,$y,$z; $nf3 = NF' + echo x=1,y=2,z=3 | mlr put '$nf1 = NF; $u = 4; $nf2 = NF; unset $x,$y,$z; $nf3 = NF' nf1=3,u=4,nf2=5,nf3=3 ``NR``, by contrast (and ``FNR`` as well), retains the value from the original input stream, and records may be dropped by a ``filter`` within a ``then``-chain. To recover consecutive record numbers, you can use out-of-stream variables as follows: @@ -100,7 +100,7 @@ The reason is that ``NR`` is computed for the original input records and isn't d .. code-block:: none :emphasize-lines: 1-11 - $ mlr --opprint --from data/small put ' + mlr --opprint --from data/small put ' begin{ @nr1 = 0 } @nr1 += 1; $nr1 = @nr1 @@ -120,6 +120,6 @@ Or, simply use ``mlr cat -n``: .. code-block:: none :emphasize-lines: 1-1 - $ mlr filter '$x > 0.5' then cat -n data/small + mlr filter '$x > 0.5' then cat -n data/small n=1,a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797 n=2,a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729