Mirrors/miller
1
0
Fork 0
mirror of https://github.com/johnkerl/miller.git synced 2026-08-01 20:12:19 +00:00
Code Issues Projects Releases Wiki Activity

Proofreads for Miller 6 docs

Browse source
This commit is contained in:
John Kerl 2021-08-22 11:35:55 -04:00
parent fa098e82e9
commit 28ba0ba798
146 changed files with 23271 additions and 21615 deletions
Download patch file Download diff file Expand all files Collapse all files

333
docs6/docs/10min.md
View file

@ -8,8 +8,7 @@ You can install Miller for various platforms as follows:
* Linux: `yum install miller` or `apt-get install miller` depending on your flavor of Linux
* MacOS: `brew install miller` or `port install miller` depending on your preference of [Homebrew](https://brew.sh) or [MacPorts](https://macports.org).
* Windows: `choco install miller` using [Chocolatey](https://chocolatey.org).
* You can get latest builds for Linux, MacOS, and Windows by visiting [https://github.com/johnkerl/miller/actions](https://github.com/johnkerl/miller/actions).
, selecting the latest build, and clicking _Artifacts_. (These are retained for 5 days after each commit.)
* You can get latest builds for Linux, MacOS, and Windows by visiting [https://github.com/johnkerl/miller/actions](https://github.com/johnkerl/miller/actions), selecting the latest build, and clicking _Artifacts_. (These are retained for 5 days after each commit.)
* See also the [build page](build.md) if you prefer -- in particular, if your platform's package manager doesn't have the latest release.
As a first check, you should be able to run `mlr --version` at your system's command prompt and see something like the following:
@ -27,34 +26,34 @@ As a second check, given [example.csv](./example.csv) you should be able to do
<b>mlr --csv cat example.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
color,shape,flag,index,quantity,rate
yellow,triangle,true,11,43.6498,9.8870
red,square,true,15,79.2778,0.0130
red,circle,true,16,13.8103,2.9010
red,square,false,48,77.5542,7.4670
purple,triangle,false,51,81.2290,8.5910
red,square,false,64,77.1991,9.5310
purple,triangle,false,65,80.1405,5.8240
yellow,circle,true,73,63.9785,4.2370
yellow,circle,true,87,63.5058,8.3350
purple,square,false,91,72.3735,8.2430
color,shape,flag,k,index,quantity,rate
yellow,triangle,true,1,11,43.6498,9.8870
red,square,true,2,15,79.2778,0.0130
red,circle,true,3,16,13.8103,2.9010
red,square,false,4,48,77.5542,7.4670
purple,triangle,false,5,51,81.2290,8.5910
red,square,false,6,64,77.1991,9.5310
purple,triangle,false,7,65,80.1405,5.8240
yellow,circle,true,8,73,63.9785,4.2370
yellow,circle,true,9,87,63.5058,8.3350
purple,square,false,10,91,72.3735,8.2430
</pre>
<pre class="pre-highlight-in-pair">
<b>mlr --icsv --opprint cat example.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
color shape flag index quantity rate
yellow triangle true 11 43.6498 9.8870
red square true 15 79.2778 0.0130
red circle true 16 13.8103 2.9010
red square false 48 77.5542 7.4670
purple triangle false 51 81.2290 8.5910
red square false 64 77.1991 9.5310
purple triangle false 65 80.1405 5.8240
yellow circle true 73 63.9785 4.2370
yellow circle true 87 63.5058 8.3350
purple square false 91 72.3735 8.2430
color shape flag k index quantity rate
yellow triangle true 1 11 43.6498 9.8870
red square true 2 15 79.2778 0.0130
red circle true 3 16 13.8103 2.9010
red square false 4 48 77.5542 7.4670
purple triangle false 5 51 81.2290 8.5910
red square false 6 64 77.1991 9.5310
purple triangle false 7 65 80.1405 5.8240
yellow circle true 8 73 63.9785 4.2370
yellow circle true 9 87 63.5058 8.3350
purple square false 10 91 72.3735 8.2430
</pre>
If you run into issues on these checks, please check out the resources on the [community page](community.md) for help.
@ -69,17 +68,17 @@ Let's take a quick look at some of the most useful Miller verbs -- file-format-a
<b>mlr --csv cat example.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
color,shape,flag,index,quantity,rate
yellow,triangle,true,11,43.6498,9.8870
red,square,true,15,79.2778,0.0130
red,circle,true,16,13.8103,2.9010
red,square,false,48,77.5542,7.4670
purple,triangle,false,51,81.2290,8.5910
red,square,false,64,77.1991,9.5310
purple,triangle,false,65,80.1405,5.8240
yellow,circle,true,73,63.9785,4.2370
yellow,circle,true,87,63.5058,8.3350
purple,square,false,91,72.3735,8.2430
color,shape,flag,k,index,quantity,rate
yellow,triangle,true,1,11,43.6498,9.8870
red,square,true,2,15,79.2778,0.0130
red,circle,true,3,16,13.8103,2.9010
red,square,false,4,48,77.5542,7.4670
purple,triangle,false,5,51,81.2290,8.5910
red,square,false,6,64,77.1991,9.5310
purple,triangle,false,7,65,80.1405,5.8240
yellow,circle,true,8,73,63.9785,4.2370
yellow,circle,true,9,87,63.5058,8.3350
purple,square,false,10,91,72.3735,8.2430
</pre>
But `mlr cat` can also do format conversion -- for example, you can pretty-print in tabular format:
@ -88,17 +87,17 @@ But `mlr cat` can also do format conversion -- for example, you can pretty-print
<b>mlr --icsv --opprint cat example.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
color shape flag index quantity rate
yellow triangle true 11 43.6498 9.8870
red square true 15 79.2778 0.0130
red circle true 16 13.8103 2.9010
red square false 48 77.5542 7.4670
purple triangle false 51 81.2290 8.5910
red square false 64 77.1991 9.5310
purple triangle false 65 80.1405 5.8240
yellow circle true 73 63.9785 4.2370
yellow circle true 87 63.5058 8.3350
purple square false 91 72.3735 8.2430
color shape flag k index quantity rate
yellow triangle true 1 11 43.6498 9.8870
red square true 2 15 79.2778 0.0130
red circle true 3 16 13.8103 2.9010
red square false 4 48 77.5542 7.4670
purple triangle false 5 51 81.2290 8.5910
red square false 6 64 77.1991 9.5310
purple triangle false 7 65 80.1405 5.8240
yellow circle true 8 73 63.9785 4.2370
yellow circle true 9 87 63.5058 8.3350
purple square false 10 91 72.3735 8.2430
</pre>
`mlr head` and `mlr tail` count records rather than lines. Whether you're getting the first few records or the last few, the CSV header is included either way:
@ -107,22 +106,22 @@ purple square false 91 72.3735 8.2430
<b>mlr --csv head -n 4 example.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
color,shape,flag,index,quantity,rate
yellow,triangle,true,11,43.6498,9.8870
red,square,true,15,79.2778,0.0130
red,circle,true,16,13.8103,2.9010
red,square,false,48,77.5542,7.4670
color,shape,flag,k,index,quantity,rate
yellow,triangle,true,1,11,43.6498,9.8870
red,square,true,2,15,79.2778,0.0130
red,circle,true,3,16,13.8103,2.9010
red,square,false,4,48,77.5542,7.4670
</pre>
<pre class="pre-highlight-in-pair">
<b>mlr --csv tail -n 4 example.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
color,shape,flag,index,quantity,rate
purple,triangle,false,65,80.1405,5.8240
yellow,circle,true,73,63.9785,4.2370
yellow,circle,true,87,63.5058,8.3350
purple,square,false,91,72.3735,8.2430
color,shape,flag,k,index,quantity,rate
purple,triangle,false,7,65,80.1405,5.8240
yellow,circle,true,8,73,63.9785,4.2370
yellow,circle,true,9,87,63.5058,8.3350
purple,square,false,10,91,72.3735,8.2430
</pre>
<pre class="pre-highlight-in-pair">
@ -133,6 +132,7 @@ purple,square,false,91,72.3735,8.2430
"color": "yellow",
"shape": "circle",
"flag": true,
"k": 9,
"index": 87,
"quantity": 63.5058,
"rate": 8.3350
@ -141,6 +141,7 @@ purple,square,false,91,72.3735,8.2430
"color": "purple",
"shape": "square",
"flag": false,
"k": 10,
"index": 91,
"quantity": 72.3735,
"rate": 8.2430
@ -153,17 +154,17 @@ You can sort on a single field:
<b>mlr --icsv --opprint sort -f shape example.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
color shape flag index quantity rate
red circle true 16 13.8103 2.9010
yellow circle true 73 63.9785 4.2370
yellow circle true 87 63.5058 8.3350
red square true 15 79.2778 0.0130
red square false 48 77.5542 7.4670
red square false 64 77.1991 9.5310
purple square false 91 72.3735 8.2430
yellow triangle true 11 43.6498 9.8870
purple triangle false 51 81.2290 8.5910
purple triangle false 65 80.1405 5.8240
color shape flag k index quantity rate
red circle true 3 16 13.8103 2.9010
yellow circle true 8 73 63.9785 4.2370
yellow circle true 9 87 63.5058 8.3350
red square true 2 15 79.2778 0.0130
red square false 4 48 77.5542 7.4670
red square false 6 64 77.1991 9.5310
purple square false 10 91 72.3735 8.2430
yellow triangle true 1 11 43.6498 9.8870
purple triangle false 5 51 81.2290 8.5910
purple triangle false 7 65 80.1405 5.8240
</pre>
Or, you can sort primarily alphabetically on one field, then secondarily numerically descending on another field, and so on:
@ -172,17 +173,17 @@ Or, you can sort primarily alphabetically on one field, then secondarily numeric
<b>mlr --icsv --opprint sort -f shape -nr index example.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
color shape flag index quantity rate
yellow circle true 87 63.5058 8.3350
yellow circle true 73 63.9785 4.2370
red circle true 16 13.8103 2.9010
purple square false 91 72.3735 8.2430
red square false 64 77.1991 9.5310
red square false 48 77.5542 7.4670
red square true 15 79.2778 0.0130
purple triangle false 65 80.1405 5.8240
purple triangle false 51 81.2290 8.5910
yellow triangle true 11 43.6498 9.8870
color shape flag k index quantity rate
yellow circle true 9 87 63.5058 8.3350
yellow circle true 8 73 63.9785 4.2370
red circle true 3 16 13.8103 2.9010
purple square false 10 91 72.3735 8.2430
red square false 6 64 77.1991 9.5310
red square false 4 48 77.5542 7.4670
red square true 2 15 79.2778 0.0130
purple triangle false 7 65 80.1405 5.8240
purple triangle false 5 51 81.2290 8.5910
yellow triangle true 1 11 43.6498 9.8870
</pre>
If there are fields you don't want to see in your data, you can use `cut` to keep only the ones you want, in the same order they appeared in the input data:
@ -229,17 +230,17 @@ You can use `cut -x` to omit fields you don't care about:
<b>mlr --icsv --opprint cut -x -f flag,shape example.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
color index quantity rate
yellow 11 43.6498 9.8870
red 15 79.2778 0.0130
red 16 13.8103 2.9010
red 48 77.5542 7.4670
purple 51 81.2290 8.5910
red 64 77.1991 9.5310
purple 65 80.1405 5.8240
yellow 73 63.9785 4.2370
yellow 87 63.5058 8.3350
purple 91 72.3735 8.2430
color k index quantity rate
yellow 1 11 43.6498 9.8870
red 2 15 79.2778 0.0130
red 3 16 13.8103 2.9010
red 4 48 77.5542 7.4670
purple 5 51 81.2290 8.5910
red 6 64 77.1991 9.5310
purple 7 65 80.1405 5.8240
yellow 8 73 63.9785 4.2370
yellow 9 87 63.5058 8.3350
purple 10 91 72.3735 8.2430
</pre>
You can use `filter` to keep only records you care about:
@ -248,20 +249,20 @@ You can use `filter` to keep only records you care about:
<b>mlr --icsv --opprint filter '$color == "red"' example.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
color shape flag index quantity rate
red square true 15 79.2778 0.0130
red circle true 16 13.8103 2.9010
red square false 48 77.5542 7.4670
red square false 64 77.1991 9.5310
color shape flag k index quantity rate
red square true 2 15 79.2778 0.0130
red circle true 3 16 13.8103 2.9010
red square false 4 48 77.5542 7.4670
red square false 6 64 77.1991 9.5310
</pre>
<pre class="pre-highlight-in-pair">
<b>mlr --icsv --opprint filter '$color == "red" && $flag == true' example.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
color shape flag index quantity rate
red square true 15 79.2778 0.0130
red circle true 16 13.8103 2.9010
color shape flag k index quantity rate
red square true 2 15 79.2778 0.0130
red circle true 3 16 13.8103 2.9010
</pre>
You can use `put` to create new fields which are computed from other fields:
@ -273,17 +274,17 @@ You can use `put` to create new fields which are computed from other fields:
<b>' example.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
color shape flag index quantity rate ratio color_shape
yellow triangle true 11 43.6498 9.8870 4.414868008496004 yellow_triangle
red square true 15 79.2778 0.0130 6098.292307692308 red_square
red circle true 16 13.8103 2.9010 4.760530851430541 red_circle
red square false 48 77.5542 7.4670 10.386259541984733 red_square
purple triangle false 51 81.2290 8.5910 9.455127458968688 purple_triangle
red square false 64 77.1991 9.5310 8.099790158430384 red_square
purple triangle false 65 80.1405 5.8240 13.760388049450551 purple_triangle
yellow circle true 73 63.9785 4.2370 15.09995279679018 yellow_circle
yellow circle true 87 63.5058 8.3350 7.619172165566886 yellow_circle
purple square false 91 72.3735 8.2430 8.779995147397793 purple_square
color shape flag k index quantity rate ratio color_shape
yellow triangle true 1 11 43.6498 9.8870 4.414868008496004 yellow_triangle
red square true 2 15 79.2778 0.0130 6098.292307692308 red_square
red circle true 3 16 13.8103 2.9010 4.760530851430541 red_circle
red square false 4 48 77.5542 7.4670 10.386259541984733 red_square
purple triangle false 5 51 81.2290 8.5910 9.455127458968688 purple_triangle
red square false 6 64 77.1991 9.5310 8.099790158430384 red_square
purple triangle false 7 65 80.1405 5.8240 13.760388049450551 purple_triangle
yellow circle true 8 73 63.9785 4.2370 15.09995279679018 yellow_circle
yellow circle true 9 87 63.5058 8.3350 7.619172165566886 yellow_circle
purple square false 10 91 72.3735 8.2430 8.779995147397793 purple_square
</pre>
Even though Miller's main selling point is name-indexing, sometimes you really want to refer to a field name by its positional index. Use `$[[3]]` to access the name of field 3 or `$[[[3]]]` to access the value of field 3:
@ -292,41 +293,41 @@ Even though Miller's main selling point is name-indexing, sometimes you really w
<b>mlr --icsv --opprint put '$[[3]] = "NEW"' example.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
color shape NEW index quantity rate
yellow triangle true 11 43.6498 9.8870
red square true 15 79.2778 0.0130
red circle true 16 13.8103 2.9010
red square false 48 77.5542 7.4670
purple triangle false 51 81.2290 8.5910
red square false 64 77.1991 9.5310
purple triangle false 65 80.1405 5.8240
yellow circle true 73 63.9785 4.2370
yellow circle true 87 63.5058 8.3350
purple square false 91 72.3735 8.2430
color shape NEW k index quantity rate
yellow triangle true 1 11 43.6498 9.8870
red square true 2 15 79.2778 0.0130
red circle true 3 16 13.8103 2.9010
red square false 4 48 77.5542 7.4670
purple triangle false 5 51 81.2290 8.5910
red square false 6 64 77.1991 9.5310
purple triangle false 7 65 80.1405 5.8240
yellow circle true 8 73 63.9785 4.2370
yellow circle true 9 87 63.5058 8.3350
purple square false 10 91 72.3735 8.2430
</pre>
<pre class="pre-highlight-in-pair">
<b>mlr --icsv --opprint put '$[[[3]]] = "NEW"' example.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
color shape flag index quantity rate
yellow triangle NEW 11 43.6498 9.8870
red square NEW 15 79.2778 0.0130
red circle NEW 16 13.8103 2.9010
red square NEW 48 77.5542 7.4670
purple triangle NEW 51 81.2290 8.5910
red square NEW 64 77.1991 9.5310
purple triangle NEW 65 80.1405 5.8240
yellow circle NEW 73 63.9785 4.2370
yellow circle NEW 87 63.5058 8.3350
purple square NEW 91 72.3735 8.2430
color shape flag k index quantity rate
yellow triangle NEW 1 11 43.6498 9.8870
red square NEW 2 15 79.2778 0.0130
red circle NEW 3 16 13.8103 2.9010
red square NEW 4 48 77.5542 7.4670
purple triangle NEW 5 51 81.2290 8.5910
red square NEW 6 64 77.1991 9.5310
purple triangle NEW 7 65 80.1405 5.8240
yellow circle NEW 8 73 63.9785 4.2370
yellow circle NEW 9 87 63.5058 8.3350
purple square NEW 10 91 72.3735 8.2430
</pre>
You can find the full list of verbs at the [Verbs Reference](reference-verbs.md) page.
## Multiple input files
Miller takes all the files from the command line as an input stream. But it's format-aware, so it doesn't repeat CSV header lines. For example, with input files [data/a.csv](data/a.csv and [data/b.csv](data/b.csv), the system `cat` command will repeat header lines:
Miller takes all the files from the command line as an input stream. But it's format-aware, so it doesn't repeat CSV header lines. For example, with input files [data/a.csv](data/a.csv) and [data/b.csv](data/b.csv), the system `cat` command will repeat header lines:
<pre class="pre-highlight-in-pair">
<b>cat data/a.csv</b>
@ -376,10 +377,10 @@ Often we want to chain queries together -- for example, sorting by a field and t
<b>mlr --csv sort -nr index example.csv | mlr --icsv --opprint head -n 3</b>
</pre>
<pre class="pre-non-highlight-in-pair">
color shape flag index quantity rate
purple square false 91 72.3735 8.2430
yellow circle true 87 63.5058 8.3350
yellow circle true 73 63.9785 4.2370
color shape flag k index quantity rate
purple square false 10 91 72.3735 8.2430
yellow circle true 9 87 63.5058 8.3350
yellow circle true 8 73 63.9785 4.2370
</pre>
This works fine -- but Miller also lets you chain verbs together using the word `then`. Think of this as a Miller-internal pipe that lets you use fewer keystrokes:
@ -388,10 +389,10 @@ This works fine -- but Miller also lets you chain verbs together using the word
<b>mlr --icsv --opprint sort -nr index then head -n 3 example.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
color shape flag index quantity rate
purple square false 91 72.3735 8.2430
yellow circle true 87 63.5058 8.3350
yellow circle true 73 63.9785 4.2370
color shape flag k index quantity rate
purple square false 10 91 72.3735 8.2430
yellow circle true 9 87 63.5058 8.3350
yellow circle true 8 73 63.9785 4.2370
</pre>
As another convenience, you can put the filename first using `--from`. When you're interacting with your data at the command line, this makes it easier to up-arrow and append to the previous command:
@ -400,10 +401,10 @@ As another convenience, you can put the filename first using `--from`. When you'
<b>mlr --icsv --opprint --from example.csv sort -nr index then head -n 3</b>
</pre>
<pre class="pre-non-highlight-in-pair">
color shape flag index quantity rate
purple square false 91 72.3735 8.2430
yellow circle true 87 63.5058 8.3350
yellow circle true 73 63.9785 4.2370
color shape flag k index quantity rate
purple square false 10 91 72.3735 8.2430
yellow circle true 9 87 63.5058 8.3350
yellow circle true 8 73 63.9785 4.2370
</pre>
<pre class="pre-highlight-in-pair">
@ -429,10 +430,10 @@ Here are the records with the top three `index` values:
<b>mlr --icsv --opprint sort -nr index then head -n 3 example.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
color shape flag index quantity rate
purple square false 91 72.3735 8.2430
yellow circle true 87 63.5058 8.3350
yellow circle true 73 63.9785 4.2370
color shape flag k index quantity rate
purple square false 10 91 72.3735 8.2430
yellow circle true 9 87 63.5058 8.3350
yellow circle true 8 73 63.9785 4.2370
</pre>
Lots of Miller commands take a `-g` option for group-by: here, `head -n 1 -g shape` outputs the first record for each distinct value of the `shape` field. This means we're finding the record with highest `index` field for each distinct `shape` field:
@ -441,10 +442,10 @@ Lots of Miller commands take a `-g` option for group-by: here, `head -n 1 -g sha
<b>mlr --icsv --opprint sort -f shape -nr index then head -n 1 -g shape example.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
color shape flag index quantity rate
yellow circle true 87 63.5058 8.3350
purple square false 91 72.3735 8.2430
purple triangle false 65 80.1405 5.8240
color shape flag k index quantity rate
yellow circle true 9 87 63.5058 8.3350
purple square false 10 91 72.3735 8.2430
purple triangle false 7 65 80.1405 5.8240
</pre>
Statistics can be computed with or without group-by field(s):
@ -559,7 +560,7 @@ You can read more about this at the [File Formats](file-formats.md) page.
Often we want to print output to the screen. Miller does this by default, as we've seen in the previous examples.
Sometimes, though, we want to print output to another file. Just use **> outputfilenamegoeshere** at the end of your command:
Sometimes, though, we want to print output to another file. Just use `> outputfilenamegoeshere` at the end of your command:
<pre class="pre-highlight-in-pair">
<b>mlr --icsv --opprint cat example.csv > newfile.csv</b>
@ -586,7 +587,7 @@ yellow circle true 87 63.5058 8.3350
purple square false 91 72.3735 8.2430
</pre>
Other times we just want our files to be **changed in-place**: just use **mlr -I**:
Other times we just want our files to be **changed in-place**: just use `mlr -I`:
<pre class="pre-highlight-non-pair">
<b>cp example.csv newfile.txt</b>
@ -648,29 +649,29 @@ Lastly, using `tee` within `put`, you can split your input data into separate fi
<b>cat circle.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
color,shape,flag,index,quantity,rate
red,circle,true,16,13.8103,2.9010
yellow,circle,true,73,63.9785,4.2370
yellow,circle,true,87,63.5058,8.3350
color,shape,flag,k,index,quantity,rate
red,circle,true,3,16,13.8103,2.9010
yellow,circle,true,8,73,63.9785,4.2370
yellow,circle,true,9,87,63.5058,8.3350
</pre>
<pre class="pre-highlight-in-pair">
<b>cat square.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
color,shape,flag,index,quantity,rate
red,square,true,15,79.2778,0.0130
red,square,false,48,77.5542,7.4670
red,square,false,64,77.1991,9.5310
purple,square,false,91,72.3735,8.2430
color,shape,flag,k,index,quantity,rate
red,square,true,2,15,79.2778,0.0130
red,square,false,4,48,77.5542,7.4670
red,square,false,6,64,77.1991,9.5310
purple,square,false,10,91,72.3735,8.2430
</pre>
<pre class="pre-highlight-in-pair">
<b>cat triangle.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
color,shape,flag,index,quantity,rate
yellow,triangle,true,11,43.6498,9.8870
purple,triangle,false,51,81.2290,8.5910
purple,triangle,false,65,80.1405,5.8240
color,shape,flag,k,index,quantity,rate
yellow,triangle,true,1,11,43.6498,9.8870
purple,triangle,false,5,51,81.2290,8.5910
purple,triangle,false,7,65,80.1405,5.8240
</pre>

9
docs6/docs/10min.md.in
View file

@ -7,8 +7,7 @@ You can install Miller for various platforms as follows:
* Linux: `yum install miller` or `apt-get install miller` depending on your flavor of Linux
* MacOS: `brew install miller` or `port install miller` depending on your preference of [Homebrew](https://brew.sh) or [MacPorts](https://macports.org).
* Windows: `choco install miller` using [Chocolatey](https://chocolatey.org).
* You can get latest builds for Linux, MacOS, and Windows by visiting [https://github.com/johnkerl/miller/actions](https://github.com/johnkerl/miller/actions).
, selecting the latest build, and clicking _Artifacts_. (These are retained for 5 days after each commit.)
* You can get latest builds for Linux, MacOS, and Windows by visiting [https://github.com/johnkerl/miller/actions](https://github.com/johnkerl/miller/actions), selecting the latest build, and clicking _Artifacts_. (These are retained for 5 days after each commit.)
* See also the [build page](build.md) if you prefer -- in particular, if your platform's package manager doesn't have the latest release.
As a first check, you should be able to run `mlr --version` at your system's command prompt and see something like the following:
@ -122,7 +121,7 @@ You can find the full list of verbs at the [Verbs Reference](reference-verbs.md)
## Multiple input files
Miller takes all the files from the command line as an input stream. But it's format-aware, so it doesn't repeat CSV header lines. For example, with input files [data/a.csv](data/a.csv and [data/b.csv](data/b.csv), the system `cat` command will repeat header lines:
Miller takes all the files from the command line as an input stream. But it's format-aware, so it doesn't repeat CSV header lines. For example, with input files [data/a.csv](data/a.csv) and [data/b.csv](data/b.csv), the system `cat` command will repeat header lines:
GENMD_RUN_COMMAND
cat data/a.csv
@ -272,7 +271,7 @@ You can read more about this at the [File Formats](file-formats.md) page.
Often we want to print output to the screen. Miller does this by default, as we've seen in the previous examples.
Sometimes, though, we want to print output to another file. Just use **> outputfilenamegoeshere** at the end of your command:
Sometimes, though, we want to print output to another file. Just use `> outputfilenamegoeshere` at the end of your command:
GENMD_CARDIFY_HIGHLIGHT_ONE
mlr --icsv --opprint cat example.csv > newfile.csv
@ -295,7 +294,7 @@ yellow circle true 87 63.5058 8.3350
purple square false 91 72.3735 8.2430
GENMD_EOF
Other times we just want our files to be **changed in-place**: just use **mlr -I**:
Other times we just want our files to be **changed in-place**: just use `mlr -I`:
GENMD_CARDIFY_HIGHLIGHT_ONE
cp example.csv newfile.txt

10
docs6/docs/build.md
View file

@ -3,6 +3,8 @@
Please also see [Installation](installation.md) for information about pre-built executables.
You will need to first install Go version 1.15 or higher: please see [https://go.dev](https://go.dev).
## Miller license
Two-clause BSD license [https://github.com/johnkerl/miller/blob/master/LICENSE.txt](https://github.com/johnkerl/miller/blob/master/LICENSE.txt).
@ -33,12 +35,12 @@ If you have any build errors, feel free to open an issue with "New Issue" at [ht
These are necessary to produce the `mlr` executable.
* Go version 1.16 or higher
* Go version 1.15 or higher: please see [https://go.dev](https://go.dev)
* Others packaged within `go.mod` and `go.sum` which you don't need to deal with manually -- the Go build process handles them for us
### Optional external dependencies
This documentation pageset is built using Sphinx. Please see [https://github.com/johnkerl/miller/blob/main/docs6/README.md](https://github.com/johnkerl/miller/blob/main/docs6/README.md) for details.
This documentation pageset is built using [https://www.mkdocs.org/](MkDocs). Please see [https://github.com/johnkerl/miller/blob/main/docs6/README.md](https://github.com/johnkerl/miller/blob/main/docs6/README.md) for details.
## Creating a new release: for developers
@ -49,7 +51,6 @@ In this example I am using version 6.1.0 to 6.2.0; of course that will change fo
* Update version found in `mlr --version` and `man mlr`:
* Edit `go/src/version/version.go` from `6.1.0-dev` to `6.2.0`.
* Likewise `docs6/conf.py`
* `cd ../docs6`
* `export PATH=../go:$PATH`
* `make html`
@ -80,6 +81,7 @@ In this example I am using version 6.1.0 to 6.2.0; of course that will change fo
* Social-media updates.
<pre class="pre-non-highlight-non-pair">
# brew notes:
git remote add upstream https://github.com/Homebrew/homebrew-core # one-time setup only
git fetch upstream
git rebase upstream/master
@ -99,7 +101,7 @@ git push -u origin miller-6.1.0
* Afterwork:
* Edit `go/src/version/version.go` and `docs6/conf.py` to change version from `6.2.0` to `6.2.0-dev`.
* Edit `go/src/version/version.go` to change version from `6.2.0` to `6.2.0-dev`.
* `cd go`
* `./build`
* Commit and push.

10
docs6/docs/build.md.in
View file

@ -2,6 +2,8 @@
Please also see [Installation](installation.md) for information about pre-built executables.
You will need to first install Go version 1.15 or higher: please see [https://go.dev](https://go.dev).
## Miller license
Two-clause BSD license [https://github.com/johnkerl/miller/blob/master/LICENSE.txt](https://github.com/johnkerl/miller/blob/master/LICENSE.txt).
@ -32,12 +34,12 @@ If you have any build errors, feel free to open an issue with "New Issue" at [ht
These are necessary to produce the `mlr` executable.
* Go version 1.16 or higher
* Go version 1.15 or higher: please see [https://go.dev](https://go.dev)
* Others packaged within `go.mod` and `go.sum` which you don't need to deal with manually -- the Go build process handles them for us
### Optional external dependencies
This documentation pageset is built using Sphinx. Please see [https://github.com/johnkerl/miller/blob/main/docs6/README.md](https://github.com/johnkerl/miller/blob/main/docs6/README.md) for details.
This documentation pageset is built using [https://www.mkdocs.org/](MkDocs). Please see [https://github.com/johnkerl/miller/blob/main/docs6/README.md](https://github.com/johnkerl/miller/blob/main/docs6/README.md) for details.
## Creating a new release: for developers
@ -48,7 +50,6 @@ In this example I am using version 6.1.0 to 6.2.0; of course that will change fo
* Update version found in `mlr --version` and `man mlr`:
* Edit `go/src/version/version.go` from `6.1.0-dev` to `6.2.0`.
* Likewise `docs6/conf.py`
* `cd ../docs6`
* `export PATH=../go:$PATH`
* `make html`
@ -79,6 +80,7 @@ In this example I am using version 6.1.0 to 6.2.0; of course that will change fo
* Social-media updates.
GENMD_CARDIFY
# brew notes:
git remote add upstream https://github.com/Homebrew/homebrew-core # one-time setup only
git fetch upstream
git rebase upstream/master
@ -98,7 +100,7 @@ GENMD_EOF
* Afterwork:
* Edit `go/src/version/version.go` and `docs6/conf.py` to change version from `6.2.0` to `6.2.0-dev`.
* Edit `go/src/version/version.go` to change version from `6.2.0` to `6.2.0-dev`.
* `cd go`
* `./build`
* Commit and push.

8
docs6/docs/circle.csv
View file

@ -1,4 +1,4 @@
color,shape,flag,index,quantity,rate
red,circle,true,16,13.8103,2.9010
yellow,circle,true,73,63.9785,4.2370
yellow,circle,true,87,63.5058,8.3350
color,shape,flag,k,index,quantity,rate
red,circle,true,3,16,13.8103,2.9010
yellow,circle,true,8,73,63.9785,4.2370
yellow,circle,true,9,87,63.5058,8.3350

1 color shape flag k index quantity rate
2 red circle true 3 16 13.8103 2.9010
3 yellow circle true 8 73 63.9785 4.2370
4 yellow circle true 9 87 63.5058 8.3350

2
docs6/docs/contributing.md
View file

@ -29,7 +29,7 @@ Once PRs are merged, readthedocs creates [https://miller.readthedocs.io](https:/
As of Miller-6's current pre-release status, the best way to test is to either build from source via [Building from source](build.md), or by getting a recent binary at [https://github.com/johnkerl/miller/actions](https://github.com/johnkerl/miller/actions), then click latest build, then *Artifacts*. Then simply use Miller for whatever you do, and create an issue at [https://github.com/johnkerl/miller/issues](https://github.com/johnkerl/miller/issues).
Do note that as of 2021-06-17 a few things have not been ported to Miller 6 -- most notably, including regex captures and localtime DSL functions.
Do note that as of 2021-06-17 a few things have not been ported to Miller 6 -- most notably, including localtime DSL functions and other issues.
## Feature development

2
docs6/docs/contributing.md.in
View file

@ -28,7 +28,7 @@ Once PRs are merged, readthedocs creates [https://miller.readthedocs.io](https:/
As of Miller-6's current pre-release status, the best way to test is to either build from source via [Building from source](build.md), or by getting a recent binary at [https://github.com/johnkerl/miller/actions](https://github.com/johnkerl/miller/actions), then click latest build, then *Artifacts*. Then simply use Miller for whatever you do, and create an issue at [https://github.com/johnkerl/miller/issues](https://github.com/johnkerl/miller/issues).
Do note that as of 2021-06-17 a few things have not been ported to Miller 6 -- most notably, including regex captures and localtime DSL functions.
Do note that as of 2021-06-17 a few things have not been ported to Miller 6 -- most notably, including localtime DSL functions and other issues.
## Feature development

20
docs6/docs/csv-with-and-without-headers.md
View file

@ -48,22 +48,22 @@ Likewise, if you need to produce CSV which is lacking its header, you can pipe M
</pre>
<pre class="pre-non-highlight-in-pair">
color,shape,flag,i,u,v,w,x
yellow,triangle,1,11,0.6321695890307647,0.9887207810889004,0.4364983936735774,5.7981881667050565
red,square,1,15,0.21966833570651523,0.001257332190235938,0.7927778364718627,2.944117399716207
red,circle,1,16,0.20901671281497636,0.29005231936593445,0.13810280912907674,5.065034003400998
red,square,0,48,0.9562743938458542,0.7467203085342884,0.7755423050923582,7.117831369597269
purple,triangle,0,51,0.4355354501763202,0.8591292672156728,0.8122903963006748,5.753094629505863
yellow,triangle,1,56,0.6321695890307647,0.9887207810889004,0.4364983936735774,5.7981881667050565
red,square,1,80,0.21966833570651523,0.001257332190235938,0.7927778364718627,2.944117399716207
red,circle,1,84,0.20901671281497636,0.29005231936593445,0.13810280912907674,5.065034003400998
red,square,0,243,0.9562743938458542,0.7467203085342884,0.7755423050923582,7.117831369597269
purple,triangle,0,257,0.4355354501763202,0.8591292672156728,0.8122903963006748,5.753094629505863
</pre>
<pre class="pre-highlight-in-pair">
<b>head -5 data/colored-shapes.dkvp | mlr --ocsv --headerless-csv-output cat</b>
</pre>
<pre class="pre-non-highlight-in-pair">
yellow,triangle,1,11,0.6321695890307647,0.9887207810889004,0.4364983936735774,5.7981881667050565
red,square,1,15,0.21966833570651523,0.001257332190235938,0.7927778364718627,2.944117399716207
red,circle,1,16,0.20901671281497636,0.29005231936593445,0.13810280912907674,5.065034003400998
red,square,0,48,0.9562743938458542,0.7467203085342884,0.7755423050923582,7.117831369597269
purple,triangle,0,51,0.4355354501763202,0.8591292672156728,0.8122903963006748,5.753094629505863
yellow,triangle,1,56,0.6321695890307647,0.9887207810889004,0.4364983936735774,5.7981881667050565
red,square,1,80,0.21966833570651523,0.001257332190235938,0.7927778364718627,2.944117399716207
red,circle,1,84,0.20901671281497636,0.29005231936593445,0.13810280912907674,5.065034003400998
red,square,0,243,0.9562743938458542,0.7467203085342884,0.7755423050923582,7.117831369597269
purple,triangle,0,257,0.4355354501763202,0.8591292672156728,0.8122903963006748,5.753094629505863
</pre>
Lastly, often we say "CSV" or "TSV" when we have positionally indexed data in columns which are separated by commas or tabs, respectively. In this case it's perhaps simpler to **just use NIDX format** which was designed for this purpose. (See also [File Formats](file-formats.md).) For example:

6
docs6/docs/customization.md
View file

@ -29,13 +29,13 @@ Then you can just type things like
<b>mlr sort -n id mydata.csv</b>
</pre>
and the `--csv` part will automatically be understood. (If you do want to process, say, a JSON file then `mlr --json ...` at the command line will override the default from your `.mlrrc`.)
and the `--csv` part will automatically be understood. If you do want to process, say, a JSON file then `mlr --json ...` at the command line will still override the defaults you've placed in your `.mlrrc`.
## What you can put in your .mlrrc
* You can include any command-line flags, except the "terminal" ones such as `--help`.
* The `--prepipe`, `--load`, and `--mload` flags aren't allowed in `.mlrrc` as they control code execution, and could result in your scripts running things you don't expect if you receive data from someone with a `.mlrrc` in it.
* The `--prepipe`, `--load`, and `--mload` flags aren't allowed in `.mlrrc` as they control code execution, and could result in your scripts running things you don't expect if you receive data from someone with a `./.mlrrc` in it.
* The formatting rule is you need to put one flag beginning with `--` per line: for example, `--csv` on one line and `--nr-progress-mod 1000` on a separate line.
@ -83,4 +83,4 @@ Otherwise:
* If `./.mlrrc` exists, it's then also processed as above.
* The idea is you can have all your settings in your `$HOME/.mlrrc`, then override maybe one or two for your current directory if you like.
* The idea is you can have all your settings in your `$HOME/.mlrrc`, then maybe more project-specific settings for your current directory if you like.

6
docs6/docs/customization.md.in
View file

@ -28,13 +28,13 @@ GENMD_CARDIFY_HIGHLIGHT_ONE
mlr sort -n id mydata.csv
GENMD_EOF
and the `--csv` part will automatically be understood. (If you do want to process, say, a JSON file then `mlr --json ...` at the command line will override the default from your `.mlrrc`.)
and the `--csv` part will automatically be understood. If you do want to process, say, a JSON file then `mlr --json ...` at the command line will still override the defaults you've placed in your `.mlrrc`.
## What you can put in your .mlrrc
* You can include any command-line flags, except the "terminal" ones such as `--help`.
* The `--prepipe`, `--load`, and `--mload` flags aren't allowed in `.mlrrc` as they control code execution, and could result in your scripts running things you don't expect if you receive data from someone with a `.mlrrc` in it.
* The `--prepipe`, `--load`, and `--mload` flags aren't allowed in `.mlrrc` as they control code execution, and could result in your scripts running things you don't expect if you receive data from someone with a `./.mlrrc` in it.
* The formatting rule is you need to put one flag beginning with `--` per line: for example, `--csv` on one line and `--nr-progress-mod 1000` on a separate line.
@ -66,4 +66,4 @@ Otherwise:
* If `./.mlrrc` exists, it's then also processed as above.
* The idea is you can have all your settings in your `$HOME/.mlrrc`, then override maybe one or two for your current directory if you like.
* The idea is you can have all your settings in your `$HOME/.mlrrc`, then maybe more project-specific settings for your current directory if you like.

4
docs6/docs/data-cleaning-examples.md
View file

@ -1,7 +1,7 @@
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
# Data-cleaning examples
Here are some ways to use the type-checking options as described in [Type-checking](reference-dsl-variables.md#type-checking). Suppose you have the following data file, with inconsistent typing for boolean. (Also imagine that, for the sake of discussion, we have a million-line file rather than a four-line file, so we can't see it all at once and some automation is called for.)
Here are some ways to use the type-checking options as described in the [Type-checking page](reference-dsl-variables.md#type-checking). Suppose you have the following data file, with inconsistent typing for boolean. (Also imagine that, for the sake of discussion, we have a million-line file rather than a four-line file, so we can't see it all at once and some automation is called for.)
<pre class="pre-highlight-in-pair">
<b>cat data/het-bool.csv</b>
@ -70,7 +70,7 @@ fred true
wilma 1
</pre>
A third way is to abort the process on fimd.instance of bad data:
A third way is to abort the process on first instance of bad data:
<pre class="pre-highlight-in-pair">
<b>mlr --csv put '$reachable = asserting_string($reachable)' data/het-bool.csv</b>

4
docs6/docs/data-cleaning-examples.md.in
View file

@ -1,6 +1,6 @@
# Data-cleaning examples
Here are some ways to use the type-checking options as described in [Type-checking](reference-dsl-variables.md#type-checking). Suppose you have the following data file, with inconsistent typing for boolean. (Also imagine that, for the sake of discussion, we have a million-line file rather than a four-line file, so we can't see it all at once and some automation is called for.)
Here are some ways to use the type-checking options as described in the [Type-checking page](reference-dsl-variables.md#type-checking). Suppose you have the following data file, with inconsistent typing for boolean. (Also imagine that, for the sake of discussion, we have a million-line file rather than a four-line file, so we can't see it all at once and some automation is called for.)
GENMD_RUN_COMMAND
cat data/het-bool.csv
@ -30,7 +30,7 @@ mlr --icsv --opprint put '
' data/het-bool.csv
GENMD_EOF
A third way is to abort the process on fimd.instance of bad data:
A third way is to abort the process on first instance of bad data:
GENMD_RUN_COMMAND_TOLERATING_ERROR
mlr --csv put '$reachable = asserting_string($reachable)' data/het-bool.csv

21
docs6/docs/data-diving-examples.md
View file

@ -8,13 +8,18 @@ The [flins.csv](data/flins.csv) file is some sample data obtained from [https://
Vertical-tabular format is good for a quick look at CSV data layout -- seeing what columns you have to work with:
<pre class="pre-highlight-in-pair">
<b>head -n 2 data/flins.csv | mlr --icsv --oxtab cat</b>
<b>mlr --icsv --oxtab --from data/flins.csv head -n 2</b>
</pre>
<pre class="pre-non-highlight-in-pair">
county Seminole
tiv_2011 22890.55
tiv_2012 20848.71
line Residential
county Miami Dade
tiv_2011 1158674.85
tiv_2012 1076001.08
line Residential
</pre>
A few simple queries:
@ -122,13 +127,13 @@ Peek at the data:
<b>head -n 6 data/colored-shapes.dkvp | mlr --opprint cat</b>
</pre>
<pre class="pre-non-highlight-in-pair">
color shape flag i u v w x
yellow triangle 1 11 0.6321695890307647 0.9887207810889004 0.4364983936735774 5.7981881667050565
red square 1 15 0.21966833570651523 0.001257332190235938 0.7927778364718627 2.944117399716207
red circle 1 16 0.20901671281497636 0.29005231936593445 0.13810280912907674 5.065034003400998
red square 0 48 0.9562743938458542 0.7467203085342884 0.7755423050923582 7.117831369597269
purple triangle 0 51 0.4355354501763202 0.8591292672156728 0.8122903963006748 5.753094629505863
red square 0 64 0.2015510269821953 0.9531098083420033 0.7719912015786777 5.612050466474166
color shape flag i u v w x
yellow triangle 1 56 0.6321695890307647 0.9887207810889004 0.4364983936735774 5.7981881667050565
red square 1 80 0.21966833570651523 0.001257332190235938 0.7927778364718627 2.944117399716207
red circle 1 84 0.20901671281497636 0.29005231936593445 0.13810280912907674 5.065034003400998
red square 0 243 0.9562743938458542 0.7467203085342884 0.7755423050923582 7.117831369597269
purple triangle 0 257 0.4355354501763202 0.8591292672156728 0.8122903963006748 5.753094629505863
red square 0 322 0.2015510269821953 0.9531098083420033 0.7719912015786777 5.612050466474166
</pre>
Look at uncategorized stats (using [creach](https://github.com/johnkerl/scripts/blob/master/fundam/creach) for spacing).

2
docs6/docs/data-diving-examples.md.in
View file

@ -7,7 +7,7 @@ The [flins.csv](data/flins.csv) file is some sample data obtained from [https://
Vertical-tabular format is good for a quick look at CSV data layout -- seeing what columns you have to work with:
GENMD_RUN_COMMAND
head -n 2 data/flins.csv | mlr --icsv --oxtab cat
mlr --icsv --oxtab --from data/flins.csv head -n 2
GENMD_EOF
A few simple queries:

20156
docs6/docs/data/colored-shapes.csv
View file

File diff suppressed because it is too large Load diff

20156
docs6/docs/data/colored-shapes.dkvp
View file

File diff suppressed because it is too large Load diff

11
docs6/docs/data/dynamic-nr.sh
View file

@ -1,11 +0,0 @@
mlr --opprint --from data/small put '
begin{ @nr1 = 0 }
@nr1 += 1;
$nr1 = @nr1
' \
then filter '$x>0.5' \
then put '
begin{ @nr2 = 0 }
@nr2 += 1;
$nr2 = @nr2
'

23
docs6/docs/data/het.json Normal file
View file

@ -0,0 +1,23 @@
{
"resource": "/path/to/file",
"loadsec": 0.45,
"ok": true
}
{
"record_count": 100,
"resource": "/path/to/file"
}
{
"resource": "/path/to/second/file",
"loadsec": 0.32,
"ok": true
}
{
"record_count": 150,
"resource": "/path/to/second/file"
}
{
"resource": "/some/other/path",
"loadsec": 0.97,
"ok": false
}

4
docs6/docs/data/short.csv Normal file
View file

@ -0,0 +1,4 @@
word,value
apple,37
ball,28
cat,54
1 word value
2 apple 37
3 ball 28
4 cat 54

13
docs6/docs/data/then-chaining-performance.txt
View file

@ -2,15 +2,14 @@
mlr cut -x -f i,y data/big | mlr sort -n y > /dev/null
% time sh piped.sh
real 0m2.828s
user 0m3.183s
sys 0m0.137s
real 0m2.321s
user 0m4.878s
sys 0m1.564s
% cat chained.sh
mlr cut -x -f i,y then sort -n y data/big > /dev/null
% time sh chained.sh
real 0m2.082s
user 0m1.933s
sys 0m0.137s
real 0m2.070s
user 0m2.738s
sys 0m1.259s

16
docs6/docs/data/two-pass-percentage.sh
View file

@ -1,16 +0,0 @@
mlr --from data/small --opprint put -q '
# These are executed once per record, which is the first pass.
# The key is to use NR to index an out-of-stream variable to
# retain all the x-field values.
@x_min = min($x, @x_min);
@x_max = max($x, @x_max);
@x[NR] = $x;
# The second pass is in a for-loop in an end-block.
end {
for (nr, x in @x) {
@x_pct[nr] = 100 * (x - @x_min) / (@x_max - @x_min);
}
emit (@x, @x_pct), "NR"
}
'

11
docs6/docs/data/two-pass-record-numbers.sh
View file

@ -1,11 +0,0 @@
mlr --opprint --from data/small put -q '
@records[NR] = $*;
end {
for((I,k),v in @records) {
@records[I]["I"] = I;
@records[I]["N"] = NR;
@records[I]["PCT"] = 100*I/NR
}
emit @records,"I"
}
' then reorder -f I,N,PCT

75
docs6/docs/data/type-decl-example.mlr
View file

@ -1,37 +1,38 @@
subr s(a, str b, int c) { # a is implicitly var (untyped).
# b is explicitly str.
# c is explicitly int.
# The type-checking is done at the callsite
# when arguments are bound to parameters.
#
var b = 100; # error # Re-declaration in the same scope is disallowed.
int n = 10; # Declaration of variable local to the subroutine.
n = 20; # Assignment is OK.
int n = 30; # error # Re-declaration in the same scope is disallowed.
str n = "abc"; # error # Re-declaration in the same scope is disallowed.
#
float f1 = 1; # error # 1 is an int, not a float.
float f2 = 2.0; # 2.0 is a float.
num f3 = 3; # 3 is a num.
num f4 = 4.0; # 4.0 is a num.
} #
#
call s(1, 2, 3); # Type-assertion '3 is int' is done here at the callsite.
#
k = "def"; # Top-level variable k.
#
for (str k, v in $*) { # k and v are bound here, masking outer k.
print k . ":" . v; # k is explicitly str; v is implicitly var.
} #
#
print "k is".k; # k at this scope level is still "def".
print "v is".v; # v is undefined in this scope.
#
i = -1; #
for (i = 1, int j = 2; i <= 10; i += 1, j *= 2) { # C-style triple-for variables use enclosing scope, unless
# declared local: i is outer, j is local to the loop.
print "inner i =" . i; #
print "inner j =" . j; #
} #
print "outer i =" . i; # i has been modified by the loop.
print "outer j =" . j; # j is undefined in this scope.
subr s(a, str b, int c) { # a is implicitly var (untyped).
# b is explicitly str.
# c is explicitly int.
# The type-checking is done at the callsite
# when arguments are bound to parameters.
#
var b = 100; # error # Re-declaration in the same scope is disallowed.
int n = 10; # Declaration of variable local to the subroutine.
n = 20; # Assignment is OK.
int n = 30; # error # Re-declaration in the same scope is disallowed.
str n = "abc"; # error # Re-declaration in the same scope is disallowed.
#
float f1 = 1; # error # 1 is an int, not a float.
float f2 = 2.0; # 2.0 is a float.
num f3 = 3; # 3 is a num.
num f4 = 4.0; # 4.0 is a num.
} #
#
call s(1, 2, 3); # Type-assertion '3 is int' is done here at the callsite.
#
k = "def"; # Top-level variable k.
#
for (str k, v in $*) { # k and v are bound here, masking outer k.
print k . ":" . v; # k is explicitly str; v is implicitly var.
} #
#
print "k is".k; # k at this scope level is still "def".
print "v is".v; # v is undefined in this scope.
#
i = -1; #
for (i = 1, int j = 2; i <= 10; i += 1, j *= 2) {
# C-style triple-for variables use enclosing scope,
# unless declared local: i is outer, j is local to the loop.
print "inner i =" . i; #
print "inner j =" . j; #
} #
print "outer i =" . i; # i has been modified by the loop.
print "outer j =" . j; # j is undefined in this scope.

22
docs6/docs/example.csv
View file

@ -1,11 +1,11 @@
color,shape,flag,index,quantity,rate
yellow,triangle,true,11,43.6498,9.8870
red,square,true,15,79.2778,0.0130
red,circle,true,16,13.8103,2.9010
red,square,false,48,77.5542,7.4670
purple,triangle,false,51,81.2290,8.5910
red,square,false,64,77.1991,9.5310
purple,triangle,false,65,80.1405,5.8240
yellow,circle,true,73,63.9785,4.2370
yellow,circle,true,87,63.5058,8.3350
purple,square,false,91,72.3735,8.2430
color,shape,flag,k,index,quantity,rate
yellow,triangle,true,1,11,43.6498,9.8870
red,square,true,2,15,79.2778,0.0130
red,circle,true,3,16,13.8103,2.9010
red,square,false,4,48,77.5542,7.4670
purple,triangle,false,5,51,81.2290,8.5910
red,square,false,6,64,77.1991,9.5310
purple,triangle,false,7,65,80.1405,5.8240
yellow,circle,true,8,73,63.9785,4.2370
yellow,circle,true,9,87,63.5058,8.3350
purple,square,false,10,91,72.3735,8.2430

1 color shape flag k index quantity rate
2 yellow triangle true 1 11 43.6498 9.8870
3 red square true 2 15 79.2778 0.0130
4 red circle true 3 16 13.8103 2.9010
5 red square false 4 48 77.5542 7.4670
6 purple triangle false 5 51 81.2290 8.5910
7 red square false 6 64 77.1991 9.5310
8 purple triangle false 7 65 80.1405 5.8240
9 yellow circle true 8 73 63.9785 4.2370
10 yellow circle true 9 87 63.5058 8.3350
11 purple square false 10 91 72.3735 8.2430

91
docs6/docs/extra.css
View file

@ -10,32 +10,46 @@
/* Top-of-page banner */
.md-header {
background-color: #800000;
background-color: #800000;
}
.md-nav--primary .md-nav__title[for="__drawer"] {
color: #ffffff;
background-color: #800000;
padding: 3px;
border-radius: 6px;
.md-footer {
background-color: #e0e0e0;
}
.md-footer {
background-color: #e0e0e0;
.md-nav--primary .md-nav__title[for="__drawer"] {
color: #ffffff;
background-color: #800000;
padding: 3px;
border-radius: 6px;
}
/* Section headings in the navbar: bold them and stand them out a little */
.md-nav__item--nested > .md-nav__link {
color: inherit;
font-weight: bold;
margin: 3px;
padding: 3px;
/*border: 1px solid white;*/
color: inherit;
font-weight: bold;
margin: 3px;
padding: 3px;
/*border: 1px solid white;*/
}
.md-nav__link:hover {
/*border: 1px solid gray;*/
background-color: #e0e0e0;
}
/* Centered quicklinks for Introduction page */
div.quicklinks {
text-align: center;
overflow: hidden;
}
span.quicklinks {
border: 1px solid #e0e0e0;
border-radius: 6px;
padding: 6px;
text-align: center;
display:table;
margin:0 auto;
}
/* Code samples using <pre> (which is currently in use) */
/* For command-and-output situations (the most common) -- zero margin between.
* Example:
@ -49,8 +63,12 @@
* docs/genmd-filter script when it produces the docs/*.md files from their
* respective docs/*.md.in files.
*/
.md-typeset code {
font-size: 12px;
}
pre {
color: #000000;
font-size: 12px;
padding: 10px;
@ -58,13 +76,11 @@ pre {
white-space: -moz-pre-wrap; /* Mozilla, since 1999 */
white-space: -pre-wrap; /* Opera 4-6 */
white-space: -o-pre-wrap; /* Opera 7 */
word-wrap: break-word; /* Internet Explorer 5.5+ */
}
.md-typeset code, .md-typeset kbd, .md-typeset pre {
color: #000000;
/*background-color: #e0e0e0;*/
background-color: #eae2cb;
/*background-color: #c5b690;*/
font-size: 12px;
direction: ltr;
border-radius: 3px;
@ -122,20 +138,20 @@ pre.pre-non-highlight-non-pair {
border-bottom-left-radius: 6px;
}
/* Section titles in content pages */
h1 {
font-weight: bold;
pre.pre-no-wrap {
font-size: 12px;
padding: 10px;
white-space: nowrap; /* css-3 */
white-space: -moz-nowrap; /* Mozilla, since 1999 */
white-space: -no-wrap; /* Opera 4-6 */
white-space: -o-nowrap; /* Opera 7 */
}
/* Section titles in content pages */
.md-typeset h1 {
font-weight: bold;
}
h1, h2, h3 {
color: #800000;
font-weight: bold;
border-radius: 6px;
padding: 6px;
background-color: #e0e0e0;
}
.md-typeset h1, .md-typeset h2, .md-typeset h3 {
color: #800000;
font-weight: bold;
@ -145,13 +161,16 @@ h1, h2, h3 {
}
/* Hyperlinks in content pages */
a, a:before, a:focus, a:hover, a:visited {
a, a:before, a:focus, a:hover, a:link, a:visited {
color: #800000;
}
.md-typeset a {
color: #800000;
}
.md-typeset a:link {
color: #800000;
}
.md-typeset a:visited {
color: #800000;
}
@ -180,6 +199,7 @@ a, a:before, a:focus, a:hover, a:visited {
.md-nav__link[data-md-state="blur"] {
color: #800000;
}
nav a:hover {
border-radius: 3px;
background-color: #e0e0e0;
@ -189,3 +209,18 @@ nav a:hover {
img {
border-radius: 6px;
}
.md-footer__link:hover {
background-color: #f0f0f0;
}
:root > * {
--md-code-fg-color: #000000;
--md-code-bg-color: #eae2cb;
--md-typeset-color: #000000;
--md-typeset-a-color: #800000;
--md-footer-fg-color: #800000;
--md-footer-fg-color: #eae2cb;
}

6
docs6/docs/feature-comparison.md
View file

@ -58,11 +58,9 @@ Likewise with `mlr sort`, `mlr tac`, and so on.
* Like `awk`, Miller (as of v5.0.0) allows you to define new functions within its `put` and `filter` expression language. Further programmability comes from chaining with `then`.
* As with `awk`, `$`-variables are stream variables and all verbs (such as `cut`, `stats1`, `put`, etc.) as well as `put`/`filter` statements operate on streams. This means that you define actions to be done on each record and then stream your data through those actions. The built-in variables `NF`, `NR`, etc. change from one line to another, `$x` is a label for field `x` in the current record, and the input to `sqrt($x)` changes from one record to the next. The expression language for the `put` and `filter` verbs additionally allows you to define `begin {...}` and `end {...}` blocks for actions to be taken before and after records are processed, respectively.
* As with `awk`, `$`-variables are stream variables and all verbs (such as `cut`, `stats1`, `put`, etc.) as well as `put`/`filter` statements operate on streams. This means that you define actions to be done on each record and then stream your data through those actions. The built-in variables `NF`, `NR`, etc. change from one record to another, `$x` is a label for field `x` in the current record, and the input to `sqrt($x)` changes from one record to the next. The expression language for the `put` and `filter` verbs additionally allows you to define `begin {...}` and `end {...}` blocks for actions to be taken before and after records are processed, respectively.
* As with `awk`, Miller's `put`/`filter` language lets you set `@sum=0` before records are read, then update that sum on each record, then print its value at the end. Unlike `awk`, Miller makes syntactically explicit the difference between variables with extent across all records (names starting with `@`, such as `@sum`) and variables which are local to the current expression (names starting without `@`, such as `sum`).
* Miller can be faster than `awk`, `cut`, and so on, depending on platform; see also [Performance](performance.md). In particular, Miller's DSL syntax is parsed into Go control structures at startup time, with the bulk data-stream processing all done in Go.
* As with `awk`, Miller's `put`/`filter` language lets you set `@sum=0` before records are read, then update that sum on each record, then print its value at the end. Unlike `awk`, Miller makes syntactically explicit the difference between variables with extent across all records (names starting with `@`, such as `@sum`) and variables which are local to the current expression invocation (names starting without `@`, such as `sum`).
## See also

6
docs6/docs/feature-comparison.md.in
View file

@ -36,11 +36,9 @@ Likewise with `mlr sort`, `mlr tac`, and so on.
* Like `awk`, Miller (as of v5.0.0) allows you to define new functions within its `put` and `filter` expression language. Further programmability comes from chaining with `then`.
* As with `awk`, `$`-variables are stream variables and all verbs (such as `cut`, `stats1`, `put`, etc.) as well as `put`/`filter` statements operate on streams. This means that you define actions to be done on each record and then stream your data through those actions. The built-in variables `NF`, `NR`, etc. change from one line to another, `$x` is a label for field `x` in the current record, and the input to `sqrt($x)` changes from one record to the next. The expression language for the `put` and `filter` verbs additionally allows you to define `begin {...}` and `end {...}` blocks for actions to be taken before and after records are processed, respectively.
* As with `awk`, `$`-variables are stream variables and all verbs (such as `cut`, `stats1`, `put`, etc.) as well as `put`/`filter` statements operate on streams. This means that you define actions to be done on each record and then stream your data through those actions. The built-in variables `NF`, `NR`, etc. change from one record to another, `$x` is a label for field `x` in the current record, and the input to `sqrt($x)` changes from one record to the next. The expression language for the `put` and `filter` verbs additionally allows you to define `begin {...}` and `end {...}` blocks for actions to be taken before and after records are processed, respectively.
* As with `awk`, Miller's `put`/`filter` language lets you set `@sum=0` before records are read, then update that sum on each record, then print its value at the end. Unlike `awk`, Miller makes syntactically explicit the difference between variables with extent across all records (names starting with `@`, such as `@sum`) and variables which are local to the current expression (names starting without `@`, such as `sum`).
* Miller can be faster than `awk`, `cut`, and so on, depending on platform; see also [Performance](performance.md). In particular, Miller's DSL syntax is parsed into Go control structures at startup time, with the bulk data-stream processing all done in Go.
* As with `awk`, Miller's `put`/`filter` language lets you set `@sum=0` before records are read, then update that sum on each record, then print its value at the end. Unlike `awk`, Miller makes syntactically explicit the difference between variables with extent across all records (names starting with `@`, such as `@sum`) and variables which are local to the current expression invocation (names starting without `@`, such as `sum`).
## See also

13
docs6/docs/file-formats.md
View file

@ -156,7 +156,7 @@ resource=/some/other/path,loadsec=0.97,ok=false
etc. and I just log them as needed. Then later, I can use `grep`, `mlr --opprint group-like`, etc.
to analyze my logs.
See [Reference: I/O options](reference-main-io-options.md) regarding how to specify separators other than the default equals-sign and comma.
See the [I/O options reference](reference-main-io-options.md) regarding how to specify separators other than the default equals-sign and comma.
## NIDX: Index-numbered (toolkit style)
@ -423,7 +423,7 @@ JSON isn't a parameterized format, so `RS`, `FS`, `PS` aren't specifiable. Nonet
* Use `--jquoteall` to double-quote all object values. By default, integers, floating-point numbers, and booleans `true` and `false` are not double-quoted when they appear as JSON-object keys.
* Use `--jflatsep youmd.inghere` to specify the string used for key concatenation: this defaults to a single colon.
* Use `--jflatsep yourseparatorhere` to specify the string used for key concatenation: this defaults to a single dot.
Again, please see [jq](https://stedolan.github.io/jq/) for a truly powerful, JSON-specific tool.
@ -517,9 +517,10 @@ _networkd * 24 24 Network Services /var/networkd /usr/bin/false
<pre class="pre-highlight-in-pair">
<b>$ grep -v '^#' /etc/passwd | head -n 2 | \</b>
<b> mlr --nidx --fs : --ojson --jvstack --jlistwrap \</b>
<b> label name,password,uid,gid,gecos,home_dir,shell</b>
</pre>
<pre class="pre-non-highlight-in-pair">
mlr --nidx --fs : --ojson --jvstack --jlistwrap label name,password,uid,gid,gecos,home_dir,shell
[
{
"name": "nobody",
@ -529,8 +530,8 @@ _networkd * 24 24 Network Services /var/networkd /usr/bin/false
"gecos": "Unprivileged User",
"home_dir": "/var/empty",
"shell": "/usr/bin/false"
}
,{
},
{
"name": "root",
"password": "*",
"uid": 0,
@ -596,7 +597,7 @@ The line-ending autodetector triggers on the first line ending detected in the i
If you use `--ors {something else}` with (default or explicitly specified) `--irs auto` then line endings are autodetected on input and set to what you specify on output.
If you use `--irs {something else}` with (default or explicitly specified) `--ors auto` then the output line endings used are LF on Unix/Linux/BSD/MacOSX, and CRLF on Windows.
If you use `--irs {something else}` with (default or explicitly specified) `--ors auto` then the output line endings used are LF on Unix/Linux/BSD/MacOS X, and CRLF on Windows.
See also [Record/field/pair separators](reference-main-io-options.md#recordfieldpair-separators) for more information about record/field/pair separators.

15
docs6/docs/file-formats.md.in
View file

@ -86,7 +86,7 @@ GENMD_EOF
etc. and I just log them as needed. Then later, I can use `grep`, `mlr --opprint group-like`, etc.
to analyze my logs.
See [Reference: I/O options](reference-main-io-options.md) regarding how to specify separators other than the default equals-sign and comma.
See the [I/O options reference](reference-main-io-options.md) regarding how to specify separators other than the default equals-sign and comma.
## NIDX: Index-numbered (toolkit style)
@ -208,7 +208,7 @@ JSON isn't a parameterized format, so `RS`, `FS`, `PS` aren't specifiable. Nonet
* Use `--jquoteall` to double-quote all object values. By default, integers, floating-point numbers, and booleans `true` and `false` are not double-quoted when they appear as JSON-object keys.
* Use `--jflatsep youmd.inghere` to specify the string used for key concatenation: this defaults to a single colon.
* Use `--jflatsep yourseparatorhere` to specify the string used for key concatenation: this defaults to a single dot.
Again, please see [jq](https://stedolan.github.io/jq/) for a truly powerful, JSON-specific tool.
@ -269,9 +269,10 @@ $ grep -v '^#' /etc/passwd | head -n 2 | mlr --nidx --fs : --oxtab cat
7 /bin/sh
GENMD_EOF
GENMD_CARDIFY_HIGHLIGHT_ONE
GENMD_CARDIFY_HIGHLIGHT_THREE
$ grep -v '^#' /etc/passwd | head -n 2 | \
mlr --nidx --fs : --ojson --jvstack --jlistwrap label name,password,uid,gid,gecos,home_dir,shell
mlr --nidx --fs : --ojson --jvstack --jlistwrap \
label name,password,uid,gid,gecos,home_dir,shell
[
{
"name": "nobody",
@ -281,8 +282,8 @@ $ grep -v '^#' /etc/passwd | head -n 2 | \
"gecos": "Unprivileged User",
"home_dir": "/var/empty",
"shell": "/usr/bin/false"
}
,{
},
{
"name": "root",
"password": "*",
"uid": 0,
@ -326,7 +327,7 @@ The line-ending autodetector triggers on the first line ending detected in the i
If you use `--ors {something else}` with (default or explicitly specified) `--irs auto` then line endings are autodetected on input and set to what you specify on output.
If you use `--irs {something else}` with (default or explicitly specified) `--ors auto` then the output line endings used are LF on Unix/Linux/BSD/MacOSX, and CRLF on Windows.
If you use `--irs {something else}` with (default or explicitly specified) `--ors auto` then the output line endings used are LF on Unix/Linux/BSD/MacOS X, and CRLF on Windows.
See also [Record/field/pair separators](reference-main-io-options.md#recordfieldpair-separators) for more information about record/field/pair separators.

8
docs6/docs/genmd-filter
View file

@ -36,6 +36,14 @@ def main
line1 = lines.shift
write_card([line1], lines, output_handle)
elsif content_line =~ /^GENMD_CARDIFY_HIGHLIGHT_TWO$/
lines = read_until_genmd_eof(input_handle)
write_card(lines.slice(0,2), lines.slice(2, lines.length), output_handle)
elsif content_line =~ /^GENMD_CARDIFY_HIGHLIGHT_THREE$/
lines = read_until_genmd_eof(input_handle)
write_card(lines.slice(0,3), lines.slice(3, lines.length), output_handle)
elsif content_line =~ /^GENMD_RUN_COMMAND_TOLERATING_ERROR$/
cmd_lines = read_until_genmd_eof(input_handle)
run_command_tolerating_error(cmd_lines, output_handle)

12
docs6/docs/index.md
View file

@ -1,6 +1,18 @@
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
# Introduction
<div>
<span class="quicklinks">
Quick links:
&nbsp;
<a href="reference-verbs/index.html">[verbs]</a>
&nbsp;
<a href="reference-dsl-builtin-functions/index.html">[functions]</a>
&nbsp;
<a href="https://github.com/johnkerl/miller">[repository]</a>
</span>
</div>
**Miller is a command-line tool for querying, shaping, and reformatting data files in various formats including CSV and JSON.**
In several senses, Miller is more than one tool:

12
docs6/docs/index.md.in
View file

@ -1,5 +1,17 @@
# Introduction
<div>
<span class="quicklinks">
Quick links:
&nbsp;
<a href="reference-verbs/index.html">[verbs]</a>
&nbsp;
<a href="reference-dsl-builtin-functions/index.html">[functions]</a>
&nbsp;
<a href="https://github.com/johnkerl/miller">[repository]</a>
</span>
</div>
**Miller is a command-line tool for querying, shaping, and reformatting data files in various formats including CSV and JSON.**
In several senses, Miller is more than one tool:

6
docs6/docs/installation.md
View file

@ -7,13 +7,13 @@
Once Miller 6 is released, the commands in this section will install Miller 6 for you.
Until then, please see the following sections for how to get Miller 6.*
[Homebrew](https://brew.sh/) installation support for OSX is available via
[Homebrew](https://brew.sh/) installation support for OS X is available via
<pre class="pre-highlight-non-pair">
<b>brew update && brew install miller</b>
</pre>
...and also via [MacPorts](https://www.macports.org/):
... and also via [MacPorts](https://www.macports.org/):
<pre class="pre-highlight-non-pair">
<b>sudo port selfupdate && sudo port install miller</b>
@ -41,7 +41,7 @@ On Windows, Miller is available via [Chocolatey](https://chocolatey.org/):
## Prebuilt executables via GitHub per release
Please see [https://github.com/johnkerl/miller/releases](https://github.com/johnkerl/miller/releases) where there are builds for OSX Yosemite, Linux x86-64 (dynamically linked), and Windows.
Please see [https://github.com/johnkerl/miller/releases](https://github.com/johnkerl/miller/releases) where there are builds for OS X Yosemite, Linux x86-64 (dynamically linked), and Windows.
## Prebuilt executables via GitHub per commit

6
docs6/docs/installation.md.in
View file

@ -6,13 +6,13 @@
Once Miller 6 is released, the commands in this section will install Miller 6 for you.
Until then, please see the following sections for how to get Miller 6.*
[Homebrew](https://brew.sh/) installation support for OSX is available via
[Homebrew](https://brew.sh/) installation support for OS X is available via
GENMD_CARDIFY_HIGHLIGHT_ONE
brew update && brew install miller
GENMD_EOF
...and also via [MacPorts](https://www.macports.org/):
... and also via [MacPorts](https://www.macports.org/):
GENMD_CARDIFY_HIGHLIGHT_ONE
sudo port selfupdate && sudo port install miller
@ -40,7 +40,7 @@ GENMD_EOF
## Prebuilt executables via GitHub per release
Please see [https://github.com/johnkerl/miller/releases](https://github.com/johnkerl/miller/releases) where there are builds for OSX Yosemite, Linux x86-64 (dynamically linked), and Windows.
Please see [https://github.com/johnkerl/miller/releases](https://github.com/johnkerl/miller/releases) where there are builds for OS X Yosemite, Linux x86-64 (dynamically linked), and Windows.
## Prebuilt executables via GitHub per commit

2
docs6/docs/internationalization.md
View file

@ -9,4 +9,4 @@ Support for internationalization includes:
* The [strlen](reference-dsl-builtin-functions.md#strlen) function correctly counts UTF-8 codepoints rather than bytes.
* The [toupper](reference-dsl-builtin-functions.md#toupper), [tolower](reference-dsl-builtin-functions.md#tolower), and [capitalize](reference-dsl-builtin-functions.md#capitalize) DSL functions operate within the capabilities of the Go libraries.
Please file an issue at [https://github.com/johnkerl/miller](https://github.com/johnkerl/miller) if you encounter bugs related to internationalization (or anything else for that matter).
Please file an issue at [https://github.com/johnkerl/miller/issues/new](https://github.com/johnkerl/miller/issues/new) if you encounter bugs related to internationalization (or anything else for that matter).

2
docs6/docs/internationalization.md.in
View file

@ -8,4 +8,4 @@ Support for internationalization includes:
* The [strlen](reference-dsl-builtin-functions.md#strlen) function correctly counts UTF-8 codepoints rather than bytes.
* The [toupper](reference-dsl-builtin-functions.md#toupper), [tolower](reference-dsl-builtin-functions.md#tolower), and [capitalize](reference-dsl-builtin-functions.md#capitalize) DSL functions operate within the capabilities of the Go libraries.
Please file an issue at [https://github.com/johnkerl/miller](https://github.com/johnkerl/miller) if you encounter bugs related to internationalization (or anything else for that matter).
Please file an issue at [https://github.com/johnkerl/miller/issues/new](https://github.com/johnkerl/miller/issues/new) if you encounter bugs related to internationalization (or anything else for that matter).

26
docs6/docs/keystroke-savers.md
View file

@ -1,17 +1,17 @@
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
# Keystroke-savers
## Short format specifiers
## Short format specifiers, including --c2p
In our examples so far we've often made use of `mlr --icsv --opprint` or `mlr --icsv --ojson`. These are such frequently occurring patterns that they have short options like **--c2p** and **--c2j**:
In our examples so far we've often made use of `mlr --icsv --opprint` or `mlr --icsv --ojson`. These are such frequently occurring patterns that they have short options like `--c2p` and `--c2j`:
<pre class="pre-highlight-in-pair">
<b>mlr --c2p head -n 2 example.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
color shape flag index quantity rate
yellow triangle true 11 43.6498 9.8870
red square true 15 79.2778 0.0130
color shape flag k index quantity rate
yellow triangle true 1 11 43.6498 9.8870
red square true 2 15 79.2778 0.0130
</pre>
<pre class="pre-highlight-in-pair">
@ -22,6 +22,7 @@ red square true 15 79.2778 0.0130
"color": "yellow",
"shape": "triangle",
"flag": true,
"k": 1,
"index": 11,
"quantity": 43.6498,
"rate": 9.8870
@ -30,15 +31,16 @@ red square true 15 79.2778 0.0130
"color": "red",
"shape": "square",
"flag": true,
"k": 2,
"index": 15,
"quantity": 79.2778,
"rate": 0.0130
}
</pre>
You can get the full list here (TODO:linkify).
You can get the full list [here](file-formats.md#data-conversion-keystroke-savers).
## File names up front
## File names up front, including --from
Already we saw that you can put the filename first using `--from`. When you're interacting with your data at the command line, this makes it easier to up-arrow and append to the previous command:
@ -46,10 +48,10 @@ Already we saw that you can put the filename first using `--from`. When you're i
<b>mlr --c2p --from example.csv sort -nr index then head -n 3</b>
</pre>
<pre class="pre-non-highlight-in-pair">
color shape flag index quantity rate
purple square false 91 72.3735 8.2430
yellow circle true 87 63.5058 8.3350
yellow circle true 73 63.9785 4.2370
color shape flag k index quantity rate
purple square false 10 91 72.3735 8.2430
yellow circle true 9 87 63.5058 8.3350
yellow circle true 8 73 63.9785 4.2370
</pre>
<pre class="pre-highlight-in-pair">
@ -70,6 +72,6 @@ If there's more than one input file, you can use `--mfrom`, then however many fi
## .mlrrc file
If you want the default file format for Miller to be CSV you can simply put `--csv` on a line by itself in your `~/.mlrrc` file. Then instead of `mlr --csv cat example.csv` you can just do `mlr cat example.csv`. This is just the default, though, so `mlr --opprint cat example.csv` will still use default CSV format for input, and PPRINT (tabular) for output.
If you want the default file format for Miller to be CSV, you can simply put `--csv` on a line by itself in your `~/.mlrrc` file. Then instead of `mlr --csv cat example.csv` you can just do `mlr cat example.csv`. This is just a personal default, though, so `mlr --opprint cat example.csv` will use default CSV format for input, and PPRINT (tabular) for output.
You can read more about this at the [Customization](customization.md) page.

10
docs6/docs/keystroke-savers.md.in
View file

@ -1,8 +1,8 @@
# Keystroke-savers
## Short format specifiers
## Short format specifiers, including --c2p
In our examples so far we've often made use of `mlr --icsv --opprint` or `mlr --icsv --ojson`. These are such frequently occurring patterns that they have short options like **--c2p** and **--c2j**:
In our examples so far we've often made use of `mlr --icsv --opprint` or `mlr --icsv --ojson`. These are such frequently occurring patterns that they have short options like `--c2p` and `--c2j`:
GENMD_RUN_COMMAND
mlr --c2p head -n 2 example.csv
@ -12,9 +12,9 @@ GENMD_RUN_COMMAND
mlr --c2j head -n 2 example.csv
GENMD_EOF
You can get the full list here (TODO:linkify).
You can get the full list [here](file-formats.md#data-conversion-keystroke-savers).
## File names up front
## File names up front, including --from
Already we saw that you can put the filename first using `--from`. When you're interacting with your data at the command line, this makes it easier to up-arrow and append to the previous command:
@ -34,6 +34,6 @@ GENMD_EOF
## .mlrrc file
If you want the default file format for Miller to be CSV you can simply put `--csv` on a line by itself in your `~/.mlrrc` file. Then instead of `mlr --csv cat example.csv` you can just do `mlr cat example.csv`. This is just the default, though, so `mlr --opprint cat example.csv` will still use default CSV format for input, and PPRINT (tabular) for output.
If you want the default file format for Miller to be CSV, you can simply put `--csv` on a line by itself in your `~/.mlrrc` file. Then instead of `mlr --csv cat example.csv` you can just do `mlr cat example.csv`. This is just a personal default, though, so `mlr --opprint cat example.csv` will use default CSV format for input, and PPRINT (tabular) for output.
You can read more about this at the [Customization](customization.md) page.

2
docs6/docs/manpage.md
View file

@ -1,5 +1,5 @@
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
# Manpage
# Manual page
This is simply a copy of what you should see on running **man mlr** at a command prompt, once Miller is installed on your system.

2
docs6/docs/manpage.md.in
View file

@ -1,4 +1,4 @@
# Manpage
# Manual page
This is simply a copy of what you should see on running **man mlr** at a command prompt, once Miller is installed on your system.

6
docs6/docs/miller-on-windows.md
View file

@ -3,11 +3,11 @@
## Native builds as of Miller 6
Miller was originally developed for Unix-like operating systems including Linux and MacOS. Since the initial release of Miller in 2015, support for Windows has been partial. But as of version 6.0.0, Miller builds directly on Windows.
Miller was originally developed for Unix-like operating systems including Linux and MacOS. Since Miller 5.2.0 which was the first version to support Windows at all, that support has been partial. But as of version 6.0.0, Miller builds directly on Windows.
The experience is now almost the same as on Linux, NetBSD/FreeBSD, and MacOS.
**The experience is now almost the same on Windows as it is on Linux, NetBSD/FreeBSD, and MacOS.**
MSYS2 is no longer required, although you can use Miller from within MSYS2 if you like. There is now simply a single `mlr.exe`, with no `msys2.dll` alongside anymore.
MSYS2 is no longer required -- although you can of course still use Miller from within MSYS2 if you prefer. There is now simply a single `mlr.exe`, with no `msys2.dll` alongside anymore.
See [Installation](installation.md) for how to get a copy of `mlr.exe`.

6
docs6/docs/miller-on-windows.md.in
View file

@ -2,11 +2,11 @@
## Native builds as of Miller 6
Miller was originally developed for Unix-like operating systems including Linux and MacOS. Since the initial release of Miller in 2015, support for Windows has been partial. But as of version 6.0.0, Miller builds directly on Windows.
Miller was originally developed for Unix-like operating systems including Linux and MacOS. Since Miller 5.2.0 which was the first version to support Windows at all, that support has been partial. But as of version 6.0.0, Miller builds directly on Windows.
The experience is now almost the same as on Linux, NetBSD/FreeBSD, and MacOS.
**The experience is now almost the same on Windows as it is on Linux, NetBSD/FreeBSD, and MacOS.**
MSYS2 is no longer required, although you can use Miller from within MSYS2 if you like. There is now simply a single `mlr.exe`, with no `msys2.dll` alongside anymore.
MSYS2 is no longer required -- although you can of course still use Miller from within MSYS2 if you prefer. There is now simply a single `mlr.exe`, with no `msys2.dll` alongside anymore.
See [Installation](installation.md) for how to get a copy of `mlr.exe`.

71
docs6/docs/mk-func-h2s.sh
View file

@ -4,6 +4,8 @@
export MLRRC=__none__
mlr help list-functions | grep -v '^[a-zA-Z]' | uniq | while read funcname; do
# Displayname is for HTML rendered in the document body. Linkname is for
# #-style links which don't do so well with special characters.
displayname=$funcname
linkname="$funcname"
if [ "$funcname" = '+' ]; then
@ -18,24 +20,50 @@ mlr help list-functions | grep -v '^[a-zA-Z]' | uniq | while read funcname; do
elif [ "$funcname" = '**' ]; then
displayname='\**'
linkname='exponentiation'
elif [ "$funcname" = '|' ]; then
displayname='\|'
linkname='bitwise-or'
elif [ "$funcname" = '&' ]; then
displayname='&'
linkname='bitwise-and'
elif [ "$funcname" = '^' ]; then
displayname='^'
linkname='bitwise-xor'
elif [ "$funcname" = '||' ]; then
displayname='\|\|'
linkname='logical-or'
elif [ "$funcname" = '&&' ]; then
displayname='&&'
linkname='logical-and'
elif [ "$funcname" = '^^' ]; then
displayname='^^'
linkname='logical-xor'
elif [ "$funcname" = '?' ]; then
displayname='\?'
linkname='question-mark'
elif [ "$funcname" = ':' ]; then
displayname='\:'
linkname='colon'
elif [ "$funcname" = '!' ]; then
displayname='\!'
linkname='colon'
elif [ "$funcname" = '? :' ]; then
displayname='\?'
linkname='question-mark-colon'
elif [ "$funcname" = '?:' ]; then
displayname='\?'
linkname='question-mark-colon'
elif [ "$funcname" = '!' ]; then
displayname='\!'
linkname='exclamation-point'
elif [ "$funcname" = '=~' ]; then
displayname='=~'
linkname='regmatch'
elif [ "$funcname" = '!=~' ]; then
displayname='!=~'
linkname='regnotmatch'
elif [ "$funcname" = '>>' ]; then
displayname='\>\>'
linkname='srsh'
@ -67,43 +95,6 @@ done
mlr help list-functions | grep '^[a-zA-Z]' | sort -u | while read funcname; do
displayname=$funcname
linkname="$funcname"
if [ "$funcname" = '+' ]; then
displayname='\+'
linkname='plus'
elif [ "$funcname" = '-' ]; then
displayname='\-'
linkname='minus'
elif [ "$funcname" = '*' ]; then
displayname='\*'
linkname='times'
elif [ "$funcname" = '**' ]; then
displayname='\**'
linkname='exponentiation'
elif [ "$funcname" = '|' ]; then
displayname='\|'
linkname='bitwise-or'
elif [ "$funcname" = '?' ]; then
displayname='\?'
linkname='question-mark'
elif [ "$funcname" = ':' ]; then
displayname='\:'
linkname='colon'
elif [ "$funcname" = '!' ]; then
displayname='\!'
linkname='colon'
elif [ "$funcname" = '? :' ]; then
displayname='\?'
linkname='question-mark-colon'
elif [ "$funcname" = '>>' ]; then
displayname='\>\>'
linkname='srsh'
elif [ "$funcname" = '>>>' ]; then
displayname='\>\>\>'
linkname='ursh'
elif [ "$funcname" = '>>>=' ]; then
displayname='\>\>\>='
linkname='ursheq'
fi
# TODO: fix section-links for mnkdocs
#echo ''

33
docs6/docs/new-in-miller-6.md
View file

@ -8,26 +8,27 @@ See also the [list of issues tagged with go-port](https://github.com/johnkerl/mi
Documentation (what you're reading here) and on-line help (`mlr --help`) have been completely reworked.
In the initial release, the focus was convincing users already familiar with
`awk`/`grep`/`cut` that Miller was a viable option. Over time it's become
clear that many users aren't expert with these. The focus has shifted toward a
higher quantity of more introductory/accessible material for command-line data
processing.
`awk`/`grep`/`cut` that Miller was a viable alternative -- but over time it's
become clear that many Miller users aren't expert with those tools. The focus
has shifted toward a higher quantity of more introductory/accessible material
for command-line data processing.
Similarly, the FAQ/recipe material has been expanded to include more, and simpler,
use-cases including resolved questions from
https://github.com/johnkerl/miller/issues and
https://github.com/johnkerl/miller/discussions. More complex/niche material has
been pushed farther down. The long reference pages have been split up into
separate pages.
Similarly, the FAQ/recipe material has been expanded to include more, and
simpler, use-cases including resolved questions from
[https://github.com/johnkerl/miller/issues](https://github.com/johnkerl/miller/issues)
and
[https://github.com/johnkerl/miller/discussions](https://github.com/johnkerl/miller/discussions).
More complex/niche material has been pushed farther down. The long reference
pages have been split up into separate pages.
Since CSV is overwhelmingly the most popular data format for Miller, it is
now discussed first, and more examples use CSV.
## JSON support, and arrays
Arrays are now supported in Miller's `put`/`filter` programming language,
as described at [Reference: arrays](reference-dsl-arrays.md). Also, `array` is now a keyword
so this is no longer usable as a local-variable or UDF name.
Arrays are now supported in Miller's `put`/`filter` programming language, as
described in the [Arrays reference](reference-dsl-arrays.md). Also, `array` is
now a keyword so this is no longer usable as a local-variable or UDF name.
JSON support is improved:
@ -106,11 +107,11 @@ Miller now has a read-evaluate-print-loop ([REPL](repl.md)) where you can single
## Improved command-line parsing
Miller 6 has getoptish command-line parsing (https://github.com/johnkerl/miller/pull/467):
Miller 6 has getoptish command-line parsing ([pull request 467](https://github.com/johnkerl/miller/pull/467)):
* `-xyz` expands automatically to `-x -y -z`, so (for example) `mlr cut -of shape,flag` is the same as `mlr cut -o -f shape,flag`.
* `--foo=bar` expands automatically to `--foo bar`, so (for example) `mlr --ifs=comma` is the same as `mlr --ifs comma`.
* `--mfrom`, `--load`, `--mload` as described at [TODO:linkify].
* `--mfrom`, `--load`, `--mload` as described on the [keystroke-savers page](keystroke-savers.md).
## Improved error messages for DSL parsing
@ -124,4 +125,4 @@ Parse error on token ">" at line 63 columnn 7.
## Developer-specific aspects
* Miller has been ported from C to Go. Developer notes: [https://github.com/johnkerl/miller/blob/main/go/README.md](https://github.com/johnkerl/miller/blob/main/go/README.md).
* Completely reworked regression testing, including regresstion-testing now running on Windows.
* Regression testing has been completely reworked, including regression-testing now running fully on Windows (alongside Linux and Mac) [on each GitHub commit](https://github.com/johnkerl/miller/actions).

33
docs6/docs/new-in-miller-6.md.in
View file

@ -7,26 +7,27 @@ See also the [list of issues tagged with go-port](https://github.com/johnkerl/mi
Documentation (what you're reading here) and on-line help (`mlr --help`) have been completely reworked.
In the initial release, the focus was convincing users already familiar with
`awk`/`grep`/`cut` that Miller was a viable option. Over time it's become
clear that many users aren't expert with these. The focus has shifted toward a
higher quantity of more introductory/accessible material for command-line data
processing.
`awk`/`grep`/`cut` that Miller was a viable alternative -- but over time it's
become clear that many Miller users aren't expert with those tools. The focus
has shifted toward a higher quantity of more introductory/accessible material
for command-line data processing.
Similarly, the FAQ/recipe material has been expanded to include more, and simpler,
use-cases including resolved questions from
https://github.com/johnkerl/miller/issues and
https://github.com/johnkerl/miller/discussions. More complex/niche material has
been pushed farther down. The long reference pages have been split up into
separate pages.
Similarly, the FAQ/recipe material has been expanded to include more, and
simpler, use-cases including resolved questions from
[https://github.com/johnkerl/miller/issues](https://github.com/johnkerl/miller/issues)
and
[https://github.com/johnkerl/miller/discussions](https://github.com/johnkerl/miller/discussions).
More complex/niche material has been pushed farther down. The long reference
pages have been split up into separate pages.
Since CSV is overwhelmingly the most popular data format for Miller, it is
now discussed first, and more examples use CSV.
## JSON support, and arrays
Arrays are now supported in Miller's `put`/`filter` programming language,
as described at [Reference: arrays](reference-dsl-arrays.md). Also, `array` is now a keyword
so this is no longer usable as a local-variable or UDF name.
Arrays are now supported in Miller's `put`/`filter` programming language, as
described in the [Arrays reference](reference-dsl-arrays.md). Also, `array` is
now a keyword so this is no longer usable as a local-variable or UDF name.
JSON support is improved:
@ -94,11 +95,11 @@ Miller now has a read-evaluate-print-loop ([REPL](repl.md)) where you can single
## Improved command-line parsing
Miller 6 has getoptish command-line parsing (https://github.com/johnkerl/miller/pull/467):
Miller 6 has getoptish command-line parsing ([pull request 467](https://github.com/johnkerl/miller/pull/467)):
* `-xyz` expands automatically to `-x -y -z`, so (for example) `mlr cut -of shape,flag` is the same as `mlr cut -o -f shape,flag`.
* `--foo=bar` expands automatically to `--foo bar`, so (for example) `mlr --ifs=comma` is the same as `mlr --ifs comma`.
* `--mfrom`, `--load`, `--mload` as described at [TODO:linkify].
* `--mfrom`, `--load`, `--mload` as described on the [keystroke-savers page](keystroke-savers.md).
## Improved error messages for DSL parsing
@ -112,4 +113,4 @@ GENMD_EOF
## Developer-specific aspects
* Miller has been ported from C to Go. Developer notes: [https://github.com/johnkerl/miller/blob/main/go/README.md](https://github.com/johnkerl/miller/blob/main/go/README.md).
* Completely reworked regression testing, including regresstion-testing now running on Windows.
* Regression testing has been completely reworked, including regression-testing now running fully on Windows (alongside Linux and Mac) [on each GitHub commit](https://github.com/johnkerl/miller/actions).

372
docs6/docs/operating-on-all-records.md Normal file
View file

@ -0,0 +1,372 @@
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
# Operating on all records
As we saw in the DSL-overview page, the Miller programming language has an
[implicit loop over records for main statements](reference-dsl.md#implicit-loop-over-records-for-main-statements).
Miller's feature of _streaming operation over records_ is implemented by the
main statements (everything outside `begin`/`end`/`func`/`subr`) getting
invoked once per record. You don't explicitly loop over records, as you would
in some dataframes contexts; rather, _Miller loops over records for you_, and
it lets you specify what to do on each record: you write the body of the loop.
That's fine for most simple use-cases, but sometimes you _do_ want to loop over
all records. Here we describe a few options.
## Sums/counters
The first option is to leverage the fact that main DSL statements are already
invoked in a loop over records, and use
[out-of-stream variables](reference-dsl-variables.md#out-of-stream-variables)
to retain sums, counters, etc.
For example, let's look at our short data file [data/short.csv](data/short.csv):
<pre class="pre-highlight-in-pair">
<b>cat data/short.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
word,value
apple,37
ball,28
cat,54
</pre>
We can track count and sum using
[out-of-stream variables](reference-dsl-variables.md#out-of-stream-variables) -- the ones that
start with the `@` sigil -- then
[emit](reference-dsl-output-statements.md#emit-statements) them as a new record
after all the input is read.
<pre class="pre-highlight-in-pair">
<b>mlr --icsv --ojson --from data/short.csv put '</b>
<b> begin {</b>
<b> @count = 0;</b>
<b> @sum = 0;</b>
<b> }</b>
<b> @count += 1;</b>
<b> @sum += $value;</b>
<b> end {</b>
<b> emit (@count, @sum);</b>
<b> }</b>
<b>'</b>
</pre>
<pre class="pre-non-highlight-in-pair">
{
"word": "apple",
"value": 37
}
{
"word": "ball",
"value": 28
}
{
"word": "cat",
"value": 54
}
{
"count": 3,
"sum": 119
}
</pre>
And if all we want is the final output and not the input data, we can use `put
-q` to not pass through the input records:
<pre class="pre-highlight-in-pair">
<b>mlr --icsv --ojson --from data/short.csv put -q '</b>
<b> begin {</b>
<b> @count = 0;</b>
<b> @sum = 0;</b>
<b> }</b>
<b> @count += 1;</b>
<b> @sum += $value;</b>
<b> end {</b>
<b> emit (@count, @sum);</b>
<b> }</b>
<b>'</b>
</pre>
<pre class="pre-non-highlight-in-pair">
{
"count": 3,
"sum": 119
}
</pre>
As discussed a bit more on the (TODO) page on streaming and memory, this doesn't keep all records in memory, only the count and sum variables. You can use this on very large files without running out of memory.
## Retain records in a map
The second option is to retain entire records in a map, then loop over them in an `end` block.
Let's use the same short data file [data/short.csv](data/short.csv):
<pre class="pre-highlight-in-pair">
<b>cat data/short.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
word,value
apple,37
ball,28
cat,54
</pre>
<pre class="pre-highlight-in-pair">
<b>mlr --icsv --ojson --from data/short.csv put -q '</b>
<b> # map</b>
<b> begin {</b>
<b> @records = {};</b>
<b> }</b>
<b> @records[NR] = $*;</b>
<b> end {</b>
<b> count = length(@records);</b>
<b> sum = 0;</b>
<b> for (i = 1; i <= NR; i += 1) {</b>
<b> sum += @records[i]["value"];</b>
<b> }</b>
<b> dump @records; # show the map</b>
<b> emit (count, sum);</b>
<b> }</b>
<b>'</b>
</pre>
<pre class="pre-non-highlight-in-pair">
{
"1": {
"word": "apple",
"value": 37
},
"2": {
"word": "ball",
"value": 28
},
"3": {
"word": "cat",
"value": 54
}
}
{
"count": 3,
"sum": 119
}
</pre>
The downside to this, of course, is that this retains all records (plus data-structure overhead) in memory, so you're limited to processing files that fit in your computer's memory. The upside, though, is that you can do random access over the records using things like
<pre class="pre-non-highlight-non-pair">
output = 0;
for (i = 1; i <= NR; i += 1) {
for (j = 1; j <= NR; j += 1) {
for (k = 1; k <= NR; k += 1) {
output += call_some_function_of(@records[i], @records[j], @record[k])
}
}
}
# do something with the output
</pre>
## Retain records in an array
The third option is to retain records in an array, then loop over them in an `end` block.
<pre class="pre-highlight-in-pair">
<b>mlr --icsv --ojson --from data/short.csv put -q '</b>
<b> # array</b>
<b> begin {</b>
<b> @records = [];</b>
<b> }</b>
<b> @records[NR] = $*;</b>
<b> end {</b>
<b> count = length(@records);</b>
<b> sum = 0;</b>
<b> for (i = 1; i <= NR; i += 1) {</b>
<b> sum += @records[i]["value"];</b>
<b> }</b>
<b> dump @records; # show the array</b>
<b> emit (count, sum);</b>
<b> }</b>
<b>'</b>
</pre>
<pre class="pre-non-highlight-in-pair">
[
{
"word": "apple",
"value": 37
},
{
"word": "ball",
"value": 28
},
{
"word": "cat",
"value": 54
}
]
{
"count": 3,
"sum": 119
}
</pre>
Just as with the retain-as-map approach, the downside is the overhead of
retaining all records in memory, and the upside is that you get random access
over records.
## Using maps vs using arrays
Retaining records as a map or as an array is a matter of taste. Some things to note:
If we initialize `@records = {}` in the `begin` block (or, if we don't initialize it at all and just start writing to it in the main statements) then `@records` is a map (TODO: linkify). If we initialize `@records=[]` then it's an array.
Arrays are, of course, contiguously indexed. (And, in Miller, their indices
start with 1, not 0 (TODO:linkify).) This means that if you are only retaining
a subset of records then your array will have null-gaps (TODO:linkify) in it:
<pre class="pre-highlight-in-pair">
<b>mlr --icsv --ojson --from data/short.csv put -q '</b>
<b> begin {</b>
<b> @records = [];</b>
<b> }</b>
<b> if (NR != 2) {</b>
<b> @records[NR] = $*</b>
<b> }</b>
<b> end {</b>
<b> dump @records;</b>
<b> }</b>
<b>'</b>
</pre>
<pre class="pre-non-highlight-in-pair">
[
{
"word": "apple",
"value": 37
},
null,
{
"word": "cat",
"value": 54
}
]
</pre>
You can index `@records` by `@count` rather than `NR` to get a contiguous array:
<pre class="pre-highlight-in-pair">
<b>mlr --icsv --ojson --from data/short.csv put -q '</b>
<b> begin {</b>
<b> @records = [];</b>
<b> @count = 0;</b>
<b> }</b>
<b> # main statement</b>
<b> if (NR != 2) {</b>
<b> @count += 1;</b>
<b> @records[@count] = $*;</b>
<b> }</b>
<b> end {</b>
<b> dump @records;</b>
<b> count = length(@records);</b>
<b> sum = 0;</b>
<b> for (record in @records) {</b>
<b> sum += record["value"];</b>
<b> }</b>
<b> emit (count, sum);</b>
<b> }</b>
<b>'</b>
</pre>
<pre class="pre-non-highlight-in-pair">
[
{
"word": "apple",
"value": 37
},
{
"word": "cat",
"value": 54
}
]
{
"count": 2,
"sum": 91
}
</pre>
If you use a map to retain records, then this is a non-issue: maps can retain whatever values you like:
<pre class="pre-highlight-in-pair">
<b>mlr --icsv --ojson --from data/short.csv put -q '</b>
<b> begin {</b>
<b> @records = {};</b>
<b> }</b>
<b> # main statement</b>
<b> if (NR != 2) {</b>
<b> @records[NR] = $*;</b>
<b> }</b>
<b> end {</b>
<b> dump @records;</b>
<b> count = length(@records);</b>
<b> sum = 0;</b>
<b> for (key in @records) {</b>
<b> sum += @records[key]["value"];</b>
<b> }</b>
<b> emit (count, sum);</b>
<b> }</b>
<b>'</b>
</pre>
<pre class="pre-non-highlight-in-pair">
{
"1": {
"word": "apple",
"value": 37
},
"3": {
"word": "cat",
"value": 54
}
}
{
"count": 2,
"sum": 91
}
</pre>
Do note that Miller maps (TODO:linkify) preserve insertion order, so at the end you're guaranteed to loop over records in the same order you read them. Also note that when you index a Miller hashmap with an integer key, this works, but (TODO:linkify) the key is stringified.
## Retain partial records in map or array
If all you need is one or a few attributes out of a record, you don't need to
retain full records. You can retain a map, or array, of just the fields you're
interested in:
<pre class="pre-highlight-in-pair">
<b>mlr --icsv --ojson --from data/short.csv put -q '</b>
<b> begin {</b>
<b> @values = {};</b>
<b> }</b>
<b> # main statement</b>
<b> if (NR != 2) {</b>
<b> @values[NR] = $value;</b>
<b> }</b>
<b> end {</b>
<b> dump @values;</b>
<b> count = length(@values);</b>
<b> sum = 0;</b>
<b> for (key in @values) {</b>
<b> sum += @values[key];</b>
<b> }</b>
<b> emit (count, sum);</b>
<b> }</b>
<b>'</b>
</pre>
<pre class="pre-non-highlight-in-pair">
{
"1": 37,
"3": 54
}
{
"count": 2,
"sum": 91
}
</pre>
## For more information
For more information please see the page on [two-pass algorithms](two-pass-algorithms.md).

240
docs6/docs/operating-on-all-records.md.in Normal file
View file

@ -0,0 +1,240 @@
# Operating on all records
As we saw in the DSL-overview page, the Miller programming language has an
[implicit loop over records for main statements](reference-dsl.md#implicit-loop-over-records-for-main-statements).
Miller's feature of _streaming operation over records_ is implemented by the
main statements (everything outside `begin`/`end`/`func`/`subr`) getting
invoked once per record. You don't explicitly loop over records, as you would
in some dataframes contexts; rather, _Miller loops over records for you_, and
it lets you specify what to do on each record: you write the body of the loop.
That's fine for most simple use-cases, but sometimes you _do_ want to loop over
all records. Here we describe a few options.
## Sums/counters
The first option is to leverage the fact that main DSL statements are already
invoked in a loop over records, and use
[out-of-stream variables](reference-dsl-variables.md#out-of-stream-variables)
to retain sums, counters, etc.
For example, let's look at our short data file [data/short.csv](data/short.csv):
GENMD_RUN_COMMAND
cat data/short.csv
GENMD_EOF
We can track count and sum using
[out-of-stream variables](reference-dsl-variables.md#out-of-stream-variables) -- the ones that
start with the `@` sigil -- then
[emit](reference-dsl-output-statements.md#emit-statements) them as a new record
after all the input is read.
GENMD_RUN_COMMAND
mlr --icsv --ojson --from data/short.csv put '
begin {
@count = 0;
@sum = 0;
}
@count += 1;
@sum += $value;
end {
emit (@count, @sum);
}
'
GENMD_EOF
And if all we want is the final output and not the input data, we can use `put
-q` to not pass through the input records:
GENMD_RUN_COMMAND
mlr --icsv --ojson --from data/short.csv put -q '
begin {
@count = 0;
@sum = 0;
}
@count += 1;
@sum += $value;
end {
emit (@count, @sum);
}
'
GENMD_EOF
As discussed a bit more on the (TODO) page on streaming and memory, this doesn't keep all records in memory, only the count and sum variables. You can use this on very large files without running out of memory.
## Retain records in a map
The second option is to retain entire records in a map, then loop over them in an `end` block.
Let's use the same short data file [data/short.csv](data/short.csv):
GENMD_RUN_COMMAND
cat data/short.csv
GENMD_EOF
GENMD_RUN_COMMAND
mlr --icsv --ojson --from data/short.csv put -q '
# map
begin {
@records = {};
}
@records[NR] = $*;
end {
count = length(@records);
sum = 0;
for (i = 1; i <= NR; i += 1) {
sum += @records[i]["value"];
}
dump @records; # show the map
emit (count, sum);
}
'
GENMD_EOF
The downside to this, of course, is that this retains all records (plus data-structure overhead) in memory, so you're limited to processing files that fit in your computer's memory. The upside, though, is that you can do random access over the records using things like
GENMD_CARDIFY
output = 0;
for (i = 1; i <= NR; i += 1) {
for (j = 1; j <= NR; j += 1) {
for (k = 1; k <= NR; k += 1) {
output += call_some_function_of(@records[i], @records[j], @record[k])
}
}
}
# do something with the output
GENMD_EOF
## Retain records in an array
The third option is to retain records in an array, then loop over them in an `end` block.
GENMD_RUN_COMMAND
mlr --icsv --ojson --from data/short.csv put -q '
# array
begin {
@records = [];
}
@records[NR] = $*;
end {
count = length(@records);
sum = 0;
for (i = 1; i <= NR; i += 1) {
sum += @records[i]["value"];
}
dump @records; # show the array
emit (count, sum);
}
'
GENMD_EOF
Just as with the retain-as-map approach, the downside is the overhead of
retaining all records in memory, and the upside is that you get random access
over records.
## Using maps vs using arrays
Retaining records as a map or as an array is a matter of taste. Some things to note:
If we initialize `@records = {}` in the `begin` block (or, if we don't initialize it at all and just start writing to it in the main statements) then `@records` is a map (TODO: linkify). If we initialize `@records=[]` then it's an array.
Arrays are, of course, contiguously indexed. (And, in Miller, their indices
start with 1, not 0 (TODO:linkify).) This means that if you are only retaining
a subset of records then your array will have null-gaps (TODO:linkify) in it:
GENMD_RUN_COMMAND
mlr --icsv --ojson --from data/short.csv put -q '
begin {
@records = [];
}
if (NR != 2) {
@records[NR] = $*
}
end {
dump @records;
}
'
GENMD_EOF
You can index `@records` by `@count` rather than `NR` to get a contiguous array:
GENMD_RUN_COMMAND
mlr --icsv --ojson --from data/short.csv put -q '
begin {
@records = [];
@count = 0;
}
# main statement
if (NR != 2) {
@count += 1;
@records[@count] = $*;
}
end {
dump @records;
count = length(@records);
sum = 0;
for (record in @records) {
sum += record["value"];
}
emit (count, sum);
}
'
GENMD_EOF
If you use a map to retain records, then this is a non-issue: maps can retain whatever values you like:
GENMD_RUN_COMMAND
mlr --icsv --ojson --from data/short.csv put -q '
begin {
@records = {};
}
# main statement
if (NR != 2) {
@records[NR] = $*;
}
end {
dump @records;
count = length(@records);
sum = 0;
for (key in @records) {
sum += @records[key]["value"];
}
emit (count, sum);
}
'
GENMD_EOF
Do note that Miller maps (TODO:linkify) preserve insertion order, so at the end you're guaranteed to loop over records in the same order you read them. Also note that when you index a Miller hashmap with an integer key, this works, but (TODO:linkify) the key is stringified.
## Retain partial records in map or array
If all you need is one or a few attributes out of a record, you don't need to
retain full records. You can retain a map, or array, of just the fields you're
interested in:
GENMD_RUN_COMMAND
mlr --icsv --ojson --from data/short.csv put -q '
begin {
@values = {};
}
# main statement
if (NR != 2) {
@values[NR] = $value;
}
end {
dump @values;
count = length(@values);
sum = 0;
for (key in @values) {
sum += @values[key];
}
emit (count, sum);
}
'
GENMD_EOF
## For more information
For more information please see the page on [two-pass algorithms](two-pass-algorithms.md).

2
docs6/docs/output-colorization.md
View file

@ -47,6 +47,6 @@ How you can control colorization:
If environment-variable settings and command-line flags are both provided, the latter take precedence.
Please do `mlr --list-color-codes` to see the available color codes (like `170`), and `mlr --list-color-names` to see available names (like `orchid`).
You can use `mlr --list-color-codes` to see the available color codes (like `170`), and `mlr --list-color-names` to see available names (like `orchid`).
![pix/colorization2.png](pix/colorization2.png)

2
docs6/docs/output-colorization.md.in
View file

@ -46,6 +46,6 @@ How you can control colorization:
If environment-variable settings and command-line flags are both provided, the latter take precedence.
Please do `mlr --list-color-codes` to see the available color codes (like `170`), and `mlr --list-color-names` to see available names (like `orchid`).
You can use `mlr --list-color-codes` to see the available color codes (like `170`), and `mlr --list-color-names` to see available names (like `orchid`).
![pix/colorization2.png](pix/colorization2.png)

2
docs6/docs/performance.md
View file

@ -15,4 +15,4 @@ In a previous version of this page, I compared Miller to some items in the Unix
## Summary
Miller can do many kinds of processing on key-value-pair data using elapsed time roughly of the same order of magnitude as items in the Unix toolkit can handle positionally indexed data. Specific results vary widely by platform, implementation details, multi-core use (or not). Lastly, specific special-purpose non-record-aware processing will run far faster in `grep`, `sed`, etc.
Miller can do many kinds of processing on key-value-pair data using elapsed time roughly of the same order of magnitude as items in the Unix toolkit can handle positionally indexed data. Specific results vary widely by platform, implementation details, and multi-core use (or not). Lastly, specific special-purpose non-record-aware processing will run far faster if implemented in `grep`, `sed`, etc.

2
docs6/docs/performance.md.in
View file

@ -14,4 +14,4 @@ In a previous version of this page, I compared Miller to some items in the Unix
## Summary
Miller can do many kinds of processing on key-value-pair data using elapsed time roughly of the same order of magnitude as items in the Unix toolkit can handle positionally indexed data. Specific results vary widely by platform, implementation details, multi-core use (or not). Lastly, specific special-purpose non-record-aware processing will run far faster in `grep`, `sed`, etc.
Miller can do many kinds of processing on key-value-pair data using elapsed time roughly of the same order of magnitude as items in the Unix toolkit can handle positionally indexed data. Specific results vary widely by platform, implementation details, and multi-core use (or not). Lastly, specific special-purpose non-record-aware processing will run far faster if implemented in `grep`, `sed`, etc.

BIN
docs6/docs/pix/miller-windows.png
View file

Binary file not shown.
Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 645 KiB

After

Width:  |  Height:  |  Size: 645 KiB

Before After
Before After

162
docs6/docs/programming-examples.md
View file

@ -78,9 +78,9 @@ end {
## Mandelbrot-set generator
The [Mandelbrot set](http://en.wikipedia.org/wiki/Mandelbrot_set) is also easily expressed. This isn't an important case of data-processing in the vein for which Miller was designed, but it is an example of Miller as a general-purpose programming language -- a test case for the expressiveness of the language.
The [Mandelbrot set](http://en.wikipedia.org/wiki/Mandelbrot_set) is also easily expressed. This isn't an important case of data processing (the use-case Miller was designed for), but it is an example of Miller as a general-purpose programming language -- a test case for the expressiveness of the language.
The (approximate) computation of points in the complex plane which are and aren't members is just a few lines of complex arithmetic (see the Wikipedia article); how to render them is another task. Using graphics libraries you can create PNG or JPEG files, but another fun way to do this is by printing various characters to the screen:
The (approximate) computation of points in the complex plane which are and aren't members is just a few lines of complex arithmetic (see the [Wikipedia article](https://en.wikipedia.org/wiki/Mandelbrot_set)); how to render them visually is another task. Using graphics libraries you can create PNG or JPEG files, but another fun way to do this is by printing various characters to the screen:
<pre class="pre-highlight-in-pair">
<b>cat programs/mand.mlr</b>
@ -88,40 +88,26 @@ The (approximate) computation of points in the complex plane which are and aren'
<pre class="pre-non-highlight-in-pair">
# Mandelbrot set generator: simple example of Miller DSL as programming language.
begin {
# Set defaults
@rcorn = -2.0;
@icorn = -2.0;
@side = 4.0;
@iheight = 50;
@iwidth = 100;
@maxits = 100;
@levelstep = 5;
@chars = "@X*o-."; # Palette of characters to print to the screen.
@verbose = false;
@do_julia = false;
@jr = 0.0; # Real part of Julia point, if any
@ji = 0.0; # Imaginary part of Julia point, if any
# Set defaults. They can be overridden by e.g.
# mlr -n put -e 'begin{@maxits=200}' -f nameofthisfile.mlr
# or
# mlr -n put -s maxits=200 -f nameofthisfile.mlr
@rcorn ??= -2.0;
@icorn ??= -2.0;
@side ??= 4.0;
@iheight ??= 50;
@iwidth ??= 100;
@maxits ??= 100;
@levelstep ??= 5;
@chars ??= "@X*o-.";
@silent ??= false;
@do_julia ??= false;
@jr ??= 0.0; # Real part of Julia point, if any
@ji ??= 0.0; # Imaginary part of Julia point, if any
}
# Here, we can override defaults from an input file (if any). In Miller's
# put/filter DSL, absent-null right-hand sides result in no assignment so we
# can simply put @rcorn = $rcorn: if there is a field in the input like
# 'rcorn = -1.847' we'll read and use it, else we'll keep the default.
@rcorn = $rcorn;
@icorn = $icorn;
@side = $side;
@iheight = $iheight;
@iwidth = $iwidth;
@maxits = $maxits;
@levelstep = $levelstep;
@chars = $chars;
@verbose = $verbose;
@do_julia = $do_julia;
@jr = $jr;
@ji = $ji;
end {
if (@verbose) {
if (!@silent) {
print "RCORN = ".@rcorn;
print "ICORN = ".@icorn;
print "SIDE = ".@side;
@ -132,21 +118,22 @@ end {
print "CHARS = ".@chars;
}
# Iterate over a matrix of rows and columns, printing one character for each cell.
for (int ii = @iheight-1; ii >= 0; ii -= 1) {
num pi = @icorn + (ii/@iheight) * @side;
num ci = @icorn + (ii/@iheight) * @side;
for (int ir = 0; ir < @iwidth; ir += 1) {
num pr = @rcorn + (ir/@iwidth) * @side;
printn get_point_plot(pr, pi, @maxits, @do_julia, @jr, @ji);
num cr = @rcorn + (ir/@iwidth) * @side;
str c = get_point_plot(cr, ci, @maxits, @do_julia, @jr, @ji);
if (!@silent) {
printn c
}
}
if (!@silent) {
print;
}
print;
}
}
# This is a function to approximate membership in the Mandelbrot set (or Julia
# set for a given Julia point if do_julia == true) for a given point in the
# complex plane.
func get_point_plot(pr, pi, maxits, do_julia, jr, ji) {
func get_point_plot(num pr, num pi, int maxits, bool do_julia, num jr, num ji): str {
num zr = 0.0;
num zi = 0.0;
num cr = 0.0;
@ -181,7 +168,6 @@ func get_point_plot(pr, pi, maxits, do_julia, jr, ji) {
if (!escaped) {
return ".";
} else {
# The // operator is Miller's (pythonic) integer-division operator
int level = (iti // @levelstep) % strlen(@chars);
return substr(@chars, level, level);
}
@ -191,59 +177,42 @@ func get_point_plot(pr, pi, maxits, do_julia, jr, ji) {
At standard resolution this makes a nice little ASCII plot:
<pre class="pre-highlight-in-pair">
<b>mlr -n put -f ./programs/mand.mlr</b>
<b>mlr -n put -s iheight=25 -s iwidth=50 -f ./programs/mand.mlr</b>
</pre>
<pre class="pre-non-highlight-in-pair">
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXX.XXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXooXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXX**o..*XXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXX*-....-oXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXX@XXXXXXXXXX*......o*XXXXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXXX**oo*-.-........oo.XXXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXXXXXX....................X..o-XXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@XXXXXXXXXXXXXXX*oo......................oXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@XXX*XXXXXXXXXXXX**o........................*X*X@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@XXXXXXooo***o*.*XX**X..........................o-XX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@XXXXXXXX*-.......-***.............................oXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@XXXXXXXX*@..........Xo............................*XX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@XXXX@XXXXXXXX*o@oX...........@...........................oXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
.........................................................o*XXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@XXXXXXXXX*-.oX...........@...........................oXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@XXXXXXXXXX**@..........*o............................*XXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@XXXXXXXXXXXXX-........***.............................oXXXXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@XXXXXXXXXXXXoo****o*.XX***@..........................o-XXXXXXXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@XXXXX*XXXX*XXXXXXX**-........................***XXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@XXXXXXXXXXXXX*o*.....................@o*XXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXXXXX*....................*..o-XX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXX*ooo*-.o........oo.X*XXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXX**@.....*XXXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXX*o....-o*XXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXXXo*o..*XXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXXXXXX*o*XXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXXXXX@XXXXXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXXXXXX@@XXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@XXXXX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
RCORN = -2
ICORN = -2
SIDE = 4.0
IHEIGHT = 25
IWIDTH = 50
MAXITS = 100
LEVELSTEP = 5
CHARS = @X*o-.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@XX.XX@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@XX*o.XXX@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@XX@XXXXX...oXXXXX@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@XXXXXX..........X.-X@@@@@@@@@@@@@@@@@@
@@@@@@@@XXXXXXXX*o............XX@@@@@@@@@@@@@@@@@@
@@@@@@XXXX-...-*...............X@@@@@@@@@@@@@@@@@@
@XX@XXXXoo....................XX@@@@@@@@@@@@@@@@@@
@@@XXXXX-o....................XXX@@@@@@@@@@@@@@@@@
@@@@XXXXXX-....*...............XXXXX@@@@@@@@@@@@@@
@@@@@@@XXXXX*XXX*-............*XXX@@@@@@@@@@@@@@@@
@@@@@@@@@@@@XXXXXX..........*.-X@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@XXXX*@..*XXXX@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@XXXXXoo.XXXX@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@XXXXXX@XXXX@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@XX@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
</pre>
But using a very small font size (as small as my Mac will let me go), and by choosing the coordinates to zoom in on a particular part of the complex plane, we can get a nice little picture:
@ -253,8 +222,9 @@ But using a very small font size (as small as my Mac will let me go), and by cho
# Get the number of rows and columns from the terminal window dimensions
iheight=$(stty size | mlr --nidx --fs space cut -f 1)
iwidth=$(stty size | mlr --nidx --fs space cut -f 2)
echo "rcorn=-1.755350,icorn=+0.014230,side=0.000020,maxits=10000,iheight=$iheight,iwidth=$iwidth" \
| mlr put -f programs/mand.mlr
mlr -n put \
-s rcorn=-1.755350 -s icorn=0.014230 -s side=0.000020 -s maxits=10000 -s iheight=$iheight -s iwidth=$iwidth \
-f programs/mand.mlr
</pre>
![pix/mand.png](pix/mand.png)

11
docs6/docs/programming-examples.md.in
View file

@ -16,9 +16,9 @@ GENMD_EOF
## Mandelbrot-set generator
The [Mandelbrot set](http://en.wikipedia.org/wiki/Mandelbrot_set) is also easily expressed. This isn't an important case of data-processing in the vein for which Miller was designed, but it is an example of Miller as a general-purpose programming language -- a test case for the expressiveness of the language.
The [Mandelbrot set](http://en.wikipedia.org/wiki/Mandelbrot_set) is also easily expressed. This isn't an important case of data processing (the use-case Miller was designed for), but it is an example of Miller as a general-purpose programming language -- a test case for the expressiveness of the language.
The (approximate) computation of points in the complex plane which are and aren't members is just a few lines of complex arithmetic (see the Wikipedia article); how to render them is another task. Using graphics libraries you can create PNG or JPEG files, but another fun way to do this is by printing various characters to the screen:
The (approximate) computation of points in the complex plane which are and aren't members is just a few lines of complex arithmetic (see the [Wikipedia article](https://en.wikipedia.org/wiki/Mandelbrot_set)); how to render them visually is another task. Using graphics libraries you can create PNG or JPEG files, but another fun way to do this is by printing various characters to the screen:
GENMD_RUN_COMMAND
cat programs/mand.mlr
@ -27,7 +27,7 @@ GENMD_EOF
At standard resolution this makes a nice little ASCII plot:
GENMD_RUN_COMMAND
mlr -n put -f ./programs/mand.mlr
mlr -n put -s iheight=25 -s iwidth=50 -f ./programs/mand.mlr
GENMD_EOF
But using a very small font size (as small as my Mac will let me go), and by choosing the coordinates to zoom in on a particular part of the complex plane, we can get a nice little picture:
@ -37,8 +37,9 @@ GENMD_CARDIFY
# Get the number of rows and columns from the terminal window dimensions
iheight=$(stty size | mlr --nidx --fs space cut -f 1)
iwidth=$(stty size | mlr --nidx --fs space cut -f 2)
echo "rcorn=-1.755350,icorn=+0.014230,side=0.000020,maxits=10000,iheight=$iheight,iwidth=$iwidth" \
| mlr put -f programs/mand.mlr
mlr -n put \
-s rcorn=-1.755350 -s icorn=0.014230 -s side=0.000020 -s maxits=10000 -s iheight=$iheight -s iwidth=$iwidth \
-f programs/mand.mlr
GENMD_EOF
![pix/mand.png](pix/mand.png)

200
docs6/docs/programming-language.md
View file

@ -1,7 +1,7 @@
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
# Intro to Miller's programming language
In the [Miller in 10 minutes](10min.md) page we took a tour of some of Miller's most-used verbs including `cat`, `head`, `tail`, `cut`, and `sort`. These are analogs of familiar system commands, but empowered by field-name indexing and file-format awareness: the system `sort` command only knows about lines and column names like `1,2,3,4`, while `mlr sort` knows about CSV/TSV/JSON/etc records, and field names like `color,shape,flag,index`.
In the [Miller in 10 minutes](10min.md) page we took a tour of some of Miller's most-used [verbs](reference-verbs.md) including `cat`, `head`, `tail`, `cut`, and `sort`. These are analogs of familiar system commands, but empowered by field-name indexing and file-format awareness: the system `sort` command only knows about lines and column names like `1,2,3,4`, while `mlr sort` knows about CSV/TSV/JSON/etc records, and field names like `color,shape,flag,index`.
We also caught a glimpse of Miller's `put` and `filter` verbs. These two are special since they let you express statements using Miller's programming language. It's a *embedded domain-specific language* since it's inside Miller: often referred to simply as the *Miller DSL*.
@ -9,31 +9,31 @@ In the [DSL reference](reference-dsl.md) page we have a complete reference to Mi
## Records and fields
Let's keep using the sample [example.csv](./example.csv). For example:
Let's keep using the [example.csv](./example.csv) file:
<pre class="pre-highlight-in-pair">
<b>mlr --c2p put '$cost = $quantity * $rate' example.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
color shape flag index quantity rate cost
yellow triangle true 11 43.6498 9.8870 431.5655726
red square true 15 79.2778 0.0130 1.0306114
red circle true 16 13.8103 2.9010 40.063680299999994
red square false 48 77.5542 7.4670 579.0972113999999
purple triangle false 51 81.2290 8.5910 697.8383389999999
red square false 64 77.1991 9.5310 735.7846221000001
purple triangle false 65 80.1405 5.8240 466.738272
yellow circle true 73 63.9785 4.2370 271.0769045
yellow circle true 87 63.5058 8.3350 529.3208430000001
purple square false 91 72.3735 8.2430 596.5747605000001
color shape flag k index quantity rate cost
yellow triangle true 1 11 43.6498 9.8870 431.5655726
red square true 2 15 79.2778 0.0130 1.0306114
red circle true 3 16 13.8103 2.9010 40.063680299999994
red square false 4 48 77.5542 7.4670 579.0972113999999
purple triangle false 5 51 81.2290 8.5910 697.8383389999999
red square false 6 64 77.1991 9.5310 735.7846221000001
purple triangle false 7 65 80.1405 5.8240 466.738272
yellow circle true 8 73 63.9785 4.2370 271.0769045
yellow circle true 9 87 63.5058 8.3350 529.3208430000001
purple square false 10 91 72.3735 8.2430 596.5747605000001
</pre>
When we type that, a few things are happening:
* We refer to fields in the input data using a dollar sign and then the field name, e.g. `$quantity`. (If a field name contains special characters like a dot or slash, just use curly braces: `${field.name}`.)
* The expression `$cost = $quantity * $rate` is executed once per record of the data file. Our [example.csv](./example.csv) has 10 records so this expression was executed 10 times, with the field names `$quantity` and `$rate` bound to the current record's values for those fields.
* The expression `$cost = $quantity * $rate` is executed once per record of the data file. Our [example.csv](./example.csv) has 10 records so this expression was executed 10 times, with the field names `$quantity` and `$rate` each time bound to the current record's values for those fields.
* On the left-hand side we have the new field name `$cost` which didn't come from the input data. Assignments to new variables result in a new field being placed after all the other ones. If we'd assigned to an existing field name, it would have been updated in-place.
* The entire expression is surrounded by single quotes, to get it past the system shell. Inside those, only double quotes have meaning in Miller's programming language.
* The entire expression is surrounded by single quotes (with an adjustment needed on [Windows](miller-on-windows.md)), to get it past the system shell. Inside those, only double quotes have meaning in Miller's programming language.
## Multi-line statements, and statements-from-file
@ -43,17 +43,17 @@ You can use more than one statement, separating them with semicolons, and option
<b>mlr --c2p put '$cost = $quantity * $rate; $index = $index * 100' example.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
color shape flag index quantity rate cost
yellow triangle true 1100 43.6498 9.8870 431.5655726
red square true 1500 79.2778 0.0130 1.0306114
red circle true 1600 13.8103 2.9010 40.063680299999994
red square false 4800 77.5542 7.4670 579.0972113999999
purple triangle false 5100 81.2290 8.5910 697.8383389999999
red square false 6400 77.1991 9.5310 735.7846221000001
purple triangle false 6500 80.1405 5.8240 466.738272
yellow circle true 7300 63.9785 4.2370 271.0769045
yellow circle true 8700 63.5058 8.3350 529.3208430000001
purple square false 9100 72.3735 8.2430 596.5747605000001
color shape flag k index quantity rate cost
yellow triangle true 1 1100 43.6498 9.8870 431.5655726
red square true 2 1500 79.2778 0.0130 1.0306114
red circle true 3 1600 13.8103 2.9010 40.063680299999994
red square false 4 4800 77.5542 7.4670 579.0972113999999
purple triangle false 5 5100 81.2290 8.5910 697.8383389999999
red square false 6 6400 77.1991 9.5310 735.7846221000001
purple triangle false 7 6500 80.1405 5.8240 466.738272
yellow circle true 8 7300 63.9785 4.2370 271.0769045
yellow circle true 9 8700 63.5058 8.3350 529.3208430000001
purple square false 10 9100 72.3735 8.2430 596.5747605000001
</pre>
<pre class="pre-highlight-in-pair">
@ -63,17 +63,17 @@ purple square false 9100 72.3735 8.2430 596.5747605000001
<b>' example.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
color shape flag index quantity rate cost
yellow triangle true 1100 43.6498 9.8870 431.5655726
red square true 1500 79.2778 0.0130 1.0306114
red circle true 1600 13.8103 2.9010 40.063680299999994
red square false 4800 77.5542 7.4670 579.0972113999999
purple triangle false 5100 81.2290 8.5910 697.8383389999999
red square false 6400 77.1991 9.5310 735.7846221000001
purple triangle false 6500 80.1405 5.8240 466.738272
yellow circle true 7300 63.9785 4.2370 271.0769045
yellow circle true 8700 63.5058 8.3350 529.3208430000001
purple square false 9100 72.3735 8.2430 596.5747605000001
color shape flag k index quantity rate cost
yellow triangle true 1 1100 43.6498 9.8870 431.5655726
red square true 2 1500 79.2778 0.0130 1.0306114
red circle true 3 1600 13.8103 2.9010 40.063680299999994
red square false 4 4800 77.5542 7.4670 579.0972113999999
purple triangle false 5 5100 81.2290 8.5910 697.8383389999999
red square false 6 6400 77.1991 9.5310 735.7846221000001
purple triangle false 7 6500 80.1405 5.8240 466.738272
yellow circle true 8 7300 63.9785 4.2370 271.0769045
yellow circle true 9 8700 63.5058 8.3350 529.3208430000001
purple square false 10 9100 72.3735 8.2430 596.5747605000001
</pre>
One of Miller's key features is the ability to express data-transformation right there at the keyboard, interactively. But if you find yourself using expressions repeatedly, you can put everything between the single quotes into a file and refer to that using `put -f`:
@ -90,20 +90,20 @@ $index *= 100
<b>mlr --c2p put -f dsl-example.mlr example.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
color shape flag index quantity rate cost
yellow triangle true 1100 43.6498 9.8870 431.5655726
red square true 1500 79.2778 0.0130 1.0306114
red circle true 1600 13.8103 2.9010 40.063680299999994
red square false 4800 77.5542 7.4670 579.0972113999999
purple triangle false 5100 81.2290 8.5910 697.8383389999999
red square false 6400 77.1991 9.5310 735.7846221000001
purple triangle false 6500 80.1405 5.8240 466.738272
yellow circle true 7300 63.9785 4.2370 271.0769045
yellow circle true 8700 63.5058 8.3350 529.3208430000001
purple square false 9100 72.3735 8.2430 596.5747605000001
color shape flag k index quantity rate cost
yellow triangle true 1 1100 43.6498 9.8870 431.5655726
red square true 2 1500 79.2778 0.0130 1.0306114
red circle true 3 1600 13.8103 2.9010 40.063680299999994
red square false 4 4800 77.5542 7.4670 579.0972113999999
purple triangle false 5 5100 81.2290 8.5910 697.8383389999999
red square false 6 6400 77.1991 9.5310 735.7846221000001
purple triangle false 7 6500 80.1405 5.8240 466.738272
yellow circle true 8 7300 63.9785 4.2370 271.0769045
yellow circle true 9 8700 63.5058 8.3350 529.3208430000001
purple square false 10 9100 72.3735 8.2430 596.5747605000001
</pre>
This becomes particularly important on Windows. Quite a bit of effort was put into making Miller on Windows be able to handle the kinds of single-quoted expressions we're showing here, but if you get syntax-error messages on Windows using examples in this documentation, you can put the parts between single quotes into a file and refer to that using `mlr put -f`.
This becomes particularly important on Windows. Quite a bit of effort was put into making Miller on Windows be able to handle the kinds of single-quoted expressions we're showing here, but if you get syntax-error messages on Windows using examples in this documentation, you can put the parts between single quotes into a file and refer to that using `mlr put -f` -- or, use the triple-double-quote trick as described in the [Miller on Windows page](miller-on-windows.md).
## Out-of-stream variables, begin, and end
@ -111,29 +111,29 @@ Above we saw that your expression is executed once per record -- if a file has a
Above we also saw that names like `$quantity` are bound to each record in turn.
To make `begin` and `end` statements useful, we need somewhere to put things that persist across the duration of the record stream, and a way to emit them. Miller uses **out-of-stream variables** (or **oosvars** for short) whose names start with an `@` sigil, and the **emit** keyword to write them into the output record stream:
To make `begin` and `end` statements useful, we need somewhere to put things that persist across the duration of the record stream, and a way to emit them. Miller uses [**out-of-stream variables**](reference-dsl-variables.md#out-of-stream-variables) (or **oosvars** for short) whose names start with an `@` sigil, along with the [`emit`](reference-dsl-output-statements.md#emit-statements) keyword to write them into the output record stream:
<pre class="pre-highlight-in-pair">
<b>mlr --c2p --from example.csv put 'begin { @sum = 0 } @sum += $quantity; end {emit @sum}'</b>
</pre>
<pre class="pre-non-highlight-in-pair">
color shape flag index quantity rate
yellow triangle true 11 43.6498 9.8870
red square true 15 79.2778 0.0130
red circle true 16 13.8103 2.9010
red square false 48 77.5542 7.4670
purple triangle false 51 81.2290 8.5910
red square false 64 77.1991 9.5310
purple triangle false 65 80.1405 5.8240
yellow circle true 73 63.9785 4.2370
yellow circle true 87 63.5058 8.3350
purple square false 91 72.3735 8.2430
color shape flag k index quantity rate
yellow triangle true 1 11 43.6498 9.8870
red square true 2 15 79.2778 0.0130
red circle true 3 16 13.8103 2.9010
red square false 4 48 77.5542 7.4670
purple triangle false 5 51 81.2290 8.5910
red square false 6 64 77.1991 9.5310
purple triangle false 7 65 80.1405 5.8240
yellow circle true 8 73 63.9785 4.2370
yellow circle true 9 87 63.5058 8.3350
purple square false 10 91 72.3735 8.2430
sum
652.7185
</pre>
If you want the end-block output to be the only output, and not include the input data, you can use `mlr put -q`:
If you want the end-block output to be the only output, and not include the records from the input data, you can use `mlr put -q`:
<pre class="pre-highlight-in-pair">
<b>mlr --c2p --from example.csv put -q 'begin { @sum = 0 } @sum += $quantity; end {emit @sum}'</b>
@ -167,11 +167,23 @@ sum
}
</pre>
We'll see in the documentation for [stats1](reference-verbs.md#stats1) that there's a lower-keystroking way to get counts and sums of things -- so, take this sum/count example as an indication of the kinds of things you can do using Miller's programming language.
We'll see in the documentation for [stats1](reference-verbs.md#stats1) that there's a lower-keystroking way to get counts and sums of things:
<pre class="pre-highlight-in-pair">
<b>mlr --c2j --from example.csv stats1 -a sum,count -f quantity</b>
</pre>
<pre class="pre-non-highlight-in-pair">
{
"quantity_sum": 652.7185,
"quantity_count": 10
}
</pre>
So, take this sum/count example as an indication of the kinds of things you can do using Miller's programming language.
## Context variables
Also inspired by [AWK](https://en.wikipedia.org/wiki/AWK), the Miller DSL has the following special **context variables**:
Also inspired by [AWK](https://en.wikipedia.org/wiki/AWK), the Miller DSL has the following special [**context variables**](reference-dsl-variables.md#built-in-variables):
* `FILENAME` -- the filename the current record came from. Especially useful in things like `mlr ... *.csv`.
* `FILENUM` -- similarly, but integer 1,2,3,... rather than filenam.e
@ -191,6 +203,23 @@ $filenum = FILENUM;
$newnf = NF;
</pre>
<pre class="pre-highlight-in-pair">
<b>cat data/a.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
a,b,c
1,2,3
4,5,6
</pre>
<pre class="pre-highlight-in-pair">
<b>cat data/b.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
a,b,c
7,8,9
</pre>
<pre class="pre-highlight-in-pair">
<b>mlr --c2p put -f context-example.mlr data/a.csv data/b.csv</b>
</pre>
@ -203,7 +232,7 @@ a b c nf nr fnr filename filenum newnf
## Functions and local variables
You can define your own functions:
You can [define your own functions](reference-dsl-user-defined-functions.md):
<pre class="pre-highlight-in-pair">
<b>cat factorial-example.mlr</b>
@ -222,17 +251,17 @@ func factorial(n) {
<b>mlr --c2p --from example.csv put -f factorial-example.mlr -e '$fact = factorial(NR)'</b>
</pre>
<pre class="pre-non-highlight-in-pair">
color shape flag index quantity rate fact
yellow triangle true 11 43.6498 9.8870 1
red square true 15 79.2778 0.0130 2
red circle true 16 13.8103 2.9010 6
red square false 48 77.5542 7.4670 24
purple triangle false 51 81.2290 8.5910 120
red square false 64 77.1991 9.5310 720
purple triangle false 65 80.1405 5.8240 5040
yellow circle true 73 63.9785 4.2370 40320
yellow circle true 87 63.5058 8.3350 362880
purple square false 91 72.3735 8.2430 3628800
color shape flag k index quantity rate fact
yellow triangle true 1 11 43.6498 9.8870 1
red square true 2 15 79.2778 0.0130 2
red circle true 3 16 13.8103 2.9010 6
red square false 4 48 77.5542 7.4670 24
purple triangle false 5 51 81.2290 8.5910 120
red square false 6 64 77.1991 9.5310 720
purple triangle false 7 65 80.1405 5.8240 5040
yellow circle true 8 73 63.9785 4.2370 40320
yellow circle true 9 87 63.5058 8.3350 362880
purple square false 10 91 72.3735 8.2430 3628800
</pre>
Note that here we used the `-f` flag to `put` to load our function
@ -274,12 +303,13 @@ count_of_red sum_of_red
Miller's else-if is spelled `elif`.
As we'll see more of in section (TODO:linkify), Miller has a few kinds of
for-loops. In addition to the usual 3-part `for (i = 0; i < 10; i += 1)` kind
that many programming languages have, Miller also lets you loop over arrays and
hashmaps. We haven't encountered arrays and hashmaps yet in this introduction,
but for now it suffices to know that `$*` is a special variable holding the
current record as a hashmap:
As we'll see more of in the [control-structures reference
page](reference-dsl-control-structures.md#for-loops), Miller has a few kinds
of for-loops. In addition to the usual 3-part `for (i = 0; i < 10; i += 1)`
kind that many programming languages have, Miller also lets you loop over
arrays and hashmaps. We haven't encountered arrays and hashmaps yet in this
introduction, but for now it suffices to know that `$*` is a special variable
holding the current record as a hashmap:
<pre class="pre-highlight-in-pair">
<b>cat for-example.mlr</b>
@ -320,7 +350,7 @@ Here we used the local variables `k` and `v`. Now we've seen four kinds of varia
* Local variables like `k`
* Built-in context variables like `NF` and `NR`
If you're curious about scope and extent of local variables, you can read more in (TODO:linkify) the section on variables.
If you're curious about scope and extent of local variables, you can read more in the [section on variables](reference-dsl-variables.md).
## Arithmetic
@ -330,11 +360,11 @@ Numbers in Miller's programming language are intended to operate with the princi
* Sums, differences, and products of integers are also integers (so `2*3=6` not `6.0`) -- unless the result of the operation would overflow a 64-bit signed integer in which case the result is automatically converted to float. (If you ever want integer-to-integer arithmetic, use `x .+ y`, `x .* y`, etc.)
* Quotients of integers are integers if the division is exact, else floating-point: so `6/2=3` but `7/2=3.5`.
You can read more about this at (TODO:linkify).
You can read more about this in the [arithmetic reference](reference-main-arithmetic.md).
## Absent data
In addition to types including string, number (int/float), arrays, and hashmaps, Miller varibles can also be **absent**. This is when a variable never had a value assigned to it. Miller's treatment of absent data is intended to make it easy for you to handle non-heterogeneous data. We'll see more in section (TODO:linkify) but the basic idea is:
In addition to types including string, number (int/float), arrays, and hashmaps, Miller varibles can also be **absent**. This is when a variable never had a value assigned to it. Miller's treatment of absent data is intended to make it easy for you to handle non-heterogeneous data. We'll see more in the [null-data reference](reference-main-null-data.md) but the basic idea is:
* Adding a number to absent gives the number back. This means you don't have to put `@sum = 0` in your `begin` blocks.
* Any variable which has the absent value is not assigned. This means you don't have to check presence of things from one record to the next.

53
docs6/docs/programming-language.md.in
View file

@ -1,6 +1,6 @@
# Intro to Miller's programming language
In the [Miller in 10 minutes](10min.md) page we took a tour of some of Miller's most-used verbs including `cat`, `head`, `tail`, `cut`, and `sort`. These are analogs of familiar system commands, but empowered by field-name indexing and file-format awareness: the system `sort` command only knows about lines and column names like `1,2,3,4`, while `mlr sort` knows about CSV/TSV/JSON/etc records, and field names like `color,shape,flag,index`.
In the [Miller in 10 minutes](10min.md) page we took a tour of some of Miller's most-used [verbs](reference-verbs.md) including `cat`, `head`, `tail`, `cut`, and `sort`. These are analogs of familiar system commands, but empowered by field-name indexing and file-format awareness: the system `sort` command only knows about lines and column names like `1,2,3,4`, while `mlr sort` knows about CSV/TSV/JSON/etc records, and field names like `color,shape,flag,index`.
We also caught a glimpse of Miller's `put` and `filter` verbs. These two are special since they let you express statements using Miller's programming language. It's a *embedded domain-specific language* since it's inside Miller: often referred to simply as the *Miller DSL*.
@ -8,7 +8,7 @@ In the [DSL reference](reference-dsl.md) page we have a complete reference to Mi
## Records and fields
Let's keep using the sample [example.csv](./example.csv). For example:
Let's keep using the [example.csv](./example.csv) file:
GENMD_RUN_COMMAND
mlr --c2p put '$cost = $quantity * $rate' example.csv
@ -17,9 +17,9 @@ GENMD_EOF
When we type that, a few things are happening:
* We refer to fields in the input data using a dollar sign and then the field name, e.g. `$quantity`. (If a field name contains special characters like a dot or slash, just use curly braces: `${field.name}`.)
* The expression `$cost = $quantity * $rate` is executed once per record of the data file. Our [example.csv](./example.csv) has 10 records so this expression was executed 10 times, with the field names `$quantity` and `$rate` bound to the current record's values for those fields.
* The expression `$cost = $quantity * $rate` is executed once per record of the data file. Our [example.csv](./example.csv) has 10 records so this expression was executed 10 times, with the field names `$quantity` and `$rate` each time bound to the current record's values for those fields.
* On the left-hand side we have the new field name `$cost` which didn't come from the input data. Assignments to new variables result in a new field being placed after all the other ones. If we'd assigned to an existing field name, it would have been updated in-place.
* The entire expression is surrounded by single quotes, to get it past the system shell. Inside those, only double quotes have meaning in Miller's programming language.
* The entire expression is surrounded by single quotes (with an adjustment needed on [Windows](miller-on-windows.md)), to get it past the system shell. Inside those, only double quotes have meaning in Miller's programming language.
## Multi-line statements, and statements-from-file
@ -41,7 +41,7 @@ GENMD_RUN_COMMAND
mlr --c2p put -f dsl-example.mlr example.csv
GENMD_EOF
This becomes particularly important on Windows. Quite a bit of effort was put into making Miller on Windows be able to handle the kinds of single-quoted expressions we're showing here, but if you get syntax-error messages on Windows using examples in this documentation, you can put the parts between single quotes into a file and refer to that using `mlr put -f`.
This becomes particularly important on Windows. Quite a bit of effort was put into making Miller on Windows be able to handle the kinds of single-quoted expressions we're showing here, but if you get syntax-error messages on Windows using examples in this documentation, you can put the parts between single quotes into a file and refer to that using `mlr put -f` -- or, use the triple-double-quote trick as described in the [Miller on Windows page](miller-on-windows.md).
## Out-of-stream variables, begin, and end
@ -49,13 +49,13 @@ Above we saw that your expression is executed once per record -- if a file has a
Above we also saw that names like `$quantity` are bound to each record in turn.
To make `begin` and `end` statements useful, we need somewhere to put things that persist across the duration of the record stream, and a way to emit them. Miller uses **out-of-stream variables** (or **oosvars** for short) whose names start with an `@` sigil, and the **emit** keyword to write them into the output record stream:
To make `begin` and `end` statements useful, we need somewhere to put things that persist across the duration of the record stream, and a way to emit them. Miller uses [**out-of-stream variables**](reference-dsl-variables.md#out-of-stream-variables) (or **oosvars** for short) whose names start with an `@` sigil, along with the [`emit`](reference-dsl-output-statements.md#emit-statements) keyword to write them into the output record stream:
GENMD_RUN_COMMAND
mlr --c2p --from example.csv put 'begin { @sum = 0 } @sum += $quantity; end {emit @sum}'
GENMD_EOF
If you want the end-block output to be the only output, and not include the input data, you can use `mlr put -q`:
If you want the end-block output to be the only output, and not include the records from the input data, you can use `mlr put -q`:
GENMD_RUN_COMMAND
mlr --c2p --from example.csv put -q 'begin { @sum = 0 } @sum += $quantity; end {emit @sum}'
@ -74,11 +74,17 @@ mlr --c2j --from example.csv put -q '
'
GENMD_EOF
We'll see in the documentation for [stats1](reference-verbs.md#stats1) that there's a lower-keystroking way to get counts and sums of things -- so, take this sum/count example as an indication of the kinds of things you can do using Miller's programming language.
We'll see in the documentation for [stats1](reference-verbs.md#stats1) that there's a lower-keystroking way to get counts and sums of things:
GENMD_RUN_COMMAND
mlr --c2j --from example.csv stats1 -a sum,count -f quantity
GENMD_EOF
So, take this sum/count example as an indication of the kinds of things you can do using Miller's programming language.
## Context variables
Also inspired by [AWK](https://en.wikipedia.org/wiki/AWK), the Miller DSL has the following special **context variables**:
Also inspired by [AWK](https://en.wikipedia.org/wiki/AWK), the Miller DSL has the following special [**context variables**](reference-dsl-variables.md#built-in-variables):
* `FILENAME` -- the filename the current record came from. Especially useful in things like `mlr ... *.csv`.
* `FILENUM` -- similarly, but integer 1,2,3,... rather than filenam.e
@ -90,13 +96,21 @@ GENMD_RUN_COMMAND
cat context-example.mlr
GENMD_EOF
GENMD_RUN_COMMAND
cat data/a.csv
GENMD_EOF
GENMD_RUN_COMMAND
cat data/b.csv
GENMD_EOF
GENMD_RUN_COMMAND
mlr --c2p put -f context-example.mlr data/a.csv data/b.csv
GENMD_EOF
## Functions and local variables
You can define your own functions:
You can [define your own functions](reference-dsl-user-defined-functions.md):
GENMD_RUN_COMMAND
cat factorial-example.mlr
@ -126,12 +140,13 @@ GENMD_EOF
Miller's else-if is spelled `elif`.
As we'll see more of in section (TODO:linkify), Miller has a few kinds of
for-loops. In addition to the usual 3-part `for (i = 0; i < 10; i += 1)` kind
that many programming languages have, Miller also lets you loop over arrays and
hashmaps. We haven't encountered arrays and hashmaps yet in this introduction,
but for now it suffices to know that `$*` is a special variable holding the
current record as a hashmap:
As we'll see more of in the [control-structures reference
page](reference-dsl-control-structures.md#for-loops), Miller has a few kinds
of for-loops. In addition to the usual 3-part `for (i = 0; i < 10; i += 1)`
kind that many programming languages have, Miller also lets you loop over
arrays and hashmaps. We haven't encountered arrays and hashmaps yet in this
introduction, but for now it suffices to know that `$*` is a special variable
holding the current record as a hashmap:
GENMD_RUN_COMMAND
cat for-example.mlr
@ -152,7 +167,7 @@ Here we used the local variables `k` and `v`. Now we've seen four kinds of varia
* Local variables like `k`
* Built-in context variables like `NF` and `NR`
If you're curious about scope and extent of local variables, you can read more in (TODO:linkify) the section on variables.
If you're curious about scope and extent of local variables, you can read more in the [section on variables](reference-dsl-variables.md).
## Arithmetic
@ -162,11 +177,11 @@ Numbers in Miller's programming language are intended to operate with the princi
* Sums, differences, and products of integers are also integers (so `2*3=6` not `6.0`) -- unless the result of the operation would overflow a 64-bit signed integer in which case the result is automatically converted to float. (If you ever want integer-to-integer arithmetic, use `x .+ y`, `x .* y`, etc.)
* Quotients of integers are integers if the division is exact, else floating-point: so `6/2=3` but `7/2=3.5`.
You can read more about this at (TODO:linkify).
You can read more about this in the [arithmetic reference](reference-main-arithmetic.md).
## Absent data
In addition to types including string, number (int/float), arrays, and hashmaps, Miller varibles can also be **absent**. This is when a variable never had a value assigned to it. Miller's treatment of absent data is intended to make it easy for you to handle non-heterogeneous data. We'll see more in section (TODO:linkify) but the basic idea is:
In addition to types including string, number (int/float), arrays, and hashmaps, Miller varibles can also be **absent**. This is when a variable never had a value assigned to it. Miller's treatment of absent data is intended to make it easy for you to handle non-heterogeneous data. We'll see more in the [null-data reference](reference-main-null-data.md) but the basic idea is:
* Adding a number to absent gives the number back. This means you don't have to put `@sum = 0` in your `begin` blocks.
* Any variable which has the absent value is not assigned. This means you don't have to check presence of things from one record to the next.

68
docs6/docs/programs/mand.mlr
View file

@ -1,39 +1,25 @@
# Mandelbrot set generator: simple example of Miller DSL as programming language.
begin {
# Set defaults
@rcorn = -2.0;
@icorn = -2.0;
@side = 4.0;
@iheight = 50;
@iwidth = 100;
@maxits = 100;
@levelstep = 5;
@chars = "@X*o-."; # Palette of characters to print to the screen.
@verbose = false;
@do_julia = false;
@jr = 0.0; # Real part of Julia point, if any
@ji = 0.0; # Imaginary part of Julia point, if any
# Set defaults. They can be overridden by e.g.
# mlr -n put -e 'begin{@maxits=200}' -f nameofthisfile.mlr
# or
# mlr -n put -s maxits=200 -f nameofthisfile.mlr
@rcorn ??= -2.0;
@icorn ??= -2.0;
@side ??= 4.0;
@iheight ??= 50;
@iwidth ??= 100;
@maxits ??= 100;
@levelstep ??= 5;
@chars ??= "@X*o-.";
@silent ??= false;
@do_julia ??= false;
@jr ??= 0.0; # Real part of Julia point, if any
@ji ??= 0.0; # Imaginary part of Julia point, if any
}
# Here, we can override defaults from an input file (if any). In Miller's
# put/filter DSL, absent-null right-hand sides result in no assignment so we
# can simply put @rcorn = $rcorn: if there is a field in the input like
# 'rcorn = -1.847' we'll read and use it, else we'll keep the default.
@rcorn = $rcorn;
@icorn = $icorn;
@side = $side;
@iheight = $iheight;
@iwidth = $iwidth;
@maxits = $maxits;
@levelstep = $levelstep;
@chars = $chars;
@verbose = $verbose;
@do_julia = $do_julia;
@jr = $jr;
@ji = $ji;
end {
if (@verbose) {
if (!@silent) {
print "RCORN = ".@rcorn;
print "ICORN = ".@icorn;
print "SIDE = ".@side;
@ -44,21 +30,22 @@ end {
print "CHARS = ".@chars;
}
# Iterate over a matrix of rows and columns, printing one character for each cell.
for (int ii = @iheight-1; ii >= 0; ii -= 1) {
num pi = @icorn + (ii/@iheight) * @side;
num ci = @icorn + (ii/@iheight) * @side;
for (int ir = 0; ir < @iwidth; ir += 1) {
num pr = @rcorn + (ir/@iwidth) * @side;
printn get_point_plot(pr, pi, @maxits, @do_julia, @jr, @ji);
num cr = @rcorn + (ir/@iwidth) * @side;
str c = get_point_plot(cr, ci, @maxits, @do_julia, @jr, @ji);
if (!@silent) {
printn c
}
}
if (!@silent) {
print;
}
print;
}
}
# This is a function to approximate membership in the Mandelbrot set (or Julia
# set for a given Julia point if do_julia == true) for a given point in the
# complex plane.
func get_point_plot(pr, pi, maxits, do_julia, jr, ji) {
func get_point_plot(num pr, num pi, int maxits, bool do_julia, num jr, num ji): str {
num zr = 0.0;
num zi = 0.0;
num cr = 0.0;
@ -93,7 +80,6 @@ func get_point_plot(pr, pi, maxits, do_julia, jr, ji) {
if (!escaped) {
return ".";
} else {
# The // operator is Miller's (pythonic) integer-division operator
int level = (iti // @levelstep) % strlen(@chars);
return substr(@chars, level, level);
}

55
docs6/docs/programs/mand.sh
View file

@ -1,33 +1,56 @@
#!/bin/bash
set -euo pipefail
iheight=$(stty size | mlr --nidx --fs space cut -f 1)
iwidth=$(stty size | mlr --nidx --fs space cut -f 2)
rcorn=-2.0
icorn=-2.0
side=4.0
iheight=50
iwidth=100
maxits=100
do_julia=false
jr=0.0
ji=0.0
if [ "$1" = "1" ]; then
echo "rcorn=-1.787582,icorn=-0.000002,side=0.000004,maxits=1000,iheight=$iheight,iwidth=$iwidth"
rcorn=-1.787582;icorn=-0.000002;side=0.000004;maxits=1000
elif [ "$1" = "2" ]; then
echo "rcorn=-0.162950,icorn=+1.026100,side=0.000200,maxits=100000,iheight=$iheight,iwidth=$iwidth"
rcorn=-0.162950;icorn=1.026100;side=0.000200;maxits=100000
elif [ "$1" = "3" ]; then
echo "rcorn=-1.755350,icorn=+0.014230,side=0.000020,maxits=10000,iheight=$iheight,iwidth=$iwidth"
rcorn=-1.755350;icorn=0.014230;side=0.000020;maxits=10000
elif [ "$1" = 4 ]; then
echo "do_julia=true,jr= 0.35,ji=0.35,maxits=1000,iheight=$iheight,iwidth=$iwidth"
do_julia=true;jr=0.35;ji=0.35;maxits=1000
elif [ "$1" = 5 ]; then
echo "do_julia=true,jr= 0.0,ji=0.63567,iheight=$iheight,iwidth=$iwidth"
do_julia=true;jr=0.0;ji=0.63567
elif [ "$1" = 6 ]; then
echo "do_julia=true,jr= 0.4,ji=0.34745,maxits=1000,iheight=$iheight,iwidth=$iwidth"
do_julia=true;jr=0.4;ji=0.34745;maxits=1000
elif [ "$1" = 7 ]; then
echo "do_julia=true,jr= 0.36,ji=0.36,maxits=80,iheight=$iheight,iwidth=$iwidth"
do_julia=true;jr=0.36;ji=0.36;maxits=80
elif [ "$1" = 8 ]; then
echo "do_julia=true,jr=-0.55,ji=0.55,maxits=100,iheight=$iheight,iwidth=$iwidth"
do_julia=true;jr=-0.55;ji=0.55;maxits=100
elif [ "$1" = 9 ]; then
echo "do_julia=true,jr=-0.51,ji=0.51,maxits=1000,iheight=$iheight,iwidth=$iwidth"
do_julia=true;jr=-0.51;ji=0.51;maxits=1000
elif [ "$1" = 10 ]; then
echo "do_julia=true,jr=-1.26,ji=-0.03,rcorn=-0.3,icorn=-0.3,side=0.6,iheight=$iheight,iwidth=$iwidth"
do_julia=true;jr=-1.26;ji=-0.03;rcorn=-0.3;icorn=-0.3;side=0.6
elif [ "$1" = 11 ]; then
echo "do_julia=true,jr=-1.26,ji=-0.03,rcorn=-0.6,icorn=-0.6,side=.2,iheight=$iheight,iwidth=$iwidth"
do_julia=true;jr=-1.26;ji=-0.03;rcorn=-0.6;icorn=-0.6;side=0.2
elif [ "$1" = 12 ]; then
echo "do_julia=true,jr=-1.26,ji=-0.03,rcorn=-0.75,icorn=-0.03125,side=.0625,iheight=$iheight,iwidth=$iwidth"
do_julia=true;jr=-1.26;ji=-0.03;rcorn=-0.75;icorn=-0.03125;side=.0625
elif [ "$1" = 13 ]; then
echo "do_julia=true,jr=-1.26,ji=-0.03,rcorn=-0.75,icorn=-0.01,side=.02,iheight=$iheight,iwidth=$iwidth"
else
echo "iheight=$iheight,iwidth=$iwidth"
fi | mlr put -f programs/mand.mlr
do_julia=true;jr=-1.26;ji=-0.03;rcorn=-0.75;icorn=-0.01;side=.02
fi
mlr -n put \
-s rcorn=$rcorn \
-s icorn=$icorn \
-s side=$side \
-s iheight=$iheight \
-s iwidth=$iwidth \
-s maxits=$maxits \
-s do_julia=$do_julia \
-s jr=$jr \
-s ji=$ji \
-f programs/mand.mlr

199
docs6/docs/proofreads.txt Normal file
View file

@ -0,0 +1,199 @@
----------------------------------------------------------------
ALL:
* unvisited links are still blue -- ?!?
* GENMD_INCLUDE_AND_RUN_ESCAPED -> remove and replace with GENMD_RUN_COMMAND
* example.csv rename index to something else, and add i column, update any code samples which use index
* csv to csv,tsv throughout
* rid of explicitly passing around os.Stdout in all various help functions, annoying
* het.dkvp > het.json in more places
* check each page for adequate h2 coverage
----------------------------------------------------------------
w compression page: make one! :)
w flatten/unflatten page: make one! :)
w flesh out arrays page!
c GOMAXPROCS -- up it? separate page maybe -- ?
- note one goroutine for in, out, & each verb
- check and respect env-var
* move aux-cmds down lower. maybe some other reorders as well.
* new different-from-other-languages page
- no ++
- 1-up arrays
for (i = 1; i <= n; i += 1) { ... }
- hash-maps are order-preserving
- single-for over array: var is value; over map: var is key
----------------------------------------------------------------
index:
E CSV -> CSV/TSV
features:
E CSV -> CSV/TSV
feature-comparison:
E even with newlines -- show an example
file-formats:
* line-wrap the static text
* => to :
* remove "." on numbers
w tabularize by concatenating keys: update
o JSON-to-JSON
o JSON-to-not
o not-to-JSON
o not-to-not
l arrays linkify
? Formatting JSON Options: check each flag is still current
? Data-conversion keystroke-savers: to/from separate page. and/or merge.
? autodetect: hmmmmmmmm
record-heterogeneity:
? current csv-reader code handles this? i think not. check csvlite-reader too.
new-in-miller-6:
w flatten/unflatten -- needs a new separate page
l gzin/bz2in linkify
? TODO marks
contributing:
L add a pre-release note about https://johnkerl.org/miller6 & why no double RTD
E update for sphinx -> mkdocs. and/or link to r.md.
csv-with-and-without-headers:
? Headerless CSV with duplicate field values -> typo-fix -- duplicate keys actually -- ?!?
? cat out nas.csv first
? ag -> just grep ?
l Miller handles compliant CSV: include rfc4180 link
? Regularizing ragged CSV: use regularize verb?!?
shapes-of-data:
? autodetect -- remove?
E data/small -> example.csv (and throughout ...)
E Rectangularizing data: cf regularize verb, & other page?
special-symbols-and-formatting:
? mlr --icsv --odkvp --ofs '\001' cat commas.csv | cat -v
o test this
o look for unbackslash callsites in the C impl
o SeparatorFromArg
? How to escape '?' in regexes? -> still true? link to torbiak297?
operating-on-all-records:
o linkify map/array/etc
o 1-up link at end
o insord link at end
o int2str@key link at end
dates-and-times:
? Caveat: localtime-handling in timezones with DST is still a work in progress -- fixed by torbiak?
shell-commands:
e while-read example from issues
data-diving-examples:
? mlr --from data/flins.csv --icsv --opprint \
stats1 -a min,mean,max -f tiv_2012 -g construction,line & next -- why zero output?
? mlr --from data/flins.csv --icsv --opprint \
stats2 -a corr,linreg-ols,r2 -f tiv_2011,tiv_2012 -g county -- more "_" b04k
E head -n 6 data/colored-shapes.dkvp | mlr --opprint cat -- regen w/ fewer decimal places to avoid line-wraps
E have a "conclusions:" ... w/ backrefs to generator scceipt & code-snippets
log-processing-examples:
l or Miller's having-fields, or is_present -> DSL is_present, & linkify
? grep 'various sorts' *.log \
| sed 's/.*} //' \
| mlr --fs space --repifs --oxtab stats1 -a min,p10,p50,p90,max -f time -g status -- why no output?
sql-examples:
l Again, all the examples in the CSV section apply here -- linkify "CSV section"
data-cleaning-examples:
? mlr --icsv --opprint put '$format_ok = is_string($reachable)' data/het-bool.csv -- do false/true autotype -- ?
statistics-examples:
? 2nd card why no output?
two-pass-algorithms:
l link to "new" verbs x everywhere possible
l Of course, Miller verbs such as sort, tac, etc. all must ... -> linkify to new memory/streaming page
misc examples:
? Program timing & subsequents -> another page
reference-verbs:
E mlr --opprint bar --auto -f x,y data/small -> use some int data to avoid line-wraps
E data/colored-shapes.dkvp (this page & throughout) a CSV file ...
* ... I ONLY READ UP TO CUT & PAUSED ...
reference-main-io-options:
E Compression: include new options
reference-main-then-chaining:
c On the other hand, Miller is single-threaded, while modern systems are usually multi-processor, and when streaming-data programs operate through pipes, each one can use a CPU. -> update for Go ... MAXPROCS ...
reference-main-auxiliary-commands:
c mlr lecat -> have the code use colors stuff not hard-coded -- ? patch into env vars -- ?
c mlr lecat -> code check isatty
reference-main-data-types:
E field values are usually strings -> update
E Field values are treated as numeric for the following: -> update
? Miller's types for function processing are -> add JSON null; & check for others (array/map)
E true/false -> add info about NaN and Inf et al.
!! very much missing stuff!! where's the listing of mlrval types?!?
- reference-dsl-variables.md.in should link to it
- ditto programming-language.md.in
c UTs for r"a" . r"b" and so on and so on
reference-main-null-data:
? more variants of is_... ?
? Records with one or more empty sort-field values sort after records with all sort-field values present -> apparently not true for sort -nr
reference-main-arithmetic:
? remove references to put/filter -F
? test stats1/step -F flag
? The following, however, produce integer output if their inputs are integers: + - * / // % abs ceil floor max min round roundm sgn. As well, stats1 -a min, stats1 -a max, stats1 -a sum, step -a delta, and step -a rsum produce integer output if their inputs are integers. -> test all
reference-main-regular-expressions:
? regular expressions (of type POSIX.2) -> check/update; also link to Go page
d separate escaping for "\t" etc in arg-2/regex position -- "\t"."\t" example as well ...
reference-main-overview:
E is that it is designed for -> write and link to streaming-operations page
reference-dsl-variables:
E cat data/type-decl-example.mlr -> narrow it
reference-dsl-operators:
c Operator precedence: add >>>; check for other new operators (like ??)
c code: ** operator have int-int-to-int check?
reference-dsl-control-structures:
E emph p-a blocks are syntactic sugar for if-statements
? mlr put '$x > 0.0; $y = log10($x); $z = sqrt($y)' data/put-gating-example-1.dkvp -> no longer true -- wtf? is this doc-snippet non-live?
e while (NR < 10) will never terminate as NR is only incremented between records -> and each expression is invoked once per record so once for NR=1, once for NR=2, etc.
E C-style triple-for loops: loop to NR -> NO!!!
? or if a same-name variable is present in outer scope then it is modified. -> check that and make sure there is a UT
l Since uninitialized out-of-stream variables default to 0 for addition/substraction and 1 for multiplication when they appear on expression right-hand sides (not quite as in awk, where they'd default to 0 either way) <-> xlink to other page
reference-dsl-user-defined-functions:
E User-defined subroutines -> non-factorial example -- maybe some useful aggregator
reference-dsl-builtin-functions:
c mlr: option "--list-all-functions-as-table" not recognized. Please run "mlr --help" for usage information -> fix
! ... need to proofread entire list ...
reference-dsl-output-statements:
? NR %% 1000 == 0 -> why %%
E Keep in mind that out-of-stream variables are a nested, multi-level hashmap (directly viewable as JSON using dump), whereas Miller output records are lists of single-level key-value pairs -> link to flatten page
? Use --oflatsep -> check
? mlr --oxtab put -q --oflatsep / '
o 2 examples not 3?
o why not '--oflatsep /' respected?
manpage:
? [NEEDS READ-THROUGH]

1
docs6/docs/randomizing-examples.md
View file

@ -41,7 +41,6 @@ Namely:
* Set the Miller random-number seed so this webdoc looks the same every time I regenerate it.
* Use pretty-printed tabular output.
* Use pretty-printed tabular output.
* Use `seqgen` to produce 100,000 records `i=0`, `i=1`, etc.
* Send those to a `put` step which defines an inverse-transform-sampling function and calls it twice, then computes the sum and product of samples.
* Send those to a histogram, and from there to a bar-plotter. This is just for visualization; you could just as well output CSV and send that off to your own plotting tool, etc.

1
docs6/docs/randomizing-examples.md.in
View file

@ -12,7 +12,6 @@ Namely:
* Set the Miller random-number seed so this webdoc looks the same every time I regenerate it.
* Use pretty-printed tabular output.
* Use pretty-printed tabular output.
* Use `seqgen` to produce 100,000 records `i=0`, `i=1`, etc.
* Send those to a `put` step which defines an inverse-transform-sampling function and calls it twice, then computes the sum and product of samples.
* Send those to a histogram, and from there to a bar-plotter. This is just for visualization; you could just as well output CSV and send that off to your own plotting tool, etc.

114
docs6/docs/record-heterogeneity.md
View file

@ -12,38 +12,36 @@ But heterogeneous data abound (today's no-SQL databases for example). Miller han
Miller simply prints a newline and a new header when there is a schema change. When there is no schema change, you get CSV per se as a special case. Likewise, Miller reads heterogeneous CSV or pretty-print input the same way. The difference between CSV and CSV-lite is that the former is RFC4180-compliant, while the latter readily handles heterogeneous data (which is non-compliant). For example:
<pre class="pre-highlight-in-pair">
<b>cat data/het.dkvp</b>
<b>cat data/het.json</b>
</pre>
<pre class="pre-non-highlight-in-pair">
resource=/path/to/file,loadsec=0.45,ok=true
record_count=100,resource=/path/to/file
resource=/path/to/second/file,loadsec=0.32,ok=true
record_count=150,resource=/path/to/second/file
resource=/some/other/path,loadsec=0.97,ok=false
{
"resource": "/path/to/file",
"loadsec": 0.45,
"ok": true
}
{
"record_count": 100,
"resource": "/path/to/file"
}
{
"resource": "/path/to/second/file",
"loadsec": 0.32,
"ok": true
}
{
"record_count": 150,
"resource": "/path/to/second/file"
}
{
"resource": "/some/other/path",
"loadsec": 0.97,
"ok": false
}
</pre>
<pre class="pre-highlight-in-pair">
<b>mlr --ocsvlite cat data/het.dkvp</b>
</pre>
<pre class="pre-non-highlight-in-pair">
resource,loadsec,ok
/path/to/file,0.45,true
record_count,resource
100,/path/to/file
resource,loadsec,ok
/path/to/second/file,0.32,true
record_count,resource
150,/path/to/second/file
resource,loadsec,ok
/some/other/path,0.97,false
</pre>
<pre class="pre-highlight-in-pair">
<b>mlr --opprint cat data/het.dkvp</b>
<b>mlr --ijson --opprint cat data/het.json</b>
</pre>
<pre class="pre-non-highlight-in-pair">
resource loadsec ok
@ -62,7 +60,7 @@ resource loadsec ok
/some/other/path 0.97 false
</pre>
Miller handles explicit header changes as just shown. If your CSV input contains ragged data -- if there are implicit header changes -- you can use `--allow-ragged-csv-input` (or keystroke-saver `--ragged`). For too-short data lines, values are filled with empty string; for too-long data lines, missing field names are replaced with positional indices (counting up from 1, not 0), as follows:
Miller handles explicit header changes as just shown. If your CSV input contains ragged data -- if there are implicit header changes (no intervening blank line and new header line) -- you can use `--allow-ragged-csv-input` (or keystroke-saver `--ragged`). For too-short data lines, values are filled with empty string; for too-long data lines, missing field names are replaced with positional indices (counting up from 1, not 0), as follows:
<pre class="pre-highlight-in-pair">
<b>cat data/ragged.csv</b>
@ -95,21 +93,27 @@ c 8
You may also find Miller's `group-like` feature handy (see also [Verbs Reference](reference-verbs.md)):
<pre class="pre-highlight-in-pair">
<b>mlr --ocsvlite group-like data/het.dkvp</b>
<b>mlr --j2p cat data/het.json</b>
</pre>
<pre class="pre-non-highlight-in-pair">
resource,loadsec,ok
/path/to/file,0.45,true
/path/to/second/file,0.32,true
/some/other/path,0.97,false
resource loadsec ok
/path/to/file 0.45 true
record_count,resource
100,/path/to/file
150,/path/to/second/file
record_count resource
100 /path/to/file
resource loadsec ok
/path/to/second/file 0.32 true
record_count resource
150 /path/to/second/file
resource loadsec ok
/some/other/path 0.97 false
</pre>
<pre class="pre-highlight-in-pair">
<b>mlr --opprint group-like data/het.dkvp</b>
<b>mlr --j2p group-like data/het.json</b>
</pre>
<pre class="pre-non-highlight-in-pair">
resource loadsec ok
@ -127,18 +131,36 @@ record_count resource
For these formats, record-heterogeneity comes naturally:
<pre class="pre-highlight-in-pair">
<b>cat data/het.dkvp</b>
<b>cat data/het.json</b>
</pre>
<pre class="pre-non-highlight-in-pair">
resource=/path/to/file,loadsec=0.45,ok=true
record_count=100,resource=/path/to/file
resource=/path/to/second/file,loadsec=0.32,ok=true
record_count=150,resource=/path/to/second/file
resource=/some/other/path,loadsec=0.97,ok=false
{
"resource": "/path/to/file",
"loadsec": 0.45,
"ok": true
}
{
"record_count": 100,
"resource": "/path/to/file"
}
{
"resource": "/path/to/second/file",
"loadsec": 0.32,
"ok": true
}
{
"record_count": 150,
"resource": "/path/to/second/file"
}
{
"resource": "/some/other/path",
"loadsec": 0.97,
"ok": false
}
</pre>
<pre class="pre-highlight-in-pair">
<b>mlr --onidx --ofs ' ' cat data/het.dkvp</b>
<b>mlr --ijson --onidx --ofs ' ' cat data/het.json</b>
</pre>
<pre class="pre-non-highlight-in-pair">
/path/to/file 0.45 true
@ -149,7 +171,7 @@ resource=/some/other/path,loadsec=0.97,ok=false
</pre>
<pre class="pre-highlight-in-pair">
<b>mlr --oxtab cat data/het.dkvp</b>
<b>mlr --ijson --oxtab cat data/het.json</b>
</pre>
<pre class="pre-non-highlight-in-pair">
resource /path/to/file
@ -172,7 +194,7 @@ ok false
</pre>
<pre class="pre-highlight-in-pair">
<b>mlr --oxtab group-like data/het.dkvp</b>
<b>mlr --ijson --oxtab group-like data/het.json</b>
</pre>
<pre class="pre-non-highlight-in-pair">
resource /path/to/file

22
docs6/docs/record-heterogeneity.md.in
View file

@ -11,18 +11,14 @@ But heterogeneous data abound (today's no-SQL databases for example). Miller han
Miller simply prints a newline and a new header when there is a schema change. When there is no schema change, you get CSV per se as a special case. Likewise, Miller reads heterogeneous CSV or pretty-print input the same way. The difference between CSV and CSV-lite is that the former is RFC4180-compliant, while the latter readily handles heterogeneous data (which is non-compliant). For example:
GENMD_RUN_COMMAND
cat data/het.dkvp
cat data/het.json
GENMD_EOF
GENMD_RUN_COMMAND
mlr --ocsvlite cat data/het.dkvp
mlr --ijson --opprint cat data/het.json
GENMD_EOF
GENMD_RUN_COMMAND
mlr --opprint cat data/het.dkvp
GENMD_EOF
Miller handles explicit header changes as just shown. If your CSV input contains ragged data -- if there are implicit header changes -- you can use `--allow-ragged-csv-input` (or keystroke-saver `--ragged`). For too-short data lines, values are filled with empty string; for too-long data lines, missing field names are replaced with positional indices (counting up from 1, not 0), as follows:
Miller handles explicit header changes as just shown. If your CSV input contains ragged data -- if there are implicit header changes (no intervening blank line and new header line) -- you can use `--allow-ragged-csv-input` (or keystroke-saver `--ragged`). For too-short data lines, values are filled with empty string; for too-long data lines, missing field names are replaced with positional indices (counting up from 1, not 0), as follows:
GENMD_RUN_COMMAND
cat data/ragged.csv
@ -35,11 +31,11 @@ GENMD_EOF
You may also find Miller's `group-like` feature handy (see also [Verbs Reference](reference-verbs.md)):
GENMD_RUN_COMMAND
mlr --ocsvlite group-like data/het.dkvp
mlr --j2p cat data/het.json
GENMD_EOF
GENMD_RUN_COMMAND
mlr --opprint group-like data/het.dkvp
mlr --j2p group-like data/het.json
GENMD_EOF
### Key-value-pair, vertical-tabular, and index-numbered formats
@ -47,19 +43,19 @@ GENMD_EOF
For these formats, record-heterogeneity comes naturally:
GENMD_RUN_COMMAND
cat data/het.dkvp
cat data/het.json
GENMD_EOF
GENMD_RUN_COMMAND
mlr --onidx --ofs ' ' cat data/het.dkvp
mlr --ijson --onidx --ofs ' ' cat data/het.json
GENMD_EOF
GENMD_RUN_COMMAND
mlr --oxtab cat data/het.dkvp
mlr --ijson --oxtab cat data/het.json
GENMD_EOF
GENMD_RUN_COMMAND
mlr --oxtab group-like data/het.dkvp
mlr --ijson --oxtab group-like data/het.json
GENMD_EOF
## For processing

2
docs6/docs/reference-dsl-arrays.md
View file

@ -1,5 +1,5 @@
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
# Reference: arrays
# Arrays
TODO

2
docs6/docs/reference-dsl-arrays.md.in
View file

@ -1,4 +1,4 @@
# Reference: arrays
# Arrays
TODO

13
docs6/docs/reference-dsl-builtin-functions.md
View file

@ -1,5 +1,5 @@
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
# DSL reference: built-in functions
# DSL built-in functions
## Summary
@ -13,7 +13,7 @@ Each function takes a specific number of arguments, as shown below, except for f
You can get a list of all functions using **mlr -f**, with details using **mlr -F**.
<a id=colon />
<a id=exclamation-point />
## \!
<pre class="pre-non-highlight-non-pair">
@ -28,6 +28,7 @@ You can get a list of all functions using **mlr -f**, with details using **mlr -
</pre>
<a id=regnotmatch />
## !=~
<pre class="pre-non-highlight-non-pair">
@ -42,6 +43,7 @@ You can get a list of all functions using **mlr -f**, with details using **mlr -
</pre>
<a id=bitwise-and />
## &
<pre class="pre-non-highlight-non-pair">
@ -49,6 +51,7 @@ You can get a list of all functions using **mlr -f**, with details using **mlr -
</pre>
<a id=logical-and />
## &&
<pre class="pre-non-highlight-non-pair">
@ -165,6 +168,7 @@ You can get a list of all functions using **mlr -f**, with details using **mlr -
</pre>
<a id=regmatch />
## =~
<pre class="pre-non-highlight-non-pair">
@ -224,6 +228,7 @@ You can get a list of all functions using **mlr -f**, with details using **mlr -
</pre>
<a id=bitwise-xor />
## ^
<pre class="pre-non-highlight-non-pair">
@ -231,6 +236,7 @@ You can get a list of all functions using **mlr -f**, with details using **mlr -
</pre>
<a id=logical-xor />
## ^^
<pre class="pre-non-highlight-non-pair">
@ -246,7 +252,8 @@ You can get a list of all functions using **mlr -f**, with details using **mlr -
</pre>
## ||
<a id=logical-or />
## \|\|
<pre class="pre-non-highlight-non-pair">
|| (class=boolean #args=2) Logical OR.

2
docs6/docs/reference-dsl-builtin-functions.md.in
View file

@ -1,4 +1,4 @@
# DSL reference: built-in functions
# DSL built-in functions
## Summary

33
docs6/docs/reference-dsl-complexity.md
View file

@ -1,9 +1,34 @@
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
# DSL reference: a note on the complexity of Miller's expression language
# A note on the complexity of Miller's expression language
One of Miller's strengths is its brevity: it's much quicker -- and less error-prone -- to type `mlr stats1 -a sum -f x,y -g a,b` than having to track summation variables as in `awk`, or using Miller's out-of-stream variables. And the more language features Miller's put-DSL has (for-loops, if-statements, nested control structures, user-defined functions, etc.) then the *less* powerful it begins to seem: because of the other programming-language features it *doesn't* have (classes, exceptions, and so on).
One of Miller's strengths is its brevity: it's much quicker -- and less
error-prone -- to type `mlr stats1 -a sum -f x,y -g a,b` than having to track
summation variables as in `awk`, or using Miller's [out-of-stream
variables](reference-dsl-variables.md#out-of-stream-variables). And the more
language features Miller's put-DSL has (for-loops, if-statements, nested
control structures, user-defined functions, etc.) then the *less* powerful it
begins to seem: because of the other programming-language features it *doesn't*
have (classes, exceptions, and so on).
When I was originally prototyping Miller in 2015, the decision I had was whether to hand-code in a low-level language like C or Rust or Go, with my own hand-rolled DSL, or whether to use a higher-level language (like Python or Lua or Nim) and let the `put` statements be handled by the implementation language's own `eval`: the implementation language would take the place of a DSL. Multiple performance experiments showed me I could get better throughput using the former, by a wide margin. So Miller is Go under the hood with a hand-rolled DSL.
When I was originally prototyping Miller in 2015, the primary decision I had
was whether to hand-code in a low-level language like C or Rust or Go, with my
own hand-rolled DSL, or whether to use a higher-level language (like Python or
Lua or Nim) and let the `put` statements be handled by the implementation
language's own `eval`: the implementation language would take the place of a
DSL. Multiple performance experiments showed me I could get better throughput
using the former, by a wide margin. So Miller is Go under the hood with a
hand-rolled DSL.
I do want to keep focusing on what Miller is good at -- concise notation, low latency, and high throughput -- and not add too much in terms of high-level-language features to the DSL. That said, some sort of customizability is a basic thing to want. As of 4.1.0 we have recursive for/while/if structures on about the same complexity level as `awk`; as of 5.0.0 we have user-defined functions and map-valued variables, again on about the same complexity level as `awk` along with optional type-declaration syntax; as of Miller 6 we have full support for arrays. While I'm excited by these powerful language features, I hope to keep new features focused on Miller's sweet spot which is speed plus simplicity.
I do want to keep focusing on what Miller is good at -- concise notation, low
latency, and high throughput -- and not add too much in terms of
high-level-language features to the DSL. That said, some sort of
customizability is a basic thing to want. As of 4.1.0 we have recursive
`for`/`while`/`if` [structures](reference-dsl-control-structures.md) on about
the same complexity level as `awk`; as of 5.0.0 we have [user-defined
functions](reference-dsl-user-defined-functions.md) and [map-valued
variables](reference-dsl-variables.md), again on about the same complexity level
as `awk` along with optional type-declaration syntax; as of Miller 6 we have
full support for [arrays](reference-dsl-arrays.md). While I'm excited by these
powerful language features, I hope to keep new features focused on Miller's
sweet spot which is speed plus simplicity.

33
docs6/docs/reference-dsl-complexity.md.in
View file

@ -1,8 +1,33 @@
# DSL reference: a note on the complexity of Miller's expression language
# A note on the complexity of Miller's expression language
One of Miller's strengths is its brevity: it's much quicker -- and less error-prone -- to type `mlr stats1 -a sum -f x,y -g a,b` than having to track summation variables as in `awk`, or using Miller's out-of-stream variables. And the more language features Miller's put-DSL has (for-loops, if-statements, nested control structures, user-defined functions, etc.) then the *less* powerful it begins to seem: because of the other programming-language features it *doesn't* have (classes, exceptions, and so on).
One of Miller's strengths is its brevity: it's much quicker -- and less
error-prone -- to type `mlr stats1 -a sum -f x,y -g a,b` than having to track
summation variables as in `awk`, or using Miller's [out-of-stream
variables](reference-dsl-variables.md#out-of-stream-variables). And the more
language features Miller's put-DSL has (for-loops, if-statements, nested
control structures, user-defined functions, etc.) then the *less* powerful it
begins to seem: because of the other programming-language features it *doesn't*
have (classes, exceptions, and so on).
When I was originally prototyping Miller in 2015, the decision I had was whether to hand-code in a low-level language like C or Rust or Go, with my own hand-rolled DSL, or whether to use a higher-level language (like Python or Lua or Nim) and let the `put` statements be handled by the implementation language's own `eval`: the implementation language would take the place of a DSL. Multiple performance experiments showed me I could get better throughput using the former, by a wide margin. So Miller is Go under the hood with a hand-rolled DSL.
When I was originally prototyping Miller in 2015, the primary decision I had
was whether to hand-code in a low-level language like C or Rust or Go, with my
own hand-rolled DSL, or whether to use a higher-level language (like Python or
Lua or Nim) and let the `put` statements be handled by the implementation
language's own `eval`: the implementation language would take the place of a
DSL. Multiple performance experiments showed me I could get better throughput
using the former, by a wide margin. So Miller is Go under the hood with a
hand-rolled DSL.
I do want to keep focusing on what Miller is good at -- concise notation, low latency, and high throughput -- and not add too much in terms of high-level-language features to the DSL. That said, some sort of customizability is a basic thing to want. As of 4.1.0 we have recursive for/while/if structures on about the same complexity level as `awk`; as of 5.0.0 we have user-defined functions and map-valued variables, again on about the same complexity level as `awk` along with optional type-declaration syntax; as of Miller 6 we have full support for arrays. While I'm excited by these powerful language features, I hope to keep new features focused on Miller's sweet spot which is speed plus simplicity.
I do want to keep focusing on what Miller is good at -- concise notation, low
latency, and high throughput -- and not add too much in terms of
high-level-language features to the DSL. That said, some sort of
customizability is a basic thing to want. As of 4.1.0 we have recursive
`for`/`while`/`if` [structures](reference-dsl-control-structures.md) on about
the same complexity level as `awk`; as of 5.0.0 we have [user-defined
functions](reference-dsl-user-defined-functions.md) and [map-valued
variables](reference-dsl-variables.md), again on about the same complexity level
as `awk` along with optional type-declaration syntax; as of Miller 6 we have
full support for [arrays](reference-dsl-arrays.md). While I'm excited by these
powerful language features, I hope to keep new features focused on Miller's
sweet spot which is speed plus simplicity.

41
docs6/docs/reference-dsl-control-structures.md
View file

@ -1,5 +1,5 @@
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
# DSL reference: control structures
# DSL control structures
## Pattern-action blocks
@ -44,9 +44,9 @@ a=xyz_789
<b> data/put-gating-example-2.dkvp</b>
</pre>
<pre class="pre-non-highlight-in-pair">
a=abc_123,b=left_\1,c=right_\2
a=abc_123,b=left_abc,c=right_123
a=some other name
a=xyz_789,b=left_\1,c=right_\2
a=xyz_789,b=left_xyz,c=right_789
</pre>
This produces heteregenous output which Miller, of course, has no problems with (see [Record Heterogeneity](record-heterogeneity.md)). But if you want homogeneous output, the curly braces can be replaced with a semicolon between the expression and the body statements. This causes `put` to evaluate the boolean expression (along with any side effects, namely, regex-captures `\1`, `\2`, etc.) but doesn't use it as a criterion for whether subsequent assignments should be executed. Instead, subsequent assignments are done unconditionally:
@ -55,6 +55,8 @@ This produces heteregenous output which Miller, of course, has no problems with
<b>mlr put '$x > 0.0; $y = log10($x); $z = sqrt($y)' data/put-gating-example-1.dkvp</b>
</pre>
<pre class="pre-non-highlight-in-pair">
x=-1,y=NaN,z=NaN
x=0,y=-Inf,z=NaN
x=1,y=0,z=0
x=2,y=0.3010299956639812,z=0.5486620049392715
x=3,y=0.4771212547196624,z=0.6907396432228734
@ -68,8 +70,9 @@ x=3,y=0.4771212547196624,z=0.6907396432228734
<b>' data/put-gating-example-2.dkvp</b>
</pre>
<pre class="pre-non-highlight-in-pair">
a=abc_123,b=left_\1,c=right_\2
a=xyz_789,b=left_\1,c=right_\2
a=abc_123,b=left_abc,c=right_123
a=some other name,b=left_,c=right_
a=xyz_789,b=left_xyz,c=right_789
</pre>
## If-statements
@ -131,12 +134,26 @@ x=1,y=2,3=,4=,5=,6=,7=,8=,9=,10=,foo=bar
x=1,y=2,3=,4=,5=,foo=bar
</pre>
A `break` or `continue` within nested conditional blocks or if-statements will, of course, propagate to the innermost loop enclosing them, if any. A `break` or `continue` outside a loop is a syntax error that will be flagged as soon as the expression is parsed, before any input records are ingested.
The existence of `while`, `do-while`, and `for` loops in Miller's DSL means that you can create infinite-loop scenarios inadvertently. In particular, please recall that DSL statements are executed once if in `begin` or `end` blocks, and once *per record* otherwise. For example, **while (NR < 10) will never terminate as NR is only incremented between records**.
A `break` or `continue` within nested conditional blocks or if-statements will,
of course, propagate to the innermost loop enclosing them, if any. A `break` or
`continue` outside a loop is a syntax error that will be flagged as soon as the
expression is parsed, before any input records are ingested.
The existence of `while`, `do-while`, and `for` loops in Miller's DSL means
that you can create infinite-loop scenarios inadvertently. In particular,
please recall that DSL statements are executed once if in `begin` or `end`
blocks, and once *per record* otherwise. For example, **while (NR < 10) will
never terminate**. The [`NR`
variable](reference-dsl-variables.md#built-in-variables) is only incremented
between records, and each DSL expression is invoked once per record: so, once
for `NR=1`, once for `NR=2`, etc.
If you do want to loop over records, see [Operating on all
records](operating-on-all-records.md) for some options.
## For-loops
While Miller's `while` and `do-while` statements are much as in many other languages, `for` loops are more idiosyncratic to Miller. They are loops over key-value pairs, whether in stream records, out-of-stream variables, local variables, or map-literals: more reminiscent of `foreach`, as in (for example) PHP. There are **for-loops over map keys** and **for-loops over key-value tuples**. Additionally, Miller has a **C-style triple-for loop** with initialize, test, and update statements.
While Miller's `while` and `do-while` statements are much as in many other languages, `for` loops are more idiosyncratic to Miller. They are loops over key-value pairs, whether in stream records, out-of-stream variables, local variables, or map-literals: more reminiscent of `foreach`, as in (for example) PHP. There are **for-loops over map keys** and **for-loops over key-value tuples**. Additionally, Miller has a **C-style triple-for loop** with initialize, test, and update statements. Each is described below.
As with `while` and `do-while`, a `break` or `continue` within nested control structures will propagate to the innermost loop enclosing them, if any, and a `break` or `continue` outside a loop is a syntax error that will be flagged as soon as the expression is parsed, before any input records are ingested.
@ -471,7 +488,7 @@ Notes:
* Miller has no `++` or `--` operators.
* As with all for/if/while statements in Miller, the curly braces are required even if the body is a single statement, or empty.
* As with all `for`/`if`/`while` statements in Miller, the curly braces are required even if the body is a single statement, or empty.
## Begin/end blocks
@ -510,7 +527,7 @@ a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
x_sum=2.264761728567491
</pre>
The **put -q** option is a shorthand which suppresses printing of each output record, with only `emit` statements being output. So to get only summary outputs, one could write
The **put -q** option suppresses printing of each output record, with only `emit` statements being output. So to get only summary outputs, you could write
<pre class="pre-highlight-in-pair">
<b>mlr put -q '</b>
@ -539,7 +556,7 @@ x_count=5
x_sum=2.264761728567491
</pre>
This is of course not much different than
This is of course (see also [here](reference-dsl.md#verbs-compared-to-dsl)) not much different than
<pre class="pre-highlight-in-pair">
<b>mlr stats1 -a count,sum -f x ./data/small</b>
@ -548,5 +565,5 @@ This is of course not much different than
x_count=5,x_sum=2.264761728567491
</pre>
Note that it's a syntax error for begin/end blocks to refer to field names (beginning with `$`), since these execute outside the context of input records.
Note that it's a syntax error for begin/end blocks to refer to field names (beginning with `$`), since begin/end blocks execute outside the context of input records.

30
docs6/docs/reference-dsl-control-structures.md.in
View file

@ -1,4 +1,4 @@
# DSL reference: control structures
# DSL control structures
## Pattern-action blocks
@ -74,12 +74,26 @@ GENMD_INCLUDE_AND_RUN_ESCAPED(data/while-example-1.sh)
GENMD_INCLUDE_AND_RUN_ESCAPED(data/while-example-2.sh)
A `break` or `continue` within nested conditional blocks or if-statements will, of course, propagate to the innermost loop enclosing them, if any. A `break` or `continue` outside a loop is a syntax error that will be flagged as soon as the expression is parsed, before any input records are ingested.
The existence of `while`, `do-while`, and `for` loops in Miller's DSL means that you can create infinite-loop scenarios inadvertently. In particular, please recall that DSL statements are executed once if in `begin` or `end` blocks, and once *per record* otherwise. For example, **while (NR < 10) will never terminate as NR is only incremented between records**.
A `break` or `continue` within nested conditional blocks or if-statements will,
of course, propagate to the innermost loop enclosing them, if any. A `break` or
`continue` outside a loop is a syntax error that will be flagged as soon as the
expression is parsed, before any input records are ingested.
The existence of `while`, `do-while`, and `for` loops in Miller's DSL means
that you can create infinite-loop scenarios inadvertently. In particular,
please recall that DSL statements are executed once if in `begin` or `end`
blocks, and once *per record* otherwise. For example, **while (NR < 10) will
never terminate**. The [`NR`
variable](reference-dsl-variables.md#built-in-variables) is only incremented
between records, and each DSL expression is invoked once per record: so, once
for `NR=1`, once for `NR=2`, etc.
If you do want to loop over records, see [Operating on all
records](operating-on-all-records.md) for some options.
## For-loops
While Miller's `while` and `do-while` statements are much as in many other languages, `for` loops are more idiosyncratic to Miller. They are loops over key-value pairs, whether in stream records, out-of-stream variables, local variables, or map-literals: more reminiscent of `foreach`, as in (for example) PHP. There are **for-loops over map keys** and **for-loops over key-value tuples**. Additionally, Miller has a **C-style triple-for loop** with initialize, test, and update statements.
While Miller's `while` and `do-while` statements are much as in many other languages, `for` loops are more idiosyncratic to Miller. They are loops over key-value pairs, whether in stream records, out-of-stream variables, local variables, or map-literals: more reminiscent of `foreach`, as in (for example) PHP. There are **for-loops over map keys** and **for-loops over key-value tuples**. Additionally, Miller has a **C-style triple-for loop** with initialize, test, and update statements. Each is described below.
As with `while` and `do-while`, a `break` or `continue` within nested control structures will propagate to the innermost loop enclosing them, if any, and a `break` or `continue` outside a loop is a syntax error that will be flagged as soon as the expression is parsed, before any input records are ingested.
@ -151,7 +165,7 @@ Notes:
* Miller has no `++` or `--` operators.
* As with all for/if/while statements in Miller, the curly braces are required even if the body is a single statement, or empty.
* As with all `for`/`if`/`while` statements in Miller, the curly braces are required even if the body is a single statement, or empty.
## Begin/end blocks
@ -163,7 +177,7 @@ Since uninitialized out-of-stream variables default to 0 for addition/substracti
GENMD_INCLUDE_AND_RUN_ESCAPED(data/begin-end-example-2.sh)
The **put -q** option is a shorthand which suppresses printing of each output record, with only `emit` statements being output. So to get only summary outputs, one could write
The **put -q** option suppresses printing of each output record, with only `emit` statements being output. So to get only summary outputs, you could write
GENMD_INCLUDE_AND_RUN_ESCAPED(data/begin-end-example-3.sh)
@ -171,9 +185,9 @@ We can do similarly with multiple out-of-stream variables:
GENMD_INCLUDE_AND_RUN_ESCAPED(data/begin-end-example-4.sh)
This is of course not much different than
This is of course (see also [here](reference-dsl.md#verbs-compared-to-dsl)) not much different than
GENMD_INCLUDE_AND_RUN_ESCAPED(data/begin-end-example-5.sh)
Note that it's a syntax error for begin/end blocks to refer to field names (beginning with `$`), since these execute outside the context of input records.
Note that it's a syntax error for begin/end blocks to refer to field names (beginning with `$`), since begin/end blocks execute outside the context of input records.

12
docs6/docs/reference-dsl-errors.md
View file

@ -1,9 +1,9 @@
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
# DSL reference: errors and transparency
# DSL errors and transparency
As soon as you have a programming language, you start having the problem *What is my code doing, and why?* This includes getting syntax errors -- which are always annoying -- as well as the even more annoying problem of a program which parses without syntax error but doesn't do what you expect.
As soon as you have a [programming language](programming-language.md), you start having the problem *What is my code doing, and why?* This includes getting syntax errors -- which are always annoying -- as well as the even more annoying problem of a program which parses without syntax error but doesn't do what you expect.
The `syntax error` message is cryptic: it says `syntax error at ` followed by the next symbol it couldn't parse. This is good, but (as of 5.0.0) it doesn't say things like `syntax error at line 17, character 22`. Here are some common causes of syntax errors:
The syntax-error message gives you line/column position for the syntax that couldn't be parsed. The cause may be clear from that information, or perhaps not. Here are some common causes of syntax errors:
* Don't forget `;` at end of line, before another statement on the next line.
@ -17,8 +17,4 @@ As for transparency:
* The `-v` option to `mlr put` and `mlr filter` prints abstract syntax trees for your code. While not all details here will be of interest to everyone, certainly this makes questions such as operator precedence completely unambiguous.
* The `-T` option prints a trace of each statement executed.
* The `-t` and `-a` options show low-level details for the parsing process and for stack-variable-index allocation, respectively. These will likely be of interest to people who enjoy compilers, and probably less useful for a more general audience.
* Please see [type-checking](reference-dsl-variables.md#type-checking) for type declarations and type-assertions you can use to make sure expressions and the data flowing them are evaluating as you expect. I made them optional because one of Miller's important use-cases is being able to say simple things like `mlr put '$y = $x + 1' myfile.dat` with a minimum of punctuational bric-a-brac -- but for programs over a few lines I generally find that the more type-specification, the better.
* Please see [type-checking](reference-dsl-variables.md#type-checking) for type declarations and type-assertions you can use to make sure expressions and the data flowing them are evaluating as you expect. I made them optional because one of Miller's important use-cases is being able to say simple things like `mlr put '$y = $x + 1' myfile.dat` with a minimum of punctuational bric-a-brac -- but for programs over a few lines long, I generally find that the more type-specification, the better.

12
docs6/docs/reference-dsl-errors.md.in
View file

@ -1,8 +1,8 @@
# DSL reference: errors and transparency
# DSL errors and transparency
As soon as you have a programming language, you start having the problem *What is my code doing, and why?* This includes getting syntax errors -- which are always annoying -- as well as the even more annoying problem of a program which parses without syntax error but doesn't do what you expect.
As soon as you have a [programming language](programming-language.md), you start having the problem *What is my code doing, and why?* This includes getting syntax errors -- which are always annoying -- as well as the even more annoying problem of a program which parses without syntax error but doesn't do what you expect.
The `syntax error` message is cryptic: it says `syntax error at ` followed by the next symbol it couldn't parse. This is good, but (as of 5.0.0) it doesn't say things like `syntax error at line 17, character 22`. Here are some common causes of syntax errors:
The syntax-error message gives you line/column position for the syntax that couldn't be parsed. The cause may be clear from that information, or perhaps not. Here are some common causes of syntax errors:
* Don't forget `;` at end of line, before another statement on the next line.
@ -16,8 +16,4 @@ As for transparency:
* The `-v` option to `mlr put` and `mlr filter` prints abstract syntax trees for your code. While not all details here will be of interest to everyone, certainly this makes questions such as operator precedence completely unambiguous.
* The `-T` option prints a trace of each statement executed.
* The `-t` and `-a` options show low-level details for the parsing process and for stack-variable-index allocation, respectively. These will likely be of interest to people who enjoy compilers, and probably less useful for a more general audience.
* Please see [type-checking](reference-dsl-variables.md#type-checking) for type declarations and type-assertions you can use to make sure expressions and the data flowing them are evaluating as you expect. I made them optional because one of Miller's important use-cases is being able to say simple things like `mlr put '$y = $x + 1' myfile.dat` with a minimum of punctuational bric-a-brac -- but for programs over a few lines I generally find that the more type-specification, the better.
* Please see [type-checking](reference-dsl-variables.md#type-checking) for type declarations and type-assertions you can use to make sure expressions and the data flowing them are evaluating as you expect. I made them optional because one of Miller's important use-cases is being able to say simple things like `mlr put '$y = $x + 1' myfile.dat` with a minimum of punctuational bric-a-brac -- but for programs over a few lines long, I generally find that the more type-specification, the better.

37
docs6/docs/reference-dsl-filter-statements.md
View file

@ -1,39 +1,44 @@
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
# DSL reference: filter statements
# DSL filter statements
You can use `filter` within `put`. In fact, the following two are synonymous:
You can use the `filter` DSL keyword within the `put` verb. In fact, the following two are synonymous:
<pre class="pre-highlight-in-pair">
<b>mlr filter 'NR==2 || NR==3' data/small</b>
<b>mlr --csv filter 'NR==2 || NR==3' example.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
color,shape,flag,k,index,quantity,rate
red,square,true,2,15,79.2778,0.0130
red,circle,true,3,16,13.8103,2.9010
</pre>
<pre class="pre-highlight-in-pair">
<b>mlr put 'filter NR==2 || NR==3' data/small</b>
<b>mlr --csv put 'filter NR==2 || NR==3' example.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
color,shape,flag,k,index,quantity,rate
red,square,true,2,15,79.2778,0.0130
red,circle,true,3,16,13.8103,2.9010
</pre>
The former, of course, is much easier to type. But the latter allows you to define more complex expressions for the filter, and/or do other things in addition to the filter:
The former, of course, is a little easier to type. For another example:
<pre class="pre-highlight-in-pair">
<b>mlr put '@running_sum += $x; filter @running_sum > 1.3' data/small</b>
<b>mlr --csv put '@running_sum += $quantity; filter @running_sum > 500' example.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
color,shape,flag,k,index,quantity,rate
yellow,circle,true,8,73,63.9785,4.2370
yellow,circle,true,9,87,63.5058,8.3350
purple,square,false,10,91,72.3735,8.2430
</pre>
<pre class="pre-highlight-in-pair">
<b>mlr put '$z = $x * $y; filter $z > 0.3' data/small</b>
<b>mlr --csv filter '@running_sum += $quantity; @running_sum > 500' example.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797,z=0.3961455844854848
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729,z=0.4951063394654227
color,shape,flag,k,index,quantity,rate
yellow,circle,true,8,73,63.9785,4.2370
yellow,circle,true,9,87,63.5058,8.3350
purple,square,false,10,91,72.3735,8.2430
</pre>

14
docs6/docs/reference-dsl-filter-statements.md.in
View file

@ -1,21 +1,21 @@
# DSL reference: filter statements
# DSL filter statements
You can use `filter` within `put`. In fact, the following two are synonymous:
You can use the `filter` DSL keyword within the `put` verb. In fact, the following two are synonymous:
GENMD_RUN_COMMAND
mlr filter 'NR==2 || NR==3' data/small
mlr --csv filter 'NR==2 || NR==3' example.csv
GENMD_EOF
GENMD_RUN_COMMAND
mlr put 'filter NR==2 || NR==3' data/small
mlr --csv put 'filter NR==2 || NR==3' example.csv
GENMD_EOF
The former, of course, is much easier to type. But the latter allows you to define more complex expressions for the filter, and/or do other things in addition to the filter:
The former, of course, is a little easier to type. For another example:
GENMD_RUN_COMMAND
mlr put '@running_sum += $x; filter @running_sum > 1.3' data/small
mlr --csv put '@running_sum += $quantity; filter @running_sum > 500' example.csv
GENMD_EOF
GENMD_RUN_COMMAND
mlr put '$z = $x * $y; filter $z > 0.3' data/small
mlr --csv filter '@running_sum += $quantity; @running_sum > 500' example.csv
GENMD_EOF

18
docs6/docs/reference-dsl-operators.md
View file

@ -1,5 +1,5 @@
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
# DSL reference: operators
# DSL operators
## Operator precedence
@ -9,7 +9,7 @@ Operators are listed in order of decreasing precedence, highest first.
Operators Associativity
--------- -------------
() left to right
** right to left
** right to left
! ~ unary+ unary- & right to left
binary* / // % left to right
binary+ binary- . left to right
@ -30,11 +30,17 @@ binary+ binary- . left to right
* Functions are often pass-throughs straight to the system-standard Go libraries.
* The `min` and `max` functions are different from other multi-argument functions which return null if any of their inputs are null: for `min` and `max`, by contrast, if one argument is absent-null, the other is returned. Empty-null loses min or max against numeric or boolean; empty-null is less than any other string.
* The [`min`](reference-dsl-builtin-functions.md#min) and [`max`](reference-dsl-builtin-functions.md#max) functions are different from other multi-argument functions which return null if any of their inputs are null: for [`min`](reference-dsl-builtin-functions.md#min) and [`max`](reference-dsl-builtin-functions.md#max), by contrast, if one argument is absent-null, the other is returned. Empty-null loses min or max against numeric or boolean; empty-null is less than any other string.
* Symmetrically with respect to the bitwise OR, XOR, and AND operators `|`, `^`, `&`, Miller has logical operators `||`, `^^`, `&&`: the logical XOR not existing in Go.
* Symmetrically with respect to the bitwise OR, XOR, and AND operators
[`|`](reference-dsl-builtin-functions.md#bitwise-or),
[`&`](reference-dsl-builtin-functions.md#bitwise-and), and
[`^`](reference-dsl-builtin-functions.md#bitwise-xor), Miller has logical operators
[`||`](reference-dsl-builtin-functions.md#logical-or),
[`&&`](reference-dsl-builtin-functions.md#logical-and), and
[`^^`](reference-dsl-builtin-functions.md#logical-xor).
* The exponentiation operator `**` is familiar from many languages.
* The exponentiation operator [`**`](reference-dsl-builtin-functions.md#exponentiation) is familiar from many languages, except that an integer raised to an int power is int, not float.
* The regex-match and regex-not-match operators `=~` and `!=~` are similar to those in Ruby and Perl.
* The regex-match and regex-not-match operators [`=~`](reference-dsl-builtin-functions.md#regmatch) and [`!=~`](reference-dsl-builtin-functions.md#regnotmatch) are similar to those in Ruby and Perl.

18
docs6/docs/reference-dsl-operators.md.in
View file

@ -1,4 +1,4 @@
# DSL reference: operators
# DSL operators
## Operator precedence
@ -8,7 +8,7 @@ GENMD_CARDIFY
Operators Associativity
--------- -------------
() left to right
** right to left
** right to left
! ~ unary+ unary- & right to left
binary* / // % left to right
binary+ binary- . left to right
@ -29,11 +29,17 @@ GENMD_EOF
* Functions are often pass-throughs straight to the system-standard Go libraries.
* The `min` and `max` functions are different from other multi-argument functions which return null if any of their inputs are null: for `min` and `max`, by contrast, if one argument is absent-null, the other is returned. Empty-null loses min or max against numeric or boolean; empty-null is less than any other string.
* The [`min`](reference-dsl-builtin-functions.md#min) and [`max`](reference-dsl-builtin-functions.md#max) functions are different from other multi-argument functions which return null if any of their inputs are null: for [`min`](reference-dsl-builtin-functions.md#min) and [`max`](reference-dsl-builtin-functions.md#max), by contrast, if one argument is absent-null, the other is returned. Empty-null loses min or max against numeric or boolean; empty-null is less than any other string.
* Symmetrically with respect to the bitwise OR, XOR, and AND operators `|`, `^`, `&`, Miller has logical operators `||`, `^^`, `&&`: the logical XOR not existing in Go.
* Symmetrically with respect to the bitwise OR, XOR, and AND operators
[`|`](reference-dsl-builtin-functions.md#bitwise-or),
[`&`](reference-dsl-builtin-functions.md#bitwise-and), and
[`^`](reference-dsl-builtin-functions.md#bitwise-xor), Miller has logical operators
[`||`](reference-dsl-builtin-functions.md#logical-or),
[`&&`](reference-dsl-builtin-functions.md#logical-and), and
[`^^`](reference-dsl-builtin-functions.md#logical-xor).
* The exponentiation operator `**` is familiar from many languages.
* The exponentiation operator [`**`](reference-dsl-builtin-functions.md#exponentiation) is familiar from many languages, except that an integer raised to an int power is int, not float.
* The regex-match and regex-not-match operators `=~` and `!=~` are similar to those in Ruby and Perl.
* The regex-match and regex-not-match operators [`=~`](reference-dsl-builtin-functions.md#regmatch) and [`!=~`](reference-dsl-builtin-functions.md#regnotmatch) are similar to those in Ruby and Perl.

30
docs6/docs/reference-dsl-output-statements.md
View file

@ -1,5 +1,5 @@
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
# DSL reference: output statements
# DSL output statements
You can **output** variable-values or expressions in **five ways**:
@ -11,7 +11,7 @@ You can **output** variable-values or expressions in **five ways**:
* Use **tee** which formats the current stream record (not just an arbitrary string as with **print**) to a specific file.
* Use **emit**/**emitp**/**emitf** to send out-of-stream variables' current values to the output record stream, e.g. `@sum += $x; emit @sum` which produces an extra output record such as `sum=3.1648382`.
* Use **emit**/**emitp**/**emitf** to send out-of-stream variables' current values to the output record stream, e.g. `@sum += $x; emit @sum` which produces an extra record such as `sum=3.1648382`. These records, just like records from input file(s), participate in downstream [then-chaining](reference-main-then-chaining.md) to other verbs.
For the first two options you are populating the output-records stream which feeds into the next verb in a `then`-chain (if any), or which otherwise is formatted for output using `--o...` flags.
@ -25,11 +25,21 @@ The `print` statement is perhaps self-explanatory, but with a few light caveats:
* Output goes directly to stdout/stderr, respectively: data produced this way do not go downstream to the next verb in a `then`-chain. (Use `emit` for that.)
* Print statements are for strings (`print "hello"`), or things which can be made into strings: numbers (`print 3`, `print $a + $b`, or concatenations thereof (`print "a + b = " . ($a + $b)`). Maps (in `$*`, map-valued out-of-stream or local variables, and map literals) aren't convertible into strings. If you print a map, you get `{is-a-map}` as output. Please use `dump` to print maps.
* Print statements are for strings (`print "hello"`), or things which can be made into strings: numbers (`print 3`, `print $a + $b`), or concatenations thereof (`print "a + b = " . ($a + $b)`). Maps (in `$*`, map-valued out-of-stream or local variables, and map literals) as well as arrays are printed as JSON.
* You can redirect print output to a file: `mlr --from myfile.dat put 'print > "tap.txt", $x'` `mlr --from myfile.dat put 'o=$*; print > $a.".txt", $x'`.
* You can redirect print output to a file:
* See also [Redirected-output statements](reference-dsl-output-statements.md#redirected-output-statements) for examples.
<pre class="pre-highlight-non-pair">
<b>mlr --from myfile.dat put 'print > "tap.txt", $x'</b>
</pre>
* You can redirect print output to multiple files, split by values present in various records:
<pre class="pre-highlight-non-pair">
<b>mlr --from myfile.dat put 'print > $a.".txt", $x'</b>
</pre>
See also [Redirected-output statements](reference-dsl-output-statements.md#redirected-output-statements) for examples.
## Dump statements
@ -39,7 +49,7 @@ The `dump` statement is for printing expressions, including maps, directly to st
* Output goes directly to stdout/stderr, respectively: data produced this way do not go downstream to the next verb in a `then`-chain. (Use `emit` for that.)
* You can use `dump` to output single strings, numbers, or expressions including map-valued data. Map-valued data are printed as JSON. Miller allows string and integer keys in its map literals while JSON allows only string keys, so use `mlr put --jknquoteint` if you want integer-valued map keys not double-quoted.
* You can use `dump` to output single strings, numbers, or expressions including map-valued data. Map-valued data are printed as JSON.
* If you use `dump` (or `edump`) with no arguments, you get a JSON structure representing the current values of all out-of-stream variables.
@ -51,7 +61,13 @@ The `dump` statement is for printing expressions, including maps, directly to st
Records produced by a `mlr put` go downstream to the next verb in your `then`-chain, if any, or otherwise to standard output. If you want to additionally copy out records to files, you can do that using `tee`.
The syntax is, by example, `mlr --from myfile.dat put 'tee > "tap.dat", $*' then sort -n index`. First is `tee >`, then the filename expression (which can be an expression such as `"tap.".$a.".dat"`), then a comma, then `$*`. (Nothing else but `$*` is teeable.)
The syntax is, by example:
<pre class="pre-highlight-non-pair">
<b>mlr --from myfile.dat put 'tee > "tap.dat", $*' then sort -n index</b>
</pre>
First is `tee >`, then the filename expression (which can be an expression such as `"tap.".$a.".dat"`), then a comma, then `$*`. (Nothing else but `$*` is teeable.)
See also [Redirected-output statements](reference-dsl-output-statements.md#redirected-output-statements) for examples.

30
docs6/docs/reference-dsl-output-statements.md.in
View file

@ -1,4 +1,4 @@
# DSL reference: output statements
# DSL output statements
You can **output** variable-values or expressions in **five ways**:
@ -10,7 +10,7 @@ You can **output** variable-values or expressions in **five ways**:
* Use **tee** which formats the current stream record (not just an arbitrary string as with **print**) to a specific file.
* Use **emit**/**emitp**/**emitf** to send out-of-stream variables' current values to the output record stream, e.g. `@sum += $x; emit @sum` which produces an extra output record such as `sum=3.1648382`.
* Use **emit**/**emitp**/**emitf** to send out-of-stream variables' current values to the output record stream, e.g. `@sum += $x; emit @sum` which produces an extra record such as `sum=3.1648382`. These records, just like records from input file(s), participate in downstream [then-chaining](reference-main-then-chaining.md) to other verbs.
For the first two options you are populating the output-records stream which feeds into the next verb in a `then`-chain (if any), or which otherwise is formatted for output using `--o...` flags.
@ -24,11 +24,21 @@ The `print` statement is perhaps self-explanatory, but with a few light caveats:
* Output goes directly to stdout/stderr, respectively: data produced this way do not go downstream to the next verb in a `then`-chain. (Use `emit` for that.)
* Print statements are for strings (`print "hello"`), or things which can be made into strings: numbers (`print 3`, `print $a + $b`, or concatenations thereof (`print "a + b = " . ($a + $b)`). Maps (in `$*`, map-valued out-of-stream or local variables, and map literals) aren't convertible into strings. If you print a map, you get `{is-a-map}` as output. Please use `dump` to print maps.
* Print statements are for strings (`print "hello"`), or things which can be made into strings: numbers (`print 3`, `print $a + $b`), or concatenations thereof (`print "a + b = " . ($a + $b)`). Maps (in `$*`, map-valued out-of-stream or local variables, and map literals) as well as arrays are printed as JSON.
* You can redirect print output to a file: `mlr --from myfile.dat put 'print > "tap.txt", $x'` `mlr --from myfile.dat put 'o=$*; print > $a.".txt", $x'`.
* You can redirect print output to a file:
* See also [Redirected-output statements](reference-dsl-output-statements.md#redirected-output-statements) for examples.
GENMD_CARDIFY_HIGHLIGHT_ONE
mlr --from myfile.dat put 'print > "tap.txt", $x'
GENMD_EOF
* You can redirect print output to multiple files, split by values present in various records:
GENMD_CARDIFY_HIGHLIGHT_ONE
mlr --from myfile.dat put 'print > $a.".txt", $x'
GENMD_EOF
See also [Redirected-output statements](reference-dsl-output-statements.md#redirected-output-statements) for examples.
## Dump statements
@ -38,7 +48,7 @@ The `dump` statement is for printing expressions, including maps, directly to st
* Output goes directly to stdout/stderr, respectively: data produced this way do not go downstream to the next verb in a `then`-chain. (Use `emit` for that.)
* You can use `dump` to output single strings, numbers, or expressions including map-valued data. Map-valued data are printed as JSON. Miller allows string and integer keys in its map literals while JSON allows only string keys, so use `mlr put --jknquoteint` if you want integer-valued map keys not double-quoted.
* You can use `dump` to output single strings, numbers, or expressions including map-valued data. Map-valued data are printed as JSON.
* If you use `dump` (or `edump`) with no arguments, you get a JSON structure representing the current values of all out-of-stream variables.
@ -50,7 +60,13 @@ The `dump` statement is for printing expressions, including maps, directly to st
Records produced by a `mlr put` go downstream to the next verb in your `then`-chain, if any, or otherwise to standard output. If you want to additionally copy out records to files, you can do that using `tee`.
The syntax is, by example, `mlr --from myfile.dat put 'tee > "tap.dat", $*' then sort -n index`. First is `tee >`, then the filename expression (which can be an expression such as `"tap.".$a.".dat"`), then a comma, then `$*`. (Nothing else but `$*` is teeable.)
The syntax is, by example:
GENMD_CARDIFY_HIGHLIGHT_ONE
mlr --from myfile.dat put 'tee > "tap.dat", $*' then sort -n index
GENMD_EOF
First is `tee >`, then the filename expression (which can be an expression such as `"tap.".$a.".dat"`), then a comma, then `$*`. (Nothing else but `$*` is teeable.)
See also [Redirected-output statements](reference-dsl-output-statements.md#redirected-output-statements) for examples.

6
docs6/docs/reference-dsl-syntax.md
View file

@ -1,5 +1,5 @@
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
# DSL reference: syntax
# DSL syntax
## Expression formatting
@ -59,7 +59,7 @@ x_y_corr
## Expressions from files
The simplest way to enter expressions for `put` and `filter` is between single quotes on the command line, e.g.
The simplest way to enter expressions for `put` and `filter` is between single quotes on the command line (see also [here](miller-on-windows.md) for Windows). For example:
<pre class="pre-highlight-in-pair">
<b>mlr --from data/small put '$xy = sqrt($x**2 + $y**2)'</b>
@ -133,7 +133,7 @@ A suggested use-case here is defining functions in files, and calling them from
Another suggested use-case is putting default parameter values in files, e.g. using `begin{@count=is_present(@count)?@count:10}` in the file, where you can precede that using `begin{@count=40}` using `-e`.
Moreover, you can have one or more `-f` expressions (maybe one function per file, for example) and one or more `-e` expressions on the command line. If you mix `-f` and `-e` then the expressions are evaluated in the order encountered. (Since the expressions are all simply concatenated together in order, don't forget intervening semicolons: e.g. not `mlr put -e '$x=1' -e '$y=2 ...'` but rather `mlr put -e '$x=1;' -e '$y=2' ...`.)
Moreover, you can have one or more `-f` expressions (maybe one function per file, for example) and one or more `-e` expressions on the command line. If you mix `-f` and `-e` then the expressions are evaluated in the order encountered.
## Semicolons, commas, newlines, and curly braces

6
docs6/docs/reference-dsl-syntax.md.in
View file

@ -1,4 +1,4 @@
# DSL reference: syntax
# DSL syntax
## Expression formatting
@ -20,7 +20,7 @@ GENMD_EOF
## Expressions from files
The simplest way to enter expressions for `put` and `filter` is between single quotes on the command line, e.g.
The simplest way to enter expressions for `put` and `filter` is between single quotes on the command line (see also [here](miller-on-windows.md) for Windows). For example:
GENMD_INCLUDE_AND_RUN_ESCAPED(data/fe-example-1.sh)
@ -51,7 +51,7 @@ A suggested use-case here is defining functions in files, and calling them from
Another suggested use-case is putting default parameter values in files, e.g. using `begin{@count=is_present(@count)?@count:10}` in the file, where you can precede that using `begin{@count=40}` using `-e`.
Moreover, you can have one or more `-f` expressions (maybe one function per file, for example) and one or more `-e` expressions on the command line. If you mix `-f` and `-e` then the expressions are evaluated in the order encountered. (Since the expressions are all simply concatenated together in order, don't forget intervening semicolons: e.g. not `mlr put -e '$x=1' -e '$y=2 ...'` but rather `mlr put -e '$x=1;' -e '$y=2' ...`.)
Moreover, you can have one or more `-f` expressions (maybe one function per file, for example) and one or more `-e` expressions on the command line. If you mix `-f` and `-e` then the expressions are evaluated in the order encountered.
## Semicolons, commas, newlines, and curly braces

6
docs6/docs/reference-dsl-unset-statements.md
View file

@ -1,6 +1,6 @@
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
DSL reference: unset statements
#
# DSL unset statements
You can clear a map key by assigning the empty string as its value: `$x=""` or `@x=""`. Using `unset` you can remove the key entirely. Examples:
<pre class="pre-highlight-in-pair">
@ -81,4 +81,4 @@ This can also be done, of course, using `mlr cut -x`. You can also clear out-of-
}
</pre>
If you use `unset all` (or `unset @*` which is synonymous), that will unset all out-of-stream variables which have been defined up to that point.
If you use `unset all` (or `unset @*` which is synonymous), that will unset all out-of-stream variables which have been assigned up to that point.

6
docs6/docs/reference-dsl-unset-statements.md.in
View file

@ -1,5 +1,5 @@
DSL reference: unset statements
#
# DSL unset statements
You can clear a map key by assigning the empty string as its value: `$x=""` or `@x=""`. Using `unset` you can remove the key entirely. Examples:
GENMD_RUN_COMMAND
@ -20,4 +20,4 @@ GENMD_RUN_COMMAND
mlr put -q '@sum[$a][$b] += $x; end { dump; unset @sum["eks"]; dump }' data/small
GENMD_EOF
If you use `unset all` (or `unset @*` which is synonymous), that will unset all out-of-stream variables which have been defined up to that point.
If you use `unset all` (or `unset @*` which is synonymous), that will unset all out-of-stream variables which have been assigned up to that point.

14
docs6/docs/reference-dsl-user-defined-functions.md
View file

@ -1,5 +1,5 @@
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
# DSL reference: user-defined functions
# DSL user-defined functions
As of Miller 5.0.0 you can define your own functions, as well as subroutines.
@ -36,17 +36,15 @@ Properties of user-defined functions:
* Function bodies start with `func` and a parameter list, defined outside of `begin`, `end`, or other `func` or `subr` blocks. (I.e. the Miller DSL has no nested functions.)
* A function (uniqified by its name) may not be redefined: either by redefining a user-defined function, or by redefining a built-in function. However, functions and subroutines have separate namespaces: you can define a subroutine `log` which does not clash with the mathematical `log` function.
* A function (uniqified by its name) may not be redefined: either by redefining a user-defined function, or by redefining a built-in function. However, functions and subroutines have separate namespaces: you can define a subroutine `log` (for logging messages to stderr, say) which does not clash with the mathematical `log` (logarithm) function.
* Functions may be defined either before or after use (there is an object-binding/linkage step at startup). More specifically, functions may be either recursive or mutually recursive. Functions may not call subroutines.
* Functions may be defined either before or after use -- there is an object-binding/linkage step at startup. More specifically, functions may be either recursive or mutually recursive.
* Functions may be defined and called either within `mlr put` or `mlr put`.
* Functions have read access to `$`-variables and `@`-variables but may not modify them. See also [Memoization with out-of-stream variables](misc-examples.md#memoization-with-out-of-stream-variables) for an example.
* Functions may be defined and called either within `mlr filter` or `mlr put`.
* Argument values may be reassigned: they are not read-only.
* When a return value is not implicitly returned, this results in a return value of absent-null. (In the example above, if there were records for which the argument to `f` is non-numeric, the assignments would be skipped.) See also the section on [xxxx](reference-main-null-data.md).
* When a return value is not implicitly returned, this results in a return value of [absent-null](reference-main-null-data.md). (In the example above, if there were records for which the argument to `f` is non-numeric, the assignments would be skipped.) See also the [null-data reference page](reference-main-null-data.md).
* See the section on [Local variables](reference-dsl-variables.md#local-variables) for information on scope and extent of arguments, as well as for information on the use of local variables within functions.
@ -94,7 +92,7 @@ Properties of user-defined subroutines:
* A subroutine (uniqified by its name) may not be redefined. However, functions and subroutines have separate namespaces: you can define a subroutine `log` which does not clash with the mathematical `log` function.
* Subroutines may be defined either before or after use (there is an object-binding/linkage step at startup). More specifically, subroutines may be either recursive or mutually recursive. Subroutines may call functions.
* Subroutines may be defined either before or after use -- there is an object-binding/linkage step at startup. More specifically, subroutines may be either recursive or mutually recursive. Subroutines may call functions.
* Subroutines may be defined and called either within `mlr put` or `mlr put`.

14
docs6/docs/reference-dsl-user-defined-functions.md.in
View file

@ -1,4 +1,4 @@
# DSL reference: user-defined functions
# DSL user-defined functions
As of Miller 5.0.0 you can define your own functions, as well as subroutines.
@ -12,17 +12,15 @@ Properties of user-defined functions:
* Function bodies start with `func` and a parameter list, defined outside of `begin`, `end`, or other `func` or `subr` blocks. (I.e. the Miller DSL has no nested functions.)
* A function (uniqified by its name) may not be redefined: either by redefining a user-defined function, or by redefining a built-in function. However, functions and subroutines have separate namespaces: you can define a subroutine `log` which does not clash with the mathematical `log` function.
* A function (uniqified by its name) may not be redefined: either by redefining a user-defined function, or by redefining a built-in function. However, functions and subroutines have separate namespaces: you can define a subroutine `log` (for logging messages to stderr, say) which does not clash with the mathematical `log` (logarithm) function.
* Functions may be defined either before or after use (there is an object-binding/linkage step at startup). More specifically, functions may be either recursive or mutually recursive. Functions may not call subroutines.
* Functions may be defined either before or after use -- there is an object-binding/linkage step at startup. More specifically, functions may be either recursive or mutually recursive.
* Functions may be defined and called either within `mlr put` or `mlr put`.
* Functions have read access to `$`-variables and `@`-variables but may not modify them. See also [Memoization with out-of-stream variables](misc-examples.md#memoization-with-out-of-stream-variables) for an example.
* Functions may be defined and called either within `mlr filter` or `mlr put`.
* Argument values may be reassigned: they are not read-only.
* When a return value is not implicitly returned, this results in a return value of absent-null. (In the example above, if there were records for which the argument to `f` is non-numeric, the assignments would be skipped.) See also the section on [xxxx](reference-main-null-data.md).
* When a return value is not implicitly returned, this results in a return value of [absent-null](reference-main-null-data.md). (In the example above, if there were records for which the argument to `f` is non-numeric, the assignments would be skipped.) See also the [null-data reference page](reference-main-null-data.md).
* See the section on [Local variables](reference-dsl-variables.md#local-variables) for information on scope and extent of arguments, as well as for information on the use of local variables within functions.
@ -40,7 +38,7 @@ Properties of user-defined subroutines:
* A subroutine (uniqified by its name) may not be redefined. However, functions and subroutines have separate namespaces: you can define a subroutine `log` which does not clash with the mathematical `log` function.
* Subroutines may be defined either before or after use (there is an object-binding/linkage step at startup). More specifically, subroutines may be either recursive or mutually recursive. Subroutines may call functions.
* Subroutines may be defined either before or after use -- there is an object-binding/linkage step at startup. More specifically, subroutines may be either recursive or mutually recursive. Subroutines may call functions.
* Subroutines may be defined and called either within `mlr put` or `mlr put`.

101
docs6/docs/reference-dsl-variables.md
View file

@ -1,5 +1,5 @@
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
# DSL reference: variables
# DSL variables
Miller has the following kinds of variables:
@ -9,7 +9,7 @@ Miller has the following kinds of variables:
**Out-of-stream variables** accessed using the `@` prefix. These refer to data which persist from one record to the next, including in `begin` and `end` blocks (which execute before/after the record stream is consumed, respectively). You use them to remember values across records, such as sums, differences, counters, and so on. In a few contexts, presented below, you can refer to the entire out-of-stream-variables collection as `@*`.
**Local variables** are limited in scope and extent to the current statements being executed: these include function arguments, bound variables in for loops, and explicitly declared local variables.
**Local variables** are limited in scope and extent to the current statements being executed: these include function arguments, bound variables in for loops, and local variables.
**Keywords** are not variables, but since their names are reserved, you cannot use these names for local variables.
@ -76,9 +76,9 @@ a,b,c,nr
4,5,6,2
</pre>
The **extent** is for the duration of the put/filter: in a `begin` statement (which executes before the fimd.input record is consumed) you will find `NR=1` and in an `end` statement (which is executed after the last input record is consumed) you will find `NR` to be the total number of records ingested.
The **extent** is for the duration of the put/filter: in a `begin` statement (which executes before the first input record is consumed) you will find `NR=1` and in an `end` statement (which is executed after the last input record is consumed) you will find `NR` to be the total number of records ingested.
These are all **read-only** for the `mlr put` and `mlr filter` DSLs: they may be assigned from, e.g. `$nr=NR`, but they may not be assigned to: `NR=100` is a syntax error.
These are all **read-only** for the `mlr put` and `mlr filter` DSL: they may be assigned from, e.g. `$nr=NR`, but they may not be assigned to: `NR=100` is a syntax error.
## Field names
@ -393,7 +393,7 @@ Things which are completely unsurprising, resembling many other languages:
* However, you cannot redeclare the *type* of an argument or a local: `var a=1; var a=2` is an error but `var a=1; a=2` is OK.
* All argument-passing is positional rather than by name; arguments are passed by value, not by reference. (This is also true for map-valued variables: they are not, and cannot be, passed by reference)
* All argument-passing is positional rather than by name; arguments are passed by value, not by reference. (This is also true for map-valued variables: they are not, and cannot be, passed by reference.)
* You can define locals (using `var`, `num`, etc.) at any scope (if-statements, else-statements, while-loops, for-loops, or the top-level scope), and nested scopes will have access (more details on scope in the next section). If you define a local variable with the same name inside an inner scope, then a new variable is created with the narrower scope.
@ -403,13 +403,13 @@ Things which are completely unsurprising, resembling many other languages:
Things which are perhaps surprising compared to other languages:
* Type declarations using `var`, or typed using `num`, `int`, `float`, `str`, and `bool` are necessary to declare local variables. Function arguments and variables bound in for-loops over stream records and out-of-stream variables are *implicitly* declared using `var`. (Some examples are shown below.)
* Type declarations using `var`, or typed using `num`, `int`, `float`, `str`, and `bool` are not necessary to declare local variables. Function arguments and variables bound in for-loops over stream records and out-of-stream variables are *implicitly* declared using `var`. (Some examples are shown below.)
* Type-checking is done at assignment time. For example, `float f = 0` is an error (since `0` is an integer), as is `float f = 0.0; f = 1`. For this reason I prefer to use `num` over `float` in most contexts since `num` encompasses integer and floating-point values. More information is at [Type-checking](reference-dsl-variables.md#type-checking).
* Bound variables in for-loops over stream records and out-of-stream variables are implicitly local to that block. E.g. in `for (k, v in $*) { ... }` `for ((k1, k2), v in @*) { ... }` if there are `k`, `v`, etc. in the enclosing scope then those will be masked by the loop-local bound variables in the loop, and moreover the values of the loop-local bound variables are not available after the end of the loop.
* For C-style triple-for loops, if a for-loop variable is defined using `var`, `int`, etc. then it is scoped to that for-loop. E.g. `for (i = 0; i < 10; i += 1) { ... }` and `for (int i = 0; i < 10; i += 1) { ... }`. (This is unsurprising.). If there is no typedecl and an outer-scope variable of that name exists, then it is used. (This is also unsurprising.) But of there is no outer-scope variable of that name then the variable is scoped to the for-loop only.
* For C-style triple-for loops, if a for-loop variable is defined using `var`, `int`, etc. then it is scoped to that for-loop. E.g. `for (i = 0; i < 10; i += 1) { ... }` and `for (int i = 0; i < 10; i += 1) { ... }`. (This is unsurprising.). If there is no typedecl and an outer-scope variable of that name exists, then it is used. (This is also unsurprising.) But if there is no outer-scope variable of that name, then the variable is scoped to the for-loop only.
The following example demonstrates the scope rules:
@ -484,48 +484,49 @@ And this example demonstrates the type-declaration rules:
<b>cat data/type-decl-example.mlr</b>
</pre>
<pre class="pre-non-highlight-in-pair">
subr s(a, str b, int c) { # a is implicitly var (untyped).
# b is explicitly str.
# c is explicitly int.
# The type-checking is done at the callsite
# when arguments are bound to parameters.
#
var b = 100; # error # Re-declaration in the same scope is disallowed.
int n = 10; # Declaration of variable local to the subroutine.
n = 20; # Assignment is OK.
int n = 30; # error # Re-declaration in the same scope is disallowed.
str n = "abc"; # error # Re-declaration in the same scope is disallowed.
#
float f1 = 1; # error # 1 is an int, not a float.
float f2 = 2.0; # 2.0 is a float.
num f3 = 3; # 3 is a num.
num f4 = 4.0; # 4.0 is a num.
} #
#
call s(1, 2, 3); # Type-assertion '3 is int' is done here at the callsite.
#
k = "def"; # Top-level variable k.
#
for (str k, v in $*) { # k and v are bound here, masking outer k.
print k . ":" . v; # k is explicitly str; v is implicitly var.
} #
#
print "k is".k; # k at this scope level is still "def".
print "v is".v; # v is undefined in this scope.
#
i = -1; #
for (i = 1, int j = 2; i <= 10; i += 1, j *= 2) { # C-style triple-for variables use enclosing scope, unless
# declared local: i is outer, j is local to the loop.
print "inner i =" . i; #
print "inner j =" . j; #
} #
print "outer i =" . i; # i has been modified by the loop.
print "outer j =" . j; # j is undefined in this scope.
subr s(a, str b, int c) { # a is implicitly var (untyped).
# b is explicitly str.
# c is explicitly int.
# The type-checking is done at the callsite
# when arguments are bound to parameters.
#
var b = 100; # error # Re-declaration in the same scope is disallowed.
int n = 10; # Declaration of variable local to the subroutine.
n = 20; # Assignment is OK.
int n = 30; # error # Re-declaration in the same scope is disallowed.
str n = "abc"; # error # Re-declaration in the same scope is disallowed.
#
float f1 = 1; # error # 1 is an int, not a float.
float f2 = 2.0; # 2.0 is a float.
num f3 = 3; # 3 is a num.
num f4 = 4.0; # 4.0 is a num.
} #
#
call s(1, 2, 3); # Type-assertion '3 is int' is done here at the callsite.
#
k = "def"; # Top-level variable k.
#
for (str k, v in $*) { # k and v are bound here, masking outer k.
print k . ":" . v; # k is explicitly str; v is implicitly var.
} #
#
print "k is".k; # k at this scope level is still "def".
print "v is".v; # v is undefined in this scope.
#
i = -1; #
for (i = 1, int j = 2; i <= 10; i += 1, j *= 2) {
# C-style triple-for variables use enclosing scope,
# unless declared local: i is outer, j is local to the loop.
print "inner i =" . i; #
print "inner j =" . j; #
} #
print "outer i =" . i; # i has been modified by the loop.
print "outer j =" . j; # j is undefined in this scope.
</pre>
## Map literals
Miller's `put`/`filter` DSL has four kinds of hashmaps. **Stream records** are (single-level) maps from name to value. **Out-of-stream variables** and **local variables** can also be maps, although they can be multi-level hashmaps (e.g. `@sum[$x][$y]`). The fourth kind is **map literals**. These cannot be on the left-hand side of assignment expressions. Syntactically they look like JSON, although Miller allows string and integer keys in its map literals while JSON allows only string keys (e.g. `"3"` rather than `3`).
Miller's `put`/`filter` DSL has four kinds of hashmaps. **Stream records** are (single-level) maps from name to value. **Out-of-stream variables** and **local variables** can also be maps, although they can be multi-level hashmaps (e.g. `@sum[$x][$y]`). The fourth kind is **map literals**. These cannot be on the left-hand side of assignment expressions. Syntactically they look like JSON, although Miller allows string and integer keys in its map literals while JSON allows only string keys (e.g. `"3"` rather than `3`). Note though that integer keys become stringified in Miller: `@mymap[3]=4` results in `@mymap` being `{"3":4}`.
For example, the following swaps the input stream's `a` and `i` fields, modifies `y`, and drops the rest:
@ -599,17 +600,15 @@ Like out-of-stream and local variables, map literals can be multi-level:
}
</pre>
By default, map-valued expressions are dumped using JSON formatting. If you use `dump` to print a hashmap with integer keys and you don't want them double-quoted (JSON-style) then you can use `mlr put --jknquoteint`. See also `mlr put --help`.
## Type-checking
Miller's `put`/`filter` DSLs support two optional kinds of type-checking. One is inline **type-tests** and **type-assertions** within expressions. The other is **type declarations** for assignments to local variables, binding of arguments to user-defined functions, and return values from user-defined functions, These are discussed in the following subsections.
Miller's `put`/`filter` DSL supports two optional kinds of type-checking. One is inline **type-tests** and **type-assertions** within expressions. The other is **type declarations** for assignments to local variables, binding of arguments to user-defined functions, and return values from user-defined functions, These are discussed in the following subsections.
Use of type-checking is entirely up to you: omit it if you want flexibility with heterogeneous data; use it if you want to help catch misspellings in your DSL code or unexpected irregularities in your input data.
### Type-test and type-assertion expressions
The following `is...` functions take a value and return a boolean indicating whether the argument is of the indicated type. The `assert_...` functions return their argument if it is of the specified type, and cause a fatal error otherwise:
The following `is_...` functions take a value and return a boolean indicating whether the argument is of the indicated type. The `assert_...` functions return their argument if it is of the specified type, and cause a fatal error otherwise:
<pre class="pre-highlight-in-pair">
<b>mlr -f | grep ^is</b>
@ -736,7 +735,7 @@ func f(map m, int i): bool {
## Null data: empty and absent
Please see [xxxx](reference-main-null-data.md).
Please see the [null-data reference page](reference-main-null-data.md).
## Aggregate variable assignments
@ -798,7 +797,7 @@ a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
<pre class="pre-highlight-in-pair">
<b>mlr --opprint put -q '</b>
<b> is_null(@xmax) || $x > @xmax {@xmax=$x; @recmax=$*};</b>
<b> is_null(@xmax) || $x > @xmax {@xmax = $x; @recmax = $*};</b>
<b> end {emit @recmax}</b>
<b>' data/small</b>
</pre>

26
docs6/docs/reference-dsl-variables.md.in
View file

@ -1,4 +1,4 @@
# DSL reference: variables
# DSL variables
Miller has the following kinds of variables:
@ -8,7 +8,7 @@ Miller has the following kinds of variables:
**Out-of-stream variables** accessed using the `@` prefix. These refer to data which persist from one record to the next, including in `begin` and `end` blocks (which execute before/after the record stream is consumed, respectively). You use them to remember values across records, such as sums, differences, counters, and so on. In a few contexts, presented below, you can refer to the entire out-of-stream-variables collection as `@*`.
**Local variables** are limited in scope and extent to the current statements being executed: these include function arguments, bound variables in for loops, and explicitly declared local variables.
**Local variables** are limited in scope and extent to the current statements being executed: these include function arguments, bound variables in for loops, and local variables.
**Keywords** are not variables, but since their names are reserved, you cannot use these names for local variables.
@ -38,9 +38,9 @@ GENMD_RUN_COMMAND
mlr --csv repeat -n 3 then put '$nr = NR' data/a.csv
GENMD_EOF
The **extent** is for the duration of the put/filter: in a `begin` statement (which executes before the fimd.input record is consumed) you will find `NR=1` and in an `end` statement (which is executed after the last input record is consumed) you will find `NR` to be the total number of records ingested.
The **extent** is for the duration of the put/filter: in a `begin` statement (which executes before the first input record is consumed) you will find `NR=1` and in an `end` statement (which is executed after the last input record is consumed) you will find `NR` to be the total number of records ingested.
These are all **read-only** for the `mlr put` and `mlr filter` DSLs: they may be assigned from, e.g. `$nr=NR`, but they may not be assigned to: `NR=100` is a syntax error.
These are all **read-only** for the `mlr put` and `mlr filter` DSL: they may be assigned from, e.g. `$nr=NR`, but they may not be assigned to: `NR=100` is a syntax error.
## Field names
@ -172,7 +172,7 @@ Things which are completely unsurprising, resembling many other languages:
* However, you cannot redeclare the *type* of an argument or a local: `var a=1; var a=2` is an error but `var a=1; a=2` is OK.
* All argument-passing is positional rather than by name; arguments are passed by value, not by reference. (This is also true for map-valued variables: they are not, and cannot be, passed by reference)
* All argument-passing is positional rather than by name; arguments are passed by value, not by reference. (This is also true for map-valued variables: they are not, and cannot be, passed by reference.)
* You can define locals (using `var`, `num`, etc.) at any scope (if-statements, else-statements, while-loops, for-loops, or the top-level scope), and nested scopes will have access (more details on scope in the next section). If you define a local variable with the same name inside an inner scope, then a new variable is created with the narrower scope.
@ -182,13 +182,13 @@ Things which are completely unsurprising, resembling many other languages:
Things which are perhaps surprising compared to other languages:
* Type declarations using `var`, or typed using `num`, `int`, `float`, `str`, and `bool` are necessary to declare local variables. Function arguments and variables bound in for-loops over stream records and out-of-stream variables are *implicitly* declared using `var`. (Some examples are shown below.)
* Type declarations using `var`, or typed using `num`, `int`, `float`, `str`, and `bool` are not necessary to declare local variables. Function arguments and variables bound in for-loops over stream records and out-of-stream variables are *implicitly* declared using `var`. (Some examples are shown below.)
* Type-checking is done at assignment time. For example, `float f = 0` is an error (since `0` is an integer), as is `float f = 0.0; f = 1`. For this reason I prefer to use `num` over `float` in most contexts since `num` encompasses integer and floating-point values. More information is at [Type-checking](reference-dsl-variables.md#type-checking).
* Bound variables in for-loops over stream records and out-of-stream variables are implicitly local to that block. E.g. in `for (k, v in $*) { ... }` `for ((k1, k2), v in @*) { ... }` if there are `k`, `v`, etc. in the enclosing scope then those will be masked by the loop-local bound variables in the loop, and moreover the values of the loop-local bound variables are not available after the end of the loop.
* For C-style triple-for loops, if a for-loop variable is defined using `var`, `int`, etc. then it is scoped to that for-loop. E.g. `for (i = 0; i < 10; i += 1) { ... }` and `for (int i = 0; i < 10; i += 1) { ... }`. (This is unsurprising.). If there is no typedecl and an outer-scope variable of that name exists, then it is used. (This is also unsurprising.) But of there is no outer-scope variable of that name then the variable is scoped to the for-loop only.
* For C-style triple-for loops, if a for-loop variable is defined using `var`, `int`, etc. then it is scoped to that for-loop. E.g. `for (i = 0; i < 10; i += 1) { ... }` and `for (int i = 0; i < 10; i += 1) { ... }`. (This is unsurprising.). If there is no typedecl and an outer-scope variable of that name exists, then it is used. (This is also unsurprising.) But if there is no outer-scope variable of that name, then the variable is scoped to the for-loop only.
The following example demonstrates the scope rules:
@ -212,7 +212,7 @@ GENMD_EOF
## Map literals
Miller's `put`/`filter` DSL has four kinds of hashmaps. **Stream records** are (single-level) maps from name to value. **Out-of-stream variables** and **local variables** can also be maps, although they can be multi-level hashmaps (e.g. `@sum[$x][$y]`). The fourth kind is **map literals**. These cannot be on the left-hand side of assignment expressions. Syntactically they look like JSON, although Miller allows string and integer keys in its map literals while JSON allows only string keys (e.g. `"3"` rather than `3`).
Miller's `put`/`filter` DSL has four kinds of hashmaps. **Stream records** are (single-level) maps from name to value. **Out-of-stream variables** and **local variables** can also be maps, although they can be multi-level hashmaps (e.g. `@sum[$x][$y]`). The fourth kind is **map literals**. These cannot be on the left-hand side of assignment expressions. Syntactically they look like JSON, although Miller allows string and integer keys in its map literals while JSON allows only string keys (e.g. `"3"` rather than `3`). Note though that integer keys become stringified in Miller: `@mymap[3]=4` results in `@mymap` being `{"3":4}`.
For example, the following swaps the input stream's `a` and `i` fields, modifies `y`, and drops the rest:
@ -226,17 +226,15 @@ Like out-of-stream and local variables, map literals can be multi-level:
GENMD_INCLUDE_AND_RUN_ESCAPED(data/map-literal-example-3.sh)
By default, map-valued expressions are dumped using JSON formatting. If you use `dump` to print a hashmap with integer keys and you don't want them double-quoted (JSON-style) then you can use `mlr put --jknquoteint`. See also `mlr put --help`.
## Type-checking
Miller's `put`/`filter` DSLs support two optional kinds of type-checking. One is inline **type-tests** and **type-assertions** within expressions. The other is **type declarations** for assignments to local variables, binding of arguments to user-defined functions, and return values from user-defined functions, These are discussed in the following subsections.
Miller's `put`/`filter` DSL supports two optional kinds of type-checking. One is inline **type-tests** and **type-assertions** within expressions. The other is **type declarations** for assignments to local variables, binding of arguments to user-defined functions, and return values from user-defined functions, These are discussed in the following subsections.
Use of type-checking is entirely up to you: omit it if you want flexibility with heterogeneous data; use it if you want to help catch misspellings in your DSL code or unexpected irregularities in your input data.
### Type-test and type-assertion expressions
The following `is...` functions take a value and return a boolean indicating whether the argument is of the indicated type. The `assert_...` functions return their argument if it is of the specified type, and cause a fatal error otherwise:
The following `is_...` functions take a value and return a boolean indicating whether the argument is of the indicated type. The `assert_...` functions return their argument if it is of the specified type, and cause a fatal error otherwise:
GENMD_RUN_COMMAND
mlr -f | grep ^is
@ -321,7 +319,7 @@ GENMD_EOF
## Null data: empty and absent
Please see [xxxx](reference-main-null-data.md).
Please see the [null-data reference page](reference-main-null-data.md).
## Aggregate variable assignments
@ -351,7 +349,7 @@ GENMD_EOF
GENMD_RUN_COMMAND
mlr --opprint put -q '
is_null(@xmax) || $x > @xmax {@xmax=$x; @recmax=$*};
is_null(@xmax) || $x > @xmax {@xmax = $x; @recmax = $*};
end {emit @recmax}
' data/small
GENMD_EOF

132
docs6/docs/reference-dsl.md
View file

@ -1,7 +1,7 @@
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
# DSL reference: overview
# DSL overview
## Overview
## Verbs compared to DSL
Here's comparison of verbs and `put`/`filter` DSL expressions:
@ -41,7 +41,79 @@ a=wye,x_sum=0.7778922255683036
Please see [Verbs Reference](reference-verbs.md) for information on verbs other than `put` and `filter`.
The essential usages of `mlr filter` and `mlr put` are for record-selection and record-updating expressions, respectively. For example, given the following input data:
## Implicit loop over records for main statements
The most important point about the Miller DSL is that it is designed for _streaming operation over records_.
DSL statements include:
* `func` and `subr` for user-defined functions and subroutines, which we'll look at later in the [separate page about them](reference-dsl-user-defined-functions.md);
* `begin` and `end` blocks, for statements you want to run before the first record, or after the last one;
* everything else, which collectively are called _main statements_.
The feature of _streaming operation over records_ is implemented by the main
statements getting invoked once per record. You don't explicitly loop over
records, as you would in some dataframes contexts; rather, _Miller loops over
records for you_, and it lets you specify what to do on each record: you write
the body of the loop.
(You can, if you like, use the per-record statements to grow a list of records,
then loop over them all in an `end` block. This is described in the page on
[operating over all records](operating-over-all-records.md)).
To see this in action, let's take a look at the [data/short.csv](./data/short.csv) file:
<pre class="pre-highlight-in-pair">
<b>cat data/short.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
word,value
apple,37
ball,28
cat,54
</pre>
There are three records in this file, with `word=apple`, `word=ball`, and
`word=cat`, respectively. Let's print something in a `begin` statement, add a
field in a main statement, and print something else in an `end` statement:
<pre class="pre-highlight-in-pair">
<b>mlr --csv --from data/short.csv put '</b>
<b> begin {</b>
<b> print "begin";</b>
<b> }</b>
<b> $nr = NR;</b>
<b> end {</b>
<b> print "end";</b>
<b> }</b>
<b>'</b>
</pre>
<pre class="pre-non-highlight-in-pair">
begin
word,value,nr
apple,37,1
ball,28,2
cat,54,3
end
</pre>
The `print` statements for `begin` and `end` went out before the first record
was seen and after the last was seen; the field-creation statement `$nr = NR`
was invoked three times, once for each record. We didn't explicitly loop over
records, since Miller was already looping over records, and invoked our main
statement on each loop iteration.
For almost all simple uses of the Miller programming language, this implicit
looping over records is probably all you will need. (For more involved cases you
can see the pages on [operating over all records](operating-on-all-records.md),
[out-of-stream variables](reference-dsl-variables.md#out-of-stream-variables),
and [two-pass algorithms](two-pass-algorithms.md).)
## Essential use: record-selection and record-updating
The essential usages of `mlr filter` and `mlr put` are for record-selection and
record-updating expressions, respectively. For example, given the following
input data:
<pre class="pre-highlight-in-pair">
<b>cat data/small</b>
@ -77,15 +149,61 @@ a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463,ab=eks_wye
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729,ab=wye_pan
</pre>
## Differences between put and filter
The two verbs `mlr filter` and `mlr put` are essentially the same. The only differences are:
* Expressions sent to `mlr filter` must end with a boolean expression, which is the filtering criterion;
* Expressions sent to `mlr filter` should contain a boolean expression, which is the filtering criterion. (If not, all records pass through.)
* `mlr filter` expressions may not reference the `filter` keyword within them; and
* `mlr filter` expressions may not reference the `filter` keyword within them.
* `mlr filter` expressions may not use `tee`, `emit`, `emitp`, or `emitf`.
## Location of boolean expression for filter
You can define and invoke functions and subroutines to help produce the bare-boolean statement, and record fields may be assigned in the statements before or after the bare-boolean statement. For example:
<pre class="pre-highlight-in-pair">
<b>mlr --c2p --from example.csv filter '</b>
<b> # Bare-boolean filter expression: only records matching this pass through:</b>
<b> $quantity >= 70;</b>
<b> # For records that do pass through, set these:</b>
<b> if ($rate > 8) {</b>
<b> $description = "high rate";</b>
<b> } else {</b>
<b> $description = "low rate";</b>
<b> }</b>
<b>'</b>
</pre>
<pre class="pre-non-highlight-in-pair">
color shape flag k index quantity rate description
red square true 2 15 79.2778 0.0130 low rate
red square false 4 48 77.5542 7.4670 low rate
purple triangle false 5 51 81.2290 8.5910 high rate
red square false 6 64 77.1991 9.5310 high rate
purple triangle false 7 65 80.1405 5.8240 low rate
purple square false 10 91 72.3735 8.2430 high rate
</pre>
<pre class="pre-highlight-in-pair">
<b>mlr --c2p --from example.csv filter '</b>
<b> # Bare-boolean filter expression: only records matching this pass through:</b>
<b> $shape =~ "^(...)(...)$";</b>
<b> # For records that do pass through, capture the first "(...)" into $left and</b>
<b> # the second "(...)" into $right</b>
<b> $left = "\1";</b>
<b> $right = "\2";</b>
<b>'</b>
</pre>
<pre class="pre-non-highlight-in-pair">
color shape flag k index quantity rate left right
red square true 2 15 79.2778 0.0130 squ are
red circle true 3 16 13.8103 2.9010 cir cle
red square false 4 48 77.5542 7.4670 squ are
red square false 6 64 77.1991 9.5310 squ are
yellow circle true 8 73 63.9785 4.2370 cir cle
yellow circle true 9 87 63.5058 8.3350 cir cle
purple square false 10 91 72.3735 8.2430 squ are
</pre>
All the rest is the same: in particular, you can define and invoke functions and subroutines to help produce the final boolean statement, and record fields may be assigned to in the statements preceding the final boolean statement.
There are more details and more choices, of course, as detailed in the following sections.

99
docs6/docs/reference-dsl.md.in
View file

@ -1,6 +1,6 @@
# DSL reference: overview
# DSL overview
## Overview
## Verbs compared to DSL
Here's comparison of verbs and `put`/`filter` DSL expressions:
@ -30,7 +30,65 @@ GENMD_EOF
Please see [Verbs Reference](reference-verbs.md) for information on verbs other than `put` and `filter`.
The essential usages of `mlr filter` and `mlr put` are for record-selection and record-updating expressions, respectively. For example, given the following input data:
## Implicit loop over records for main statements
The most important point about the Miller DSL is that it is designed for _streaming operation over records_.
DSL statements include:
* `func` and `subr` for user-defined functions and subroutines, which we'll look at later in the [separate page about them](reference-dsl-user-defined-functions.md);
* `begin` and `end` blocks, for statements you want to run before the first record, or after the last one;
* everything else, which collectively are called _main statements_.
The feature of _streaming operation over records_ is implemented by the main
statements getting invoked once per record. You don't explicitly loop over
records, as you would in some dataframes contexts; rather, _Miller loops over
records for you_, and it lets you specify what to do on each record: you write
the body of the loop.
(You can, if you like, use the per-record statements to grow a list of records,
then loop over them all in an `end` block. This is described in the page on
[operating over all records](operating-over-all-records.md)).
To see this in action, let's take a look at the [data/short.csv](./data/short.csv) file:
GENMD_RUN_COMMAND
cat data/short.csv
GENMD_EOF
There are three records in this file, with `word=apple`, `word=ball`, and
`word=cat`, respectively. Let's print something in a `begin` statement, add a
field in a main statement, and print something else in an `end` statement:
GENMD_RUN_COMMAND
mlr --csv --from data/short.csv put '
begin {
print "begin";
}
$nr = NR;
end {
print "end";
}
'
GENMD_EOF
The `print` statements for `begin` and `end` went out before the first record
was seen and after the last was seen; the field-creation statement `$nr = NR`
was invoked three times, once for each record. We didn't explicitly loop over
records, since Miller was already looping over records, and invoked our main
statement on each loop iteration.
For almost all simple uses of the Miller programming language, this implicit
looping over records is probably all you will need. (For more involved cases you
can see the pages on [operating over all records](operating-on-all-records.md),
[out-of-stream variables](reference-dsl-variables.md#out-of-stream-variables),
and [two-pass algorithms](two-pass-algorithms.md).)
## Essential use: record-selection and record-updating
The essential usages of `mlr filter` and `mlr put` are for record-selection and
record-updating expressions, respectively. For example, given the following
input data:
GENMD_RUN_COMMAND
cat data/small
@ -48,15 +106,42 @@ GENMD_RUN_COMMAND
mlr put '$ab = $a . "_" . $b ' data/small
GENMD_EOF
## Differences between put and filter
The two verbs `mlr filter` and `mlr put` are essentially the same. The only differences are:
* Expressions sent to `mlr filter` must end with a boolean expression, which is the filtering criterion;
* Expressions sent to `mlr filter` should contain a boolean expression, which is the filtering criterion. (If not, all records pass through.)
* `mlr filter` expressions may not reference the `filter` keyword within them; and
* `mlr filter` expressions may not reference the `filter` keyword within them.
* `mlr filter` expressions may not use `tee`, `emit`, `emitp`, or `emitf`.
## Location of boolean expression for filter
You can define and invoke functions and subroutines to help produce the bare-boolean statement, and record fields may be assigned in the statements before or after the bare-boolean statement. For example:
GENMD_RUN_COMMAND
mlr --c2p --from example.csv filter '
# Bare-boolean filter expression: only records matching this pass through:
$quantity >= 70;
# For records that do pass through, set these:
if ($rate > 8) {
$description = "high rate";
} else {
$description = "low rate";
}
'
GENMD_EOF
GENMD_RUN_COMMAND
mlr --c2p --from example.csv filter '
# Bare-boolean filter expression: only records matching this pass through:
$shape =~ "^(...)(...)$";
# For records that do pass through, capture the first "(...)" into $left and
# the second "(...)" into $right
$left = "\1";
$right = "\2";
'
GENMD_EOF
All the rest is the same: in particular, you can define and invoke functions and subroutines to help produce the final boolean statement, and record fields may be assigned to in the statements preceding the final boolean statement.
There are more details and more choices, of course, as detailed in the following sections.

6
docs6/docs/reference-main-arithmetic.md
View file

@ -1,5 +1,5 @@
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
# Reference: arithmetic
# Arithmetic
## Input scanning
@ -17,7 +17,7 @@ The sum, difference, and product of integers is again integer, except for when t
The short of it is that Miller does this transparently for you so you needn't think about it.
Implementation details of this, for the interested: integer adds and subtracts overflow by at most one bit so it suffices to check sign-changes. Thus, Miller allows you to add and subtract arbitrary 64-bit signed integers, converting only to float precisely when the result is less than -2\ :sup:`63` or greater than 2\ :sup:`63`\ -1. Multiplies, on the other hand, can overflow by a word size and a sign-change technique does not suffice to detect overflow. Instead Miller tests whether the floating-point product exceeds the representable integer range. Now, 64-bit integers have 64-bit precision while IEEE-doubles have only 52-bit mantissas -- so, there are 53 bits including implicit leading one. The following experiment explicitly demonstrates the resolution at this range:
Implementation details of this, for the interested: integer adds and subtracts overflow by at most one bit so it suffices to check sign-changes. Thus, Miller allows you to add and subtract arbitrary 64-bit signed integers, converting only to float precisely when the result is less than -2\*\*63 or greater than 2\*\*63 - 1. Multiplies, on the other hand, can overflow by a word size and a sign-change technique does not suffice to detect overflow. Instead, Miller tests whether the floating-point product exceeds the representable integer range. Now, 64-bit integers have 64-bit precision while IEEE-doubles have only 52-bit mantissas -- so, there are 53 bits including implicit leading one. The following experiment explicitly demonstrates the resolution at this range:
<pre class="pre-non-highlight-non-pair">
64-bit integer 64-bit integer Casted to double Back to 64-bit
@ -32,7 +32,7 @@ in hex in decimal integer
0x7fffffffffffffff 9223372036854775807 9223372036854775808.000000 0x8000000000000000
</pre>
That is, one cannot check an integer product to see if it is precisely greater than 2\ :sup:`63`\ -1 or less than -2\ :sup:`63` using either integer arithmetic (it may have already overflowed) or using double-precision (due to granularity). Instead Miller checks for overflow in 64-bit integer multiplication by seeing whether the absolute value of the double-precision product exceeds the largest representable IEEE double less than 2\ :sup:`63`, which we see from the listing above is 9223372036854774784. (An alternative would be to do all integer multiplies using handcrafted multi-word 128-bit arithmetic. This approach is not taken.)
That is, one cannot check an integer product to see if it is precisely greater than 2\*\*63 - 1 or less than -2\*\*63 using either integer arithmetic (it may have already overflowed) or using double-precision (due to granularity). Instead, Miller checks for overflow in 64-bit integer multiplication by seeing whether the absolute value of the double-precision product exceeds the largest representable IEEE double less than 2\*\*63, which we see from the listing above is 9223372036854774784. (An alternative would be to do all integer multiplies using handcrafted multi-word 128-bit arithmetic. This approach is not taken.)
## Pythonic division

6
docs6/docs/reference-main-arithmetic.md.in
View file

@ -1,4 +1,4 @@
# Reference: arithmetic
# Arithmetic
## Input scanning
@ -16,7 +16,7 @@ The sum, difference, and product of integers is again integer, except for when t
The short of it is that Miller does this transparently for you so you needn't think about it.
Implementation details of this, for the interested: integer adds and subtracts overflow by at most one bit so it suffices to check sign-changes. Thus, Miller allows you to add and subtract arbitrary 64-bit signed integers, converting only to float precisely when the result is less than -2\ :sup:`63` or greater than 2\ :sup:`63`\ -1. Multiplies, on the other hand, can overflow by a word size and a sign-change technique does not suffice to detect overflow. Instead Miller tests whether the floating-point product exceeds the representable integer range. Now, 64-bit integers have 64-bit precision while IEEE-doubles have only 52-bit mantissas -- so, there are 53 bits including implicit leading one. The following experiment explicitly demonstrates the resolution at this range:
Implementation details of this, for the interested: integer adds and subtracts overflow by at most one bit so it suffices to check sign-changes. Thus, Miller allows you to add and subtract arbitrary 64-bit signed integers, converting only to float precisely when the result is less than -2\*\*63 or greater than 2\*\*63 - 1. Multiplies, on the other hand, can overflow by a word size and a sign-change technique does not suffice to detect overflow. Instead, Miller tests whether the floating-point product exceeds the representable integer range. Now, 64-bit integers have 64-bit precision while IEEE-doubles have only 52-bit mantissas -- so, there are 53 bits including implicit leading one. The following experiment explicitly demonstrates the resolution at this range:
GENMD_CARDIFY
64-bit integer 64-bit integer Casted to double Back to 64-bit
@ -31,7 +31,7 @@ in hex in decimal integer
0x7fffffffffffffff 9223372036854775807 9223372036854775808.000000 0x8000000000000000
GENMD_EOF
That is, one cannot check an integer product to see if it is precisely greater than 2\ :sup:`63`\ -1 or less than -2\ :sup:`63` using either integer arithmetic (it may have already overflowed) or using double-precision (due to granularity). Instead Miller checks for overflow in 64-bit integer multiplication by seeing whether the absolute value of the double-precision product exceeds the largest representable IEEE double less than 2\ :sup:`63`, which we see from the listing above is 9223372036854774784. (An alternative would be to do all integer multiplies using handcrafted multi-word 128-bit arithmetic. This approach is not taken.)
That is, one cannot check an integer product to see if it is precisely greater than 2\*\*63 - 1 or less than -2\*\*63 using either integer arithmetic (it may have already overflowed) or using double-precision (due to granularity). Instead, Miller checks for overflow in 64-bit integer multiplication by seeing whether the absolute value of the double-precision product exceeds the largest representable IEEE double less than 2\*\*63, which we see from the listing above is 9223372036854774784. (An alternative would be to do all integer multiplies using handcrafted multi-word 128-bit arithmetic. This approach is not taken.)
## Pythonic division

4
docs6/docs/reference-main-auxiliary-commands.md
View file

@ -1,7 +1,7 @@
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
# Reference: auxiliary commands
# Auxiliary commands
There are a few nearly-standalone programs which have nothing to do with the rest of Miller, do not participate in record streams, and do not deal with file formats. They might as well be little standalone executables but they're delivered within the main Miller executable for convenience.
There are a few nearly-standalone programs which have nothing to do with the rest of Miller, do not participate in record streams, and do not deal with file formats. They might as well be little standalone executables, but instead they're delivered within the main Miller executable for convenience.
<pre class="pre-highlight-in-pair">
<b>mlr aux-list</b>

4
docs6/docs/reference-main-auxiliary-commands.md.in
View file

@ -1,6 +1,6 @@
# Reference: auxiliary commands
# Auxiliary commands
There are a few nearly-standalone programs which have nothing to do with the rest of Miller, do not participate in record streams, and do not deal with file formats. They might as well be little standalone executables but they're delivered within the main Miller executable for convenience.
There are a few nearly-standalone programs which have nothing to do with the rest of Miller, do not participate in record streams, and do not deal with file formats. They might as well be little standalone executables, but instead they're delivered within the main Miller executable for convenience.
GENMD_RUN_COMMAND
mlr aux-list

32
docs6/docs/reference-main-data-types.md
View file

@ -1,7 +1,29 @@
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
# Reference: data types
# Data types
Miller's input and output are all string-oriented: there is (as of August 2015 anyway) no support for binary record packing. In this sense, everything is a string in and out of Miller. During processing, field names are always strings, even if they have names like "3"; field values are usually strings. Field values' ability to be interpreted as a non-string type only has meaning when comparison or function operations are done on them. And it is an error condition if Miller encounters non-numeric (or otherwise mistyped) data in a field in which it has been asked to do numeric (or otherwise type-specific) operations.
## List of types
Miller's types for function processing are:
* Scalars:
* **string**
* **float** (double-precision)
* **int** (64-bit signed)
* **boolean**
* Collections:
* **map**
* **array**
* Nulls and error:
* **empty-null** (empty string)
* **absent-null** (reads of unset right-hand sides, or fall-through non-explicit return values from user-defined functions)
* **JSON-null** (TODO)
* **error** (TODO: give an example)
TODO: point to null-data page; arrays page; arithmetic page; what else?
## To be ported
Miller's input and output are all string-oriented: there is (as of August 2015 anyway) no support for binary record packing. In this sense, everything is a string in and out of Miller. During processing, field names are always strings, even if they have names like "3"; field values can have various data types. Field values' ability to be interpreted as a non-string type only has meaning when comparison or function operations are done on them. And it is an error condition if Miller encounters non-numeric (or otherwise mistyped) data in a field in which it has been asked to do numeric (or otherwise type-specific) operations.
Field values are treated as numeric for the following:
@ -13,7 +35,7 @@ For `mlr put` and `mlr filter`:
* Miller's types for function processing are **empty-null** (empty string), **absent-null** (reads of unset right-hand sides, or fall-through non-explicit return values from user-defined functions), **error**, **string**, **float** (double-precision), **int** (64-bit signed), and **boolean**.
* On input, string values representable as numbers, e.g. "3" or "3.1", are treated as int or float, respectively. If a record has `x=1,y=2` then `mlr put '$z=$x+$y'` will produce `x=1,y=2,z=3`, and `mlr put '$z=$x.$y'` does not give an error simply because the dot operator has been generalized to stringify non-strings. To coerce back to string for processing, use the `string` function: `mlr put '$z=string($x).string($y)'` will produce `x=1,y=2,z=12`.
* On input, string values representable as numbers, e.g. "3" or "3.1", are treated as int or float, respectively. If a record has `x=1,y=2` then `mlr put '$z=$x+$y'` will produce `x=1,y=2,z=3`, and `mlr put '$z=$x.$y'` does not give an error, but only because the dot operator has been generalized to stringify non-strings. To coerce back to string for processing, use the `string` function: `mlr put '$z=string($x).string($y)'` will produce `x=1,y=2,z=12`.
* On input, string values representable as boolean (e.g. `"true"`, `"false"`) are *not* automatically treated as boolean. (This is because `"true"` and `"false"` are ordinary words, and auto string-to-boolean on a column consisting of words would result in some strings mixed with some booleans.) Use the `boolean` function to coerce: e.g. giving the record `x=1,y=2,w=false` to `mlr put '$z=($x<$y) || boolean($w)'`.
@ -21,7 +43,7 @@ For `mlr put` and `mlr filter`:
* All math functions described in `mlr --help-all-functions` take integer as well as float input.
## String literals
## Escape sequences for string literals
You can use the following backslash escapes for strings such as between the double quotes in contexts such as `mlr filter '$name =~ "..."'`, `mlr put '$name = $othername . "..."'`, `mlr put '$name = sub($name, "...", "...")`, etc.:
@ -41,5 +63,5 @@ See also [https://en.wikipedia.org/wiki/Escape_sequences_in_C](https://en.wikipe
These replacements apply only to strings you key in for the DSL expressions for `filter` and `put`: that is, if you type `\t` in a string literal for a `filter`/`put` expression, it will be turned into a tab character. If you want a backslash followed by a `t`, then please type `\\t`.
However, these replacements are not done automatically within your data stream. If you wish to make these replacements, you can do, for example, for a field named `field`, `mlr put '$field = gsub($field, "\\t", "\t")'`. If you need to make such a replacement for all fields in your data, you should probably simply use the system `sed` command.
However, these replacements are done automatically only for string literals within DSL expressions -- they are not done automatically to fields within your data stream. If you wish to make these replacements, you can do (for example) `mlr put '$field = gsub($field, "\\t", "\t")'`. If you need to make such a replacement for all fields in your data, you should probably use the system `sed` command instead.

32
docs6/docs/reference-main-data-types.md.in
View file

@ -1,6 +1,28 @@
# Reference: data types
# Data types
Miller's input and output are all string-oriented: there is (as of August 2015 anyway) no support for binary record packing. In this sense, everything is a string in and out of Miller. During processing, field names are always strings, even if they have names like "3"; field values are usually strings. Field values' ability to be interpreted as a non-string type only has meaning when comparison or function operations are done on them. And it is an error condition if Miller encounters non-numeric (or otherwise mistyped) data in a field in which it has been asked to do numeric (or otherwise type-specific) operations.
## List of types
Miller's types for function processing are:
* Scalars:
* **string**
* **float** (double-precision)
* **int** (64-bit signed)
* **boolean**
* Collections:
* **map**
* **array**
* Nulls and error:
* **empty-null** (empty string)
* **absent-null** (reads of unset right-hand sides, or fall-through non-explicit return values from user-defined functions)
* **JSON-null** (TODO)
* **error** (TODO: give an example)
TODO: point to null-data page; arrays page; arithmetic page; what else?
## To be ported
Miller's input and output are all string-oriented: there is (as of August 2015 anyway) no support for binary record packing. In this sense, everything is a string in and out of Miller. During processing, field names are always strings, even if they have names like "3"; field values can have various data types. Field values' ability to be interpreted as a non-string type only has meaning when comparison or function operations are done on them. And it is an error condition if Miller encounters non-numeric (or otherwise mistyped) data in a field in which it has been asked to do numeric (or otherwise type-specific) operations.
Field values are treated as numeric for the following:
@ -12,7 +34,7 @@ For `mlr put` and `mlr filter`:
* Miller's types for function processing are **empty-null** (empty string), **absent-null** (reads of unset right-hand sides, or fall-through non-explicit return values from user-defined functions), **error**, **string**, **float** (double-precision), **int** (64-bit signed), and **boolean**.
* On input, string values representable as numbers, e.g. "3" or "3.1", are treated as int or float, respectively. If a record has `x=1,y=2` then `mlr put '$z=$x+$y'` will produce `x=1,y=2,z=3`, and `mlr put '$z=$x.$y'` does not give an error simply because the dot operator has been generalized to stringify non-strings. To coerce back to string for processing, use the `string` function: `mlr put '$z=string($x).string($y)'` will produce `x=1,y=2,z=12`.
* On input, string values representable as numbers, e.g. "3" or "3.1", are treated as int or float, respectively. If a record has `x=1,y=2` then `mlr put '$z=$x+$y'` will produce `x=1,y=2,z=3`, and `mlr put '$z=$x.$y'` does not give an error, but only because the dot operator has been generalized to stringify non-strings. To coerce back to string for processing, use the `string` function: `mlr put '$z=string($x).string($y)'` will produce `x=1,y=2,z=12`.
* On input, string values representable as boolean (e.g. `"true"`, `"false"`) are *not* automatically treated as boolean. (This is because `"true"` and `"false"` are ordinary words, and auto string-to-boolean on a column consisting of words would result in some strings mixed with some booleans.) Use the `boolean` function to coerce: e.g. giving the record `x=1,y=2,w=false` to `mlr put '$z=($x<$y) || boolean($w)'`.
@ -20,7 +42,7 @@ For `mlr put` and `mlr filter`:
* All math functions described in `mlr --help-all-functions` take integer as well as float input.
## String literals
## Escape sequences for string literals
You can use the following backslash escapes for strings such as between the double quotes in contexts such as `mlr filter '$name =~ "..."'`, `mlr put '$name = $othername . "..."'`, `mlr put '$name = sub($name, "...", "...")`, etc.:
@ -40,5 +62,5 @@ See also [https://en.wikipedia.org/wiki/Escape_sequences_in_C](https://en.wikipe
These replacements apply only to strings you key in for the DSL expressions for `filter` and `put`: that is, if you type `\t` in a string literal for a `filter`/`put` expression, it will be turned into a tab character. If you want a backslash followed by a `t`, then please type `\\t`.
However, these replacements are not done automatically within your data stream. If you wish to make these replacements, you can do, for example, for a field named `field`, `mlr put '$field = gsub($field, "\\t", "\t")'`. If you need to make such a replacement for all fields in your data, you should probably simply use the system `sed` command.
However, these replacements are done automatically only for string literals within DSL expressions -- they are not done automatically to fields within your data stream. If you wish to make these replacements, you can do (for example) `mlr put '$field = gsub($field, "\\t", "\t")'`. If you need to make such a replacement for all fields in your data, you should probably use the system `sed` command instead.

2
docs6/docs/reference-main-env-vars.md
View file

@ -1,5 +1,5 @@
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
# Reference: Miller environment variables
# Miller environment variables
The following environment variables affect how Miller works:

2
docs6/docs/reference-main-env-vars.md.in
View file

@ -1,4 +1,4 @@
# Reference: Miller environment variables
# Miller environment variables
The following environment variables affect how Miller works:

16
docs6/docs/reference-main-io-options.md
View file

@ -1,5 +1,5 @@
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
# Reference: I/O options
# I/O options
## Formats
@ -49,7 +49,7 @@ Additional notes:
* DKVP (key-value-pair) format is the default for input and output. So, `--oxtab` is the same as `--idkvp --oxtab`.
**Pro-tip:** Please use either **--format1**, or **--iformat1 --oformat2**. If you use **--format1 --oformat2** then what happens is that flags are set up for input *and* output for format1, some of which are overwritten for output in format2. For technical reasons, having `--oformat2` clobber all the output-related effects of `--format1` also removes some flexibility from the command-line interface. See also [https://github.com/johnkerl/miller/issues/180](https://github.com/johnkerl/miller/issues/180) and [https://github.com/johnkerl/miller/issues/199](https://github.com/johnkerl/miller/issues/199).
**Pro-tip:** Please use either **--format1**, or **--iformat1 --oformat2**. If you use **--format1 --oformat2** then what happens is that flags are set up for input *and* output for format1, some of which are overwritten for output in format2. For technical reasons, having `--oformat2` clobber all the output-related effects of `--format1` also removes some flexibility from the command-line interface. See also Miller issues [180](https://github.com/johnkerl/miller/issues/180) and [199](https://github.com/johnkerl/miller/issues/199).
## In-place mode
@ -120,25 +120,25 @@ Options:
The command-line option `--ofmt {format string}` is the global number format for commands which generate numeric output, e.g. `stats1`, `stats2`, `histogram`, and `step`, as well as `mlr put`. Examples:
<pre class="pre-non-highlight-non-pair">
--ofmt %.9le --ofmt %.6lf --ofmt %.0lf
--ofmt %.9e --ofmt %.6f --ofmt %.0f
</pre>
These are just familiar `printf` formats applied to double-precision numbers. Please don't use `%s` or `%d`. Additionally, if you use leading width (e.g. `%18.12lf`) then the output will contain embedded whitespace, which may not be what you want if you pipe the output to something else, particularly CSV. I use Miller's pretty-print format (`mlr --opprint`) to column-align numerical data.
These are just familiar `printf` formats. (TODO: write about type-checking once that's implemented.) Additionally, if you use leading width (e.g. `%18.12f`) then the output will contain embedded whitespace, which may not be what you want if you pipe the output to something else, particularly CSV. I use Miller's pretty-print format (`mlr --opprint`) to column-align numerical data.
To apply formatting to a single field, overriding the global `ofmt`, use `fmtnum` function within `mlr put`. For example:
<pre class="pre-highlight-in-pair">
<b>echo 'x=3.1,y=4.3' | mlr put '$z=fmtnum($x*$y,"%08lf")'</b>
<b>echo 'x=3.1,y=4.3' | mlr put '$z=fmtnum($x*$y,"%08f")'</b>
</pre>
<pre class="pre-non-highlight-in-pair">
x=3.1,y=4.3,z=%!l(float64=00013.33)f
x=3.1,y=4.3,z=13.330000
</pre>
<pre class="pre-highlight-in-pair">
<b>echo 'x=0xffff,y=0xff' | mlr put '$z=fmtnum(int($x*$y),"%08llx")'</b>
<b>echo 'x=0xffff,y=0xff' | mlr put '$z=fmtnum(int($x*$y),"%08x")'</b>
</pre>
<pre class="pre-non-highlight-in-pair">
x=0xffff,y=0xff,z=%!l(int=16711425)lx
x=0xffff,y=0xff,z=00feff01
</pre>
Input conversion from hexadecimal is done automatically on fields handled by `mlr put` and `mlr filter` as long as the field value begins with "0x". To apply output conversion to hexadecimal on a single column, you may use `fmtnum`, or the keystroke-saving `hexfmt` function. Example:

12
docs6/docs/reference-main-io-options.md.in
View file

@ -1,4 +1,4 @@
# Reference: I/O options
# I/O options
## Formats
@ -32,7 +32,7 @@ Additional notes:
* DKVP (key-value-pair) format is the default for input and output. So, `--oxtab` is the same as `--idkvp --oxtab`.
**Pro-tip:** Please use either **--format1**, or **--iformat1 --oformat2**. If you use **--format1 --oformat2** then what happens is that flags are set up for input *and* output for format1, some of which are overwritten for output in format2. For technical reasons, having `--oformat2` clobber all the output-related effects of `--format1` also removes some flexibility from the command-line interface. See also [https://github.com/johnkerl/miller/issues/180](https://github.com/johnkerl/miller/issues/180) and [https://github.com/johnkerl/miller/issues/199](https://github.com/johnkerl/miller/issues/199).
**Pro-tip:** Please use either **--format1**, or **--iformat1 --oformat2**. If you use **--format1 --oformat2** then what happens is that flags are set up for input *and* output for format1, some of which are overwritten for output in format2. For technical reasons, having `--oformat2` clobber all the output-related effects of `--format1` also removes some flexibility from the command-line interface. See also Miller issues [180](https://github.com/johnkerl/miller/issues/180) and [199](https://github.com/johnkerl/miller/issues/199).
## In-place mode
@ -103,19 +103,19 @@ GENMD_EOF
The command-line option `--ofmt {format string}` is the global number format for commands which generate numeric output, e.g. `stats1`, `stats2`, `histogram`, and `step`, as well as `mlr put`. Examples:
GENMD_CARDIFY
--ofmt %.9le --ofmt %.6lf --ofmt %.0lf
--ofmt %.9e --ofmt %.6f --ofmt %.0f
GENMD_EOF
These are just familiar `printf` formats applied to double-precision numbers. Please don't use `%s` or `%d`. Additionally, if you use leading width (e.g. `%18.12lf`) then the output will contain embedded whitespace, which may not be what you want if you pipe the output to something else, particularly CSV. I use Miller's pretty-print format (`mlr --opprint`) to column-align numerical data.
These are just familiar `printf` formats. (TODO: write about type-checking once that's implemented.) Additionally, if you use leading width (e.g. `%18.12f`) then the output will contain embedded whitespace, which may not be what you want if you pipe the output to something else, particularly CSV. I use Miller's pretty-print format (`mlr --opprint`) to column-align numerical data.
To apply formatting to a single field, overriding the global `ofmt`, use `fmtnum` function within `mlr put`. For example:
GENMD_RUN_COMMAND
echo 'x=3.1,y=4.3' | mlr put '$z=fmtnum($x*$y,"%08lf")'
echo 'x=3.1,y=4.3' | mlr put '$z=fmtnum($x*$y,"%08f")'
GENMD_EOF
GENMD_RUN_COMMAND
echo 'x=0xffff,y=0xff' | mlr put '$z=fmtnum(int($x*$y),"%08llx")'
echo 'x=0xffff,y=0xff' | mlr put '$z=fmtnum(int($x*$y),"%08x")'
GENMD_EOF
Input conversion from hexadecimal is done automatically on fields handled by `mlr put` and `mlr filter` as long as the field value begins with "0x". To apply output conversion to hexadecimal on a single column, you may use `fmtnum`, or the keystroke-saving `hexfmt` function. Example:

6
docs6/docs/reference-main-null-data.md
View file

@ -1,5 +1,5 @@
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
# Reference: null data
# null data
One of Miller's key features is its support for **heterogeneous** data. For example, take `mlr sort`: if you try to sort on field `hostname` when not all records in the data stream *have* a field named `hostname`, it is not an error (although you could pre-filter the data stream using `mlr having-fields --at-least hostname then sort ...`). Rather, records lacking one or more sort keys are simply output contiguously by `mlr sort`.
@ -27,7 +27,7 @@ a=5,b=7
</pre>
<pre class="pre-highlight-in-pair">
<b>mlr sort -n a data/sort-null.dat</b>
<b>mlr sort -n a data/sort-null.dat</b>
</pre>
<pre class="pre-non-highlight-in-pair">
a=1,b=8
@ -106,7 +106,7 @@ The reasoning is as follows:
* Absent out-of-stream-variable values are precisely what allow you to write `mlr put '@sum += $x'`. Otherwise you would have to write `mlr put 'begin{@sum = 0}; @sum += $x'` -- which is tolerable -- but for `mlr put 'begin{...}; @sum[$a][$b] += $x'` you'd have to pre-initialize `@sum` for all values of `$a` and `$b` in your input data stream, which is intolerable.
* The penalty for the absent feature is that misspelled variables can be hard to find: e.g. in `mlr put 'begin{@sumx = 10}; ...; update @sumx somehow per-record; ...; end {@something = @sum * 2}'` the accumulator is spelt `@sumx` in the begin-block but `@sum` in the end-block, where since it is absent, `@sum*2` evaluates to 2. See also the section on [DSL reference: errors and transparency](reference-dsl-errors.md).
* The penalty for the absent feature is that misspelled variables can be hard to find: e.g. in `mlr put 'begin{@sumx = 10}; ...; update @sumx somehow per-record; ...; end {@something = @sum * 2}'` the accumulator is spelt `@sumx` in the begin-block but `@sum` in the end-block, where since it is absent, `@sum*2` evaluates to 2. See also the section on [DSL errors and transparency](reference-dsl-errors.md).
Since absent plus absent is absent (and likewise for other operators), accumulations such as `@sum += $x` work correctly on heterogenous data, as do within-record formulas if both operands are absent. If one operand is present, you may get behavior you don't desire. To work around this -- namely, to set an output field only for records which have all the inputs present -- you can use a pattern-action block with `is_present`:

Some files were not shown because too many files have changed in this diff Show more