Mirrors/miller
1
0
Fork 0
mirror of https://github.com/johnkerl/miller.git synced 2026-07-17 16:38:54 +00:00
Code Issues Projects Releases Wiki Activity

More miller-6 mkdocs porting from sphinx (#617)

Browse source 
* More mkdocs porting

* mkdocs-ify http/external links

* move README.md up one level

* Further sharpen code-sample CSS

* fix last internal-ref links
This commit is contained in:
John Kerl 2021-08-04 21:50:41 -04:00 • committed by GitHub
parent 11eac853d2
commit fa3ee05822
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
141 changed files with 4449 additions and 4352 deletions
Download patch file Download diff file Expand all files Collapse all files

42
docs6b/README.md Normal file
View file

@ -0,0 +1,42 @@
# Miller docs
## Why use Mkdocs
* Connects to https://miller.readthedocs.io so people can get their docmods onto the web instead of the self-hosted https://johnkerl.org/miller/doc. Thanks to @pabloab for the great advice!
* More standard look and feel -- lots of people use readthedocs for other things so this should feel familiar.
* We get a Search feature for free.
* Mkdocs vs Sphinx: these are similar tools, but I find that I more easily get better desktop+mobile formatting using Mkdocs.
## Contributing
* You need `pip install mkdocs` (or `pip3 install mkdocs`).
* The docs include lots of live code examples which will be invoked using `mlr` which must be somewhere in your `$PATH`.
* Clone https://github.com/johnkerl/miller and cd into `docs/` within your clone.
* Quick-editing loop:
* In one terminal, cd to this directory and leave `mkdocs serve` running.
* In another terminal, cd to the `docs` subdirectory and edit `*.md.in`.
* In your browser, visit http://127.0.0.1:8000
* Alternate editing loop:
* Leave one terminal open as a place you will run `mkdocs build`
* In one terminal, cd to the `docs` subdirectory and edit `*.md.in`.
* In the first terminal, run `mkdocs build` which will populate the `site` directory.
* In your browser, visit `file:///your/path/to/miller/docs/site/index.html`
* Link-checking:
* `sudo pip3 install git+https://github.com/linkchecker/linkchecker.git`
* `cd site` and `linkchecker .`
* Submitting:
* `git add` your modified files, `git commit`, `git push`, and submit a PR at https://github.com/johnkerl/miller.
## Notes
* CSS:
* I used the Mkdocs Readthedocs theme which I like a lot. I customized `docs/extra.css` for Miller coloring/branding.
* Live code:
* I didn't find a way to include non-Python live-code examples within Mkdocs so I adapted the pre-Mkdocs Miller-doc strategy which is to have a generator script read a template file (here, `foo.md.in`), run the marked lines, and generate the output file (`foo.md`). This is `genmds`.
* Edit the `*.md.in` files, not `*.md` directly.
* Within the `*.md.in` files are lines like `GENMD_RUN_COMMAND`. These will be run, and their output included, by `genmds` which calls the `genmds` script for you.
* readthedocs:
* https://readthedocs.org/
* https://readthedocs.org/projects/miller/
* https://readthedocs.org/projects/miller/builds/
* https://miller.readthedocs.io/en/latest/

1
docs6b/docs/.vimrc
View file

@ -1,2 +1 @@
map \d :w<C-m>:!clear;build-one %<C-m>
map \f :w<C-m>:!clear;make html<C-m>

342
docs6b/docs/10min.md
View file

@ -5,23 +5,28 @@
You can install Miller for various platforms as follows:
* Linux: ``yum install miller`` or ``apt-get install miller`` depending on your flavor of Linux
* MacOS: ``brew install miller`` or ``port install miller`` depending on your preference of [Homebrew](https://brew.sh>`_ or `MacPorts <https://macports.org).
* Windows: ``choco install miller`` using [Chocolatey](https://chocolatey.org).
* You can get latest builds for Linux, MacOS, and Windows by visiting https://github.com/johnkerl/miller/actions, selecting the latest build, and clicking _Artifacts_. (These are retained for 5 days after each commit.)
* Linux: `yum install miller` or `apt-get install miller` depending on your flavor of Linux
* MacOS: `brew install miller` or `port install miller` depending on your preference of [Homebrew](https://brew.sh) or [MacPorts](https://macports.org).
* Windows: `choco install miller` using [Chocolatey](https://chocolatey.org).
* You can get latest builds for Linux, MacOS, and Windows by visiting [https://github.com/johnkerl/miller/actions](https://github.com/johnkerl/miller/actions).
, selecting the latest build, and clicking _Artifacts_. (These are retained for 5 days after each commit.)
* See also the [build page](build.md) if you prefer -- in particular, if your platform's package manager doesn't have the latest release.
As a first check, you should be able to run ``mlr --version`` at your system's command prompt and see something like the following:
As a first check, you should be able to run `mlr --version` at your system's command prompt and see something like the following:
<pre>
<pre class="pre-highlight">
<b>mlr --version</b>
</pre>
<pre class="pre-non-highlight">
Miller v6.0.0-dev
</pre>
As a second check, given [example.csv](./example.csv) you should be able to do
<pre>
<pre class="pre-highlight">
<b>mlr --csv cat example.csv</b>
</pre>
<pre class="pre-non-highlight">
color,shape,flag,index,quantity,rate
yellow,triangle,true,11,43.6498,9.8870
red,square,true,15,79.2778,0.0130
@ -35,8 +40,10 @@ yellow,circle,true,87,63.5058,8.3350
purple,square,false,91,72.3735,8.2430
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --icsv --opprint cat example.csv</b>
</pre>
<pre class="pre-non-highlight">
color shape flag index quantity rate
yellow triangle true 11 43.6498 9.8870
red square true 15 79.2778 0.0130
@ -56,10 +63,12 @@ If you run into issues on these checks, please check out the resources on the [c
Let's take a quick look at some of the most useful Miller verbs -- file-format-aware, name-index-empowered equivalents of standard system commands.
``mlr cat`` is like system ``cat`` (or ``type`` on Windows) -- it passes the data through unmodified:
`mlr cat` is like system `cat` (or `type` on Windows) -- it passes the data through unmodified:
<pre>
<pre class="pre-highlight">
<b>mlr --csv cat example.csv</b>
</pre>
<pre class="pre-non-highlight">
color,shape,flag,index,quantity,rate
yellow,triangle,true,11,43.6498,9.8870
red,square,true,15,79.2778,0.0130
@ -73,10 +82,12 @@ yellow,circle,true,87,63.5058,8.3350
purple,square,false,91,72.3735,8.2430
</pre>
But ``mlr cat`` can also do format conversion -- for example, you can pretty-print in tabular format:
But `mlr cat` can also do format conversion -- for example, you can pretty-print in tabular format:
<pre>
<pre class="pre-highlight">
<b>mlr --icsv --opprint cat example.csv</b>
</pre>
<pre class="pre-non-highlight">
color shape flag index quantity rate
yellow triangle true 11 43.6498 9.8870
red square true 15 79.2778 0.0130
@ -90,10 +101,12 @@ yellow circle true 87 63.5058 8.3350
purple square false 91 72.3735 8.2430
</pre>
``mlr head`` and ``mlr tail`` count records rather than lines. Whether you're getting the first few records or the last few, the CSV header is included either way:
`mlr head` and `mlr tail` count records rather than lines. Whether you're getting the first few records or the last few, the CSV header is included either way:
<pre>
<pre class="pre-highlight">
<b>mlr --csv head -n 4 example.csv</b>
</pre>
<pre class="pre-non-highlight">
color,shape,flag,index,quantity,rate
yellow,triangle,true,11,43.6498,9.8870
red,square,true,15,79.2778,0.0130
@ -101,8 +114,10 @@ red,circle,true,16,13.8103,2.9010
red,square,false,48,77.5542,7.4670
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --csv tail -n 4 example.csv</b>
</pre>
<pre class="pre-non-highlight">
color,shape,flag,index,quantity,rate
purple,triangle,false,65,80.1405,5.8240
yellow,circle,true,73,63.9785,4.2370
@ -110,8 +125,10 @@ yellow,circle,true,87,63.5058,8.3350
purple,square,false,91,72.3735,8.2430
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --icsv --ojson tail -n 2 example.csv</b>
</pre>
<pre class="pre-non-highlight">
{
"color": "yellow",
"shape": "circle",
@ -132,8 +149,10 @@ purple,square,false,91,72.3735,8.2430
You can sort on a single field:
<pre>
<pre class="pre-highlight">
<b>mlr --icsv --opprint sort -f shape example.csv</b>
</pre>
<pre class="pre-non-highlight">
color shape flag index quantity rate
red circle true 16 13.8103 2.9010
yellow circle true 73 63.9785 4.2370
@ -149,8 +168,10 @@ purple triangle false 65 80.1405 5.8240
Or, you can sort primarily alphabetically on one field, then secondarily numerically descending on another field, and so on:
<pre>
<pre class="pre-highlight">
<b>mlr --icsv --opprint sort -f shape -nr index example.csv</b>
</pre>
<pre class="pre-non-highlight">
color shape flag index quantity rate
yellow circle true 87 63.5058 8.3350
yellow circle true 73 63.9785 4.2370
@ -164,10 +185,12 @@ purple triangle false 51 81.2290 8.5910
yellow triangle true 11 43.6498 9.8870
</pre>
If there are fields you don't want to see in your data, you can use ``cut`` to keep only the ones you want, in the same order they appeared in the input data:
If there are fields you don't want to see in your data, you can use `cut` to keep only the ones you want, in the same order they appeared in the input data:
<pre>
<pre class="pre-highlight">
<b>mlr --icsv --opprint cut -f flag,shape example.csv</b>
</pre>
<pre class="pre-non-highlight">
shape flag
triangle true
square true
@ -181,10 +204,12 @@ circle true
square false
</pre>
You can also use ``cut -o`` to keep specified fields, but in your preferred order:
You can also use `cut -o` to keep specified fields, but in your preferred order:
<pre>
<pre class="pre-highlight">
<b>mlr --icsv --opprint cut -o -f flag,shape example.csv</b>
</pre>
<pre class="pre-non-highlight">
flag shape
true triangle
true square
@ -198,10 +223,12 @@ true circle
false square
</pre>
You can use ``cut -x`` to omit fields you don't care about:
You can use `cut -x` to omit fields you don't care about:
<pre>
<pre class="pre-highlight">
<b>mlr --icsv --opprint cut -x -f flag,shape example.csv</b>
</pre>
<pre class="pre-non-highlight">
color index quantity rate
yellow 11 43.6498 9.8870
red 15 79.2778 0.0130
@ -215,10 +242,12 @@ yellow 87 63.5058 8.3350
purple 91 72.3735 8.2430
</pre>
You can use ``filter`` to keep only records you care about:
You can use `filter` to keep only records you care about:
<pre>
<pre class="pre-highlight">
<b>mlr --icsv --opprint filter '$color == "red"' example.csv</b>
</pre>
<pre class="pre-non-highlight">
color shape flag index quantity rate
red square true 15 79.2778 0.0130
red circle true 16 13.8103 2.9010
@ -226,20 +255,24 @@ red square false 48 77.5542 7.4670
red square false 64 77.1991 9.5310
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --icsv --opprint filter '$color == "red" && $flag == true' example.csv</b>
</pre>
<pre class="pre-non-highlight">
color shape flag index quantity rate
red square true 15 79.2778 0.0130
red circle true 16 13.8103 2.9010
</pre>
You can use ``put`` to create new fields which are computed from other fields:
You can use `put` to create new fields which are computed from other fields:
<pre>
<pre class="pre-highlight">
<b>mlr --icsv --opprint put '</b>
<b> $ratio = $quantity / $rate;</b>
<b> $color_shape = $color . "_" . $shape</b>
<b>' example.csv</b>
</pre>
<pre class="pre-non-highlight">
color shape flag index quantity rate ratio color_shape
yellow triangle true 11 43.6498 9.8870 4.414868008496004 yellow_triangle
red square true 15 79.2778 0.0130 6098.292307692308 red_square
@ -253,10 +286,12 @@ yellow circle true 87 63.5058 8.3350 7.619172165566886 yellow_circle
purple square false 91 72.3735 8.2430 8.779995147397793 purple_square
</pre>
Even though Miller's main selling point is name-indexing, sometimes you really want to refer to a field name by its positional index. Use ``$[[3]]`` to access the name of field 3 or ``$[[[3]]]`` to access the value of field 3:
Even though Miller's main selling point is name-indexing, sometimes you really want to refer to a field name by its positional index. Use `$[[3]]` to access the name of field 3 or `$[[[3]]]` to access the value of field 3:
<pre>
<pre class="pre-highlight">
<b>mlr --icsv --opprint put '$[[3]] = "NEW"' example.csv</b>
</pre>
<pre class="pre-non-highlight">
color shape NEW index quantity rate
yellow triangle true 11 43.6498 9.8870
red square true 15 79.2778 0.0130
@ -270,8 +305,10 @@ yellow circle true 87 63.5058 8.3350
purple square false 91 72.3735 8.2430
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --icsv --opprint put '$[[[3]]] = "NEW"' example.csv</b>
</pre>
<pre class="pre-non-highlight">
color shape flag index quantity rate
yellow triangle NEW 11 43.6498 9.8870
red square NEW 15 79.2778 0.0130
@ -289,23 +326,29 @@ You can find the full list of verbs at the [Verbs Reference](reference-verbs.md)
## Multiple input files
Miller takes all the files from the command line as an input stream. But it's format-aware, so it doesn't repeat CSV header lines. For example, with input files [data/a.csv](data/a.csv and [data/b.csv](data/b.csv), the system ``cat`` command will repeat header lines:
Miller takes all the files from the command line as an input stream. But it's format-aware, so it doesn't repeat CSV header lines. For example, with input files [data/a.csv](data/a.csv and [data/b.csv](data/b.csv), the system `cat` command will repeat header lines:
<pre>
<pre class="pre-highlight">
<b>cat data/a.csv</b>
</pre>
<pre class="pre-non-highlight">
a,b,c
1,2,3
4,5,6
</pre>
<pre>
<pre class="pre-highlight">
<b>cat data/b.csv</b>
</pre>
<pre class="pre-non-highlight">
a,b,c
7,8,9
</pre>
<pre>
<pre class="pre-highlight">
<b>cat data/a.csv data/b.csv</b>
</pre>
<pre class="pre-non-highlight">
a,b,c
1,2,3
4,5,6
@ -313,10 +356,12 @@ a,b,c
7,8,9
</pre>
However, ``mlr cat`` will not:
However, `mlr cat` will not:
<pre>
<pre class="pre-highlight">
<b>mlr --csv cat data/a.csv data/b.csv</b>
</pre>
<pre class="pre-non-highlight">
a,b,c
1,2,3
4,5,6
@ -327,39 +372,47 @@ a,b,c
Often we want to chain queries together -- for example, sorting by a field and taking the top few values. We can do this using pipes:
<pre>
<pre class="pre-highlight">
<b>mlr --csv sort -nr index example.csv | mlr --icsv --opprint head -n 3</b>
</pre>
<pre class="pre-non-highlight">
color shape flag index quantity rate
purple square false 91 72.3735 8.2430
yellow circle true 87 63.5058 8.3350
yellow circle true 73 63.9785 4.2370
</pre>
This works fine -- but Miller also lets you chain verbs together using the word ``then``. Think of this as a Miller-internal pipe that lets you use fewer keystrokes:
This works fine -- but Miller also lets you chain verbs together using the word `then`. Think of this as a Miller-internal pipe that lets you use fewer keystrokes:
<pre>
<pre class="pre-highlight">
<b>mlr --icsv --opprint sort -nr index then head -n 3 example.csv</b>
</pre>
<pre class="pre-non-highlight">
color shape flag index quantity rate
purple square false 91 72.3735 8.2430
yellow circle true 87 63.5058 8.3350
yellow circle true 73 63.9785 4.2370
</pre>
As another convenience, you can put the filename first using ``--from``. When you're interacting with your data at the command line, this makes it easier to up-arrow and append to the previous command:
As another convenience, you can put the filename first using `--from`. When you're interacting with your data at the command line, this makes it easier to up-arrow and append to the previous command:
<pre>
<pre class="pre-highlight">
<b>mlr --icsv --opprint --from example.csv sort -nr index then head -n 3</b>
</pre>
<pre class="pre-non-highlight">
color shape flag index quantity rate
purple square false 91 72.3735 8.2430
yellow circle true 87 63.5058 8.3350
yellow circle true 73 63.9785 4.2370
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --icsv --opprint --from example.csv \</b>
<b> sort -nr index \</b>
<b> then head -n 3 \</b>
<b> then cut -f shape,quantity</b>
</pre>
<pre class="pre-non-highlight">
shape quantity
square 72.3735
circle 63.5058
@ -368,22 +421,26 @@ circle 63.9785
## Sorts and stats
Now suppose you want to sort the data on a given column, *and then* take the top few in that ordering. You can use Miller's ``then`` feature to pipe commands together.
Now suppose you want to sort the data on a given column, *and then* take the top few in that ordering. You can use Miller's `then` feature to pipe commands together.
Here are the records with the top three ``index`` values:
Here are the records with the top three `index` values:
<pre>
<pre class="pre-highlight">
<b>mlr --icsv --opprint sort -nr index then head -n 3 example.csv</b>
</pre>
<pre class="pre-non-highlight">
color shape flag index quantity rate
purple square false 91 72.3735 8.2430
yellow circle true 87 63.5058 8.3350
yellow circle true 73 63.9785 4.2370
</pre>
Lots of Miller commands take a ``-g`` option for group-by: here, ``head -n 1 -g shape`` outputs the first record for each distinct value of the ``shape`` field. This means we're finding the record with highest ``index`` field for each distinct ``shape`` field:
Lots of Miller commands take a `-g` option for group-by: here, `head -n 1 -g shape` outputs the first record for each distinct value of the `shape` field. This means we're finding the record with highest `index` field for each distinct `shape` field:
<pre>
<pre class="pre-highlight">
<b>mlr --icsv --opprint sort -f shape -nr index then head -n 1 -g shape example.csv</b>
</pre>
<pre class="pre-non-highlight">
color shape flag index quantity rate
yellow circle true 87 63.5058 8.3350
purple square false 91 72.3735 8.2430
@ -392,18 +449,22 @@ purple triangle false 65 80.1405 5.8240
Statistics can be computed with or without group-by field(s):
<pre>
<pre class="pre-highlight">
<b>mlr --icsv --opprint --from example.csv \</b>
<b> stats1 -a count,min,mean,max -f quantity -g shape</b>
</pre>
<pre class="pre-non-highlight">
shape quantity_count quantity_min quantity_mean quantity_max
triangle 3 43.6498 68.33976666666666 81.229
square 4 72.3735 76.60114999999999 79.2778
circle 3 13.8103 47.0982 63.9785
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --icsv --opprint --from example.csv \</b>
<b> stats1 -a count,min,mean,max -f quantity -g shape,color</b>
</pre>
<pre class="pre-non-highlight">
shape color quantity_count quantity_min quantity_mean quantity_max
triangle yellow 1 43.6498 43.6498 43.6498
square red 3 77.1991 78.01036666666666 79.2778
@ -415,9 +476,11 @@ square purple 1 72.3735 72.3735 72.3735
If your output has a lot of columns, you can use XTAB format to line things up vertically for you instead:
<pre>
<pre class="pre-highlight">
<b>mlr --icsv --oxtab --from example.csv \</b>
<b> stats1 -a p0,p10,p25,p50,p75,p90,p99,p100 -f rate</b>
</pre>
<pre class="pre-non-highlight">
rate_p0 0.0130
rate_p10 2.9010
rate_p25 4.2370
@ -438,14 +501,14 @@ Miller supports the following formats:
* JSON (JavaScript Object Notation)
* PPRINT (pretty-printed tabular)
* XTAB (vertical-tabular or sideways-tabular)
* NIDX (numerically indexed, label-free, with implicit labels ``"1"``, ``"2"``, etc.)
* NIDX (numerically indexed, label-free, with implicit labels `"1"`, `"2"`, etc.)
* DKVP (delimited key-value pairs).
What's a CSV file, really? It's an array of rows, or *records*, each being a list of key-value pairs, or *fields*: for CSV it so happens that all the keys are shared in the header line and the values vary from one data line to another.
For example, if you have:
<pre>
<pre class="pre-non-highlight">
shape,flag,index
circle,1,24
square,0,36
@ -453,14 +516,14 @@ square,0,36
then that's a way of saying:
<pre>
<pre class="pre-non-highlight">
shape=circle,flag=1,index=24
shape=square,flag=0,index=36
</pre>
Other ways to write the same data:
<pre>
<pre class="pre-non-highlight">
CSV PPRINT
shape,flag,index shape flag index
circle,1,24 circle 1 24
@ -487,111 +550,106 @@ Anything we can do with CSV input data, we can do with any other format input da
How to specify these to Miller:
* If you use ``--csv`` or ``--json`` or ``--pprint``, etc., then Miller will use that format for input and output.
* If you use ``--icsv`` and ``--ojson`` (note the extra ``i`` and ``o``) then Miller will use CSV for input and JSON for output, etc. See also [Keystroke Savers](keystroke-savers.md) for even shorter options like ``--c2j``.
* If you use `--csv` or `--json` or `--pprint`, etc., then Miller will use that format for input and output.
* If you use `--icsv` and `--ojson` (note the extra `i` and `o`) then Miller will use CSV for input and JSON for output, etc. See also [Keystroke Savers](keystroke-savers.md) for even shorter options like `--c2j`.
You can read more about this at the [File Formats](file-formats.md) page.
.. _10min-choices-for-printing-to-files:
## Choices for printing to files
Often we want to print output to the screen. Miller does this by default, as we've seen in the previous examples.
Sometimes, though, we want to print output to another file. Just use **> outputfilenamegoeshere** at the end of your command:
.. code-block:: none
:emphasize-lines: 1,1
mlr --icsv --opprint cat example.csv > newfile.csv
# Output goes to the new file;
# nothing is printed to the screen.
.. code-block:: none
:emphasize-lines: 1,1
cat newfile.csv
color shape flag index quantity rate
yellow triangle true 11 43.6498 9.8870
red square true 15 79.2778 0.0130
red circle true 16 13.8103 2.9010
red square false 48 77.5542 7.4670
purple triangle false 51 81.2290 8.5910
red square false 64 77.1991 9.5310
purple triangle false 65 80.1405 5.8240
yellow circle true 73 63.9785 4.2370
yellow circle true 87 63.5058 8.3350
purple square false 91 72.3735 8.2430
Other times we just want our files to be **changed in-place**: just use **mlr -I**:
.. code-block:: none
:emphasize-lines: 1,1
cp example.csv newfile.txt
.. code-block:: none
:emphasize-lines: 1,1
cat newfile.txt
color,shape,flag,index,quantity,rate
yellow,triangle,true,11,43.6498,9.8870
red,square,true,15,79.2778,0.0130
red,circle,true,16,13.8103,2.9010
red,square,false,48,77.5542,7.4670
purple,triangle,false,51,81.2290,8.5910
red,square,false,64,77.1991,9.5310
purple,triangle,false,65,80.1405,5.8240
yellow,circle,true,73,63.9785,4.2370
yellow,circle,true,87,63.5058,8.3350
purple,square,false,91,72.3735,8.2430
.. code-block:: none
:emphasize-lines: 1,1
mlr -I --csv sort -f shape newfile.txt
.. code-block:: none
:emphasize-lines: 1,1
cat newfile.txt
color,shape,flag,index,quantity,rate
red,circle,true,16,13.8103,2.9010
yellow,circle,true,73,63.9785,4.2370
yellow,circle,true,87,63.5058,8.3350
red,square,true,15,79.2778,0.0130
red,square,false,48,77.5542,7.4670
red,square,false,64,77.1991,9.5310
purple,square,false,91,72.3735,8.2430
yellow,triangle,true,11,43.6498,9.8870
purple,triangle,false,51,81.2290,8.5910
purple,triangle,false,65,80.1405,5.8240
Also using ``mlr -I`` you can bulk-operate on lots of files: e.g.:
.. code-block:: none
:emphasize-lines: 1,1
mlr -I --csv cut -x -f unwanted_column_name *.csv
If you like, you can first copy off your original data somewhere else, before doing in-place operations.
Lastly, using ``tee`` within ``put``, you can split your input data into separate files per one or more field names:
<pre>
<b>mlr --csv --from example.csv put -q 'tee > $shape.".csv", $*'</b>
<b>mlr --icsv --opprint cat example.csv > newfile.csv</b>
# Output goes to the new file;
# nothing is printed to the screen.
</pre>
<pre>
<b>cat newfile.csv</b>
color shape flag index quantity rate
yellow triangle true 11 43.6498 9.8870
red square true 15 79.2778 0.0130
red circle true 16 13.8103 2.9010
red square false 48 77.5542 7.4670
purple triangle false 51 81.2290 8.5910
red square false 64 77.1991 9.5310
purple triangle false 65 80.1405 5.8240
yellow circle true 73 63.9785 4.2370
yellow circle true 87 63.5058 8.3350
purple square false 91 72.3735 8.2430
</pre>
Other times we just want our files to be **changed in-place**: just use **mlr -I**:
<pre>
<b>cp example.csv newfile.txt</b>
</pre>
<pre>
<b>cat newfile.txt</b>
color,shape,flag,index,quantity,rate
yellow,triangle,true,11,43.6498,9.8870
red,square,true,15,79.2778,0.0130
red,circle,true,16,13.8103,2.9010
red,square,false,48,77.5542,7.4670
purple,triangle,false,51,81.2290,8.5910
red,square,false,64,77.1991,9.5310
purple,triangle,false,65,80.1405,5.8240
yellow,circle,true,73,63.9785,4.2370
yellow,circle,true,87,63.5058,8.3350
purple,square,false,91,72.3735,8.2430
</pre>
<pre>
<b>mlr -I --csv sort -f shape newfile.txt</b>
</pre>
<pre>
<b>cat newfile.txt</b>
color,shape,flag,index,quantity,rate
red,circle,true,16,13.8103,2.9010
yellow,circle,true,73,63.9785,4.2370
yellow,circle,true,87,63.5058,8.3350
red,square,true,15,79.2778,0.0130
red,square,false,48,77.5542,7.4670
red,square,false,64,77.1991,9.5310
purple,square,false,91,72.3735,8.2430
yellow,triangle,true,11,43.6498,9.8870
purple,triangle,false,51,81.2290,8.5910
purple,triangle,false,65,80.1405,5.8240
</pre>
Also using `mlr -I` you can bulk-operate on lots of files: e.g.:
<pre>
<b>mlr -I --csv cut -x -f unwanted_column_name *.csv</b>
</pre>
If you like, you can first copy off your original data somewhere else, before doing in-place operations.
Lastly, using `tee` within `put`, you can split your input data into separate files per one or more field names:
<pre class="pre-highlight">
<b>mlr --csv --from example.csv put -q 'tee > $shape.".csv", $*'</b>
</pre>
<pre class="pre-highlight">
<b>cat circle.csv</b>
</pre>
<pre class="pre-non-highlight">
color,shape,flag,index,quantity,rate
red,circle,true,16,13.8103,2.9010
yellow,circle,true,73,63.9785,4.2370
yellow,circle,true,87,63.5058,8.3350
</pre>
<pre>
<pre class="pre-highlight">
<b>cat square.csv</b>
</pre>
<pre class="pre-non-highlight">
color,shape,flag,index,quantity,rate
red,square,true,15,79.2778,0.0130
red,square,false,48,77.5542,7.4670
@ -599,8 +657,10 @@ red,square,false,64,77.1991,9.5310
purple,square,false,91,72.3735,8.2430
</pre>
<pre>
<pre class="pre-highlight">
<b>cat triangle.csv</b>
</pre>
<pre class="pre-non-highlight">
color,shape,flag,index,quantity,rate
yellow,triangle,true,11,43.6498,9.8870
purple,triangle,false,51,81.2290,8.5910

174
docs6b/docs/10min.md.in
View file

@ -4,13 +4,14 @@
You can install Miller for various platforms as follows:
* Linux: ``yum install miller`` or ``apt-get install miller`` depending on your flavor of Linux
* MacOS: ``brew install miller`` or ``port install miller`` depending on your preference of [Homebrew](https://brew.sh>`_ or `MacPorts <https://macports.org).
* Windows: ``choco install miller`` using [Chocolatey](https://chocolatey.org).
* You can get latest builds for Linux, MacOS, and Windows by visiting https://github.com/johnkerl/miller/actions, selecting the latest build, and clicking _Artifacts_. (These are retained for 5 days after each commit.)
* Linux: `yum install miller` or `apt-get install miller` depending on your flavor of Linux
* MacOS: `brew install miller` or `port install miller` depending on your preference of [Homebrew](https://brew.sh) or [MacPorts](https://macports.org).
* Windows: `choco install miller` using [Chocolatey](https://chocolatey.org).
* You can get latest builds for Linux, MacOS, and Windows by visiting [https://github.com/johnkerl/miller/actions](https://github.com/johnkerl/miller/actions).
, selecting the latest build, and clicking _Artifacts_. (These are retained for 5 days after each commit.)
* See also the [build page](build.md) if you prefer -- in particular, if your platform's package manager doesn't have the latest release.
As a first check, you should be able to run ``mlr --version`` at your system's command prompt and see something like the following:
As a first check, you should be able to run `mlr --version` at your system's command prompt and see something like the following:
GENMD_RUN_COMMAND
mlr --version
@ -32,19 +33,19 @@ If you run into issues on these checks, please check out the resources on the [c
Let's take a quick look at some of the most useful Miller verbs -- file-format-aware, name-index-empowered equivalents of standard system commands.
``mlr cat`` is like system ``cat`` (or ``type`` on Windows) -- it passes the data through unmodified:
`mlr cat` is like system `cat` (or `type` on Windows) -- it passes the data through unmodified:
GENMD_RUN_COMMAND
mlr --csv cat example.csv
GENMD_EOF
But ``mlr cat`` can also do format conversion -- for example, you can pretty-print in tabular format:
But `mlr cat` can also do format conversion -- for example, you can pretty-print in tabular format:
GENMD_RUN_COMMAND
mlr --icsv --opprint cat example.csv
GENMD_EOF
``mlr head`` and ``mlr tail`` count records rather than lines. Whether you're getting the first few records or the last few, the CSV header is included either way:
`mlr head` and `mlr tail` count records rather than lines. Whether you're getting the first few records or the last few, the CSV header is included either way:
GENMD_RUN_COMMAND
mlr --csv head -n 4 example.csv
@ -70,25 +71,25 @@ GENMD_RUN_COMMAND
mlr --icsv --opprint sort -f shape -nr index example.csv
GENMD_EOF
If there are fields you don't want to see in your data, you can use ``cut`` to keep only the ones you want, in the same order they appeared in the input data:
If there are fields you don't want to see in your data, you can use `cut` to keep only the ones you want, in the same order they appeared in the input data:
GENMD_RUN_COMMAND
mlr --icsv --opprint cut -f flag,shape example.csv
GENMD_EOF
You can also use ``cut -o`` to keep specified fields, but in your preferred order:
You can also use `cut -o` to keep specified fields, but in your preferred order:
GENMD_RUN_COMMAND
mlr --icsv --opprint cut -o -f flag,shape example.csv
GENMD_EOF
You can use ``cut -x`` to omit fields you don't care about:
You can use `cut -x` to omit fields you don't care about:
GENMD_RUN_COMMAND
mlr --icsv --opprint cut -x -f flag,shape example.csv
GENMD_EOF
You can use ``filter`` to keep only records you care about:
You can use `filter` to keep only records you care about:
GENMD_RUN_COMMAND
mlr --icsv --opprint filter '$color == "red"' example.csv
@ -98,7 +99,7 @@ GENMD_RUN_COMMAND
mlr --icsv --opprint filter '$color == "red" && $flag == true' example.csv
GENMD_EOF
You can use ``put`` to create new fields which are computed from other fields:
You can use `put` to create new fields which are computed from other fields:
GENMD_RUN_COMMAND
mlr --icsv --opprint put '
@ -107,7 +108,7 @@ mlr --icsv --opprint put '
' example.csv
GENMD_EOF
Even though Miller's main selling point is name-indexing, sometimes you really want to refer to a field name by its positional index. Use ``$[[3]]`` to access the name of field 3 or ``$[[[3]]]`` to access the value of field 3:
Even though Miller's main selling point is name-indexing, sometimes you really want to refer to a field name by its positional index. Use `$[[3]]` to access the name of field 3 or `$[[[3]]]` to access the value of field 3:
GENMD_RUN_COMMAND
mlr --icsv --opprint put '$[[3]] = "NEW"' example.csv
@ -121,7 +122,7 @@ You can find the full list of verbs at the [Verbs Reference](reference-verbs.md)
## Multiple input files
Miller takes all the files from the command line as an input stream. But it's format-aware, so it doesn't repeat CSV header lines. For example, with input files [data/a.csv](data/a.csv and [data/b.csv](data/b.csv), the system ``cat`` command will repeat header lines:
Miller takes all the files from the command line as an input stream. But it's format-aware, so it doesn't repeat CSV header lines. For example, with input files [data/a.csv](data/a.csv and [data/b.csv](data/b.csv), the system `cat` command will repeat header lines:
GENMD_RUN_COMMAND
cat data/a.csv
@ -135,7 +136,7 @@ GENMD_RUN_COMMAND
cat data/a.csv data/b.csv
GENMD_EOF
However, ``mlr cat`` will not:
However, `mlr cat` will not:
GENMD_RUN_COMMAND
mlr --csv cat data/a.csv data/b.csv
@ -149,13 +150,13 @@ GENMD_RUN_COMMAND
mlr --csv sort -nr index example.csv | mlr --icsv --opprint head -n 3
GENMD_EOF
This works fine -- but Miller also lets you chain verbs together using the word ``then``. Think of this as a Miller-internal pipe that lets you use fewer keystrokes:
This works fine -- but Miller also lets you chain verbs together using the word `then`. Think of this as a Miller-internal pipe that lets you use fewer keystrokes:
GENMD_RUN_COMMAND
mlr --icsv --opprint sort -nr index then head -n 3 example.csv
GENMD_EOF
As another convenience, you can put the filename first using ``--from``. When you're interacting with your data at the command line, this makes it easier to up-arrow and append to the previous command:
As another convenience, you can put the filename first using `--from`. When you're interacting with your data at the command line, this makes it easier to up-arrow and append to the previous command:
GENMD_RUN_COMMAND
mlr --icsv --opprint --from example.csv sort -nr index then head -n 3
@ -170,15 +171,15 @@ GENMD_EOF
## Sorts and stats
Now suppose you want to sort the data on a given column, *and then* take the top few in that ordering. You can use Miller's ``then`` feature to pipe commands together.
Now suppose you want to sort the data on a given column, *and then* take the top few in that ordering. You can use Miller's `then` feature to pipe commands together.
Here are the records with the top three ``index`` values:
Here are the records with the top three `index` values:
GENMD_RUN_COMMAND
mlr --icsv --opprint sort -nr index then head -n 3 example.csv
GENMD_EOF
Lots of Miller commands take a ``-g`` option for group-by: here, ``head -n 1 -g shape`` outputs the first record for each distinct value of the ``shape`` field. This means we're finding the record with highest ``index`` field for each distinct ``shape`` field:
Lots of Miller commands take a `-g` option for group-by: here, `head -n 1 -g shape` outputs the first record for each distinct value of the `shape` field. This means we're finding the record with highest `index` field for each distinct `shape` field:
GENMD_RUN_COMMAND
mlr --icsv --opprint sort -f shape -nr index then head -n 1 -g shape example.csv
@ -213,7 +214,7 @@ Miller supports the following formats:
* JSON (JavaScript Object Notation)
* PPRINT (pretty-printed tabular)
* XTAB (vertical-tabular or sideways-tabular)
* NIDX (numerically indexed, label-free, with implicit labels ``"1"``, ``"2"``, etc.)
* NIDX (numerically indexed, label-free, with implicit labels `"1"`, `"2"`, etc.)
* DKVP (delimited key-value pairs).
What's a CSV file, really? It's an array of rows, or *records*, each being a list of key-value pairs, or *fields*: for CSV it so happens that all the keys are shared in the header line and the values vary from one data line to another.
@ -262,96 +263,87 @@ Anything we can do with CSV input data, we can do with any other format input da
How to specify these to Miller:
* If you use ``--csv`` or ``--json`` or ``--pprint``, etc., then Miller will use that format for input and output.
* If you use ``--icsv`` and ``--ojson`` (note the extra ``i`` and ``o``) then Miller will use CSV for input and JSON for output, etc. See also [Keystroke Savers](keystroke-savers.md) for even shorter options like ``--c2j``.
* If you use `--csv` or `--json` or `--pprint`, etc., then Miller will use that format for input and output.
* If you use `--icsv` and `--ojson` (note the extra `i` and `o`) then Miller will use CSV for input and JSON for output, etc. See also [Keystroke Savers](keystroke-savers.md) for even shorter options like `--c2j`.
You can read more about this at the [File Formats](file-formats.md) page.
.. _10min-choices-for-printing-to-files:
## Choices for printing to files
Often we want to print output to the screen. Miller does this by default, as we've seen in the previous examples.
Sometimes, though, we want to print output to another file. Just use **> outputfilenamegoeshere** at the end of your command:
.. code-block:: none
:emphasize-lines: 1,1
<pre>
<b>mlr --icsv --opprint cat example.csv > newfile.csv</b>
# Output goes to the new file;
# nothing is printed to the screen.
</pre>
mlr --icsv --opprint cat example.csv > newfile.csv
# Output goes to the new file;
# nothing is printed to the screen.
.. code-block:: none
:emphasize-lines: 1,1
cat newfile.csv
color shape flag index quantity rate
yellow triangle true 11 43.6498 9.8870
red square true 15 79.2778 0.0130
red circle true 16 13.8103 2.9010
red square false 48 77.5542 7.4670
purple triangle false 51 81.2290 8.5910
red square false 64 77.1991 9.5310
purple triangle false 65 80.1405 5.8240
yellow circle true 73 63.9785 4.2370
yellow circle true 87 63.5058 8.3350
purple square false 91 72.3735 8.2430
<pre>
<b>cat newfile.csv</b>
color shape flag index quantity rate
yellow triangle true 11 43.6498 9.8870
red square true 15 79.2778 0.0130
red circle true 16 13.8103 2.9010
red square false 48 77.5542 7.4670
purple triangle false 51 81.2290 8.5910
red square false 64 77.1991 9.5310
purple triangle false 65 80.1405 5.8240
yellow circle true 73 63.9785 4.2370
yellow circle true 87 63.5058 8.3350
purple square false 91 72.3735 8.2430
</pre>
Other times we just want our files to be **changed in-place**: just use **mlr -I**:
.. code-block:: none
:emphasize-lines: 1,1
<pre>
<b>cp example.csv newfile.txt</b>
</pre>
cp example.csv newfile.txt
<pre>
<b>cat newfile.txt</b>
color,shape,flag,index,quantity,rate
yellow,triangle,true,11,43.6498,9.8870
red,square,true,15,79.2778,0.0130
red,circle,true,16,13.8103,2.9010
red,square,false,48,77.5542,7.4670
purple,triangle,false,51,81.2290,8.5910
red,square,false,64,77.1991,9.5310
purple,triangle,false,65,80.1405,5.8240
yellow,circle,true,73,63.9785,4.2370
yellow,circle,true,87,63.5058,8.3350
purple,square,false,91,72.3735,8.2430
</pre>
.. code-block:: none
:emphasize-lines: 1,1
<pre>
<b>mlr -I --csv sort -f shape newfile.txt</b>
</pre>
cat newfile.txt
color,shape,flag,index,quantity,rate
yellow,triangle,true,11,43.6498,9.8870
red,square,true,15,79.2778,0.0130
red,circle,true,16,13.8103,2.9010
red,square,false,48,77.5542,7.4670
purple,triangle,false,51,81.2290,8.5910
red,square,false,64,77.1991,9.5310
purple,triangle,false,65,80.1405,5.8240
yellow,circle,true,73,63.9785,4.2370
yellow,circle,true,87,63.5058,8.3350
purple,square,false,91,72.3735,8.2430
<pre>
<b>cat newfile.txt</b>
color,shape,flag,index,quantity,rate
red,circle,true,16,13.8103,2.9010
yellow,circle,true,73,63.9785,4.2370
yellow,circle,true,87,63.5058,8.3350
red,square,true,15,79.2778,0.0130
red,square,false,48,77.5542,7.4670
red,square,false,64,77.1991,9.5310
purple,square,false,91,72.3735,8.2430
yellow,triangle,true,11,43.6498,9.8870
purple,triangle,false,51,81.2290,8.5910
purple,triangle,false,65,80.1405,5.8240
</pre>
.. code-block:: none
:emphasize-lines: 1,1
Also using `mlr -I` you can bulk-operate on lots of files: e.g.:
mlr -I --csv sort -f shape newfile.txt
.. code-block:: none
:emphasize-lines: 1,1
cat newfile.txt
color,shape,flag,index,quantity,rate
red,circle,true,16,13.8103,2.9010
yellow,circle,true,73,63.9785,4.2370
yellow,circle,true,87,63.5058,8.3350
red,square,true,15,79.2778,0.0130
red,square,false,48,77.5542,7.4670
red,square,false,64,77.1991,9.5310
purple,square,false,91,72.3735,8.2430
yellow,triangle,true,11,43.6498,9.8870
purple,triangle,false,51,81.2290,8.5910
purple,triangle,false,65,80.1405,5.8240
Also using ``mlr -I`` you can bulk-operate on lots of files: e.g.:
.. code-block:: none
:emphasize-lines: 1,1
mlr -I --csv cut -x -f unwanted_column_name *.csv
<pre>
<b>mlr -I --csv cut -x -f unwanted_column_name *.csv</b>
</pre>
If you like, you can first copy off your original data somewhere else, before doing in-place operations.
Lastly, using ``tee`` within ``put``, you can split your input data into separate files per one or more field names:
Lastly, using `tee` within `put`, you can split your input data into separate files per one or more field names:
GENMD_RUN_COMMAND
mlr --csv --from example.csv put -q 'tee > $shape.".csv", $*'

40
docs6b/docs/README.md
View file

@ -1,40 +0,0 @@
# Miller Sphinx docs
## Why use Sphinx
* Connects to https://miller.readthedocs.io so people can get their docmods onto the web instead of the self-hosted https://johnkerl.org/miller/doc. Thanks to @pabloab for the great advice!
* More standard look and feel -- lots of people use readthedocs for other things so this should feel familiar
* We get a Search feature for free
## Contributing
* You need `pip install sphinx` (or `pip3 install sphinx`)
* The docs include lots of live code examples which will be invoked using `mlr` which must be somewhere in your `$PATH`
* Clone https://github.com/johnkerl/miller and cd into `docs/` within your clone
* Editing loop:
* Edit `*.md.in`
* Run `make html`
* Either `open _build/html/index.html` (MacOS) or point your browser to `file:///path/to/your/clone/of/miller/docs/_build/html/index.html`
* Submitting:
* `git add` your modified files, `git commit`, `git push`, and submit a PR at https://github.com/johnkerl/miller
* A nice markup reference: https://www.sphinx-doc.org/en/1.8/usage/restructuredtext/basics.html
## Notes
* CSS:
* I used the Sphinx Classic theme which I like a lot except the colors -- it's a blue scheme and Miller has never been blue.
* Files are in `docs/_static/*.css` where I marked my mods with `/* CHANGE ME */`.
* If you modify the CSS you must run `make clean html` (not just `make html`) then reload in your browser.
* Live code:
* I didn't find a way to include non-Python live-code examples within Sphinx so I adapted the pre-Sphinx Miller-doc strategy which is to have a generator script read a template file (here, `foo.md.in`), run the marked lines, and generate the output file (`foo.md`).
* Edit the `*.md.in` files, not `*.md` directly.
* Within the `*.md.in` files are lines like `GENMD_RUN_COMMAND`. These will be run, and their output included, by `make html` which calls the `genmds` script for you.
* readthedocs:
* https://readthedocs.org/
* https://readthedocs.org/projects/miller/
* https://readthedocs.org/projects/miller/builds/
* https://miller.readthedocs.io/en/latest/
## To do
* Let's all discuss if/how we want the v2 docs to be structured better than the v1 docs.

67
docs6b/docs/build.md
View file

@ -5,40 +5,40 @@ Please also see [Installation](installation.md) for information about pre-built
## Miller license
Two-clause BSD license https://github.com/johnkerl/miller/blob/master/LICENSE.txt.
Two-clause BSD license [https://github.com/johnkerl/miller/blob/master/LICENSE.txt](https://github.com/johnkerl/miller/blob/master/LICENSE.txt).
## From release tarball
* Obtain ``mlr-i.j.k.tar.gz`` from https://github.com/johnkerl/miller/tags, replacing ``i.j.k`` with the desired release, e.g. ``6.1.0``.
* ``tar zxvf mlr-i.j.k.tar.gz``
* ``cd mlr-i.j.k``
* ``cd go``
* ``./build`` creates the ``go/mlr`` executable and runs regression tests
* ``go build mlr.go`` creates the ``go/mlr`` executable without running regression tests
* Obtain `mlr-i.j.k.tar.gz` from [https://github.com/johnkerl/miller/tags](https://github.com/johnkerl/miller/tags), replacing `i.j.k` with the desired release, e.g. `6.1.0`.
* `tar zxvf mlr-i.j.k.tar.gz`
* `cd mlr-i.j.k`
* `cd go`
* `./build` creates the `go/mlr` executable and runs regression tests
* `go build mlr.go` creates the `go/mlr` executable without running regression tests
## From git clone
* ``git clone https://github.com/johnkerl/miller``
* ``cd miller/go``
* ``./build`` creates the ``go/mlr`` executable and runs regression tests
* ``go build mlr.go`` creates the ``go/mlr`` executable without running regression tests
* `git clone https://github.com/johnkerl/miller`
* `cd miller/go`
* `./build` creates the `go/mlr` executable and runs regression tests
* `go build mlr.go` creates the `go/mlr` executable without running regression tests
## In case of problems
If you have any build errors, feel free to open an issue with "New Issue" at https://github.com/johnkerl/miller/issues.
If you have any build errors, feel free to open an issue with "New Issue" at [https://github.com/johnkerl/miller/issues](https://github.com/johnkerl/miller/issues).
## Dependencies
### Required external dependencies
These are necessary to produce the ``mlr`` executable.
These are necessary to produce the `mlr` executable.
* Go version 1.16 or higher
* Others packaged within ``go.mod`` and ``go.sum`` which you don't need to deal with manually -- the Go build process handles them for us
* Others packaged within `go.mod` and `go.sum` which you don't need to deal with manually -- the Go build process handles them for us
### Optional external dependencies
This documentation pageset is built using Sphinx. Please see https://github.com/johnkerl/miller/blob/main/docs6/README.md for details.
This documentation pageset is built using Sphinx. Please see [https://github.com/johnkerl/miller/blob/main/docs6/README.md](https://github.com/johnkerl/miller/blob/main/docs6/README.md) for details.
## Creating a new release: for developers
@ -46,40 +46,40 @@ At present I'm the primary developer so this is just my checklist for making new
In this example I am using version 6.1.0 to 6.2.0; of course that will change for subsequent revisions.
* Update version found in ``mlr --version`` and ``man mlr``:
* Update version found in `mlr --version` and `man mlr`:
* Edit ``go/src/version/version.go`` from ``6.1.0-dev`` to ``6.2.0``.
* Likewise ``docs6/conf.py``
* ``cd ../docs6``
* ``export PATH=../go:$PATH``
* ``make html``
* The ordering is important: the first build creates ``mlr``; the second runs ``mlr`` to create ``manpage.txt``; the third includes ``manpage.txt`` into one of its outputs.
* Edit `go/src/version/version.go` from `6.1.0-dev` to `6.2.0`.
* Likewise `docs6/conf.py`
* `cd ../docs6`
* `export PATH=../go:$PATH`
* `make html`
* The ordering is important: the first build creates `mlr`; the second runs `mlr` to create `manpage.txt`; the third includes `manpage.txt` into one of its outputs.
* Commit and push.
* Create the release tarball and SRPM:
* TBD for the Go port ...
* Linux/MacOS/Windows binaries from GitHub Actions ...
* Pull back release tarball ``mlr-6.2.0.tar.gz`` from buildbox, and ``mlr.{arch}`` binaries from whatever buildboxes.
* Pull back release tarball `mlr-6.2.0.tar.gz` from buildbox, and `mlr.{arch}` binaries from whatever buildboxes.
* Create the Github release tag:
* Don't forget the ``v`` in ``v6.2.0``
* Don't forget the `v` in `v6.2.0`
* Write the release notes
* Attach the release tarball and binaries. Double-check assets were successfully uploaded.
* Publish the release
* Check the release-specific docs:
* Look at https://miller.readthedocs.io for new-version docs, after a few minutes' propagation time.
* Look at [https://miller.readthedocs.io](https://miller.readthedocs.io) for new-version docs, after a few minutes' propagation time.
* Notify:
* Submit ``brew`` pull request; notify any other distros which don't appear to have autoupdated since the previous release (notes below)
* Similarly for ``macports``: https://github.com/macports/macports-ports/blob/master/textproc/miller/Portfile.
* Submit `brew` pull request; notify any other distros which don't appear to have autoupdated since the previous release (notes below)
* Similarly for `macports`: [https://github.com/macports/macports-ports/blob/master/textproc/miller/Portfile](https://github.com/macports/macports-ports/blob/master/textproc/miller/Portfile)
* Social-media updates.
<pre>
<pre class="pre-non-highlight">
git remote add upstream https://github.com/Homebrew/homebrew-core # one-time setup only
git fetch upstream
git rebase upstream/master
@ -88,8 +88,9 @@ shasum -a 256 /path/to/mlr-6.1.0.tar.gz
edit Formula/miller.rb
# Test the URL from the line like
# url "https://github.com/johnkerl/miller/releases/download/v6.1.0/mlr-6.1.0.tar.gz"
# in a browser for typos
# A '@BrewTestBot Test this please' comment within the homebrew-core pull request will restart the homebrew travis build
# in a browser for typos.
# A '@BrewTestBot Test this please' comment within the homebrew-core pull request
# will restart the homebrew travis build.
git add Formula/miller.rb
git commit -m 'miller 6.1.0'
git push -u origin miller-6.1.0
@ -98,9 +99,9 @@ git push -u origin miller-6.1.0
* Afterwork:
* Edit ``go/src/version/version.go`` and ``docs6/conf.py`` to change version from ``6.2.0`` to ``6.2.0-dev``.
* ``cd go``
* ``./build``
* Edit `go/src/version/version.go` and `docs6/conf.py` to change version from `6.2.0` to `6.2.0-dev`.
* `cd go`
* `./build`
* Commit and push.
## Misc. development notes

65
docs6b/docs/build.md.in
View file

@ -4,40 +4,40 @@ Please also see [Installation](installation.md) for information about pre-built
## Miller license
Two-clause BSD license https://github.com/johnkerl/miller/blob/master/LICENSE.txt.
Two-clause BSD license [https://github.com/johnkerl/miller/blob/master/LICENSE.txt](https://github.com/johnkerl/miller/blob/master/LICENSE.txt).
## From release tarball
* Obtain ``mlr-i.j.k.tar.gz`` from https://github.com/johnkerl/miller/tags, replacing ``i.j.k`` with the desired release, e.g. ``6.1.0``.
* ``tar zxvf mlr-i.j.k.tar.gz``
* ``cd mlr-i.j.k``
* ``cd go``
* ``./build`` creates the ``go/mlr`` executable and runs regression tests
* ``go build mlr.go`` creates the ``go/mlr`` executable without running regression tests
* Obtain `mlr-i.j.k.tar.gz` from [https://github.com/johnkerl/miller/tags](https://github.com/johnkerl/miller/tags), replacing `i.j.k` with the desired release, e.g. `6.1.0`.
* `tar zxvf mlr-i.j.k.tar.gz`
* `cd mlr-i.j.k`
* `cd go`
* `./build` creates the `go/mlr` executable and runs regression tests
* `go build mlr.go` creates the `go/mlr` executable without running regression tests
## From git clone
* ``git clone https://github.com/johnkerl/miller``
* ``cd miller/go``
* ``./build`` creates the ``go/mlr`` executable and runs regression tests
* ``go build mlr.go`` creates the ``go/mlr`` executable without running regression tests
* `git clone https://github.com/johnkerl/miller`
* `cd miller/go`
* `./build` creates the `go/mlr` executable and runs regression tests
* `go build mlr.go` creates the `go/mlr` executable without running regression tests
## In case of problems
If you have any build errors, feel free to open an issue with "New Issue" at https://github.com/johnkerl/miller/issues.
If you have any build errors, feel free to open an issue with "New Issue" at [https://github.com/johnkerl/miller/issues](https://github.com/johnkerl/miller/issues).
## Dependencies
### Required external dependencies
These are necessary to produce the ``mlr`` executable.
These are necessary to produce the `mlr` executable.
* Go version 1.16 or higher
* Others packaged within ``go.mod`` and ``go.sum`` which you don't need to deal with manually -- the Go build process handles them for us
* Others packaged within `go.mod` and `go.sum` which you don't need to deal with manually -- the Go build process handles them for us
### Optional external dependencies
This documentation pageset is built using Sphinx. Please see https://github.com/johnkerl/miller/blob/main/docs6/README.md for details.
This documentation pageset is built using Sphinx. Please see [https://github.com/johnkerl/miller/blob/main/docs6/README.md](https://github.com/johnkerl/miller/blob/main/docs6/README.md) for details.
## Creating a new release: for developers
@ -45,37 +45,37 @@ At present I'm the primary developer so this is just my checklist for making new
In this example I am using version 6.1.0 to 6.2.0; of course that will change for subsequent revisions.
* Update version found in ``mlr --version`` and ``man mlr``:
* Update version found in `mlr --version` and `man mlr`:
* Edit ``go/src/version/version.go`` from ``6.1.0-dev`` to ``6.2.0``.
* Likewise ``docs6/conf.py``
* ``cd ../docs6``
* ``export PATH=../go:$PATH``
* ``make html``
* The ordering is important: the first build creates ``mlr``; the second runs ``mlr`` to create ``manpage.txt``; the third includes ``manpage.txt`` into one of its outputs.
* Edit `go/src/version/version.go` from `6.1.0-dev` to `6.2.0`.
* Likewise `docs6/conf.py`
* `cd ../docs6`
* `export PATH=../go:$PATH`
* `make html`
* The ordering is important: the first build creates `mlr`; the second runs `mlr` to create `manpage.txt`; the third includes `manpage.txt` into one of its outputs.
* Commit and push.
* Create the release tarball and SRPM:
* TBD for the Go port ...
* Linux/MacOS/Windows binaries from GitHub Actions ...
* Pull back release tarball ``mlr-6.2.0.tar.gz`` from buildbox, and ``mlr.{arch}`` binaries from whatever buildboxes.
* Pull back release tarball `mlr-6.2.0.tar.gz` from buildbox, and `mlr.{arch}` binaries from whatever buildboxes.
* Create the Github release tag:
* Don't forget the ``v`` in ``v6.2.0``
* Don't forget the `v` in `v6.2.0`
* Write the release notes
* Attach the release tarball and binaries. Double-check assets were successfully uploaded.
* Publish the release
* Check the release-specific docs:
* Look at https://miller.readthedocs.io for new-version docs, after a few minutes' propagation time.
* Look at [https://miller.readthedocs.io](https://miller.readthedocs.io) for new-version docs, after a few minutes' propagation time.
* Notify:
* Submit ``brew`` pull request; notify any other distros which don't appear to have autoupdated since the previous release (notes below)
* Similarly for ``macports``: https://github.com/macports/macports-ports/blob/master/textproc/miller/Portfile.
* Submit `brew` pull request; notify any other distros which don't appear to have autoupdated since the previous release (notes below)
* Similarly for `macports`: [https://github.com/macports/macports-ports/blob/master/textproc/miller/Portfile](https://github.com/macports/macports-ports/blob/master/textproc/miller/Portfile)
* Social-media updates.
GENMD_CARDIFY
@ -87,8 +87,9 @@ shasum -a 256 /path/to/mlr-6.1.0.tar.gz
edit Formula/miller.rb
# Test the URL from the line like
# url "https://github.com/johnkerl/miller/releases/download/v6.1.0/mlr-6.1.0.tar.gz"
# in a browser for typos
# A '@BrewTestBot Test this please' comment within the homebrew-core pull request will restart the homebrew travis build
# in a browser for typos.
# A '@BrewTestBot Test this please' comment within the homebrew-core pull request
# will restart the homebrew travis build.
git add Formula/miller.rb
git commit -m 'miller 6.1.0'
git push -u origin miller-6.1.0
@ -97,9 +98,9 @@ GENMD_EOF
* Afterwork:
* Edit ``go/src/version/version.go`` and ``docs6/conf.py`` to change version from ``6.2.0`` to ``6.2.0-dev``.
* ``cd go``
* ``./build``
* Edit `go/src/version/version.go` and `docs6/conf.py` to change version from `6.2.0` to `6.2.0-dev`.
* `cd go`
* `./build`
* Commit and push.
## Misc. development notes

6
docs6b/docs/community.md
View file

@ -2,6 +2,6 @@
# Community
* See [Miller GitHub Discussions](https://github.com/johnkerl/miller/discussions) for general Q&A, advice, sharing success stories, etc.
* See also [Miller-tagged questions on Stack Overflow](https://stackoverflow.com/questions/tagged/miller?tab=Newest)
* See [Miller GitHub Issues](https://github.com/johnkerl/miller/issues) for bug reports and feature requests
* Other correspondence: [mailto:kerl.john.r+miller@gmail.com](mailto:kerl.john.r+miller@gmail.com)
* See also [Miller-tagged questions on Stack Overflow](https://stackoverflow.com/questions/tagged/miller?tab=Newest).
* See [Miller GitHub Issues](https://github.com/johnkerl/miller/issues) for bug reports and feature requests.
* Other correspondence: [mailto:kerl.john.r+miller@gmail.com](mailto:kerl.john.r+miller@gmail.com).

6
docs6b/docs/community.md.in
View file

@ -1,6 +1,6 @@
# Community
* See [Miller GitHub Discussions](https://github.com/johnkerl/miller/discussions) for general Q&A, advice, sharing success stories, etc.
* See also [Miller-tagged questions on Stack Overflow](https://stackoverflow.com/questions/tagged/miller?tab=Newest)
* See [Miller GitHub Issues](https://github.com/johnkerl/miller/issues) for bug reports and feature requests
* Other correspondence: [mailto:kerl.john.r+miller@gmail.com](mailto:kerl.john.r+miller@gmail.com)
* See also [Miller-tagged questions on Stack Overflow](https://stackoverflow.com/questions/tagged/miller?tab=Newest).
* See [Miller GitHub Issues](https://github.com/johnkerl/miller/issues) for bug reports and feature requests.
* Other correspondence: [mailto:kerl.john.r+miller@gmail.com](mailto:kerl.john.r+miller@gmail.com).

8
docs6b/docs/contributing.md
View file

@ -11,15 +11,15 @@ Pre-release Miller documentation is at [https://github.com/johnkerl/miller/tree/
Clone [https://github.com/johnkerl/miller](https://github.com/johnkerl/miller) and `cd` into `docs6`.
After ``sudo pip install sphinx`` (or ``pip3``) you should be able to do ``make html``.
After `sudo pip install sphinx` (or `pip3`) you should be able to do `make html`.
Edit ``*.md.in`` files, then ``make html`` to generate ``*.md``, then run the Sphinx document-generator.
Edit `*.md.in` files, then `make html` to generate `*.md`, then run the Sphinx document-generator.
Open ``_build/html/index.html`` in your browser, e.g. ``file:////Users/yourname/git/miller/docs6/_build/html/contributing.html``, to verify.
Open `_build/html/index.html` in your browser, e.g. `file:////Users/yourname/git/miller/docs6/_build/html/contributing.html`, to verify.
PRs are welcome at [https://github.com/johnkerl/miller](https://github.com/johnkerl/miller).
Once PRs are merged, readthedocs creates https://miller.readthedocs.io using the following configs:
Once PRs are merged, readthedocs creates [https://miller.readthedocs.io](https://miller.readthedocs.io) using the following configs:
* [https://readthedocs.org/projects/miller](https://readthedocs.org/projects/miller)
* [https://readthedocs.org/projects/miller/builds](https://readthedocs.org/projects/miller/builds)

8
docs6b/docs/contributing.md.in
View file

@ -10,15 +10,15 @@ Pre-release Miller documentation is at [https://github.com/johnkerl/miller/tree/
Clone [https://github.com/johnkerl/miller](https://github.com/johnkerl/miller) and `cd` into `docs6`.
After ``sudo pip install sphinx`` (or ``pip3``) you should be able to do ``make html``.
After `sudo pip install sphinx` (or `pip3`) you should be able to do `make html`.
Edit ``*.md.in`` files, then ``make html`` to generate ``*.md``, then run the Sphinx document-generator.
Edit `*.md.in` files, then `make html` to generate `*.md`, then run the Sphinx document-generator.
Open ``_build/html/index.html`` in your browser, e.g. ``file:////Users/yourname/git/miller/docs6/_build/html/contributing.html``, to verify.
Open `_build/html/index.html` in your browser, e.g. `file:////Users/yourname/git/miller/docs6/_build/html/contributing.html`, to verify.
PRs are welcome at [https://github.com/johnkerl/miller](https://github.com/johnkerl/miller).
Once PRs are merged, readthedocs creates https://miller.readthedocs.io using the following configs:
Once PRs are merged, readthedocs creates [https://miller.readthedocs.io](https://miller.readthedocs.io) using the following configs:
* [https://readthedocs.org/projects/miller](https://readthedocs.org/projects/miller)
* [https://readthedocs.org/projects/miller/builds](https://readthedocs.org/projects/miller/builds)

54
docs6b/docs/csv-with-and-without-headers.md
View file

@ -3,20 +3,24 @@
## Headerless CSV on input or output
Sometimes we get CSV files which lack a header. For example (`data/headerless.csv <./data/headerless.csv>`_):
Sometimes we get CSV files which lack a header. For example, [data/headerless.csv](./data/headerless.csv):
<pre>
<pre class="pre-highlight">
<b>cat data/headerless.csv</b>
</pre>
<pre class="pre-non-highlight">
John,23,present
Fred,34,present
Alice,56,missing
Carol,45,present
</pre>
You can use Miller to add a header. The ``--implicit-csv-header`` applies positionally indexed labels:
You can use Miller to add a header. The `--implicit-csv-header` applies positionally indexed labels:
<pre>
<pre class="pre-highlight">
<b>mlr --csv --implicit-csv-header cat data/headerless.csv</b>
</pre>
<pre class="pre-non-highlight">
1,2,3
John,23,present
Fred,34,present
@ -26,8 +30,10 @@ Carol,45,present
Following that, you can rename the positionally indexed labels to names with meaning for your context. For example:
<pre>
<pre class="pre-highlight">
<b>mlr --csv --implicit-csv-header label name,age,status data/headerless.csv</b>
</pre>
<pre class="pre-non-highlight">
name,age,status
John,23,present
Fred,34,present
@ -35,10 +41,12 @@ Alice,56,missing
Carol,45,present
</pre>
Likewise, if you need to produce CSV which is lacking its header, you can pipe Miller's output to the system command ``sed 1d``, or you can use Miller's ``--headerless-csv-output`` option:
Likewise, if you need to produce CSV which is lacking its header, you can pipe Miller's output to the system command `sed 1d`, or you can use Miller's `--headerless-csv-output` option:
<pre>
<pre class="pre-highlight">
<b>head -5 data/colored-shapes.dkvp | mlr --ocsv cat</b>
</pre>
<pre class="pre-non-highlight">
color,shape,flag,i,u,v,w,x
yellow,triangle,1,11,0.6321695890307647,0.9887207810889004,0.4364983936735774,5.7981881667050565
red,square,1,15,0.21966833570651523,0.001257332190235938,0.7927778364718627,2.944117399716207
@ -47,8 +55,10 @@ red,square,0,48,0.9562743938458542,0.7467203085342884,0.7755423050923582,7.11783
purple,triangle,0,51,0.4355354501763202,0.8591292672156728,0.8122903963006748,5.753094629505863
</pre>
<pre>
<pre class="pre-highlight">
<b>head -5 data/colored-shapes.dkvp | mlr --ocsv --headerless-csv-output cat</b>
</pre>
<pre class="pre-non-highlight">
yellow,triangle,1,11,0.6321695890307647,0.9887207810889004,0.4364983936735774,5.7981881667050565
red,square,1,15,0.21966833570651523,0.001257332190235938,0.7927778364718627,2.944117399716207
red,circle,1,16,0.20901671281497636,0.29005231936593445,0.13810280912907674,5.065034003400998
@ -58,8 +68,10 @@ purple,triangle,0,51,0.4355354501763202,0.8591292672156728,0.8122903963006748,5.
Lastly, often we say "CSV" or "TSV" when we have positionally indexed data in columns which are separated by commas or tabs, respectively. In this case it's perhaps simpler to **just use NIDX format** which was designed for this purpose. (See also [File Formats](file-formats.md).) For example:
<pre>
<pre class="pre-highlight">
<b>mlr --inidx --ifs comma --oxtab cut -f 1,3 data/headerless.csv</b>
</pre>
<pre class="pre-non-highlight">
1 John
3 present
@ -75,18 +87,20 @@ Lastly, often we say "CSV" or "TSV" when we have positionally indexed data in co
## Headerless CSV with duplicate field values
Miller is (by central design) a mapping from name to value, rather than integer position to value as in most tools in the Unix toolkit such as ``sort``, ``cut``, ``awk``, etc. So given input ``Yea=1,Yea=2`` on the same input line, first ``Yea=1`` is stored, then updated with ``Yea=2``. This is in the input-parser and the value ``Yea=1`` is unavailable to any further processing. The following example line comes from a headerless CSV file and includes 5 times the string (value) ``'NA'``:
Miller is (by central design) a mapping from name to value, rather than integer position to value as in most tools in the Unix toolkit such as `sort`, `cut`, `awk`, etc. So given input `Yea=1,Yea=2` on the same input line, first `Yea=1` is stored, then updated with `Yea=2`. This is in the input-parser and the value `Yea=1` is unavailable to any further processing. The following example line comes from a headerless CSV file and includes 5 times the string (value) `'NA'`:
<pre>
<pre class="pre-highlight">
<b>ag '0.9' nas.csv | head -1</b>
</pre>
<pre class="pre-non-highlight">
2:-349801.10097848,4537221.43295653,2,1,NA,NA,NA,NA,NA
</pre>
The repeated ``'NA'`` strings (values) in the same line will be treated as fields (columns) with same name, thus only one is kept in the output.
The repeated `'NA'` strings (values) in the same line will be treated as fields (columns) with same name, thus only one is kept in the output.
This can be worked around by telling ``mlr`` that there is no header row by using ``--implicit-csv-header`` or changing the input format by using ``nidx`` like so:
This can be worked around by telling `mlr` that there is no header row by using `--implicit-csv-header` or changing the input format by using `nidx` like so:
<pre>
<pre class="pre-non-highlight">
ag '0.9' nas.csv | mlr --n2c --fs "," label xsn,ysn,x,y,t,a,e29,e31,e32 then head
</pre>
@ -94,15 +108,17 @@ ag '0.9' nas.csv | mlr --n2c --fs "," label xsn,ysn,x,y,t,a,e29,e31,e32 then hea
Miller handles compliant CSV: in particular, it's an error if the number of data fields in a given data line don't match the number of header lines. But in the event that you have a CSV file in which some lines have less than the full number of fields, you can use Miller to pad them out. The trick is to use NIDX format, for which each line stands on its own without respect to a header line.
<pre>
<pre class="pre-highlight">
<b>cat data/ragged.csv</b>
</pre>
<pre class="pre-non-highlight">
a,b,c
1,2,3
4,5
6,7,8,9
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --from data/ragged.csv --fs comma --nidx put '</b>
<b> @maxnf = max(@maxnf, NF);</b>
<b> @nf = NF;</b>
@ -111,6 +127,8 @@ a,b,c
<b> $[@nf] = ""</b>
<b> }</b>
<b>'</b>
</pre>
<pre class="pre-non-highlight">
a,b,c
1,2,3
4,5
@ -119,13 +137,15 @@ a,b,c
or, more simply,
<pre>
<pre class="pre-highlight">
<b>mlr --from data/ragged.csv --fs comma --nidx put '</b>
<b> @maxnf = max(@maxnf, NF);</b>
<b> while(NF < @maxnf) {</b>
<b> $[NF+1] = "";</b>
<b> }</b>
<b>'</b>
</pre>
<pre class="pre-non-highlight">
a,b,c
1,2,3
4,5

12
docs6b/docs/csv-with-and-without-headers.md.in
View file

@ -2,13 +2,13 @@
## Headerless CSV on input or output
Sometimes we get CSV files which lack a header. For example (`data/headerless.csv <./data/headerless.csv>`_):
Sometimes we get CSV files which lack a header. For example, [data/headerless.csv](./data/headerless.csv):
GENMD_RUN_COMMAND
cat data/headerless.csv
GENMD_EOF
You can use Miller to add a header. The ``--implicit-csv-header`` applies positionally indexed labels:
You can use Miller to add a header. The `--implicit-csv-header` applies positionally indexed labels:
GENMD_RUN_COMMAND
mlr --csv --implicit-csv-header cat data/headerless.csv
@ -20,7 +20,7 @@ GENMD_RUN_COMMAND
mlr --csv --implicit-csv-header label name,age,status data/headerless.csv
GENMD_EOF
Likewise, if you need to produce CSV which is lacking its header, you can pipe Miller's output to the system command ``sed 1d``, or you can use Miller's ``--headerless-csv-output`` option:
Likewise, if you need to produce CSV which is lacking its header, you can pipe Miller's output to the system command `sed 1d`, or you can use Miller's `--headerless-csv-output` option:
GENMD_RUN_COMMAND
head -5 data/colored-shapes.dkvp | mlr --ocsv cat
@ -38,16 +38,16 @@ GENMD_EOF
## Headerless CSV with duplicate field values
Miller is (by central design) a mapping from name to value, rather than integer position to value as in most tools in the Unix toolkit such as ``sort``, ``cut``, ``awk``, etc. So given input ``Yea=1,Yea=2`` on the same input line, first ``Yea=1`` is stored, then updated with ``Yea=2``. This is in the input-parser and the value ``Yea=1`` is unavailable to any further processing. The following example line comes from a headerless CSV file and includes 5 times the string (value) ``'NA'``:
Miller is (by central design) a mapping from name to value, rather than integer position to value as in most tools in the Unix toolkit such as `sort`, `cut`, `awk`, etc. So given input `Yea=1,Yea=2` on the same input line, first `Yea=1` is stored, then updated with `Yea=2`. This is in the input-parser and the value `Yea=1` is unavailable to any further processing. The following example line comes from a headerless CSV file and includes 5 times the string (value) `'NA'`:
GENMD_CARDIFY_HIGHLIGHT_ONE
ag '0.9' nas.csv | head -1
2:-349801.10097848,4537221.43295653,2,1,NA,NA,NA,NA,NA
GENMD_EOF
The repeated ``'NA'`` strings (values) in the same line will be treated as fields (columns) with same name, thus only one is kept in the output.
The repeated `'NA'` strings (values) in the same line will be treated as fields (columns) with same name, thus only one is kept in the output.
This can be worked around by telling ``mlr`` that there is no header row by using ``--implicit-csv-header`` or changing the input format by using ``nidx`` like so:
This can be worked around by telling `mlr` that there is no header row by using `--implicit-csv-header` or changing the input format by using `nidx` like so:
GENMD_CARDIFY
ag '0.9' nas.csv | mlr --n2c --fs "," label xsn,ysn,x,y,t,a,e29,e31,e32 then head

46
docs6b/docs/customization.md
View file

@ -3,51 +3,51 @@
## How to use .mlrrc
Suppose you always use CSV files. Then instead of always having to type ``--csv`` as in
Suppose you always use CSV files. Then instead of always having to type `--csv` as in
<pre>
<pre class="pre-highlight">
<b>mlr --csv cut -x -f extra mydata.csv</b>
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --csv sort -n id mydata.csv</b>
</pre>
and so on, you can instead put the following into your ``$HOME/.mlrrc``:
and so on, you can instead put the following into your `$HOME/.mlrrc`:
<pre>
<pre class="pre-non-highlight">
--csv
</pre>
Then you can just type things like
<pre>
<pre class="pre-highlight">
<b>mlr cut -x -f extra mydata.csv</b>
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr sort -n id mydata.csv</b>
</pre>
and the ``--csv`` part will automatically be understood. (If you do want to process, say, a JSON file then ``mlr --json ...`` at the command line will override the default from your ``.mlrrc``.)
and the `--csv` part will automatically be understood. (If you do want to process, say, a JSON file then `mlr --json ...` at the command line will override the default from your `.mlrrc`.)
## What you can put in your .mlrrc
* You can include any command-line flags, except the "terminal" ones such as ``--help``.
* You can include any command-line flags, except the "terminal" ones such as `--help`.
* The ``--prepipe``, ``--load``, and ``--mload`` flags aren't allowed in ``.mlrrc`` as they control code execution, and could result in your scripts running things you don't expect if you receive data from someone with a ``.mlrrc`` in it.
* The `--prepipe`, `--load`, and `--mload` flags aren't allowed in `.mlrrc` as they control code execution, and could result in your scripts running things you don't expect if you receive data from someone with a `.mlrrc` in it.
* The formatting rule is you need to put one flag beginning with ``--`` per line: for example, ``--csv`` on one line and ``--nr-progress-mod 1000`` on a separate line.
* The formatting rule is you need to put one flag beginning with `--` per line: for example, `--csv` on one line and `--nr-progress-mod 1000` on a separate line.
* Since every line starts with a ``--`` option, you can leave off the initial ``--`` if you want. For example, ``ojson`` is the same as ``--ojson``, and ``nr-progress-mod 1000`` is the same as ``--nr-progress-mod 1000``.
* Since every line starts with a `--` option, you can leave off the initial `--` if you want. For example, `ojson` is the same as `--ojson`, and `nr-progress-mod 1000` is the same as `--nr-progress-mod 1000`.
* Comments are from a ``#`` to the end of the line.
* Comments are from a `#` to the end of the line.
* Empty lines are ignored -- including lines which are empty after comments are removed.
Here is an example ``.mlrrc`` file:
Here is an example `.mlrrc` file:
<pre>
<pre class="pre-non-highlight">
# Input and output formats are CSV by default (unless otherwise specified
# on the mlr command line):
csv
@ -67,20 +67,20 @@ skip-comments-with @
## Where to put your .mlrrc
If the environment variable ``MLRRC`` is set:
If the environment variable `MLRRC` is set:
* If its value is ``__none__`` then no ``.mlrrc`` files are processed. (This is nice for things like regression testing.)
* If its value is `__none__` then no `.mlrrc` files are processed. (This is nice for things like regression testing.)
* Otherwise, its value (as a filename) is loaded and processed. If there are syntax errors, they abort ``mlr`` with a usage message (as if you had mistyped something on the command line). If the file can't be loaded at all, though, it is silently skipped.
* Otherwise, its value (as a filename) is loaded and processed. If there are syntax errors, they abort `mlr` with a usage message (as if you had mistyped something on the command line). If the file can't be loaded at all, though, it is silently skipped.
* Any ``.mlrrc`` in your home directory or current directory is ignored whenever ``MLRRC`` is set in the environment.
* Any `.mlrrc` in your home directory or current directory is ignored whenever `MLRRC` is set in the environment.
* Example line in your shell's rc file: ``export MLRRC=/path/to/my/mlrrc``
* Example line in your shell's rc file: `export MLRRC=/path/to/my/mlrrc`
Otherwise:
* If ``$HOME/.mlrrc`` exists, it's processed as above.
* If `$HOME/.mlrrc` exists, it's processed as above.
* If ``./.mlrrc`` exists, it's then also processed as above.
* If `./.mlrrc` exists, it's then also processed as above.
* The idea is you can have all your settings in your ``$HOME/.mlrrc``, then override maybe one or two for your current directory if you like.
* The idea is you can have all your settings in your `$HOME/.mlrrc`, then override maybe one or two for your current directory if you like.

34
docs6b/docs/customization.md.in
View file

@ -2,7 +2,7 @@
## How to use .mlrrc
Suppose you always use CSV files. Then instead of always having to type ``--csv`` as in
Suppose you always use CSV files. Then instead of always having to type `--csv` as in
GENMD_CARDIFY_HIGHLIGHT_ONE
mlr --csv cut -x -f extra mydata.csv
@ -12,7 +12,7 @@ GENMD_CARDIFY_HIGHLIGHT_ONE
mlr --csv sort -n id mydata.csv
GENMD_EOF
and so on, you can instead put the following into your ``$HOME/.mlrrc``:
and so on, you can instead put the following into your `$HOME/.mlrrc`:
GENMD_CARDIFY
--csv
@ -28,42 +28,42 @@ GENMD_CARDIFY_HIGHLIGHT_ONE
mlr sort -n id mydata.csv
GENMD_EOF
and the ``--csv`` part will automatically be understood. (If you do want to process, say, a JSON file then ``mlr --json ...`` at the command line will override the default from your ``.mlrrc``.)
and the `--csv` part will automatically be understood. (If you do want to process, say, a JSON file then `mlr --json ...` at the command line will override the default from your `.mlrrc`.)
## What you can put in your .mlrrc
* You can include any command-line flags, except the "terminal" ones such as ``--help``.
* You can include any command-line flags, except the "terminal" ones such as `--help`.
* The ``--prepipe``, ``--load``, and ``--mload`` flags aren't allowed in ``.mlrrc`` as they control code execution, and could result in your scripts running things you don't expect if you receive data from someone with a ``.mlrrc`` in it.
* The `--prepipe`, `--load`, and `--mload` flags aren't allowed in `.mlrrc` as they control code execution, and could result in your scripts running things you don't expect if you receive data from someone with a `.mlrrc` in it.
* The formatting rule is you need to put one flag beginning with ``--`` per line: for example, ``--csv`` on one line and ``--nr-progress-mod 1000`` on a separate line.
* The formatting rule is you need to put one flag beginning with `--` per line: for example, `--csv` on one line and `--nr-progress-mod 1000` on a separate line.
* Since every line starts with a ``--`` option, you can leave off the initial ``--`` if you want. For example, ``ojson`` is the same as ``--ojson``, and ``nr-progress-mod 1000`` is the same as ``--nr-progress-mod 1000``.
* Since every line starts with a `--` option, you can leave off the initial `--` if you want. For example, `ojson` is the same as `--ojson`, and `nr-progress-mod 1000` is the same as `--nr-progress-mod 1000`.
* Comments are from a ``#`` to the end of the line.
* Comments are from a `#` to the end of the line.
* Empty lines are ignored -- including lines which are empty after comments are removed.
Here is an example ``.mlrrc`` file:
Here is an example `.mlrrc` file:
GENMD_INCLUDE_ESCAPED(sample_mlrrc)
## Where to put your .mlrrc
If the environment variable ``MLRRC`` is set:
If the environment variable `MLRRC` is set:
* If its value is ``__none__`` then no ``.mlrrc`` files are processed. (This is nice for things like regression testing.)
* If its value is `__none__` then no `.mlrrc` files are processed. (This is nice for things like regression testing.)
* Otherwise, its value (as a filename) is loaded and processed. If there are syntax errors, they abort ``mlr`` with a usage message (as if you had mistyped something on the command line). If the file can't be loaded at all, though, it is silently skipped.
* Otherwise, its value (as a filename) is loaded and processed. If there are syntax errors, they abort `mlr` with a usage message (as if you had mistyped something on the command line). If the file can't be loaded at all, though, it is silently skipped.
* Any ``.mlrrc`` in your home directory or current directory is ignored whenever ``MLRRC`` is set in the environment.
* Any `.mlrrc` in your home directory or current directory is ignored whenever `MLRRC` is set in the environment.
* Example line in your shell's rc file: ``export MLRRC=/path/to/my/mlrrc``
* Example line in your shell's rc file: `export MLRRC=/path/to/my/mlrrc`
Otherwise:
* If ``$HOME/.mlrrc`` exists, it's processed as above.
* If `$HOME/.mlrrc` exists, it's processed as above.
* If ``./.mlrrc`` exists, it's then also processed as above.
* If `./.mlrrc` exists, it's then also processed as above.
* The idea is you can have all your settings in your ``$HOME/.mlrrc``, then override maybe one or two for your current directory if you like.
* The idea is you can have all your settings in your `$HOME/.mlrrc`, then override maybe one or two for your current directory if you like.

26
docs6b/docs/data-cleaning-examples.md
View file

@ -1,10 +1,12 @@
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
# Data-cleaning examples
Here are some ways to use the type-checking options as described in :ref:`reference-dsl-type-tests-and-assertions` Suppose you have the following data file, with inconsistent typing for boolean. (Also imagine that, for the sake of discussion, we have a million-line file rather than a four-line file, so we can't see it all at once and some automation is called for.)
Here are some ways to use the type-checking options as described in [Type-checking](reference-dsl-variables.md#type-checking). Suppose you have the following data file, with inconsistent typing for boolean. (Also imagine that, for the sake of discussion, we have a million-line file rather than a four-line file, so we can't see it all at once and some automation is called for.)
<pre>
<pre class="pre-highlight">
<b>cat data/het-bool.csv</b>
</pre>
<pre class="pre-non-highlight">
name,reachable
barney,false
betty,true
@ -14,8 +16,10 @@ wilma,1
One option is to coerce everything to boolean, or integer:
<pre>
<pre class="pre-highlight">
<b>mlr --icsv --opprint put '$reachable = boolean($reachable)' data/het-bool.csv</b>
</pre>
<pre class="pre-non-highlight">
name reachable
barney false
betty true
@ -23,8 +27,10 @@ fred true
wilma true
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --icsv --opprint put '$reachable = int(boolean($reachable))' data/het-bool.csv</b>
</pre>
<pre class="pre-non-highlight">
name reachable
barney 0
betty 1
@ -34,8 +40,10 @@ wilma 1
A second option is to flag badly formatted data within the output stream:
<pre>
<pre class="pre-highlight">
<b>mlr --icsv --opprint put '$format_ok = is_string($reachable)' data/het-bool.csv</b>
</pre>
<pre class="pre-non-highlight">
name reachable format_ok
barney false false
betty true false
@ -45,10 +53,12 @@ wilma 1 false
Or perhaps to flag badly formatted data outside the output stream:
<pre>
<pre class="pre-highlight">
<b>mlr --icsv --opprint put '</b>
<b> if (!is_string($reachable)) {eprint "Malformed at NR=".NR}</b>
<b>' data/het-bool.csv</b>
</pre>
<pre class="pre-non-highlight">
Malformed at NR=1
Malformed at NR=2
Malformed at NR=3
@ -62,7 +72,9 @@ wilma 1
A third way is to abort the process on fimd.instance of bad data:
<pre>
<pre class="pre-highlight">
<b>mlr --csv put '$reachable = asserting_string($reachable)' data/het-bool.csv</b>
</pre>
<pre class="pre-non-highlight">
Miller: is_string type-assertion failed at NR=1 FNR=1 FILENAME=data/het-bool.csv
</pre>

2
docs6b/docs/data-cleaning-examples.md.in
View file

@ -1,6 +1,6 @@
# Data-cleaning examples
Here are some ways to use the type-checking options as described in :ref:`reference-dsl-type-tests-and-assertions` Suppose you have the following data file, with inconsistent typing for boolean. (Also imagine that, for the sake of discussion, we have a million-line file rather than a four-line file, so we can't see it all at once and some automation is called for.)
Here are some ways to use the type-checking options as described in [Type-checking](reference-dsl-variables.md#type-checking). Suppose you have the following data file, with inconsistent typing for boolean. (Also imagine that, for the sake of discussion, we have a million-line file rather than a four-line file, so we can't see it all at once and some automation is called for.)
GENMD_RUN_COMMAND
cat data/het-bool.csv

78
docs6b/docs/data-diving-examples.md
View file

@ -7,8 +7,10 @@ The [flins.csv](data/flins.csv) file is some sample data obtained from [https://
Vertical-tabular format is good for a quick look at CSV data layout -- seeing what columns you have to work with:
<pre>
<pre class="pre-highlight">
<b>head -n 2 data/flins.csv | mlr --icsv --oxtab cat</b>
</pre>
<pre class="pre-non-highlight">
county Seminole
tiv_2011 22890.55
tiv_2012 20848.71
@ -17,8 +19,10 @@ line Residential
A few simple queries:
<pre>
<pre class="pre-highlight">
<b>mlr --from data/flins.csv --icsv --opprint count-distinct -f county | head</b>
</pre>
<pre class="pre-non-highlight">
county count
Seminole 1
Miami Dade 2
@ -28,32 +32,36 @@ Duval 1
St. Johns 1
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --from data/flins.csv --icsv --opprint count-distinct -f construction,line</b>
</pre>
Categorization of total insured value:
<pre>
<pre class="pre-highlight">
<b>mlr --from data/flins.csv --icsv --opprint stats1 -a min,mean,max -f tiv_2012</b>
</pre>
<pre class="pre-non-highlight">
tiv_2012_min tiv_2012_mean tiv_2012_max
19757.91 1061531.4637499999 2785551.63
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --from data/flins.csv --icsv --opprint \</b>
<b> stats1 -a min,mean,max -f tiv_2012 -g construction,line</b>
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --from data/flins.csv --icsv --oxtab \</b>
<b> stats1 -a p0,p10,p50,p90,p95,p99,p100 -f hu_site_deductible</b>
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --from data/flins.csv --icsv --opprint \</b>
<b> stats1 -a p95,p99,p100 -f hu_site_deductible -g county \</b>
<b> then sort -f county | head</b>
</pre>
<pre class="pre-non-highlight">
county
Duval
Highlands
@ -63,9 +71,11 @@ Seminole
St. Johns
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --from data/flins.csv --icsv --oxtab \</b>
<b> stats2 -a corr,linreg-ols,r2 -f tiv_2011,tiv_2012</b>
</pre>
<pre class="pre-non-highlight">
tiv_2011_tiv_2012_corr 0.9353629581411828
tiv_2011_tiv_2012_ols_m 1.0890905877734807
tiv_2011_tiv_2012_ols_b 103095.52335638746
@ -73,9 +83,11 @@ tiv_2011_tiv_2012_ols_n 8
tiv_2011_tiv_2012_r2 0.8749038634626236
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --from data/flins.csv --icsv --opprint \</b>
<b> stats2 -a corr,linreg-ols,r2 -f tiv_2011,tiv_2012 -g county</b>
</pre>
<pre class="pre-non-highlight">
county tiv_2011_tiv_2012_corr tiv_2011_tiv_2012_ols_m tiv_2011_tiv_2012_ols_b tiv_2011_tiv_2012_ols_n tiv_2011_tiv_2012_r2
Seminole - - - 1 -
Miami Dade 1 0.9306426512386247 -2311.1543275160047 2 0.9999999999999999
@ -87,25 +99,29 @@ St. Johns - - -
## Color/shape data
The [data/colored-shapes.dkvp](data/colored-shapes.dkvp) file is some sample data produced by the [mkdat2](data/mkdat2) script. The idea is:
The [data/colored-shapes.dkvp](data/colored-shapes.dkvp) file is some sample data produced by the [mkdat2](../data/mkdat2) script. The idea is:
* Produce some data with known distributions and correlations, and verify that Miller recovers those properties empirically.
* Each record is labeled with one of a few colors and one of a few shapes.
* The ``flag`` field is 0 or 1, with probability dependent on color
* The ``u`` field is plain uniform on the unit interval.
* The ``v`` field is the same, except tightly correlated with ``u`` for red circles.
* The ``w`` field is autocorrelated for each color/shape pair.
* The ``x`` field is boring Gaussian with mean 5 and standard deviation about 1.2, with no dependence on color or shape.
* The `flag` field is 0 or 1, with probability dependent on color
* The `u` field is plain uniform on the unit interval.
* The `v` field is the same, except tightly correlated with `u` for red circles.
* The `w` field is autocorrelated for each color/shape pair.
* The `x` field is boring Gaussian with mean 5 and standard deviation about 1.2, with no dependence on color or shape.
Peek at the data:
<pre>
<pre class="pre-highlight">
<b>wc -l data/colored-shapes.dkvp</b>
</pre>
<pre class="pre-non-highlight">
10078 data/colored-shapes.dkvp
</pre>
<pre>
<pre class="pre-highlight">
<b>head -n 6 data/colored-shapes.dkvp | mlr --opprint cat</b>
</pre>
<pre class="pre-non-highlight">
color shape flag i u v w x
yellow triangle 1 11 0.6321695890307647 0.9887207810889004 0.4364983936735774 5.7981881667050565
red square 1 15 0.21966833570651523 0.001257332190235938 0.7927778364718627 2.944117399716207
@ -117,10 +133,12 @@ red square 0 64 0.2015510269821953 0.9531098083420033 0.7719912015786
Look at uncategorized stats (using [creach](https://github.com/johnkerl/scripts/blob/master/fundam/creach) for spacing).
Here it looks reasonable that ``u`` is unit-uniform; something's up with ``v`` but we can't yet see what:
Here it looks reasonable that `u` is unit-uniform; something's up with `v` but we can't yet see what:
<pre>
<pre class="pre-highlight">
<b>mlr --oxtab stats1 -a min,mean,max -f flag,u,v data/colored-shapes.dkvp | creach 3</b>
</pre>
<pre class="pre-non-highlight">
flag_min 0
flag_mean 0.39888866838658465
flag_max 1
@ -136,8 +154,10 @@ v_max 1.0724998185026013
The histogram shows the different distribution of 0/1 flags:
<pre>
<pre class="pre-highlight">
<b>mlr --opprint histogram -f flag,u,v --lo -0.1 --hi 1.1 --nbins 12 data/colored-shapes.dkvp</b>
</pre>
<pre class="pre-non-highlight">
bin_lo bin_hi flag_count u_count v_count
-0.010000000000000002 0.09000000000000002 6058 0 36
0.09000000000000002 0.19000000000000003 0 1062 988
@ -155,10 +175,12 @@ bin_lo bin_hi flag_count u_count v_count
Look at univariate stats by color and shape. In particular, color-dependent flag probabilities pop out, aligning with their original Bernoulli probablities from the data-generator script:
<pre>
<pre class="pre-highlight">
<b>mlr --opprint stats1 -a min,mean,max -f flag,u,v -g color \</b>
<b> then sort -f color \</b>
<b> data/colored-shapes.dkvp</b>
</pre>
<pre class="pre-non-highlight">
color flag_min flag_mean flag_max u_min u_mean u_max v_min v_mean v_max
blue 0 0.5843537414965987 1 0.000043912454007477564 0.517717155039078 0.9999687954968421 0.0014886830387470518 0.49105642841387653 0.9995761761685742
green 0 0.20919747520288548 1 0.00048750676198217047 0.5048610622924616 0.9999361779701204 0.0005012669003675585 0.49908475928072205 0.9996764373885353
@ -168,28 +190,34 @@ red 0 0.3031674208144796 1 0.0006711367180041172 0.492559648
yellow 0 0.8924274593064402 1 0.001300228762057487 0.49712912165196765 0.99992313390574 0.0007109695568577878 0.510626599360317 0.9999189897724752
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --opprint stats1 -a min,mean,max -f flag,u,v -g shape \</b>
<b> then sort -f shape \</b>
<b> data/colored-shapes.dkvp</b>
</pre>
<pre class="pre-non-highlight">
shape flag_min flag_mean flag_max u_min u_mean u_max v_min v_mean v_max
circle 0 0.3998456194519491 1 0.000043912454007477564 0.49855450951394115 0.99992313390574 -0.09270905318501277 0.49552415740048406 1.0724998185026013
square 0 0.39611178614823817 1 0.0001881939925673093 0.499385458061097 0.9999687954968421 0.00008930277299445954 0.49653825501903986 0.9999751864255598
triangle 0 0.4015421115065243 1 0.000881025170573424 0.4968585405884252 0.9996614910922645 0.000716883409890845 0.501049532862137 0.9999946837499262
</pre>
Look at bivariate stats by color and shape. In particular, ``u,v`` pairwise correlation for red circles pops out:
Look at bivariate stats by color and shape. In particular, `u,v` pairwise correlation for red circles pops out:
<pre>
<pre class="pre-highlight">
<b>mlr --opprint --right stats2 -a corr -f u,v,w,x data/colored-shapes.dkvp</b>
</pre>
<pre class="pre-non-highlight">
u_v_corr w_x_corr
0.13341803768384553 -0.011319938208638764
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --opprint --right \</b>
<b> stats2 -a corr -f u,v,w,x -g color,shape then sort -nr u_v_corr \</b>
<b> data/colored-shapes.dkvp</b>
</pre>
<pre class="pre-non-highlight">
color shape u_v_corr w_x_corr
red circle 0.9807984157534667 -0.018565046320623148
orange square 0.17685846147882145 -0.07104374629148885

16
docs6b/docs/data-diving-examples.md.in
View file

@ -54,15 +54,15 @@ GENMD_EOF
## Color/shape data
The [data/colored-shapes.dkvp](data/colored-shapes.dkvp) file is some sample data produced by the [mkdat2](data/mkdat2) script. The idea is:
The [data/colored-shapes.dkvp](data/colored-shapes.dkvp) file is some sample data produced by the [mkdat2](../data/mkdat2) script. The idea is:
* Produce some data with known distributions and correlations, and verify that Miller recovers those properties empirically.
* Each record is labeled with one of a few colors and one of a few shapes.
* The ``flag`` field is 0 or 1, with probability dependent on color
* The ``u`` field is plain uniform on the unit interval.
* The ``v`` field is the same, except tightly correlated with ``u`` for red circles.
* The ``w`` field is autocorrelated for each color/shape pair.
* The ``x`` field is boring Gaussian with mean 5 and standard deviation about 1.2, with no dependence on color or shape.
* The `flag` field is 0 or 1, with probability dependent on color
* The `u` field is plain uniform on the unit interval.
* The `v` field is the same, except tightly correlated with `u` for red circles.
* The `w` field is autocorrelated for each color/shape pair.
* The `x` field is boring Gaussian with mean 5 and standard deviation about 1.2, with no dependence on color or shape.
Peek at the data:
@ -76,7 +76,7 @@ GENMD_EOF
Look at uncategorized stats (using [creach](https://github.com/johnkerl/scripts/blob/master/fundam/creach) for spacing).
Here it looks reasonable that ``u`` is unit-uniform; something's up with ``v`` but we can't yet see what:
Here it looks reasonable that `u` is unit-uniform; something's up with `v` but we can't yet see what:
GENMD_RUN_COMMAND
mlr --oxtab stats1 -a min,mean,max -f flag,u,v data/colored-shapes.dkvp | creach 3
@ -102,7 +102,7 @@ mlr --opprint stats1 -a min,mean,max -f flag,u,v -g shape \
data/colored-shapes.dkvp
GENMD_EOF
Look at bivariate stats by color and shape. In particular, ``u,v`` pairwise correlation for red circles pops out:
Look at bivariate stats by color and shape. In particular, `u,v` pairwise correlation for red circles pops out:
GENMD_RUN_COMMAND
mlr --opprint --right stats2 -a corr -f u,v,w,x data/colored-shapes.dkvp

6
docs6b/docs/data/sar.mlr
View file

@ -1,3 +1,3 @@
for (k in $*) {
$[k] = gsub($[k], "e", "X");
}
for (k in $*) {
$[k] = gsub($[k], "e", "X");
}

38
docs6b/docs/dates-and-times.md
View file

@ -5,32 +5,38 @@
Given input like
<pre>
<pre class="pre-highlight">
<b>cat dates.csv</b>
</pre>
<pre class="pre-non-highlight">
date,event
2018-02-03,initialization
2018-03-07,discovery
2018-02-03,allocation
</pre>
we can use ``strptime`` to parse the date field into seconds-since-epoch and then do numeric comparisons. Simply match your input dataset's date-formatting to the :ref:`reference-dsl-strptime` format-string. For example:
we can use [strptime](reference-verbs.md#strptime) to parse the date field into seconds-since-epoch and then do numeric comparisons. Simply match your input dataset's date-formatting to the [strptime](reference-verbs.md#strptime) format-string. For example:
<pre>
<pre class="pre-highlight">
<b>mlr --csv filter '</b>
<b> strptime($date, "%Y-%m-%d") > strptime("2018-03-03", "%Y-%m-%d")</b>
<b>' dates.csv</b>
</pre>
<pre class="pre-non-highlight">
date,event
2018-03-07,discovery
</pre>
Caveat: localtime-handling in timezones with DST is still a work in progress; see https://github.com/johnkerl/miller/issues/170. See also https://github.com/johnkerl/miller/issues/208 -- thanks @aborruso!
Caveat: localtime-handling in timezones with DST is still a work in progress; see [https://github.com/johnkerl/miller/issues/170](https://github.com/johnkerl/miller/issues/170) . See also [https://github.com/johnkerl/miller/issues/208](https://github.com/johnkerl/miller/issues/208) -- thanks @aborruso!
## Finding missing dates
Suppose you have some date-stamped data which may (or may not) be missing entries for one or more dates:
<pre>
<pre class="pre-highlight">
<b>head -n 10 data/miss-date.csv</b>
</pre>
<pre class="pre-non-highlight">
date,qoh
2012-03-05,10055
2012-03-06,10486
@ -43,19 +49,23 @@ date,qoh
2012-03-13,11177
</pre>
<pre>
<pre class="pre-highlight">
<b>wc -l data/miss-date.csv</b>
</pre>
<pre class="pre-non-highlight">
1372 data/miss-date.csv
</pre>
Since there are 1372 lines in the data file, some automation is called for. To find the missing dates, you can convert the dates to seconds since the epoch using ``strptime``, then compute adjacent differences (the ``cat -n`` simply inserts record-counters):
Since there are 1372 lines in the data file, some automation is called for. To find the missing dates, you can convert the dates to seconds since the epoch using `strptime`, then compute adjacent differences (the `cat -n` simply inserts record-counters):
<pre>
<pre class="pre-highlight">
<b>mlr --from data/miss-date.csv --icsv \</b>
<b> cat -n \</b>
<b> then put '$datestamp = strptime($date, "%Y-%m-%d")' \</b>
<b> then step -a delta -f datestamp \</b>
<b>| head</b>
</pre>
<pre class="pre-non-highlight">
n=1,date=2012-03-05,qoh=10055,datestamp=1330905600,datestamp_delta=0
n=2,date=2012-03-06,qoh=10486,datestamp=1330992000,datestamp_delta=86400
n=3,date=2012-03-07,qoh=10430,datestamp=1331078400,datestamp_delta=86400
@ -70,20 +80,24 @@ n=10,date=2012-03-14,qoh=11498,datestamp=1331683200,datestamp_delta=86400
Then, filter for adjacent difference not being 86400 (the number of seconds in a day):
<pre>
<pre class="pre-highlight">
<b>mlr --from data/miss-date.csv --icsv \</b>
<b> cat -n \</b>
<b> then put '$datestamp = strptime($date, "%Y-%m-%d")' \</b>
<b> then step -a delta -f datestamp \</b>
<b> then filter '$datestamp_delta != 86400 && $n != 1'</b>
</pre>
<pre class="pre-non-highlight">
n=774,date=2014-04-19,qoh=130140,datestamp=1397865600,datestamp_delta=259200
n=1119,date=2015-03-31,qoh=181625,datestamp=1427760000,datestamp_delta=172800
</pre>
Given this, it's now easy to see where the gaps are:
<pre>
<pre class="pre-highlight">
<b>mlr cat -n then filter '$n >= 770 && $n <= 780' data/miss-date.csv</b>
</pre>
<pre class="pre-non-highlight">
n=770,1=2014-04-12,2=129435
n=771,1=2014-04-13,2=129868
n=772,1=2014-04-14,2=129797
@ -97,8 +111,10 @@ n=779,1=2014-04-23,2=130849
n=780,1=2014-04-24,2=131026
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr cat -n then filter '$n >= 1115 && $n <= 1125' data/miss-date.csv</b>
</pre>
<pre class="pre-non-highlight">
n=1115,1=2015-03-25,2=181006
n=1116,1=2015-03-26,2=180995
n=1117,1=2015-03-27,2=181043

6
docs6b/docs/dates-and-times.md.in
View file

@ -8,7 +8,7 @@ GENMD_RUN_COMMAND
cat dates.csv
GENMD_EOF
we can use ``strptime`` to parse the date field into seconds-since-epoch and then do numeric comparisons. Simply match your input dataset's date-formatting to the :ref:`reference-dsl-strptime` format-string. For example:
we can use [strptime](reference-verbs.md#strptime) to parse the date field into seconds-since-epoch and then do numeric comparisons. Simply match your input dataset's date-formatting to the [strptime](reference-verbs.md#strptime) format-string. For example:
GENMD_RUN_COMMAND
mlr --csv filter '
@ -16,7 +16,7 @@ mlr --csv filter '
' dates.csv
GENMD_EOF
Caveat: localtime-handling in timezones with DST is still a work in progress; see https://github.com/johnkerl/miller/issues/170. See also https://github.com/johnkerl/miller/issues/208 -- thanks @aborruso!
Caveat: localtime-handling in timezones with DST is still a work in progress; see [https://github.com/johnkerl/miller/issues/170](https://github.com/johnkerl/miller/issues/170) . See also [https://github.com/johnkerl/miller/issues/208](https://github.com/johnkerl/miller/issues/208) -- thanks @aborruso!
## Finding missing dates
@ -30,7 +30,7 @@ GENMD_RUN_COMMAND
wc -l data/miss-date.csv
GENMD_EOF
Since there are 1372 lines in the data file, some automation is called for. To find the missing dates, you can convert the dates to seconds since the epoch using ``strptime``, then compute adjacent differences (the ``cat -n`` simply inserts record-counters):
Since there are 1372 lines in the data file, some automation is called for. To find the missing dates, you can convert the dates to seconds since the epoch using `strptime`, then compute adjacent differences (the `cat -n` simply inserts record-counters):
GENMD_INCLUDE_AND_RUN_ESCAPED(data/miss-date-1.sh)

28
docs6b/docs/dkvp-examples.md
View file

@ -5,7 +5,7 @@
Here are the I/O routines:
<pre>
<pre class="pre-non-highlight">
#!/usr/bin/env python
# ================================================================
@ -68,8 +68,10 @@ def map2dkvpline(map , ops, ofs):
And here is an example using them:
<pre>
<pre class="pre-highlight">
<b>cat polyglot-dkvp-io/example.py</b>
</pre>
<pre class="pre-non-highlight">
#!/usr/bin/env python
import sys
@ -108,8 +110,10 @@ while True:
Run as-is:
<pre>
<pre class="pre-highlight">
<b>python polyglot-dkvp-io/example.py < data/small</b>
</pre>
<pre class="pre-non-highlight">
a=pan,b=pan,i=1,y=0.7268028627434533,ab=panpan,iy=1.7268028627434533,ta=str,tb=str,ti=int,ty=float,tab=str,tiy=float
a=eks,b=pan,i=2,y=0.5221511083334797,ab=ekspan,iy=2.5221511083334796,ta=str,tb=str,ti=int,ty=float,tab=str,tiy=float
a=wye,b=wye,i=3,y=0.33831852551664776,ab=wyewye,iy=3.3383185255166477,ta=str,tb=str,ti=int,ty=float,tab=str,tiy=float
@ -119,8 +123,10 @@ a=wye,b=pan,i=5,y=0.8636244699032729,ab=wyepan,iy=5.863624469903273,ta=str,tb=st
Run as-is, then pipe to Miller for pretty-printing:
<pre>
<pre class="pre-highlight">
<b>python polyglot-dkvp-io/example.py < data/small | mlr --opprint cat</b>
</pre>
<pre class="pre-non-highlight">
a b i y ab iy ta tb ti ty tab tiy
pan pan 1 0.7268028627434533 panpan 1.7268028627434533 str str int float str float
eks pan 2 0.5221511083334797 ekspan 2.5221511083334796 str str int float str float
@ -133,7 +139,7 @@ wye pan 5 0.8636244699032729 wyepan 5.863624469903273 str str int float str fl
Here are the I/O routines:
<pre>
<pre class="pre-non-highlight">
#!/usr/bin/env ruby
# ================================================================
@ -190,8 +196,10 @@ end
And here is an example using them:
<pre>
<pre class="pre-highlight">
<b>cat polyglot-dkvp-io/example.rb</b>
</pre>
<pre class="pre-non-highlight">
#!/usr/bin/env ruby
require 'dkvp_io'
@ -220,8 +228,10 @@ end
Run as-is:
<pre>
<pre class="pre-highlight">
<b>ruby -I./polyglot-dkvp-io polyglot-dkvp-io/example.rb data/small</b>
</pre>
<pre class="pre-non-highlight">
a=pan,b=pan,i=1,y=0.7268028627434533,ab=panpan,iy=1.7268028627434533,ta=String,tb=String,ti=Integer,ty=Float,tab=String,tiy=Float
a=eks,b=pan,i=2,y=0.5221511083334797,ab=ekspan,iy=2.5221511083334796,ta=String,tb=String,ti=Integer,ty=Float,tab=String,tiy=Float
a=wye,b=wye,i=3,y=0.33831852551664776,ab=wyewye,iy=3.3383185255166477,ta=String,tb=String,ti=Integer,ty=Float,tab=String,tiy=Float
@ -231,8 +241,10 @@ a=wye,b=pan,i=5,y=0.8636244699032729,ab=wyepan,iy=5.863624469903273,ta=String,tb
Run as-is, then pipe to Miller for pretty-printing:
<pre>
<pre class="pre-highlight">
<b>ruby -I./polyglot-dkvp-io polyglot-dkvp-io/example.rb data/small | mlr --opprint cat</b>
</pre>
<pre class="pre-non-highlight">
a b i y ab iy ta tb ti ty tab tiy
pan pan 1 0.7268028627434533 panpan 1.7268028627434533 String String Integer Float String Float
eks pan 2 0.5221511083334797 ekspan 2.5221511083334796 String String Integer Float String Float

2
docs6b/docs/etymology.md
View file

@ -1,6 +1,6 @@
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
# Why call it Miller?
The Unix toolkit was created in the **1970s** and is a mainstay to this day. Miller's look and feel adheres closely to the [classic toolkit style](http://en.wikipedia.org/wiki/Unix_philosophy): if this were music, Miller would be a **tribute album**. Likewise, since commands are subcommands of the ``mlr`` executable, the result is a **band**, if you will, of command-line tools. Put these together and the namesake is another classic product of the 1970s: the [Steve Miller Band](http://en.wikipedia.org/wiki/Steve%5fMiller%5fBand).
The Unix toolkit was created in the **1970s** and is a mainstay to this day. Miller's look and feel adheres closely to the [classic toolkit style](http://en.wikipedia.org/wiki/Unix_philosophy): if this were music, Miller would be a **tribute album**. Likewise, since commands are subcommands of the `mlr` executable, the result is a **band**, if you will, of command-line tools. Put these together and the namesake is another classic product of the 1970s: the [Steve Miller Band](http://en.wikipedia.org/wiki/Steve%5fMiller%5fBand).
(Additionally, and far more prosaically ... just as a miller is someone who grinds and mixes grain into flour to extend its usefulness, Miller grinds and mixes data for you.)

2
docs6b/docs/etymology.md.in
View file

@ -1,5 +1,5 @@
# Why call it Miller?
The Unix toolkit was created in the **1970s** and is a mainstay to this day. Miller's look and feel adheres closely to the [classic toolkit style](http://en.wikipedia.org/wiki/Unix_philosophy): if this were music, Miller would be a **tribute album**. Likewise, since commands are subcommands of the ``mlr`` executable, the result is a **band**, if you will, of command-line tools. Put these together and the namesake is another classic product of the 1970s: the [Steve Miller Band](http://en.wikipedia.org/wiki/Steve%5fMiller%5fBand).
The Unix toolkit was created in the **1970s** and is a mainstay to this day. Miller's look and feel adheres closely to the [classic toolkit style](http://en.wikipedia.org/wiki/Unix_philosophy): if this were music, Miller would be a **tribute album**. Likewise, since commands are subcommands of the `mlr` executable, the result is a **band**, if you will, of command-line tools. Put these together and the namesake is another classic product of the 1970s: the [Steve Miller Band](http://en.wikipedia.org/wiki/Steve%5fMiller%5fBand).
(Additionally, and far more prosaically ... just as a miller is someone who grinds and mixes grain into flour to extend its usefulness, Miller grinds and mixes data for you.)

55
docs6b/docs/extra.css
View file

@ -1,11 +1,17 @@
/* Top left, desktop */
.wy-menu {
background-color: #c0c0c0;
/*background-color: #404040;*/
}
/* Top left, mobile */
.wy-nav-top {
background-color: #c0c0c0;
}
/* Left navbar */
.wy-side-nav-search {
background-color: #c0c0c0;
/*background-color: #404040;*/
}
/* Code samples without <pre> (none currently in use) */
.rst-content code {
color: #000000;
background-color: #e0e0e0;
@ -18,11 +24,27 @@
color: #0FF1CE;
}
.caption-text {
/*background-color: #808080; */
color: maroon;
padding: 4px;
}
/* Code samples using <pre> (which is currently in use) */
.pre-highlight {
color: #000000;
background-color: #c5b690;
font-size: 12px;
padding: 10px;
margin-bottom: 0px;
}
.pre-non-highlight {
color: #000000;
background-color: #eae2cb;
font-size: 12px;
padding: 10px;
margin-top: 0px;
}
/* Section titles in content pages */
h1 {
color: maroon;
background-color: #e0e0e0;
@ -36,31 +58,7 @@ h3 {
background-color: #e0e0e0;
}
pre {
color: #000000;
background-color: #eae2cb;
font-size: 12px;
padding: 10px;
}
pre b {
background-color: #c5b690;
padding: 2px;
}
a {
color: orange;
}
a:visited {
color: green;
}
a:hover {
color: blue;
}
a .internal {
color: red;
}
/* Hyperlinks in content pages */
a {
color: maroon;
}
@ -72,6 +70,7 @@ a:hover {
color: maroon;
}
/* Hyperlinks in navbar */
.wy-menu-vertical li ul li a {
margin-bottom:0;
color:#000000;

40
docs6b/docs/feature-comparison.md
View file

@ -1,61 +1,69 @@
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
# Unix-toolkit context
How does Miller fit within the Unix toolkit (``grep``, ``sed``, ``awk``, etc.)?
How does Miller fit within the Unix toolkit (`grep`, `sed`, `awk`, etc.)?
## File-format awareness
Miller respects CSV headers. If you do ``mlr --csv cat *.csv`` then the header line is written once:
Miller respects CSV headers. If you do `mlr --csv cat *.csv` then the header line is written once:
<pre>
<pre class="pre-highlight">
<b>cat data/a.csv</b>
</pre>
<pre class="pre-non-highlight">
a,b,c
1,2,3
4,5,6
</pre>
<pre>
<pre class="pre-highlight">
<b>cat data/b.csv</b>
</pre>
<pre class="pre-non-highlight">
a,b,c
7,8,9
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --csv cat data/a.csv data/b.csv</b>
</pre>
<pre class="pre-non-highlight">
a,b,c
1,2,3
4,5,6
7,8,9
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --csv sort -nr b data/a.csv data/b.csv</b>
</pre>
<pre class="pre-non-highlight">
a,b,c
7,8,9
4,5,6
1,2,3
</pre>
Likewise with ``mlr sort``, ``mlr tac``, and so on.
Likewise with `mlr sort`, `mlr tac`, and so on.
## awk-like features: mlr filter and mlr put
* ``mlr filter`` includes/excludes records based on a filter expression, e.g. ``mlr filter '$count > 10'``.
* `mlr filter` includes/excludes records based on a filter expression, e.g. `mlr filter '$count > 10'`.
* ``mlr put`` adds a new field as a function of others, e.g. ``mlr put '$xy = $x * $y'`` or ``mlr put '$counter = NR'``.
* `mlr put` adds a new field as a function of others, e.g. `mlr put '$xy = $x * $y'` or `mlr put '$counter = NR'`.
* The ``$name`` syntax is straight from ``awk``'s ``$1 $2 $3`` (adapted to name-based indexing), as are the variables ``FS``, ``OFS``, ``RS``, ``ORS``, ``NF``, ``NR``, and ``FILENAME``. The ``ENV[...]`` syntax is from Ruby.
* The `$name` syntax is straight from `awk`'s `$1 $2 $3` (adapted to name-based indexing), as are the variables `FS`, `OFS`, `RS`, `ORS`, `NF`, `NR`, and `FILENAME`. The `ENV[...]` syntax is from Ruby.
* While ``awk`` functions are record-based, Miller subcommands (or *verbs*) are stream-based: each of them maps a stream of records into another stream of records.
* While `awk` functions are record-based, Miller subcommands (or *verbs*) are stream-based: each of them maps a stream of records into another stream of records.
* Like ``awk``, Miller (as of v5.0.0) allows you to define new functions within its ``put`` and ``filter`` expression language. Further programmability comes from chaining with ``then``.
* Like `awk`, Miller (as of v5.0.0) allows you to define new functions within its `put` and `filter` expression language. Further programmability comes from chaining with `then`.
* As with ``awk``, ``$``-variables are stream variables and all verbs (such as ``cut``, ``stats1``, ``put``, etc.) as well as ``put``/``filter`` statements operate on streams. This means that you define actions to be done on each record and then stream your data through those actions. The built-in variables ``NF``, ``NR``, etc. change from one line to another, ``$x`` is a label for field ``x`` in the current record, and the input to ``sqrt($x)`` changes from one record to the next. The expression language for the ``put`` and ``filter`` verbs additionally allows you to define ``begin {...}`` and ``end {...}`` blocks for actions to be taken before and after records are processed, respectively.
* As with `awk`, `$`-variables are stream variables and all verbs (such as `cut`, `stats1`, `put`, etc.) as well as `put`/`filter` statements operate on streams. This means that you define actions to be done on each record and then stream your data through those actions. The built-in variables `NF`, `NR`, etc. change from one line to another, `$x` is a label for field `x` in the current record, and the input to `sqrt($x)` changes from one record to the next. The expression language for the `put` and `filter` verbs additionally allows you to define `begin {...}` and `end {...}` blocks for actions to be taken before and after records are processed, respectively.
* As with ``awk``, Miller's ``put``/``filter`` language lets you set ``@sum=0`` before records are read, then update that sum on each record, then print its value at the end. Unlike ``awk``, Miller makes syntactically explicit the difference between variables with extent across all records (names starting with ``@``, such as ``@sum``) and variables which are local to the current expression (names starting without ``@``, such as ``sum``).
* As with `awk`, Miller's `put`/`filter` language lets you set `@sum=0` before records are read, then update that sum on each record, then print its value at the end. Unlike `awk`, Miller makes syntactically explicit the difference between variables with extent across all records (names starting with `@`, such as `@sum`) and variables which are local to the current expression (names starting without `@`, such as `sum`).
* Miller can be faster than ``awk``, ``cut``, and so on, depending on platform; see also [Performance](performance.md). In particular, Miller's DSL syntax is parsed into Go control structures at startup time, with the bulk data-stream processing all done in Go.
* Miller can be faster than `awk`, `cut`, and so on, depending on platform; see also [Performance](performance.md). In particular, Miller's DSL syntax is parsed into Go control structures at startup time, with the bulk data-stream processing all done in Go.
## See also
See [Verbs Reference](reference-verbs.md) for more on Miller's subcommands ``cat``, ``cut``, ``head``, ``sort``, ``tac``, ``tail``, ``top``, and ``uniq``, as well as [DSL reference](reference-dsl.md) for more on the awk-like ``mlr filter`` and ``mlr put``.
See [Verbs Reference](reference-verbs.md) for more on Miller's subcommands `cat`, `cut`, `head`, `sort`, `tac`, `tail`, `top`, and `uniq`, as well as [DSL reference](reference-dsl.md) for more on the awk-like `mlr filter` and `mlr put`.

24
docs6b/docs/feature-comparison.md.in
View file

@ -1,10 +1,10 @@
# Unix-toolkit context
How does Miller fit within the Unix toolkit (``grep``, ``sed``, ``awk``, etc.)?
How does Miller fit within the Unix toolkit (`grep`, `sed`, `awk`, etc.)?
## File-format awareness
Miller respects CSV headers. If you do ``mlr --csv cat *.csv`` then the header line is written once:
Miller respects CSV headers. If you do `mlr --csv cat *.csv` then the header line is written once:
GENMD_RUN_COMMAND
cat data/a.csv
@ -22,26 +22,26 @@ GENMD_RUN_COMMAND
mlr --csv sort -nr b data/a.csv data/b.csv
GENMD_EOF
Likewise with ``mlr sort``, ``mlr tac``, and so on.
Likewise with `mlr sort`, `mlr tac`, and so on.
## awk-like features: mlr filter and mlr put
* ``mlr filter`` includes/excludes records based on a filter expression, e.g. ``mlr filter '$count > 10'``.
* `mlr filter` includes/excludes records based on a filter expression, e.g. `mlr filter '$count > 10'`.
* ``mlr put`` adds a new field as a function of others, e.g. ``mlr put '$xy = $x * $y'`` or ``mlr put '$counter = NR'``.
* `mlr put` adds a new field as a function of others, e.g. `mlr put '$xy = $x * $y'` or `mlr put '$counter = NR'`.
* The ``$name`` syntax is straight from ``awk``'s ``$1 $2 $3`` (adapted to name-based indexing), as are the variables ``FS``, ``OFS``, ``RS``, ``ORS``, ``NF``, ``NR``, and ``FILENAME``. The ``ENV[...]`` syntax is from Ruby.
* The `$name` syntax is straight from `awk`'s `$1 $2 $3` (adapted to name-based indexing), as are the variables `FS`, `OFS`, `RS`, `ORS`, `NF`, `NR`, and `FILENAME`. The `ENV[...]` syntax is from Ruby.
* While ``awk`` functions are record-based, Miller subcommands (or *verbs*) are stream-based: each of them maps a stream of records into another stream of records.
* While `awk` functions are record-based, Miller subcommands (or *verbs*) are stream-based: each of them maps a stream of records into another stream of records.
* Like ``awk``, Miller (as of v5.0.0) allows you to define new functions within its ``put`` and ``filter`` expression language. Further programmability comes from chaining with ``then``.
* Like `awk`, Miller (as of v5.0.0) allows you to define new functions within its `put` and `filter` expression language. Further programmability comes from chaining with `then`.
* As with ``awk``, ``$``-variables are stream variables and all verbs (such as ``cut``, ``stats1``, ``put``, etc.) as well as ``put``/``filter`` statements operate on streams. This means that you define actions to be done on each record and then stream your data through those actions. The built-in variables ``NF``, ``NR``, etc. change from one line to another, ``$x`` is a label for field ``x`` in the current record, and the input to ``sqrt($x)`` changes from one record to the next. The expression language for the ``put`` and ``filter`` verbs additionally allows you to define ``begin {...}`` and ``end {...}`` blocks for actions to be taken before and after records are processed, respectively.
* As with `awk`, `$`-variables are stream variables and all verbs (such as `cut`, `stats1`, `put`, etc.) as well as `put`/`filter` statements operate on streams. This means that you define actions to be done on each record and then stream your data through those actions. The built-in variables `NF`, `NR`, etc. change from one line to another, `$x` is a label for field `x` in the current record, and the input to `sqrt($x)` changes from one record to the next. The expression language for the `put` and `filter` verbs additionally allows you to define `begin {...}` and `end {...}` blocks for actions to be taken before and after records are processed, respectively.
* As with ``awk``, Miller's ``put``/``filter`` language lets you set ``@sum=0`` before records are read, then update that sum on each record, then print its value at the end. Unlike ``awk``, Miller makes syntactically explicit the difference between variables with extent across all records (names starting with ``@``, such as ``@sum``) and variables which are local to the current expression (names starting without ``@``, such as ``sum``).
* As with `awk`, Miller's `put`/`filter` language lets you set `@sum=0` before records are read, then update that sum on each record, then print its value at the end. Unlike `awk`, Miller makes syntactically explicit the difference between variables with extent across all records (names starting with `@`, such as `@sum`) and variables which are local to the current expression (names starting without `@`, such as `sum`).
* Miller can be faster than ``awk``, ``cut``, and so on, depending on platform; see also [Performance](performance.md). In particular, Miller's DSL syntax is parsed into Go control structures at startup time, with the bulk data-stream processing all done in Go.
* Miller can be faster than `awk`, `cut`, and so on, depending on platform; see also [Performance](performance.md). In particular, Miller's DSL syntax is parsed into Go control structures at startup time, with the bulk data-stream processing all done in Go.
## See also
See [Verbs Reference](reference-verbs.md) for more on Miller's subcommands ``cat``, ``cut``, ``head``, ``sort``, ``tac``, ``tail``, ``top``, and ``uniq``, as well as [DSL reference](reference-dsl.md) for more on the awk-like ``mlr filter`` and ``mlr put``.
See [Verbs Reference](reference-verbs.md) for more on Miller's subcommands `cat`, `cut`, `head`, `sort`, `tac`, `tail`, `top`, and `uniq`, as well as [DSL reference](reference-dsl.md) for more on the awk-like `mlr filter` and `mlr put`.

10
docs6b/docs/features.md
View file

@ -19,11 +19,11 @@ including but not limited to the familiar CSV, TSV, and JSON. (Miller can handl
* Miller complements **data-analysis tools** such as **R**, **pandas**, etc.: you can use Miller to **clean** and **prepare** your data. While you can do **basic statistics** entirely in Miller, its streaming-data feature and single-pass algorithms enable you to **reduce very large data sets**.
* Miller complements SQL **databases**: you can slice, dice, and reformat data on the client side on its way into or out of a database. (Examples :ref:`here <sql-input-examples>` and :ref:`here <sql-output-examples>`.) You can also reap some of the benefits of databases for quick, setup-free one-off tasks when you just need to query some data in disk files in a hurry.
* Miller complements SQL **databases**: you can slice, dice, and reformat data on the client side on its way into or out of a database. (See [SQL Examples](sql-examples.md).) You can also reap some of the benefits of databases for quick, setup-free one-off tasks when you just need to query some data in disk files in a hurry.
* Miller also goes beyond the classic Unix tools by stepping fully into our modern, **no-SQL** world: its essential record-heterogeneity property allows Miller to operate on data where records with different schema (field names) are interleaved.
* Miller is **streaming**: most operations need only a single record in memory at a time, rather than ingesting all input before producing any output. For those operations which require deeper retention (``sort``, ``tac``, ``stats1``), Miller retains only as much data as needed. This means that whenever functionally possible, you can operate on files which are larger than your system's available RAM, and you can use Miller in **tail -f** contexts.
* Miller is **streaming**: most operations need only a single record in memory at a time, rather than ingesting all input before producing any output. For those operations which require deeper retention (`sort`, `tac`, `stats1`), Miller retains only as much data as needed. This means that whenever functionally possible, you can operate on files which are larger than your system's available RAM, and you can use Miller in **tail -f** contexts.
* Miller is **pipe-friendly** and interoperates with the Unix toolkit
@ -31,10 +31,10 @@ including but not limited to the familiar CSV, TSV, and JSON. (Miller can handl
* Miller does **conversion** between formats
* Miller's **processing is format-aware**: e.g. CSV ``sort`` and ``tac`` keep header lines first
* Miller's **processing is format-aware**: e.g. CSV `sort` and `tac` keep header lines first
* Miller has high-throughput **performance** on par with the Unix toolkit
* Not unlike [jq](https://stedolan.github.io/jq/) (for JSON), Miller is written in Go which is a portable, modern language, and Miller has no runtime dependencies. You can download or compile a single binary, ``scp`` it to a faraway machine, and expect it to work.
* Not unlike [jq](https://stedolan.github.io/jq/) (for JSON), Miller is written in Go which is a portable, modern language, and Miller has no runtime dependencies. You can download or compile a single binary, `scp` it to a faraway machine, and expect it to work.
Releases and release notes: https://github.com/johnkerl/miller/releases.
Releases and release notes: [https://github.com/johnkerl/miller/releases](https://github.com/johnkerl/miller/releases).

10
docs6b/docs/features.md.in
View file

@ -18,11 +18,11 @@ including but not limited to the familiar CSV, TSV, and JSON. (Miller can handl
* Miller complements **data-analysis tools** such as **R**, **pandas**, etc.: you can use Miller to **clean** and **prepare** your data. While you can do **basic statistics** entirely in Miller, its streaming-data feature and single-pass algorithms enable you to **reduce very large data sets**.
* Miller complements SQL **databases**: you can slice, dice, and reformat data on the client side on its way into or out of a database. (Examples :ref:`here <sql-input-examples>` and :ref:`here <sql-output-examples>`.) You can also reap some of the benefits of databases for quick, setup-free one-off tasks when you just need to query some data in disk files in a hurry.
* Miller complements SQL **databases**: you can slice, dice, and reformat data on the client side on its way into or out of a database. (See [SQL Examples](sql-examples.md).) You can also reap some of the benefits of databases for quick, setup-free one-off tasks when you just need to query some data in disk files in a hurry.
* Miller also goes beyond the classic Unix tools by stepping fully into our modern, **no-SQL** world: its essential record-heterogeneity property allows Miller to operate on data where records with different schema (field names) are interleaved.
* Miller is **streaming**: most operations need only a single record in memory at a time, rather than ingesting all input before producing any output. For those operations which require deeper retention (``sort``, ``tac``, ``stats1``), Miller retains only as much data as needed. This means that whenever functionally possible, you can operate on files which are larger than your system's available RAM, and you can use Miller in **tail -f** contexts.
* Miller is **streaming**: most operations need only a single record in memory at a time, rather than ingesting all input before producing any output. For those operations which require deeper retention (`sort`, `tac`, `stats1`), Miller retains only as much data as needed. This means that whenever functionally possible, you can operate on files which are larger than your system's available RAM, and you can use Miller in **tail -f** contexts.
* Miller is **pipe-friendly** and interoperates with the Unix toolkit
@ -30,10 +30,10 @@ including but not limited to the familiar CSV, TSV, and JSON. (Miller can handl
* Miller does **conversion** between formats
* Miller's **processing is format-aware**: e.g. CSV ``sort`` and ``tac`` keep header lines first
* Miller's **processing is format-aware**: e.g. CSV `sort` and `tac` keep header lines first
* Miller has high-throughput **performance** on par with the Unix toolkit
* Not unlike [jq](https://stedolan.github.io/jq/) (for JSON), Miller is written in Go which is a portable, modern language, and Miller has no runtime dependencies. You can download or compile a single binary, ``scp`` it to a faraway machine, and expect it to work.
* Not unlike [jq](https://stedolan.github.io/jq/) (for JSON), Miller is written in Go which is a portable, modern language, and Miller has no runtime dependencies. You can download or compile a single binary, `scp` it to a faraway machine, and expect it to work.
Releases and release notes: https://github.com/johnkerl/miller/releases.
Releases and release notes: [https://github.com/johnkerl/miller/releases](https://github.com/johnkerl/miller/releases).

200
docs6b/docs/file-formats.md
View file

@ -7,8 +7,10 @@ Additionally, Miller gives you the option of including comments within your data
## Examples
<pre>
<pre class="pre-highlight">
<b>mlr help data-formats</b>
</pre>
<pre class="pre-non-highlight">
CSV/CSV-lite: comma-separated values with separate header line
TSV: same but with tabs in places of commas
+---------------------+
@ -71,28 +73,26 @@ NIDX: implicitly numerically indexed (Unix-toolkit style)
+---------------------+
</pre>
.. _file-formats-csv:
## CSV/TSV/ASV/USV/etc.
When ``mlr`` is invoked with the ``--csv`` or ``--csvlite`` option, key names are found on the first record and values are taken from subsequent records. This includes the case of CSV-formatted files. See [Record Heterogeneity](record-heterogeneity.md) for how Miller handles changes of field names within a single data stream.
When `mlr` is invoked with the `--csv` or `--csvlite` option, key names are found on the first record and values are taken from subsequent records. This includes the case of CSV-formatted files. See [Record Heterogeneity](record-heterogeneity.md) for how Miller handles changes of field names within a single data stream.
Miller has record separator ``RS`` and field separator ``FS``, just as ``awk`` does. For TSV, use ``--fs tab``; to convert TSV to CSV, use ``--ifs tab --ofs comma``, etc. (See also :ref:`reference-separators`.)
Miller has record separator `RS` and field separator `FS`, just as `awk` does. For TSV, use `--fs tab`; to convert TSV to CSV, use `--ifs tab --ofs comma`, etc. (See also [Record/field/pair separators](reference-main-io-options.md#recordfieldpair-separators).)
**TSV (tab-separated values):** the following are synonymous pairs:
* ``--tsv`` and ``--csv --fs tab``
* ``--itsv`` and ``--icsv --ifs tab``
* ``--otsv`` and ``--ocsv --ofs tab``
* ``--tsvlite`` and ``--csvlite --fs tab``
* ``--itsvlite`` and ``--icsvlite --ifs tab``
* ``--otsvlite`` and ``--ocsvlite --ofs tab``
* `--tsv` and `--csv --fs tab`
* `--itsv` and `--icsv --ifs tab`
* `--otsv` and `--ocsv --ofs tab`
* `--tsvlite` and `--csvlite --fs tab`
* `--itsvlite` and `--icsvlite --ifs tab`
* `--otsvlite` and `--ocsvlite --ofs tab`
**ASV (ASCII-separated values):** the flags ``--asv``, ``--iasv``, ``--oasv``, ``--asvlite``, ``--iasvlite``, and ``--oasvlite`` are analogous except they use ASCII FS and RS 0x1f and 0x1e, respectively.
**ASV (ASCII-separated values):** the flags `--asv`, `--iasv`, `--oasv`, `--asvlite`, `--iasvlite`, and `--oasvlite` are analogous except they use ASCII FS and RS 0x1f and 0x1e, respectively.
**USV (Unicode-separated values):** likewise, the flags ``--usv``, ``--iusv``, ``--ousv``, ``--usvlite``, ``--iusvlite``, and ``--ousvlite`` use Unicode FS and RS U+241F (UTF-8 0x0xe2909f) and U+241E (UTF-8 0xe2909e), respectively.
**USV (Unicode-separated values):** likewise, the flags `--usv`, `--iusv`, `--ousv`, `--usvlite`, `--iusvlite`, and `--ousvlite` use Unicode FS and RS U+241F (UTF-8 0x0xe2909f) and U+241E (UTF-8 0xe2909e), respectively.
Miller's ``--csv`` flag supports [RFC-4180 CSV](https://tools.ietf.org/html/rfc4180). This includes CRLF line-terminators by default, regardless of platform.
Miller's `--csv` flag supports [RFC-4180 CSV](https://tools.ietf.org/html/rfc4180). This includes CRLF line-terminators by default, regardless of platform.
Here are the differences between CSV and CSV-lite:
@ -106,16 +106,16 @@ Here are things they have in common:
* The ability to specify record/field separators other than the default, e.g. CR-LF vs. LF, or tab instead of comma for TSV, and so on.
* The ``--implicit-csv-header`` flag for input and the ``--headerless-csv-output`` flag for output.
.. _file-formats-dkvp:
* The `--implicit-csv-header` flag for input and the `--headerless-csv-output` flag for output.
## DKVP: Key-value pairs
Miller's default file format is DKVP, for **delimited key-value pairs**. Example:
<pre>
<pre class="pre-highlight">
<b>mlr cat data/small</b>
</pre>
<pre class="pre-non-highlight">
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
@ -125,51 +125,51 @@ a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
Such data are easy to generate, e.g. in Ruby with
<pre>
<pre class="pre-non-highlight">
puts "host=#{hostname},seconds=#{t2-t1},message=#{msg}"
</pre>
<pre>
<pre class="pre-non-highlight">
puts mymap.collect{|k,v| "#{k}=#{v}"}.join(',')
</pre>
or ``print`` statements in various languages, e.g.
or `print` statements in various languages, e.g.
<pre>
<pre class="pre-non-highlight">
echo "type=3,user=$USER,date=$date\n";
</pre>
<pre>
<pre class="pre-non-highlight">
logger.log("type=3,user=$USER,date=$date\n");
</pre>
Fields lacking an IPS will have positional index (starting at 1) used as the key, as in NIDX format. For example, ``dish=7,egg=8,flint`` is parsed as ``"dish" => "7", "egg" => "8", "3" => "flint"`` and ``dish,egg,flint`` is parsed as ``"1" => "dish", "2" => "egg", "3" => "flint"``.
Fields lacking an IPS will have positional index (starting at 1) used as the key, as in NIDX format. For example, `dish=7,egg=8,flint` is parsed as `"dish" => "7", "egg" => "8", "3" => "flint"` and `dish,egg,flint` is parsed as `"1" => "dish", "2" => "egg", "3" => "flint"`.
As discussed in [Record Heterogeneity](record-heterogeneity.md), Miller handles changes of field names within the same data stream. But using DKVP format this is particularly natural. One of my favorite use-cases for Miller is in application/server logs, where I log all sorts of lines such as
<pre>
<pre class="pre-non-highlight">
resource=/path/to/file,loadsec=0.45,ok=true
record_count=100, resource=/path/to/file
resource=/some/other/path,loadsec=0.97,ok=false
</pre>
etc. and I just log them as needed. Then later, I can use ``grep``, ``mlr --opprint group-like``, etc.
etc. and I just log them as needed. Then later, I can use `grep`, `mlr --opprint group-like`, etc.
to analyze my logs.
See :doc:`reference-main-io-options` regarding how to specify separators other than the default equals-sign and comma.
.. _file-formats-nidx:
See [Reference: I/O options](reference-main-io-options.md) regarding how to specify separators other than the default equals-sign and comma.
## NIDX: Index-numbered (toolkit style)
With ``--inidx --ifs ' ' --repifs``, Miller splits lines on whitespace and assigns integer field names starting with 1.
With `--inidx --ifs ' ' --repifs`, Miller splits lines on whitespace and assigns integer field names starting with 1.
This recapitulates Unix-toolkit behavior.
Example with index-numbered output:
<pre>
<pre class="pre-highlight">
<b>cat data/small</b>
</pre>
<pre class="pre-non-highlight">
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
@ -177,8 +177,10 @@ a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --onidx --ofs ' ' cat data/small</b>
</pre>
<pre class="pre-non-highlight">
pan pan 1 0.3467901443380824 0.7268028627434533
eks pan 2 0.7586799647899636 0.5221511083334797
wye wye 3 0.20460330576630303 0.33831852551664776
@ -188,15 +190,19 @@ wye pan 5 0.5732889198020006 0.8636244699032729
Example with index-numbered input:
<pre>
<pre class="pre-highlight">
<b>cat data/mydata.txt</b>
</pre>
<pre class="pre-non-highlight">
oh say can you see
by the dawn's
early light
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --inidx --ifs ' ' --odkvp cat data/mydata.txt</b>
</pre>
<pre class="pre-non-highlight">
1=oh,2=say,3=can,4=you,5=see
1=by,2=the,3=dawn's
1=early,2=light
@ -204,23 +210,25 @@ early light
Example with index-numbered input and output:
<pre>
<pre class="pre-highlight">
<b>cat data/mydata.txt</b>
</pre>
<pre class="pre-non-highlight">
oh say can you see
by the dawn's
early light
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --nidx --fs ' ' --repifs cut -f 2,3 data/mydata.txt</b>
</pre>
<pre class="pre-non-highlight">
say can
the dawn's
light
</pre>
.. _file-formats-json:
## Tabular JSON
## JSON
JSON is a format which supports arbitrarily deep nesting of "objects" (hashmaps) and "arrays" (lists), while Miller is a tool for handling **tabular data** only. This means Miller cannot (and should not) handle arbitrary JSON. (Check out [jq](https://stedolan.github.io/jq/).)
@ -232,8 +240,10 @@ By *tabular JSON* I mean the data is either a sequence of one or more objects, o
An **array of single-level objects** is, quite simply, **a table**:
<pre>
<pre class="pre-highlight">
<b>mlr --json head -n 2 then cut -f color,shape data/json-example-1.json</b>
</pre>
<pre class="pre-non-highlight">
{
"color": "yellow",
"shape": "triangle"
@ -244,8 +254,10 @@ An **array of single-level objects** is, quite simply, **a table**:
}
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --json --jvstack head -n 2 then cut -f color,u,v data/json-example-1.json</b>
</pre>
<pre class="pre-non-highlight">
{
"color": "yellow",
"u": 0.6321695890307647,
@ -258,8 +270,10 @@ An **array of single-level objects** is, quite simply, **a table**:
}
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --ijson --opprint stats1 -a mean,stddev,count -f u -g shape data/json-example-1.json</b>
</pre>
<pre class="pre-non-highlight">
shape u_mean u_stddev u_count
triangle 0.5839952367477192 0.13118354465618046 3
square 0.409355036804889 0.3654281755508655 4
@ -270,8 +284,10 @@ circle 0.36601268553826866 0.2090944565900053 3
Additionally, Miller can **tabularize nested objects by concatentating keys**:
<pre>
<pre class="pre-highlight">
<b>mlr --json --jvstack head -n 2 data/json-example-2.json</b>
</pre>
<pre class="pre-non-highlight">
{
"flag": 1,
"i": 11,
@ -302,8 +318,10 @@ Additionally, Miller can **tabularize nested objects by concatentating keys**:
}
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --ijson --opprint head -n 4 data/json-example-2.json</b>
</pre>
<pre class="pre-non-highlight">
flag i attributes.color attributes.shape values.u values.v values.w values.x
1 11 yellow triangle 0.632170 0.988721 0.436498 5.798188
1 15 red square 0.219668 0.001257 0.792778 2.944117
@ -311,12 +329,14 @@ flag i attributes.color attributes.shape values.u values.v values.w values.x
0 48 red square 0.956274 0.746720 0.775542 7.117831
</pre>
Note in particular that as far as Miller's ``put`` and ``filter``, as well as other I/O formats, are concerned, these are simply field names with colons in them:
Note in particular that as far as Miller's `put` and `filter`, as well as other I/O formats, are concerned, these are simply field names with colons in them:
<pre>
<pre class="pre-highlight">
<b>mlr --json --jvstack head -n 1 \</b>
<b> then put '${values:uv} = ${values:u} * ${values:v}' \</b>
<b> data/json-example-2.json</b>
</pre>
<pre class="pre-non-highlight">
{
"flag": 1,
"i": 11,
@ -335,12 +355,14 @@ Note in particular that as far as Miller's ``put`` and ``filter``, as well as ot
### Arrays
Arrays (TODO: update for Miller6) aren't supported in Miller's ``put``/``filter`` DSL. By default, JSON arrays are read in as integer-keyed maps.
Arrays (TODO: update for Miller6) aren't supported in Miller's `put`/`filter` DSL. By default, JSON arrays are read in as integer-keyed maps.
Suppose we have arrays like this in our input data:
<pre>
<pre class="pre-highlight">
<b>cat data/json-example-3.json</b>
</pre>
<pre class="pre-non-highlight">
{
"label": "orange",
"values": [12.2, 13.8, 17.2]
@ -353,8 +375,10 @@ Suppose we have arrays like this in our input data:
Then integer indices (starting from 0 and counting up) are used as map keys:
<pre>
<pre class="pre-highlight">
<b>mlr --ijson --oxtab cat data/json-example-3.json</b>
</pre>
<pre class="pre-non-highlight">
label orange
values.1 12.2
values.2 13.8
@ -367,8 +391,10 @@ values.2 32.4
When the data are written back out as JSON, field names are re-expanded as above, but what were arrays on input are now maps on output:
<pre>
<pre class="pre-highlight">
<b>mlr --json --jvstack cat data/json-example-3.json</b>
</pre>
<pre class="pre-non-highlight">
{
"label": "orange",
"values": [12.2, 13.8, 17.2]
@ -381,34 +407,34 @@ When the data are written back out as JSON, field names are re-expanded as above
This is non-ideal, but it allows Miller (5.x release being latest as of this writing) to handle JSON arrays at all.
You might also use ``mlr --json-skip-arrays-on-input`` or ``mlr --json-fatal-arrays-on-input``.
You might also use `mlr --json-skip-arrays-on-input` or `mlr --json-fatal-arrays-on-input`.
To truly handle JSON, please use a JSON-processing tool such as [jq](https://stedolan.github.io/jq/).
### Formatting JSON options
JSON isn't a parameterized format, so ``RS``, ``FS``, ``PS`` aren't specifiable. Nonetheless, you can do the following:
JSON isn't a parameterized format, so `RS`, `FS`, `PS` aren't specifiable. Nonetheless, you can do the following:
* Use ``--jvstack`` to pretty-print JSON objects with multi-line (vertically stacked) spacing. By default, each Miller record (JSON object) is one per line.
* Use `--jvstack` to pretty-print JSON objects with multi-line (vertically stacked) spacing. By default, each Miller record (JSON object) is one per line.
* Keystroke-savers: ``--jsonx`` simply means ``--json --jvstack``, and ``--ojsonx`` simply means ``--ojson --jvstack``.
* Keystroke-savers: `--jsonx` simply means `--json --jvstack`, and `--ojsonx` simply means `--ojson --jvstack`.
* Use ``--jlistwrap`` to print the sequence of JSON objects wrapped in an outermost ``[`` and ``]``. By default, these aren't printed.
* Use `--jlistwrap` to print the sequence of JSON objects wrapped in an outermost `[` and `]`. By default, these aren't printed.
* Use ``--jquoteall`` to double-quote all object values. By default, integers, floating-point numbers, and booleans ``true`` and ``false`` are not double-quoted when they appear as JSON-object keys.
* Use `--jquoteall` to double-quote all object values. By default, integers, floating-point numbers, and booleans `true` and `false` are not double-quoted when they appear as JSON-object keys.
* Use ``--jflatsep youmd.inghere`` to specify the string used for key concatenation: this defaults to a single colon.
* Use `--jflatsep youmd.inghere` to specify the string used for key concatenation: this defaults to a single colon.
Again, please see [jq](https://stedolan.github.io/jq/) for a truly powerful, JSON-specific tool.
.. _file-formats-pprint:
## PPRINT: Pretty-printed tabular
Miller's pretty-print format is like CSV, but column-aligned. For example, compare
<pre>
<pre class="pre-highlight">
<b>mlr --ocsv cat data/small</b>
</pre>
<pre class="pre-non-highlight">
a,b,i,x,y
pan,pan,1,0.3467901443380824,0.7268028627434533
eks,pan,2,0.7586799647899636,0.5221511083334797
@ -417,8 +443,10 @@ eks,wye,4,0.38139939387114097,0.13418874328430463
wye,pan,5,0.5732889198020006,0.8636244699032729
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --opprint cat data/small</b>
</pre>
<pre class="pre-non-highlight">
a b i x y
pan pan 1 0.3467901443380824 0.7268028627434533
eks pan 2 0.7586799647899636 0.5221511083334797
@ -427,14 +455,16 @@ eks wye 4 0.38139939387114097 0.13418874328430463
wye pan 5 0.5732889198020006 0.8636244699032729
</pre>
Note that while Miller is a line-at-a-time processor and retains input lines in memory only where necessary (e.g. for sort), pretty-print output requires it to accumulate all input lines (so that it can compute maximum column widths) before producing any output. This has two consequences: (a) pretty-print output won't work on ``tail -f`` contexts, where Miller will be waiting for an end-of-file marker which never arrives; (b) pretty-print output for large files is constrained by available machine memory.
Note that while Miller is a line-at-a-time processor and retains input lines in memory only where necessary (e.g. for sort), pretty-print output requires it to accumulate all input lines (so that it can compute maximum column widths) before producing any output. This has two consequences: (a) pretty-print output won't work on `tail -f` contexts, where Miller will be waiting for an end-of-file marker which never arrives; (b) pretty-print output for large files is constrained by available machine memory.
See [Record Heterogeneity](record-heterogeneity.md) for how Miller handles changes of field names within a single data stream.
For output only (this isn't supported in the input-scanner as of 5.0.0) you can use ``--barred`` with pprint output format:
For output only (this isn't supported in the input-scanner as of 5.0.0) you can use `--barred` with pprint output format:
<pre>
<pre class="pre-highlight">
<b>mlr --opprint --barred cat data/small</b>
</pre>
<pre class="pre-non-highlight">
+-----+-----+---+---------------------+---------------------+
| a | b | i | x | y |
+-----+-----+---+---------------------+---------------------+
@ -446,14 +476,12 @@ For output only (this isn't supported in the input-scanner as of 5.0.0) you can
+-----+-----+---+---------------------+---------------------+
</pre>
.. _file-formats-xtab:
## XTAB: Vertical tabular
This is perhaps most useful for looking a very wide and/or multi-column data which causes line-wraps on the screen (but see also
[ngrid](https://github.com/twosigma/ngrid/) for an entirely different, very powerful option). Namely:
<pre>
<pre class="pre-non-highlight">
$ grep -v '^#' /etc/passwd | head -n 6 | mlr --nidx --fs : --opprint cat
1 2 3 4 5 6 7
nobody * -2 -2 Unprivileged User /var/empty /usr/bin/false
@ -464,7 +492,7 @@ _taskgated * 13 13 Task Gate Daemon /var/empty /usr/bin/false
_networkd * 24 24 Network Services /var/networkd /usr/bin/false
</pre>
<pre>
<pre class="pre-non-highlight">
$ grep -v '^#' /etc/passwd | head -n 2 | mlr --nidx --fs : --oxtab cat
1 nobody
2 *
@ -483,7 +511,7 @@ $ grep -v '^#' /etc/passwd | head -n 2 | mlr --nidx --fs : --oxtab cat
7 /bin/sh
</pre>
<pre>
<pre class="pre-non-highlight">
$ grep -v '^#' /etc/passwd | head -n 2 | \
mlr --nidx --fs : --ojson --jvstack --jlistwrap label name,password,uid,gid,gecos,home_dir,shell
[
@ -512,8 +540,10 @@ $ grep -v '^#' /etc/passwd | head -n 2 | \
Markdown format looks like this:
<pre>
<pre class="pre-highlight">
<b>mlr --omd cat data/small</b>
</pre>
<pre class="pre-non-highlight">
| a | b | i | x | y |
| --- | --- | --- | --- | --- |
| pan | pan | 1 | 0.3467901443380824 | 0.7268028627434533 |
@ -531,10 +561,12 @@ As of Miller 4.3.0, markdown format is supported only for output, not input.
## Data-conversion keystroke-savers
While you can do format conversion using ``mlr --icsv --ojson cat myfile.csv``, there are also keystroke-savers for this purpose, such as ``mlr --c2j cat myfile.csv``. For a complete list:
While you can do format conversion using `mlr --icsv --ojson cat myfile.csv`, there are also keystroke-savers for this purpose, such as `mlr --c2j cat myfile.csv`. For a complete list:
<pre>
<pre class="pre-highlight">
<b>mlr help format-conversion</b>
</pre>
<pre class="pre-non-highlight">
As keystroke-savers for format-conversion you may use the following:
--c2t --c2d --c2n --c2j --c2x --c2p --c2m
--t2c --t2d --t2n --t2j --t2x --t2p --t2m
@ -550,24 +582,26 @@ output only.
## Autodetect of line endings
Default line endings (``--irs`` and ``--ors``) are ``'auto'`` which means **autodetect from the input file format**, as long as the input file(s) have lines ending in either LF (also known as linefeed, ``'\n'``, ``0x0a``, Unix-style) or CRLF (also known as carriage-return/linefeed pairs, ``'\r\n'``, ``0x0d 0x0a``, Windows style).
Default line endings (`--irs` and `--ors`) are `'auto'` which means **autodetect from the input file format**, as long as the input file(s) have lines ending in either LF (also known as linefeed, `'\n'`, `0x0a`, Unix-style) or CRLF (also known as carriage-return/linefeed pairs, `'\r\n'`, `0x0d 0x0a`, Windows style).
**If both IRS and ORS are auto (which is the default) then LF input will lead to LF output and CRLF input will lead to CRLF output, regardless of the platform you're running on.**
The line-ending autodetector triggers on the first line ending detected in the input stream. E.g. if you specify a CRLF-terminated file on the command line followed by an LF-terminated file then autodetected line endings will be CRLF.
If you use ``--ors {something else}`` with (default or explicitly specified) ``--irs auto`` then line endings are autodetected on input and set to what you specify on output.
If you use `--ors {something else}` with (default or explicitly specified) `--irs auto` then line endings are autodetected on input and set to what you specify on output.
If you use ``--irs {something else}`` with (default or explicitly specified) ``--ors auto`` then the output line endings used are LF on Unix/Linux/BSD/MacOSX, and CRLF on Windows.
If you use `--irs {something else}` with (default or explicitly specified) `--ors auto` then the output line endings used are LF on Unix/Linux/BSD/MacOSX, and CRLF on Windows.
See also :ref:`reference-separators` for more information about record/field/pair separators.
See also [Record/field/pair separators](reference-main-io-options.md#recordfieldpair-separators) for more information about record/field/pair separators.
## Comments in data
You can include comments within your data files, and either have them ignored, or passed directly through to the standard output as soon as they are encountered:
<pre>
<pre class="pre-highlight">
<b>mlr help comments-in-data</b>
</pre>
<pre class="pre-non-highlight">
--skip-comments Ignore commented lines (prefixed by "#")
within the input.
--skip-comments-with {string} Ignore commented lines within input, with
@ -589,8 +623,10 @@ Notes:
Examples:
<pre>
<pre class="pre-highlight">
<b>cat data/budget.csv</b>
</pre>
<pre class="pre-non-highlight">
# Asana -- here are the budget figures you asked for!
type,quantity
purple,456.78
@ -598,16 +634,20 @@ green,678.12
orange,123.45
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --skip-comments --icsv --opprint sort -nr quantity data/budget.csv</b>
</pre>
<pre class="pre-non-highlight">
type quantity
green 678.12
purple 456.78
orange 123.45
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --pass-comments --icsv --opprint sort -nr quantity data/budget.csv</b>
</pre>
<pre class="pre-non-highlight">
# Asana -- here are the budget figures you asked for!
type quantity
green 678.12

80
docs6b/docs/file-formats.md.in
View file

@ -10,28 +10,26 @@ GENMD_RUN_COMMAND
mlr help data-formats
GENMD_EOF
.. _file-formats-csv:
## CSV/TSV/ASV/USV/etc.
When ``mlr`` is invoked with the ``--csv`` or ``--csvlite`` option, key names are found on the first record and values are taken from subsequent records. This includes the case of CSV-formatted files. See [Record Heterogeneity](record-heterogeneity.md) for how Miller handles changes of field names within a single data stream.
When `mlr` is invoked with the `--csv` or `--csvlite` option, key names are found on the first record and values are taken from subsequent records. This includes the case of CSV-formatted files. See [Record Heterogeneity](record-heterogeneity.md) for how Miller handles changes of field names within a single data stream.
Miller has record separator ``RS`` and field separator ``FS``, just as ``awk`` does. For TSV, use ``--fs tab``; to convert TSV to CSV, use ``--ifs tab --ofs comma``, etc. (See also :ref:`reference-separators`.)
Miller has record separator `RS` and field separator `FS`, just as `awk` does. For TSV, use `--fs tab`; to convert TSV to CSV, use `--ifs tab --ofs comma`, etc. (See also [Record/field/pair separators](reference-main-io-options.md#recordfieldpair-separators).)
**TSV (tab-separated values):** the following are synonymous pairs:
* ``--tsv`` and ``--csv --fs tab``
* ``--itsv`` and ``--icsv --ifs tab``
* ``--otsv`` and ``--ocsv --ofs tab``
* ``--tsvlite`` and ``--csvlite --fs tab``
* ``--itsvlite`` and ``--icsvlite --ifs tab``
* ``--otsvlite`` and ``--ocsvlite --ofs tab``
* `--tsv` and `--csv --fs tab`
* `--itsv` and `--icsv --ifs tab`
* `--otsv` and `--ocsv --ofs tab`
* `--tsvlite` and `--csvlite --fs tab`
* `--itsvlite` and `--icsvlite --ifs tab`
* `--otsvlite` and `--ocsvlite --ofs tab`
**ASV (ASCII-separated values):** the flags ``--asv``, ``--iasv``, ``--oasv``, ``--asvlite``, ``--iasvlite``, and ``--oasvlite`` are analogous except they use ASCII FS and RS 0x1f and 0x1e, respectively.
**ASV (ASCII-separated values):** the flags `--asv`, `--iasv`, `--oasv`, `--asvlite`, `--iasvlite`, and `--oasvlite` are analogous except they use ASCII FS and RS 0x1f and 0x1e, respectively.
**USV (Unicode-separated values):** likewise, the flags ``--usv``, ``--iusv``, ``--ousv``, ``--usvlite``, ``--iusvlite``, and ``--ousvlite`` use Unicode FS and RS U+241F (UTF-8 0x0xe2909f) and U+241E (UTF-8 0xe2909e), respectively.
**USV (Unicode-separated values):** likewise, the flags `--usv`, `--iusv`, `--ousv`, `--usvlite`, `--iusvlite`, and `--ousvlite` use Unicode FS and RS U+241F (UTF-8 0x0xe2909f) and U+241E (UTF-8 0xe2909e), respectively.
Miller's ``--csv`` flag supports [RFC-4180 CSV](https://tools.ietf.org/html/rfc4180). This includes CRLF line-terminators by default, regardless of platform.
Miller's `--csv` flag supports [RFC-4180 CSV](https://tools.ietf.org/html/rfc4180). This includes CRLF line-terminators by default, regardless of platform.
Here are the differences between CSV and CSV-lite:
@ -45,9 +43,7 @@ Here are things they have in common:
* The ability to specify record/field separators other than the default, e.g. CR-LF vs. LF, or tab instead of comma for TSV, and so on.
* The ``--implicit-csv-header`` flag for input and the ``--headerless-csv-output`` flag for output.
.. _file-formats-dkvp:
* The `--implicit-csv-header` flag for input and the `--headerless-csv-output` flag for output.
## DKVP: Key-value pairs
@ -67,7 +63,7 @@ GENMD_CARDIFY
puts mymap.collect{|k,v| "#{k}=#{v}"}.join(',')
GENMD_EOF
or ``print`` statements in various languages, e.g.
or `print` statements in various languages, e.g.
GENMD_CARDIFY
echo "type=3,user=$USER,date=$date\n";
@ -77,7 +73,7 @@ GENMD_CARDIFY
logger.log("type=3,user=$USER,date=$date\n");
GENMD_EOF
Fields lacking an IPS will have positional index (starting at 1) used as the key, as in NIDX format. For example, ``dish=7,egg=8,flint`` is parsed as ``"dish" => "7", "egg" => "8", "3" => "flint"`` and ``dish,egg,flint`` is parsed as ``"1" => "dish", "2" => "egg", "3" => "flint"``.
Fields lacking an IPS will have positional index (starting at 1) used as the key, as in NIDX format. For example, `dish=7,egg=8,flint` is parsed as `"dish" => "7", "egg" => "8", "3" => "flint"` and `dish,egg,flint` is parsed as `"1" => "dish", "2" => "egg", "3" => "flint"`.
As discussed in [Record Heterogeneity](record-heterogeneity.md), Miller handles changes of field names within the same data stream. But using DKVP format this is particularly natural. One of my favorite use-cases for Miller is in application/server logs, where I log all sorts of lines such as
@ -87,16 +83,14 @@ record_count=100, resource=/path/to/file
resource=/some/other/path,loadsec=0.97,ok=false
GENMD_EOF
etc. and I just log them as needed. Then later, I can use ``grep``, ``mlr --opprint group-like``, etc.
etc. and I just log them as needed. Then later, I can use `grep`, `mlr --opprint group-like`, etc.
to analyze my logs.
See :doc:`reference-main-io-options` regarding how to specify separators other than the default equals-sign and comma.
.. _file-formats-nidx:
See [Reference: I/O options](reference-main-io-options.md) regarding how to specify separators other than the default equals-sign and comma.
## NIDX: Index-numbered (toolkit style)
With ``--inidx --ifs ' ' --repifs``, Miller splits lines on whitespace and assigns integer field names starting with 1.
With `--inidx --ifs ' ' --repifs`, Miller splits lines on whitespace and assigns integer field names starting with 1.
This recapitulates Unix-toolkit behavior.
@ -130,9 +124,7 @@ GENMD_RUN_COMMAND
mlr --nidx --fs ' ' --repifs cut -f 2,3 data/mydata.txt
GENMD_EOF
.. _file-formats-json:
## Tabular JSON
## JSON
JSON is a format which supports arbitrarily deep nesting of "objects" (hashmaps) and "arrays" (lists), while Miller is a tool for handling **tabular data** only. This means Miller cannot (and should not) handle arbitrary JSON. (Check out [jq](https://stedolan.github.io/jq/).)
@ -168,7 +160,7 @@ GENMD_RUN_COMMAND
mlr --ijson --opprint head -n 4 data/json-example-2.json
GENMD_EOF
Note in particular that as far as Miller's ``put`` and ``filter``, as well as other I/O formats, are concerned, these are simply field names with colons in them:
Note in particular that as far as Miller's `put` and `filter`, as well as other I/O formats, are concerned, these are simply field names with colons in them:
GENMD_RUN_COMMAND
mlr --json --jvstack head -n 1 \
@ -178,7 +170,7 @@ GENMD_EOF
### Arrays
Arrays (TODO: update for Miller6) aren't supported in Miller's ``put``/``filter`` DSL. By default, JSON arrays are read in as integer-keyed maps.
Arrays (TODO: update for Miller6) aren't supported in Miller's `put`/`filter` DSL. By default, JSON arrays are read in as integer-keyed maps.
Suppose we have arrays like this in our input data:
@ -200,28 +192,26 @@ GENMD_EOF
This is non-ideal, but it allows Miller (5.x release being latest as of this writing) to handle JSON arrays at all.
You might also use ``mlr --json-skip-arrays-on-input`` or ``mlr --json-fatal-arrays-on-input``.
You might also use `mlr --json-skip-arrays-on-input` or `mlr --json-fatal-arrays-on-input`.
To truly handle JSON, please use a JSON-processing tool such as [jq](https://stedolan.github.io/jq/).
### Formatting JSON options
JSON isn't a parameterized format, so ``RS``, ``FS``, ``PS`` aren't specifiable. Nonetheless, you can do the following:
JSON isn't a parameterized format, so `RS`, `FS`, `PS` aren't specifiable. Nonetheless, you can do the following:
* Use ``--jvstack`` to pretty-print JSON objects with multi-line (vertically stacked) spacing. By default, each Miller record (JSON object) is one per line.
* Use `--jvstack` to pretty-print JSON objects with multi-line (vertically stacked) spacing. By default, each Miller record (JSON object) is one per line.
* Keystroke-savers: ``--jsonx`` simply means ``--json --jvstack``, and ``--ojsonx`` simply means ``--ojson --jvstack``.
* Keystroke-savers: `--jsonx` simply means `--json --jvstack`, and `--ojsonx` simply means `--ojson --jvstack`.
* Use ``--jlistwrap`` to print the sequence of JSON objects wrapped in an outermost ``[`` and ``]``. By default, these aren't printed.
* Use `--jlistwrap` to print the sequence of JSON objects wrapped in an outermost `[` and `]`. By default, these aren't printed.
* Use ``--jquoteall`` to double-quote all object values. By default, integers, floating-point numbers, and booleans ``true`` and ``false`` are not double-quoted when they appear as JSON-object keys.
* Use `--jquoteall` to double-quote all object values. By default, integers, floating-point numbers, and booleans `true` and `false` are not double-quoted when they appear as JSON-object keys.
* Use ``--jflatsep youmd.inghere`` to specify the string used for key concatenation: this defaults to a single colon.
* Use `--jflatsep youmd.inghere` to specify the string used for key concatenation: this defaults to a single colon.
Again, please see [jq](https://stedolan.github.io/jq/) for a truly powerful, JSON-specific tool.
.. _file-formats-pprint:
## PPRINT: Pretty-printed tabular
Miller's pretty-print format is like CSV, but column-aligned. For example, compare
@ -234,18 +224,16 @@ GENMD_RUN_COMMAND
mlr --opprint cat data/small
GENMD_EOF
Note that while Miller is a line-at-a-time processor and retains input lines in memory only where necessary (e.g. for sort), pretty-print output requires it to accumulate all input lines (so that it can compute maximum column widths) before producing any output. This has two consequences: (a) pretty-print output won't work on ``tail -f`` contexts, where Miller will be waiting for an end-of-file marker which never arrives; (b) pretty-print output for large files is constrained by available machine memory.
Note that while Miller is a line-at-a-time processor and retains input lines in memory only where necessary (e.g. for sort), pretty-print output requires it to accumulate all input lines (so that it can compute maximum column widths) before producing any output. This has two consequences: (a) pretty-print output won't work on `tail -f` contexts, where Miller will be waiting for an end-of-file marker which never arrives; (b) pretty-print output for large files is constrained by available machine memory.
See [Record Heterogeneity](record-heterogeneity.md) for how Miller handles changes of field names within a single data stream.
For output only (this isn't supported in the input-scanner as of 5.0.0) you can use ``--barred`` with pprint output format:
For output only (this isn't supported in the input-scanner as of 5.0.0) you can use `--barred` with pprint output format:
GENMD_RUN_COMMAND
mlr --opprint --barred cat data/small
GENMD_EOF
.. _file-formats-xtab:
## XTAB: Vertical tabular
This is perhaps most useful for looking a very wide and/or multi-column data which causes line-wraps on the screen (but see also
@ -273,7 +261,7 @@ As of Miller 4.3.0, markdown format is supported only for output, not input.
## Data-conversion keystroke-savers
While you can do format conversion using ``mlr --icsv --ojson cat myfile.csv``, there are also keystroke-savers for this purpose, such as ``mlr --c2j cat myfile.csv``. For a complete list:
While you can do format conversion using `mlr --icsv --ojson cat myfile.csv`, there are also keystroke-savers for this purpose, such as `mlr --c2j cat myfile.csv`. For a complete list:
GENMD_RUN_COMMAND
mlr help format-conversion
@ -281,17 +269,17 @@ GENMD_EOF
## Autodetect of line endings
Default line endings (``--irs`` and ``--ors``) are ``'auto'`` which means **autodetect from the input file format**, as long as the input file(s) have lines ending in either LF (also known as linefeed, ``'\n'``, ``0x0a``, Unix-style) or CRLF (also known as carriage-return/linefeed pairs, ``'\r\n'``, ``0x0d 0x0a``, Windows style).
Default line endings (`--irs` and `--ors`) are `'auto'` which means **autodetect from the input file format**, as long as the input file(s) have lines ending in either LF (also known as linefeed, `'\n'`, `0x0a`, Unix-style) or CRLF (also known as carriage-return/linefeed pairs, `'\r\n'`, `0x0d 0x0a`, Windows style).
**If both IRS and ORS are auto (which is the default) then LF input will lead to LF output and CRLF input will lead to CRLF output, regardless of the platform you're running on.**
The line-ending autodetector triggers on the first line ending detected in the input stream. E.g. if you specify a CRLF-terminated file on the command line followed by an LF-terminated file then autodetected line endings will be CRLF.
If you use ``--ors {something else}`` with (default or explicitly specified) ``--irs auto`` then line endings are autodetected on input and set to what you specify on output.
If you use `--ors {something else}` with (default or explicitly specified) `--irs auto` then line endings are autodetected on input and set to what you specify on output.
If you use ``--irs {something else}`` with (default or explicitly specified) ``--ors auto`` then the output line endings used are LF on Unix/Linux/BSD/MacOSX, and CRLF on Windows.
If you use `--irs {something else}` with (default or explicitly specified) `--ors auto` then the output line endings used are LF on Unix/Linux/BSD/MacOSX, and CRLF on Windows.
See also :ref:`reference-separators` for more information about record/field/pair separators.
See also [Record/field/pair separators](reference-main-io-options.md#recordfieldpair-separators) for more information about record/field/pair separators.
## Comments in data

18
docs6b/docs/genmd-filter
View file

@ -132,16 +132,22 @@ end
# ----------------------------------------------------------------
def write_card(command_lines, output_lines, output_handle)
output_handle.puts("<pre>")
command_lines.each do |command_line|
output_handle.puts("<b>"+command_line+"</b>")
if command_lines.length > 0
output_handle.puts('<pre class="pre-highlight">')
command_lines.each do |command_line|
output_handle.puts("<b>"+command_line+"</b>")
end
output_handle.puts("</pre>")
end
output_lines.each do |output_line|
output_handle.puts(output_line)
if output_lines.length > 0
output_handle.puts('<pre class="pre-non-highlight">')
output_lines.each do |output_line|
output_handle.puts(output_line)
end
output_handle.puts("</pre>")
end
output_handle.puts("</pre>")
end

4
docs6b/docs/index.md
View file

@ -8,11 +8,11 @@ In several senses, Miller is more than one tool:
**Format conversion:** You can convert CSV files to JSON, or vice versa, or
pretty-print your data horizontally or vertically to make it easier to read.
**Data manipulation:** With a few keystrokes you can remove columns you don't care about -- or, make new ones using expressions like ``$rate = $units / $seconds``.
**Data manipulation:** With a few keystrokes you can remove columns you don't care about -- or, make new ones using expressions like `$rate = $units / $seconds`.
**Pre-processing/post-processing vs standalone use:** You can use Miller to clean data files and put them into standard formats, perhaps in preparation to load them into a database or a hands-off data-processing pipeline. Or you can use it post-process and summary database-query output. As well, you can use Miller to explore and analyze your data interactively.
**Compact verbs vs programming language:** For low-keystroking you can do things like ``mlr --csv sort -f name input.csv`` or ``mlr --json head -n 1 myfile.json``. The ``sort``, ``head``, etc are called *verbs*. They're analogs of familiar command-line tools like ``sort``, ``head``, and so on -- but they're aware of name-indexed, multi-line file formats like CSV and JSON. In addition, though, using Miller's ``put`` verb you can use programming-language statements for expressions like ``$rate = $units / $seconds`` which allow you to succintly express your own logic.
**Compact verbs vs programming language:** For low-keystroking you can do things like `mlr --csv sort -f name input.csv` or `mlr --json head -n 1 myfile.json`. The `sort`, `head`, etc are called *verbs*. They're analogs of familiar command-line tools like `sort`, `head`, and so on -- but they're aware of name-indexed, multi-line file formats like CSV and JSON. In addition, though, using Miller's `put` verb you can use programming-language statements for expressions like `$rate = $units / $seconds` which allow you to succintly express your own logic.
**Multiple domains:** People use Miller for data analysis, data science, software engineering, devops/system-administration, journalism, scientific research, and more.

4
docs6b/docs/index.md.in
View file

@ -7,11 +7,11 @@ In several senses, Miller is more than one tool:
**Format conversion:** You can convert CSV files to JSON, or vice versa, or
pretty-print your data horizontally or vertically to make it easier to read.
**Data manipulation:** With a few keystrokes you can remove columns you don't care about -- or, make new ones using expressions like ``$rate = $units / $seconds``.
**Data manipulation:** With a few keystrokes you can remove columns you don't care about -- or, make new ones using expressions like `$rate = $units / $seconds`.
**Pre-processing/post-processing vs standalone use:** You can use Miller to clean data files and put them into standard formats, perhaps in preparation to load them into a database or a hands-off data-processing pipeline. Or you can use it post-process and summary database-query output. As well, you can use Miller to explore and analyze your data interactively.
**Compact verbs vs programming language:** For low-keystroking you can do things like ``mlr --csv sort -f name input.csv`` or ``mlr --json head -n 1 myfile.json``. The ``sort``, ``head``, etc are called *verbs*. They're analogs of familiar command-line tools like ``sort``, ``head``, and so on -- but they're aware of name-indexed, multi-line file formats like CSV and JSON. In addition, though, using Miller's ``put`` verb you can use programming-language statements for expressions like ``$rate = $units / $seconds`` which allow you to succintly express your own logic.
**Compact verbs vs programming language:** For low-keystroking you can do things like `mlr --csv sort -f name input.csv` or `mlr --json head -n 1 myfile.json`. The `sort`, `head`, etc are called *verbs*. They're analogs of familiar command-line tools like `sort`, `head`, and so on -- but they're aware of name-indexed, multi-line file formats like CSV and JSON. In addition, though, using Miller's `put` verb you can use programming-language statements for expressions like `$rate = $units / $seconds` which allow you to succintly express your own logic.
**Multiple domains:** People use Miller for data analysis, data science, software engineering, devops/system-administration, journalism, scientific research, and more.

14
docs6b/docs/installation.md
View file

@ -9,33 +9,33 @@ Until then, please see the following sections for how to get Miller 6.*
[Homebrew](https://brew.sh/) installation support for OSX is available via
<pre>
<pre class="pre-highlight">
<b>brew update && brew install miller</b>
</pre>
...and also via [MacPorts](https://www.macports.org/):
<pre>
<pre class="pre-highlight">
<b>sudo port selfupdate && sudo port install miller</b>
</pre>
You may already have the ``mlr`` executable available in your platform's package manager on NetBSD, Debian Linux, Ubuntu Xenial and upward, Arch Linux, or perhaps other distributions. For example, on various Linux distributions you might do one of the following:
You may already have the `mlr` executable available in your platform's package manager on NetBSD, Debian Linux, Ubuntu Xenial and upward, Arch Linux, or perhaps other distributions. For example, on various Linux distributions you might do one of the following:
<pre>
<pre class="pre-highlight">
<b>sudo apt-get install miller</b>
</pre>
<pre>
<pre class="pre-highlight">
<b>sudo apt install miller</b>
</pre>
<pre>
<pre class="pre-highlight">
<b>sudo yum install miller</b>
</pre>
On Windows, Miller is available via [Chocolatey](https://chocolatey.org/):
<pre>
<pre class="pre-highlight">
<b>choco install miller</b>
</pre>

2
docs6b/docs/installation.md.in
View file

@ -18,7 +18,7 @@ GENMD_CARDIFY_HIGHLIGHT_ONE
sudo port selfupdate && sudo port install miller
GENMD_EOF
You may already have the ``mlr`` executable available in your platform's package manager on NetBSD, Debian Linux, Ubuntu Xenial and upward, Arch Linux, or perhaps other distributions. For example, on various Linux distributions you might do one of the following:
You may already have the `mlr` executable available in your platform's package manager on NetBSD, Debian Linux, Ubuntu Xenial and upward, Arch Linux, or perhaps other distributions. For example, on various Linux distributions you might do one of the following:
GENMD_CARDIFY_HIGHLIGHT_ONE
sudo apt-get install miller

6
docs6b/docs/internationalization.md
View file

@ -6,7 +6,7 @@ Miller handles ASCII and UTF-8 strings. (I have no plans to support UTF-16 or IS
Support for internationalization includes:
* Tabular output formats such pprint and xtab (see [File Formats](file-formats.md)) are aligned correctly.
* The :ref:`reference-dsl-strlen` function correctly counts UTF-8 codepoints rather than bytes.
* The :ref:`reference-dsl-toupper`, :ref:`reference-dsl-tolower`, and :ref:`reference-dsl-capitalize` DSL functions operate within the capabilities of the Go libraries.
* The [strlen](reference-dsl-builtin-functions.md#strlen) function correctly counts UTF-8 codepoints rather than bytes.
* The [toupper](reference-dsl-builtin-functions.md#toupper), [tolower](reference-dsl-builtin-functions.md#tolower), and [capitalize](reference-dsl-builtin-functions.md#capitalize) DSL functions operate within the capabilities of the Go libraries.
Please file an issue at https://github.com/johnkerl/miller if you encounter bugs related to internationalization (or anything else for that matter).
Please file an issue at [https://github.com/johnkerl/miller](https://github.com/johnkerl/miller) if you encounter bugs related to internationalization (or anything else for that matter).

6
docs6b/docs/internationalization.md.in
View file

@ -5,7 +5,7 @@ Miller handles ASCII and UTF-8 strings. (I have no plans to support UTF-16 or IS
Support for internationalization includes:
* Tabular output formats such pprint and xtab (see [File Formats](file-formats.md)) are aligned correctly.
* The :ref:`reference-dsl-strlen` function correctly counts UTF-8 codepoints rather than bytes.
* The :ref:`reference-dsl-toupper`, :ref:`reference-dsl-tolower`, and :ref:`reference-dsl-capitalize` DSL functions operate within the capabilities of the Go libraries.
* The [strlen](reference-dsl-builtin-functions.md#strlen) function correctly counts UTF-8 codepoints rather than bytes.
* The [toupper](reference-dsl-builtin-functions.md#toupper), [tolower](reference-dsl-builtin-functions.md#tolower), and [capitalize](reference-dsl-builtin-functions.md#capitalize) DSL functions operate within the capabilities of the Go libraries.
Please file an issue at https://github.com/johnkerl/miller if you encounter bugs related to internationalization (or anything else for that matter).
Please file an issue at [https://github.com/johnkerl/miller](https://github.com/johnkerl/miller) if you encounter bugs related to internationalization (or anything else for that matter).

58
docs6b/docs/joins.md
View file

@ -5,18 +5,22 @@
**This section describes behavior before Miller 5.1.0. As of 5.1.0, -u is the default.**
For example, the right file here has nine records, and the left file should add in the ``hostname`` column -- so the join output should also have 9 records:
For example, the right file here has nine records, and the left file should add in the `hostname` column -- so the join output should also have 9 records:
<pre>
<pre class="pre-highlight">
<b>mlr --icsvlite --opprint cat data/join-u-left.csv</b>
</pre>
<pre class="pre-non-highlight">
hostname ipaddr
nadir.east.our.org 10.3.1.18
zenith.west.our.org 10.3.1.27
apoapsis.east.our.org 10.4.5.94
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --icsvlite --opprint cat data/join-u-right.csv</b>
</pre>
<pre class="pre-non-highlight">
ipaddr timestamp bytes
10.3.1.27 1448762579 4568
10.3.1.18 1448762578 8729
@ -29,8 +33,10 @@ ipaddr timestamp bytes
10.4.5.94 1448762599 12200
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --icsvlite --opprint join -s -j ipaddr -f data/join-u-left.csv data/join-u-right.csv</b>
</pre>
<pre class="pre-non-highlight">
ipaddr hostname timestamp bytes
10.3.1.27 zenith.west.our.org 1448762579 4568
10.4.5.94 apoapsis.east.our.org 1448762579 17445
@ -38,12 +44,14 @@ ipaddr hostname timestamp bytes
10.4.5.94 apoapsis.east.our.org 1448762599 12200
</pre>
The issue is that Miller's ``join``, by default (before 5.1.0), took input sorted (lexically ascending) by the sort keys on both the left and right files. This design decision was made intentionally to parallel the Unix/Linux system ``join`` command, which has the same semantics. The benefit of this default is that the joiner program can stream through the left and right files, needing to load neither entirely into memory. The drawback, of course, is that is requires sorted input.
The issue is that Miller's `join`, by default (before 5.1.0), took input sorted (lexically ascending) by the sort keys on both the left and right files. This design decision was made intentionally to parallel the Unix/Linux system `join` command, which has the same semantics. The benefit of this default is that the joiner program can stream through the left and right files, needing to load neither entirely into memory. The drawback, of course, is that is requires sorted input.
The solution (besides pre-sorting the input files on the join keys) is to simply use **mlr join -u** (which is now the default). This loads the left file entirely into memory (while the right file is still streamed one line at a time) and does all possible joins without requiring sorted input:
<pre>
<pre class="pre-highlight">
<b>mlr --icsvlite --opprint join -u -j ipaddr -f data/join-u-left.csv data/join-u-right.csv</b>
</pre>
<pre class="pre-non-highlight">
ipaddr hostname timestamp bytes
10.3.1.27 zenith.west.our.org 1448762579 4568
10.3.1.18 nadir.east.our.org 1448762578 8729
@ -62,14 +70,14 @@ General advice is to make sure the left-file is relatively small, e.g. containin
Suppose you have the following two data files:
<pre>
<pre class="pre-non-highlight">
id,code
3,0000ff
2,00ff00
4,ff0000
</pre>
<pre>
<pre class="pre-non-highlight">
id,color
4,red
2,green
@ -77,17 +85,21 @@ id,color
Joining on color the results are as expected:
<pre>
<pre class="pre-highlight">
<b>mlr --csv join -j id -f data/color-codes.csv data/color-names.csv</b>
</pre>
<pre class="pre-non-highlight">
id,code,color
4,ff0000,red
2,00ff00,green
</pre>
However, if we ask for left-unpaireds, since there's no ``color`` column, we get a row not having the same column names as the other:
However, if we ask for left-unpaireds, since there's no `color` column, we get a row not having the same column names as the other:
<pre>
<pre class="pre-highlight">
<b>mlr --csv join --ul -j id -f data/color-codes.csv data/color-names.csv</b>
</pre>
<pre class="pre-non-highlight">
id,code,color
4,ff0000,red
2,00ff00,green
@ -98,10 +110,12 @@ id,code
To fix this, we can use **unsparsify**:
<pre>
<pre class="pre-highlight">
<b>mlr --csv join --ul -j id -f data/color-codes.csv \</b>
<b> then unsparsify --fill-with "" \</b>
<b> data/color-names.csv</b>
</pre>
<pre class="pre-non-highlight">
id,code,color
4,ff0000,red
2,00ff00,green
@ -114,8 +128,10 @@ Thanks to @aborruso for the tip!
Suppose we have the following data:
<pre>
<pre class="pre-highlight">
<b>cat multi-join/input.csv</b>
</pre>
<pre class="pre-non-highlight">
id,task
10,chop
20,puree
@ -127,30 +143,36 @@ id,task
30,clean
</pre>
And we want to augment the ``id`` column with lookups from the following data files:
And we want to augment the `id` column with lookups from the following data files:
<pre>
<pre class="pre-highlight">
<b>cat multi-join/name-lookup.csv</b>
</pre>
<pre class="pre-non-highlight">
id,name
30,Alice
10,Bob
20,Carol
</pre>
<pre>
<pre class="pre-highlight">
<b>cat multi-join/status-lookup.csv</b>
</pre>
<pre class="pre-non-highlight">
id,status
30,occupied
10,idle
20,idle
</pre>
We can run the input file through multiple ``join`` commands in a ``then``-chain:
We can run the input file through multiple `join` commands in a `then`-chain:
<pre>
<pre class="pre-highlight">
<b>mlr --icsv --opprint join -f multi-join/name-lookup.csv -j id \</b>
<b> then join -f multi-join/status-lookup.csv -j id \</b>
<b> multi-join/input.csv</b>
</pre>
<pre class="pre-non-highlight">
id status name task
10 idle Bob chop
20 idle Carol puree

10
docs6b/docs/joins.md.in
View file

@ -4,7 +4,7 @@
**This section describes behavior before Miller 5.1.0. As of 5.1.0, -u is the default.**
For example, the right file here has nine records, and the left file should add in the ``hostname`` column -- so the join output should also have 9 records:
For example, the right file here has nine records, and the left file should add in the `hostname` column -- so the join output should also have 9 records:
GENMD_RUN_COMMAND
mlr --icsvlite --opprint cat data/join-u-left.csv
@ -18,7 +18,7 @@ GENMD_RUN_COMMAND
mlr --icsvlite --opprint join -s -j ipaddr -f data/join-u-left.csv data/join-u-right.csv
GENMD_EOF
The issue is that Miller's ``join``, by default (before 5.1.0), took input sorted (lexically ascending) by the sort keys on both the left and right files. This design decision was made intentionally to parallel the Unix/Linux system ``join`` command, which has the same semantics. The benefit of this default is that the joiner program can stream through the left and right files, needing to load neither entirely into memory. The drawback, of course, is that is requires sorted input.
The issue is that Miller's `join`, by default (before 5.1.0), took input sorted (lexically ascending) by the sort keys on both the left and right files. This design decision was made intentionally to parallel the Unix/Linux system `join` command, which has the same semantics. The benefit of this default is that the joiner program can stream through the left and right files, needing to load neither entirely into memory. The drawback, of course, is that is requires sorted input.
The solution (besides pre-sorting the input files on the join keys) is to simply use **mlr join -u** (which is now the default). This loads the left file entirely into memory (while the right file is still streamed one line at a time) and does all possible joins without requiring sorted input:
@ -42,7 +42,7 @@ GENMD_RUN_COMMAND
mlr --csv join -j id -f data/color-codes.csv data/color-names.csv
GENMD_EOF
However, if we ask for left-unpaireds, since there's no ``color`` column, we get a row not having the same column names as the other:
However, if we ask for left-unpaireds, since there's no `color` column, we get a row not having the same column names as the other:
GENMD_RUN_COMMAND
mlr --csv join --ul -j id -f data/color-codes.csv data/color-names.csv
@ -66,7 +66,7 @@ GENMD_RUN_COMMAND
cat multi-join/input.csv
GENMD_EOF
And we want to augment the ``id`` column with lookups from the following data files:
And we want to augment the `id` column with lookups from the following data files:
GENMD_RUN_COMMAND
cat multi-join/name-lookup.csv
@ -76,7 +76,7 @@ GENMD_RUN_COMMAND
cat multi-join/status-lookup.csv
GENMD_EOF
We can run the input file through multiple ``join`` commands in a ``then``-chain:
We can run the input file through multiple `join` commands in a `then`-chain:
GENMD_RUN_COMMAND
mlr --icsv --opprint join -f multi-join/name-lookup.csv -j id \

26
docs6b/docs/keystroke-savers.md
View file

@ -3,17 +3,21 @@
## Short format specifiers
In our examples so far we've often made use of ``mlr --icsv --opprint`` or ``mlr --icsv --ojson``. These are such frequently occurring patterns that they have short options like **--c2p** and **--c2j**:
In our examples so far we've often made use of `mlr --icsv --opprint` or `mlr --icsv --ojson`. These are such frequently occurring patterns that they have short options like **--c2p** and **--c2j**:
<pre>
<pre class="pre-highlight">
<b>mlr --c2p head -n 2 example.csv</b>
</pre>
<pre class="pre-non-highlight">
color shape flag index quantity rate
yellow triangle true 11 43.6498 9.8870
red square true 15 79.2778 0.0130
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --c2j head -n 2 example.csv</b>
</pre>
<pre class="pre-non-highlight">
{
"color": "yellow",
"shape": "triangle",
@ -36,32 +40,36 @@ You can get the full list here (TODO:linkify).
## File names up front
Already we saw that you can put the filename first using ``--from``. When you're interacting with your data at the command line, this makes it easier to up-arrow and append to the previous command:
Already we saw that you can put the filename first using `--from`. When you're interacting with your data at the command line, this makes it easier to up-arrow and append to the previous command:
<pre>
<pre class="pre-highlight">
<b>mlr --c2p --from example.csv sort -nr index then head -n 3</b>
</pre>
<pre class="pre-non-highlight">
color shape flag index quantity rate
purple square false 91 72.3735 8.2430
yellow circle true 87 63.5058 8.3350
yellow circle true 73 63.9785 4.2370
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --c2p --from example.csv sort -nr index then head -n 3 then cut -f shape,quantity</b>
</pre>
<pre class="pre-non-highlight">
shape quantity
square 72.3735
circle 63.5058
circle 63.9785
</pre>
If there's more than one input file, you can use ``--mfrom``, then however many file names, then ``--`` to indicate the end of your input-file-name list:
If there's more than one input file, you can use `--mfrom`, then however many file names, then `--` to indicate the end of your input-file-name list:
<pre>
<pre class="pre-highlight">
<b>mlr --c2p --mfrom data/*.csv -- sort -n index</b>
</pre>
## .mlrrc file
If you want the default file format for Miller to be CSV you can simply put ``--csv`` on a line by itself in your ``~/.mlrrc`` file. Then instead of ``mlr --csv cat example.csv`` you can just do ``mlr cat example.csv``. This is just the default, though, so ``mlr --opprint cat example.csv`` will still use default CSV format for input, and PPRINT (tabular) for output.
If you want the default file format for Miller to be CSV you can simply put `--csv` on a line by itself in your `~/.mlrrc` file. Then instead of `mlr --csv cat example.csv` you can just do `mlr cat example.csv`. This is just the default, though, so `mlr --opprint cat example.csv` will still use default CSV format for input, and PPRINT (tabular) for output.
You can read more about this at the [Customization](customization.md) page.

8
docs6b/docs/keystroke-savers.md.in
View file

@ -2,7 +2,7 @@
## Short format specifiers
In our examples so far we've often made use of ``mlr --icsv --opprint`` or ``mlr --icsv --ojson``. These are such frequently occurring patterns that they have short options like **--c2p** and **--c2j**:
In our examples so far we've often made use of `mlr --icsv --opprint` or `mlr --icsv --ojson`. These are such frequently occurring patterns that they have short options like **--c2p** and **--c2j**:
GENMD_RUN_COMMAND
mlr --c2p head -n 2 example.csv
@ -16,7 +16,7 @@ You can get the full list here (TODO:linkify).
## File names up front
Already we saw that you can put the filename first using ``--from``. When you're interacting with your data at the command line, this makes it easier to up-arrow and append to the previous command:
Already we saw that you can put the filename first using `--from`. When you're interacting with your data at the command line, this makes it easier to up-arrow and append to the previous command:
GENMD_RUN_COMMAND
mlr --c2p --from example.csv sort -nr index then head -n 3
@ -26,7 +26,7 @@ GENMD_RUN_COMMAND
mlr --c2p --from example.csv sort -nr index then head -n 3 then cut -f shape,quantity
GENMD_EOF
If there's more than one input file, you can use ``--mfrom``, then however many file names, then ``--`` to indicate the end of your input-file-name list:
If there's more than one input file, you can use `--mfrom`, then however many file names, then `--` to indicate the end of your input-file-name list:
GENMD_SHOW_COMMAND
mlr --c2p --mfrom data/*.csv -- sort -n index
@ -34,6 +34,6 @@ GENMD_EOF
## .mlrrc file
If you want the default file format for Miller to be CSV you can simply put ``--csv`` on a line by itself in your ``~/.mlrrc`` file. Then instead of ``mlr --csv cat example.csv`` you can just do ``mlr cat example.csv``. This is just the default, though, so ``mlr --opprint cat example.csv`` will still use default CSV format for input, and PPRINT (tabular) for output.
If you want the default file format for Miller to be CSV you can simply put `--csv` on a line by itself in your `~/.mlrrc` file. Then instead of `mlr --csv cat example.csv` you can just do `mlr cat example.csv`. This is just the default, though, so `mlr --opprint cat example.csv` will still use default CSV format for input, and PPRINT (tabular) for output.
You can read more about this at the [Customization](customization.md) page.

30
docs6b/docs/log-processing-examples.md
View file

@ -1,7 +1,7 @@
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
# Log-processing examples
Another of my favorite use-cases for Miller is doing ad-hoc processing of log-file data. Here's where DKVP format really shines: one, since the field names and field values are present on every line, every line stands on its own. That means you can ``grep`` or what have you. Also it means not every line needs to have the same list of field names ("schema").
Another of my favorite use-cases for Miller is doing ad-hoc processing of log-file data. Here's where DKVP format really shines: one, since the field names and field values are present on every line, every line stands on its own. That means you can `grep` or what have you. Also it means not every line needs to have the same list of field names ("schema").
## Generating and aggregating log-file output
@ -11,8 +11,10 @@ Writing a program -- in any language whatsoever -- you can have it print out log
Suppose your program has printed something like this [log.txt](./log.txt):
<pre>
<pre class="pre-highlight">
<b>cat log.txt</b>
</pre>
<pre class="pre-non-highlight">
op=enter,time=1472819681
op=cache,type=A9,hit=0
op=cache,type=A4,hit=1
@ -58,22 +60,26 @@ op=cache,type=A9,hit=0
time=1472819742,batch_size=100,num_filtered=728
</pre>
Each print statement simply contains local information: the current timestamp, whether a particular cache was hit or not, etc. Then using either the system ``grep`` command, or Miller's ``having-fields``, or ``is_present``, we can pick out the parts we want and analyze them:
Each print statement simply contains local information: the current timestamp, whether a particular cache was hit or not, etc. Then using either the system `grep` command, or Miller's `having-fields`, or `is_present`, we can pick out the parts we want and analyze them:
<pre>
<pre class="pre-highlight">
<b>grep op=cache log.txt \</b>
<b> | mlr --idkvp --opprint stats1 -a mean -f hit -g type then sort -f type</b>
</pre>
<pre class="pre-non-highlight">
type hit_mean
A1 0.8571428571428571
A4 0.7142857142857143
A9 0.09090909090909091
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --from log.txt --opprint \</b>
<b> filter 'is_present($batch_size)' \</b>
<b> then step -a delta -f time,num_filtered \</b>
<b> then sec2gmt time</b>
</pre>
<pre class="pre-non-highlight">
time batch_size num_filtered time_delta num_filtered_delta
2016-09-02T12:34:50Z 100 237 0 0
2016-09-02T12:35:05Z 100 348 15 111
@ -85,8 +91,10 @@ time batch_size num_filtered time_delta num_filtered_delta
Alternatively, we can simply group the similar data for a better look:
<pre>
<pre class="pre-highlight">
<b>mlr --opprint group-like log.txt</b>
</pre>
<pre class="pre-non-highlight">
op time
enter 1472819681
@ -137,8 +145,10 @@ time batch_size num_filtered
1472819742 100 728
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --opprint group-like then sec2gmt time log.txt</b>
</pre>
<pre class="pre-non-highlight">
op time
enter 2016-09-02T12:34:41Z
@ -193,13 +203,13 @@ time batch_size num_filtered
This, of course, depends highly on what's in your log files. But, as an example, suppose you have log-file lines such as
<pre>
<pre class="pre-non-highlight">
2015-10-08 08:29:09,445 INFO com.company.path.to.ClassName @ [sometext] various/sorts/of data {& punctuation} hits=1 status=0 time=2.378
</pre>
I prefer to pre-filter with ``grep`` and/or ``sed`` to extract the structured text, then hand that to Miller. Example:
I prefer to pre-filter with `grep` and/or `sed` to extract the structured text, then hand that to Miller. Example:
<pre>
<pre class="pre-highlight">
<b>grep 'various sorts' *.log \</b>
<b> | sed 's/.*} //' \</b>
<b> | mlr --fs space --repifs --oxtab stats1 -a min,p10,p50,p90,max -f time -g status</b>

6
docs6b/docs/log-processing-examples.md.in
View file

@ -1,6 +1,6 @@
# Log-processing examples
Another of my favorite use-cases for Miller is doing ad-hoc processing of log-file data. Here's where DKVP format really shines: one, since the field names and field values are present on every line, every line stands on its own. That means you can ``grep`` or what have you. Also it means not every line needs to have the same list of field names ("schema").
Another of my favorite use-cases for Miller is doing ad-hoc processing of log-file data. Here's where DKVP format really shines: one, since the field names and field values are present on every line, every line stands on its own. That means you can `grep` or what have you. Also it means not every line needs to have the same list of field names ("schema").
## Generating and aggregating log-file output
@ -14,7 +14,7 @@ GENMD_RUN_COMMAND
cat log.txt
GENMD_EOF
Each print statement simply contains local information: the current timestamp, whether a particular cache was hit or not, etc. Then using either the system ``grep`` command, or Miller's ``having-fields``, or ``is_present``, we can pick out the parts we want and analyze them:
Each print statement simply contains local information: the current timestamp, whether a particular cache was hit or not, etc. Then using either the system `grep` command, or Miller's `having-fields`, or `is_present`, we can pick out the parts we want and analyze them:
GENMD_INCLUDE_AND_RUN_ESCAPED(10-1.sh)
@ -38,7 +38,7 @@ GENMD_CARDIFY
2015-10-08 08:29:09,445 INFO com.company.path.to.ClassName @ [sometext] various/sorts/of data {& punctuation} hits=1 status=0 time=2.378
GENMD_EOF
I prefer to pre-filter with ``grep`` and/or ``sed`` to extract the structured text, then hand that to Miller. Example:
I prefer to pre-filter with `grep` and/or `sed` to extract the structured text, then hand that to Miller. Example:
GENMD_SHOW_COMMAND
grep 'various sorts' *.log \

2
docs6b/docs/manpage.md
View file

@ -3,7 +3,7 @@
This is simply a copy of what you should see on running **man mlr** at a command prompt, once Miller is installed on your system.
<pre>
<pre class="pre-non-highlight">
MILLER(1) MILLER(1)

14
docs6b/docs/miller-on-windows.md
View file

@ -7,17 +7,17 @@ Miller was originally developed for Unix-like operating systems including Linux
The experience is now almost the same as on Linux, NetBSD/FreeBSD, and MacOS.
MSYS2 is no longer required, although you can use Miller from within MSYS2 if you like. There is now simply a single ``mlr.exe``, with no ``msys2.dll`` alongside anymore.
MSYS2 is no longer required, although you can use Miller from within MSYS2 if you like. There is now simply a single `mlr.exe`, with no `msys2.dll` alongside anymore.
See [Installation](installation.md) for how to get a copy of ``mlr.exe``.
See [Installation](installation.md) for how to get a copy of `mlr.exe`.
## Setup
Simply place ``mlr.exe`` somewhere within your ``PATH`` variable.
Simply place `mlr.exe` somewhere within your `PATH` variable.
![pix/miller-windows.png](pix/miller-windows.png)
To use Miller from within MSYS2/Cygwin, also make sure ``mlr.exe`` is within the ``PATH`` variable.
To use Miller from within MSYS2/Cygwin, also make sure `mlr.exe` is within the `PATH` variable.
![pix/miller-msys.png](pix/miller-msys.png)
@ -25,18 +25,18 @@ To use Miller from within MSYS2/Cygwin, also make sure ``mlr.exe`` is within the
[Output Colorization](output-colorization.md) doesn't work on Windows, outside of MSYS2.
The Windows-support code within Miller makes effort to support Linux/Unix/MacOS-like command-line syntax including single-quoting of expressions for ``mlr put`` and ``mlr filter`` -- and in the examples above, this often works. However, there are still some cases where more complex expressions aren't successfully parsed from the Windows prompt, even though they are from MSYS2:
The Windows-support code within Miller makes effort to support Linux/Unix/MacOS-like command-line syntax including single-quoting of expressions for `mlr put` and `mlr filter` -- and in the examples above, this often works. However, there are still some cases where more complex expressions aren't successfully parsed from the Windows prompt, even though they are from MSYS2:
![pix/miller-windows-complex.png](pix/miller-windows-complex.png)
![pix/miller-msys-complex.png](pix/miller-msys-complex.png)
Single quotes with ``&&`` or ``||`` inside are fundamentally unhandleable within Windows; there is nothing Miller can do here as the Windows command line is split before Miller ever receives it.
Single quotes with `&&` or `||` inside are fundamentally unhandleable within Windows; there is nothing Miller can do here as the Windows command line is split before Miller ever receives it.
One workaround is to use MSYS2. Another workaround is to put more complex DSL expressions into a file:
![pix/miller-windows-complex-workaround.png](pix/miller-windows-complex-workaround.png)
A third workaround is to replace ``"`` with ``"""``, then ``'`` with ``"``:
A third workaround is to replace `"` with `"""`, then `'` with `"`:
![pix/miller-windows-triple-double-quote.png](pix/miller-windows-triple-double-quote.png)

14
docs6b/docs/miller-on-windows.md.in
View file

@ -6,17 +6,17 @@ Miller was originally developed for Unix-like operating systems including Linux
The experience is now almost the same as on Linux, NetBSD/FreeBSD, and MacOS.
MSYS2 is no longer required, although you can use Miller from within MSYS2 if you like. There is now simply a single ``mlr.exe``, with no ``msys2.dll`` alongside anymore.
MSYS2 is no longer required, although you can use Miller from within MSYS2 if you like. There is now simply a single `mlr.exe`, with no `msys2.dll` alongside anymore.
See [Installation](installation.md) for how to get a copy of ``mlr.exe``.
See [Installation](installation.md) for how to get a copy of `mlr.exe`.
## Setup
Simply place ``mlr.exe`` somewhere within your ``PATH`` variable.
Simply place `mlr.exe` somewhere within your `PATH` variable.
![pix/miller-windows.png](pix/miller-windows.png)
To use Miller from within MSYS2/Cygwin, also make sure ``mlr.exe`` is within the ``PATH`` variable.
To use Miller from within MSYS2/Cygwin, also make sure `mlr.exe` is within the `PATH` variable.
![pix/miller-msys.png](pix/miller-msys.png)
@ -24,18 +24,18 @@ To use Miller from within MSYS2/Cygwin, also make sure ``mlr.exe`` is within the
[Output Colorization](output-colorization.md) doesn't work on Windows, outside of MSYS2.
The Windows-support code within Miller makes effort to support Linux/Unix/MacOS-like command-line syntax including single-quoting of expressions for ``mlr put`` and ``mlr filter`` -- and in the examples above, this often works. However, there are still some cases where more complex expressions aren't successfully parsed from the Windows prompt, even though they are from MSYS2:
The Windows-support code within Miller makes effort to support Linux/Unix/MacOS-like command-line syntax including single-quoting of expressions for `mlr put` and `mlr filter` -- and in the examples above, this often works. However, there are still some cases where more complex expressions aren't successfully parsed from the Windows prompt, even though they are from MSYS2:
![pix/miller-windows-complex.png](pix/miller-windows-complex.png)
![pix/miller-msys-complex.png](pix/miller-msys-complex.png)
Single quotes with ``&&`` or ``||`` inside are fundamentally unhandleable within Windows; there is nothing Miller can do here as the Windows command line is split before Miller ever receives it.
Single quotes with `&&` or `||` inside are fundamentally unhandleable within Windows; there is nothing Miller can do here as the Windows command line is split before Miller ever receives it.
One workaround is to use MSYS2. Another workaround is to put more complex DSL expressions into a file:
![pix/miller-windows-complex-workaround.png](pix/miller-windows-complex-workaround.png)
A third workaround is to replace ``"`` with ``"""``, then ``'`` with ``"``:
A third workaround is to replace `"` with `"""`, then `'` with `"`:
![pix/miller-windows-triple-double-quote.png](pix/miller-windows-triple-double-quote.png)

74
docs6b/docs/misc-examples.md
View file

@ -3,62 +3,64 @@
Column select:
<pre>
<pre class="pre-highlight">
<b>mlr --csv cut -f hostname,uptime mydata.csv</b>
</pre>
Add new columns as function of other columns:
<pre>
<pre class="pre-highlight">
<b>mlr --nidx put '$sum = $7 < 0.0 ? 3.5 : $7 + 2.1*$8' *.dat</b>
</pre>
Row filter:
<pre>
<pre class="pre-highlight">
<b>mlr --csv filter '$status != "down" && $upsec >= 10000' *.csv</b>
</pre>
Apply column labels and pretty-print:
<pre>
<pre class="pre-highlight">
<b>grep -v '^#' /etc/group | mlr --ifs : --nidx --opprint label group,pass,gid,member then sort -f group</b>
</pre>
Join multiple data sources on key columns:
<pre>
<pre class="pre-highlight">
<b>mlr join -j account_id -f accounts.dat then group-by account_name balances.dat</b>
</pre>
Mulltiple formats including JSON:
<pre>
<pre class="pre-highlight">
<b>mlr --json put '$attr = sub($attr, "([0-9]+)_([0-9]+)_.*", "\1:\2")' data/*.json</b>
</pre>
Aggregate per-column statistics:
<pre>
<pre class="pre-highlight">
<b>mlr stats1 -a min,mean,max,p10,p50,p90 -f flag,u,v data/*</b>
</pre>
Linear regression:
<pre>
<pre class="pre-highlight">
<b>mlr stats2 -a linreg-pca -f u,v -g shape data/*</b>
</pre>
Aggregate custom per-column statistics:
<pre>
<pre class="pre-highlight">
<b>mlr put -q '@sum[$a][$b] += $x; end {emit @sum, "a", "b"}' data/*</b>
</pre>
Iterate over data using DSL expressions:
<pre>
<pre class="pre-highlight">
<b>mlr --from estimates.tbl put '</b>
</pre>
<pre class="pre-non-highlight">
for (k,v in $*) {
if (is_numeric(v) && k =~ "^[t-z].*$") {
$sum += v; $count += 1
@ -70,39 +72,39 @@ Iterate over data using DSL expressions:
Run DSL expressions from a script file:
<pre>
<pre class="pre-highlight">
<b>mlr --from infile.dat put -f analyze.mlr</b>
</pre>
Split/reduce output to multiple filenames:
<pre>
<pre class="pre-highlight">
<b>mlr --from infile.dat put 'tee > "./taps/data-".$a."-".$b, $*'</b>
</pre>
Compressed I/O:
<pre>
<pre class="pre-highlight">
<b>mlr --from infile.dat put 'tee | "gzip > ./taps/data-".$a."-".$b.".gz", $*'</b>
</pre>
Interoperate with other data-processing tools using standard pipes:
<pre>
<pre class="pre-highlight">
<b>mlr --from infile.dat put -q '@v=$*; dump | "jq .[]"'</b>
</pre>
Tap/trace:
<pre>
<pre class="pre-highlight">
<b>mlr --from infile.dat put '(NR % 1000 == 0) { print > stderr, "Checkpoint ".NR}'</b>
</pre>
## Program timing
This admittedly artificial example demonstrates using Miller time and stats functions to introspectively acquire some information about Miller's own runtime. The ``delta`` function computes the difference between successive timestamps.
This admittedly artificial example demonstrates using Miller time and stats functions to introspectively acquire some information about Miller's own runtime. The `delta` function computes the difference between successive timestamps.
<pre>
<pre class="pre-non-highlight">
$ ruby -e '10000.times{|i|puts "i=#{i+1}"}' > lines.txt
$ head -n 5 lines.txt
@ -136,8 +138,10 @@ t_delta_max 5.388259888e-05
Suppose you have a database query which you run at one point in time, producing the output on the left, then again later producing the output on the right:
<pre>
<pre class="pre-highlight">
<b>cat data/previous_counters.csv</b>
</pre>
<pre class="pre-non-highlight">
color,count
red,3472
blue,6838
@ -145,8 +149,10 @@ orange,694
purple,12
</pre>
<pre>
<pre class="pre-highlight">
<b>cat data/current_counters.csv</b>
</pre>
<pre class="pre-non-highlight">
color,count
red,3467
orange,670
@ -158,12 +164,14 @@ And, suppose you want to compute the differences in the counters between adjacen
First, rename counter columns to make them distinct:
<pre>
<pre class="pre-highlight">
<b>mlr --csv rename count,previous_count data/previous_counters.csv > data/prevtemp.csv</b>
</pre>
<pre>
<pre class="pre-highlight">
<b>cat data/prevtemp.csv</b>
</pre>
<pre class="pre-non-highlight">
color,previous_count
red,3472
blue,6838
@ -171,12 +179,14 @@ orange,694
purple,12
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --csv rename count,current_count data/current_counters.csv > data/currtemp.csv</b>
</pre>
<pre>
<pre class="pre-highlight">
<b>cat data/currtemp.csv</b>
</pre>
<pre class="pre-non-highlight">
color,current_count
red,3467
orange,670
@ -184,14 +194,16 @@ yellow,27
blue,6944
</pre>
Then, join on the key field(s), and use unsparsify to zero-fill counters absent on one side but present on the other. Use ``--ul`` and ``--ur`` to emit unpaired records (namely, purple on the left and yellow on the right):
Then, join on the key field(s), and use unsparsify to zero-fill counters absent on one side but present on the other. Use `--ul` and `--ur` to emit unpaired records (namely, purple on the left and yellow on the right):
<pre>
<pre class="pre-highlight">
<b>mlr --icsv --opprint \</b>
<b> join -j color --ul --ur -f data/prevtemp.csv \</b>
<b> then unsparsify --fill-with 0 \</b>
<b> then put '$count_delta = $current_count - $previous_count' \</b>
<b> data/currtemp.csv</b>
</pre>
<pre class="pre-non-highlight">
color previous_count current_count count_delta
red 3472 3467 -5
orange 694 670 -24
@ -200,13 +212,11 @@ blue 6838 6944 106
purple 12 0 (error)
</pre>
.. _cookbook-memoization-with-oosvars:
## Memoization with out-of-stream variables
The recursive function for the Fibonacci sequence is famous for its computational complexity. Namely, using f(0)=1, f(1)=1, f(n)=f(n-1)+f(n-2) for n>=2, the evaluation tree branches left as well as right at each non-trivial level, resulting in millions or more paths to the root 0/1 nodes for larger n. This program
<pre>
<pre class="pre-non-highlight">
mlr --ofmt '%.9lf' --opprint seqgen --start 1 --stop 28 then put '
func f(n) {
@fcount += 1; # count number of calls to the function
@ -226,7 +236,7 @@ mlr --ofmt '%.9lf' --opprint seqgen --start 1 --stop 28 then put '
produces output like this:
<pre>
<pre class="pre-non-highlight">
i o fcount seconds_delta
1 1 1 0
2 2 3 0.000039101
@ -258,9 +268,9 @@ i o fcount seconds_delta
28 514229 1028457 0.971235037
</pre>
Note that the time it takes to evaluate the function is blowing up exponentially as the input argument increases. Using ``@``-variables, which persist across records, we can cache and reuse the results of previous computations:
Note that the time it takes to evaluate the function is blowing up exponentially as the input argument increases. Using `@`-variables, which persist across records, we can cache and reuse the results of previous computations:
<pre>
<pre class="pre-non-highlight">
mlr --ofmt '%.9lf' --opprint seqgen --start 1 --stop 28 then put '
func f(n) {
@fcount += 1; # count number of calls to the function
@ -283,7 +293,7 @@ mlr --ofmt '%.9lf' --opprint seqgen --start 1 --stop 28 then put '
with output like this:
<pre>
<pre class="pre-non-highlight">
i o fcount seconds_delta
1 1 1 0
2 2 3 0.000053883

8
docs6b/docs/misc-examples.md.in
View file

@ -99,7 +99,7 @@ GENMD_EOF
## Program timing
This admittedly artificial example demonstrates using Miller time and stats functions to introspectively acquire some information about Miller's own runtime. The ``delta`` function computes the difference between successive timestamps.
This admittedly artificial example demonstrates using Miller time and stats functions to introspectively acquire some information about Miller's own runtime. The `delta` function computes the difference between successive timestamps.
GENMD_INCLUDE_ESCAPED(data/timing-example.txt)
@ -135,12 +135,10 @@ GENMD_RUN_COMMAND
cat data/currtemp.csv
GENMD_EOF
Then, join on the key field(s), and use unsparsify to zero-fill counters absent on one side but present on the other. Use ``--ul`` and ``--ur`` to emit unpaired records (namely, purple on the left and yellow on the right):
Then, join on the key field(s), and use unsparsify to zero-fill counters absent on one side but present on the other. Use `--ul` and `--ur` to emit unpaired records (namely, purple on the left and yellow on the right):
GENMD_INCLUDE_AND_RUN_ESCAPED(data/previous-to-current.sh)
.. _cookbook-memoization-with-oosvars:
## Memoization with out-of-stream variables
The recursive function for the Fibonacci sequence is famous for its computational complexity. Namely, using f(0)=1, f(1)=1, f(n)=f(n-1)+f(n-2) for n>=2, the evaluation tree branches left as well as right at each non-trivial level, resulting in millions or more paths to the root 0/1 nodes for larger n. This program
@ -181,7 +179,7 @@ i o fcount seconds_delta
28 514229 1028457 0.971235037
GENMD_EOF
Note that the time it takes to evaluate the function is blowing up exponentially as the input argument increases. Using ``@``-variables, which persist across records, we can cache and reuse the results of previous computations:
Note that the time it takes to evaluate the function is blowing up exponentially as the input argument increases. Using `@`-variables, which persist across records, we can cache and reuse the results of previous computations:
GENMD_INCLUDE_ESCAPED(data/fibo-cached.sh)

40
docs6b/docs/mk-func-h2s.sh
View file

@ -47,16 +47,20 @@ mlr help list-functions | grep -v '^[a-zA-Z]' | uniq | while read funcname; do
linkname='ursheq'
fi
echo ""
echo ".. _reference-dsl-${linkname}:"
echo ""
echo "$displayname"
echo "^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^"
echo ""
echo '.. code-block:: none'
# TODO: fix section-links for mnkdocs
#echo ''
#echo ".. _reference-dsl-${linkname}:"
echo ''
mlr help function "$funcname" | sed 's/^/ /'
if [ "$linkname" = "$displayname" ]; then
echo "## $displayname"
else
echo "<a id=$linkname />"
echo "## $displayname"
fi
echo ''
echo '<pre class="pre-non-highlight">'
mlr help function "$funcname"
echo '</pre>'
echo ''
done
@ -101,15 +105,19 @@ mlr help list-functions | grep '^[a-zA-Z]' | sort -u | while read funcname; do
linkname='ursheq'
fi
echo ""
echo ".. _reference-dsl-${linkname}:"
echo ""
echo "$displayname"
echo "^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^"
echo ""
echo '.. code-block:: none'
# TODO: fix section-links for mnkdocs
#echo ''
#echo ".. _reference-dsl-${linkname}:"
echo ''
mlr help function "$funcname" | sed 's/^/ /'
if [ "$linkname" = "$displayname" ]; then
echo "## $displayname"
else
echo "<a id=$linkname />"
echo "## $displayname"
fi
echo ''
echo '<pre class="pre-non-highlight">'
mlr help function "$funcname"
echo '</pre>'
echo ''
done

54
docs6b/docs/new-in-miller-6.md
View file

@ -5,10 +5,10 @@ See also the [list of issues tagged with go-port](https://github.com/johnkerl/mi
## Documentation improvements
Documentation (what you're reading here) and on-line help (``mlr --help``) have been completely reworked.
Documentation (what you're reading here) and on-line help (`mlr --help`) have been completely reworked.
In the initial release, the focus was convincing users already familiar with
``awk``/``grep``/``cut`` that Miller was a viable option. Over time it's become
`awk`/`grep`/`cut` that Miller was a viable option. Over time it's become
clear that many users aren't expert with these. The focus has shifted toward a
higher quantity of more introductory/accessible material for command-line data
processing.
@ -25,14 +25,14 @@ now discussed first, and more examples use CSV.
## JSON support, and arrays
Arrays are now supported in Miller's ``put``/``filter`` programming language,
as described at :doc:`reference-dsl-arrays`. Also, ``array`` is now a keyword
Arrays are now supported in Miller's `put`/`filter` programming language,
as described at [Reference: arrays](reference-dsl-arrays.md). Also, `array` is now a keyword
so this is no longer usable as a local-variable or UDF name.
JSON support is improved:
* Direct support for arrays means that you can now use Miller to process more JSON files.
* Streamable JSON parsing: Miller's internal record-processing pipeline starts as soon as the first record is read (which was already the case for other file formats). This means that, unless records are wrapped with outermost ``[...]``, Miller now handles JSON in ``tail -f`` contexts like it does for other file formats.
* Streamable JSON parsing: Miller's internal record-processing pipeline starts as soon as the first record is read (which was already the case for other file formats). This means that, unless records are wrapped with outermost `[...]`, Miller now handles JSON in `tail -f` contexts like it does for other file formats.
* Flatten/unflatten -- TODO pick a name and link to a separate page/section
## Improved Windows experience
@ -44,11 +44,7 @@ Binaries are reliably available using GitHub Actions: see also [Installation](in
## In-process support for compressed input
In addition to ``--prepipe gunzip``, you can now use the ``--gzin`` flag. In
fact, if your files end in ``.gz`` you don't even need to do that -- Miller
will autodetect by file extension and automatically uncompress ``mlr --csv cat
foo.csv.gz``. Similarly for ``.z`` and ``.bz2`` files. Please see section
[TODO:linkify] for more information.
In addition to `--prepipe gunzip`, you can now use the `--gzin` flag. In fact, if your files end in `.gz` you don't even need to do that -- Miller will autodetect by file extension and automatically uncompress `mlr --csv cat foo.csv.gz`. Similarly for `.z` and `.bz2` files. Please see section [TODO:linkify] for more information.
## Output colorization
@ -60,12 +56,12 @@ The most central part of Miller 6 is a deep refactor of how data values are pars
from file contents, how types are inferred, and how they're converted back to
text into output files.
This was all initiated by https://github.com/johnkerl/miller/issues/151.
This was all initiated by [https://github.com/johnkerl/miller/issues/151](https://github.com/johnkerl/miller/issues/151).
In Miller 5 and below, all values were stored as strings, then only converted
to int/float as-needed, for example when a particular field was referenced in
the ``stats1`` or ``put`` verbs. This led to awkwardnesses such as the ``-S``
and ``-F`` flags for ``put`` and ``filter``.
the `stats1` or `put` verbs. This led to awkwardnesses such as the `-S`
and `-F` flags for `put` and `filter`.
In Miller 6, things parseable as int/float are treated as such from the moment
the input data is read, and these are passed along through the verb chain. All
@ -75,17 +71,21 @@ a numeric field isn't modified during the processing chain, it's printed out
the way it arrived. Also, quoted values in JSON strings are flagged as being
strings throughout the processing chain.
For example (see https://github.com/johnkerl/miller/issues/178) you can now do
For example (see [https://github.com/johnkerl/miller/issues/178](https://github.com/johnkerl/miller/issues/178)) you can now do
<pre>
<pre class="pre-highlight">
<b>echo '{ "a": "0123" }' | mlr --json cat</b>
</pre>
<pre class="pre-non-highlight">
{
"a": "0123"
}
</pre>
<pre>
<pre class="pre-highlight">
<b>echo '{ "x": 1.230, "y": 1.230000000 }' | mlr --json cat</b>
</pre>
<pre class="pre-non-highlight">
{
"x": 1.230,
"y": 1.230000000
@ -98,28 +98,30 @@ Miller now has a read-evaluate-print-loop ([REPL](repl.md)) where you can single
## New DSL functions / operators
* String-hashing functions :ref:`reference-dsl-md5`, :ref:`reference-dsl-sha1`, :ref:`reference-dsl-sha256`, and :ref:`reference-dsl-sha512`.
* Platform-property functions :ref:`reference-dsl-hostname`, :ref:`reference-dsl-os`, and :ref:`reference-dsl-version`.
* Unsigned right-shift :ref:`reference-dsl-ursh` along with ``>>>=``.
* String-hashing functions [md5](reference-dsl-builtin-functions.md#md5), [sha1](reference-dsl-builtin-functions.md#sha1), [sha256](reference-dsl-builtin-functions.md#sha256), and [sha512](reference-dsl-builtin-functions.md#sha512).
* Platform-property functions [hostname](reference-dsl-builtin-functions.md#hostname), [os](reference-dsl-builtin-functions.md#os), and [version](reference-dsl-builtin-functions.md#version).
* Unsigned right-shift [`>>>`](reference-dsl-builtin-functions.md#ursh) along with `>>>=`.
## Improved command-line parsing
Miller 6 has getoptish command-line parsing (https://github.com/johnkerl/miller/pull/467):
* ``-xyz`` expands automatically to ``-x -y -z``, so (for example) ``mlr cut -of shape,flag`` is the same as ``mlr cut -o -f shape,flag``.
* ``--foo=bar`` expands automatically to ``--foo bar``, so (for example) ``mlr --ifs=comma`` is the same as ``mlr --ifs comma``.
* ``--mfrom``, ``--load``, ``--mload`` as described at [TODO:linkify].
* `-xyz` expands automatically to `-x -y -z`, so (for example) `mlr cut -of shape,flag` is the same as `mlr cut -o -f shape,flag`.
* `--foo=bar` expands automatically to `--foo bar`, so (for example) `mlr --ifs=comma` is the same as `mlr --ifs comma`.
* `--mfrom`, `--load`, `--mload` as described at [TODO:linkify].
## Improved error messages for DSL parsing
For ``mlr put`` and ``mlr filter``, parse-error messages now include location information:
For `mlr put` and `mlr filter`, parse-error messages now include location information:
<pre>
<pre class="pre-non-highlight">
mlr: cannot parse DSL expression.
Parse error on token ">" at line 63 columnn 7.
</pre>
## Developer-specific aspects
* Miller has been ported from C to Go. Developer notes: https://github.com/johnkerl/miller/blob/main/go/README.md
* Completely reworked regression testing, including running on Windows
* Miller has been ported from C to Go. Developer notes: [https://github.com/johnkerl/miller/blob/main/go/README.md](https://github.com/johnkerl/miller/blob/main/go/README.md).
* Completely reworked regression testing, including regresstion-testing now running on Windows.

44
docs6b/docs/new-in-miller-6.md.in
View file

@ -4,10 +4,10 @@ See also the [list of issues tagged with go-port](https://github.com/johnkerl/mi
## Documentation improvements
Documentation (what you're reading here) and on-line help (``mlr --help``) have been completely reworked.
Documentation (what you're reading here) and on-line help (`mlr --help`) have been completely reworked.
In the initial release, the focus was convincing users already familiar with
``awk``/``grep``/``cut`` that Miller was a viable option. Over time it's become
`awk`/`grep`/`cut` that Miller was a viable option. Over time it's become
clear that many users aren't expert with these. The focus has shifted toward a
higher quantity of more introductory/accessible material for command-line data
processing.
@ -24,14 +24,14 @@ now discussed first, and more examples use CSV.
## JSON support, and arrays
Arrays are now supported in Miller's ``put``/``filter`` programming language,
as described at :doc:`reference-dsl-arrays`. Also, ``array`` is now a keyword
Arrays are now supported in Miller's `put`/`filter` programming language,
as described at [Reference: arrays](reference-dsl-arrays.md). Also, `array` is now a keyword
so this is no longer usable as a local-variable or UDF name.
JSON support is improved:
* Direct support for arrays means that you can now use Miller to process more JSON files.
* Streamable JSON parsing: Miller's internal record-processing pipeline starts as soon as the first record is read (which was already the case for other file formats). This means that, unless records are wrapped with outermost ``[...]``, Miller now handles JSON in ``tail -f`` contexts like it does for other file formats.
* Streamable JSON parsing: Miller's internal record-processing pipeline starts as soon as the first record is read (which was already the case for other file formats). This means that, unless records are wrapped with outermost `[...]`, Miller now handles JSON in `tail -f` contexts like it does for other file formats.
* Flatten/unflatten -- TODO pick a name and link to a separate page/section
## Improved Windows experience
@ -43,11 +43,7 @@ Binaries are reliably available using GitHub Actions: see also [Installation](in
## In-process support for compressed input
In addition to ``--prepipe gunzip``, you can now use the ``--gzin`` flag. In
fact, if your files end in ``.gz`` you don't even need to do that -- Miller
will autodetect by file extension and automatically uncompress ``mlr --csv cat
foo.csv.gz``. Similarly for ``.z`` and ``.bz2`` files. Please see section
[TODO:linkify] for more information.
In addition to `--prepipe gunzip`, you can now use the `--gzin` flag. In fact, if your files end in `.gz` you don't even need to do that -- Miller will autodetect by file extension and automatically uncompress `mlr --csv cat foo.csv.gz`. Similarly for `.z` and `.bz2` files. Please see section [TODO:linkify] for more information.
## Output colorization
@ -59,12 +55,12 @@ The most central part of Miller 6 is a deep refactor of how data values are pars
from file contents, how types are inferred, and how they're converted back to
text into output files.
This was all initiated by https://github.com/johnkerl/miller/issues/151.
This was all initiated by [https://github.com/johnkerl/miller/issues/151](https://github.com/johnkerl/miller/issues/151).
In Miller 5 and below, all values were stored as strings, then only converted
to int/float as-needed, for example when a particular field was referenced in
the ``stats1`` or ``put`` verbs. This led to awkwardnesses such as the ``-S``
and ``-F`` flags for ``put`` and ``filter``.
the `stats1` or `put` verbs. This led to awkwardnesses such as the `-S`
and `-F` flags for `put` and `filter`.
In Miller 6, things parseable as int/float are treated as such from the moment
the input data is read, and these are passed along through the verb chain. All
@ -74,7 +70,7 @@ a numeric field isn't modified during the processing chain, it's printed out
the way it arrived. Also, quoted values in JSON strings are flagged as being
strings throughout the processing chain.
For example (see https://github.com/johnkerl/miller/issues/178) you can now do
For example (see [https://github.com/johnkerl/miller/issues/178](https://github.com/johnkerl/miller/issues/178)) you can now do
GENMD_RUN_COMMAND
echo '{ "a": "0123" }' | mlr --json cat
@ -90,21 +86,23 @@ Miller now has a read-evaluate-print-loop ([REPL](repl.md)) where you can single
## New DSL functions / operators
* String-hashing functions :ref:`reference-dsl-md5`, :ref:`reference-dsl-sha1`, :ref:`reference-dsl-sha256`, and :ref:`reference-dsl-sha512`.
* Platform-property functions :ref:`reference-dsl-hostname`, :ref:`reference-dsl-os`, and :ref:`reference-dsl-version`.
* Unsigned right-shift :ref:`reference-dsl-ursh` along with ``>>>=``.
* String-hashing functions [md5](reference-dsl-builtin-functions.md#md5), [sha1](reference-dsl-builtin-functions.md#sha1), [sha256](reference-dsl-builtin-functions.md#sha256), and [sha512](reference-dsl-builtin-functions.md#sha512).
* Platform-property functions [hostname](reference-dsl-builtin-functions.md#hostname), [os](reference-dsl-builtin-functions.md#os), and [version](reference-dsl-builtin-functions.md#version).
* Unsigned right-shift [`>>>`](reference-dsl-builtin-functions.md#ursh) along with `>>>=`.
## Improved command-line parsing
Miller 6 has getoptish command-line parsing (https://github.com/johnkerl/miller/pull/467):
* ``-xyz`` expands automatically to ``-x -y -z``, so (for example) ``mlr cut -of shape,flag`` is the same as ``mlr cut -o -f shape,flag``.
* ``--foo=bar`` expands automatically to ``--foo bar``, so (for example) ``mlr --ifs=comma`` is the same as ``mlr --ifs comma``.
* ``--mfrom``, ``--load``, ``--mload`` as described at [TODO:linkify].
* `-xyz` expands automatically to `-x -y -z`, so (for example) `mlr cut -of shape,flag` is the same as `mlr cut -o -f shape,flag`.
* `--foo=bar` expands automatically to `--foo bar`, so (for example) `mlr --ifs=comma` is the same as `mlr --ifs comma`.
* `--mfrom`, `--load`, `--mload` as described at [TODO:linkify].
## Improved error messages for DSL parsing
For ``mlr put`` and ``mlr filter``, parse-error messages now include location information:
For `mlr put` and `mlr filter`, parse-error messages now include location information:
GENMD_CARDIFY
mlr: cannot parse DSL expression.
@ -113,5 +111,5 @@ GENMD_EOF
## Developer-specific aspects
* Miller has been ported from C to Go. Developer notes: https://github.com/johnkerl/miller/blob/main/go/README.md
* Completely reworked regression testing, including running on Windows
* Miller has been ported from C to Go. Developer notes: [https://github.com/johnkerl/miller/blob/main/go/README.md](https://github.com/johnkerl/miller/blob/main/go/README.md).
* Completely reworked regression testing, including regresstion-testing now running on Windows.

52
docs6b/docs/operating-on-all-fields.md
View file

@ -5,26 +5,32 @@
Suppose you want to replace spaces with underscores in your column names:
<pre>
<pre class="pre-highlight">
<b>cat data/spaces.csv</b>
</pre>
<pre class="pre-non-highlight">
a b c,def,g h i
123,4567,890
2468,1357,3579
9987,3312,4543
</pre>
The simplest way is to use ``mlr rename`` with ``-g`` (for global replace, not just first occurrence of space within each field) and ``-r`` for pattern-matching (rather than explicit single-column renames):
The simplest way is to use `mlr rename` with `-g` (for global replace, not just first occurrence of space within each field) and `-r` for pattern-matching (rather than explicit single-column renames):
<pre>
<pre class="pre-highlight">
<b>mlr --csv rename -g -r ' ,_' data/spaces.csv</b>
</pre>
<pre class="pre-non-highlight">
a_b_c,def,g_h_i
123,4567,890
2468,1357,3579
9987,3312,4543
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --csv --opprint rename -g -r ' ,_' data/spaces.csv</b>
</pre>
<pre class="pre-non-highlight">
a_b_c def g_h_i
123 4567 890
2468 1357 3579
@ -33,8 +39,10 @@ a_b_c def g_h_i
You can also do this with a for-loop:
<pre>
<pre class="pre-highlight">
<b>cat data/bulk-rename-for-loop.mlr</b>
</pre>
<pre class="pre-non-highlight">
map newrec = {};
for (oldk, v in $*) {
newrec[gsub(oldk, " ", "_")] = v;
@ -42,8 +50,10 @@ for (oldk, v in $*) {
$* = newrec
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --icsv --opprint put -f data/bulk-rename-for-loop.mlr data/spaces.csv</b>
</pre>
<pre class="pre-non-highlight">
a_b_c def g_h_i
123 4567 890
2468 1357 3579
@ -52,24 +62,30 @@ a_b_c def g_h_i
## Search-and-replace over all fields
How to do ``$name = gsub($name, "old", "new")`` for all fields?
How to do `$name = gsub($name, "old", "new")` for all fields?
<pre>
<pre class="pre-highlight">
<b>cat data/sar.csv</b>
</pre>
<pre class="pre-non-highlight">
a,b,c
the quick,brown fox,jumped
over,the,lazy dogs
</pre>
<pre>
<pre class="pre-highlight">
<b>cat data/sar.mlr</b>
for (k in $*) {
$[k] = gsub($[k], "e", "X");
}
</pre>
<pre class="pre-non-highlight">
for (k in $*) {
$[k] = gsub($[k], "e", "X");
}
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --csv put -f data/sar.mlr data/sar.csv</b>
</pre>
<pre class="pre-non-highlight">
a,b,c
thX quick,brown fox,jumpXd
ovXr,thX,lazy dogs
@ -77,10 +93,12 @@ ovXr,thX,lazy dogs
## Full field renames and reassigns
Using Miller 5.0.0's map literals and assigning to ``$*``, you can fully generalize :ref:`mlr rename <reference-verbs-rename>`, :ref:`mlr reorder <reference-verbs-reorder>`, etc.
Using Miller 5.0.0's map literals and assigning to `$*`, you can fully generalize [rename](reference-verbs.md#rename), [reorder](reference-verbs.md#reorder), etc.
<pre>
<pre class="pre-highlight">
<b>cat data/small</b>
</pre>
<pre class="pre-non-highlight">
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
@ -88,7 +106,7 @@ a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr put '</b>
<b> begin {</b>
<b> @i_cumu = 0;</b>
@ -104,6 +122,8 @@ a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
<b> "x": $y,</b>
<b> };</b>
<b>' data/small</b>
</pre>
<pre class="pre-non-highlight">
z=0.3467901443380824,KEYFIELD=pan,i=1,b=pan,y=0.3467901443380824,x=0.7268028627434533
z=0.7586799647899636,KEYFIELD=eks,i=3,b=pan,y=0.7586799647899636,x=0.5221511083334797
z=0.20460330576630303,KEYFIELD=wye,i=6,b=wye,y=0.20460330576630303,x=0.33831852551664776

6
docs6b/docs/operating-on-all-fields.md.in
View file

@ -8,7 +8,7 @@ GENMD_RUN_COMMAND
cat data/spaces.csv
GENMD_EOF
The simplest way is to use ``mlr rename`` with ``-g`` (for global replace, not just first occurrence of space within each field) and ``-r`` for pattern-matching (rather than explicit single-column renames):
The simplest way is to use `mlr rename` with `-g` (for global replace, not just first occurrence of space within each field) and `-r` for pattern-matching (rather than explicit single-column renames):
GENMD_RUN_COMMAND
mlr --csv rename -g -r ' ,_' data/spaces.csv
@ -30,7 +30,7 @@ GENMD_EOF
## Search-and-replace over all fields
How to do ``$name = gsub($name, "old", "new")`` for all fields?
How to do `$name = gsub($name, "old", "new")` for all fields?
GENMD_RUN_COMMAND
cat data/sar.csv
@ -46,7 +46,7 @@ GENMD_EOF
## Full field renames and reassigns
Using Miller 5.0.0's map literals and assigning to ``$*``, you can fully generalize :ref:`mlr rename <reference-verbs-rename>`, :ref:`mlr reorder <reference-verbs-reorder>`, etc.
Using Miller 5.0.0's map literals and assigning to `$*`, you can fully generalize [rename](reference-verbs.md#rename), [reorder](reference-verbs.md#reorder), etc.
GENMD_RUN_COMMAND
cat data/small

26
docs6b/docs/originality.md
View file

@ -1,40 +1,40 @@
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
# How original is Miller?
It isn't. Miller is one of many, many participants in the online-analytical-processing culture. Other key participants include ``awk``, SQL, spreadsheets, etc. etc. etc. Far from being an original concept, Miller explicitly strives to imitate several existing tools:
It isn't. Miller is one of many, many participants in the online-analytical-processing culture. Other key participants include `awk`, SQL, spreadsheets, etc. etc. etc. Far from being an original concept, Miller explicitly strives to imitate several existing tools:
**The Unix toolkit**: Intentional similarities as described in [Unix-toolkit Context](feature-comparison.md).
Recipes abound for command-line data analysis using the Unix toolkit. Here are just a couple of my favorites:
* http://en.wikibooks.org/wiki/Ad_Hoc_Data_Analysis_From_The_Unix_Command_Line
* http://www.gregreda.com/2013/07/15/unix-commands-for-data-science
* https://github.com/dbohdan/structured-text-tools
* [http://en.wikibooks.org/wiki/Ad_Hoc_Data_Analysis_From_The_Unix_Command_Line](http://en.wikibooks.org/wiki/Ad_Hoc_Data_Analysis_From_The_Unix_Command_Line)
* [http://www.gregreda.com/2013/07/15/unix-commands-for-data-science](http://www.gregreda.com/2013/07/15/unix-commands-for-data-science)
* [https://github.com/dbohdan/structured-text-tools](https://github.com/dbohdan/structured-text-tools)
**RecordStream**: Miller owes particular inspiration to [RecordStream](https://github.com/benbernard/RecordStream). The key difference is that RecordStream is a Perl-based tool for manipulating JSON (including requiring it to separately manipulate other formats such as CSV into and out of JSON), while Miller is fast Go which handles its formats natively. The similarities include the ``sort``, ``stats1`` (analog of RecordStream's ``collate``), and ``delta`` operations, as well as ``filter`` and ``put``, and pretty-print formatting.
**RecordStream**: Miller owes particular inspiration to [RecordStream](https://github.com/benbernard/RecordStream). The key difference is that RecordStream is a Perl-based tool for manipulating JSON (including requiring it to separately manipulate other formats such as CSV into and out of JSON), while Miller is fast Go which handles its formats natively. The similarities include the `sort`, `stats1` (analog of RecordStream's `collate`), and `delta` operations, as well as `filter` and `put`, and pretty-print formatting.
**stats_m**: A third source of lineage is my Python [stats_m](https://github.com/johnkerl/scripts-math/tree/master/stats) module. This includes simple single-pass algorithms which form Miller's ``stats1`` and ``stats2`` subcommands.
**stats_m**: A third source of lineage is my Python [stats_m](https://github.com/johnkerl/scripts-math/tree/master/stats) module. This includes simple single-pass algorithms which form Miller's `stats1` and `stats2` subcommands.
**SQL**: Fourthly, Miller's ``group-by`` command name is from SQL, as is the term ``aggregate``.
**SQL**: Fourthly, Miller's `group-by` command name is from SQL, as is the term `aggregate`.
**Added value**: Miller's added values include:
* Name-indexing, compared to the Unix toolkit's positional indexing.
* Raw speed, compared to ``awk``, RecordStream, ``stats_m``, or various other kinds of Python/Ruby/etc. scripts one can easily create.
* Raw speed, compared to `awk`, RecordStream, `stats_m`, or various other kinds of Python/Ruby/etc. scripts one can easily create.
* Compact keystroking for many common tasks, with a decent amount of flexibility.
* Ability to handle text files on the Unix pipe, without need for creating database tables, compared to SQL databases.
* Various file formats, and on-the-fly format conversion.
**jq**: Miller does for name-indexed text what [jq](https://stedolan.github.io/jq/) does for JSON. If you're not already familiar with ``jq``, please check it out!.
**jq**: Miller does for name-indexed text what [jq](https://stedolan.github.io/jq/) does for JSON. If you're not already familiar with `jq`, please check it out!.
**What about similar tools?**
Here's a comprehensive list: https://github.com/dbohdan/structured-text-tools. Last I knew it doesn't mention [rows](https://github.com/turicas/rows) so here's a plug for that as well. As it turns out, I learned about most of these after writing Miller.
Here's a comprehensive list: [https://github.com/dbohdan/structured-text-tools](https://github.com/dbohdan/structured-text-tools). Last I knew it doesn't mention [rows](https://github.com/turicas/rows) so here's a plug for that as well. As it turns out, I learned about most of these after writing Miller.
**What about DOTADIW?** One of the key points of the [Unix philosophy](http://en.wikipedia.org/wiki/Unix_philosophy) is that a tool should do one thing and do it well. Hence ``sort`` and ``cut`` do just one thing. Why does Miller put ``awk``-like processing, a few SQL-like operations, and statistical reduction all into one tool? This is a fair question. First note that many standard tools, such as ``awk`` and ``perl``, do quite a few things -- as does ``jq``. But I could have pushed for putting format awareness and name-indexing options into ``cut``, ``awk``, and so on (so you could do ``cut -f hostname,uptime`` or ``awk '{sum += $x*$y}END{print sum}'``). Patching ``cut``, ``sort``, etc. on multiple operating systems is a non-starter in terms of uptake. Moreover, it makes sense for me to have Miller be a tool which collects together format-aware record-stream processing into one place, with good reuse of Miller-internal library code for its various features.
**What about DOTADIW?** One of the key points of the [Unix philosophy](http://en.wikipedia.org/wiki/Unix_philosophy) is that a tool should do one thing and do it well. Hence `sort` and `cut` do just one thing. Why does Miller put `awk`-like processing, a few SQL-like operations, and statistical reduction all into one tool? This is a fair question. First note that many standard tools, such as `awk` and `perl`, do quite a few things -- as does `jq`. But I could have pushed for putting format awareness and name-indexing options into `cut`, `awk`, and so on (so you could do `cut -f hostname,uptime` or `awk '{sum += $x*$y}END{print sum}'`). Patching `cut`, `sort`, etc. on multiple operating systems is a non-starter in terms of uptake. Moreover, it makes sense for me to have Miller be a tool which collects together format-aware record-stream processing into one place, with good reuse of Miller-internal library code for its various features.
**Why not use Perl/Python/Ruby etc.?** Maybe you should. With those tools you'll get far more expressive power, and sufficiently quick turnaround time for small-to-medium-sized data. Using Miller you'll get something less than a complete programming language, but which is fast, with moderate amounts of flexibility and much less keystroking.
When I was first developing Miller I made a survey of several languages. Using low-level implementation languages like C, Go, Rust, and Nim, I'd need to create my own domain-specific language (DSL) which would always be less featured than a full programming language, but I'd get better performance. Using high-level interpreted languages such as Perl/Python/Ruby I'd get the language's ``eval`` for free and I wouldn't need a DSL; Miller would have mainly been a set of format-specific I/O hooks. If I'd gotten good enough performance from the latter I'd have done it without question and Miller would be far more flexible. But high-level languages win the performance criteria by a landslide so we have Miller in Go with a custom DSL.
When I was first developing Miller I made a survey of several languages. Using low-level implementation languages like C, Go, Rust, and Nim, I'd need to create my own domain-specific language (DSL) which would always be less featured than a full programming language, but I'd get better performance. Using high-level interpreted languages such as Perl/Python/Ruby I'd get the language's `eval` for free and I wouldn't need a DSL; Miller would have mainly been a set of format-specific I/O hooks. If I'd gotten good enough performance from the latter I'd have done it without question and Miller would be far more flexible. But high-level languages win the performance criteria by a landslide so we have Miller in Go with a custom DSL.
**No, really, why one more command-line data-manipulation tool?** I wrote Miller because I was frustrated with tools like ``grep``, ``sed``, and so on being *line-aware* without being *format-aware*. The single most poignant example I can think of is seeing people grep data lines out of their CSV files and sadly losing their header lines. While some lighter-than-SQL processing is very nice to have, at core I wanted the format-awareness of [RecordStream](https://github.com/benbernard/RecordStream) combined with the raw speed of the Unix toolkit. Miller does precisely that.
**No, really, why one more command-line data-manipulation tool?** I wrote Miller because I was frustrated with tools like `grep`, `sed`, and so on being *line-aware* without being *format-aware*. The single most poignant example I can think of is seeing people grep data lines out of their CSV files and sadly losing their header lines. While some lighter-than-SQL processing is very nice to have, at core I wanted the format-awareness of [RecordStream](https://github.com/benbernard/RecordStream) combined with the raw speed of the Unix toolkit. Miller does precisely that.

26
docs6b/docs/originality.md.in
View file

@ -1,39 +1,39 @@
# How original is Miller?
It isn't. Miller is one of many, many participants in the online-analytical-processing culture. Other key participants include ``awk``, SQL, spreadsheets, etc. etc. etc. Far from being an original concept, Miller explicitly strives to imitate several existing tools:
It isn't. Miller is one of many, many participants in the online-analytical-processing culture. Other key participants include `awk`, SQL, spreadsheets, etc. etc. etc. Far from being an original concept, Miller explicitly strives to imitate several existing tools:
**The Unix toolkit**: Intentional similarities as described in [Unix-toolkit Context](feature-comparison.md).
Recipes abound for command-line data analysis using the Unix toolkit. Here are just a couple of my favorites:
* http://en.wikibooks.org/wiki/Ad_Hoc_Data_Analysis_From_The_Unix_Command_Line
* http://www.gregreda.com/2013/07/15/unix-commands-for-data-science
* https://github.com/dbohdan/structured-text-tools
* [http://en.wikibooks.org/wiki/Ad_Hoc_Data_Analysis_From_The_Unix_Command_Line](http://en.wikibooks.org/wiki/Ad_Hoc_Data_Analysis_From_The_Unix_Command_Line)
* [http://www.gregreda.com/2013/07/15/unix-commands-for-data-science](http://www.gregreda.com/2013/07/15/unix-commands-for-data-science)
* [https://github.com/dbohdan/structured-text-tools](https://github.com/dbohdan/structured-text-tools)
**RecordStream**: Miller owes particular inspiration to [RecordStream](https://github.com/benbernard/RecordStream). The key difference is that RecordStream is a Perl-based tool for manipulating JSON (including requiring it to separately manipulate other formats such as CSV into and out of JSON), while Miller is fast Go which handles its formats natively. The similarities include the ``sort``, ``stats1`` (analog of RecordStream's ``collate``), and ``delta`` operations, as well as ``filter`` and ``put``, and pretty-print formatting.
**RecordStream**: Miller owes particular inspiration to [RecordStream](https://github.com/benbernard/RecordStream). The key difference is that RecordStream is a Perl-based tool for manipulating JSON (including requiring it to separately manipulate other formats such as CSV into and out of JSON), while Miller is fast Go which handles its formats natively. The similarities include the `sort`, `stats1` (analog of RecordStream's `collate`), and `delta` operations, as well as `filter` and `put`, and pretty-print formatting.
**stats_m**: A third source of lineage is my Python [stats_m](https://github.com/johnkerl/scripts-math/tree/master/stats) module. This includes simple single-pass algorithms which form Miller's ``stats1`` and ``stats2`` subcommands.
**stats_m**: A third source of lineage is my Python [stats_m](https://github.com/johnkerl/scripts-math/tree/master/stats) module. This includes simple single-pass algorithms which form Miller's `stats1` and `stats2` subcommands.
**SQL**: Fourthly, Miller's ``group-by`` command name is from SQL, as is the term ``aggregate``.
**SQL**: Fourthly, Miller's `group-by` command name is from SQL, as is the term `aggregate`.
**Added value**: Miller's added values include:
* Name-indexing, compared to the Unix toolkit's positional indexing.
* Raw speed, compared to ``awk``, RecordStream, ``stats_m``, or various other kinds of Python/Ruby/etc. scripts one can easily create.
* Raw speed, compared to `awk`, RecordStream, `stats_m`, or various other kinds of Python/Ruby/etc. scripts one can easily create.
* Compact keystroking for many common tasks, with a decent amount of flexibility.
* Ability to handle text files on the Unix pipe, without need for creating database tables, compared to SQL databases.
* Various file formats, and on-the-fly format conversion.
**jq**: Miller does for name-indexed text what [jq](https://stedolan.github.io/jq/) does for JSON. If you're not already familiar with ``jq``, please check it out!.
**jq**: Miller does for name-indexed text what [jq](https://stedolan.github.io/jq/) does for JSON. If you're not already familiar with `jq`, please check it out!.
**What about similar tools?**
Here's a comprehensive list: https://github.com/dbohdan/structured-text-tools. Last I knew it doesn't mention [rows](https://github.com/turicas/rows) so here's a plug for that as well. As it turns out, I learned about most of these after writing Miller.
Here's a comprehensive list: [https://github.com/dbohdan/structured-text-tools](https://github.com/dbohdan/structured-text-tools). Last I knew it doesn't mention [rows](https://github.com/turicas/rows) so here's a plug for that as well. As it turns out, I learned about most of these after writing Miller.
**What about DOTADIW?** One of the key points of the [Unix philosophy](http://en.wikipedia.org/wiki/Unix_philosophy) is that a tool should do one thing and do it well. Hence ``sort`` and ``cut`` do just one thing. Why does Miller put ``awk``-like processing, a few SQL-like operations, and statistical reduction all into one tool? This is a fair question. First note that many standard tools, such as ``awk`` and ``perl``, do quite a few things -- as does ``jq``. But I could have pushed for putting format awareness and name-indexing options into ``cut``, ``awk``, and so on (so you could do ``cut -f hostname,uptime`` or ``awk '{sum += $x*$y}END{print sum}'``). Patching ``cut``, ``sort``, etc. on multiple operating systems is a non-starter in terms of uptake. Moreover, it makes sense for me to have Miller be a tool which collects together format-aware record-stream processing into one place, with good reuse of Miller-internal library code for its various features.
**What about DOTADIW?** One of the key points of the [Unix philosophy](http://en.wikipedia.org/wiki/Unix_philosophy) is that a tool should do one thing and do it well. Hence `sort` and `cut` do just one thing. Why does Miller put `awk`-like processing, a few SQL-like operations, and statistical reduction all into one tool? This is a fair question. First note that many standard tools, such as `awk` and `perl`, do quite a few things -- as does `jq`. But I could have pushed for putting format awareness and name-indexing options into `cut`, `awk`, and so on (so you could do `cut -f hostname,uptime` or `awk '{sum += $x*$y}END{print sum}'`). Patching `cut`, `sort`, etc. on multiple operating systems is a non-starter in terms of uptake. Moreover, it makes sense for me to have Miller be a tool which collects together format-aware record-stream processing into one place, with good reuse of Miller-internal library code for its various features.
**Why not use Perl/Python/Ruby etc.?** Maybe you should. With those tools you'll get far more expressive power, and sufficiently quick turnaround time for small-to-medium-sized data. Using Miller you'll get something less than a complete programming language, but which is fast, with moderate amounts of flexibility and much less keystroking.
When I was first developing Miller I made a survey of several languages. Using low-level implementation languages like C, Go, Rust, and Nim, I'd need to create my own domain-specific language (DSL) which would always be less featured than a full programming language, but I'd get better performance. Using high-level interpreted languages such as Perl/Python/Ruby I'd get the language's ``eval`` for free and I wouldn't need a DSL; Miller would have mainly been a set of format-specific I/O hooks. If I'd gotten good enough performance from the latter I'd have done it without question and Miller would be far more flexible. But high-level languages win the performance criteria by a landslide so we have Miller in Go with a custom DSL.
When I was first developing Miller I made a survey of several languages. Using low-level implementation languages like C, Go, Rust, and Nim, I'd need to create my own domain-specific language (DSL) which would always be less featured than a full programming language, but I'd get better performance. Using high-level interpreted languages such as Perl/Python/Ruby I'd get the language's `eval` for free and I wouldn't need a DSL; Miller would have mainly been a set of format-specific I/O hooks. If I'd gotten good enough performance from the latter I'd have done it without question and Miller would be far more flexible. But high-level languages win the performance criteria by a landslide so we have Miller in Go with a custom DSL.
**No, really, why one more command-line data-manipulation tool?** I wrote Miller because I was frustrated with tools like ``grep``, ``sed``, and so on being *line-aware* without being *format-aware*. The single most poignant example I can think of is seeing people grep data lines out of their CSV files and sadly losing their header lines. While some lighter-than-SQL processing is very nice to have, at core I wanted the format-awareness of [RecordStream](https://github.com/benbernard/RecordStream) combined with the raw speed of the Unix toolkit. Miller does precisely that.
**No, really, why one more command-line data-manipulation tool?** I wrote Miller because I was frustrated with tools like `grep`, `sed`, and so on being *line-aware* without being *format-aware*. The single most poignant example I can think of is seeing people grep data lines out of their CSV files and sadly losing their header lines. While some lighter-than-SQL processing is very nice to have, at core I wanted the format-awareness of [RecordStream](https://github.com/benbernard/RecordStream) combined with the raw speed of the Unix toolkit. Miller does precisely that.

29
docs6b/docs/output-colorization.md
View file

@ -3,7 +3,7 @@
As of version 6.0.0, Miller supports output-colorization. Here are examples using side-by-side black-background and white-background terminals:
.. image:: pix/colorization.png
![pix/colorization.png](pix/colorization.png)
Things having colors:
@ -16,38 +16,37 @@ Rules for colorization:
* By default, colorize output only if writing to stdout and stdout is a TTY.
* Example: color: ``mlr --csv cat foo.csv``
* Example: no color: ``mlr --csv cat foo.csv > bar.csv``
* Example: no color: ``mlr --csv cat foo.csv | less``
* Example: color: `mlr --csv cat foo.csv`
* Example: no color: `mlr --csv cat foo.csv > bar.csv`
* Example: no color: `mlr --csv cat foo.csv | less`
* The default colors were chosen since they look OK with white or black terminal background, and are differentiable with common varieties of human color vision.
Mechanisms for colorization:
* Miller uses ANSI escape sequences only. This does not work on Windows except on Cygwin.
* Requires ``TERM`` environment variable to be set to non-empty string.
* Requires `TERM` environment variable to be set to non-empty string.
* Doesn't try to check to see whether the terminal is capable of 256-color ANSI vs 16-color ANSI. Note that if colors are in the range 0..15 then 16-color ANSI escapes are used, so this is in the user's control.
How you can control colorization:
* Suppression/unsuppression:
* ``export MLR_NO_COLOR=true`` means Miller won't color even when it normally would.
* ``export MLR_ALWAYS_COLOR=true`` means Miller will color even when it normally would not. For example, you might want to use this when piping ``mlr`` output to ``less -r``.
* Command-line flags ``--no-color`` or ``-M``, ``--always-color`` or ``-C``.
* `export MLR_NO_COLOR=true` means Miller won't color even when it normally would.
* `export MLR_ALWAYS_COLOR=true` means Miller will color even when it normally would not. For example, you might want to use this when piping `mlr` output to `less -r`.
* Command-line flags `--no-color` or `-M`, `--always-color` or `-C`.
* Color choices can be specified by using environment variables or command-line flags, with values 0..255:
* ``export MLR_KEY_COLOR=208``
* ``export MLR_VALUE_COLOR=33``
* Likewise for ``MLR_PASS_COLOR``, ``MLR_FAIL_COLOR``, ``MLR_HELP_COLOR``, ``MLR_REPL_PS1_COLOR``, and ``MLR_REPL_PS2_COLOR``.
* Command-line flags ``--key-color 208``, ``--value-color 33``, etc., and likewise for ``--pass-color``, ``--fail-color``, ``--repl-ps1-color``, ``--repl-ps2-color``, and ``--help-color``.
* `export MLR_KEY_COLOR=208`
* `export MLR_VALUE_COLOR=33`
* Likewise for `MLR_PASS_COLOR`, `MLR_FAIL_COLOR`, `MLR_HELP_COLOR`, `MLR_REPL_PS1_COLOR`, and `MLR_REPL_PS2_COLOR`.
* Command-line flags `--key-color 208`, `--value-color 33`, etc., and likewise for `--pass-color`, `--fail-color`, `--repl-ps1-color`, `--repl-ps2-color`, and `--help-color`.
* This is particularly useful if your terminal's background color clashes with current settings.
If environment-variable settings and command-line flags are both provided, the latter take precedence.
Please do ``mlr --list-color-codes`` to see the available color codes (like ``170``), and ``mlr --list-color-names`` to see available names (like ``orchid``).
.. image:: pix/colorization2.png
Please do `mlr --list-color-codes` to see the available color codes (like `170`), and `mlr --list-color-names` to see available names (like `orchid`).
![pix/colorization2.png](pix/colorization2.png)

29
docs6b/docs/output-colorization.md.in
View file

@ -2,7 +2,7 @@
As of version 6.0.0, Miller supports output-colorization. Here are examples using side-by-side black-background and white-background terminals:
.. image:: pix/colorization.png
![pix/colorization.png](pix/colorization.png)
Things having colors:
@ -15,38 +15,37 @@ Rules for colorization:
* By default, colorize output only if writing to stdout and stdout is a TTY.
* Example: color: ``mlr --csv cat foo.csv``
* Example: no color: ``mlr --csv cat foo.csv > bar.csv``
* Example: no color: ``mlr --csv cat foo.csv | less``
* Example: color: `mlr --csv cat foo.csv`
* Example: no color: `mlr --csv cat foo.csv > bar.csv`
* Example: no color: `mlr --csv cat foo.csv | less`
* The default colors were chosen since they look OK with white or black terminal background, and are differentiable with common varieties of human color vision.
Mechanisms for colorization:
* Miller uses ANSI escape sequences only. This does not work on Windows except on Cygwin.
* Requires ``TERM`` environment variable to be set to non-empty string.
* Requires `TERM` environment variable to be set to non-empty string.
* Doesn't try to check to see whether the terminal is capable of 256-color ANSI vs 16-color ANSI. Note that if colors are in the range 0..15 then 16-color ANSI escapes are used, so this is in the user's control.
How you can control colorization:
* Suppression/unsuppression:
* ``export MLR_NO_COLOR=true`` means Miller won't color even when it normally would.
* ``export MLR_ALWAYS_COLOR=true`` means Miller will color even when it normally would not. For example, you might want to use this when piping ``mlr`` output to ``less -r``.
* Command-line flags ``--no-color`` or ``-M``, ``--always-color`` or ``-C``.
* `export MLR_NO_COLOR=true` means Miller won't color even when it normally would.
* `export MLR_ALWAYS_COLOR=true` means Miller will color even when it normally would not. For example, you might want to use this when piping `mlr` output to `less -r`.
* Command-line flags `--no-color` or `-M`, `--always-color` or `-C`.
* Color choices can be specified by using environment variables or command-line flags, with values 0..255:
* ``export MLR_KEY_COLOR=208``
* ``export MLR_VALUE_COLOR=33``
* Likewise for ``MLR_PASS_COLOR``, ``MLR_FAIL_COLOR``, ``MLR_HELP_COLOR``, ``MLR_REPL_PS1_COLOR``, and ``MLR_REPL_PS2_COLOR``.
* Command-line flags ``--key-color 208``, ``--value-color 33``, etc., and likewise for ``--pass-color``, ``--fail-color``, ``--repl-ps1-color``, ``--repl-ps2-color``, and ``--help-color``.
* `export MLR_KEY_COLOR=208`
* `export MLR_VALUE_COLOR=33`
* Likewise for `MLR_PASS_COLOR`, `MLR_FAIL_COLOR`, `MLR_HELP_COLOR`, `MLR_REPL_PS1_COLOR`, and `MLR_REPL_PS2_COLOR`.
* Command-line flags `--key-color 208`, `--value-color 33`, etc., and likewise for `--pass-color`, `--fail-color`, `--repl-ps1-color`, `--repl-ps2-color`, and `--help-color`.
* This is particularly useful if your terminal's background color clashes with current settings.
If environment-variable settings and command-line flags are both provided, the latter take precedence.
Please do ``mlr --list-color-codes`` to see the available color codes (like ``170``), and ``mlr --list-color-names`` to see available names (like ``orchid``).
.. image:: pix/colorization2.png
Please do `mlr --list-color-codes` to see the available color codes (like `170`), and `mlr --list-color-names` to see available names (like `orchid`).
![pix/colorization2.png](pix/colorization2.png)

8
docs6b/docs/performance.md
View file

@ -5,14 +5,14 @@
In a previous version of this page, I compared Miller to some items in the Unix toolkit in terms of run time. But such comparisons are very much not apples-to-apples:
* Miller's principal strength is that it handles **key-value data in various formats** while the system tools **do not**. So if you time ``mlr sort`` on a CSV file against system ``sort``, it's not relevant to say which is faster by how many percent -- Miller will respect the header line, leaving it in place, while the system sort will move it, sorting it along with all the other header lines. This would be comparing the run times of two programs produce different outputs. Likewise, ``awk`` doesn't respect header lines, although you can code up some CSV-handling using ``if (NR==1) { ... } else { ... }``. And that's just CSV: I don't know any simple way to get ``sort``, ``awk``, etc. to handle DKVP, JSON, etc. -- which is the main reason I wrote Miller.
* Miller's principal strength is that it handles **key-value data in various formats** while the system tools **do not**. So if you time `mlr sort` on a CSV file against system `sort`, it's not relevant to say which is faster by how many percent -- Miller will respect the header line, leaving it in place, while the system sort will move it, sorting it along with all the other header lines. This would be comparing the run times of two programs produce different outputs. Likewise, `awk` doesn't respect header lines, although you can code up some CSV-handling using `if (NR==1) { ... } else { ... }`. And that's just CSV: I don't know any simple way to get `sort`, `awk`, etc. to handle DKVP, JSON, etc. -- which is the main reason I wrote Miller.
* **Implementations differ by platform**: one ``awk`` may be fundamentally faster than another, and ``mawk`` has a very efficient bytecode implementation -- which handles positionally indexed data far faster than Miller does.
* **Implementations differ by platform**: one `awk` may be fundamentally faster than another, and `mawk` has a very efficient bytecode implementation -- which handles positionally indexed data far faster than Miller does.
* The system ``sort`` command will, on some systems, handle too-large-for-RAM datasets by spilling to disk; Miller (as of version 5.2.0, mid-2017) does not. Miller sorts are always stable; GNU supports stable and unstable variants.
* The system `sort` command will, on some systems, handle too-large-for-RAM datasets by spilling to disk; Miller (as of version 5.2.0, mid-2017) does not. Miller sorts are always stable; GNU supports stable and unstable variants.
* Etc.
## Summary
Miller can do many kinds of processing on key-value-pair data using elapsed time roughly of the same order of magnitude as items in the Unix toolkit can handle positionally indexed data. Specific results vary widely by platform, implementation details, multi-core use (or not). Lastly, specific special-purpose non-record-aware processing will run far faster in ``grep``, ``sed``, etc.
Miller can do many kinds of processing on key-value-pair data using elapsed time roughly of the same order of magnitude as items in the Unix toolkit can handle positionally indexed data. Specific results vary widely by platform, implementation details, multi-core use (or not). Lastly, specific special-purpose non-record-aware processing will run far faster in `grep`, `sed`, etc.

8
docs6b/docs/performance.md.in
View file

@ -4,14 +4,14 @@
In a previous version of this page, I compared Miller to some items in the Unix toolkit in terms of run time. But such comparisons are very much not apples-to-apples:
* Miller's principal strength is that it handles **key-value data in various formats** while the system tools **do not**. So if you time ``mlr sort`` on a CSV file against system ``sort``, it's not relevant to say which is faster by how many percent -- Miller will respect the header line, leaving it in place, while the system sort will move it, sorting it along with all the other header lines. This would be comparing the run times of two programs produce different outputs. Likewise, ``awk`` doesn't respect header lines, although you can code up some CSV-handling using ``if (NR==1) { ... } else { ... }``. And that's just CSV: I don't know any simple way to get ``sort``, ``awk``, etc. to handle DKVP, JSON, etc. -- which is the main reason I wrote Miller.
* Miller's principal strength is that it handles **key-value data in various formats** while the system tools **do not**. So if you time `mlr sort` on a CSV file against system `sort`, it's not relevant to say which is faster by how many percent -- Miller will respect the header line, leaving it in place, while the system sort will move it, sorting it along with all the other header lines. This would be comparing the run times of two programs produce different outputs. Likewise, `awk` doesn't respect header lines, although you can code up some CSV-handling using `if (NR==1) { ... } else { ... }`. And that's just CSV: I don't know any simple way to get `sort`, `awk`, etc. to handle DKVP, JSON, etc. -- which is the main reason I wrote Miller.
* **Implementations differ by platform**: one ``awk`` may be fundamentally faster than another, and ``mawk`` has a very efficient bytecode implementation -- which handles positionally indexed data far faster than Miller does.
* **Implementations differ by platform**: one `awk` may be fundamentally faster than another, and `mawk` has a very efficient bytecode implementation -- which handles positionally indexed data far faster than Miller does.
* The system ``sort`` command will, on some systems, handle too-large-for-RAM datasets by spilling to disk; Miller (as of version 5.2.0, mid-2017) does not. Miller sorts are always stable; GNU supports stable and unstable variants.
* The system `sort` command will, on some systems, handle too-large-for-RAM datasets by spilling to disk; Miller (as of version 5.2.0, mid-2017) does not. Miller sorts are always stable; GNU supports stable and unstable variants.
* Etc.
## Summary
Miller can do many kinds of processing on key-value-pair data using elapsed time roughly of the same order of magnitude as items in the Unix toolkit can handle positionally indexed data. Specific results vary widely by platform, implementation details, multi-core use (or not). Lastly, specific special-purpose non-record-aware processing will run far faster in ``grep``, ``sed``, etc.
Miller can do many kinds of processing on key-value-pair data using elapsed time roughly of the same order of magnitude as items in the Unix toolkit can handle positionally indexed data. Specific results vary widely by platform, implementation details, multi-core use (or not). Lastly, specific special-purpose non-record-aware processing will run far faster in `grep`, `sed`, etc.

20
docs6b/docs/programming-examples.md
View file

@ -5,10 +5,12 @@ Here are a few things focusing on Miller's DSL as a programming language per se,
## Sieve of Eratosthenes
The [Sieve of Eratosthenes](http://en.wikipedia.org/wiki/Sieve_of_Eratosthenes) is a standard introductory programming topic. The idea is to find all primes up to some *N* by making a list of the numbers 1 to *N*, then striking out all multiples of 2 except 2 itself, all multiples of 3 except 3 itself, all multiples of 4 except 4 itself, and so on. Whatever survives that without getting marked is a prime. This is easy enough in Miller. Notice that here all the work is in ``begin`` and ``end`` statements; there is no file input (so we use ``mlr -n`` to keep Miller from waiting for input data).
The [Sieve of Eratosthenes](http://en.wikipedia.org/wiki/Sieve_of_Eratosthenes) is a standard introductory programming topic. The idea is to find all primes up to some *N* by making a list of the numbers 1 to *N*, then striking out all multiples of 2 except 2 itself, all multiples of 3 except 3 itself, all multiples of 4 except 4 itself, and so on. Whatever survives that without getting marked is a prime. This is easy enough in Miller. Notice that here all the work is in `begin` and `end` statements; there is no file input (so we use `mlr -n` to keep Miller from waiting for input data).
<pre>
<pre class="pre-highlight">
<b>cat programs/sieve.mlr</b>
</pre>
<pre class="pre-non-highlight">
# ================================================================
# Sieve of Eratosthenes: simple example of Miller DSL as programming language.
# ================================================================
@ -43,8 +45,10 @@ end {
}
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr -n put -f programs/sieve.mlr</b>
</pre>
<pre class="pre-non-highlight">
2
3
5
@ -78,8 +82,10 @@ The [Mandelbrot set](http://en.wikipedia.org/wiki/Mandelbrot_set) is also easily
The (approximate) computation of points in the complex plane which are and aren't members is just a few lines of complex arithmetic (see the Wikipedia article); how to render them is another task. Using graphics libraries you can create PNG or JPEG files, but another fun way to do this is by printing various characters to the screen:
<pre>
<pre class="pre-highlight">
<b>cat programs/mand.mlr</b>
</pre>
<pre class="pre-non-highlight">
# Mandelbrot set generator: simple example of Miller DSL as programming language.
begin {
# Set defaults
@ -184,8 +190,10 @@ func get_point_plot(pr, pi, maxits, do_julia, jr, ji) {
At standard resolution this makes a nice little ASCII plot:
<pre>
<pre class="pre-highlight">
<b>mlr -n put -f ./programs/mand.mlr</b>
</pre>
<pre class="pre-non-highlight">
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ -240,7 +248,7 @@ At standard resolution this makes a nice little ASCII plot:
But using a very small font size (as small as my Mac will let me go), and by choosing the coordinates to zoom in on a particular part of the complex plane, we can get a nice little picture:
<pre>
<pre class="pre-non-highlight">
#!/bin/bash
# Get the number of rows and columns from the terminal window dimensions
iheight=$(stty size | mlr --nidx --fs space cut -f 1)

2
docs6b/docs/programming-examples.md.in
View file

@ -4,7 +4,7 @@ Here are a few things focusing on Miller's DSL as a programming language per se,
## Sieve of Eratosthenes
The [Sieve of Eratosthenes](http://en.wikipedia.org/wiki/Sieve_of_Eratosthenes) is a standard introductory programming topic. The idea is to find all primes up to some *N* by making a list of the numbers 1 to *N*, then striking out all multiples of 2 except 2 itself, all multiples of 3 except 3 itself, all multiples of 4 except 4 itself, and so on. Whatever survives that without getting marked is a prime. This is easy enough in Miller. Notice that here all the work is in ``begin`` and ``end`` statements; there is no file input (so we use ``mlr -n`` to keep Miller from waiting for input data).
The [Sieve of Eratosthenes](http://en.wikipedia.org/wiki/Sieve_of_Eratosthenes) is a standard introductory programming topic. The idea is to find all primes up to some *N* by making a list of the numbers 1 to *N*, then striking out all multiples of 2 except 2 itself, all multiples of 3 except 3 itself, all multiples of 4 except 4 itself, and so on. Whatever survives that without getting marked is a prime. This is easy enough in Miller. Notice that here all the work is in `begin` and `end` statements; there is no file input (so we use `mlr -n` to keep Miller from waiting for input data).
GENMD_RUN_COMMAND
cat programs/sieve.mlr

152
docs6b/docs/programming-language.md
View file

@ -1,18 +1,20 @@
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
# Intro to Miller's programming language
In the [Miller in 10 minutes](10min.md) page we took a tour of some of Miller's most-used verbs including ``cat``, ``head``, ``tail``, ``cut``, and ``sort``. These are analogs of familiar system commands, but empowered by field-name indexing and file-format awareness: the system ``sort`` command only knows about lines and column names like ``1,2,3,4``, while ``mlr sort`` knows about CSV/TSV/JSON/etc records, and field names like ``color,shape,flag,index``.
In the [Miller in 10 minutes](10min.md) page we took a tour of some of Miller's most-used verbs including `cat`, `head`, `tail`, `cut`, and `sort`. These are analogs of familiar system commands, but empowered by field-name indexing and file-format awareness: the system `sort` command only knows about lines and column names like `1,2,3,4`, while `mlr sort` knows about CSV/TSV/JSON/etc records, and field names like `color,shape,flag,index`.
We also caught a glimpse of Miller's ``put`` and ``filter`` verbs. These two are special since they let you express statements using Miller's programming language. It's a *embedded domain-specific language* since it's inside Miller: often referred to simply as the *Miller DSL*.
We also caught a glimpse of Miller's `put` and `filter` verbs. These two are special since they let you express statements using Miller's programming language. It's a *embedded domain-specific language* since it's inside Miller: often referred to simply as the *Miller DSL*.
In the [DSL reference](reference-dsl.md) page we have a complete reference to Miller's programming language. For now, let's take a quick look at key features -- you can use as few or as many features as you like.
## Records and fields
Let's keep using the sample [example.csv](./example.csv). When we type
Let's keep using the sample [example.csv](./example.csv). For example:
<pre>
<pre class="pre-highlight">
<b>mlr --c2p put '$cost = $quantity * $rate' example.csv</b>
</pre>
<pre class="pre-non-highlight">
color shape flag index quantity rate cost
yellow triangle true 11 43.6498 9.8870 431.5655726
red square true 15 79.2778 0.0130 1.0306114
@ -26,19 +28,21 @@ yellow circle true 87 63.5058 8.3350 529.3208430000001
purple square false 91 72.3735 8.2430 596.5747605000001
</pre>
a few things are happening:
When we type that, a few things are happening:
* We refer to fields in the input data using a dollar sign and then the field name, e.g. ``$quantity``. (If a field name has special characters like a dot or slash, just use curly braces: ``${field.name}``.)
* The expression ``$cost = $quantity * $rate`` is executed once per record of the data file. Our [example.csv](./example.csv) has 10 records so this expression was executed 10 times, with the field names ``$quantity`` and ``$rate`` bound to the current record's values for those fields.
* On the left-hand side we have the new field name ``$cost`` which didn't come from the input data. Assignments to new variables result in a new field being placed after all the other ones. If we'd assigned to an existing field name, it would have been updated in-place.
* We refer to fields in the input data using a dollar sign and then the field name, e.g. `$quantity`. (If a field name contains special characters like a dot or slash, just use curly braces: `${field.name}`.)
* The expression `$cost = $quantity * $rate` is executed once per record of the data file. Our [example.csv](./example.csv) has 10 records so this expression was executed 10 times, with the field names `$quantity` and `$rate` bound to the current record's values for those fields.
* On the left-hand side we have the new field name `$cost` which didn't come from the input data. Assignments to new variables result in a new field being placed after all the other ones. If we'd assigned to an existing field name, it would have been updated in-place.
* The entire expression is surrounded by single quotes, to get it past the system shell. Inside those, only double quotes have meaning in Miller's programming language.
## Multi-line statements, and statements-from-file
You can use more than one statement, separating them with semicolons, and optionally putting them on lines of their own:
<pre>
<pre class="pre-highlight">
<b>mlr --c2p put '$cost = $quantity * $rate; $index = $index * 100' example.csv</b>
</pre>
<pre class="pre-non-highlight">
color shape flag index quantity rate cost
yellow triangle true 1100 43.6498 9.8870 431.5655726
red square true 1500 79.2778 0.0130 1.0306114
@ -52,11 +56,13 @@ yellow circle true 8700 63.5058 8.3350 529.3208430000001
purple square false 9100 72.3735 8.2430 596.5747605000001
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --c2p put '</b>
<b> $cost = $quantity * $rate;</b>
<b> $index *= 100</b>
<b>' example.csv</b>
</pre>
<pre class="pre-non-highlight">
color shape flag index quantity rate cost
yellow triangle true 1100 43.6498 9.8870 431.5655726
red square true 1500 79.2778 0.0130 1.0306114
@ -70,16 +76,20 @@ yellow circle true 8700 63.5058 8.3350 529.3208430000001
purple square false 9100 72.3735 8.2430 596.5747605000001
</pre>
One of Miller's key features is the ability to express data-transformation right there at the keyboard, interactively. But if you find yourself using expressions repeatedly, you can put everything between the single quotes into a file and refer to that using ``put -f``:
One of Miller's key features is the ability to express data-transformation right there at the keyboard, interactively. But if you find yourself using expressions repeatedly, you can put everything between the single quotes into a file and refer to that using `put -f`:
<pre>
<pre class="pre-highlight">
<b>cat dsl-example.mlr</b>
</pre>
<pre class="pre-non-highlight">
$cost = $quantity * $rate;
$index *= 100
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --c2p put -f dsl-example.mlr example.csv</b>
</pre>
<pre class="pre-non-highlight">
color shape flag index quantity rate cost
yellow triangle true 1100 43.6498 9.8870 431.5655726
red square true 1500 79.2778 0.0130 1.0306114
@ -93,18 +103,20 @@ yellow circle true 8700 63.5058 8.3350 529.3208430000001
purple square false 9100 72.3735 8.2430 596.5747605000001
</pre>
This becomes particularly important on Windows. Quite a bit of effort was put into making Miller on Windows be able to handle the kinds of single-quoted expressions we're showing here, but if you get syntax-error messages on Windows using examples in this documentation, you can put the parts between single quotes into a file and refer to that using ``mlr put -f``.
This becomes particularly important on Windows. Quite a bit of effort was put into making Miller on Windows be able to handle the kinds of single-quoted expressions we're showing here, but if you get syntax-error messages on Windows using examples in this documentation, you can put the parts between single quotes into a file and refer to that using `mlr put -f`.
## Out-of-stream variables, begin, and end
Above we saw that your expression is executed once per record -- if a file has a million records, your expression will be executed a million times, once for each record. But you can mark statements to only be executed once, either before the record stream begins, or after the record stream is ended. If you know about [AWK](https://en.wikipedia.org/wiki/AWK), you might have noticed that Miller's programming language is loosely inspired by it, including the ``begin`` and ``end`` statements.
Above we saw that your expression is executed once per record -- if a file has a million records, your expression will be executed a million times, once for each record. But you can mark statements to only be executed once, either before the record stream begins, or after the record stream is ended. If you know about [AWK](https://en.wikipedia.org/wiki/AWK), you might have noticed that Miller's programming language is loosely inspired by it, including the `begin` and `end` statements.
Above we also saw that names like ``$quantity`` are bound to each record in turn.
Above we also saw that names like `$quantity` are bound to each record in turn.
To make ``begin`` and ``end`` statements useful, we need somewhere to put things that persist across the duration of the record stream, and a way to emit them. Miller uses **out-of-stream variables** (or **oosvars** for short) whose names start with an ``@`` sigil, and the **emit** keyword to write them into the output record stream:
To make `begin` and `end` statements useful, we need somewhere to put things that persist across the duration of the record stream, and a way to emit them. Miller uses **out-of-stream variables** (or **oosvars** for short) whose names start with an `@` sigil, and the **emit** keyword to write them into the output record stream:
<pre>
<pre class="pre-highlight">
<b>mlr --c2p --from example.csv put 'begin { @sum = 0 } @sum += $quantity; end {emit @sum}'</b>
</pre>
<pre class="pre-non-highlight">
color shape flag index quantity rate
yellow triangle true 11 43.6498 9.8870
red square true 15 79.2778 0.0130
@ -121,48 +133,56 @@ sum
652.7185
</pre>
If you want the end-block output to be the only output, and not include the input data, you can use ``mlr put -q``:
If you want the end-block output to be the only output, and not include the input data, you can use `mlr put -q`:
<pre>
<pre class="pre-highlight">
<b>mlr --c2p --from example.csv put -q 'begin { @sum = 0 } @sum += $quantity; end {emit @sum}'</b>
</pre>
<pre class="pre-non-highlight">
sum
652.7185
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --c2j --from example.csv put -q 'begin { @sum = 0 } @sum += $quantity; end {emit @sum}'</b>
</pre>
<pre class="pre-non-highlight">
{
"sum": 652.7185
}
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --c2j --from example.csv put -q '</b>
<b> begin { @count = 0; @sum = 0 }</b>
<b> @count += 1;</b>
<b> @sum += $quantity;</b>
<b> end {emit (@count, @sum)}</b>
<b>'</b>
</pre>
<pre class="pre-non-highlight">
{
"count": 10,
"sum": 652.7185
}
</pre>
We'll see in the documentation for :ref:`reference-verbs-stats1` that there's a lower-keystroking way to get counts and sums of things -- so, take this sum/count example as an indication of the kinds of things you can do using Miller's programming language.
We'll see in the documentation for [stats1](reference-verbs.md#stats1) that there's a lower-keystroking way to get counts and sums of things -- so, take this sum/count example as an indication of the kinds of things you can do using Miller's programming language.
## Context variables
Also inspired by [AWK](https://en.wikipedia.org/wiki/AWK), the Miller DSL has the following special **context variables**:
* ``FILENAME`` -- the filename the current record came from. Especially useful in things like ``mlr ... *.csv``.
* ``FILENUM`` -- similarly, but integer 1,2,3,... rather than filenam.e
* ``NF`` -- the number of fields in the current record. Note that if you assign ``$newcolumn = some value`` then ``NF`` will increment.
* ``NR`` -- starting from 1, counter of how many records processed so far.
* ``FNR`` -- similar, but resets to 1 at the start of each file.
* `FILENAME` -- the filename the current record came from. Especially useful in things like `mlr ... *.csv`.
* `FILENUM` -- similarly, but integer 1,2,3,... rather than filenam.e
* `NF` -- the number of fields in the current record. Note that if you assign `$newcolumn = some value` then `NF` will increment.
* `NR` -- starting from 1, counter of how many records processed so far.
* `FNR` -- similar, but resets to 1 at the start of each file.
<pre>
<pre class="pre-highlight">
<b>cat context-example.mlr</b>
</pre>
<pre class="pre-non-highlight">
$nf = NF;
$nr = NR;
$fnr = FNR;
@ -171,8 +191,10 @@ $filenum = FILENUM;
$newnf = NF;
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --c2p put -f context-example.mlr data/a.csv data/b.csv</b>
</pre>
<pre class="pre-non-highlight">
a b c nf nr fnr filename filenum newnf
1 2 3 3 1 1 data/a.csv 1 8
4 5 6 3 2 2 data/a.csv 1 8
@ -183,8 +205,10 @@ a b c nf nr fnr filename filenum newnf
You can define your own functions:
<pre>
<pre class="pre-highlight">
<b>cat factorial-example.mlr</b>
</pre>
<pre class="pre-non-highlight">
func factorial(n) {
if (n <= 1) {
return n
@ -194,8 +218,10 @@ func factorial(n) {
}
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --c2p --from example.csv put -f factorial-example.mlr -e '$fact = factorial(NR)'</b>
</pre>
<pre class="pre-non-highlight">
color shape flag index quantity rate fact
yellow triangle true 11 43.6498 9.8870 1
red square true 15 79.2778 0.0130 2
@ -209,18 +235,20 @@ yellow circle true 87 63.5058 8.3350 362880
purple square false 91 72.3735 8.2430 3628800
</pre>
Note that here we used the ``-f`` flag to ``put`` to load our function
definition, and also the ``-e`` flag to add another statement on the command
line. (We could have also put ``$fact = factorial(NR)`` inside
``factorial-example.mlr`` but that would have made that file less flexible for our
Note that here we used the `-f` flag to `put` to load our function
definition, and also the `-e` flag to add another statement on the command
line. (We could have also put `$fact = factorial(NR)` inside
`factorial-example.mlr` but that would have made that file less flexible for our
future use.)
## If-statements, loops, and local variables
Suppose you want to only compute sums conditionally -- you can use an ``if`` statement:
Suppose you want to only compute sums conditionally -- you can use an `if` statement:
<pre>
<pre class="pre-highlight">
<b>cat if-example.mlr</b>
</pre>
<pre class="pre-non-highlight">
begin {
@count_of_red = 0;
@sum_of_red = 0
@ -236,38 +264,46 @@ end {
}
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --c2p --from example.csv put -q -f if-example.mlr</b>
</pre>
<pre class="pre-non-highlight">
count_of_red sum_of_red
4 247.84139999999996
</pre>
Miller's else-if is spelled ``elif``.
Miller's else-if is spelled `elif`.
As we'll see more of in section (TODO:linkify), Miller has a few kinds of
for-loops. In addition to the usual 3-part ``for (i = 0; i < 10; i += 1)`` kind
for-loops. In addition to the usual 3-part `for (i = 0; i < 10; i += 1)` kind
that many programming languages have, Miller also lets you loop over arrays and
hashmaps. We haven't encountered arrays and hashmaps yet in this introduction,
but for now it suffices to know that ``$*`` is a special variable holding the
but for now it suffices to know that `$*` is a special variable holding the
current record as a hashmap:
<pre>
<pre class="pre-highlight">
<b>cat for-example.mlr</b>
</pre>
<pre class="pre-non-highlight">
for (k, v in $*) {
print "KEY IS ". k . " VALUE IS ". v;
}
print
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --csv cat data/a.csv</b>
</pre>
<pre class="pre-non-highlight">
a,b,c
1,2,3
4,5,6
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --csv --from data/a.csv put -qf for-example.mlr</b>
</pre>
<pre class="pre-non-highlight">
KEY IS a VALUE IS 1
KEY IS b VALUE IS 2
KEY IS c VALUE IS 3
@ -277,12 +313,12 @@ KEY IS b VALUE IS 5
KEY IS c VALUE IS 6
</pre>
Here we used the local variables ``k`` and ``v``. Now we've seen four kinds of variables:
Here we used the local variables `k` and `v`. Now we've seen four kinds of variables:
* Record fields like ``$shape``
* Out-of-stream variables like ``@sum``
* Local variables like ``k``
* Built-in context variables like ``NF`` and ``NR``
* Record fields like `$shape`
* Out-of-stream variables like `@sum`
* Local variables like `k`
* Built-in context variables like `NF` and `NR`
If you're curious about scope and extent of local variables, you can read more in (TODO:linkify) the section on variables.
@ -291,8 +327,8 @@ If you're curious about scope and extent of local variables, you can read more i
Numbers in Miller's programming language are intended to operate with the principle of least surprise:
* Internally, numbers are either 64-bit signed integers or double-precision floating-point.
* Sums, differences, and products of integers are also integers (so ``2*3=6`` not ``6.0``) -- unless the result of the operation would overflow a 64-bit signed integer in which case the result is automatically converted to float. (If you ever want integer-to-integer arithmetic, use ``x .+ y``, ``x .* y``, etc.)
* Quotients of integers are integers if the division is exact, else floating-point: so ``6/2=3`` but ``7/2=3.5``.
* Sums, differences, and products of integers are also integers (so `2*3=6` not `6.0`) -- unless the result of the operation would overflow a 64-bit signed integer in which case the result is automatically converted to float. (If you ever want integer-to-integer arithmetic, use `x .+ y`, `x .* y`, etc.)
* Quotients of integers are integers if the division is exact, else floating-point: so `6/2=3` but `7/2=3.5`.
You can read more about this at (TODO:linkify).
@ -300,13 +336,15 @@ You can read more about this at (TODO:linkify).
In addition to types including string, number (int/float), arrays, and hashmaps, Miller varibles can also be **absent**. This is when a variable never had a value assigned to it. Miller's treatment of absent data is intended to make it easy for you to handle non-heterogeneous data. We'll see more in section (TODO:linkify) but the basic idea is:
* Adding a number to absent gives the number back. This means you don't have to put ``@sum = 0`` in your ``begin`` blocks.
* Adding a number to absent gives the number back. This means you don't have to put `@sum = 0` in your `begin` blocks.
* Any variable which has the absent value is not assigned. This means you don't have to check presence of things from one record to the next.
For example, you can sum up all the ``$a`` values across records without having to check whether they're present or not:
For example, you can sum up all the `$a` values across records without having to check whether they're present or not:
<pre>
<pre class="pre-highlight">
<b>mlr --json cat absent-example.json</b>
</pre>
<pre class="pre-non-highlight">
{
"a": 1,
"b": 2
@ -320,8 +358,10 @@ For example, you can sum up all the ``$a`` values across records without having
}
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --json put '@sum_of_a += $a; end {emit @sum_of_a}' absent-example.json</b>
</pre>
<pre class="pre-non-highlight">
{
"a": 1,
"b": 2

72
docs6b/docs/programming-language.md.in
View file

@ -1,24 +1,24 @@
# Intro to Miller's programming language
In the [Miller in 10 minutes](10min.md) page we took a tour of some of Miller's most-used verbs including ``cat``, ``head``, ``tail``, ``cut``, and ``sort``. These are analogs of familiar system commands, but empowered by field-name indexing and file-format awareness: the system ``sort`` command only knows about lines and column names like ``1,2,3,4``, while ``mlr sort`` knows about CSV/TSV/JSON/etc records, and field names like ``color,shape,flag,index``.
In the [Miller in 10 minutes](10min.md) page we took a tour of some of Miller's most-used verbs including `cat`, `head`, `tail`, `cut`, and `sort`. These are analogs of familiar system commands, but empowered by field-name indexing and file-format awareness: the system `sort` command only knows about lines and column names like `1,2,3,4`, while `mlr sort` knows about CSV/TSV/JSON/etc records, and field names like `color,shape,flag,index`.
We also caught a glimpse of Miller's ``put`` and ``filter`` verbs. These two are special since they let you express statements using Miller's programming language. It's a *embedded domain-specific language* since it's inside Miller: often referred to simply as the *Miller DSL*.
We also caught a glimpse of Miller's `put` and `filter` verbs. These two are special since they let you express statements using Miller's programming language. It's a *embedded domain-specific language* since it's inside Miller: often referred to simply as the *Miller DSL*.
In the [DSL reference](reference-dsl.md) page we have a complete reference to Miller's programming language. For now, let's take a quick look at key features -- you can use as few or as many features as you like.
## Records and fields
Let's keep using the sample [example.csv](./example.csv). When we type
Let's keep using the sample [example.csv](./example.csv). For example:
GENMD_RUN_COMMAND
mlr --c2p put '$cost = $quantity * $rate' example.csv
GENMD_EOF
a few things are happening:
When we type that, a few things are happening:
* We refer to fields in the input data using a dollar sign and then the field name, e.g. ``$quantity``. (If a field name has special characters like a dot or slash, just use curly braces: ``${field.name}``.)
* The expression ``$cost = $quantity * $rate`` is executed once per record of the data file. Our [example.csv](./example.csv) has 10 records so this expression was executed 10 times, with the field names ``$quantity`` and ``$rate`` bound to the current record's values for those fields.
* On the left-hand side we have the new field name ``$cost`` which didn't come from the input data. Assignments to new variables result in a new field being placed after all the other ones. If we'd assigned to an existing field name, it would have been updated in-place.
* We refer to fields in the input data using a dollar sign and then the field name, e.g. `$quantity`. (If a field name contains special characters like a dot or slash, just use curly braces: `${field.name}`.)
* The expression `$cost = $quantity * $rate` is executed once per record of the data file. Our [example.csv](./example.csv) has 10 records so this expression was executed 10 times, with the field names `$quantity` and `$rate` bound to the current record's values for those fields.
* On the left-hand side we have the new field name `$cost` which didn't come from the input data. Assignments to new variables result in a new field being placed after all the other ones. If we'd assigned to an existing field name, it would have been updated in-place.
* The entire expression is surrounded by single quotes, to get it past the system shell. Inside those, only double quotes have meaning in Miller's programming language.
## Multi-line statements, and statements-from-file
@ -31,7 +31,7 @@ GENMD_EOF
GENMD_INCLUDE_AND_RUN_ESCAPED(dsl-example-multiline.sh)
One of Miller's key features is the ability to express data-transformation right there at the keyboard, interactively. But if you find yourself using expressions repeatedly, you can put everything between the single quotes into a file and refer to that using ``put -f``:
One of Miller's key features is the ability to express data-transformation right there at the keyboard, interactively. But if you find yourself using expressions repeatedly, you can put everything between the single quotes into a file and refer to that using `put -f`:
GENMD_RUN_COMMAND
cat dsl-example.mlr
@ -41,21 +41,21 @@ GENMD_RUN_COMMAND
mlr --c2p put -f dsl-example.mlr example.csv
GENMD_EOF
This becomes particularly important on Windows. Quite a bit of effort was put into making Miller on Windows be able to handle the kinds of single-quoted expressions we're showing here, but if you get syntax-error messages on Windows using examples in this documentation, you can put the parts between single quotes into a file and refer to that using ``mlr put -f``.
This becomes particularly important on Windows. Quite a bit of effort was put into making Miller on Windows be able to handle the kinds of single-quoted expressions we're showing here, but if you get syntax-error messages on Windows using examples in this documentation, you can put the parts between single quotes into a file and refer to that using `mlr put -f`.
## Out-of-stream variables, begin, and end
Above we saw that your expression is executed once per record -- if a file has a million records, your expression will be executed a million times, once for each record. But you can mark statements to only be executed once, either before the record stream begins, or after the record stream is ended. If you know about [AWK](https://en.wikipedia.org/wiki/AWK), you might have noticed that Miller's programming language is loosely inspired by it, including the ``begin`` and ``end`` statements.
Above we saw that your expression is executed once per record -- if a file has a million records, your expression will be executed a million times, once for each record. But you can mark statements to only be executed once, either before the record stream begins, or after the record stream is ended. If you know about [AWK](https://en.wikipedia.org/wiki/AWK), you might have noticed that Miller's programming language is loosely inspired by it, including the `begin` and `end` statements.
Above we also saw that names like ``$quantity`` are bound to each record in turn.
Above we also saw that names like `$quantity` are bound to each record in turn.
To make ``begin`` and ``end`` statements useful, we need somewhere to put things that persist across the duration of the record stream, and a way to emit them. Miller uses **out-of-stream variables** (or **oosvars** for short) whose names start with an ``@`` sigil, and the **emit** keyword to write them into the output record stream:
To make `begin` and `end` statements useful, we need somewhere to put things that persist across the duration of the record stream, and a way to emit them. Miller uses **out-of-stream variables** (or **oosvars** for short) whose names start with an `@` sigil, and the **emit** keyword to write them into the output record stream:
GENMD_RUN_COMMAND
mlr --c2p --from example.csv put 'begin { @sum = 0 } @sum += $quantity; end {emit @sum}'
GENMD_EOF
If you want the end-block output to be the only output, and not include the input data, you can use ``mlr put -q``:
If you want the end-block output to be the only output, and not include the input data, you can use `mlr put -q`:
GENMD_RUN_COMMAND
mlr --c2p --from example.csv put -q 'begin { @sum = 0 } @sum += $quantity; end {emit @sum}'
@ -74,17 +74,17 @@ mlr --c2j --from example.csv put -q '
'
GENMD_EOF
We'll see in the documentation for :ref:`reference-verbs-stats1` that there's a lower-keystroking way to get counts and sums of things -- so, take this sum/count example as an indication of the kinds of things you can do using Miller's programming language.
We'll see in the documentation for [stats1](reference-verbs.md#stats1) that there's a lower-keystroking way to get counts and sums of things -- so, take this sum/count example as an indication of the kinds of things you can do using Miller's programming language.
## Context variables
Also inspired by [AWK](https://en.wikipedia.org/wiki/AWK), the Miller DSL has the following special **context variables**:
* ``FILENAME`` -- the filename the current record came from. Especially useful in things like ``mlr ... *.csv``.
* ``FILENUM`` -- similarly, but integer 1,2,3,... rather than filenam.e
* ``NF`` -- the number of fields in the current record. Note that if you assign ``$newcolumn = some value`` then ``NF`` will increment.
* ``NR`` -- starting from 1, counter of how many records processed so far.
* ``FNR`` -- similar, but resets to 1 at the start of each file.
* `FILENAME` -- the filename the current record came from. Especially useful in things like `mlr ... *.csv`.
* `FILENUM` -- similarly, but integer 1,2,3,... rather than filenam.e
* `NF` -- the number of fields in the current record. Note that if you assign `$newcolumn = some value` then `NF` will increment.
* `NR` -- starting from 1, counter of how many records processed so far.
* `FNR` -- similar, but resets to 1 at the start of each file.
GENMD_RUN_COMMAND
cat context-example.mlr
@ -106,15 +106,15 @@ GENMD_RUN_COMMAND
mlr --c2p --from example.csv put -f factorial-example.mlr -e '$fact = factorial(NR)'
GENMD_EOF
Note that here we used the ``-f`` flag to ``put`` to load our function
definition, and also the ``-e`` flag to add another statement on the command
line. (We could have also put ``$fact = factorial(NR)`` inside
``factorial-example.mlr`` but that would have made that file less flexible for our
Note that here we used the `-f` flag to `put` to load our function
definition, and also the `-e` flag to add another statement on the command
line. (We could have also put `$fact = factorial(NR)` inside
`factorial-example.mlr` but that would have made that file less flexible for our
future use.)
## If-statements, loops, and local variables
Suppose you want to only compute sums conditionally -- you can use an ``if`` statement:
Suppose you want to only compute sums conditionally -- you can use an `if` statement:
GENMD_RUN_COMMAND
cat if-example.mlr
@ -124,13 +124,13 @@ GENMD_RUN_COMMAND
mlr --c2p --from example.csv put -q -f if-example.mlr
GENMD_EOF
Miller's else-if is spelled ``elif``.
Miller's else-if is spelled `elif`.
As we'll see more of in section (TODO:linkify), Miller has a few kinds of
for-loops. In addition to the usual 3-part ``for (i = 0; i < 10; i += 1)`` kind
for-loops. In addition to the usual 3-part `for (i = 0; i < 10; i += 1)` kind
that many programming languages have, Miller also lets you loop over arrays and
hashmaps. We haven't encountered arrays and hashmaps yet in this introduction,
but for now it suffices to know that ``$*`` is a special variable holding the
but for now it suffices to know that `$*` is a special variable holding the
current record as a hashmap:
GENMD_RUN_COMMAND
@ -145,12 +145,12 @@ GENMD_RUN_COMMAND
mlr --csv --from data/a.csv put -qf for-example.mlr
GENMD_EOF
Here we used the local variables ``k`` and ``v``. Now we've seen four kinds of variables:
Here we used the local variables `k` and `v`. Now we've seen four kinds of variables:
* Record fields like ``$shape``
* Out-of-stream variables like ``@sum``
* Local variables like ``k``
* Built-in context variables like ``NF`` and ``NR``
* Record fields like `$shape`
* Out-of-stream variables like `@sum`
* Local variables like `k`
* Built-in context variables like `NF` and `NR`
If you're curious about scope and extent of local variables, you can read more in (TODO:linkify) the section on variables.
@ -159,8 +159,8 @@ If you're curious about scope and extent of local variables, you can read more i
Numbers in Miller's programming language are intended to operate with the principle of least surprise:
* Internally, numbers are either 64-bit signed integers or double-precision floating-point.
* Sums, differences, and products of integers are also integers (so ``2*3=6`` not ``6.0``) -- unless the result of the operation would overflow a 64-bit signed integer in which case the result is automatically converted to float. (If you ever want integer-to-integer arithmetic, use ``x .+ y``, ``x .* y``, etc.)
* Quotients of integers are integers if the division is exact, else floating-point: so ``6/2=3`` but ``7/2=3.5``.
* Sums, differences, and products of integers are also integers (so `2*3=6` not `6.0`) -- unless the result of the operation would overflow a 64-bit signed integer in which case the result is automatically converted to float. (If you ever want integer-to-integer arithmetic, use `x .+ y`, `x .* y`, etc.)
* Quotients of integers are integers if the division is exact, else floating-point: so `6/2=3` but `7/2=3.5`.
You can read more about this at (TODO:linkify).
@ -168,10 +168,10 @@ You can read more about this at (TODO:linkify).
In addition to types including string, number (int/float), arrays, and hashmaps, Miller varibles can also be **absent**. This is when a variable never had a value assigned to it. Miller's treatment of absent data is intended to make it easy for you to handle non-heterogeneous data. We'll see more in section (TODO:linkify) but the basic idea is:
* Adding a number to absent gives the number back. This means you don't have to put ``@sum = 0`` in your ``begin`` blocks.
* Adding a number to absent gives the number back. This means you don't have to put `@sum = 0` in your `begin` blocks.
* Any variable which has the absent value is not assigned. This means you don't have to check presence of things from one record to the next.
For example, you can sum up all the ``$a`` values across records without having to check whether they're present or not:
For example, you can sum up all the `$a` values across records without having to check whether they're present or not:
GENMD_RUN_COMMAND
mlr --json cat absent-example.json

24
docs6b/docs/randomizing-examples.md
View file

@ -5,8 +5,10 @@
Here we can chain together a few simple building blocks:
<pre>
<pre class="pre-highlight">
<b>cat expo-sample.sh</b>
</pre>
<pre class="pre-non-highlight">
# Generate 100,000 pairs of independent and identically distributed
# exponentially distributed random variables with the same rate parameter
# (namely, 2.5). Then compute histograms of one of them, along with
@ -40,14 +42,16 @@ Namely:
* Set the Miller random-number seed so this webdoc looks the same every time I regenerate it.
* Use pretty-printed tabular output.
* Use pretty-printed tabular output.
* Use ``seqgen`` to produce 100,000 records ``i=0``, ``i=1``, etc.
* Send those to a ``put`` step which defines an inverse-transform-sampling function and calls it twice, then computes the sum and product of samples.
* Use `seqgen` to produce 100,000 records `i=0`, `i=1`, etc.
* Send those to a `put` step which defines an inverse-transform-sampling function and calls it twice, then computes the sum and product of samples.
* Send those to a histogram, and from there to a bar-plotter. This is just for visualization; you could just as well output CSV and send that off to your own plotting tool, etc.
The output is as follows:
<pre>
<pre class="pre-highlight">
<b>sh expo-sample.sh</b>
</pre>
<pre class="pre-non-highlight">
bin_lo bin_hi u_count s_count
0 0.04 [64]*******************#[9554] [326]#...................[3703]
0.04 0.08 [64]*****************...[9554] [326]*****...............[3703]
@ -105,8 +109,10 @@ bin_lo bin_hi u_count s_count
Given this [word list](./data/english-words.txt), first take a look to see what the first few lines look like:
<pre>
<pre class="pre-highlight">
<b>head data/english-words.txt</b>
</pre>
<pre class="pre-non-highlight">
a
aa
aal
@ -121,8 +127,10 @@ abaca
Then the following will randomly sample ten words with four to eight characters in them:
<pre>
<pre class="pre-highlight">
<b>mlr --from data/english-words.txt --nidx filter -S 'n=strlen($1);4<=n&&n<=8' then sample -k 10</b>
</pre>
<pre class="pre-non-highlight">
thionine
birchman
mildewy
@ -141,8 +149,10 @@ These are simple *n*-grams as [described here](http://johnkerl.org/randspell/ran
The idea is that words from the input file are consumed, then taken apart and pasted back together in ways which imitate the letter-to-letter transitions found in the word list -- giving us automatically generated words in the same vein as *bromance* and *spork*:
<pre>
<pre class="pre-highlight">
<b>mlr --nidx --from ./ngrams/gsl-2000.txt put -q -f ./ngrams/ngfuncs.mlr -f ./ngrams/ng5.mlr</b>
</pre>
<pre class="pre-non-highlight">
beard
plastinguish
politicially

4
docs6b/docs/randomizing-examples.md.in
View file

@ -13,8 +13,8 @@ Namely:
* Set the Miller random-number seed so this webdoc looks the same every time I regenerate it.
* Use pretty-printed tabular output.
* Use pretty-printed tabular output.
* Use ``seqgen`` to produce 100,000 records ``i=0``, ``i=1``, etc.
* Send those to a ``put`` step which defines an inverse-transform-sampling function and calls it twice, then computes the sum and product of samples.
* Use `seqgen` to produce 100,000 records `i=0`, `i=1`, etc.
* Send those to a `put` step which defines an inverse-transform-sampling function and calls it twice, then computes the sum and product of samples.
* Send those to a histogram, and from there to a bar-plotter. This is just for visualization; you could just as well output CSV and send that off to your own plotting tool, etc.
The output is as follows:

58
docs6b/docs/record-heterogeneity.md
View file

@ -11,8 +11,10 @@ But heterogeneous data abound (today's no-SQL databases for example). Miller han
Miller simply prints a newline and a new header when there is a schema change. When there is no schema change, you get CSV per se as a special case. Likewise, Miller reads heterogeneous CSV or pretty-print input the same way. The difference between CSV and CSV-lite is that the former is RFC4180-compliant, while the latter readily handles heterogeneous data (which is non-compliant). For example:
<pre>
<pre class="pre-highlight">
<b>cat data/het.dkvp</b>
</pre>
<pre class="pre-non-highlight">
resource=/path/to/file,loadsec=0.45,ok=true
record_count=100,resource=/path/to/file
resource=/path/to/second/file,loadsec=0.32,ok=true
@ -20,8 +22,10 @@ record_count=150,resource=/path/to/second/file
resource=/some/other/path,loadsec=0.97,ok=false
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --ocsvlite cat data/het.dkvp</b>
</pre>
<pre class="pre-non-highlight">
resource,loadsec,ok
/path/to/file,0.45,true
@ -38,8 +42,10 @@ resource,loadsec,ok
/some/other/path,0.97,false
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --opprint cat data/het.dkvp</b>
</pre>
<pre class="pre-non-highlight">
resource loadsec ok
/path/to/file 0.45 true
@ -56,18 +62,22 @@ resource loadsec ok
/some/other/path 0.97 false
</pre>
Miller handles explicit header changes as just shown. If your CSV input contains ragged data -- if there are implicit header changes -- you can use ``--allow-ragged-csv-input`` (or keystroke-saver ``--ragged``). For too-short data lines, values are filled with empty string; for too-long data lines, missing field names are replaced with positional indices (counting up from 1, not 0), as follows:
Miller handles explicit header changes as just shown. If your CSV input contains ragged data -- if there are implicit header changes -- you can use `--allow-ragged-csv-input` (or keystroke-saver `--ragged`). For too-short data lines, values are filled with empty string; for too-long data lines, missing field names are replaced with positional indices (counting up from 1, not 0), as follows:
<pre>
<pre class="pre-highlight">
<b>cat data/ragged.csv</b>
</pre>
<pre class="pre-non-highlight">
a,b,c
1,2,3
4,5
6,7,8,9
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --icsv --oxtab --allow-ragged-csv-input cat data/ragged.csv</b>
</pre>
<pre class="pre-non-highlight">
a 1
b 2
c 3
@ -82,10 +92,12 @@ c 8
4 9
</pre>
You may also find Miller's ``group-like`` feature handy (see also [Verbs Reference](reference-verbs.md)):
You may also find Miller's `group-like` feature handy (see also [Verbs Reference](reference-verbs.md)):
<pre>
<pre class="pre-highlight">
<b>mlr --ocsvlite group-like data/het.dkvp</b>
</pre>
<pre class="pre-non-highlight">
resource,loadsec,ok
/path/to/file,0.45,true
/path/to/second/file,0.32,true
@ -96,8 +108,10 @@ record_count,resource
150,/path/to/second/file
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --opprint group-like data/het.dkvp</b>
</pre>
<pre class="pre-non-highlight">
resource loadsec ok
/path/to/file 0.45 true
/path/to/second/file 0.32 true
@ -112,8 +126,10 @@ record_count resource
For these formats, record-heterogeneity comes naturally:
<pre>
<pre class="pre-highlight">
<b>cat data/het.dkvp</b>
</pre>
<pre class="pre-non-highlight">
resource=/path/to/file,loadsec=0.45,ok=true
record_count=100,resource=/path/to/file
resource=/path/to/second/file,loadsec=0.32,ok=true
@ -121,8 +137,10 @@ record_count=150,resource=/path/to/second/file
resource=/some/other/path,loadsec=0.97,ok=false
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --onidx --ofs ' ' cat data/het.dkvp</b>
</pre>
<pre class="pre-non-highlight">
/path/to/file 0.45 true
100 /path/to/file
/path/to/second/file 0.32 true
@ -130,8 +148,10 @@ resource=/some/other/path,loadsec=0.97,ok=false
/some/other/path 0.97 false
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --oxtab cat data/het.dkvp</b>
</pre>
<pre class="pre-non-highlight">
resource /path/to/file
loadsec 0.45
ok true
@ -151,8 +171,10 @@ loadsec 0.97
ok false
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --oxtab group-like data/het.dkvp</b>
</pre>
<pre class="pre-non-highlight">
resource /path/to/file
loadsec 0.45
ok true
@ -174,10 +196,12 @@ resource /path/to/second/file
## For processing
Miller operates on specified fields and takes the rest along: for example, if you are sorting on the ``count`` field then all records in the input stream must have a ``count`` field but the other fields can vary, and moreover the sorted-on field name(s) don't need to be in the same position on each line:
Miller operates on specified fields and takes the rest along: for example, if you are sorting on the `count` field then all records in the input stream must have a `count` field but the other fields can vary, and moreover the sorted-on field name(s) don't need to be in the same position on each line:
<pre>
<pre class="pre-highlight">
<b>cat data/sort-het.dkvp</b>
</pre>
<pre class="pre-non-highlight">
count=500,color=green
count=600
status=ok,count=250,hours=0.22
@ -187,8 +211,10 @@ count=100,color=green
count=450
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr sort -n count data/sort-het.dkvp</b>
</pre>
<pre class="pre-non-highlight">
count=100,color=green
status=ok,count=200,hours=3.4
status=ok,count=250,hours=0.22

6
docs6b/docs/record-heterogeneity.md.in
View file

@ -22,7 +22,7 @@ GENMD_RUN_COMMAND
mlr --opprint cat data/het.dkvp
GENMD_EOF
Miller handles explicit header changes as just shown. If your CSV input contains ragged data -- if there are implicit header changes -- you can use ``--allow-ragged-csv-input`` (or keystroke-saver ``--ragged``). For too-short data lines, values are filled with empty string; for too-long data lines, missing field names are replaced with positional indices (counting up from 1, not 0), as follows:
Miller handles explicit header changes as just shown. If your CSV input contains ragged data -- if there are implicit header changes -- you can use `--allow-ragged-csv-input` (or keystroke-saver `--ragged`). For too-short data lines, values are filled with empty string; for too-long data lines, missing field names are replaced with positional indices (counting up from 1, not 0), as follows:
GENMD_RUN_COMMAND
cat data/ragged.csv
@ -32,7 +32,7 @@ GENMD_RUN_COMMAND
mlr --icsv --oxtab --allow-ragged-csv-input cat data/ragged.csv
GENMD_EOF
You may also find Miller's ``group-like`` feature handy (see also [Verbs Reference](reference-verbs.md)):
You may also find Miller's `group-like` feature handy (see also [Verbs Reference](reference-verbs.md)):
GENMD_RUN_COMMAND
mlr --ocsvlite group-like data/het.dkvp
@ -64,7 +64,7 @@ GENMD_EOF
## For processing
Miller operates on specified fields and takes the rest along: for example, if you are sorting on the ``count`` field then all records in the input stream must have a ``count`` field but the other fields can vary, and moreover the sorted-on field name(s) don't need to be in the same position on each line:
Miller operates on specified fields and takes the rest along: for example, if you are sorting on the `count` field then all records in the input stream must have a `count` field but the other fields can vary, and moreover the sorted-on field name(s) don't need to be in the same position on each line:
GENMD_RUN_COMMAND
cat data/sort-het.dkvp

4
docs6b/docs/reference-dsl-arrays.md
View file

@ -3,8 +3,10 @@
TODO
<pre>
<pre class="pre-highlight">
<b>mlr --json cat data/array-example.json</b>
</pre>
<pre class="pre-non-highlight">
{
"key": "ax04",
"samples": [45, 67, 22]

2437
docs6b/docs/reference-dsl-builtin-functions.md
View file

File diff suppressed because it is too large Load diff

2
docs6b/docs/reference-dsl-builtin-functions.md.in
View file

@ -6,7 +6,7 @@ GENMD_RUN_CONTENT_GENERATOR(mk-func-table.rb)
## List of functions
Each function takes a specific number of arguments, as shown below, except for functions marked as variadic such as ``min`` and ``max``. (The latter compute min and max of any number of numerical arguments.) There is no notion of optional or default-on-absent arguments. All argument-passing is positional rather than by name; arguments are passed by value, not by reference.
Each function takes a specific number of arguments, as shown below, except for functions marked as variadic such as `min` and `max`. (The latter compute min and max of any number of numerical arguments.) There is no notion of optional or default-on-absent arguments. All argument-passing is positional rather than by name; arguments are passed by value, not by reference.
You can get a list of all functions using **mlr -f**, with details using **mlr -F**.

6
docs6b/docs/reference-dsl-complexity.md
View file

@ -1,9 +1,9 @@
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
# DSL reference: a note on the complexity of Miller's expression language
One of Miller's strengths is its brevity: it's much quicker -- and less error-prone -- to type ``mlr stats1 -a sum -f x,y -g a,b`` than having to track summation variables as in ``awk``, or using Miller's out-of-stream variables. And the more language features Miller's put-DSL has (for-loops, if-statements, nested control structures, user-defined functions, etc.) then the *less* powerful it begins to seem: because of the other programming-language features it *doesn't* have (classes, exceptions, and so on).
One of Miller's strengths is its brevity: it's much quicker -- and less error-prone -- to type `mlr stats1 -a sum -f x,y -g a,b` than having to track summation variables as in `awk`, or using Miller's out-of-stream variables. And the more language features Miller's put-DSL has (for-loops, if-statements, nested control structures, user-defined functions, etc.) then the *less* powerful it begins to seem: because of the other programming-language features it *doesn't* have (classes, exceptions, and so on).
When I was originally prototyping Miller in 2015, the decision I had was whether to hand-code in a low-level language like C or Rust or Go, with my own hand-rolled DSL, or whether to use a higher-level language (like Python or Lua or Nim) and let the ``put`` statements be handled by the implementation language's own ``eval``: the implementation language would take the place of a DSL. Multiple performance experiments showed me I could get better throughput using the former, by a wide margin. So Miller is Go under the hood with a hand-rolled DSL.
When I was originally prototyping Miller in 2015, the decision I had was whether to hand-code in a low-level language like C or Rust or Go, with my own hand-rolled DSL, or whether to use a higher-level language (like Python or Lua or Nim) and let the `put` statements be handled by the implementation language's own `eval`: the implementation language would take the place of a DSL. Multiple performance experiments showed me I could get better throughput using the former, by a wide margin. So Miller is Go under the hood with a hand-rolled DSL.
I do want to keep focusing on what Miller is good at -- concise notation, low latency, and high throughput -- and not add too much in terms of high-level-language features to the DSL. That said, some sort of customizability is a basic thing to want. As of 4.1.0 we have recursive for/while/if structures on about the same complexity level as ``awk``; as of 5.0.0 we have user-defined functions and map-valued variables, again on about the same complexity level as ``awk`` along with optional type-declaration syntax; as of Miller 6 we have full support for arrays. While I'm excited by these powerful language features, I hope to keep new features focused on Miller's sweet spot which is speed plus simplicity.
I do want to keep focusing on what Miller is good at -- concise notation, low latency, and high throughput -- and not add too much in terms of high-level-language features to the DSL. That said, some sort of customizability is a basic thing to want. As of 4.1.0 we have recursive for/while/if structures on about the same complexity level as `awk`; as of 5.0.0 we have user-defined functions and map-valued variables, again on about the same complexity level as `awk` along with optional type-declaration syntax; as of Miller 6 we have full support for arrays. While I'm excited by these powerful language features, I hope to keep new features focused on Miller's sweet spot which is speed plus simplicity.

6
docs6b/docs/reference-dsl-complexity.md.in
View file

@ -1,8 +1,8 @@
# DSL reference: a note on the complexity of Miller's expression language
One of Miller's strengths is its brevity: it's much quicker -- and less error-prone -- to type ``mlr stats1 -a sum -f x,y -g a,b`` than having to track summation variables as in ``awk``, or using Miller's out-of-stream variables. And the more language features Miller's put-DSL has (for-loops, if-statements, nested control structures, user-defined functions, etc.) then the *less* powerful it begins to seem: because of the other programming-language features it *doesn't* have (classes, exceptions, and so on).
One of Miller's strengths is its brevity: it's much quicker -- and less error-prone -- to type `mlr stats1 -a sum -f x,y -g a,b` than having to track summation variables as in `awk`, or using Miller's out-of-stream variables. And the more language features Miller's put-DSL has (for-loops, if-statements, nested control structures, user-defined functions, etc.) then the *less* powerful it begins to seem: because of the other programming-language features it *doesn't* have (classes, exceptions, and so on).
When I was originally prototyping Miller in 2015, the decision I had was whether to hand-code in a low-level language like C or Rust or Go, with my own hand-rolled DSL, or whether to use a higher-level language (like Python or Lua or Nim) and let the ``put`` statements be handled by the implementation language's own ``eval``: the implementation language would take the place of a DSL. Multiple performance experiments showed me I could get better throughput using the former, by a wide margin. So Miller is Go under the hood with a hand-rolled DSL.
When I was originally prototyping Miller in 2015, the decision I had was whether to hand-code in a low-level language like C or Rust or Go, with my own hand-rolled DSL, or whether to use a higher-level language (like Python or Lua or Nim) and let the `put` statements be handled by the implementation language's own `eval`: the implementation language would take the place of a DSL. Multiple performance experiments showed me I could get better throughput using the former, by a wide margin. So Miller is Go under the hood with a hand-rolled DSL.
I do want to keep focusing on what Miller is good at -- concise notation, low latency, and high throughput -- and not add too much in terms of high-level-language features to the DSL. That said, some sort of customizability is a basic thing to want. As of 4.1.0 we have recursive for/while/if structures on about the same complexity level as ``awk``; as of 5.0.0 we have user-defined functions and map-valued variables, again on about the same complexity level as ``awk`` along with optional type-declaration syntax; as of Miller 6 we have full support for arrays. While I'm excited by these powerful language features, I hope to keep new features focused on Miller's sweet spot which is speed plus simplicity.
I do want to keep focusing on what Miller is good at -- concise notation, low latency, and high throughput -- and not add too much in terms of high-level-language features to the DSL. That said, some sort of customizability is a basic thing to want. As of 4.1.0 we have recursive for/while/if structures on about the same complexity level as `awk`; as of 5.0.0 we have user-defined functions and map-valued variables, again on about the same complexity level as `awk` along with optional type-declaration syntax; as of Miller 6 we have full support for arrays. While I'm excited by these powerful language features, I hope to keep new features focused on Miller's sweet spot which is speed plus simplicity.

158
docs6b/docs/reference-dsl-control-structures.md
View file

@ -3,10 +3,12 @@
## Pattern-action blocks
These are reminiscent of ``awk`` syntax. They can be used to allow assignments to be done only when appropriate -- e.g. for math-function domain restrictions, regex-matching, and so on:
These are reminiscent of `awk` syntax. They can be used to allow assignments to be done only when appropriate -- e.g. for math-function domain restrictions, regex-matching, and so on:
<pre>
<pre class="pre-highlight">
<b>mlr cat data/put-gating-example-1.dkvp</b>
</pre>
<pre class="pre-non-highlight">
x=-1
x=0
x=1
@ -14,8 +16,10 @@ x=2
x=3
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr put '$x > 0.0 { $y = log10($x); $z = sqrt($y) }' data/put-gating-example-1.dkvp</b>
</pre>
<pre class="pre-non-highlight">
x=-1
x=0
x=1,y=0,z=0
@ -23,58 +27,66 @@ x=2,y=0.3010299956639812,z=0.5486620049392715
x=3,y=0.4771212547196624,z=0.6907396432228734
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr cat data/put-gating-example-2.dkvp</b>
</pre>
<pre class="pre-non-highlight">
a=abc_123
a=some other name
a=xyz_789
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr put '</b>
<b> $a =~ "([a-z]+)_([0-9]+)" {</b>
<b> $b = "left_\1"; $c = "right_\2"</b>
<b> }' \</b>
<b> data/put-gating-example-2.dkvp</b>
</pre>
<pre class="pre-non-highlight">
a=abc_123,b=left_\1,c=right_\2
a=some other name
a=xyz_789,b=left_\1,c=right_\2
</pre>
This produces heteregenous output which Miller, of course, has no problems with (see [Record Heterogeneity](record-heterogeneity.md)). But if you want homogeneous output, the curly braces can be replaced with a semicolon between the expression and the body statements. This causes ``put`` to evaluate the boolean expression (along with any side effects, namely, regex-captures ``\1``, ``\2``, etc.) but doesn't use it as a criterion for whether subsequent assignments should be executed. Instead, subsequent assignments are done unconditionally:
This produces heteregenous output which Miller, of course, has no problems with (see [Record Heterogeneity](record-heterogeneity.md)). But if you want homogeneous output, the curly braces can be replaced with a semicolon between the expression and the body statements. This causes `put` to evaluate the boolean expression (along with any side effects, namely, regex-captures `\1`, `\2`, etc.) but doesn't use it as a criterion for whether subsequent assignments should be executed. Instead, subsequent assignments are done unconditionally:
<pre>
<pre class="pre-highlight">
<b>mlr put '$x > 0.0; $y = log10($x); $z = sqrt($y)' data/put-gating-example-1.dkvp</b>
</pre>
<pre class="pre-non-highlight">
x=1,y=0,z=0
x=2,y=0.3010299956639812,z=0.5486620049392715
x=3,y=0.4771212547196624,z=0.6907396432228734
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr put '</b>
<b> $a =~ "([a-z]+)_([0-9]+)";</b>
<b> $b = "left_\1";</b>
<b> $c = "right_\2"</b>
<b>' data/put-gating-example-2.dkvp</b>
</pre>
<pre class="pre-non-highlight">
a=abc_123,b=left_\1,c=right_\2
a=xyz_789,b=left_\1,c=right_\2
</pre>
## If-statements
These are again reminiscent of ``awk``. Pattern-action blocks are a special case of ``if`` with no ``elif`` or ``else`` blocks, no ``if`` keyword, and parentheses optional around the boolean expression:
These are again reminiscent of `awk`. Pattern-action blocks are a special case of `if` with no `elif` or `else` blocks, no `if` keyword, and parentheses optional around the boolean expression:
<pre>
<pre class="pre-highlight">
<b>mlr put 'NR == 4 {$foo = "bar"}'</b>
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr put 'if (NR == 4) {$foo = "bar"}'</b>
</pre>
Compound statements use ``elif`` (rather than ``elsif`` or ``else if``):
Compound statements use `elif` (rather than `elsif` or `else if`):
<pre>
<pre class="pre-non-highlight">
mlr put '
if (NR == 2) {
...
@ -90,19 +102,21 @@ mlr put '
## While and do-while loops
Miller's ``while`` and ``do-while`` are unsurprising in comparison to various languages, as are ``break`` and ``continue``:
Miller's `while` and `do-while` are unsurprising in comparison to various languages, as are `break` and `continue`:
<pre>
<pre class="pre-highlight">
<b>echo x=1,y=2 | mlr put '</b>
<b> while (NF < 10) {</b>
<b> $[NF+1] = ""</b>
<b> }</b>
<b> $foo = "bar"</b>
<b>'</b>
</pre>
<pre class="pre-non-highlight">
x=1,y=2,3=,4=,5=,6=,7=,8=,9=,10=,foo=bar
</pre>
<pre>
<pre class="pre-highlight">
<b>echo x=1,y=2 | mlr put '</b>
<b> do {</b>
<b> $[NF+1] = "";</b>
@ -112,24 +126,26 @@ x=1,y=2,3=,4=,5=,6=,7=,8=,9=,10=,foo=bar
<b> } while (NF < 10);</b>
<b> $foo = "bar"</b>
<b>'</b>
</pre>
<pre class="pre-non-highlight">
x=1,y=2,3=,4=,5=,foo=bar
</pre>
A ``break`` or ``continue`` within nested conditional blocks or if-statements will, of course, propagate to the innermost loop enclosing them, if any. A ``break`` or ``continue`` outside a loop is a syntax error that will be flagged as soon as the expression is parsed, before any input records are ingested.
The existence of ``while``, ``do-while``, and ``for`` loops in Miller's DSL means that you can create infinite-loop scenarios inadvertently. In particular, please recall that DSL statements are executed once if in ``begin`` or ``end`` blocks, and once *per record* otherwise. For example, **while (NR < 10) will never terminate as NR is only incremented between records**.
A `break` or `continue` within nested conditional blocks or if-statements will, of course, propagate to the innermost loop enclosing them, if any. A `break` or `continue` outside a loop is a syntax error that will be flagged as soon as the expression is parsed, before any input records are ingested.
The existence of `while`, `do-while`, and `for` loops in Miller's DSL means that you can create infinite-loop scenarios inadvertently. In particular, please recall that DSL statements are executed once if in `begin` or `end` blocks, and once *per record* otherwise. For example, **while (NR < 10) will never terminate as NR is only incremented between records**.
## For-loops
While Miller's ``while`` and ``do-while`` statements are much as in many other languages, ``for`` loops are more idiosyncratic to Miller. They are loops over key-value pairs, whether in stream records, out-of-stream variables, local variables, or map-literals: more reminiscent of ``foreach``, as in (for example) PHP. There are **for-loops over map keys** and **for-loops over key-value tuples**. Additionally, Miller has a **C-style triple-for loop** with initialize, test, and update statements.
While Miller's `while` and `do-while` statements are much as in many other languages, `for` loops are more idiosyncratic to Miller. They are loops over key-value pairs, whether in stream records, out-of-stream variables, local variables, or map-literals: more reminiscent of `foreach`, as in (for example) PHP. There are **for-loops over map keys** and **for-loops over key-value tuples**. Additionally, Miller has a **C-style triple-for loop** with initialize, test, and update statements.
As with ``while`` and ``do-while``, a ``break`` or ``continue`` within nested control structures will propagate to the innermost loop enclosing them, if any, and a ``break`` or ``continue`` outside a loop is a syntax error that will be flagged as soon as the expression is parsed, before any input records are ingested.
As with `while` and `do-while`, a `break` or `continue` within nested control structures will propagate to the innermost loop enclosing them, if any, and a `break` or `continue` outside a loop is a syntax error that will be flagged as soon as the expression is parsed, before any input records are ingested.
Key-only for-loops
................................................................
The ``key`` variable is always bound to the *key* of key-value pairs:
The `key` variable is always bound to the *key* of key-value pairs:
<pre>
<pre class="pre-highlight">
<b>mlr --from data/small put '</b>
<b> print "NR = ".NR;</b>
<b> for (key in $*) {</b>
@ -138,6 +154,8 @@ The ``key`` variable is always bound to the *key* of key-value pairs:
<b> }</b>
<b></b>
<b>'</b>
</pre>
<pre class="pre-non-highlight">
NR = 1
key:a value:pan
key:b value:pan
@ -175,7 +193,7 @@ NR = 5
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr -n put '</b>
<b> end {</b>
<b> o = {1:2, 3:{4:5}};</b>
@ -184,26 +202,30 @@ a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
<b> }</b>
<b> }</b>
<b>'</b>
</pre>
<pre class="pre-non-highlight">
key:1 valuetype:int
key:3 valuetype:map
</pre>
Note that the value corresponding to a given key may be gotten as through a **computed field name** using square brackets as in ``$[key]`` for stream records, or by indexing the looped-over variable using square brackets.
Note that the value corresponding to a given key may be gotten as through a **computed field name** using square brackets as in `$[key]` for stream records, or by indexing the looped-over variable using square brackets.
Key-value for-loops
................................................................
Single-level keys may be gotten at using either ``for(k,v)`` or ``for((k),v)``; multi-level keys may be gotten at using ``for((k1,k2,k3),v)`` and so on. The ``v`` variable will be bound to to a scalar value (a string or a number) if the map stops at that level, or to a map-valued variable if the map goes deeper. If the map isn't deep enough then the loop body won't be executed.
Single-level keys may be gotten at using either `for(k,v)` or `for((k),v)`; multi-level keys may be gotten at using `for((k1,k2,k3),v)` and so on. The `v` variable will be bound to to a scalar value (a string or a number) if the map stops at that level, or to a map-valued variable if the map goes deeper. If the map isn't deep enough then the loop body won't be executed.
<pre>
<pre class="pre-highlight">
<b>cat data/for-srec-example.tbl</b>
</pre>
<pre class="pre-non-highlight">
label1 label2 f1 f2 f3
blue green 100 240 350
red green 120 11 195
yellow blue 140 0 240
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --pprint --from data/for-srec-example.tbl put '</b>
<b> $sum1 = $f1 + $f2 + $f3;</b>
<b> $sum2 = 0;</b>
@ -215,14 +237,18 @@ yellow blue 140 0 240
<b> }</b>
<b> }</b>
<b>'</b>
</pre>
<pre class="pre-non-highlight">
label1 label2 f1 f2 f3 sum1 sum2 sum3
blue green 100 240 350 690 690 690
red green 120 11 195 326 326 326
yellow blue 140 0 240 380 380 380
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --from data/small --opprint put 'for (k,v in $*) { $[k."_type"] = typeof(v) }'</b>
</pre>
<pre class="pre-non-highlight">
a b i x y a_type b_type i_type x_type y_type
pan pan 1 0.3467901443380824 0.7268028627434533 string string int float float
eks pan 2 0.7586799647899636 0.5221511083334797 string string int float float
@ -231,11 +257,11 @@ eks wye 4 0.38139939387114097 0.13418874328430463 string string int float fl
wye pan 5 0.5732889198020006 0.8636244699032729 string string int float float
</pre>
Note that the value of the current field in the for-loop can be gotten either using the bound variable ``value``, or through a **computed field name** using square brackets as in ``$[key]``.
Note that the value of the current field in the for-loop can be gotten either using the bound variable `value`, or through a **computed field name** using square brackets as in `$[key]`.
Important note: to avoid inconsistent looping behavior in case you're setting new fields (and/or unsetting existing ones) while looping over the record, **Miller makes a copy of the record before the loop: loop variables are bound from the copy and all other reads/writes involve the record itself**:
<pre>
<pre class="pre-highlight">
<b>mlr --from data/small --opprint put '</b>
<b> $sum1 = 0;</b>
<b> $sum2 = 0;</b>
@ -246,6 +272,8 @@ Important note: to avoid inconsistent looping behavior in case you're setting ne
<b> }</b>
<b> }</b>
<b>'</b>
</pre>
<pre class="pre-non-highlight">
a b i x y sum1 sum2
pan pan 1 0.3467901443380824 0.7268028627434533 2.0735930070815356 8.294372028326142
eks pan 2 0.7586799647899636 0.5221511083334797 3.280831073123443 13.123324292493772
@ -256,7 +284,7 @@ wye pan 5 0.5732889198020006 0.8636244699032729 6.436913389705273 25.74765355
It can be confusing to modify the stream record while iterating over a copy of it, so instead you might find it simpler to use a local variable in the loop and only update the stream record after the loop:
<pre>
<pre class="pre-highlight">
<b>mlr --from data/small --opprint put '</b>
<b> sum = 0;</b>
<b> for (k,v in $*) {</b>
@ -266,6 +294,8 @@ It can be confusing to modify the stream record while iterating over a copy of i
<b> }</b>
<b> $sum = sum</b>
<b>'</b>
</pre>
<pre class="pre-non-highlight">
a b i x y sum
pan pan 1 0.3467901443380824 0.7268028627434533 2.0735930070815356
eks pan 2 0.7586799647899636 0.5221511083334797 3.280831073123443
@ -274,9 +304,9 @@ eks wye 4 0.38139939387114097 0.13418874328430463 4.515588137155445
wye pan 5 0.5732889198020006 0.8636244699032729 6.436913389705273
</pre>
You can also start iterating on sub-hashmaps of an out-of-stream or local variable; you can loop over nested keys; you can loop over all out-of-stream variables. The bound variables are bound to a copy of the sub-hashmap as it was before the loop started. The sub-hashmap is specified by square-bracketed indices after ``in``, and additional deeper indices are bound to loop key-variables. The terminal values are bound to the loop value-variable whenever the keys are not too shallow. The value-variable may refer to a terminal (string, number) or it may be map-valued if the map goes deeper. Example indexing is as follows:
You can also start iterating on sub-hashmaps of an out-of-stream or local variable; you can loop over nested keys; you can loop over all out-of-stream variables. The bound variables are bound to a copy of the sub-hashmap as it was before the loop started. The sub-hashmap is specified by square-bracketed indices after `in`, and additional deeper indices are bound to loop key-variables. The terminal values are bound to the loop value-variable whenever the keys are not too shallow. The value-variable may refer to a terminal (string, number) or it may be map-valued if the map goes deeper. Example indexing is as follows:
<pre>
<pre class="pre-non-highlight">
# Parentheses are optional for single key:
for (k1, v in @a["b"]["c"]) { ... }
for ((k1), v in @a["b"]["c"]) { ... }
@ -287,9 +317,9 @@ for ((k1, k2, k3), v in @a { ... } # Loop over variable starting from
for ((k1, k2, k3), v in @* { ... } # Loop over all variables (k1 is bound to basename)
</pre>
That's confusing in the abstract, so a concrete example is in order. Suppose the out-of-stream variable ``@myvar`` is populated as follows:
That's confusing in the abstract, so a concrete example is in order. Suppose the out-of-stream variable `@myvar` is populated as follows:
<pre>
<pre class="pre-highlight">
<b>mlr -n put --jknquoteint -q '</b>
<b> begin {</b>
<b> @myvar = {</b>
@ -300,6 +330,8 @@ That's confusing in the abstract, so a concrete example is in order. Suppose the
<b> }</b>
<b> end { dump }</b>
<b>'</b>
</pre>
<pre class="pre-non-highlight">
{
"myvar": {
"1": 2,
@ -317,7 +349,7 @@ That's confusing in the abstract, so a concrete example is in order. Suppose the
Then we can get at various values as follows:
<pre>
<pre class="pre-highlight">
<b>mlr -n put --jknquoteint -q '</b>
<b> begin {</b>
<b> @myvar = {</b>
@ -334,12 +366,14 @@ Then we can get at various values as follows:
<b> }</b>
<b> }</b>
<b>'</b>
</pre>
<pre class="pre-non-highlight">
key=1,valuetype=int
key=3,valuetype=map
key=6,valuetype=map
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr -n put --jknquoteint -q '</b>
<b> begin {</b>
<b> @myvar = {</b>
@ -357,11 +391,13 @@ key=6,valuetype=map
<b> }</b>
<b> }</b>
<b>'</b>
</pre>
<pre class="pre-non-highlight">
key1=3,key2=4,valuetype=int
key1=6,key2=7,valuetype=map
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr -n put --jknquoteint -q '</b>
<b> begin {</b>
<b> @myvar = {</b>
@ -379,6 +415,8 @@ key1=6,key2=7,valuetype=map
<b> }</b>
<b> }</b>
<b>'</b>
</pre>
<pre class="pre-non-highlight">
key1=7,key2=8,valuetype=int
</pre>
@ -387,7 +425,7 @@ C-style triple-for loops
These are supported as follows:
<pre>
<pre class="pre-highlight">
<b>mlr --from data/small --opprint put '</b>
<b> num suma = 0;</b>
<b> for (a = 1; a <= NR; a += 1) {</b>
@ -395,6 +433,8 @@ These are supported as follows:
<b> }</b>
<b> $suma = suma;</b>
<b>'</b>
</pre>
<pre class="pre-non-highlight">
a b i x y suma
pan pan 1 0.3467901443380824 0.7268028627434533 1
eks pan 2 0.7586799647899636 0.5221511083334797 3
@ -403,7 +443,7 @@ eks wye 4 0.38139939387114097 0.13418874328430463 10
wye pan 5 0.5732889198020006 0.8636244699032729 15
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --from data/small --opprint put '</b>
<b> num suma = 0;</b>
<b> num sumb = 0;</b>
@ -414,6 +454,8 @@ wye pan 5 0.5732889198020006 0.8636244699032729 15
<b> $suma = suma;</b>
<b> $sumb = sumb;</b>
<b>'</b>
</pre>
<pre class="pre-non-highlight">
a b i x y suma sumb
pan pan 1 0.3467901443380824 0.7268028627434533 1 1
eks pan 2 0.7586799647899636 0.5221511083334797 3 3
@ -424,26 +466,28 @@ wye pan 5 0.5732889198020006 0.8636244699032729 15 31
Notes:
* In ``for (start; continuation; update) { body }``, the start, continuation, and update statements may be empty, single statements, or multiple comma-separated statements. If the continuation is empty (e.g. ``for(i=1;;i+=1)``) it defaults to true.
* In `for (start; continuation; update) { body }`, the start, continuation, and update statements may be empty, single statements, or multiple comma-separated statements. If the continuation is empty (e.g. `for(i=1;;i+=1)`) it defaults to true.
* In particular, you may use ``$``-variables and/or ``@``-variables in the start, continuation, and/or update steps (as well as the body, of course).
* In particular, you may use `$`-variables and/or `@`-variables in the start, continuation, and/or update steps (as well as the body, of course).
* The typedecls such as ``int`` or ``num`` are optional. If a typedecl is provided (for a local variable), it binds a variable scoped to the for-loop regardless of whether a same-name variable is present in outer scope. If a typedecl is not provided, then the variable is scoped to the for-loop if no same-name variable is present in outer scope, or if a same-name variable is present in outer scope then it is modified.
* The typedecls such as `int` or `num` are optional. If a typedecl is provided (for a local variable), it binds a variable scoped to the for-loop regardless of whether a same-name variable is present in outer scope. If a typedecl is not provided, then the variable is scoped to the for-loop if no same-name variable is present in outer scope, or if a same-name variable is present in outer scope then it is modified.
* Miller has no ``++`` or ``--`` operators.
* Miller has no `++` or `--` operators.
* As with all for/if/while statements in Miller, the curly braces are required even if the body is a single statement, or empty.
## Begin/end blocks
Miller supports an ``awk``-like ``begin/end`` syntax. The statements in the ``begin`` block are executed before any input records are read; the statements in the ``end`` block are executed after the last input record is read. (If you want to execute some statement at the start of each file, not at the start of the first file as with ``begin``, you might use a pattern/action block of the form ``FNR == 1 { ... }``.) All statements outside of ``begin`` or ``end`` are, of course, executed on every input record. Semicolons separate statements inside or outside of begin/end blocks; semicolons are required between begin/end block bodies and any subsequent statement. For example:
Miller supports an `awk`-like `begin/end` syntax. The statements in the `begin` block are executed before any input records are read; the statements in the `end` block are executed after the last input record is read. (If you want to execute some statement at the start of each file, not at the start of the first file as with `begin`, you might use a pattern/action block of the form `FNR == 1 { ... }`.) All statements outside of `begin` or `end` are, of course, executed on every input record. Semicolons separate statements inside or outside of begin/end blocks; semicolons are required between begin/end block bodies and any subsequent statement. For example:
<pre>
<pre class="pre-highlight">
<b>mlr put '</b>
<b> begin { @sum = 0 };</b>
<b> @x_sum += $x;</b>
<b> end { emit @x_sum }</b>
<b>' ./data/small</b>
</pre>
<pre class="pre-non-highlight">
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
@ -452,13 +496,15 @@ a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
x_sum=2.264761728567491
</pre>
Since uninitialized out-of-stream variables default to 0 for addition/substraction and 1 for multiplication when they appear on expression right-hand sides (not quite as in ``awk``, where they'd default to 0 either way), the above can be written more succinctly as
Since uninitialized out-of-stream variables default to 0 for addition/substraction and 1 for multiplication when they appear on expression right-hand sides (not quite as in `awk`, where they'd default to 0 either way), the above can be written more succinctly as
<pre>
<pre class="pre-highlight">
<b>mlr put '</b>
<b> @x_sum += $x;</b>
<b> end { emit @x_sum }</b>
<b>' ./data/small</b>
</pre>
<pre class="pre-non-highlight">
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
@ -467,19 +513,21 @@ a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
x_sum=2.264761728567491
</pre>
The **put -q** option is a shorthand which suppresses printing of each output record, with only ``emit`` statements being output. So to get only summary outputs, one could write
The **put -q** option is a shorthand which suppresses printing of each output record, with only `emit` statements being output. So to get only summary outputs, one could write
<pre>
<pre class="pre-highlight">
<b>mlr put -q '</b>
<b> @x_sum += $x;</b>
<b> end { emit @x_sum }</b>
<b>' ./data/small</b>
</pre>
<pre class="pre-non-highlight">
x_sum=2.264761728567491
</pre>
We can do similarly with multiple out-of-stream variables:
<pre>
<pre class="pre-highlight">
<b>mlr put -q '</b>
<b> @x_count += 1;</b>
<b> @x_sum += $x;</b>
@ -488,16 +536,20 @@ We can do similarly with multiple out-of-stream variables:
<b> emit @x_sum;</b>
<b> }</b>
<b>' ./data/small</b>
</pre>
<pre class="pre-non-highlight">
x_count=5
x_sum=2.264761728567491
</pre>
This is of course not much different than
<pre>
<pre class="pre-highlight">
<b>mlr stats1 -a count,sum -f x ./data/small</b>
</pre>
<pre class="pre-non-highlight">
x_count=5,x_sum=2.264761728567491
</pre>
Note that it's a syntax error for begin/end blocks to refer to field names (beginning with ``$``), since these execute outside the context of input records.
Note that it's a syntax error for begin/end blocks to refer to field names (beginning with `$`), since these execute outside the context of input records.

55
docs6b/docs/reference-dsl-control-structures.md.in
View file

@ -2,7 +2,7 @@
## Pattern-action blocks
These are reminiscent of ``awk`` syntax. They can be used to allow assignments to be done only when appropriate -- e.g. for math-function domain restrictions, regex-matching, and so on:
These are reminiscent of `awk` syntax. They can be used to allow assignments to be done only when appropriate -- e.g. for math-function domain restrictions, regex-matching, and so on:
GENMD_RUN_COMMAND
mlr cat data/put-gating-example-1.dkvp
@ -24,7 +24,7 @@ mlr put '
data/put-gating-example-2.dkvp
GENMD_EOF
This produces heteregenous output which Miller, of course, has no problems with (see [Record Heterogeneity](record-heterogeneity.md)). But if you want homogeneous output, the curly braces can be replaced with a semicolon between the expression and the body statements. This causes ``put`` to evaluate the boolean expression (along with any side effects, namely, regex-captures ``\1``, ``\2``, etc.) but doesn't use it as a criterion for whether subsequent assignments should be executed. Instead, subsequent assignments are done unconditionally:
This produces heteregenous output which Miller, of course, has no problems with (see [Record Heterogeneity](record-heterogeneity.md)). But if you want homogeneous output, the curly braces can be replaced with a semicolon between the expression and the body statements. This causes `put` to evaluate the boolean expression (along with any side effects, namely, regex-captures `\1`, `\2`, etc.) but doesn't use it as a criterion for whether subsequent assignments should be executed. Instead, subsequent assignments are done unconditionally:
GENMD_RUN_COMMAND
mlr put '$x > 0.0; $y = log10($x); $z = sqrt($y)' data/put-gating-example-1.dkvp
@ -40,7 +40,7 @@ GENMD_EOF
## If-statements
These are again reminiscent of ``awk``. Pattern-action blocks are a special case of ``if`` with no ``elif`` or ``else`` blocks, no ``if`` keyword, and parentheses optional around the boolean expression:
These are again reminiscent of `awk`. Pattern-action blocks are a special case of `if` with no `elif` or `else` blocks, no `if` keyword, and parentheses optional around the boolean expression:
GENMD_SHOW_COMMAND
mlr put 'NR == 4 {$foo = "bar"}'
@ -50,42 +50,40 @@ GENMD_SHOW_COMMAND
mlr put 'if (NR == 4) {$foo = "bar"}'
GENMD_EOF
Compound statements use ``elif`` (rather than ``elsif`` or ``else if``):
Compound statements use `elif` (rather than `elsif` or `else if`):
GENMD_INCLUDE_ESCAPED(data/if-chain.sh)
## While and do-while loops
Miller's ``while`` and ``do-while`` are unsurprising in comparison to various languages, as are ``break`` and ``continue``:
Miller's `while` and `do-while` are unsurprising in comparison to various languages, as are `break` and `continue`:
GENMD_INCLUDE_AND_RUN_ESCAPED(data/while-example-1.sh)
GENMD_INCLUDE_AND_RUN_ESCAPED(data/while-example-2.sh)
A ``break`` or ``continue`` within nested conditional blocks or if-statements will, of course, propagate to the innermost loop enclosing them, if any. A ``break`` or ``continue`` outside a loop is a syntax error that will be flagged as soon as the expression is parsed, before any input records are ingested.
The existence of ``while``, ``do-while``, and ``for`` loops in Miller's DSL means that you can create infinite-loop scenarios inadvertently. In particular, please recall that DSL statements are executed once if in ``begin`` or ``end`` blocks, and once *per record* otherwise. For example, **while (NR < 10) will never terminate as NR is only incremented between records**.
A `break` or `continue` within nested conditional blocks or if-statements will, of course, propagate to the innermost loop enclosing them, if any. A `break` or `continue` outside a loop is a syntax error that will be flagged as soon as the expression is parsed, before any input records are ingested.
The existence of `while`, `do-while`, and `for` loops in Miller's DSL means that you can create infinite-loop scenarios inadvertently. In particular, please recall that DSL statements are executed once if in `begin` or `end` blocks, and once *per record* otherwise. For example, **while (NR < 10) will never terminate as NR is only incremented between records**.
## For-loops
While Miller's ``while`` and ``do-while`` statements are much as in many other languages, ``for`` loops are more idiosyncratic to Miller. They are loops over key-value pairs, whether in stream records, out-of-stream variables, local variables, or map-literals: more reminiscent of ``foreach``, as in (for example) PHP. There are **for-loops over map keys** and **for-loops over key-value tuples**. Additionally, Miller has a **C-style triple-for loop** with initialize, test, and update statements.
While Miller's `while` and `do-while` statements are much as in many other languages, `for` loops are more idiosyncratic to Miller. They are loops over key-value pairs, whether in stream records, out-of-stream variables, local variables, or map-literals: more reminiscent of `foreach`, as in (for example) PHP. There are **for-loops over map keys** and **for-loops over key-value tuples**. Additionally, Miller has a **C-style triple-for loop** with initialize, test, and update statements.
As with ``while`` and ``do-while``, a ``break`` or ``continue`` within nested control structures will propagate to the innermost loop enclosing them, if any, and a ``break`` or ``continue`` outside a loop is a syntax error that will be flagged as soon as the expression is parsed, before any input records are ingested.
As with `while` and `do-while`, a `break` or `continue` within nested control structures will propagate to the innermost loop enclosing them, if any, and a `break` or `continue` outside a loop is a syntax error that will be flagged as soon as the expression is parsed, before any input records are ingested.
Key-only for-loops
................................................................
### Key-only for-loops
The ``key`` variable is always bound to the *key* of key-value pairs:
The `key` variable is always bound to the *key* of key-value pairs:
GENMD_INCLUDE_AND_RUN_ESCAPED(data/single-for-example-1.sh)
GENMD_INCLUDE_AND_RUN_ESCAPED(data/single-for-example-2.sh)
Note that the value corresponding to a given key may be gotten as through a **computed field name** using square brackets as in ``$[key]`` for stream records, or by indexing the looped-over variable using square brackets.
Note that the value corresponding to a given key may be gotten as through a **computed field name** using square brackets as in `$[key]` for stream records, or by indexing the looped-over variable using square brackets.
Key-value for-loops
................................................................
### Key-value for-loops
Single-level keys may be gotten at using either ``for(k,v)`` or ``for((k),v)``; multi-level keys may be gotten at using ``for((k1,k2,k3),v)`` and so on. The ``v`` variable will be bound to to a scalar value (a string or a number) if the map stops at that level, or to a map-valued variable if the map goes deeper. If the map isn't deep enough then the loop body won't be executed.
Single-level keys may be gotten at using either `for(k,v)` or `for((k),v)`; multi-level keys may be gotten at using `for((k1,k2,k3),v)` and so on. The `v` variable will be bound to to a scalar value (a string or a number) if the map stops at that level, or to a map-valued variable if the map goes deeper. If the map isn't deep enough then the loop body won't be executed.
GENMD_RUN_COMMAND
cat data/for-srec-example.tbl
@ -97,7 +95,7 @@ GENMD_RUN_COMMAND
mlr --from data/small --opprint put 'for (k,v in $*) { $[k."_type"] = typeof(v) }'
GENMD_EOF
Note that the value of the current field in the for-loop can be gotten either using the bound variable ``value``, or through a **computed field name** using square brackets as in ``$[key]``.
Note that the value of the current field in the for-loop can be gotten either using the bound variable `value`, or through a **computed field name** using square brackets as in `$[key]`.
Important note: to avoid inconsistent looping behavior in case you're setting new fields (and/or unsetting existing ones) while looping over the record, **Miller makes a copy of the record before the loop: loop variables are bound from the copy and all other reads/writes involve the record itself**:
@ -107,11 +105,11 @@ It can be confusing to modify the stream record while iterating over a copy of i
GENMD_INCLUDE_AND_RUN_ESCAPED(data/for-srec-example-3.sh)
You can also start iterating on sub-hashmaps of an out-of-stream or local variable; you can loop over nested keys; you can loop over all out-of-stream variables. The bound variables are bound to a copy of the sub-hashmap as it was before the loop started. The sub-hashmap is specified by square-bracketed indices after ``in``, and additional deeper indices are bound to loop key-variables. The terminal values are bound to the loop value-variable whenever the keys are not too shallow. The value-variable may refer to a terminal (string, number) or it may be map-valued if the map goes deeper. Example indexing is as follows:
You can also start iterating on sub-hashmaps of an out-of-stream or local variable; you can loop over nested keys; you can loop over all out-of-stream variables. The bound variables are bound to a copy of the sub-hashmap as it was before the loop started. The sub-hashmap is specified by square-bracketed indices after `in`, and additional deeper indices are bound to loop key-variables. The terminal values are bound to the loop value-variable whenever the keys are not too shallow. The value-variable may refer to a terminal (string, number) or it may be map-valued if the map goes deeper. Example indexing is as follows:
GENMD_INCLUDE_ESCAPED(data/for-oosvar-example-0a.txt)
That's confusing in the abstract, so a concrete example is in order. Suppose the out-of-stream variable ``@myvar`` is populated as follows:
That's confusing in the abstract, so a concrete example is in order. Suppose the out-of-stream variable `@myvar` is populated as follows:
GENMD_INCLUDE_AND_RUN_ESCAPED(data/for-oosvar-example-0b.sh)
@ -123,8 +121,7 @@ GENMD_INCLUDE_AND_RUN_ESCAPED(data/for-oosvar-example-0d.sh)
GENMD_INCLUDE_AND_RUN_ESCAPED(data/for-oosvar-example-0e.sh)
C-style triple-for loops
................................................................
### C-style triple-for loops
These are supported as follows:
@ -134,27 +131,27 @@ GENMD_INCLUDE_AND_RUN_ESCAPED(data/triple-for-example-2.sh)
Notes:
* In ``for (start; continuation; update) { body }``, the start, continuation, and update statements may be empty, single statements, or multiple comma-separated statements. If the continuation is empty (e.g. ``for(i=1;;i+=1)``) it defaults to true.
* In `for (start; continuation; update) { body }`, the start, continuation, and update statements may be empty, single statements, or multiple comma-separated statements. If the continuation is empty (e.g. `for(i=1;;i+=1)`) it defaults to true.
* In particular, you may use ``$``-variables and/or ``@``-variables in the start, continuation, and/or update steps (as well as the body, of course).
* In particular, you may use `$`-variables and/or `@`-variables in the start, continuation, and/or update steps (as well as the body, of course).
* The typedecls such as ``int`` or ``num`` are optional. If a typedecl is provided (for a local variable), it binds a variable scoped to the for-loop regardless of whether a same-name variable is present in outer scope. If a typedecl is not provided, then the variable is scoped to the for-loop if no same-name variable is present in outer scope, or if a same-name variable is present in outer scope then it is modified.
* The typedecls such as `int` or `num` are optional. If a typedecl is provided (for a local variable), it binds a variable scoped to the for-loop regardless of whether a same-name variable is present in outer scope. If a typedecl is not provided, then the variable is scoped to the for-loop if no same-name variable is present in outer scope, or if a same-name variable is present in outer scope then it is modified.
* Miller has no ``++`` or ``--`` operators.
* Miller has no `++` or `--` operators.
* As with all for/if/while statements in Miller, the curly braces are required even if the body is a single statement, or empty.
## Begin/end blocks
Miller supports an ``awk``-like ``begin/end`` syntax. The statements in the ``begin`` block are executed before any input records are read; the statements in the ``end`` block are executed after the last input record is read. (If you want to execute some statement at the start of each file, not at the start of the first file as with ``begin``, you might use a pattern/action block of the form ``FNR == 1 { ... }``.) All statements outside of ``begin`` or ``end`` are, of course, executed on every input record. Semicolons separate statements inside or outside of begin/end blocks; semicolons are required between begin/end block bodies and any subsequent statement. For example:
Miller supports an `awk`-like `begin/end` syntax. The statements in the `begin` block are executed before any input records are read; the statements in the `end` block are executed after the last input record is read. (If you want to execute some statement at the start of each file, not at the start of the first file as with `begin`, you might use a pattern/action block of the form `FNR == 1 { ... }`.) All statements outside of `begin` or `end` are, of course, executed on every input record. Semicolons separate statements inside or outside of begin/end blocks; semicolons are required between begin/end block bodies and any subsequent statement. For example:
GENMD_INCLUDE_AND_RUN_ESCAPED(data/begin-end-example-1.sh)
Since uninitialized out-of-stream variables default to 0 for addition/substraction and 1 for multiplication when they appear on expression right-hand sides (not quite as in ``awk``, where they'd default to 0 either way), the above can be written more succinctly as
Since uninitialized out-of-stream variables default to 0 for addition/substraction and 1 for multiplication when they appear on expression right-hand sides (not quite as in `awk`, where they'd default to 0 either way), the above can be written more succinctly as
GENMD_INCLUDE_AND_RUN_ESCAPED(data/begin-end-example-2.sh)
The **put -q** option is a shorthand which suppresses printing of each output record, with only ``emit`` statements being output. So to get only summary outputs, one could write
The **put -q** option is a shorthand which suppresses printing of each output record, with only `emit` statements being output. So to get only summary outputs, one could write
GENMD_INCLUDE_AND_RUN_ESCAPED(data/begin-end-example-3.sh)
@ -166,5 +163,5 @@ This is of course not much different than
GENMD_INCLUDE_AND_RUN_ESCAPED(data/begin-end-example-5.sh)
Note that it's a syntax error for begin/end blocks to refer to field names (beginning with ``$``), since these execute outside the context of input records.
Note that it's a syntax error for begin/end blocks to refer to field names (beginning with `$`), since these execute outside the context of input records.

20
docs6b/docs/reference-dsl-errors.md
View file

@ -3,22 +3,22 @@
As soon as you have a programming language, you start having the problem *What is my code doing, and why?* This includes getting syntax errors -- which are always annoying -- as well as the even more annoying problem of a program which parses without syntax error but doesn't do what you expect.
The ``syntax error`` message is cryptic: it says ``syntax error at `` followed by the next symbol it couldn't parse. This is good, but (as of 5.0.0) it doesn't say things like ``syntax error at line 17, character 22``. Here are some common causes of syntax errors:
The `syntax error` message is cryptic: it says `syntax error at ` followed by the next symbol it couldn't parse. This is good, but (as of 5.0.0) it doesn't say things like `syntax error at line 17, character 22`. Here are some common causes of syntax errors:
* Don't forget ``;`` at end of line, before another statement on the next line.
* Don't forget `;` at end of line, before another statement on the next line.
* Miller's DSL lacks the ``++`` and ``--`` operators.
* Miller's DSL lacks the `++` and `--` operators.
* Curly braces are required for the bodies of ``if``/``while``/``for`` blocks, even when the body is a single statement.
* Curly braces are required for the bodies of `if`/`while`/`for` blocks, even when the body is a single statement.
Now for transparency:
As for transparency:
* As in any language, you can do (see :ref:`reference-dsl-print-statements`) ``print`` (or ``eprint`` to print to stderr). See also :ref:`reference-dsl-dump-statements` and :ref:`reference-dsl-emit-statements`.
* As in any language, you can do `print`, or `eprint` to print to stderr. See [Print statements](reference-dsl-output-statements.md#print-statements); see also [Dump statements](reference-dsl-output-statements.md#dump-statements) and [Emit statements](reference-dsl-output-statements.md#emit-statements).
* The ``-v`` option to ``mlr put`` and ``mlr filter`` prints abstract syntax trees for your code. While not all details here will be of interest to everyone, certainly this makes questions such as operator precedence completely unambiguous.
* The `-v` option to `mlr put` and `mlr filter` prints abstract syntax trees for your code. While not all details here will be of interest to everyone, certainly this makes questions such as operator precedence completely unambiguous.
* The ``-T`` option prints a trace of each statement executed.
* The `-T` option prints a trace of each statement executed.
* The ``-t`` and ``-a`` options show low-level details for the parsing process and for stack-variable-index allocation, respectively. These will likely be of interest to people who enjoy compilers, and probably less useful for a more general audience.
* The `-t` and `-a` options show low-level details for the parsing process and for stack-variable-index allocation, respectively. These will likely be of interest to people who enjoy compilers, and probably less useful for a more general audience.
* Please see :ref:`reference-dsl-type-checking` for type declarations and type-assertions you can use to make sure expressions and the data flowing them are evaluating as you expect. I made them optional because one of Miller's important use-cases is being able to say simple things like ``mlr put '$y = $x + 1' myfile.dat`` with a minimum of punctuational bric-a-brac -- but for programs over a few lines I generally find that the more type-specification, the better.
* Please see [type-checking](reference-dsl-variables.md#type-checking) for type declarations and type-assertions you can use to make sure expressions and the data flowing them are evaluating as you expect. I made them optional because one of Miller's important use-cases is being able to say simple things like `mlr put '$y = $x + 1' myfile.dat` with a minimum of punctuational bric-a-brac -- but for programs over a few lines I generally find that the more type-specification, the better.

20
docs6b/docs/reference-dsl-errors.md.in
View file

@ -2,22 +2,22 @@
As soon as you have a programming language, you start having the problem *What is my code doing, and why?* This includes getting syntax errors -- which are always annoying -- as well as the even more annoying problem of a program which parses without syntax error but doesn't do what you expect.
The ``syntax error`` message is cryptic: it says ``syntax error at `` followed by the next symbol it couldn't parse. This is good, but (as of 5.0.0) it doesn't say things like ``syntax error at line 17, character 22``. Here are some common causes of syntax errors:
The `syntax error` message is cryptic: it says `syntax error at ` followed by the next symbol it couldn't parse. This is good, but (as of 5.0.0) it doesn't say things like `syntax error at line 17, character 22`. Here are some common causes of syntax errors:
* Don't forget ``;`` at end of line, before another statement on the next line.
* Don't forget `;` at end of line, before another statement on the next line.
* Miller's DSL lacks the ``++`` and ``--`` operators.
* Miller's DSL lacks the `++` and `--` operators.
* Curly braces are required for the bodies of ``if``/``while``/``for`` blocks, even when the body is a single statement.
* Curly braces are required for the bodies of `if`/`while`/`for` blocks, even when the body is a single statement.
Now for transparency:
As for transparency:
* As in any language, you can do (see :ref:`reference-dsl-print-statements`) ``print`` (or ``eprint`` to print to stderr). See also :ref:`reference-dsl-dump-statements` and :ref:`reference-dsl-emit-statements`.
* As in any language, you can do `print`, or `eprint` to print to stderr. See [Print statements](reference-dsl-output-statements.md#print-statements); see also [Dump statements](reference-dsl-output-statements.md#dump-statements) and [Emit statements](reference-dsl-output-statements.md#emit-statements).
* The ``-v`` option to ``mlr put`` and ``mlr filter`` prints abstract syntax trees for your code. While not all details here will be of interest to everyone, certainly this makes questions such as operator precedence completely unambiguous.
* The `-v` option to `mlr put` and `mlr filter` prints abstract syntax trees for your code. While not all details here will be of interest to everyone, certainly this makes questions such as operator precedence completely unambiguous.
* The ``-T`` option prints a trace of each statement executed.
* The `-T` option prints a trace of each statement executed.
* The ``-t`` and ``-a`` options show low-level details for the parsing process and for stack-variable-index allocation, respectively. These will likely be of interest to people who enjoy compilers, and probably less useful for a more general audience.
* The `-t` and `-a` options show low-level details for the parsing process and for stack-variable-index allocation, respectively. These will likely be of interest to people who enjoy compilers, and probably less useful for a more general audience.
* Please see :ref:`reference-dsl-type-checking` for type declarations and type-assertions you can use to make sure expressions and the data flowing them are evaluating as you expect. I made them optional because one of Miller's important use-cases is being able to say simple things like ``mlr put '$y = $x + 1' myfile.dat`` with a minimum of punctuational bric-a-brac -- but for programs over a few lines I generally find that the more type-specification, the better.
* Please see [type-checking](reference-dsl-variables.md#type-checking) for type declarations and type-assertions you can use to make sure expressions and the data flowing them are evaluating as you expect. I made them optional because one of Miller's important use-cases is being able to say simple things like `mlr put '$y = $x + 1' myfile.dat` with a minimum of punctuational bric-a-brac -- but for programs over a few lines I generally find that the more type-specification, the better.

18
docs6b/docs/reference-dsl-filter-statements.md
View file

@ -1,31 +1,39 @@
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
# DSL reference: filter statements
You can use ``filter`` within ``put``. In fact, the following two are synonymous:
You can use `filter` within `put`. In fact, the following two are synonymous:
<pre>
<pre class="pre-highlight">
<b>mlr filter 'NR==2 || NR==3' data/small</b>
</pre>
<pre class="pre-non-highlight">
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr put 'filter NR==2 || NR==3' data/small</b>
</pre>
<pre class="pre-non-highlight">
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
</pre>
The former, of course, is much easier to type. But the latter allows you to define more complex expressions for the filter, and/or do other things in addition to the filter:
<pre>
<pre class="pre-highlight">
<b>mlr put '@running_sum += $x; filter @running_sum > 1.3' data/small</b>
</pre>
<pre class="pre-non-highlight">
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr put '$z = $x * $y; filter $z > 0.3' data/small</b>
</pre>
<pre class="pre-non-highlight">
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797,z=0.3961455844854848
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729,z=0.4951063394654227
</pre>

2
docs6b/docs/reference-dsl-filter-statements.md.in
View file

@ -1,6 +1,6 @@
# DSL reference: filter statements
You can use ``filter`` within ``put``. In fact, the following two are synonymous:
You can use `filter` within `put`. In fact, the following two are synonymous:
GENMD_RUN_COMMAND
mlr filter 'NR==2 || NR==3' data/small

10
docs6b/docs/reference-dsl-operators.md
View file

@ -5,7 +5,7 @@
Operators are listed in order of decreasing precedence, highest first.
<pre>
<pre class="pre-non-highlight">
Operators Associativity
--------- -------------
() left to right
@ -30,11 +30,11 @@ binary+ binary- . left to right
* Functions are often pass-throughs straight to the system-standard Go libraries.
* The ``min`` and ``max`` functions are different from other multi-argument functions which return null if any of their inputs are null: for ``min`` and ``max``, by contrast, if one argument is absent-null, the other is returned. Empty-null loses min or max against numeric or boolean; empty-null is less than any other string.
* The `min` and `max` functions are different from other multi-argument functions which return null if any of their inputs are null: for `min` and `max`, by contrast, if one argument is absent-null, the other is returned. Empty-null loses min or max against numeric or boolean; empty-null is less than any other string.
* Symmetrically with respect to the bitwise OR, XOR, and AND operators ``|``, ``^``, ``&``, Miller has logical operators ``||``, ``^^``, ``&&``: the logical XOR not existing in Go.
* Symmetrically with respect to the bitwise OR, XOR, and AND operators `|`, `^`, `&`, Miller has logical operators `||`, `^^`, `&&`: the logical XOR not existing in Go.
* The exponentiation operator ``**`` is familiar from many languages.
* The exponentiation operator `**` is familiar from many languages.
* The regex-match and regex-not-match operators ``=~`` and ``!=~`` are similar to those in Ruby and Perl.
* The regex-match and regex-not-match operators `=~` and `!=~` are similar to those in Ruby and Perl.

8
docs6b/docs/reference-dsl-operators.md.in
View file

@ -29,11 +29,11 @@ GENMD_EOF
* Functions are often pass-throughs straight to the system-standard Go libraries.
* The ``min`` and ``max`` functions are different from other multi-argument functions which return null if any of their inputs are null: for ``min`` and ``max``, by contrast, if one argument is absent-null, the other is returned. Empty-null loses min or max against numeric or boolean; empty-null is less than any other string.
* The `min` and `max` functions are different from other multi-argument functions which return null if any of their inputs are null: for `min` and `max`, by contrast, if one argument is absent-null, the other is returned. Empty-null loses min or max against numeric or boolean; empty-null is less than any other string.
* Symmetrically with respect to the bitwise OR, XOR, and AND operators ``|``, ``^``, ``&``, Miller has logical operators ``||``, ``^^``, ``&&``: the logical XOR not existing in Go.
* Symmetrically with respect to the bitwise OR, XOR, and AND operators `|`, `^`, `&`, Miller has logical operators `||`, `^^`, `&&`: the logical XOR not existing in Go.
* The exponentiation operator ``**`` is familiar from many languages.
* The exponentiation operator `**` is familiar from many languages.
* The regex-match and regex-not-match operators ``=~`` and ``!=~`` are similar to those in Ruby and Perl.
* The regex-match and regex-not-match operators `=~` and `!=~` are similar to those in Ruby and Perl.

184
docs6b/docs/reference-dsl-output-statements.md
View file

@ -3,63 +3,57 @@
You can **output** variable-values or expressions in **five ways**:
* **Assign** them to stream-record fields. For example, ``$cumulative_sum = @sum``. For another example, ``$nr = NR`` adds a field named ``nr`` to each output record, containing the value of the built-in variable ``NR`` as of when that record was ingested.
* **Assign** them to stream-record fields. For example, `$cumulative_sum = @sum`. For another example, `$nr = NR` adds a field named `nr` to each output record, containing the value of the built-in variable `NR` as of when that record was ingested.
* Use the **print** or **eprint** keywords which immediately print an expression *directly to standard output or standard error*, respectively. Note that ``dump``, ``edump``, ``print``, and ``eprint`` don't output records which participate in ``then``-chaining; rather, they're just immediate prints to stdout/stderr. The ``printn`` and ``eprintn`` keywords are the same except that they don't print final newlines. Additionally, you can print to a specified file instead of stdout/stderr.
* Use the **print** or **eprint** keywords which immediately print an expression *directly to standard output or standard error*, respectively. Note that `dump`, `edump`, `print`, and `eprint` don't output records which participate in `then`-chaining; rather, they're just immediate prints to stdout/stderr. The `printn` and `eprintn` keywords are the same except that they don't print final newlines. Additionally, you can print to a specified file instead of stdout/stderr.
* Use the **dump** or **edump** keywords, which *immediately print all out-of-stream variables as a JSON data structure to the standard output or standard error* (respectively).
* Use **tee** which formats the current stream record (not just an arbitrary string as with **print**) to a specific file.
* Use **emit**/**emitp**/**emitf** to send out-of-stream variables' current values to the output record stream, e.g. ``@sum += $x; emit @sum`` which produces an extra output record such as ``sum=3.1648382``.
* Use **emit**/**emitp**/**emitf** to send out-of-stream variables' current values to the output record stream, e.g. `@sum += $x; emit @sum` which produces an extra output record such as `sum=3.1648382`.
For the first two options you are populating the output-records stream which feeds into the next verb in a ``then``-chain (if any), or which otherwise is formatted for output using ``--o...`` flags.
For the first two options you are populating the output-records stream which feeds into the next verb in a `then`-chain (if any), or which otherwise is formatted for output using `--o...` flags.
For the last three options you are sending output directly to standard output, standard error, or a file.
.. _reference-dsl-print-statements:
## Print statements
The ``print`` statement is perhaps self-explanatory, but with a few light caveats:
The `print` statement is perhaps self-explanatory, but with a few light caveats:
* There are four variants: ``print`` goes to stdout with final newline, ``printn`` goes to stdout without final newline (you can include one using "\n" in your output string), ``eprint`` goes to stderr with final newline, and ``eprintn`` goes to stderr without final newline.
* There are four variants: `print` goes to stdout with final newline, `printn` goes to stdout without final newline (you can include one using "\n" in your output string), `eprint` goes to stderr with final newline, and `eprintn` goes to stderr without final newline.
* Output goes directly to stdout/stderr, respectively: data produced this way do not go downstream to the next verb in a ``then``-chain. (Use ``emit`` for that.)
* Output goes directly to stdout/stderr, respectively: data produced this way do not go downstream to the next verb in a `then`-chain. (Use `emit` for that.)
* Print statements are for strings (``print "hello"``), or things which can be made into strings: numbers (``print 3``, ``print $a + $b``, or concatenations thereof (``print "a + b = " . ($a + $b)``). Maps (in ``$*``, map-valued out-of-stream or local variables, and map literals) aren't convertible into strings. If you print a map, you get ``{is-a-map}`` as output. Please use ``dump`` to print maps.
* Print statements are for strings (`print "hello"`), or things which can be made into strings: numbers (`print 3`, `print $a + $b`, or concatenations thereof (`print "a + b = " . ($a + $b)`). Maps (in `$*`, map-valued out-of-stream or local variables, and map literals) aren't convertible into strings. If you print a map, you get `{is-a-map}` as output. Please use `dump` to print maps.
* You can redirect print output to a file: ``mlr --from myfile.dat put 'print > "tap.txt", $x'`` ``mlr --from myfile.dat put 'o=$*; print > $a.".txt", $x'``.
* You can redirect print output to a file: `mlr --from myfile.dat put 'print > "tap.txt", $x'` `mlr --from myfile.dat put 'o=$*; print > $a.".txt", $x'`.
* See also :ref:`reference-dsl-redirected-output-statements` for examples.
.. _reference-dsl-dump-statements:
* See also [Redirected-output statements](reference-dsl-output-statements.md#redirected-output-statements) for examples.
## Dump statements
The ``dump`` statement is for printing expressions, including maps, directly to stdout/stderr, respectively:
The `dump` statement is for printing expressions, including maps, directly to stdout/stderr, respectively:
* There are two variants: ``dump`` prints to stdout; ``edump`` prints to stderr.
* There are two variants: `dump` prints to stdout; `edump` prints to stderr.
* Output goes directly to stdout/stderr, respectively: data produced this way do not go downstream to the next verb in a ``then``-chain. (Use ``emit`` for that.)
* Output goes directly to stdout/stderr, respectively: data produced this way do not go downstream to the next verb in a `then`-chain. (Use `emit` for that.)
* You can use ``dump`` to output single strings, numbers, or expressions including map-valued data. Map-valued data are printed as JSON. Miller allows string and integer keys in its map literals while JSON allows only string keys, so use ``mlr put --jknquoteint`` if you want integer-valued map keys not double-quoted.
* You can use `dump` to output single strings, numbers, or expressions including map-valued data. Map-valued data are printed as JSON. Miller allows string and integer keys in its map literals while JSON allows only string keys, so use `mlr put --jknquoteint` if you want integer-valued map keys not double-quoted.
* If you use ``dump`` (or ``edump``) with no arguments, you get a JSON structure representing the current values of all out-of-stream variables.
* If you use `dump` (or `edump`) with no arguments, you get a JSON structure representing the current values of all out-of-stream variables.
* As with ``print``, you can redirect output to files.
* As with `print`, you can redirect output to files.
* See also :ref:`reference-dsl-redirected-output-statements` for examples.
* See also [Redirected-output statements](reference-dsl-output-statements.md#redirected-output-statements) for examples.
## Tee statements
Records produced by a ``mlr put`` go downstream to the next verb in your ``then``-chain, if any, or otherwise to standard output. If you want to additionally copy out records to files, you can do that using ``tee``.
Records produced by a `mlr put` go downstream to the next verb in your `then`-chain, if any, or otherwise to standard output. If you want to additionally copy out records to files, you can do that using `tee`.
The syntax is, by example, ``mlr --from myfile.dat put 'tee > "tap.dat", $*' then sort -n index``. First is ``tee >``, then the filename expression (which can be an expression such as ``"tap.".$a.".dat"``), then a comma, then ``$*``. (Nothing else but ``$*`` is teeable.)
The syntax is, by example, `mlr --from myfile.dat put 'tee > "tap.dat", $*' then sort -n index`. First is `tee >`, then the filename expression (which can be an expression such as `"tap.".$a.".dat"`), then a comma, then `$*`. (Nothing else but `$*` is teeable.)
See also :ref:`reference-dsl-redirected-output-statements` for examples.
.. _reference-dsl-redirected-output-statements:
See also [Redirected-output statements](reference-dsl-output-statements.md#redirected-output-statements) for examples.
## Redirected-output statements
@ -67,10 +61,12 @@ The **print**, **dump** **tee**, **emitf**, **emit**, and **emitp** keywords all
Details:
* The ``print`` and ``dump`` keywords produce output immediately to standard output, or to specified file(s) or pipe-to command if present.
* The `print` and `dump` keywords produce output immediately to standard output, or to specified file(s) or pipe-to command if present.
<pre>
<pre class="pre-highlight">
<b>mlr help keyword print</b>
</pre>
<pre class="pre-non-highlight">
print: prints expression immediately to stdout.
Example: mlr --from f.dat put -q 'print "The sum of x and y is ".($x+$y)'
@ -78,8 +74,10 @@ print: prints expression immediately to stdout.
Example: mlr --from f.dat put '(NR %% 1000 == 0) { print > stderr, "Checkpoint ".NR}'
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr help keyword dump</b>
</pre>
<pre class="pre-non-highlight">
dump: prints all currently defined out-of-stream variables immediately
to stdout as JSON.
@ -99,10 +97,12 @@ the main command line.
Example: mlr --from f.dat put -q '@v[NR]=$*; end { dump | "jq .[]"}'
</pre>
* ``mlr put`` sends the current record (possibly modified by the ``put`` expression) to the output record stream. Records are then input to the following verb in a ``then``-chain (if any), else printed to standard output (unless ``put -q``). The **tee** keyword *additionally* writes the output record to specified file(s) or pipe-to command, or immediately to ``stdout``/``stderr``.
* `mlr put` sends the current record (possibly modified by the `put` expression) to the output record stream. Records are then input to the following verb in a `then`-chain (if any), else printed to standard output (unless `put -q`). The **tee** keyword *additionally* writes the output record to specified file(s) or pipe-to command, or immediately to `stdout`/`stderr`.
<pre>
<pre class="pre-highlight">
<b>mlr help keyword tee</b>
</pre>
<pre class="pre-non-highlight">
tee: prints the current record to specified file.
This is an immediate print to the specified file (except for pprint format
which of course waits until the end of the input stream to format all output).
@ -129,10 +129,12 @@ output $*.
Example: mlr --from f.dat put -q --ojson 'tee | "gzip > /tmp/data-".$a.".gz", $*'
</pre>
* ``mlr put``'s ``emitf``, ``emitp``, and ``emit`` send out-of-stream variables to the output record stream. These are then input to the following verb in a ``then``-chain (if any), else printed to standard output. When redirected with ``>``, ``>>``, or ``|``, they *instead* write the out-of-stream variable(s) to specified file(s) or pipe-to command, or immediately to ``stdout``/``stderr``.
* `mlr put`'s `emitf`, `emitp`, and `emit` send out-of-stream variables to the output record stream. These are then input to the following verb in a `then`-chain (if any), else printed to standard output. When redirected with `>`, `>>`, or `|`, they *instead* write the out-of-stream variable(s) to specified file(s) or pipe-to command, or immediately to `stdout`/`stderr`.
<pre>
<pre class="pre-highlight">
<b>mlr help keyword emitf</b>
</pre>
<pre class="pre-non-highlight">
emitf: inserts non-indexed out-of-stream variable(s) side-by-side into the
output record stream.
@ -161,8 +163,10 @@ etc., to control the format of the output if the output is redirected. See also
Please see https://johnkerl.org/miller6://johnkerl.org/miller/doc for more information.
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr help keyword emitp</b>
</pre>
<pre class="pre-non-highlight">
emitp: inserts an out-of-stream variable into the output record stream.
Hashmap indices present in the data but not slotted by emitp arguments are
output concatenated with ":".
@ -193,8 +197,10 @@ etc., to control the format of the output if the output is redirected. See also
Please see https://johnkerl.org/miller6://johnkerl.org/miller/doc for more information.
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr help keyword emit</b>
</pre>
<pre class="pre-non-highlight">
emit: inserts an out-of-stream variable into the output record stream. Hashmap
indices present in the data but not slotted by emit arguments are not output.
@ -226,28 +232,30 @@ etc., to control the format of the output if the output is redirected. See also
Please see https://johnkerl.org/miller6://johnkerl.org/miller/doc for more information.
</pre>
.. _reference-dsl-emit-statements:
## Emit statements
There are three variants: ``emitf``, ``emit``, and ``emitp``. Keep in mind that out-of-stream variables are a nested, multi-level hashmap (directly viewable as JSON using ``dump``), whereas Miller output records are lists of single-level key-value pairs. The three emit variants allow you to control how the multilevel hashmaps are flatten down to output records. You can emit any map-valued expression, including ``$*``, map-valued out-of-stream variables, the entire out-of-stream-variable collection ``@*``, map-valued local variables, map literals, or map-valued function return values.
There are three variants: `emitf`, `emit`, and `emitp`. Keep in mind that out-of-stream variables are a nested, multi-level hashmap (directly viewable as JSON using `dump`), whereas Miller output records are lists of single-level key-value pairs. The three emit variants allow you to control how the multilevel hashmaps are flatten down to output records. You can emit any map-valued expression, including `$*`, map-valued out-of-stream variables, the entire out-of-stream-variable collection `@*`, map-valued local variables, map literals, or map-valued function return values.
Use **emitf** to output several out-of-stream variables side-by-side in the same output record. For ``emitf`` these mustn't have indexing using ``@name[...]``. Example:
Use **emitf** to output several out-of-stream variables side-by-side in the same output record. For `emitf` these mustn't have indexing using `@name[...]`. Example:
<pre>
<pre class="pre-highlight">
<b>mlr put -q '</b>
<b> @count += 1;</b>
<b> @x_sum += $x;</b>
<b> @y_sum += $y;</b>
<b> end { emitf @count, @x_sum, @y_sum}</b>
<b>' data/small</b>
</pre>
<pre class="pre-non-highlight">
count=5,x_sum=2.264761728567491,y_sum=2.585085709781158
</pre>
Use **emit** to output an out-of-stream variable. If it's non-indexed you'll get a simple key-value pair:
<pre>
<pre class="pre-highlight">
<b>cat data/small</b>
</pre>
<pre class="pre-non-highlight">
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
@ -255,22 +263,28 @@ a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr put -q '@sum += $x; end { dump }' data/small</b>
</pre>
<pre class="pre-non-highlight">
{
"sum": 2.264761728567491
}
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr put -q '@sum += $x; end { emit @sum }' data/small</b>
</pre>
<pre class="pre-non-highlight">
sum=2.264761728567491
</pre>
If it's indexed then use as many names after ``emit`` as there are indices:
If it's indexed then use as many names after `emit` as there are indices:
<pre>
<pre class="pre-highlight">
<b>mlr put -q '@sum[$a] += $x; end { dump }' data/small</b>
</pre>
<pre class="pre-non-highlight">
{
"sum": {
"pan": 0.3467901443380824,
@ -280,15 +294,19 @@ If it's indexed then use as many names after ``emit`` as there are indices:
}
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr put -q '@sum[$a] += $x; end { emit @sum, "a" }' data/small</b>
</pre>
<pre class="pre-non-highlight">
a=pan,sum=0.3467901443380824
a=eks,sum=1.1400793586611044
a=wye,sum=0.7778922255683036
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr put -q '@sum[$a][$b] += $x; end { dump }' data/small</b>
</pre>
<pre class="pre-non-highlight">
{
"sum": {
"pan": {
@ -306,8 +324,10 @@ a=wye,sum=0.7778922255683036
}
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr put -q '@sum[$a][$b] += $x; end { emit @sum, "a", "b" }' data/small</b>
</pre>
<pre class="pre-non-highlight">
a=pan,b=pan,sum=0.3467901443380824
a=eks,b=pan,sum=0.7586799647899636
a=eks,b=wye,sum=0.38139939387114097
@ -315,8 +335,10 @@ a=wye,b=wye,sum=0.20460330576630303
a=wye,b=pan,sum=0.5732889198020006
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr put -q '@sum[$a][$b][$i] += $x; end { dump }' data/small</b>
</pre>
<pre class="pre-non-highlight">
{
"sum": {
"pan": {
@ -344,11 +366,13 @@ a=wye,b=pan,sum=0.5732889198020006
}
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr put -q '</b>
<b> @sum[$a][$b][$i] += $x;</b>
<b> end { emit @sum, "a", "b", "i" }</b>
<b>' data/small</b>
</pre>
<pre class="pre-non-highlight">
a=pan,b=pan,i=1,sum=0.3467901443380824
a=eks,b=pan,i=2,sum=0.7586799647899636
a=eks,b=wye,i=4,sum=0.38139939387114097
@ -356,10 +380,12 @@ a=wye,b=wye,i=3,sum=0.20460330576630303
a=wye,b=pan,i=5,sum=0.5732889198020006
</pre>
Now for **emitp**: if you have as many names following ``emit`` as there are levels in the out-of-stream variable's hashmap, then ``emit`` and ``emitp`` do the same thing. Where they differ is when you don't specify as many names as there are hashmap levels. In this case, Miller needs to flatten multiple map indices down to output-record keys: ``emitp`` includes full prefixing (hence the ``p`` in ``emitp``) while ``emit`` takes the deepest hashmap key as the output-record key:
Now for **emitp**: if you have as many names following `emit` as there are levels in the out-of-stream variable's hashmap, then `emit` and `emitp` do the same thing. Where they differ is when you don't specify as many names as there are hashmap levels. In this case, Miller needs to flatten multiple map indices down to output-record keys: `emitp` includes full prefixing (hence the `p` in `emitp`) while `emit` takes the deepest hashmap key as the output-record key:
<pre>
<pre class="pre-highlight">
<b>mlr put -q '@sum[$a][$b] += $x; end { dump }' data/small</b>
</pre>
<pre class="pre-non-highlight">
{
"sum": {
"pan": {
@ -377,34 +403,44 @@ Now for **emitp**: if you have as many names following ``emit`` as there are lev
}
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr put -q '@sum[$a][$b] += $x; end { emit @sum, "a" }' data/small</b>
</pre>
<pre class="pre-non-highlight">
a=pan,pan=0.3467901443380824
a=eks,pan=0.7586799647899636,wye=0.38139939387114097
a=wye,wye=0.20460330576630303,pan=0.5732889198020006
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr put -q '@sum[$a][$b] += $x; end { emit @sum }' data/small</b>
</pre>
<pre class="pre-non-highlight">
pan=0.3467901443380824
pan=0.7586799647899636,wye=0.38139939387114097
wye=0.20460330576630303,pan=0.5732889198020006
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr put -q '@sum[$a][$b] += $x; end { emitp @sum, "a" }' data/small</b>
</pre>
<pre class="pre-non-highlight">
a=pan,sum.pan=0.3467901443380824
a=eks,sum.pan=0.7586799647899636,sum.wye=0.38139939387114097
a=wye,sum.wye=0.20460330576630303,sum.pan=0.5732889198020006
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr put -q '@sum[$a][$b] += $x; end { emitp @sum }' data/small</b>
</pre>
<pre class="pre-non-highlight">
sum.pan.pan=0.3467901443380824,sum.eks.pan=0.7586799647899636,sum.eks.wye=0.38139939387114097,sum.wye.wye=0.20460330576630303,sum.wye.pan=0.5732889198020006
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --oxtab put -q '@sum[$a][$b] += $x; end { emitp @sum }' data/small</b>
</pre>
<pre class="pre-non-highlight">
sum.pan.pan 0.3467901443380824
sum.eks.pan 0.7586799647899636
sum.eks.wye 0.38139939387114097
@ -413,25 +449,31 @@ sum.wye.pan 0.5732889198020006
</pre>
Use **--oflatsep** to specify the character which joins multilevel
keys for ``emitp`` (it defaults to a colon):
keys for `emitp` (it defaults to a colon):
<pre>
<pre class="pre-highlight">
<b>mlr put -q --oflatsep / '@sum[$a][$b] += $x; end { emitp @sum, "a" }' data/small</b>
</pre>
<pre class="pre-non-highlight">
a=pan,sum.pan=0.3467901443380824
a=eks,sum.pan=0.7586799647899636,sum.wye=0.38139939387114097
a=wye,sum.wye=0.20460330576630303,sum.pan=0.5732889198020006
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr put -q --oflatsep / '@sum[$a][$b] += $x; end { emitp @sum }' data/small</b>
</pre>
<pre class="pre-non-highlight">
sum.pan.pan=0.3467901443380824,sum.eks.pan=0.7586799647899636,sum.eks.wye=0.38139939387114097,sum.wye.wye=0.20460330576630303,sum.wye.pan=0.5732889198020006
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --oxtab put -q --oflatsep / '</b>
<b> @sum[$a][$b] += $x;</b>
<b> end { emitp @sum }</b>
<b>' data/small</b>
</pre>
<pre class="pre-non-highlight">
sum.pan.pan 0.3467901443380824
sum.eks.pan 0.7586799647899636
sum.eks.wye 0.38139939387114097
@ -444,7 +486,7 @@ sum.wye.pan 0.5732889198020006
You can emit **multiple map-valued expressions side-by-side** by
including their names in parentheses:
<pre>
<pre class="pre-highlight">
<b>mlr --from data/medium --opprint put -q '</b>
<b> @x_count[$a][$b] += 1;</b>
<b> @x_sum[$a][$b] += $x;</b>
@ -455,6 +497,8 @@ including their names in parentheses:
<b> emit (@x_sum, @x_count, @x_mean), "a", "b"</b>
<b> }</b>
<b>'</b>
</pre>
<pre class="pre-non-highlight">
a b x_sum x_count x_mean
pan pan 219.1851288316854 427 0.5133141190437597
pan wye 198.43293070748447 395 0.5023618498923658
@ -483,18 +527,20 @@ hat hat 182.8535323148762 381 0.47993053101017374
hat pan 168.5538067327806 363 0.4643355557376876
</pre>
What this does is walk through the first out-of-stream variable (``@x_sum`` in this example) as usual, then for each keylist found (e.g. ``pan,wye``), include the values for the remaining out-of-stream variables (here, ``@x_count`` and ``@x_mean``). You should use this when all out-of-stream variables in the emit statement have **the same shape and the same keylists**.
What this does is walk through the first out-of-stream variable (`@x_sum` in this example) as usual, then for each keylist found (e.g. `pan,wye`), include the values for the remaining out-of-stream variables (here, `@x_count` and `@x_mean`). You should use this when all out-of-stream variables in the emit statement have **the same shape and the same keylists**.
## Emit-all statements
Use **emit all** (or ``emit @*`` which is synonymous) to output all out-of-stream variables. You can use the following idiom to get various accumulators output side-by-side (reminiscent of ``mlr stats1``):
Use **emit all** (or `emit @*` which is synonymous) to output all out-of-stream variables. You can use the following idiom to get various accumulators output side-by-side (reminiscent of `mlr stats1`):
<pre>
<pre class="pre-highlight">
<b>mlr --from data/small --opprint put -q '</b>
<b> @v[$a][$b]["sum"] += $x;</b>
<b> @v[$a][$b]["count"] += 1;</b>
<b> end{emit @*,"a","b"}</b>
<b>'</b>
</pre>
<pre class="pre-non-highlight">
a b v.sum v.count
pan pan 0.3467901443380824 1
eks pan 0.7586799647899636 1
@ -503,12 +549,14 @@ wye wye 0.20460330576630303 1
wye pan 0.5732889198020006 1
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --from data/small --opprint put -q '</b>
<b> @sum[$a][$b] += $x;</b>
<b> @count[$a][$b] += 1;</b>
<b> end{emit @*,"a","b"}</b>
<b>'</b>
</pre>
<pre class="pre-non-highlight">
a b sum
pan pan 0.3467901443380824
eks pan 0.7586799647899636
@ -524,12 +572,14 @@ wye wye 1
wye pan 1
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --from data/small --opprint put -q '</b>
<b> @sum[$a][$b] += $x;</b>
<b> @count[$a][$b] += 1;</b>
<b> end{emit (@sum, @count),"a","b"}</b>
<b>'</b>
</pre>
<pre class="pre-non-highlight">
a b sum count
pan pan 0.3467901443380824 1
eks pan 0.7586799647899636 1

68
docs6b/docs/reference-dsl-output-statements.md.in
View file

@ -2,63 +2,57 @@
You can **output** variable-values or expressions in **five ways**:
* **Assign** them to stream-record fields. For example, ``$cumulative_sum = @sum``. For another example, ``$nr = NR`` adds a field named ``nr`` to each output record, containing the value of the built-in variable ``NR`` as of when that record was ingested.
* **Assign** them to stream-record fields. For example, `$cumulative_sum = @sum`. For another example, `$nr = NR` adds a field named `nr` to each output record, containing the value of the built-in variable `NR` as of when that record was ingested.
* Use the **print** or **eprint** keywords which immediately print an expression *directly to standard output or standard error*, respectively. Note that ``dump``, ``edump``, ``print``, and ``eprint`` don't output records which participate in ``then``-chaining; rather, they're just immediate prints to stdout/stderr. The ``printn`` and ``eprintn`` keywords are the same except that they don't print final newlines. Additionally, you can print to a specified file instead of stdout/stderr.
* Use the **print** or **eprint** keywords which immediately print an expression *directly to standard output or standard error*, respectively. Note that `dump`, `edump`, `print`, and `eprint` don't output records which participate in `then`-chaining; rather, they're just immediate prints to stdout/stderr. The `printn` and `eprintn` keywords are the same except that they don't print final newlines. Additionally, you can print to a specified file instead of stdout/stderr.
* Use the **dump** or **edump** keywords, which *immediately print all out-of-stream variables as a JSON data structure to the standard output or standard error* (respectively).
* Use **tee** which formats the current stream record (not just an arbitrary string as with **print**) to a specific file.
* Use **emit**/**emitp**/**emitf** to send out-of-stream variables' current values to the output record stream, e.g. ``@sum += $x; emit @sum`` which produces an extra output record such as ``sum=3.1648382``.
* Use **emit**/**emitp**/**emitf** to send out-of-stream variables' current values to the output record stream, e.g. `@sum += $x; emit @sum` which produces an extra output record such as `sum=3.1648382`.
For the first two options you are populating the output-records stream which feeds into the next verb in a ``then``-chain (if any), or which otherwise is formatted for output using ``--o...`` flags.
For the first two options you are populating the output-records stream which feeds into the next verb in a `then`-chain (if any), or which otherwise is formatted for output using `--o...` flags.
For the last three options you are sending output directly to standard output, standard error, or a file.
.. _reference-dsl-print-statements:
## Print statements
The ``print`` statement is perhaps self-explanatory, but with a few light caveats:
The `print` statement is perhaps self-explanatory, but with a few light caveats:
* There are four variants: ``print`` goes to stdout with final newline, ``printn`` goes to stdout without final newline (you can include one using "\n" in your output string), ``eprint`` goes to stderr with final newline, and ``eprintn`` goes to stderr without final newline.
* There are four variants: `print` goes to stdout with final newline, `printn` goes to stdout without final newline (you can include one using "\n" in your output string), `eprint` goes to stderr with final newline, and `eprintn` goes to stderr without final newline.
* Output goes directly to stdout/stderr, respectively: data produced this way do not go downstream to the next verb in a ``then``-chain. (Use ``emit`` for that.)
* Output goes directly to stdout/stderr, respectively: data produced this way do not go downstream to the next verb in a `then`-chain. (Use `emit` for that.)
* Print statements are for strings (``print "hello"``), or things which can be made into strings: numbers (``print 3``, ``print $a + $b``, or concatenations thereof (``print "a + b = " . ($a + $b)``). Maps (in ``$*``, map-valued out-of-stream or local variables, and map literals) aren't convertible into strings. If you print a map, you get ``{is-a-map}`` as output. Please use ``dump`` to print maps.
* Print statements are for strings (`print "hello"`), or things which can be made into strings: numbers (`print 3`, `print $a + $b`, or concatenations thereof (`print "a + b = " . ($a + $b)`). Maps (in `$*`, map-valued out-of-stream or local variables, and map literals) aren't convertible into strings. If you print a map, you get `{is-a-map}` as output. Please use `dump` to print maps.
* You can redirect print output to a file: ``mlr --from myfile.dat put 'print > "tap.txt", $x'`` ``mlr --from myfile.dat put 'o=$*; print > $a.".txt", $x'``.
* You can redirect print output to a file: `mlr --from myfile.dat put 'print > "tap.txt", $x'` `mlr --from myfile.dat put 'o=$*; print > $a.".txt", $x'`.
* See also :ref:`reference-dsl-redirected-output-statements` for examples.
.. _reference-dsl-dump-statements:
* See also [Redirected-output statements](reference-dsl-output-statements.md#redirected-output-statements) for examples.
## Dump statements
The ``dump`` statement is for printing expressions, including maps, directly to stdout/stderr, respectively:
The `dump` statement is for printing expressions, including maps, directly to stdout/stderr, respectively:
* There are two variants: ``dump`` prints to stdout; ``edump`` prints to stderr.
* There are two variants: `dump` prints to stdout; `edump` prints to stderr.
* Output goes directly to stdout/stderr, respectively: data produced this way do not go downstream to the next verb in a ``then``-chain. (Use ``emit`` for that.)
* Output goes directly to stdout/stderr, respectively: data produced this way do not go downstream to the next verb in a `then`-chain. (Use `emit` for that.)
* You can use ``dump`` to output single strings, numbers, or expressions including map-valued data. Map-valued data are printed as JSON. Miller allows string and integer keys in its map literals while JSON allows only string keys, so use ``mlr put --jknquoteint`` if you want integer-valued map keys not double-quoted.
* You can use `dump` to output single strings, numbers, or expressions including map-valued data. Map-valued data are printed as JSON. Miller allows string and integer keys in its map literals while JSON allows only string keys, so use `mlr put --jknquoteint` if you want integer-valued map keys not double-quoted.
* If you use ``dump`` (or ``edump``) with no arguments, you get a JSON structure representing the current values of all out-of-stream variables.
* If you use `dump` (or `edump`) with no arguments, you get a JSON structure representing the current values of all out-of-stream variables.
* As with ``print``, you can redirect output to files.
* As with `print`, you can redirect output to files.
* See also :ref:`reference-dsl-redirected-output-statements` for examples.
* See also [Redirected-output statements](reference-dsl-output-statements.md#redirected-output-statements) for examples.
## Tee statements
Records produced by a ``mlr put`` go downstream to the next verb in your ``then``-chain, if any, or otherwise to standard output. If you want to additionally copy out records to files, you can do that using ``tee``.
Records produced by a `mlr put` go downstream to the next verb in your `then`-chain, if any, or otherwise to standard output. If you want to additionally copy out records to files, you can do that using `tee`.
The syntax is, by example, ``mlr --from myfile.dat put 'tee > "tap.dat", $*' then sort -n index``. First is ``tee >``, then the filename expression (which can be an expression such as ``"tap.".$a.".dat"``), then a comma, then ``$*``. (Nothing else but ``$*`` is teeable.)
The syntax is, by example, `mlr --from myfile.dat put 'tee > "tap.dat", $*' then sort -n index`. First is `tee >`, then the filename expression (which can be an expression such as `"tap.".$a.".dat"`), then a comma, then `$*`. (Nothing else but `$*` is teeable.)
See also :ref:`reference-dsl-redirected-output-statements` for examples.
.. _reference-dsl-redirected-output-statements:
See also [Redirected-output statements](reference-dsl-output-statements.md#redirected-output-statements) for examples.
## Redirected-output statements
@ -66,7 +60,7 @@ The **print**, **dump** **tee**, **emitf**, **emit**, and **emitp** keywords all
Details:
* The ``print`` and ``dump`` keywords produce output immediately to standard output, or to specified file(s) or pipe-to command if present.
* The `print` and `dump` keywords produce output immediately to standard output, or to specified file(s) or pipe-to command if present.
GENMD_RUN_COMMAND
mlr help keyword print
@ -76,13 +70,13 @@ GENMD_RUN_COMMAND
mlr help keyword dump
GENMD_EOF
* ``mlr put`` sends the current record (possibly modified by the ``put`` expression) to the output record stream. Records are then input to the following verb in a ``then``-chain (if any), else printed to standard output (unless ``put -q``). The **tee** keyword *additionally* writes the output record to specified file(s) or pipe-to command, or immediately to ``stdout``/``stderr``.
* `mlr put` sends the current record (possibly modified by the `put` expression) to the output record stream. Records are then input to the following verb in a `then`-chain (if any), else printed to standard output (unless `put -q`). The **tee** keyword *additionally* writes the output record to specified file(s) or pipe-to command, or immediately to `stdout`/`stderr`.
GENMD_RUN_COMMAND
mlr help keyword tee
GENMD_EOF
* ``mlr put``'s ``emitf``, ``emitp``, and ``emit`` send out-of-stream variables to the output record stream. These are then input to the following verb in a ``then``-chain (if any), else printed to standard output. When redirected with ``>``, ``>>``, or ``|``, they *instead* write the out-of-stream variable(s) to specified file(s) or pipe-to command, or immediately to ``stdout``/``stderr``.
* `mlr put`'s `emitf`, `emitp`, and `emit` send out-of-stream variables to the output record stream. These are then input to the following verb in a `then`-chain (if any), else printed to standard output. When redirected with `>`, `>>`, or `|`, they *instead* write the out-of-stream variable(s) to specified file(s) or pipe-to command, or immediately to `stdout`/`stderr`.
GENMD_RUN_COMMAND
mlr help keyword emitf
@ -96,13 +90,11 @@ GENMD_RUN_COMMAND
mlr help keyword emit
GENMD_EOF
.. _reference-dsl-emit-statements:
## Emit statements
There are three variants: ``emitf``, ``emit``, and ``emitp``. Keep in mind that out-of-stream variables are a nested, multi-level hashmap (directly viewable as JSON using ``dump``), whereas Miller output records are lists of single-level key-value pairs. The three emit variants allow you to control how the multilevel hashmaps are flatten down to output records. You can emit any map-valued expression, including ``$*``, map-valued out-of-stream variables, the entire out-of-stream-variable collection ``@*``, map-valued local variables, map literals, or map-valued function return values.
There are three variants: `emitf`, `emit`, and `emitp`. Keep in mind that out-of-stream variables are a nested, multi-level hashmap (directly viewable as JSON using `dump`), whereas Miller output records are lists of single-level key-value pairs. The three emit variants allow you to control how the multilevel hashmaps are flatten down to output records. You can emit any map-valued expression, including `$*`, map-valued out-of-stream variables, the entire out-of-stream-variable collection `@*`, map-valued local variables, map literals, or map-valued function return values.
Use **emitf** to output several out-of-stream variables side-by-side in the same output record. For ``emitf`` these mustn't have indexing using ``@name[...]``. Example:
Use **emitf** to output several out-of-stream variables side-by-side in the same output record. For `emitf` these mustn't have indexing using `@name[...]`. Example:
GENMD_RUN_COMMAND
mlr put -q '
@ -127,7 +119,7 @@ GENMD_RUN_COMMAND
mlr put -q '@sum += $x; end { emit @sum }' data/small
GENMD_EOF
If it's indexed then use as many names after ``emit`` as there are indices:
If it's indexed then use as many names after `emit` as there are indices:
GENMD_RUN_COMMAND
mlr put -q '@sum[$a] += $x; end { dump }' data/small
@ -156,7 +148,7 @@ mlr put -q '
' data/small
GENMD_EOF
Now for **emitp**: if you have as many names following ``emit`` as there are levels in the out-of-stream variable's hashmap, then ``emit`` and ``emitp`` do the same thing. Where they differ is when you don't specify as many names as there are hashmap levels. In this case, Miller needs to flatten multiple map indices down to output-record keys: ``emitp`` includes full prefixing (hence the ``p`` in ``emitp``) while ``emit`` takes the deepest hashmap key as the output-record key:
Now for **emitp**: if you have as many names following `emit` as there are levels in the out-of-stream variable's hashmap, then `emit` and `emitp` do the same thing. Where they differ is when you don't specify as many names as there are hashmap levels. In this case, Miller needs to flatten multiple map indices down to output-record keys: `emitp` includes full prefixing (hence the `p` in `emitp`) while `emit` takes the deepest hashmap key as the output-record key:
GENMD_RUN_COMMAND
mlr put -q '@sum[$a][$b] += $x; end { dump }' data/small
@ -183,7 +175,7 @@ mlr --oxtab put -q '@sum[$a][$b] += $x; end { emitp @sum }' data/small
GENMD_EOF
Use **--oflatsep** to specify the character which joins multilevel
keys for ``emitp`` (it defaults to a colon):
keys for `emitp` (it defaults to a colon):
GENMD_RUN_COMMAND
mlr put -q --oflatsep / '@sum[$a][$b] += $x; end { emitp @sum, "a" }' data/small
@ -207,11 +199,11 @@ including their names in parentheses:
GENMD_INCLUDE_AND_RUN_ESCAPED(data/emit-lashed.sh)
What this does is walk through the first out-of-stream variable (``@x_sum`` in this example) as usual, then for each keylist found (e.g. ``pan,wye``), include the values for the remaining out-of-stream variables (here, ``@x_count`` and ``@x_mean``). You should use this when all out-of-stream variables in the emit statement have **the same shape and the same keylists**.
What this does is walk through the first out-of-stream variable (`@x_sum` in this example) as usual, then for each keylist found (e.g. `pan,wye`), include the values for the remaining out-of-stream variables (here, `@x_count` and `@x_mean`). You should use this when all out-of-stream variables in the emit statement have **the same shape and the same keylists**.
## Emit-all statements
Use **emit all** (or ``emit @*`` which is synonymous) to output all out-of-stream variables. You can use the following idiom to get various accumulators output side-by-side (reminiscent of ``mlr stats1``):
Use **emit all** (or `emit @*` which is synonymous) to output all out-of-stream variables. You can use the following idiom to get various accumulators output side-by-side (reminiscent of `mlr stats1`):
GENMD_RUN_COMMAND
mlr --from data/small --opprint put -q '

66
docs6b/docs/reference-dsl-syntax.md
View file

@ -5,8 +5,10 @@
Multiple expressions may be given, separated by semicolons, and each may refer to the ones before:
<pre>
<pre class="pre-highlight">
<b>ruby -e '10.times{|i|puts "i=#{i}"}' | mlr --opprint put '$j = $i + 1; $k = $i +$j'</b>
</pre>
<pre class="pre-non-highlight">
i j k
0 1 1
1 2 3
@ -22,7 +24,7 @@ i j k
Newlines within the expression are ignored, which can help increase legibility of complex expressions:
<pre>
<pre class="pre-highlight">
<b>mlr --opprint put '</b>
<b> $nf = NF;</b>
<b> $nr = NR;</b>
@ -30,6 +32,8 @@ Newlines within the expression are ignored, which can help increase legibility o
<b> $filenum = FILENUM;</b>
<b> $filename = FILENAME</b>
<b>' data/small data/small2</b>
</pre>
<pre class="pre-non-highlight">
a b i x y nf nr fnr filenum filename
pan pan 1 0.3467901443380824 0.7268028627434533 5 1 1 1 data/small
eks pan 2 0.7586799647899636 0.5221511083334797 5 2 2 1 data/small
@ -43,22 +47,24 @@ hat wye 10002 0.321507044286237609 0.568893318795083758 5 9 4 2 data/s
pan zee 10003 0.272054845593895200 0.425789896597056627 5 10 5 2 data/small2
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --opprint filter '($x > 0.5 && $y < 0.5) || ($x < 0.5 && $y > 0.5)' \</b>
<b> then stats2 -a corr -f x,y \</b>
<b> data/medium</b>
</pre>
<pre class="pre-non-highlight">
x_y_corr
-0.7479940285189345
</pre>
.. _reference-dsl-expressions-from-files:
## Expressions from files
The simplest way to enter expressions for ``put`` and ``filter`` is between single quotes on the command line, e.g.
The simplest way to enter expressions for `put` and `filter` is between single quotes on the command line, e.g.
<pre>
<pre class="pre-highlight">
<b>mlr --from data/small put '$xy = sqrt($x**2 + $y**2)'</b>
</pre>
<pre class="pre-non-highlight">
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533,xy=0.8052985815845617
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797,xy=0.9209978658539777
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776,xy=0.3953756915115773
@ -66,8 +72,10 @@ a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463,xy=0.404316851577441
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729,xy=1.036584492737304
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --from data/small put 'func f(a, b) { return sqrt(a**2 + b**2) } $xy = f($x, $y)'</b>
</pre>
<pre class="pre-non-highlight">
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533,xy=0.8052985815845617
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797,xy=0.9209978658539777
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776,xy=0.3953756915115773
@ -78,16 +86,20 @@ a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729,xy=1.036584492737304
You may, though, find it convenient to put expressions into files for reuse, and read them
**using the -f option**. For example:
<pre>
<pre class="pre-highlight">
<b>cat data/fe-example-3.mlr</b>
</pre>
<pre class="pre-non-highlight">
func f(a, b) {
return sqrt(a**2 + b**2)
}
$xy = f($x, $y)
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --from data/small put -f data/fe-example-3.mlr</b>
</pre>
<pre class="pre-non-highlight">
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533,xy=0.8052985815845617
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797,xy=0.9209978658539777
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776,xy=0.3953756915115773
@ -97,15 +109,19 @@ a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729,xy=1.036584492737304
If you have some of the logic in a file and you want to write the rest on the command line, you can **use the -f and -e options together**:
<pre>
<pre class="pre-highlight">
<b>cat data/fe-example-4.mlr</b>
</pre>
<pre class="pre-non-highlight">
func f(a, b) {
return sqrt(a**2 + b**2)
}
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --from data/small put -f data/fe-example-4.mlr -e '$xy = f($x, $y)'</b>
</pre>
<pre class="pre-non-highlight">
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533,xy=0.8052985815845617
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797,xy=0.9209978658539777
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776,xy=0.3953756915115773
@ -115,15 +131,15 @@ a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729,xy=1.036584492737304
A suggested use-case here is defining functions in files, and calling them from command-line expressions.
Another suggested use-case is putting default parameter values in files, e.g. using ``begin{@count=is_present(@count)?@count:10}`` in the file, where you can precede that using ``begin{@count=40}`` using ``-e``.
Another suggested use-case is putting default parameter values in files, e.g. using `begin{@count=is_present(@count)?@count:10}` in the file, where you can precede that using `begin{@count=40}` using `-e`.
Moreover, you can have one or more ``-f`` expressions (maybe one function per file, for example) and one or more ``-e`` expressions on the command line. If you mix ``-f`` and ``-e`` then the expressions are evaluated in the order encountered. (Since the expressions are all simply concatenated together in order, don't forget intervening semicolons: e.g. not ``mlr put -e '$x=1' -e '$y=2 ...'`` but rather ``mlr put -e '$x=1;' -e '$y=2' ...``.)
Moreover, you can have one or more `-f` expressions (maybe one function per file, for example) and one or more `-e` expressions on the command line. If you mix `-f` and `-e` then the expressions are evaluated in the order encountered. (Since the expressions are all simply concatenated together in order, don't forget intervening semicolons: e.g. not `mlr put -e '$x=1' -e '$y=2 ...'` but rather `mlr put -e '$x=1;' -e '$y=2' ...`.)
## Semicolons, commas, newlines, and curly braces
Miller uses **semicolons as statement separators**, not statement terminators. This means you can write:
<pre>
<pre class="pre-non-highlight">
mlr put 'x=1'
mlr put 'x=1;$y=2'
mlr put 'x=1;$y=2;'
@ -132,19 +148,23 @@ mlr put 'x=1;;;;$y=2;'
Semicolons are optional after closing curly braces (which close conditionals and loops as discussed below).
<pre>
<pre class="pre-highlight">
<b>echo x=1,y=2 | mlr put 'while (NF < 10) { $[NF+1] = ""} $foo = "bar"'</b>
</pre>
<pre class="pre-non-highlight">
x=1,y=2,3=,4=,5=,6=,7=,8=,9=,10=,foo=bar
</pre>
<pre>
<pre class="pre-highlight">
<b>echo x=1,y=2 | mlr put 'while (NF < 10) { $[NF+1] = ""}; $foo = "bar"'</b>
</pre>
<pre class="pre-non-highlight">
x=1,y=2,3=,4=,5=,6=,7=,8=,9=,10=,foo=bar
</pre>
Semicolons are required between statements even if those statements are on separate lines. **Newlines** are for your convenience but have no syntactic meaning: line endings do not terminate statements. For example, adjacent assignment statements must be separated by semicolons even if those statements are on separate lines:
<pre>
<pre class="pre-non-highlight">
mlr put '
$x = 1
$y = 2 # Syntax error
@ -158,7 +178,7 @@ mlr put '
**Trailing commas** are allowed in function/subroutine definitions, function/subroutine callsites, and map literals. This is intended for (although not restricted to) the multi-line case:
<pre>
<pre class="pre-highlight">
<b>mlr --csvlite --from data/a.csv put '</b>
<b> func f(</b>
<b> num a,</b>
@ -176,6 +196,8 @@ mlr put '
<b> "v": NR,</b>
<b> }</b>
<b>'</b>
</pre>
<pre class="pre-non-highlight">
s,t,u,v
3,-1,5,1
9,-1,41,2
@ -183,17 +205,17 @@ s,t,u,v
Bodies for all compound statements must be enclosed in **curly braces**, even if the body is a single statement:
<pre>
<pre class="pre-highlight">
<b>mlr put 'if ($x == 1) $y = 2' # Syntax error</b>
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr put 'if ($x == 1) { $y = 2 }' # This is OK</b>
</pre>
Bodies for compound statements may be empty:
<pre>
<pre class="pre-highlight">
<b>mlr put 'if ($x == 1) { }' # This no-op is syntactically acceptable</b>
</pre>

8
docs6b/docs/reference-dsl-syntax.md.in
View file

@ -18,11 +18,9 @@ mlr --opprint filter '($x > 0.5 && $y < 0.5) || ($x < 0.5 && $y > 0.5)' \
data/medium
GENMD_EOF
.. _reference-dsl-expressions-from-files:
## Expressions from files
The simplest way to enter expressions for ``put`` and ``filter`` is between single quotes on the command line, e.g.
The simplest way to enter expressions for `put` and `filter` is between single quotes on the command line, e.g.
GENMD_INCLUDE_AND_RUN_ESCAPED(data/fe-example-1.sh)
@ -51,9 +49,9 @@ GENMD_EOF
A suggested use-case here is defining functions in files, and calling them from command-line expressions.
Another suggested use-case is putting default parameter values in files, e.g. using ``begin{@count=is_present(@count)?@count:10}`` in the file, where you can precede that using ``begin{@count=40}`` using ``-e``.
Another suggested use-case is putting default parameter values in files, e.g. using `begin{@count=is_present(@count)?@count:10}` in the file, where you can precede that using `begin{@count=40}` using `-e`.
Moreover, you can have one or more ``-f`` expressions (maybe one function per file, for example) and one or more ``-e`` expressions on the command line. If you mix ``-f`` and ``-e`` then the expressions are evaluated in the order encountered. (Since the expressions are all simply concatenated together in order, don't forget intervening semicolons: e.g. not ``mlr put -e '$x=1' -e '$y=2 ...'`` but rather ``mlr put -e '$x=1;' -e '$y=2' ...``.)
Moreover, you can have one or more `-f` expressions (maybe one function per file, for example) and one or more `-e` expressions on the command line. If you mix `-f` and `-e` then the expressions are evaluated in the order encountered. (Since the expressions are all simply concatenated together in order, don't forget intervening semicolons: e.g. not `mlr put -e '$x=1' -e '$y=2 ...'` but rather `mlr put -e '$x=1;' -e '$y=2' ...`.)
## Semicolons, commas, newlines, and curly braces

22
docs6b/docs/reference-dsl-unset-statements.md
View file

@ -1,10 +1,12 @@
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
DSL reference: unset statements
#
You can clear a map key by assigning the empty string as its value: ``$x=""`` or ``@x=""``. Using ``unset`` you can remove the key entirely. Examples:
You can clear a map key by assigning the empty string as its value: `$x=""` or `@x=""`. Using `unset` you can remove the key entirely. Examples:
<pre>
<pre class="pre-highlight">
<b>cat data/small</b>
</pre>
<pre class="pre-non-highlight">
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
@ -12,8 +14,10 @@ a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr put 'unset $x, $a' data/small</b>
</pre>
<pre class="pre-non-highlight">
b=pan,i=1,y=0.7268028627434533
b=pan,i=2,y=0.5221511083334797
b=wye,i=3,y=0.33831852551664776
@ -21,10 +25,12 @@ b=wye,i=4,y=0.13418874328430463
b=pan,i=5,y=0.8636244699032729
</pre>
This can also be done, of course, using ``mlr cut -x``. You can also clear out-of-stream or local variables, at the base name level, or at an indexed sublevel:
This can also be done, of course, using `mlr cut -x`. You can also clear out-of-stream or local variables, at the base name level, or at an indexed sublevel:
<pre>
<pre class="pre-highlight">
<b>mlr put -q '@sum[$a][$b] += $x; end { dump; unset @sum; dump }' data/small</b>
</pre>
<pre class="pre-non-highlight">
{
"sum": {
"pan": {
@ -43,8 +49,10 @@ This can also be done, of course, using ``mlr cut -x``. You can also clear out-o
{}
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr put -q '@sum[$a][$b] += $x; end { dump; unset @sum["eks"]; dump }' data/small</b>
</pre>
<pre class="pre-non-highlight">
{
"sum": {
"pan": {
@ -73,4 +81,4 @@ This can also be done, of course, using ``mlr cut -x``. You can also clear out-o
}
</pre>
If you use ``unset all`` (or ``unset @*`` which is synonymous), that will unset all out-of-stream variables which have been defined up to that point.
If you use `unset all` (or `unset @*` which is synonymous), that will unset all out-of-stream variables which have been defined up to that point.

6
docs6b/docs/reference-dsl-unset-statements.md.in
View file

@ -1,6 +1,6 @@
DSL reference: unset statements
#
You can clear a map key by assigning the empty string as its value: ``$x=""`` or ``@x=""``. Using ``unset`` you can remove the key entirely. Examples:
You can clear a map key by assigning the empty string as its value: `$x=""` or `@x=""`. Using `unset` you can remove the key entirely. Examples:
GENMD_RUN_COMMAND
cat data/small
@ -10,7 +10,7 @@ GENMD_RUN_COMMAND
mlr put 'unset $x, $a' data/small
GENMD_EOF
This can also be done, of course, using ``mlr cut -x``. You can also clear out-of-stream or local variables, at the base name level, or at an indexed sublevel:
This can also be done, of course, using `mlr cut -x`. You can also clear out-of-stream or local variables, at the base name level, or at an indexed sublevel:
GENMD_RUN_COMMAND
mlr put -q '@sum[$a][$b] += $x; end { dump; unset @sum; dump }' data/small
@ -20,4 +20,4 @@ GENMD_RUN_COMMAND
mlr put -q '@sum[$a][$b] += $x; end { dump; unset @sum["eks"]; dump }' data/small
GENMD_EOF
If you use ``unset all`` (or ``unset @*`` which is synonymous), that will unset all out-of-stream variables which have been defined up to that point.
If you use `unset all` (or `unset @*` which is synonymous), that will unset all out-of-stream variables which have been defined up to that point.

34
docs6b/docs/reference-dsl-user-defined-functions.md
View file

@ -7,7 +7,7 @@ As of Miller 5.0.0 you can define your own functions, as well as subroutines.
Here's the obligatory example of a recursive function to compute the factorial function:
<pre>
<pre class="pre-highlight">
<b>mlr --opprint --from data/small put '</b>
<b> func f(n) {</b>
<b> if (is_numeric(n)) {</b>
@ -22,6 +22,8 @@ Here's the obligatory example of a recursive function to compute the factorial f
<b> $ox = f($x + NR);</b>
<b> $oi = f($i);</b>
<b>'</b>
</pre>
<pre class="pre-non-highlight">
a b i x y ox oi
pan pan 1 0.3467901443380824 0.7268028627434533 0.46705354854811026 1
eks pan 2 0.7586799647899636 0.5221511083334797 3.680838410072862 2
@ -32,29 +34,29 @@ wye pan 5 0.5732889198020006 0.8636244699032729 211.38730958519247 120
Properties of user-defined functions:
* Function bodies start with ``func`` and a parameter list, defined outside of ``begin``, ``end``, or other ``func`` or ``subr`` blocks. (I.e. the Miller DSL has no nested functions.)
* Function bodies start with `func` and a parameter list, defined outside of `begin`, `end`, or other `func` or `subr` blocks. (I.e. the Miller DSL has no nested functions.)
* A function (uniqified by its name) may not be redefined: either by redefining a user-defined function, or by redefining a built-in function. However, functions and subroutines have separate namespaces: you can define a subroutine ``log`` which does not clash with the mathematical ``log`` function.
* A function (uniqified by its name) may not be redefined: either by redefining a user-defined function, or by redefining a built-in function. However, functions and subroutines have separate namespaces: you can define a subroutine `log` which does not clash with the mathematical `log` function.
* Functions may be defined either before or after use (there is an object-binding/linkage step at startup). More specifically, functions may be either recursive or mutually recursive. Functions may not call subroutines.
* Functions may be defined and called either within ``mlr put`` or ``mlr put``.
* Functions may be defined and called either within `mlr put` or `mlr put`.
* Functions have read access to ``$``-variables and ``@``-variables but may not modify them. See also :ref:`cookbook-memoization-with-oosvars` for an example.
* Functions have read access to `$`-variables and `@`-variables but may not modify them. See also [Memoization with out-of-stream variables](misc-examples.md#memoization-with-out-of-stream-variables) for an example.
* Argument values may be reassigned: they are not read-only.
* When a return value is not implicitly returned, this results in a return value of absent-null. (In the example above, if there were records for which the argument to ``f`` is non-numeric, the assignments would be skipped.) See also the section on [xxxx](reference-main-null-data.md).
* When a return value is not implicitly returned, this results in a return value of absent-null. (In the example above, if there were records for which the argument to `f` is non-numeric, the assignments would be skipped.) See also the section on [xxxx](reference-main-null-data.md).
* See the section on :ref:`reference-dsl-local-variables` for information on scope and extent of arguments, as well as for information on the use of local variables within functions.
* See the section on [Local variables](reference-dsl-variables.md#local-variables) for information on scope and extent of arguments, as well as for information on the use of local variables within functions.
* See the section on :ref:`reference-dsl-expressions-from-files` for information on the use of ``-f`` and ``-e`` flags.
* See the section on [Expressions from files](reference-dsl-syntax.md#expressions-from-files) for information on the use of `-f` and `-e` flags.
## User-defined subroutines
Example:
<pre>
<pre class="pre-highlight">
<b>mlr --opprint --from data/small put -q '</b>
<b> begin {</b>
<b> @call_count = 0;</b>
@ -72,6 +74,8 @@ Example:
<b> print "NR=" . NR;</b>
<b> call s(NR);</b>
<b>'</b>
</pre>
<pre class="pre-non-highlight">
NR=1
numcalls=1
NR=2
@ -86,18 +90,18 @@ numcalls=15
Properties of user-defined subroutines:
* Subroutine bodies start with ``subr`` and a parameter list, defined outside of ``begin``, ``end``, or other ``func`` or ``subr`` blocks. (I.e. the Miller DSL has no nested subroutines.)
* Subroutine bodies start with `subr` and a parameter list, defined outside of `begin`, `end`, or other `func` or `subr` blocks. (I.e. the Miller DSL has no nested subroutines.)
* A subroutine (uniqified by its name) may not be redefined. However, functions and subroutines have separate namespaces: you can define a subroutine ``log`` which does not clash with the mathematical ``log`` function.
* A subroutine (uniqified by its name) may not be redefined. However, functions and subroutines have separate namespaces: you can define a subroutine `log` which does not clash with the mathematical `log` function.
* Subroutines may be defined either before or after use (there is an object-binding/linkage step at startup). More specifically, subroutines may be either recursive or mutually recursive. Subroutines may call functions.
* Subroutines may be defined and called either within ``mlr put`` or ``mlr put``.
* Subroutines may be defined and called either within `mlr put` or `mlr put`.
* Subroutines have read/write access to ``$``-variables and ``@``-variables.
* Subroutines have read/write access to `$`-variables and `@`-variables.
* Argument values may be reassigned: they are not read-only.
* See the section on :ref:`reference-dsl-local-variables` for information on scope and extent of arguments, as well as for information on the use of local variables within functions.
* See the section on [local variables](reference-dsl-variables.md#local-variables) for information on scope and extent of arguments, as well as for information on the use of local variables within functions.
* See the section on :ref:`reference-dsl-expressions-from-files` for information on the use of ``-f`` and ``-e`` flags.
* See the section on [Expressions from files](reference-dsl-syntax.md#expressions-from-files) for information on the use of `-f` and `-e` flags.

26
docs6b/docs/reference-dsl-user-defined-functions.md.in
View file

@ -10,23 +10,23 @@ GENMD_INCLUDE_AND_RUN_ESCAPED(data/factorial-example.sh)
Properties of user-defined functions:
* Function bodies start with ``func`` and a parameter list, defined outside of ``begin``, ``end``, or other ``func`` or ``subr`` blocks. (I.e. the Miller DSL has no nested functions.)
* Function bodies start with `func` and a parameter list, defined outside of `begin`, `end`, or other `func` or `subr` blocks. (I.e. the Miller DSL has no nested functions.)
* A function (uniqified by its name) may not be redefined: either by redefining a user-defined function, or by redefining a built-in function. However, functions and subroutines have separate namespaces: you can define a subroutine ``log`` which does not clash with the mathematical ``log`` function.
* A function (uniqified by its name) may not be redefined: either by redefining a user-defined function, or by redefining a built-in function. However, functions and subroutines have separate namespaces: you can define a subroutine `log` which does not clash with the mathematical `log` function.
* Functions may be defined either before or after use (there is an object-binding/linkage step at startup). More specifically, functions may be either recursive or mutually recursive. Functions may not call subroutines.
* Functions may be defined and called either within ``mlr put`` or ``mlr put``.
* Functions may be defined and called either within `mlr put` or `mlr put`.
* Functions have read access to ``$``-variables and ``@``-variables but may not modify them. See also :ref:`cookbook-memoization-with-oosvars` for an example.
* Functions have read access to `$`-variables and `@`-variables but may not modify them. See also [Memoization with out-of-stream variables](misc-examples.md#memoization-with-out-of-stream-variables) for an example.
* Argument values may be reassigned: they are not read-only.
* When a return value is not implicitly returned, this results in a return value of absent-null. (In the example above, if there were records for which the argument to ``f`` is non-numeric, the assignments would be skipped.) See also the section on [xxxx](reference-main-null-data.md).
* When a return value is not implicitly returned, this results in a return value of absent-null. (In the example above, if there were records for which the argument to `f` is non-numeric, the assignments would be skipped.) See also the section on [xxxx](reference-main-null-data.md).
* See the section on :ref:`reference-dsl-local-variables` for information on scope and extent of arguments, as well as for information on the use of local variables within functions.
* See the section on [Local variables](reference-dsl-variables.md#local-variables) for information on scope and extent of arguments, as well as for information on the use of local variables within functions.
* See the section on :ref:`reference-dsl-expressions-from-files` for information on the use of ``-f`` and ``-e`` flags.
* See the section on [Expressions from files](reference-dsl-syntax.md#expressions-from-files) for information on the use of `-f` and `-e` flags.
## User-defined subroutines
@ -36,18 +36,18 @@ GENMD_INCLUDE_AND_RUN_ESCAPED(data/subr-example.sh)
Properties of user-defined subroutines:
* Subroutine bodies start with ``subr`` and a parameter list, defined outside of ``begin``, ``end``, or other ``func`` or ``subr`` blocks. (I.e. the Miller DSL has no nested subroutines.)
* Subroutine bodies start with `subr` and a parameter list, defined outside of `begin`, `end`, or other `func` or `subr` blocks. (I.e. the Miller DSL has no nested subroutines.)
* A subroutine (uniqified by its name) may not be redefined. However, functions and subroutines have separate namespaces: you can define a subroutine ``log`` which does not clash with the mathematical ``log`` function.
* A subroutine (uniqified by its name) may not be redefined. However, functions and subroutines have separate namespaces: you can define a subroutine `log` which does not clash with the mathematical `log` function.
* Subroutines may be defined either before or after use (there is an object-binding/linkage step at startup). More specifically, subroutines may be either recursive or mutually recursive. Subroutines may call functions.
* Subroutines may be defined and called either within ``mlr put`` or ``mlr put``.
* Subroutines may be defined and called either within `mlr put` or `mlr put`.
* Subroutines have read/write access to ``$``-variables and ``@``-variables.
* Subroutines have read/write access to `$`-variables and `@`-variables.
* Argument values may be reassigned: they are not read-only.
* See the section on :ref:`reference-dsl-local-variables` for information on scope and extent of arguments, as well as for information on the use of local variables within functions.
* See the section on [local variables](reference-dsl-variables.md#local-variables) for information on scope and extent of arguments, as well as for information on the use of local variables within functions.
* See the section on :ref:`reference-dsl-expressions-from-files` for information on the use of ``-f`` and ``-e`` flags.
* See the section on [Expressions from files](reference-dsl-syntax.md#expressions-from-files) for information on the use of `-f` and `-e` flags.

254
docs6b/docs/reference-dsl-variables.md
View file

@ -3,11 +3,11 @@
Miller has the following kinds of variables:
**Built-in variables** such as ``NF``, ``NF``, ``FILENAME``, ``M_PI``, and ``M_E``. These are all capital letters and are read-only (although some of them change value from one record to another).
**Built-in variables** such as `NF`, `NF`, `FILENAME`, `M_PI`, and `M_E`. These are all capital letters and are read-only (although some of them change value from one record to another).
**Fields of stream records**, accessed using the ``$`` prefix. These refer to fields of the current data-stream record. For example, in ``echo x=1,y=2 | mlr put '$z = $x + $y'``, ``$x`` and ``$y`` refer to input fields, and ``$z`` refers to a new, computed output field. In a few contexts, presented below, you can refer to the entire record as ``$*``.
**Fields of stream records**, accessed using the `$` prefix. These refer to fields of the current data-stream record. For example, in `echo x=1,y=2 | mlr put '$z = $x + $y'`, `$x` and `$y` refer to input fields, and `$z` refers to a new, computed output field. In a few contexts, presented below, you can refer to the entire record as `$*`.
**Out-of-stream variables** accessed using the ``@`` prefix. These refer to data which persist from one record to the next, including in ``begin`` and ``end`` blocks (which execute before/after the record stream is consumed, respectively). You use them to remember values across records, such as sums, differences, counters, and so on. In a few contexts, presented below, you can refer to the entire out-of-stream-variables collection as ``@*``.
**Out-of-stream variables** accessed using the `@` prefix. These refer to data which persist from one record to the next, including in `begin` and `end` blocks (which execute before/after the record stream is consumed, respectively). You use them to remember values across records, such as sums, differences, counters, and so on. In a few contexts, presented below, you can refer to the entire out-of-stream-variables collection as `@*`.
**Local variables** are limited in scope and extent to the current statements being executed: these include function arguments, bound variables in for loops, and explicitly declared local variables.
@ -15,19 +15,23 @@ Miller has the following kinds of variables:
## Built-in variables
These are written all in capital letters, such as ``NR``, ``NF``, ``FILENAME``, and only a small, specific set of them is defined by Miller.
These are written all in capital letters, such as `NR`, `NF`, `FILENAME`, and only a small, specific set of them is defined by Miller.
Namely, Miller supports the following five built-in variables for :doc:`filter and put <reference-dsl>`, all ``awk``-inspired: ``NF``, ``NR``, ``FNR``, ``FILENUM``, and ``FILENAME``, as well as the mathematical constants ``M_PI`` and ``M_E``. Lastly, the ``ENV`` hashmap allows read access to environment variables, e.g. ``ENV["HOME"]`` or ``ENV["foo_".$hostname]``.
Namely, Miller supports the following five built-in variables for [filter and put](reference-dsl.md), all `awk`-inspired: `NF`, `NR`, `FNR`, `FILENUM`, and `FILENAME`, as well as the mathematical constants `M_PI` and `M_E`. Lastly, the `ENV` hashmap allows read access to environment variables, e.g. `ENV["HOME"]` or `ENV["foo_".$hostname]`.
<pre>
<pre class="pre-highlight">
<b>mlr filter 'FNR == 2' data/small*</b>
</pre>
<pre class="pre-non-highlight">
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
1=pan,2=pan,3=1,4=0.3467901443380824,5=0.7268028627434533
a=wye,b=eks,i=10000,x=0.734806020620654365,y=0.884788571337605134
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr put '$fnr = FNR' data/small*</b>
</pre>
<pre class="pre-non-highlight">
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533,fnr=1
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797,fnr=2
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776,fnr=3
@ -46,19 +50,23 @@ a=hat,b=wye,i=10002,x=0.321507044286237609,y=0.568893318795083758,fnr=4
a=pan,b=zee,i=10003,x=0.272054845593895200,y=0.425789896597056627,fnr=5
</pre>
Their values of ``NF``, ``NR``, ``FNR``, ``FILENUM``, and ``FILENAME`` change from one record to the next as Miller scans through your input data stream. The mathematical constants, of course, do not change; ``ENV`` is populated from the system environment variables at the time Miller starts and is read-only for the remainder of program execution.
Their values of `NF`, `NR`, `FNR`, `FILENUM`, and `FILENAME` change from one record to the next as Miller scans through your input data stream. The mathematical constants, of course, do not change; `ENV` is populated from the system environment variables at the time Miller starts and is read-only for the remainder of program execution.
Their **scope is global**: you can refer to them in any ``filter`` or ``put`` statement. Their values are assigned by the input-record reader:
Their **scope is global**: you can refer to them in any `filter` or `put` statement. Their values are assigned by the input-record reader:
<pre>
<pre class="pre-highlight">
<b>mlr --csv put '$nr = NR' data/a.csv</b>
</pre>
<pre class="pre-non-highlight">
a,b,c,nr
1,2,3,1
4,5,6,2
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --csv repeat -n 3 then put '$nr = NR' data/a.csv</b>
</pre>
<pre class="pre-non-highlight">
a,b,c,nr
1,2,3,1
1,2,3,1
@ -68,24 +76,26 @@ a,b,c,nr
4,5,6,2
</pre>
The **extent** is for the duration of the put/filter: in a ``begin`` statement (which executes before the fimd.input record is consumed) you will find ``NR=1`` and in an ``end`` statement (which is executed after the last input record is consumed) you will find ``NR`` to be the total number of records ingested.
The **extent** is for the duration of the put/filter: in a `begin` statement (which executes before the fimd.input record is consumed) you will find `NR=1` and in an `end` statement (which is executed after the last input record is consumed) you will find `NR` to be the total number of records ingested.
These are all **read-only** for the ``mlr put`` and ``mlr filter`` DSLs: they may be assigned from, e.g. ``$nr=NR``, but they may not be assigned to: ``NR=100`` is a syntax error.
These are all **read-only** for the `mlr put` and `mlr filter` DSLs: they may be assigned from, e.g. `$nr=NR`, but they may not be assigned to: `NR=100` is a syntax error.
## Field names
Names of fields within stream records must be specified using a ``$`` in :doc:`filter and put expressions <reference-dsl>`, even though the dollar signs don't appear in the data stream itself. For integer-indexed data, this looks like ``awk``'s ``$1,$2,$3``, except that Miller allows non-numeric names such as ``$quantity`` or ``$hostname``. Likewise, enclose string literals in double quotes in ``filter`` expressions even though they don't appear in file data. In particular, ``mlr filter '$x=="abc"'`` passes through the record ``x=abc``.
Names of fields within stream records must be specified using a `$` in [filter and put expressions](reference-dsl.md), even though the dollar signs don't appear in the data stream itself. For integer-indexed data, this looks like `awk`'s `$1,$2,$3`, except that Miller allows non-numeric names such as `$quantity` or `$hostname`. Likewise, enclose string literals in double quotes in `filter` expressions even though they don't appear in file data. In particular, `mlr filter '$x=="abc"'` passes through the record `x=abc`.
If field names have **special characters** such as ``.`` then you can use braces, e.g. ``'${field.name}'``.
If field names have **special characters** such as `.` then you can use braces, e.g. `'${field.name}'`.
You may also use a **computed field name** in square brackets, e.g.
<pre>
<pre class="pre-highlight">
<b>echo a=3,b=4 | mlr filter '$["x"] < 0.5'</b>
</pre>
<pre>
<pre class="pre-highlight">
<b>echo s=green,t=blue,a=3,b=4 | mlr put '$[$s."_".$t] = $a * $b'</b>
</pre>
<pre class="pre-non-highlight">
s=green,t=blue,a=3,b=4,green_blue=12
</pre>
@ -93,22 +103,24 @@ Notes:
The names of record fields depend on the contents of your input data stream, and their values change from one record to the next as Miller scans through your input data stream.
Their **extent** is limited to the current record; their **scope** is the ``filter`` or ``put`` command in which they appear.
Their **extent** is limited to the current record; their **scope** is the `filter` or `put` command in which they appear.
These are **read-write**: you can do ``$y=2*$x``, ``$x=$x+1``, etc.
These are **read-write**: you can do `$y=2*$x`, `$x=$x+1`, etc.
Records are Miller's output: field names present in the input stream are passed through to output (written to standard output) unless fields are removed with ``cut``, or records are excluded with ``filter`` or ``put -q``, etc. Simply assign a value to a field and it will be output.
Records are Miller's output: field names present in the input stream are passed through to output (written to standard output) unless fields are removed with `cut`, or records are excluded with `filter` or `put -q`, etc. Simply assign a value to a field and it will be output.
## Positional field names
Even though Miller's main selling point is name-indexing, sometimes you really want to refer to a field name by its positional index (starting from 1).
Use ``$[[3]]`` to access the name of field 3. More generally, any expression evaluating to an integer can go between ``$[[`` and ``]]``.
Use `$[[3]]` to access the name of field 3. More generally, any expression evaluating to an integer can go between `$[[` and `]]`.
Then using a computed field name, ``$[ $[[3]] ]`` is the value in the third field. This has the shorter equivalent notation ``$[[[3]]]``.
Then using a computed field name, `$[ $[[3]] ]` is the value in the third field. This has the shorter equivalent notation `$[[[3]]]`.
<pre>
<pre class="pre-highlight">
<b>mlr cat data/small</b>
</pre>
<pre class="pre-non-highlight">
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
@ -116,8 +128,10 @@ a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr put '$[[3]] = "NEW"' data/small</b>
</pre>
<pre class="pre-non-highlight">
a=pan,b=pan,NEW=1,x=0.3467901443380824,y=0.7268028627434533
a=eks,b=pan,NEW=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,NEW=3,x=0.20460330576630303,y=0.33831852551664776
@ -125,8 +139,10 @@ a=eks,b=wye,NEW=4,x=0.38139939387114097,y=0.13418874328430463
a=wye,b=pan,NEW=5,x=0.5732889198020006,y=0.8636244699032729
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr put '$[[[3]]] = "NEW"' data/small</b>
</pre>
<pre class="pre-non-highlight">
a=pan,b=pan,i=NEW,x=0.3467901443380824,y=0.7268028627434533
a=eks,b=pan,i=NEW,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,i=NEW,x=0.20460330576630303,y=0.33831852551664776
@ -134,8 +150,10 @@ a=eks,b=wye,i=NEW,x=0.38139939387114097,y=0.13418874328430463
a=wye,b=pan,i=NEW,x=0.5732889198020006,y=0.8636244699032729
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr put '$NEW = $[[NR]]' data/small</b>
</pre>
<pre class="pre-non-highlight">
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533,NEW=a
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797,NEW=b
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776,NEW=i
@ -143,8 +161,10 @@ a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463,NEW=x
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729,NEW=y
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr put '$NEW = $[[[NR]]]' data/small</b>
</pre>
<pre class="pre-non-highlight">
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533,NEW=pan
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797,NEW=pan
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776,NEW=3
@ -152,8 +172,10 @@ a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463,NEW=0.38139939387114
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729,NEW=0.8636244699032729
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr put '$[[[NR]]] = "NEW"' data/small</b>
</pre>
<pre class="pre-non-highlight">
a=NEW,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
a=eks,b=NEW,i=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,i=NEW,x=0.20460330576630303,y=0.33831852551664776
@ -161,10 +183,12 @@ a=eks,b=wye,i=4,x=NEW,y=0.13418874328430463
a=wye,b=pan,i=5,x=0.5732889198020006,y=NEW
</pre>
Right-hand side accesses to non-existent fields -- i.e. with index less than 1 or greater than ``NF`` -- return an absent value. Likewise, left-hand side accesses only refer to fields which already exist. For example, if a field has 5 records then assigning the name or value of the 6th (or 600th) field results in a no-op.
Right-hand side accesses to non-existent fields -- i.e. with index less than 1 or greater than `NF` -- return an absent value. Likewise, left-hand side accesses only refer to fields which already exist. For example, if a field has 5 records then assigning the name or value of the 6th (or 600th) field results in a no-op.
<pre>
<pre class="pre-highlight">
<b>mlr put '$[[6]] = "NEW"' data/small</b>
</pre>
<pre class="pre-non-highlight">
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
@ -172,8 +196,10 @@ a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr put '$[[[6]]] = "NEW"' data/small</b>
</pre>
<pre class="pre-non-highlight">
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
@ -183,46 +209,52 @@ a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
## Out-of-stream variables
These are prefixed with an at-sign, e.g. ``@sum``. Furthermore, unlike built-in variables and stream-record fields, they are maintained in an arbitrarily nested hashmap: you can do ``@sum += $quanity``, or ``@sum[$color] += $quanity``, or ``@sum[$color][$shape] += $quanity``. The keys for the multi-level hashmap can be any expression which evaluates to string or integer: e.g. ``@sum[NR] = $a + $b``, ``@sum[$a."-".$b] = $x``, etc.
These are prefixed with an at-sign, e.g. `@sum`. Furthermore, unlike built-in variables and stream-record fields, they are maintained in an arbitrarily nested hashmap: you can do `@sum += $quanity`, or `@sum[$color] += $quanity`, or `@sum[$color][$shape] += $quanity`. The keys for the multi-level hashmap can be any expression which evaluates to string or integer: e.g. `@sum[NR] = $a + $b`, `@sum[$a."-".$b] = $x`, etc.
Their names and their values are entirely under your control; they change only when you assign to them.
Just as for field names in stream records, if you want to define out-of-stream variables with **special characters** such as ``.`` then you can use braces, e.g. ``'@{variable.name}["index"]'``.
Just as for field names in stream records, if you want to define out-of-stream variables with **special characters** such as `.` then you can use braces, e.g. `'@{variable.name}["index"]'`.
You may use a **computed key** in square brackets, e.g.
<pre>
<pre class="pre-highlight">
<b>echo s=green,t=blue,a=3,b=4 | mlr put -q '@[$s."_".$t] = $a * $b; emit all'</b>
</pre>
<pre class="pre-non-highlight">
green_blue=12
</pre>
Out-of-stream variables are **scoped** to the ``put`` command in which they appear. In particular, if you have two or more ``put`` commands separated by ``then``, each put will have its own set of out-of-stream variables:
Out-of-stream variables are **scoped** to the `put` command in which they appear. In particular, if you have two or more `put` commands separated by `then`, each put will have its own set of out-of-stream variables:
<pre>
<pre class="pre-highlight">
<b>cat data/a.dkvp</b>
</pre>
<pre class="pre-non-highlight">
a=1,b=2,c=3
a=4,b=5,c=6
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr put '@sum += $a; end {emit @sum}' \</b>
<b> then put 'is_present($a) {$a=10*$a; @sum += $a}; end {emit @sum}' \</b>
<b> data/a.dkvp</b>
</pre>
<pre class="pre-non-highlight">
a=10,b=2,c=3
a=40,b=5,c=6
sum=5
sum=50
</pre>
Out-of-stream variables' **extent** is from the start to the end of the record stream, i.e. every time the ``put`` or ``filter`` statement referring to them is executed.
Out-of-stream variables' **extent** is from the start to the end of the record stream, i.e. every time the `put` or `filter` statement referring to them is executed.
Out-of-stream variables are **read-write**: you can do ``$sum=@sum``, ``@sum=$sum``, etc.
Out-of-stream variables are **read-write**: you can do `$sum=@sum`, `@sum=$sum`, etc.
## Indexed out-of-stream variables
Using an index on the ``@count`` and ``@sum`` variables, we get the benefit of the ``-g`` (group-by) option which ``mlr stats1`` and various other Miller commands have:
Using an index on the `@count` and `@sum` variables, we get the benefit of the `-g` (group-by) option which `mlr stats1` and various other Miller commands have:
<pre>
<pre class="pre-highlight">
<b>mlr put -q '</b>
<b> @x_count[$a] += 1;</b>
<b> @x_sum[$a] += $x;</b>
@ -231,6 +263,8 @@ Using an index on the ``@count`` and ``@sum`` variables, we get the benefit of t
<b> emit @x_sum, "a";</b>
<b> }</b>
<b>' ./data/small</b>
</pre>
<pre class="pre-non-highlight">
a=pan,x_count=1
a=eks,x_count=2
a=wye,x_count=2
@ -239,8 +273,10 @@ a=eks,x_sum=1.1400793586611044
a=wye,x_sum=0.7778922255683036
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr stats1 -a count,sum -f x -g a ./data/small</b>
</pre>
<pre class="pre-non-highlight">
a=pan,x_count=1,x_sum=0.3467901443380824
a=eks,x_count=2,x_sum=1.1400793586611044
a=wye,x_count=2,x_sum=0.7778922255683036
@ -248,7 +284,7 @@ a=wye,x_count=2,x_sum=0.7778922255683036
Indices can be arbitrarily deep -- here there are two or more of them:
<pre>
<pre class="pre-highlight">
<b>mlr --from data/medium put -q '</b>
<b> @x_count[$a][$b] += 1;</b>
<b> @x_sum[$a][$b] += $x;</b>
@ -256,6 +292,8 @@ Indices can be arbitrarily deep -- here there are two or more of them:
<b> emit (@x_count, @x_sum), "a", "b";</b>
<b> }</b>
<b>'</b>
</pre>
<pre class="pre-non-highlight">
a=pan,b=pan,x_count=427,x_sum=219.1851288316854
a=pan,b=wye,x_count=395,x_sum=198.43293070748447
a=pan,b=eks,x_count=429,x_sum=216.07522773165525
@ -283,11 +321,11 @@ a=hat,b=hat,x_count=381,x_sum=182.8535323148762
a=hat,b=pan,x_count=363,x_sum=168.5538067327806
</pre>
The idea is that ``stats1``, and other Miller verbs, encapsulate frequently-used patterns with a minimum of keystroking (and run a little faster), whereas using out-of-stream variables you have more flexibility and control in what you do.
The idea is that `stats1`, and other Miller verbs, encapsulate frequently-used patterns with a minimum of keystroking (and run a little faster), whereas using out-of-stream variables you have more flexibility and control in what you do.
Begin/end blocks can be mixed with pattern/action blocks. For example:
<pre>
<pre class="pre-highlight">
<b>mlr put '</b>
<b> begin {</b>
<b> @num_total = 0;</b>
@ -302,6 +340,8 @@ Begin/end blocks can be mixed with pattern/action blocks. For example:
<b> emitf @num_total, @num_positive</b>
<b> }</b>
<b>' data/put-gating-example-1.dkvp</b>
</pre>
<pre class="pre-non-highlight">
x=-1
x=0
x=1,y=0,z=0
@ -310,15 +350,13 @@ x=3,y=0.4771212547196624,z=0.6907396432228734
num_total=5,num_positive=3
</pre>
.. _reference-dsl-local-variables:
## Local variables
Local variables are similar to out-of-stream variables, except that their extent is limited to the expressions in which they appear (and their basenames can't be computed using square brackets). There are three kinds of local variables: **arguments** to functions/subroutines, **variables bound within for-loops**, and **locals** defined within control blocks. They may be untyped using ``var``, or typed using ``num``, ``int``, ``float``, ``str``, ``bool``, and ``map``.
Local variables are similar to out-of-stream variables, except that their extent is limited to the expressions in which they appear (and their basenames can't be computed using square brackets). There are three kinds of local variables: **arguments** to functions/subroutines, **variables bound within for-loops**, and **locals** defined within control blocks. They may be untyped using `var`, or typed using `num`, `int`, `float`, `str`, `bool`, and `map`.
For example:
<pre>
<pre class="pre-highlight">
<b># Here I'm using a specified random-number seed so this example always</b>
<b># produces the same output for this web document: in everyday practice we</b>
<b># would leave off the --seed 12345 part.</b>
@ -335,6 +373,8 @@ For example:
<b> num o = f(10, 20); # local to the top-level scope</b>
<b> $o = o;</b>
<b>'</b>
</pre>
<pre class="pre-non-highlight">
i=1,o=15.952526011537227
i=2,o=12.782237754999116
i=3,o=15.126606630220966
@ -349,32 +389,34 @@ i=10,o=15.37686787628025
Things which are completely unsurprising, resembling many other languages:
* Parameter names are bound to their arguments but can be reassigned, e.g. if there is a parameter named ``a`` then you can reassign the value of ``a`` to be something else within the function if you like.
* Parameter names are bound to their arguments but can be reassigned, e.g. if there is a parameter named `a` then you can reassign the value of `a` to be something else within the function if you like.
* However, you cannot redeclare the *type* of an argument or a local: ``var a=1; var a=2`` is an error but ``var a=1; a=2`` is OK.
* However, you cannot redeclare the *type* of an argument or a local: `var a=1; var a=2` is an error but `var a=1; a=2` is OK.
* All argument-passing is positional rather than by name; arguments are passed by value, not by reference. (This is also true for map-valued variables: they are not, and cannot be, passed by reference)
* You can define locals (using ``var``, ``num``, etc.) at any scope (if-statements, else-statements, while-loops, for-loops, or the top-level scope), and nested scopes will have access (more details on scope in the next section). If you define a local variable with the same name inside an inner scope, then a new variable is created with the narrower scope.
* You can define locals (using `var`, `num`, etc.) at any scope (if-statements, else-statements, while-loops, for-loops, or the top-level scope), and nested scopes will have access (more details on scope in the next section). If you define a local variable with the same name inside an inner scope, then a new variable is created with the narrower scope.
* If you assign to a local variable for the first time in a scope without declaring it as ``var``, ``num``, etc. then: if it exists in an outer scope, that outer-scope variable will be updated; if not, it will be defined in the current scope as if ``var`` had been used. (See also :ref:`reference-dsl-type-checking` for an example.) I recommend always declaring variables explicitly to make the intended scoping clear.
* If you assign to a local variable for the first time in a scope without declaring it as `var`, `num`, etc. then: if it exists in an outer scope, that outer-scope variable will be updated; if not, it will be defined in the current scope as if `var` had been used. (See also [Type-checking](reference-dsl-variables.md#type-checking) for an example.) I recommend always declaring variables explicitly to make the intended scoping clear.
* Functions and subroutines never have access to locals from their callee (unless passed by value as arguments).
Things which are perhaps surprising compared to other languages:
* Type declarations using ``var``, or typed using ``num``, ``int``, ``float``, ``str``, and ``bool`` are necessary to declare local variables. Function arguments and variables bound in for-loops over stream records and out-of-stream variables are *implicitly* declared using ``var``. (Some examples are shown below.)
* Type declarations using `var`, or typed using `num`, `int`, `float`, `str`, and `bool` are necessary to declare local variables. Function arguments and variables bound in for-loops over stream records and out-of-stream variables are *implicitly* declared using `var`. (Some examples are shown below.)
* Type-checking is done at assignment time. For example, ``float f = 0`` is an error (since ``0`` is an integer), as is ``float f = 0.0; f = 1``. For this reason I prefer to use ``num`` over ``float`` in most contexts since ``num`` encompasses integer and floating-point values. More information about type-checking is at :ref:`reference-dsl-type-checking`.
* Type-checking is done at assignment time. For example, `float f = 0` is an error (since `0` is an integer), as is `float f = 0.0; f = 1`. For this reason I prefer to use `num` over `float` in most contexts since `num` encompasses integer and floating-point values. More information is at [Type-checking](reference-dsl-variables.md#type-checking).
* Bound variables in for-loops over stream records and out-of-stream variables are implicitly local to that block. E.g. in ``for (k, v in $*) { ... }`` ``for ((k1, k2), v in @*) { ... }`` if there are ``k``, ``v``, etc. in the enclosing scope then those will be masked by the loop-local bound variables in the loop, and moreover the values of the loop-local bound variables are not available after the end of the loop.
* Bound variables in for-loops over stream records and out-of-stream variables are implicitly local to that block. E.g. in `for (k, v in $*) { ... }` `for ((k1, k2), v in @*) { ... }` if there are `k`, `v`, etc. in the enclosing scope then those will be masked by the loop-local bound variables in the loop, and moreover the values of the loop-local bound variables are not available after the end of the loop.
* For C-style triple-for loops, if a for-loop variable is defined using ``var``, ``int``, etc. then it is scoped to that for-loop. E.g. ``for (i = 0; i < 10; i += 1) { ... }`` and ``for (int i = 0; i < 10; i += 1) { ... }``. (This is unsurprising.). If there is no typedecl and an outer-scope variable of that name exists, then it is used. (This is also unsurprising.) But of there is no outer-scope variable of that name then the variable is scoped to the for-loop only.
* For C-style triple-for loops, if a for-loop variable is defined using `var`, `int`, etc. then it is scoped to that for-loop. E.g. `for (i = 0; i < 10; i += 1) { ... }` and `for (int i = 0; i < 10; i += 1) { ... }`. (This is unsurprising.). If there is no typedecl and an outer-scope variable of that name exists, then it is used. (This is also unsurprising.) But of there is no outer-scope variable of that name then the variable is scoped to the for-loop only.
The following example demonstrates the scope rules:
<pre>
<pre class="pre-highlight">
<b>cat data/scope-example.mlr</b>
</pre>
<pre class="pre-non-highlight">
func f(a) { # argument is local to the function
var b = 100; # local to the function
c = 100; # local to the function; does not overwrite outer c
@ -400,15 +442,19 @@ $outer_c = c;
$outer_d = d; # there is no outer d defined so no assignment happens
</pre>
<pre>
<pre class="pre-highlight">
<b>cat data/scope-example.dat</b>
</pre>
<pre class="pre-non-highlight">
n=1,x=123
n=2,x=456
n=3,x=789
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --oxtab --from data/scope-example.dat put -f data/scope-example.mlr</b>
</pre>
<pre class="pre-non-highlight">
n 1
x 123
outer_a 10
@ -434,8 +480,10 @@ outer_c 60
And this example demonstrates the type-declaration rules:
<pre>
<pre class="pre-highlight">
<b>cat data/type-decl-example.mlr</b>
</pre>
<pre class="pre-non-highlight">
subr s(a, str b, int c) { # a is implicitly var (untyped).
# b is explicitly str.
# c is explicitly int.
@ -477,11 +525,11 @@ print "outer j =" . j; # j is undefined in this scope
## Map literals
Miller's ``put``/``filter`` DSL has four kinds of hashmaps. **Stream records** are (single-level) maps from name to value. **Out-of-stream variables** and **local variables** can also be maps, although they can be multi-level hashmaps (e.g. ``@sum[$x][$y]``). The fourth kind is **map literals**. These cannot be on the left-hand side of assignment expressions. Syntactically they look like JSON, although Miller allows string and integer keys in its map literals while JSON allows only string keys (e.g. ``"3"`` rather than ``3``).
Miller's `put`/`filter` DSL has four kinds of hashmaps. **Stream records** are (single-level) maps from name to value. **Out-of-stream variables** and **local variables** can also be maps, although they can be multi-level hashmaps (e.g. `@sum[$x][$y]`). The fourth kind is **map literals**. These cannot be on the left-hand side of assignment expressions. Syntactically they look like JSON, although Miller allows string and integer keys in its map literals while JSON allows only string keys (e.g. `"3"` rather than `3`).
For example, the following swaps the input stream's ``a`` and ``i`` fields, modifies ``y``, and drops the rest:
For example, the following swaps the input stream's `a` and `i` fields, modifies `y`, and drops the rest:
<pre>
<pre class="pre-highlight">
<b>mlr --opprint put '</b>
<b> $* = {</b>
<b> "a": $i,</b>
@ -489,6 +537,8 @@ For example, the following swaps the input stream's ``a`` and ``i`` fields, modi
<b> "y": $y * 10,</b>
<b> }</b>
<b>' data/small</b>
</pre>
<pre class="pre-non-highlight">
a i y
1 pan 7.268028627434533
2 eks 5.221511083334796
@ -499,7 +549,7 @@ a i y
Likewise, you can assign map literals to out-of-stream variables or local variables; pass them as arguments to user-defined functions, return them from functions, and so on:
<pre>
<pre class="pre-highlight">
<b>mlr --from data/small put '</b>
<b> func f(map m): map {</b>
<b> m["x"] *= 200;</b>
@ -507,6 +557,8 @@ Likewise, you can assign map literals to out-of-stream variables or local variab
<b> }</b>
<b> $* = f({"a": $a, "x": $x});</b>
<b>'</b>
</pre>
<pre class="pre-non-highlight">
a=pan,x=69.35802886761648
a=eks,x=151.73599295799272
a=wye,x=40.92066115326061
@ -516,7 +568,7 @@ a=wye,x=114.65778396040011
Like out-of-stream and local variables, map literals can be multi-level:
<pre>
<pre class="pre-highlight">
<b>mlr --from data/small put -q '</b>
<b> begin {</b>
<b> @o = {</b>
@ -536,6 +588,8 @@ Like out-of-stream and local variables, map literals can be multi-level:
<b> dump @o;</b>
<b> }</b>
<b>'</b>
</pre>
<pre class="pre-non-highlight">
{
"nrec": 5,
"nkey": {
@ -545,25 +599,22 @@ Like out-of-stream and local variables, map literals can be multi-level:
}
</pre>
By default, map-valued expressions are dumped using JSON formatting. If you use ``dump`` to print a hashmap with integer keys and you don't want them double-quoted (JSON-style) then you can use ``mlr put --jknquoteint``. See also ``mlr put --help``.
.. _reference-dsl-type-checking:
By default, map-valued expressions are dumped using JSON formatting. If you use `dump` to print a hashmap with integer keys and you don't want them double-quoted (JSON-style) then you can use `mlr put --jknquoteint`. See also `mlr put --help`.
## Type-checking
Miller's ``put``/``filter`` DSLs support two optional kinds of type-checking. One is inline **type-tests** and **type-assertions** within expressions. The other is **type declarations** for assignments to local variables, binding of arguments to user-defined functions, and return values from user-defined functions, These are discussed in the following subsections.
Miller's `put`/`filter` DSLs support two optional kinds of type-checking. One is inline **type-tests** and **type-assertions** within expressions. The other is **type declarations** for assignments to local variables, binding of arguments to user-defined functions, and return values from user-defined functions, These are discussed in the following subsections.
Use of type-checking is entirely up to you: omit it if you want flexibility with heterogeneous data; use it if you want to help catch misspellings in your DSL code or unexpected irregularities in your input data.
.. _reference-dsl-type-tests-and-assertions:
### Type-test and type-assertion expressions
Type-test and type-assertion expressions
................................................................
The following `is...` functions take a value and return a boolean indicating whether the argument is of the indicated type. The `assert_...` functions return their argument if it is of the specified type, and cause a fatal error otherwise:
The following ``is...`` functions take a value and return a boolean indicating whether the argument is of the indicated type. The ``assert_...`` functions return their argument if it is of the specified type, and cause a fatal error otherwise:
<pre>
<pre class="pre-highlight">
<b>mlr -f | grep ^is</b>
</pre>
<pre class="pre-non-highlight">
is_absent
is_array
is_bool
@ -585,8 +636,10 @@ is_present
is_string
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr -f | grep ^assert</b>
</pre>
<pre class="pre-non-highlight">
asserting_absent
asserting_array
asserting_bool
@ -610,25 +663,24 @@ asserting_string
See [Data-cleaning Examples](data-cleaning-examples.md) for examples of how to use these.
Type-declarations for local variables, function parameter, and function return values
...............................................................................................
### Type-declarations for local variables, function parameter, and function return values
Local variables can be defined either untyped as in ``x = 1``, or typed as in ``int x = 1``. Types include **var** (explicitly untyped), **int**, **float**, **num** (int or float), **str**, **bool**, and **map**. These optional type declarations are enforced at the time values are assigned to variables: whether at the initial value assignment as in ``int x = 1`` or in any subsequent assignments to the same variable farther down in the scope.
Local variables can be defined either untyped as in `x = 1`, or typed as in `int x = 1`. Types include **var** (explicitly untyped), **int**, **float**, **num** (int or float), **str**, **bool**, and **map**. These optional type declarations are enforced at the time values are assigned to variables: whether at the initial value assignment as in `int x = 1` or in any subsequent assignments to the same variable farther down in the scope.
The reason for ``num`` is that ``int`` and ``float`` typedecls are very precise:
The reason for `num` is that `int` and `float` typedecls are very precise:
<pre>
<pre class="pre-non-highlight">
float a = 0; # Runtime error since 0 is int not float
int b = 1.0; # Runtime error since 1.0 is float not int
num c = 0; # OK
num d = 1.0; # OK
</pre>
A suggestion is to use ``num`` for general use when you want numeric content, and use ``int`` when you genuinely want integer-only values, e.g. in loop indices or map keys (since Miller map keys can only be strings or ints).
A suggestion is to use `num` for general use when you want numeric content, and use `int` when you genuinely want integer-only values, e.g. in loop indices or map keys (since Miller map keys can only be strings or ints).
The ``var`` type declaration indicates no type restrictions, e.g. ``var x = 1`` has the same type restrictions on ``x`` as ``x = 1``. The difference is in intentional shadowing: if you have ``x = 1`` in outer scope and ``x = 2`` in inner scope (e.g. within a for-loop or an if-statement) then outer-scope ``x`` has value 2 after the second assignment. But if you have ``var x = 2`` in the inner scope, then you are declaring a variable scoped to the inner block.) For example:
The `var` type declaration indicates no type restrictions, e.g. `var x = 1` has the same type restrictions on `x` as `x = 1`. The difference is in intentional shadowing: if you have `x = 1` in outer scope and `x = 2` in inner scope (e.g. within a for-loop or an if-statement) then outer-scope `x` has value 2 after the second assignment. But if you have `var x = 2` in the inner scope, then you are declaring a variable scoped to the inner block.) For example:
<pre>
<pre class="pre-non-highlight">
x = 1;
if (NR == 4) {
x = 2; # Refers to outer-scope x: value changes from 1 to 2.
@ -636,7 +688,7 @@ if (NR == 4) {
print x; # Value of x is now two
</pre>
<pre>
<pre class="pre-non-highlight">
x = 1;
if (NR == 4) {
var x = 2; # Defines a new inner-scope x with value 2
@ -646,7 +698,7 @@ print x; # Value of this x is still 1
Likewise function arguments can optionally be typed, with type enforced when the function is called:
<pre>
<pre class="pre-non-highlight">
func f(map m, int i) {
...
}
@ -659,9 +711,9 @@ if (NR == 4) {
print x; # Value of this x is still 1
</pre>
Thirdly, function return values can be type-checked at the point of ``return`` using ``:`` and a typedecl after the parameter list:
Thirdly, function return values can be type-checked at the point of `return` using `:` and a typedecl after the parameter list:
<pre>
<pre class="pre-non-highlight">
func f(map m, int i): bool {
...
...
@ -688,7 +740,7 @@ Please see [xxxx](reference-main-null-data.md).
## Aggregate variable assignments
There are three remaining kinds of variable assignment using out-of-stream variables, the last two of which use the ``$*`` syntax:
There are three remaining kinds of variable assignment using out-of-stream variables, the last two of which use the `$*` syntax:
* Recursive copy of out-of-stream variables
* Out-of-stream variable assigned to full stream record
@ -696,8 +748,10 @@ There are three remaining kinds of variable assignment using out-of-stream varia
Example recursive copy of out-of-stream variables:
<pre>
<pre class="pre-highlight">
<b>mlr --opprint put -q '@v["sum"] += $x; @v["count"] += 1; end{dump; @w = @v; dump}' data/small</b>
</pre>
<pre class="pre-non-highlight">
{
"v": {
"sum": 2.264761728567491,
@ -718,8 +772,10 @@ Example recursive copy of out-of-stream variables:
Example of out-of-stream variable assigned to full stream record, where the 2nd record is stashed, and the 4th record is overwritten with that:
<pre>
<pre class="pre-highlight">
<b>mlr put 'NR == 2 {@keep = $*}; NR == 4 {$* = @keep}' data/small</b>
</pre>
<pre class="pre-non-highlight">
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
@ -727,10 +783,12 @@ a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
</pre>
Example of full stream record assigned to an out-of-stream variable, finding the record for which the ``x`` field has the largest value in the input stream:
Example of full stream record assigned to an out-of-stream variable, finding the record for which the `x` field has the largest value in the input stream:
<pre>
<pre class="pre-highlight">
<b>cat data/small</b>
</pre>
<pre class="pre-non-highlight">
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
@ -738,19 +796,23 @@ a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr --opprint put -q '</b>
<b> is_null(@xmax) || $x > @xmax {@xmax=$x; @recmax=$*};</b>
<b> end {emit @recmax}</b>
<b>' data/small</b>
</pre>
<pre class="pre-non-highlight">
a b i x y
eks pan 2 0.7586799647899636 0.5221511083334797
</pre>
## Keywords for filter and put
<pre>
<pre class="pre-highlight">
<b>mlr help usage-keywords</b>
</pre>
<pre class="pre-non-highlight">
all: used in "emit", "emitp", and "unset" as a synonym for @*
begin: defines a block of statements to be executed before input records

102
docs6b/docs/reference-dsl-variables.md.in
View file

@ -2,11 +2,11 @@
Miller has the following kinds of variables:
**Built-in variables** such as ``NF``, ``NF``, ``FILENAME``, ``M_PI``, and ``M_E``. These are all capital letters and are read-only (although some of them change value from one record to another).
**Built-in variables** such as `NF`, `NF`, `FILENAME`, `M_PI`, and `M_E`. These are all capital letters and are read-only (although some of them change value from one record to another).
**Fields of stream records**, accessed using the ``$`` prefix. These refer to fields of the current data-stream record. For example, in ``echo x=1,y=2 | mlr put '$z = $x + $y'``, ``$x`` and ``$y`` refer to input fields, and ``$z`` refers to a new, computed output field. In a few contexts, presented below, you can refer to the entire record as ``$*``.
**Fields of stream records**, accessed using the `$` prefix. These refer to fields of the current data-stream record. For example, in `echo x=1,y=2 | mlr put '$z = $x + $y'`, `$x` and `$y` refer to input fields, and `$z` refers to a new, computed output field. In a few contexts, presented below, you can refer to the entire record as `$*`.
**Out-of-stream variables** accessed using the ``@`` prefix. These refer to data which persist from one record to the next, including in ``begin`` and ``end`` blocks (which execute before/after the record stream is consumed, respectively). You use them to remember values across records, such as sums, differences, counters, and so on. In a few contexts, presented below, you can refer to the entire out-of-stream-variables collection as ``@*``.
**Out-of-stream variables** accessed using the `@` prefix. These refer to data which persist from one record to the next, including in `begin` and `end` blocks (which execute before/after the record stream is consumed, respectively). You use them to remember values across records, such as sums, differences, counters, and so on. In a few contexts, presented below, you can refer to the entire out-of-stream-variables collection as `@*`.
**Local variables** are limited in scope and extent to the current statements being executed: these include function arguments, bound variables in for loops, and explicitly declared local variables.
@ -14,9 +14,9 @@ Miller has the following kinds of variables:
## Built-in variables
These are written all in capital letters, such as ``NR``, ``NF``, ``FILENAME``, and only a small, specific set of them is defined by Miller.
These are written all in capital letters, such as `NR`, `NF`, `FILENAME`, and only a small, specific set of them is defined by Miller.
Namely, Miller supports the following five built-in variables for :doc:`filter and put <reference-dsl>`, all ``awk``-inspired: ``NF``, ``NR``, ``FNR``, ``FILENUM``, and ``FILENAME``, as well as the mathematical constants ``M_PI`` and ``M_E``. Lastly, the ``ENV`` hashmap allows read access to environment variables, e.g. ``ENV["HOME"]`` or ``ENV["foo_".$hostname]``.
Namely, Miller supports the following five built-in variables for [filter and put](reference-dsl.md), all `awk`-inspired: `NF`, `NR`, `FNR`, `FILENUM`, and `FILENAME`, as well as the mathematical constants `M_PI` and `M_E`. Lastly, the `ENV` hashmap allows read access to environment variables, e.g. `ENV["HOME"]` or `ENV["foo_".$hostname]`.
GENMD_RUN_COMMAND
mlr filter 'FNR == 2' data/small*
@ -26,9 +26,9 @@ GENMD_RUN_COMMAND
mlr put '$fnr = FNR' data/small*
GENMD_EOF
Their values of ``NF``, ``NR``, ``FNR``, ``FILENUM``, and ``FILENAME`` change from one record to the next as Miller scans through your input data stream. The mathematical constants, of course, do not change; ``ENV`` is populated from the system environment variables at the time Miller starts and is read-only for the remainder of program execution.
Their values of `NF`, `NR`, `FNR`, `FILENUM`, and `FILENAME` change from one record to the next as Miller scans through your input data stream. The mathematical constants, of course, do not change; `ENV` is populated from the system environment variables at the time Miller starts and is read-only for the remainder of program execution.
Their **scope is global**: you can refer to them in any ``filter`` or ``put`` statement. Their values are assigned by the input-record reader:
Their **scope is global**: you can refer to them in any `filter` or `put` statement. Their values are assigned by the input-record reader:
GENMD_RUN_COMMAND
mlr --csv put '$nr = NR' data/a.csv
@ -38,15 +38,15 @@ GENMD_RUN_COMMAND
mlr --csv repeat -n 3 then put '$nr = NR' data/a.csv
GENMD_EOF
The **extent** is for the duration of the put/filter: in a ``begin`` statement (which executes before the fimd.input record is consumed) you will find ``NR=1`` and in an ``end`` statement (which is executed after the last input record is consumed) you will find ``NR`` to be the total number of records ingested.
The **extent** is for the duration of the put/filter: in a `begin` statement (which executes before the fimd.input record is consumed) you will find `NR=1` and in an `end` statement (which is executed after the last input record is consumed) you will find `NR` to be the total number of records ingested.
These are all **read-only** for the ``mlr put`` and ``mlr filter`` DSLs: they may be assigned from, e.g. ``$nr=NR``, but they may not be assigned to: ``NR=100`` is a syntax error.
These are all **read-only** for the `mlr put` and `mlr filter` DSLs: they may be assigned from, e.g. `$nr=NR`, but they may not be assigned to: `NR=100` is a syntax error.
## Field names
Names of fields within stream records must be specified using a ``$`` in :doc:`filter and put expressions <reference-dsl>`, even though the dollar signs don't appear in the data stream itself. For integer-indexed data, this looks like ``awk``'s ``$1,$2,$3``, except that Miller allows non-numeric names such as ``$quantity`` or ``$hostname``. Likewise, enclose string literals in double quotes in ``filter`` expressions even though they don't appear in file data. In particular, ``mlr filter '$x=="abc"'`` passes through the record ``x=abc``.
Names of fields within stream records must be specified using a `$` in [filter and put expressions](reference-dsl.md), even though the dollar signs don't appear in the data stream itself. For integer-indexed data, this looks like `awk`'s `$1,$2,$3`, except that Miller allows non-numeric names such as `$quantity` or `$hostname`. Likewise, enclose string literals in double quotes in `filter` expressions even though they don't appear in file data. In particular, `mlr filter '$x=="abc"'` passes through the record `x=abc`.
If field names have **special characters** such as ``.`` then you can use braces, e.g. ``'${field.name}'``.
If field names have **special characters** such as `.` then you can use braces, e.g. `'${field.name}'`.
You may also use a **computed field name** in square brackets, e.g.
@ -62,19 +62,19 @@ Notes:
The names of record fields depend on the contents of your input data stream, and their values change from one record to the next as Miller scans through your input data stream.
Their **extent** is limited to the current record; their **scope** is the ``filter`` or ``put`` command in which they appear.
Their **extent** is limited to the current record; their **scope** is the `filter` or `put` command in which they appear.
These are **read-write**: you can do ``$y=2*$x``, ``$x=$x+1``, etc.
These are **read-write**: you can do `$y=2*$x`, `$x=$x+1`, etc.
Records are Miller's output: field names present in the input stream are passed through to output (written to standard output) unless fields are removed with ``cut``, or records are excluded with ``filter`` or ``put -q``, etc. Simply assign a value to a field and it will be output.
Records are Miller's output: field names present in the input stream are passed through to output (written to standard output) unless fields are removed with `cut`, or records are excluded with `filter` or `put -q`, etc. Simply assign a value to a field and it will be output.
## Positional field names
Even though Miller's main selling point is name-indexing, sometimes you really want to refer to a field name by its positional index (starting from 1).
Use ``$[[3]]`` to access the name of field 3. More generally, any expression evaluating to an integer can go between ``$[[`` and ``]]``.
Use `$[[3]]` to access the name of field 3. More generally, any expression evaluating to an integer can go between `$[[` and `]]`.
Then using a computed field name, ``$[ $[[3]] ]`` is the value in the third field. This has the shorter equivalent notation ``$[[[3]]]``.
Then using a computed field name, `$[ $[[3]] ]` is the value in the third field. This has the shorter equivalent notation `$[[[3]]]`.
GENMD_RUN_COMMAND
mlr cat data/small
@ -100,7 +100,7 @@ GENMD_RUN_COMMAND
mlr put '$[[[NR]]] = "NEW"' data/small
GENMD_EOF
Right-hand side accesses to non-existent fields -- i.e. with index less than 1 or greater than ``NF`` -- return an absent value. Likewise, left-hand side accesses only refer to fields which already exist. For example, if a field has 5 records then assigning the name or value of the 6th (or 600th) field results in a no-op.
Right-hand side accesses to non-existent fields -- i.e. with index less than 1 or greater than `NF` -- return an absent value. Likewise, left-hand side accesses only refer to fields which already exist. For example, if a field has 5 records then assigning the name or value of the 6th (or 600th) field results in a no-op.
GENMD_RUN_COMMAND
mlr put '$[[6]] = "NEW"' data/small
@ -112,11 +112,11 @@ GENMD_EOF
## Out-of-stream variables
These are prefixed with an at-sign, e.g. ``@sum``. Furthermore, unlike built-in variables and stream-record fields, they are maintained in an arbitrarily nested hashmap: you can do ``@sum += $quanity``, or ``@sum[$color] += $quanity``, or ``@sum[$color][$shape] += $quanity``. The keys for the multi-level hashmap can be any expression which evaluates to string or integer: e.g. ``@sum[NR] = $a + $b``, ``@sum[$a."-".$b] = $x``, etc.
These are prefixed with an at-sign, e.g. `@sum`. Furthermore, unlike built-in variables and stream-record fields, they are maintained in an arbitrarily nested hashmap: you can do `@sum += $quanity`, or `@sum[$color] += $quanity`, or `@sum[$color][$shape] += $quanity`. The keys for the multi-level hashmap can be any expression which evaluates to string or integer: e.g. `@sum[NR] = $a + $b`, `@sum[$a."-".$b] = $x`, etc.
Their names and their values are entirely under your control; they change only when you assign to them.
Just as for field names in stream records, if you want to define out-of-stream variables with **special characters** such as ``.`` then you can use braces, e.g. ``'@{variable.name}["index"]'``.
Just as for field names in stream records, if you want to define out-of-stream variables with **special characters** such as `.` then you can use braces, e.g. `'@{variable.name}["index"]'`.
You may use a **computed key** in square brackets, e.g.
@ -124,7 +124,7 @@ GENMD_RUN_COMMAND
echo s=green,t=blue,a=3,b=4 | mlr put -q '@[$s."_".$t] = $a * $b; emit all'
GENMD_EOF
Out-of-stream variables are **scoped** to the ``put`` command in which they appear. In particular, if you have two or more ``put`` commands separated by ``then``, each put will have its own set of out-of-stream variables:
Out-of-stream variables are **scoped** to the `put` command in which they appear. In particular, if you have two or more `put` commands separated by `then`, each put will have its own set of out-of-stream variables:
GENMD_RUN_COMMAND
cat data/a.dkvp
@ -136,13 +136,13 @@ mlr put '@sum += $a; end {emit @sum}' \
data/a.dkvp
GENMD_EOF
Out-of-stream variables' **extent** is from the start to the end of the record stream, i.e. every time the ``put`` or ``filter`` statement referring to them is executed.
Out-of-stream variables' **extent** is from the start to the end of the record stream, i.e. every time the `put` or `filter` statement referring to them is executed.
Out-of-stream variables are **read-write**: you can do ``$sum=@sum``, ``@sum=$sum``, etc.
Out-of-stream variables are **read-write**: you can do `$sum=@sum`, `@sum=$sum`, etc.
## Indexed out-of-stream variables
Using an index on the ``@count`` and ``@sum`` variables, we get the benefit of the ``-g`` (group-by) option which ``mlr stats1`` and various other Miller commands have:
Using an index on the `@count` and `@sum` variables, we get the benefit of the `-g` (group-by) option which `mlr stats1` and various other Miller commands have:
GENMD_INCLUDE_AND_RUN_ESCAPED(data/begin-end-example-6.sh)
@ -152,17 +152,15 @@ Indices can be arbitrarily deep -- here there are two or more of them:
GENMD_INCLUDE_AND_RUN_ESCAPED(data/begin-end-example-6a.sh)
The idea is that ``stats1``, and other Miller verbs, encapsulate frequently-used patterns with a minimum of keystroking (and run a little faster), whereas using out-of-stream variables you have more flexibility and control in what you do.
The idea is that `stats1`, and other Miller verbs, encapsulate frequently-used patterns with a minimum of keystroking (and run a little faster), whereas using out-of-stream variables you have more flexibility and control in what you do.
Begin/end blocks can be mixed with pattern/action blocks. For example:
GENMD_INCLUDE_AND_RUN_ESCAPED(data/begin-end-example-8.sh)
.. _reference-dsl-local-variables:
## Local variables
Local variables are similar to out-of-stream variables, except that their extent is limited to the expressions in which they appear (and their basenames can't be computed using square brackets). There are three kinds of local variables: **arguments** to functions/subroutines, **variables bound within for-loops**, and **locals** defined within control blocks. They may be untyped using ``var``, or typed using ``num``, ``int``, ``float``, ``str``, ``bool``, and ``map``.
Local variables are similar to out-of-stream variables, except that their extent is limited to the expressions in which they appear (and their basenames can't be computed using square brackets). There are three kinds of local variables: **arguments** to functions/subroutines, **variables bound within for-loops**, and **locals** defined within control blocks. They may be untyped using `var`, or typed using `num`, `int`, `float`, `str`, `bool`, and `map`.
For example:
@ -170,27 +168,27 @@ GENMD_INCLUDE_AND_RUN_ESCAPED(data/local-example-1.sh)
Things which are completely unsurprising, resembling many other languages:
* Parameter names are bound to their arguments but can be reassigned, e.g. if there is a parameter named ``a`` then you can reassign the value of ``a`` to be something else within the function if you like.
* Parameter names are bound to their arguments but can be reassigned, e.g. if there is a parameter named `a` then you can reassign the value of `a` to be something else within the function if you like.
* However, you cannot redeclare the *type* of an argument or a local: ``var a=1; var a=2`` is an error but ``var a=1; a=2`` is OK.
* However, you cannot redeclare the *type* of an argument or a local: `var a=1; var a=2` is an error but `var a=1; a=2` is OK.
* All argument-passing is positional rather than by name; arguments are passed by value, not by reference. (This is also true for map-valued variables: they are not, and cannot be, passed by reference)
* You can define locals (using ``var``, ``num``, etc.) at any scope (if-statements, else-statements, while-loops, for-loops, or the top-level scope), and nested scopes will have access (more details on scope in the next section). If you define a local variable with the same name inside an inner scope, then a new variable is created with the narrower scope.
* You can define locals (using `var`, `num`, etc.) at any scope (if-statements, else-statements, while-loops, for-loops, or the top-level scope), and nested scopes will have access (more details on scope in the next section). If you define a local variable with the same name inside an inner scope, then a new variable is created with the narrower scope.
* If you assign to a local variable for the first time in a scope without declaring it as ``var``, ``num``, etc. then: if it exists in an outer scope, that outer-scope variable will be updated; if not, it will be defined in the current scope as if ``var`` had been used. (See also :ref:`reference-dsl-type-checking` for an example.) I recommend always declaring variables explicitly to make the intended scoping clear.
* If you assign to a local variable for the first time in a scope without declaring it as `var`, `num`, etc. then: if it exists in an outer scope, that outer-scope variable will be updated; if not, it will be defined in the current scope as if `var` had been used. (See also [Type-checking](reference-dsl-variables.md#type-checking) for an example.) I recommend always declaring variables explicitly to make the intended scoping clear.
* Functions and subroutines never have access to locals from their callee (unless passed by value as arguments).
Things which are perhaps surprising compared to other languages:
* Type declarations using ``var``, or typed using ``num``, ``int``, ``float``, ``str``, and ``bool`` are necessary to declare local variables. Function arguments and variables bound in for-loops over stream records and out-of-stream variables are *implicitly* declared using ``var``. (Some examples are shown below.)
* Type declarations using `var`, or typed using `num`, `int`, `float`, `str`, and `bool` are necessary to declare local variables. Function arguments and variables bound in for-loops over stream records and out-of-stream variables are *implicitly* declared using `var`. (Some examples are shown below.)
* Type-checking is done at assignment time. For example, ``float f = 0`` is an error (since ``0`` is an integer), as is ``float f = 0.0; f = 1``. For this reason I prefer to use ``num`` over ``float`` in most contexts since ``num`` encompasses integer and floating-point values. More information about type-checking is at :ref:`reference-dsl-type-checking`.
* Type-checking is done at assignment time. For example, `float f = 0` is an error (since `0` is an integer), as is `float f = 0.0; f = 1`. For this reason I prefer to use `num` over `float` in most contexts since `num` encompasses integer and floating-point values. More information is at [Type-checking](reference-dsl-variables.md#type-checking).
* Bound variables in for-loops over stream records and out-of-stream variables are implicitly local to that block. E.g. in ``for (k, v in $*) { ... }`` ``for ((k1, k2), v in @*) { ... }`` if there are ``k``, ``v``, etc. in the enclosing scope then those will be masked by the loop-local bound variables in the loop, and moreover the values of the loop-local bound variables are not available after the end of the loop.
* Bound variables in for-loops over stream records and out-of-stream variables are implicitly local to that block. E.g. in `for (k, v in $*) { ... }` `for ((k1, k2), v in @*) { ... }` if there are `k`, `v`, etc. in the enclosing scope then those will be masked by the loop-local bound variables in the loop, and moreover the values of the loop-local bound variables are not available after the end of the loop.
* For C-style triple-for loops, if a for-loop variable is defined using ``var``, ``int``, etc. then it is scoped to that for-loop. E.g. ``for (i = 0; i < 10; i += 1) { ... }`` and ``for (int i = 0; i < 10; i += 1) { ... }``. (This is unsurprising.). If there is no typedecl and an outer-scope variable of that name exists, then it is used. (This is also unsurprising.) But of there is no outer-scope variable of that name then the variable is scoped to the for-loop only.
* For C-style triple-for loops, if a for-loop variable is defined using `var`, `int`, etc. then it is scoped to that for-loop. E.g. `for (i = 0; i < 10; i += 1) { ... }` and `for (int i = 0; i < 10; i += 1) { ... }`. (This is unsurprising.). If there is no typedecl and an outer-scope variable of that name exists, then it is used. (This is also unsurprising.) But of there is no outer-scope variable of that name then the variable is scoped to the for-loop only.
The following example demonstrates the scope rules:
@ -214,9 +212,9 @@ GENMD_EOF
## Map literals
Miller's ``put``/``filter`` DSL has four kinds of hashmaps. **Stream records** are (single-level) maps from name to value. **Out-of-stream variables** and **local variables** can also be maps, although they can be multi-level hashmaps (e.g. ``@sum[$x][$y]``). The fourth kind is **map literals**. These cannot be on the left-hand side of assignment expressions. Syntactically they look like JSON, although Miller allows string and integer keys in its map literals while JSON allows only string keys (e.g. ``"3"`` rather than ``3``).
Miller's `put`/`filter` DSL has four kinds of hashmaps. **Stream records** are (single-level) maps from name to value. **Out-of-stream variables** and **local variables** can also be maps, although they can be multi-level hashmaps (e.g. `@sum[$x][$y]`). The fourth kind is **map literals**. These cannot be on the left-hand side of assignment expressions. Syntactically they look like JSON, although Miller allows string and integer keys in its map literals while JSON allows only string keys (e.g. `"3"` rather than `3`).
For example, the following swaps the input stream's ``a`` and ``i`` fields, modifies ``y``, and drops the rest:
For example, the following swaps the input stream's `a` and `i` fields, modifies `y`, and drops the rest:
GENMD_INCLUDE_AND_RUN_ESCAPED(data/map-literal-example-1.sh)
@ -228,22 +226,17 @@ Like out-of-stream and local variables, map literals can be multi-level:
GENMD_INCLUDE_AND_RUN_ESCAPED(data/map-literal-example-3.sh)
By default, map-valued expressions are dumped using JSON formatting. If you use ``dump`` to print a hashmap with integer keys and you don't want them double-quoted (JSON-style) then you can use ``mlr put --jknquoteint``. See also ``mlr put --help``.
.. _reference-dsl-type-checking:
By default, map-valued expressions are dumped using JSON formatting. If you use `dump` to print a hashmap with integer keys and you don't want them double-quoted (JSON-style) then you can use `mlr put --jknquoteint`. See also `mlr put --help`.
## Type-checking
Miller's ``put``/``filter`` DSLs support two optional kinds of type-checking. One is inline **type-tests** and **type-assertions** within expressions. The other is **type declarations** for assignments to local variables, binding of arguments to user-defined functions, and return values from user-defined functions, These are discussed in the following subsections.
Miller's `put`/`filter` DSLs support two optional kinds of type-checking. One is inline **type-tests** and **type-assertions** within expressions. The other is **type declarations** for assignments to local variables, binding of arguments to user-defined functions, and return values from user-defined functions, These are discussed in the following subsections.
Use of type-checking is entirely up to you: omit it if you want flexibility with heterogeneous data; use it if you want to help catch misspellings in your DSL code or unexpected irregularities in your input data.
.. _reference-dsl-type-tests-and-assertions:
### Type-test and type-assertion expressions
Type-test and type-assertion expressions
................................................................
The following ``is...`` functions take a value and return a boolean indicating whether the argument is of the indicated type. The ``assert_...`` functions return their argument if it is of the specified type, and cause a fatal error otherwise:
The following `is...` functions take a value and return a boolean indicating whether the argument is of the indicated type. The `assert_...` functions return their argument if it is of the specified type, and cause a fatal error otherwise:
GENMD_RUN_COMMAND
mlr -f | grep ^is
@ -255,12 +248,11 @@ GENMD_EOF
See [Data-cleaning Examples](data-cleaning-examples.md) for examples of how to use these.
Type-declarations for local variables, function parameter, and function return values
...............................................................................................
### Type-declarations for local variables, function parameter, and function return values
Local variables can be defined either untyped as in ``x = 1``, or typed as in ``int x = 1``. Types include **var** (explicitly untyped), **int**, **float**, **num** (int or float), **str**, **bool**, and **map**. These optional type declarations are enforced at the time values are assigned to variables: whether at the initial value assignment as in ``int x = 1`` or in any subsequent assignments to the same variable farther down in the scope.
Local variables can be defined either untyped as in `x = 1`, or typed as in `int x = 1`. Types include **var** (explicitly untyped), **int**, **float**, **num** (int or float), **str**, **bool**, and **map**. These optional type declarations are enforced at the time values are assigned to variables: whether at the initial value assignment as in `int x = 1` or in any subsequent assignments to the same variable farther down in the scope.
The reason for ``num`` is that ``int`` and ``float`` typedecls are very precise:
The reason for `num` is that `int` and `float` typedecls are very precise:
GENMD_CARDIFY
float a = 0; # Runtime error since 0 is int not float
@ -269,9 +261,9 @@ num c = 0; # OK
num d = 1.0; # OK
GENMD_EOF
A suggestion is to use ``num`` for general use when you want numeric content, and use ``int`` when you genuinely want integer-only values, e.g. in loop indices or map keys (since Miller map keys can only be strings or ints).
A suggestion is to use `num` for general use when you want numeric content, and use `int` when you genuinely want integer-only values, e.g. in loop indices or map keys (since Miller map keys can only be strings or ints).
The ``var`` type declaration indicates no type restrictions, e.g. ``var x = 1`` has the same type restrictions on ``x`` as ``x = 1``. The difference is in intentional shadowing: if you have ``x = 1`` in outer scope and ``x = 2`` in inner scope (e.g. within a for-loop or an if-statement) then outer-scope ``x`` has value 2 after the second assignment. But if you have ``var x = 2`` in the inner scope, then you are declaring a variable scoped to the inner block.) For example:
The `var` type declaration indicates no type restrictions, e.g. `var x = 1` has the same type restrictions on `x` as `x = 1`. The difference is in intentional shadowing: if you have `x = 1` in outer scope and `x = 2` in inner scope (e.g. within a for-loop or an if-statement) then outer-scope `x` has value 2 after the second assignment. But if you have `var x = 2` in the inner scope, then you are declaring a variable scoped to the inner block.) For example:
GENMD_CARDIFY
x = 1;
@ -304,7 +296,7 @@ if (NR == 4) {
print x; # Value of this x is still 1
GENMD_EOF
Thirdly, function return values can be type-checked at the point of ``return`` using ``:`` and a typedecl after the parameter list:
Thirdly, function return values can be type-checked at the point of `return` using `:` and a typedecl after the parameter list:
GENMD_CARDIFY
func f(map m, int i): bool {
@ -333,7 +325,7 @@ Please see [xxxx](reference-main-null-data.md).
## Aggregate variable assignments
There are three remaining kinds of variable assignment using out-of-stream variables, the last two of which use the ``$*`` syntax:
There are three remaining kinds of variable assignment using out-of-stream variables, the last two of which use the `$*` syntax:
* Recursive copy of out-of-stream variables
* Out-of-stream variable assigned to full stream record
@ -351,7 +343,7 @@ GENMD_RUN_COMMAND
mlr put 'NR == 2 {@keep = $*}; NR == 4 {$* = @keep}' data/small
GENMD_EOF
Example of full stream record assigned to an out-of-stream variable, finding the record for which the ``x`` field has the largest value in the input stream:
Example of full stream record assigned to an out-of-stream variable, finding the record for which the `x` field has the largest value in the input stream:
GENMD_RUN_COMMAND
cat data/small

36
docs6b/docs/reference-dsl.md
View file

@ -3,12 +3,14 @@
## Overview
Here's comparison of verbs and ``put``/``filter`` DSL expressions:
Here's comparison of verbs and `put`/`filter` DSL expressions:
Example:
<pre>
<pre class="pre-highlight">
<b>mlr stats1 -a sum -f x -g a data/small</b>
</pre>
<pre class="pre-non-highlight">
a=pan,x_sum=0.3467901443380824
a=eks,x_sum=1.1400793586611044
a=wye,x_sum=0.7778922255683036
@ -22,8 +24,10 @@ a=wye,x_sum=0.7778922255683036
Example:
<pre>
<pre class="pre-highlight">
<b>mlr put -q '@x_sum[$a] += $x; end{emit @x_sum, "a"}' data/small</b>
</pre>
<pre class="pre-non-highlight">
a=pan,x_sum=0.3467901443380824
a=eks,x_sum=1.1400793586611044
a=wye,x_sum=0.7778922255683036
@ -35,12 +39,14 @@ a=wye,x_sum=0.7778922255683036
* There is more to learn
* They are highly customizable
Please see [Verbs Reference](reference-verbs.md) for information on verbs other than ``put`` and ``filter``.
Please see [Verbs Reference](reference-verbs.md) for information on verbs other than `put` and `filter`.
The essential usages of ``mlr filter`` and ``mlr put`` are for record-selection and record-updating expressions, respectively. For example, given the following input data:
The essential usages of `mlr filter` and `mlr put` are for record-selection and record-updating expressions, respectively. For example, given the following input data:
<pre>
<pre class="pre-highlight">
<b>cat data/small</b>
</pre>
<pre class="pre-non-highlight">
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
@ -48,18 +54,22 @@ a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
</pre>
you might retain only the records whose ``a`` field has value ``eks``:
you might retain only the records whose `a` field has value `eks`:
<pre>
<pre class="pre-highlight">
<b>mlr filter '$a == "eks"' data/small</b>
</pre>
<pre class="pre-non-highlight">
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463
</pre>
or you might add a new field which is a function of existing fields:
<pre>
<pre class="pre-highlight">
<b>mlr put '$ab = $a . "_" . $b ' data/small</b>
</pre>
<pre class="pre-non-highlight">
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533,ab=pan_pan
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797,ab=eks_pan
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776,ab=wye_wye
@ -67,13 +77,13 @@ a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463,ab=eks_wye
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729,ab=wye_pan
</pre>
The two verbs ``mlr filter`` and ``mlr put`` are essentially the same. The only differences are:
The two verbs `mlr filter` and `mlr put` are essentially the same. The only differences are:
* Expressions sent to ``mlr filter`` must end with a boolean expression, which is the filtering criterion;
* Expressions sent to `mlr filter` must end with a boolean expression, which is the filtering criterion;
* ``mlr filter`` expressions may not reference the ``filter`` keyword within them; and
* `mlr filter` expressions may not reference the `filter` keyword within them; and
* ``mlr filter`` expressions may not use ``tee``, ``emit``, ``emitp``, or ``emitf``.
* `mlr filter` expressions may not use `tee`, `emit`, `emitp`, or `emitf`.
All the rest is the same: in particular, you can define and invoke functions and subroutines to help produce the final boolean statement, and record fields may be assigned to in the statements preceding the final boolean statement.

16
docs6b/docs/reference-dsl.md.in
View file

@ -2,7 +2,7 @@
## Overview
Here's comparison of verbs and ``put``/``filter`` DSL expressions:
Here's comparison of verbs and `put`/`filter` DSL expressions:
Example:
@ -28,15 +28,15 @@ GENMD_EOF
* There is more to learn
* They are highly customizable
Please see [Verbs Reference](reference-verbs.md) for information on verbs other than ``put`` and ``filter``.
Please see [Verbs Reference](reference-verbs.md) for information on verbs other than `put` and `filter`.
The essential usages of ``mlr filter`` and ``mlr put`` are for record-selection and record-updating expressions, respectively. For example, given the following input data:
The essential usages of `mlr filter` and `mlr put` are for record-selection and record-updating expressions, respectively. For example, given the following input data:
GENMD_RUN_COMMAND
cat data/small
GENMD_EOF
you might retain only the records whose ``a`` field has value ``eks``:
you might retain only the records whose `a` field has value `eks`:
GENMD_RUN_COMMAND
mlr filter '$a == "eks"' data/small
@ -48,13 +48,13 @@ GENMD_RUN_COMMAND
mlr put '$ab = $a . "_" . $b ' data/small
GENMD_EOF
The two verbs ``mlr filter`` and ``mlr put`` are essentially the same. The only differences are:
The two verbs `mlr filter` and `mlr put` are essentially the same. The only differences are:
* Expressions sent to ``mlr filter`` must end with a boolean expression, which is the filtering criterion;
* Expressions sent to `mlr filter` must end with a boolean expression, which is the filtering criterion;
* ``mlr filter`` expressions may not reference the ``filter`` keyword within them; and
* `mlr filter` expressions may not reference the `filter` keyword within them; and
* ``mlr filter`` expressions may not use ``tee``, ``emit``, ``emitp``, or ``emitf``.
* `mlr filter` expressions may not use `tee`, `emit`, `emitp`, or `emitf`.
All the rest is the same: in particular, you can define and invoke functions and subroutines to help produce the final boolean statement, and record fields may be assigned to in the statements preceding the final boolean statement.

14
docs6b/docs/reference-main-arithmetic.md
View file

@ -3,13 +3,13 @@
## Input scanning
Numbers in Miller are double-precision float or 64-bit signed integers. Anything scannable as int, e.g ``123`` or ``0xabcd``, is treated as an integer; otherwise, input scannable as float (``4.56`` or ``8e9``) is treated as float; everything else is a string.
Numbers in Miller are double-precision float or 64-bit signed integers. Anything scannable as int, e.g `123` or `0xabcd`, is treated as an integer; otherwise, input scannable as float (`4.56` or `8e9`) is treated as float; everything else is a string.
If you want all numbers to be treated as floats, then you may use ``float()`` in your filter/put expressions (e.g. replacing ``$c = $a * $b`` with ``$c = float($a) * float($b)``) -- or, more simply, use ``mlr filter -F`` and ``mlr put -F`` which forces all numeric input, whether from expression literals or field values, to float. Likewise ``mlr stats1 -F`` and ``mlr step -F`` force integerable accumulators (such as ``count``) to be done in floating-point.
If you want all numbers to be treated as floats, then you may use `float()` in your filter/put expressions (e.g. replacing `$c = $a * $b` with `$c = float($a) * float($b)`) -- or, more simply, use `mlr filter -F` and `mlr put -F` which forces all numeric input, whether from expression literals or field values, to float. Likewise `mlr stats1 -F` and `mlr step -F` force integerable accumulators (such as `count`) to be done in floating-point.
## Conversion by math routines
For most math functions, integers are cast to float on input, and produce float output: e.g. ``exp(0) = 1.0`` rather than ``1``. The following, however, produce integer output if their inputs are integers: ``+`` ``-`` ``*`` ``/`` ``//`` ``%`` ``abs`` ``ceil`` ``floor`` ``max`` ``min`` ``round`` ``roundm`` ``sgn``. As well, ``stats1 -a min``, ``stats1 -a max``, ``stats1 -a sum``, ``step -a delta``, and ``step -a rsum`` produce integer output if their inputs are integers.
For most math functions, integers are cast to float on input, and produce float output: e.g. `exp(0) = 1.0` rather than `1`. The following, however, produce integer output if their inputs are integers: `+` `-` `*` `/` `//` `%` `abs` `ceil` `floor` `max` `min` `round` `roundm` `sgn`. As well, `stats1 -a min`, `stats1 -a max`, `stats1 -a sum`, `step -a delta`, and `step -a rsum` produce integer output if their inputs are integers.
## Conversion by arithmetic operators
@ -19,9 +19,9 @@ The short of it is that Miller does this transparently for you so you needn't th
Implementation details of this, for the interested: integer adds and subtracts overflow by at most one bit so it suffices to check sign-changes. Thus, Miller allows you to add and subtract arbitrary 64-bit signed integers, converting only to float precisely when the result is less than -2\ :sup:`63` or greater than 2\ :sup:`63`\ -1. Multiplies, on the other hand, can overflow by a word size and a sign-change technique does not suffice to detect overflow. Instead Miller tests whether the floating-point product exceeds the representable integer range. Now, 64-bit integers have 64-bit precision while IEEE-doubles have only 52-bit mantissas -- so, there are 53 bits including implicit leading one. The following experiment explicitly demonstrates the resolution at this range:
<pre>
<pre class="pre-non-highlight">
64-bit integer 64-bit integer Casted to double Back to 64-bit
in hex in decimal integer
in hex in decimal integer
0x7ffffffffffff9ff 9223372036854774271 9223372036854773760.000000 0x7ffffffffffff800
0x7ffffffffffffa00 9223372036854774272 9223372036854773760.000000 0x7ffffffffffff800
0x7ffffffffffffbff 9223372036854774783 9223372036854774784.000000 0x7ffffffffffffc00
@ -38,6 +38,6 @@ That is, one cannot check an integer product to see if it is precisely greater t
Division and remainder are [pythonic](http://python-history.blogspot.com/2010/08/why-pythons-integer-division-floors.html):
* Quotient of integers is floating-point: ``7/2`` is ``3.5``.
* Integer division is done with ``//``: ``7//2`` is ``3``. This rounds toward the negative.
* Quotient of integers is floating-point: `7/2` is `3.5`.
* Integer division is done with `//`: `7//2` is `3`. This rounds toward the negative.
* Remainders are non-negative.

12
docs6b/docs/reference-main-arithmetic.md.in
View file

@ -2,13 +2,13 @@
## Input scanning
Numbers in Miller are double-precision float or 64-bit signed integers. Anything scannable as int, e.g ``123`` or ``0xabcd``, is treated as an integer; otherwise, input scannable as float (``4.56`` or ``8e9``) is treated as float; everything else is a string.
Numbers in Miller are double-precision float or 64-bit signed integers. Anything scannable as int, e.g `123` or `0xabcd`, is treated as an integer; otherwise, input scannable as float (`4.56` or `8e9`) is treated as float; everything else is a string.
If you want all numbers to be treated as floats, then you may use ``float()`` in your filter/put expressions (e.g. replacing ``$c = $a * $b`` with ``$c = float($a) * float($b)``) -- or, more simply, use ``mlr filter -F`` and ``mlr put -F`` which forces all numeric input, whether from expression literals or field values, to float. Likewise ``mlr stats1 -F`` and ``mlr step -F`` force integerable accumulators (such as ``count``) to be done in floating-point.
If you want all numbers to be treated as floats, then you may use `float()` in your filter/put expressions (e.g. replacing `$c = $a * $b` with `$c = float($a) * float($b)`) -- or, more simply, use `mlr filter -F` and `mlr put -F` which forces all numeric input, whether from expression literals or field values, to float. Likewise `mlr stats1 -F` and `mlr step -F` force integerable accumulators (such as `count`) to be done in floating-point.
## Conversion by math routines
For most math functions, integers are cast to float on input, and produce float output: e.g. ``exp(0) = 1.0`` rather than ``1``. The following, however, produce integer output if their inputs are integers: ``+`` ``-`` ``*`` ``/`` ``//`` ``%`` ``abs`` ``ceil`` ``floor`` ``max`` ``min`` ``round`` ``roundm`` ``sgn``. As well, ``stats1 -a min``, ``stats1 -a max``, ``stats1 -a sum``, ``step -a delta``, and ``step -a rsum`` produce integer output if their inputs are integers.
For most math functions, integers are cast to float on input, and produce float output: e.g. `exp(0) = 1.0` rather than `1`. The following, however, produce integer output if their inputs are integers: `+` `-` `*` `/` `//` `%` `abs` `ceil` `floor` `max` `min` `round` `roundm` `sgn`. As well, `stats1 -a min`, `stats1 -a max`, `stats1 -a sum`, `step -a delta`, and `step -a rsum` produce integer output if their inputs are integers.
## Conversion by arithmetic operators
@ -20,7 +20,7 @@ Implementation details of this, for the interested: integer adds and subtracts o
GENMD_CARDIFY
64-bit integer 64-bit integer Casted to double Back to 64-bit
in hex in decimal integer
in hex in decimal integer
0x7ffffffffffff9ff 9223372036854774271 9223372036854773760.000000 0x7ffffffffffff800
0x7ffffffffffffa00 9223372036854774272 9223372036854773760.000000 0x7ffffffffffff800
0x7ffffffffffffbff 9223372036854774783 9223372036854774784.000000 0x7ffffffffffffc00
@ -37,6 +37,6 @@ That is, one cannot check an integer product to see if it is precisely greater t
Division and remainder are [pythonic](http://python-history.blogspot.com/2010/08/why-pythons-integer-division-floors.html):
* Quotient of integers is floating-point: ``7/2`` is ``3.5``.
* Integer division is done with ``//``: ``7//2`` is ``3``. This rounds toward the negative.
* Quotient of integers is floating-point: `7/2` is `3.5`.
* Integer division is done with `//`: `7//2` is `3`. This rounds toward the negative.
* Remainders are non-negative.

40
docs6b/docs/reference-main-auxiliary-commands.md
View file

@ -3,8 +3,10 @@
There are a few nearly-standalone programs which have nothing to do with the rest of Miller, do not participate in record streams, and do not deal with file formats. They might as well be little standalone executables but they're delivered within the main Miller executable for convenience.
<pre>
<pre class="pre-highlight">
<b>mlr aux-list</b>
</pre>
<pre class="pre-non-highlight">
Available subcommands:
aux-list
hex
@ -17,8 +19,10 @@ Available subcommands:
For more information, please invoke mlr {subcommand} --help.
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr lecat --help</b>
</pre>
<pre class="pre-non-highlight">
Usage: mlr lecat [options] {zero or more file names}
Simply echoes input, but flags CR characters in red and LF characters in green.
If zero file names are supplied, standard input is read.
@ -27,8 +31,10 @@ Options:
-h or --help: print this message
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr termcvt --help</b>
</pre>
<pre class="pre-non-highlight">
Usage: mlr termcvt [option] {zero or more file names}
Option (exactly one is required):
--cr2crlf
@ -43,8 +49,10 @@ Zero file names means read from standard input.
Output is always to standard output; files are not written in-place.
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr hex --help</b>
</pre>
<pre class="pre-non-highlight">
Usage: mlr hex [options] {zero or more file names}
Simple hex-dump.
If zero file names are supplied, standard input is read.
@ -53,8 +61,10 @@ Options:
-h or --help: print this message
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr unhex --help</b>
</pre>
<pre class="pre-non-highlight">
Usage: mlr unhex [option] {zero or more file names}
Options:
-h or --help: print this message
@ -64,18 +74,24 @@ Output is always to standard output; files are not written in-place.
Examples:
<pre>
<pre class="pre-highlight">
<b>echo 'Hello, world!' | mlr lecat --mono</b>
</pre>
<pre class="pre-non-highlight">
Hello, world![LF]
</pre>
<pre>
<pre class="pre-highlight">
<b>echo 'Hello, world!' | mlr termcvt --lf2crlf | mlr lecat --mono</b>
</pre>
<pre class="pre-non-highlight">
Hello, world![CR][LF]
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr hex data/budget.csv</b>
</pre>
<pre class="pre-non-highlight">
00000000: 23 20 41 73 61 6e 61 20 2d 2d 20 68 65 72 65 20 |# Asana -- here |
00000010: 61 72 65 20 74 68 65 20 62 75 64 67 65 74 20 66 |are the budget f|
00000020: 69 67 75 72 65 73 20 79 6f 75 20 61 73 6b 65 64 |igures you asked|
@ -85,8 +101,10 @@ Hello, world![CR][LF]
00000060: 72 61 6e 67 65 2c 31 32 33 2e 34 35 0a |range,123.45.|
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr hex -r data/budget.csv</b>
</pre>
<pre class="pre-non-highlight">
23 20 41 73 61 6e 61 20 2d 2d 20 68 65 72 65 20
61 72 65 20 74 68 65 20 62 75 64 67 65 74 20 66
69 67 75 72 65 73 20 79 6f 75 20 61 73 6b 65 64
@ -96,8 +114,10 @@ Hello, world![CR][LF]
72 61 6e 67 65 2c 31 32 33 2e 34 35 0a
</pre>
<pre>
<pre class="pre-highlight">
<b>mlr hex -r data/budget.csv | sed 's/20/2a/g' | mlr unhex</b>
</pre>
<pre class="pre-non-highlight">
#*Asana*--*here*are*the*budget*figures*you*asked*for!
type,quantity
purple,456.78

46
docs6b/docs/reference-main-data-types.md
View file

@ -5,41 +5,41 @@ Miller's input and output are all string-oriented: there is (as of August 2015 a
Field values are treated as numeric for the following:
* Numeric sort: ``mlr sort -n``, ``mlr sort -nr``.
* Statistics: ``mlr histogram``, ``mlr stats1``, ``mlr stats2``.
* Cross-record arithmetic: ``mlr step``.
* Numeric sort: `mlr sort -n`, `mlr sort -nr`.
* Statistics: `mlr histogram`, `mlr stats1`, `mlr stats2`.
* Cross-record arithmetic: `mlr step`.
For ``mlr put`` and ``mlr filter``:
For `mlr put` and `mlr filter`:
* Miller's types for function processing are **empty-null** (empty string), **absent-null** (reads of unset right-hand sides, or fall-through non-explicit return values from user-defined functions), **error**, **string**, **float** (double-precision), **int** (64-bit signed), and **boolean**.
* On input, string values representable as numbers, e.g. "3" or "3.1", are treated as int or float, respectively. If a record has ``x=1,y=2`` then ``mlr put '$z=$x+$y'`` will produce ``x=1,y=2,z=3``, and ``mlr put '$z=$x.$y'`` does not give an error simply because the dot operator has been generalized to stringify non-strings. To coerce back to string for processing, use the ``string`` function: ``mlr put '$z=string($x).string($y)'`` will produce ``x=1,y=2,z=12``.
* On input, string values representable as numbers, e.g. "3" or "3.1", are treated as int or float, respectively. If a record has `x=1,y=2` then `mlr put '$z=$x+$y'` will produce `x=1,y=2,z=3`, and `mlr put '$z=$x.$y'` does not give an error simply because the dot operator has been generalized to stringify non-strings. To coerce back to string for processing, use the `string` function: `mlr put '$z=string($x).string($y)'` will produce `x=1,y=2,z=12`.
* On input, string values representable as boolean (e.g. ``"true"``, ``"false"``) are *not* automatically treated as boolean. (This is because ``"true"`` and ``"false"`` are ordinary words, and auto string-to-boolean on a column consisting of words would result in some strings mixed with some booleans.) Use the ``boolean`` function to coerce: e.g. giving the record ``x=1,y=2,w=false`` to ``mlr put '$z=($x<$y) || boolean($w)'``.
* On input, string values representable as boolean (e.g. `"true"`, `"false"`) are *not* automatically treated as boolean. (This is because `"true"` and `"false"` are ordinary words, and auto string-to-boolean on a column consisting of words would result in some strings mixed with some booleans.) Use the `boolean` function to coerce: e.g. giving the record `x=1,y=2,w=false` to `mlr put '$z=($x<$y) || boolean($w)'`.
* Functions take types as described in ``mlr --help-all-functions``: for example, ``log10`` takes float input and produces float output, ``gmt2sec`` maps string to int, and ``sec2gmt`` maps int to string.
* Functions take types as described in `mlr --help-all-functions`: for example, `log10` takes float input and produces float output, `gmt2sec` maps string to int, and `sec2gmt` maps int to string.
* All math functions described in ``mlr --help-all-functions`` take integer as well as float input.
* All math functions described in `mlr --help-all-functions` take integer as well as float input.
## String literals
You can use the following backslash escapes for strings such as between the double quotes in contexts such as ``mlr filter '$name =~ "..."'``, ``mlr put '$name = $othername . "..."'``, ``mlr put '$name = sub($name, "...", "...")``, etc.:
You can use the following backslash escapes for strings such as between the double quotes in contexts such as `mlr filter '$name =~ "..."'`, `mlr put '$name = $othername . "..."'`, `mlr put '$name = sub($name, "...", "...")`, etc.:
* ``\a``: ASCII code 0x07 (alarm/bell)
* ``\b``: ASCII code 0x08 (backspace)
* ``\f``: ASCII code 0x0c (formfeed)
* ``\n``: ASCII code 0x0a (LF/linefeed/newline)
* ``\r``: ASCII code 0x0d (CR/carriage return)
* ``\t``: ASCII code 0x09 (tab)
* ``\v``: ASCII code 0x0b (vertical tab)
* ``\\``: backslash
* ``\"``: double quote
* ``\123``: Octal 123, etc. for ``\000`` up to ``\377``
* ``\x7f``: Hexadecimal 7f, etc. for ``\x00`` up to ``\xff``
* `\a`: ASCII code 0x07 (alarm/bell)
* `\b`: ASCII code 0x08 (backspace)
* `\f`: ASCII code 0x0c (formfeed)
* `\n`: ASCII code 0x0a (LF/linefeed/newline)
* `\r`: ASCII code 0x0d (CR/carriage return)
* `\t`: ASCII code 0x09 (tab)
* `\v`: ASCII code 0x0b (vertical tab)
* `\\`: backslash
* `\"`: double quote
* `\123`: Octal 123, etc. for `\000` up to `\377`
* `\x7f`: Hexadecimal 7f, etc. for `\x00` up to `\xff`
See also https://en.wikipedia.org/wiki/Escape_sequences_in_C.
See also [https://en.wikipedia.org/wiki/Escape_sequences_in_C](https://en.wikipedia.org/wiki/Escape_sequences_in_C).
These replacements apply only to strings you key in for the DSL expressions for ``filter`` and ``put``: that is, if you type ``\t`` in a string literal for a ``filter``/``put`` expression, it will be turned into a tab character. If you want a backslash followed by a ``t``, then please type ``\\t``.
These replacements apply only to strings you key in for the DSL expressions for `filter` and `put`: that is, if you type `\t` in a string literal for a `filter`/`put` expression, it will be turned into a tab character. If you want a backslash followed by a `t`, then please type `\\t`.
However, these replacements are not done automatically within your data stream. If you wish to make these replacements, you can do, for example, for a field named ``field``, ``mlr put '$field = gsub($field, "\\t", "\t")'``. If you need to make such a replacement for all fields in your data, you should probably simply use the system ``sed`` command.
However, these replacements are not done automatically within your data stream. If you wish to make these replacements, you can do, for example, for a field named `field`, `mlr put '$field = gsub($field, "\\t", "\t")'`. If you need to make such a replacement for all fields in your data, you should probably simply use the system `sed` command.

46
docs6b/docs/reference-main-data-types.md.in
View file

@ -4,41 +4,41 @@ Miller's input and output are all string-oriented: there is (as of August 2015 a
Field values are treated as numeric for the following:
* Numeric sort: ``mlr sort -n``, ``mlr sort -nr``.
* Statistics: ``mlr histogram``, ``mlr stats1``, ``mlr stats2``.
* Cross-record arithmetic: ``mlr step``.
* Numeric sort: `mlr sort -n`, `mlr sort -nr`.
* Statistics: `mlr histogram`, `mlr stats1`, `mlr stats2`.
* Cross-record arithmetic: `mlr step`.
For ``mlr put`` and ``mlr filter``:
For `mlr put` and `mlr filter`:
* Miller's types for function processing are **empty-null** (empty string), **absent-null** (reads of unset right-hand sides, or fall-through non-explicit return values from user-defined functions), **error**, **string**, **float** (double-precision), **int** (64-bit signed), and **boolean**.
* On input, string values representable as numbers, e.g. "3" or "3.1", are treated as int or float, respectively. If a record has ``x=1,y=2`` then ``mlr put '$z=$x+$y'`` will produce ``x=1,y=2,z=3``, and ``mlr put '$z=$x.$y'`` does not give an error simply because the dot operator has been generalized to stringify non-strings. To coerce back to string for processing, use the ``string`` function: ``mlr put '$z=string($x).string($y)'`` will produce ``x=1,y=2,z=12``.
* On input, string values representable as numbers, e.g. "3" or "3.1", are treated as int or float, respectively. If a record has `x=1,y=2` then `mlr put '$z=$x+$y'` will produce `x=1,y=2,z=3`, and `mlr put '$z=$x.$y'` does not give an error simply because the dot operator has been generalized to stringify non-strings. To coerce back to string for processing, use the `string` function: `mlr put '$z=string($x).string($y)'` will produce `x=1,y=2,z=12`.
* On input, string values representable as boolean (e.g. ``"true"``, ``"false"``) are *not* automatically treated as boolean. (This is because ``"true"`` and ``"false"`` are ordinary words, and auto string-to-boolean on a column consisting of words would result in some strings mixed with some booleans.) Use the ``boolean`` function to coerce: e.g. giving the record ``x=1,y=2,w=false`` to ``mlr put '$z=($x<$y) || boolean($w)'``.
* On input, string values representable as boolean (e.g. `"true"`, `"false"`) are *not* automatically treated as boolean. (This is because `"true"` and `"false"` are ordinary words, and auto string-to-boolean on a column consisting of words would result in some strings mixed with some booleans.) Use the `boolean` function to coerce: e.g. giving the record `x=1,y=2,w=false` to `mlr put '$z=($x<$y) || boolean($w)'`.
* Functions take types as described in ``mlr --help-all-functions``: for example, ``log10`` takes float input and produces float output, ``gmt2sec`` maps string to int, and ``sec2gmt`` maps int to string.
* Functions take types as described in `mlr --help-all-functions`: for example, `log10` takes float input and produces float output, `gmt2sec` maps string to int, and `sec2gmt` maps int to string.
* All math functions described in ``mlr --help-all-functions`` take integer as well as float input.
* All math functions described in `mlr --help-all-functions` take integer as well as float input.
## String literals
You can use the following backslash escapes for strings such as between the double quotes in contexts such as ``mlr filter '$name =~ "..."'``, ``mlr put '$name = $othername . "..."'``, ``mlr put '$name = sub($name, "...", "...")``, etc.:
You can use the following backslash escapes for strings such as between the double quotes in contexts such as `mlr filter '$name =~ "..."'`, `mlr put '$name = $othername . "..."'`, `mlr put '$name = sub($name, "...", "...")`, etc.:
* ``\a``: ASCII code 0x07 (alarm/bell)
* ``\b``: ASCII code 0x08 (backspace)
* ``\f``: ASCII code 0x0c (formfeed)
* ``\n``: ASCII code 0x0a (LF/linefeed/newline)
* ``\r``: ASCII code 0x0d (CR/carriage return)
* ``\t``: ASCII code 0x09 (tab)
* ``\v``: ASCII code 0x0b (vertical tab)
* ``\\``: backslash
* ``\"``: double quote
* ``\123``: Octal 123, etc. for ``\000`` up to ``\377``
* ``\x7f``: Hexadecimal 7f, etc. for ``\x00`` up to ``\xff``
* `\a`: ASCII code 0x07 (alarm/bell)
* `\b`: ASCII code 0x08 (backspace)
* `\f`: ASCII code 0x0c (formfeed)
* `\n`: ASCII code 0x0a (LF/linefeed/newline)
* `\r`: ASCII code 0x0d (CR/carriage return)
* `\t`: ASCII code 0x09 (tab)
* `\v`: ASCII code 0x0b (vertical tab)
* `\\`: backslash
* `\"`: double quote
* `\123`: Octal 123, etc. for `\000` up to `\377`
* `\x7f`: Hexadecimal 7f, etc. for `\x00` up to `\xff`
See also https://en.wikipedia.org/wiki/Escape_sequences_in_C.
See also [https://en.wikipedia.org/wiki/Escape_sequences_in_C](https://en.wikipedia.org/wiki/Escape_sequences_in_C).
These replacements apply only to strings you key in for the DSL expressions for ``filter`` and ``put``: that is, if you type ``\t`` in a string literal for a ``filter``/``put`` expression, it will be turned into a tab character. If you want a backslash followed by a ``t``, then please type ``\\t``.
These replacements apply only to strings you key in for the DSL expressions for `filter` and `put`: that is, if you type `\t` in a string literal for a `filter`/`put` expression, it will be turned into a tab character. If you want a backslash followed by a `t`, then please type `\\t`.
However, these replacements are not done automatically within your data stream. If you wish to make these replacements, you can do, for example, for a field named ``field``, ``mlr put '$field = gsub($field, "\\t", "\t")'``. If you need to make such a replacement for all fields in your data, you should probably simply use the system ``sed`` command.
However, these replacements are not done automatically within your data stream. If you wish to make these replacements, you can do, for example, for a field named `field`, `mlr put '$field = gsub($field, "\\t", "\t")'`. If you need to make such a replacement for all fields in your data, you should probably simply use the system `sed` command.

6
docs6b/docs/reference-main-env-vars.md
View file

@ -3,7 +3,7 @@
The following environment variables affect how Miller works:
* ``MLRRC``: see [Customization](customization.md)
* ``MLR_NO_COLOR``, ``MLR_ALWAYS_COLOR``, ``MLR_KEY_COLOR``, ``MLR_VALUE_COLOR``, ``MLR_PASS_COLOR``, ``MLR_FAIL_COLOR``, ``MLR_REPL_PS1_COLOR``, ``MLR_REPL_PS2_COLOR``, ``MLR_HELP_COLOR``, ``TERM``, * ``MSYSTEM``: see [Output Colorization](output-colorization.md)
* ``MLR_REPL_PS1``, ``MLR_REPL_PS2``: see [REPL](repl.md)
* `MLRRC`: see [Customization](customization.md).
* `MLR_NO_COLOR`, `MLR_ALWAYS_COLOR`, `MLR_KEY_COLOR`, `MLR_VALUE_COLOR`, `MLR_PASS_COLOR`, `MLR_FAIL_COLOR`, `MLR_REPL_PS1_COLOR`, `MLR_REPL_PS2_COLOR`, `MLR_HELP_COLOR`, `TERM`, * `MSYSTEM`: see [Output Colorization](output-colorization.md).
* `MLR_REPL_PS1`, `MLR_REPL_PS2`: see [REPL](repl.md).

Some files were not shown because too many files have changed in this diff Show more