mirror of
https://github.com/johnkerl/miller.git
synced 2026-07-25 17:04:01 +00:00
Misc doc-mods for Miller 6 (#575)
* Ongoing docs6 reorg * Misc: --foo=bar regtest, MLR_* unsetenv @ regtest
This commit is contained in:
parent
bc024993c1
commit
a731230d8a
31 changed files with 526 additions and 347 deletions
|
|
@ -211,7 +211,9 @@ code {
|
|||
font-family: 'Bitstream Vera Sans Mono', 'Monaco', monospace;
|
||||
color: black;
|
||||
padding: 1px 2px 1px 2px;
|
||||
background-color: #fafafa;
|
||||
/* background-color: #e9e2cd; */ /* CHANGE */
|
||||
/* background-color: #c3b694; */
|
||||
background-color: #e8e8e8;
|
||||
border-bottom: 1px solid #eee;
|
||||
}
|
||||
|
||||
|
|
|
|||
25
docs6/contributing.rst
Normal file
25
docs6/contributing.rst
Normal file
|
|
@ -0,0 +1,25 @@
|
|||
..
|
||||
PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE.
|
||||
|
||||
How to contribute
|
||||
================================================================
|
||||
|
||||
Community
|
||||
----------------------------------------------------------------
|
||||
|
||||
TODO
|
||||
|
||||
Documentation improvements
|
||||
----------------------------------------------------------------
|
||||
|
||||
TODO
|
||||
|
||||
Testing
|
||||
----------------------------------------------------------------
|
||||
|
||||
TODO
|
||||
|
||||
Feature development
|
||||
----------------------------------------------------------------
|
||||
|
||||
TODO
|
||||
22
docs6/contributing.rst.in
Normal file
22
docs6/contributing.rst.in
Normal file
|
|
@ -0,0 +1,22 @@
|
|||
How to contribute
|
||||
================================================================
|
||||
|
||||
Community
|
||||
----------------------------------------------------------------
|
||||
|
||||
TODO
|
||||
|
||||
Documentation improvements
|
||||
----------------------------------------------------------------
|
||||
|
||||
TODO
|
||||
|
||||
Testing
|
||||
----------------------------------------------------------------
|
||||
|
||||
TODO
|
||||
|
||||
Feature development
|
||||
----------------------------------------------------------------
|
||||
|
||||
TODO
|
||||
|
|
@ -79,20 +79,20 @@ Here is an example ``.mlrrc file``:
|
|||
Where to put your .mlrrc
|
||||
----------------------------------------------------------------
|
||||
|
||||
* If the environment variable ``MLRRC`` is set:
|
||||
If the environment variable ``MLRRC`` is set:
|
||||
|
||||
* If its value is ``__none__`` then no ``.mlrrc`` files are processed. (This is nice for things like regression testing.)
|
||||
* If its value is ``__none__`` then no ``.mlrrc`` files are processed. (This is nice for things like regression testing.)
|
||||
|
||||
* Otherwise, its value (as a filename) is loaded and processed. If there are syntax errors, they abort ``mlr`` with a usage message (as if you had mistyped something on the command line). If the file can't be loaded at all, though, it is silently skipped.
|
||||
* Otherwise, its value (as a filename) is loaded and processed. If there are syntax errors, they abort ``mlr`` with a usage message (as if you had mistyped something on the command line). If the file can't be loaded at all, though, it is silently skipped.
|
||||
|
||||
* Any ``.mlrrc`` in your home directory or current directory is ignored whenever ``MLRRC`` is set in the environment.
|
||||
* Any ``.mlrrc`` in your home directory or current directory is ignored whenever ``MLRRC`` is set in the environment.
|
||||
|
||||
* Example line in your shell's rc file: ``export MLRRC=/path/to/my/mlrrc``
|
||||
* Example line in your shell's rc file: ``export MLRRC=/path/to/my/mlrrc``
|
||||
|
||||
* Otherwise:
|
||||
Otherwise:
|
||||
|
||||
* If ``$HOME/.mlrrc`` exists, it's processed as above.
|
||||
* If ``$HOME/.mlrrc`` exists, it's processed as above.
|
||||
|
||||
* If ``./.mlrrc`` exists, it's then also processed as above.
|
||||
* If ``./.mlrrc`` exists, it's then also processed as above.
|
||||
|
||||
* The idea is you can have all your settings in your ``$HOME/.mlrrc``, then override maybe one or two for your current directory if you like.
|
||||
* The idea is you can have all your settings in your ``$HOME/.mlrrc``, then override maybe one or two for your current directory if you like.
|
||||
|
|
|
|||
|
|
@ -58,20 +58,20 @@ GENRST_INCLUDE_ESCAPED(sample_mlrrc)
|
|||
Where to put your .mlrrc
|
||||
----------------------------------------------------------------
|
||||
|
||||
* If the environment variable ``MLRRC`` is set:
|
||||
If the environment variable ``MLRRC`` is set:
|
||||
|
||||
* If its value is ``__none__`` then no ``.mlrrc`` files are processed. (This is nice for things like regression testing.)
|
||||
* If its value is ``__none__`` then no ``.mlrrc`` files are processed. (This is nice for things like regression testing.)
|
||||
|
||||
* Otherwise, its value (as a filename) is loaded and processed. If there are syntax errors, they abort ``mlr`` with a usage message (as if you had mistyped something on the command line). If the file can't be loaded at all, though, it is silently skipped.
|
||||
* Otherwise, its value (as a filename) is loaded and processed. If there are syntax errors, they abort ``mlr`` with a usage message (as if you had mistyped something on the command line). If the file can't be loaded at all, though, it is silently skipped.
|
||||
|
||||
* Any ``.mlrrc`` in your home directory or current directory is ignored whenever ``MLRRC`` is set in the environment.
|
||||
* Any ``.mlrrc`` in your home directory or current directory is ignored whenever ``MLRRC`` is set in the environment.
|
||||
|
||||
* Example line in your shell's rc file: ``export MLRRC=/path/to/my/mlrrc``
|
||||
* Example line in your shell's rc file: ``export MLRRC=/path/to/my/mlrrc``
|
||||
|
||||
* Otherwise:
|
||||
Otherwise:
|
||||
|
||||
* If ``$HOME/.mlrrc`` exists, it's processed as above.
|
||||
* If ``$HOME/.mlrrc`` exists, it's processed as above.
|
||||
|
||||
* If ``./.mlrrc`` exists, it's then also processed as above.
|
||||
* If ``./.mlrrc`` exists, it's then also processed as above.
|
||||
|
||||
* The idea is you can have all your settings in your ``$HOME/.mlrrc``, then override maybe one or two for your current directory if you like.
|
||||
* The idea is you can have all your settings in your ``$HOME/.mlrrc``, then override maybe one or two for your current directory if you like.
|
||||
|
|
|
|||
|
|
@ -26,7 +26,9 @@ Miller in more detail
|
|||
internationalization
|
||||
output-colorization
|
||||
customization
|
||||
miller-on-windows
|
||||
new-in-miller-6
|
||||
contributing
|
||||
|
||||
FAQs and recipes
|
||||
----------------------------------------------------------------
|
||||
|
|
@ -76,6 +78,7 @@ Reference
|
|||
:maxdepth: 1
|
||||
|
||||
reference-main-overview
|
||||
reference-verbs
|
||||
reference-main-io-options
|
||||
reference-main-then-chaining
|
||||
reference-main-auxiliary-commands
|
||||
|
|
@ -83,8 +86,8 @@ Reference
|
|||
reference-main-null-data
|
||||
reference-main-arithmetic
|
||||
reference-main-regular-expressions
|
||||
reference-main-env-vars
|
||||
reference-main-online-help
|
||||
reference-verbs
|
||||
reference-dsl
|
||||
reference-dsl-syntax
|
||||
reference-dsl-variables
|
||||
|
|
@ -105,5 +108,4 @@ Reference
|
|||
Index
|
||||
----------------------------------------------------------------
|
||||
|
||||
* :ref:`genindex`
|
||||
* :ref:`search`
|
||||
|
|
|
|||
14
docs6/miller-on-windows.rst
Normal file
14
docs6/miller-on-windows.rst
Normal file
|
|
@ -0,0 +1,14 @@
|
|||
..
|
||||
PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE.
|
||||
|
||||
Miller on Windows
|
||||
================================================================
|
||||
|
||||
TODO
|
||||
|
||||
* ``mlr.exe``, no ``msys2.dll``
|
||||
* easier to build yourself
|
||||
* where to get binaries
|
||||
* ``PATH`` seutp w/ examples
|
||||
* MSYS2 ``PATH`` seutp w/ examples
|
||||
* examples of the one gotcha: DSL @ CLI, and the ``put -f`` workaround
|
||||
11
docs6/miller-on-windows.rst.in
Normal file
11
docs6/miller-on-windows.rst.in
Normal file
|
|
@ -0,0 +1,11 @@
|
|||
Miller on Windows
|
||||
================================================================
|
||||
|
||||
TODO
|
||||
|
||||
* ``mlr.exe``, no ``msys2.dll``
|
||||
* easier to build yourself
|
||||
* where to get binaries
|
||||
* ``PATH`` seutp w/ examples
|
||||
* MSYS2 ``PATH`` seutp w/ examples
|
||||
* examples of the one gotcha: DSL @ CLI, and the ``put -f`` workaround
|
||||
|
|
@ -41,7 +41,7 @@ mlr -F | grep -v '^[a-zA-Z]' | uniq | while read funcname; do
|
|||
echo "$displayname"
|
||||
echo "^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^"
|
||||
echo ""
|
||||
echo '::'
|
||||
echo '.. code-block:: none'
|
||||
echo ''
|
||||
mlr --help-function "$funcname" | sed 's/^/ /'
|
||||
echo ''
|
||||
|
|
@ -83,7 +83,7 @@ mlr -F | grep '^[a-zA-Z]' | sort -u | while read funcname; do
|
|||
echo "$displayname"
|
||||
echo "^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^"
|
||||
echo ""
|
||||
echo '::'
|
||||
echo '.. code-block:: none'
|
||||
echo ''
|
||||
mlr --help-function "$funcname" | sed 's/^/ /'
|
||||
echo ''
|
||||
|
|
|
|||
|
|
@ -8,6 +8,9 @@ What's new in Miller 6
|
|||
|
||||
* Completely reworked documentation (here) and on-line help (``mlr --help``)
|
||||
* Arrays in the ``put``/``filter`` DSL
|
||||
|
||||
* also ``array`` is now a keyword so you can't use that as a local-variable or UDF name
|
||||
|
||||
* JSON:
|
||||
|
||||
* Improved JSON support
|
||||
|
|
@ -19,14 +22,22 @@ What's new in Miller 6
|
|||
|
||||
* Build artifacts (binaries) using GitHub Actions
|
||||
* In-process support for compressed input
|
||||
* Built-in ``md5`` ``sha1`` etc
|
||||
* ``hostname`` ``os`` ``version`` functions
|
||||
* Input-preservation -- find a way to describe this -- link to the issue ...
|
||||
* REPL TBD
|
||||
* :doc:`output-colorization`
|
||||
* Minor:
|
||||
|
||||
* Getoptish (#467)
|
||||
|
||||
* ``-xyz`` to ``-x -y -z`` -- show an example. And back-compat for ``sort``
|
||||
* ``--foo=bar`` to ``--foo bar`` -- show an example.
|
||||
|
||||
* ``--mfrom``, ``--load``, ``--mload``
|
||||
* Better syntax-error messages for the DSL, including line number
|
||||
* ``substr0`` and ``substr1``, and why
|
||||
* ``>>>`` and ``>>>=``
|
||||
* Completely reworked regression-testing
|
||||
|
||||
* Dev: ported to Go
|
||||
|
|
|
|||
|
|
@ -5,6 +5,9 @@ What's new in Miller 6
|
|||
|
||||
* Completely reworked documentation (here) and on-line help (``mlr --help``)
|
||||
* Arrays in the ``put``/``filter`` DSL
|
||||
|
||||
* also ``array`` is now a keyword so you can't use that as a local-variable or UDF name
|
||||
|
||||
* JSON:
|
||||
|
||||
* Improved JSON support
|
||||
|
|
@ -16,14 +19,22 @@ What's new in Miller 6
|
|||
|
||||
* Build artifacts (binaries) using GitHub Actions
|
||||
* In-process support for compressed input
|
||||
* Built-in ``md5`` ``sha1`` etc
|
||||
* ``hostname`` ``os`` ``version`` functions
|
||||
* Input-preservation -- find a way to describe this -- link to the issue ...
|
||||
* REPL TBD
|
||||
* :doc:`output-colorization`
|
||||
* Minor:
|
||||
|
||||
* Getoptish (#467)
|
||||
|
||||
* ``-xyz`` to ``-x -y -z`` -- show an example. And back-compat for ``sort``
|
||||
* ``--foo=bar`` to ``--foo bar`` -- show an example.
|
||||
|
||||
* ``--mfrom``, ``--load``, ``--mload``
|
||||
* Better syntax-error messages for the DSL, including line number
|
||||
* ``substr0`` and ``substr1``, and why
|
||||
* ``>>>`` and ``>>>=``
|
||||
* Completely reworked regression-testing
|
||||
|
||||
* Dev: ported to Go
|
||||
|
|
|
|||
Binary file not shown.
|
Before Width: | Height: | Size: 441 KiB After Width: | Height: | Size: 306 KiB |
BIN
docs6/pix/colorization.png.000
Normal file
BIN
docs6/pix/colorization.png.000
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 441 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 694 KiB After Width: | Height: | Size: 424 KiB |
BIN
docs6/pix/colorization2.png.000
Normal file
BIN
docs6/pix/colorization2.png.000
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 694 KiB |
File diff suppressed because it is too large
Load diff
21
docs6/reference-main-env-vars.rst
Normal file
21
docs6/reference-main-env-vars.rst
Normal file
|
|
@ -0,0 +1,21 @@
|
|||
..
|
||||
PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE.
|
||||
|
||||
Reference: Miller environment variables
|
||||
================================================================
|
||||
|
||||
TODO
|
||||
|
||||
* ``MLRRC``
|
||||
* ``HOME``
|
||||
* ``MLR_NO_COLOR``
|
||||
* ``MLR_ALWAYS_COLOR``
|
||||
* ``MLR_KEY_COLOR``
|
||||
* ``MLR_VALUE_COLOR``
|
||||
* ``MLR_PASS_COLOR``
|
||||
* ``MLR_FAIL_COLOR``
|
||||
* ``MLR_HELP_COLOR``
|
||||
* ``TERM``
|
||||
* ``MSYSTEM``
|
||||
* ``MLR_REPL_PS1``
|
||||
* ``MLR_REPL_PS2``
|
||||
18
docs6/reference-main-env-vars.rst.in
Normal file
18
docs6/reference-main-env-vars.rst.in
Normal file
|
|
@ -0,0 +1,18 @@
|
|||
Reference: Miller environment variables
|
||||
================================================================
|
||||
|
||||
TODO
|
||||
|
||||
* ``MLRRC``
|
||||
* ``HOME``
|
||||
* ``MLR_NO_COLOR``
|
||||
* ``MLR_ALWAYS_COLOR``
|
||||
* ``MLR_KEY_COLOR``
|
||||
* ``MLR_VALUE_COLOR``
|
||||
* ``MLR_PASS_COLOR``
|
||||
* ``MLR_FAIL_COLOR``
|
||||
* ``MLR_HELP_COLOR``
|
||||
* ``TERM``
|
||||
* ``MSYSTEM``
|
||||
* ``MLR_REPL_PS1``
|
||||
* ``MLR_REPL_PS2``
|
||||
|
|
@ -4,6 +4,9 @@
|
|||
Reference: Miller commands
|
||||
===============================
|
||||
|
||||
Overview
|
||||
----------------------------------------------------------------
|
||||
|
||||
TODO: push this into reference-verbs overview, and make this page an overview of miller invocations
|
||||
|
||||
Whereas the Unix toolkit is made of the separate executables ``cat``, ``tail``, ``cut``,
|
||||
|
|
@ -26,3 +29,45 @@ These fall into categories as follows:
|
|||
* Particularly oriented toward :doc:`record-heterogeneity`, although all Miller commands can handle heterogeneous records: :ref:`reference-verbs-group-by`, :ref:`reference-verbs-group-like`, :ref:`reference-verbs-having-fields`.
|
||||
|
||||
* These draw from other sources (see also :doc:`originality`): :ref:`reference-verbs-count-distinct` is SQL-ish, and :ref:`reference-verbs-rename` can be done by ``sed`` (which does it faster: see :doc:`performance`. Verbs: :ref:`reference-verbs-check`, :ref:`reference-verbs-count-distinct`, :ref:`reference-verbs-label`, :ref:`reference-verbs-merge-fields`, :ref:`reference-verbs-nest`, :ref:`reference-verbs-nothing`, :ref:`reference-verbs-regularize`, :ref:`reference-verbs-rename`, :ref:`reference-verbs-reorder`, :ref:`reference-verbs-reshape`, :ref:`reference-verbs-seqgen`.
|
||||
|
||||
|
||||
Verbs vs DSL
|
||||
----------------------------------------------------------------
|
||||
|
||||
When you type ``mlr {something} myfile.dat``, the ``{something}`` part is called a **verb**. It specifies how you want to transform your data. (See also :doc:`reference-main-overview` for a breakdown.) The following is an alphabetical list of verbs with their descriptions.
|
||||
|
||||
The verbs ``put`` and ``filter`` are special in that they have a rich expression language (domain-specific language, or "DSL"). More information about them can be found at :doc:`reference-dsl`.
|
||||
|
||||
Here's a comparison of verbs and ``put``/``filter`` DSL expressions:
|
||||
|
||||
Example:
|
||||
|
||||
.. code-block:: none
|
||||
:emphasize-lines: 1-1
|
||||
|
||||
$ mlr stats1 -a sum -f x -g a data/small
|
||||
a=pan,x_sum=0.3467901443380824
|
||||
a=eks,x_sum=1.1400793586611044
|
||||
a=wye,x_sum=0.7778922255683036
|
||||
|
||||
* Verbs are coded in Go
|
||||
* They run a bit faster
|
||||
* They take fewer keystrokes
|
||||
* There is less to learn
|
||||
* Their customization is limited to each verb's options
|
||||
|
||||
Example:
|
||||
|
||||
.. code-block:: none
|
||||
:emphasize-lines: 1-1
|
||||
|
||||
$ mlr put -q '@x_sum[$a] += $x; end{emit @x_sum, "a"}' data/small
|
||||
a=pan,x_sum=0.3467901443380824
|
||||
a=eks,x_sum=1.1400793586611044
|
||||
a=wye,x_sum=0.7778922255683036
|
||||
|
||||
* You get to write your own DSL expressions
|
||||
* They run a bit slower
|
||||
* They take more keystrokes
|
||||
* There is more to learn
|
||||
* They are highly customizable
|
||||
|
|
|
|||
|
|
@ -1,6 +1,9 @@
|
|||
Reference: Miller commands
|
||||
===============================
|
||||
|
||||
Overview
|
||||
----------------------------------------------------------------
|
||||
|
||||
TODO: push this into reference-verbs overview, and make this page an overview of miller invocations
|
||||
|
||||
Whereas the Unix toolkit is made of the separate executables ``cat``, ``tail``, ``cut``,
|
||||
|
|
@ -19,3 +22,37 @@ These fall into categories as follows:
|
|||
* Particularly oriented toward :doc:`record-heterogeneity`, although all Miller commands can handle heterogeneous records: :ref:`reference-verbs-group-by`, :ref:`reference-verbs-group-like`, :ref:`reference-verbs-having-fields`.
|
||||
|
||||
* These draw from other sources (see also :doc:`originality`): :ref:`reference-verbs-count-distinct` is SQL-ish, and :ref:`reference-verbs-rename` can be done by ``sed`` (which does it faster: see :doc:`performance`. Verbs: :ref:`reference-verbs-check`, :ref:`reference-verbs-count-distinct`, :ref:`reference-verbs-label`, :ref:`reference-verbs-merge-fields`, :ref:`reference-verbs-nest`, :ref:`reference-verbs-nothing`, :ref:`reference-verbs-regularize`, :ref:`reference-verbs-rename`, :ref:`reference-verbs-reorder`, :ref:`reference-verbs-reshape`, :ref:`reference-verbs-seqgen`.
|
||||
|
||||
|
||||
Verbs vs DSL
|
||||
----------------------------------------------------------------
|
||||
|
||||
When you type ``mlr {something} myfile.dat``, the ``{something}`` part is called a **verb**. It specifies how you want to transform your data. (See also :doc:`reference-main-overview` for a breakdown.) The following is an alphabetical list of verbs with their descriptions.
|
||||
|
||||
The verbs ``put`` and ``filter`` are special in that they have a rich expression language (domain-specific language, or "DSL"). More information about them can be found at :doc:`reference-dsl`.
|
||||
|
||||
Here's a comparison of verbs and ``put``/``filter`` DSL expressions:
|
||||
|
||||
Example:
|
||||
|
||||
GENRST_RUN_COMMAND
|
||||
mlr stats1 -a sum -f x -g a data/small
|
||||
GENRST_EOF
|
||||
|
||||
* Verbs are coded in Go
|
||||
* They run a bit faster
|
||||
* They take fewer keystrokes
|
||||
* There is less to learn
|
||||
* Their customization is limited to each verb's options
|
||||
|
||||
Example:
|
||||
|
||||
GENRST_RUN_COMMAND
|
||||
mlr put -q '@x_sum[$a] += $x; end{emit @x_sum, "a"}' data/small
|
||||
GENRST_EOF
|
||||
|
||||
* You get to write your own DSL expressions
|
||||
* They run a bit slower
|
||||
* They take more keystrokes
|
||||
* There is more to learn
|
||||
* They are highly customizable
|
||||
|
|
|
|||
|
|
@ -1,50 +1,9 @@
|
|||
..
|
||||
PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE.
|
||||
|
||||
Verbs reference
|
||||
Reference: List of verbs
|
||||
================================================================
|
||||
|
||||
Overview
|
||||
----------------------------------------------------------------
|
||||
|
||||
When you type ``mlr {something} myfile.dat``, the ``{something}`` part is called a **verb**. It specifies how you want to transform your data. (See also :doc:`reference-main-overview` for a breakdown.) The following is an alphabetical list of verbs with their descriptions.
|
||||
|
||||
The verbs ``put`` and ``filter`` are special in that they have a rich expression language (domain-specific language, or "DSL"). More information about them can be found at :doc:`reference-dsl`.
|
||||
|
||||
Here's a comparison of verbs and ``put``/``filter`` DSL expressions:
|
||||
|
||||
Example:
|
||||
|
||||
.. code-block:: none
|
||||
:emphasize-lines: 1-1
|
||||
|
||||
$ mlr stats1 -a sum -f x -g a data/small
|
||||
a=pan,x_sum=0.3467901443380824
|
||||
a=eks,x_sum=1.1400793586611044
|
||||
a=wye,x_sum=0.7778922255683036
|
||||
|
||||
* Verbs are coded in Go
|
||||
* They run a bit faster
|
||||
* They take fewer keystrokes
|
||||
* There is less to learn
|
||||
* Their customization is limited to each verb's options
|
||||
|
||||
Example:
|
||||
|
||||
.. code-block:: none
|
||||
:emphasize-lines: 1-1
|
||||
|
||||
$ mlr put -q '@x_sum[$a] += $x; end{emit @x_sum, "a"}' data/small
|
||||
a=pan,x_sum=0.3467901443380824
|
||||
a=eks,x_sum=1.1400793586611044
|
||||
a=wye,x_sum=0.7778922255683036
|
||||
|
||||
* You get to write your own DSL expressions
|
||||
* They run a bit slower
|
||||
* They take more keystrokes
|
||||
* There is more to learn
|
||||
* They are highly customizable
|
||||
|
||||
.. _reference-verbs-altkv:
|
||||
|
||||
altkv
|
||||
|
|
|
|||
|
|
@ -1,39 +1,6 @@
|
|||
Verbs reference
|
||||
Reference: List of verbs
|
||||
================================================================
|
||||
|
||||
Overview
|
||||
----------------------------------------------------------------
|
||||
|
||||
When you type ``mlr {something} myfile.dat``, the ``{something}`` part is called a **verb**. It specifies how you want to transform your data. (See also :doc:`reference-main-overview` for a breakdown.) The following is an alphabetical list of verbs with their descriptions.
|
||||
|
||||
The verbs ``put`` and ``filter`` are special in that they have a rich expression language (domain-specific language, or "DSL"). More information about them can be found at :doc:`reference-dsl`.
|
||||
|
||||
Here's a comparison of verbs and ``put``/``filter`` DSL expressions:
|
||||
|
||||
Example:
|
||||
|
||||
GENRST_RUN_COMMAND
|
||||
mlr stats1 -a sum -f x -g a data/small
|
||||
GENRST_EOF
|
||||
|
||||
* Verbs are coded in Go
|
||||
* They run a bit faster
|
||||
* They take fewer keystrokes
|
||||
* There is less to learn
|
||||
* Their customization is limited to each verb's options
|
||||
|
||||
Example:
|
||||
|
||||
GENRST_RUN_COMMAND
|
||||
mlr put -q '@x_sum[$a] += $x; end{emit @x_sum, "a"}' data/small
|
||||
GENRST_EOF
|
||||
|
||||
* You get to write your own DSL expressions
|
||||
* They run a bit slower
|
||||
* They take more keystrokes
|
||||
* There is more to learn
|
||||
* They are highly customizable
|
||||
|
||||
.. _reference-verbs-altkv:
|
||||
|
||||
altkv
|
||||
|
|
|
|||
1
go/regtest/cases/cli-getoptish/0001/cmd
Normal file
1
go/regtest/cases/cli-getoptish/0001/cmd
Normal file
|
|
@ -0,0 +1 @@
|
|||
mlr --ofs pipe cat regtest/input/abixy
|
||||
0
go/regtest/cases/cli-getoptish/0001/experr
Normal file
0
go/regtest/cases/cli-getoptish/0001/experr
Normal file
10
go/regtest/cases/cli-getoptish/0001/expout
Normal file
10
go/regtest/cases/cli-getoptish/0001/expout
Normal file
|
|
@ -0,0 +1,10 @@
|
|||
a=pan|b=pan|i=1|x=0.3467901443380824|y=0.7268028627434533
|
||||
a=eks|b=pan|i=2|x=0.7586799647899636|y=0.5221511083334797
|
||||
a=wye|b=wye|i=3|x=0.20460330576630303|y=0.33831852551664776
|
||||
a=eks|b=wye|i=4|x=0.38139939387114097|y=0.13418874328430463
|
||||
a=wye|b=pan|i=5|x=0.5732889198020006|y=0.8636244699032729
|
||||
a=zee|b=pan|i=6|x=0.5271261600918548|y=0.49322128674835697
|
||||
a=eks|b=zee|i=7|x=0.6117840605678454|y=0.1878849191181694
|
||||
a=zee|b=wye|i=8|x=0.5985540091064224|y=0.976181385699006
|
||||
a=hat|b=wye|i=9|x=0.03144187646093577|y=0.7495507603507059
|
||||
a=pan|b=wye|i=10|x=0.5026260055412137|y=0.9526183602969864
|
||||
1
go/regtest/cases/cli-getoptish/0002/cmd
Normal file
1
go/regtest/cases/cli-getoptish/0002/cmd
Normal file
|
|
@ -0,0 +1 @@
|
|||
mlr --ofs=pipe cat regtest/input/abixy
|
||||
0
go/regtest/cases/cli-getoptish/0002/experr
Normal file
0
go/regtest/cases/cli-getoptish/0002/experr
Normal file
10
go/regtest/cases/cli-getoptish/0002/expout
Normal file
10
go/regtest/cases/cli-getoptish/0002/expout
Normal file
|
|
@ -0,0 +1,10 @@
|
|||
a=pan|b=pan|i=1|x=0.3467901443380824|y=0.7268028627434533
|
||||
a=eks|b=pan|i=2|x=0.7586799647899636|y=0.5221511083334797
|
||||
a=wye|b=wye|i=3|x=0.20460330576630303|y=0.33831852551664776
|
||||
a=eks|b=wye|i=4|x=0.38139939387114097|y=0.13418874328430463
|
||||
a=wye|b=pan|i=5|x=0.5732889198020006|y=0.8636244699032729
|
||||
a=zee|b=pan|i=6|x=0.5271261600918548|y=0.49322128674835697
|
||||
a=eks|b=zee|i=7|x=0.6117840605678454|y=0.1878849191181694
|
||||
a=zee|b=wye|i=8|x=0.5985540091064224|y=0.976181385699006
|
||||
a=hat|b=wye|i=9|x=0.03144187646093577|y=0.7495507603507059
|
||||
a=pan|b=wye|i=10|x=0.5026260055412137|y=0.9526183602969864
|
||||
|
|
@ -28,6 +28,14 @@ const ShouldFailName = "should-fail"
|
|||
const MajorSeparator = "================================================================"
|
||||
const MinorSeparator = "----------------------------------------------------------------"
|
||||
|
||||
var envVarsToUnset = []string{
|
||||
"MLRRC",
|
||||
"MLR_KEY_COLOR",
|
||||
"MLR_VALUE_COLOR",
|
||||
"MLR_REPL_PS1",
|
||||
"MLR_REPL_PS2",
|
||||
}
|
||||
|
||||
type stringPair struct {
|
||||
first string
|
||||
second string
|
||||
|
|
@ -85,6 +93,11 @@ func (regtester *RegTester) Execute(
|
|||
paths []string,
|
||||
) bool {
|
||||
|
||||
// Don't let the current user's settings affect expected results
|
||||
for _, name := range envVarsToUnset {
|
||||
os.Unsetenv(name)
|
||||
}
|
||||
|
||||
regtester.resetCounts()
|
||||
|
||||
if len(paths) == 0 {
|
||||
|
|
|
|||
|
|
@ -2,6 +2,7 @@ package lib
|
|||
|
||||
import (
|
||||
"regexp"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Getoptify expands "-xyz" into "-x -y -z" while leaving "--xyz" intact. This
|
||||
|
|
@ -11,14 +12,22 @@ import (
|
|||
// verbs, and auxents) that multi-character options start with two dashes, e.g.
|
||||
// "--csv". (The sole exception is the sort verb's -nf/-nr which are handled
|
||||
// specially there.)
|
||||
//
|
||||
// Secondly, we split "--foo=bar" into "--foo" and "bar".
|
||||
func Getoptify(inargs []string) []string {
|
||||
regex := regexp.MustCompile("^-[a-zA-Z0-9]+$")
|
||||
expandRegex := regexp.MustCompile("^-[a-zA-Z0-9]+$")
|
||||
splitRegex := regexp.MustCompile("^--[^=]+=.+$")
|
||||
outargs := make([]string, 0)
|
||||
for _, inarg := range inargs {
|
||||
if regex.MatchString(inarg) {
|
||||
if expandRegex.MatchString(inarg) {
|
||||
for _, c := range inarg[1:] {
|
||||
outargs = append(outargs, "-"+string(c))
|
||||
}
|
||||
} else if splitRegex.MatchString(inarg) {
|
||||
pair := strings.SplitN(inarg, "=", 2)
|
||||
InternalCodingErrorIf(len(pair) != 2)
|
||||
outargs = append(outargs, pair[0])
|
||||
outargs = append(outargs, pair[1])
|
||||
} else {
|
||||
outargs = append(outargs, inarg)
|
||||
}
|
||||
|
|
|
|||
104
go/todo.txt
104
go/todo.txt
|
|
@ -1,14 +1,13 @@
|
|||
================================================================
|
||||
TOP OF LIST:
|
||||
|
||||
* sat:
|
||||
o convert all code-blocks to cardify
|
||||
o most/least-frequent: map -> odict
|
||||
|
||||
* emit * regex * fmtnum * localtime *
|
||||
* --nr-progress-mod * mlr -k *
|
||||
* cli-audits * doc6 * survey * check issues *
|
||||
* split up mlr -h
|
||||
|
||||
* convert all code-blocks to cardify
|
||||
* most/least-frequent: map -> odict
|
||||
* snk postntn
|
||||
|
||||
----------------------------------------------------------------
|
||||
|
|
@ -26,6 +25,50 @@ doclink etc:
|
|||
* data/small -> csv throughout. and/or just use example.csv
|
||||
----------------------------------------------------------------
|
||||
|
||||
----------------------------------------------------------------
|
||||
DOC6
|
||||
|
||||
! mlr --help, manpage, etc total refactor & audit
|
||||
|
||||
? quick-examples needs fewer inputs, and outputs for all retained inputs
|
||||
? or q-e more links to 10min
|
||||
|
||||
! fill out:
|
||||
o how-to-help guide
|
||||
o windows guide
|
||||
o repl guide
|
||||
o env-vars section
|
||||
|
||||
! thruout pitch the docs at primarily the first whenever possible:
|
||||
- python/ruby-level user w/o heavy sysadm
|
||||
- sysadm w/o dev
|
||||
> do not assume people know awk! XD
|
||||
- devs
|
||||
|
||||
* code to-do items:
|
||||
o stats1 --fr
|
||||
cookbook2.rst.in
|
||||
o mlr --print-type-arithmetic-info
|
||||
reference.rst.in
|
||||
o mlr --help-all-keywords
|
||||
reference-dsl.rst.in
|
||||
o mlr format-values --help
|
||||
reference-verbs.rst.in
|
||||
o mlr --usage-comments-in-data
|
||||
file-formats.rst.in
|
||||
o file-formats: colon -> dot
|
||||
o unset multiple lvalues in the BNF
|
||||
faq.rst.in
|
||||
o list at end of todo.txt
|
||||
o trailing space on pprint right
|
||||
o file-formats.rst.in & olh: dkvp not first anymore
|
||||
|
||||
* data-cleaning examples -> full section -- ?
|
||||
* c2 & c3 are all esoterica
|
||||
----------------------------------------------------------------
|
||||
|
||||
* case auxfiles: cat them too
|
||||
|
||||
----------------------------------------------------------------
|
||||
* mlr -k
|
||||
o c,go
|
||||
|
|
@ -47,59 +90,6 @@ w survey
|
|||
o head -n1 / tail -n1 -- special-case? but not head -n10 since getopts to head -n -1 -0 ...
|
||||
o --from=foo => split (at getopt-level) to --from foo
|
||||
|
||||
! how-to-help guide
|
||||
- testing
|
||||
- doc mods
|
||||
- dev
|
||||
- community
|
||||
> backport to docs5
|
||||
|
||||
----------------------------------------------------------------
|
||||
! get doc6 going sooner than later! lots accumulating ...
|
||||
o big:
|
||||
? quick-examples needs fewer inputs, and outputs for all retained inputs
|
||||
? or q-e more links to 10min
|
||||
! thruout pitch the docs at primarily the first whenever possible:
|
||||
- python/ruby-level user w/o heavy sysadm
|
||||
- sysadm w/o dev
|
||||
> do not assume people know awk! XD
|
||||
- devs
|
||||
- in that vein re-think the faq -- split up, and/or move down
|
||||
- likewise cookbook2/3 -- overly nerdy; perhaps just annoying
|
||||
- re-work data-sharing <-> SQL ...
|
||||
- env-vars section
|
||||
- windows section
|
||||
o little:
|
||||
- stats1 --fr
|
||||
cookbook2.rst.in
|
||||
- mlr --print-type-arithmetic-info
|
||||
reference.rst.in
|
||||
- mlr --help-all-keywords
|
||||
reference-dsl.rst.in
|
||||
- mlr format-values --help
|
||||
reference-verbs.rst.in
|
||||
- mlr --usage-comments-in-data
|
||||
file-formats.rst.in
|
||||
- file-formats: colon -> dot
|
||||
- unset multiple lvalues in the BNF
|
||||
faq.rst.in
|
||||
- list at end of todo.txt
|
||||
- maybe mlr -> mlrc as live-todo & 6make by default -- ?
|
||||
- trailing space on pprint right
|
||||
- sphinx more mobile-friendly somehow -- ?
|
||||
- file-formats.rst.in & olh: dkvp not first anymore
|
||||
o plan:
|
||||
- first landing page w/ toplinks -- ?
|
||||
- current FAQ -> esoterica (mostly ...) & distill new FAQ from GH issues/discussions
|
||||
- integrations section: log-processing, sql, mixing
|
||||
- cookbook 1/2/3 total teardown/rebuild:
|
||||
> data-cleaning examples -> full section -- ?
|
||||
> c2 & c3 are all esoterica
|
||||
! mlr --help, manpage, etc total refactor & audit
|
||||
----------------------------------------------------------------
|
||||
|
||||
* case auxfiles: cat them too
|
||||
|
||||
----------------------------------------------------------------
|
||||
|
||||
* blocker: regexes
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue