Mirrors/miller
1
0
Fork 0
mirror of https://github.com/johnkerl/miller.git synced 2026-07-28 10:13:59 +00:00
Code Issues Projects Releases Wiki Activity

Miller-6 docs iteration (#576)

Browse source 
* Windows screenshots
* Miller-on-Windows page
* Iterating
This commit is contained in:
John Kerl 2021-06-17 03:34:06 +00:00 • committed by GitHub
parent a731230d8a
commit 0e1b233bcc
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
42 changed files with 6842 additions and 221 deletions
Download patch file Download diff file Expand all files Collapse all files

BIN
docs/_static/top.png vendored
View file

Binary file not shown.
Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 5.8 KiB

After

Width:  |  Height:  |  Size: 17 KiB

Before After
Before After

13
docs/build-one Executable file
View file

@ -0,0 +1,13 @@
#!/bin/bash
set -euo pipefail
if [ $# -ge 1 ]; then
for name; do
if [[ $name == *.rst.in ]]; then
genrst $name;
fi
done
else
for rstin in *.rst.in; do genrst $rstin; done
fi
sphinx-build -M html . _build

2
docs/file-formats.rst
View file

@ -95,7 +95,7 @@ Miller has record separator ``RS`` and field separator ``FS``, just as ``awk`` d
**USV (Unicode-separated values):** likewise, the flags ``--usv``, ``--iusv``, ``--ousv``, ``--usvlite``, ``--iusvlite``, and ``--ousvlite`` use Unicode FS and RS U+241F (UTF-8 0x0xe2909f) and U+241E (UTF-8 0xe2909e), respectively.
Miller's ``--csv`` flag supports `RFC-4180 CSV <https://tools.ietf.org/html/rfc4180">`_. This includes CRLF line-terminators by default, regardless of platform.
Miller's ``--csv`` flag supports `RFC-4180 CSV <https://tools.ietf.org/html/rfc4180>`_. This includes CRLF line-terminators by default, regardless of platform.
Here are the differences between CSV and CSV-lite:

2
docs/file-formats.rst.in
View file

@ -30,7 +30,7 @@ Miller has record separator ``RS`` and field separator ``FS``, just as ``awk`` d
**USV (Unicode-separated values):** likewise, the flags ``--usv``, ``--iusv``, ``--ousv``, ``--usvlite``, ``--iusvlite``, and ``--ousvlite`` use Unicode FS and RS U+241F (UTF-8 0x0xe2909f) and U+241E (UTF-8 0xe2909e), respectively.
Miller's ``--csv`` flag supports `RFC-4180 CSV <https://tools.ietf.org/html/rfc4180">`_. This includes CRLF line-terminators by default, regardless of platform.
Miller's ``--csv`` flag supports `RFC-4180 CSV <https://tools.ietf.org/html/rfc4180>`_. This includes CRLF line-terminators by default, regardless of platform.
Here are the differences between CSV and CSV-lite:

1
docs/index.rst
View file

@ -68,5 +68,4 @@ Background
Index
----------------------------------------------------------------
* :ref:`genindex`
* :ref:`search`

1
docs6/10min.rst
View file

@ -12,6 +12,7 @@ You can install Miller for various platforms as follows:
* Linux: ``yum install miller`` or ``apt-get install miller`` depending on your flavor of Linux
* MacOS: ``brew install miller`` or ``port install miller`` depending on your preference of `Homebrew <https://brew.sh>`_ or `MacPorts <https://macports.org>`_.
* Windows: ``choco install miller`` using `Chocolatey <https://chocolatey.org>`_.
* You can get latest builds for Linux, MacOS, and Windows by visiting https://github.com/johnkerl/miller/actions, selecting the latest build, and clicking _Artifacts_. (These are retained for 5 days after each commit.)
* See also :doc:`build` if you prefer -- in particular, if your platform's package manager doesn't have the latest release.
As a first check, you should be able to run ``mlr --version`` at your system's command prompt and see something like the following:

1
docs6/10min.rst.in
View file

@ -9,6 +9,7 @@ You can install Miller for various platforms as follows:
* Linux: ``yum install miller`` or ``apt-get install miller`` depending on your flavor of Linux
* MacOS: ``brew install miller`` or ``port install miller`` depending on your preference of `Homebrew <https://brew.sh>`_ or `MacPorts <https://macports.org>`_.
* Windows: ``choco install miller`` using `Chocolatey <https://chocolatey.org>`_.
* You can get latest builds for Linux, MacOS, and Windows by visiting https://github.com/johnkerl/miller/actions, selecting the latest build, and clicking _Artifacts_. (These are retained for 5 days after each commit.)
* See also :doc:`build` if you prefer -- in particular, if your platform's package manager doesn't have the latest release.
As a first check, you should be able to run ``mlr --version`` at your system's command prompt and see something like the following:

2
docs6/_static/scrolls.css
View file

@ -151,7 +151,7 @@ blockquote {
p {
line-height: 20px;
margin-bottom: 0;
margin-bottom: 10px;
margin-top: 10px;
}

BIN
docs6/_static/top.png
View file

Binary file not shown.
Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 5.8 KiB

After

Width:  |  Height:  |  Size: 17 KiB

Before After
Before After

2
docs6/conf.py
View file

@ -18,7 +18,7 @@
# -- Project information -----------------------------------------------------
project = 'Miller'
copyright = '2020, John Kerl'
copyright = '2021, John Kerl'
author = 'John Kerl'
# The full version, including alpha/beta/rc tags

8
docs6/data/array-example.json Normal file
View file

@ -0,0 +1,8 @@
{
"key": "ax04",
"samples": [45, 67, 22]
}
{
"key": "cz09",
"samples": [11, 29, 84, 91]
}

62
docs6/file-formats.rst
View file

@ -4,7 +4,9 @@
File formats
================================================================
Miller handles name-indexed data using several formats: some you probably know by name, such as CSV, TSV, and JSON -- and other formats you're likely already seeing and using in your structured data. Additionally, Miller gives you the option of including comments within your data.
Miller handles name-indexed data using several formats: some you probably know by name, such as CSV, TSV, and JSON -- and other formats you're likely already seeing and using in your structured data.
Additionally, Miller gives you the option of including comments within your data.
Examples
----------------------------------------------------------------
@ -13,26 +15,15 @@ Examples
:emphasize-lines: 1-1
$ mlr --usage-data-format-examples
DKVP: delimited key-value pairs (Miller default format)
+---------------------+
| apple=1,bat=2,cog=3 | Record 1: "apple" => "1", "bat" => "2", "cog" => "3"
| dish=7,egg=8,flint | Record 2: "dish" => "7", "egg" => "8", "3" => "flint"
+---------------------+
NIDX: implicitly numerically indexed (Unix-toolkit style)
+---------------------+
| the quick brown | Record 1: "1" => "the", "2" => "quick", "3" => "brown"
| fox jumped | Record 2: "1" => "fox", "2" => "jumped"
+---------------------+
CSV/CSV-lite: comma-separated values with separate header line
CSV/CSV-lite: comma-separated values with separate header line
TSV: same but with tabs in places of commas
+---------------------+
| apple,bat,cog |
| 1,2,3 | Record 1: "apple => "1", "bat" => "2", "cog" => "3"
| 4,5,6 | Record 2: "apple" => "4", "bat" => "5", "cog" => "6"
+---------------------+
Tabular JSON: nested objects are supported, although arrays within them are not:
JSON (sequence or array of objects):
+---------------------+
| { |
| "apple": 1, | Record 1: "apple" => "1", "bat" => "2", "cog" => "3"
@ -55,6 +46,14 @@ Examples
| 4 5 6 | Record 2: "apple" => "4", "bat" => "5", "cog" => "6"
+---------------------+
Markdown tabular (supported for output only):
+-----------------------+
| | apple | bat | cog | |
| | --- | --- | --- | |
| | 1 | 2 | 3 | | Record 1: "apple => "1", "bat" => "2", "cog" => "3"
| | 4 | 5 | 6 | | Record 2: "apple" => "4", "bat" => "5", "cog" => "6"
+-----------------------+
XTAB: pretty-printed transposed tabular
+---------------------+
| apple 1 | Record 1: "apple" => "1", "bat" => "2", "cog" => "3"
@ -65,13 +64,17 @@ Examples
| egg 8 |
+---------------------+
Markdown tabular (supported for output only):
+-----------------------+
| | apple | bat | cog | |
| | --- | --- | --- | |
| | 1 | 2 | 3 | | Record 1: "apple => "1", "bat" => "2", "cog" => "3"
| | 4 | 5 | 6 | | Record 2: "apple" => "4", "bat" => "5", "cog" => "6"
+-----------------------+
DKVP: delimited key-value pairs (Miller default format)
+---------------------+
| apple=1,bat=2,cog=3 | Record 1: "apple" => "1", "bat" => "2", "cog" => "3"
| dish=7,egg=8,flint | Record 2: "dish" => "7", "egg" => "8", "3" => "flint"
+---------------------+
NIDX: implicitly numerically indexed (Unix-toolkit style)
+---------------------+
| the quick brown | Record 1: "1" => "the", "2" => "quick", "3" => "brown"
| fox jumped | Record 2: "1" => "fox", "2" => "jumped"
+---------------------+
.. _file-formats-csv:
@ -95,7 +98,7 @@ Miller has record separator ``RS`` and field separator ``FS``, just as ``awk`` d
**USV (Unicode-separated values):** likewise, the flags ``--usv``, ``--iusv``, ``--ousv``, ``--usvlite``, ``--iusvlite``, and ``--ousvlite`` use Unicode FS and RS U+241F (UTF-8 0x0xe2909f) and U+241E (UTF-8 0xe2909e), respectively.
Miller's ``--csv`` flag supports `RFC-4180 CSV <https://tools.ietf.org/html/rfc4180">`_. This includes CRLF line-terminators by default, regardless of platform.
Miller's ``--csv`` flag supports `RFC-4180 CSV <https://tools.ietf.org/html/rfc4180>`_. This includes CRLF line-terminators by default, regardless of platform.
Here are the differences between CSV and CSV-lite:
@ -168,7 +171,9 @@ See :doc:`reference-main-io-options` regarding how to specify separators other t
NIDX: Index-numbered (toolkit style)
----------------------------------------------------------------
With ``--inidx --ifs ' ' --repifs``, Miller splits lines on whitespace and assigns integer field names starting with 1. This recapitulates Unix-toolkit behavior.
With ``--inidx --ifs ' ' --repifs``, Miller splits lines on whitespace and assigns integer field names starting with 1.
This recapitulates Unix-toolkit behavior.
Example with index-numbered output:
@ -237,6 +242,8 @@ JSON is a format which supports arbitrarily deep nesting of "objects" (hashmaps)
But if you have tabular data represented in JSON then Miller can handle that for you.
By *tabular JSON* I mean the data is either a sequence of one or more objects, or an array consisting of one or more orjects. Miller treats JSON objects as name-indexed records.
Single-level JSON objects
^^^^^^^^^^^^^^^^^^^^^^^^^
@ -421,15 +428,8 @@ JSON isn't a parameterized format, so ``RS``, ``FS``, ``PS`` aren't specifiable.
* Use ``--jflatsep yourstringhere`` to specify the string used for key concatenation: this defaults to a single colon.
* Use ``--jofmt`` to force Miller to apply the global ``--ofmt`` to floating-point values. First note: please use sprintf-style codes for double precision, e.g. ending in ``%lf``, ``%le``, or ``%lg``. Miller floats are double-precision so behavior using ``%f``, ``%d``, etc. is undefined. Second note: ``0.123`` is valid JSON; ``.123`` is not. Thus this feature allows you to emit JSON which may be unparseable by other tools.
Again, please see `jq <https://stedolan.github.io/jq/>`_ for a truly powerful, JSON-specific tool.
JSON non-streaming
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The JSON parser Miller uses does not return until all input is parsed: in particular this means that, unlike for other file formats, Miller does not (at present) handle JSON files in ``tail -f`` contexts.
.. _file-formats-pprint:
PPRINT: Pretty-printed tabular

19
docs6/file-formats.rst.in
View file

@ -1,7 +1,9 @@
File formats
================================================================
Miller handles name-indexed data using several formats: some you probably know by name, such as CSV, TSV, and JSON -- and other formats you're likely already seeing and using in your structured data. Additionally, Miller gives you the option of including comments within your data.
Miller handles name-indexed data using several formats: some you probably know by name, such as CSV, TSV, and JSON -- and other formats you're likely already seeing and using in your structured data.
Additionally, Miller gives you the option of including comments within your data.
Examples
----------------------------------------------------------------
@ -32,7 +34,7 @@ Miller has record separator ``RS`` and field separator ``FS``, just as ``awk`` d
**USV (Unicode-separated values):** likewise, the flags ``--usv``, ``--iusv``, ``--ousv``, ``--usvlite``, ``--iusvlite``, and ``--ousvlite`` use Unicode FS and RS U+241F (UTF-8 0x0xe2909f) and U+241E (UTF-8 0xe2909e), respectively.
Miller's ``--csv`` flag supports `RFC-4180 CSV <https://tools.ietf.org/html/rfc4180">`_. This includes CRLF line-terminators by default, regardless of platform.
Miller's ``--csv`` flag supports `RFC-4180 CSV <https://tools.ietf.org/html/rfc4180>`_. This includes CRLF line-terminators by default, regardless of platform.
Here are the differences between CSV and CSV-lite:
@ -99,7 +101,9 @@ See :doc:`reference-main-io-options` regarding how to specify separators other t
NIDX: Index-numbered (toolkit style)
----------------------------------------------------------------
With ``--inidx --ifs ' ' --repifs``, Miller splits lines on whitespace and assigns integer field names starting with 1. This recapitulates Unix-toolkit behavior.
With ``--inidx --ifs ' ' --repifs``, Miller splits lines on whitespace and assigns integer field names starting with 1.
This recapitulates Unix-toolkit behavior.
Example with index-numbered output:
@ -140,6 +144,8 @@ JSON is a format which supports arbitrarily deep nesting of "objects" (hashmaps)
But if you have tabular data represented in JSON then Miller can handle that for you.
By *tabular JSON* I mean the data is either a sequence of one or more objects, or an array consisting of one or more orjects. Miller treats JSON objects as name-indexed records.
Single-level JSON objects
^^^^^^^^^^^^^^^^^^^^^^^^^
@ -222,15 +228,8 @@ JSON isn't a parameterized format, so ``RS``, ``FS``, ``PS`` aren't specifiable.
* Use ``--jflatsep yourstringhere`` to specify the string used for key concatenation: this defaults to a single colon.
* Use ``--jofmt`` to force Miller to apply the global ``--ofmt`` to floating-point values. First note: please use sprintf-style codes for double precision, e.g. ending in ``%lf``, ``%le``, or ``%lg``. Miller floats are double-precision so behavior using ``%f``, ``%d``, etc. is undefined. Second note: ``0.123`` is valid JSON; ``.123`` is not. Thus this feature allows you to emit JSON which may be unparseable by other tools.
Again, please see `jq <https://stedolan.github.io/jq/>`_ for a truly powerful, JSON-specific tool.
JSON non-streaming
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The JSON parser Miller uses does not return until all input is parsed: in particular this means that, unlike for other file formats, Miller does not (at present) handle JSON files in ``tail -f`` contexts.
.. _file-formats-pprint:
PPRINT: Pretty-printed tabular

1
docs6/index.rst
View file

@ -83,6 +83,7 @@ Reference
reference-main-then-chaining
reference-main-auxiliary-commands
reference-main-data-types
reference-dsl-arrays
reference-main-null-data
reference-main-arithmetic
reference-main-regular-expressions

5
docs6/installation.rst
View file

@ -50,7 +50,10 @@ Prebuilt executables via GitHub per release
Please see https://github.com/johnkerl/miller/releases where there are builds for OSX Yosemite, Linux x86-64 (dynamically linked), and Windows.
Miller is autobuilt for **Linux**, **MacOS**, and **Windows** using **GitHub Actions** on every commit (https://github.com/johnkerl/miller/actions).
Prebuilt executables via GitHub per commit
----------------------------------------------------------------
Miller is autobuilt for **Linux**, **MacOS**, and **Windows** using **GitHub Actions** on every commit (https://github.com/johnkerl/miller/actions): select the latest build and click _Artifacts_. (These are retained for 5 days after each commit.)
Building from source
----------------------------------------------------------------

5
docs6/installation.rst.in
View file

@ -47,7 +47,10 @@ Prebuilt executables via GitHub per release
Please see https://github.com/johnkerl/miller/releases where there are builds for OSX Yosemite, Linux x86-64 (dynamically linked), and Windows.
Miller is autobuilt for **Linux**, **MacOS**, and **Windows** using **GitHub Actions** on every commit (https://github.com/johnkerl/miller/actions).
Prebuilt executables via GitHub per commit
----------------------------------------------------------------
Miller is autobuilt for **Linux**, **MacOS**, and **Windows** using **GitHub Actions** on every commit (https://github.com/johnkerl/miller/actions): select the latest build and click _Artifacts_. (These are retained for 5 days after each commit.)
Building from source
----------------------------------------------------------------

40
docs6/miller-on-windows.rst
View file

@ -4,11 +4,37 @@
Miller on Windows
================================================================
TODO
Native builds as of Miller 6
----------------------------------------------------------------
* ``mlr.exe``, no ``msys2.dll``
* easier to build yourself
* where to get binaries
* ``PATH`` seutp w/ examples
* MSYS2 ``PATH`` seutp w/ examples
* examples of the one gotcha: DSL @ CLI, and the ``put -f`` workaround
As of version 6.0.0, Miller builds directly on Windows. The experience is now almost the same as on Linux, NetBSD/FreeBSD, and MacOS.
MSYS2 is no longer required, although you can use Miller from within MSYS2 if you like. There is now simply a single ``mlr.exe``, with no ``msys2.dll`` alongside anymore.
See :doc:`installation` for how to get a copy of ``mlr.exe``.
Setup
----------------------------------------------------------------
Simply place ``mlr.exe`` somewhere within your ``PATH`` variable.
.. image:: pix/miller-windows.png
To use Miller from within MSYS2/Cygwin, also make sure ``mlr.exe`` is within the ``PATH`` variable.
.. image:: pix/miller-msys.png
Differences
----------------------------------------------------------------
:doc:`output-colorization` doesn't work on Windows, outside of MSYS2.
The Windows-support code within Miller makes effort to support Linux/Unix/MacOS-like command-line syntax including single-quoting of expressions for ``mlr put`` and ``mlr filter`` -- and in the examples above, this often works. However, there are still some cases where more complex expressions aren't successfully parsed from the Windows prompt, even though they are from MSYS2:
.. image:: pix/miller-windows-complex.png
.. image:: pix/miller-msys-complex.png
One workaround is to use MSYS2. Another is to put more complex DSL expressions into a file:
.. image:: pix/miller-windows-complex-workaround.png

40
docs6/miller-on-windows.rst.in
View file

@ -1,11 +1,37 @@
Miller on Windows
================================================================
TODO
Native builds as of Miller 6
----------------------------------------------------------------
* ``mlr.exe``, no ``msys2.dll``
* easier to build yourself
* where to get binaries
* ``PATH`` seutp w/ examples
* MSYS2 ``PATH`` seutp w/ examples
* examples of the one gotcha: DSL @ CLI, and the ``put -f`` workaround
As of version 6.0.0, Miller builds directly on Windows. The experience is now almost the same as on Linux, NetBSD/FreeBSD, and MacOS.
MSYS2 is no longer required, although you can use Miller from within MSYS2 if you like. There is now simply a single ``mlr.exe``, with no ``msys2.dll`` alongside anymore.
See :doc:`installation` for how to get a copy of ``mlr.exe``.
Setup
----------------------------------------------------------------
Simply place ``mlr.exe`` somewhere within your ``PATH`` variable.
.. image:: pix/miller-windows.png
To use Miller from within MSYS2/Cygwin, also make sure ``mlr.exe`` is within the ``PATH`` variable.
.. image:: pix/miller-msys.png
Differences
----------------------------------------------------------------
:doc:`output-colorization` doesn't work on Windows, outside of MSYS2.
The Windows-support code within Miller makes effort to support Linux/Unix/MacOS-like command-line syntax including single-quoting of expressions for ``mlr put`` and ``mlr filter`` -- and in the examples above, this often works. However, there are still some cases where more complex expressions aren't successfully parsed from the Windows prompt, even though they are from MSYS2:
.. image:: pix/miller-windows-complex.png
.. image:: pix/miller-msys-complex.png
One workaround is to use MSYS2. Another is to put more complex DSL expressions into a file:
.. image:: pix/miller-windows-complex-workaround.png

24
docs6/mk-func-h2s.sh
View file

@ -27,12 +27,24 @@ mlr -F | grep -v '^[a-zA-Z]' | uniq | while read funcname; do
elif [ "$funcname" = ':' ]; then
displayname='\:'
linkname='colon'
elif [ "$funcname" = '!' ]; then
displayname='\!'
linkname='colon'
elif [ "$funcname" = '? :' ]; then
displayname='\?'
linkname='question-mark-colon'
elif [ "$funcname" = '?:' ]; then
displayname='\?'
linkname='question-mark-colon'
elif [ "$funcname" = '>>' ]; then
displayname='\>\>'
linkname='srsh'
elif [ "$funcname" = '>>>' ]; then
displayname='\>\>\>'
linkname='ursh'
elif [ "$funcname" = '>>>=' ]; then
displayname='\>\>\>='
linkname='ursheq'
fi
echo ""
@ -72,9 +84,21 @@ mlr -F | grep '^[a-zA-Z]' | sort -u | while read funcname; do
elif [ "$funcname" = ':' ]; then
displayname='\:'
linkname='colon'
elif [ "$funcname" = '!' ]; then
displayname='\!'
linkname='colon'
elif [ "$funcname" = '? :' ]; then
displayname='\?'
linkname='question-mark-colon'
elif [ "$funcname" = '>>' ]; then
displayname='\>\>'
linkname='srsh'
elif [ "$funcname" = '>>>' ]; then
displayname='\>\>\>'
linkname='ursh'
elif [ "$funcname" = '>>>=' ]; then
displayname='\>\>\>='
linkname='ursheq'
fi
echo ""

84
docs6/new-in-miller-6.rst
View file

@ -4,44 +4,76 @@
What's new in Miller 6
================================================================
[doc is WIP]
See also https://github.com/johnkerl/miller/labels/go-port
* Completely reworked documentation (here) and on-line help (``mlr --help``)
* Arrays in the ``put``/``filter`` DSL
Documentation improvements
----------------------------------------------------------------
* also ``array`` is now a keyword so you can't use that as a local-variable or UDF name
Documentation (here) and on-line help (``mlr --help``) has been completely reworked.
* JSON:
* Intro material is pulled out front, no longer assuming users are comparing to tools like AWK.
* Since CSV is the file format most used by Miller users, it is discussed first, and more examples use CSV.
* There is more focus on simpler use-cases, and detailed how-to material has been moved further down.
* Long reference sections have been split up.
* Improved JSON support
* Streamable JSON parsing
JSON support, and arrays
----------------------------------------------------------------
* Full* support for Windows
Arrays are now supported in Miller's ``put``/``filter`` programming language, as described at :doc:`reference-dsl-arrays`. Also, ``array`` is now a keyword so this is no longer usable as a local-variable or UDF name.
* Make a Windows docpage
JSON support is improved:
* Direct support for arrays means that you can now use Miller to process more JSON files.
* Streamable JSON parsing: Miller's internal record-processing pipeline starts as soon as the first record is read (which was already the case for other file formats). This means that, unless records are wrapped with outermost ``[...]``, Miller now handles JSON in ``tail -f`` contexts like it does for other file formats.
* Flatten/unflatten -- TODO pick a name and link to a separate page/section
Improved Windows experience
----------------------------------------------------------------
Stronger support for Windows (with or without MSYS2), with a couple of exceptions. See :doc:`miller-on-windows` for more information.
Binaries are reliably available using GitHub Actions: see also :doc:`installation`.
In-process support for compressed input
----------------------------------------------------------------
In addition to ``--prepipe gunzip``, you can now use the ``--gzin`` flag, and similarly for BZIP files. In fact, Miller will autodetect (by file extension) and automatically uncompress ``mlr --csv cat foo.csv.gz``. Please see section [TODO:linkify] for more information.
TODO: type up
Output colorization
----------------------------------------------------------------
See :doc:`output-colorization`.
New DSL functions / operators
----------------------------------------------------------------
* String-hashing functions :ref:`reference-dsl-md5`, :ref:`reference-dsl-sha1`, :ref:`reference-dsl-sha256`, and :ref:`reference-dsl-sha512`
* Platform-property functions :ref:`reference-dsl-hostname`, :ref:`reference-dsl-os`, and :ref:`reference-dsl-version`
* Unsigned right-shift :ref:`reference-dsl-ursh` along with ``>>>=``
Improved command-line parsing
----------------------------------------------------------------
Getoptish command-line parsing (https://github.com/johnkerl/miller/pull/467):
* ``-xyz`` expands automatically to ``-x -y -z``, so (for example) ``mlr cut -of shape,flag`` is the same as ``mlr cut -o -f shape,flag``.
* ``--foo=bar`` expands automatically to ``--foo bar``, so (for example) ``mlr --ifs=comma`` is the same as ``mlr --ifs comma``.
* ``--mfrom``, ``--load``, ``--mload`` as described at [TODO:linkify].
To file
----------------------------------------------------------------
* Build artifacts (binaries) using GitHub Actions
* In-process support for compressed input
* Built-in ``md5`` ``sha1`` etc
* ``hostname`` ``os`` ``version`` functions
* Input-preservation -- find a way to describe this -- link to the issue ...
* REPL TBD
* :doc:`output-colorization`
* Minor:
* Getoptish (#467)
* ``-xyz`` to ``-x -y -z`` -- show an example. And back-compat for ``sort``
* ``--foo=bar`` to ``--foo bar`` -- show an example.
* ``--mfrom``, ``--load``, ``--mload``
* Better syntax-error messages for the DSL, including line number
* ``substr0`` and ``substr1``, and why
* ``>>>`` and ``>>>=``
* Completely reworked regression-testing
* Dev: ported to Go
Developer-specific aspects
----------------------------------------------------------------
* Developer notes: https://github.com/johnkerl/miller/blob/main/go/README.md
See also https://github.com/johnkerl/miller/issues/372
* Miller has been ported from C to Go. Developer notes: https://github.com/johnkerl/miller/blob/main/go/README.md
* Completely reworked regression testing, including running on Windows

82
docs6/new-in-miller-6.rst.in
View file

@ -1,44 +1,74 @@
What's new in Miller 6
================================================================
[doc is WIP]
See also https://github.com/johnkerl/miller/labels/go-port
* Completely reworked documentation (here) and on-line help (``mlr --help``)
* Arrays in the ``put``/``filter`` DSL
Documentation improvements
----------------------------------------------------------------
* also ``array`` is now a keyword so you can't use that as a local-variable or UDF name
Documentation (here) and on-line help (``mlr --help``) has been completely reworked.
* JSON:
* Intro material is pulled out front, no longer assuming users are comparing to tools like AWK.
* Since CSV is the file format most used by Miller users, it is discussed first, and more examples use CSV.
* There is more focus on simpler use-cases, and detailed how-to material has been moved further down.
* Long reference sections have been split up.
* Improved JSON support
* Streamable JSON parsing
JSON support, and arrays
----------------------------------------------------------------
* Full* support for Windows
Arrays are now supported in Miller's ``put``/``filter`` programming language, as described at :doc:`reference-dsl-arrays`. Also, ``array`` is now a keyword so this is no longer usable as a local-variable or UDF name.
* Make a Windows docpage
JSON support is improved:
* Direct support for arrays means that you can now use Miller to process more JSON files.
* Streamable JSON parsing: Miller's internal record-processing pipeline starts as soon as the first record is read (which was already the case for other file formats). This means that, unless records are wrapped with outermost ``[...]``, Miller now handles JSON in ``tail -f`` contexts like it does for other file formats.
* Flatten/unflatten -- TODO pick a name and link to a separate page/section
Improved Windows experience
----------------------------------------------------------------
Stronger support for Windows (with or without MSYS2), with a couple of exceptions. See :doc:`miller-on-windows` for more information.
Binaries are reliably available using GitHub Actions: see also :doc:`installation`.
In-process support for compressed input
----------------------------------------------------------------
In addition to ``--prepipe gunzip``, you can now use the ``--gzin`` flag, and similarly for BZIP files. In fact, Miller will autodetect (by file extension) and automatically uncompress ``mlr --csv cat foo.csv.gz``. Please see section [TODO:linkify] for more information.
Output colorization
----------------------------------------------------------------
See :doc:`output-colorization`.
New DSL functions / operators
----------------------------------------------------------------
* String-hashing functions :ref:`reference-dsl-md5`, :ref:`reference-dsl-sha1`, :ref:`reference-dsl-sha256`, and :ref:`reference-dsl-sha512`
* Platform-property functions :ref:`reference-dsl-hostname`, :ref:`reference-dsl-os`, and :ref:`reference-dsl-version`
* Unsigned right-shift :ref:`reference-dsl-ursh` along with ``>>>=``
Improved command-line parsing
----------------------------------------------------------------
Getoptish command-line parsing (https://github.com/johnkerl/miller/pull/467):
* ``-xyz`` expands automatically to ``-x -y -z``, so (for example) ``mlr cut -of shape,flag`` is the same as ``mlr cut -o -f shape,flag``.
* ``--foo=bar`` expands automatically to ``--foo bar``, so (for example) ``mlr --ifs=comma`` is the same as ``mlr --ifs comma``.
* ``--mfrom``, ``--load``, ``--mload`` as described at [TODO:linkify].
To file
----------------------------------------------------------------
* Build artifacts (binaries) using GitHub Actions
* In-process support for compressed input
* Built-in ``md5`` ``sha1`` etc
* ``hostname`` ``os`` ``version`` functions
* Input-preservation -- find a way to describe this -- link to the issue ...
* REPL TBD
* :doc:`output-colorization`
* Minor:
* Getoptish (#467)
* ``-xyz`` to ``-x -y -z`` -- show an example. And back-compat for ``sort``
* ``--foo=bar`` to ``--foo bar`` -- show an example.
* ``--mfrom``, ``--load``, ``--mload``
* Better syntax-error messages for the DSL, including line number
* ``substr0`` and ``substr1``, and why
* ``>>>`` and ``>>>=``
* Completely reworked regression-testing
* Dev: ported to Go
Developer-specific aspects
----------------------------------------------------------------
* Developer notes: https://github.com/johnkerl/miller/blob/main/go/README.md
See also https://github.com/johnkerl/miller/issues/372
* Miller has been ported from C to Go. Developer notes: https://github.com/johnkerl/miller/blob/main/go/README.md
* Completely reworked regression testing, including running on Windows

BIN
docs6/pix/miller-msys-complex.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 54 KiB

BIN
docs6/pix/miller-msys.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 226 KiB

BIN
docs6/pix/miller-windows-complex-workaround.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 100 KiB

BIN
docs6/pix/miller-windows-complex.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 250 KiB

BIN
docs6/pix/miller-windows.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 645 KiB

20
docs6/reference-dsl-arrays.rst Normal file
View file

@ -0,0 +1,20 @@
..
PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE.
Reference: arrays
================================================================
TODO
.. code-block:: none
:emphasize-lines: 1-1
$ mlr --json cat data/array-example.json
{
"key": "ax04",
"samples": [45, 67, 22]
}
{
"key": "cz09",
"samples": [11, 29, 84, 91]
}

8
docs6/reference-dsl-arrays.rst.in Normal file
View file

@ -0,0 +1,8 @@
Reference: arrays
================================================================
TODO
GENRST_RUN_COMMAND
mlr --json cat data/array-example.json
GENRST_EOF

12
docs6/reference-dsl-builtin-functions.rst
View file

@ -195,9 +195,9 @@ You can get a list of all functions using **mlr -F**.
.. _reference-dsl->>:
.. _reference-dsl-srsh:
>>
\>\>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code-block:: none
@ -206,9 +206,9 @@ You can get a list of all functions using **mlr -F**.
.. _reference-dsl->>>:
.. _reference-dsl-ursh:
>>>
\>\>\>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code-block:: none
@ -217,9 +217,9 @@ You can get a list of all functions using **mlr -F**.
.. _reference-dsl-!:
.. _reference-dsl-colon:
!
\!
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code-block:: none

43
docs6/reference-main-online-help.rst
View file

@ -38,26 +38,15 @@ Examples:
mlr --from infile.dat put '(NR % 1000 == 0) { print > os.Stderr, "Checkpoint ".NR}'
DATA-FORMAT EXAMPLES:
DKVP: delimited key-value pairs (Miller default format)
+---------------------+
| apple=1,bat=2,cog=3 | Record 1: "apple" => "1", "bat" => "2", "cog" => "3"
| dish=7,egg=8,flint | Record 2: "dish" => "7", "egg" => "8", "3" => "flint"
+---------------------+
NIDX: implicitly numerically indexed (Unix-toolkit style)
+---------------------+
| the quick brown | Record 1: "1" => "the", "2" => "quick", "3" => "brown"
| fox jumped | Record 2: "1" => "fox", "2" => "jumped"
+---------------------+
CSV/CSV-lite: comma-separated values with separate header line
CSV/CSV-lite: comma-separated values with separate header line
TSV: same but with tabs in places of commas
+---------------------+
| apple,bat,cog |
| 1,2,3 | Record 1: "apple => "1", "bat" => "2", "cog" => "3"
| 4,5,6 | Record 2: "apple" => "4", "bat" => "5", "cog" => "6"
+---------------------+
Tabular JSON: nested objects are supported, although arrays within them are not:
JSON (sequence or array of objects):
+---------------------+
| { |
| "apple": 1, | Record 1: "apple" => "1", "bat" => "2", "cog" => "3"
@ -80,6 +69,14 @@ Examples:
| 4 5 6 | Record 2: "apple" => "4", "bat" => "5", "cog" => "6"
+---------------------+
Markdown tabular (supported for output only):
+-----------------------+
| | apple | bat | cog | |
| | --- | --- | --- | |
| | 1 | 2 | 3 | | Record 1: "apple => "1", "bat" => "2", "cog" => "3"
| | 4 | 5 | 6 | | Record 2: "apple" => "4", "bat" => "5", "cog" => "6"
+-----------------------+
XTAB: pretty-printed transposed tabular
+---------------------+
| apple 1 | Record 1: "apple" => "1", "bat" => "2", "cog" => "3"
@ -90,13 +87,17 @@ Examples:
| egg 8 |
+---------------------+
Markdown tabular (supported for output only):
+-----------------------+
| | apple | bat | cog | |
| | --- | --- | --- | |
| | 1 | 2 | 3 | | Record 1: "apple => "1", "bat" => "2", "cog" => "3"
| | 4 | 5 | 6 | | Record 2: "apple" => "4", "bat" => "5", "cog" => "6"
+-----------------------+
DKVP: delimited key-value pairs (Miller default format)
+---------------------+
| apple=1,bat=2,cog=3 | Record 1: "apple" => "1", "bat" => "2", "cog" => "3"
| dish=7,egg=8,flint | Record 2: "dish" => "7", "egg" => "8", "3" => "flint"
+---------------------+
NIDX: implicitly numerically indexed (Unix-toolkit style)
+---------------------+
| the quick brown | Record 1: "1" => "the", "2" => "quick", "3" => "brown"
| fox jumped | Record 2: "1" => "fox", "2" => "jumped"
+---------------------+
HELP OPTIONS:
-h or --help Show this message.

30
docs6/reference-main-overview.rst
View file

@ -7,29 +7,7 @@ Reference: Miller commands
Overview
----------------------------------------------------------------
TODO: push this into reference-verbs overview, and make this page an overview of miller invocations
Whereas the Unix toolkit is made of the separate executables ``cat``, ``tail``, ``cut``,
``sort``, etc., Miller has subcommands, or **verbs**, invoked as follows:
.. code-block:: none
mlr tac *.dat
mlr cut --complement -f os_version *.dat
mlr sort -f hostname,uptime *.dat
These fall into categories as follows:
* Analogs of their Unix-toolkit namesakes, discussed below as well as in :doc:`feature-comparison`: :ref:`reference-verbs-cat`, :ref:`reference-verbs-cut`, :ref:`reference-verbs-grep`, :ref:`reference-verbs-head`, :ref:`reference-verbs-join`, :ref:`reference-verbs-sort`, :ref:`reference-verbs-tac`, :ref:`reference-verbs-tail`, :ref:`reference-verbs-top`, :ref:`reference-verbs-uniq`.
* ``awk``-like functionality: :ref:`reference-verbs-filter`, :ref:`reference-verbs-put`, :ref:`reference-verbs-sec2gmt`, :ref:`reference-verbs-sec2gmtdate`, :ref:`reference-verbs-step`, :ref:`reference-verbs-tee`.
* Statistically oriented: :ref:`reference-verbs-bar`, :ref:`reference-verbs-bootstrap`, :ref:`reference-verbs-decimate`, :ref:`reference-verbs-histogram`, :ref:`reference-verbs-least-frequent`, :ref:`reference-verbs-most-frequent`, :ref:`reference-verbs-sample`, :ref:`reference-verbs-shuffle`, :ref:`reference-verbs-stats1`, :ref:`reference-verbs-stats2`.
* Particularly oriented toward :doc:`record-heterogeneity`, although all Miller commands can handle heterogeneous records: :ref:`reference-verbs-group-by`, :ref:`reference-verbs-group-like`, :ref:`reference-verbs-having-fields`.
* These draw from other sources (see also :doc:`originality`): :ref:`reference-verbs-count-distinct` is SQL-ish, and :ref:`reference-verbs-rename` can be done by ``sed`` (which does it faster: see :doc:`performance`. Verbs: :ref:`reference-verbs-check`, :ref:`reference-verbs-count-distinct`, :ref:`reference-verbs-label`, :ref:`reference-verbs-merge-fields`, :ref:`reference-verbs-nest`, :ref:`reference-verbs-nothing`, :ref:`reference-verbs-regularize`, :ref:`reference-verbs-rename`, :ref:`reference-verbs-reorder`, :ref:`reference-verbs-reshape`, :ref:`reference-verbs-seqgen`.
TODO: overview of miller invocations
Verbs vs DSL
----------------------------------------------------------------
@ -66,8 +44,8 @@ Example:
a=eks,x_sum=1.1400793586611044
a=wye,x_sum=0.7778922255683036
* You get to write your own DSL expressions
* You get to write your own expressions
* They run a bit slower
* They take more keystrokes
* There is more to learn
* They are highly customizable
* There's more to learn
* They're highly customizable

26
docs6/reference-main-overview.rst.in
View file

@ -4,25 +4,7 @@ Reference: Miller commands
Overview
----------------------------------------------------------------
TODO: push this into reference-verbs overview, and make this page an overview of miller invocations
Whereas the Unix toolkit is made of the separate executables ``cat``, ``tail``, ``cut``,
``sort``, etc., Miller has subcommands, or **verbs**, invoked as follows:
GENRST_INCLUDE_ESCAPED(data/subcommand-example.txt)
These fall into categories as follows:
* Analogs of their Unix-toolkit namesakes, discussed below as well as in :doc:`feature-comparison`: :ref:`reference-verbs-cat`, :ref:`reference-verbs-cut`, :ref:`reference-verbs-grep`, :ref:`reference-verbs-head`, :ref:`reference-verbs-join`, :ref:`reference-verbs-sort`, :ref:`reference-verbs-tac`, :ref:`reference-verbs-tail`, :ref:`reference-verbs-top`, :ref:`reference-verbs-uniq`.
* ``awk``-like functionality: :ref:`reference-verbs-filter`, :ref:`reference-verbs-put`, :ref:`reference-verbs-sec2gmt`, :ref:`reference-verbs-sec2gmtdate`, :ref:`reference-verbs-step`, :ref:`reference-verbs-tee`.
* Statistically oriented: :ref:`reference-verbs-bar`, :ref:`reference-verbs-bootstrap`, :ref:`reference-verbs-decimate`, :ref:`reference-verbs-histogram`, :ref:`reference-verbs-least-frequent`, :ref:`reference-verbs-most-frequent`, :ref:`reference-verbs-sample`, :ref:`reference-verbs-shuffle`, :ref:`reference-verbs-stats1`, :ref:`reference-verbs-stats2`.
* Particularly oriented toward :doc:`record-heterogeneity`, although all Miller commands can handle heterogeneous records: :ref:`reference-verbs-group-by`, :ref:`reference-verbs-group-like`, :ref:`reference-verbs-having-fields`.
* These draw from other sources (see also :doc:`originality`): :ref:`reference-verbs-count-distinct` is SQL-ish, and :ref:`reference-verbs-rename` can be done by ``sed`` (which does it faster: see :doc:`performance`. Verbs: :ref:`reference-verbs-check`, :ref:`reference-verbs-count-distinct`, :ref:`reference-verbs-label`, :ref:`reference-verbs-merge-fields`, :ref:`reference-verbs-nest`, :ref:`reference-verbs-nothing`, :ref:`reference-verbs-regularize`, :ref:`reference-verbs-rename`, :ref:`reference-verbs-reorder`, :ref:`reference-verbs-reshape`, :ref:`reference-verbs-seqgen`.
TODO: overview of miller invocations
Verbs vs DSL
----------------------------------------------------------------
@ -51,8 +33,8 @@ GENRST_RUN_COMMAND
mlr put -q '@x_sum[$a] += $x; end{emit @x_sum, "a"}' data/small
GENRST_EOF
* You get to write your own DSL expressions
* You get to write your own expressions
* They run a bit slower
* They take more keystrokes
* There is more to learn
* They are highly customizable
* There's more to learn
* They're highly customizable

26
docs6/reference-verbs.rst
View file

@ -4,6 +4,30 @@
Reference: List of verbs
================================================================
Overview
----------------------------------------------------------------
Whereas the Unix toolkit is made of the separate executables ``cat``, ``tail``, ``cut``,
``sort``, etc., Miller has subcommands, or **verbs**, invoked as follows:
.. code-block:: none
mlr tac *.dat
mlr cut --complement -f os_version *.dat
mlr sort -f hostname,uptime *.dat
These fall into categories as follows:
* Analogs of their Unix-toolkit namesakes, discussed below as well as in :doc:`feature-comparison`: :ref:`reference-verbs-cat`, :ref:`reference-verbs-cut`, :ref:`reference-verbs-grep`, :ref:`reference-verbs-head`, :ref:`reference-verbs-join`, :ref:`reference-verbs-sort`, :ref:`reference-verbs-tac`, :ref:`reference-verbs-tail`, :ref:`reference-verbs-top`, :ref:`reference-verbs-uniq`.
* ``awk``-like functionality: :ref:`reference-verbs-filter`, :ref:`reference-verbs-put`, :ref:`reference-verbs-sec2gmt`, :ref:`reference-verbs-sec2gmtdate`, :ref:`reference-verbs-step`, :ref:`reference-verbs-tee`.
* Statistically oriented: :ref:`reference-verbs-bar`, :ref:`reference-verbs-bootstrap`, :ref:`reference-verbs-decimate`, :ref:`reference-verbs-histogram`, :ref:`reference-verbs-least-frequent`, :ref:`reference-verbs-most-frequent`, :ref:`reference-verbs-sample`, :ref:`reference-verbs-shuffle`, :ref:`reference-verbs-stats1`, :ref:`reference-verbs-stats2`.
* Particularly oriented toward :doc:`record-heterogeneity`, although all Miller commands can handle heterogeneous records: :ref:`reference-verbs-group-by`, :ref:`reference-verbs-group-like`, :ref:`reference-verbs-having-fields`.
* These draw from other sources (see also :doc:`originality`): :ref:`reference-verbs-count-distinct` is SQL-ish, and :ref:`reference-verbs-rename` can be done by ``sed`` (which does it faster: see :doc:`performance`. Verbs: :ref:`reference-verbs-check`, :ref:`reference-verbs-count-distinct`, :ref:`reference-verbs-label`, :ref:`reference-verbs-merge-fields`, :ref:`reference-verbs-nest`, :ref:`reference-verbs-nothing`, :ref:`reference-verbs-regularize`, :ref:`reference-verbs-rename`, :ref:`reference-verbs-reorder`, :ref:`reference-verbs-reshape`, :ref:`reference-verbs-seqgen`.
.. _reference-verbs-altkv:
altkv
@ -1817,8 +1841,8 @@ most-frequent
square red
triangle red
circle red
square blue
square yellow
square blue
See also :ref:`reference-verbs-least-frequent`.

20
docs6/reference-verbs.rst.in
View file

@ -1,6 +1,26 @@
Reference: List of verbs
================================================================
Overview
----------------------------------------------------------------
Whereas the Unix toolkit is made of the separate executables ``cat``, ``tail``, ``cut``,
``sort``, etc., Miller has subcommands, or **verbs**, invoked as follows:
GENRST_INCLUDE_ESCAPED(data/subcommand-example.txt)
These fall into categories as follows:
* Analogs of their Unix-toolkit namesakes, discussed below as well as in :doc:`feature-comparison`: :ref:`reference-verbs-cat`, :ref:`reference-verbs-cut`, :ref:`reference-verbs-grep`, :ref:`reference-verbs-head`, :ref:`reference-verbs-join`, :ref:`reference-verbs-sort`, :ref:`reference-verbs-tac`, :ref:`reference-verbs-tail`, :ref:`reference-verbs-top`, :ref:`reference-verbs-uniq`.
* ``awk``-like functionality: :ref:`reference-verbs-filter`, :ref:`reference-verbs-put`, :ref:`reference-verbs-sec2gmt`, :ref:`reference-verbs-sec2gmtdate`, :ref:`reference-verbs-step`, :ref:`reference-verbs-tee`.
* Statistically oriented: :ref:`reference-verbs-bar`, :ref:`reference-verbs-bootstrap`, :ref:`reference-verbs-decimate`, :ref:`reference-verbs-histogram`, :ref:`reference-verbs-least-frequent`, :ref:`reference-verbs-most-frequent`, :ref:`reference-verbs-sample`, :ref:`reference-verbs-shuffle`, :ref:`reference-verbs-stats1`, :ref:`reference-verbs-stats2`.
* Particularly oriented toward :doc:`record-heterogeneity`, although all Miller commands can handle heterogeneous records: :ref:`reference-verbs-group-by`, :ref:`reference-verbs-group-like`, :ref:`reference-verbs-having-fields`.
* These draw from other sources (see also :doc:`originality`): :ref:`reference-verbs-count-distinct` is SQL-ish, and :ref:`reference-verbs-rename` can be done by ``sed`` (which does it faster: see :doc:`performance`. Verbs: :ref:`reference-verbs-check`, :ref:`reference-verbs-count-distinct`, :ref:`reference-verbs-label`, :ref:`reference-verbs-merge-fields`, :ref:`reference-verbs-nest`, :ref:`reference-verbs-nothing`, :ref:`reference-verbs-regularize`, :ref:`reference-verbs-rename`, :ref:`reference-verbs-reorder`, :ref:`reference-verbs-reshape`, :ref:`reference-verbs-seqgen`.
.. _reference-verbs-altkv:
altkv

88
go/help.txt
View file

@ -1,3 +1,91 @@
================================================================
manpage:
mlr --usage-synopsis
mlr --version
mlr --usage-examples
mlr --usage-data-format-examples
mlr --usage-help-options
mlr --usage-list-all-verbs
mlr --usage-functions
mlr --usage-data-format-options
mlr --usage-comments-in-data
mlr --usage-format-conversion-keystroke-saver-options
mlr --usage-compressed-data-options
mlr --usage-separator-options
mlr --usage-csv-options
mlr --usage-double-quoting
mlr --usage-numerical-formatting
mlr --usage-other-options
mlr --usage-then-chaining
mlr --usage-auxents
mlr --list-all-verbs-raw
mlr #{verb} -h
mlr --list-all-functions-raw
mlr --help-function '#{function}
mlr --list-all-keywords-raw`
mlr --help-keyword '#{keyword}'
----------------------------------------------------------------
old mlr --help:
mainUsageSynopsis(o, argv0)
mainUsageExamples(o, argv0, " ")
mainUsageDataFormatExamples(o, argv0)
mainUsageHelpOptions(o, argv0)
mainUsageMlrrc(o, argv0)
listAllVerbs(o, " ")
mainUsageFunctions(o)
mainUsageDataFormatOptions(o, argv0)
mainUsageCommentsInData(o, argv0)
mainUsageFormatConversionKeystrokeSaverOptions(o, argv0)
mainUsageCompressedDataOptions(o, argv0)
mainUsageSeparatorOptions(o, argv0);
mainUsageCsvOptions(o, argv0)
mainUsageDoubleQuoting(o, argv0);
mainUsageNumericalFormatting(o, argv0)
mainUsageOutputColorization(o, argv0)
mainUsageOtherOptions(o, argv0)
mainUsageThenChaining(o, argv0)
mainUsageAuxents(o)
mainUsageSeeAlso(o, argv0)
----------------------------------------------------------------
new -- ?!?
mlr --help
= mlr help
< mlr --usage-synopsis
mlr help {foo} w/ lookup -- ?
mlr --version
= mlr version
mlr --usage-examples
mlr --usage-data-format-examples
mlr --usage-help-options
mlr --usage-list-all-verbs
mlr --usage-functions
mlr --usage-data-format-options
mlr --usage-comments-in-data
mlr --usage-format-conversion-keystroke-saver-options
mlr --usage-compressed-data-options
mlr --usage-separator-options
mlr --usage-csv-options
mlr --usage-double-quoting
mlr --usage-numerical-formatting
mlr --usage-other-options
mlr --usage-then-chaining
mlr --usage-auxents
mlr --list-all-verbs-raw
mlr #{verb} -h
mlr --list-all-functions-raw
mlr --help-function '#{function}
mlr --list-all-keywords-raw`
mlr --help-keyword '#{keyword}'
================================================================
Usage: mlr [I/O options] {verb} [verb-dependent options ...] {zero or more file names}
----------------------------------------------------------------

4
go/src/auxents/regtest/regtester.go
View file

@ -28,6 +28,10 @@ const ShouldFailName = "should-fail"
const MajorSeparator = "================================================================"
const MinorSeparator = "----------------------------------------------------------------"
// Don't unset MLR_PASS_COLOR or MLR_FAIL_COLOR -- if people want to change the
// output-coloring used by this regression-tester, we should let them. We
// should only unset environment variables which can cause functional tests to
// fail.
var envVarsToUnset = []string{
"MLRRC",
"MLR_KEY_COLOR",

43
go/src/cli/mlrcli_usage.go
View file

@ -185,26 +185,15 @@ func mainUsageFunctions(o *os.File) {
// TODO: rid of argv0 throughout, replacing with lib.MlrExeName()
func mainUsageDataFormatExamples(o *os.File, argv0 string) {
fmt.Fprintf(o,
` DKVP: delimited key-value pairs (Miller default format)
+---------------------+
| apple=1,bat=2,cog=3 | Record 1: "apple" => "1", "bat" => "2", "cog" => "3"
| dish=7,egg=8,flint | Record 2: "dish" => "7", "egg" => "8", "3" => "flint"
+---------------------+
NIDX: implicitly numerically indexed (Unix-toolkit style)
+---------------------+
| the quick brown | Record 1: "1" => "the", "2" => "quick", "3" => "brown"
| fox jumped | Record 2: "1" => "fox", "2" => "jumped"
+---------------------+
CSV/CSV-lite: comma-separated values with separate header line
` CSV/CSV-lite: comma-separated values with separate header line
TSV: same but with tabs in places of commas
+---------------------+
| apple,bat,cog |
| 1,2,3 | Record 1: "apple => "1", "bat" => "2", "cog" => "3"
| 4,5,6 | Record 2: "apple" => "4", "bat" => "5", "cog" => "6"
+---------------------+
Tabular JSON: nested objects are supported, although arrays within them are not:
JSON (sequence or array of objects):
+---------------------+
| { |
| "apple": 1, | Record 1: "apple" => "1", "bat" => "2", "cog" => "3"
@ -227,6 +216,14 @@ func mainUsageDataFormatExamples(o *os.File, argv0 string) {
| 4 5 6 | Record 2: "apple" => "4", "bat" => "5", "cog" => "6"
+---------------------+
Markdown tabular (supported for output only):
+-----------------------+
| | apple | bat | cog | |
| | --- | --- | --- | |
| | 1 | 2 | 3 | | Record 1: "apple => "1", "bat" => "2", "cog" => "3"
| | 4 | 5 | 6 | | Record 2: "apple" => "4", "bat" => "5", "cog" => "6"
+-----------------------+
XTAB: pretty-printed transposed tabular
+---------------------+
| apple 1 | Record 1: "apple" => "1", "bat" => "2", "cog" => "3"
@ -237,13 +234,17 @@ func mainUsageDataFormatExamples(o *os.File, argv0 string) {
| egg 8 |
+---------------------+
Markdown tabular (supported for output only):
+-----------------------+
| | apple | bat | cog | |
| | --- | --- | --- | |
| | 1 | 2 | 3 | | Record 1: "apple => "1", "bat" => "2", "cog" => "3"
| | 4 | 5 | 6 | | Record 2: "apple" => "4", "bat" => "5", "cog" => "6"
+-----------------------+
DKVP: delimited key-value pairs (Miller default format)
+---------------------+
| apple=1,bat=2,cog=3 | Record 1: "apple" => "1", "bat" => "2", "cog" => "3"
| dish=7,egg=8,flint | Record 2: "dish" => "7", "egg" => "8", "3" => "flint"
+---------------------+
NIDX: implicitly numerically indexed (Unix-toolkit style)
+---------------------+
| the quick brown | Record 1: "1" => "the", "2" => "quick", "3" => "brown"
| fox jumped | Record 2: "1" => "fox", "2" => "jumped"
+---------------------+
`)
}

26
go/todo.txt
View file

@ -13,10 +13,6 @@ TOP OF LIST:
----------------------------------------------------------------
colorizer:
* mrpl output?
* test on:
o mac (various emulators)
o linux (laptop+instances; screen, tmux, various emulators)
o windows (cmd & PS)
* REPL PS1 color?
----------------------------------------------------------------
@ -35,15 +31,18 @@ DOC6
! fill out:
o how-to-help guide
o windows guide
o repl guide
o env-vars section
o new-in-miller-6: flatten/unflatten
o JSON flatten/unflatten section
o Arrays section including 1-up and why
! thruout pitch the docs at primarily the first whenever possible:
- python/ruby-level user w/o heavy sysadm
- sysadm w/o dev
> do not assume people know awk! XD
- devs
* items:
o file-formats:
o NIDX link to headerless CSV
o JSON re-write re arrays & key-concat -- needs its own section
o format ordering: pick one for sphinx, manpage, olh
o autodetect: tbd
* code to-do items:
o stats1 --fr
@ -65,6 +64,13 @@ DOC6
* data-cleaning examples -> full section -- ?
* c2 & c3 are all esoterica
! thruout pitch the docs at primarily the first whenever possible:
- python/ruby-level user w/o heavy sysadm
- sysadm w/o dev
> do not assume people know awk! XD
- devs
----------------------------------------------------------------
* case auxfiles: cat them too

23
man6/Makefile Normal file
View file

@ -0,0 +1,23 @@
# Respective MANPATH entries would include /usr/local/share/man or $HOME/man.
# This should be run after make in the ../c directory but before make in the ../docs directory,
# since ../go/mlr is used to autogenerate ./manpage.txt which is used in ../docs.
# See also https://miller.readthedocs.io/en/latest/build.html#creating-a-new-release-for-developers
INSTALLDIR=/usr/local/share/man/man1
INSTALLHOME=$(HOME)/man/man1
top: .always
echo mkman start
mkman.rb > mlr6.1
mkman.rb | groff -man -Tascii | col -b > manpage.txt
echo mkman end
# These targets are only for local dev work.
install:
mkdir -p $(INSTALLDIR)
cp mlr6.1 $(INSTALLDIR)/mlr6.1
installhome:
mkdir -p $(INSTALLHOME)
cp mlr6.1 $(INSTALLHOME)/mlr6.1
.always:
true

2223
man6/manpage.txt Normal file
View file

File diff suppressed because it is too large Load diff

239
man6/mkman.rb Executable file
View file

@ -0,0 +1,239 @@
#!/usr/bin/env ruby
# ================================================================
# This is a manpage autogenerator for Miller. There are various tools out there
# for creating xroff-formatted manpages, but I wanted something with minimal
# external dependencies which would also automatically generate most of its
# output from the mlr executable itself. It turns out it's easy enough to get
# this in just a few lines of Ruby.
#
# Note for dev-viewing of the output:
# ./mkman.rb | groff -man -Tascii | less
# ================================================================
# ----------------------------------------------------------------
def main
# In case the user running this has a .mlrrc
ENV['MLRRC'] = '__none__'
print make_top
print make_section('NAME', [
"miller \\\- like awk, sed, cut, join, and sort for name-indexed data such as CSV and tabular JSON."
])
print make_section('SYNOPSIS', [
`mlr --usage-synopsis`
])
print make_section('DESCRIPTION', [
"""Miller operates on key-value-pair data while the familiar Unix tools operate
on integer-indexed fields: if the natural data structure for the latter is the
array, then Miller's natural data structure is the insertion-ordered hash map.
This encompasses a variety of data formats, including but not limited to the
familiar CSV, TSV, and JSON. (Miller can handle positionally-indexed data as
a special case.) This manpage documents #{`mlr --version`.chomp}."""
])
print make_section('EXAMPLES', [
""
])
print make_subsection('COMMAND-LINE SYNTAX', [])
print make_code_block(`mlr --usage-examples`)
print make_subsection('DATA FORMATS', [])
print make_code_block(`mlr --usage-data-format-examples`)
print make_section('OPTIONS', [
"""In the following option flags, the version with \"i\" designates the input
stream, \"o\" the output stream, and the version without prefix sets the option
for both input and output stream. For example: --irs sets the input record
separator, --ors the output record separator, and --rs sets both the input and
output separator to the given value."""
])
print make_subsection('HELP OPTIONS', [])
print make_code_block(`mlr --usage-help-options`)
print make_subsection('VERB LIST', [])
print make_code_block(`mlr --usage-list-all-verbs`)
print make_subsection('FUNCTION LIST', [])
print make_code_block(`mlr --usage-functions`)
print make_subsection('I/O FORMATTING', [])
print make_code_block(`mlr --usage-data-format-options`)
print make_subsection('COMMENTS IN DATA', [])
print make_code_block(`mlr --usage-comments-in-data`)
print make_subsection('FORMAT-CONVERSION KEYSTROKE-SAVERS', [])
print make_code_block(`mlr --usage-format-conversion-keystroke-saver-options`)
print make_subsection('COMPRESSED I/O', [])
print make_code_block(`mlr --usage-compressed-data-options`)
print make_subsection('SEPARATORS', [])
print make_code_block(`mlr --usage-separator-options`)
print make_subsection('CSV-SPECIFIC OPTIONS', [])
print make_code_block(`mlr --usage-csv-options`)
print make_subsection('DOUBLE-QUOTING FOR CSV/CSVLITE OUTPUT', [])
print make_code_block(`mlr --usage-double-quoting`)
print make_subsection('NUMERICAL FORMATTING', [])
print make_code_block(`mlr --usage-numerical-formatting`)
print make_subsection('OTHER OPTIONS', [])
print make_code_block(`mlr --usage-other-options`)
print make_subsection('THEN-CHAINING', [])
print make_code_block(`mlr --usage-then-chaining`)
print make_subsection('AUXILIARY COMMANDS', [])
print make_code_block(`mlr --usage-auxents`)
print make_section('MLRRC', [])
print make_code_block(`mlr --usage-mlrrc`)
verbs = `mlr --list-all-verbs-raw`
print make_section('VERBS', [
""
])
verbs = verbs.strip.split("\n")
for verb in verbs
print make_subsection(verb, [])
print make_code_block(`mlr #{verb} -h`)
end
functions = `mlr --list-all-functions-raw`
print make_section('FUNCTIONS FOR FILTER/PUT', [
""
])
functions = functions.strip.split("\n").uniq
for function in functions
print make_subsection(function, [])
text = `mlr --help-function '#{function}'`
text = text.sub(function + ' ', '')
print make_code_block(text)
end
keywords = `mlr --list-all-keywords-raw`
print make_section('KEYWORDS FOR PUT AND FILTER', [
""
])
keywords = keywords.strip.split("\n").uniq
for keyword in keywords
print make_subsection(keyword, [])
text = `mlr --help-keyword '#{keyword}'`
print make_code_block(text)
end
print make_section('AUTHOR', [
"Miller is written by John Kerl <kerl.john.r@gmail.com>.",
"This manual page has been composed from Miller's help output by Eric MSP Veith <eveith@veith-m.de>."
])
print make_section('SEE ALSO', [
"awk(1), sed(1), cut(1), join(1), sort(1), RFC 4180: Common Format and MIME Type for " +
"Comma-Separated Values (CSV) Files, the miller website http://johnkerl.org/miller/doc"
])
end
# ================================================================
def make_top()
t = Time::new
stamp = t.gmtime.strftime("%Y-%m-%d")
# Portability definitions thanks to some asciidoc output
"""'\\\" t
.\\\" Title: mlr
.\\\" Author: [see the \"AUTHOR\" section]
.\\\" Generator: #{$0}
.\\\" Date: #{stamp}
.\\\" Manual: \\ \\&
.\\\" Source: \\ \\&
.\\\" Language: English
.\\\"
.TH \"MILLER\" \"1\" \"#{stamp}\" \"\\ \\&\" \"\\ \\&\"
.\\\" -----------------------------------------------------------------
.\\\" * Portability definitions
.\\\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.\\\" http://bugs.debian.org/507673
.\\\" http://lists.gnu.org/archive/html/groff/2009-02/msg00013.html
.\\\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.ie \\n(.g .ds Aq \(aq
.el .ds Aq '
.\\\" -----------------------------------------------------------------
.\\\" * set default formatting
.\\\" -----------------------------------------------------------------
.\\\" disable hyphenation
.nh
.\\\" disable justification (adjust text to left margin only)
.ad l
.\\\" -----------------------------------------------------------------
"""
end
# ----------------------------------------------------------------
def make_section(title, paragraphs)
retval = ".SH \"#{title}\"\n"
paragraphs.each do |paragraph|
retval += ".sp\n"
retval += groff_encode(paragraph) + "\n"
end
retval
end
# ----------------------------------------------------------------
def make_subsection(title, paragraphs)
retval = ".SS \"#{title}\"\n"
paragraphs.each do |paragraph|
retval += ".sp\n"
retval += groff_encode(paragraph) + "\n"
end
retval
end
# ----------------------------------------------------------------
def make_subsubsection(title, paragraphs)
retval = ".sp\n";
retval += "\\fB#{title}\\fR\n"
paragraphs.each do |paragraph|
retval += ".sp\n"
retval += groff_encode(paragraph) + "\n"
end
retval
end
# ----------------------------------------------------------------
def make_code_block(block)
retval = ".if n \\{\\\n"
retval += ".RS 0\n"
retval += ".\\}\n"
retval += ".nf\n"
# In case the line starts with a dot:
retval += block.gsub('\\', '\e').gsub(/^\./){'\&.'}
# In case the line starts with a single quote:
retval = retval.gsub(/^'/, '\(cq')
retval += ".fi\n"
retval += ".if n \\{\\\n"
retval += ".RE\n"
end
# ----------------------------------------------------------------
def groff_encode(line)
#line = line.gsub(/'/, '\(cq')
#line = line.gsub(/"/, '\(dq')
#line = line.gsub(/\./, '\&')
#line = line.gsub(/-/, '\-')
line = line.gsub(/\\([^-])/, '\e\1')
line = line.gsub(/^\./){'\&.'}
line
end
# ================================================================
main

3838
man6/mlr6.1 Normal file
View file

File diff suppressed because it is too large Load diff