diff --git a/docs/_static/top.png b/docs/_static/top.png index f1387d1ee..f502d8ac7 100644 Binary files a/docs/_static/top.png and b/docs/_static/top.png differ diff --git a/docs/build-one b/docs/build-one new file mode 100755 index 000000000..7cc07f15f --- /dev/null +++ b/docs/build-one @@ -0,0 +1,13 @@ +#!/bin/bash +set -euo pipefail + +if [ $# -ge 1 ]; then + for name; do + if [[ $name == *.rst.in ]]; then + genrst $name; + fi + done +else + for rstin in *.rst.in; do genrst $rstin; done +fi +sphinx-build -M html . _build diff --git a/docs/file-formats.rst b/docs/file-formats.rst index ea551ef8a..143a9b204 100644 --- a/docs/file-formats.rst +++ b/docs/file-formats.rst @@ -95,7 +95,7 @@ Miller has record separator ``RS`` and field separator ``FS``, just as ``awk`` d **USV (Unicode-separated values):** likewise, the flags ``--usv``, ``--iusv``, ``--ousv``, ``--usvlite``, ``--iusvlite``, and ``--ousvlite`` use Unicode FS and RS U+241F (UTF-8 0x0xe2909f) and U+241E (UTF-8 0xe2909e), respectively. -Miller's ``--csv`` flag supports `RFC-4180 CSV `_. This includes CRLF line-terminators by default, regardless of platform. +Miller's ``--csv`` flag supports `RFC-4180 CSV `_. This includes CRLF line-terminators by default, regardless of platform. Here are the differences between CSV and CSV-lite: diff --git a/docs/file-formats.rst.in b/docs/file-formats.rst.in index fb6c89652..8204f90f4 100644 --- a/docs/file-formats.rst.in +++ b/docs/file-formats.rst.in @@ -30,7 +30,7 @@ Miller has record separator ``RS`` and field separator ``FS``, just as ``awk`` d **USV (Unicode-separated values):** likewise, the flags ``--usv``, ``--iusv``, ``--ousv``, ``--usvlite``, ``--iusvlite``, and ``--ousvlite`` use Unicode FS and RS U+241F (UTF-8 0x0xe2909f) and U+241E (UTF-8 0xe2909e), respectively. -Miller's ``--csv`` flag supports `RFC-4180 CSV `_. This includes CRLF line-terminators by default, regardless of platform. +Miller's ``--csv`` flag supports `RFC-4180 CSV `_. This includes CRLF line-terminators by default, regardless of platform. Here are the differences between CSV and CSV-lite: diff --git a/docs/index.rst b/docs/index.rst index b4bbf0868..184ae58d5 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -68,5 +68,4 @@ Background Index ---------------------------------------------------------------- -* :ref:`genindex` * :ref:`search` diff --git a/docs6/10min.rst b/docs6/10min.rst index 814d9eb0b..312b169f8 100644 --- a/docs6/10min.rst +++ b/docs6/10min.rst @@ -12,6 +12,7 @@ You can install Miller for various platforms as follows: * Linux: ``yum install miller`` or ``apt-get install miller`` depending on your flavor of Linux * MacOS: ``brew install miller`` or ``port install miller`` depending on your preference of `Homebrew `_ or `MacPorts `_. * Windows: ``choco install miller`` using `Chocolatey `_. +* You can get latest builds for Linux, MacOS, and Windows by visiting https://github.com/johnkerl/miller/actions, selecting the latest build, and clicking _Artifacts_. (These are retained for 5 days after each commit.) * See also :doc:`build` if you prefer -- in particular, if your platform's package manager doesn't have the latest release. As a first check, you should be able to run ``mlr --version`` at your system's command prompt and see something like the following: diff --git a/docs6/10min.rst.in b/docs6/10min.rst.in index b567c4265..d29f6b169 100644 --- a/docs6/10min.rst.in +++ b/docs6/10min.rst.in @@ -9,6 +9,7 @@ You can install Miller for various platforms as follows: * Linux: ``yum install miller`` or ``apt-get install miller`` depending on your flavor of Linux * MacOS: ``brew install miller`` or ``port install miller`` depending on your preference of `Homebrew `_ or `MacPorts `_. * Windows: ``choco install miller`` using `Chocolatey `_. +* You can get latest builds for Linux, MacOS, and Windows by visiting https://github.com/johnkerl/miller/actions, selecting the latest build, and clicking _Artifacts_. (These are retained for 5 days after each commit.) * See also :doc:`build` if you prefer -- in particular, if your platform's package manager doesn't have the latest release. As a first check, you should be able to run ``mlr --version`` at your system's command prompt and see something like the following: diff --git a/docs6/_static/scrolls.css b/docs6/_static/scrolls.css index 2b356d041..51f58c99e 100644 --- a/docs6/_static/scrolls.css +++ b/docs6/_static/scrolls.css @@ -151,7 +151,7 @@ blockquote { p { line-height: 20px; - margin-bottom: 0; + margin-bottom: 10px; margin-top: 10px; } diff --git a/docs6/_static/top.png b/docs6/_static/top.png index f1387d1ee..f502d8ac7 100644 Binary files a/docs6/_static/top.png and b/docs6/_static/top.png differ diff --git a/docs6/conf.py b/docs6/conf.py index 1b0bf9e58..f1695c807 100644 --- a/docs6/conf.py +++ b/docs6/conf.py @@ -18,7 +18,7 @@ # -- Project information ----------------------------------------------------- project = 'Miller' -copyright = '2020, John Kerl' +copyright = '2021, John Kerl' author = 'John Kerl' # The full version, including alpha/beta/rc tags diff --git a/docs6/data/array-example.json b/docs6/data/array-example.json new file mode 100644 index 000000000..9409d73a9 --- /dev/null +++ b/docs6/data/array-example.json @@ -0,0 +1,8 @@ +{ + "key": "ax04", + "samples": [45, 67, 22] +} +{ + "key": "cz09", + "samples": [11, 29, 84, 91] +} diff --git a/docs6/file-formats.rst b/docs6/file-formats.rst index 2e1c43143..be1165787 100644 --- a/docs6/file-formats.rst +++ b/docs6/file-formats.rst @@ -4,7 +4,9 @@ File formats ================================================================ -Miller handles name-indexed data using several formats: some you probably know by name, such as CSV, TSV, and JSON -- and other formats you're likely already seeing and using in your structured data. Additionally, Miller gives you the option of including comments within your data. +Miller handles name-indexed data using several formats: some you probably know by name, such as CSV, TSV, and JSON -- and other formats you're likely already seeing and using in your structured data. + +Additionally, Miller gives you the option of including comments within your data. Examples ---------------------------------------------------------------- @@ -13,26 +15,15 @@ Examples :emphasize-lines: 1-1 $ mlr --usage-data-format-examples - DKVP: delimited key-value pairs (Miller default format) - +---------------------+ - | apple=1,bat=2,cog=3 | Record 1: "apple" => "1", "bat" => "2", "cog" => "3" - | dish=7,egg=8,flint | Record 2: "dish" => "7", "egg" => "8", "3" => "flint" - +---------------------+ - - NIDX: implicitly numerically indexed (Unix-toolkit style) - +---------------------+ - | the quick brown | Record 1: "1" => "the", "2" => "quick", "3" => "brown" - | fox jumped | Record 2: "1" => "fox", "2" => "jumped" - +---------------------+ - - CSV/CSV-lite: comma-separated values with separate header line + CSV/CSV-lite: comma-separated values with separate header line + TSV: same but with tabs in places of commas +---------------------+ | apple,bat,cog | | 1,2,3 | Record 1: "apple => "1", "bat" => "2", "cog" => "3" | 4,5,6 | Record 2: "apple" => "4", "bat" => "5", "cog" => "6" +---------------------+ - Tabular JSON: nested objects are supported, although arrays within them are not: + JSON (sequence or array of objects): +---------------------+ | { | | "apple": 1, | Record 1: "apple" => "1", "bat" => "2", "cog" => "3" @@ -55,6 +46,14 @@ Examples | 4 5 6 | Record 2: "apple" => "4", "bat" => "5", "cog" => "6" +---------------------+ + Markdown tabular (supported for output only): + +-----------------------+ + | | apple | bat | cog | | + | | --- | --- | --- | | + | | 1 | 2 | 3 | | Record 1: "apple => "1", "bat" => "2", "cog" => "3" + | | 4 | 5 | 6 | | Record 2: "apple" => "4", "bat" => "5", "cog" => "6" + +-----------------------+ + XTAB: pretty-printed transposed tabular +---------------------+ | apple 1 | Record 1: "apple" => "1", "bat" => "2", "cog" => "3" @@ -65,13 +64,17 @@ Examples | egg 8 | +---------------------+ - Markdown tabular (supported for output only): - +-----------------------+ - | | apple | bat | cog | | - | | --- | --- | --- | | - | | 1 | 2 | 3 | | Record 1: "apple => "1", "bat" => "2", "cog" => "3" - | | 4 | 5 | 6 | | Record 2: "apple" => "4", "bat" => "5", "cog" => "6" - +-----------------------+ + DKVP: delimited key-value pairs (Miller default format) + +---------------------+ + | apple=1,bat=2,cog=3 | Record 1: "apple" => "1", "bat" => "2", "cog" => "3" + | dish=7,egg=8,flint | Record 2: "dish" => "7", "egg" => "8", "3" => "flint" + +---------------------+ + + NIDX: implicitly numerically indexed (Unix-toolkit style) + +---------------------+ + | the quick brown | Record 1: "1" => "the", "2" => "quick", "3" => "brown" + | fox jumped | Record 2: "1" => "fox", "2" => "jumped" + +---------------------+ .. _file-formats-csv: @@ -95,7 +98,7 @@ Miller has record separator ``RS`` and field separator ``FS``, just as ``awk`` d **USV (Unicode-separated values):** likewise, the flags ``--usv``, ``--iusv``, ``--ousv``, ``--usvlite``, ``--iusvlite``, and ``--ousvlite`` use Unicode FS and RS U+241F (UTF-8 0x0xe2909f) and U+241E (UTF-8 0xe2909e), respectively. -Miller's ``--csv`` flag supports `RFC-4180 CSV `_. This includes CRLF line-terminators by default, regardless of platform. +Miller's ``--csv`` flag supports `RFC-4180 CSV `_. This includes CRLF line-terminators by default, regardless of platform. Here are the differences between CSV and CSV-lite: @@ -168,7 +171,9 @@ See :doc:`reference-main-io-options` regarding how to specify separators other t NIDX: Index-numbered (toolkit style) ---------------------------------------------------------------- -With ``--inidx --ifs ' ' --repifs``, Miller splits lines on whitespace and assigns integer field names starting with 1. This recapitulates Unix-toolkit behavior. +With ``--inidx --ifs ' ' --repifs``, Miller splits lines on whitespace and assigns integer field names starting with 1. + +This recapitulates Unix-toolkit behavior. Example with index-numbered output: @@ -237,6 +242,8 @@ JSON is a format which supports arbitrarily deep nesting of "objects" (hashmaps) But if you have tabular data represented in JSON then Miller can handle that for you. +By *tabular JSON* I mean the data is either a sequence of one or more objects, or an array consisting of one or more orjects. Miller treats JSON objects as name-indexed records. + Single-level JSON objects ^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -421,15 +428,8 @@ JSON isn't a parameterized format, so ``RS``, ``FS``, ``PS`` aren't specifiable. * Use ``--jflatsep yourstringhere`` to specify the string used for key concatenation: this defaults to a single colon. -* Use ``--jofmt`` to force Miller to apply the global ``--ofmt`` to floating-point values. First note: please use sprintf-style codes for double precision, e.g. ending in ``%lf``, ``%le``, or ``%lg``. Miller floats are double-precision so behavior using ``%f``, ``%d``, etc. is undefined. Second note: ``0.123`` is valid JSON; ``.123`` is not. Thus this feature allows you to emit JSON which may be unparseable by other tools. - Again, please see `jq `_ for a truly powerful, JSON-specific tool. -JSON non-streaming -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The JSON parser Miller uses does not return until all input is parsed: in particular this means that, unlike for other file formats, Miller does not (at present) handle JSON files in ``tail -f`` contexts. - .. _file-formats-pprint: PPRINT: Pretty-printed tabular diff --git a/docs6/file-formats.rst.in b/docs6/file-formats.rst.in index c1ede52b0..50dabd184 100644 --- a/docs6/file-formats.rst.in +++ b/docs6/file-formats.rst.in @@ -1,7 +1,9 @@ File formats ================================================================ -Miller handles name-indexed data using several formats: some you probably know by name, such as CSV, TSV, and JSON -- and other formats you're likely already seeing and using in your structured data. Additionally, Miller gives you the option of including comments within your data. +Miller handles name-indexed data using several formats: some you probably know by name, such as CSV, TSV, and JSON -- and other formats you're likely already seeing and using in your structured data. + +Additionally, Miller gives you the option of including comments within your data. Examples ---------------------------------------------------------------- @@ -32,7 +34,7 @@ Miller has record separator ``RS`` and field separator ``FS``, just as ``awk`` d **USV (Unicode-separated values):** likewise, the flags ``--usv``, ``--iusv``, ``--ousv``, ``--usvlite``, ``--iusvlite``, and ``--ousvlite`` use Unicode FS and RS U+241F (UTF-8 0x0xe2909f) and U+241E (UTF-8 0xe2909e), respectively. -Miller's ``--csv`` flag supports `RFC-4180 CSV `_. This includes CRLF line-terminators by default, regardless of platform. +Miller's ``--csv`` flag supports `RFC-4180 CSV `_. This includes CRLF line-terminators by default, regardless of platform. Here are the differences between CSV and CSV-lite: @@ -99,7 +101,9 @@ See :doc:`reference-main-io-options` regarding how to specify separators other t NIDX: Index-numbered (toolkit style) ---------------------------------------------------------------- -With ``--inidx --ifs ' ' --repifs``, Miller splits lines on whitespace and assigns integer field names starting with 1. This recapitulates Unix-toolkit behavior. +With ``--inidx --ifs ' ' --repifs``, Miller splits lines on whitespace and assigns integer field names starting with 1. + +This recapitulates Unix-toolkit behavior. Example with index-numbered output: @@ -140,6 +144,8 @@ JSON is a format which supports arbitrarily deep nesting of "objects" (hashmaps) But if you have tabular data represented in JSON then Miller can handle that for you. +By *tabular JSON* I mean the data is either a sequence of one or more objects, or an array consisting of one or more orjects. Miller treats JSON objects as name-indexed records. + Single-level JSON objects ^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -222,15 +228,8 @@ JSON isn't a parameterized format, so ``RS``, ``FS``, ``PS`` aren't specifiable. * Use ``--jflatsep yourstringhere`` to specify the string used for key concatenation: this defaults to a single colon. -* Use ``--jofmt`` to force Miller to apply the global ``--ofmt`` to floating-point values. First note: please use sprintf-style codes for double precision, e.g. ending in ``%lf``, ``%le``, or ``%lg``. Miller floats are double-precision so behavior using ``%f``, ``%d``, etc. is undefined. Second note: ``0.123`` is valid JSON; ``.123`` is not. Thus this feature allows you to emit JSON which may be unparseable by other tools. - Again, please see `jq `_ for a truly powerful, JSON-specific tool. -JSON non-streaming -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The JSON parser Miller uses does not return until all input is parsed: in particular this means that, unlike for other file formats, Miller does not (at present) handle JSON files in ``tail -f`` contexts. - .. _file-formats-pprint: PPRINT: Pretty-printed tabular diff --git a/docs6/index.rst b/docs6/index.rst index 1713d4039..948303b93 100644 --- a/docs6/index.rst +++ b/docs6/index.rst @@ -83,6 +83,7 @@ Reference reference-main-then-chaining reference-main-auxiliary-commands reference-main-data-types + reference-dsl-arrays reference-main-null-data reference-main-arithmetic reference-main-regular-expressions diff --git a/docs6/installation.rst b/docs6/installation.rst index f8544b2ec..f9444df2c 100644 --- a/docs6/installation.rst +++ b/docs6/installation.rst @@ -50,7 +50,10 @@ Prebuilt executables via GitHub per release Please see https://github.com/johnkerl/miller/releases where there are builds for OSX Yosemite, Linux x86-64 (dynamically linked), and Windows. -Miller is autobuilt for **Linux**, **MacOS**, and **Windows** using **GitHub Actions** on every commit (https://github.com/johnkerl/miller/actions). +Prebuilt executables via GitHub per commit +---------------------------------------------------------------- + +Miller is autobuilt for **Linux**, **MacOS**, and **Windows** using **GitHub Actions** on every commit (https://github.com/johnkerl/miller/actions): select the latest build and click _Artifacts_. (These are retained for 5 days after each commit.) Building from source ---------------------------------------------------------------- diff --git a/docs6/installation.rst.in b/docs6/installation.rst.in index 350ad9db6..4ccf942c2 100644 --- a/docs6/installation.rst.in +++ b/docs6/installation.rst.in @@ -47,7 +47,10 @@ Prebuilt executables via GitHub per release Please see https://github.com/johnkerl/miller/releases where there are builds for OSX Yosemite, Linux x86-64 (dynamically linked), and Windows. -Miller is autobuilt for **Linux**, **MacOS**, and **Windows** using **GitHub Actions** on every commit (https://github.com/johnkerl/miller/actions). +Prebuilt executables via GitHub per commit +---------------------------------------------------------------- + +Miller is autobuilt for **Linux**, **MacOS**, and **Windows** using **GitHub Actions** on every commit (https://github.com/johnkerl/miller/actions): select the latest build and click _Artifacts_. (These are retained for 5 days after each commit.) Building from source ---------------------------------------------------------------- diff --git a/docs6/miller-on-windows.rst b/docs6/miller-on-windows.rst index 5954b216c..394097806 100644 --- a/docs6/miller-on-windows.rst +++ b/docs6/miller-on-windows.rst @@ -4,11 +4,37 @@ Miller on Windows ================================================================ -TODO +Native builds as of Miller 6 +---------------------------------------------------------------- -* ``mlr.exe``, no ``msys2.dll`` -* easier to build yourself -* where to get binaries -* ``PATH`` seutp w/ examples -* MSYS2 ``PATH`` seutp w/ examples -* examples of the one gotcha: DSL @ CLI, and the ``put -f`` workaround +As of version 6.0.0, Miller builds directly on Windows. The experience is now almost the same as on Linux, NetBSD/FreeBSD, and MacOS. + +MSYS2 is no longer required, although you can use Miller from within MSYS2 if you like. There is now simply a single ``mlr.exe``, with no ``msys2.dll`` alongside anymore. + +See :doc:`installation` for how to get a copy of ``mlr.exe``. + +Setup +---------------------------------------------------------------- + +Simply place ``mlr.exe`` somewhere within your ``PATH`` variable. + +.. image:: pix/miller-windows.png + +To use Miller from within MSYS2/Cygwin, also make sure ``mlr.exe`` is within the ``PATH`` variable. + +.. image:: pix/miller-msys.png + +Differences +---------------------------------------------------------------- + +:doc:`output-colorization` doesn't work on Windows, outside of MSYS2. + +The Windows-support code within Miller makes effort to support Linux/Unix/MacOS-like command-line syntax including single-quoting of expressions for ``mlr put`` and ``mlr filter`` -- and in the examples above, this often works. However, there are still some cases where more complex expressions aren't successfully parsed from the Windows prompt, even though they are from MSYS2: + +.. image:: pix/miller-windows-complex.png + +.. image:: pix/miller-msys-complex.png + +One workaround is to use MSYS2. Another is to put more complex DSL expressions into a file: + +.. image:: pix/miller-windows-complex-workaround.png diff --git a/docs6/miller-on-windows.rst.in b/docs6/miller-on-windows.rst.in index 2034d15ac..3fc673128 100644 --- a/docs6/miller-on-windows.rst.in +++ b/docs6/miller-on-windows.rst.in @@ -1,11 +1,37 @@ Miller on Windows ================================================================ -TODO +Native builds as of Miller 6 +---------------------------------------------------------------- -* ``mlr.exe``, no ``msys2.dll`` -* easier to build yourself -* where to get binaries -* ``PATH`` seutp w/ examples -* MSYS2 ``PATH`` seutp w/ examples -* examples of the one gotcha: DSL @ CLI, and the ``put -f`` workaround +As of version 6.0.0, Miller builds directly on Windows. The experience is now almost the same as on Linux, NetBSD/FreeBSD, and MacOS. + +MSYS2 is no longer required, although you can use Miller from within MSYS2 if you like. There is now simply a single ``mlr.exe``, with no ``msys2.dll`` alongside anymore. + +See :doc:`installation` for how to get a copy of ``mlr.exe``. + +Setup +---------------------------------------------------------------- + +Simply place ``mlr.exe`` somewhere within your ``PATH`` variable. + +.. image:: pix/miller-windows.png + +To use Miller from within MSYS2/Cygwin, also make sure ``mlr.exe`` is within the ``PATH`` variable. + +.. image:: pix/miller-msys.png + +Differences +---------------------------------------------------------------- + +:doc:`output-colorization` doesn't work on Windows, outside of MSYS2. + +The Windows-support code within Miller makes effort to support Linux/Unix/MacOS-like command-line syntax including single-quoting of expressions for ``mlr put`` and ``mlr filter`` -- and in the examples above, this often works. However, there are still some cases where more complex expressions aren't successfully parsed from the Windows prompt, even though they are from MSYS2: + +.. image:: pix/miller-windows-complex.png + +.. image:: pix/miller-msys-complex.png + +One workaround is to use MSYS2. Another is to put more complex DSL expressions into a file: + +.. image:: pix/miller-windows-complex-workaround.png diff --git a/docs6/mk-func-h2s.sh b/docs6/mk-func-h2s.sh index 09b5596d1..db7876dae 100755 --- a/docs6/mk-func-h2s.sh +++ b/docs6/mk-func-h2s.sh @@ -27,12 +27,24 @@ mlr -F | grep -v '^[a-zA-Z]' | uniq | while read funcname; do elif [ "$funcname" = ':' ]; then displayname='\:' linkname='colon' + elif [ "$funcname" = '!' ]; then + displayname='\!' + linkname='colon' elif [ "$funcname" = '? :' ]; then displayname='\?' linkname='question-mark-colon' elif [ "$funcname" = '?:' ]; then displayname='\?' linkname='question-mark-colon' + elif [ "$funcname" = '>>' ]; then + displayname='\>\>' + linkname='srsh' + elif [ "$funcname" = '>>>' ]; then + displayname='\>\>\>' + linkname='ursh' + elif [ "$funcname" = '>>>=' ]; then + displayname='\>\>\>=' + linkname='ursheq' fi echo "" @@ -72,9 +84,21 @@ mlr -F | grep '^[a-zA-Z]' | sort -u | while read funcname; do elif [ "$funcname" = ':' ]; then displayname='\:' linkname='colon' + elif [ "$funcname" = '!' ]; then + displayname='\!' + linkname='colon' elif [ "$funcname" = '? :' ]; then displayname='\?' linkname='question-mark-colon' + elif [ "$funcname" = '>>' ]; then + displayname='\>\>' + linkname='srsh' + elif [ "$funcname" = '>>>' ]; then + displayname='\>\>\>' + linkname='ursh' + elif [ "$funcname" = '>>>=' ]; then + displayname='\>\>\>=' + linkname='ursheq' fi echo "" diff --git a/docs6/new-in-miller-6.rst b/docs6/new-in-miller-6.rst index b68bfb638..4dd5b2e75 100644 --- a/docs6/new-in-miller-6.rst +++ b/docs6/new-in-miller-6.rst @@ -4,44 +4,76 @@ What's new in Miller 6 ================================================================ -[doc is WIP] +See also https://github.com/johnkerl/miller/labels/go-port -* Completely reworked documentation (here) and on-line help (``mlr --help``) -* Arrays in the ``put``/``filter`` DSL +Documentation improvements +---------------------------------------------------------------- - * also ``array`` is now a keyword so you can't use that as a local-variable or UDF name +Documentation (here) and on-line help (``mlr --help``) has been completely reworked. -* JSON: +* Intro material is pulled out front, no longer assuming users are comparing to tools like AWK. +* Since CSV is the file format most used by Miller users, it is discussed first, and more examples use CSV. +* There is more focus on simpler use-cases, and detailed how-to material has been moved further down. +* Long reference sections have been split up. - * Improved JSON support - * Streamable JSON parsing +JSON support, and arrays +---------------------------------------------------------------- -* Full* support for Windows +Arrays are now supported in Miller's ``put``/``filter`` programming language, as described at :doc:`reference-dsl-arrays`. Also, ``array`` is now a keyword so this is no longer usable as a local-variable or UDF name. - * Make a Windows docpage +JSON support is improved: + +* Direct support for arrays means that you can now use Miller to process more JSON files. +* Streamable JSON parsing: Miller's internal record-processing pipeline starts as soon as the first record is read (which was already the case for other file formats). This means that, unless records are wrapped with outermost ``[...]``, Miller now handles JSON in ``tail -f`` contexts like it does for other file formats. +* Flatten/unflatten -- TODO pick a name and link to a separate page/section + +Improved Windows experience +---------------------------------------------------------------- + +Stronger support for Windows (with or without MSYS2), with a couple of exceptions. See :doc:`miller-on-windows` for more information. + +Binaries are reliably available using GitHub Actions: see also :doc:`installation`. + +In-process support for compressed input +---------------------------------------------------------------- + +In addition to ``--prepipe gunzip``, you can now use the ``--gzin`` flag, and similarly for BZIP files. In fact, Miller will autodetect (by file extension) and automatically uncompress ``mlr --csv cat foo.csv.gz``. Please see section [TODO:linkify] for more information. + +TODO: type up + +Output colorization +---------------------------------------------------------------- + +See :doc:`output-colorization`. + +New DSL functions / operators +---------------------------------------------------------------- + +* String-hashing functions :ref:`reference-dsl-md5`, :ref:`reference-dsl-sha1`, :ref:`reference-dsl-sha256`, and :ref:`reference-dsl-sha512` +* Platform-property functions :ref:`reference-dsl-hostname`, :ref:`reference-dsl-os`, and :ref:`reference-dsl-version` +* Unsigned right-shift :ref:`reference-dsl-ursh` along with ``>>>=`` + +Improved command-line parsing +---------------------------------------------------------------- + +Getoptish command-line parsing (https://github.com/johnkerl/miller/pull/467): + +* ``-xyz`` expands automatically to ``-x -y -z``, so (for example) ``mlr cut -of shape,flag`` is the same as ``mlr cut -o -f shape,flag``. +* ``--foo=bar`` expands automatically to ``--foo bar``, so (for example) ``mlr --ifs=comma`` is the same as ``mlr --ifs comma``. +* ``--mfrom``, ``--load``, ``--mload`` as described at [TODO:linkify]. + +To file +---------------------------------------------------------------- -* Build artifacts (binaries) using GitHub Actions -* In-process support for compressed input -* Built-in ``md5`` ``sha1`` etc -* ``hostname`` ``os`` ``version`` functions * Input-preservation -- find a way to describe this -- link to the issue ... * REPL TBD -* :doc:`output-colorization` * Minor: - * Getoptish (#467) - - * ``-xyz`` to ``-x -y -z`` -- show an example. And back-compat for ``sort`` - * ``--foo=bar`` to ``--foo bar`` -- show an example. - - * ``--mfrom``, ``--load``, ``--mload`` * Better syntax-error messages for the DSL, including line number * ``substr0`` and ``substr1``, and why - * ``>>>`` and ``>>>=`` - * Completely reworked regression-testing -* Dev: ported to Go +Developer-specific aspects +---------------------------------------------------------------- - * Developer notes: https://github.com/johnkerl/miller/blob/main/go/README.md - -See also https://github.com/johnkerl/miller/issues/372 +* Miller has been ported from C to Go. Developer notes: https://github.com/johnkerl/miller/blob/main/go/README.md +* Completely reworked regression testing, including running on Windows diff --git a/docs6/new-in-miller-6.rst.in b/docs6/new-in-miller-6.rst.in index 973ae193c..e7886aa5e 100644 --- a/docs6/new-in-miller-6.rst.in +++ b/docs6/new-in-miller-6.rst.in @@ -1,44 +1,74 @@ What's new in Miller 6 ================================================================ -[doc is WIP] +See also https://github.com/johnkerl/miller/labels/go-port -* Completely reworked documentation (here) and on-line help (``mlr --help``) -* Arrays in the ``put``/``filter`` DSL +Documentation improvements +---------------------------------------------------------------- - * also ``array`` is now a keyword so you can't use that as a local-variable or UDF name +Documentation (here) and on-line help (``mlr --help``) has been completely reworked. -* JSON: +* Intro material is pulled out front, no longer assuming users are comparing to tools like AWK. +* Since CSV is the file format most used by Miller users, it is discussed first, and more examples use CSV. +* There is more focus on simpler use-cases, and detailed how-to material has been moved further down. +* Long reference sections have been split up. - * Improved JSON support - * Streamable JSON parsing +JSON support, and arrays +---------------------------------------------------------------- -* Full* support for Windows +Arrays are now supported in Miller's ``put``/``filter`` programming language, as described at :doc:`reference-dsl-arrays`. Also, ``array`` is now a keyword so this is no longer usable as a local-variable or UDF name. - * Make a Windows docpage +JSON support is improved: + +* Direct support for arrays means that you can now use Miller to process more JSON files. +* Streamable JSON parsing: Miller's internal record-processing pipeline starts as soon as the first record is read (which was already the case for other file formats). This means that, unless records are wrapped with outermost ``[...]``, Miller now handles JSON in ``tail -f`` contexts like it does for other file formats. +* Flatten/unflatten -- TODO pick a name and link to a separate page/section + +Improved Windows experience +---------------------------------------------------------------- + +Stronger support for Windows (with or without MSYS2), with a couple of exceptions. See :doc:`miller-on-windows` for more information. + +Binaries are reliably available using GitHub Actions: see also :doc:`installation`. + +In-process support for compressed input +---------------------------------------------------------------- + +In addition to ``--prepipe gunzip``, you can now use the ``--gzin`` flag, and similarly for BZIP files. In fact, Miller will autodetect (by file extension) and automatically uncompress ``mlr --csv cat foo.csv.gz``. Please see section [TODO:linkify] for more information. + +Output colorization +---------------------------------------------------------------- + +See :doc:`output-colorization`. + +New DSL functions / operators +---------------------------------------------------------------- + +* String-hashing functions :ref:`reference-dsl-md5`, :ref:`reference-dsl-sha1`, :ref:`reference-dsl-sha256`, and :ref:`reference-dsl-sha512` +* Platform-property functions :ref:`reference-dsl-hostname`, :ref:`reference-dsl-os`, and :ref:`reference-dsl-version` +* Unsigned right-shift :ref:`reference-dsl-ursh` along with ``>>>=`` + +Improved command-line parsing +---------------------------------------------------------------- + +Getoptish command-line parsing (https://github.com/johnkerl/miller/pull/467): + +* ``-xyz`` expands automatically to ``-x -y -z``, so (for example) ``mlr cut -of shape,flag`` is the same as ``mlr cut -o -f shape,flag``. +* ``--foo=bar`` expands automatically to ``--foo bar``, so (for example) ``mlr --ifs=comma`` is the same as ``mlr --ifs comma``. +* ``--mfrom``, ``--load``, ``--mload`` as described at [TODO:linkify]. + +To file +---------------------------------------------------------------- -* Build artifacts (binaries) using GitHub Actions -* In-process support for compressed input -* Built-in ``md5`` ``sha1`` etc -* ``hostname`` ``os`` ``version`` functions * Input-preservation -- find a way to describe this -- link to the issue ... * REPL TBD -* :doc:`output-colorization` * Minor: - * Getoptish (#467) - - * ``-xyz`` to ``-x -y -z`` -- show an example. And back-compat for ``sort`` - * ``--foo=bar`` to ``--foo bar`` -- show an example. - - * ``--mfrom``, ``--load``, ``--mload`` * Better syntax-error messages for the DSL, including line number * ``substr0`` and ``substr1``, and why - * ``>>>`` and ``>>>=`` - * Completely reworked regression-testing -* Dev: ported to Go +Developer-specific aspects +---------------------------------------------------------------- - * Developer notes: https://github.com/johnkerl/miller/blob/main/go/README.md - -See also https://github.com/johnkerl/miller/issues/372 +* Miller has been ported from C to Go. Developer notes: https://github.com/johnkerl/miller/blob/main/go/README.md +* Completely reworked regression testing, including running on Windows diff --git a/docs6/pix/miller-msys-complex.png b/docs6/pix/miller-msys-complex.png new file mode 100644 index 000000000..ac9bb9512 Binary files /dev/null and b/docs6/pix/miller-msys-complex.png differ diff --git a/docs6/pix/miller-msys.png b/docs6/pix/miller-msys.png new file mode 100644 index 000000000..a1e918ee9 Binary files /dev/null and b/docs6/pix/miller-msys.png differ diff --git a/docs6/pix/miller-windows-complex-workaround.png b/docs6/pix/miller-windows-complex-workaround.png new file mode 100644 index 000000000..c8d06015f Binary files /dev/null and b/docs6/pix/miller-windows-complex-workaround.png differ diff --git a/docs6/pix/miller-windows-complex.png b/docs6/pix/miller-windows-complex.png new file mode 100644 index 000000000..7b6217b14 Binary files /dev/null and b/docs6/pix/miller-windows-complex.png differ diff --git a/docs6/pix/miller-windows.png b/docs6/pix/miller-windows.png new file mode 100644 index 000000000..46e1a155b Binary files /dev/null and b/docs6/pix/miller-windows.png differ diff --git a/docs6/reference-dsl-arrays.rst b/docs6/reference-dsl-arrays.rst new file mode 100644 index 000000000..d4ab4ff44 --- /dev/null +++ b/docs6/reference-dsl-arrays.rst @@ -0,0 +1,20 @@ +.. + PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE. + +Reference: arrays +================================================================ + +TODO + +.. code-block:: none + :emphasize-lines: 1-1 + + $ mlr --json cat data/array-example.json + { + "key": "ax04", + "samples": [45, 67, 22] + } + { + "key": "cz09", + "samples": [11, 29, 84, 91] + } diff --git a/docs6/reference-dsl-arrays.rst.in b/docs6/reference-dsl-arrays.rst.in new file mode 100644 index 000000000..f006f445d --- /dev/null +++ b/docs6/reference-dsl-arrays.rst.in @@ -0,0 +1,8 @@ +Reference: arrays +================================================================ + +TODO + +GENRST_RUN_COMMAND +mlr --json cat data/array-example.json +GENRST_EOF diff --git a/docs6/reference-dsl-builtin-functions.rst b/docs6/reference-dsl-builtin-functions.rst index 1f20f8447..90ab42163 100644 --- a/docs6/reference-dsl-builtin-functions.rst +++ b/docs6/reference-dsl-builtin-functions.rst @@ -195,9 +195,9 @@ You can get a list of all functions using **mlr -F**. -.. _reference-dsl->>: +.. _reference-dsl-srsh: ->> +\>\> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: none @@ -206,9 +206,9 @@ You can get a list of all functions using **mlr -F**. -.. _reference-dsl->>>: +.. _reference-dsl-ursh: ->>> +\>\>\> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: none @@ -217,9 +217,9 @@ You can get a list of all functions using **mlr -F**. -.. _reference-dsl-!: +.. _reference-dsl-colon: -! +\! ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: none diff --git a/docs6/reference-main-online-help.rst b/docs6/reference-main-online-help.rst index 5a132a1b5..1280f3cae 100644 --- a/docs6/reference-main-online-help.rst +++ b/docs6/reference-main-online-help.rst @@ -38,26 +38,15 @@ Examples: mlr --from infile.dat put '(NR % 1000 == 0) { print > os.Stderr, "Checkpoint ".NR}' DATA-FORMAT EXAMPLES: - DKVP: delimited key-value pairs (Miller default format) - +---------------------+ - | apple=1,bat=2,cog=3 | Record 1: "apple" => "1", "bat" => "2", "cog" => "3" - | dish=7,egg=8,flint | Record 2: "dish" => "7", "egg" => "8", "3" => "flint" - +---------------------+ - - NIDX: implicitly numerically indexed (Unix-toolkit style) - +---------------------+ - | the quick brown | Record 1: "1" => "the", "2" => "quick", "3" => "brown" - | fox jumped | Record 2: "1" => "fox", "2" => "jumped" - +---------------------+ - - CSV/CSV-lite: comma-separated values with separate header line + CSV/CSV-lite: comma-separated values with separate header line + TSV: same but with tabs in places of commas +---------------------+ | apple,bat,cog | | 1,2,3 | Record 1: "apple => "1", "bat" => "2", "cog" => "3" | 4,5,6 | Record 2: "apple" => "4", "bat" => "5", "cog" => "6" +---------------------+ - Tabular JSON: nested objects are supported, although arrays within them are not: + JSON (sequence or array of objects): +---------------------+ | { | | "apple": 1, | Record 1: "apple" => "1", "bat" => "2", "cog" => "3" @@ -80,6 +69,14 @@ Examples: | 4 5 6 | Record 2: "apple" => "4", "bat" => "5", "cog" => "6" +---------------------+ + Markdown tabular (supported for output only): + +-----------------------+ + | | apple | bat | cog | | + | | --- | --- | --- | | + | | 1 | 2 | 3 | | Record 1: "apple => "1", "bat" => "2", "cog" => "3" + | | 4 | 5 | 6 | | Record 2: "apple" => "4", "bat" => "5", "cog" => "6" + +-----------------------+ + XTAB: pretty-printed transposed tabular +---------------------+ | apple 1 | Record 1: "apple" => "1", "bat" => "2", "cog" => "3" @@ -90,13 +87,17 @@ Examples: | egg 8 | +---------------------+ - Markdown tabular (supported for output only): - +-----------------------+ - | | apple | bat | cog | | - | | --- | --- | --- | | - | | 1 | 2 | 3 | | Record 1: "apple => "1", "bat" => "2", "cog" => "3" - | | 4 | 5 | 6 | | Record 2: "apple" => "4", "bat" => "5", "cog" => "6" - +-----------------------+ + DKVP: delimited key-value pairs (Miller default format) + +---------------------+ + | apple=1,bat=2,cog=3 | Record 1: "apple" => "1", "bat" => "2", "cog" => "3" + | dish=7,egg=8,flint | Record 2: "dish" => "7", "egg" => "8", "3" => "flint" + +---------------------+ + + NIDX: implicitly numerically indexed (Unix-toolkit style) + +---------------------+ + | the quick brown | Record 1: "1" => "the", "2" => "quick", "3" => "brown" + | fox jumped | Record 2: "1" => "fox", "2" => "jumped" + +---------------------+ HELP OPTIONS: -h or --help Show this message. diff --git a/docs6/reference-main-overview.rst b/docs6/reference-main-overview.rst index 4350043bc..6ce797f20 100644 --- a/docs6/reference-main-overview.rst +++ b/docs6/reference-main-overview.rst @@ -7,29 +7,7 @@ Reference: Miller commands Overview ---------------------------------------------------------------- -TODO: push this into reference-verbs overview, and make this page an overview of miller invocations - -Whereas the Unix toolkit is made of the separate executables ``cat``, ``tail``, ``cut``, -``sort``, etc., Miller has subcommands, or **verbs**, invoked as follows: - -.. code-block:: none - - mlr tac *.dat - mlr cut --complement -f os_version *.dat - mlr sort -f hostname,uptime *.dat - -These fall into categories as follows: - -* Analogs of their Unix-toolkit namesakes, discussed below as well as in :doc:`feature-comparison`: :ref:`reference-verbs-cat`, :ref:`reference-verbs-cut`, :ref:`reference-verbs-grep`, :ref:`reference-verbs-head`, :ref:`reference-verbs-join`, :ref:`reference-verbs-sort`, :ref:`reference-verbs-tac`, :ref:`reference-verbs-tail`, :ref:`reference-verbs-top`, :ref:`reference-verbs-uniq`. - -* ``awk``-like functionality: :ref:`reference-verbs-filter`, :ref:`reference-verbs-put`, :ref:`reference-verbs-sec2gmt`, :ref:`reference-verbs-sec2gmtdate`, :ref:`reference-verbs-step`, :ref:`reference-verbs-tee`. - -* Statistically oriented: :ref:`reference-verbs-bar`, :ref:`reference-verbs-bootstrap`, :ref:`reference-verbs-decimate`, :ref:`reference-verbs-histogram`, :ref:`reference-verbs-least-frequent`, :ref:`reference-verbs-most-frequent`, :ref:`reference-verbs-sample`, :ref:`reference-verbs-shuffle`, :ref:`reference-verbs-stats1`, :ref:`reference-verbs-stats2`. - -* Particularly oriented toward :doc:`record-heterogeneity`, although all Miller commands can handle heterogeneous records: :ref:`reference-verbs-group-by`, :ref:`reference-verbs-group-like`, :ref:`reference-verbs-having-fields`. - -* These draw from other sources (see also :doc:`originality`): :ref:`reference-verbs-count-distinct` is SQL-ish, and :ref:`reference-verbs-rename` can be done by ``sed`` (which does it faster: see :doc:`performance`. Verbs: :ref:`reference-verbs-check`, :ref:`reference-verbs-count-distinct`, :ref:`reference-verbs-label`, :ref:`reference-verbs-merge-fields`, :ref:`reference-verbs-nest`, :ref:`reference-verbs-nothing`, :ref:`reference-verbs-regularize`, :ref:`reference-verbs-rename`, :ref:`reference-verbs-reorder`, :ref:`reference-verbs-reshape`, :ref:`reference-verbs-seqgen`. - +TODO: overview of miller invocations Verbs vs DSL ---------------------------------------------------------------- @@ -66,8 +44,8 @@ Example: a=eks,x_sum=1.1400793586611044 a=wye,x_sum=0.7778922255683036 -* You get to write your own DSL expressions +* You get to write your own expressions * They run a bit slower * They take more keystrokes -* There is more to learn -* They are highly customizable +* There's more to learn +* They're highly customizable diff --git a/docs6/reference-main-overview.rst.in b/docs6/reference-main-overview.rst.in index 334d1abac..161c79208 100644 --- a/docs6/reference-main-overview.rst.in +++ b/docs6/reference-main-overview.rst.in @@ -4,25 +4,7 @@ Reference: Miller commands Overview ---------------------------------------------------------------- -TODO: push this into reference-verbs overview, and make this page an overview of miller invocations - -Whereas the Unix toolkit is made of the separate executables ``cat``, ``tail``, ``cut``, -``sort``, etc., Miller has subcommands, or **verbs**, invoked as follows: - -GENRST_INCLUDE_ESCAPED(data/subcommand-example.txt) - -These fall into categories as follows: - -* Analogs of their Unix-toolkit namesakes, discussed below as well as in :doc:`feature-comparison`: :ref:`reference-verbs-cat`, :ref:`reference-verbs-cut`, :ref:`reference-verbs-grep`, :ref:`reference-verbs-head`, :ref:`reference-verbs-join`, :ref:`reference-verbs-sort`, :ref:`reference-verbs-tac`, :ref:`reference-verbs-tail`, :ref:`reference-verbs-top`, :ref:`reference-verbs-uniq`. - -* ``awk``-like functionality: :ref:`reference-verbs-filter`, :ref:`reference-verbs-put`, :ref:`reference-verbs-sec2gmt`, :ref:`reference-verbs-sec2gmtdate`, :ref:`reference-verbs-step`, :ref:`reference-verbs-tee`. - -* Statistically oriented: :ref:`reference-verbs-bar`, :ref:`reference-verbs-bootstrap`, :ref:`reference-verbs-decimate`, :ref:`reference-verbs-histogram`, :ref:`reference-verbs-least-frequent`, :ref:`reference-verbs-most-frequent`, :ref:`reference-verbs-sample`, :ref:`reference-verbs-shuffle`, :ref:`reference-verbs-stats1`, :ref:`reference-verbs-stats2`. - -* Particularly oriented toward :doc:`record-heterogeneity`, although all Miller commands can handle heterogeneous records: :ref:`reference-verbs-group-by`, :ref:`reference-verbs-group-like`, :ref:`reference-verbs-having-fields`. - -* These draw from other sources (see also :doc:`originality`): :ref:`reference-verbs-count-distinct` is SQL-ish, and :ref:`reference-verbs-rename` can be done by ``sed`` (which does it faster: see :doc:`performance`. Verbs: :ref:`reference-verbs-check`, :ref:`reference-verbs-count-distinct`, :ref:`reference-verbs-label`, :ref:`reference-verbs-merge-fields`, :ref:`reference-verbs-nest`, :ref:`reference-verbs-nothing`, :ref:`reference-verbs-regularize`, :ref:`reference-verbs-rename`, :ref:`reference-verbs-reorder`, :ref:`reference-verbs-reshape`, :ref:`reference-verbs-seqgen`. - +TODO: overview of miller invocations Verbs vs DSL ---------------------------------------------------------------- @@ -51,8 +33,8 @@ GENRST_RUN_COMMAND mlr put -q '@x_sum[$a] += $x; end{emit @x_sum, "a"}' data/small GENRST_EOF -* You get to write your own DSL expressions +* You get to write your own expressions * They run a bit slower * They take more keystrokes -* There is more to learn -* They are highly customizable +* There's more to learn +* They're highly customizable diff --git a/docs6/reference-verbs.rst b/docs6/reference-verbs.rst index a8cf00c0c..9d412a39f 100644 --- a/docs6/reference-verbs.rst +++ b/docs6/reference-verbs.rst @@ -4,6 +4,30 @@ Reference: List of verbs ================================================================ +Overview +---------------------------------------------------------------- + +Whereas the Unix toolkit is made of the separate executables ``cat``, ``tail``, ``cut``, +``sort``, etc., Miller has subcommands, or **verbs**, invoked as follows: + +.. code-block:: none + + mlr tac *.dat + mlr cut --complement -f os_version *.dat + mlr sort -f hostname,uptime *.dat + +These fall into categories as follows: + +* Analogs of their Unix-toolkit namesakes, discussed below as well as in :doc:`feature-comparison`: :ref:`reference-verbs-cat`, :ref:`reference-verbs-cut`, :ref:`reference-verbs-grep`, :ref:`reference-verbs-head`, :ref:`reference-verbs-join`, :ref:`reference-verbs-sort`, :ref:`reference-verbs-tac`, :ref:`reference-verbs-tail`, :ref:`reference-verbs-top`, :ref:`reference-verbs-uniq`. + +* ``awk``-like functionality: :ref:`reference-verbs-filter`, :ref:`reference-verbs-put`, :ref:`reference-verbs-sec2gmt`, :ref:`reference-verbs-sec2gmtdate`, :ref:`reference-verbs-step`, :ref:`reference-verbs-tee`. + +* Statistically oriented: :ref:`reference-verbs-bar`, :ref:`reference-verbs-bootstrap`, :ref:`reference-verbs-decimate`, :ref:`reference-verbs-histogram`, :ref:`reference-verbs-least-frequent`, :ref:`reference-verbs-most-frequent`, :ref:`reference-verbs-sample`, :ref:`reference-verbs-shuffle`, :ref:`reference-verbs-stats1`, :ref:`reference-verbs-stats2`. + +* Particularly oriented toward :doc:`record-heterogeneity`, although all Miller commands can handle heterogeneous records: :ref:`reference-verbs-group-by`, :ref:`reference-verbs-group-like`, :ref:`reference-verbs-having-fields`. + +* These draw from other sources (see also :doc:`originality`): :ref:`reference-verbs-count-distinct` is SQL-ish, and :ref:`reference-verbs-rename` can be done by ``sed`` (which does it faster: see :doc:`performance`. Verbs: :ref:`reference-verbs-check`, :ref:`reference-verbs-count-distinct`, :ref:`reference-verbs-label`, :ref:`reference-verbs-merge-fields`, :ref:`reference-verbs-nest`, :ref:`reference-verbs-nothing`, :ref:`reference-verbs-regularize`, :ref:`reference-verbs-rename`, :ref:`reference-verbs-reorder`, :ref:`reference-verbs-reshape`, :ref:`reference-verbs-seqgen`. + .. _reference-verbs-altkv: altkv @@ -1817,8 +1841,8 @@ most-frequent square red triangle red circle red - square blue square yellow + square blue See also :ref:`reference-verbs-least-frequent`. diff --git a/docs6/reference-verbs.rst.in b/docs6/reference-verbs.rst.in index 6dcee47c6..7bf32dd28 100644 --- a/docs6/reference-verbs.rst.in +++ b/docs6/reference-verbs.rst.in @@ -1,6 +1,26 @@ Reference: List of verbs ================================================================ +Overview +---------------------------------------------------------------- + +Whereas the Unix toolkit is made of the separate executables ``cat``, ``tail``, ``cut``, +``sort``, etc., Miller has subcommands, or **verbs**, invoked as follows: + +GENRST_INCLUDE_ESCAPED(data/subcommand-example.txt) + +These fall into categories as follows: + +* Analogs of their Unix-toolkit namesakes, discussed below as well as in :doc:`feature-comparison`: :ref:`reference-verbs-cat`, :ref:`reference-verbs-cut`, :ref:`reference-verbs-grep`, :ref:`reference-verbs-head`, :ref:`reference-verbs-join`, :ref:`reference-verbs-sort`, :ref:`reference-verbs-tac`, :ref:`reference-verbs-tail`, :ref:`reference-verbs-top`, :ref:`reference-verbs-uniq`. + +* ``awk``-like functionality: :ref:`reference-verbs-filter`, :ref:`reference-verbs-put`, :ref:`reference-verbs-sec2gmt`, :ref:`reference-verbs-sec2gmtdate`, :ref:`reference-verbs-step`, :ref:`reference-verbs-tee`. + +* Statistically oriented: :ref:`reference-verbs-bar`, :ref:`reference-verbs-bootstrap`, :ref:`reference-verbs-decimate`, :ref:`reference-verbs-histogram`, :ref:`reference-verbs-least-frequent`, :ref:`reference-verbs-most-frequent`, :ref:`reference-verbs-sample`, :ref:`reference-verbs-shuffle`, :ref:`reference-verbs-stats1`, :ref:`reference-verbs-stats2`. + +* Particularly oriented toward :doc:`record-heterogeneity`, although all Miller commands can handle heterogeneous records: :ref:`reference-verbs-group-by`, :ref:`reference-verbs-group-like`, :ref:`reference-verbs-having-fields`. + +* These draw from other sources (see also :doc:`originality`): :ref:`reference-verbs-count-distinct` is SQL-ish, and :ref:`reference-verbs-rename` can be done by ``sed`` (which does it faster: see :doc:`performance`. Verbs: :ref:`reference-verbs-check`, :ref:`reference-verbs-count-distinct`, :ref:`reference-verbs-label`, :ref:`reference-verbs-merge-fields`, :ref:`reference-verbs-nest`, :ref:`reference-verbs-nothing`, :ref:`reference-verbs-regularize`, :ref:`reference-verbs-rename`, :ref:`reference-verbs-reorder`, :ref:`reference-verbs-reshape`, :ref:`reference-verbs-seqgen`. + .. _reference-verbs-altkv: altkv diff --git a/go/help.txt b/go/help.txt index 23d0f2ea0..fcbd0cbc7 100644 --- a/go/help.txt +++ b/go/help.txt @@ -1,3 +1,91 @@ +================================================================ +manpage: + +mlr --usage-synopsis +mlr --version +mlr --usage-examples +mlr --usage-data-format-examples +mlr --usage-help-options +mlr --usage-list-all-verbs +mlr --usage-functions +mlr --usage-data-format-options +mlr --usage-comments-in-data +mlr --usage-format-conversion-keystroke-saver-options +mlr --usage-compressed-data-options +mlr --usage-separator-options +mlr --usage-csv-options +mlr --usage-double-quoting +mlr --usage-numerical-formatting +mlr --usage-other-options +mlr --usage-then-chaining +mlr --usage-auxents +mlr --list-all-verbs-raw +mlr #{verb} -h +mlr --list-all-functions-raw +mlr --help-function '#{function} +mlr --list-all-keywords-raw` +mlr --help-keyword '#{keyword}' + +---------------------------------------------------------------- +old mlr --help: + +mainUsageSynopsis(o, argv0) +mainUsageExamples(o, argv0, " ") +mainUsageDataFormatExamples(o, argv0) +mainUsageHelpOptions(o, argv0) +mainUsageMlrrc(o, argv0) +listAllVerbs(o, " ") +mainUsageFunctions(o) +mainUsageDataFormatOptions(o, argv0) +mainUsageCommentsInData(o, argv0) +mainUsageFormatConversionKeystrokeSaverOptions(o, argv0) +mainUsageCompressedDataOptions(o, argv0) +mainUsageSeparatorOptions(o, argv0); +mainUsageCsvOptions(o, argv0) +mainUsageDoubleQuoting(o, argv0); +mainUsageNumericalFormatting(o, argv0) +mainUsageOutputColorization(o, argv0) +mainUsageOtherOptions(o, argv0) +mainUsageThenChaining(o, argv0) +mainUsageAuxents(o) +mainUsageSeeAlso(o, argv0) + +---------------------------------------------------------------- +new -- ?!? + +mlr --help += mlr help +< mlr --usage-synopsis + +mlr help {foo} w/ lookup -- ? + +mlr --version += mlr version + +mlr --usage-examples +mlr --usage-data-format-examples +mlr --usage-help-options +mlr --usage-list-all-verbs +mlr --usage-functions +mlr --usage-data-format-options +mlr --usage-comments-in-data +mlr --usage-format-conversion-keystroke-saver-options +mlr --usage-compressed-data-options +mlr --usage-separator-options +mlr --usage-csv-options +mlr --usage-double-quoting +mlr --usage-numerical-formatting +mlr --usage-other-options +mlr --usage-then-chaining +mlr --usage-auxents +mlr --list-all-verbs-raw +mlr #{verb} -h +mlr --list-all-functions-raw +mlr --help-function '#{function} +mlr --list-all-keywords-raw` +mlr --help-keyword '#{keyword}' + +================================================================ Usage: mlr [I/O options] {verb} [verb-dependent options ...] {zero or more file names} ---------------------------------------------------------------- diff --git a/go/src/auxents/regtest/regtester.go b/go/src/auxents/regtest/regtester.go index b004fa547..cc62df263 100644 --- a/go/src/auxents/regtest/regtester.go +++ b/go/src/auxents/regtest/regtester.go @@ -28,6 +28,10 @@ const ShouldFailName = "should-fail" const MajorSeparator = "================================================================" const MinorSeparator = "----------------------------------------------------------------" +// Don't unset MLR_PASS_COLOR or MLR_FAIL_COLOR -- if people want to change the +// output-coloring used by this regression-tester, we should let them. We +// should only unset environment variables which can cause functional tests to +// fail. var envVarsToUnset = []string{ "MLRRC", "MLR_KEY_COLOR", diff --git a/go/src/cli/mlrcli_usage.go b/go/src/cli/mlrcli_usage.go index 66e80a141..5d609a9cf 100644 --- a/go/src/cli/mlrcli_usage.go +++ b/go/src/cli/mlrcli_usage.go @@ -185,26 +185,15 @@ func mainUsageFunctions(o *os.File) { // TODO: rid of argv0 throughout, replacing with lib.MlrExeName() func mainUsageDataFormatExamples(o *os.File, argv0 string) { fmt.Fprintf(o, - ` DKVP: delimited key-value pairs (Miller default format) - +---------------------+ - | apple=1,bat=2,cog=3 | Record 1: "apple" => "1", "bat" => "2", "cog" => "3" - | dish=7,egg=8,flint | Record 2: "dish" => "7", "egg" => "8", "3" => "flint" - +---------------------+ - - NIDX: implicitly numerically indexed (Unix-toolkit style) - +---------------------+ - | the quick brown | Record 1: "1" => "the", "2" => "quick", "3" => "brown" - | fox jumped | Record 2: "1" => "fox", "2" => "jumped" - +---------------------+ - - CSV/CSV-lite: comma-separated values with separate header line + ` CSV/CSV-lite: comma-separated values with separate header line + TSV: same but with tabs in places of commas +---------------------+ | apple,bat,cog | | 1,2,3 | Record 1: "apple => "1", "bat" => "2", "cog" => "3" | 4,5,6 | Record 2: "apple" => "4", "bat" => "5", "cog" => "6" +---------------------+ - Tabular JSON: nested objects are supported, although arrays within them are not: + JSON (sequence or array of objects): +---------------------+ | { | | "apple": 1, | Record 1: "apple" => "1", "bat" => "2", "cog" => "3" @@ -227,6 +216,14 @@ func mainUsageDataFormatExamples(o *os.File, argv0 string) { | 4 5 6 | Record 2: "apple" => "4", "bat" => "5", "cog" => "6" +---------------------+ + Markdown tabular (supported for output only): + +-----------------------+ + | | apple | bat | cog | | + | | --- | --- | --- | | + | | 1 | 2 | 3 | | Record 1: "apple => "1", "bat" => "2", "cog" => "3" + | | 4 | 5 | 6 | | Record 2: "apple" => "4", "bat" => "5", "cog" => "6" + +-----------------------+ + XTAB: pretty-printed transposed tabular +---------------------+ | apple 1 | Record 1: "apple" => "1", "bat" => "2", "cog" => "3" @@ -237,13 +234,17 @@ func mainUsageDataFormatExamples(o *os.File, argv0 string) { | egg 8 | +---------------------+ - Markdown tabular (supported for output only): - +-----------------------+ - | | apple | bat | cog | | - | | --- | --- | --- | | - | | 1 | 2 | 3 | | Record 1: "apple => "1", "bat" => "2", "cog" => "3" - | | 4 | 5 | 6 | | Record 2: "apple" => "4", "bat" => "5", "cog" => "6" - +-----------------------+ + DKVP: delimited key-value pairs (Miller default format) + +---------------------+ + | apple=1,bat=2,cog=3 | Record 1: "apple" => "1", "bat" => "2", "cog" => "3" + | dish=7,egg=8,flint | Record 2: "dish" => "7", "egg" => "8", "3" => "flint" + +---------------------+ + + NIDX: implicitly numerically indexed (Unix-toolkit style) + +---------------------+ + | the quick brown | Record 1: "1" => "the", "2" => "quick", "3" => "brown" + | fox jumped | Record 2: "1" => "fox", "2" => "jumped" + +---------------------+ `) } diff --git a/go/todo.txt b/go/todo.txt index ffcd0f84e..62554282e 100644 --- a/go/todo.txt +++ b/go/todo.txt @@ -13,10 +13,6 @@ TOP OF LIST: ---------------------------------------------------------------- colorizer: * mrpl output? -* test on: - o mac (various emulators) - o linux (laptop+instances; screen, tmux, various emulators) - o windows (cmd & PS) * REPL PS1 color? ---------------------------------------------------------------- @@ -35,15 +31,18 @@ DOC6 ! fill out: o how-to-help guide - o windows guide o repl guide o env-vars section + o new-in-miller-6: flatten/unflatten + o JSON flatten/unflatten section + o Arrays section including 1-up and why -! thruout pitch the docs at primarily the first whenever possible: - - python/ruby-level user w/o heavy sysadm - - sysadm w/o dev - > do not assume people know awk! XD - - devs +* items: + o file-formats: + o NIDX link to headerless CSV + o JSON re-write re arrays & key-concat -- needs its own section + o format ordering: pick one for sphinx, manpage, olh + o autodetect: tbd * code to-do items: o stats1 --fr @@ -65,6 +64,13 @@ DOC6 * data-cleaning examples -> full section -- ? * c2 & c3 are all esoterica + +! thruout pitch the docs at primarily the first whenever possible: + - python/ruby-level user w/o heavy sysadm + - sysadm w/o dev + > do not assume people know awk! XD + - devs + ---------------------------------------------------------------- * case auxfiles: cat them too diff --git a/man6/Makefile b/man6/Makefile new file mode 100644 index 000000000..5a617929a --- /dev/null +++ b/man6/Makefile @@ -0,0 +1,23 @@ +# Respective MANPATH entries would include /usr/local/share/man or $HOME/man. +# This should be run after make in the ../c directory but before make in the ../docs directory, +# since ../go/mlr is used to autogenerate ./manpage.txt which is used in ../docs. +# See also https://miller.readthedocs.io/en/latest/build.html#creating-a-new-release-for-developers +INSTALLDIR=/usr/local/share/man/man1 +INSTALLHOME=$(HOME)/man/man1 + +top: .always + echo mkman start + mkman.rb > mlr6.1 + mkman.rb | groff -man -Tascii | col -b > manpage.txt + echo mkman end + +# These targets are only for local dev work. +install: + mkdir -p $(INSTALLDIR) + cp mlr6.1 $(INSTALLDIR)/mlr6.1 +installhome: + mkdir -p $(INSTALLHOME) + cp mlr6.1 $(INSTALLHOME)/mlr6.1 + +.always: + true diff --git a/man6/manpage.txt b/man6/manpage.txt new file mode 100644 index 000000000..c04248359 --- /dev/null +++ b/man6/manpage.txt @@ -0,0 +1,2223 @@ +MILLER(1) MILLER(1) + + + +NAME + miller - like awk, sed, cut, join, and sort for name-indexed data such + as CSV and tabular JSON. + +SYNOPSIS + Usage: mlr [I/O options] {verb} [verb-dependent options ...] {zero or + more file names} + + +DESCRIPTION + Miller operates on key-value-pair data while the familiar Unix tools + operate on integer-indexed fields: if the natural data structure for + the latter is the array, then Miller's natural data structure is the + insertion-ordered hash map. This encompasses a variety of data + formats, including but not limited to the familiar CSV, TSV, and JSON. + (Miller can handle positionally-indexed data as a special case.) This + manpage documents Miller v6.0.0-dev. + +EXAMPLES + COMMAND-LINE SYNTAX + mlr --csv cut -f hostname,uptime mydata.csv + mlr --tsv --rs lf filter '$status != "down" && $upsec >= 10000' *.tsv + mlr --nidx put '$sum = $7 < 0.0 ? 3.5 : $7 + 2.1*$8' *.dat + grep -v '^#' /etc/group | mlr --ifs : --nidx --opprint label group,pass,gid,member then sort -f group + mlr join -j account_id -f accounts.dat then group-by account_name balances.dat + mlr --json put '$attr = sub($attr, "([0-9]+)_([0-9]+)_.*", "\1:\2")' data/*.json + mlr stats1 -a min,mean,max,p10,p50,p90 -f flag,u,v data/* + mlr stats2 -a linreg-pca -f u,v -g shape data/* + mlr put -q '@sum[$a][$b] += $x; end {emit @sum, "a", "b"}' data/* + mlr --from estimates.tbl put ' + for (k,v in $*) { + if (is_numeric(v) && k =~ "^[t-z].*$") { + $sum += v; $count += 1 + } + } + $mean = $sum / $count # no assignment if count unset' + mlr --from infile.dat put -f analyze.mlr + mlr --from infile.dat put 'tee > "./taps/data-".$a."-".$b, $*' + mlr --from infile.dat put 'tee | "gzip > ./taps/data-".$a."-".$b.".gz", $*' + mlr --from infile.dat put -q '@v=$*; dump | "jq .[]"' + mlr --from infile.dat put '(NR % 1000 == 0) { print > os.Stderr, "Checkpoint ".NR}' + + DATA FORMATS + CSV/CSV-lite: comma-separated values with separate header line + TSV: same but with tabs in places of commas + +---------------------+ + | apple,bat,cog | + | 1,2,3 | Record 1: "apple => "1", "bat" => "2", "cog" => "3" + | 4,5,6 | Record 2: "apple" => "4", "bat" => "5", "cog" => "6" + +---------------------+ + + JSON (sequence or array of objects): + +---------------------+ + | { | + | "apple": 1, | Record 1: "apple" => "1", "bat" => "2", "cog" => "3" + | "bat": 2, | + | "cog": 3 | + | } | + | { | + | "dish": { | Record 2: "dish:egg" => "7", "dish:flint" => "8", "garlic" => "" + | "egg": 7, | + | "flint": 8 | + | }, | + | "garlic": "" | + | } | + +---------------------+ + + PPRINT: pretty-printed tabular + +---------------------+ + | apple bat cog | + | 1 2 3 | Record 1: "apple => "1", "bat" => "2", "cog" => "3" + | 4 5 6 | Record 2: "apple" => "4", "bat" => "5", "cog" => "6" + +---------------------+ + + Markdown tabular (supported for output only): + +-----------------------+ + | | apple | bat | cog | | + | | --- | --- | --- | | + | | 1 | 2 | 3 | | Record 1: "apple => "1", "bat" => "2", "cog" => "3" + | | 4 | 5 | 6 | | Record 2: "apple" => "4", "bat" => "5", "cog" => "6" + +-----------------------+ + + XTAB: pretty-printed transposed tabular + +---------------------+ + | apple 1 | Record 1: "apple" => "1", "bat" => "2", "cog" => "3" + | bat 2 | + | cog 3 | + | | + | dish 7 | Record 2: "dish" => "7", "egg" => "8" + | egg 8 | + +---------------------+ + + DKVP: delimited key-value pairs (Miller default format) + +---------------------+ + | apple=1,bat=2,cog=3 | Record 1: "apple" => "1", "bat" => "2", "cog" => "3" + | dish=7,egg=8,flint | Record 2: "dish" => "7", "egg" => "8", "3" => "flint" + +---------------------+ + + NIDX: implicitly numerically indexed (Unix-toolkit style) + +---------------------+ + | the quick brown | Record 1: "1" => "the", "2" => "quick", "3" => "brown" + | fox jumped | Record 2: "1" => "fox", "2" => "jumped" + +---------------------+ + +OPTIONS + In the following option flags, the version with "i" designates the + input stream, "o" the output stream, and the version without prefix + sets the option for both input and output stream. For example: --irs + sets the input record separator, --ors the output record separator, and + --rs sets both the input and output separator to the given value. + + HELP OPTIONS + -h or --help Show this message. + --version Show the software version. + {verb name} --help Show verb-specific help. + --help-all-verbs Show help on all verbs. + -l or --list-all-verbs List only verb names. + -L List only verb names, one per line. + -f or --help-all-functions Show help on all built-in functions. + -F Show a bare listing of built-in functions by name. + -k or --help-all-keywords Show help on all keywords. + -K Show a bare listing of keywords by name. + + VERB LIST + altkv bar bootstrap cat check clean-whitespace count-distinct count + count-similar cut decimate fill-down fill-empty filter flatten format-values + fraction gap grep group-by group-like having-fields head histogram json-parse + json-stringify join label least-frequent merge-fields most-frequent nest + nothing put regularize remove-empty-columns rename reorder repeat reshape + sample sec2gmtdate sec2gmt seqgen shuffle skip-trivial-records sort + sort-within-records stats1 stats2 step tac tail tee top unflatten uniq + unsparsify + + FUNCTION LIST + + + - + * + / + // + ** + pow + .+ + .- + .* + ./ + % + ~ + & + | + ^ + << + >> + >>> + bitcount + madd + msub + mmul + mexp + ! + == + != + > + >= + < + <= + =~ + !=~ + && + || + ^^ + ?? + ??? + ?: + . + capitalize + clean_whitespace + collapse_whitespace + gsub + lstrip + regextract + regextract_or_else + rstrip + strip + strlen + ssub + sub + substr0 + substr1 + substr + tolower + toupper + truncate + md5 + sha1 + sha256 + sha512 + abs + acos + acosh + asin + asinh + atan + atan2 + atanh + cbrt + ceil + cos + cosh + erf + erfc + exp + expm1 + floor + invqnorm + log + log10 + log1p + logifit + max + min + qnorm + round + sgn + sin + sinh + sqrt + tan + tanh + roundm + urand + urandint + urandrange + urand32 + gmt2sec + sec2gmt + sec2gmtdate + systime + systimeint + uptime + strftime + strptime + dhms2fsec + dhms2sec + fsec2dhms + fsec2hms + hms2fsec + hms2sec + sec2dhms + sec2hms + is_absent + is_array + is_bool + is_boolean + is_empty + is_empty_map + is_error + is_float + is_int + is_map + is_nonempty_map + is_not_empty + is_not_map + is_not_array + is_not_null + is_null + is_numeric + is_present + is_string + asserting_absent + asserting_array + asserting_bool + asserting_boolean + asserting_error + asserting_empty + asserting_empty_map + asserting_float + asserting_int + asserting_map + asserting_nonempty_map + asserting_not_empty + asserting_not_map + asserting_not_array + asserting_not_null + asserting_null + asserting_numeric + asserting_present + asserting_string + typeof + boolean + float + fmtnum + hexfmt + int + joink + joinv + joinkv + splita + splitax + splitkv + splitkvx + splitnv + splitnvx + string + append + arrayify + depth + flatten + get_keys + get_values + haskey + json_parse + json_stringify + leafcount + length + mapdiff + mapexcept + mapselect + mapsum + unflatten + hostname + os + system + version + Please use "mlr --help-function {function name}" for function-specific help. + + I/O FORMATTING + --idkvp --odkvp --dkvp Delimited key-value pairs, e.g "a=1,b=2" + (this is Miller's default format). + + --inidx --onidx --nidx Implicitly-integer-indexed fields + (Unix-toolkit style). + -T Synonymous with "--nidx --fs tab". + + --icsv --ocsv --csv Comma-separated value (or tab-separated + with --fs tab, etc.) + + --itsv --otsv --tsv Keystroke-savers for "--icsv --ifs tab", + "--ocsv --ofs tab", "--csv --fs tab". + --iasv --oasv --asv Similar but using ASCII FS 0x1f and RS 0x1e\n", + --iusv --ousv --usv Similar but using Unicode FS U+241F (UTF-8 0xe2909f)\n", + and RS U+241E (UTF-8 0xe2909e)\n", + + --icsvlite --ocsvlite --csvlite Comma-separated value (or tab-separated + with --fs tab, etc.). The 'lite' CSV does not handle + RFC-CSV double-quoting rules; is slightly faster; + and handles heterogeneity in the input stream via + empty newline followed by new header line. See also + http://johnkerl.org/miller/doc/file-formats.html#CSV/TSV/etc. + + --itsvlite --otsvlite --tsvlite Keystroke-savers for "--icsvlite --ifs tab", + "--ocsvlite --ofs tab", "--csvlite --fs tab". + -t Synonymous with --tsvlite. + --iasvlite --oasvlite --asvlite Similar to --itsvlite et al. but using ASCII FS 0x1f and RS 0x1e\n", + --iusvlite --ousvlite --usvlite Similar to --itsvlite et al. but using Unicode FS U+241F (UTF-8 0xe2909f)\n", + and RS U+241E (UTF-8 0xe2909e)\n", + + --ipprint --opprint --pprint Pretty-printed tabular (produces no + output until all input is in). + --right Right-justifies all fields for PPRINT output. + --barred Prints a border around PPRINT output + (only available for output). + + --omd Markdown-tabular (only available for output). + + --ixtab --oxtab --xtab Pretty-printed vertical-tabular. + --xvright Right-justifies values for XTAB format. + + --ijson --ojson --json JSON tabular: sequence or list of one-level + maps: {...}{...} or [{...},{...}]. + --json-map-arrays-on-input JSON arrays are unmillerable. --json-map-arrays-on-input + --json-skip-arrays-on-input is the default: arrays are converted to integer-indexed + --json-fatal-arrays-on-input maps. The other two options cause them to be skipped, or + to be treated as errors. Please use the jq tool for full + JSON (pre)processing. + --jvstack Put one key-value pair per line for JSON output. + --no-jvstack Put objects/arrays all on one line for JSON output. + --jsonx --ojsonx Keystroke-savers for --json --jvstack + --jsonx --ojsonx and --ojson --jvstack, respectively. + --jlistwrap Wrap JSON output in outermost [ ]. + --jknquoteint Do not quote non-string map keys in JSON output. + --jvquoteall Quote map values in JSON output, even if they're + numeric. + --oflatsep {string} Separator for flattening multi-level JSON keys, + e.g. '{"a":{"b":3}}' becomes a:b => 3 for + non-JSON formats. Defaults to ..\n", + + -p is a keystroke-saver for --nidx --fs space --repifs + + Examples: --csv for CSV-formatted input and output; --idkvp --opprint for + DKVP-formatted input and pretty-printed output. + + Please use --iformat1 --oformat2 rather than --format1 --oformat2. + The latter sets up input and output flags for format1, not all of which + are overridden in all cases by setting output format to format2. + + + COMMENTS IN DATA + --skip-comments Ignore commented lines (prefixed by "#") + within the input. + --skip-comments-with {string} Ignore commented lines within input, with + specified prefix. + --pass-comments Immediately print commented lines (prefixed by "#") + within the input. + --pass-comments-with {string} Immediately print commented lines within input, with + specified prefix. + Notes: + * Comments are only honored at the start of a line. + * In the absence of any of the above four options, comments are data like + any other text. + * When pass-comments is used, comment lines are written to standard output + immediately upon being read; they are not part of the record stream. + Results may be counterintuitive. A suggestion is to place comments at the + start of data files. + + FORMAT-CONVERSION KEYSTROKE-SAVERS + As keystroke-savers for format-conversion you may use the following: + --c2t --c2d --c2n --c2j --c2x --c2p --c2m + --t2c --t2d --t2n --t2j --t2x --t2p --t2m + --d2c --d2t --d2n --d2j --d2x --d2p --d2m + --n2c --n2t --n2d --n2j --n2x --n2p --n2m + --j2c --j2t --j2d --j2n --j2x --j2p --j2m + --x2c --x2t --x2d --x2n --x2j --x2p --x2m + --p2c --p2t --p2d --p2n --p2j --p2x --p2m + The letters c t d n j x p m refer to formats CSV, TSV, DKVP, NIDX, JSON, XTAB, + PPRINT, and markdown, respectively. Note that markdown format is available for + output only. + + COMPRESSED I/O + Decompression done within the Miller process itself: + --gzin Uncompress gzip within the Miller process. Done by default if file ends in ".gz". + --bz2in Uncompress bz2ip within the Miller process. Done by default if file ends in ".bz2". + --zin Uncompress zlib within the Miller process. Done by default if file ends in ".z". + Decompression done outside the Miller processn --prepipe {command} You can, of course, already do without this for single input files, + e.g. "gunzip < myfile.csv.gz | mlr ...". + However, when multiple input files are present, between-file separations are + lost; also, the FILENAME variable doesn't iterate. Using --prepipe you can + specify an action to be taken on each input file. This prepipe command must + be able to read from standard input; it will be invoked with + {command} < {filename}. + --prepipex {command} Like --prepipe with one exception: doesn't insert '<' between + command and filename at runtime. Useful for some commands like 'unzip -qc' which don't + read standard input. + Examples: + mlr --prepipe 'gunzip' + mlr --prepipe 'zcat -cf' + mlr --prepipe 'xz -cd' + mlr --prepipe cat + Note that this feature is quite general and is not limited to decompression + utilities. You can use it to apply per-file filters of your choice. + For output compression (or other) utilities, simply pipe the output: + mlr ... | {your compression command} + Lastly, note that if --prepipe is specified, it replaces any decisions that might + have been made based on the file suffix. Also, --gzin/--bz2in/--zin are ignored + if --prepipe is also specified. + + SEPARATORS + CSV-SPECIFIC OPTIONS + --implicit-csv-header Use 1,2,3,... as field labels, rather than from line 1 + of input files. Tip: combine with "label" to recreate + missing headers. + --no-implicit-csv-header Do not use --implicit-csv-header. This is the default + anyway -- the main use is for the flags to 'mlr join' if you have + main file(s) which are headerless but you want to join in on + a file which does have a CSV header. Then you could use + 'mlr --csv --implicit-csv-header join --no-implicit-csv-header + -l your-join-in-with-header.csv ... your-headerless.csv' + --allow-ragged-csv-input|--ragged If a data line has fewer fields than the header line, + fill remaining keys with empty string. If a data line has more + fields than the header line, use integer field labels as in + the implicit-header case. + --headerless-csv-output Print only CSV data lines. + -N Keystroke-saver for --implicit-csv-header --headerless-csv-output. + + DOUBLE-QUOTING FOR CSV/CSVLITE OUTPUT + NUMERICAL FORMATTING + OTHER OPTIONS + --seed {n} with n of the form 12345678 or 0xcafefeed. For put/filter + urand()/urandint()/urand32(). + --nr-progress-mod {m}, with m a positive integer: print filename and record + count to os.Stderr every m input records. + --from {filename} Use this to specify an input file before the verb(s), + rather than after. May be used more than once. Example: + "mlr --from a.dat --from b.dat cat" is the same as + "mlr cat a.dat b.dat". + --mfrom {filenames} -- Use this to specify one of more input files before the verb(s), + rather than after. May be used more than once. + The list of filename must end with "--". This is useful + for example since "--from *.csv" doesn't do what you might + hope but "--mfrom *.csv --" does. + --load {filename} Load DSL script file for all put/filter operations on the command line. + If the name following --load is a directory, load all "*.mlr" files + in that directory. This is just like "put -f" and "filter -f" + except it's up-front on the command line, so you can do something like + alias mlr='mlr --load ~/myscripts' if you like. + --mload {names} -- Like --load but works with more than one filename, + e.g. '--mload *.mlr --'. + -n Process no input files, nor standard input either. Useful + for mlr put with begin/end statements only. (Same as --from + /dev/null.) Also useful in "mlr -n put -v '...'" for + analyzing abstract syntax trees (if that's your thing). + -I Process files in-place. For each file name on the command + line, output is written to a temp file in the same + directory, which is then renamed over the original. Each + file is processed in isolation: if the output format is + CSV, CSV headers will be present in each output file; + statistics are only over each file's own records; and so on. + + THEN-CHAINING + Output of one verb may be chained as input to another using "then", e.g. + mlr stats1 -a min,mean,max -f flag,u,v -g color then sort -f color + + AUXILIARY COMMANDS + Miller has a few otherwise-standalone executables packaged within it. + They do not participate in any other parts of Miller. + Please use "mlr aux-list" for more information. + +MLRRC + You can set up personal defaults via a $HOME/.mlrrc and/or ./.mlrrc. + For example, if you usually process CSV, then you can put "--csv" in your .mlrrc file + and that will be the default input/output format unless otherwise specified on the command line. + + The .mlrrc file format is one "--flag" or "--option value" per line, with the leading "--" optional. + Hash-style comments and blank lines are ignored. + + Sample .mlrrc: + # Input and output formats are CSV by default (unless otherwise specified + # on the mlr command line): + csv + # These are no-ops for CSV, but when I do use JSON output, I want these + # pretty-printing options to be used: + jvstack + jlistwrap + + How to specify location of .mlrrc: + * If $MLRRC is set: + o If its value is "__none__" then no .mlrrc files are processed. + o Otherwise, its value (as a filename) is loaded and processed. If there are syntax + errors, they abort mlr with a usage message (as if you had mistyped something on the + command line). If the file can't be loaded at all, though, it is silently skipped. + o Any .mlrrc in your home directory or current directory is ignored whenever $MLRRC is + set in the environment. + * Otherwise: + o If $HOME/.mlrrc exists, it's then processed as above. + o If ./.mlrrc exists, it's then also processed as above. + (I.e. current-directory .mlrrc defaults are stacked over home-directory .mlrrc defaults.) + + See also: + https://miller.readthedocs.io/en/latest/customization.html + +VERBS + altkv + Usage: mlr altkv [options] + Given fields with values of the form a,b,c,d,e,f emits a=b,c=d,e=f pairs. + Options: + -h|--help Show this message. + + bar + Usage: mlr bar [options] + Replaces a numeric field with a number of asterisks, allowing for cheesy + bar plots. These align best with --opprint or --oxtab output format. + Options: + -f {a,b,c} Field names to convert to bars. + --lo {lo} Lower-limit value for min-width bar: default '0.000000'. + --hi {hi} Upper-limit value for max-width bar: default '100.000000'. + -w {n} Bar-field width: default '40'. + --auto Automatically computes limits, ignoring --lo and --hi. + Holds all records in memory before producing any output. + -c {character} Fill character: default '*'. + -x {character} Out-of-bounds character: default '#'. + -b {character} Blank character: default '.'. + Nominally the fill, out-of-bounds, and blank characters will be strings of length 1. + However you can make them all longer if you so desire. + -h|--help Show this message. + + bootstrap + Usage: mlr bootstrap [options] + Emits an n-sample, with replacement, of the input records. + See also mlr sample and mlr shuffle. + Options: + -n Number of samples to output. Defaults to number of input records. + Must be non-negative. + -h|--help Show this message. + + cat + Usage: mlr cat [options] + Passes input records directly to output. Most useful for format conversion. + Options: + -n Prepend field "n" to each record with record-counter starting at 1. + -N {name} Prepend field {name} to each record with record-counter starting at 1. + -g {a,b,c} Optional group-by-field names for counters, e.g. a,b,c + -h|--help Show this message. + + check + Usage: mlr check [options] + Consumes records without printing any output. + Useful for doing a well-formatted check on input data. + Options: + -h|--help Show this message. + + clean-whitespace + Usage: mlr clean-whitespace [options] + For each record, for each field in the record, whitespace-cleans the keys and/or + values. Whitespace-cleaning entails stripping leading and trailing whitespace, + and replacing multiple whitespace with singles. For finer-grained control, + please see the DSL functions lstrip, rstrip, strip, collapse_whitespace, + and clean_whitespace. + + Options: + -k|--keys-only Do not touch values. + -v|--values-only Do not touch keys. + It is an error to specify -k as well as -v -- to clean keys and values, + leave off -k as well as -v. + -h|--help Show this message. + + count-distinct + Usage: mlr count-distinct [options] + Prints number of records having distinct values for specified field names. + Same as uniq -c. + + Options: + -f {a,b,c} Field names for distinct count. + -n Show only the number of distinct values. Not compatible with -u. + -o {name} Field name for output count. Default "count". + Ignored with -u. + -u Do unlashed counts for multiple field names. With -f a,b and + without -u, computes counts for distinct combinations of a + and b field values. With -f a,b and with -u, computes counts + for distinct a field values and counts for distinct b field + values separately. + + count + Usage: mlr count [options] + Prints number of records, optionally grouped by distinct values for specified field names. + Options: + -g {a,b,c} Optional group-by-field names for counts, e.g. a,b,c + -n {n} Show only the number of distinct values. Not interesting without -g. + -o {name} Field name for output-count. Default "count". + -h|--help Show this message. + + count-similar + Usage: mlr count-similar [options] + Ingests all records, then emits each record augmented by a count of + the number of other records having the same group-by field values. + Options: + -g {a,b,c} Group-by-field names for counts, e.g. a,b,c + -o {name} Field name for output-counts. Defaults to "count". + -h|--help Show this message. + + cut + Usage: mlr cut [options] + Passes through input records with specified fields included/excluded. + Options: + -f {a,b,c} Comma-separated field names for cut, e.g. a,b,c. + -o Retain fields in the order specified here in the argument list. + Default is to retain them in the order found in the input data. + -x|--complement Exclude, rather than include, field names specified by -f. + -r Treat field names as regular expressions. "ab", "a.*b" will + match any field name containing the substring "ab" or matching + "a.*b", respectively; anchors of the form "^ab$", "^a.*b$" may + be used. The -o flag is ignored when -r is present. + -h|--help Show this message. + Examples: + mlr cut -f hostname,status + mlr cut -x -f hostname,status + mlr cut -r -f '^status$,sda[0-9]' + mlr cut -r -f '^status$,"sda[0-9]"' + mlr cut -r -f '^status$,"sda[0-9]"i' (this is case-insensitive) + + decimate + Usage: mlr decimate [options] + Passes through one of every n records, optionally by category. + Options: + -b Decimate by printing first of every n. + -e Decimate by printing last of every n (default). + -g {a,b,c} Optional group-by-field names for decimate counts, e.g. a,b,c. + -n {n} Decimation factor (default 10). + -h|--help Show this message. + + fill-down + Usage: mlr fill-down [options] + If a given record has a missing value for a given field, fill that from + the corresponding value from a previous record, if any. + By default, a 'missing' field either is absent, or has the empty-string value. + With -a, a field is 'missing' only if it is absent. + + Options: + --all Operate on all fields in the input. + -a|--only-if-absent If a given record has a missing value for a given field, + fill that from the corresponding value from a previous record, if any. + By default, a 'missing' field either is absent, or has the empty-string value. + With -a, a field is 'missing' only if it is absent. + -f Field names for fill-down. + -h|--help Show this message. + + fill-empty + Usage: mlr fill-empty [options] + Fills empty-string fields with specified fill-value. + Options: + -v {string} Fill-value: defaults to "N/A" + + filter + Usage: mlr put [options] {DSL expression} + Options: + -f {file name} File containing a DSL expression. If the filename is a directory, + all *.mlr files in that directory are loaded. + + -e {expression} You can use this after -f to add an expression. Example use + case: define functions/subroutines in a file you specify with -f, then call + them with an expression you specify with -e. + + (If you mix -e and -f then the expressions are evaluated in the order encountered. + Since the expression pieces are simply concatenated, please be sure to use intervening + semicolons to separate expressions.) + + -s name=value: Predefines out-of-stream variable @name to have + Thus mlr put -s foo=97 '$column += @foo' is like + mlr put 'begin {@foo = 97} $column += @foo'. + The value part is subject to type-inferencing. + May be specified more than once, e.g. -s name1=value1 -s name2=value2. + Note: the value may be an environment variable, e.g. -s sequence=$SEQUENCE + + -x (default false) Prints records for which {expression} evaluates to false, not true, + i.e. invert the sense of the filter expression. + + -q Does not include the modified record in the output stream. + Useful for when all desired output is in begin and/or end blocks. + + -S and -F: There are no-ops in Miller 6 and above, since now type-inferencing is done + by the record-readers before filter/put is executed. Supported as no-op pass-through + flags for backward compatibility. + + -h|--help Show this message. + + Parser-info options: + + -w Print warnings about things like uninitialized variables. + + -W Same as -w, but exit the process if there are any warnings. + + -p Prints the expressions's AST (abstract syntax tree), which gives full + transparency on the precedence and associativity rules of Miller's grammar, + to stdout. + + -d Like -p but uses a parenthesized-expression format for the AST. + + -D Like -d but with output all on one line. + + -E Echo DSL expression before printing parse-tree + + -v Same as -E -p. + + -X Exit after parsing but before stream-processing. Useful with -v/-d/-D, if you + only want to look at parser information. + + flatten + Usage: mlr flatten [options] + Flattens multi-level maps to single-level ones. Example: field with name 'a' + and value '{"b": { "c": 4 }}' becomes name 'a.b.c' and value 4. + Options: + -f Comma-separated list of field names to flatten (default all). + -s Separator, defaulting to mlr --oflatsep value. + -h|--help Show this message. + + format-values + Usage: mlr format-values [options] + Applies format strings to all field values, depending on autodetected type. + * If a field value is detected to be integer, applies integer format. + * Else, if a field value is detected to be float, applies float format. + * Else, applies string format. + + Note: this is a low-keystroke way to apply formatting to many fields. To get + finer control, please see the fmtnum function within the mlr put DSL. + + Note: this verb lets you apply arbitrary format strings, which can produce + undefined behavior and/or program crashes. See your system's "man printf". + + Options: + -i {integer format} Defaults to "%d". + Examples: "%06lld", "%08llx". + Note that Miller integers are long long so you must use + formats which apply to long long, e.g. with ll in them. + Undefined behavior results otherwise. + -f {float format} Defaults to "%f". + Examples: "%8.3lf", "%.6le". + Note that Miller floats are double-precision so you must + use formats which apply to double, e.g. with l[efg] in them. + Undefined behavior results otherwise. + -s {string format} Defaults to "%s". + Examples: "_%s", "%08s". + Note that you must use formats which apply to string, e.g. + with s in them. Undefined behavior results otherwise. + -n Coerce field values autodetected as int to float, and then + apply the float format. + + fraction + Usage: mlr fraction [options] + For each record's value in specified fields, computes the ratio of that + value to the sum of values in that field over all input records. + E.g. with input records x=1 x=2 x=3 and x=4, emits output records + x=1,x_fraction=0.1 x=2,x_fraction=0.2 x=3,x_fraction=0.3 and x=4,x_fraction=0.4 + + Note: this is internally a two-pass algorithm: on the first pass it retains + input records and accumulates sums; on the second pass it computes quotients + and emits output records. This means it produces no output until all input is read. + + Options: + -f {a,b,c} Field name(s) for fraction calculation + -g {d,e,f} Optional group-by-field name(s) for fraction counts + -p Produce percents [0..100], not fractions [0..1]. Output field names + end with "_percent" rather than "_fraction" + -c Produce cumulative distributions, i.e. running sums: each output + value folds in the sum of the previous for the specified group + E.g. with input records x=1 x=2 x=3 and x=4, emits output records + x=1,x_cumulative_fraction=0.1 x=2,x_cumulative_fraction=0.3 + x=3,x_cumulative_fraction=0.6 and x=4,x_cumulative_fraction=1.0 + + gap + Usage: mlr gap [options] + Emits an empty record every n records, or when certain values change. + Options: + Emits an empty record every n records, or when certain values change. + -g {a,b,c} Print a gap whenever values of these fields (e.g. a,b,c) changes. + -n {n} Print a gap every n records. + One of -f or -g is required. + -n is ignored if -g is present. + -h|--help Show this message. + + grep + Usage: mlr grep [options] {regular expression} + Passes through records which match the regular expression. + Options: + -i Use case-insensitive search. + -v Invert: pass through records which do not match the regex. + -h|--help Show this message. + Note that "mlr filter" is more powerful, but requires you to know field names. + By contrast, "mlr grep" allows you to regex-match the entire record. It does + this by formatting each record in memory as DKVP, using command-line-specified + ORS/OFS/OPS, and matching the resulting line against the regex specified + here. In particular, the regex is not applied to the input stream: if you + have CSV with header line "x,y,z" and data line "1,2,3" then the regex will + be matched, not against either of these lines, but against the DKVP line + "x=1,y=2,z=3". Furthermore, not all the options to system grep are supported, + and this command is intended to be merely a keystroke-saver. To get all the + features of system grep, you can do + "mlr --odkvp ... | grep ... | mlr --idkvp ..." + + group-by + Usage: mlr group-by [options] {comma-separated field names} + Outputs records in batches having identical values at specified field names.Options: + -h|--help Show this message. + + group-like + Usage: mlr group-like [options] + Outputs records in batches having identical field names.Options: + -h|--help Show this message. + + having-fields + Usage: mlr having-fields [options] + Conditionally passes through records depending on each record's field names. + Options: + --at-least {comma-separated names} + --which-are {comma-separated names} + --at-most {comma-separated names} + --all-matching {regular expression} + --any-matching {regular expression} + --none-matching {regular expression} + Examples: + mlr having-fields --which-are amount,status,owner + mlr having-fields --any-matching 'sda[0-9]' + mlr having-fields --any-matching '"sda[0-9]"' + mlr having-fields --any-matching '"sda[0-9]"i' (this is case-insensitive) + + head + Usage: mlr head [options] + Passes through the first n records, optionally by category. + Options: + -g {a,b,c} Optional group-by-field names for head counts, e.g. a,b,c. + -n {n} Head-count to print. Default 10. + -h|--help Show this message. + + histogram + Just a histogram. Input values < lo or > hi are not counted. + Usage: mlr histogram [options] + -f {a,b,c} Value-field names for histogram counts + --lo {lo} Histogram low value + --hi {hi} Histogram high value + --nbins {n} Number of histogram bins + --auto Automatically computes limits, ignoring --lo and --hi. + Holds all values in memory before producing any output. + -o {prefix} Prefix for output field name. Default: no prefix. + -h|--help Show this message. + + json-parse + Usage: mlr json-parse [options] + Tries to convert string field values to parsed JSON, e.g. "[1,2,3]" -> [1,2,3]. + Options: + -f {...} Comma-separated list of field names to json-parse (default all). + -h|--help Show this message. + + json-stringify + Usage: mlr json-stringify [options] + Produces string field values from field-value data, e.g. [1,2,3] -> "[1,2,3]". + Options: + -f {...} Comma-separated list of field names to json-parse (default all). + --jvstack Produce multi-line JSON output. + --no-jvstack Produce single-line JSON output per record (default). + -h|--help Show this message. + + join + Usage: mlr sort {flags} + Sorts records primarily by the first specified field, secondarily by the second + field, and so on. (Any records not having all specified sort keys will appear + at the end of the output, in the order they were encountered, regardless of the + specified sort order.) The sort is stable: records that compare equal will sort + in the order they were encountered in the input record stream. + + Options: + -f {comma-separated field names} Lexical ascending + -n {comma-separated field names} Numerical ascending; nulls sort last + -nf {comma-separated field names} Same as -n + -r {comma-separated field names} Lexical descending + -nr {comma-separated field names} Numerical descending; nulls sort first + -h|--help Show this message. + + Example: + mlr sort -f a,b -nr x,y,z + which is the same as: + mlr sort -f a -f b -nr x -nr y -nr z + + label + Usage: mlr label [options] {new1,new2,new3,...} + Given n comma-separated names, renames the first n fields of each record to + have the respective name. (Fields past the nth are left with their original + names.) Particularly useful with --inidx or --implicit-csv-header, to give + useful names to otherwise integer-indexed fields. + + Options: + -h|--help Show this message. + + least-frequent + Usage: mlr least-frequent [options] + Shows the least frequently occurring distinct values for specified field names. + The first entry is the statistical anti-mode; the remaining are runners-up. + Options: + -f {one or more comma-separated field names}. Required flag. + -n {count}. Optional flag defaulting to 10. + -b Suppress counts; show only field values. + -o {name} Field name for output count. Default "count". + See also "mlr most-frequent". + + merge-fields + Usage: mlr merge-fields [options] + Computes univariate statistics for each input record, accumulated across + specified fields. + Options: + -a {sum,count,...} Names of accumulators. One or more of: + count Count instances of fields + mode Find most-frequently-occurring values for fields; first-found wins tie + antimode Find least-frequently-occurring values for fields; first-found wins tie + sum Compute sums of specified fields + mean Compute averages (sample means) of specified fields + var Compute sample variance of specified fields + stddev Compute sample standard deviation of specified fields + meaneb Estimate error bars for averages (assuming no sample autocorrelation) + skewness Compute sample skewness of specified fields + kurtosis Compute sample kurtosis of specified fields + min Compute minimum values of specified fields + max Compute maximum values of specified fields + -f {a,b,c} Value-field names on which to compute statistics. Requires -o. + -r {a,b,c} Regular expressions for value-field names on which to compute + statistics. Requires -o. + -c {a,b,c} Substrings for collapse mode. All fields which have the same names + after removing substrings will be accumulated together. Please see + examples below. + -i Use interpolated percentiles, like R's type=7; default like type=1. + Not sensical for string-valued fields. + -o {name} Output field basename for -f/-r. + -k Keep the input fields which contributed to the output statistics; + the default is to omit them. + + String-valued data make sense unless arithmetic on them is required, + e.g. for sum, mean, interpolated percentiles, etc. In case of mixed data, + numbers are less than strings. + + Example input data: "a_in_x=1,a_out_x=2,b_in_y=4,b_out_x=8". + Example: mlr merge-fields -a sum,count -f a_in_x,a_out_x -o foo + produces "b_in_y=4,b_out_x=8,foo_sum=3,foo_count=2" since "a_in_x,a_out_x" are + summed over. + Example: mlr merge-fields -a sum,count -r in_,out_ -o bar + produces "bar_sum=15,bar_count=4" since all four fields are summed over. + Example: mlr merge-fields -a sum,count -c in_,out_ + produces "a_x_sum=3,a_x_count=2,b_y_sum=4,b_y_count=1,b_x_sum=8,b_x_count=1" + since "a_in_x" and "a_out_x" both collapse to "a_x", "b_in_y" collapses to + "b_y", and "b_out_x" collapses to "b_x". + + most-frequent + Usage: mlr most-frequent [options] + Shows the most frequently occurring distinct values for specified field names. + The first entry is the statistical mode; the remaining are runners-up. + Options: + -f {one or more comma-separated field names}. Required flag. + -n {count}. Optional flag defaulting to 10. + -b Suppress counts; show only field values. + -o {name} Field name for output count. Default "count". + See also "mlr least-frequent". + + nest + Usage: mlr nest [options] + Explodes specified field values into separate fields/records, or reverses this. + Options: + --explode,--implode One is required. + --values,--pairs One is required. + --across-records,--across-fields One is required. + -f {field name} Required. + --nested-fs {string} Defaults to ";". Field separator for nested values. + --nested-ps {string} Defaults to ":". Pair separator for nested key-value pairs. + --evar {string} Shorthand for --explode --values ---across-records --nested-fs {string} + --ivar {string} Shorthand for --implode --values ---across-records --nested-fs {string} + Please use "mlr --usage-separator-options" for information on specifying separators. + + Examples: + + mlr nest --explode --values --across-records -f x + with input record "x=a;b;c,y=d" produces output records + "x=a,y=d" + "x=b,y=d" + "x=c,y=d" + Use --implode to do the reverse. + + mlr nest --explode --values --across-fields -f x + with input record "x=a;b;c,y=d" produces output records + "x_1=a,x_2=b,x_3=c,y=d" + Use --implode to do the reverse. + + mlr nest --explode --pairs --across-records -f x + with input record "x=a:1;b:2;c:3,y=d" produces output records + "a=1,y=d" + "b=2,y=d" + "c=3,y=d" + + mlr nest --explode --pairs --across-fields -f x + with input record "x=a:1;b:2;c:3,y=d" produces output records + "a=1,b=2,c=3,y=d" + + Notes: + * With --pairs, --implode doesn't make sense since the original field name has + been lost. + * The combination "--implode --values --across-records" is non-streaming: + no output records are produced until all input records have been read. In + particular, this means it won't work in tail -f contexts. But all other flag + combinations result in streaming (tail -f friendly) data processing. + * It's up to you to ensure that the nested-fs is distinct from your data's IFS: + e.g. by default the former is semicolon and the latter is comma. + See also mlr reshape. + + nothing + Usage: mlr nothing [options] + Drops all input records. Useful for testing, or after tee/print/etc. have + produced other output. + Options: + -h|--help Show this message. + + put + Usage: mlr put [options] {DSL expression} + Options: + -f {file name} File containing a DSL expression. If the filename is a directory, + all *.mlr files in that directory are loaded. + + -e {expression} You can use this after -f to add an expression. Example use + case: define functions/subroutines in a file you specify with -f, then call + them with an expression you specify with -e. + + (If you mix -e and -f then the expressions are evaluated in the order encountered. + Since the expression pieces are simply concatenated, please be sure to use intervening + semicolons to separate expressions.) + + -s name=value: Predefines out-of-stream variable @name to have + Thus mlr put -s foo=97 '$column += @foo' is like + mlr put 'begin {@foo = 97} $column += @foo'. + The value part is subject to type-inferencing. + May be specified more than once, e.g. -s name1=value1 -s name2=value2. + Note: the value may be an environment variable, e.g. -s sequence=$SEQUENCE + + -x (default false) Prints records for which {expression} evaluates to false, not true, + i.e. invert the sense of the filter expression. + + -q Does not include the modified record in the output stream. + Useful for when all desired output is in begin and/or end blocks. + + -S and -F: There are no-ops in Miller 6 and above, since now type-inferencing is done + by the record-readers before filter/put is executed. Supported as no-op pass-through + flags for backward compatibility. + + -h|--help Show this message. + + Parser-info options: + + -w Print warnings about things like uninitialized variables. + + -W Same as -w, but exit the process if there are any warnings. + + -p Prints the expressions's AST (abstract syntax tree), which gives full + transparency on the precedence and associativity rules of Miller's grammar, + to stdout. + + -d Like -p but uses a parenthesized-expression format for the AST. + + -D Like -d but with output all on one line. + + -E Echo DSL expression before printing parse-tree + + -v Same as -E -p. + + -X Exit after parsing but before stream-processing. Useful with -v/-d/-D, if you + only want to look at parser information. + + regularize + Usage: mlr regularize [options] + Outputs records sorted lexically ascending by keys.Options: + -h|--help Show this message. + + remove-empty-columns + Usage: mlr remove-empty-columns [options] + Omits fields which are empty on every input row. Non-streaming. + Options: + -h|--help Show this message. + + rename + Usage: mlr rename [options] {old1,new1,old2,new2,...} + Renames specified fields. + Options: + -r Treat old field names as regular expressions. "ab", "a.*b" + will match any field name containing the substring "ab" or + matching "a.*b", respectively; anchors of the form "^ab$", + "^a.*b$" may be used. New field names may be plain strings, + or may contain capture groups of the form "\1" through + "\9". Wrapping the regex in double quotes is optional, but + is required if you wish to follow it with 'i' to indicate + case-insensitivity. + -g Do global replacement within each field name rather than + first-match replacement. + -h|--help Show this message. + Examples: + mlr rename old_name,new_name' + mlr rename old_name_1,new_name_1,old_name_2,new_name_2' + mlr rename -r 'Date_[0-9]+,Date,' Rename all such fields to be "Date" + mlr rename -r '"Date_[0-9]+",Date' Same + mlr rename -r 'Date_([0-9]+).*,\1' Rename all such fields to be of the form 20151015 + mlr rename -r '"name"i,Name' Rename "name", "Name", "NAME", etc. to "Name" + + reorder + Usage: mlr reorder [options] + Moves specified names to start of record, or end of record. + Options: + -e Put specified field names at record end: default is to put them at record start. + -f {a,b,c} Field names to reorder. + -b {x} Put field names specified with -f before field name specified by {x}, + if any. If {x} isn't present in a given record, the specified fields + will not be moved. + -a {x} Put field names specified with -f after field name specified by {x}, + if any. If {x} isn't present in a given record, the specified fields + will not be moved. + -h|--help Show this message. + + Examples: + mlr reorder -f a,b sends input record "d=4,b=2,a=1,c=3" to "a=1,b=2,d=4,c=3". + mlr reorder -e -f a,b sends input record "d=4,b=2,a=1,c=3" to "d=4,c=3,a=1,b=2". + + repeat + Usage: mlr repeat [options] + Copies input records to output records multiple times. + Options must be exactly one of the following: + -n {repeat count} Repeat each input record this many times. + -f {field name} Same, but take the repeat count from the specified + field name of each input record. + -h|--help Show this message. + Example: + echo x=0 | mlr repeat -n 4 then put '$x=urand()' + produces: + x=0.488189 + x=0.484973 + x=0.704983 + x=0.147311 + Example: + echo a=1,b=2,c=3 | mlr repeat -f b + produces: + a=1,b=2,c=3 + a=1,b=2,c=3 + Example: + echo a=1,b=2,c=3 | mlr repeat -f c + produces: + a=1,b=2,c=3 + a=1,b=2,c=3 + a=1,b=2,c=3 + + reshape + Usage: mlr reshape [options] + Wide-to-long options: + -i {input field names} -o {key-field name,value-field name} + -r {input field regexes} -o {key-field name,value-field name} + These pivot/reshape the input data such that the input fields are removed + and separate records are emitted for each key/value pair. + Note: this works with tail -f and produces output records for each input + record seen. + Long-to-wide options: + -s {key-field name,value-field name} + These pivot/reshape the input data to undo the wide-to-long operation. + Note: this does not work with tail -f; it produces output records only after + all input records have been read. + + Examples: + + Input file "wide.txt": + time X Y + 2009-01-01 0.65473572 2.4520609 + 2009-01-02 -0.89248112 0.2154713 + 2009-01-03 0.98012375 1.3179287 + + mlr --pprint reshape -i X,Y -o item,value wide.txt + time item value + 2009-01-01 X 0.65473572 + 2009-01-01 Y 2.4520609 + 2009-01-02 X -0.89248112 + 2009-01-02 Y 0.2154713 + 2009-01-03 X 0.98012375 + 2009-01-03 Y 1.3179287 + + mlr --pprint reshape -r '[A-Z]' -o item,value wide.txt + time item value + 2009-01-01 X 0.65473572 + 2009-01-01 Y 2.4520609 + 2009-01-02 X -0.89248112 + 2009-01-02 Y 0.2154713 + 2009-01-03 X 0.98012375 + 2009-01-03 Y 1.3179287 + + Input file "long.txt": + time item value + 2009-01-01 X 0.65473572 + 2009-01-01 Y 2.4520609 + 2009-01-02 X -0.89248112 + 2009-01-02 Y 0.2154713 + 2009-01-03 X 0.98012375 + 2009-01-03 Y 1.3179287 + + mlr --pprint reshape -s item,value long.txt + time X Y + 2009-01-01 0.65473572 2.4520609 + 2009-01-02 -0.89248112 0.2154713 + 2009-01-03 0.98012375 1.3179287 + See also mlr nest. + + sample + Usage: mlr sample [options] + Reservoir sampling (subsampling without replacement), optionally by category. + See also mlr bootstrap and mlr shuffle. + Options: + -g {a,b,c} Optional: group-by-field names for samples, e.g. a,b,c. + -k {k} Required: number of records to output in total, or by group if using -g. + -h|--help Show this message. + + sec2gmtdate + Usage: ../c/mlr sec2gmtdate {comma-separated list of field names} + Replaces a numeric field representing seconds since the epoch with the + corresponding GMT year-month-day timestamp; leaves non-numbers as-is. + This is nothing more than a keystroke-saver for the sec2gmtdate function: + ../c/mlr sec2gmtdate time1,time2 + is the same as + ../c/mlr put '$time1=sec2gmtdate($time1);$time2=sec2gmtdate($time2)' + + sec2gmt + Usage: mlr sec2gmt [options] {comma-separated list of field names} + Replaces a numeric field representing seconds since the epoch with the + corresponding GMT timestamp; leaves non-numbers as-is. This is nothing + more than a keystroke-saver for the sec2gmt function: + mlr sec2gmt time1,time2 + is the same as + mlr put '$time1 = sec2gmt($time1); $time2 = sec2gmt($time2)' + Options: + -1 through -9: format the seconds using 1..9 decimal places, respectively. + --millis Input numbers are treated as milliseconds since the epoch. + --micros Input numbers are treated as microseconds since the epoch. + --nanos Input numbers are treated as nanoseconds since the epoch. + -h|--help Show this message. + + seqgen + Usage: mlr seqgen [options] + Passes input records directly to output. Most useful for format conversion. + Produces a sequence of counters. Discards the input record stream. Produces + output as specified by the options + + Options: + -f {name} (default "i") Field name for counters. + --start {value} (default 1) Inclusive start value. + --step {value} (default 1) Step value. + --stop {value} (default 100) Inclusive stop value. + -h|--help Show this message. + Start, stop, and/or step may be floating-point. Output is integer if start, + stop, and step are all integers. Step may be negative. It may not be zero + unless start == stop. + + shuffle + Usage: mlr shuffle [options] + Outputs records randomly permuted. No output records are produced until + all input records are read. See also mlr bootstrap and mlr sample. + Options: + -h|--help Show this message. + + skip-trivial-records + Usage: mlr skip-trivial-records [options] + Passes through all records except those with zero fields, + or those for which all fields have empty value. + Options: + -h|--help Show this message. + + sort + Usage: mlr sort {flags} + Sorts records primarily by the first specified field, secondarily by the second + field, and so on. (Any records not having all specified sort keys will appear + at the end of the output, in the order they were encountered, regardless of the + specified sort order.) The sort is stable: records that compare equal will sort + in the order they were encountered in the input record stream. + + Options: + -f {comma-separated field names} Lexical ascending + -n {comma-separated field names} Numerical ascending; nulls sort last + -nf {comma-separated field names} Same as -n + -r {comma-separated field names} Lexical descending + -nr {comma-separated field names} Numerical descending; nulls sort first + -h|--help Show this message. + + Example: + mlr sort -f a,b -nr x,y,z + which is the same as: + mlr sort -f a -f b -nr x -nr y -nr z + + sort-within-records + Usage: mlr sort-within-records [options] + Outputs records sorted lexically ascending by keys. + Options: + -r Recursively sort subobjects/submaps, e.g. for JSON input. + -h|--help Show this message. + + stats1 + Usage: mlr stats1 [options] + Computes univariate statistics for one or more given fields, accumulated across + the input record stream. + Options: + -a {sum,count,...} Names of accumulators: one or more of: + median This is the same as p50 + p10 p25.2 p50 p98 p100 etc. + TODO: flags for interpolated percentiles + count Count instances of fields + mode Find most-frequently-occurring values for fields; first-found wins tie + antimode Find least-frequently-occurring values for fields; first-found wins tie + sum Compute sums of specified fields + mean Compute averages (sample means) of specified fields + var Compute sample variance of specified fields + stddev Compute sample standard deviation of specified fields + meaneb Estimate error bars for averages (assuming no sample autocorrelation) + skewness Compute sample skewness of specified fields + kurtosis Compute sample kurtosis of specified fields + min Compute minimum values of specified fields + max Compute maximum values of specified fields + + -f {a,b,c} Value-field names on which to compute statistics + -g {d,e,f} Optional group-by-field names + + -i Use interpolated percentiles, like R's type=7; default like type=1.\n"); + Not sensical for string-valued fields.\n"); + -s Print iterative stats. Useful in tail -f contexts (in which + case please avoid pprint-format output since end of input + stream will never be seen). + -h|--help Show this message. + [TODO: more] + Example: mlr stats1 -a min,p10,p50,p90,max -f value -g size,shape + mlr stats1 + Example: mlr stats1 -a count,mode -f size + mlr stats1 + Example: mlr stats1 -a count,mode -f size -g shape + mlr stats1 + Example: mlr stats1 -a count,mode --fr '^[a-h].*$' -gr '^k.*$' + mlr stats1 + This computes count and mode statistics on all field names beginning + with a through h, grouped by all field names starting with k. + + Notes: + * p50 and median are synonymous. + * min and max output the same results as p0 and p100, respectively, but use + less memory. + * String-valued data make sense unless arithmetic on them is required, + e.g. for sum, mean, interpolated percentiles, etc. In case of mixed data, + numbers are less than strings. + * count and mode allow text input; the rest require numeric input. + In particular, 1 and 1.0 are distinct text for count and mode. + * When there are mode ties, the first-encountered datum wins. + + stats2 + Usage: mlr stats2 [options] + Computes bivariate statistics for one or more given field-name pairs, + accumulated across the input record stream. + -a {linreg-ols,corr,...} Names of accumulators: one or more of: + linreg-ols Linear regression using ordinary least squares + linreg-pca Linear regression using principal component analysis + r2 Quality metric for linreg-ols (linreg-pca emits its own) + logireg Logistic regression + corr Sample correlation + cov Sample covariance + covx Sample-covariance matrix + -f {a,b,c,d} Value-field name-pairs on which to compute statistics. + There must be an even number of names. + -g {e,f,g} Optional group-by-field names. + -v Print additional output for linreg-pca. + -s Print iterative stats. Useful in tail -f contexts (in which + case please avoid pprint-format output since end of input + stream will never be seen). + --fit Rather than printing regression parameters, applies them to + the input data to compute new fit fields. All input records are + held in memory until end of input stream. Has effect only for + linreg-ols, linreg-pca, and logireg. + Only one of -s or --fit may be used. + Example: mlr stats2 -a linreg-pca -f x,y + Example: mlr stats2 -a linreg-ols,r2 -f x,y -g size,shape + Example: mlr stats2 -a corr -f x,y + + step + Usage: mlr step [options] + Computes values dependent on the previous record, optionally grouped by category. + Options: + -a {delta,rsum,...} Names of steppers: comma-separated, one or more of: + delta Compute differences in field(s) between successive records + shift Include value(s) in field(s) from previous record, if any + from-first Compute differences in field(s) from first record + ratio Compute ratios in field(s) between successive records + rsum Compute running sums of field(s) between successive records + counter Count instances of field(s) between successive records + ewma Exponentially weighted moving average over successive records + + -f {a,b,c} Value-field names on which to compute statistics + -g {d,e,f} Optional group-by-field names + -F Computes integerable things (e.g. counter) in floating point. + As of Miller 6 this happens automatically, but the flag is accepted + as a no-op for backward compatibility with Miller 5 and below. + -d {x,y,z} Weights for ewma. 1 means current sample gets all weight (no + smoothing), near under under 1 is light smoothing, near over 0 is + heavy smoothing. Multiple weights may be specified, e.g. + "mlr step -a ewma -f sys_load -d 0.01,0.1,0.9". Default if omitted + is "-d 0.5". + -o {a,b,c} Custom suffixes for EWMA output fields. If omitted, these default to + the -d values. If supplied, the number of -o values must be the same + as the number of -d values. + -h|--help Show this message. + + Examples: + mlr step -a rsum -f request_size + mlr step -a delta -f request_size -g hostname + mlr step -a ewma -d 0.1,0.9 -f x,y + mlr step -a ewma -d 0.1,0.9 -o smooth,rough -f x,y + mlr step -a ewma -d 0.1,0.9 -o smooth,rough -f x,y -g group_name + + Please see https://miller.readthedocs.io/en/latest/reference-verbs.html#filter or + https://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average + for more information on EWMA. + + tac + Usage: mlr tac [options] + Prints records in reverse order from the order in which they were encountered. + Options: + -h|--help Show this message. + + tail + Usage: mlr tail [options] + Passes through the last n records, optionally by category. + Options: + -g {a,b,c} Optional group-by-field names for head counts, e.g. a,b,c. + -n {n} Head-count to print. Default 10. + -h|--help Show this message. + + tee + Usage: mlr tee [options] {filename} + Options: + -a Append to existing file, if any, rather than overwriting. + -p Treat filename as a pipe-to command. + Any of the output-format command-line flags (see mlr -h). Example: using + mlr --icsv --opprint put '...' then tee --ojson ./mytap.dat then stats1 ... + the input is CSV, the output is pretty-print tabular, but the tee-file output + is written in JSON format. + + -h|--help Show this message. + + top + Usage: mlr top [options] + -f {a,b,c} Value-field names for top counts. + -g {d,e,f} Optional group-by-field names for top counts. + -n {count} How many records to print per category; default 1. + -a Print all fields for top-value records; default is + to print only value and group-by fields. Requires a single + value-field name only. + --min Print top smallest values; default is top largest values. + -F Keep top values as floats even if they look like integers. + -o {name} Field name for output indices. Default "top_idx". + Prints the n records with smallest/largest values at specified fields, + optionally by category. + + unflatten + Usage: mlr unflatten [options] + Reverses flatten. Example: field with name 'a.b.c' and value 4 + becomes name 'a' and value '{"b": { "c": 4 }}'. + Options: + -f {a,b,c} Comma-separated list of field names to unflatten (default all). + -s {string} Separator, defaulting to mlr --oflatsep value. + -h|--help Show this message. + + uniq + Usage: mlr uniq [options] + Prints distinct values for specified field names. With -c, same as + count-distinct. For uniq, -f is a synonym for -g. + + Options: + -g {d,e,f} Group-by-field names for uniq counts. + -c Show repeat counts in addition to unique values. + -n Show only the number of distinct values. + -o {name} Field name for output count. Default "count". + -a Output each unique record only once. Incompatible with -g. + With -c, produces unique records, with repeat counts for each. + With -n, produces only one record which is the unique-record count. + With neither -c nor -n, produces unique records. + + unsparsify + Usage: mlr unsparsify [options] + Prints records with the union of field names over all input records. + For field names absent in a given record but present in others, fills in + a value. This verb retains all input before producing any output. + Options: + --fill-with {filler string} What to fill absent fields with. Defaults to + the empty string. + -f {a,b,c} Specify field names to be operated on. Any other fields won't be + modified, and operation will be streaming. + -h|--help Show this message. + Example: if the input is two records, one being 'a=1,b=2' and the other + being 'b=3,c=4', then the output is the two records 'a=1,b=2,c=' and + 'a=,b=3,c=4'. + +FUNCTIONS FOR FILTER/PUT + + + (class=arithmetic #args=1,2) Addition as binary operator; unary plus operator. + + - + (class=arithmetic #args=1,2) Subtraction as binary operator; unary negation operator. + + * + (class=arithmetic #args=2) Multiplication, with integer*integer overflow to float. + + / + (class=arithmetic #args=2) Division. Integer / integer is floating-point. + + // + (class=arithmetic #args=2) Pythonic integer division, rounding toward negative. + + ** + (class=arithmetic #args=2) Exponentiation. Same as pow, but as an infix operator. + + pow + (class=arithmetic #args=2) Exponentiation. Same as **, but as a function. + + .+ + (class=arithmetic #args=2) Addition, with integer-to-integer overflow. + + .- + (class=arithmetic #args=2) Subtraction, with integer-to-integer overflow. + + .* + (class=arithmetic #args=2) Multiplication, with integer-to-integer overflow. + + ./ + (class=arithmetic #args=2) Integer division; not pythonic. + + % + (class=arithmetic #args=2) Remainder; never negative-valued (pythonic). + + ~ + (class=arithmetic #args=1) Bitwise NOT. Beware '$y=~$x' since =~ is the + regex-match operator: try '$y = ~$x'. + + & + (class=arithmetic #args=2) Bitwise AND. + + | + (class=arithmetic #args=2) Bitwise OR. + + ^ + (class=arithmetic #args=2) Bitwise XOR. + + << + (class=arithmetic #args=2) Bitwise left-shift. + + >> + (class=arithmetic #args=2) Bitwise signed right-shift. + + >>> + (class=arithmetic #args=2) Bitwise unsigned right-shift. + + bitcount + (class=arithmetic #args=1) Count of 1-bits. + + madd + (class=arithmetic #args=3) a + b mod m (integers) + + msub + (class=arithmetic #args=3) a - b mod m (integers) + + mmul + (class=arithmetic #args=3) a * b mod m (integers) + + mexp + (class=arithmetic #args=3) a ** b mod m (integers) + + ! + (class=boolean #args=1) Logical negation. + + == + (class=boolean #args=2) String/numeric equality. Mixing number and string results in string compare. + + != + (class=boolean #args=2) String/numeric inequality. Mixing number and string results in string compare. + + > + (class=boolean #args=2) String/numeric greater-than. Mixing number and string results in string compare. + + >= + (class=boolean #args=2) String/numeric greater-than-or-equals. Mixing number and string results in string compare. + + < + (class=boolean #args=2) String/numeric less-than. Mixing number and string results in string compare. + + <= + (class=boolean #args=2) String/numeric less-than-or-equals. Mixing number and string results in string compare. + + =~ + (class=boolean #args=2) String (left-hand side) matches regex (right-hand side), e.g. '$name =~ "^a.*b$"'. + + !=~ + (class=boolean #args=2) String (left-hand side) does not match regex (right-hand side), e.g. '$name !=~ "^a.*b$"'. + + && + (class=boolean #args=2) Logical AND. + + || + (class=boolean #args=2) Logical OR. + + ^^ + (class=boolean #args=2) Logical XOR. + + ?? + (class=boolean #args=2) Absent-coalesce operator. $a ?? 1 evaluates to 1 if $a isn't defined in the current record. + + ??? + (class=boolean #args=2) Absent-coalesce operator. $a ?? 1 evaluates to 1 if $a isn't defined in the current record, or has empty value. + + ?: + (class=boolean #args=3) Standard ternary operator. + + . + (class=string #args=2) String concatenation. + + capitalize + (class=string #args=1) Convert string's first character to uppercase. + + clean_whitespace + (class=string #args=1) Same as collapse_whitespace and strip. + + collapse_whitespace + (class=string #args=1) Strip repeated whitespace from string. + + gsub + (class=string #args=3) Example: '$name=gsub($name, "old", "new")' (replace all). + + lstrip + (class=string #args=1) Strip leading whitespace from string. + + regextract + (class=string #args=2) Example: '$name=regextract($name, "[A-Z]{3}[0-9]{2}")' + + regextract_or_else + (class=string #args=3) Example: '$name=regextract_or_else($name, "[A-Z]{3}[0-9]{2}", "default")' + + rstrip + (class=string #args=1) Strip trailing whitespace from string. + + strip + (class=string #args=1) Strip leading and trailing whitespace from string. + + strlen + (class=string #args=1) String length. + + ssub + (class=string #args=3) Like sub but does no regexing. No characters are special. + + sub + (class=string #args=3) Example: '$name=sub($name, "old", "new")' (replace once). + + substr0 + (class=string #args=3) substr0(s,m,n) gives substring of s from 0-up position m to n + inclusive. Negative indices -len .. -1 alias to 0 .. len-1. + + substr1 + (class=string #args=3) substr1(s,m,n) gives substring of s from 1-up position m to n + inclusive. Negative indices -len .. -1 alias to 1 .. len. + + substr + (class=string #args=3) substr is an alias for substr0. See also substr1. Miller is generally 1-up + with all array indices, but, this is a backward-compatibility issue with Miller 5 and below. + Arrays are new in Miller 6; the substr function is older. + + tolower + (class=string #args=1) Convert string to lowercase. + + toupper + (class=string #args=1) Convert string to uppercase. + + truncate + (class=string #args=2) Truncates string first argument to max length of int second argument. + + md5 + (class=hashing #args=1) MD5 hash. + + sha1 + (class=hashing #args=1) SHA1 hash. + + sha256 + (class=hashing #args=1) SHA256 hash. + + sha512 + (class=hashing #args=1) SHA512 hash. + + abs + (class=math #args=1) Absolute value. + + acos + (class=math #args=1) Inverse trigonometric cosine. + + acosh + (class=math #args=1) Inverse hyperbolic cosine. + + asin + (class=math #args=1) Inverse trigonometric sine. + + asinh + (class=math #args=1) Inverse hyperbolic sine. + + atan + (class=math #args=1) One-argument arctangent. + + atan2 + (class=math #args=2) Two-argument arctangent. + + atanh + (class=math #args=1) Inverse hyperbolic tangent. + + cbrt + (class=math #args=1) Cube root. + + ceil + (class=math #args=1) Ceiling: nearest integer at or above. + + cos + (class=math #args=1) Trigonometric cosine. + + cosh + (class=math #args=1) Hyperbolic cosine. + + erf + (class=math #args=1) Error function. + + erfc + (class=math #args=1) Complementary error function. + + exp + (class=math #args=1) Exponential function e**x. + + expm1 + (class=math #args=1) e**x - 1. + + floor + (class=math #args=1) Floor: nearest integer at or below. + + invqnorm + (class=math #args=1) Inverse of normal cumulative distribution function. + Note that invqorm(urand()) is normally distributed. + + log + (class=math #args=1) Natural (base-e) logarithm. + + log10 + (class=math #args=1) Base-10 logarithm. + + log1p + (class=math #args=1) log(1-x). + + logifit + (class=math #args=3) Given m and b from logistic regression, compute fit: + $yhat=logifit($x,$m,$b). + + max + (class=math #args=variadic) Max of n numbers; null loses. + + min + (class=math #args=variadic) Min of n numbers; null loses. + + qnorm + (class=math #args=1) Normal cumulative distribution function. + + round + (class=math #args=1) Round to nearest integer. + + sgn + (class=math #args=1) +1, 0, -1 for positive, zero, negative input respectively. + + sin + (class=math #args=1) Trigonometric sine. + + sinh + (class=math #args=1) Hyperbolic sine. + + sqrt + (class=math #args=1) Square root. + + tan + (class=math #args=1) Trigonometric tangent. + + tanh + (class=math #args=1) Hyperbolic tangent. + + roundm + (class=math #args=2) Round to nearest multiple of m: roundm($x,$m) is + the same as round($x/$m)*$m. + + urand + (class=math #args=0) Floating-point numbers uniformly distributed on the unit interval. + Int-valued example: '$n=floor(20+urand()*11)'. + + urandint + (class=math #args=2) Integer uniformly distributed between inclusive integer endpoints. + + urandrange + (class=math #args=2) Floating-point numbers uniformly distributed on the interval [a, b). + + urand32 + (class=math #args=0) Integer uniformly distributed 0 and 2**32-1 inclusive. + + gmt2sec + (class=time #args=1) Parses GMT timestamp as integer seconds since the epoch. + + sec2gmt + (class=time #args=1,2) Formats seconds since epoch (integer part) + as GMT timestamp, e.g. sec2gmt(1440768801.7) = "2015-08-28T13:33:21Z". + Leaves non-numbers as-is. With second integer argument n, includes n decimal places + for the seconds part + + sec2gmtdate + (class=time #args=1) Formats seconds since epoch (integer part) + as GMT timestamp with year-month-date, e.g. sec2gmtdate(1440768801.7) = "2015-08-28". + Leaves non-numbers as-is. + + + systime + (class=time #args=0) help string will go here + + systimeint + (class=time #args=0) help string will go here + + uptime + (class=time #args=0) help string will go here + + strftime + (class=time #args=2) Formats seconds since the epoch as timestamp, e.g. + strftime(1440768801.7,"%Y-%m-%dT%H:%M:%SZ") = "2015-08-28T13:33:21Z", and + strftime(1440768801.7,"%Y-%m-%dT%H:%M:%3SZ") = "2015-08-28T13:33:21.700Z". + Format strings are as in the C library (please see "man strftime" on your system), + with the Miller-specific addition of "%1S" through "%9S" which format the seconds + with 1 through 9 decimal places, respectively. ("%S" uses no decimal places.) + See also strftime_local. + + + strptime + (class=time #args=2) strptime: Parses timestamp as floating-point seconds since the epoch, + e.g. strptime("2015-08-28T13:33:21Z","%Y-%m-%dT%H:%M:%SZ") = 1440768801.000000, + and strptime("2015-08-28T13:33:21.345Z","%Y-%m-%dT%H:%M:%SZ") = 1440768801.345000. + See also strptime_local. + + + dhms2fsec + (class=time #args=1) Recovers floating-point seconds as in dhms2fsec("5d18h53m20.250000s") = 500000.250000 + + + dhms2sec + (class=time #args=1) Recovers integer seconds as in dhms2sec("5d18h53m20s") = 500000 + + + fsec2dhms + (class=time #args=1) Formats floating-point seconds as in fsec2dhms(500000.25) = "5d18h53m20.250000s" + + + fsec2hms + (class=time #args=1) Formats floating-point seconds as in fsec2hms(5000.25) = "01:23:20.250000" + + + hms2fsec + (class=time #args=1) Recovers floating-point seconds as in hms2fsec("01:23:20.250000") = 5000.250000 + + + hms2sec + (class=time #args=1) Recovers integer seconds as in hms2sec("01:23:20") = 5000 + + + sec2dhms + (class=time #args=1) Formats integer seconds as in sec2dhms(500000) = "5d18h53m20s" + + + sec2hms + (class=time #args=1) Formats integer seconds as in sec2hms(5000) = "01:23:20" + + + is_absent + (class=typing #args=1) False if field is present in input, true otherwise + + is_array + (class=typing #args=1) True if argument is an array. + + is_bool + (class=typing #args=1) True if field is present with boolean value. Synonymous with is_boolean. + + is_boolean + (class=typing #args=1) True if field is present with boolean value. Synonymous with is_bool. + + is_empty + (class=typing #args=1) True if field is present in input with empty string value, false otherwise. + + is_empty_map + (class=typing #args=1) True if argument is a map which is empty. + + is_error + (class=typing #args=1) True if if argument is an error, such as taking string length of an integer. + + is_float + (class=typing #args=1) True if field is present with value inferred to be float + + is_int + (class=typing #args=1) True if field is present with value inferred to be int + + is_map + (class=typing #args=1) True if argument is a map. + + is_nonempty_map + (class=typing #args=1) True if argument is a map which is non-empty. + + is_not_empty + (class=typing #args=1) False if field is present in input with empty value, true otherwise + + is_not_map + (class=typing #args=1) True if argument is not a map. + + is_not_array + (class=typing #args=1) True if argument is not an array. + + is_not_null + (class=typing #args=1) False if argument is null (empty or absent), true otherwise. + + is_null + (class=typing #args=1) True if argument is null (empty or absent), false otherwise. + + is_numeric + (class=typing #args=1) True if field is present with value inferred to be int or float + + is_present + (class=typing #args=1) True if field is present in input, false otherwise. + + is_string + (class=typing #args=1) True if field is present with string (including empty-string) value + + asserting_absent + (class=typing #args=1) Aborts with an error if is_absent on the argument returns false, + else returns its argument. + + asserting_array + (class=typing #args=1) Aborts with an error if is_array on the argument returns false, + else returns its argument. + + asserting_bool + (class=typing #args=1) Aborts with an error if is_bool on the argument returns false, + else returns its argument. + + asserting_boolean + (class=typing #args=1) Aborts with an error if is_boolean on the argument returns false, + else returns its argument. + + asserting_error + (class=typing #args=1) Aborts with an error if is_error on the argument returns false, + else returns its argument. + + asserting_empty + (class=typing #args=1) Aborts with an error if is_empty on the argument returns false, + else returns its argument. + + asserting_empty_map + (class=typing #args=1) Aborts with an error if is_empty_map on the argument returns false, + else returns its argument. + + asserting_float + (class=typing #args=1) Aborts with an error if is_float on the argument returns false, + else returns its argument. + + asserting_int + (class=typing #args=1) Aborts with an error if is_int on the argument returns false, + else returns its argument. + + asserting_map + (class=typing #args=1) Aborts with an error if is_map on the argument returns false, + else returns its argument. + + asserting_nonempty_map + (class=typing #args=1) Aborts with an error if is_nonempty_map on the argument returns false, + else returns its argument. + + asserting_not_empty + (class=typing #args=1) Aborts with an error if is_not_empty on the argument returns false, + else returns its argument. + + asserting_not_map + (class=typing #args=1) Aborts with an error if is_not_map on the argument returns false, + else returns its argument. + + asserting_not_array + (class=typing #args=1) Aborts with an error if is_not_array on the argument returns false, + else returns its argument. + + asserting_not_null + (class=typing #args=1) Aborts with an error if is_not_null on the argument returns false, + else returns its argument. + + asserting_null + (class=typing #args=1) Aborts with an error if is_null on the argument returns false, + else returns its argument. + + asserting_numeric + (class=typing #args=1) Aborts with an error if is_numeric on the argument returns false, + else returns its argument. + + asserting_present + (class=typing #args=1) Aborts with an error if is_present on the argument returns false, + else returns its argument. + + asserting_string + (class=typing #args=1) Aborts with an error if is_string on the argument returns false, + else returns its argument. + + typeof + (class=typing #args=1) Convert argument to type of argument (e.g. "str"). For debug. + + boolean + (class=conversion #args=1) Convert int/float/bool/string to boolean. + + float + (class=conversion #args=1) Convert int/float/bool/string to float. + + fmtnum + (class=conversion #args=2) Convert int/float/bool to string using + printf-style format string, e.g. '$s = fmtnum($n, "%06lld")'. + + hexfmt + (class=conversion #args=1) Convert int to hex string, e.g. 255 to "0xff". + + int + (class=conversion #args=1) Convert int/float/bool/string to int. + + joink + (class=conversion #args=2) Makes string from map/array keys. Examples: + joink({"a":3,"b":4,"c":5}, ",") = "a,b,c" + joink([1,2,3], ",") = "1,2,3". + + joinv + (class=conversion #args=2) Makes string from map/array values. + joinv([3,4,5], ",") = "3,4,5" + joinv({"a":3,"b":4,"c":5}, ",") = "3,4,5" + + joinkv + (class=conversion #args=3) Makes string from map/array key-value pairs. Examples: + joinkv([3,4,5], "=", ",") = "1=3,2=4,3=5" + joinkv({"a":3,"b":4,"c":5}, "=", ",") = "a=3,b=4,c=5" + + splita + (class=conversion #args=2) Splits string into array with type inference. Example: + splita("3,4,5", ",") = [3,4,5] + + splitax + (class=conversion #args=2) Splits string into array without type inference. Example: + splita("3,4,5", ",") = ["3","4","5"] + + splitkv + (class=conversion #args=3) Splits string by separators into map with type inference. Example: + splitkv("a=3,b=4,c=5", "=", ",") = {"a":3,"b":4,"c":5} + + splitkvx + (class=conversion #args=3) Splits string by separators into map without type inference (keys and + values are strings). Example: + splitkvx("a=3,b=4,c=5", "=", ",") = {"a":"3","b":"4","c":"5"} + + splitnv + (class=conversion #args=2) Splits string by separator into integer-indexed map with type inference. Example: + splitnv("a,b,c", ",") = {"1":"a","2":"b","3":"c"} + + splitnvx + (class=conversion #args=2) Splits string by separator into integer-indexed map without type + inference (values are strings). Example: + splitnvx("3,4,5", ",") = {"1":"3","2":"4","3":"5"} + + string + (class=conversion #args=1) Convert int/float/bool/string/array/map to string. + + append + (class=maps/arrays #args=2) Appends second argument to end of first argument, which must be an array. + + arrayify + (class=maps/arrays #args=1) Walks through a nested map/array, converting any map with consecutive keys + "1", "2", ... into an array. Useful to wrap the output of unflatten. + + depth + (class=maps/arrays #args=1) Prints maximum depth of map/array. Scalars have depth 0. + + flatten + (class=maps/arrays #args=3) Flattens multi-level maps to single-level ones. Examples: + flatten("a", ".", {"b": { "c": 4 }}) is {"a.b.c" : 4}. + flatten("", ".", {"a": { "b": 3 }}) is {"a.b" : 3}. + Two-argument version: flatten($*, ".") is the same as flatten("", ".", $*). + Useful for nested JSON-like structures for non-JSON file formats like CSV. + + get_keys + (class=maps/arrays #args=1) Returns array of keys of map or array + + get_values + (class=maps/arrays #args=1) Returns array of keys of map or array -- in the latter case, returns a copy of the array + + haskey + (class=maps/arrays #args=2) True/false if map has/hasn't key, e.g. 'haskey($*, "a")' or + 'haskey(mymap, mykey)', or true/false if array index is in bounds / out of bounds. + Error if 1st argument is not a map or array. Note -n..-1 alias to 1..n in Miller arrays. + + json_parse + (class=maps/arrays #args=1) Converts value from JSON-formatted string. + + json_stringify + (class=maps/arrays #args=1,2) Converts value to JSON-formatted string. Default output is single-line. + With optional second boolean argument set to true, produces multiline output. + + leafcount + (class=maps/arrays #args=1) Counts total number of terminal values in map/array. For single-level + map/array, same as length. + + length + (class=maps/arrays #args=1) Counts number of top-level entries in array/map. Scalars have length 1. + + mapdiff + (class=maps/arrays #args=variadic) With 0 args, returns empty map. With 1 arg, returns copy of arg. + With 2 or more, returns copy of arg 1 with all keys from any of remaining + argument maps removed. + + mapexcept + (class=maps/arrays #args=variadic) Returns a map with keys from remaining arguments, if any, unset. + Remaining arguments can be strings or arrays of string. + E.g. 'mapexcept({1:2,3:4,5:6}, 1, 5, 7)' is '{3:4}' + and 'mapexcept({1:2,3:4,5:6}, [1, 5, 7])' is '{3:4}'. + + mapselect + (class=maps/arrays #args=variadic) Returns a map with only keys from remaining arguments set. + Remaining arguments can be strings or arrays of string. + E.g. 'mapselect({1:2,3:4,5:6}, 1, 5, 7)' is '{1:2,5:6}' + and 'mapselect({1:2,3:4,5:6}, [1, 5, 7])' is '{1:2,5:6}'. + + mapsum + (class=maps/arrays #args=variadic) With 0 args, returns empty map. With >= 1 arg, returns a map with + key-value pairs from all arguments. Rightmost collisions win, e.g. + 'mapsum({1:2,3:4},{1:5})' is '{1:5,3:4}'. + + unflatten + (class=maps/arrays #args=2) Reverses flatten. Example: + unflatten({"a.b.c" : 4}, ".") is {"a": "b": { "c": 4 }}}. + Useful for nested JSON-like structures for non-JSON file formats like CSV. + See also arrayify. + + hostname + (class=system #args=0) Returns the hostname as a string. + + os + (class=system #args=0) Returns the operating-system name as a string. + + system + (class=system #args=1) Run command string, yielding its stdout minus final carriage return. + + version + (class=system #args=0) Returns the Miller version as a string. + +KEYWORDS FOR PUT AND FILTER + TODO: port mlr_dsl_list_all_keywords_raw + TODO: port mlr_dsl_keyword_usage + +AUTHOR + Miller is written by John Kerl . + + This manual page has been composed from Miller's help output by Eric + MSP Veith . + +SEE ALSO + awk(1), sed(1), cut(1), join(1), sort(1), RFC 4180: Common Format and + MIME Type for Comma-Separated Values (CSV) Files, the miller website + http://johnkerl.org/miller/doc + + + + 2021-06-16 MILLER(1) diff --git a/man6/mkman.rb b/man6/mkman.rb new file mode 100755 index 000000000..1cadc1dcd --- /dev/null +++ b/man6/mkman.rb @@ -0,0 +1,239 @@ +#!/usr/bin/env ruby + +# ================================================================ +# This is a manpage autogenerator for Miller. There are various tools out there +# for creating xroff-formatted manpages, but I wanted something with minimal +# external dependencies which would also automatically generate most of its +# output from the mlr executable itself. It turns out it's easy enough to get +# this in just a few lines of Ruby. +# +# Note for dev-viewing of the output: +# ./mkman.rb | groff -man -Tascii | less +# ================================================================ + +# ---------------------------------------------------------------- +def main + # In case the user running this has a .mlrrc + ENV['MLRRC'] = '__none__' + + print make_top + + print make_section('NAME', [ + "miller \\\- like awk, sed, cut, join, and sort for name-indexed data such as CSV and tabular JSON." + ]) + + print make_section('SYNOPSIS', [ +`mlr --usage-synopsis` + ]) + + print make_section('DESCRIPTION', [ +"""Miller operates on key-value-pair data while the familiar Unix tools operate +on integer-indexed fields: if the natural data structure for the latter is the +array, then Miller's natural data structure is the insertion-ordered hash map. +This encompasses a variety of data formats, including but not limited to the +familiar CSV, TSV, and JSON. (Miller can handle positionally-indexed data as +a special case.) This manpage documents #{`mlr --version`.chomp}.""" + ]) + + print make_section('EXAMPLES', [ + "" + ]) + + print make_subsection('COMMAND-LINE SYNTAX', []) + print make_code_block(`mlr --usage-examples`) + + print make_subsection('DATA FORMATS', []) + print make_code_block(`mlr --usage-data-format-examples`) + + print make_section('OPTIONS', [ +"""In the following option flags, the version with \"i\" designates the input +stream, \"o\" the output stream, and the version without prefix sets the option +for both input and output stream. For example: --irs sets the input record +separator, --ors the output record separator, and --rs sets both the input and +output separator to the given value.""" + ]) + + print make_subsection('HELP OPTIONS', []) + print make_code_block(`mlr --usage-help-options`) + + print make_subsection('VERB LIST', []) + print make_code_block(`mlr --usage-list-all-verbs`) + + print make_subsection('FUNCTION LIST', []) + print make_code_block(`mlr --usage-functions`) + + print make_subsection('I/O FORMATTING', []) + print make_code_block(`mlr --usage-data-format-options`) + + print make_subsection('COMMENTS IN DATA', []) + print make_code_block(`mlr --usage-comments-in-data`) + + print make_subsection('FORMAT-CONVERSION KEYSTROKE-SAVERS', []) + print make_code_block(`mlr --usage-format-conversion-keystroke-saver-options`) + + print make_subsection('COMPRESSED I/O', []) + print make_code_block(`mlr --usage-compressed-data-options`) + + print make_subsection('SEPARATORS', []) + print make_code_block(`mlr --usage-separator-options`) + + print make_subsection('CSV-SPECIFIC OPTIONS', []) + print make_code_block(`mlr --usage-csv-options`) + + print make_subsection('DOUBLE-QUOTING FOR CSV/CSVLITE OUTPUT', []) + print make_code_block(`mlr --usage-double-quoting`) + + print make_subsection('NUMERICAL FORMATTING', []) + print make_code_block(`mlr --usage-numerical-formatting`) + + print make_subsection('OTHER OPTIONS', []) + print make_code_block(`mlr --usage-other-options`) + + print make_subsection('THEN-CHAINING', []) + print make_code_block(`mlr --usage-then-chaining`) + + print make_subsection('AUXILIARY COMMANDS', []) + print make_code_block(`mlr --usage-auxents`) + + print make_section('MLRRC', []) + + print make_code_block(`mlr --usage-mlrrc`) + + verbs = `mlr --list-all-verbs-raw` + print make_section('VERBS', [ + "" + ]) + verbs = verbs.strip.split("\n") + for verb in verbs + print make_subsection(verb, []) + print make_code_block(`mlr #{verb} -h`) + end + + functions = `mlr --list-all-functions-raw` + print make_section('FUNCTIONS FOR FILTER/PUT', [ + "" + ]) + functions = functions.strip.split("\n").uniq + for function in functions + print make_subsection(function, []) + text = `mlr --help-function '#{function}'` + text = text.sub(function + ' ', '') + print make_code_block(text) + end + + keywords = `mlr --list-all-keywords-raw` + print make_section('KEYWORDS FOR PUT AND FILTER', [ + "" + ]) + keywords = keywords.strip.split("\n").uniq + for keyword in keywords + print make_subsection(keyword, []) + text = `mlr --help-keyword '#{keyword}'` + print make_code_block(text) + end + + print make_section('AUTHOR', [ + "Miller is written by John Kerl .", + "This manual page has been composed from Miller's help output by Eric MSP Veith ." + ]) + print make_section('SEE ALSO', [ + "awk(1), sed(1), cut(1), join(1), sort(1), RFC 4180: Common Format and MIME Type for " + + "Comma-Separated Values (CSV) Files, the miller website http://johnkerl.org/miller/doc" + ]) +end + +# ================================================================ +def make_top() + t = Time::new + stamp = t.gmtime.strftime("%Y-%m-%d") + + # Portability definitions thanks to some asciidoc output + +"""'\\\" t +.\\\" Title: mlr +.\\\" Author: [see the \"AUTHOR\" section] +.\\\" Generator: #{$0} +.\\\" Date: #{stamp} +.\\\" Manual: \\ \\& +.\\\" Source: \\ \\& +.\\\" Language: English +.\\\" +.TH \"MILLER\" \"1\" \"#{stamp}\" \"\\ \\&\" \"\\ \\&\" +.\\\" ----------------------------------------------------------------- +.\\\" * Portability definitions +.\\\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +.\\\" http://bugs.debian.org/507673 +.\\\" http://lists.gnu.org/archive/html/groff/2009-02/msg00013.html +.\\\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +.ie \\n(.g .ds Aq \(aq +.el .ds Aq ' +.\\\" ----------------------------------------------------------------- +.\\\" * set default formatting +.\\\" ----------------------------------------------------------------- +.\\\" disable hyphenation +.nh +.\\\" disable justification (adjust text to left margin only) +.ad l +.\\\" ----------------------------------------------------------------- +""" +end + +# ---------------------------------------------------------------- +def make_section(title, paragraphs) + retval = ".SH \"#{title}\"\n" + paragraphs.each do |paragraph| + retval += ".sp\n" + retval += groff_encode(paragraph) + "\n" + end + retval +end + +# ---------------------------------------------------------------- +def make_subsection(title, paragraphs) + retval = ".SS \"#{title}\"\n" + paragraphs.each do |paragraph| + retval += ".sp\n" + retval += groff_encode(paragraph) + "\n" + end + retval +end + +# ---------------------------------------------------------------- +def make_subsubsection(title, paragraphs) + retval = ".sp\n"; + retval += "\\fB#{title}\\fR\n" + paragraphs.each do |paragraph| + retval += ".sp\n" + retval += groff_encode(paragraph) + "\n" + end + retval +end + +# ---------------------------------------------------------------- +def make_code_block(block) + retval = ".if n \\{\\\n" + retval += ".RS 0\n" + retval += ".\\}\n" + retval += ".nf\n" + # In case the line starts with a dot: + retval += block.gsub('\\', '\e').gsub(/^\./){'\&.'} + # In case the line starts with a single quote: + retval = retval.gsub(/^'/, '\(cq') + retval += ".fi\n" + retval += ".if n \\{\\\n" + retval += ".RE\n" +end + +# ---------------------------------------------------------------- +def groff_encode(line) + #line = line.gsub(/'/, '\(cq') + #line = line.gsub(/"/, '\(dq') + #line = line.gsub(/\./, '\&') + #line = line.gsub(/-/, '\-') + line = line.gsub(/\\([^-])/, '\e\1') + line = line.gsub(/^\./){'\&.'} + line +end + +# ================================================================ +main diff --git a/man6/mlr6.1 b/man6/mlr6.1 new file mode 100644 index 000000000..4ffd32fa7 --- /dev/null +++ b/man6/mlr6.1 @@ -0,0 +1,3838 @@ +'\" t +.\" Title: mlr +.\" Author: [see the "AUTHOR" section] +.\" Generator: ./mkman.rb +.\" Date: 2021-06-16 +.\" Manual: \ \& +.\" Source: \ \& +.\" Language: English +.\" +.TH "MILLER" "1" "2021-06-16" "\ \&" "\ \&" +.\" ----------------------------------------------------------------- +.\" * Portability definitions +.\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +.\" http://bugs.debian.org/507673 +.\" http://lists.gnu.org/archive/html/groff/2009-02/msg00013.html +.\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +.ie \n(.g .ds Aq (aq +.el .ds Aq ' +.\" ----------------------------------------------------------------- +.\" * set default formatting +.\" ----------------------------------------------------------------- +.\" disable hyphenation +.nh +.\" disable justification (adjust text to left margin only) +.ad l +.\" ----------------------------------------------------------------- +.SH "NAME" +.sp +miller \- like awk, sed, cut, join, and sort for name-indexed data such as CSV and tabular JSON. +.SH "SYNOPSIS" +.sp +Usage: mlr [I/O options] {verb} [verb-dependent options ...] {zero or more file names} + +.SH "DESCRIPTION" +.sp +Miller operates on key-value-pair data while the familiar Unix tools operate +on integer-indexed fields: if the natural data structure for the latter is the +array, then Miller's natural data structure is the insertion-ordered hash map. +This encompasses a variety of data formats, including but not limited to the +familiar CSV, TSV, and JSON. (Miller can handle positionally-indexed data as +a special case.) This manpage documents Miller v6.0.0-dev. +.SH "EXAMPLES" +.sp + +.SS "COMMAND-LINE SYNTAX" +.if n \{\ +.RS 0 +.\} +.nf +mlr --csv cut -f hostname,uptime mydata.csv +mlr --tsv --rs lf filter '$status != "down" && $upsec >= 10000' *.tsv +mlr --nidx put '$sum = $7 < 0.0 ? 3.5 : $7 + 2.1*$8' *.dat +grep -v '^#' /etc/group | mlr --ifs : --nidx --opprint label group,pass,gid,member then sort -f group +mlr join -j account_id -f accounts.dat then group-by account_name balances.dat +mlr --json put '$attr = sub($attr, "([0-9]+)_([0-9]+)_.*", "\e1:\e2")' data/*.json +mlr stats1 -a min,mean,max,p10,p50,p90 -f flag,u,v data/* +mlr stats2 -a linreg-pca -f u,v -g shape data/* +mlr put -q '@sum[$a][$b] += $x; end {emit @sum, "a", "b"}' data/* +mlr --from estimates.tbl put ' + for (k,v in $*) { + if (is_numeric(v) && k =~ "^[t-z].*$") { + $sum += v; $count += 1 + } + } + $mean = $sum / $count # no assignment if count unset' +mlr --from infile.dat put -f analyze.mlr +mlr --from infile.dat put 'tee > "./taps/data-".$a."-".$b, $*' +mlr --from infile.dat put 'tee | "gzip > ./taps/data-".$a."-".$b.".gz", $*' +mlr --from infile.dat put -q '@v=$*; dump | "jq .[]"' +mlr --from infile.dat put '(NR % 1000 == 0) { print > os.Stderr, "Checkpoint ".NR}' +.fi +.if n \{\ +.RE +.SS "DATA FORMATS" +.if n \{\ +.RS 0 +.\} +.nf + CSV/CSV-lite: comma-separated values with separate header line + TSV: same but with tabs in places of commas + +---------------------+ + | apple,bat,cog | + | 1,2,3 | Record 1: "apple => "1", "bat" => "2", "cog" => "3" + | 4,5,6 | Record 2: "apple" => "4", "bat" => "5", "cog" => "6" + +---------------------+ + + JSON (sequence or array of objects): + +---------------------+ + | { | + | "apple": 1, | Record 1: "apple" => "1", "bat" => "2", "cog" => "3" + | "bat": 2, | + | "cog": 3 | + | } | + | { | + | "dish": { | Record 2: "dish:egg" => "7", "dish:flint" => "8", "garlic" => "" + | "egg": 7, | + | "flint": 8 | + | }, | + | "garlic": "" | + | } | + +---------------------+ + + PPRINT: pretty-printed tabular + +---------------------+ + | apple bat cog | + | 1 2 3 | Record 1: "apple => "1", "bat" => "2", "cog" => "3" + | 4 5 6 | Record 2: "apple" => "4", "bat" => "5", "cog" => "6" + +---------------------+ + + Markdown tabular (supported for output only): + +-----------------------+ + | | apple | bat | cog | | + | | --- | --- | --- | | + | | 1 | 2 | 3 | | Record 1: "apple => "1", "bat" => "2", "cog" => "3" + | | 4 | 5 | 6 | | Record 2: "apple" => "4", "bat" => "5", "cog" => "6" + +-----------------------+ + + XTAB: pretty-printed transposed tabular + +---------------------+ + | apple 1 | Record 1: "apple" => "1", "bat" => "2", "cog" => "3" + | bat 2 | + | cog 3 | + | | + | dish 7 | Record 2: "dish" => "7", "egg" => "8" + | egg 8 | + +---------------------+ + + DKVP: delimited key-value pairs (Miller default format) + +---------------------+ + | apple=1,bat=2,cog=3 | Record 1: "apple" => "1", "bat" => "2", "cog" => "3" + | dish=7,egg=8,flint | Record 2: "dish" => "7", "egg" => "8", "3" => "flint" + +---------------------+ + + NIDX: implicitly numerically indexed (Unix-toolkit style) + +---------------------+ + | the quick brown | Record 1: "1" => "the", "2" => "quick", "3" => "brown" + | fox jumped | Record 2: "1" => "fox", "2" => "jumped" + +---------------------+ +.fi +.if n \{\ +.RE +.SH "OPTIONS" +.sp +In the following option flags, the version with "i" designates the input +stream, "o" the output stream, and the version without prefix sets the option +for both input and output stream. For example: --irs sets the input record +separator, --ors the output record separator, and --rs sets both the input and +output separator to the given value. +.SS "HELP OPTIONS" +.if n \{\ +.RS 0 +.\} +.nf + -h or --help Show this message. + --version Show the software version. + {verb name} --help Show verb-specific help. + --help-all-verbs Show help on all verbs. + -l or --list-all-verbs List only verb names. + -L List only verb names, one per line. + -f or --help-all-functions Show help on all built-in functions. + -F Show a bare listing of built-in functions by name. + -k or --help-all-keywords Show help on all keywords. + -K Show a bare listing of keywords by name. +.fi +.if n \{\ +.RE +.SS "VERB LIST" +.if n \{\ +.RS 0 +.\} +.nf + altkv bar bootstrap cat check clean-whitespace count-distinct count + count-similar cut decimate fill-down fill-empty filter flatten format-values + fraction gap grep group-by group-like having-fields head histogram json-parse + json-stringify join label least-frequent merge-fields most-frequent nest + nothing put regularize remove-empty-columns rename reorder repeat reshape + sample sec2gmtdate sec2gmt seqgen shuffle skip-trivial-records sort + sort-within-records stats1 stats2 step tac tail tee top unflatten uniq + unsparsify +.fi +.if n \{\ +.RE +.SS "FUNCTION LIST" +.if n \{\ +.RS 0 +.\} +.nf ++ +- +* +/ +// +** +pow +\&.+ +\&.- +\&.* +\&./ +% +~ +& +| +^ +<< +>> +>>> +bitcount +madd +msub +mmul +mexp +! +== +!= +> +>= +< +<= +=~ +!=~ +&& +|| +^^ +?? +??? +?: +\&. +capitalize +clean_whitespace +collapse_whitespace +gsub +lstrip +regextract +regextract_or_else +rstrip +strip +strlen +ssub +sub +substr0 +substr1 +substr +tolower +toupper +truncate +md5 +sha1 +sha256 +sha512 +abs +acos +acosh +asin +asinh +atan +atan2 +atanh +cbrt +ceil +cos +cosh +erf +erfc +exp +expm1 +floor +invqnorm +log +log10 +log1p +logifit +max +min +qnorm +round +sgn +sin +sinh +sqrt +tan +tanh +roundm +urand +urandint +urandrange +urand32 +gmt2sec +sec2gmt +sec2gmtdate +systime +systimeint +uptime +strftime +strptime +dhms2fsec +dhms2sec +fsec2dhms +fsec2hms +hms2fsec +hms2sec +sec2dhms +sec2hms +is_absent +is_array +is_bool +is_boolean +is_empty +is_empty_map +is_error +is_float +is_int +is_map +is_nonempty_map +is_not_empty +is_not_map +is_not_array +is_not_null +is_null +is_numeric +is_present +is_string +asserting_absent +asserting_array +asserting_bool +asserting_boolean +asserting_error +asserting_empty +asserting_empty_map +asserting_float +asserting_int +asserting_map +asserting_nonempty_map +asserting_not_empty +asserting_not_map +asserting_not_array +asserting_not_null +asserting_null +asserting_numeric +asserting_present +asserting_string +typeof +boolean +float +fmtnum +hexfmt +int +joink +joinv +joinkv +splita +splitax +splitkv +splitkvx +splitnv +splitnvx +string +append +arrayify +depth +flatten +get_keys +get_values +haskey +json_parse +json_stringify +leafcount +length +mapdiff +mapexcept +mapselect +mapsum +unflatten +hostname +os +system +version +Please use "mlr --help-function {function name}" for function-specific help. +.fi +.if n \{\ +.RE +.SS "I/O FORMATTING" +.if n \{\ +.RS 0 +.\} +.nf + + --idkvp --odkvp --dkvp Delimited key-value pairs, e.g "a=1,b=2" + (this is Miller's default format). + + --inidx --onidx --nidx Implicitly-integer-indexed fields + (Unix-toolkit style). + -T Synonymous with "--nidx --fs tab". + + --icsv --ocsv --csv Comma-separated value (or tab-separated + with --fs tab, etc.) + + --itsv --otsv --tsv Keystroke-savers for "--icsv --ifs tab", + "--ocsv --ofs tab", "--csv --fs tab". + --iasv --oasv --asv Similar but using ASCII FS 0x1f and RS 0x1e\en", + --iusv --ousv --usv Similar but using Unicode FS U+241F (UTF-8 0xe2909f)\en", + and RS U+241E (UTF-8 0xe2909e)\en", + + --icsvlite --ocsvlite --csvlite Comma-separated value (or tab-separated + with --fs tab, etc.). The 'lite' CSV does not handle + RFC-CSV double-quoting rules; is slightly faster; + and handles heterogeneity in the input stream via + empty newline followed by new header line. See also + http://johnkerl.org/miller/doc/file-formats.html#CSV/TSV/etc. + + --itsvlite --otsvlite --tsvlite Keystroke-savers for "--icsvlite --ifs tab", + "--ocsvlite --ofs tab", "--csvlite --fs tab". + -t Synonymous with --tsvlite. + --iasvlite --oasvlite --asvlite Similar to --itsvlite et al. but using ASCII FS 0x1f and RS 0x1e\en", + --iusvlite --ousvlite --usvlite Similar to --itsvlite et al. but using Unicode FS U+241F (UTF-8 0xe2909f)\en", + and RS U+241E (UTF-8 0xe2909e)\en", + + --ipprint --opprint --pprint Pretty-printed tabular (produces no + output until all input is in). + --right Right-justifies all fields for PPRINT output. + --barred Prints a border around PPRINT output + (only available for output). + + --omd Markdown-tabular (only available for output). + + --ixtab --oxtab --xtab Pretty-printed vertical-tabular. + --xvright Right-justifies values for XTAB format. + + --ijson --ojson --json JSON tabular: sequence or list of one-level + maps: {...}{...} or [{...},{...}]. + --json-map-arrays-on-input JSON arrays are unmillerable. --json-map-arrays-on-input + --json-skip-arrays-on-input is the default: arrays are converted to integer-indexed + --json-fatal-arrays-on-input maps. The other two options cause them to be skipped, or + to be treated as errors. Please use the jq tool for full + JSON (pre)processing. + --jvstack Put one key-value pair per line for JSON output. + --no-jvstack Put objects/arrays all on one line for JSON output. + --jsonx --ojsonx Keystroke-savers for --json --jvstack + --jsonx --ojsonx and --ojson --jvstack, respectively. + --jlistwrap Wrap JSON output in outermost [ ]. + --jknquoteint Do not quote non-string map keys in JSON output. + --jvquoteall Quote map values in JSON output, even if they're + numeric. + --oflatsep {string} Separator for flattening multi-level JSON keys, + e.g. '{"a":{"b":3}}' becomes a:b => 3 for + non-JSON formats. Defaults to ..\en", + + -p is a keystroke-saver for --nidx --fs space --repifs + + Examples: --csv for CSV-formatted input and output; --idkvp --opprint for + DKVP-formatted input and pretty-printed output. + + Please use --iformat1 --oformat2 rather than --format1 --oformat2. + The latter sets up input and output flags for format1, not all of which + are overridden in all cases by setting output format to format2. + +.fi +.if n \{\ +.RE +.SS "COMMENTS IN DATA" +.if n \{\ +.RS 0 +.\} +.nf + --skip-comments Ignore commented lines (prefixed by "#") + within the input. + --skip-comments-with {string} Ignore commented lines within input, with + specified prefix. + --pass-comments Immediately print commented lines (prefixed by "#") + within the input. + --pass-comments-with {string} Immediately print commented lines within input, with + specified prefix. +Notes: +* Comments are only honored at the start of a line. +* In the absence of any of the above four options, comments are data like + any other text. +* When pass-comments is used, comment lines are written to standard output + immediately upon being read; they are not part of the record stream. + Results may be counterintuitive. A suggestion is to place comments at the + start of data files. +.fi +.if n \{\ +.RE +.SS "FORMAT-CONVERSION KEYSTROKE-SAVERS" +.if n \{\ +.RS 0 +.\} +.nf +As keystroke-savers for format-conversion you may use the following: + --c2t --c2d --c2n --c2j --c2x --c2p --c2m + --t2c --t2d --t2n --t2j --t2x --t2p --t2m + --d2c --d2t --d2n --d2j --d2x --d2p --d2m + --n2c --n2t --n2d --n2j --n2x --n2p --n2m + --j2c --j2t --j2d --j2n --j2x --j2p --j2m + --x2c --x2t --x2d --x2n --x2j --x2p --x2m + --p2c --p2t --p2d --p2n --p2j --p2x --p2m +The letters c t d n j x p m refer to formats CSV, TSV, DKVP, NIDX, JSON, XTAB, +PPRINT, and markdown, respectively. Note that markdown format is available for +output only. +.fi +.if n \{\ +.RE +.SS "COMPRESSED I/O" +.if n \{\ +.RS 0 +.\} +.nf + Decompression done within the Miller process itself: + --gzin Uncompress gzip within the Miller process. Done by default if file ends in ".gz". + --bz2in Uncompress bz2ip within the Miller process. Done by default if file ends in ".bz2". + --zin Uncompress zlib within the Miller process. Done by default if file ends in ".z". + Decompression done outside the Miller processn --prepipe {command} You can, of course, already do without this for single input files, + e.g. "gunzip < myfile.csv.gz | mlr ...". + However, when multiple input files are present, between-file separations are + lost; also, the FILENAME variable doesn't iterate. Using --prepipe you can + specify an action to be taken on each input file. This prepipe command must + be able to read from standard input; it will be invoked with + {command} < {filename}. + --prepipex {command} Like --prepipe with one exception: doesn't insert '<' between + command and filename at runtime. Useful for some commands like 'unzip -qc' which don't + read standard input. + Examples: + mlr --prepipe 'gunzip' + mlr --prepipe 'zcat -cf' + mlr --prepipe 'xz -cd' + mlr --prepipe cat + Note that this feature is quite general and is not limited to decompression + utilities. You can use it to apply per-file filters of your choice. + For output compression (or other) utilities, simply pipe the output: + mlr ... | {your compression command} + Lastly, note that if --prepipe is specified, it replaces any decisions that might + have been made based on the file suffix. Also, --gzin/--bz2in/--zin are ignored + if --prepipe is also specified. +.fi +.if n \{\ +.RE +.SS "SEPARATORS" +.if n \{\ +.RS 0 +.\} +.nf +.fi +.if n \{\ +.RE +.SS "CSV-SPECIFIC OPTIONS" +.if n \{\ +.RS 0 +.\} +.nf + --implicit-csv-header Use 1,2,3,... as field labels, rather than from line 1 + of input files. Tip: combine with "label" to recreate + missing headers. + --no-implicit-csv-header Do not use --implicit-csv-header. This is the default + anyway -- the main use is for the flags to 'mlr join' if you have + main file(s) which are headerless but you want to join in on + a file which does have a CSV header. Then you could use + 'mlr --csv --implicit-csv-header join --no-implicit-csv-header + -l your-join-in-with-header.csv ... your-headerless.csv' + --allow-ragged-csv-input|--ragged If a data line has fewer fields than the header line, + fill remaining keys with empty string. If a data line has more + fields than the header line, use integer field labels as in + the implicit-header case. + --headerless-csv-output Print only CSV data lines. + -N Keystroke-saver for --implicit-csv-header --headerless-csv-output. +.fi +.if n \{\ +.RE +.SS "DOUBLE-QUOTING FOR CSV/CSVLITE OUTPUT" +.if n \{\ +.RS 0 +.\} +.nf +.fi +.if n \{\ +.RE +.SS "NUMERICAL FORMATTING" +.if n \{\ +.RS 0 +.\} +.nf +.fi +.if n \{\ +.RE +.SS "OTHER OPTIONS" +.if n \{\ +.RS 0 +.\} +.nf + --seed {n} with n of the form 12345678 or 0xcafefeed. For put/filter + urand()/urandint()/urand32(). + --nr-progress-mod {m}, with m a positive integer: print filename and record + count to os.Stderr every m input records. + --from {filename} Use this to specify an input file before the verb(s), + rather than after. May be used more than once. Example: + "mlr --from a.dat --from b.dat cat" is the same as + "mlr cat a.dat b.dat". + --mfrom {filenames} -- Use this to specify one of more input files before the verb(s), + rather than after. May be used more than once. + The list of filename must end with "--". This is useful + for example since "--from *.csv" doesn't do what you might + hope but "--mfrom *.csv --" does. + --load {filename} Load DSL script file for all put/filter operations on the command line. + If the name following --load is a directory, load all "*.mlr" files + in that directory. This is just like "put -f" and "filter -f" + except it's up-front on the command line, so you can do something like + alias mlr='mlr --load ~/myscripts' if you like. + --mload {names} -- Like --load but works with more than one filename, + e.g. '--mload *.mlr --'. + -n Process no input files, nor standard input either. Useful + for mlr put with begin/end statements only. (Same as --from + /dev/null.) Also useful in "mlr -n put -v '...'" for + analyzing abstract syntax trees (if that's your thing). + -I Process files in-place. For each file name on the command + line, output is written to a temp file in the same + directory, which is then renamed over the original. Each + file is processed in isolation: if the output format is + CSV, CSV headers will be present in each output file; + statistics are only over each file's own records; and so on. +.fi +.if n \{\ +.RE +.SS "THEN-CHAINING" +.if n \{\ +.RS 0 +.\} +.nf +Output of one verb may be chained as input to another using "then", e.g. + mlr stats1 -a min,mean,max -f flag,u,v -g color then sort -f color +.fi +.if n \{\ +.RE +.SS "AUXILIARY COMMANDS" +.if n \{\ +.RS 0 +.\} +.nf +Miller has a few otherwise-standalone executables packaged within it. +They do not participate in any other parts of Miller. +Please use "mlr aux-list" for more information. +.fi +.if n \{\ +.RE +.SH "MLRRC" +.if n \{\ +.RS 0 +.\} +.nf +You can set up personal defaults via a $HOME/.mlrrc and/or ./.mlrrc. +For example, if you usually process CSV, then you can put "--csv" in your .mlrrc file +and that will be the default input/output format unless otherwise specified on the command line. + +The .mlrrc file format is one "--flag" or "--option value" per line, with the leading "--" optional. +Hash-style comments and blank lines are ignored. + +Sample .mlrrc: +# Input and output formats are CSV by default (unless otherwise specified +# on the mlr command line): +csv +# These are no-ops for CSV, but when I do use JSON output, I want these +# pretty-printing options to be used: +jvstack +jlistwrap + +How to specify location of .mlrrc: +* If $MLRRC is set: + o If its value is "__none__" then no .mlrrc files are processed. + o Otherwise, its value (as a filename) is loaded and processed. If there are syntax + errors, they abort mlr with a usage message (as if you had mistyped something on the + command line). If the file can't be loaded at all, though, it is silently skipped. + o Any .mlrrc in your home directory or current directory is ignored whenever $MLRRC is + set in the environment. +* Otherwise: + o If $HOME/.mlrrc exists, it's then processed as above. + o If ./.mlrrc exists, it's then also processed as above. + (I.e. current-directory .mlrrc defaults are stacked over home-directory .mlrrc defaults.) + +See also: +https://miller.readthedocs.io/en/latest/customization.html +.fi +.if n \{\ +.RE +.SH "VERBS" +.sp + +.SS "altkv" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr altkv [options] +Given fields with values of the form a,b,c,d,e,f emits a=b,c=d,e=f pairs. +Options: +-h|--help Show this message. +.fi +.if n \{\ +.RE +.SS "bar" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr bar [options] +Replaces a numeric field with a number of asterisks, allowing for cheesy +bar plots. These align best with --opprint or --oxtab output format. +Options: +-f {a,b,c} Field names to convert to bars. +--lo {lo} Lower-limit value for min-width bar: default '0.000000'. +--hi {hi} Upper-limit value for max-width bar: default '100.000000'. +-w {n} Bar-field width: default '40'. +--auto Automatically computes limits, ignoring --lo and --hi. + Holds all records in memory before producing any output. +-c {character} Fill character: default '*'. +-x {character} Out-of-bounds character: default '#'. +-b {character} Blank character: default '.'. +Nominally the fill, out-of-bounds, and blank characters will be strings of length 1. +However you can make them all longer if you so desire. +-h|--help Show this message. +.fi +.if n \{\ +.RE +.SS "bootstrap" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr bootstrap [options] +Emits an n-sample, with replacement, of the input records. +See also mlr sample and mlr shuffle. +Options: + -n Number of samples to output. Defaults to number of input records. + Must be non-negative. +-h|--help Show this message. +.fi +.if n \{\ +.RE +.SS "cat" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr cat [options] +Passes input records directly to output. Most useful for format conversion. +Options: +-n Prepend field "n" to each record with record-counter starting at 1. +-N {name} Prepend field {name} to each record with record-counter starting at 1. +-g {a,b,c} Optional group-by-field names for counters, e.g. a,b,c +-h|--help Show this message. +.fi +.if n \{\ +.RE +.SS "check" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr check [options] +Consumes records without printing any output. +Useful for doing a well-formatted check on input data. +Options: +-h|--help Show this message. +.fi +.if n \{\ +.RE +.SS "clean-whitespace" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr clean-whitespace [options] +For each record, for each field in the record, whitespace-cleans the keys and/or +values. Whitespace-cleaning entails stripping leading and trailing whitespace, +and replacing multiple whitespace with singles. For finer-grained control, +please see the DSL functions lstrip, rstrip, strip, collapse_whitespace, +and clean_whitespace. + +Options: +-k|--keys-only Do not touch values. +-v|--values-only Do not touch keys. +It is an error to specify -k as well as -v -- to clean keys and values, +leave off -k as well as -v. +-h|--help Show this message. +.fi +.if n \{\ +.RE +.SS "count-distinct" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr count-distinct [options] +Prints number of records having distinct values for specified field names. +Same as uniq -c. + +Options: +-f {a,b,c} Field names for distinct count. +-n Show only the number of distinct values. Not compatible with -u. +-o {name} Field name for output count. Default "count". + Ignored with -u. +-u Do unlashed counts for multiple field names. With -f a,b and + without -u, computes counts for distinct combinations of a + and b field values. With -f a,b and with -u, computes counts + for distinct a field values and counts for distinct b field + values separately. +.fi +.if n \{\ +.RE +.SS "count" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr count [options] +Prints number of records, optionally grouped by distinct values for specified field names. +Options: +-g {a,b,c} Optional group-by-field names for counts, e.g. a,b,c +-n {n} Show only the number of distinct values. Not interesting without -g. +-o {name} Field name for output-count. Default "count". +-h|--help Show this message. +.fi +.if n \{\ +.RE +.SS "count-similar" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr count-similar [options] +Ingests all records, then emits each record augmented by a count of +the number of other records having the same group-by field values. +Options: +-g {a,b,c} Group-by-field names for counts, e.g. a,b,c +-o {name} Field name for output-counts. Defaults to "count". +-h|--help Show this message. +.fi +.if n \{\ +.RE +.SS "cut" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr cut [options] +Passes through input records with specified fields included/excluded. +Options: + -f {a,b,c} Comma-separated field names for cut, e.g. a,b,c. + -o Retain fields in the order specified here in the argument list. + Default is to retain them in the order found in the input data. + -x|--complement Exclude, rather than include, field names specified by -f. + -r Treat field names as regular expressions. "ab", "a.*b" will + match any field name containing the substring "ab" or matching + "a.*b", respectively; anchors of the form "^ab$", "^a.*b$" may + be used. The -o flag is ignored when -r is present. +-h|--help Show this message. +Examples: + mlr cut -f hostname,status + mlr cut -x -f hostname,status + mlr cut -r -f '^status$,sda[0-9]' + mlr cut -r -f '^status$,"sda[0-9]"' + mlr cut -r -f '^status$,"sda[0-9]"i' (this is case-insensitive) +.fi +.if n \{\ +.RE +.SS "decimate" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr decimate [options] +Passes through one of every n records, optionally by category. +Options: + -b Decimate by printing first of every n. + -e Decimate by printing last of every n (default). + -g {a,b,c} Optional group-by-field names for decimate counts, e.g. a,b,c. + -n {n} Decimation factor (default 10). +-h|--help Show this message. +.fi +.if n \{\ +.RE +.SS "fill-down" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr fill-down [options] +If a given record has a missing value for a given field, fill that from +the corresponding value from a previous record, if any. +By default, a 'missing' field either is absent, or has the empty-string value. +With -a, a field is 'missing' only if it is absent. + +Options: + --all Operate on all fields in the input. + -a|--only-if-absent If a given record has a missing value for a given field, + fill that from the corresponding value from a previous record, if any. + By default, a 'missing' field either is absent, or has the empty-string value. + With -a, a field is 'missing' only if it is absent. + -f Field names for fill-down. + -h|--help Show this message. +.fi +.if n \{\ +.RE +.SS "fill-empty" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr fill-empty [options] +Fills empty-string fields with specified fill-value. +Options: +-v {string} Fill-value: defaults to "N/A" +.fi +.if n \{\ +.RE +.SS "filter" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr put [options] {DSL expression} +Options: +-f {file name} File containing a DSL expression. If the filename is a directory, + all *.mlr files in that directory are loaded. + +-e {expression} You can use this after -f to add an expression. Example use + case: define functions/subroutines in a file you specify with -f, then call + them with an expression you specify with -e. + +(If you mix -e and -f then the expressions are evaluated in the order encountered. +Since the expression pieces are simply concatenated, please be sure to use intervening +semicolons to separate expressions.) + +-s name=value: Predefines out-of-stream variable @name to have + Thus mlr put -s foo=97 '$column += @foo' is like + mlr put 'begin {@foo = 97} $column += @foo'. + The value part is subject to type-inferencing. + May be specified more than once, e.g. -s name1=value1 -s name2=value2. + Note: the value may be an environment variable, e.g. -s sequence=$SEQUENCE + +-x (default false) Prints records for which {expression} evaluates to false, not true, + i.e. invert the sense of the filter expression. + +-q Does not include the modified record in the output stream. + Useful for when all desired output is in begin and/or end blocks. + +-S and -F: There are no-ops in Miller 6 and above, since now type-inferencing is done + by the record-readers before filter/put is executed. Supported as no-op pass-through + flags for backward compatibility. + +-h|--help Show this message. + +Parser-info options: + +-w Print warnings about things like uninitialized variables. + +-W Same as -w, but exit the process if there are any warnings. + +-p Prints the expressions's AST (abstract syntax tree), which gives full + transparency on the precedence and associativity rules of Miller's grammar, + to stdout. + +-d Like -p but uses a parenthesized-expression format for the AST. + +-D Like -d but with output all on one line. + +-E Echo DSL expression before printing parse-tree + +-v Same as -E -p. + +-X Exit after parsing but before stream-processing. Useful with -v/-d/-D, if you + only want to look at parser information. +.fi +.if n \{\ +.RE +.SS "flatten" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr flatten [options] +Flattens multi-level maps to single-level ones. Example: field with name 'a' +and value '{"b": { "c": 4 }}' becomes name 'a.b.c' and value 4. +Options: +-f Comma-separated list of field names to flatten (default all). +-s Separator, defaulting to mlr --oflatsep value. +-h|--help Show this message. +.fi +.if n \{\ +.RE +.SS "format-values" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr format-values [options] +Applies format strings to all field values, depending on autodetected type. +* If a field value is detected to be integer, applies integer format. +* Else, if a field value is detected to be float, applies float format. +* Else, applies string format. + +Note: this is a low-keystroke way to apply formatting to many fields. To get +finer control, please see the fmtnum function within the mlr put DSL. + +Note: this verb lets you apply arbitrary format strings, which can produce +undefined behavior and/or program crashes. See your system's "man printf". + +Options: +-i {integer format} Defaults to "%d". + Examples: "%06lld", "%08llx". + Note that Miller integers are long long so you must use + formats which apply to long long, e.g. with ll in them. + Undefined behavior results otherwise. +-f {float format} Defaults to "%f". + Examples: "%8.3lf", "%.6le". + Note that Miller floats are double-precision so you must + use formats which apply to double, e.g. with l[efg] in them. + Undefined behavior results otherwise. +-s {string format} Defaults to "%s". + Examples: "_%s", "%08s". + Note that you must use formats which apply to string, e.g. + with s in them. Undefined behavior results otherwise. +-n Coerce field values autodetected as int to float, and then + apply the float format. +.fi +.if n \{\ +.RE +.SS "fraction" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr fraction [options] +For each record's value in specified fields, computes the ratio of that +value to the sum of values in that field over all input records. +E.g. with input records x=1 x=2 x=3 and x=4, emits output records +x=1,x_fraction=0.1 x=2,x_fraction=0.2 x=3,x_fraction=0.3 and x=4,x_fraction=0.4 + +Note: this is internally a two-pass algorithm: on the first pass it retains +input records and accumulates sums; on the second pass it computes quotients +and emits output records. This means it produces no output until all input is read. + +Options: +-f {a,b,c} Field name(s) for fraction calculation +-g {d,e,f} Optional group-by-field name(s) for fraction counts +-p Produce percents [0..100], not fractions [0..1]. Output field names + end with "_percent" rather than "_fraction" +-c Produce cumulative distributions, i.e. running sums: each output + value folds in the sum of the previous for the specified group + E.g. with input records x=1 x=2 x=3 and x=4, emits output records + x=1,x_cumulative_fraction=0.1 x=2,x_cumulative_fraction=0.3 + x=3,x_cumulative_fraction=0.6 and x=4,x_cumulative_fraction=1.0 +.fi +.if n \{\ +.RE +.SS "gap" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr gap [options] +Emits an empty record every n records, or when certain values change. +Options: +Emits an empty record every n records, or when certain values change. +-g {a,b,c} Print a gap whenever values of these fields (e.g. a,b,c) changes. +-n {n} Print a gap every n records. +One of -f or -g is required. +-n is ignored if -g is present. +-h|--help Show this message. +.fi +.if n \{\ +.RE +.SS "grep" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr grep [options] {regular expression} +Passes through records which match the regular expression. +Options: +-i Use case-insensitive search. +-v Invert: pass through records which do not match the regex. +-h|--help Show this message. +Note that "mlr filter" is more powerful, but requires you to know field names. +By contrast, "mlr grep" allows you to regex-match the entire record. It does +this by formatting each record in memory as DKVP, using command-line-specified +ORS/OFS/OPS, and matching the resulting line against the regex specified +here. In particular, the regex is not applied to the input stream: if you +have CSV with header line "x,y,z" and data line "1,2,3" then the regex will +be matched, not against either of these lines, but against the DKVP line +"x=1,y=2,z=3". Furthermore, not all the options to system grep are supported, +and this command is intended to be merely a keystroke-saver. To get all the +features of system grep, you can do + "mlr --odkvp ... | grep ... | mlr --idkvp ..." +.fi +.if n \{\ +.RE +.SS "group-by" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr group-by [options] {comma-separated field names} +Outputs records in batches having identical values at specified field names.Options: +-h|--help Show this message. +.fi +.if n \{\ +.RE +.SS "group-like" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr group-like [options] +Outputs records in batches having identical field names.Options: +-h|--help Show this message. +.fi +.if n \{\ +.RE +.SS "having-fields" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr having-fields [options] +Conditionally passes through records depending on each record's field names. +Options: + --at-least {comma-separated names} + --which-are {comma-separated names} + --at-most {comma-separated names} + --all-matching {regular expression} + --any-matching {regular expression} + --none-matching {regular expression} +Examples: + mlr having-fields --which-are amount,status,owner + mlr having-fields --any-matching 'sda[0-9]' + mlr having-fields --any-matching '"sda[0-9]"' + mlr having-fields --any-matching '"sda[0-9]"i' (this is case-insensitive) +.fi +.if n \{\ +.RE +.SS "head" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr head [options] +Passes through the first n records, optionally by category. +Options: +-g {a,b,c} Optional group-by-field names for head counts, e.g. a,b,c. +-n {n} Head-count to print. Default 10. +-h|--help Show this message. +.fi +.if n \{\ +.RE +.SS "histogram" +.if n \{\ +.RS 0 +.\} +.nf +Just a histogram. Input values < lo or > hi are not counted. +Usage: mlr histogram [options] +-f {a,b,c} Value-field names for histogram counts +--lo {lo} Histogram low value +--hi {hi} Histogram high value +--nbins {n} Number of histogram bins +--auto Automatically computes limits, ignoring --lo and --hi. + Holds all values in memory before producing any output. +-o {prefix} Prefix for output field name. Default: no prefix. +-h|--help Show this message. +.fi +.if n \{\ +.RE +.SS "json-parse" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr json-parse [options] +Tries to convert string field values to parsed JSON, e.g. "[1,2,3]" -> [1,2,3]. +Options: +-f {...} Comma-separated list of field names to json-parse (default all). +-h|--help Show this message. +.fi +.if n \{\ +.RE +.SS "json-stringify" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr json-stringify [options] +Produces string field values from field-value data, e.g. [1,2,3] -> "[1,2,3]". +Options: +-f {...} Comma-separated list of field names to json-parse (default all). +--jvstack Produce multi-line JSON output. +--no-jvstack Produce single-line JSON output per record (default). +-h|--help Show this message. +.fi +.if n \{\ +.RE +.SS "join" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr sort {flags} +Sorts records primarily by the first specified field, secondarily by the second +field, and so on. (Any records not having all specified sort keys will appear +at the end of the output, in the order they were encountered, regardless of the +specified sort order.) The sort is stable: records that compare equal will sort +in the order they were encountered in the input record stream. + +Options: +-f {comma-separated field names} Lexical ascending +-n {comma-separated field names} Numerical ascending; nulls sort last +-nf {comma-separated field names} Same as -n +-r {comma-separated field names} Lexical descending +-nr {comma-separated field names} Numerical descending; nulls sort first +-h|--help Show this message. + +Example: + mlr sort -f a,b -nr x,y,z +which is the same as: + mlr sort -f a -f b -nr x -nr y -nr z +.fi +.if n \{\ +.RE +.SS "label" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr label [options] {new1,new2,new3,...} +Given n comma-separated names, renames the first n fields of each record to +have the respective name. (Fields past the nth are left with their original +names.) Particularly useful with --inidx or --implicit-csv-header, to give +useful names to otherwise integer-indexed fields. + +Options: +-h|--help Show this message. +.fi +.if n \{\ +.RE +.SS "least-frequent" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr least-frequent [options] +Shows the least frequently occurring distinct values for specified field names. +The first entry is the statistical anti-mode; the remaining are runners-up. +Options: +-f {one or more comma-separated field names}. Required flag. +-n {count}. Optional flag defaulting to 10. +-b Suppress counts; show only field values. +-o {name} Field name for output count. Default "count". +See also "mlr most-frequent". +.fi +.if n \{\ +.RE +.SS "merge-fields" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr merge-fields [options] +Computes univariate statistics for each input record, accumulated across +specified fields. +Options: +-a {sum,count,...} Names of accumulators. One or more of: + count Count instances of fields + mode Find most-frequently-occurring values for fields; first-found wins tie + antimode Find least-frequently-occurring values for fields; first-found wins tie + sum Compute sums of specified fields + mean Compute averages (sample means) of specified fields + var Compute sample variance of specified fields + stddev Compute sample standard deviation of specified fields + meaneb Estimate error bars for averages (assuming no sample autocorrelation) + skewness Compute sample skewness of specified fields + kurtosis Compute sample kurtosis of specified fields + min Compute minimum values of specified fields + max Compute maximum values of specified fields +-f {a,b,c} Value-field names on which to compute statistics. Requires -o. +-r {a,b,c} Regular expressions for value-field names on which to compute + statistics. Requires -o. +-c {a,b,c} Substrings for collapse mode. All fields which have the same names + after removing substrings will be accumulated together. Please see + examples below. +-i Use interpolated percentiles, like R's type=7; default like type=1. + Not sensical for string-valued fields. +-o {name} Output field basename for -f/-r. +-k Keep the input fields which contributed to the output statistics; + the default is to omit them. + +String-valued data make sense unless arithmetic on them is required, +e.g. for sum, mean, interpolated percentiles, etc. In case of mixed data, +numbers are less than strings. + +Example input data: "a_in_x=1,a_out_x=2,b_in_y=4,b_out_x=8". +Example: mlr merge-fields -a sum,count -f a_in_x,a_out_x -o foo + produces "b_in_y=4,b_out_x=8,foo_sum=3,foo_count=2" since "a_in_x,a_out_x" are + summed over. +Example: mlr merge-fields -a sum,count -r in_,out_ -o bar + produces "bar_sum=15,bar_count=4" since all four fields are summed over. +Example: mlr merge-fields -a sum,count -c in_,out_ + produces "a_x_sum=3,a_x_count=2,b_y_sum=4,b_y_count=1,b_x_sum=8,b_x_count=1" + since "a_in_x" and "a_out_x" both collapse to "a_x", "b_in_y" collapses to + "b_y", and "b_out_x" collapses to "b_x". +.fi +.if n \{\ +.RE +.SS "most-frequent" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr most-frequent [options] +Shows the most frequently occurring distinct values for specified field names. +The first entry is the statistical mode; the remaining are runners-up. +Options: +-f {one or more comma-separated field names}. Required flag. +-n {count}. Optional flag defaulting to 10. +-b Suppress counts; show only field values. +-o {name} Field name for output count. Default "count". +See also "mlr least-frequent". +.fi +.if n \{\ +.RE +.SS "nest" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr nest [options] +Explodes specified field values into separate fields/records, or reverses this. +Options: + --explode,--implode One is required. + --values,--pairs One is required. + --across-records,--across-fields One is required. + -f {field name} Required. + --nested-fs {string} Defaults to ";". Field separator for nested values. + --nested-ps {string} Defaults to ":". Pair separator for nested key-value pairs. + --evar {string} Shorthand for --explode --values ---across-records --nested-fs {string} + --ivar {string} Shorthand for --implode --values ---across-records --nested-fs {string} +Please use "mlr --usage-separator-options" for information on specifying separators. + +Examples: + + mlr nest --explode --values --across-records -f x + with input record "x=a;b;c,y=d" produces output records + "x=a,y=d" + "x=b,y=d" + "x=c,y=d" + Use --implode to do the reverse. + + mlr nest --explode --values --across-fields -f x + with input record "x=a;b;c,y=d" produces output records + "x_1=a,x_2=b,x_3=c,y=d" + Use --implode to do the reverse. + + mlr nest --explode --pairs --across-records -f x + with input record "x=a:1;b:2;c:3,y=d" produces output records + "a=1,y=d" + "b=2,y=d" + "c=3,y=d" + + mlr nest --explode --pairs --across-fields -f x + with input record "x=a:1;b:2;c:3,y=d" produces output records + "a=1,b=2,c=3,y=d" + +Notes: +* With --pairs, --implode doesn't make sense since the original field name has + been lost. +* The combination "--implode --values --across-records" is non-streaming: + no output records are produced until all input records have been read. In + particular, this means it won't work in tail -f contexts. But all other flag + combinations result in streaming (tail -f friendly) data processing. +* It's up to you to ensure that the nested-fs is distinct from your data's IFS: + e.g. by default the former is semicolon and the latter is comma. +See also mlr reshape. +.fi +.if n \{\ +.RE +.SS "nothing" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr nothing [options] +Drops all input records. Useful for testing, or after tee/print/etc. have +produced other output. +Options: +-h|--help Show this message. +.fi +.if n \{\ +.RE +.SS "put" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr put [options] {DSL expression} +Options: +-f {file name} File containing a DSL expression. If the filename is a directory, + all *.mlr files in that directory are loaded. + +-e {expression} You can use this after -f to add an expression. Example use + case: define functions/subroutines in a file you specify with -f, then call + them with an expression you specify with -e. + +(If you mix -e and -f then the expressions are evaluated in the order encountered. +Since the expression pieces are simply concatenated, please be sure to use intervening +semicolons to separate expressions.) + +-s name=value: Predefines out-of-stream variable @name to have + Thus mlr put -s foo=97 '$column += @foo' is like + mlr put 'begin {@foo = 97} $column += @foo'. + The value part is subject to type-inferencing. + May be specified more than once, e.g. -s name1=value1 -s name2=value2. + Note: the value may be an environment variable, e.g. -s sequence=$SEQUENCE + +-x (default false) Prints records for which {expression} evaluates to false, not true, + i.e. invert the sense of the filter expression. + +-q Does not include the modified record in the output stream. + Useful for when all desired output is in begin and/or end blocks. + +-S and -F: There are no-ops in Miller 6 and above, since now type-inferencing is done + by the record-readers before filter/put is executed. Supported as no-op pass-through + flags for backward compatibility. + +-h|--help Show this message. + +Parser-info options: + +-w Print warnings about things like uninitialized variables. + +-W Same as -w, but exit the process if there are any warnings. + +-p Prints the expressions's AST (abstract syntax tree), which gives full + transparency on the precedence and associativity rules of Miller's grammar, + to stdout. + +-d Like -p but uses a parenthesized-expression format for the AST. + +-D Like -d but with output all on one line. + +-E Echo DSL expression before printing parse-tree + +-v Same as -E -p. + +-X Exit after parsing but before stream-processing. Useful with -v/-d/-D, if you + only want to look at parser information. +.fi +.if n \{\ +.RE +.SS "regularize" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr regularize [options] +Outputs records sorted lexically ascending by keys.Options: +-h|--help Show this message. +.fi +.if n \{\ +.RE +.SS "remove-empty-columns" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr remove-empty-columns [options] +Omits fields which are empty on every input row. Non-streaming. +Options: +-h|--help Show this message. +.fi +.if n \{\ +.RE +.SS "rename" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr rename [options] {old1,new1,old2,new2,...} +Renames specified fields. +Options: +-r Treat old field names as regular expressions. "ab", "a.*b" + will match any field name containing the substring "ab" or + matching "a.*b", respectively; anchors of the form "^ab$", + "^a.*b$" may be used. New field names may be plain strings, + or may contain capture groups of the form "\e1" through + "\e9". Wrapping the regex in double quotes is optional, but + is required if you wish to follow it with 'i' to indicate + case-insensitivity. +-g Do global replacement within each field name rather than + first-match replacement. +-h|--help Show this message. +Examples: +mlr rename old_name,new_name' +mlr rename old_name_1,new_name_1,old_name_2,new_name_2' +mlr rename -r 'Date_[0-9]+,Date,' Rename all such fields to be "Date" +mlr rename -r '"Date_[0-9]+",Date' Same +mlr rename -r 'Date_([0-9]+).*,\e1' Rename all such fields to be of the form 20151015 +mlr rename -r '"name"i,Name' Rename "name", "Name", "NAME", etc. to "Name" +.fi +.if n \{\ +.RE +.SS "reorder" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr reorder [options] +Moves specified names to start of record, or end of record. +Options: +-e Put specified field names at record end: default is to put them at record start. +-f {a,b,c} Field names to reorder. +-b {x} Put field names specified with -f before field name specified by {x}, + if any. If {x} isn't present in a given record, the specified fields + will not be moved. +-a {x} Put field names specified with -f after field name specified by {x}, + if any. If {x} isn't present in a given record, the specified fields + will not be moved. +-h|--help Show this message. + +Examples: +mlr reorder -f a,b sends input record "d=4,b=2,a=1,c=3" to "a=1,b=2,d=4,c=3". +mlr reorder -e -f a,b sends input record "d=4,b=2,a=1,c=3" to "d=4,c=3,a=1,b=2". +.fi +.if n \{\ +.RE +.SS "repeat" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr repeat [options] +Copies input records to output records multiple times. +Options must be exactly one of the following: +-n {repeat count} Repeat each input record this many times. +-f {field name} Same, but take the repeat count from the specified + field name of each input record. +-h|--help Show this message. +Example: + echo x=0 | mlr repeat -n 4 then put '$x=urand()' +produces: + x=0.488189 + x=0.484973 + x=0.704983 + x=0.147311 +Example: + echo a=1,b=2,c=3 | mlr repeat -f b +produces: + a=1,b=2,c=3 + a=1,b=2,c=3 +Example: + echo a=1,b=2,c=3 | mlr repeat -f c +produces: + a=1,b=2,c=3 + a=1,b=2,c=3 + a=1,b=2,c=3 +.fi +.if n \{\ +.RE +.SS "reshape" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr reshape [options] +Wide-to-long options: + -i {input field names} -o {key-field name,value-field name} + -r {input field regexes} -o {key-field name,value-field name} + These pivot/reshape the input data such that the input fields are removed + and separate records are emitted for each key/value pair. + Note: this works with tail -f and produces output records for each input + record seen. +Long-to-wide options: + -s {key-field name,value-field name} + These pivot/reshape the input data to undo the wide-to-long operation. + Note: this does not work with tail -f; it produces output records only after + all input records have been read. + +Examples: + + Input file "wide.txt": + time X Y + 2009-01-01 0.65473572 2.4520609 + 2009-01-02 -0.89248112 0.2154713 + 2009-01-03 0.98012375 1.3179287 + + mlr --pprint reshape -i X,Y -o item,value wide.txt + time item value + 2009-01-01 X 0.65473572 + 2009-01-01 Y 2.4520609 + 2009-01-02 X -0.89248112 + 2009-01-02 Y 0.2154713 + 2009-01-03 X 0.98012375 + 2009-01-03 Y 1.3179287 + + mlr --pprint reshape -r '[A-Z]' -o item,value wide.txt + time item value + 2009-01-01 X 0.65473572 + 2009-01-01 Y 2.4520609 + 2009-01-02 X -0.89248112 + 2009-01-02 Y 0.2154713 + 2009-01-03 X 0.98012375 + 2009-01-03 Y 1.3179287 + + Input file "long.txt": + time item value + 2009-01-01 X 0.65473572 + 2009-01-01 Y 2.4520609 + 2009-01-02 X -0.89248112 + 2009-01-02 Y 0.2154713 + 2009-01-03 X 0.98012375 + 2009-01-03 Y 1.3179287 + + mlr --pprint reshape -s item,value long.txt + time X Y + 2009-01-01 0.65473572 2.4520609 + 2009-01-02 -0.89248112 0.2154713 + 2009-01-03 0.98012375 1.3179287 +See also mlr nest. +.fi +.if n \{\ +.RE +.SS "sample" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr sample [options] +Reservoir sampling (subsampling without replacement), optionally by category. +See also mlr bootstrap and mlr shuffle. +Options: +-g {a,b,c} Optional: group-by-field names for samples, e.g. a,b,c. +-k {k} Required: number of records to output in total, or by group if using -g. +-h|--help Show this message. +.fi +.if n \{\ +.RE +.SS "sec2gmtdate" +.if n \{\ +.RS 0 +.\} +.nf +Usage: ../c/mlr sec2gmtdate {comma-separated list of field names} +Replaces a numeric field representing seconds since the epoch with the +corresponding GMT year-month-day timestamp; leaves non-numbers as-is. +This is nothing more than a keystroke-saver for the sec2gmtdate function: + ../c/mlr sec2gmtdate time1,time2 +is the same as + ../c/mlr put '$time1=sec2gmtdate($time1);$time2=sec2gmtdate($time2)' +.fi +.if n \{\ +.RE +.SS "sec2gmt" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr sec2gmt [options] {comma-separated list of field names} +Replaces a numeric field representing seconds since the epoch with the +corresponding GMT timestamp; leaves non-numbers as-is. This is nothing +more than a keystroke-saver for the sec2gmt function: + mlr sec2gmt time1,time2 +is the same as + mlr put '$time1 = sec2gmt($time1); $time2 = sec2gmt($time2)' +Options: +-1 through -9: format the seconds using 1..9 decimal places, respectively. +--millis Input numbers are treated as milliseconds since the epoch. +--micros Input numbers are treated as microseconds since the epoch. +--nanos Input numbers are treated as nanoseconds since the epoch. +-h|--help Show this message. +.fi +.if n \{\ +.RE +.SS "seqgen" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr seqgen [options] +Passes input records directly to output. Most useful for format conversion. +Produces a sequence of counters. Discards the input record stream. Produces +output as specified by the options + +Options: +-f {name} (default "i") Field name for counters. +--start {value} (default 1) Inclusive start value. +--step {value} (default 1) Step value. +--stop {value} (default 100) Inclusive stop value. +-h|--help Show this message. +Start, stop, and/or step may be floating-point. Output is integer if start, +stop, and step are all integers. Step may be negative. It may not be zero +unless start == stop. +.fi +.if n \{\ +.RE +.SS "shuffle" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr shuffle [options] +Outputs records randomly permuted. No output records are produced until +all input records are read. See also mlr bootstrap and mlr sample. +Options: +-h|--help Show this message. +.fi +.if n \{\ +.RE +.SS "skip-trivial-records" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr skip-trivial-records [options] +Passes through all records except those with zero fields, +or those for which all fields have empty value. +Options: +-h|--help Show this message. +.fi +.if n \{\ +.RE +.SS "sort" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr sort {flags} +Sorts records primarily by the first specified field, secondarily by the second +field, and so on. (Any records not having all specified sort keys will appear +at the end of the output, in the order they were encountered, regardless of the +specified sort order.) The sort is stable: records that compare equal will sort +in the order they were encountered in the input record stream. + +Options: +-f {comma-separated field names} Lexical ascending +-n {comma-separated field names} Numerical ascending; nulls sort last +-nf {comma-separated field names} Same as -n +-r {comma-separated field names} Lexical descending +-nr {comma-separated field names} Numerical descending; nulls sort first +-h|--help Show this message. + +Example: + mlr sort -f a,b -nr x,y,z +which is the same as: + mlr sort -f a -f b -nr x -nr y -nr z +.fi +.if n \{\ +.RE +.SS "sort-within-records" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr sort-within-records [options] +Outputs records sorted lexically ascending by keys. +Options: +-r Recursively sort subobjects/submaps, e.g. for JSON input. +-h|--help Show this message. +.fi +.if n \{\ +.RE +.SS "stats1" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr stats1 [options] +Computes univariate statistics for one or more given fields, accumulated across +the input record stream. +Options: +-a {sum,count,...} Names of accumulators: one or more of: + median This is the same as p50 + p10 p25.2 p50 p98 p100 etc. + TODO: flags for interpolated percentiles + count Count instances of fields + mode Find most-frequently-occurring values for fields; first-found wins tie + antimode Find least-frequently-occurring values for fields; first-found wins tie + sum Compute sums of specified fields + mean Compute averages (sample means) of specified fields + var Compute sample variance of specified fields + stddev Compute sample standard deviation of specified fields + meaneb Estimate error bars for averages (assuming no sample autocorrelation) + skewness Compute sample skewness of specified fields + kurtosis Compute sample kurtosis of specified fields + min Compute minimum values of specified fields + max Compute maximum values of specified fields + +-f {a,b,c} Value-field names on which to compute statistics +-g {d,e,f} Optional group-by-field names + +-i Use interpolated percentiles, like R's type=7; default like type=1.\en"); + Not sensical for string-valued fields.\en"); +-s Print iterative stats. Useful in tail -f contexts (in which + case please avoid pprint-format output since end of input + stream will never be seen). +-h|--help Show this message. +[TODO: more] +Example: mlr stats1 -a min,p10,p50,p90,max -f value -g size,shape + mlr stats1 +Example: mlr stats1 -a count,mode -f size + mlr stats1 +Example: mlr stats1 -a count,mode -f size -g shape + mlr stats1 +Example: mlr stats1 -a count,mode --fr '^[a-h].*$' -gr '^k.*$' + mlr stats1 + This computes count and mode statistics on all field names beginning + with a through h, grouped by all field names starting with k. + +Notes: +* p50 and median are synonymous. +* min and max output the same results as p0 and p100, respectively, but use + less memory. +* String-valued data make sense unless arithmetic on them is required, + e.g. for sum, mean, interpolated percentiles, etc. In case of mixed data, + numbers are less than strings. +* count and mode allow text input; the rest require numeric input. + In particular, 1 and 1.0 are distinct text for count and mode. +* When there are mode ties, the first-encountered datum wins. +.fi +.if n \{\ +.RE +.SS "stats2" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr stats2 [options] +Computes bivariate statistics for one or more given field-name pairs, +accumulated across the input record stream. +-a {linreg-ols,corr,...} Names of accumulators: one or more of: + linreg-ols Linear regression using ordinary least squares + linreg-pca Linear regression using principal component analysis + r2 Quality metric for linreg-ols (linreg-pca emits its own) + logireg Logistic regression + corr Sample correlation + cov Sample covariance + covx Sample-covariance matrix +-f {a,b,c,d} Value-field name-pairs on which to compute statistics. + There must be an even number of names. +-g {e,f,g} Optional group-by-field names. +-v Print additional output for linreg-pca. +-s Print iterative stats. Useful in tail -f contexts (in which + case please avoid pprint-format output since end of input + stream will never be seen). +--fit Rather than printing regression parameters, applies them to + the input data to compute new fit fields. All input records are + held in memory until end of input stream. Has effect only for + linreg-ols, linreg-pca, and logireg. +Only one of -s or --fit may be used. +Example: mlr stats2 -a linreg-pca -f x,y +Example: mlr stats2 -a linreg-ols,r2 -f x,y -g size,shape +Example: mlr stats2 -a corr -f x,y +.fi +.if n \{\ +.RE +.SS "step" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr step [options] +Computes values dependent on the previous record, optionally grouped by category. +Options: +-a {delta,rsum,...} Names of steppers: comma-separated, one or more of: + delta Compute differences in field(s) between successive records + shift Include value(s) in field(s) from previous record, if any + from-first Compute differences in field(s) from first record + ratio Compute ratios in field(s) between successive records + rsum Compute running sums of field(s) between successive records + counter Count instances of field(s) between successive records + ewma Exponentially weighted moving average over successive records + +-f {a,b,c} Value-field names on which to compute statistics +-g {d,e,f} Optional group-by-field names +-F Computes integerable things (e.g. counter) in floating point. + As of Miller 6 this happens automatically, but the flag is accepted + as a no-op for backward compatibility with Miller 5 and below. +-d {x,y,z} Weights for ewma. 1 means current sample gets all weight (no + smoothing), near under under 1 is light smoothing, near over 0 is + heavy smoothing. Multiple weights may be specified, e.g. + "mlr step -a ewma -f sys_load -d 0.01,0.1,0.9". Default if omitted + is "-d 0.5". +-o {a,b,c} Custom suffixes for EWMA output fields. If omitted, these default to + the -d values. If supplied, the number of -o values must be the same + as the number of -d values. +-h|--help Show this message. + +Examples: + mlr step -a rsum -f request_size + mlr step -a delta -f request_size -g hostname + mlr step -a ewma -d 0.1,0.9 -f x,y + mlr step -a ewma -d 0.1,0.9 -o smooth,rough -f x,y + mlr step -a ewma -d 0.1,0.9 -o smooth,rough -f x,y -g group_name + +Please see https://miller.readthedocs.io/en/latest/reference-verbs.html#filter or +https://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average +for more information on EWMA. +.fi +.if n \{\ +.RE +.SS "tac" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr tac [options] +Prints records in reverse order from the order in which they were encountered. +Options: +-h|--help Show this message. +.fi +.if n \{\ +.RE +.SS "tail" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr tail [options] +Passes through the last n records, optionally by category. +Options: +-g {a,b,c} Optional group-by-field names for head counts, e.g. a,b,c. +-n {n} Head-count to print. Default 10. +-h|--help Show this message. +.fi +.if n \{\ +.RE +.SS "tee" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr tee [options] {filename} +Options: +-a Append to existing file, if any, rather than overwriting. +-p Treat filename as a pipe-to command. +Any of the output-format command-line flags (see mlr -h). Example: using + mlr --icsv --opprint put '...' then tee --ojson ./mytap.dat then stats1 ... +the input is CSV, the output is pretty-print tabular, but the tee-file output +is written in JSON format. + +-h|--help Show this message. +.fi +.if n \{\ +.RE +.SS "top" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr top [options] +-f {a,b,c} Value-field names for top counts. +-g {d,e,f} Optional group-by-field names for top counts. +-n {count} How many records to print per category; default 1. +-a Print all fields for top-value records; default is + to print only value and group-by fields. Requires a single + value-field name only. +--min Print top smallest values; default is top largest values. +-F Keep top values as floats even if they look like integers. +-o {name} Field name for output indices. Default "top_idx". +Prints the n records with smallest/largest values at specified fields, +optionally by category. +.fi +.if n \{\ +.RE +.SS "unflatten" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr unflatten [options] +Reverses flatten. Example: field with name 'a.b.c' and value 4 +becomes name 'a' and value '{"b": { "c": 4 }}'. +Options: +-f {a,b,c} Comma-separated list of field names to unflatten (default all). +-s {string} Separator, defaulting to mlr --oflatsep value. +-h|--help Show this message. +.fi +.if n \{\ +.RE +.SS "uniq" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr uniq [options] +Prints distinct values for specified field names. With -c, same as +count-distinct. For uniq, -f is a synonym for -g. + +Options: +-g {d,e,f} Group-by-field names for uniq counts. +-c Show repeat counts in addition to unique values. +-n Show only the number of distinct values. +-o {name} Field name for output count. Default "count". +-a Output each unique record only once. Incompatible with -g. + With -c, produces unique records, with repeat counts for each. + With -n, produces only one record which is the unique-record count. + With neither -c nor -n, produces unique records. +.fi +.if n \{\ +.RE +.SS "unsparsify" +.if n \{\ +.RS 0 +.\} +.nf +Usage: mlr unsparsify [options] +Prints records with the union of field names over all input records. +For field names absent in a given record but present in others, fills in +a value. This verb retains all input before producing any output. +Options: +--fill-with {filler string} What to fill absent fields with. Defaults to + the empty string. +-f {a,b,c} Specify field names to be operated on. Any other fields won't be + modified, and operation will be streaming. +-h|--help Show this message. +Example: if the input is two records, one being 'a=1,b=2' and the other +being 'b=3,c=4', then the output is the two records 'a=1,b=2,c=' and +\(cqa=,b=3,c=4'. +.fi +.if n \{\ +.RE +.SH "FUNCTIONS FOR FILTER/PUT" +.sp + +.SS "+" +.if n \{\ +.RS 0 +.\} +.nf + (class=arithmetic #args=1,2) Addition as binary operator; unary plus operator. +.fi +.if n \{\ +.RE +.SS "-" +.if n \{\ +.RS 0 +.\} +.nf + (class=arithmetic #args=1,2) Subtraction as binary operator; unary negation operator. +.fi +.if n \{\ +.RE +.SS "*" +.if n \{\ +.RS 0 +.\} +.nf + (class=arithmetic #args=2) Multiplication, with integer*integer overflow to float. +.fi +.if n \{\ +.RE +.SS "/" +.if n \{\ +.RS 0 +.\} +.nf + (class=arithmetic #args=2) Division. Integer / integer is floating-point. +.fi +.if n \{\ +.RE +.SS "//" +.if n \{\ +.RS 0 +.\} +.nf + (class=arithmetic #args=2) Pythonic integer division, rounding toward negative. +.fi +.if n \{\ +.RE +.SS "**" +.if n \{\ +.RS 0 +.\} +.nf + (class=arithmetic #args=2) Exponentiation. Same as pow, but as an infix operator. +.fi +.if n \{\ +.RE +.SS "pow" +.if n \{\ +.RS 0 +.\} +.nf + (class=arithmetic #args=2) Exponentiation. Same as **, but as a function. +.fi +.if n \{\ +.RE +.SS ".+" +.if n \{\ +.RS 0 +.\} +.nf + (class=arithmetic #args=2) Addition, with integer-to-integer overflow. +.fi +.if n \{\ +.RE +.SS ".-" +.if n \{\ +.RS 0 +.\} +.nf + (class=arithmetic #args=2) Subtraction, with integer-to-integer overflow. +.fi +.if n \{\ +.RE +.SS ".*" +.if n \{\ +.RS 0 +.\} +.nf + (class=arithmetic #args=2) Multiplication, with integer-to-integer overflow. +.fi +.if n \{\ +.RE +.SS "./" +.if n \{\ +.RS 0 +.\} +.nf + (class=arithmetic #args=2) Integer division; not pythonic. +.fi +.if n \{\ +.RE +.SS "%" +.if n \{\ +.RS 0 +.\} +.nf + (class=arithmetic #args=2) Remainder; never negative-valued (pythonic). +.fi +.if n \{\ +.RE +.SS "~" +.if n \{\ +.RS 0 +.\} +.nf + (class=arithmetic #args=1) Bitwise NOT. Beware '$y=~$x' since =~ is the +regex-match operator: try '$y = ~$x'. +.fi +.if n \{\ +.RE +.SS "&" +.if n \{\ +.RS 0 +.\} +.nf + (class=arithmetic #args=2) Bitwise AND. +.fi +.if n \{\ +.RE +.SS "|" +.if n \{\ +.RS 0 +.\} +.nf + (class=arithmetic #args=2) Bitwise OR. +.fi +.if n \{\ +.RE +.SS "^" +.if n \{\ +.RS 0 +.\} +.nf + (class=arithmetic #args=2) Bitwise XOR. +.fi +.if n \{\ +.RE +.SS "<<" +.if n \{\ +.RS 0 +.\} +.nf + (class=arithmetic #args=2) Bitwise left-shift. +.fi +.if n \{\ +.RE +.SS ">>" +.if n \{\ +.RS 0 +.\} +.nf + (class=arithmetic #args=2) Bitwise signed right-shift. +.fi +.if n \{\ +.RE +.SS ">>>" +.if n \{\ +.RS 0 +.\} +.nf + (class=arithmetic #args=2) Bitwise unsigned right-shift. +.fi +.if n \{\ +.RE +.SS "bitcount" +.if n \{\ +.RS 0 +.\} +.nf + (class=arithmetic #args=1) Count of 1-bits. +.fi +.if n \{\ +.RE +.SS "madd" +.if n \{\ +.RS 0 +.\} +.nf + (class=arithmetic #args=3) a + b mod m (integers) +.fi +.if n \{\ +.RE +.SS "msub" +.if n \{\ +.RS 0 +.\} +.nf + (class=arithmetic #args=3) a - b mod m (integers) +.fi +.if n \{\ +.RE +.SS "mmul" +.if n \{\ +.RS 0 +.\} +.nf + (class=arithmetic #args=3) a * b mod m (integers) +.fi +.if n \{\ +.RE +.SS "mexp" +.if n \{\ +.RS 0 +.\} +.nf + (class=arithmetic #args=3) a ** b mod m (integers) +.fi +.if n \{\ +.RE +.SS "!" +.if n \{\ +.RS 0 +.\} +.nf + (class=boolean #args=1) Logical negation. +.fi +.if n \{\ +.RE +.SS "==" +.if n \{\ +.RS 0 +.\} +.nf + (class=boolean #args=2) String/numeric equality. Mixing number and string results in string compare. +.fi +.if n \{\ +.RE +.SS "!=" +.if n \{\ +.RS 0 +.\} +.nf + (class=boolean #args=2) String/numeric inequality. Mixing number and string results in string compare. +.fi +.if n \{\ +.RE +.SS ">" +.if n \{\ +.RS 0 +.\} +.nf + (class=boolean #args=2) String/numeric greater-than. Mixing number and string results in string compare. +.fi +.if n \{\ +.RE +.SS ">=" +.if n \{\ +.RS 0 +.\} +.nf + (class=boolean #args=2) String/numeric greater-than-or-equals. Mixing number and string results in string compare. +.fi +.if n \{\ +.RE +.SS "<" +.if n \{\ +.RS 0 +.\} +.nf + (class=boolean #args=2) String/numeric less-than. Mixing number and string results in string compare. +.fi +.if n \{\ +.RE +.SS "<=" +.if n \{\ +.RS 0 +.\} +.nf + (class=boolean #args=2) String/numeric less-than-or-equals. Mixing number and string results in string compare. +.fi +.if n \{\ +.RE +.SS "=~" +.if n \{\ +.RS 0 +.\} +.nf + (class=boolean #args=2) String (left-hand side) matches regex (right-hand side), e.g. '$name =~ "^a.*b$"'. +.fi +.if n \{\ +.RE +.SS "!=~" +.if n \{\ +.RS 0 +.\} +.nf + (class=boolean #args=2) String (left-hand side) does not match regex (right-hand side), e.g. '$name !=~ "^a.*b$"'. +.fi +.if n \{\ +.RE +.SS "&&" +.if n \{\ +.RS 0 +.\} +.nf + (class=boolean #args=2) Logical AND. +.fi +.if n \{\ +.RE +.SS "||" +.if n \{\ +.RS 0 +.\} +.nf + (class=boolean #args=2) Logical OR. +.fi +.if n \{\ +.RE +.SS "^^" +.if n \{\ +.RS 0 +.\} +.nf + (class=boolean #args=2) Logical XOR. +.fi +.if n \{\ +.RE +.SS "??" +.if n \{\ +.RS 0 +.\} +.nf + (class=boolean #args=2) Absent-coalesce operator. $a ?? 1 evaluates to 1 if $a isn't defined in the current record. +.fi +.if n \{\ +.RE +.SS "???" +.if n \{\ +.RS 0 +.\} +.nf + (class=boolean #args=2) Absent-coalesce operator. $a ?? 1 evaluates to 1 if $a isn't defined in the current record, or has empty value. +.fi +.if n \{\ +.RE +.SS "?:" +.if n \{\ +.RS 0 +.\} +.nf + (class=boolean #args=3) Standard ternary operator. +.fi +.if n \{\ +.RE +.SS "." +.if n \{\ +.RS 0 +.\} +.nf + (class=string #args=2) String concatenation. +.fi +.if n \{\ +.RE +.SS "capitalize" +.if n \{\ +.RS 0 +.\} +.nf + (class=string #args=1) Convert string's first character to uppercase. +.fi +.if n \{\ +.RE +.SS "clean_whitespace" +.if n \{\ +.RS 0 +.\} +.nf + (class=string #args=1) Same as collapse_whitespace and strip. +.fi +.if n \{\ +.RE +.SS "collapse_whitespace" +.if n \{\ +.RS 0 +.\} +.nf + (class=string #args=1) Strip repeated whitespace from string. +.fi +.if n \{\ +.RE +.SS "gsub" +.if n \{\ +.RS 0 +.\} +.nf + (class=string #args=3) Example: '$name=gsub($name, "old", "new")' (replace all). +.fi +.if n \{\ +.RE +.SS "lstrip" +.if n \{\ +.RS 0 +.\} +.nf + (class=string #args=1) Strip leading whitespace from string. +.fi +.if n \{\ +.RE +.SS "regextract" +.if n \{\ +.RS 0 +.\} +.nf + (class=string #args=2) Example: '$name=regextract($name, "[A-Z]{3}[0-9]{2}")' +.fi +.if n \{\ +.RE +.SS "regextract_or_else" +.if n \{\ +.RS 0 +.\} +.nf + (class=string #args=3) Example: '$name=regextract_or_else($name, "[A-Z]{3}[0-9]{2}", "default")' +.fi +.if n \{\ +.RE +.SS "rstrip" +.if n \{\ +.RS 0 +.\} +.nf + (class=string #args=1) Strip trailing whitespace from string. +.fi +.if n \{\ +.RE +.SS "strip" +.if n \{\ +.RS 0 +.\} +.nf + (class=string #args=1) Strip leading and trailing whitespace from string. +.fi +.if n \{\ +.RE +.SS "strlen" +.if n \{\ +.RS 0 +.\} +.nf + (class=string #args=1) String length. +.fi +.if n \{\ +.RE +.SS "ssub" +.if n \{\ +.RS 0 +.\} +.nf + (class=string #args=3) Like sub but does no regexing. No characters are special. +.fi +.if n \{\ +.RE +.SS "sub" +.if n \{\ +.RS 0 +.\} +.nf + (class=string #args=3) Example: '$name=sub($name, "old", "new")' (replace once). +.fi +.if n \{\ +.RE +.SS "substr0" +.if n \{\ +.RS 0 +.\} +.nf + (class=string #args=3) substr0(s,m,n) gives substring of s from 0-up position m to n +inclusive. Negative indices -len .. -1 alias to 0 .. len-1. +.fi +.if n \{\ +.RE +.SS "substr1" +.if n \{\ +.RS 0 +.\} +.nf + (class=string #args=3) substr1(s,m,n) gives substring of s from 1-up position m to n +inclusive. Negative indices -len .. -1 alias to 1 .. len. +.fi +.if n \{\ +.RE +.SS "substr" +.if n \{\ +.RS 0 +.\} +.nf + (class=string #args=3) substr is an alias for substr0. See also substr1. Miller is generally 1-up +with all array indices, but, this is a backward-compatibility issue with Miller 5 and below. +Arrays are new in Miller 6; the substr function is older. +.fi +.if n \{\ +.RE +.SS "tolower" +.if n \{\ +.RS 0 +.\} +.nf + (class=string #args=1) Convert string to lowercase. +.fi +.if n \{\ +.RE +.SS "toupper" +.if n \{\ +.RS 0 +.\} +.nf + (class=string #args=1) Convert string to uppercase. +.fi +.if n \{\ +.RE +.SS "truncate" +.if n \{\ +.RS 0 +.\} +.nf + (class=string #args=2) Truncates string first argument to max length of int second argument. +.fi +.if n \{\ +.RE +.SS "md5" +.if n \{\ +.RS 0 +.\} +.nf + (class=hashing #args=1) MD5 hash. +.fi +.if n \{\ +.RE +.SS "sha1" +.if n \{\ +.RS 0 +.\} +.nf + (class=hashing #args=1) SHA1 hash. +.fi +.if n \{\ +.RE +.SS "sha256" +.if n \{\ +.RS 0 +.\} +.nf + (class=hashing #args=1) SHA256 hash. +.fi +.if n \{\ +.RE +.SS "sha512" +.if n \{\ +.RS 0 +.\} +.nf + (class=hashing #args=1) SHA512 hash. +.fi +.if n \{\ +.RE +.SS "abs" +.if n \{\ +.RS 0 +.\} +.nf + (class=math #args=1) Absolute value. +.fi +.if n \{\ +.RE +.SS "acos" +.if n \{\ +.RS 0 +.\} +.nf + (class=math #args=1) Inverse trigonometric cosine. +.fi +.if n \{\ +.RE +.SS "acosh" +.if n \{\ +.RS 0 +.\} +.nf + (class=math #args=1) Inverse hyperbolic cosine. +.fi +.if n \{\ +.RE +.SS "asin" +.if n \{\ +.RS 0 +.\} +.nf + (class=math #args=1) Inverse trigonometric sine. +.fi +.if n \{\ +.RE +.SS "asinh" +.if n \{\ +.RS 0 +.\} +.nf + (class=math #args=1) Inverse hyperbolic sine. +.fi +.if n \{\ +.RE +.SS "atan" +.if n \{\ +.RS 0 +.\} +.nf + (class=math #args=1) One-argument arctangent. +.fi +.if n \{\ +.RE +.SS "atan2" +.if n \{\ +.RS 0 +.\} +.nf + (class=math #args=2) Two-argument arctangent. +.fi +.if n \{\ +.RE +.SS "atanh" +.if n \{\ +.RS 0 +.\} +.nf + (class=math #args=1) Inverse hyperbolic tangent. +.fi +.if n \{\ +.RE +.SS "cbrt" +.if n \{\ +.RS 0 +.\} +.nf + (class=math #args=1) Cube root. +.fi +.if n \{\ +.RE +.SS "ceil" +.if n \{\ +.RS 0 +.\} +.nf + (class=math #args=1) Ceiling: nearest integer at or above. +.fi +.if n \{\ +.RE +.SS "cos" +.if n \{\ +.RS 0 +.\} +.nf + (class=math #args=1) Trigonometric cosine. +.fi +.if n \{\ +.RE +.SS "cosh" +.if n \{\ +.RS 0 +.\} +.nf + (class=math #args=1) Hyperbolic cosine. +.fi +.if n \{\ +.RE +.SS "erf" +.if n \{\ +.RS 0 +.\} +.nf + (class=math #args=1) Error function. +.fi +.if n \{\ +.RE +.SS "erfc" +.if n \{\ +.RS 0 +.\} +.nf + (class=math #args=1) Complementary error function. +.fi +.if n \{\ +.RE +.SS "exp" +.if n \{\ +.RS 0 +.\} +.nf + (class=math #args=1) Exponential function e**x. +.fi +.if n \{\ +.RE +.SS "expm1" +.if n \{\ +.RS 0 +.\} +.nf + (class=math #args=1) e**x - 1. +.fi +.if n \{\ +.RE +.SS "floor" +.if n \{\ +.RS 0 +.\} +.nf + (class=math #args=1) Floor: nearest integer at or below. +.fi +.if n \{\ +.RE +.SS "invqnorm" +.if n \{\ +.RS 0 +.\} +.nf + (class=math #args=1) Inverse of normal cumulative distribution function. +Note that invqorm(urand()) is normally distributed. +.fi +.if n \{\ +.RE +.SS "log" +.if n \{\ +.RS 0 +.\} +.nf + (class=math #args=1) Natural (base-e) logarithm. +.fi +.if n \{\ +.RE +.SS "log10" +.if n \{\ +.RS 0 +.\} +.nf + (class=math #args=1) Base-10 logarithm. +.fi +.if n \{\ +.RE +.SS "log1p" +.if n \{\ +.RS 0 +.\} +.nf + (class=math #args=1) log(1-x). +.fi +.if n \{\ +.RE +.SS "logifit" +.if n \{\ +.RS 0 +.\} +.nf + (class=math #args=3) Given m and b from logistic regression, compute fit: +$yhat=logifit($x,$m,$b). +.fi +.if n \{\ +.RE +.SS "max" +.if n \{\ +.RS 0 +.\} +.nf + (class=math #args=variadic) Max of n numbers; null loses. +.fi +.if n \{\ +.RE +.SS "min" +.if n \{\ +.RS 0 +.\} +.nf + (class=math #args=variadic) Min of n numbers; null loses. +.fi +.if n \{\ +.RE +.SS "qnorm" +.if n \{\ +.RS 0 +.\} +.nf + (class=math #args=1) Normal cumulative distribution function. +.fi +.if n \{\ +.RE +.SS "round" +.if n \{\ +.RS 0 +.\} +.nf + (class=math #args=1) Round to nearest integer. +.fi +.if n \{\ +.RE +.SS "sgn" +.if n \{\ +.RS 0 +.\} +.nf + (class=math #args=1) +1, 0, -1 for positive, zero, negative input respectively. +.fi +.if n \{\ +.RE +.SS "sin" +.if n \{\ +.RS 0 +.\} +.nf + (class=math #args=1) Trigonometric sine. +.fi +.if n \{\ +.RE +.SS "sinh" +.if n \{\ +.RS 0 +.\} +.nf + (class=math #args=1) Hyperbolic sine. +.fi +.if n \{\ +.RE +.SS "sqrt" +.if n \{\ +.RS 0 +.\} +.nf + (class=math #args=1) Square root. +.fi +.if n \{\ +.RE +.SS "tan" +.if n \{\ +.RS 0 +.\} +.nf + (class=math #args=1) Trigonometric tangent. +.fi +.if n \{\ +.RE +.SS "tanh" +.if n \{\ +.RS 0 +.\} +.nf + (class=math #args=1) Hyperbolic tangent. +.fi +.if n \{\ +.RE +.SS "roundm" +.if n \{\ +.RS 0 +.\} +.nf + (class=math #args=2) Round to nearest multiple of m: roundm($x,$m) is +the same as round($x/$m)*$m. +.fi +.if n \{\ +.RE +.SS "urand" +.if n \{\ +.RS 0 +.\} +.nf + (class=math #args=0) Floating-point numbers uniformly distributed on the unit interval. +Int-valued example: '$n=floor(20+urand()*11)'. +.fi +.if n \{\ +.RE +.SS "urandint" +.if n \{\ +.RS 0 +.\} +.nf + (class=math #args=2) Integer uniformly distributed between inclusive integer endpoints. +.fi +.if n \{\ +.RE +.SS "urandrange" +.if n \{\ +.RS 0 +.\} +.nf + (class=math #args=2) Floating-point numbers uniformly distributed on the interval [a, b). +.fi +.if n \{\ +.RE +.SS "urand32" +.if n \{\ +.RS 0 +.\} +.nf + (class=math #args=0) Integer uniformly distributed 0 and 2**32-1 inclusive. +.fi +.if n \{\ +.RE +.SS "gmt2sec" +.if n \{\ +.RS 0 +.\} +.nf + (class=time #args=1) Parses GMT timestamp as integer seconds since the epoch. +.fi +.if n \{\ +.RE +.SS "sec2gmt" +.if n \{\ +.RS 0 +.\} +.nf + (class=time #args=1,2) Formats seconds since epoch (integer part) +as GMT timestamp, e.g. sec2gmt(1440768801.7) = "2015-08-28T13:33:21Z". +Leaves non-numbers as-is. With second integer argument n, includes n decimal places +for the seconds part +.fi +.if n \{\ +.RE +.SS "sec2gmtdate" +.if n \{\ +.RS 0 +.\} +.nf + (class=time #args=1) Formats seconds since epoch (integer part) +as GMT timestamp with year-month-date, e.g. sec2gmtdate(1440768801.7) = "2015-08-28". +Leaves non-numbers as-is. + +.fi +.if n \{\ +.RE +.SS "systime" +.if n \{\ +.RS 0 +.\} +.nf + (class=time #args=0) help string will go here +.fi +.if n \{\ +.RE +.SS "systimeint" +.if n \{\ +.RS 0 +.\} +.nf + (class=time #args=0) help string will go here +.fi +.if n \{\ +.RE +.SS "uptime" +.if n \{\ +.RS 0 +.\} +.nf + (class=time #args=0) help string will go here +.fi +.if n \{\ +.RE +.SS "strftime" +.if n \{\ +.RS 0 +.\} +.nf + (class=time #args=2) Formats seconds since the epoch as timestamp, e.g. + strftime(1440768801.7,"%Y-%m-%dT%H:%M:%SZ") = "2015-08-28T13:33:21Z", and + strftime(1440768801.7,"%Y-%m-%dT%H:%M:%3SZ") = "2015-08-28T13:33:21.700Z". + Format strings are as in the C library (please see "man strftime" on your system), + with the Miller-specific addition of "%1S" through "%9S" which format the seconds + with 1 through 9 decimal places, respectively. ("%S" uses no decimal places.) + See also strftime_local. + +.fi +.if n \{\ +.RE +.SS "strptime" +.if n \{\ +.RS 0 +.\} +.nf + (class=time #args=2) strptime: Parses timestamp as floating-point seconds since the epoch, + e.g. strptime("2015-08-28T13:33:21Z","%Y-%m-%dT%H:%M:%SZ") = 1440768801.000000, + and strptime("2015-08-28T13:33:21.345Z","%Y-%m-%dT%H:%M:%SZ") = 1440768801.345000. + See also strptime_local. + +.fi +.if n \{\ +.RE +.SS "dhms2fsec" +.if n \{\ +.RS 0 +.\} +.nf + (class=time #args=1) Recovers floating-point seconds as in dhms2fsec("5d18h53m20.250000s") = 500000.250000 + +.fi +.if n \{\ +.RE +.SS "dhms2sec" +.if n \{\ +.RS 0 +.\} +.nf + (class=time #args=1) Recovers integer seconds as in dhms2sec("5d18h53m20s") = 500000 + +.fi +.if n \{\ +.RE +.SS "fsec2dhms" +.if n \{\ +.RS 0 +.\} +.nf + (class=time #args=1) Formats floating-point seconds as in fsec2dhms(500000.25) = "5d18h53m20.250000s" + +.fi +.if n \{\ +.RE +.SS "fsec2hms" +.if n \{\ +.RS 0 +.\} +.nf + (class=time #args=1) Formats floating-point seconds as in fsec2hms(5000.25) = "01:23:20.250000" + +.fi +.if n \{\ +.RE +.SS "hms2fsec" +.if n \{\ +.RS 0 +.\} +.nf + (class=time #args=1) Recovers floating-point seconds as in hms2fsec("01:23:20.250000") = 5000.250000 + +.fi +.if n \{\ +.RE +.SS "hms2sec" +.if n \{\ +.RS 0 +.\} +.nf + (class=time #args=1) Recovers integer seconds as in hms2sec("01:23:20") = 5000 + +.fi +.if n \{\ +.RE +.SS "sec2dhms" +.if n \{\ +.RS 0 +.\} +.nf + (class=time #args=1) Formats integer seconds as in sec2dhms(500000) = "5d18h53m20s" + +.fi +.if n \{\ +.RE +.SS "sec2hms" +.if n \{\ +.RS 0 +.\} +.nf + (class=time #args=1) Formats integer seconds as in sec2hms(5000) = "01:23:20" + +.fi +.if n \{\ +.RE +.SS "is_absent" +.if n \{\ +.RS 0 +.\} +.nf + (class=typing #args=1) False if field is present in input, true otherwise +.fi +.if n \{\ +.RE +.SS "is_array" +.if n \{\ +.RS 0 +.\} +.nf + (class=typing #args=1) True if argument is an array. +.fi +.if n \{\ +.RE +.SS "is_bool" +.if n \{\ +.RS 0 +.\} +.nf + (class=typing #args=1) True if field is present with boolean value. Synonymous with is_boolean. +.fi +.if n \{\ +.RE +.SS "is_boolean" +.if n \{\ +.RS 0 +.\} +.nf + (class=typing #args=1) True if field is present with boolean value. Synonymous with is_bool. +.fi +.if n \{\ +.RE +.SS "is_empty" +.if n \{\ +.RS 0 +.\} +.nf + (class=typing #args=1) True if field is present in input with empty string value, false otherwise. +.fi +.if n \{\ +.RE +.SS "is_empty_map" +.if n \{\ +.RS 0 +.\} +.nf + (class=typing #args=1) True if argument is a map which is empty. +.fi +.if n \{\ +.RE +.SS "is_error" +.if n \{\ +.RS 0 +.\} +.nf + (class=typing #args=1) True if if argument is an error, such as taking string length of an integer. +.fi +.if n \{\ +.RE +.SS "is_float" +.if n \{\ +.RS 0 +.\} +.nf + (class=typing #args=1) True if field is present with value inferred to be float +.fi +.if n \{\ +.RE +.SS "is_int" +.if n \{\ +.RS 0 +.\} +.nf + (class=typing #args=1) True if field is present with value inferred to be int +.fi +.if n \{\ +.RE +.SS "is_map" +.if n \{\ +.RS 0 +.\} +.nf + (class=typing #args=1) True if argument is a map. +.fi +.if n \{\ +.RE +.SS "is_nonempty_map" +.if n \{\ +.RS 0 +.\} +.nf + (class=typing #args=1) True if argument is a map which is non-empty. +.fi +.if n \{\ +.RE +.SS "is_not_empty" +.if n \{\ +.RS 0 +.\} +.nf + (class=typing #args=1) False if field is present in input with empty value, true otherwise +.fi +.if n \{\ +.RE +.SS "is_not_map" +.if n \{\ +.RS 0 +.\} +.nf + (class=typing #args=1) True if argument is not a map. +.fi +.if n \{\ +.RE +.SS "is_not_array" +.if n \{\ +.RS 0 +.\} +.nf + (class=typing #args=1) True if argument is not an array. +.fi +.if n \{\ +.RE +.SS "is_not_null" +.if n \{\ +.RS 0 +.\} +.nf + (class=typing #args=1) False if argument is null (empty or absent), true otherwise. +.fi +.if n \{\ +.RE +.SS "is_null" +.if n \{\ +.RS 0 +.\} +.nf + (class=typing #args=1) True if argument is null (empty or absent), false otherwise. +.fi +.if n \{\ +.RE +.SS "is_numeric" +.if n \{\ +.RS 0 +.\} +.nf + (class=typing #args=1) True if field is present with value inferred to be int or float +.fi +.if n \{\ +.RE +.SS "is_present" +.if n \{\ +.RS 0 +.\} +.nf + (class=typing #args=1) True if field is present in input, false otherwise. +.fi +.if n \{\ +.RE +.SS "is_string" +.if n \{\ +.RS 0 +.\} +.nf + (class=typing #args=1) True if field is present with string (including empty-string) value +.fi +.if n \{\ +.RE +.SS "asserting_absent" +.if n \{\ +.RS 0 +.\} +.nf + (class=typing #args=1) Aborts with an error if is_absent on the argument returns false, +else returns its argument. +.fi +.if n \{\ +.RE +.SS "asserting_array" +.if n \{\ +.RS 0 +.\} +.nf + (class=typing #args=1) Aborts with an error if is_array on the argument returns false, +else returns its argument. +.fi +.if n \{\ +.RE +.SS "asserting_bool" +.if n \{\ +.RS 0 +.\} +.nf + (class=typing #args=1) Aborts with an error if is_bool on the argument returns false, +else returns its argument. +.fi +.if n \{\ +.RE +.SS "asserting_boolean" +.if n \{\ +.RS 0 +.\} +.nf + (class=typing #args=1) Aborts with an error if is_boolean on the argument returns false, +else returns its argument. +.fi +.if n \{\ +.RE +.SS "asserting_error" +.if n \{\ +.RS 0 +.\} +.nf + (class=typing #args=1) Aborts with an error if is_error on the argument returns false, +else returns its argument. +.fi +.if n \{\ +.RE +.SS "asserting_empty" +.if n \{\ +.RS 0 +.\} +.nf + (class=typing #args=1) Aborts with an error if is_empty on the argument returns false, +else returns its argument. +.fi +.if n \{\ +.RE +.SS "asserting_empty_map" +.if n \{\ +.RS 0 +.\} +.nf + (class=typing #args=1) Aborts with an error if is_empty_map on the argument returns false, +else returns its argument. +.fi +.if n \{\ +.RE +.SS "asserting_float" +.if n \{\ +.RS 0 +.\} +.nf + (class=typing #args=1) Aborts with an error if is_float on the argument returns false, +else returns its argument. +.fi +.if n \{\ +.RE +.SS "asserting_int" +.if n \{\ +.RS 0 +.\} +.nf + (class=typing #args=1) Aborts with an error if is_int on the argument returns false, +else returns its argument. +.fi +.if n \{\ +.RE +.SS "asserting_map" +.if n \{\ +.RS 0 +.\} +.nf + (class=typing #args=1) Aborts with an error if is_map on the argument returns false, +else returns its argument. +.fi +.if n \{\ +.RE +.SS "asserting_nonempty_map" +.if n \{\ +.RS 0 +.\} +.nf + (class=typing #args=1) Aborts with an error if is_nonempty_map on the argument returns false, +else returns its argument. +.fi +.if n \{\ +.RE +.SS "asserting_not_empty" +.if n \{\ +.RS 0 +.\} +.nf + (class=typing #args=1) Aborts with an error if is_not_empty on the argument returns false, +else returns its argument. +.fi +.if n \{\ +.RE +.SS "asserting_not_map" +.if n \{\ +.RS 0 +.\} +.nf + (class=typing #args=1) Aborts with an error if is_not_map on the argument returns false, +else returns its argument. +.fi +.if n \{\ +.RE +.SS "asserting_not_array" +.if n \{\ +.RS 0 +.\} +.nf + (class=typing #args=1) Aborts with an error if is_not_array on the argument returns false, +else returns its argument. +.fi +.if n \{\ +.RE +.SS "asserting_not_null" +.if n \{\ +.RS 0 +.\} +.nf + (class=typing #args=1) Aborts with an error if is_not_null on the argument returns false, +else returns its argument. +.fi +.if n \{\ +.RE +.SS "asserting_null" +.if n \{\ +.RS 0 +.\} +.nf + (class=typing #args=1) Aborts with an error if is_null on the argument returns false, +else returns its argument. +.fi +.if n \{\ +.RE +.SS "asserting_numeric" +.if n \{\ +.RS 0 +.\} +.nf + (class=typing #args=1) Aborts with an error if is_numeric on the argument returns false, +else returns its argument. +.fi +.if n \{\ +.RE +.SS "asserting_present" +.if n \{\ +.RS 0 +.\} +.nf + (class=typing #args=1) Aborts with an error if is_present on the argument returns false, +else returns its argument. +.fi +.if n \{\ +.RE +.SS "asserting_string" +.if n \{\ +.RS 0 +.\} +.nf + (class=typing #args=1) Aborts with an error if is_string on the argument returns false, +else returns its argument. +.fi +.if n \{\ +.RE +.SS "typeof" +.if n \{\ +.RS 0 +.\} +.nf + (class=typing #args=1) Convert argument to type of argument (e.g. "str"). For debug. +.fi +.if n \{\ +.RE +.SS "boolean" +.if n \{\ +.RS 0 +.\} +.nf + (class=conversion #args=1) Convert int/float/bool/string to boolean. +.fi +.if n \{\ +.RE +.SS "float" +.if n \{\ +.RS 0 +.\} +.nf + (class=conversion #args=1) Convert int/float/bool/string to float. +.fi +.if n \{\ +.RE +.SS "fmtnum" +.if n \{\ +.RS 0 +.\} +.nf + (class=conversion #args=2) Convert int/float/bool to string using +printf-style format string, e.g. '$s = fmtnum($n, "%06lld")'. +.fi +.if n \{\ +.RE +.SS "hexfmt" +.if n \{\ +.RS 0 +.\} +.nf + (class=conversion #args=1) Convert int to hex string, e.g. 255 to "0xff". +.fi +.if n \{\ +.RE +.SS "int" +.if n \{\ +.RS 0 +.\} +.nf + (class=conversion #args=1) Convert int/float/bool/string to int. +.fi +.if n \{\ +.RE +.SS "joink" +.if n \{\ +.RS 0 +.\} +.nf + (class=conversion #args=2) Makes string from map/array keys. Examples: +joink({"a":3,"b":4,"c":5}, ",") = "a,b,c" +joink([1,2,3], ",") = "1,2,3". +.fi +.if n \{\ +.RE +.SS "joinv" +.if n \{\ +.RS 0 +.\} +.nf + (class=conversion #args=2) Makes string from map/array values. +joinv([3,4,5], ",") = "3,4,5" +joinv({"a":3,"b":4,"c":5}, ",") = "3,4,5" +.fi +.if n \{\ +.RE +.SS "joinkv" +.if n \{\ +.RS 0 +.\} +.nf + (class=conversion #args=3) Makes string from map/array key-value pairs. Examples: +joinkv([3,4,5], "=", ",") = "1=3,2=4,3=5" +joinkv({"a":3,"b":4,"c":5}, "=", ",") = "a=3,b=4,c=5" +.fi +.if n \{\ +.RE +.SS "splita" +.if n \{\ +.RS 0 +.\} +.nf + (class=conversion #args=2) Splits string into array with type inference. Example: +splita("3,4,5", ",") = [3,4,5] +.fi +.if n \{\ +.RE +.SS "splitax" +.if n \{\ +.RS 0 +.\} +.nf + (class=conversion #args=2) Splits string into array without type inference. Example: +splita("3,4,5", ",") = ["3","4","5"] +.fi +.if n \{\ +.RE +.SS "splitkv" +.if n \{\ +.RS 0 +.\} +.nf + (class=conversion #args=3) Splits string by separators into map with type inference. Example: +splitkv("a=3,b=4,c=5", "=", ",") = {"a":3,"b":4,"c":5} +.fi +.if n \{\ +.RE +.SS "splitkvx" +.if n \{\ +.RS 0 +.\} +.nf + (class=conversion #args=3) Splits string by separators into map without type inference (keys and +values are strings). Example: +splitkvx("a=3,b=4,c=5", "=", ",") = {"a":"3","b":"4","c":"5"} +.fi +.if n \{\ +.RE +.SS "splitnv" +.if n \{\ +.RS 0 +.\} +.nf + (class=conversion #args=2) Splits string by separator into integer-indexed map with type inference. Example: +splitnv("a,b,c", ",") = {"1":"a","2":"b","3":"c"} +.fi +.if n \{\ +.RE +.SS "splitnvx" +.if n \{\ +.RS 0 +.\} +.nf + (class=conversion #args=2) Splits string by separator into integer-indexed map without type +inference (values are strings). Example: +splitnvx("3,4,5", ",") = {"1":"3","2":"4","3":"5"} +.fi +.if n \{\ +.RE +.SS "string" +.if n \{\ +.RS 0 +.\} +.nf + (class=conversion #args=1) Convert int/float/bool/string/array/map to string. +.fi +.if n \{\ +.RE +.SS "append" +.if n \{\ +.RS 0 +.\} +.nf + (class=maps/arrays #args=2) Appends second argument to end of first argument, which must be an array. +.fi +.if n \{\ +.RE +.SS "arrayify" +.if n \{\ +.RS 0 +.\} +.nf + (class=maps/arrays #args=1) Walks through a nested map/array, converting any map with consecutive keys +"1", "2", ... into an array. Useful to wrap the output of unflatten. +.fi +.if n \{\ +.RE +.SS "depth" +.if n \{\ +.RS 0 +.\} +.nf + (class=maps/arrays #args=1) Prints maximum depth of map/array. Scalars have depth 0. +.fi +.if n \{\ +.RE +.SS "flatten" +.if n \{\ +.RS 0 +.\} +.nf + (class=maps/arrays #args=3) Flattens multi-level maps to single-level ones. Examples: +flatten("a", ".", {"b": { "c": 4 }}) is {"a.b.c" : 4}. +flatten("", ".", {"a": { "b": 3 }}) is {"a.b" : 3}. +Two-argument version: flatten($*, ".") is the same as flatten("", ".", $*). +Useful for nested JSON-like structures for non-JSON file formats like CSV. +.fi +.if n \{\ +.RE +.SS "get_keys" +.if n \{\ +.RS 0 +.\} +.nf + (class=maps/arrays #args=1) Returns array of keys of map or array +.fi +.if n \{\ +.RE +.SS "get_values" +.if n \{\ +.RS 0 +.\} +.nf + (class=maps/arrays #args=1) Returns array of keys of map or array -- in the latter case, returns a copy of the array +.fi +.if n \{\ +.RE +.SS "haskey" +.if n \{\ +.RS 0 +.\} +.nf + (class=maps/arrays #args=2) True/false if map has/hasn't key, e.g. 'haskey($*, "a")' or +\(cqhaskey(mymap, mykey)', or true/false if array index is in bounds / out of bounds. +Error if 1st argument is not a map or array. Note -n..-1 alias to 1..n in Miller arrays. +.fi +.if n \{\ +.RE +.SS "json_parse" +.if n \{\ +.RS 0 +.\} +.nf + (class=maps/arrays #args=1) Converts value from JSON-formatted string. +.fi +.if n \{\ +.RE +.SS "json_stringify" +.if n \{\ +.RS 0 +.\} +.nf + (class=maps/arrays #args=1,2) Converts value to JSON-formatted string. Default output is single-line. +With optional second boolean argument set to true, produces multiline output. +.fi +.if n \{\ +.RE +.SS "leafcount" +.if n \{\ +.RS 0 +.\} +.nf + (class=maps/arrays #args=1) Counts total number of terminal values in map/array. For single-level +map/array, same as length. +.fi +.if n \{\ +.RE +.SS "length" +.if n \{\ +.RS 0 +.\} +.nf + (class=maps/arrays #args=1) Counts number of top-level entries in array/map. Scalars have length 1. +.fi +.if n \{\ +.RE +.SS "mapdiff" +.if n \{\ +.RS 0 +.\} +.nf + (class=maps/arrays #args=variadic) With 0 args, returns empty map. With 1 arg, returns copy of arg. +With 2 or more, returns copy of arg 1 with all keys from any of remaining +argument maps removed. +.fi +.if n \{\ +.RE +.SS "mapexcept" +.if n \{\ +.RS 0 +.\} +.nf + (class=maps/arrays #args=variadic) Returns a map with keys from remaining arguments, if any, unset. +Remaining arguments can be strings or arrays of string. +E.g. 'mapexcept({1:2,3:4,5:6}, 1, 5, 7)' is '{3:4}' +and 'mapexcept({1:2,3:4,5:6}, [1, 5, 7])' is '{3:4}'. +.fi +.if n \{\ +.RE +.SS "mapselect" +.if n \{\ +.RS 0 +.\} +.nf + (class=maps/arrays #args=variadic) Returns a map with only keys from remaining arguments set. +Remaining arguments can be strings or arrays of string. +E.g. 'mapselect({1:2,3:4,5:6}, 1, 5, 7)' is '{1:2,5:6}' +and 'mapselect({1:2,3:4,5:6}, [1, 5, 7])' is '{1:2,5:6}'. +.fi +.if n \{\ +.RE +.SS "mapsum" +.if n \{\ +.RS 0 +.\} +.nf + (class=maps/arrays #args=variadic) With 0 args, returns empty map. With >= 1 arg, returns a map with +key-value pairs from all arguments. Rightmost collisions win, e.g. +\(cqmapsum({1:2,3:4},{1:5})' is '{1:5,3:4}'. +.fi +.if n \{\ +.RE +.SS "unflatten" +.if n \{\ +.RS 0 +.\} +.nf + (class=maps/arrays #args=2) Reverses flatten. Example: +unflatten({"a.b.c" : 4}, ".") is {"a": "b": { "c": 4 }}}. +Useful for nested JSON-like structures for non-JSON file formats like CSV. +See also arrayify. +.fi +.if n \{\ +.RE +.SS "hostname" +.if n \{\ +.RS 0 +.\} +.nf + (class=system #args=0) Returns the hostname as a string. +.fi +.if n \{\ +.RE +.SS "os" +.if n \{\ +.RS 0 +.\} +.nf + (class=system #args=0) Returns the operating-system name as a string. +.fi +.if n \{\ +.RE +.SS "system" +.if n \{\ +.RS 0 +.\} +.nf + (class=system #args=1) Run command string, yielding its stdout minus final carriage return. +.fi +.if n \{\ +.RE +.SS "version" +.if n \{\ +.RS 0 +.\} +.nf + (class=system #args=0) Returns the Miller version as a string. +.fi +.if n \{\ +.RE +.SH "KEYWORDS FOR PUT AND FILTER" +.sp + +.SS "TODO: port mlr_dsl_list_all_keywords_raw" +.if n \{\ +.RS 0 +.\} +.nf +TODO: port mlr_dsl_keyword_usage +.fi +.if n \{\ +.RE +.SH "AUTHOR" +.sp +Miller is written by John Kerl . +.sp +This manual page has been composed from Miller's help output by Eric MSP Veith . +.SH "SEE ALSO" +.sp +awk(1), sed(1), cut(1), join(1), sort(1), RFC 4180: Common Format and MIME Type for Comma-Separated Values (CSV) Files, the miller website http://johnkerl.org/miller/doc