From a535b9901dd8f79219ec4b9caabe09de7eef7019 Mon Sep 17 00:00:00 2001 From: John Kerl Date: Thu, 17 Jun 2021 00:03:58 -0400 Subject: [PATCH] Add REPL docs --- docs6/index.rst | 1 + docs6/repl.rst | 171 +++++++++++++++++++++++++++++++++++ docs6/repl.rst.in | 149 ++++++++++++++++++++++++++++++ go/src/auxents/repl/verbs.go | 2 +- 4 files changed, 322 insertions(+), 1 deletion(-) create mode 100644 docs6/repl.rst create mode 100644 docs6/repl.rst.in diff --git a/docs6/index.rst b/docs6/index.rst index 948303b93..5502ad55d 100644 --- a/docs6/index.rst +++ b/docs6/index.rst @@ -26,6 +26,7 @@ Miller in more detail internationalization output-colorization customization + repl miller-on-windows new-in-miller-6 contributing diff --git a/docs6/repl.rst b/docs6/repl.rst new file mode 100644 index 000000000..ea5d044e2 --- /dev/null +++ b/docs6/repl.rst @@ -0,0 +1,171 @@ +.. + PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE. + +The REPL +================================================================ + +The Miller REPL (read-evaluate-print loop) is an interactive counterpart to record-processing using the ``put``/``filter`` language. + +It isn't a source-level debugger which lets you execute one source-code *statement* at a time -- however, it does let you operate on one *record* at a time. Further, it lets you use "immediate expressions", namely, you can interact with the language without having to provide data from an input file. + +Using Miller without the REPL +---------------------------------------------------------------- + +Using ``put`` and ``filter``, you can do the following as we've seen above: + +* Specify input format (e.g. ``--icsv``), output format (e.g. ``--ojson``), etc. using command-line flags. +* Specify filenames on the command line. +* Define ``begin {...}`` blocks which are executed before the first record is read. +* Define ``end {...}`` blocks which are executed after the last record is read. +* Define user-defined functions/subroutines using ``func`` and ``subr``. +* Specify statements to be executed on each record -- which are anything outside of ``begin``/``end``/``func``/``subr``. +* Example: + +.. code-block:: none + :emphasize-lines: 1-2 + + $ mlr --icsv --ojson --from example.csv head -n 2 \ + then put 'begin {print "HELLO"} $z = $x + $y; end {print "GOODBYE"}' + HELLO + { + "color": "yellow", + "shape": "triangle", + "flag": true, + "index": 11, + "quantity": 43.6498, + "rate": 9.8870 + } + { + "color": "red", + "shape": "square", + "flag": true, + "index": 15, + "quantity": 79.2778, + "rate": 0.0130 + } + GOODBYE + +Using Miller with the REPL +---------------------------------------------------------------- + +Using the REPL, by contrast, you get interactive control over those same steps: + +* Specify input format (e.g. ``--icsv``), output format (e.g. ``--ojson``), etc. using command-line flags. +* REPL-only statements (non-DSL statements) start with ``:``, such as ``:help`` or ``:quit`` + or ``:open``. +* Specify filenames either on the command line or via ``:open`` at the Miller REPL. +* Read records one at a time using ``:read``. +* Write the current record (maybe after you've modified it with things like ``$z = $x + $y``) + using ``:write``. This goes to the terminal; you can use ``:> {filename}`` to make writes + go to a file, or ``:>> {filename}`` to append. +* You can type ``:reopen`` to go back to the start of the same file(s) you specified + with ``:open``. +* Skip ahead using statements ``:skip 10`` or ``:skip until NR == 100`` or + ``:skip until $status_code != 200``. +* Similarly, but processing records rather than skipping past them, using + ``:process`` rather than ``:skip``. Like ``:write``, these go to the screen; + use ``:> {filename}`` or ``:>> {filename}`` to log to a file instead. +* Define ``begin {...}`` blocks; invoke them at will using ``:begin``. +* Define ``end {...}`` blocks; invoke them at will using ``:end``. +* Define user-defined functions/subroutines using ``func``/``subr``; call them from other statements. +* Interactively specify statements to be executed immediately on the current record. +* Load any of the above from Miller-script files using ``:load``. + +The input "record" by default is the empty map but you can do things like +``$x=3``, or ``unset $y``, or ``$* = {"x": 3, "y": 4}`` to populate it. Or, ``:open +foo.dat`` followed by ``:read`` to populate it from a data file. + +Non-assignment expressions, such as ``7`` or ``true``, operate as filter conditions +in the ``put`` DSL: they can be used to specify whether a record will or won't be +included in the output-record stream. But here in the REPL, they are simply +printed to the terminal, e.g. if you type ``1+2``, you will see ``3``. + +Entering multi-line statements +---------------------------------------------------------------- + +* To enter multi-line statements, enter ``<`` on a line by itself, then the code (taking care + for semicolons), then ">" on a line by itself. These will be executed immediately. +* If you enter ``<<`` on a line by itself, then the code, then ``>>`` on a line by + itself, the statements will be remembered for executing on records with + ``:main``, as if you had done ``:load`` to load statements from a file. + +Examples +---------------------------------------------------------------- + +Use the REPL to look at arithmetic: + +.. code-block:: none + + $ mlr repl + + [mlr] 6/3 + 2 + + [mlr] 6/5 + 1.2 + + [mlr] typeof(6/3) + int + + [mlr] typeof(6/5) + float + +Read the first record from a small file: + +.. code-block:: none + + $ mlr repl + + [mlr] :open foo.dat + + [mlr] :read + + [mlr] :context + FILENAME="foo.dat",FILENUM=1,NR=1,FNR=1 + + [mlr] $* + { + "a": "eks", + "b": "wye", + "i": 4, + "x": 0.38139939387114097, + "y": 0.13418874328430463 + } + + [mlr] $z = $x + $i + + [mlr] :write + a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463,z=4.381399393871141 + +Skip until deep into a larger file, then inspect a record: + +.. code-block:: none + + $ mlr repl --csv + + [mlr] :open data/colored-shapes.csv + [mlr] :skip until NR == 10000 + [mlr] :r + [mlr] $* + { + "color": "yellow", + "shape": "circle", + "flag": 1, + "i": 99284, + "u": 0.6530503199545348, + "v": 0.23908588907834516, + "w": 0.4799125551304738, + "x": 6.379888206335166 + } + +History-editing +---------------------------------------------------------------- + +No command-line-history-editing feature is built in but **rlwrap mlr repl** is a +delight. You may need ``brew install rlwrap``, ``sudo apt-get install rlwrap``, +etc. depending on your platform. + +On-line help +---------------------------------------------------------------- + +After ``mlr repl``, type ``:help`` to see more about your options. In particular, ``:help examples``. diff --git a/docs6/repl.rst.in b/docs6/repl.rst.in new file mode 100644 index 000000000..ead45cd73 --- /dev/null +++ b/docs6/repl.rst.in @@ -0,0 +1,149 @@ +The REPL +================================================================ + +The Miller REPL (read-evaluate-print loop) is an interactive counterpart to record-processing using the ``put``/``filter`` language. + +It isn't a source-level debugger which lets you execute one source-code *statement* at a time -- however, it does let you operate on one *record* at a time. Further, it lets you use "immediate expressions", namely, you can interact with the language without having to provide data from an input file. + +Using Miller without the REPL +---------------------------------------------------------------- + +Using ``put`` and ``filter``, you can do the following as we've seen above: + +* Specify input format (e.g. ``--icsv``), output format (e.g. ``--ojson``), etc. using command-line flags. +* Specify filenames on the command line. +* Define ``begin {...}`` blocks which are executed before the first record is read. +* Define ``end {...}`` blocks which are executed after the last record is read. +* Define user-defined functions/subroutines using ``func`` and ``subr``. +* Specify statements to be executed on each record -- which are anything outside of ``begin``/``end``/``func``/``subr``. +* Example: + +GENRST_RUN_COMMAND +mlr --icsv --ojson --from example.csv head -n 2 \ + then put 'begin {print "HELLO"} $z = $x + $y; end {print "GOODBYE"}' +GENRST_EOF + +Using Miller with the REPL +---------------------------------------------------------------- + +Using the REPL, by contrast, you get interactive control over those same steps: + +* Specify input format (e.g. ``--icsv``), output format (e.g. ``--ojson``), etc. using command-line flags. +* REPL-only statements (non-DSL statements) start with ``:``, such as ``:help`` or ``:quit`` + or ``:open``. +* Specify filenames either on the command line or via ``:open`` at the Miller REPL. +* Read records one at a time using ``:read``. +* Write the current record (maybe after you've modified it with things like ``$z = $x + $y``) + using ``:write``. This goes to the terminal; you can use ``:> {filename}`` to make writes + go to a file, or ``:>> {filename}`` to append. +* You can type ``:reopen`` to go back to the start of the same file(s) you specified + with ``:open``. +* Skip ahead using statements ``:skip 10`` or ``:skip until NR == 100`` or + ``:skip until $status_code != 200``. +* Similarly, but processing records rather than skipping past them, using + ``:process`` rather than ``:skip``. Like ``:write``, these go to the screen; + use ``:> {filename}`` or ``:>> {filename}`` to log to a file instead. +* Define ``begin {...}`` blocks; invoke them at will using ``:begin``. +* Define ``end {...}`` blocks; invoke them at will using ``:end``. +* Define user-defined functions/subroutines using ``func``/``subr``; call them from other statements. +* Interactively specify statements to be executed immediately on the current record. +* Load any of the above from Miller-script files using ``:load``. + +The input "record" by default is the empty map but you can do things like +``$x=3``, or ``unset $y``, or ``$* = {"x": 3, "y": 4}`` to populate it. Or, ``:open +foo.dat`` followed by ``:read`` to populate it from a data file. + +Non-assignment expressions, such as ``7`` or ``true``, operate as filter conditions +in the ``put`` DSL: they can be used to specify whether a record will or won't be +included in the output-record stream. But here in the REPL, they are simply +printed to the terminal, e.g. if you type ``1+2``, you will see ``3``. + +Entering multi-line statements +---------------------------------------------------------------- + +* To enter multi-line statements, enter ``<`` on a line by itself, then the code (taking care + for semicolons), then ">" on a line by itself. These will be executed immediately. +* If you enter ``<<`` on a line by itself, then the code, then ``>>`` on a line by + itself, the statements will be remembered for executing on records with + ``:main``, as if you had done ``:load`` to load statements from a file. + +Examples +---------------------------------------------------------------- + +Use the REPL to look at arithmetic: + +.. code-block:: none + + $ mlr repl + + [mlr] 6/3 + 2 + + [mlr] 6/5 + 1.2 + + [mlr] typeof(6/3) + int + + [mlr] typeof(6/5) + float + +Read the first record from a small file: + +.. code-block:: none + + $ mlr repl + + [mlr] :open foo.dat + + [mlr] :read + + [mlr] :context + FILENAME="foo.dat",FILENUM=1,NR=1,FNR=1 + + [mlr] $* + { + "a": "eks", + "b": "wye", + "i": 4, + "x": 0.38139939387114097, + "y": 0.13418874328430463 + } + + [mlr] $z = $x + $i + + [mlr] :write + a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463,z=4.381399393871141 + +Skip until deep into a larger file, then inspect a record: + +.. code-block:: none + + $ mlr repl --csv + + [mlr] :open data/colored-shapes.csv + [mlr] :skip until NR == 10000 + [mlr] :r + [mlr] $* + { + "color": "yellow", + "shape": "circle", + "flag": 1, + "i": 99284, + "u": 0.6530503199545348, + "v": 0.23908588907834516, + "w": 0.4799125551304738, + "x": 6.379888206335166 + } + +History-editing +---------------------------------------------------------------- + +No command-line-history-editing feature is built in but **rlwrap mlr repl** is a +delight. You may need ``brew install rlwrap``, ``sudo apt-get install rlwrap``, +etc. depending on your platform. + +On-line help +---------------------------------------------------------------- + +After ``mlr repl``, type ``:help`` to see more about your options. In particular, ``:help examples``. diff --git a/go/src/auxents/repl/verbs.go b/go/src/auxents/repl/verbs.go index cea9454aa..79203ab09 100644 --- a/go/src/auxents/repl/verbs.go +++ b/go/src/auxents/repl/verbs.go @@ -901,7 +901,7 @@ to record-processing using the put/filter DSL (domain-specific language).`) * Define user-defined functions/subroutines using func and subr. * Specify statements to be executed on each record -- which are anything outside of begin/end/func/subr. * Example: - %s --icsv --ojson put 'begin {print "HELLO"} $z = $x + $y end {print "GOODBYE"}`, + %s --icsv --ojson put 'begin {print "HELLO"} $z = $x + $y; end {print "GOODBYE"}`, repl.exeName) fmt.Println() fmt.Println()