mirror of
https://github.com/johnkerl/miller.git
synced 2026-07-18 00:45:47 +00:00
More Miller 6 doc info (#849)
* Info on operator precedence and the REPL * Information on indexed emit/emit1 in Miller 6
This commit is contained in:
John Kerl
GitHub
parent
b8698f0a9b
commit
ebef818a00
13 changed files with 240 additions and 36 deletions
|
|
@ -3054,5 +3054,5 @@ SEE ALSO
|
|||
|
||||
|
||||
|
||||
2022-01-09 MILLER(1)
|
||||
2022-01-10 MILLER(1)
|
||||
</pre>
|
||||
|
|
|
|||
|
|
@ -3033,4 +3033,4 @@ SEE ALSO
|
|||
|
||||
|
||||
|
||||
2022-01-09 MILLER(1)
|
||||
2022-01-10 MILLER(1)
|
||||
|
|
|
|||
|
|
@ -237,6 +237,18 @@ For example (see [https://github.com/johnkerl/miller/issues/178](https://github.
|
|||
]
|
||||
</pre>
|
||||
|
||||
### Deduping of repeated field names
|
||||
|
||||
By default, field names are deduped for all file formats except JSON / JSON Lines. So if you
|
||||
have an input record with `x=8,x=9` then the second field's key is renamed to
|
||||
`x_2` and so on -- the record scans as `x=8,x_2=9`. Use `mlr
|
||||
--no-dedupe-field-names` to suppress this, and have the record be scanned as
|
||||
`x=9`.
|
||||
|
||||
For JSON and JSON Lines, the last duplicated key in an input record is always retained,
|
||||
regardless of `mlr --no-dedupe-field-names`: `{"x":8,"x":9}` scans as if it
|
||||
were `{"x":9}`.
|
||||
|
||||
### Regex support for IFS and IPS
|
||||
|
||||
You can now split fields on whitespace when whitespace is a mix of tabs and
|
||||
|
|
@ -304,9 +316,24 @@ IFS and IPS can be regular expressions now. Please see the section on [multi-cha
|
|||
|
||||
### Emit statements
|
||||
|
||||
* Emitting a map-valued expression now requires either a temporary variable or the new `emit1` keyword. Please see the
|
||||
[page on emit statements](reference-dsl-output-statements.md#emit1-and-emitemitpemitf) for more information.
|
||||
* By default, field names are deduped for all file formats except JSON. So if you have an input record with `x=8,x=9` then the second field's key is renamed to `x_2` and so on -- the record scans as `x=8,x_2=9`. Use `mlr --no-dedupe-field-names` to suppress this, and have the record be scanned as `x=9`. For JSON, the last duplicated key in an input record is always retained, regardless of `mlr --no-dedupe-field-names`: `{"x":8,"x":9}` scans as if it were `{"x":9}`.
|
||||
Variables must be non-indexed on `emit`. To emit an indexed variable now requires the new `emit1` keyword.
|
||||
|
||||
<pre class="pre-highlight-in-pair">
|
||||
<b>mlr5 -n put 'end {@input={"a":1}; emit @input["a"]}'</b>
|
||||
</pre>
|
||||
<pre class="pre-non-highlight-in-pair">
|
||||
input=1
|
||||
</pre>
|
||||
|
||||
<pre class="pre-highlight-in-pair">
|
||||
<b>mlr -n put 'end {@input={"a":1}; emit1 {"input":@input["a"]}}'</b>
|
||||
</pre>
|
||||
<pre class="pre-non-highlight-in-pair">
|
||||
input=1
|
||||
</pre>
|
||||
|
||||
Please see the [section on emit statements](reference-dsl-output-statements.md#emit1-and-emitemitpemitf)
|
||||
for more information.
|
||||
|
||||
## Developer-specific aspects
|
||||
|
||||
|
|
|
|||
|
|
@ -191,6 +191,18 @@ GENMD-RUN-COMMAND
|
|||
echo '{ "x": 1.230, "y": 1.230000000 }' | mlr --json cat
|
||||
GENMD-EOF
|
||||
|
||||
### Deduping of repeated field names
|
||||
|
||||
By default, field names are deduped for all file formats except JSON / JSON Lines. So if you
|
||||
have an input record with `x=8,x=9` then the second field's key is renamed to
|
||||
`x_2` and so on -- the record scans as `x=8,x_2=9`. Use `mlr
|
||||
--no-dedupe-field-names` to suppress this, and have the record be scanned as
|
||||
`x=9`.
|
||||
|
||||
For JSON and JSON Lines, the last duplicated key in an input record is always retained,
|
||||
regardless of `mlr --no-dedupe-field-names`: `{"x":8,"x":9}` scans as if it
|
||||
were `{"x":9}`.
|
||||
|
||||
### Regex support for IFS and IPS
|
||||
|
||||
You can now split fields on whitespace when whitespace is a mix of tabs and
|
||||
|
|
@ -258,9 +270,19 @@ IFS and IPS can be regular expressions now. Please see the section on [multi-cha
|
|||
|
||||
### Emit statements
|
||||
|
||||
* Emitting a map-valued expression now requires either a temporary variable or the new `emit1` keyword. Please see the
|
||||
[page on emit statements](reference-dsl-output-statements.md#emit1-and-emitemitpemitf) for more information.
|
||||
* By default, field names are deduped for all file formats except JSON. So if you have an input record with `x=8,x=9` then the second field's key is renamed to `x_2` and so on -- the record scans as `x=8,x_2=9`. Use `mlr --no-dedupe-field-names` to suppress this, and have the record be scanned as `x=9`. For JSON, the last duplicated key in an input record is always retained, regardless of `mlr --no-dedupe-field-names`: `{"x":8,"x":9}` scans as if it were `{"x":9}`.
|
||||
Variables must be non-indexed on `emit`. To emit an indexed variable now requires the new `emit1` keyword.
|
||||
|
||||
GENMD-CARDIFY-HIGHLIGHT-ONE
|
||||
mlr5 -n put 'end {@input={"a":1}; emit @input["a"]}'
|
||||
input=1
|
||||
GENMD-EOF
|
||||
|
||||
GENMD-RUN-COMMAND
|
||||
mlr -n put 'end {@input={"a":1}; emit1 {"input":@input["a"]}}'
|
||||
GENMD-EOF
|
||||
|
||||
Please see the [section on emit statements](reference-dsl-output-statements.md#emit1-and-emitemitpemitf)
|
||||
for more information.
|
||||
|
||||
## Developer-specific aspects
|
||||
|
||||
|
|
|
|||
|
|
@ -46,6 +46,9 @@ Operators are listed in order of decreasing precedence, highest first.
|
|||
| `? :` | right to left |
|
||||
| `=` | N/A for Miller (there is no $a=$b=$c) |
|
||||
|
||||
See also the [section on parsing and operator precedence in the REPL](repl.md#parsing-and-operator-precedence)
|
||||
for information on how to examine operator precedence interactively.
|
||||
|
||||
## Operator and function semantics
|
||||
|
||||
* Functions are often pass-throughs straight to the system-standard Go libraries.
|
||||
|
|
|
|||
|
|
@ -30,6 +30,9 @@ Operators are listed in order of decreasing precedence, highest first.
|
|||
| `? :` | right to left |
|
||||
| `=` | N/A for Miller (there is no $a=$b=$c) |
|
||||
|
||||
See also the [section on parsing and operator precedence in the REPL](repl.md#parsing-and-operator-precedence)
|
||||
for information on how to examine operator precedence interactively.
|
||||
|
||||
## Operator and function semantics
|
||||
|
||||
* Functions are often pass-throughs straight to the system-standard Go libraries.
|
||||
|
|
|
|||
|
|
@ -363,23 +363,49 @@ So, `emit1` can handle syntactic richness in the one thing being emitted;
|
|||
`emitf`, `emit`, and `emitp` can handle syntactic richness in the side-by-side
|
||||
placement, indexing, and redirection.
|
||||
|
||||
(Mnemonic: If all you want is to insert a new record into the record stream, `emit1` is probably the one you want.)
|
||||
(Mnemonic: If all you want is to insert a new record into the record stream, `emit1` is probably the _one_ you want.)
|
||||
|
||||
What this means is that if you want to emit an expression which evaluates to a map, you can do quite simply
|
||||
|
||||
<pre class="pre-non-highlight-non-pair">
|
||||
mlr --c2p --from example.csv put -q '
|
||||
emit1 mapsum({"id": NR}, $*)
|
||||
'
|
||||
<pre class="pre-highlight-in-pair">
|
||||
<b>mlr --c2p --from example.csv put -q '</b>
|
||||
<b> emit1 mapsum({"id": NR}, $*)</b>
|
||||
<b>'</b>
|
||||
</pre>
|
||||
<pre class="pre-non-highlight-in-pair">
|
||||
id color shape flag k index quantity rate
|
||||
1 yellow triangle true 1 11 43.6498 9.8870
|
||||
2 red square true 2 15 79.2778 0.0130
|
||||
3 red circle true 3 16 13.8103 2.9010
|
||||
4 red square false 4 48 77.5542 7.4670
|
||||
5 purple triangle false 5 51 81.2290 8.5910
|
||||
6 red square false 6 64 77.1991 9.5310
|
||||
7 purple triangle false 7 65 80.1405 5.8240
|
||||
8 yellow circle true 8 73 63.9785 4.2370
|
||||
9 yellow circle true 9 87 63.5058 8.3350
|
||||
10 purple square false 10 91 72.3735 8.2430
|
||||
</pre>
|
||||
|
||||
And if you want indexing, redirects, etc., just assign to a temporary variable and use one of the other emit variants:
|
||||
|
||||
<pre class="pre-non-highlight-non-pair">
|
||||
mlr --c2p --from example.csv put -q '
|
||||
o = mapsum({"id": NR}, $*);
|
||||
emit > stderr, o;
|
||||
'
|
||||
<pre class="pre-highlight-in-pair">
|
||||
<b>mlr --c2p --from example.csv put -q '</b>
|
||||
<b> o = mapsum({"id": NR}, $*);</b>
|
||||
<b> emit o;</b>
|
||||
<b>'</b>
|
||||
</pre>
|
||||
<pre class="pre-non-highlight-in-pair">
|
||||
id color shape flag k index quantity rate
|
||||
1 yellow triangle true 1 11 43.6498 9.8870
|
||||
2 red square true 2 15 79.2778 0.0130
|
||||
3 red circle true 3 16 13.8103 2.9010
|
||||
4 red square false 4 48 77.5542 7.4670
|
||||
5 purple triangle false 5 51 81.2290 8.5910
|
||||
6 red square false 6 64 77.1991 9.5310
|
||||
7 purple triangle false 7 65 80.1405 5.8240
|
||||
8 yellow circle true 8 73 63.9785 4.2370
|
||||
9 yellow circle true 9 87 63.5058 8.3350
|
||||
10 purple square false 10 91 72.3735 8.2430
|
||||
</pre>
|
||||
|
||||
## Emitf statements
|
||||
|
|
|
|||
|
|
@ -174,23 +174,23 @@ So, `emit1` can handle syntactic richness in the one thing being emitted;
|
|||
`emitf`, `emit`, and `emitp` can handle syntactic richness in the side-by-side
|
||||
placement, indexing, and redirection.
|
||||
|
||||
(Mnemonic: If all you want is to insert a new record into the record stream, `emit1` is probably the one you want.)
|
||||
(Mnemonic: If all you want is to insert a new record into the record stream, `emit1` is probably the _one_ you want.)
|
||||
|
||||
What this means is that if you want to emit an expression which evaluates to a map, you can do quite simply
|
||||
|
||||
GENMD-CARDIFY
|
||||
mlr --c2p --from example.csv put -q '
|
||||
emit1 mapsum({"id": NR}, $*)
|
||||
'
|
||||
GENMD-RUN-COMMAND
|
||||
mlr --c2p --from example.csv put -q '
|
||||
emit1 mapsum({"id": NR}, $*)
|
||||
'
|
||||
GENMD-EOF
|
||||
|
||||
And if you want indexing, redirects, etc., just assign to a temporary variable and use one of the other emit variants:
|
||||
|
||||
GENMD-CARDIFY
|
||||
mlr --c2p --from example.csv put -q '
|
||||
o = mapsum({"id": NR}, $*);
|
||||
emit > stderr, o;
|
||||
'
|
||||
GENMD-RUN-COMMAND
|
||||
mlr --c2p --from example.csv put -q '
|
||||
o = mapsum({"id": NR}, $*);
|
||||
emit o;
|
||||
'
|
||||
GENMD-EOF
|
||||
|
||||
## Emitf statements
|
||||
|
|
|
|||
|
|
@ -207,6 +207,71 @@ Skip until deep into a larger file, then inspect a record:
|
|||
}
|
||||
</pre>
|
||||
|
||||
## Parsing and operator precedence
|
||||
|
||||
You can invoke `mlr repl` with the `-v` or `-d` flags to show parse trees for expressions you enter.
|
||||
(The `-v` and `-d` flags differ only in the format they use to present the parse trees.)
|
||||
For example, if you have any questions about operator precedence, you can check the
|
||||
[operator-precedence section](reference-dsl-operators.md#operator-precedence) -- or, you can
|
||||
try it out and see for yourself:
|
||||
|
||||
<pre class="pre-highlight-in-pair">
|
||||
<b>mlr repl -v</b>
|
||||
</pre>
|
||||
<pre class="pre-non-highlight-in-pair">
|
||||
Miller 6.0.0-rc1 REPL for darwin/amd64/go1.16.5
|
||||
Docs: https://miller.readthedocs.io
|
||||
Type ':h' or ':help' for online help; ':q' or ':quit' to quit.
|
||||
[mlr] x = 1 * 2 + 3
|
||||
* statement block
|
||||
* assignment "="
|
||||
* local variable "x"
|
||||
* operator "+"
|
||||
* operator "*"
|
||||
* int literal "1"
|
||||
* int literal "2"
|
||||
* int literal "3"
|
||||
[mlr] x = 1 + 2 * 3
|
||||
* statement block
|
||||
* assignment "="
|
||||
* local variable "x"
|
||||
* operator "+"
|
||||
* int literal "1"
|
||||
* operator "*"
|
||||
* int literal "2"
|
||||
* int literal "3"
|
||||
</pre>
|
||||
|
||||
<pre class="pre-highlight-in-pair">
|
||||
<b>mlr repl -d</b>
|
||||
</pre>
|
||||
<pre class="pre-non-highlight-in-pair">
|
||||
$ mrpl -d
|
||||
Miller 6.0.0-rc1 REPL for darwin/amd64/go1.16.5
|
||||
Docs: https://miller.readthedocs.io
|
||||
Type ':h' or ':help' for online help; ':q' or ':quit' to quit.
|
||||
[mlr] x = 1 * 2 + 3
|
||||
(statement-block
|
||||
(=
|
||||
x
|
||||
(+
|
||||
(* 1 2)
|
||||
3
|
||||
)
|
||||
)
|
||||
)
|
||||
[mlr] x = 1 + 2 * 3
|
||||
(statement-block
|
||||
(=
|
||||
x
|
||||
(+
|
||||
1
|
||||
(* 2 3)
|
||||
)
|
||||
)
|
||||
)
|
||||
</pre>
|
||||
|
||||
## History-editing
|
||||
|
||||
No command-line-history-editing feature is built in but **rlwrap mlr repl** is a
|
||||
|
|
|
|||
|
|
@ -157,6 +157,67 @@ mlr repl --csv
|
|||
}
|
||||
GENMD-EOF
|
||||
|
||||
## Parsing and operator precedence
|
||||
|
||||
You can invoke `mlr repl` with the `-v` or `-d` flags to show parse trees for expressions you enter.
|
||||
(The `-v` and `-d` flags differ only in the format they use to present the parse trees.)
|
||||
For example, if you have any questions about operator precedence, you can check the
|
||||
[operator-precedence section](reference-dsl-operators.md#operator-precedence) -- or, you can
|
||||
try it out and see for yourself:
|
||||
|
||||
GENMD-CARDIFY-HIGHLIGHT-ONE
|
||||
mlr repl -v
|
||||
Miller 6.0.0-rc1 REPL for darwin/amd64/go1.16.5
|
||||
Docs: https://miller.readthedocs.io
|
||||
Type ':h' or ':help' for online help; ':q' or ':quit' to quit.
|
||||
[mlr] x = 1 * 2 + 3
|
||||
* statement block
|
||||
* assignment "="
|
||||
* local variable "x"
|
||||
* operator "+"
|
||||
* operator "*"
|
||||
* int literal "1"
|
||||
* int literal "2"
|
||||
* int literal "3"
|
||||
[mlr] x = 1 + 2 * 3
|
||||
* statement block
|
||||
* assignment "="
|
||||
* local variable "x"
|
||||
* operator "+"
|
||||
* int literal "1"
|
||||
* operator "*"
|
||||
* int literal "2"
|
||||
* int literal "3"
|
||||
GENMD-EOF
|
||||
|
||||
GENMD-CARDIFY-HIGHLIGHT-ONE
|
||||
mlr repl -d
|
||||
$ mrpl -d
|
||||
Miller 6.0.0-rc1 REPL for darwin/amd64/go1.16.5
|
||||
Docs: https://miller.readthedocs.io
|
||||
Type ':h' or ':help' for online help; ':q' or ':quit' to quit.
|
||||
[mlr] x = 1 * 2 + 3
|
||||
(statement-block
|
||||
(=
|
||||
x
|
||||
(+
|
||||
(* 1 2)
|
||||
3
|
||||
)
|
||||
)
|
||||
)
|
||||
[mlr] x = 1 + 2 * 3
|
||||
(statement-block
|
||||
(=
|
||||
x
|
||||
(+
|
||||
1
|
||||
(* 2 3)
|
||||
)
|
||||
)
|
||||
)
|
||||
GENMD-EOF
|
||||
|
||||
## History-editing
|
||||
|
||||
No command-line-history-editing feature is built in but **rlwrap mlr repl** is a
|
||||
|
|
|
|||
|
|
@ -3033,4 +3033,4 @@ SEE ALSO
|
|||
|
||||
|
||||
|
||||
2022-01-09 MILLER(1)
|
||||
2022-01-10 MILLER(1)
|
||||
|
|
|
|||
|
|
@ -2,12 +2,12 @@
|
|||
.\" Title: mlr
|
||||
.\" Author: [see the "AUTHOR" section]
|
||||
.\" Generator: ./mkman.rb
|
||||
.\" Date: 2022-01-09
|
||||
.\" Date: 2022-01-10
|
||||
.\" Manual: \ \&
|
||||
.\" Source: \ \&
|
||||
.\" Language: English
|
||||
.\"
|
||||
.TH "MILLER" "1" "2022-01-09" "\ \&" "\ \&"
|
||||
.TH "MILLER" "1" "2022-01-10" "\ \&" "\ \&"
|
||||
.\" -----------------------------------------------------------------
|
||||
.\" * Portability definitions
|
||||
.\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
|
|
|||
7
todo.txt
7
todo.txt
|
|
@ -3,13 +3,10 @@ PUNCHDOWN LIST
|
|||
|
||||
----------------------------------------------------------------
|
||||
|
||||
* field-dedupe separate section at nim6
|
||||
* mrpl -v/-d doc
|
||||
* emit1 note
|
||||
* document cloudthings, e.g.
|
||||
o go.yml
|
||||
o codespell.yml
|
||||
- codespell --check-filenames --skip *.csv,*.dkvp,*.txt,*.js,*.html,*.map,./tags,./test/cases --ignore-words-list denom,inTerm,inout,iput,nd,nin,numer,Wit,te,wee
|
||||
- codespell --check-filenames --skip *.csv,*.dkvp,*.txt,*.js,*.html,*.map,./tags,./test/cases --ignore-words-list denom,inTerm,inout,iput,nd,nin,numer,Wit,te,wee
|
||||
o readthedocs triggers
|
||||
* sync-print option; or (yuck) another xprint variant; or ...; emph dump/eprint
|
||||
? trace-mode ?
|
||||
|
|
@ -318,7 +315,7 @@ MAYBES
|
|||
|
||||
* dotted-syntax support in verbs?
|
||||
|
||||
* repl as verb -- ? 'put --repl' maybe
|
||||
* repl as verb -- ? 'put --repl' maybe
|
||||
|
||||
* json-triple-quote -- what can be done here?
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue