Ran "make dev" after #949

This commit is contained in:
John Kerl 2022-02-18 18:45:46 -05:00
parent 8aecba305f
commit f2fa246540
18 changed files with 110 additions and 66 deletions

View file

@ -74,7 +74,6 @@ def dkvpline2map(line, ips, ifs):
# ----------------------------------------------------------------
# ops and ofs (output pair separator and output field separator) are nominally '=' and ','.
def map2dkvpline(map , ops, ofs):
line = ''
pairs = []
for key in map:
pairs.append(str(key) + ops + str(map[key]))

View file

@ -597,11 +597,28 @@ light
While you can do format conversion using `mlr --icsv --ojson cat myfile.csv`, there are also keystroke-savers for this purpose, such as `mlr --c2j cat myfile.csv`. For a complete list:
<pre class="pre-highlight-in-pair">
<b>mlr help format-conversion</b>
<b>mlr help format-conversion-keystroke-saver-flags</b>
</pre>
<pre class="pre-non-highlight-in-pair">
No help found for "format-conversion". Please try 'mlr help find format-conversion' for approximate match.
See also 'mlr help topics'.
FORMAT-CONVERSION KEYSTROKE-SAVER FLAGS
As keystroke-savers for format-conversion you may use the following.
The letters c, t, j, l, d, n, x, p, and m refer to formats CSV, TSV, DKVP, NIDX,
JSON, JSON Lines, XTAB, PPRINT, and markdown, respectively. Note that markdown
format is available for output only.
| In\out | CSV | TSV | JSON | JSONL | DKVP | NIDX | XTAB | PPRINT | Markdown |
+--------+-------+-------+--------+--------+--------+--------+--------+----------+
| CSV | | --c2t | --c2j | --c2l | --c2d | --c2n | --c2x | --c2p | --c2m |
| TSV | --t2c | | --t2j | --t2l | --t2d | --t2n | --t2x | --t2p | --t2m |
| JSON | --j2c | --j2t | | --j2l | --j2d | --j2n | --j2x | --j2p | --j2m |
| JSONL | --l2c | --l2t | | | --l2d | --l2n | --l2x | --l2p | --l2m |
| DKVP | --d2c | --d2t | --d2j | --d2l | | --d2n | --d2x | --d2p | --d2m |
| NIDX | --n2c | --n2t | --n2j | --n2l | --n2d | | --n2x | --n2p | --n2m |
| XTAB | --x2c | --x2t | --x2j | --x2l | --x2d | --x2n | | --x2p | --x2m |
| PPRINT | --p2c | --p2t | --p2j | --p2l | --p2d | --p2n | --p2x | | --p2m |
-p Keystroke-saver for `--nidx --fs space --repifs`.
-T Keystroke-saver for `--nidx --fs tab`.
</pre>
## Comments in data
@ -609,11 +626,37 @@ See also 'mlr help topics'.
You can include comments within your data files, and either have them ignored, or passed directly through to the standard output as soon as they are encountered:
<pre class="pre-highlight-in-pair">
<b>mlr help comments-in-data</b>
<b>mlr help comments-in-data-flags</b>
</pre>
<pre class="pre-non-highlight-in-pair">
No help found for "comments-in-data". Please try 'mlr help find comments-in-data' for approximate match.
See also 'mlr help topics'.
COMMENTS-IN-DATA FLAGS
Miller lets you put comments in your data, such as
# This is a comment for a CSV file
a,b,c
1,2,3
4,5,6
Notes:
* Comments are only honored at the start of a line.
* In the absence of any of the below four options, comments are data like
any other text. (The comments-in-data feature is opt-in.)
* When `--pass-comments` is used, comment lines are written to standard output
immediately upon being read; they are not part of the record stream. Results
may be counterintuitive. A suggestion is to place comments at the start of
data files.
--pass-comments Immediately print commented lines (prefixed by `#`)
within the input.
--pass-comments-with {string}
Immediately print commented lines within input, with
specified prefix.
--skip-comments Ignore commented lines (prefixed by `#`) within the
input.
--skip-comments-with {string}
Ignore commented lines within input, with specified
prefix.
</pre>
Examples:

View file

@ -141,7 +141,7 @@ _delimiter_ can be used as a synonym for [_separator_](#separator).
## division
Miller uses [pythonic division](http://127.0.0.1:8000/reference-main-arithmetic.md#pythonic-division)
Miller uses [pythonic division](reference-main-arithmetic.md#pythonic-division)
for quotients of integers, with the exception that integer divided by integer
is integer (not float) if the quotient can be represented exactly as an
integer.
@ -212,7 +212,8 @@ See also the [emit-statements section](reference-dsl-output-statements.md#emit-s
## empty
Refers to the string with zero characters. For example, in a CSV file with [header line](#header)
`a,b,c` and data
`a,b,c` and data `,,` the three fields are empty; with data `1,2,` the first two fields (`a` and `b`)
are not empty and the third field `c` is empty.
## end
@ -240,7 +241,7 @@ Same as [`printn`](#printn), except it prints to [stderr](#stderr) rather than [
## false
A [keyword](#keyword) in the [Miller programming language](miller-programming-language.md) for the
boolean literal; signified by `true` in Python; in some languages (such as C)
boolean literal; signified by `False` in Python; in some languages (such as C)
signified by the zero integer value.
## field
@ -335,7 +336,7 @@ to write your own functions.
## function literal
A function without a name, like `func(a,b) { return a + 2*b + 7}`, assigned to
A function without a name, like `func(a,b) {return a + 2*b + 7}`, assigned to
a local variable or passed to a [higher-order
function](reference-dsl-higher-order-functions.md) like `apply` or `sort`. See
the [section on function literals](reference-dsl-user-defined-functions.md#function-literals).
@ -343,7 +344,7 @@ the [section on function literals](reference-dsl-user-defined-functions.md#funct
## GZIP / .gz
A [data-compression format supported by Miller](reference-main-compressed-data.md).
Files compressed using GZIP compression normally end in`.gz`.
Files compressed using GZIP compression normally end in `.gz`.
## hashmap
@ -362,8 +363,8 @@ and the [Miller CSV section](file-formats.md#csvtsvasvusvetc).
## heterogeneity
Referring to data where all records have the same keys, in the same order. See the
[record-heterogeneity page](record-heterogeneity.md#homogeneousrectangular-data).
Referring to data where not all records have the same keys, in the same order. See the
[record-heterogeneity page](record-heterogeneity.md#ragged-data).
## higher-order function
@ -372,6 +373,11 @@ A function which takes another function as an argument, such as
[`apply`](reference-dsl-builtin-functions.md#apply). See the [page on
higher-order functions](reference-dsl-higher-order-functions.md).
## homogeneity
Referring to data where all records have the same keys, in the same order. See the
[record-heterogeneity page](record-heterogeneity.md#homogeneousrectangular-data).
## if
A [keyword](#keyword) which is used to indicate the start of an [if-statement](reference-dsl-control-structures.md)
@ -647,7 +653,7 @@ See also the [Miller command structure page](reference-main-overview.md).
## rectangular
Referring to data where all records have the same keys, in the same order. Synonymous
with [heterogeneous](#heterogeneity). See the
with [homogeneous](#homogeneity). See the
[record-heterogeneity page](record-heterogeneity.md#homogeneousrectangular-data).
## REPL
@ -694,14 +700,14 @@ page](record-heterogeneity.md#sparse-data).
A [keyword](#keyword) in the
[Miller programming language](miller-programming-language.md)
for [print, dump, and tee statements](http://127.0.0.1:8000/reference-dsl-output-statements.md#tee-statements)
indicating that data are to be sent to the [_standard output_](https://en.wikipedia.org/wiki/Standard_streams#Standard_output_(stderr)).
for [print, dump, and tee statements](reference-dsl-output-statements.md#tee-statements)
indicating that data are to be sent to the [_standard error_](https://en.wikipedia.org/wiki/Standard_streams#Standard_error_(stderr)).
## stdout
A [keyword](#keyword) in the
[Miller programming language](miller-programming-language.md)
for [print, dump, and tee statements](http://127.0.0.1:8000/reference-dsl-output-statements.md#tee-statements)
for [print, dump, and tee statements](reference-dsl-output-statements.md#tee-statements)
indicating that data are to be sent to the [_standard output_](https://en.wikipedia.org/wiki/Standard_streams#Standard_output_(stdout)).
## str

View file

@ -2222,7 +2222,7 @@ FUNCTIONS FOR FILTER/PUT
(class=collections #args=1) Returns array of keys of map or array
get_values
(class=collections #args=1) Returns array of keys of map or array -- in the latter case, returns a copy of the array
(class=collections #args=1) Returns array of values of map or array -- in the latter case, returns a copy of the array
gmt2localtime
(class=time #args=1,2) Convert from a GMT-time string to a local-time string. Consulting $TZ unless second argument is supplied.
@ -2825,8 +2825,8 @@ KEYWORDS FOR PUT AND FILTER
dump: prints all currently defined out-of-stream variables immediately
to stdout as JSON.
With &gt;, &gt;&gt;, or |, the data do not become part of the output record stream but
are instead redirected.
With &gt;, &gt;&gt;, or |, the data do not go directly to stdout but are instead
redirected.
The &gt; and &gt;&gt; are for write and append, as in the shell, but (as with awk) the
file-overwrite for &gt; is on first write, not per record. The | is for piping to
@ -3181,5 +3181,5 @@ SEE ALSO
2022-02-15 MILLER(1)
2022-02-18 MILLER(1)
</pre>

View file

@ -2201,7 +2201,7 @@ FUNCTIONS FOR FILTER/PUT
(class=collections #args=1) Returns array of keys of map or array
get_values
(class=collections #args=1) Returns array of keys of map or array -- in the latter case, returns a copy of the array
(class=collections #args=1) Returns array of values of map or array -- in the latter case, returns a copy of the array
gmt2localtime
(class=time #args=1,2) Convert from a GMT-time string to a local-time string. Consulting $TZ unless second argument is supplied.
@ -2804,8 +2804,8 @@ KEYWORDS FOR PUT AND FILTER
dump: prints all currently defined out-of-stream variables immediately
to stdout as JSON.
With >, >>, or |, the data do not become part of the output record stream but
are instead redirected.
With >, >>, or |, the data do not go directly to stdout but are instead
redirected.
The > and >> are for write and append, as in the shell, but (as with awk) the
file-overwrite for > is on first write, not per record. The | is for piping to
@ -3160,4 +3160,4 @@ SEE ALSO
2022-02-15 MILLER(1)
2022-02-18 MILLER(1)

View file

@ -72,17 +72,15 @@ Aggregate custom per-column statistics:
Iterate over data using DSL expressions:
<pre class="pre-highlight-in-pair">
<pre class="pre-highlight-non-pair">
<b>mlr --from estimates.tbl put '</b>
</pre>
<pre class="pre-non-highlight-in-pair">
for (k,v in $*) {
if (is_numeric(v) && k =~ "^[t-z].*$") {
$sum += v; $count += 1
}
}
$mean = $sum / $count # no assignment if count unset
'
<b> for (k,v in $*) {</b>
<b> if (is_numeric(v) && k =~ "^[t-z].*$") {</b>
<b> $sum += v; $count += 1</b>
<b> }</b>
<b> }</b>
<b> $mean = $sum / $count # no assignment if count unset</b>
<b>'</b>
</pre>
Run DSL expressions from a script file:

View file

@ -314,7 +314,7 @@ IFS and IPS can be regular expressions now. Please see the section on [multi-cha
### Type-inference
* The `-S` and `-F` flags to `mlr put` and `mlr filter` are ignored, since type-inference is no longer done in `mlr put` and `mlr filter`, but rather, when records are first read. You can use `mlr -S` and `mlr -A`, respectively, instead to control type-inference within the record-readers.
* Octal numbers like `0123` and `07` are type-inferred as string. Use `mlr -O` to infer them as octal integers. Note that `08` and `09` will then infer as deicmal integers.
* Octal numbers like `0123` and `07` are type-inferred as string. Use `mlr -O` to infer them as octal integers. Note that `08` and `09` will then infer as decimal integers.
* Any numbers prefix with `0o`, e.g. `0o377`, are already treated as octal regardless of `mlr -O` -- `mlr -O` only affects how leading-zero integers are handled.
* See also the [miscellaneous-flags reference](reference-main-flag-list.md#miscellaneous-flags).

View file

@ -94,7 +94,7 @@ a,b,c
This example is still homogeneous, though: every row has the same keys, in the same order: `a,b,c`.
Empty values don't make the data heterogeneous.
Note however that we can use the [`fill-down`](reference-verbs.md#fill-empty) verb to make these
Note however that we can use the [`fill-empty`](reference-verbs.md#fill-empty) verb to make these
values non-empty, if we like:
<pre class="pre-highlight-in-pair">
@ -121,7 +121,7 @@ a,b,c
7,8,9,10
</pre>
If you `mlr csv cat` this, you'll get an error message:
If you `mlr --csv cat` this, you'll get an error message:
<pre class="pre-highlight-in-pair">
<b>mlr --csv cat data/het/ragged.csv</b>

View file

@ -408,7 +408,7 @@ get_keys (class=collections #args=1) Returns array of keys of map or array
### get_values
<pre class="pre-non-highlight-non-pair">
get_values (class=collections #args=1) Returns array of keys of map or array -- in the latter case, returns a copy of the array
get_values (class=collections #args=1) Returns array of values of map or array -- in the latter case, returns a copy of the array
</pre>

View file

@ -55,7 +55,7 @@ for information on how to examine operator precedence interactively.
* The [`min`](reference-dsl-builtin-functions.md#min) and [`max`](reference-dsl-builtin-functions.md#max) functions are different from other multi-argument functions which return null if any of their inputs are null: for [`min`](reference-dsl-builtin-functions.md#min) and [`max`](reference-dsl-builtin-functions.md#max), by contrast, if one argument is absent-null, the other is returned. Empty-null loses min or max against numeric or boolean; empty-null is less than any other string.
* Symmetrically with respect to the bitwise OR, XOR, and AND operators
* Symmetrically with respect to the bitwise OR, AND, and XOR operators
[`|`](reference-dsl-builtin-functions.md#bitwise-or),
[`&`](reference-dsl-builtin-functions.md#bitwise-and), and
[`^`](reference-dsl-builtin-functions.md#bitwise-xor), Miller has logical operators

View file

@ -20,14 +20,14 @@ You can **output** variable-values or expressions in **five ways**:
* **Assign** them to stream-record fields. For example, `$cumulative_sum = @sum`. For another example, `$nr = NR` adds a field named `nr` to each output record, containing the value of the built-in variable `NR` as of when that record was ingested.
* Use **emit1**/**emit**/**emitp**/**emitf** to send out-of-stream variables' current values to the output record stream, e.g. `@sum += $x; emit1 @sum` which produces an extra record such as `sum=3.1648382`. These records, just like records from input file(s), participate in downstream [then-chaining](reference-main-then-chaining.md) to other verbs.
* Use the **print** or **eprint** keywords which immediately print an expression *directly to standard output or standard error*, respectively. Note that `dump`, `edump`, `print`, and `eprint` don't output records which participate in `then`-chaining; rather, they're just immediate prints to stdout/stderr. The `printn` and `eprintn` keywords are the same except that they don't print final newlines. Additionally, you can print to a specified file instead of stdout/stderr.
* Use the **dump** or **edump** keywords, which *immediately print all out-of-stream variables as a JSON data structure to the standard output or standard error* (respectively).
* Use **tee** which formats the current stream record (not just an arbitrary string as with **print**) to a specific file.
* Use **emit1**/**emit**/**emitp**/**emitf** to send out-of-stream variables' current values to the output record stream, e.g. `@sum += $x; emit1 @sum` which produces an extra record such as `sum=3.1648382`. These records, just like records from input file(s), participate in downstream [then-chaining](reference-main-then-chaining.md) to other verbs.
For the first two options you are populating the output-records stream which feeds into the next verb in a `then`-chain (if any), or which otherwise is formatted for output using `--o...` flags.
For the last three options you are sending output directly to standard output, standard error, or a file.
@ -167,8 +167,8 @@ print: prints expression immediately to stdout.
dump: prints all currently defined out-of-stream variables immediately
to stdout as JSON.
With >, >>, or |, the data do not become part of the output record stream but
are instead redirected.
With >, >>, or |, the data do not go directly to stdout but are instead
redirected.
The > and >> are for write and append, as in the shell, but (as with awk) the
file-overwrite for > is on first write, not per record. The | is for piping to
@ -763,4 +763,3 @@ eks wye 0.381399 1
wye wye 0.204603 1
wye pan 0.573288 1
</pre>

View file

@ -131,7 +131,7 @@ the role of subroutine quite well.
If you have a file with UDFs you use frequently, say `my-udfs.mlr`, you can use
`--load` or `--mload` to define them for your Miller scripts. For example, in
your shell,
your shell,
<pre class="pre-highlight-non-pair">
<b>alias mlr='mlr --load ~/my-functions.mlr'</b>
@ -147,7 +147,7 @@ See the [miscellaneous-flags page](reference-main-flag-list.md#miscellaneous-fla
## Function literals
You can define unmnamed functions and assign the to variables, or pass them to functions.
You can define unnamed functions and assign them to variables, or pass them to functions.
See also the [page on higher-order functions](reference-dsl-higher-order-functions.md)
for more information on

View file

@ -300,7 +300,7 @@ num_total=5,num_positive=3
## Local variables
Local variables are similar to out-of-stream variables, except that their extent is limited to the expressions in which they appear (and their basenames can't be computed using square brackets). There are three kinds of local variables: **arguments** to functions/subroutines, **variables bound within for-loops**, and **locals** defined within control blocks. They may be untyped using `var`, or typed using `num`, `int`, `float`, `str`, `bool`, and `map`.
Local variables are similar to out-of-stream variables, except that their extent is limited to the expressions in which they appear (and their basenames can't be computed using square brackets). There are three kinds of local variables: **arguments** to functions/subroutines, **variables bound within for-loops**, and **locals** defined within control blocks. They may be untyped using `var`, or typed using `num`, `int`, `float`, `str`, `bool`, `arr`, and `map`.
For example:
@ -351,7 +351,7 @@ Things which are completely unsurprising, resembling many other languages:
Things which are perhaps surprising compared to other languages:
* Type declarations using `var`, or typed using `num`, `int`, `float`, `str`, and `bool` are not necessary to declare local variables. Function arguments and variables bound in for-loops over stream records and out-of-stream variables are *implicitly* declared using `var`. (Some examples are shown below.)
* Type declarations using `var`, or typed using `num`, `int`, `float`, `str`, `arr`, and `bool` are not necessary to declare local variables. Function arguments and variables bound in for-loops over stream records and out-of-stream variables are *implicitly* declared using `var`. (Some examples are shown below.)
* Type-checking is done at assignment time. For example, `float f = 0` is an error (since `0` is an integer), as is `float f = 0.0; f = 1`. For this reason I prefer to use `num` over `float` in most contexts since `num` encompasses integer and floating-point values. More information is at [Type-checking](reference-dsl-variables.md#type-checking).
@ -697,7 +697,7 @@ See [Data-cleaning Examples](data-cleaning-examples.md) for examples of how to u
### Type-declarations for local variables, function parameter, and function return values
Local variables can be defined either untyped as in `x = 1`, or typed as in `int x = 1`. Types include **var** (explicitly untyped), **int**, **float**, **num** (int or float), **str**, **bool**, and **map**. These optional type declarations are enforced at the time values are assigned to variables: whether at the initial value assignment as in `int x = 1` or in any subsequent assignments to the same variable farther down in the scope.
Local variables can be defined either untyped as in `x = 1`, or typed as in `int x = 1`. Types include **var** (explicitly untyped), **int**, **float**, **num** (int or float), **str**, **bool**, **arr**, and **map**. These optional type declarations are enforced at the time values are assigned to variables: whether at the initial value assignment as in `int x = 1` or in any subsequent assignments to the same variable farther down in the scope.
The reason for `num` is that `int` and `float` typedecls are very precise:
@ -936,8 +936,8 @@ in curly braces.
dump: prints all currently defined out-of-stream variables immediately
to stdout as JSON.
With >, >>, or |, the data do not become part of the output record stream but
are instead redirected.
With >, >>, or |, the data do not go directly to stdout but are instead
redirected.
The > and >> are for write and append, as in the shell, but (as with awk) the
file-overwrite for > is on first write, not per record. The | is for piping to
@ -1232,4 +1232,3 @@ ORS: evaluates to the output record separator from the command line,
or to LF or CRLF from the input data if in autodetect mode (which is
the default).
</pre>

View file

@ -98,7 +98,7 @@ allowed as they could be used for unexpected code execution. You can use
Note that this feature is quite general and is not limited to decompression
utilities. You can use it to apply per-file filters of your choice: e.g. `mlr
--prepipe head -n 10 ...`, if you like.
--prepipe 'head -n 10' ...`, if you like.
There is a `--prepipe` and a `--prepipex`:

View file

@ -580,7 +580,7 @@ alpha,5;2;8;6;1;4;9;10;3;7
</pre>
In the following example we sort data in several ways -- the first two just
recaptiulate (for reference) what `sort` with default flags already does; the third is novel:
recapitulate (for reference) what `sort` with default flags already does; the third is novel:
<pre class="pre-highlight-in-pair">
<b>mlr --icsv --ojson --from data/sortaf-example.csv put '</b>

View file

@ -64,7 +64,7 @@ much data as needed. For example, the [`sort`](reference-verbs.md#sort) and
before emitting any -- the last input record may well end up being the first
one to be emitted.
[`stats1`](reference-verbs.md#stats1) Other verbs, such as
Other verbs, such as
[`tail`](reference-verbs.md#tail) and [`top`](reference-verbs.md#top), need to
retain only a fixed number of records -- 10, perhaps, even if the input data
has a million records.

View file

@ -2201,7 +2201,7 @@ FUNCTIONS FOR FILTER/PUT
(class=collections #args=1) Returns array of keys of map or array
get_values
(class=collections #args=1) Returns array of keys of map or array -- in the latter case, returns a copy of the array
(class=collections #args=1) Returns array of values of map or array -- in the latter case, returns a copy of the array
gmt2localtime
(class=time #args=1,2) Convert from a GMT-time string to a local-time string. Consulting $TZ unless second argument is supplied.
@ -2804,8 +2804,8 @@ KEYWORDS FOR PUT AND FILTER
dump: prints all currently defined out-of-stream variables immediately
to stdout as JSON.
With >, >>, or |, the data do not become part of the output record stream but
are instead redirected.
With >, >>, or |, the data do not go directly to stdout but are instead
redirected.
The > and >> are for write and append, as in the shell, but (as with awk) the
file-overwrite for > is on first write, not per record. The | is for piping to
@ -3160,4 +3160,4 @@ SEE ALSO
2022-02-15 MILLER(1)
2022-02-18 MILLER(1)

View file

@ -2,12 +2,12 @@
.\" Title: mlr
.\" Author: [see the "AUTHOR" section]
.\" Generator: ./mkman.rb
.\" Date: 2022-02-15
.\" Date: 2022-02-18
.\" Manual: \ \&
.\" Source: \ \&
.\" Language: English
.\"
.TH "MILLER" "1" "2022-02-15" "\ \&" "\ \&"
.TH "MILLER" "1" "2022-02-18" "\ \&" "\ \&"
.\" -----------------------------------------------------------------
.\" * Portability definitions
.\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@ -3086,7 +3086,7 @@ format("{}:{}:{}", 1,2,3,4) gives "1:2:3".
.RS 0
.\}
.nf
(class=collections #args=1) Returns array of keys of map or array -- in the latter case, returns a copy of the array
(class=collections #args=1) Returns array of values of map or array -- in the latter case, returns a copy of the array
.fi
.if n \{\
.RE
@ -4645,8 +4645,8 @@ in curly braces.
dump: prints all currently defined out-of-stream variables immediately
to stdout as JSON.
With >, >>, or |, the data do not become part of the output record stream but
are instead redirected.
With >, >>, or |, the data do not go directly to stdout but are instead
redirected.
The > and >> are for write and append, as in the shell, but (as with awk) the
file-overwrite for > is on first write, not per record. The | is for piping to