mirror of
https://github.com/johnkerl/miller.git
synced 2026-07-17 16:38:54 +00:00
1a20b32bb5
By default, records with an empty-string value in a join field are paired just like any other value, so two records both missing an ID get matched with each other -- rarely the intended behavior. --ignore-empty treats an empty-string join-field value as if the field were absent, on both the left and right files, so such records fall through to unpaired handling (--np/--ul/--ur) instead of cross-joining on the empty string. Wires the check through both the default half-streaming join and the -s/--sorted-input doubly-streaming join, which track left-file buckets independently and needed the same empty-aware key-presence check in JoinBucketKeeper. Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
1653 lines
78 KiB
Text
1653 lines
78 KiB
Text
================================================================
|
|
altkv
|
|
Usage: mlr altkv [options]
|
|
Given fields with values of the form a,b,c,d,e,f emits a=b,c=d,e=f pairs.
|
|
Options:
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
bar
|
|
Usage: mlr bar [options]
|
|
Replaces a numeric field with a number of asterisks, allowing for cheesy
|
|
bar plots. These align best with --opprint or --oxtab output format.
|
|
Options:
|
|
-f {a,b,c} Field names to convert to bars.
|
|
--lo {lo} Lower-limit value for min-width bar: default '0.000000'.
|
|
--hi {hi} Upper-limit value for max-width bar: default '100.000000'.
|
|
-w {n} Bar-field width: default '40'.
|
|
--auto Automatically computes limits, ignoring --lo and --hi. Holds all
|
|
records in memory before producing any output.
|
|
-c {character} Fill character: default '*'.
|
|
-x {character} Out-of-bounds character: default '#'.
|
|
-b {character} Blank character: default '.'.
|
|
-h|--help Show this message.
|
|
Nominally the fill, out-of-bounds, and blank characters will be strings of length 1.
|
|
However you can make them all longer if you so desire.
|
|
|
|
================================================================
|
|
bootstrap
|
|
Usage: mlr bootstrap [options]
|
|
Emits an n-sample, with replacement, of the input records.
|
|
See also mlr sample and mlr shuffle.
|
|
Options:
|
|
-n {n} Number of samples to output. Defaults to number of input records. Must
|
|
be non-negative.
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
bootstrap-ci
|
|
Usage: mlr bootstrap-ci [options]
|
|
Computes bootstrap confidence intervals for statistics of given fields,
|
|
accumulated across the input record stream: values are resampled with
|
|
replacement many times, the statistic is computed on each resample, and the
|
|
confidence interval is taken from percentiles of the resampled statistics.
|
|
For each value field and statistic, outputs the full-data statistic in
|
|
{field}_{stat}, along with confidence-interval endpoints in {field}_{stat}_lo
|
|
and {field}_{stat}_hi. Use mlr --seed for reproducible results.
|
|
See also mlr bootstrap and mlr stats1.
|
|
Options:
|
|
-a {mean,...} Names of statistics to bootstrap: one or more of the listed
|
|
values, as in mlr stats1 -a. Also accepts median (same as p50) and
|
|
percentiles p{n} for n in 0..100. Defaults to mean.
|
|
-f {a,b,c} Value-field names on which to compute statistics. Required.
|
|
-g {d,e,f} Optional group-by-field names.
|
|
-n {n} Number of bootstrap resamples. Must be positive. Defaults to 1000.
|
|
-c {level} Confidence level, strictly between 0 and 1. Defaults to 0.95.
|
|
-i Use interpolated percentiles, like R's type=7, for percentile
|
|
statistics as well as for the confidence-interval endpoints;
|
|
default like type=1.
|
|
-h|--help Show this message.
|
|
Example: mlr --seed 12345 bootstrap-ci -f x,y
|
|
Example: mlr --seed 12345 bootstrap-ci -a mean,median -f x -g shape -n 5000 -c 0.99
|
|
|
|
================================================================
|
|
case
|
|
Usage: mlr case [options]
|
|
Uppercases strings in record keys and/or values.
|
|
Options:
|
|
-k Case only keys, not keys and values.
|
|
-v Case only values, not keys and values.
|
|
-f {a,b,c} Specify which field names to case (default: all).
|
|
-u Convert to uppercase.
|
|
-l Convert to lowercase.
|
|
-s Convert to sentence case (capitalize first letter).
|
|
-t Convert to title case (capitalize words).
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
cat
|
|
Usage: mlr cat [options]
|
|
Passes input records directly to output. Most useful for format conversion.
|
|
Options:
|
|
-n Prepend field "n" to each record with record-counter starting at 1.
|
|
-N {name} Prepend field {name} to each record with record-counter starting at
|
|
1.
|
|
-g {a,b,c} Optional group-by-field names for counters, e.g. a,b,c.
|
|
--filename Prepend current filename to each record.
|
|
--filenum Prepend current filenum (1-up) to each record.
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
check
|
|
Usage: mlr check [options]
|
|
Consumes records without printing any output,
|
|
with the exception that warnings are printed to stderr.
|
|
Useful for doing a well-formatted check on input data.
|
|
Current checks are:
|
|
* Data are parseable
|
|
* If any key is the empty string
|
|
Options:
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
clean-whitespace
|
|
Usage: mlr clean-whitespace [options]
|
|
For each record, for each field in the record, whitespace-cleans the keys and/or
|
|
values. Whitespace-cleaning entails stripping leading and trailing whitespace,
|
|
and replacing multiple whitespace with singles. For finer-grained control,
|
|
please see the DSL functions lstrip, rstrip, strip, collapse_whitespace,
|
|
and clean_whitespace.
|
|
|
|
Options:
|
|
-k|--keys-only Do not touch values.
|
|
-v|--values-only Do not touch keys.
|
|
-h|--help Show this message.
|
|
It is an error to specify -k as well as -v -- to clean keys and values,
|
|
leave off -k as well as -v.
|
|
|
|
================================================================
|
|
count-distinct
|
|
Usage: mlr count-distinct [options]
|
|
Prints number of records having distinct values for specified field names.
|
|
Same as uniq -c.
|
|
|
|
Options:
|
|
-f {a,b,c} Field names for distinct count (synonym for -g).
|
|
-g {a,b,c} Field names for distinct count.
|
|
-x {a,b,c} Field names to exclude for distinct count; use each record's other
|
|
fields instead.
|
|
-n Show only the number of distinct values. Not compatible with -u.
|
|
-o {name} Field name for output count. Default "count". Ignored with -u.
|
|
-u Do unlashed counts for multiple field names. With -f a,b and without
|
|
-u, computes counts for distinct combinations of a and b field
|
|
values. With -f a,b and with -u, computes counts for distinct a field
|
|
values and counts for distinct b field values separately.
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
count
|
|
Usage: mlr count [options]
|
|
Prints number of records, optionally grouped by distinct values for specified field names.
|
|
Options:
|
|
-g {a,b,c} Optional group-by-field names for counts, e.g. a,b,c
|
|
-n Show only the number of distinct values. Not interesting without -g.
|
|
-o {name} Field name for output-count. Default "count".
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
count-similar
|
|
Usage: mlr count-similar [options]
|
|
Ingests all records, then emits each record augmented by a count of
|
|
the number of other records having the same group-by field values.
|
|
Options:
|
|
-g {a,b,c} Group-by-field names for counts, e.g. a,b,c
|
|
-o {name} Field name for output-counts. Defaults to "count".
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
cut
|
|
Usage: mlr cut [options]
|
|
Passes through input records with specified fields included/excluded.
|
|
Options:
|
|
-f {a,b,c} Comma-separated field names to include or exclude, e.g. a,b,c.
|
|
-o Retain fields in the order specified by -f rather than in
|
|
input-record order.
|
|
-x|--complement Exclude, rather than include, the field names specified by -f.
|
|
-r Treat field names as regular expressions. "ab", "a.*b" will
|
|
match any field name containing the substring "ab" or matching
|
|
"a.*b", respectively; anchors of the form "^ab$", "^a.*b$" may
|
|
be used.
|
|
-h|--help Show this message.
|
|
Examples:
|
|
mlr cut -f hostname,status
|
|
mlr cut -x -f hostname,status
|
|
mlr cut -r -f '^status$,sda[0-9]'
|
|
mlr cut -r -f '^status$,"sda[0-9]"'
|
|
mlr cut -r -f '^status$,"sda[0-9]"i' (this is case-insensitive)
|
|
|
|
================================================================
|
|
decimate
|
|
Usage: mlr decimate [options]
|
|
Passes through one of every n records, optionally by category.
|
|
Options:
|
|
-b Decimate by printing first of every n.
|
|
-e Decimate by printing last of every n (default).
|
|
-g {a,b,c} Optional group-by-field names for decimate counts, e.g. a,b,c.
|
|
-n {n} Decimation factor (default 10).
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
describe
|
|
Usage: mlr describe [options]
|
|
Shows a compact schema for the input data: field names, types, and value shape.
|
|
Emits one output record per input field, with types seen, counts, cardinality,
|
|
min/max, and (for low-cardinality fields) the complete set of distinct values.
|
|
|
|
Output fields, one record per input field:
|
|
field_name name of the input field
|
|
types map from type name (int, string, etc.) to occurrence count
|
|
count number of records in which the field appears
|
|
null_count count of field values either empty string or JSON null
|
|
distinct_count count of distinct values for the field
|
|
min, max minimum/maximum field value (works for strings as well as numbers)
|
|
values all distinct values, in order first seen -- only for fields
|
|
whose distinct_count is within the -n limit
|
|
|
|
Notes:
|
|
* Distinctness is computed on string representations -- so 4.1 and 4.10 are counted as distinct here.
|
|
* Use `mlr --ojson describe` for a machine-readable JSON document; in tabular
|
|
output formats the types map and values array are flattened.
|
|
* See also the summary verb, which reports summary statistics (mean, percentiles, etc.).
|
|
|
|
Options:
|
|
-n|--max-values {n} List a field's distinct values only if it has at most {n} of
|
|
them; 0 suppresses the values array entirely. Defaults to
|
|
20.
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
fill-down
|
|
Usage: mlr fill-down [options]
|
|
If a given record has a missing value for a given field, fill that from
|
|
the corresponding value from a previous record, if any.
|
|
By default, a 'missing' field either is absent, or has the empty-string value.
|
|
With -a, a field is 'missing' only if it is absent.
|
|
|
|
Options:
|
|
--all Operate on all fields in the input.
|
|
-a|--only-if-absent If a given record has a missing value for a given field,
|
|
fill that from the corresponding value from a previous
|
|
record, if any. By default, a 'missing' field either is
|
|
absent, or has the empty-string value. With -a, a field is
|
|
'missing' only if it is absent.
|
|
-f {a,b,c} Field names for fill-down.
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
fill-empty
|
|
Usage: mlr fill-empty [options]
|
|
Fills empty-string fields with specified fill-value.
|
|
Options:
|
|
-v {string} Fill-value: defaults to "N/A"
|
|
-S Don't infer type -- so '-v 0' would fill string 0 not int 0.
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
filter
|
|
Usage: mlr filter [options] {DSL expression}
|
|
Lets you use a domain-specific language to programmatically filter which
|
|
stream records will be output.
|
|
See also: https://miller.readthedocs.io/en/latest/reference-verbs
|
|
|
|
Options:
|
|
-f {file name} File containing a DSL expression (see examples below). If the
|
|
filename is a directory, all *.mlr files in that directory are
|
|
loaded.
|
|
-e {expression} DSL expression to evaluate. You can use this after -f to add an
|
|
expression. Example use case: define functions/subroutines in a
|
|
file you specify with -f, then call them with an expression you
|
|
specify with -e.
|
|
-s {name=value} Predefines out-of-stream variable @name to have the given value.
|
|
Thus mlr put -s foo=97 '$column += @foo' is like mlr put 'begin
|
|
{@foo = 97} $column += @foo'. The value part is subject to
|
|
type-inferencing. May be specified more than once, e.g. -s
|
|
name1=value1 -s name2=value2. Note: the value may be an
|
|
environment variable, e.g. -s sequence=$SEQUENCE.
|
|
-x Prints records for which {expression} evaluates to false, not
|
|
true, i.e. invert the sense of the filter expression. Default
|
|
false.
|
|
-q Does not include the modified record in the output stream.
|
|
Useful for when all desired output is in begin and/or end
|
|
blocks.
|
|
-S No-op in Miller 6 and above, since type-inferencing is now done
|
|
by the record-readers before filter/put is executed. Supported
|
|
as a no-op pass-through flag for backward compatibility.
|
|
-F No-op in Miller 6 and above, since type-inferencing is now done
|
|
by the record-readers before filter/put is executed. Supported
|
|
as a no-op pass-through flag for backward compatibility.
|
|
-w Print warnings about things like uninitialized variables.
|
|
-W Same as -w, but exit the process if there are any warnings.
|
|
-p Prints the expression's AST (abstract syntax tree), which gives
|
|
full transparency on the precedence and associativity rules of
|
|
Miller's grammar, to stdout.
|
|
-d Like -p but uses a parenthesized-expression format for the AST.
|
|
-D Like -d but with output all on one line.
|
|
-E Echo DSL expression before printing parse-tree.
|
|
-v Same as -E -p.
|
|
-X Exit after parsing but before stream-processing. Useful with
|
|
-v/-d/-D, if you only want to look at parser information.
|
|
--explain Parse and type-check the DSL expression, report whether it is
|
|
valid, and exit without reading the input stream. Exit status is
|
|
0 if the expression is valid and non-zero otherwise; combine
|
|
with --errors-json for a machine-readable error.
|
|
-h|--help Show this message.
|
|
|
|
If you mix -e and -f then the expressions are evaluated in the order encountered.
|
|
Since the expression pieces are simply concatenated, please be sure to use intervening
|
|
semicolons to separate expressions.
|
|
|
|
Parser-info options are -w, -W, -p, -d, -D, -E, -v, and -X.
|
|
|
|
Records will pass the filter depending on the last bare-boolean statement in
|
|
the DSL expression. That can be the result of <, ==, >, etc., the return value of a function call
|
|
which returns boolean, etc.
|
|
|
|
Examples:
|
|
mlr --csv --from example.csv filter '$color == "red"'
|
|
mlr --csv --from example.csv filter '$color == "red" && flag == true'
|
|
More example filter expressions:
|
|
First record in each file:
|
|
'FNR == 1'
|
|
Subsampling:
|
|
'urand() < 0.001'
|
|
Compound booleans:
|
|
'$color != "blue" && $value > 4.2'
|
|
'($x < 0.5 && $y < 0.5) || ($x > 0.5 && $y > 0.5)'
|
|
Regexes with case-insensitive flag
|
|
'($name =~ "^sys.*east$") || ($name =~ "^dev.[0-9]+"i)'
|
|
Assignments, then bare-boolean filter statement:
|
|
'$ab = $a+$b; $cd = $c+$d; $ab != $cd'
|
|
Bare-boolean filter statement within a conditional:
|
|
'if (NR < 100) {
|
|
$x > 0.3;
|
|
} else {
|
|
$x > 0.002;
|
|
}
|
|
'
|
|
Using 'any' higher-order function to see if $index is 10, 20, or 30:
|
|
'any([10,20,30], func(e) {return $index == e})'
|
|
|
|
See also https://miller.readthedocs.io/reference-dsl for more context.
|
|
|
|
================================================================
|
|
flatten
|
|
Usage: mlr flatten [options]
|
|
Flattens multi-level maps to single-level ones. Example: field with name 'a'
|
|
and value '{"b": { "c": 4 }}' becomes name 'a.b.c' and value 4.
|
|
Options:
|
|
-f Comma-separated list of field names to flatten (default all).
|
|
-s {string} Separator, defaulting to mlr --flatsep value.
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
format-values
|
|
Usage: mlr format-values [options]
|
|
Applies format strings to all field values, depending on autodetected type.
|
|
* If a field value is detected to be integer, applies integer format.
|
|
* Else, if a field value is detected to be float, applies float format.
|
|
* Else, applies string format.
|
|
|
|
Note: this is a low-keystroke way to apply formatting to many fields. To get
|
|
finer control, please see the fmtnum function within the mlr put DSL.
|
|
|
|
Note: this verb lets you apply arbitrary format strings, which can produce
|
|
undefined behavior and/or program crashes. See your system's "man printf".
|
|
|
|
Options:
|
|
-i {integer format} Integer format string; defaults to "%d". Examples: "%06lld",
|
|
"%08llx". Note that Miller integers are long long so you
|
|
must use formats which apply to long long, e.g. with ll in
|
|
them. Undefined behavior results otherwise.
|
|
-f {float format} Float format string; defaults to "%f". Examples: "%8.3lf",
|
|
"%.6le". Note that Miller floats are double-precision so you
|
|
must use formats which apply to double, e.g. with l[efg] in
|
|
them. Undefined behavior results otherwise.
|
|
-s {string format} String format string; defaults to "%s". Examples: "_%s",
|
|
"%08s". Note that you must use formats which apply to
|
|
string, e.g. with s in them. Undefined behavior results
|
|
otherwise.
|
|
-n Coerce field values autodetected as int to float, and then
|
|
apply the float format.
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
fraction
|
|
Usage: mlr fraction [options]
|
|
For each record's value in specified fields, computes the ratio of that
|
|
value to the sum of values in that field over all input records.
|
|
E.g. with input records x=1 x=2 x=3 and x=4, emits output records
|
|
x=1,x_fraction=0.1 x=2,x_fraction=0.2 x=3,x_fraction=0.3 and x=4,x_fraction=0.4
|
|
|
|
Note: this is internally a two-pass algorithm: on the first pass it retains
|
|
input records and accumulates sums; on the second pass it computes quotients
|
|
and emits output records. This means it produces no output until all input is read.
|
|
|
|
Options:
|
|
-f {a,b,c} Field name(s) for fraction calculation
|
|
-g {d,e,f} Optional group-by-field name(s) for fraction counts
|
|
-p Produce percents [0..100], not fractions [0..1]. Output field names
|
|
end with "_percent" rather than "_fraction"
|
|
-c Produce cumulative distributions, i.e. running sums: each output
|
|
value folds in the sum of the previous for the specified group. E.g.
|
|
with input records x=1 x=2 x=3 and x=4, emits output records
|
|
x=1,x_cumulative_fraction=0.1 x=2,x_cumulative_fraction=0.3
|
|
x=3,x_cumulative_fraction=0.6 and x=4,x_cumulative_fraction=1.0
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
gap
|
|
Usage: mlr gap [options]
|
|
Emits an empty record every n records, or when certain values change.
|
|
One of -n or -g is required.
|
|
Options:
|
|
-g {a,b,c} Print a gap whenever values of these fields (e.g. a,b,c) changes.
|
|
-n {n} Print a gap every n records. Ignored if -g is present.
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
grep
|
|
Usage: mlr grep [options] {regular expression}
|
|
Passes through records which match the regular expression.
|
|
Options:
|
|
-i Use case-insensitive search.
|
|
-v Invert: pass through records which do not match the regex.
|
|
-a Only grep for values, not keys and values.
|
|
-h|--help Show this message.
|
|
Note that "mlr filter" is more powerful, but requires you to know field names.
|
|
By contrast, "mlr grep" allows you to regex-match the entire record. It does this
|
|
by formatting each record in memory as DKVP (or NIDX, if -a is supplied), using
|
|
OFS "," and OPS "=", and matching the resulting line against the regex specified
|
|
here. In particular, the regex is not applied to the input stream: if you have
|
|
CSV with header line "x,y,z" and data line "1,2,3" then the regex will be
|
|
matched, not against either of these lines, but against the DKVP line
|
|
"x=1,y=2,z=3". Furthermore, not all the options to system grep are supported,
|
|
and this command is intended to be merely a keystroke-saver. To get all the
|
|
features of system grep, you can do
|
|
"mlr --odkvp ... | grep ... | mlr --idkvp ..."
|
|
|
|
================================================================
|
|
group-by
|
|
Usage: mlr group-by [options] {comma-separated field names}
|
|
Outputs records in batches having identical values at specified field names.
|
|
Options:
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
group-like
|
|
Usage: mlr group-like [options]
|
|
Outputs records in batches having identical field names.
|
|
Options:
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
gsub
|
|
Usage: mlr gsub [options]
|
|
Replaces old string with new string in specified field(s), with regex support
|
|
for the old string and handling multiple matches, like the `gsub` DSL function.
|
|
The replacement string supports C-style backslash escapes such as \n, \t,
|
|
and \x1f. Write \\ to get a literal backslash.
|
|
See also the `sub` and `ssub` verbs.
|
|
Options:
|
|
-f {a,b,c} Field names to apply substitution to.
|
|
-r {regex} Regular expression for field names to apply substitution to.
|
|
-a Apply substitution to all fields.
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
having-fields
|
|
Usage: mlr having-fields [options]
|
|
Conditionally passes through records depending on each record's field names.
|
|
Options:
|
|
--at-least {comma-separated names} Pass records that have at least these field
|
|
names.
|
|
--which-are {comma-separated names} Pass records whose field names are exactly
|
|
these.
|
|
--at-most {comma-separated names} Pass records that have at most these field
|
|
names.
|
|
--all-matching {regular expression} Pass records where all field names match
|
|
the regex.
|
|
--any-matching {regular expression} Pass records where any field name matches
|
|
the regex.
|
|
--none-matching {regular expression} Pass records where no field name matches
|
|
the regex.
|
|
-h|--help Show this message.
|
|
Examples:
|
|
mlr having-fields --which-are amount,status,owner
|
|
mlr having-fields --any-matching 'sda[0-9]'
|
|
mlr having-fields --any-matching '"sda[0-9]"'
|
|
mlr having-fields --any-matching '"sda[0-9]"i' (this is case-insensitive)
|
|
|
|
================================================================
|
|
head
|
|
Usage: mlr head [options]
|
|
Passes through the first n records, optionally by category.
|
|
Without -g, ceases consuming more input (i.e. is fast) when n records have been read.
|
|
Options:
|
|
-g {a,b,c} Optional group-by-field names for head counts, e.g. a,b,c.
|
|
-n {n} Head-count to print. Default 10. A negative count, e.g. -n -2, passes
|
|
through all but the last n records, optionally by category.
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
histogram
|
|
Just a histogram. Input values < lo or > hi are not counted.
|
|
Usage: mlr histogram [options]
|
|
Options:
|
|
-f {a,b,c} Value-field names for histogram counts.
|
|
--lo {lo} Histogram low value.
|
|
--hi {hi} Histogram high value.
|
|
--nbins {n} Number of histogram bins. Defaults to 20.
|
|
--auto Automatically computes limits, ignoring --lo and --hi. Holds all
|
|
values in memory before producing any output.
|
|
-o {prefix} Prefix for output field name. Default: no prefix.
|
|
-s Print a one-line Unicode sparkline per field instead of per-bin
|
|
counts.
|
|
-h|--help Show this message.
|
|
With -s, output is one record per value-field, with a sparkline field
|
|
instead of one record per bin.
|
|
|
|
================================================================
|
|
json-parse
|
|
Usage: mlr json-parse [options]
|
|
Tries to convert string field values to parsed JSON, e.g. "[1,2,3]" -> [1,2,3].
|
|
Options:
|
|
-f {a,b,c} Comma-separated list of field names to json-parse (default all).
|
|
-k If supplied, then on parse fail for any cell, keep the (unparsable)
|
|
input value for the cell.
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
json-stringify
|
|
Usage: mlr json-stringify [options]
|
|
Produces string field values from field-value data, e.g. [1,2,3] -> "[1,2,3]".
|
|
Options:
|
|
-f {a,b,c} Comma-separated list of field names to json-stringify (default
|
|
all).
|
|
--jvstack Produce multi-line JSON output.
|
|
--no-jvstack Produce single-line JSON output per record (default).
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
join
|
|
Usage: mlr join [options]
|
|
Joins records from specified left file name with records from all file names
|
|
at the end of the Miller argument list.
|
|
Functionality is essentially the same as the system "join" command, but for
|
|
record streams.
|
|
Options:
|
|
-f {left file name} Left file name for join.
|
|
-j {a,b,c} Comma-separated join-field names for
|
|
output.
|
|
-l {a,b,c} Comma-separated join-field names for left
|
|
input file; defaults to -j values if
|
|
omitted.
|
|
-r {a,b,c} Comma-separated join-field names for right
|
|
input file(s); defaults to -j values if
|
|
omitted.
|
|
--lk|--left-keep-field-names {a,b,c} If supplied, this means keep only the
|
|
specified field names from the left file.
|
|
Automatically includes the join-field
|
|
name(s). Helpful for when you only want a
|
|
limited subset of information from the left
|
|
file. Tip: you can use --lk "": this means
|
|
the left file becomes solely a row-selector
|
|
for the input files.
|
|
--lp {text} Additional prefix for non-join output field
|
|
names from the left file. Applies to paired
|
|
and unpaired output records.
|
|
--rp {text} Additional prefix for non-join output field
|
|
names from the right file(s). Applies to
|
|
paired and unpaired output records.
|
|
--np Do not emit paired records.
|
|
--ul Emit unpaired records from the left file.
|
|
--ur Emit unpaired records from the right
|
|
file(s).
|
|
--ignore-empty Treat records with empty-string values in
|
|
any join-field as if that join-field were
|
|
absent, on both the left and right files.
|
|
Such records are never paired -- not even
|
|
with one another -- and are treated as
|
|
unpaired, subject to --np/--ul/--ur as
|
|
usual.
|
|
-s|--sorted-input Require sorted input: records must be
|
|
sorted lexically by their join-field names,
|
|
else not all records will be paired. The
|
|
only likely use case for this is with a
|
|
left file which is too big to fit into
|
|
system memory otherwise.
|
|
-u Enable unsorted input. (This is the default
|
|
even without -u.) In this case, the entire
|
|
left file will be loaded into memory.
|
|
--prepipe {command} Shell command to prepipe the left-file
|
|
input through. As in main input options;
|
|
see mlr --help for details. If you wish to
|
|
use a prepipe command for the main input as
|
|
well as here, it must be specified there as
|
|
well as here.
|
|
--prepipex {command} Shell command to prepipe the left-file
|
|
input through (no shell quoting). As in
|
|
main input options; see mlr --help for
|
|
details.
|
|
-h|--help Show this message.
|
|
File-format options default to those for the right file names on the Miller
|
|
argument list, but may be overridden for the left file as follows. Please see
|
|
the main "mlr --help" for more information on syntax for these arguments:
|
|
-i {format name} for the left-file format, e.g. 'csv' or 'json'. Or, flags
|
|
like --icsv, --ijson, etc.: '--icsv' is the same as '-i csv', and so on.
|
|
--irs {record-separator character}
|
|
--ifs {field-separator character}
|
|
--ips {pair-separator character}
|
|
--repifs
|
|
--implicit-csv-header
|
|
--implicit-tsv-header
|
|
--no-implicit-csv-header
|
|
--no-implicit-tsv-header
|
|
For example, if you have 'mlr --csv ... join -l foo ... ' then the left-file format will
|
|
be specified CSV as well unless you override with 'mlr --csv ... join --ijson -l foo' etc.
|
|
Likewise, if you have 'mlr --csv --implicit-csv-header ...' then the join-in file will be
|
|
expected to be headerless as well unless you put '--no-implicit-csv-header' after 'join'.
|
|
Please use "mlr --usage-separator-options" for information on specifying separators.
|
|
Please see https://miller.readthedocs.io/en/latest/reference-verbs#join for more information
|
|
including examples.
|
|
|
|
================================================================
|
|
label
|
|
Usage: mlr label [options] {new1,new2,new3,...}
|
|
Given n comma-separated names, renames the first n fields of each record to
|
|
have the respective name. (Fields past the nth are left with their original
|
|
names.) Particularly useful with --inidx or --implicit-csv-header, to give
|
|
useful names to otherwise integer-indexed fields.
|
|
|
|
Options:
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
latin1-to-utf8
|
|
Usage: mlr latin1-to-utf8, with no options.
|
|
Recursively converts record strings from Latin-1 to UTF-8.
|
|
For field-level control, please see the latin1_to_utf8 DSL function.
|
|
Options:
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
least-frequent
|
|
Usage: mlr least-frequent [options]
|
|
Shows the least frequently occurring distinct values for specified field names.
|
|
The first entry is the statistical anti-mode; the remaining are runners-up.
|
|
Options:
|
|
-f {a,b,c} One or more comma-separated field names to group by. Required flag.
|
|
-n {count} Maximum number of results to output. Optional flag defaulting to 10.
|
|
-b Suppress counts; show only field values.
|
|
-o {name} Field name for output count. Default "count".
|
|
-h|--help Show this message.
|
|
See also "mlr most-frequent".
|
|
|
|
================================================================
|
|
merge-fields
|
|
Usage: mlr merge-fields [options]
|
|
Computes univariate statistics for each input record, accumulated across
|
|
specified fields.
|
|
Options:
|
|
-a {sum,count,...} Names of accumulators: one or more of the accumulators listed
|
|
below.
|
|
-f {a,b,c} Value-field names on which to compute statistics. Requires
|
|
-o.
|
|
-r {a,b,c} Regular expressions for value-field names on which to compute
|
|
statistics. Requires -o.
|
|
-c {a,b,c} Substrings for collapse mode: all fields which have the same
|
|
names after removing substrings will be accumulated together.
|
|
Please see examples below.
|
|
-i Use interpolated percentiles, like R's type=7; default like
|
|
type=1. Not sensical for string-valued fields.
|
|
-o {name} Output field basename for -f/-r.
|
|
-k Keep the input fields which contributed to the output
|
|
statistics; the default is to omit them.
|
|
-S No-op flag for backward compatibility with Miller 5.
|
|
-F No-op flag for backward compatibility with Miller 5.
|
|
-h|--help Show this message.
|
|
|
|
Accumulators for -a:
|
|
count Count instances of fields
|
|
null_count Count number of empty-string/JSON-null instances per field
|
|
distinct_count Count number of distinct values per field
|
|
mode Find most-frequently-occurring values for fields; first-found wins tie
|
|
antimode Find least-frequently-occurring values for fields; first-found wins tie
|
|
sum Compute sums of specified fields
|
|
mean Compute averages (sample means) of specified fields
|
|
mad Compute mean absolute deviation
|
|
var Compute sample variance of specified fields
|
|
stddev Compute sample standard deviation of specified fields
|
|
meaneb Estimate error bars for averages (assuming no sample autocorrelation)
|
|
skewness Compute sample skewness of specified fields
|
|
kurtosis Compute sample kurtosis of specified fields
|
|
min Compute minimum values of specified fields
|
|
max Compute maximum values of specified fields
|
|
minlen Compute minimum string-lengths of specified fields
|
|
maxlen Compute maximum string-lengths of specified fields
|
|
|
|
String-valued data make sense unless arithmetic on them is required,
|
|
e.g. for sum, mean, interpolated percentiles, etc. In case of mixed data,
|
|
numbers are less than strings.
|
|
|
|
Example input data: "a_in_x=1,a_out_x=2,b_in_y=4,b_out_x=8".
|
|
Example: mlr merge-fields -a sum,count -f a_in_x,a_out_x -o foo
|
|
produces "b_in_y=4,b_out_x=8,foo_sum=3,foo_count=2" since "a_in_x,a_out_x" are
|
|
summed over.
|
|
Example: mlr merge-fields -a sum,count -r in_,out_ -o bar
|
|
produces "bar_sum=15,bar_count=4" since all four fields are summed over.
|
|
Example: mlr merge-fields -a sum,count -c in_,out_
|
|
produces "a_x_sum=3,a_x_count=2,b_y_sum=4,b_y_count=1,b_x_sum=8,b_x_count=1"
|
|
since "a_in_x" and "a_out_x" both collapse to "a_x", "b_in_y" collapses to
|
|
"b_y", and "b_out_x" collapses to "b_x".
|
|
|
|
================================================================
|
|
most-frequent
|
|
Usage: mlr most-frequent [options]
|
|
Shows the most frequently occurring distinct values for specified field names.
|
|
The first entry is the statistical mode; the remaining are runners-up.
|
|
Options:
|
|
-f {a,b,c} One or more comma-separated field names to group by. Required flag.
|
|
-n {count} Maximum number of results to output. Optional flag defaulting to 10.
|
|
-b Suppress counts; show only field values.
|
|
-o {name} Field name for output count. Default "count".
|
|
-h|--help Show this message.
|
|
See also "mlr least-frequent".
|
|
|
|
================================================================
|
|
nest
|
|
Usage: mlr nest [options]
|
|
Explodes specified field values into separate fields/records, or reverses this.
|
|
Options:
|
|
--explode Explode field values into separate fields/records. One of
|
|
--explode or --implode is required.
|
|
--implode Reverse of --explode. One of --explode or --implode is
|
|
required.
|
|
--values Operate on field values. One of --values or --pairs is
|
|
required.
|
|
--pairs Operate on field key-value pairs. One of --values or
|
|
--pairs is required.
|
|
--across-records Explode/implode across records. One of --across-records or
|
|
--across-fields is required.
|
|
--across-fields Explode/implode across fields. One of --across-records or
|
|
--across-fields is required.
|
|
-f {field name} Required field name to operate on.
|
|
-r {field names} Like -f but treat arguments as a regular expression. Match
|
|
all field names and operate on each in record order.
|
|
Example: -r '^[xy]$'.
|
|
--nested-fs {string} Field separator for nested values. Defaults to ";".
|
|
--nested-ps {string} Pair separator for nested key-value pairs. Defaults to ":".
|
|
--evar {string} Shorthand for --explode --values --across-records
|
|
--nested-fs {string}.
|
|
--ivar {string} Shorthand for --implode --values --across-records
|
|
--nested-fs {string}.
|
|
-h|--help Show this message.
|
|
Please use "mlr --usage-separator-options" for information on specifying separators.
|
|
|
|
Examples:
|
|
|
|
mlr nest --explode --values --across-records -f x
|
|
with input record "x=a;b;c,y=d" produces output records
|
|
"x=a,y=d"
|
|
"x=b,y=d"
|
|
"x=c,y=d"
|
|
Use --implode to do the reverse.
|
|
|
|
mlr nest --explode --values --across-fields -f x
|
|
with input record "x=a;b;c,y=d" produces output records
|
|
"x_1=a,x_2=b,x_3=c,y=d"
|
|
Use --implode to do the reverse.
|
|
|
|
mlr nest --explode --pairs --across-records -f x
|
|
with input record "x=a:1;b:2;c:3,y=d" produces output records
|
|
"a=1,y=d"
|
|
"b=2,y=d"
|
|
"c=3,y=d"
|
|
|
|
mlr nest --explode --pairs --across-fields -f x
|
|
with input record "x=a:1;b:2;c:3,y=d" produces output records
|
|
"a=1,b=2,c=3,y=d"
|
|
|
|
Notes:
|
|
* With --pairs, --implode doesn't make sense since the original field name has
|
|
been lost.
|
|
* The combination "--implode --values --across-records" is non-streaming:
|
|
no output records are produced until all input records have been read. In
|
|
particular, this means it won't work in `tail -f` contexts. But all other flag
|
|
combinations result in streaming (`tail -f` friendly) data processing.
|
|
If input is coming from `tail -f`, be sure to use `--records-per-batch 1`.
|
|
* It's up to you to ensure that the nested-fs is distinct from your data's IFS:
|
|
e.g. by default the former is semicolon and the latter is comma.
|
|
See also mlr reshape.
|
|
|
|
================================================================
|
|
nothing
|
|
Usage: mlr nothing [options]
|
|
Drops all input records. Useful for testing, or after tee/print/etc. have
|
|
produced other output.
|
|
Options:
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
put
|
|
Usage: mlr put [options] {DSL expression}
|
|
Lets you use a domain-specific language to programmatically alter stream records.
|
|
See also: https://miller.readthedocs.io/en/latest/reference-verbs
|
|
|
|
Options:
|
|
-f {file name} File containing a DSL expression (see examples below). If the
|
|
filename is a directory, all *.mlr files in that directory are
|
|
loaded.
|
|
-e {expression} DSL expression to evaluate. You can use this after -f to add an
|
|
expression. Example use case: define functions/subroutines in a
|
|
file you specify with -f, then call them with an expression you
|
|
specify with -e.
|
|
-s {name=value} Predefines out-of-stream variable @name to have the given value.
|
|
Thus mlr put -s foo=97 '$column += @foo' is like mlr put 'begin
|
|
{@foo = 97} $column += @foo'. The value part is subject to
|
|
type-inferencing. May be specified more than once, e.g. -s
|
|
name1=value1 -s name2=value2. Note: the value may be an
|
|
environment variable, e.g. -s sequence=$SEQUENCE.
|
|
-x Prints records for which {expression} evaluates to false, not
|
|
true, i.e. invert the sense of the filter expression. Default
|
|
false.
|
|
-q Does not include the modified record in the output stream.
|
|
Useful for when all desired output is in begin and/or end
|
|
blocks.
|
|
-S No-op in Miller 6 and above, since type-inferencing is now done
|
|
by the record-readers before filter/put is executed. Supported
|
|
as a no-op pass-through flag for backward compatibility.
|
|
-F No-op in Miller 6 and above, since type-inferencing is now done
|
|
by the record-readers before filter/put is executed. Supported
|
|
as a no-op pass-through flag for backward compatibility.
|
|
-w Print warnings about things like uninitialized variables.
|
|
-W Same as -w, but exit the process if there are any warnings.
|
|
-p Prints the expression's AST (abstract syntax tree), which gives
|
|
full transparency on the precedence and associativity rules of
|
|
Miller's grammar, to stdout.
|
|
-d Like -p but uses a parenthesized-expression format for the AST.
|
|
-D Like -d but with output all on one line.
|
|
-E Echo DSL expression before printing parse-tree.
|
|
-v Same as -E -p.
|
|
-X Exit after parsing but before stream-processing. Useful with
|
|
-v/-d/-D, if you only want to look at parser information.
|
|
--explain Parse and type-check the DSL expression, report whether it is
|
|
valid, and exit without reading the input stream. Exit status is
|
|
0 if the expression is valid and non-zero otherwise; combine
|
|
with --errors-json for a machine-readable error.
|
|
-h|--help Show this message.
|
|
|
|
If you mix -e and -f then the expressions are evaluated in the order encountered.
|
|
Since the expression pieces are simply concatenated, please be sure to use intervening
|
|
semicolons to separate expressions.
|
|
|
|
Parser-info options are -w, -W, -p, -d, -D, -E, -v, and -X.
|
|
|
|
Examples:
|
|
mlr --from example.csv put '$qr = $quantity * $rate'
|
|
More example put expressions:
|
|
If-statements:
|
|
'if ($flag == true) { $quantity *= 10}'
|
|
'if ($x > 0.0) { $y=log10($x); $z=sqrt($y) } else {$y = 0.0; $z = 0.0}'
|
|
Newly created fields can be read after being written:
|
|
'$new_field = $index**2; $qn = $quantity * $new_field'
|
|
Regex-replacement:
|
|
'$name = sub($name, "http.*com"i, "")'
|
|
Regex-capture:
|
|
'if ($a =~ "([a-z]+)_([0-9]+)") { $b = "left_\1"; $c = "right_\2" }'
|
|
Built-in variables:
|
|
'$filename = FILENAME'
|
|
Aggregations (use mlr put -q):
|
|
'@sum += $x; end {emit @sum}'
|
|
'@sum[$shape] += $quantity; end {emit @sum, "shape"}'
|
|
'@sum[$shape][$color] += $x; end {emit @sum, "shape", "color"}'
|
|
'
|
|
@min = min(@min,$x);
|
|
@max=max(@max,$x);
|
|
end{emitf @min, @max}
|
|
'
|
|
|
|
See also https://miller.readthedocs.io/reference-dsl for more context.
|
|
|
|
================================================================
|
|
rank
|
|
Usage: mlr rank [options]
|
|
For each record's value in specified fields, computes the standard
|
|
competition rank (1,2,2,4,...) of that value among all input records,
|
|
optionally within groups.
|
|
E.g. with input records x=10, x=20, x=20, and x=30, emits output records
|
|
x=10,x_rank=1 x=20,x_rank=2 x=20,x_rank=2 and x=30,x_rank=4.
|
|
|
|
Note: by default this is a two-pass algorithm: on the first pass it retains
|
|
input records and their values; on the second pass it computes ranks and
|
|
emits output records, in original input order. This means it produces no
|
|
output until all input is read, but gives correct ranks regardless of input
|
|
order. Use --sorted for a single-pass streaming alternative.
|
|
|
|
Options:
|
|
-f {a,b,c} Field name(s) to rank.
|
|
-g {d,e,f} Optional group-by-field name(s).
|
|
--sorted Promise that the input is already sorted by the field(s) being ranked
|
|
(within each group, if -g is given). This computes rank in a single
|
|
streaming pass and O(1) space, by comparing each record's value only
|
|
to the immediately preceding one, rather than buffering all records
|
|
to compute an order-independent rank. Produces wrong output if the
|
|
input is not in fact sorted.
|
|
-h|--help Show this message.
|
|
Example: mlr rank -f x data/rank-example.csv
|
|
Example: mlr rank -f x -g g data/rank-example.csv
|
|
Example: mlr sort -f x then rank -f x --sorted data/rank-example.csv
|
|
|
|
================================================================
|
|
regularize
|
|
Usage: mlr regularize [options]
|
|
Outputs records sorted lexically ascending by keys.
|
|
Options:
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
remove-empty-columns
|
|
Usage: mlr remove-empty-columns [options]
|
|
Omits fields which are empty on every input row. Non-streaming.
|
|
Options:
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
rename
|
|
Usage: mlr rename [options] {old1,new1,old2,new2,...}
|
|
Renames specified fields.
|
|
Options:
|
|
-r Treat old field names as regular expressions. "ab", "a.*b" will match
|
|
any field name containing the substring "ab" or matching "a.*b",
|
|
respectively; anchors of the form "^ab$", "^a.*b$" may be used. New
|
|
field names may be plain strings, or may contain capture groups of the
|
|
form "\1" through "\9". Wrapping the regex in double quotes is
|
|
optional, but is required if you wish to follow it with 'i' to
|
|
indicate case-insensitivity.
|
|
-g Do global replacement within each field name rather than first-match
|
|
replacement.
|
|
-h|--help Show this message.
|
|
Examples:
|
|
mlr rename old_name,new_name
|
|
mlr rename old_name_1,new_name_1,old_name_2,new_name_2
|
|
mlr rename -r 'Date_[0-9]+,Date' Rename all such fields to be "Date"
|
|
mlr rename -r '"Date_[0-9]+",Date' Same
|
|
mlr rename -r 'Date_([0-9]+).*,\1' Rename all such fields to be of the form 20151015
|
|
mlr rename -r '"name"i,Name' Rename "name", "Name", "NAME", etc. to "Name"
|
|
|
|
================================================================
|
|
reorder
|
|
Usage: mlr reorder [options]
|
|
Moves specified names to start of record, or end of record.
|
|
Options:
|
|
-e Put specified field names at record end: default is to put them at
|
|
record start.
|
|
-f {a,b,c} Field names to reorder.
|
|
-r {a,b,c} Treat field names as regular expressions. Matched fields are moved to
|
|
start or end, grouped by the order the regexes are given; within each
|
|
group, fields keep their record order. Example: -r '^YYY,^XXX' puts
|
|
all YYY-prefixed fields first, then all XXX-prefixed fields, then the
|
|
rest.
|
|
-b {x} Put field names specified with -f before field name specified by {x},
|
|
if any. If {x} isn't present in a given record, the specified fields
|
|
will not be moved.
|
|
-a {x} Put field names specified with -f after field name specified by {x},
|
|
if any. If {x} isn't present in a given record, the specified fields
|
|
will not be moved.
|
|
-h|--help Show this message.
|
|
|
|
Examples:
|
|
mlr reorder -f a,b sends input record "d=4,b=2,a=1,c=3" to "a=1,b=2,d=4,c=3".
|
|
mlr reorder -e -f a,b sends input record "d=4,b=2,a=1,c=3" to "d=4,c=3,a=1,b=2".
|
|
mlr reorder -r '^YYY,^XXX' puts YYY-prefixed fields first, then XXX-prefixed fields, then rest.
|
|
|
|
================================================================
|
|
repeat
|
|
Usage: mlr repeat [options]
|
|
Copies input records to output records multiple times.
|
|
Options must be exactly one of -n or -f.
|
|
Options:
|
|
-n {repeat count} Repeat each input record this many times.
|
|
-f {field name} Same as -n, but take the repeat count from the specified field
|
|
name of each input record.
|
|
-h|--help Show this message.
|
|
Example:
|
|
echo x=0 | mlr repeat -n 4 then put '$x=urand()'
|
|
produces:
|
|
x=0.488189
|
|
x=0.484973
|
|
x=0.704983
|
|
x=0.147311
|
|
Example:
|
|
echo a=1,b=2,c=3 | mlr repeat -f b
|
|
produces:
|
|
a=1,b=2,c=3
|
|
a=1,b=2,c=3
|
|
Example:
|
|
echo a=1,b=2,c=3 | mlr repeat -f c
|
|
produces:
|
|
a=1,b=2,c=3
|
|
a=1,b=2,c=3
|
|
a=1,b=2,c=3
|
|
|
|
================================================================
|
|
reshape
|
|
Usage: mlr reshape [options]
|
|
Wide-to-long options:
|
|
-i {input field names} -o {key-field name,value-field name}
|
|
-r {input field regex} -o {key-field name,value-field name}
|
|
These pivot/reshape the input data such that the input fields are removed
|
|
and separate records are emitted for each key/value pair.
|
|
Note: if you have multiple regexes, please specify them using multiple -r,
|
|
since regexes can contain commas within them.
|
|
Note: this works with tail -f and produces output records for each input
|
|
record seen. If input is coming from `tail -f`, be sure to use
|
|
`--records-per-batch 1`.
|
|
Long-to-wide options:
|
|
-s {key-field name,value-field name}
|
|
These pivot/reshape the input data to undo the wide-to-long operation.
|
|
Note: this does not work with tail -f; it produces output records only after
|
|
all input records have been read.
|
|
Options:
|
|
-i {input field names} Input field names for wide-to-long reshape.
|
|
Use with -o.
|
|
-r {input field regex} Input field regex for wide-to-long reshape.
|
|
May be repeated. Use with -o. If you have
|
|
multiple regexes, please specify them using
|
|
multiple -r, since regexes can contain
|
|
commas within them.
|
|
-o {key-field name,value-field name} Output key-field and value-field names for
|
|
wide-to-long reshape. Requires -i or -r.
|
|
-s {key-field name,value-field name} Key-field and value-field names for
|
|
long-to-wide reshape.
|
|
-h|--help Show this message.
|
|
|
|
Examples:
|
|
|
|
Input file "wide.txt":
|
|
time X Y
|
|
2009-01-01 0.65473572 2.4520609
|
|
2009-01-02 -0.89248112 0.2154713
|
|
2009-01-03 0.98012375 1.3179287
|
|
|
|
mlr --pprint reshape -i X,Y -o item,value wide.txt
|
|
time item value
|
|
2009-01-01 X 0.65473572
|
|
2009-01-01 Y 2.4520609
|
|
2009-01-02 X -0.89248112
|
|
2009-01-02 Y 0.2154713
|
|
2009-01-03 X 0.98012375
|
|
2009-01-03 Y 1.3179287
|
|
|
|
mlr --pprint reshape -r '[A-Z]' -o item,value wide.txt
|
|
time item value
|
|
2009-01-01 X 0.65473572
|
|
2009-01-01 Y 2.4520609
|
|
2009-01-02 X -0.89248112
|
|
2009-01-02 Y 0.2154713
|
|
2009-01-03 X 0.98012375
|
|
2009-01-03 Y 1.3179287
|
|
|
|
Input file "long.txt":
|
|
time item value
|
|
2009-01-01 X 0.65473572
|
|
2009-01-01 Y 2.4520609
|
|
2009-01-02 X -0.89248112
|
|
2009-01-02 Y 0.2154713
|
|
2009-01-03 X 0.98012375
|
|
2009-01-03 Y 1.3179287
|
|
|
|
mlr --pprint reshape -s item,value long.txt
|
|
time X Y
|
|
2009-01-01 0.65473572 2.4520609
|
|
2009-01-02 -0.89248112 0.2154713
|
|
2009-01-03 0.98012375 1.3179287
|
|
See also mlr nest.
|
|
|
|
================================================================
|
|
sample
|
|
Usage: mlr sample [options]
|
|
Reservoir sampling (subsampling without replacement), optionally by category.
|
|
See also mlr bootstrap and mlr shuffle.
|
|
Options:
|
|
-g {a,b,c} Optional: group-by-field names for samples, e.g. a,b,c.
|
|
-k {k} Required: number of records to output in total, or by group if using
|
|
-g.
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
sec2gmtdate
|
|
Usage: mlr sec2gmtdate {comma-separated list of field names}
|
|
Replaces a numeric field representing seconds since the epoch with the
|
|
corresponding GMT year-month-day timestamp; leaves non-numbers as-is.
|
|
This is nothing more than a keystroke-saver for the sec2gmtdate function:
|
|
mlr sec2gmtdate time1,time2
|
|
is the same as
|
|
mlr put '$time1=sec2gmtdate($time1);$time2=sec2gmtdate($time2)'
|
|
Options:
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
sec2gmt
|
|
Usage: mlr sec2gmt [options] {comma-separated list of field names}
|
|
Replaces a numeric field representing seconds since the epoch with the
|
|
corresponding GMT timestamp; leaves non-numbers as-is. This is nothing
|
|
more than a keystroke-saver for the sec2gmt function:
|
|
mlr sec2gmt time1,time2
|
|
is the same as
|
|
mlr put '$time1 = sec2gmt($time1); $time2 = sec2gmt($time2)'
|
|
Options:
|
|
-1 Format seconds with 1 decimal place.
|
|
-2 Format seconds with 2 decimal places.
|
|
-3 Format seconds with 3 decimal places.
|
|
-4 Format seconds with 4 decimal places.
|
|
-5 Format seconds with 5 decimal places.
|
|
-6 Format seconds with 6 decimal places.
|
|
-7 Format seconds with 7 decimal places.
|
|
-8 Format seconds with 8 decimal places.
|
|
-9 Format seconds with 9 decimal places.
|
|
--millis Input numbers are treated as milliseconds since the epoch.
|
|
--micros Input numbers are treated as microseconds since the epoch.
|
|
--nanos Input numbers are treated as nanoseconds since the epoch.
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
seqgen
|
|
Usage: mlr seqgen [options]
|
|
Produces a sequence of counters. Discards the input record stream. Produces
|
|
output as specified by the options.
|
|
|
|
Options:
|
|
-f {name} Field name for counters. Default "i".
|
|
--start {value} Inclusive start value. Default 1.
|
|
--step {value} Step value. Default 1. May be negative but not zero (unless
|
|
start == stop).
|
|
--stop {value} Inclusive stop value. Default 100.
|
|
-h|--help Show this message.
|
|
Start, stop, and/or step may be floating-point. Output is integer if start,
|
|
stop, and step are all integers. Step may be negative. It may not be zero
|
|
unless start == stop.
|
|
|
|
================================================================
|
|
shuffle
|
|
Usage: mlr shuffle [options]
|
|
Outputs records randomly permuted. No output records are produced until
|
|
all input records are read. See also mlr bootstrap and mlr sample.
|
|
Options:
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
skip-trivial-records
|
|
Usage: mlr skip-trivial-records [options]
|
|
Passes through all records except those with zero fields,
|
|
or those for which all fields have empty value.
|
|
Options:
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
sort
|
|
Usage: mlr sort {flags}
|
|
Sorts records primarily by the first specified field, secondarily by the second
|
|
field, and so on. (Any records not having all specified sort keys will appear
|
|
at the end of the output, in the order they were encountered, regardless of the
|
|
specified sort order.) The sort is stable: records that compare equal will sort
|
|
in the order they were encountered in the input record stream.
|
|
|
|
Options:
|
|
-f {a,b,c} Lexical ascending sort on the specified field names.
|
|
-r {a,b,c} Lexical descending sort on the specified field names.
|
|
-c {a,b,c} Case-folded lexical ascending sort on the specified field names.
|
|
-cr {a,b,c} Case-folded lexical descending sort on the specified field
|
|
names.
|
|
-n {a,b,c} Numerical ascending sort on the specified field names; nulls
|
|
sort last.
|
|
-nf {a,b,c} Same as -n.
|
|
-nr {a,b,c} Numerical descending sort on the specified field names; nulls
|
|
sort first.
|
|
-t {a,b,c} Natural ascending sort on the specified field names.
|
|
-b Move sort fields to start of record, as in reorder -b.
|
|
-tr|-rt {a,b,c} Natural descending sort on the specified field names.
|
|
-h|--help Show this message.
|
|
|
|
Example:
|
|
mlr sort -f a,b -nr x,y,z
|
|
which is the same as:
|
|
mlr sort -f a -f b -nr x -nr y -nr z
|
|
|
|
================================================================
|
|
sort-within-records
|
|
Usage: mlr sort-within-records [options]
|
|
Outputs records sorted lexically ascending by keys.
|
|
Options:
|
|
-f {names} Sort only these keys; others preserve record order.
|
|
-r {regex} Sort only keys matching this regex; others preserve record order.
|
|
Example: -r '^[xy]' sorts keys starting with x or y. With no regex
|
|
argument, -r recursively sorts subobjects/submaps (e.g. for JSON
|
|
input), or combines with -f to treat names as regexes.
|
|
-n Sort field names naturally (e.g. 2 before 12). Combines with -f/-r.
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
sparkline
|
|
Usage: mlr sparkline [options]
|
|
Reduces numeric field(s), across all records in input order, to a compact
|
|
Unicode sparkline -- one block character per record -- for visualizing
|
|
trends. Emits one output record per field. Holds all records in memory
|
|
before producing any output.
|
|
Options:
|
|
-f {a,b,c} Field names to sparkline.
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
sparsify
|
|
Usage: mlr sparsify [options]
|
|
Unsets fields for which the key is the empty string (or, optionally, another
|
|
specified value). Only makes sense with output format not being CSV or TSV.
|
|
Options:
|
|
-s {filler string} What values to remove. Defaults to the empty string.
|
|
-f {a,b,c} Specify field names to be operated on; any other fields won't
|
|
be modified. The default is to modify all fields.
|
|
-h|--help Show this message.
|
|
Example: if input is a=1,b=,c=3 then output is a=1,c=3.
|
|
|
|
================================================================
|
|
split
|
|
Usage: mlr split [options] {filename}
|
|
Options:
|
|
-n {n} Cap output file sizes at N records.
|
|
-m {m} Produce M files, round-robining records among them.
|
|
-g {a,b,c} Write separate files with records having distinct values for the
|
|
specified field names.
|
|
--prefix {p} Output filename prefix. Default "split".
|
|
--suffix {s} Output filename suffix. Default is from the output format, e.g.
|
|
"csv".
|
|
--folder {f} Output directory. Default is current directory.
|
|
-a Append to existing files rather than overwriting.
|
|
-v Send records downstream as well as splitting to files.
|
|
-e Do NOT URL-escape names of output files.
|
|
-j {J} String used to join filename parts. Default "_".
|
|
-h|--help Show this message.
|
|
Exactly one of -m, -n, or -g must be supplied.
|
|
Any of the output-format command-line flags (see mlr -h). For example, using
|
|
mlr --icsv --from myfile.csv split --ojson -n 1000
|
|
the input is CSV, but the output files are JSON.
|
|
|
|
Examples: Suppose myfile.csv has 1,000,000 records.
|
|
|
|
100 output files, 10,000 records each. First 10,000 records in split_1.csv, next in split_2.csv, etc.
|
|
mlr --csv --from myfile.csv split -n 10000
|
|
|
|
10 output files, 100,000 records each. Records 1,11,21,etc in split_1.csv, records 2,12,22, etc in split_2.csv, etc.
|
|
mlr --csv --from myfile.csv split -m 10
|
|
Same, but with JSON output.
|
|
mlr --csv --from myfile.csv split -m 10 -o json
|
|
|
|
Same but instead of split_1.csv, split_2.csv, etc. there are test_1.dat, test_2.dat, etc.
|
|
mlr --csv --from myfile.csv split -m 10 --prefix test --suffix dat
|
|
Same, but written to the /tmp/ directory.
|
|
mlr --csv --from myfile.csv split -m 10 --prefix /tmp/test --suffix dat
|
|
Or using --folder:
|
|
mlr --csv --from myfile.csv split -m 10 --folder /tmp --prefix test --suffix dat
|
|
|
|
If the shape field has values triangle and square, then there will be split_triangle.csv and split_square.csv.
|
|
mlr --csv --from myfile.csv split -g shape
|
|
|
|
If the color field has values yellow and green, and the shape field has values triangle and square,
|
|
then there will be split_yellow_triangle.csv, split_yellow_square.csv, etc.
|
|
mlr --csv --from myfile.csv split -g color,shape
|
|
|
|
See also the "tee" DSL function which lets you do more ad-hoc customization.
|
|
|
|
================================================================
|
|
ssub
|
|
Usage: mlr ssub [options]
|
|
Replaces old string with new string in specified field(s), without regex support for
|
|
the old string, like the `ssub` DSL function.
|
|
Both the search and replacement strings support C-style backslash escapes such
|
|
as \n, \t, and \x1f. Write \\ to get a literal backslash.
|
|
See also the `gsub` and `sub` verbs.
|
|
Options:
|
|
-f {a,b,c} Field names to apply substitution to.
|
|
-r {regex} Regular expression for field names to apply substitution to.
|
|
-a Apply substitution to all fields.
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
stats1
|
|
Usage: mlr stats1 [options]
|
|
Computes univariate statistics for one or more given fields, accumulated across
|
|
the input record stream.
|
|
Options:
|
|
-a {sum,count,...} Names of accumulators: one or more of the listed values. Also
|
|
accepts median (same as p50) and percentiles p{n} for n in
|
|
0..100, e.g. p10 p25.2 p50 p98 p100.
|
|
-f {a,b,c} Value-field names on which to compute statistics.
|
|
--fr {regex} Regex for value-field names on which to compute statistics
|
|
(compute statistics on values in all field names matching the
|
|
regex).
|
|
--fx {regex} Inverted regex for value-field names on which to compute
|
|
statistics (compute statistics on values in all field names
|
|
not matching the regex).
|
|
-g {d,e,f} Optional group-by-field names.
|
|
--gr {regex} Regex for optional group-by-field names (group by values in
|
|
field names matching the regex).
|
|
--gx {regex} Inverted regex for optional group-by-field names (group by
|
|
values in field names not matching the regex).
|
|
--grfx {regex} Shorthand for --gr {regex} --fx {that same regex}.
|
|
-i Use interpolated percentiles, like R's type=7; default like
|
|
type=1. Not sensical for string-valued fields.
|
|
-s Print iterative stats. Useful in tail -f contexts, in which
|
|
case please avoid pprint-format output since end of input
|
|
stream will never be seen. Likewise, if input is coming from
|
|
`tail -f` be sure to use `--records-per-batch 1`.
|
|
-w {n} Sliding-window mode: compute statistics over a trailing
|
|
window of up to n records (including the current one), rather
|
|
than over the whole record stream. Windows are kept per group
|
|
when -g is used. One output record is emitted per input
|
|
record, with the windowed statistics appended to it. Not
|
|
compatible with -s.
|
|
-S No-op flag for backward compatibility with Miller 5.
|
|
-F No-op flag for backward compatibility with Miller 5.
|
|
-h|--help Show this message.
|
|
Names of accumulators for -a, one or more of:
|
|
median This is the same as p50
|
|
p10 p25.2 p50 p98 p100 etc.
|
|
count Count instances of fields
|
|
null_count Count number of empty-string/JSON-null instances per field
|
|
distinct_count Count number of distinct values per field
|
|
mode Find most-frequently-occurring values for fields; first-found wins tie
|
|
antimode Find least-frequently-occurring values for fields; first-found wins tie
|
|
sum Compute sums of specified fields
|
|
mean Compute averages (sample means) of specified fields
|
|
mad Compute mean absolute deviation
|
|
var Compute sample variance of specified fields
|
|
stddev Compute sample standard deviation of specified fields
|
|
meaneb Estimate error bars for averages (assuming no sample autocorrelation)
|
|
skewness Compute sample skewness of specified fields
|
|
kurtosis Compute sample kurtosis of specified fields
|
|
min Compute minimum values of specified fields
|
|
max Compute maximum values of specified fields
|
|
minlen Compute minimum string-lengths of specified fields
|
|
maxlen Compute maximum string-lengths of specified fields
|
|
Example: mlr stats1 -a min,p10,p50,p90,max -f value -g size,shape
|
|
Example: mlr stats1 -a count,mode -f size
|
|
Example: mlr stats1 -a count,mode -f size -g shape
|
|
Example: mlr stats1 -a mean,min,max -f quantity -g name -w 7
|
|
This emits one output record per input record, with sliding-window
|
|
statistics over the last up-to-7 records for each name.
|
|
Example: mlr stats1 -a count,mode --fr '^[a-h].*$' --gr '^k.*$'
|
|
This computes count and mode statistics on all field names beginning
|
|
with a through h, grouped by all field names starting with k.
|
|
|
|
Notes:
|
|
* p50 and median are synonymous.
|
|
* min and max output the same results as p0 and p100, respectively, but use
|
|
less memory.
|
|
* String-valued data make sense unless arithmetic on them is required,
|
|
e.g. for sum, mean, interpolated percentiles, etc. In case of mixed data,
|
|
numbers are less than strings.
|
|
* count and mode allow text input; the rest require numeric input.
|
|
In particular, 1 and 1.0 are distinct text for count and mode.
|
|
* When there are mode ties, the first-encountered datum wins.
|
|
|
|
================================================================
|
|
stats2
|
|
Usage: mlr stats2 [options]
|
|
Computes bivariate statistics for one or more given field-name pairs,
|
|
accumulated across the input record stream.
|
|
Options:
|
|
-a {linreg-ols,corr,...} Names of accumulators: one or more of the listed
|
|
values.
|
|
-f {a,b,c,d} Value-field name-pairs on which to compute statistics.
|
|
There must be an even number of names.
|
|
-g {e,f,g} Optional group-by-field names.
|
|
-v Print additional output for linreg-pca.
|
|
-s Print iterative stats. Useful in tail -f contexts, in
|
|
which case please avoid pprint-format output since end
|
|
of input stream will never be seen. Likewise, if input
|
|
is coming from `tail -f`, be sure to use
|
|
`--records-per-batch 1`.
|
|
--fit Rather than printing regression parameters, applies
|
|
them to the input data to compute new fit fields. All
|
|
input records are held in memory until end of input
|
|
stream. Has effect only for linreg-ols, linreg-pca, and
|
|
logireg.
|
|
-S No-op flag for backward compatibility with Miller 5.
|
|
-F No-op flag for backward compatibility with Miller 5.
|
|
-h|--help Show this message.
|
|
Names of accumulators for -a, one or more of:
|
|
linreg-ols Linear regression using ordinary least squares
|
|
linreg-pca Linear regression using principal component analysis
|
|
r2 Quality metric for linreg-ols (linreg-pca emits its own)
|
|
logireg Logistic regression
|
|
corr Sample correlation
|
|
cov Sample covariance
|
|
covx Sample-covariance matrix
|
|
Only one of -s or --fit may be used.
|
|
Example: mlr stats2 -a linreg-pca -f x,y
|
|
Example: mlr stats2 -a linreg-ols,r2 -f x,y -g size,shape
|
|
Example: mlr stats2 -a corr -f x,y
|
|
|
|
================================================================
|
|
step
|
|
Usage: mlr step [options]
|
|
Computes values dependent on earlier/later records, optionally grouped by category.
|
|
Options:
|
|
-a {delta,rsum,...} Names of steppers: comma-separated, one or more of the
|
|
listed values.
|
|
-f {a,b,c} Value-field names on which to compute statistics.
|
|
-g {d,e,f} Optional group-by-field names.
|
|
-F Computes integerable things (e.g. counter) in floating
|
|
point. As of Miller 6 this happens automatically, but the
|
|
flag is accepted as a no-op for backward compatibility with
|
|
Miller 5 and below.
|
|
-d {x,y,z} Weights for EWMA. 1 means current sample gets all weight (no
|
|
smoothing), near under 1 is light smoothing, near over 0 is
|
|
heavy smoothing. Multiple weights may be specified, e.g.
|
|
"mlr step -a ewma -f sys_load -d 0.01,0.1,0.9". Default if
|
|
omitted is "-d 0.5".
|
|
-o {a,b,c} Custom suffixes for EWMA output fields. If omitted, these
|
|
default to the -d values. If supplied, the number of -o
|
|
values must be the same as the number of -d values.
|
|
-h|--help Show this message.
|
|
|
|
Names of steppers for -a, comma-separated, one or more of:
|
|
counter Count instances of field(s) between successive records
|
|
delta Compute differences in field(s) between successive records. Use delta or equivalently delta_1 for the previous record, or delta_{n} for n records back.
|
|
ewma Exponentially weighted moving average over successive records
|
|
from-first Compute differences in field(s) from first record
|
|
ratio Compute ratios in field(s) between successive records. Use ratio or equivalently ratio_1 for the previous record, or ratio_{n} for n records back.
|
|
rprod Compute running products of field(s) between successive records
|
|
rsum Compute running sums of field(s) between successive records
|
|
shift Alias for shift_lag. Use shift or equivalently shift_1 for the previous record, or shift_{n} for n records back.
|
|
shift_lag Include value(s) in field(s) from the previous record, if any. Use shift_lag or equivalently shift_lag_1 for the previous record, or shift_lag_{n} for n records back.
|
|
shift_lead Include value(s) in field(s) from the next record, if any. Use shift_lead or equivalently shift_lead_1 for the next record, or shift_lead_{n} for n records forward.
|
|
slwin Sliding-window averages over m records back and n forward. E.g. slwin_7_2 for 7 back and 2 forward.
|
|
|
|
Examples:
|
|
mlr step -a rsum -f request_size
|
|
mlr step -a delta -f request_size -g hostname
|
|
mlr step -a ewma -d 0.1,0.9 -f x,y
|
|
mlr step -a ewma -d 0.1,0.9 -o smooth,rough -f x,y
|
|
mlr step -a ewma -d 0.1,0.9 -o smooth,rough -f x,y -g group_name
|
|
mlr step -a slwin_9_0,slwin_0_9 -f x
|
|
mlr step -a shift_lag_12 -f sales
|
|
|
|
The shift, shift_lag, shift_lead, delta, and ratio steppers accept an
|
|
optional trailing count: shift_lag_{n} refers n records back, and
|
|
shift_lead_{n} refers n records forward. The plain forms are equivalent
|
|
to a count of 1: e.g. shift_lag is the same as shift_lag_1.
|
|
|
|
Please see https://miller.readthedocs.io/en/latest/reference-verbs.html#filter or
|
|
https://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average
|
|
for more information on EWMA.
|
|
|
|
================================================================
|
|
sub
|
|
Usage: mlr sub [options]
|
|
Replaces old string with new string in specified field(s), with regex support
|
|
for the old string and not handling multiple matches, like the `sub` DSL function.
|
|
The replacement string supports C-style backslash escapes such as \n, \t,
|
|
and \x1f. Write \\ to get a literal backslash.
|
|
See also the `gsub` and `ssub` verbs.
|
|
Options:
|
|
-f {a,b,c} Field names to apply substitution to.
|
|
-r {regex} Regular expression for field names to apply substitution to.
|
|
-a Apply substitution to all fields.
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
summary
|
|
Usage: mlr summary [options]
|
|
Show summary statistics about the input data.
|
|
|
|
All summarizers:
|
|
field_type string, int, etc. -- if a column has mixed types, all encountered types are printed (see notes below)
|
|
count +1 for every instance of the field across all records in the input record stream
|
|
null_count count of field values either empty string or JSON null
|
|
distinct_count count of distinct values for the field
|
|
mode most-frequently-occurring value for the field
|
|
sum sum of field values
|
|
mean mean of the field values
|
|
stddev standard deviation of the field values
|
|
var variance of the field values
|
|
skewness skewness of the field values
|
|
minlen length of shortest string representation for the field
|
|
maxlen length of longest string representation for the field
|
|
min minimum field value
|
|
p25 first-quartile field value
|
|
median median field value
|
|
p75 third-quartile field value
|
|
max maximum field value
|
|
iqr interquartile range: p75 - p25
|
|
lof lower outer fence: p25 - 3.0 * iqr
|
|
lif lower inner fence: p25 - 1.5 * iqr
|
|
uif upper inner fence: p75 + 1.5 * iqr
|
|
uof upper outer fence: p75 + 3.0 * iqr
|
|
|
|
Default summarizers:
|
|
field_type count mean min max null_count distinct_count
|
|
|
|
Notes:
|
|
* min, p25, median, p75, and max work for strings as well as numbers
|
|
* Distinct-counts are computed on string representations -- so 4.1 and 4.10 are counted as distinct here.
|
|
* If the mode is not unique in the input data, the first-encountered value is reported as the mode.
|
|
* A field_type of "int-string", "empty-string", etc. means the column contains values of mixed types --
|
|
all types encountered are printed, hyphen-joined, in the order first encountered.
|
|
|
|
Options:
|
|
-a {mean,sum,etc.} Use only the specified summarizers.
|
|
-x {mean,sum,etc.} Use all summarizers except the specified ones.
|
|
--all Use all available summarizers.
|
|
--transpose Show output with field names as column names.
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
surv
|
|
Usage: mlr surv -d {duration-field} -s {status-field}
|
|
|
|
Estimate Kaplan-Meier survival curve (right-censored).
|
|
Options:
|
|
-d {field} Name of duration field (time-to-event or censoring).
|
|
-s {field} Name of status field (0=censored, 1=event).
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
tac
|
|
Usage: mlr tac [options]
|
|
Prints records in reverse order from the order in which they were encountered.
|
|
Options:
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
tail
|
|
Usage: mlr tail [options]
|
|
Passes through the last n records, optionally by category.
|
|
Options:
|
|
-g {a,b,c} Optional group-by-field names for tail counts, e.g. a,b,c.
|
|
-n {n} Tail-count to print. Default 10. A leading '+' means start at the nth
|
|
record rather than print the last n: e.g. -n +3 passes through all
|
|
but the first 2 records, optionally by category.
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
tee
|
|
Usage: mlr tee [options] {filename}
|
|
Options:
|
|
-a Append to existing file, if any, rather than overwriting.
|
|
-p Treat filename as a pipe-to command.
|
|
-h|--help Show this message.
|
|
Any of the output-format command-line flags (see mlr -h). Example: using
|
|
mlr --icsv --opprint put '...' then tee --ojson ./mytap.dat then stats1 ...
|
|
the input is CSV, the output is pretty-print tabular, but the tee-file output
|
|
is written in JSON format.
|
|
|
|
================================================================
|
|
template
|
|
Usage: mlr template [options]
|
|
Places input-record fields in the order specified by list of column names.
|
|
If the input record is missing a specified field, it will be filled with the fill-with.
|
|
If the input record possesses an unspecified field, it will be discarded.
|
|
Options:
|
|
-f {a,b,c} Comma-separated field names for template, e.g.
|
|
a,b,c.
|
|
-t {filename} CSV file whose header line will be used for
|
|
template.
|
|
--fill-with {filler string} What to fill absent fields with. Defaults to the
|
|
empty string.
|
|
-h|--help Show this message.
|
|
Example:
|
|
* Specified fields are a,b,c.
|
|
* Input record is c=3,a=1,f=6.
|
|
* Output record is a=1,b=,c=3.
|
|
|
|
================================================================
|
|
top
|
|
Usage: mlr top [options]
|
|
Options:
|
|
-f {a,b,c} Value-field names for top counts.
|
|
-g {d,e,f} Optional group-by-field names for top counts.
|
|
-n {count} How many records to print per category; default 1.
|
|
-a Print all fields for top-value records; default is to print only
|
|
value and group-by fields. Requires a single value-field name only.
|
|
--max Print top largest values. This is the default.
|
|
--min Print top smallest values; default is top largest values.
|
|
-F Keep top values as floats even if they look like integers (ignored in
|
|
Miller 6, kept for backward compatibility).
|
|
-o {name} Field name for output indices. Default "top_idx". Ignored if -a is
|
|
used.
|
|
-h|--help Show this message.
|
|
Prints the n records with smallest/largest values at specified fields,
|
|
optionally by category. If -a is given, then the top records are emitted
|
|
with the same fields as they appeared in the input. Without -a, only fields
|
|
from -f, fields from -g, and the top-index field are emitted. For more information
|
|
please see https://miller.readthedocs.io/en/latest/reference-verbs#top
|
|
|
|
================================================================
|
|
utf8-to-latin1
|
|
Usage: mlr utf8-to-latin1, with no options.
|
|
Recursively converts record strings from UTF-8 to Latin-1.
|
|
For field-level control, please see the utf8_to_latin1 DSL function.
|
|
Options:
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
unflatten
|
|
Usage: mlr unflatten [options]
|
|
Reverses flatten. Example: field with name 'a.b.c' and value 4
|
|
becomes name 'a' and value '{"b": { "c": 4 }}'.
|
|
Options:
|
|
-f {a,b,c} Comma-separated list of field names to unflatten (default all).
|
|
-s {string} Separator, defaulting to mlr --flatsep value.
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
uniq
|
|
Usage: mlr uniq [options]
|
|
Prints distinct values for specified field names. With -c, same as
|
|
count-distinct. For uniq, -f is a synonym for -g. Output fields are
|
|
written in the order in which they are named with -g or -f, not in the
|
|
order in which they appear in the input records.
|
|
To deduplicate records by one or more fields while keeping all other
|
|
fields, use head: e.g. "mlr head -n 1 -g hash" keeps the first record
|
|
for each distinct value of the hash field, with all fields intact.
|
|
|
|
Options:
|
|
-g {d,e,f} Group-by field names for uniq counts.
|
|
-f {d,e,f} Synonym for -g.
|
|
-x {a,b,c} Field names to exclude for uniq; use each record's other fields
|
|
instead.
|
|
-c Show repeat counts in addition to unique values.
|
|
-n Show only the number of distinct values.
|
|
-o {name} Field name for output count. Default "count".
|
|
-a Output each unique record only once. Incompatible with -g. With -c,
|
|
produces unique records, with repeat counts for each. With -n,
|
|
produces only one record which is the unique-record count. With
|
|
neither -c nor -n, produces unique records.
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
unspace
|
|
Usage: mlr unspace [options]
|
|
Replaces spaces in record keys and/or values with _. This is helpful for PPRINT output.
|
|
Options:
|
|
-f {x} Replace spaces with specified filler character. Default "_".
|
|
-k Unspace only keys, not keys and values.
|
|
-v Unspace only values, not keys and values.
|
|
-h|--help Show this message.
|
|
|
|
================================================================
|
|
unsparsify
|
|
Usage: mlr unsparsify [options]
|
|
Prints records with the union of field names over all input records.
|
|
For field names absent in a given record but present in others, fills in
|
|
a value. This verb retains all input before producing any output.
|
|
Options:
|
|
--fill-with {filler string} What to fill absent fields with. Defaults to the
|
|
empty string.
|
|
-f {a,b,c} Specify field names to be operated on; others are
|
|
not modified and operation is streaming.
|
|
-h|--help Show this message.
|
|
Example: if the input is two records, one being 'a=1,b=2' and the other
|
|
being 'b=3,c=4', then the output is the two records 'a=1,b=2,c=' and
|
|
'a=,b=3,c=4'.
|
|
================================================================
|