mirror of
https://github.com/johnkerl/miller.git
synced 2026-07-18 08:55:41 +00:00
7483 lines
255 KiB
HTML
7483 lines
255 KiB
HTML
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
|
|
<html lang="en">
|
|
|
|
<!-- PAGE GENERATED FROM template.html and content-for-reference.html BY poki. -->
|
|
<!-- PLEASE MAKE CHANGES THERE AND THEN RE-RUN poki. -->
|
|
<head>
|
|
<meta http-equiv="Content-type" content="text/html;charset=UTF-8"/>
|
|
<meta name="description" content="Miller documentation"/>
|
|
<meta name="viewport" content="width=device-width, initial-scale=1.0"/> <!-- mobile-friendly -->
|
|
<meta name="keywords"
|
|
content="John Kerl, Kerl, Miller, miller, mlr, OLAP, data analysis software, regression, correlation, variance, data tools, " />
|
|
|
|
<title> Reference </title>
|
|
<link rel="stylesheet" type="text/css" href="css/miller.css"/>
|
|
<link rel="stylesheet" type="text/css" href="css/poki-callbacks.css"/>
|
|
</head>
|
|
|
|
<!-- ================================================================ -->
|
|
<script type="text/javascript">
|
|
var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www.");
|
|
document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E"));
|
|
</script>
|
|
<script type="text/javascript">
|
|
try {
|
|
var pageTracker = _gat._getTracker("UA-15651652-1");
|
|
pageTracker._trackPageview();
|
|
} catch(err) {}
|
|
</script>
|
|
|
|
<script type="text/javascript">
|
|
function toggle(divName) {
|
|
var eleDiv = document.getElementById(divName);
|
|
if (eleDiv != null) {
|
|
if (eleDiv.style.display == "block") {
|
|
eleDiv.style.display = "none";
|
|
} else {
|
|
eleDiv.style.display = "block";
|
|
}
|
|
}
|
|
}
|
|
</script>
|
|
|
|
<!--
|
|
The background image is from a screenshot of a Google search for "data analysis
|
|
tools", lightened and sepia-toned. Over this was placed a Mac Terminal app with
|
|
very light-grey font and translucent background, in which a few statistical
|
|
Miller commands were run with pretty-print-tabular output format.
|
|
-->
|
|
<body background="pix/sepia-overlay.jpg">
|
|
|
|
<!-- ================================================================ -->
|
|
<table width="100%">
|
|
<tr>
|
|
|
|
<!-- navbar -->
|
|
<td width="15%">
|
|
<!--
|
|
<img src="pix/mlr.jpg" />
|
|
<img style="border-width:1px; color:black;" src="pix/mlr.jpg" />
|
|
-->
|
|
|
|
<div class="pokinav">
|
|
<center><titleinbody>Miller</titleinbody></center>
|
|
|
|
<!-- PAGE LIST GENERATED FROM template.html BY poki -->
|
|
<br/><b>Overview:</b>
|
|
<br/>• <a href="index.html">About Miller</a>
|
|
<br/>• <a href="10-min.html">Miller in 10 minutes</a>
|
|
<br/>• <a href="file-formats.html">File formats</a>
|
|
<br/>• <a href="feature-comparison.html">Miller features in the context of the Unix toolkit</a>
|
|
<br/>• <a href="record-heterogeneity.html">Record-heterogeneity</a>
|
|
<br/>• <a href="internationalization.html">Internationalization</a>
|
|
<br/><b>Using Miller:</b>
|
|
<br/>• <a href="faq.html">FAQ</a>
|
|
<br/>• <a href="reference.html"><b>Reference</b></a>
|
|
<br/>• <a href="manpage.html">Manpage</a>
|
|
<br/>• <a href="data-examples.html">Data-diving examples</a>
|
|
<br/>• <a href="cookbook.html">Cookbook</a>
|
|
<br/>• <a href="release-docs.html">Documents by release</a>
|
|
<br/>• <a href="build.html">Installation, portability, dependencies, and testing</a>
|
|
<br/><b>Background:</b>
|
|
<br/>• <a href="whyc.html">Why C?</a>
|
|
<br/>• <a href="etymology.html">Why call it Miller?</a>
|
|
<br/>• <a href="originality.html">How original is Miller?</a>
|
|
<br/>• <a href="performance.html">Performance</a>
|
|
<br/><b>Repository:</b>
|
|
<br/>• <a href="to-do.html">Things to do</a>
|
|
<br/>• <a href="contact.html">Contact information</a>
|
|
<br/>• <a href="https://github.com/johnkerl/miller">GitHub repo</a>
|
|
<br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/>
|
|
<br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/>
|
|
<br/> <br/> <br/> <br/> <br/> <br/>
|
|
</div>
|
|
</td>
|
|
|
|
<!-- page body -->
|
|
<td>
|
|
<!--
|
|
This is a visually gorgeous feature (here & in the CSS): it allows for
|
|
independent scroll of the nav and body panels. In particular the nav
|
|
stays on-screen as you scroll the body.
|
|
|
|
However, two problems:
|
|
|
|
(1) In Firefox & Chrome both I get janky end-of-body scrolls: there is
|
|
more content but I can't scroll down to it unless I repeatedly retry the
|
|
scrolldown. Which is weird.
|
|
|
|
(2) Worse, only the first page renders in PDF (again, Firefox & Chrome).
|
|
|
|
For now I'm disabling this separate-scroll feature. A frontender, I am
|
|
not ... maybe someday I'll find a config which gets *all* the features
|
|
I want; for now, it's a tradeoff.
|
|
-->
|
|
|
|
<!-- Implementation details: one bit is right here:
|
|
|
|
div style="overflow-y:scroll;height:1500px"
|
|
|
|
and the other bit is in css/poki-callbacks.css:
|
|
|
|
.pokinav {
|
|
display: inline-block;
|
|
background: #e8d9bc;
|
|
border: 1;
|
|
box-shadow: 0px 0px 3px 3px #C9C9C9;
|
|
margin: 10px;
|
|
padding-top: 10px;
|
|
padding-bottom: 10px;
|
|
padding-left: 10px;
|
|
padding-right: 10px;
|
|
overflow-y: scroll; < - - - - - - here
|
|
height: 1500px;
|
|
}
|
|
|
|
-->
|
|
<div>
|
|
<center> <titleinbody> Reference </titleinbody> </center>
|
|
<p/>
|
|
|
|
<!-- BODY COPIED FROM content-for-reference.html BY poki -->
|
|
<div class="pokitoc">
|
|
<center><b>Contents:</b></center>
|
|
• <a href="#Command_overview">Command overview</a><br/>
|
|
• <a href="#On-line_help">On-line help</a><br/>
|
|
• <a href="#I/O_options">I/O options</a><br/>
|
|
• <a href="#Formats">Formats</a><br/>
|
|
• <a href="#Compression">Compression</a><br/>
|
|
• <a href="#Record/field/pair_separators">Record/field/pair separators</a><br/>
|
|
• <a href="#Number_formatting">Number formatting</a><br/>
|
|
• <a href="#Data_transformations">Data transformations</a><br/>
|
|
• <a href="#bar">bar</a><br/>
|
|
• <a href="#bootstrap">bootstrap</a><br/>
|
|
• <a href="#cat">cat</a><br/>
|
|
• <a href="#check">check</a><br/>
|
|
• <a href="#count-distinct">count-distinct</a><br/>
|
|
• <a href="#cut">cut</a><br/>
|
|
• <a href="#decimate">decimate</a><br/>
|
|
• <a href="#filter">filter</a><br/>
|
|
• <a href="#Features_which_filter_shares_with_put">Features which filter shares with put</a><br/>
|
|
• <a href="#grep">grep</a><br/>
|
|
• <a href="#group-by">group-by</a><br/>
|
|
• <a href="#group-like">group-like</a><br/>
|
|
• <a href="#having-fields">having-fields</a><br/>
|
|
• <a href="#head">head</a><br/>
|
|
• <a href="#histogram">histogram</a><br/>
|
|
• <a href="#join">join</a><br/>
|
|
• <a href="#label">label</a><br/>
|
|
• <a href="#least-frequent">least-frequent</a><br/>
|
|
• <a href="#merge-fields">merge-fields</a><br/>
|
|
• <a href="#most-frequent">most-frequent</a><br/>
|
|
• <a href="#nest">nest</a><br/>
|
|
• <a href="#nothing">nothing</a><br/>
|
|
• <a href="#put">put</a><br/>
|
|
• <a href="#Features_which_put_shares_with_filter">Features which put shares with filter</a><br/>
|
|
• <a href="#regularize">regularize</a><br/>
|
|
• <a href="#rename">rename</a><br/>
|
|
• <a href="#reorder">reorder</a><br/>
|
|
• <a href="#repeat">repeat</a><br/>
|
|
• <a href="#reshape">reshape</a><br/>
|
|
• <a href="#sample">sample</a><br/>
|
|
• <a href="#sec2gmt">sec2gmt</a><br/>
|
|
• <a href="#sec2gmtdate">sec2gmtdate</a><br/>
|
|
• <a href="#seqgen">seqgen</a><br/>
|
|
• <a href="#shuffle">shuffle</a><br/>
|
|
• <a href="#sort">sort</a><br/>
|
|
• <a href="#stats1">stats1</a><br/>
|
|
• <a href="#stats2">stats2</a><br/>
|
|
• <a href="#step">step</a><br/>
|
|
• <a href="#tac">tac</a><br/>
|
|
• <a href="#tail">tail</a><br/>
|
|
• <a href="#tee">tee</a><br/>
|
|
• <a href="#top">top</a><br/>
|
|
• <a href="#uniq">uniq</a><br/>
|
|
• <a href="#Expression_language_for_filter_and_put">Expression language for filter and put</a><br/>
|
|
• <a href="#Syntax">Syntax</a><br/>
|
|
• <a href="#Expression_formatting">Expression formatting</a><br/>
|
|
• <a href="#Expressions_from_files">Expressions from files</a><br/>
|
|
• <a href="#Semicolons,_newlines,_and_curly_braces">Semicolons, newlines, and curly braces</a><br/>
|
|
• <a href="#Variables">Variables</a><br/>
|
|
• <a href="#Built-in_variables">Built-in variables</a><br/>
|
|
• <a href="#Field_names">Field names</a><br/>
|
|
• <a href="#Local_variables">Local variables</a><br/>
|
|
• <a href="#Out-of-stream_variables">Out-of-stream variables</a><br/>
|
|
• <a href="#Indexed_out-of-stream_variables">Indexed out-of-stream variables</a><br/>
|
|
• <a href="#Aggregate_variable_assignments">Aggregate variable assignments</a><br/>
|
|
• <a href="#Keywords_for_filter_and_put">Keywords for filter and put</a><br/>
|
|
• <a href="#Control_structures">Control structures</a><br/>
|
|
• <a href="#Pattern-action_blocks">Pattern-action blocks</a><br/>
|
|
• <a href="#If-statements">If-statements</a><br/>
|
|
• <a href="#While_and_do-while_loops">While and do-while loops</a><br/>
|
|
• <a href="#For-loops">For-loops</a><br/>
|
|
• <a href="#Begin/end_blocks">Begin/end blocks</a><br/>
|
|
• <a href="#Output_statements">Output statements</a><br/>
|
|
• <a href="#Emit_statements">Emit statements</a><br/>
|
|
• <a href="#Multi-emit_statements">Multi-emit statements</a><br/>
|
|
• <a href="#Emit-all_statements">Emit-all statements</a><br/>
|
|
• <a href="#Redirected-output_statements">Redirected-output statements</a><br/>
|
|
• <a href="#Unset_statements">Unset statements</a><br/>
|
|
• <a href="#Filter_statements">Filter statements</a><br/>
|
|
• <a href="#Built-in_functions_for_filter_and_put">Built-in functions for filter and put</a><br/>
|
|
• <a href="#User-defined_functions_and_subroutines">User-defined functions and subroutines</a><br/>
|
|
• <a href="#User-defined_functions">User-defined functions</a><br/>
|
|
• <a href="#User-defined_subroutines">User-defined subroutines</a><br/>
|
|
• <a href="#A_note_on_the_complexity_of_Miller’s_expression_language">A note on the complexity of Miller’s expression language</a><br/>
|
|
• <a href="#then-chaining">then-chaining</a><br/>
|
|
• <a href="#Data_types">Data types</a><br/>
|
|
• <a href="#Null_data:_empty_and_absent">Null data: empty and absent</a><br/>
|
|
• <a href="#String_literals">String literals</a><br/>
|
|
• <a href="#Regular_expressions">Regular expressions</a><br/>
|
|
• <a href="#Regex_captures">Regex captures</a><br/>
|
|
• <a href="#Operator_precedence">Operator precedence</a><br/>
|
|
• <a href="#Operator_and_function_semantics">Operator and function semantics</a><br/>
|
|
• <a href="#Arithmetic">Arithmetic</a><br/>
|
|
• <a href="#Input_scanning">Input scanning</a><br/>
|
|
• <a href="#Conversion_by_math_routines">Conversion by math routines</a><br/>
|
|
• <a href="#Conversion_by_arithmetic_operators">Conversion by arithmetic operators</a><br/>
|
|
• <a href="#Pythonic_division">Pythonic division</a><br/>
|
|
</div>
|
|
<p/>
|
|
|
|
<a id="Command_overview"/><h1>Command overview</h1>
|
|
|
|
<p>
|
|
Whereas the Unix toolkit is made of the separate executables <tt>cat</tt>, <tt>tail</tt>, <tt>cut</tt>,
|
|
<tt>sort</tt>, etc., Miller has subcommands, invoked as follows:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
mlr tac *.dat
|
|
mlr cut --complement -f os_version *.dat
|
|
mlr sort -f hostname,uptime *.dat
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>These fall into categories as follows:
|
|
|
|
<table border=1>
|
|
<tr class="mlrbg">
|
|
<th>Commands </th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>
|
|
<a href="#cat"><tt>cat</tt></a>,
|
|
<a href="#cut"><tt>cut</tt></a>,
|
|
<a href="#head"><tt>head</tt></a>,
|
|
<a href="#sort"><tt>sort</tt></a>,
|
|
<a href="#tac"><tt>tac</tt></a>,
|
|
<a href="#tail"><tt>tail</tt></a>,
|
|
<a href="#top"><tt>top</tt></a>,
|
|
<a href="#uniq"><tt>uniq</tt></a>
|
|
</td>
|
|
<td> Analogs of their Unix-toolkit namesakes, discussed below as well as in
|
|
<a href="feature-comparison.html">Miller features in the context of the Unix toolkit</a> </td>
|
|
</tr>
|
|
|
|
<tr>
|
|
<td>
|
|
<a href="#filter"><tt>filter</tt></a>,
|
|
<a href="#put"><tt>put</tt></a>,
|
|
<a href="#step"><tt>step</tt></a>
|
|
</td>
|
|
<td> <tt>awk</tt>-like functionality </td>
|
|
</tr>
|
|
|
|
<tr>
|
|
<td>
|
|
<a href="#histogram"><tt>histogram</tt></a>,
|
|
<a href="#stats1"><tt>stats1</tt></a>,
|
|
<a href="#stats2"><tt>stats2</tt></a>
|
|
</td>
|
|
<td> Statistically oriented </td>
|
|
</tr>
|
|
|
|
<tr>
|
|
<td>
|
|
<a href="#group-by"><tt>group-by</tt></a>,
|
|
<a href="#group-like"><tt>group-like</tt></a>,
|
|
<a href="#having-fields"><tt>having-fields</tt></a>
|
|
</td>
|
|
<td> Particularly oriented toward <a href="record-heterogeneity.html">Record-heterogeneity</a>, although
|
|
all Miller commands can handle heterogeneous records
|
|
</tr>
|
|
|
|
<tr>
|
|
<td>
|
|
<a href="#count-distinct"><tt>count-distinct</tt></a>,
|
|
<a href="#label"><tt>label</tt></a>,
|
|
<a href="#regularize"><tt>rename</tt></a>,
|
|
<a href="#rename"><tt>rename</tt></a>,
|
|
<a href="#reorder"><tt>reorder</tt></a>
|
|
</td>
|
|
<td> These draw from other sources (see also <a href="originality.html">How original is Miller?</a>):
|
|
<a href="#count-distinct"><tt>count-distinct</tt></a> is SQL-ish, and
|
|
<a href="#rename"><tt>rename</tt></a> can be done by <tt>sed</tt> (which does it faster:
|
|
see <a href="performance.html">Performance</a>).
|
|
</td>
|
|
</tr>
|
|
|
|
</table>
|
|
|
|
<a id="On-line_help"/><h1>On-line help</h1>
|
|
|
|
<p/>Examples:<p/>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --help
|
|
Usage: mlr [I/O options] {verb} [verb-dependent options ...] {zero or more file names}
|
|
|
|
Command-line-syntax examples:
|
|
mlr --csv cut -f hostname,uptime mydata.csv
|
|
mlr --tsv --rs lf filter '$status != "down" && $upsec >= 10000' *.tsv
|
|
mlr --nidx put '$sum = $7 < 0.0 ? 3.5 : $7 + 2.1*$8' *.dat
|
|
grep -v '^#' /etc/group | mlr --ifs : --nidx --opprint label group,pass,gid,member then sort -f group
|
|
mlr join -j account_id -f accounts.dat then group-by account_name balances.dat
|
|
mlr --json put '$attr = sub($attr, "([0-9]+)_([0-9]+)_.*", "\1:\2")' data/*.json
|
|
mlr stats1 -a min,mean,max,p10,p50,p90 -f flag,u,v data/*
|
|
mlr stats2 -a linreg-pca -f u,v -g shape data/*
|
|
mlr put -q '@sum[$a][$b] += $x; end {emit @sum, "a", "b"}' data/*
|
|
mlr --from estimates.tbl put '
|
|
for (k,v in $*) {
|
|
if (isnumeric(v) && k =~ "^[t-z].*$") {
|
|
$sum += v; $count += 1
|
|
}
|
|
}
|
|
$mean = $sum / $count # no assignment if count unset'
|
|
mlr --from infile.dat put -f analyze.mlr
|
|
mlr --from infile.dat put 'tee > "./taps/data-".$a."-".$b, $*'
|
|
mlr --from infile.dat put 'tee | "gzip > ./taps/data-".$a."-".$b.".gz", $*'
|
|
mlr --from infile.dat put -q '@v=$*; dump | "jq .[]"'
|
|
mlr --from infile.dat put '(NR % 1000 == 0) { print > stderr, "Checkpoint ".NR}'
|
|
|
|
Data-format examples:
|
|
DKVP: delimited key-value pairs (Miller default format)
|
|
+---------------------+
|
|
| apple=1,bat=2,cog=3 | Record 1: "apple" => "1", "bat" => "2", "cog" => "3"
|
|
| dish=7,egg=8,flint | Record 2: "dish" => "7", "egg" => "8", "3" => "flint"
|
|
+---------------------+
|
|
|
|
NIDX: implicitly numerically indexed (Unix-toolkit style)
|
|
+---------------------+
|
|
| the quick brown | Record 1: "1" => "the", "2" => "quick", "3" => "brown"
|
|
| fox jumped | Record 2: "1" => "fox", "2" => "jumped"
|
|
+---------------------+
|
|
|
|
CSV/CSV-lite: comma-separated values with separate header line
|
|
+---------------------+
|
|
| apple,bat,cog |
|
|
| 1,2,3 | Record 1: "apple => "1", "bat" => "2", "cog" => "3"
|
|
| 4,5,6 | Record 2: "apple" => "4", "bat" => "5", "cog" => "6"
|
|
+---------------------+
|
|
|
|
Tabular JSON: nested objects are supported, although arrays within them are not:
|
|
+---------------------+
|
|
| { |
|
|
| "apple": 1, | Record 1: "apple" => "1", "bat" => "2", "cog" => "3"
|
|
| "bat": 2, |
|
|
| "cog": 3 |
|
|
| } |
|
|
| { |
|
|
| "dish": { | Record 2: "dish:egg" => "7", "dish:flint" => "8", "garlic" => ""
|
|
| "egg": 7, |
|
|
| "flint": 8 |
|
|
| }, |
|
|
| "garlic": "" |
|
|
| } |
|
|
+---------------------+
|
|
|
|
PPRINT: pretty-printed tabular
|
|
+---------------------+
|
|
| apple bat cog |
|
|
| 1 2 3 | Record 1: "apple => "1", "bat" => "2", "cog" => "3"
|
|
| 4 5 6 | Record 2: "apple" => "4", "bat" => "5", "cog" => "6"
|
|
+---------------------+
|
|
|
|
XTAB: pretty-printed transposed tabular
|
|
+---------------------+
|
|
| apple 1 | Record 1: "apple" => "1", "bat" => "2", "cog" => "3"
|
|
| bat 2 |
|
|
| cog 3 |
|
|
| |
|
|
| dish 7 | Record 2: "dish" => "7", "egg" => "8"
|
|
| egg 8 |
|
|
+---------------------+
|
|
|
|
Markdown tabular (supported for output only):
|
|
+-----------------------+
|
|
| | apple | bat | cog | |
|
|
| | --- | --- | --- | |
|
|
| | 1 | 2 | 3 | | Record 1: "apple => "1", "bat" => "2", "cog" => "3"
|
|
| | 4 | 5 | 6 | | Record 2: "apple" => "4", "bat" => "5", "cog" => "6"
|
|
+-----------------------+
|
|
|
|
Help options:
|
|
-h or --help Show this message.
|
|
--version Show the software version.
|
|
{verb name} --help Show verb-specific help.
|
|
--list-all-verbs or -l List only verb names.
|
|
--help-all-verbs Show help on all verbs.
|
|
|
|
Verbs:
|
|
bar bootstrap cat check count-distinct cut decimate filter grep group-by
|
|
group-like having-fields head histogram join label least-frequent
|
|
merge-fields most-frequent nest nothing put regularize rename reorder repeat
|
|
reshape sample sec2gmt sec2gmtdate seqgen shuffle sort stats1 stats2 step
|
|
tac tail tee top uniq
|
|
|
|
Functions for the filter and put verbs:
|
|
+ + - - * / // % ** | ^ & ~ << >> == != =~ !=~ > >= < <= && || ^^ ! ? : .
|
|
gsub strlen sub substr tolower toupper abs acos acosh asin asinh atan atan2
|
|
atanh cbrt ceil cos cosh erf erfc exp expm1 floor invqnorm log log10 log1p
|
|
logifit madd max mexp min mmul msub pow qnorm round roundm sgn sin sinh sqrt
|
|
tan tanh urand urand32 urandint dhms2fsec dhms2sec fsec2dhms fsec2hms
|
|
gmt2sec hms2fsec hms2sec sec2dhms sec2gmt sec2gmtdate sec2hms strftime
|
|
strptime systime isabsent isbool isboolean isempty isemptymap isfloat isint
|
|
ismap isnonemptymap isnotempty isnotnull isnull isnumeric ispresent isscalar
|
|
isstring assert_notnull assert_present assert_empty assert_notempty
|
|
assert_numeric assert_int assert_float assert_bool assert_boolean
|
|
assert_string boolean float fmtnum hexfmt int string typeof depth haskey
|
|
joink joinkv joinv leafcount length mapdiff mapsum splitkv splitnv
|
|
Please use "mlr --help-function {function name}" for function-specific help.
|
|
Please use "mlr --help-all-functions" or "mlr -f" for help on all functions.
|
|
Please use "mlr --help-all-keywords" or "mlr -k" for help on all keywords.
|
|
|
|
Data-format options, for input, output, or both:
|
|
--idkvp --odkvp --dkvp Delimited key-value pairs, e.g "a=1,b=2"
|
|
(this is Miller's default format).
|
|
|
|
--inidx --onidx --nidx Implicitly-integer-indexed fields
|
|
(Unix-toolkit style).
|
|
|
|
--icsv --ocsv --csv Comma-separated value (or tab-separated
|
|
with --fs tab, etc.)
|
|
|
|
--itsv --otsv --tsv Keystroke-savers for "--icsv --ifs tab",
|
|
"--ocsv --ofs tab", "--csv --fs tab".
|
|
|
|
--ipprint --opprint --pprint Pretty-printed tabular (produces no
|
|
output until all input is in).
|
|
--right Right-justifies all fields for PPRINT output.
|
|
--barred Prints a border around PPRINT output.
|
|
|
|
--omd Markdown-tabular (only available for output).
|
|
|
|
--ixtab --oxtab --xtab Pretty-printed vertical-tabular.
|
|
--xvright Right-justifies values for XTAB format.
|
|
|
|
--ijson --ojson --json JSON tabular: sequence or list of one-level
|
|
maps: {...}{...} or [{...},{...}].
|
|
--jvstack Put one key-value pair per line for JSON
|
|
output.
|
|
--jlistwrap Wrap JSON output in outermost [ ].
|
|
--jquoteall Quote map keys in JSON output, even if they're
|
|
numeric.
|
|
--jflatsep {string} Separator for flattening multi-level JSON keys,
|
|
e.g. '{"a":{"b":3}}' becomes a:b => 3 for
|
|
non-JSON formats. Defaults to :.
|
|
|
|
-p is a keystroke-saver for --nidx --fs space --repifs
|
|
|
|
Examples: --csv for CSV-formatted input and output; --idkvp --opprint for
|
|
DKVP-formatted input and pretty-printed output.
|
|
|
|
PLEASE USE "mlr --csv --rs lf" FOR NATIVE UN*X (LINEFEED-TERMINATED) CSV FILES.
|
|
You can also have MLR_CSV_DEFAULT_RS=lf in your shell environment, e.g.
|
|
"export MLR_CSV_DEFAULT_RS=lf" or "setenv MLR_CSV_DEFAULT_RS lf" depending on
|
|
which shell you use.
|
|
|
|
As keystroke-savers for format-conversion you may use the following:
|
|
--c2t --c2d --c2n --c2j --c2x --c2p --c2m
|
|
--t2c --t2d --t2n --t2j --t2x --t2p --t2m
|
|
--d2c --d2t --d2n --d2j --d2x --d2p --d2m
|
|
--n2c --n2t --n2d --n2j --n2x --n2p --n2m
|
|
--j2c --j2t --j2d --j2n --j2x --j2p --j2m
|
|
--x2c --x2t --x2d --x2n --x2j --x2p --x2m
|
|
--p2c --p2t --p2d --p2n --p2j --p2x --p2m
|
|
The letters c t d n j x p m refer to formats CSV with LF, TSV with LF, DKVP,
|
|
NIDX, JSON, XTAB, PPRINT, and markdown, respectively. Note that markdown format
|
|
is available for output only.
|
|
|
|
Compressed-data options:
|
|
--prepipe {command} This allows Miller to handle compressed inputs. You can do
|
|
without this for single input files, e.g. "gunzip < myfile.csv.gz | mlr ...".
|
|
However, when multiple input files are present, between-file separations are
|
|
lost; also, the FILENAME variable doesn't iterate. Using --prepipe you can
|
|
specify an action to be taken on each input file. This pre-pipe command must
|
|
be able to read from standard input; it will be invoked with
|
|
{command} < {filename}.
|
|
Examples:
|
|
mlr --prepipe 'gunzip'
|
|
mlr --prepipe 'zcat -cf'
|
|
mlr --prepipe 'xz -cd'
|
|
mlr --prepipe cat
|
|
Note that this feature is quite general and is not limited to decompression
|
|
utilities. You can use it to apply per-file filters of your choice.
|
|
For output compression (or other) utilities, simply pipe the output:
|
|
mlr ... | {your compression command}
|
|
|
|
Separator options, for input, output, or both:
|
|
--rs --irs --ors Record separators, e.g. 'lf' or '\r\n'
|
|
--fs --ifs --ofs --repifs Field separators, e.g. comma
|
|
--ps --ips --ops Pair separators, e.g. equals sign
|
|
Notes:
|
|
* IPS/OPS are only used for DKVP and XTAB formats, since only in these formats
|
|
do key-value pairs appear juxtaposed.
|
|
* IRS/ORS are ignored for XTAB format. Nominally IFS and OFS are newlines;
|
|
XTAB records are separated by two or more consecutive IFS/OFS -- i.e.
|
|
a blank line.
|
|
* OFS must be single-character for PPRINT format. This is because it is used
|
|
with repetition for alignment; multi-character separators would make
|
|
alignment impossible.
|
|
* OPS may be multi-character for XTAB format, in which case alignment is
|
|
disabled.
|
|
* DKVP, NIDX, CSVLITE, PPRINT, and XTAB formats are intended to handle
|
|
platform-native text data. In particular, this means LF line-terminators
|
|
by default on Linux/OSX. You can use "--dkvp --rs crlf" for
|
|
CRLF-terminated DKVP files, and so on.
|
|
* CSV is intended to handle RFC-4180-compliant data. In particular, this means
|
|
it uses CRLF line-terminators by default. You can use "--csv --rs lf" for
|
|
Linux-native CSV files. You can also have "MLR_CSV_DEFAULT_RS=lf" in your
|
|
shell environment, e.g. "export MLR_CSV_DEFAULT_RS=lf" or "setenv
|
|
MLR_CSV_DEFAULT_RS lf" depending on which shell you use.
|
|
* TSV is simply CSV using tab as field separator ("--fs tab").
|
|
* FS/PS are ignored for markdown format; RS is used.
|
|
* All RS/FS/PS options are ignored for JSON format: JSON doesn't allow
|
|
changing these.
|
|
* You can specify separators in any of the following ways, shown by example:
|
|
- Type them out, quoting as necessary for shell escapes, e.g.
|
|
"--fs '|' --ips :"
|
|
- C-style escape sequences, e.g. "--rs '\r\n' --fs '\t'".
|
|
- To avoid backslashing, you can use any of the following names:
|
|
cr crcr newline lf lflf crlf crlfcrlf tab space comma pipe slash colon semicolon equals
|
|
* Default separators by format:
|
|
File format RS FS PS
|
|
dkvp \n , =
|
|
json (N/A) (N/A) (N/A)
|
|
nidx \n space (N/A)
|
|
csv \r\n , (N/A)
|
|
csvlite \n , (N/A)
|
|
markdown \n (N/A) (N/A)
|
|
pprint \n space (N/A)
|
|
xtab (N/A) \n space
|
|
|
|
Relevant to CSV/CSV-lite input only:
|
|
--implicit-csv-header Use 1,2,3,... as field labels, rather than from line 1
|
|
of input files. Tip: combine with "label" to recreate
|
|
missing headers.
|
|
--headerless-csv-output Print only CSV data lines.
|
|
|
|
Double-quoting for CSV output:
|
|
--quote-all Wrap all fields in double quotes
|
|
--quote-none Do not wrap any fields in double quotes, even if they have
|
|
OFS or ORS in them
|
|
--quote-minimal Wrap fields in double quotes only if they have OFS or ORS
|
|
in them (default)
|
|
--quote-numeric Wrap fields in double quotes only if they have numbers
|
|
in them
|
|
--quote-original Wrap fields in double quotes if and only if they were
|
|
quoted on input. This isn't sticky for computed fields:
|
|
e.g. if fields a and b were quoted on input and you do
|
|
"put '$c = $a . $b'" then field c won't inherit a or b's
|
|
was-quoted-on-input flag.
|
|
|
|
Numerical formatting:
|
|
--ofmt {format} E.g. %.18lf, %.0lf. Please use sprintf-style codes for
|
|
double-precision. Applies to verbs which compute new
|
|
values, e.g. put, stats1, stats2. See also the fmtnum
|
|
function within mlr put (mlr --help-all-functions).
|
|
Defaults to %lf.
|
|
|
|
Other options:
|
|
--seed {n} with n of the form 12345678 or 0xcafefeed. For put/filter
|
|
urand()/urandint()/urand32().
|
|
--nr-progress-mod {m}, with m a positive integer: print filename and record
|
|
count to stderr every m input records.
|
|
--from {filename} Use this to specify an input file before the verb(s),
|
|
rather than after. May be used more than once. Example:
|
|
"mlr --from a.dat --from b.dat cat" is the same as
|
|
"mlr cat a.dat b.dat".
|
|
-n Process no input files, nor standard input either. Useful
|
|
for mlr put with begin/end statements only. (Same as --from
|
|
/dev/null.) Also useful in "mlr -n put -v '...'" for
|
|
analyzing abstract syntax trees (if that's your thing).
|
|
|
|
Then-chaining:
|
|
Output of one verb may be chained as input to another using "then", e.g.
|
|
mlr stats1 -a min,mean,max -f flag,u,v -g color then sort -f color
|
|
|
|
For more information please see http://johnkerl.org/miller/doc and/or
|
|
http://github.com/johnkerl/miller. This is Miller version v5.0.0-alpha.
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr sort --help
|
|
Usage: mlr sort {flags}
|
|
Flags:
|
|
-f {comma-separated field names} Lexical ascending
|
|
-n {comma-separated field names} Numerical ascending; nulls sort last
|
|
-nf {comma-separated field names} Numerical ascending; nulls sort last
|
|
-r {comma-separated field names} Lexical descending
|
|
-nr {comma-separated field names} Numerical descending; nulls sort first
|
|
Sorts records primarily by the first specified field, secondarily by the second
|
|
field, and so on. Any records not having all specified sort keys will appear
|
|
at the end of the output, in the order they were encountered, regardless of the
|
|
specified sort order.
|
|
Example:
|
|
mlr sort -f a,b -nr x,y,z
|
|
which is the same as:
|
|
mlr sort -f a -f b -nr x -nr y -nr z
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="I/O_options"/><h1>I/O options</h1>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="Formats"/><h2>Formats</h2>
|
|
|
|
<p/> Options:
|
|
|
|
<pre>
|
|
--dkvp --idkvp --odkvp
|
|
--nidx --inidx --onidx
|
|
--csv --icsv --ocsv
|
|
--csvlite --icsvlite --ocsvlite
|
|
--pprint --ipprint --opprint --right
|
|
--xtab --ixtab --oxtab
|
|
--json --ijson --ojson
|
|
</pre>
|
|
|
|
<p/> These are as discussed in <a href="file-formats.html">File formats</a>, with the exception of <tt>--right</tt>
|
|
which makes pretty-printed output right-aligned:
|
|
|
|
<table><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint cat data/small
|
|
a b i x y
|
|
pan pan 1 0.3467901443380824 0.7268028627434533
|
|
eks pan 2 0.7586799647899636 0.5221511083334797
|
|
wye wye 3 0.20460330576630303 0.33831852551664776
|
|
eks wye 4 0.38139939387114097 0.13418874328430463
|
|
wye pan 5 0.5732889198020006 0.8636244699032729
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint --right cat data/small
|
|
a b i x y
|
|
pan pan 1 0.3467901443380824 0.7268028627434533
|
|
eks pan 2 0.7586799647899636 0.5221511083334797
|
|
wye wye 3 0.20460330576630303 0.33831852551664776
|
|
eks wye 4 0.38139939387114097 0.13418874328430463
|
|
wye pan 5 0.5732889198020006 0.8636244699032729
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr></table>
|
|
|
|
<p/>Additional notes:
|
|
|
|
<ul>
|
|
|
|
<li/> Use <tt>--csv</tt>, <tt>--pprint</tt>, etc. when the input and output formats are the same.
|
|
|
|
<li/> Use <tt>--icsv --opprint</tt>, etc. when you want format conversion as part of what Miller does to your data.
|
|
|
|
<li/> DKVP (key-value-pair) format is the default for input and output. So,
|
|
<tt>--oxtab</tt> is the same as <tt>--idkvp --oxtab</tt>.
|
|
|
|
</ul>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="Compression"/><h2>Compression</h2>
|
|
|
|
<p/> Options:
|
|
|
|
<pre>
|
|
--prepipe {command}
|
|
</pre>
|
|
|
|
<p/>The prepipe command is anything which reads from standard input and produces data acceptable to
|
|
Miller. Nominally this allows you to use whichever decompression utilities you have installed on your
|
|
system, on a per-file basis. If the command has flags, quote them: e.g. <tt>mlr --prepipe 'zcat -cf'</tt>. Examples:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
# These two produce the same output:
|
|
$ gunzip < myfile1.csv.gz | mlr cut -f hostname,uptime
|
|
$ mlr --prepipe gunzip cut -f hostname,uptime myfile1.csv.gz
|
|
# With multiple input files you need --prepipe:
|
|
$ mlr --prepipe gunzip cut -f hostname,uptime myfile1.csv.gz myfile2.csv.gz
|
|
$ mlr --prepipe gunzip --idkvp --oxtab cut -f hostname,uptime myfile1.dat.gz myfile2.dat.gz
|
|
|
|
# Similar to the above, but with compressed output as well as input:
|
|
$ gunzip < myfile1.csv.gz | mlr cut -f hostname,uptime | gzip > outfile.csv.gz
|
|
$ mlr --prepipe gunzip cut -f hostname,uptime myfile1.csv.gz | gzip > outfile.csv.gz
|
|
$ mlr --prepipe gunzip cut -f hostname,uptime myfile1.csv.gz myfile2.csv.gz | gzip > outfile.csv.gz
|
|
|
|
# Similar to the above, but with different compression tools for input and output:
|
|
$ gunzip < myfile1.csv.gz | mlr cut -f hostname,uptime | xz -z > outfile.csv.xz
|
|
$ xz -cd < myfile1.csv.xz | mlr cut -f hostname,uptime | gzip > outfile.csv.xz
|
|
$ mlr --prepipe 'xz -cd' cut -f hostname,uptime myfile1.csv.xz myfile2.csv.xz | xz -z > outfile.csv.xz
|
|
|
|
... etc.
|
|
</pre>
|
|
</div>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="Record/field/pair_separators"/><h2>Record/field/pair separators</h2>
|
|
|
|
<p/> Miller has record separators <tt>IRS</tt> and <tt>ORS</tt>, field
|
|
separators <tt>IFS</tt> and <tt>OFS</tt>, and pair separators <tt>IPS</tt> and
|
|
<tt>OPS</tt>. For example, in the DKVP line <tt>a=1,b=2,c=3</tt>, the record
|
|
separator is newline, field separator is comma, and pair separator is the
|
|
equals sign. These are the default values.
|
|
|
|
<p/> Options:
|
|
<pre>
|
|
--rs --irs --ors
|
|
--fs --ifs --ofs --repifs
|
|
--ps --ips --ops
|
|
</pre>
|
|
|
|
<ul>
|
|
|
|
<li/> You can change a separator from input to output via e.g. <tt>--ifs =
|
|
--ofs :</tt>. Or, you can specify that the same separator is to be used for
|
|
input and output via e.g. <tt>--fs :</tt>.
|
|
|
|
<li/> The pair separator is only relevant to DKVP format.
|
|
|
|
<li/> Pretty-print and xtab formats ignore the separator arguments altogether.
|
|
|
|
<li/> The <tt>--repifs</tt> means that multiple successive occurrences of the
|
|
field separator count as one. For example, in CSV data we often signify nulls
|
|
by empty strings, e.g. <tt>2,9,,,,,6,5,4</tt>. On the other hand, if the field
|
|
separator is a space, it might be more natural to parse <tt>2 4 5</tt> the
|
|
same as <tt>2 4 5</tt>: <tt>--repifs --ifs ' '</tt> lets this happen. In fact,
|
|
the <tt>--ipprint</tt> option above is internally implemented in terms of
|
|
<tt>--repifs</tt>.
|
|
|
|
<li/> Just write out the desired separator, e.g. <tt>--ofs '|'</tt>. But you
|
|
may use the symbolic names <tt>newline</tt>, <tt>space</tt>, <tt>tab</tt>,
|
|
<tt>pipe</tt>, or <tt>semicolon</tt> if you like.
|
|
|
|
</ul>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="Number_formatting"/><h2>Number formatting</h2>
|
|
|
|
<p/> The command-line option <tt>--ofmt {format string}</tt> is the global
|
|
number format for commands which generate numeric output, e.g.
|
|
<tt>stats1</tt>, <tt>stats2</tt>, <tt>histogram</tt>, and <tt>step</tt>, as
|
|
well as <tt>mlr put</tt>. Examples:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
--ofmt %.9le --ofmt %.6lf --ofmt %.0lf
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/> These are just C <tt>printf</tt> formats applied to double-precision
|
|
numbers. Please don’t use <tt>%s</tt> or <tt>%d</tt>. Additionally, if
|
|
you use leading width (e.g. <tt>%18.12lf</tt>) then the output will contain
|
|
embedded whitespace, which may not be what you want if you pipe the output to
|
|
something else, particularly CSV. I use Miller’s pretty-print format
|
|
(<tt>mlr --opprint</tt>) to column-align numerical data.
|
|
|
|
<p/> To apply formatting to a single field, overriding the global
|
|
<tt>ofmt</tt>, use <tt>fmtnum</tt> function within <tt>mlr put</tt>. For example:
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ echo 'x=3.1,y=4.3' | mlr put '$z=fmtnum($x*$y,"%08lf")'
|
|
x=3.1,y=4.3,z=13.330000
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ echo 'x=0xffff,y=0xff' | mlr put '$z=fmtnum(int($x*$y),"%08llx")'
|
|
x=0xffff,y=0xff,z=00feff01
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>Input conversion from hexadecimal is done automatically on fields handled
|
|
by <tt>mlr put</tt> and <tt>mlr filter</tt> as long as the field value begins
|
|
with "0x". To apply output conversion to hexadecimal on a single column, you
|
|
may use <tt>fmtnum</tt>, or the keystroke-saving <tt>hexfmt</tt> function.
|
|
Example:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ echo 'x=0xffff,y=0xff' | mlr put '$z=hexfmt($x*$y)'
|
|
x=0xffff,y=0xff,z=0xfeff01
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="Data_transformations"/><h1>Data transformations</h1>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="bar"/><h2>bar</h2>
|
|
|
|
<p/> Cheesy bar-charting.
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr bar -h
|
|
Usage: mlr bar [options]
|
|
Replaces a numeric field with a number of asterisks, allowing for cheesy
|
|
bar plots. These align best with --opprint or --oxtab output format.
|
|
Options:
|
|
-f {a,b,c} Field names to convert to bars.
|
|
-c {character} Fill character: default '*'.
|
|
-x {character} Out-of-bounds character: default '#'.
|
|
-b {character} Blank character: default '.'.
|
|
--lo {lo} Lower-limit value for min-width bar: default '0.000000'.
|
|
--hi {hi} Upper-limit value for max-width bar: default '100.000000'.
|
|
-w {n} Bar-field width: default '40'.
|
|
--auto Automatically computes limits, ignoring --lo and --hi.
|
|
Holds all records in memory before producing any output.
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint cat data/small
|
|
a b i x y
|
|
pan pan 1 0.3467901443380824 0.7268028627434533
|
|
eks pan 2 0.7586799647899636 0.5221511083334797
|
|
wye wye 3 0.20460330576630303 0.33831852551664776
|
|
eks wye 4 0.38139939387114097 0.13418874328430463
|
|
wye pan 5 0.5732889198020006 0.8636244699032729
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint bar --lo 0 --hi 1 -f x,y data/small
|
|
a b i x y
|
|
pan pan 1 *************........................... *****************************...........
|
|
eks pan 2 ******************************.......... ********************....................
|
|
wye wye 3 ********................................ *************...........................
|
|
eks wye 4 ***************......................... *****...................................
|
|
wye pan 5 **********************.................. **********************************......
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint bar --lo 0.4 --hi 0.6 -f x,y data/small
|
|
a b i x y
|
|
pan pan 1 #....................................... ***************************************#
|
|
eks pan 2 ***************************************# ************************................
|
|
wye wye 3 #....................................... #.......................................
|
|
eks wye 4 #....................................... #.......................................
|
|
wye pan 5 **********************************...... ***************************************#
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint bar --auto -f x,y data/small
|
|
a b i x y
|
|
pan pan 1 [0.204603]**********..............................[0.75868] [0.134189]********************************........[0.863624]
|
|
eks pan 2 [0.204603]***************************************#[0.75868] [0.134189]*********************...................[0.863624]
|
|
wye wye 3 [0.204603]#.......................................[0.75868] [0.134189]***********.............................[0.863624]
|
|
eks wye 4 [0.204603]************............................[0.75868] [0.134189]#.......................................[0.863624]
|
|
wye pan 5 [0.204603]**************************..............[0.75868] [0.134189]***************************************#[0.863624]
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="bootstrap"/><h2>bootstrap</h2>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr bootstrap --help
|
|
Usage: mlr bootstrap [options]
|
|
Emits an n-sample, with replacement, of the input records.
|
|
Options:
|
|
-n {number} Number of samples to output. Defaults to number of input records.
|
|
Must be non-negative.
|
|
See also mlr sample and mlr shuffle.
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/> The canonical use for bootstrap sampling is to put error bars on statistical quantities, such as mean. For example:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint stats1 -a mean,count -f u -g color data/colored-shapes.dkvp
|
|
color u_mean u_count
|
|
yellow 0.497129 1413
|
|
red 0.492560 4641
|
|
purple 0.494005 1142
|
|
green 0.504861 1109
|
|
blue 0.517717 1470
|
|
orange 0.490532 303
|
|
|
|
$ mlr --opprint bootstrap then stats1 -a mean,count -f u -g color data/colored-shapes.dkvp
|
|
color u_mean u_count
|
|
yellow 0.500651 1380
|
|
purple 0.501556 1111
|
|
green 0.503272 1068
|
|
red 0.493895 4702
|
|
blue 0.512529 1496
|
|
orange 0.521030 321
|
|
|
|
$ mlr --opprint bootstrap then stats1 -a mean,count -f u -g color data/colored-shapes.dkvp
|
|
color u_mean u_count
|
|
yellow 0.498046 1485
|
|
blue 0.513576 1417
|
|
red 0.492870 4595
|
|
orange 0.507697 307
|
|
green 0.496803 1075
|
|
purple 0.486337 1199
|
|
|
|
$ mlr --opprint bootstrap then stats1 -a mean,count -f u -g color data/colored-shapes.dkvp
|
|
color u_mean u_count
|
|
blue 0.522921 1447
|
|
red 0.490717 4617
|
|
yellow 0.496450 1419
|
|
purple 0.496523 1192
|
|
green 0.507569 1111
|
|
orange 0.468014 292
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="cat"/><h2>cat</h2>
|
|
|
|
<p/> Most useful for format conversions (see
|
|
<a href="file-formats.html">File formats</a>), and concatenating multiple
|
|
same-schema CSV files to have the same header:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr cat -h
|
|
Usage: mlr cat [options]
|
|
Passes input records directly to output. Most useful for format conversion.
|
|
Options:
|
|
-n Prepend field "n" to each record with record-counter starting at 1
|
|
-N {name} Prepend field {name} to each record with record-counter starting at 1
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<table><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ cat data/a.csv
|
|
a,b,c
|
|
1,2,3
|
|
4,5,6
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td> <td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ cat data/b.csv
|
|
a,b,c
|
|
7,8,9
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td> <td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --csv cat data/a.csv data/b.csv
|
|
a,b,c
|
|
1,2,3
|
|
4,5,6
|
|
7,8,9
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr></table>
|
|
|
|
<table><tr><td>
|
|
</td> <td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --icsv --oxtab cat data/a.csv data/b.csv
|
|
a 1
|
|
b 2
|
|
c 3
|
|
|
|
a 4
|
|
b 5
|
|
c 6
|
|
|
|
a 7
|
|
b 8
|
|
c 9
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td> <td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --csv cat -n data/a.csv data/b.csv
|
|
n,a,b,c
|
|
1,1,2,3
|
|
2,4,5,6
|
|
3,7,8,9
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr></table>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="check"/><h2>check</h2>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr check --help
|
|
Usage: mlr check
|
|
Consumes records without printing any output.
|
|
Useful for doing a well-formatted check on input data.
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="count-distinct"/><h2>count-distinct</h2>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr count-distinct --help
|
|
Usage: mlr count-distinct [options]
|
|
-f {a,b,c} Field names for distinct count.
|
|
-n Show only the number of distinct values.
|
|
Prints number of records having distinct values for specified field names.
|
|
Same as uniq -c.
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr count-distinct -f a,b then sort -nr count data/medium
|
|
a=zee,b=wye,count=455
|
|
a=pan,b=eks,count=429
|
|
a=pan,b=pan,count=427
|
|
a=wye,b=hat,count=426
|
|
a=hat,b=wye,count=423
|
|
a=pan,b=hat,count=417
|
|
a=eks,b=hat,count=417
|
|
a=eks,b=eks,count=413
|
|
a=pan,b=zee,count=413
|
|
a=zee,b=hat,count=409
|
|
a=eks,b=wye,count=407
|
|
a=zee,b=zee,count=403
|
|
a=pan,b=wye,count=395
|
|
a=wye,b=pan,count=392
|
|
a=zee,b=eks,count=391
|
|
a=zee,b=pan,count=389
|
|
a=hat,b=eks,count=389
|
|
a=wye,b=eks,count=386
|
|
a=hat,b=zee,count=385
|
|
a=wye,b=zee,count=385
|
|
a=hat,b=hat,count=381
|
|
a=wye,b=wye,count=377
|
|
a=eks,b=pan,count=371
|
|
a=hat,b=pan,count=363
|
|
a=eks,b=zee,count=357
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr count-distinct -n -f a,b data/medium
|
|
count=25
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="cut"/><h2>cut</h2>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr cut --help
|
|
Usage: mlr cut [options]
|
|
Passes through input records with specified fields included/excluded.
|
|
-f {a,b,c} Field names to include for cut.
|
|
-o Retain fields in the order specified here in the argument list.
|
|
Default is to retain them in the order found in the input data.
|
|
-x|--complement Exclude, rather than include, field names specified by -f.
|
|
-r Treat field names as regular expressions. "ab", "a.*b" will
|
|
match any field name containing the substring "ab" or matching
|
|
"a.*b", respectively; anchors of the form "^ab$", "^a.*b$" may
|
|
be used. The -o flag is ignored when -r is present.
|
|
Examples:
|
|
mlr cut -f hostname,status
|
|
mlr cut -x -f hostname,status
|
|
mlr cut -r -f '^status$,sda[0-9]'
|
|
mlr cut -r -f '^status$,"sda[0-9]"'
|
|
mlr cut -r -f '^status$,"sda[0-9]"i' (this is case-insensitive)
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<table><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint cat data/small
|
|
a b i x y
|
|
pan pan 1 0.3467901443380824 0.7268028627434533
|
|
eks pan 2 0.7586799647899636 0.5221511083334797
|
|
wye wye 3 0.20460330576630303 0.33831852551664776
|
|
eks wye 4 0.38139939387114097 0.13418874328430463
|
|
wye pan 5 0.5732889198020006 0.8636244699032729
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint cut -f y,x,i data/small
|
|
i x y
|
|
1 0.3467901443380824 0.7268028627434533
|
|
2 0.7586799647899636 0.5221511083334797
|
|
3 0.20460330576630303 0.33831852551664776
|
|
4 0.38139939387114097 0.13418874328430463
|
|
5 0.5732889198020006 0.8636244699032729
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ echo 'a=1,b=2,c=3' | mlr cut -f b,c,a
|
|
a=1,b=2,c=3
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ echo 'a=1,b=2,c=3' | mlr cut -o -f b,c,a
|
|
b=2,c=3,a=1
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr></table>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="decimate"/><h2>decimate</h2>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr decimate --help
|
|
Usage: mlr decimate [options]
|
|
-n {count} Decimation factor; default 10
|
|
-b Decimate by printing first of every n.
|
|
-e Decimate by printing last of every n (default).
|
|
-g {a,b,c} Optional group-by-field names for decimate counts
|
|
Passes through one of every n records, optionally by category.
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>
|
|
<!-- ================================================================ -->
|
|
<a id="filter"/><h2>filter</h2>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr filter --help
|
|
Usage: mlr filter [options] {expression}
|
|
Prints records for which {expression} evaluates to true.
|
|
If there are multiple semicolon-delimited expressions, all of them are
|
|
evaluated and the last one is used as the filter criterion.
|
|
|
|
Options:
|
|
-v: Prints the expressions's AST (abstract syntax tree), which gives
|
|
full transparency on the precedence and associativity rules of
|
|
Miller's grammar, to stdout.
|
|
-a: Prints a low-level stack-allocation trace to stdout.
|
|
-t: Prints a low-level parser trace to stderr.
|
|
-T: Prints a every statement to stderr as it is executed.
|
|
-x: Prints records for which {expression} evaluates to false.
|
|
-S: Keeps field values, or literals in the expression, as strings with no type
|
|
inference to int or float.
|
|
-F: Keeps field values, or literals in the expression, as strings or floats
|
|
with no inference to int.
|
|
--oflatsep {string}: Separator to use when flattening multi-level @-variables
|
|
to output records for emit. Default ":".
|
|
-f {filename}: the DSL expression is taken from the specified file rather
|
|
than from the command line. Outer single quotes wrapping the expression
|
|
should not be placed in the file. If -f is specified more than once,
|
|
all input files specified using -f are concatenated to produce the expression.
|
|
(For example, you can define functions in one file and call them from another.)
|
|
-e {expression}: You can use this after -f to add an expression. Example use
|
|
case: define functions/subroutines in a file you specify with -f, then call
|
|
them with an expression you specify with -e.
|
|
(If you mix -e and -f then the expressions are evaluated in the order encountered.
|
|
Since the expression pieces are simply concatenated, please be sure to use intervening
|
|
semicolons to separate expressions.)
|
|
--no-fflush: for emit, tee, print, and dump, don't call fflush() after every
|
|
record.
|
|
Any of the output-format command-line flags (see mlr -h). Example: using
|
|
mlr --icsv --opprint ... then put --ojson 'tee > "mytap-".$a.".dat", $*' then ...
|
|
the input is CSV, the output is pretty-print tabular, but the tee-file output
|
|
is written in JSON format.
|
|
|
|
Please use a dollar sign for field names and double-quotes for string
|
|
literals. If field names have special characters such as "." then you might
|
|
use braces, e.g. '${field.name}'. Miller built-in variables are
|
|
NF NR FNR FILENUM FILENAME PI E, and ENV["namegoeshere"] to access environment
|
|
variables. The environment-variable name may be an expression, e.g. a field
|
|
value.
|
|
|
|
Use # to comment to end of line.
|
|
|
|
Examples:
|
|
mlr filter 'log10($count) > 4.0'
|
|
mlr filter 'FNR == 2 (second record in each file)'
|
|
mlr filter 'urand() < 0.001' (subsampling)
|
|
mlr filter '$color != "blue" && $value > 4.2'
|
|
mlr filter '($x<.5 && $y<.5) || ($x>.5 && $y>.5)'
|
|
mlr filter '($name =~ "^sys.*east$") || ($name =~ "^dev.[0-9]+"i)'
|
|
mlr filter '$ab = $a+$b; $cd = $c+$d; $ab != $cd'
|
|
mlr filter '
|
|
NR == 1 ||
|
|
#NR == 2 ||
|
|
NR == 3
|
|
'
|
|
|
|
Please see http://johnkerl.org/miller/doc/reference.html for more information
|
|
including function list. Or "mlr -f". Please also also "mlr grep" which is
|
|
useful when you don't yet know which field name(s) you're looking for.
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<a id="Features_which_filter_shares_with_put"/><h3>Features which filter shares with put</h3>
|
|
|
|
<p/>Please see <a href="#Expression_language_for_filter_and_put">Expression
|
|
language for filter and put</a> for more information about the expression
|
|
language for <tt>mlr filter</tt>.
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="grep"/><h2>grep</h2>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr grep -h
|
|
Usage: mlr grep [options] {regular expression}
|
|
Passes through records which match {regex}.
|
|
Options:
|
|
-i Use case-insensitive search.
|
|
-v Invert: pass through records which do not match the regex.
|
|
Note that "mlr filter" is more powerful, but requires you to know field names.
|
|
By contrast, "mlr grep" allows you to regex-match the entire record. It does
|
|
this by formatting each record in memory as DKVP, using command-line-specified
|
|
ORS/OFS/OPS, and matching the resulting line against the regex specified
|
|
here. In particular, the regex is not applied to the input stream: if you
|
|
have CSV with header line "x,y,z" and data line "1,2,3" then the regex will
|
|
be matched, not against either of these lines, but against the DKVP line
|
|
"x=1,y=2,z=3". Furthermore, not all the options to system grep are supported,
|
|
and this command is intended to be merely a keystroke-saver. To get all the
|
|
features of system grep, you can do
|
|
"mlr --odkvp ... | grep ... | mlr --idkvp ..."
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="group-by"/><h2>group-by</h2>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr group-by --help
|
|
Usage: mlr group-by {comma-separated field names}
|
|
Outputs records in batches having identical values at specified field names.
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>This is similar to <tt>sort</tt> but with less work. Namely, Miller’s
|
|
sort has three steps: read through the data and append linked lists of records,
|
|
one for each unique combination of the key-field values; after all records
|
|
are read, sort the key-field values; then print each record-list. The group-by
|
|
operation simply omits the middle sort. An example should make this more
|
|
clear.
|
|
|
|
<table><tr> <td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint group-by a data/small
|
|
a b i x y
|
|
pan pan 1 0.3467901443380824 0.7268028627434533
|
|
eks pan 2 0.7586799647899636 0.5221511083334797
|
|
eks wye 4 0.38139939387114097 0.13418874328430463
|
|
wye wye 3 0.20460330576630303 0.33831852551664776
|
|
wye pan 5 0.5732889198020006 0.8636244699032729
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td> <td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint sort -f a data/small
|
|
a b i x y
|
|
eks pan 2 0.7586799647899636 0.5221511083334797
|
|
eks wye 4 0.38139939387114097 0.13418874328430463
|
|
pan pan 1 0.3467901443380824 0.7268028627434533
|
|
wye wye 3 0.20460330576630303 0.33831852551664776
|
|
wye pan 5 0.5732889198020006 0.8636244699032729
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td> </tr></table>
|
|
|
|
<p/>In this example, since the sort is on field <tt>a</tt>, the first step is
|
|
to group together all records having the same value for field <tt>a</tt>; the
|
|
second step is to sort the distinct <tt>a</tt>-field values <tt>pan</tt>,
|
|
<tt>eks</tt>, and <tt>wye</tt> into <tt>eks</tt>, <tt>pan</tt>, and
|
|
<tt>wye</tt>; the third step is to print out the record-list for
|
|
<tt>a=eks</tt>, then the record-list for <tt>a=pan</tt>, then the record-list
|
|
for <tt>a=wye</tt>. The group-by operation omits the middle sort and just puts
|
|
like records together, for those times when a sort isn’t desired. In
|
|
particular, the ordering of group-by fields for group-by is the order in which
|
|
they were encountered in the data stream, which in some cases may be more interesting
|
|
to you.
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="group-like"/><h2>group-like</h2>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr group-like --help
|
|
Usage: mlr group-like
|
|
Outputs records in batches having identical field names.
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/> This groups together records having the same schema (i.e. same ordered list of field names)
|
|
which is useful for making sense of time-ordered output as described in
|
|
<a href="record-heterogeneity.html">Record-heterogeneity</a> — in particular, in
|
|
preparation for CSV or pretty-print output.
|
|
|
|
<table><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr cat data/het.dkvp
|
|
resource=/path/to/file,loadsec=0.45,ok=true
|
|
record_count=100,resource=/path/to/file
|
|
resource=/path/to/second/file,loadsec=0.32,ok=true
|
|
record_count=150,resource=/path/to/second/file
|
|
resource=/some/other/path,loadsec=0.97,ok=false
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint group-like data/het.dkvp
|
|
resource loadsec ok
|
|
/path/to/file 0.45 true
|
|
/path/to/second/file 0.32 true
|
|
/some/other/path 0.97 false
|
|
|
|
record_count resource
|
|
100 /path/to/file
|
|
150 /path/to/second/file
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr></table>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="having-fields"/><h2>having-fields</h2>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr having-fields --help
|
|
Usage: mlr having-fields [options]
|
|
Conditionally passes through records depending on each record's field names.
|
|
Options:
|
|
--at-least {comma-separated names}
|
|
--which-are {comma-separated names}
|
|
--at-most {comma-separated names}
|
|
--all-matching {regular expression}
|
|
--any-matching {regular expression}
|
|
--none-matching {regular expression}
|
|
Examples:
|
|
mlr having-fields --which-are amount,status,owner
|
|
mlr having-fields --any-matching 'sda[0-9]'
|
|
mlr having-fields --any-matching '"sda[0-9]"'
|
|
mlr having-fields --any-matching '"sda[0-9]"i' (this is case-insensitive)
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/> Similar to <a href="#group-like"><tt>group-like</tt></a>, this retains records with specified schema.
|
|
|
|
<table><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr cat data/het.dkvp
|
|
resource=/path/to/file,loadsec=0.45,ok=true
|
|
record_count=100,resource=/path/to/file
|
|
resource=/path/to/second/file,loadsec=0.32,ok=true
|
|
record_count=150,resource=/path/to/second/file
|
|
resource=/some/other/path,loadsec=0.97,ok=false
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr having-fields --at-least resource data/het.dkvp
|
|
resource=/path/to/file,loadsec=0.45,ok=true
|
|
record_count=100,resource=/path/to/file
|
|
resource=/path/to/second/file,loadsec=0.32,ok=true
|
|
record_count=150,resource=/path/to/second/file
|
|
resource=/some/other/path,loadsec=0.97,ok=false
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr having-fields --which-are resource,ok,loadsec data/het.dkvp
|
|
resource=/path/to/file,loadsec=0.45,ok=true
|
|
resource=/path/to/second/file,loadsec=0.32,ok=true
|
|
resource=/some/other/path,loadsec=0.97,ok=false
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr></table>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="head"/><h2>head</h2>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr head --help
|
|
Usage: mlr head [options]
|
|
-n {count} Head count to print; default 10
|
|
-g {a,b,c} Optional group-by-field names for head counts
|
|
Passes through the first n records, optionally by category.
|
|
Without -g, ceases consuming more input (i.e. is fast) when n
|
|
records have been read.
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
Note that <tt>head</tt> is distinct from <a href="#top"><tt>top</tt></a>
|
|
— <tt>head</tt> shows fields which appear first in the data stream;
|
|
<tt>top</tt> shows fields which are numerically largest (or smallest).
|
|
|
|
<table><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint head -n 4 data/medium
|
|
a b i x y
|
|
pan pan 1 0.3467901443380824 0.7268028627434533
|
|
eks pan 2 0.7586799647899636 0.5221511083334797
|
|
wye wye 3 0.20460330576630303 0.33831852551664776
|
|
eks wye 4 0.38139939387114097 0.13418874328430463
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint head -n 1 -g b data/medium
|
|
a b i x y
|
|
pan pan 1 0.3467901443380824 0.7268028627434533
|
|
wye wye 3 0.20460330576630303 0.33831852551664776
|
|
eks zee 7 0.6117840605678454 0.1878849191181694
|
|
zee eks 17 0.29081949506712723 0.054478717073354166
|
|
wye hat 24 0.7286126830627567 0.19441962592638418
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr></table>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="histogram"/><h2>histogram</h2>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr histogram --help
|
|
Usage: mlr histogram [options]
|
|
-f {a,b,c} Value-field names for histogram counts
|
|
--lo {lo} Histogram low value
|
|
--hi {hi} Histogram high value
|
|
--nbins {n} Number of histogram bins
|
|
--auto Automatically computes limits, ignoring --lo and --hi.
|
|
Holds all values in memory before producing any output.
|
|
Just a histogram. Input values < lo or > hi are not counted.
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
This is just a histogram; there’s not too much to say here. A note about
|
|
binning, by example: Suppose you use <tt>--lo 0.0 --hi 1.0 --nbins 10 -f
|
|
x</tt>. The input numbers less than 0 or greater than 1 aren’t counted
|
|
in any bin. Input numbers equal to 1 are counted in the last bin. That is, bin
|
|
0 has <tt>0.0 ≤ x < 0.1</tt>, bin 1 has <tt>0.1 ≤ x < 0.2</tt>,
|
|
etc., but bin 9 has <tt>0.9 ≤ x ≤ 1.0</tt>.
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint put '$x2=$x**2;$x3=$x2*$x' then histogram -f x,x2,x3 --lo 0 --hi 1 --nbins 10 data/medium
|
|
bin_lo bin_hi x_count x2_count x3_count
|
|
0.000000 0.100000 1072 3231 4661
|
|
0.100000 0.200000 938 1254 1184
|
|
0.200000 0.300000 1037 988 845
|
|
0.300000 0.400000 988 832 676
|
|
0.400000 0.500000 950 774 576
|
|
0.500000 0.600000 1002 692 476
|
|
0.600000 0.700000 1007 591 438
|
|
0.700000 0.800000 1007 560 420
|
|
0.800000 0.900000 986 571 383
|
|
0.900000 1.000000 1013 507 341
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="join"/><h2>join</h2>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr join --help
|
|
Usage: mlr join [options]
|
|
Joins records from specified left file name with records from all file names
|
|
at the end of the Miller argument list.
|
|
Functionality is essentially the same as the system "join" command, but for
|
|
record streams.
|
|
Options:
|
|
-f {left file name}
|
|
-j {a,b,c} Comma-separated join-field names for output
|
|
-l {a,b,c} Comma-separated join-field names for left input file;
|
|
defaults to -j values if omitted.
|
|
-r {a,b,c} Comma-separated join-field names for right input file(s);
|
|
defaults to -j values if omitted.
|
|
--lp {text} Additional prefix for non-join output field names from
|
|
the left file
|
|
--rp {text} Additional prefix for non-join output field names from
|
|
the right file(s)
|
|
--np Do not emit paired records
|
|
--ul Emit unpaired records from the left file
|
|
--ur Emit unpaired records from the right file(s)
|
|
-u Enable unsorted input. In this case, the entire left file will
|
|
be loaded into memory. Without -u, records must be sorted
|
|
lexically by their join-field names, else not all records will
|
|
be paired.
|
|
--prepipe {command} As in main input options; see mlr --help for details.
|
|
If you wish to use a prepipe command for the main input as well
|
|
as here, it must be specified there as well as here.
|
|
File-format options default to those for the right file names on the Miller
|
|
argument list, but may be overridden for the left file as follows. Please see
|
|
the main "mlr --help" for more information on syntax for these arguments.
|
|
-i {one of csv,dkvp,nidx,pprint,xtab}
|
|
--irs {record-separator character}
|
|
--ifs {field-separator character}
|
|
--ips {pair-separator character}
|
|
--repifs
|
|
--repips
|
|
--use-mmap
|
|
--no-mmap
|
|
Please use "mlr --usage-separator-options" for information on specifying separators.
|
|
Please see http://johnkerl.org/miller/doc/reference.html for more information
|
|
including examples.
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
Examples:
|
|
|
|
<p/>Join larger table with IDs with smaller ID-to-name lookup table, showing only paired records:
|
|
|
|
<table><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --icsvlite --opprint cat data/join-left-example.csv
|
|
id name
|
|
100 alice
|
|
200 bob
|
|
300 carol
|
|
400 david
|
|
500 edgar
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --icsvlite --opprint cat data/join-right-example.csv
|
|
status idcode
|
|
present 400
|
|
present 100
|
|
missing 200
|
|
present 100
|
|
present 200
|
|
missing 100
|
|
missing 200
|
|
present 300
|
|
missing 600
|
|
present 400
|
|
present 400
|
|
present 300
|
|
present 100
|
|
missing 400
|
|
present 200
|
|
present 200
|
|
present 200
|
|
present 200
|
|
present 400
|
|
present 300
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --icsvlite --opprint join -u -j id -r idcode -f data/join-left-example.csv data/join-right-example.csv
|
|
id name status
|
|
400 david present
|
|
100 alice present
|
|
200 bob missing
|
|
100 alice present
|
|
200 bob present
|
|
100 alice missing
|
|
200 bob missing
|
|
300 carol present
|
|
400 david present
|
|
400 david present
|
|
300 carol present
|
|
100 alice present
|
|
400 david missing
|
|
200 bob present
|
|
200 bob present
|
|
200 bob present
|
|
200 bob present
|
|
400 david present
|
|
300 carol present
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr></table>
|
|
|
|
<p/>Same, but with sorting the input first:
|
|
<table><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --icsvlite --opprint sort -f idcode then join -j id -r idcode -f data/join-left-example.csv data/join-right-example.csv
|
|
id name status
|
|
100 alice present
|
|
100 alice present
|
|
100 alice missing
|
|
100 alice present
|
|
200 bob missing
|
|
200 bob present
|
|
200 bob missing
|
|
200 bob present
|
|
200 bob present
|
|
200 bob present
|
|
200 bob present
|
|
300 carol present
|
|
300 carol present
|
|
300 carol present
|
|
400 david present
|
|
400 david present
|
|
400 david present
|
|
400 david missing
|
|
400 david present
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr></table>
|
|
|
|
<p/>Same, but showing only unpaired records:
|
|
<table><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --icsvlite --opprint join --np --ul --ur -u -j id -r idcode -f data/join-left-example.csv data/join-right-example.csv
|
|
status idcode
|
|
missing 600
|
|
|
|
id name
|
|
500 edgar
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr></table>
|
|
|
|
<p/>Use prefixing options to disambiguate between otherwise identical non-join field names:
|
|
<table><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --csvlite --opprint cat data/self-join.csv data/self-join.csv
|
|
a b c
|
|
1 2 3
|
|
1 4 5
|
|
1 2 3
|
|
1 4 5
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --csvlite --opprint join -j a --lp left_ --rp right_ -f data/self-join.csv data/self-join.csv
|
|
a left_b left_c right_b right_c
|
|
1 2 3 2 3
|
|
1 4 5 2 3
|
|
1 2 3 4 5
|
|
1 4 5 4 5
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr></table>
|
|
|
|
<p/>Use zero join columns:
|
|
<table><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --csvlite --opprint join -j "" --lp left_ --rp right_ -f data/self-join.csv data/self-join.csv
|
|
left_a left_b left_c right_a right_b right_c
|
|
1 2 3 1 2 3
|
|
1 4 5 1 2 3
|
|
1 2 3 1 4 5
|
|
1 4 5 1 4 5
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr></table>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="label"/><h2>label</h2>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr label --help
|
|
Usage: mlr label {new1,new2,new3,...}
|
|
Given n comma-separated names, renames the first n fields of each record to
|
|
have the respective name. (Fields past the nth are left with their original
|
|
names.) Particularly useful with --inidx or --implicit-csv-header, to give
|
|
useful names to otherwise integer-indexed fields.
|
|
Examples:
|
|
"echo 'a b c d' | mlr --inidx --odkvp cat" gives "1=a,2=b,3=c,4=d"
|
|
"echo 'a b c d' | mlr --inidx --odkvp label s,t" gives "s=a,t=b,3=c,4=d"
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
See also <a href="#rename"><tt>rename</tt></a>.
|
|
|
|
<p/>Example: Files such as <tt>/etc/passwd</tt>, <tt>/etc/group</tt>, and so on
|
|
have implicit field names which are found in section-5 manpages. These field names may be made explicit as follows:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
% grep -v '^#' /etc/passwd | mlr --nidx --fs : --opprint label name,password,uid,gid,gecos,home_dir,shell | head
|
|
name password uid gid gecos home_dir shell
|
|
nobody * -2 -2 Unprivileged User /var/empty /usr/bin/false
|
|
root * 0 0 System Administrator /var/root /bin/sh
|
|
daemon * 1 1 System Services /var/root /usr/bin/false
|
|
_uucp * 4 4 Unix to Unix Copy Protocol /var/spool/uucp /usr/sbin/uucico
|
|
_taskgated * 13 13 Task Gate Daemon /var/empty /usr/bin/false
|
|
_networkd * 24 24 Network Services /var/networkd /usr/bin/false
|
|
_installassistant * 25 25 Install Assistant /var/empty /usr/bin/false
|
|
_lp * 26 26 Printing Services /var/spool/cups /usr/bin/false
|
|
_postfix * 27 27 Postfix Mail Server /var/spool/postfix /usr/bin/false
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>Likewise, if you have CSV/CSV-lite input data which has somehow been bereft of its header line, you can re-add a header line using <tt>--implicit-csv-header</tt> and <tt>label</tt>:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ cat data/headerless.csv
|
|
John,23,present
|
|
Fred,34,present
|
|
Alice,56,missing
|
|
Carol,45,present
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --csv --rs lf --implicit-csv-header cat data/headerless.csv
|
|
1,2,3
|
|
John,23,present
|
|
Fred,34,present
|
|
Alice,56,missing
|
|
Carol,45,present
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --csv --rs lf --implicit-csv-header label name,age,status data/headerless.csv
|
|
name,age,status
|
|
John,23,present
|
|
Fred,34,present
|
|
Alice,56,missing
|
|
Carol,45,present
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --icsv --rs lf --implicit-csv-header --opprint label name,age,status data/headerless.csv
|
|
name age status
|
|
John 23 present
|
|
Fred 34 present
|
|
Alice 56 missing
|
|
Carol 45 present
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="least-frequent"/><h2>least-frequent</h2>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr least-frequent -h
|
|
Usage: mlr least-frequent [options]
|
|
Shows the least frequently occurring distinct values for specified field names.
|
|
The first entry is the statistical anti-mode; the remaining are runners-up.
|
|
Options:
|
|
-f {one or more comma-separated field names}. Required flag.
|
|
-n {count}. Optional flag defaulting to 10.
|
|
-b Suppress counts; show only field values.
|
|
See also "mlr most".
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint --from data/colored-shapes.dkvp least-frequent -f shape -n 5
|
|
shape count
|
|
circle 2591
|
|
triangle 3372
|
|
square 4115
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint --from data/colored-shapes.dkvp least-frequent -f shape,color -n 5
|
|
shape color count
|
|
circle orange 68
|
|
triangle orange 107
|
|
square orange 128
|
|
circle green 287
|
|
circle purple 289
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint --from data/colored-shapes.dkvp least-frequent -f shape,color -n 5 -b
|
|
shape color
|
|
circle orange
|
|
triangle orange
|
|
square orange
|
|
circle green
|
|
circle purple
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
See also <a href="#most-frequent">most-frequent</a>.
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="merge-fields"/><h2>merge-fields</h2>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr merge-fields --help
|
|
Usage: mlr merge-fields [options]
|
|
Computes univariate statistics for each input record, accumulated across
|
|
specified fields.
|
|
Options:
|
|
-a {sum,count,...} Names of accumulators. One or more of:
|
|
count Count instances of fields
|
|
mode Find most-frequently-occurring values for fields; first-found wins tie
|
|
sum Compute sums of specified fields
|
|
mean Compute averages (sample means) of specified fields
|
|
stddev Compute sample standard deviation of specified fields
|
|
var Compute sample variance of specified fields
|
|
meaneb Estimate error bars for averages (assuming no sample autocorrelation)
|
|
skewness Compute sample skewness of specified fields
|
|
kurtosis Compute sample kurtosis of specified fields
|
|
min Compute minimum values of specified fields
|
|
max Compute maximum values of specified fields
|
|
-f {a,b,c} Value-field names on which to compute statistics. Requires -o.
|
|
-r {a,b,c} Regular expressions for value-field names on which to compute
|
|
statistics. Requires -o.
|
|
-c {a,b,c} Substrings for collapse mode. All fields which have the same names
|
|
after removing substrings will be accumulated together. Please see
|
|
examples below.
|
|
-i Use interpolated percentiles, like R's type=7; default like type=1.
|
|
-o {name} Output field basename for -f/-r.
|
|
-k Keep the input fields which contributed to the output statistics;
|
|
the default is to omit them.
|
|
-F Computes integerable things (e.g. count) in floating point.
|
|
Example input data: "a_in_x=1,a_out_x=2,b_in_y=4,b_out_x=8".
|
|
Example: mlr merge-fields -a sum,count -f a_in_x,a_out_x -o foo
|
|
produces "b_in_y=4,b_out_x=8,foo_sum=3,foo_count=2" since "a_in_x,a_out_x" are
|
|
summed over.
|
|
Example: mlr merge-fields -a sum,count -r in_,out_ -o bar
|
|
produces "bar_sum=15,bar_count=4" since all four fields are summed over.
|
|
Example: mlr merge-fields -a sum,count -c in_,out_
|
|
produces "a_x_sum=3,a_x_count=2,b_y_sum=4,b_y_count=1,b_x_sum=8,b_x_count=1"
|
|
since "a_in_x" and "a_out_x" both collapse to "a_x", "b_in_y" collapses to
|
|
"b_y", and "b_out_x" collapses to "b_x".
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>This is like <tt>mlr stats1</tt> but all accumulation is done across fields
|
|
within each given record: horizontal rather than vertical statistics, if you
|
|
will.
|
|
|
|
<p/>Examples:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --csvlite --opprint cat data/inout.csv
|
|
a_in a_out b_in b_out
|
|
436 490 446 195
|
|
526 320 963 780
|
|
220 888 705 831
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --csvlite --opprint merge-fields -a min,max,sum -c _in,_out data/inout.csv
|
|
a_min a_max a_sum b_min b_max b_sum
|
|
436 490 926 195 446 641
|
|
320 526 846 780 963 1743
|
|
220 888 1108 705 831 1536
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --csvlite --opprint merge-fields -k -a sum -c _in,_out data/inout.csv
|
|
a_in a_out b_in b_out a_sum b_sum
|
|
436 490 446 195 926 641
|
|
526 320 963 780 846 1743
|
|
220 888 705 831 1108 1536
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="most-frequent"/><h2>most-frequent</h2>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr most-frequent -h
|
|
Usage: mlr most-frequent [options]
|
|
Shows the most frequently occurring distinct values for specified field names.
|
|
The first entry is the statistical mode; the remaining are runners-up.
|
|
Options:
|
|
-f {one or more comma-separated field names}. Required flag.
|
|
-n {count}. Optional flag defaulting to 10.
|
|
-b Suppress counts; show only field values.
|
|
See also "mlr least".
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint --from data/colored-shapes.dkvp most-frequent -f shape -n 5
|
|
shape count
|
|
square 4115
|
|
triangle 3372
|
|
circle 2591
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint --from data/colored-shapes.dkvp most-frequent -f shape,color -n 5
|
|
shape color count
|
|
square red 1874
|
|
triangle red 1560
|
|
circle red 1207
|
|
square yellow 589
|
|
square blue 589
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint --from data/colored-shapes.dkvp most-frequent -f shape,color -n 5 -b
|
|
shape color
|
|
square red
|
|
triangle red
|
|
circle red
|
|
square yellow
|
|
square blue
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
See also <a href="#least-frequent">least-frequent</a>.
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="nest"/><h2>nest</h2>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr nest -h
|
|
Usage: mlr nest [options]
|
|
Explodes specified field values into separate fields/records, or reverses this.
|
|
Options:
|
|
--explode,--implode One is required.
|
|
--values,--pairs One is required.
|
|
--across-records,--across-fields One is required.
|
|
-f {field name} Required.
|
|
--nested-fs {string} Defaults to ";". Field separator for nested values.
|
|
--nested-ps {string} Defaults to ":". Pair separator for nested key-value pairs.
|
|
Please use "mlr --usage-separator-options" for information on specifying separators.
|
|
|
|
Examples:
|
|
|
|
mlr nest --explode --values --across-records -f x
|
|
with input record "x=a;b;c,y=d" produces output records
|
|
"x=a,y=d"
|
|
"x=b,y=d"
|
|
"x=c,y=d"
|
|
Use --implode to do the reverse.
|
|
|
|
mlr nest --explode --values --across-fields -f x
|
|
with input record "x=a;b;c,y=d" produces output records
|
|
"x_1=a,x_2=b,x_3=c,y=d"
|
|
Use --implode to do the reverse.
|
|
|
|
mlr nest --explode --pairs --across-records -f x
|
|
with input record "x=a:1;b:2;c:3,y=d" produces output records
|
|
"a=1,y=d"
|
|
"b=2,y=d"
|
|
"c=3,y=d"
|
|
|
|
mlr nest --explode --pairs --across-fields -f x
|
|
with input record "x=a:1;b:2;c:3,y=d" produces output records
|
|
"a=1,b=2,c=3,y=d"
|
|
|
|
Notes:
|
|
* With --pairs, --implode doesn't make sense since the original field name has
|
|
been lost.
|
|
* The combination "--implode --values --across-records" is non-streaming:
|
|
no output records are produced until all input records have been read. In
|
|
particular, this means it won't work in tail -f contexts. But all other flag
|
|
combinations result in streaming (tail -f friendly) data processing.
|
|
* It's up to you to ensure that the nested-fs is distinct from your data's IFS:
|
|
e.g. by default the former is semicolon and the latter is comma.
|
|
See also mlr reshape.
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="nothing"/><h2>nothing</h2>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr nothing -h
|
|
Usage: mlr nothing [options]
|
|
Drops all input records. Useful for testing, or after tee/print/etc. have
|
|
produced other output.
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="put"/><h2>put</h2>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr put --help
|
|
Usage: mlr put [options] {expression}
|
|
Adds/updates specified field(s). Expressions are semicolon-separated and must
|
|
either be assignments, or evaluate to boolean. Booleans with following
|
|
statements in curly braces control whether those statements are executed;
|
|
booleans without following curly braces do nothing except side effects (e.g.
|
|
regex-captures into \1, \2, etc.).
|
|
|
|
Options:
|
|
-v: Prints the expressions's AST (abstract syntax tree), which gives
|
|
full transparency on the precedence and associativity rules of
|
|
Miller's grammar, to stdout.
|
|
-a: Prints a low-level stack-allocation trace to stdout.
|
|
-t: Prints a low-level parser trace to stderr.
|
|
-T: Prints a every statement to stderr as it is executed.
|
|
-q: Does not include the modified record in the output stream. Useful for when
|
|
all desired output is in begin and/or end blocks.
|
|
-S: Keeps field values, or literals in the expression, as strings with no type
|
|
inference to int or float.
|
|
-F: Keeps field values, or literals in the expression, as strings or floats
|
|
with no inference to int.
|
|
--oflatsep {string}: Separator to use when flattening multi-level @-variables
|
|
to output records for emit. Default ":".
|
|
-f {filename}: the DSL expression is taken from the specified file rather
|
|
than from the command line. Outer single quotes wrapping the expression
|
|
should not be placed in the file. If -f is specified more than once,
|
|
all input files specified using -f are concatenated to produce the expression.
|
|
(For example, you can define functions in one file and call them from another.)
|
|
-e {expression}: You can use this after -f to add an expression. Example use
|
|
case: define functions/subroutines in a file you specify with -f, then call
|
|
them with an expression you specify with -e.
|
|
(If you mix -e and -f then the expressions are evaluated in the order encountered.
|
|
Since the expression pieces are simply concatenated, please be sure to use intervening
|
|
semicolons to separate expressions.)
|
|
--no-fflush: for emit, tee, print, and dump, don't call fflush() after every
|
|
record.
|
|
Any of the output-format command-line flags (see mlr -h). Example: using
|
|
mlr --icsv --opprint ... then put --ojson 'tee > "mytap-".$a.".dat", $*' then ...
|
|
the input is CSV, the output is pretty-print tabular, but the tee-file output
|
|
is written in JSON format.
|
|
|
|
Please use a dollar sign for field names and double-quotes for string
|
|
literals. If field names have special characters such as "." then you might
|
|
use braces, e.g. '${field.name}'. Miller built-in variables are
|
|
NF NR FNR FILENUM FILENAME PI E, and ENV["namegoeshere"] to access environment
|
|
variables. The environment-variable name may be an expression, e.g. a field
|
|
value.
|
|
|
|
Use # to comment to end of line.
|
|
|
|
Examples:
|
|
mlr put '$y = log10($x); $z = sqrt($y)'
|
|
mlr put '$x>0.0 { $y=log10($x); $z=sqrt($y) }' # does {...} only if $x > 0.0
|
|
mlr put '$x>0.0; $y=log10($x); $z=sqrt($y)' # does all three statements
|
|
mlr put '$a =~ "([a-z]+)_([0-9]+); $b = "left_\1"; $c = "right_\2"'
|
|
mlr put '$a =~ "([a-z]+)_([0-9]+) { $b = "left_\1"; $c = "right_\2" }'
|
|
mlr put '$filename = FILENAME'
|
|
mlr put '$colored_shape = $color . "_" . $shape'
|
|
mlr put '$y = cos($theta); $z = atan2($y, $x)'
|
|
mlr put '$name = sub($name, "http.*com"i, "")'
|
|
mlr put -q '@sum += $x; end {emit @sum}'
|
|
mlr put -q '@sum[$a] += $x; end {emit @sum, "a"}'
|
|
mlr put -q '@sum[$a][$b] += $x; end {emit @sum, "a", "b"}'
|
|
mlr put -q '@min=min(@min,$x);@max=max(@max,$x); end{emitf @min, @max}'
|
|
mlr put -q 'isnull(@xmax) || $x > @xmax {@xmax=$x; @recmax=$*}; end {emit @recmax}'
|
|
mlr put '
|
|
$x = 1;
|
|
#$y = 2;
|
|
$z = 3
|
|
'
|
|
|
|
Please see also 'mlr -k' for examples using redirected output.
|
|
|
|
Please see http://johnkerl.org/miller/doc/reference.html for more information
|
|
including function list. Or "mlr -f".
|
|
Please see in particular:
|
|
http://www.johnkerl.org/miller/doc/reference.html#put
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<a id="Features_which_put_shares_with_filter"/><h3>Features which put shares with filter</h3>
|
|
|
|
<p/>Please see <a href="#Expression_language_for_filter_and_put">Expression
|
|
language for filter and put</a> for more information about the expression
|
|
language for <tt>mlr put</tt>.
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="regularize"/><h2>regularize</h2>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr regularize --help
|
|
Usage: mlr regularize
|
|
For records seen earlier in the data stream with same field names in
|
|
a different order, outputs them with field names in the previously
|
|
encountered order.
|
|
Example: input records a=1,c=2,b=3, then e=4,d=5, then c=7,a=6,b=8
|
|
output as a=1,c=2,b=3, then e=4,d=5, then a=6,c=7,b=8
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>This exists since hash-map software in various languages and tools
|
|
encountered in the wild does not always print similar rows with fields in the
|
|
same order: <tt>mlr regularize</tt> helps clean that up.
|
|
|
|
<p/>See also <a href="#reorder"><tt>reorder</tt></a>.
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="rename"/><h2>rename</h2>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr rename --help
|
|
Usage: mlr rename [options] {old1,new1,old2,new2,...}
|
|
Renames specified fields.
|
|
Options:
|
|
-r Treat old field names as regular expressions. "ab", "a.*b"
|
|
will match any field name containing the substring "ab" or
|
|
matching "a.*b", respectively; anchors of the form "^ab$",
|
|
"^a.*b$" may be used. New field names may be plain strings,
|
|
or may contain capture groups of the form "\1" through
|
|
"\9". Wrapping the regex in double quotes is optional, but
|
|
is required if you wish to follow it with 'i' to indicate
|
|
case-insensitivity.
|
|
-g Do global replacement within each field name rather than
|
|
first-match replacement.
|
|
Examples:
|
|
mlr rename -f old_name,new_name'
|
|
mlr rename -f old_name_1,new_name_1,old_name_2,new_name_2'
|
|
mlr rename -r 'Date_[0-9]+,Date,' Rename all such fields to be "Date"
|
|
mlr rename -r '"Date_[0-9]+",Date' Same
|
|
mlr rename -r 'Date_([0-9]+).*,\1' Rename all such fields to be of the form 20151015
|
|
mlr rename -r '"name"i,Name' Rename "name", "Name", "NAME", etc. to "Name"
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<table><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint cat data/small
|
|
a b i x y
|
|
pan pan 1 0.3467901443380824 0.7268028627434533
|
|
eks pan 2 0.7586799647899636 0.5221511083334797
|
|
wye wye 3 0.20460330576630303 0.33831852551664776
|
|
eks wye 4 0.38139939387114097 0.13418874328430463
|
|
wye pan 5 0.5732889198020006 0.8636244699032729
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint rename i,INDEX,b,COLUMN2 data/small
|
|
a COLUMN2 INDEX x y
|
|
pan pan 1 0.3467901443380824 0.7268028627434533
|
|
eks pan 2 0.7586799647899636 0.5221511083334797
|
|
wye wye 3 0.20460330576630303 0.33831852551664776
|
|
eks wye 4 0.38139939387114097 0.13418874328430463
|
|
wye pan 5 0.5732889198020006 0.8636244699032729
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr></table>
|
|
|
|
<p/>As discussed in <a href="performance.html">Performance</a>, <tt>sed</tt>
|
|
is significantly faster than Miller at doing this. However, Miller is
|
|
format-aware, so it knows to do renames only within specified field keys and
|
|
not any others, nor in field values which may happen to contain the same
|
|
pattern. Example:
|
|
|
|
<table><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ sed 's/y/COLUMN5/g' data/small
|
|
a=pan,b=pan,i=1,x=0.3467901443380824,COLUMN5=0.7268028627434533
|
|
a=eks,b=pan,i=2,x=0.7586799647899636,COLUMN5=0.5221511083334797
|
|
a=wCOLUMN5e,b=wCOLUMN5e,i=3,x=0.20460330576630303,COLUMN5=0.33831852551664776
|
|
a=eks,b=wCOLUMN5e,i=4,x=0.38139939387114097,COLUMN5=0.13418874328430463
|
|
a=wCOLUMN5e,b=pan,i=5,x=0.5732889198020006,COLUMN5=0.8636244699032729
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr rename y,COLUMN5 data/small
|
|
a=pan,b=pan,i=1,x=0.3467901443380824,COLUMN5=0.7268028627434533
|
|
a=eks,b=pan,i=2,x=0.7586799647899636,COLUMN5=0.5221511083334797
|
|
a=wye,b=wye,i=3,x=0.20460330576630303,COLUMN5=0.33831852551664776
|
|
a=eks,b=wye,i=4,x=0.38139939387114097,COLUMN5=0.13418874328430463
|
|
a=wye,b=pan,i=5,x=0.5732889198020006,COLUMN5=0.8636244699032729
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr></table>
|
|
|
|
See also <a href="#label"><tt>label</tt></a>.
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="reorder"/><h2>reorder</h2>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr reorder --help
|
|
Usage: mlr reorder [options]
|
|
-f {a,b,c} Field names to reorder.
|
|
-e Put specified field names at record end: default is to put
|
|
them at record start.
|
|
Examples:
|
|
mlr reorder -f a,b sends input record "d=4,b=2,a=1,c=3" to "a=1,b=2,d=4,c=3".
|
|
mlr reorder -e -f a,b sends input record "d=4,b=2,a=1,c=3" to "d=4,c=3,a=1,b=2".
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
This pivots specified field names to the start or end of the record — for
|
|
example when you have highly multi-column data and you want to bring a field or
|
|
two to the front of line where you can give a quick visual scan.
|
|
|
|
<table><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint cat data/small
|
|
a b i x y
|
|
pan pan 1 0.3467901443380824 0.7268028627434533
|
|
eks pan 2 0.7586799647899636 0.5221511083334797
|
|
wye wye 3 0.20460330576630303 0.33831852551664776
|
|
eks wye 4 0.38139939387114097 0.13418874328430463
|
|
wye pan 5 0.5732889198020006 0.8636244699032729
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint reorder -f i,b data/small
|
|
i b a x y
|
|
1 pan pan 0.3467901443380824 0.7268028627434533
|
|
2 pan eks 0.7586799647899636 0.5221511083334797
|
|
3 wye wye 0.20460330576630303 0.33831852551664776
|
|
4 wye eks 0.38139939387114097 0.13418874328430463
|
|
5 pan wye 0.5732889198020006 0.8636244699032729
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint reorder -e -f i,b data/small
|
|
a x y i b
|
|
pan 0.3467901443380824 0.7268028627434533 1 pan
|
|
eks 0.7586799647899636 0.5221511083334797 2 pan
|
|
wye 0.20460330576630303 0.33831852551664776 3 wye
|
|
eks 0.38139939387114097 0.13418874328430463 4 wye
|
|
wye 0.5732889198020006 0.8636244699032729 5 pan
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr></table>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="repeat"/><h2>repeat</h2>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr repeat --help
|
|
Usage: mlr repeat [options]
|
|
Copies input records to output records multiple times.
|
|
Options must be exactly one of the following:
|
|
-n {repeat count} Repeat each input record this many times.
|
|
-f {field name} Same, but take the repeat count from the specified
|
|
field name of each input record.
|
|
Example:
|
|
echo x=0 | mlr repeat -n 4 then put '$x=urand()'
|
|
produces:
|
|
x=0.488189
|
|
x=0.484973
|
|
x=0.704983
|
|
x=0.147311
|
|
Example:
|
|
echo a=1,b=2,c=3 | mlr repeat -f b
|
|
produces:
|
|
a=1,b=2,c=3
|
|
a=1,b=2,c=3
|
|
Example:
|
|
echo a=1,b=2,c=3 | mlr repeat -f c
|
|
produces:
|
|
a=1,b=2,c=3
|
|
a=1,b=2,c=3
|
|
a=1,b=2,c=3
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p>This is useful in at least two ways: one, as a data-generator as in the
|
|
above example using <tt>urand()</tt>; two, for reconstructing individual
|
|
samples from data which has been count-aggregated:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ cat data/repeat-example.dat
|
|
color=blue,count=5
|
|
color=red,count=4
|
|
color=green,count=3
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr repeat -f count then cut -x -f count data/repeat-example.dat
|
|
color=blue
|
|
color=blue
|
|
color=blue
|
|
color=blue
|
|
color=blue
|
|
color=red
|
|
color=red
|
|
color=red
|
|
color=red
|
|
color=green
|
|
color=green
|
|
color=green
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p>After expansion with <tt>repeat</tt>, such data can then be sent on to
|
|
<tt>stats1 -a mode</tt>, or (if the data are numeric) to <tt>stats1 -a
|
|
p10,p50,p90</tt>, etc.
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="reshape"/><h2>reshape</h2>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr reshape --help
|
|
Usage: mlr reshape [options]
|
|
Wide-to-long options:
|
|
-i {input field names} -o {key-field name,value-field name}
|
|
-r {input field regexes} -o {key-field name,value-field name}
|
|
These pivot/reshape the input data such that the input fields are removed
|
|
and separate records are emitted for each key/value pair.
|
|
Note: this works with tail -f and produces output records for each input
|
|
record seen.
|
|
Long-to-wide options:
|
|
-s {key-field name,value-field name}
|
|
These pivot/reshape the input data to undo the wide-to-long operation.
|
|
Note: this does not work with tail -f; it produces output records only after
|
|
all input records have been read.
|
|
|
|
Examples:
|
|
|
|
Input file "wide.txt":
|
|
time X Y
|
|
2009-01-01 0.65473572 2.4520609
|
|
2009-01-02 -0.89248112 0.2154713
|
|
2009-01-03 0.98012375 1.3179287
|
|
|
|
mlr --pprint reshape -i X,Y -o item,value wide.txt
|
|
time item value
|
|
2009-01-01 X 0.65473572
|
|
2009-01-01 Y 2.4520609
|
|
2009-01-02 X -0.89248112
|
|
2009-01-02 Y 0.2154713
|
|
2009-01-03 X 0.98012375
|
|
2009-01-03 Y 1.3179287
|
|
|
|
mlr --pprint reshape -r '[A-Z]' -o item,value wide.txt
|
|
time item value
|
|
2009-01-01 X 0.65473572
|
|
2009-01-01 Y 2.4520609
|
|
2009-01-02 X -0.89248112
|
|
2009-01-02 Y 0.2154713
|
|
2009-01-03 X 0.98012375
|
|
2009-01-03 Y 1.3179287
|
|
|
|
Input file "long.txt":
|
|
time item value
|
|
2009-01-01 X 0.65473572
|
|
2009-01-01 Y 2.4520609
|
|
2009-01-02 X -0.89248112
|
|
2009-01-02 Y 0.2154713
|
|
2009-01-03 X 0.98012375
|
|
2009-01-03 Y 1.3179287
|
|
|
|
mlr --pprint reshape -s item,value long.txt
|
|
time X Y
|
|
2009-01-01 0.65473572 2.4520609
|
|
2009-01-02 -0.89248112 0.2154713
|
|
2009-01-03 0.98012375 1.3179287
|
|
See also mlr nest.
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="sample"/><h2>sample</h2>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr sample --help
|
|
Usage: mlr sample [options]
|
|
Reservoir sampling (subsampling without replacement), optionally by category.
|
|
-k {count} Required: number of records to output, total, or by group if using -g.
|
|
-g {a,b,c} Optional: group-by-field names for samples.
|
|
See also mlr bootstrap and mlr shuffle.
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>This is reservoir-sampling: select <i>k</i> items from <i>n</i> with
|
|
uniform probability and no repeats in the sample. (If <i>n</i> is less than
|
|
<i>k</i>, then of course only <i>n</i> samples are produced.) With <tt>-g
|
|
{field names}</tt>, produce a <i>k</i>-sample for each distinct value of the
|
|
specified field names.
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint sample -k 4 data/colored-shapes.dkvp
|
|
color shape flag i u v w x
|
|
purple triangle 0 90122 0.9986871176198068 0.3037738877233719 0.5154934457238382 5.365962021016529
|
|
red circle 0 3139 0.04835898233323954 -0.03964684310055758 0.5263660881848111 5.3758779366493625
|
|
orange triangle 0 67847 0.36746306902109926 0.5161574810505635 0.5176199566173642 3.1748088656576567
|
|
yellow square 1 33576 0.3098376725521097 0.8525628505287842 0.49774122460981685 4.494754378604669
|
|
|
|
$ mlr --opprint sample -k 4 data/colored-shapes.dkvp
|
|
color shape flag i u v w x
|
|
blue square 1 16783 0.09974385090654347 0.7243899920872646 0.5353718443278438 4.431057737383438
|
|
orange square 1 93291 0.5944176543007182 0.17744449786454086 0.49262281749172077 3.1548117990710653
|
|
yellow square 1 54436 0.5268161165014636 0.8785588662666121 0.5058773791931063 7.019185838783636
|
|
yellow square 1 55491 0.0025440267883102274 0.05474106287787284 0.5102729153751984 3.526301273728043
|
|
|
|
$ mlr --opprint sample -k 2 -g color data/colored-shapes.dkvp
|
|
color shape flag i u v w x
|
|
yellow triangle 1 11 0.6321695890307647 0.9887207810889004 0.4364983936735774 5.7981881667050565
|
|
yellow square 1 917 0.8547010348386344 0.7356782810796262 0.4531511689924275 5.774541777078352
|
|
red circle 1 4000 0.05490416175132373 0.07392337815122155 0.49416101516594396 5.355725080701707
|
|
red square 0 87506 0.6357719216821314 0.6970867759393995 0.4940826462055272 6.351579417310387
|
|
purple triangle 0 14898 0.7800986870203719 0.23998073813992293 0.5014775988383656 3.141006771777843
|
|
purple triangle 0 151 0.032614487569017414 0.7346633365041219 0.7812143304483805 2.6831992610568047
|
|
green triangle 1 126 0.1513010528347546 0.40346767294704544 0.051213231883952326 5.955109300797182
|
|
green circle 0 17635 0.029856606049114442 0.4724542934246524 0.49529606749929744 5.239153910272168
|
|
blue circle 1 1020 0.414263129226617 0.8304946402876182 0.13151094520189244 4.397873687920433
|
|
blue triangle 0 220 0.441773289968473 0.44597731903759075 0.6329360666849821 4.3064608776550894
|
|
orange square 0 1885 0.8079311983747106 0.8685956833908394 0.3116410800256374 4.390864584500387
|
|
orange triangle 0 1533 0.32904497195507487 0.23168161807490417 0.8722623057355134 5.164071635714438
|
|
|
|
$ mlr --opprint sample -k 2 -g color then sort -f color data/colored-shapes.dkvp
|
|
color shape flag i u v w x
|
|
blue circle 0 215 0.7803586969333292 0.33146680638888126 0.04289047852629113 5.725365736377487
|
|
blue circle 1 3616 0.8548431579124808 0.4989623130006362 0.3339426415875795 3.696785877560498
|
|
green square 0 356 0.7674272008085286 0.341578843118008 0.4570224877870851 4.830320062215299
|
|
green square 0 152 0.6684429446914862 0.016056003736548696 0.4656148241291592 5.434588759225423
|
|
orange triangle 0 587 0.5175826237797857 0.08989091493635304 0.9011709461770973 4.265854207755811
|
|
orange triangle 0 1533 0.32904497195507487 0.23168161807490417 0.8722623057355134 5.164071635714438
|
|
purple triangle 0 14192 0.5196327866973567 0.7860928603468063 0.4964368415453642 4.899167143824484
|
|
purple triangle 0 65 0.6842806710360729 0.5823723856331258 0.8014053396013747 5.805148213865135
|
|
red square 1 2431 0.38378504852300466 0.11445015005595527 0.49355539228753786 5.146756570128739
|
|
red triangle 0 57097 0.43763430414406546 0.3355450325004481 0.5322349637512487 4.144267240289442
|
|
yellow triangle 1 11 0.6321695890307647 0.9887207810889004 0.4364983936735774 5.7981881667050565
|
|
yellow square 1 158 0.41527900739142165 0.7118027080775757 0.4200799665161291 5.33279067554884
|
|
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>Note that no output is produced until all inputs are in. Another way to do
|
|
sampling, which works in the streaming case, is <tt>mlr filter 'urand() &
|
|
0.001'</tt> where you tune the 0.001 to meet your needs.
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="sec2gmt"/><h2>sec2gmt</h2>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr sec2gmt -h
|
|
Usage: mlr sec2gmt {comma-separated list of field names}
|
|
Replaces a numeric field representing seconds since the epoch with the
|
|
corresponding GMT timestamp; leaves non-numbers as-is. This is nothing
|
|
more than a keystroke-saver for the sec2gmt function:
|
|
mlr sec2gmt time1,time2
|
|
is the same as
|
|
mlr put '$time1=sec2gmt($time1);$time2=sec2gmt($time2)'
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="sec2gmtdate"/><h2>sec2gmtdate</h2>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr sec2gmtdate -h
|
|
Usage: mlr sec2gmtdate {comma-separated list of field names}
|
|
Replaces a numeric field representing seconds since the epoch with the
|
|
corresponding GMT year-month-day timestamp; leaves non-numbers as-is.
|
|
This is nothing more than a keystroke-saver for the sec2gmtdate function:
|
|
mlr sec2gmtdate time1,time2
|
|
is the same as
|
|
mlr put '$time1=sec2gmtdate($time1);$time2=sec2gmtdate($time2)'
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="seqgen"/><h2>seqgen</h2>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr seqgen -h
|
|
Usage: mlr seqgen [options]
|
|
Produces a sequence of counters. Discards the input record stream. Produces
|
|
output as specified by the following options:
|
|
-f {name} Field name for counters; default "i".
|
|
--start {number} Inclusive start value; default "1".
|
|
--stop {number} Inclusive stop value; default "100".
|
|
--step {number} Step value; default "1".
|
|
Start, stop, and/or step may be floating-point. Output is integer if start,
|
|
stop, and step are all integers. Step may be negative. It may not be zero
|
|
unless start == stop.
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr seqgen --stop 10
|
|
i=1
|
|
i=2
|
|
i=3
|
|
i=4
|
|
i=5
|
|
i=6
|
|
i=7
|
|
i=8
|
|
i=9
|
|
i=10
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr seqgen --start 20 --stop 40 --step 4
|
|
i=20
|
|
i=24
|
|
i=28
|
|
i=32
|
|
i=36
|
|
i=40
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr seqgen --start 40 --stop 20 --step -4
|
|
i=40
|
|
i=36
|
|
i=32
|
|
i=28
|
|
i=24
|
|
i=20
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="shuffle"/><h2>shuffle</h2>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr shuffle -h
|
|
Usage: mlr shuffle {no options}
|
|
Outputs records randomly permuted. No output records are produced until
|
|
all input records are read.
|
|
See also mlr bootstrap and mlr sample.
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="sort"/><h2>sort</h2>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr sort --help
|
|
Usage: mlr sort {flags}
|
|
Flags:
|
|
-f {comma-separated field names} Lexical ascending
|
|
-n {comma-separated field names} Numerical ascending; nulls sort last
|
|
-nf {comma-separated field names} Numerical ascending; nulls sort last
|
|
-r {comma-separated field names} Lexical descending
|
|
-nr {comma-separated field names} Numerical descending; nulls sort first
|
|
Sorts records primarily by the first specified field, secondarily by the second
|
|
field, and so on. Any records not having all specified sort keys will appear
|
|
at the end of the output, in the order they were encountered, regardless of the
|
|
specified sort order.
|
|
Example:
|
|
mlr sort -f a,b -nr x,y,z
|
|
which is the same as:
|
|
mlr sort -f a -f b -nr x -nr y -nr z
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>Example:
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint sort -f a -nr x data/small
|
|
a b i x y
|
|
eks pan 2 0.7586799647899636 0.5221511083334797
|
|
eks wye 4 0.38139939387114097 0.13418874328430463
|
|
pan pan 1 0.3467901443380824 0.7268028627434533
|
|
wye pan 5 0.5732889198020006 0.8636244699032729
|
|
wye wye 3 0.20460330576630303 0.33831852551664776
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>Here’s an example filtering log data: suppose multiple threads (labeled here by color) are all logging progress counts to a single log file. The log file is (by nature) chronological, so the progress of various threads is interleaved:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ head -n 10 data/multicountdown.dat
|
|
upsec=0.002,color=green,count=1203
|
|
upsec=0.083,color=red,count=3817
|
|
upsec=0.188,color=red,count=3801
|
|
upsec=0.395,color=blue,count=2697
|
|
upsec=0.526,color=purple,count=953
|
|
upsec=0.671,color=blue,count=2684
|
|
upsec=0.899,color=purple,count=926
|
|
upsec=0.912,color=red,count=3798
|
|
upsec=1.093,color=blue,count=2662
|
|
upsec=1.327,color=purple,count=917
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/> We can group these by thread by sorting on the thread ID (here,
|
|
<tt>color</tt>). Since Miller’s sort is stable, this means that
|
|
timestamps within each thread’s log data are still chronological:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ head -n 20 data/multicountdown.dat | mlr --opprint sort -f color
|
|
upsec color count
|
|
0.395 blue 2697
|
|
0.671 blue 2684
|
|
1.093 blue 2662
|
|
2.064 blue 2659
|
|
2.2880000000000003 blue 2647
|
|
0.002 green 1203
|
|
1.407 green 1187
|
|
1.448 green 1177
|
|
2.313 green 1161
|
|
0.526 purple 953
|
|
0.899 purple 926
|
|
1.327 purple 917
|
|
1.703 purple 908
|
|
0.083 red 3817
|
|
0.188 red 3801
|
|
0.912 red 3798
|
|
1.416 red 3788
|
|
1.587 red 3782
|
|
1.601 red 3755
|
|
1.832 red 3717
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>Any records not having all specified sort keys will appear at the end of the output, in the order they
|
|
were encountered, regardless of the specified sort order:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr sort -n x data/sort-missing.dkvp
|
|
x=1
|
|
x=2
|
|
x=4
|
|
a=3
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr sort -nr x data/sort-missing.dkvp
|
|
x=4
|
|
x=2
|
|
x=1
|
|
a=3
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="stats1"/><h2>stats1</h2>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr stats1 --help
|
|
Usage: mlr stats1 [options]
|
|
Computes univariate statistics for one or more given fields, accumulated across
|
|
the input record stream.
|
|
Options:
|
|
-a {sum,count,...} Names of accumulators: p10 p25.2 p50 p98 p100 etc. and/or
|
|
one or more of:
|
|
count Count instances of fields
|
|
mode Find most-frequently-occurring values for fields; first-found wins tie
|
|
sum Compute sums of specified fields
|
|
mean Compute averages (sample means) of specified fields
|
|
stddev Compute sample standard deviation of specified fields
|
|
var Compute sample variance of specified fields
|
|
meaneb Estimate error bars for averages (assuming no sample autocorrelation)
|
|
skewness Compute sample skewness of specified fields
|
|
kurtosis Compute sample kurtosis of specified fields
|
|
min Compute minimum values of specified fields
|
|
max Compute maximum values of specified fields
|
|
-f {a,b,c} Value-field names on which to compute statistics
|
|
-g {d,e,f} Optional group-by-field names
|
|
-i Use interpolated percentiles, like R's type=7; default like type=1.
|
|
-s Print iterative stats. Useful in tail -f contexts (in which
|
|
case please avoid pprint-format output since end of input
|
|
stream will never be seen).
|
|
-F Computes integerable things (e.g. count) in floating point.
|
|
Example: mlr stats1 -a min,p10,p50,p90,max -f value -g size,shape
|
|
Example: mlr stats1 -a count,mode -f size
|
|
Example: mlr stats1 -a count,mode -f size -g shape
|
|
Notes:
|
|
* p50 is a synonym for median.
|
|
* min and max output the same results as p0 and p100, respectively, but use
|
|
less memory.
|
|
* count and mode allow text input; the rest require numeric input.
|
|
In particular, 1 and 1.0 are distinct text for count and mode.
|
|
* When there are mode ties, the first-encountered datum wins.
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
These are simple univariate statistics on one or more number-valued fields
|
|
(<tt>count</tt> and <tt>mode</tt> apply to non-numeric fields as well),
|
|
optionally categorized by one or more other fields.
|
|
|
|
<table><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --oxtab stats1 -a count,sum,min,p10,p50,mean,p90,max -f x,y data/medium
|
|
x_count 10000
|
|
x_sum 4986.019682
|
|
x_min 0.000045
|
|
x_p10 0.093322
|
|
x_p50 0.501159
|
|
x_mean 0.498602
|
|
x_p90 0.900794
|
|
x_max 0.999953
|
|
y_count 10000
|
|
y_sum 5062.057445
|
|
y_min 0.000088
|
|
y_p10 0.102132
|
|
y_p50 0.506021
|
|
y_mean 0.506206
|
|
y_p90 0.905366
|
|
y_max 0.999965
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint stats1 -a mean -f x,y -g b then sort -f b data/medium
|
|
b x_mean y_mean
|
|
eks 0.506361 0.510293
|
|
hat 0.487899 0.513118
|
|
pan 0.497304 0.499599
|
|
wye 0.497593 0.504596
|
|
zee 0.504242 0.502997
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint stats1 -a p50,p99 -f u,v -g color then put '$ur=$u_p99/$u_p50;$vr=$v_p99/$v_p50' data/colored-shapes.dkvp
|
|
color u_p50 u_p99 v_p50 v_p99 ur vr
|
|
yellow 0.501019 0.989046 0.520630 0.987034 1.974069 1.895845
|
|
red 0.485038 0.990054 0.492586 0.994444 2.041189 2.018823
|
|
purple 0.501319 0.988893 0.504571 0.988287 1.972582 1.958668
|
|
green 0.502015 0.990764 0.505359 0.990175 1.973574 1.959350
|
|
blue 0.525226 0.992655 0.485170 0.993873 1.889958 2.048505
|
|
orange 0.483548 0.993635 0.480913 0.989102 2.054884 2.056717
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint count-distinct -f shape then sort -nr count data/colored-shapes.dkvp
|
|
shape count
|
|
square 4115
|
|
triangle 3372
|
|
circle 2591
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint stats1 -a mode -f color -g shape data/colored-shapes.dkvp
|
|
shape color_mode
|
|
triangle red
|
|
square red
|
|
circle red
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr></table>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="stats2"/><h2>stats2</h2>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr stats2 --help
|
|
Usage: mlr stats2 [options]
|
|
Computes bivariate statistics for one or more given field-name pairs,
|
|
accumulated across the input record stream.
|
|
-a {linreg-ols,corr,...} Names of accumulators: one or more of:
|
|
linreg-pca Linear regression using principal component analysis
|
|
linreg-ols Linear regression using ordinary least squares
|
|
r2 Quality metric for linreg-ols (linreg-pca emits its own)
|
|
logireg Logistic regression
|
|
corr Sample correlation
|
|
cov Sample covariance
|
|
covx Sample-covariance matrix
|
|
-f {a,b,c,d} Value-field name-pairs on which to compute statistics.
|
|
There must be an even number of names.
|
|
-g {e,f,g} Optional group-by-field names.
|
|
-v Print additional output for linreg-pca.
|
|
-s Print iterative stats. Useful in tail -f contexts (in which
|
|
case please avoid pprint-format output since end of input
|
|
stream will never be seen).
|
|
--fit Rather than printing regression parameters, applies them to
|
|
the input data to compute new fit fields. All input records are
|
|
held in memory until end of input stream. Has effect only for
|
|
linreg-ols, linreg-pca, and logireg.
|
|
Only one of -s or --fit may be used.
|
|
Example: mlr stats2 -a linreg-pca -f x,y
|
|
Example: mlr stats2 -a linreg-ols,r2 -f x,y -g size,shape
|
|
Example: mlr stats2 -a corr -f x,y
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
These are simple bivariate statistics on one or more pairs of number-valued
|
|
fields, optionally categorized by one or more fields.
|
|
|
|
<table><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --oxtab put '$x2=$x*$x; $xy=$x*$y; $y2=$y**2' then stats2 -a cov,corr -f x,y,y,y,x2,xy,x2,y2 data/medium
|
|
x_y_cov 0.000043
|
|
x_y_corr 0.000504
|
|
y_y_cov 0.084611
|
|
y_y_corr 1.000000
|
|
x2_xy_cov 0.041884
|
|
x2_xy_corr 0.630174
|
|
x2_y2_cov -0.000310
|
|
x2_y2_corr -0.003425
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint put '$x2=$x*$x; $xy=$x*$y; $y2=$y**2' then stats2 -a linreg-ols,r2 -f x,y,y,y,xy,y2 -g a data/medium
|
|
a x_y_ols_m x_y_ols_b x_y_ols_n x_y_r2 y_y_ols_m y_y_ols_b y_y_ols_n y_y_r2 xy_y2_ols_m xy_y2_ols_b xy_y2_ols_n xy_y2_r2
|
|
pan 0.017026 0.500403 2081 0.000287 1.000000 0.000000 2081 1.000000 0.878132 0.119082 2081 0.417498
|
|
eks 0.040780 0.481402 1965 0.001646 1.000000 0.000000 1965 1.000000 0.897873 0.107341 1965 0.455632
|
|
wye -0.039153 0.525510 1966 0.001505 1.000000 0.000000 1966 1.000000 0.853832 0.126745 1966 0.389917
|
|
zee 0.002781 0.504307 2047 0.000008 1.000000 0.000000 2047 1.000000 0.852444 0.124017 2047 0.393566
|
|
hat -0.018621 0.517901 1941 0.000352 1.000000 0.000000 1941 1.000000 0.841230 0.135573 1941 0.368794
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr></table>
|
|
|
|
<p/>Here’s an example simple line-fit. The <tt>x</tt> and <tt>y</tt>
|
|
fields of the <tt>data/medium</tt> dataset are just independent uniformly
|
|
distributed on the unit interval. Here we remove half the data and fit a line to it.
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
|
|
# Prepare input data:
|
|
mlr filter '($x<.5 && $y<.5) || ($x>.5 && $y>.5)' data/medium > data/medium-squares
|
|
|
|
# Do a linear regression and examine coefficients:
|
|
mlr --ofs newline stats2 -a linreg-pca -f x,y data/medium-squares
|
|
x_y_pca_m=1.014419
|
|
x_y_pca_b=0.000308
|
|
x_y_pca_quality=0.861354
|
|
|
|
# Option 1 to apply the regression coefficients and produce a linear fit:
|
|
# Set x_y_pca_m and x_y_pca_b as shell variables:
|
|
eval $(mlr --ofs newline stats2 -a linreg-pca -f x,y data/medium-squares)
|
|
# In addition to x and y, make a new yfit which is the line fit, then plot
|
|
# using your favorite tool:
|
|
mlr --onidx put '$yfit='$x_y_pca_m'*$x+'$x_y_pca_b then cut -x -f a,b,i data/medium-squares \
|
|
| pgr -p -title 'linreg-pca example' -xmin 0 -xmax 1 -ymin 0 -ymax 1
|
|
|
|
# Option 2 to apply the regression coefficients and produce a linear fit: use --fit option
|
|
mlr --onidx stats2 -a linreg-pca --fit -f x,y then cut -f a,b,i data/medium-squares \
|
|
| pgr -p -title 'linreg-pca example' -xmin 0 -xmax 1 -ymin 0 -ymax 1
|
|
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>I use <a href="https://github.com/johnkerl/pgr"><tt>pgr</tt></a> for
|
|
plotting; here’s a screenshot.
|
|
|
|
<center>
|
|
<img src="data/linreg-example.jpg"/>
|
|
</center>
|
|
|
|
<p/> (Thanks Drew Kunas for a good conversation about PCA!)
|
|
|
|
<p/> Here’s an example estimating time-to-completion for a set of jobs.
|
|
Input data comes from a log file, with number of work units left to do in the
|
|
<tt>count</tt> field and accumulated seconds in the <tt>upsec</tt> field,
|
|
labeled by the <tt>color</tt> field:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ head -n 10 data/multicountdown.dat
|
|
upsec=0.002,color=green,count=1203
|
|
upsec=0.083,color=red,count=3817
|
|
upsec=0.188,color=red,count=3801
|
|
upsec=0.395,color=blue,count=2697
|
|
upsec=0.526,color=purple,count=953
|
|
upsec=0.671,color=blue,count=2684
|
|
upsec=0.899,color=purple,count=926
|
|
upsec=0.912,color=red,count=3798
|
|
upsec=1.093,color=blue,count=2662
|
|
upsec=1.327,color=purple,count=917
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
We can do a linear regression on count remaining as a function of time: with <tt>c = m*u+b</tt> we want to find the
|
|
time when the count goes to zero, i.e. <tt>u=-b/m</tt>.
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --oxtab stats2 -a linreg-pca -f upsec,count -g color then put '$donesec = -$upsec_count_pca_b/$upsec_count_pca_m' data/multicountdown.dat
|
|
color green
|
|
upsec_count_pca_m -32.756917
|
|
upsec_count_pca_b 1213.722730
|
|
upsec_count_pca_n 24
|
|
upsec_count_pca_quality 0.999984
|
|
donesec 37.052410
|
|
|
|
color red
|
|
upsec_count_pca_m -37.367646
|
|
upsec_count_pca_b 3810.133400
|
|
upsec_count_pca_n 30
|
|
upsec_count_pca_quality 0.999989
|
|
donesec 101.963431
|
|
|
|
color blue
|
|
upsec_count_pca_m -29.231212
|
|
upsec_count_pca_b 2698.932820
|
|
upsec_count_pca_n 25
|
|
upsec_count_pca_quality 0.999959
|
|
donesec 92.330514
|
|
|
|
color purple
|
|
upsec_count_pca_m -39.030097
|
|
upsec_count_pca_b 979.988341
|
|
upsec_count_pca_n 21
|
|
upsec_count_pca_quality 0.999991
|
|
donesec 25.108529
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="step"/><h2>step</h2>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr step --help
|
|
Usage: mlr step [options]
|
|
Computes values dependent on the previous record, optionally grouped
|
|
by category.
|
|
|
|
Options:
|
|
-a {delta,rsum,...} Names of steppers: comma-separated, one or more of:
|
|
delta Compute differences in field(s) between successive records
|
|
shift Include value(s) in field(s) from previous record, if any
|
|
from-first Compute differences in field(s) from first record
|
|
ratio Compute ratios in field(s) between successive records
|
|
rsum Compute running sums of field(s) between successive records
|
|
counter Count instances of field(s) between successive records
|
|
ewma Exponentially weighted moving average over successive records
|
|
-f {a,b,c} Value-field names on which to compute statistics
|
|
-g {d,e,f} Optional group-by-field names
|
|
-F Computes integerable things (e.g. counter) in floating point.
|
|
-d {x,y,z} Weights for ewma. 1 means current sample gets all weight (no
|
|
smoothing), near under under 1 is light smoothing, near over 0 is
|
|
heavy smoothing. Multiple weights may be specified, e.g.
|
|
"mlr step -a ewma -f sys_load -d 0.01,0.1,0.9". Default if omitted
|
|
is "-d 0.5".
|
|
-o {a,b,c} Custom suffixes for EWMA output fields. If omitted, these default to
|
|
the -d values. If supplied, the number of -o values must be the same
|
|
as the number of -d values.
|
|
|
|
Examples:
|
|
mlr step -a rsum -f request_size
|
|
mlr step -a delta -f request_size -g hostname
|
|
mlr step -a ewma -d 0.1,0.9 -f x,y
|
|
mlr step -a ewma -d 0.1,0.9 -o smooth,rough -f x,y
|
|
mlr step -a ewma -d 0.1,0.9 -o smooth,rough -f x,y -g group_name
|
|
|
|
Please see http://johnkerl.org/miller/doc/reference.html#filter or
|
|
https://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average
|
|
for more information on EWMA.
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
Most Miller commands are record-at-a-time, with the exception of <tt>stats1</tt>,
|
|
<tt>stats2</tt>, and <tt>histogram</tt> which compute aggregate output. The
|
|
<tt>step</tt> command is intermediate: it allows the option of adding fields
|
|
which are functions of fields from previous records. Rsum is short for <i>running sum</i>.
|
|
|
|
<table><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint step -a shift,delta,rsum,counter -f x data/medium | head -15
|
|
a b i x y x_shift x_delta x_rsum x_counter
|
|
pan pan 1 0.3467901443380824 0.7268028627434533 - 0 0.346790 1
|
|
eks pan 2 0.7586799647899636 0.5221511083334797 0.3467901443380824 0.411890 1.105470 2
|
|
wye wye 3 0.20460330576630303 0.33831852551664776 0.7586799647899636 -0.554077 1.310073 3
|
|
eks wye 4 0.38139939387114097 0.13418874328430463 0.20460330576630303 0.176796 1.691473 4
|
|
wye pan 5 0.5732889198020006 0.8636244699032729 0.38139939387114097 0.191890 2.264762 5
|
|
zee pan 6 0.5271261600918548 0.49322128674835697 0.5732889198020006 -0.046163 2.791888 6
|
|
eks zee 7 0.6117840605678454 0.1878849191181694 0.5271261600918548 0.084658 3.403672 7
|
|
zee wye 8 0.5985540091064224 0.976181385699006 0.6117840605678454 -0.013230 4.002226 8
|
|
hat wye 9 0.03144187646093577 0.7495507603507059 0.5985540091064224 -0.567112 4.033668 9
|
|
pan wye 10 0.5026260055412137 0.9526183602969864 0.03144187646093577 0.471184 4.536294 10
|
|
pan pan 11 0.7930488423451967 0.6505816637259333 0.5026260055412137 0.290423 5.329343 11
|
|
zee pan 12 0.3676141320555616 0.23614420670296965 0.7930488423451967 -0.425435 5.696957 12
|
|
eks pan 13 0.4915175580479536 0.7709126592971468 0.3676141320555616 0.123903 6.188474 13
|
|
eks zee 14 0.5207382318405251 0.34141681118811673 0.4915175580479536 0.029221 6.709213 14
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint step -a shift,delta,rsum,counter -f x -g a data/medium | head -15
|
|
a b i x y x_shift x_delta x_rsum x_counter
|
|
pan pan 1 0.3467901443380824 0.7268028627434533 - 0 0.346790 1
|
|
eks pan 2 0.7586799647899636 0.5221511083334797 - 0 0.758680 1
|
|
wye wye 3 0.20460330576630303 0.33831852551664776 - 0 0.204603 1
|
|
eks wye 4 0.38139939387114097 0.13418874328430463 0.7586799647899636 -0.377281 1.140079 2
|
|
wye pan 5 0.5732889198020006 0.8636244699032729 0.20460330576630303 0.368686 0.777892 2
|
|
zee pan 6 0.5271261600918548 0.49322128674835697 - 0 0.527126 1
|
|
eks zee 7 0.6117840605678454 0.1878849191181694 0.38139939387114097 0.230385 1.751863 3
|
|
zee wye 8 0.5985540091064224 0.976181385699006 0.5271261600918548 0.071428 1.125680 2
|
|
hat wye 9 0.03144187646093577 0.7495507603507059 - 0 0.031442 1
|
|
pan wye 10 0.5026260055412137 0.9526183602969864 0.3467901443380824 0.155836 0.849416 2
|
|
pan pan 11 0.7930488423451967 0.6505816637259333 0.5026260055412137 0.290423 1.642465 3
|
|
zee pan 12 0.3676141320555616 0.23614420670296965 0.5985540091064224 -0.230940 1.493294 3
|
|
eks pan 13 0.4915175580479536 0.7709126592971468 0.6117840605678454 -0.120267 2.243381 4
|
|
eks zee 14 0.5207382318405251 0.34141681118811673 0.4915175580479536 0.029221 2.764119 5
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint step -a ewma -f x -d 0.1,0.9 ../doc/data/medium | head -15
|
|
a b i x y x_ewma_0.1 x_ewma_0.9
|
|
pan pan 1 0.3467901443380824 0.7268028627434533 0.346790 0.346790
|
|
eks pan 2 0.7586799647899636 0.5221511083334797 0.387979 0.717491
|
|
wye wye 3 0.20460330576630303 0.33831852551664776 0.369642 0.255892
|
|
eks wye 4 0.38139939387114097 0.13418874328430463 0.370817 0.368849
|
|
wye pan 5 0.5732889198020006 0.8636244699032729 0.391064 0.552845
|
|
zee pan 6 0.5271261600918548 0.49322128674835697 0.404671 0.529698
|
|
eks zee 7 0.6117840605678454 0.1878849191181694 0.425382 0.603575
|
|
zee wye 8 0.5985540091064224 0.976181385699006 0.442699 0.599056
|
|
hat wye 9 0.03144187646093577 0.7495507603507059 0.401573 0.088203
|
|
pan wye 10 0.5026260055412137 0.9526183602969864 0.411679 0.461184
|
|
pan pan 11 0.7930488423451967 0.6505816637259333 0.449816 0.759862
|
|
zee pan 12 0.3676141320555616 0.23614420670296965 0.441596 0.406839
|
|
eks pan 13 0.4915175580479536 0.7709126592971468 0.446588 0.483050
|
|
eks zee 14 0.5207382318405251 0.34141681118811673 0.454003 0.516969
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint step -a ewma -f x -d 0.1,0.9 -o smooth,rough ../doc/data/medium | head -15
|
|
a b i x y x_ewma_smooth x_ewma_rough
|
|
pan pan 1 0.3467901443380824 0.7268028627434533 0.346790 0.346790
|
|
eks pan 2 0.7586799647899636 0.5221511083334797 0.387979 0.717491
|
|
wye wye 3 0.20460330576630303 0.33831852551664776 0.369642 0.255892
|
|
eks wye 4 0.38139939387114097 0.13418874328430463 0.370817 0.368849
|
|
wye pan 5 0.5732889198020006 0.8636244699032729 0.391064 0.552845
|
|
zee pan 6 0.5271261600918548 0.49322128674835697 0.404671 0.529698
|
|
eks zee 7 0.6117840605678454 0.1878849191181694 0.425382 0.603575
|
|
zee wye 8 0.5985540091064224 0.976181385699006 0.442699 0.599056
|
|
hat wye 9 0.03144187646093577 0.7495507603507059 0.401573 0.088203
|
|
pan wye 10 0.5026260055412137 0.9526183602969864 0.411679 0.461184
|
|
pan pan 11 0.7930488423451967 0.6505816637259333 0.449816 0.759862
|
|
zee pan 12 0.3676141320555616 0.23614420670296965 0.441596 0.406839
|
|
eks pan 13 0.4915175580479536 0.7709126592971468 0.446588 0.483050
|
|
eks zee 14 0.5207382318405251 0.34141681118811673 0.454003 0.516969
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr></table>
|
|
|
|
Example deriving uptime-delta from system uptime:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ each 10 uptime | mlr -p step -a delta -f 11
|
|
...
|
|
20:08 up 36 days, 10:38, 5 users, load averages: 1.42 1.62 1.73 0.000000
|
|
20:08 up 36 days, 10:38, 5 users, load averages: 1.55 1.64 1.74 0.020000
|
|
20:08 up 36 days, 10:38, 7 users, load averages: 1.58 1.65 1.74 0.010000
|
|
20:08 up 36 days, 10:38, 9 users, load averages: 1.78 1.69 1.76 0.040000
|
|
20:08 up 36 days, 10:39, 9 users, load averages: 2.12 1.76 1.78 0.070000
|
|
20:08 up 36 days, 10:39, 9 users, load averages: 2.51 1.85 1.81 0.090000
|
|
20:08 up 36 days, 10:39, 8 users, load averages: 2.79 1.92 1.83 0.070000
|
|
20:08 up 36 days, 10:39, 4 users, load averages: 2.64 1.90 1.83 -0.020000
|
|
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="tac"/><h2>tac</h2>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr tac --help
|
|
Usage: mlr tac
|
|
Prints records in reverse order from the order in which they were encountered.
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>Prints the records in the input stream in reverse order. Note: this
|
|
requires Miller to retain all input records in memory before any output records
|
|
are produced.
|
|
|
|
<table><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --icsv --opprint cat data/a.csv
|
|
a b c
|
|
1 2 3
|
|
4 5 6
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --icsv --opprint cat data/b.csv
|
|
a b c
|
|
7 8 9
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --icsv --opprint tac data/a.csv data/b.csv
|
|
a b c
|
|
7 8 9
|
|
4 5 6
|
|
1 2 3
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr></table>
|
|
<table><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --icsv --opprint put '$filename=FILENAME' then tac data/a.csv data/b.csv
|
|
a b c filename
|
|
7 8 9 data/b.csv
|
|
4 5 6 data/a.csv
|
|
1 2 3 data/a.csv
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr></table>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="tail"/><h2>tail</h2>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr tail --help
|
|
Usage: mlr tail [options]
|
|
-n {count} Tail count to print; default 10
|
|
-g {a,b,c} Optional group-by-field names for tail counts
|
|
Passes through the last n records, optionally by category.
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/> Prints the last <i>n</i> records in the input stream, optionally by category.
|
|
|
|
<table><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint tail -n 4 data/colored-shapes.dkvp
|
|
color shape flag i u v w x
|
|
blue square 1 99974 0.6189062525431605 0.2637962404841453 0.5311465405784674 6.210738209085753
|
|
blue triangle 0 99976 0.008110504040268474 0.8267274952432482 0.4732962944898885 6.146956761817328
|
|
yellow triangle 0 99990 0.3839424618160777 0.55952913620132 0.5113763011485609 4.307973891915119
|
|
yellow circle 1 99994 0.764950884927175 0.25284227383991364 0.49969878539567425 5.013809741826425
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint tail -n 1 -g shape data/colored-shapes.dkvp
|
|
color shape flag i u v w x
|
|
yellow triangle 0 99990 0.3839424618160777 0.55952913620132 0.5113763011485609 4.307973891915119
|
|
blue square 1 99974 0.6189062525431605 0.2637962404841453 0.5311465405784674 6.210738209085753
|
|
yellow circle 1 99994 0.764950884927175 0.25284227383991364 0.49969878539567425 5.013809741826425
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr></table>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="tee"/><h2>tee</h2>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr tee --help
|
|
Usage: mlr tee [options] {filename}
|
|
Passes through input records (like mlr cat) but also writes to specified output
|
|
file, using output-format flags from the command line (e.g. --ocsv). See also
|
|
the "tee" keyword within mlr put, which allows data-dependent filenames.
|
|
Options:
|
|
-a: append to existing file, if any, rather than overwriting.
|
|
--no-fflush: don't call fflush() after every record.
|
|
Any of the output-format command-line flags (see mlr -h). Example: using
|
|
mlr --icsv --opprint put '...' then tee --ojson ./mytap.dat then stats1 ...
|
|
the input is CSV, the output is pretty-print tabular, but the tee-file output
|
|
is written in JSON format.
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="top"/><h2>top</h2>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr top --help
|
|
Usage: mlr top [options]
|
|
-f {a,b,c} Value-field names for top counts.
|
|
-g {d,e,f} Optional group-by-field names for top counts.
|
|
-n {count} How many records to print per category; default 1.
|
|
-a Print all fields for top-value records; default is
|
|
to print only value and group-by fields. Requires a single
|
|
value-field name only.
|
|
--min Print top smallest values; default is top largest values.
|
|
-F Keep top values as floats even if they look like integers.
|
|
Prints the n records with smallest/largest values at specified fields,
|
|
optionally by category.
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
Note that <tt>top</tt> is distinct from <a href="#head"><tt>head</tt></a>
|
|
— <tt>head</tt> shows fields which appear first in the data stream;
|
|
<tt>top</tt> shows fields which are numerically largest (or smallest).
|
|
|
|
<table><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint top -n 4 -f x data/medium
|
|
top_idx x_top
|
|
1 0.999953
|
|
2 0.999823
|
|
3 0.999733
|
|
4 0.999563
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint top -n 2 -f x -g a then sort -f a data/medium
|
|
a top_idx x_top
|
|
eks 1 0.998811
|
|
eks 2 0.998534
|
|
hat 1 0.999953
|
|
hat 2 0.999733
|
|
pan 1 0.999403
|
|
pan 2 0.999044
|
|
wye 1 0.999823
|
|
wye 2 0.999264
|
|
zee 1 0.999490
|
|
zee 2 0.999438
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr></table>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="uniq"/><h2>uniq</h2>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr uniq --help
|
|
Usage: mlr uniq [options]
|
|
-g {d,e,f} Group-by-field names for uniq counts.
|
|
-c Show repeat counts in addition to unique values.
|
|
-n Show only the number of distinct values.
|
|
Prints distinct values for specified field names. With -c, same as
|
|
count-distinct. For uniq, -f is a synonym for -g.
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<table><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ wc -l data/colored-shapes.dkvp
|
|
10078 data/colored-shapes.dkvp
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr uniq -g color,shape data/colored-shapes.dkvp
|
|
color=yellow,shape=triangle
|
|
color=red,shape=square
|
|
color=red,shape=circle
|
|
color=purple,shape=triangle
|
|
color=yellow,shape=circle
|
|
color=purple,shape=square
|
|
color=yellow,shape=square
|
|
color=red,shape=triangle
|
|
color=green,shape=triangle
|
|
color=green,shape=square
|
|
color=blue,shape=circle
|
|
color=blue,shape=triangle
|
|
color=purple,shape=circle
|
|
color=blue,shape=square
|
|
color=green,shape=circle
|
|
color=orange,shape=triangle
|
|
color=orange,shape=square
|
|
color=orange,shape=circle
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint uniq -g color,shape -c then sort -f color,shape data/colored-shapes.dkvp
|
|
color shape count
|
|
blue circle 384
|
|
blue square 589
|
|
blue triangle 497
|
|
green circle 287
|
|
green square 454
|
|
green triangle 368
|
|
orange circle 68
|
|
orange square 128
|
|
orange triangle 107
|
|
purple circle 289
|
|
purple square 481
|
|
purple triangle 372
|
|
red circle 1207
|
|
red square 1874
|
|
red triangle 1560
|
|
yellow circle 356
|
|
yellow square 589
|
|
yellow triangle 468
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr><tr><td>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint uniq -n -g color,shape data/colored-shapes.dkvp
|
|
count
|
|
18
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
</td></tr></table>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="Expression_language_for_filter_and_put"/><h1>Expression language for filter and put</h1>
|
|
|
|
The essential usages of <tt>mlr filter</tt> and <tt>mlr put</tt> are for
|
|
record-selection and record-updating expressions, respectively. For example:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ cat data/small
|
|
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
|
|
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
|
|
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
|
|
a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463
|
|
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr filter '$a == "eks"' data/small
|
|
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
|
|
a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr put '$ab = $a . "_" . $b ' data/small
|
|
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533,ab=pan_pan
|
|
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797,ab=eks_pan
|
|
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776,ab=wye_wye
|
|
a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463,ab=eks_wye
|
|
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729,ab=wye_pan
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>The two are essentially the same command. The only differences are:
|
|
expressions sent to <tt>mlr filter</tt> must end with a boolean expression,
|
|
which is the filtering criterion; <tt>mlr filter</tt> expressions may not
|
|
reference the <tt>filter</tt> keyword within them; and <tt>mlr filter</tt>
|
|
expressions may not use <tt>tee</tt>, <tt>emit</tt>, <tt>emitp</tt>, or
|
|
<tt>emitf</tt>. All the rest is the same: in particular, you can define and
|
|
invoke functions and subroutines to help produce the final boolean statement,
|
|
and record fields may be assigned to in the statements preceding the final
|
|
boolean statement.
|
|
|
|
<p/>There are more details and more choices, of course, as detailed in the following sections.
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="Syntax"/><h2>Syntax</h2>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="Expression_formatting"/><h3>Expression formatting</h3>
|
|
|
|
<p/>Multiple expressions may be given, separated by semicolons, and each may refer to the ones before:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ ruby -e '10.times{|i|puts "i=#{i}"}' | mlr --opprint put '$j = $i + 1; $k = $i +$j'
|
|
i j k
|
|
0 1 1
|
|
1 2 3
|
|
2 3 5
|
|
3 4 7
|
|
4 5 9
|
|
5 6 11
|
|
6 7 13
|
|
7 8 15
|
|
8 9 17
|
|
9 10 19
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
Newlines within the expression are ignored, which can help increase legibility of complex expressions:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint put '
|
|
$nf = NF;
|
|
$nr = NR;
|
|
$fnr = FNR;
|
|
$filenum = FILENUM;
|
|
$filename = FILENAME
|
|
' data/small data/small2
|
|
a b i x y nf nr fnr filenum filename
|
|
pan pan 1 0.3467901443380824 0.7268028627434533 5 1 1 1 data/small
|
|
eks pan 2 0.7586799647899636 0.5221511083334797 5 2 2 1 data/small
|
|
wye wye 3 0.20460330576630303 0.33831852551664776 5 3 3 1 data/small
|
|
eks wye 4 0.38139939387114097 0.13418874328430463 5 4 4 1 data/small
|
|
wye pan 5 0.5732889198020006 0.8636244699032729 5 5 5 1 data/small
|
|
pan eks 9999 0.267481232652199086 0.557077185510228001 5 6 1 2 data/small2
|
|
wye eks 10000 0.734806020620654365 0.884788571337605134 5 7 2 2 data/small2
|
|
pan wye 10001 0.870530722602517626 0.009854780514656930 5 8 3 2 data/small2
|
|
hat wye 10002 0.321507044286237609 0.568893318795083758 5 9 4 2 data/small2
|
|
pan zee 10003 0.272054845593895200 0.425789896597056627 5 10 5 2 data/small2
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint filter '($x > 0.5 && $y < 0.5) || ($x < 0.5 && $y > 0.5)' then stats2 -a corr -f x,y data/medium
|
|
x_y_corr
|
|
-0.747994
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="Expressions_from_files"/><h3>Expressions from files</h3>
|
|
|
|
<p/>The simplest way to enter expressions for <tt>put</tt> and <tt>filter</tt> is between single quotes on the command line, e.g.
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --from data/small put '$xy = sqrt($x**2 + $y**2)'
|
|
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533,xy=0.805299
|
|
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797,xy=0.920998
|
|
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776,xy=0.395376
|
|
a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463,xy=0.404317
|
|
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729,xy=1.036584
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --from data/small put 'func f(a, b) { return sqrt(a**2 + b**2) } $xy = f($x, $y)'
|
|
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533,xy=0.805299
|
|
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797,xy=0.920998
|
|
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776,xy=0.395376
|
|
a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463,xy=0.404317
|
|
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729,xy=1.036584
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>You may, though, find it convenient to put expressions into files for reuse, and read them
|
|
<b>using the -f option</b>. For example:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ cat data/fe-example-3.mlr
|
|
func f(a, b) {
|
|
return sqrt(a**2 + b**2)
|
|
}
|
|
$xy = f($x, $y)
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --from data/small put -f data/fe-example-3.mlr
|
|
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533,xy=0.805299
|
|
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797,xy=0.920998
|
|
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776,xy=0.395376
|
|
a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463,xy=0.404317
|
|
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729,xy=1.036584
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>If you have some of the logic in a file and you want to write the rest on the command line, you
|
|
can <b>use the -f and -e options</b>:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ cat data/fe-example-4.mlr
|
|
func f(a, b) {
|
|
return sqrt(a**2 + b**2)
|
|
}
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --from data/small put -f data/fe-example-4.mlr -e '$xy = f($x, $y)'
|
|
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533,xy=0.805299
|
|
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797,xy=0.920998
|
|
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776,xy=0.395376
|
|
a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463,xy=0.404317
|
|
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729,xy=1.036584
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>A suggested use-case here is defining functions in files, and calling them from command-line expressions.
|
|
|
|
<p/>Another suggest use-case is putting default parameter values in files, e.g. using
|
|
<tt>begin{@count=ispresent(@count)?@count:10}</tt> in the file, where you can precede that using
|
|
<tt>begin{@count=40}</tt> using <tt>-e</tt>.
|
|
|
|
<p/>Moreover, you can have one or more <tt>-f</tt> expressions (maybe one
|
|
function per file, for example) and one or more <tt>-e</tt> expressions on the
|
|
command line. If you mix <tt>-f</tt> and <tt>-e</tt> then the expressions are
|
|
evaluated in the order encountered. (Since the expressions are all simply
|
|
concatenated together in order, don’t forget intervening semicolons: e.g.
|
|
not <tt>mlr put -e '$x=1' -e '$y=2 ...'</tt> but rather <tt>mlr put -e '$x=1;' -e
|
|
'$y=2' ...</tt>.)
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="Semicolons,_newlines,_and_curly_braces"/><h3>Semicolons, newlines, and curly braces</h3>
|
|
|
|
<p/>Miller uses semicolons as statement separators, not statement terminators. This means you can write:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
mlr put 'x=1'
|
|
mlr put 'x=1;$y=2'
|
|
mlr put 'x=1;$y=2;'
|
|
mlr put 'x=1;;;;$y=2;'
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>Semicolons are optional after closing curly braces (which close conditionals and loops as discussed below).
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ echo x=1,y=2 | mlr put 'while (NF < 10) { $[NF+1] = ""} $foo = "bar"'
|
|
x=1,y=2,3=,4=,5=,6=,7=,8=,9=,10=,foo=bar
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ echo x=1,y=2 | mlr put 'while (NF < 10) { $[NF+1] = ""}; $foo = "bar"'
|
|
x=1,y=2,3=,4=,5=,6=,7=,8=,9=,10=,foo=bar
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>Semicolons are required between statements even if those statements are on
|
|
separate lines. Newlines are for your convenience but have no syntactic
|
|
meaning: line endings do not terminate statements. For example, adjacent
|
|
assignment statements must be separated by semicolons even if those statements
|
|
are on separate lines:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
mlr put '
|
|
$x = 1
|
|
$y = 2 # Syntax error
|
|
'
|
|
|
|
mlr put '
|
|
$x = 1;
|
|
$y = 2 # This is OK
|
|
'
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>Bodies for all compound statements must be enclosed in curly braces, even if the body is a single statement:
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
mlr put 'if ($x == 1) $y = 2' # Syntax error
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
mlr put 'if ($x == 1) { $y = 2 }' # This is OK
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>Bodies for compound statements may be empty:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
mlr put 'if ($x == 1) { }' # This no-op is syntactically acceptable
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="Variables"/><h2>Variables</h2>
|
|
|
|
<p/>Miller has the following kinds of variables:
|
|
|
|
<p/> <b>Built-in variables</b> such as <tt>NF</tt>, <tt>NF</tt>,
|
|
<tt>FILENAME</tt>, <tt>PI</tt>, and <tt>E</tt>. These are all capital letters
|
|
and are read-only (although some of them change value from one record to
|
|
another).
|
|
|
|
<p/> <b>Fields of stream records</b>, accessed using the <tt>$</tt> prefix.
|
|
These refer to fields of the current data-stream record. For example, in
|
|
<tt>echo x=1,y=2 | mlr put '$z = $x + $y'</tt>, <tt>$x</tt> and <tt>$y</tt>
|
|
refer to input fields, and <tt>$z</tt> refers to a new, computed output field.
|
|
In a few contexts, presented below, you can refer to the entire record as
|
|
<tt>$*</tt>.
|
|
|
|
<p/> <b>Out-of-stream variables</b> accessed using the <tt>@</tt> prefix. These
|
|
refer to data which persist from one record to the next, including in
|
|
<tt>begin</tt> and <tt>end</tt> blocks (which execute before/after the record
|
|
stream is consumed, respectively). You use them to remember values across
|
|
records, such as sums, differences, counters, and so on. In a few contexts,
|
|
presented below, you can refer to the entire out-of-stream-variables collection
|
|
as <tt>@*</tt>.
|
|
|
|
<p/> <b>Local variables</b> are limited in scope and extent to the current
|
|
statements being executed: these include function arguments, bound variables in
|
|
for loops, and explicitly declared local variables.
|
|
|
|
<p/> <b>Keywords</b> are not variables, but since their names are reserved, you
|
|
cannot use these names for local variables.
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="Built-in_variables"/><h3>Built-in variables</h3>
|
|
|
|
<p/> These are written all in capital letters, such as <tt>NR</tt>,
|
|
<tt>NF</tt>, <tt>FILENAME</tt>, and only a small, specific set of them is
|
|
defined by Miller.
|
|
|
|
<p/>Miller supports the following five built-in variables for <a
|
|
href="#filter"><tt>filter</tt></a> and <tt>put</tt>, all <tt>awk</tt>-inspired:
|
|
<tt>NF</tt>, <tt>NR</tt>, <tt>FNR</tt>, <tt>FILENUM</tt>, and
|
|
<tt>FILENAME</tt>, as well as the mathematical constants <tt>PI</tt> and
|
|
<tt>E</tt>. Lastly, the <tt>ENV</tt> hashmap allows read access to environment
|
|
variables, e.g. <tt>ENV["HOME"]</tt> or <tt>ENV["foo_".$hostname]</tt>.
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr filter 'FNR == 2' data/small*
|
|
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
|
|
1=pan,2=pan,3=1,4=0.3467901443380824,5=0.7268028627434533
|
|
a=wye,b=eks,i=10000,x=0.734806020620654365,y=0.884788571337605134
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr put '$fnr = FNR' data/small*
|
|
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533,fnr=1
|
|
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797,fnr=2
|
|
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776,fnr=3
|
|
a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463,fnr=4
|
|
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729,fnr=5
|
|
1=a,2=b,3=i,4=x,5=y,fnr=1
|
|
1=pan,2=pan,3=1,4=0.3467901443380824,5=0.7268028627434533,fnr=2
|
|
1=eks,2=pan,3=2,4=0.7586799647899636,5=0.5221511083334797,fnr=3
|
|
1=wye,2=wye,3=3,4=0.20460330576630303,5=0.33831852551664776,fnr=4
|
|
1=eks,2=wye,3=4,4=0.38139939387114097,5=0.13418874328430463,fnr=5
|
|
1=wye,2=pan,3=5,4=0.5732889198020006,5=0.8636244699032729,fnr=6
|
|
a=pan,b=eks,i=9999,x=0.267481232652199086,y=0.557077185510228001,fnr=1
|
|
a=wye,b=eks,i=10000,x=0.734806020620654365,y=0.884788571337605134,fnr=2
|
|
a=pan,b=wye,i=10001,x=0.870530722602517626,y=0.009854780514656930,fnr=3
|
|
a=hat,b=wye,i=10002,x=0.321507044286237609,y=0.568893318795083758,fnr=4
|
|
a=pan,b=zee,i=10003,x=0.272054845593895200,y=0.425789896597056627,fnr=5
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/> Their values of <tt>NF</tt>, <tt>NR</tt>, <tt>FNR</tt>, <tt>FILENUM</tt>,
|
|
and <tt>FILENAME</tt> change from one record to the next as Miller scans
|
|
through your input data stream. The mathematical constants, of course, do not
|
|
change; <tt>ENV</tt> is populated from the system environment variables at the
|
|
time Miller starts and is read-only for the remainder of program execution.
|
|
|
|
<p/> Their <b>scope is global</b>: you can refer to them in any <tt>filter</tt>
|
|
or <tt>put</tt> statement. Their values are assigned by the input-record
|
|
reader:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --csv put '$nr = NR' data/a.csv
|
|
a,b,c,nr
|
|
1,2,3,1
|
|
4,5,6,2
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --csv repeat -n 3 then put '$nr = NR' data/a.csv
|
|
a,b,c,nr
|
|
1,2,3,1
|
|
1,2,3,1
|
|
1,2,3,1
|
|
4,5,6,2
|
|
4,5,6,2
|
|
4,5,6,2
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/> The <b>extent</b> is for the duration of the put/filter: in a
|
|
<tt>begin</tt> statement (which executes before the first input record is
|
|
consumed) you will find <tt>NR=1</tt> and in an <tt>end</tt>statement (which is
|
|
executed after the last input record is consumed) you will find <tt>NR</tt> to
|
|
be the total number of records ingested.
|
|
|
|
<p/> These are all <b>read-only</b> for the <tt>mlr put</tt> and <tt>mlr
|
|
filter</tt> DSLs: they may be assigned from, e.g. <tt>$nr=NR</tt>, but they may
|
|
not be assigned to: <tt>NR=100</tt> is a syntax error.
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="Field_names"/><h3>Field names</h3>
|
|
|
|
<p/>Field names must be specified using a <tt>$</tt> in <tt>filter</tt> and <a
|
|
href="#put"><tt>put</tt></a> expressions, even though the dollar signs
|
|
don’t appear in the data stream. For integer-indexed data, this looks
|
|
like <tt>awk</tt>’s <tt>$1,$2,$3</tt>, except that Miller allows
|
|
non-numeric names such as <tt>$quantity</tt> or <tt>$hostname</tt>. (Likewise,
|
|
enclose string literals in double quotes in <tt>filter</tt> expressions even
|
|
though they don’t appear in file data. In particular, <tt>mlr filter
|
|
'$x=="abc"'</tt> passes through the record <tt>x=abc</tt>.)
|
|
|
|
<p/>If field names have <b>special characters</b> such as <tt>.</tt> then you can use
|
|
braces, e.g. <tt>'${field.name}'</tt>.
|
|
|
|
<p/>You may also use a <b>computed field name</b> in square brackets, e.g.
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ echo a=3,b=4 | mlr filter '$["x"] < 0.5'
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ echo s=green,t=blue,a=3,b=4 | mlr put '$[$s."_".$t] = $a * $b'
|
|
s=green,t=blue,a=3,b=4,green_blue=12
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/> The names of record fields depend on the contents of your input data stream, and their
|
|
values change from one record to the next as Miller scans through your input
|
|
data stream.
|
|
|
|
<p/> Their <b>extent</b> is limited to the current record; their <b>scope</b>
|
|
is the <tt>filter</tt> or <tt>put</tt> command in which they appear.
|
|
|
|
<p/> These are <b>read-write</b>: you can do <tt>$y=2*$x</tt>,
|
|
<tt>$x=$x+1</tt>, etc.
|
|
|
|
<p/> Records are Miller’s output: field names present in the input
|
|
stream are passed through to output (written to standard output) unless fields
|
|
are removed with <tt>cut</tt>, or records are excluded with <tt>filter</tt> or
|
|
<tt>put -q</tt>, etc. Simply assign a value to a field and it will be output.
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="Local_variables"/><h3>Local variables</h3>
|
|
|
|
<p/>There are three kinds of local variables: <b>arguments</b> to
|
|
functions/subroutines, <b>variables bound within for-loops</b>, and
|
|
<b>locals</b> defined within control blocks. They may be untyped using
|
|
<tt>var</tt>, or typed using <tt>num</tt>, <tt>int</tt>, <tt>float</tt>,
|
|
<tt>str</tt>, and <tt>bool</tt>.
|
|
|
|
<p/>For example:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ # Here I'm using a specified random-number seed so this example always
|
|
# produces the same output for this web document: in everyday practice we
|
|
# would leave off the --seed 12345 part.
|
|
mlr --seed 12345 seqgen --start 1 --stop 10 then put '
|
|
func f(a, b) { # function arguments a and b
|
|
r = 0.0; # local r scoped to the function
|
|
for (int i = 0; i < 6; i += 1) { # local i scoped to the for-loop
|
|
num u = urand(); # local u scoped to the for-loop
|
|
r += u; # updates r from the enclosing scope
|
|
}
|
|
r /= 6;
|
|
return a + (b - a) * r;
|
|
}
|
|
num o = f(10, 20); # local to the top-level scope
|
|
$o = o;
|
|
'
|
|
i=1,o=14.662901
|
|
i=2,o=17.881983
|
|
i=3,o=14.586560
|
|
i=4,o=16.402409
|
|
i=5,o=16.336598
|
|
i=6,o=14.622701
|
|
i=7,o=15.983753
|
|
i=8,o=13.852177
|
|
i=9,o=15.472899
|
|
i=10,o=15.643912
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>Notes:
|
|
|
|
<ul>
|
|
|
|
<li/> Parameter names are bound to their arguments but can be reassigned, e.g.
|
|
if there is a parameter named <tt>a</tt> then you can reassign the value of
|
|
<tt>a</tt> to be something else within the function if you like.
|
|
|
|
<li/> However, you cannot redeclare the <i>type</i> of an argument or a local:
|
|
<tt>var a=1; var a=2</tt> is an error but
|
|
<tt>var a=1; a=2</tt> is OK.
|
|
|
|
<li/> Type declarations using <tt>var</tt>, or typed using <tt>num</tt>,
|
|
<tt>int</tt>, <tt>float</tt>, <tt>str</tt>, and <tt>bool</tt> are necessary to
|
|
declare local variables. Function arguments and variables bound in for-loops
|
|
over stream records and out-of-stream variables are <i>implicitly</i> declared
|
|
using <tt>var</tt>. (Some examples are shown below.)
|
|
|
|
<li/> Type-checking is done at assignment time. For example, <tt>float f =
|
|
0</tt> is an error (since <tt>0</tt> is an integer), as is <tt>float f = 0.0; f
|
|
= 1</tt>. For this reason I prefer to use <tt>num</tt> over <tt>float</tt>
|
|
since <tt>num</tt> encompasses integer and floating-point values.
|
|
|
|
<li/> All argument-passing is positional rather than by name; arguments are
|
|
passed by value, not by reference.
|
|
|
|
<li/> You can define locals (using <tt>var</tt>, <tt>num</tt>, etc.) at any
|
|
scope (if-statements, else-statements, while-loops, for-loops, or the top-level
|
|
scope), and nested scopes will have access (more details on scope in the next
|
|
section). If you define a local variable with the same name inside an inner
|
|
scope, then a new variable is created with the narrower scope.
|
|
|
|
<li/> If you assign to a local variable for the first time in a scope without
|
|
declaring it as <tt>var</tt>, <tt>num</tt>, etc. then: if it exists in an outer
|
|
scope, that outer-scope variable will be updated; if not, it will be defined in
|
|
the current scope as if <tt>var</tt> had been used. I recommend always
|
|
declaring variables explicitly to make the intended scoping clear.
|
|
|
|
<li/> Functions and subroutines never have access to locals from their callee
|
|
(unless passed by value as arguments).
|
|
|
|
<li/> Bound variables in for-loops over stream records and out-of-stream
|
|
variables are implicitly local to that block. E.g. in
|
|
<tt>for (k, v in $*) { ... }</tt>
|
|
<tt>for ((k1, k2), v in @*) { ... }</tt>
|
|
if there are <tt>k</tt>, <tt>v</tt>, etc. in the enclosing scope then those
|
|
will be masked by the loop-local bound variables in the loop, and moreover
|
|
the values of the loop-local bound variables are not available after the
|
|
end of the loop.
|
|
|
|
<li/> For C-style triple-for loops, if a for-loop variable is defined using
|
|
<tt>var</tt>, <tt>int</tt>, etc. then it is scoped to that for-loop. E.g.
|
|
<tt>for (i = 0; i < 10; i += 1) { ... }</tt> and
|
|
<tt>for (int i = 0; i < 10; i += 1) { ... }</tt>.
|
|
|
|
</ul>
|
|
|
|
<p/> The following example demonstrates the scope rules:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ cat data/scope-example.mlr
|
|
func f(a) { # argument is local to the function
|
|
var b = 100; # local to the function
|
|
c = 100; # local to the function; does not overwrite outer c
|
|
return a + 1;
|
|
}
|
|
var a = 10; # local at top level
|
|
var b = 20; # local at top level
|
|
c = 30; # local at top level; there is no more-outer-scope c
|
|
if (NR == 3) {
|
|
var a = 40; # scoped to the if-statement; doesn't overwrite outer a
|
|
b = 50; # not scoped to the if-statement; overwrites outer b
|
|
c = 60; # not scoped to the if-statement; overwrites outer c
|
|
d = 70; # there is no outer d so a local d is created here
|
|
|
|
$inner_a = a;
|
|
$inner_b = b;
|
|
$inner_c = c;
|
|
$inner_d = d;
|
|
}
|
|
$outer_a = a;
|
|
$outer_b = b;
|
|
$outer_c = c;
|
|
$outer_d = d; # there is no outer d defined so no assignment happens
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ cat data/scope-example.dat
|
|
n=1,x=123
|
|
n=2,x=456
|
|
n=3,x=789
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --oxtab --from data/scope-example.dat put -f data/scope-example.mlr
|
|
n 1
|
|
x 123
|
|
outer_a 10
|
|
outer_b 20
|
|
outer_c 30
|
|
|
|
n 2
|
|
x 456
|
|
outer_a 10
|
|
outer_b 20
|
|
outer_c 30
|
|
|
|
n 3
|
|
x 789
|
|
inner_a 40
|
|
inner_b 50
|
|
inner_c 60
|
|
inner_d 70
|
|
outer_a 10
|
|
outer_b 50
|
|
outer_c 60
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/> And this example demonstrates the type-declaration rules:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ cat data/type-decl-example.mlr
|
|
subr s(a, str b, int c) { # a is implicitly var (untyped).
|
|
# b is explicitly str.
|
|
# c is explicitly int.
|
|
# The type-checking is done at the callsite
|
|
# when arguments are bound to parameters.
|
|
#
|
|
var b = 100; # error # Re-declaration in the same scope is disallowed.
|
|
int n = 10; # Declaration of variable local to the subroutine.
|
|
n = 20; # Assignment is OK.
|
|
int n = 30; # error # Re-declaration in the same scope is disallowed.
|
|
str n = "abc"; # error # Re-declaration in the same scope is disallowed.
|
|
#
|
|
float f1 = 1; # error # 1 is an int, not a float.
|
|
float f2 = 2.0; # 2.0 is a float.
|
|
num f3 = 3; # 3 is a num.
|
|
num f4 = 4.0; # 4.0 is a num.
|
|
} #
|
|
#
|
|
call s(1, 2, 3); # Type-assertion '3 is int' is done here at the callsite.
|
|
#
|
|
k = "def"; # Top-level variable k.
|
|
#
|
|
for (str k, v in $*) { # k and v are bound here, masking outer k.
|
|
print k . ":" . v; # k is explicitly str; v is implicitly var.
|
|
} #
|
|
#
|
|
print "k is".k; # k at this scope level is still "def".
|
|
print "v is".v; # v is undefined in this scope.
|
|
#
|
|
i = -1; #
|
|
for (i = 1, int j = 2; i <= 10; i += 1, j *= 2) { # C-style triple-for variables use enclosing scope, unless
|
|
# declared local: i is outer, j is local to the loop.
|
|
print "inner i =" . i; #
|
|
print "inner j =" . j; #
|
|
} #
|
|
print "outer i =" . i; # i has been modified by the loop.
|
|
print "outer j =" . j; # j is undefined in this scope.
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="Out-of-stream_variables"/><h3>Out-of-stream variables</h3>
|
|
|
|
<p/> These are prefixed with an at-sign, e.g. <tt>@sum</tt>. Furthermore,
|
|
unlike built-in variables and stream-record fields, they are maintained in an
|
|
arbitrarily nested hashmap: you can do <tt>@sum += $quanity</tt>, or
|
|
<tt>@sum[$color] += $quanity</tt>, or <tt>@sum[$color][$shape] +=
|
|
$quanity</tt>. The keys for the multi-level hashmap can be any expression which
|
|
evaluates to string or integer: e.g. <tt>@sum[NR] = $a + $b</tt>,
|
|
<tt>@sum[$a."-".$b] = $x</tt>, etc.
|
|
|
|
<p/> Their names and their values are entirely under your control; they change
|
|
only when you assign to them.
|
|
|
|
<p/> Just as for field names in stream records, if you want to define out-of-stream variables
|
|
with <b>special characters</b> such as <tt>.</tt> then you can use braces, e.g. <tt>'@{variable.name}["index"]'</tt>.
|
|
|
|
<p/>You may use a <b>computed key </b> in square brackets, e.g.
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ echo s=green,t=blue,a=3,b=4 | mlr put -q '@[$s."_".$t] = $a * $b; emit all'
|
|
green_blue=12
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/> Out-of-stream variables are <b>scoped</b> to the <tt>put</tt> command in
|
|
which they appear. In particular, if you have two or more <tt>put</tt>
|
|
commands separated by <tt>then</tt>, each put will have its own set of
|
|
out-of-stream variables:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ cat data/a.dkvp
|
|
a=1,b=2,c=3
|
|
a=4,b=5,c=6
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr put '@sum += $a; end {emit @sum}' then put 'ispresent($a) {$a=10*$a; @sum += $a}; end {emit @sum}' data/a.dkvp
|
|
a=10,b=2,c=3
|
|
a=40,b=5,c=6
|
|
sum=5
|
|
sum=50
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/> Out-of-stream variables are read-write: you can do <tt>$sum=@sum</tt>, <tt>@sum=$sum</tt>,
|
|
etc.
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="Indexed_out-of-stream_variables"/><h3>Indexed out-of-stream variables</h3>
|
|
|
|
<p/>Using an index on the <tt>@count</tt> and <tt>@sum</tt> variables, we get the benefit of the
|
|
<tt>-g</tt> (group-by) option which <tt>mlr stats1</tt> and various other Miller commands have:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr put -q '
|
|
@x_count[$a] += 1;
|
|
@x_sum[$a] += $x;
|
|
end {
|
|
emit @x_count, "a";
|
|
emit @x_sum, "a";
|
|
}
|
|
' ../data/small
|
|
a=pan,x_count=2
|
|
a=eks,x_count=3
|
|
a=wye,x_count=2
|
|
a=zee,x_count=2
|
|
a=hat,x_count=1
|
|
a=pan,x_sum=0.849416
|
|
a=eks,x_sum=1.751863
|
|
a=wye,x_sum=0.777892
|
|
a=zee,x_sum=1.125680
|
|
a=hat,x_sum=0.031442
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr stats1 -a count,sum -f x -g a ../data/small
|
|
a=pan,x_count=2,x_sum=0.849416
|
|
a=eks,x_count=3,x_sum=1.751863
|
|
a=wye,x_count=2,x_sum=0.777892
|
|
a=zee,x_count=2,x_sum=1.125680
|
|
a=hat,x_count=1,x_sum=0.031442
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>Indices can be arbitrarily deep — here there are two or more of them:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --from data/medium put -q '
|
|
@x_count[$a][$b] += 1;
|
|
@x_sum[$a][$b] += $x;
|
|
end {
|
|
emit (@x_count, @x_sum), "a", "b";
|
|
}
|
|
'
|
|
a=pan,b=pan,x_count=427,x_sum=219.185129
|
|
a=pan,b=wye,x_count=395,x_sum=198.432931
|
|
a=pan,b=eks,x_count=429,x_sum=216.075228
|
|
a=pan,b=hat,x_count=417,x_sum=205.222776
|
|
a=pan,b=zee,x_count=413,x_sum=205.097518
|
|
a=eks,b=pan,x_count=371,x_sum=179.963030
|
|
a=eks,b=wye,x_count=407,x_sum=196.945286
|
|
a=eks,b=zee,x_count=357,x_sum=176.880365
|
|
a=eks,b=eks,x_count=413,x_sum=215.916097
|
|
a=eks,b=hat,x_count=417,x_sum=208.783171
|
|
a=wye,b=wye,x_count=377,x_sum=185.295850
|
|
a=wye,b=pan,x_count=392,x_sum=195.847900
|
|
a=wye,b=hat,x_count=426,x_sum=212.033183
|
|
a=wye,b=zee,x_count=385,x_sum=194.774048
|
|
a=wye,b=eks,x_count=386,x_sum=204.812961
|
|
a=zee,b=pan,x_count=389,x_sum=202.213804
|
|
a=zee,b=wye,x_count=455,x_sum=233.991394
|
|
a=zee,b=eks,x_count=391,x_sum=190.961778
|
|
a=zee,b=zee,x_count=403,x_sum=206.640635
|
|
a=zee,b=hat,x_count=409,x_sum=191.300006
|
|
a=hat,b=wye,x_count=423,x_sum=208.883010
|
|
a=hat,b=zee,x_count=385,x_sum=196.349450
|
|
a=hat,b=eks,x_count=389,x_sum=189.006793
|
|
a=hat,b=hat,x_count=381,x_sum=182.853532
|
|
a=hat,b=pan,x_count=363,x_sum=168.553807
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
The idea is that <tt>stats1</tt>, and other Miller commands, encapsulate
|
|
frequently-used patterns with a minimum of keystroking (and run a little
|
|
faster), whereas using out-of-stream variables you have more flexibility and
|
|
control in what you do.
|
|
|
|
<p/>Begin/end blocks can be mixed with pattern/action blocks. For example:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr put '
|
|
begin {
|
|
@num_total = 0;
|
|
@num_positive = 0;
|
|
};
|
|
@num_total += 1;
|
|
$x > 0.0 {
|
|
@num_positive += 1;
|
|
$y = log10($x); $z = sqrt($y)
|
|
};
|
|
end {
|
|
emitf @num_total, @num_positive
|
|
}
|
|
' data/put-gating-example-1.dkvp
|
|
x=-1
|
|
x=0
|
|
x=1,y=0.000000,z=0.000000
|
|
x=2,y=0.301030,z=0.548662
|
|
x=3,y=0.477121,z=0.690740
|
|
num_total=5,num_positive=3
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="Aggregate_variable_assignments"/><h3>Aggregate variable assignments</h3>
|
|
|
|
<p/>There are three remaining kinds of variable assignment using out-of-stream
|
|
variables, the last two of which use the <tt>$*</tt> syntax:
|
|
<ul>
|
|
<li/> Recursive copy of out-of-stream variables
|
|
<li/> Out-of-stream variable assigned to full stream record
|
|
<li/> Full stream record assigned to an out-of-stream variable
|
|
</ul>
|
|
|
|
<p/> Example recursive copy of out-of-stream variables:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint put -q '@v["sum"] += $x; @v["count"] += 1; end{dump; @w = @v; dump}' data/small
|
|
{
|
|
"v": {
|
|
"sum": 2.264762,
|
|
"count": 5
|
|
}
|
|
}
|
|
{
|
|
"v": {
|
|
"sum": 2.264762,
|
|
"count": 5
|
|
},
|
|
"w": {
|
|
"sum": 2.264762,
|
|
"count": 5
|
|
}
|
|
}
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>Example of out-of-stream variable assigned to full stream record, where the 2nd record is stashed, and the 4th record is overwritten with that:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr put 'NR == 2 {@keep = $*}; NR == 4 {$* = @keep}' data/small
|
|
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
|
|
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
|
|
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
|
|
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
|
|
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>Example of full stream record assigned to an out-of-stream variable, finding
|
|
the record for which the <tt>x</tt> field has the largest value in the input
|
|
stream:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ cat data/small
|
|
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
|
|
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
|
|
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
|
|
a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463
|
|
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint put -q 'isnull(@xmax) || $x > @xmax {@xmax=$x; @recmax=$*}; end {emit @recmax}' data/small
|
|
a b i x y
|
|
eks pan 2 0.7586799647899636 0.5221511083334797
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="Keywords_for_filter_and_put"/><h3>Keywords for filter and put</h3>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --help-all-keywords
|
|
all: used in "emit", "emitp", and "unset" as a synonym for @*
|
|
|
|
begin: defines a block of statements to be executed before input records
|
|
are ingested. The body statements must be wrapped in curly braces.
|
|
Example: 'begin { @count = 0 }'
|
|
|
|
bool: declares a boolean local variable in the current curly-braced scope.
|
|
Type-checking happens at assignment: 'bool b = 1' is an error.
|
|
|
|
break: causes execution to continue after the body of the current
|
|
for/while/do-while loop.
|
|
|
|
call: used for invoking a user-defined subroutine.
|
|
Example: 'subr s(k,v) { print k . " is " . v} call s("a", $a)'
|
|
|
|
continue: causes execution to skip the remaining statements in the body of
|
|
the current for/while/do-while loop. For-loop increments are still applied.
|
|
|
|
do: with "while", introduces a do-while loop. The body statements must be wrapped
|
|
in curly braces.
|
|
|
|
dump: prints all currently defined out-of-stream variables immediately
|
|
to stdout as JSON.
|
|
|
|
With >, >>, or |, the data do not become part of the output record stream but
|
|
are instead redirected.
|
|
|
|
The > and >> are for write and append, as in the shell, but (as with awk) the
|
|
file-overwrite for > is on first write, not per record. The | is for piping to
|
|
a process which will process the data. There will be one open file for each
|
|
distinct file name (for > and >>) or one subordinate process for each distinct
|
|
value of the piped-to command (for |). Output-formatting flags are taken from
|
|
the main command line.
|
|
|
|
Example: mlr --from f.dat put -q '@v[NR]=$*; end { dump }'
|
|
Example: mlr --from f.dat put -q '@v[NR]=$*; end { dump > "mytap.dat"}'
|
|
Example: mlr --from f.dat put -q '@v[NR]=$*; end { dump >> "mytap.dat"}'
|
|
Example: mlr --from f.dat put -q '@v[NR]=$*; end { dump | "jq .[]"}'
|
|
|
|
edump: prints all currently defined out-of-stream variables immediately
|
|
to stderr as JSON.
|
|
|
|
Example: mlr --from f.dat put -q '@v[NR]=$*; end { edump }'
|
|
|
|
elif: the way Miller spells "else if". The body statements must be wrapped
|
|
in curly braces.
|
|
|
|
else: terminates an if/elif/elif chain. The body statements must be wrapped
|
|
in curly braces.
|
|
|
|
emit: inserts an out-of-stream variable into the output record stream. Hashmap
|
|
indices present in the data but not slotted by emit arguments are not output.
|
|
|
|
With >, >>, or |, the data do not become part of the output record stream but
|
|
are instead redirected.
|
|
|
|
The > and >> are for write and append, as in the shell, but (as with awk) the
|
|
file-overwrite for > is on first write, not per record. The | is for piping to
|
|
a process which will process the data. There will be one open file for each
|
|
distinct file name (for > and >>) or one subordinate process for each distinct
|
|
value of the piped-to command (for |). Output-formatting flags are taken from
|
|
the main command line.
|
|
|
|
You can use any of the output-format command-line flags, e.g. --ocsv, --ofs,
|
|
etc., to control the format of the output if the output is redirected. See also mlr -h.
|
|
|
|
Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emit @sums'
|
|
Example: mlr --from f.dat put --ojson '@sums[$a][$b]+=$x; emit > "tap-".$a.$b.".dat", @sums'
|
|
Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emit @sums, "index1", "index2"'
|
|
Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emit @*, "index1", "index2"'
|
|
Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emit > "mytap.dat", @*, "index1", "index2"'
|
|
Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emit >> "mytap.dat", @*, "index1", "index2"'
|
|
Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emit | "gzip > mytap.dat.gz", @*, "index1", "index2"'
|
|
Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emit > stderr, @*, "index1", "index2"'
|
|
Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emit | "grep somepattern", @*, "index1", "index2"'
|
|
|
|
Please see http://johnkerl.org/miller/doc for more information.
|
|
|
|
emitf: inserts non-indexed out-of-stream variable(s) side-by-side into the
|
|
output record stream.
|
|
|
|
With >, >>, or |, the data do not become part of the output record stream but
|
|
are instead redirected.
|
|
|
|
The > and >> are for write and append, as in the shell, but (as with awk) the
|
|
file-overwrite for > is on first write, not per record. The | is for piping to
|
|
a process which will process the data. There will be one open file for each
|
|
distinct file name (for > and >>) or one subordinate process for each distinct
|
|
value of the piped-to command (for |). Output-formatting flags are taken from
|
|
the main command line.
|
|
|
|
You can use any of the output-format command-line flags, e.g. --ocsv, --ofs,
|
|
etc., to control the format of the output if the output is redirected. See also mlr -h.
|
|
|
|
Example: mlr --from f.dat put '@a=$i;@b+=$x;@c+=$y; emitf @a'
|
|
Example: mlr --from f.dat put --oxtab '@a=$i;@b+=$x;@c+=$y; emitf > "tap-".$i.".dat", @a'
|
|
Example: mlr --from f.dat put '@a=$i;@b+=$x;@c+=$y; emitf @a, @b, @c'
|
|
Example: mlr --from f.dat put '@a=$i;@b+=$x;@c+=$y; emitf > "mytap.dat", @a, @b, @c'
|
|
Example: mlr --from f.dat put '@a=$i;@b+=$x;@c+=$y; emitf >> "mytap.dat", @a, @b, @c'
|
|
Example: mlr --from f.dat put '@a=$i;@b+=$x;@c+=$y; emitf > stderr, @a, @b, @c'
|
|
Example: mlr --from f.dat put '@a=$i;@b+=$x;@c+=$y; emitf | "grep somepattern", @a, @b, @c'
|
|
Example: mlr --from f.dat put '@a=$i;@b+=$x;@c+=$y; emitf | "grep somepattern > mytap.dat", @a, @b, @c'
|
|
|
|
Please see http://johnkerl.org/miller/doc for more information.
|
|
|
|
emitp: inserts an out-of-stream variable into the output record stream.
|
|
Hashmap indices present in the data but not slotted by emitp arguments are
|
|
output concatenated with ":".
|
|
|
|
With >, >>, or |, the data do not become part of the output record stream but
|
|
are instead redirected.
|
|
|
|
The > and >> are for write and append, as in the shell, but (as with awk) the
|
|
file-overwrite for > is on first write, not per record. The | is for piping to
|
|
a process which will process the data. There will be one open file for each
|
|
distinct file name (for > and >>) or one subordinate process for each distinct
|
|
value of the piped-to command (for |). Output-formatting flags are taken from
|
|
the main command line.
|
|
|
|
You can use any of the output-format command-line flags, e.g. --ocsv, --ofs,
|
|
etc., to control the format of the output if the output is redirected. See also mlr -h.
|
|
|
|
Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emitp @sums'
|
|
Example: mlr --from f.dat put --opprint '@sums[$a][$b]+=$x; emitp > "tap-".$a.$b.".dat", @sums'
|
|
Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emitp @sums, "index1", "index2"'
|
|
Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emitp @*, "index1", "index2"'
|
|
Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emitp > "mytap.dat", @*, "index1", "index2"'
|
|
Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emitp >> "mytap.dat", @*, "index1", "index2"'
|
|
Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emitp | "gzip > mytap.dat.gz", @*, "index1", "index2"'
|
|
Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emitp > stderr, @*, "index1", "index2"'
|
|
Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emitp | "grep somepattern", @*, "index1", "index2"'
|
|
|
|
Please see http://johnkerl.org/miller/doc for more information.
|
|
|
|
end: defines a block of statements to be executed after input records
|
|
are ingested. The body statements must be wrapped in curly braces.
|
|
Example: 'end { emit @count }'
|
|
Example: 'end { eprint "Final count is " . @count }'
|
|
|
|
eprint: prints expression immediately to stderr.
|
|
Example: mlr --from f.dat put -q 'eprint "The sum of x and y is ".($x+$y)'
|
|
Example: mlr --from f.dat put -q 'for (k, v in $*) { eprint k . " => " . v }'
|
|
Example: mlr --from f.dat put '(NR % 1000 == 0) { eprint "Checkpoint ".NR}'
|
|
|
|
eprintn: prints expression immediately to stderr, without trailing newline.
|
|
Example: mlr --from f.dat put -q 'eprintn "The sum of x and y is ".($x+$y); eprint ""'
|
|
|
|
false: the boolean literal value.
|
|
|
|
filter: includes/excludes the record in the output record stream.
|
|
|
|
Example: mlr --from f.dat put 'filter (NR == 2 || $x > 5.4)'
|
|
|
|
Instead of put with 'filter false' you can simply use put -q. The following
|
|
uses the input record to accumulate data but only prints the running sum
|
|
without printing the input record:
|
|
|
|
Example: mlr --from f.dat put -q '@running_sum += $x * $y; emit @running_sum'
|
|
|
|
float: declares a floating-point local variable in the current curly-braced scope.
|
|
Type-checking happens at assignment: 'float x = 0' is an error.
|
|
|
|
for: defines a for-loop using one of three styles. The body statements must
|
|
be wrapped in curly braces.
|
|
For-loop over stream record:
|
|
Example: 'for (k, v in $*) { ... }'
|
|
For-loop over out-of-stream variables:
|
|
Example: 'for (k, v in @counts) { ... }'
|
|
Example: 'for ((k1, k2), v in @counts) { ... }'
|
|
Example: 'for ((k1, k2, k3), v in @*) { ... }'
|
|
C-style for-loop:
|
|
Example: 'for (var i = 0, var b = 1; i < 10; i += 1, b *= 2) { ... }'
|
|
|
|
func: used for defining a user-defined function.
|
|
Example: 'func f(a,b) { return sqrt(a**2+b**2)} $d = f($x, $y)'
|
|
|
|
if: starts an if/elif/elif chain. The body statements must be wrapped
|
|
in curly braces.
|
|
|
|
in: used in for-loops over stream records or out-of-stream variables.
|
|
|
|
int: declares an integer local variable in the current curly-braced scope.
|
|
Type-checking happens at assignment: 'int x = 0.0' is an error.
|
|
|
|
num: declares an int/float local variable in the current curly-braced scope.
|
|
Type-checking happens at assignment: 'num b = true' is an error.
|
|
|
|
print: prints expression immediately to stdout.
|
|
Example: mlr --from f.dat put -q 'print "The sum of x and y is ".($x+$y)'
|
|
Example: mlr --from f.dat put -q 'for (k, v in $*) { print k . " => " . v }'
|
|
Example: mlr --from f.dat put '(NR % 1000 == 0) { print > stderr, "Checkpoint ".NR}'
|
|
|
|
printn: prints expression immediately to stdout, without trailing newline.
|
|
Example: mlr --from f.dat put -q 'printn "."; end { print "" }'
|
|
|
|
return: specifies the return value from a user-defined function.
|
|
Omitted return statements (including via if-branches) result in an absent-null
|
|
return value, which in turns results in a skipped assignment to an LHS.
|
|
|
|
stderr: Used for tee, emit, emitf, emitp, print, and dump in place of filename
|
|
to print to standard error.
|
|
|
|
stdout: Used for tee, emit, emitf, emitp, print, and dump in place of filename
|
|
to print to standard output.
|
|
|
|
str: declares a string local variable in the current curly-braced scope.
|
|
Type-checking happens at assignment.
|
|
|
|
subr: used for defining a subroutine.
|
|
Example: 'subr s(k,v) { print k . " is " . v} call s("a", $a)'
|
|
|
|
tee: prints the current record to specified file.
|
|
This is an immediate print to the specified file (except for pprint format
|
|
which of course waits until the end of the input stream to format all output).
|
|
|
|
The > and >> are for write and append, as in the shell, but (as with awk) the
|
|
file-overwrite for > is on first write, not per record. The | is for piping to
|
|
a process which will process the data. There will be one open file for each
|
|
distinct file name (for > and >>) or one subordinate process for each distinct
|
|
value of the piped-to command (for |). Output-formatting flags are taken from
|
|
the main command line.
|
|
|
|
You can use any of the output-format command-line flags, e.g. --ocsv, --ofs,
|
|
etc., to control the format of the output. See also mlr -h.
|
|
|
|
Example: mlr --from f.dat put 'tee > "/tmp/data-".$a, $*'
|
|
Example: mlr --from f.dat put 'tee >> "/tmp/data-".$a.$b, $*'
|
|
Example: mlr --from f.dat put 'tee > stderr, $*'
|
|
Example: mlr --from f.dat put -q 'tee | "tr [a-z\] [A-Z\]", $*'
|
|
Example: mlr --from f.dat put -q 'tee | "tr [a-z\] [A-Z\] > /tmp/data-".$a, $*'
|
|
Example: mlr --from f.dat put -q 'tee | "gzip > /tmp/data-".$a.".gz", $*'
|
|
Example: mlr --from f.dat put -q --ojson 'tee | "gzip > /tmp/data-".$a.".gz", $*'
|
|
|
|
true: the boolean literal value.
|
|
|
|
unset: clears field(s) from the current record, or an out-of-stream variable.
|
|
|
|
Example: mlr --from f.dat put 'unset $x'
|
|
Example: mlr --from f.dat put 'unset $*'
|
|
Example: mlr --from f.dat put 'for (k, v in $*) { if (k =~ "a.*") { unset $[k] } }'
|
|
Example: mlr --from f.dat put '...; unset @sums'
|
|
Example: mlr --from f.dat put '...; unset @sums["green"]'
|
|
Example: mlr --from f.dat put '...; unset @*'
|
|
|
|
var: declares an untyped local variable in the current curly-braced scope.
|
|
Examples: 'var a=1', 'var xyz=""'
|
|
|
|
while: introduces a while loop, or with "do", introduces a do-while loop.
|
|
The body statements must be wrapped in curly braces.
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="Control_structures"/><h2>Control structures</h2>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="Pattern-action_blocks"/><h3>Pattern-action blocks</h3>
|
|
|
|
<p/>These are reminiscent of <tt>awk</tt> syntax. They can be used to allow
|
|
assignments to be done only when appropriate — e.g. for math-function
|
|
domain restrictions, regex-matching, and so on:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr cat data/put-gating-example-1.dkvp
|
|
x=-1
|
|
x=0
|
|
x=1
|
|
x=2
|
|
x=3
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr put '$x > 0.0 { $y = log10($x); $z = sqrt($y) }' data/put-gating-example-1.dkvp
|
|
x=-1
|
|
x=0
|
|
x=1,y=0.000000,z=0.000000
|
|
x=2,y=0.301030,z=0.548662
|
|
x=3,y=0.477121,z=0.690740
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr cat data/put-gating-example-2.dkvp
|
|
a=abc_123
|
|
a=some other name
|
|
a=xyz_789
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr put '$a =~ "([a-z]+)_([0-9]+)" { $b = "left_\1"; $c = "right_\2" }' data/put-gating-example-2.dkvp
|
|
a=abc_123,b=left_abc,c=right_123
|
|
a=some other name
|
|
a=xyz_789,b=left_xyz,c=right_789
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>This produces heteregenous output which Miller, of course, has no problems
|
|
with (see <a href="record-heterogeneity.html">Record-heterogeneity</a>). But if you
|
|
want homogeneous output, the curly braces can be replaced with a semicolon
|
|
between the expression and the body statements. This causes <tt>put</tt> to
|
|
evaluate the boolean expression (along with any side effects, namely,
|
|
regex-captures <tt>\1</tt>, <tt>\2</tt>, etc.) but doesn’t use it as a
|
|
criterion for whether subsequent assignments should be executed. Instead,
|
|
subsequent assignments are done unconditionally:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr put '$x > 0.0; $y = log10($x); $z = sqrt($y)' data/put-gating-example-1.dkvp
|
|
x=-1,y=nan,z=nan
|
|
x=0,y=-inf,z=nan
|
|
x=1,y=0.000000,z=0.000000
|
|
x=2,y=0.301030,z=0.548662
|
|
x=3,y=0.477121,z=0.690740
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr put '$a =~ "([a-z]+)_([0-9]+)"; $b = "left_\1"; $c = "right_\2"' data/put-gating-example-2.dkvp
|
|
a=abc_123,b=left_abc,c=right_123
|
|
a=some other name,b=left_,c=right_
|
|
a=xyz_789,b=left_xyz,c=right_789
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="If-statements"/><h3>If-statements</h3>
|
|
|
|
<p/>These are again reminiscent of <tt>awk</tt>. Pattern-action blocks are a special case of <tt>if</tt> with no
|
|
<tt>elif</tt> or <tt>else</tt> blocks, no <tt>if</tt> keyword, and parentheses optional around the boolean expression:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
mlr put 'NR == 4 {$foo = "bar"}'
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
mlr put 'if (NR == 4) {$foo = "bar"}'
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>Compound statements use <tt>elif</tt> (rather than <tt>elsif</tt> or <tt>else if</tt>):
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
mlr put '
|
|
if (NR == 2) {
|
|
...
|
|
} elif (NR ==4) {
|
|
...
|
|
} elif (NR ==6) {
|
|
...
|
|
} else {
|
|
...
|
|
}
|
|
'
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="While_and_do-while_loops"/><h3>While and do-while loops</h3>
|
|
|
|
<p/>Miller’s <tt>while</tt> and <tt>do-while</tt> are unsurprising in
|
|
comparison to various languages, as are <tt>break</tt> and <tt>continue</tt>:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ echo x=1,y=2 | mlr put '
|
|
while (NF < 10) {
|
|
$[NF+1] = ""
|
|
}
|
|
$foo = "bar"
|
|
'
|
|
x=1,y=2,3=,4=,5=,6=,7=,8=,9=,10=,foo=bar
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ echo x=1,y=2 | mlr put '
|
|
do {
|
|
$[NF+1] = "";
|
|
if (NF == 5) {
|
|
break
|
|
}
|
|
} while (NF < 10);
|
|
$foo = "bar"
|
|
'
|
|
x=1,y=2,3=,4=,5=,foo=bar
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/> A <tt>break</tt> or <tt>continue</tt> within nested conditional blocks or
|
|
if-statements will, of course, propagate to the innermost loop enclosing them,
|
|
if any. A <tt>break</tt> or <tt>continue</tt> outside a loop is a syntax error
|
|
that will be flagged as soon as the expression is parsed, before any input
|
|
records are ingested.
|
|
|
|
<p/> The existence of <tt>while</tt>, <tt>do-while</tt>, and <tt>for</tt> loops
|
|
in Miller’s DSL means that you can create infinite-loop scenarios
|
|
inadvertently. In particular, please recall that DSL statements are executed
|
|
once if in <tt>begin</tt> or <tt>end</tt> blocks, and once <i>per record</i>
|
|
otherwise. For example, <b><tt>while (NR < 10)</tt> will never terminate as
|
|
<tt>NR</tt> is only incremented between records</b>.
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="For-loops"/><h3>For-loops</h3>
|
|
|
|
<p/>While Miller’s <tt>while</tt> and <tt>do-while</tt> statements are
|
|
much as in many other languages, <tt>for</tt> loops are more idiosyncratic to
|
|
Miller. They are loops over key-value pairs, whether in stream records or
|
|
out-of-stream variables: more reminiscent of <tt>foreach</tt>, as in (for
|
|
example) PHP.
|
|
|
|
<p/> There are three variants: <b>for-loop over key-value pairs in the current
|
|
stream record</b>, <b>for-loop over key-value pairs in an out-of-stream
|
|
variable</b>, and <b>C-style triple-for loops</b>. In each of the first two
|
|
cases the <tt>in</tt> keyword specifies the hashmap being iterated over, and
|
|
the variable names between <tt>for</tt> and <tt>in</tt> are bound to the keys
|
|
and values, respectively, of the hashmap’s key-value pairs on each loop
|
|
iteration. As with <tt>while</tt> and <tt>do-while</tt>, a <tt>break</tt> or
|
|
<tt>continue</tt> within nested control structures will propagate to the
|
|
innermost loop enclosing them, if any, and a <tt>break</tt> or
|
|
<tt>continue</tt> outside a loop is a syntax error that will be flagged as soon
|
|
as the expression is parsed, before any input records are ingested.
|
|
|
|
<p/><b>For-loop over the current stream record</b>:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ cat data/for-srec-example.tbl
|
|
label1 label2 f1 f2 f3
|
|
blue green 100 240 350
|
|
red green 120 11 195
|
|
yellow blue 140 0 240
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --pprint --from data/for-srec-example.tbl put '
|
|
$sum1 = $f1 + $f2 + $f3;
|
|
$sum2 = 0;
|
|
$sum3 = 0;
|
|
for (key, value in $*) {
|
|
if (key =~ "^f[0-9]+") {
|
|
$sum2 += value;
|
|
$sum3 += $[key];
|
|
}
|
|
}
|
|
'
|
|
label1 label2 f1 f2 f3 sum1 sum2 sum3
|
|
blue green 100 240 350 690 690 690
|
|
red green 120 11 195 326 326 326
|
|
yellow blue 140 0 240 380 380 380
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --from data/small --opprint put 'for (k,v in $*) { $[k."_type"] = typeof(v) }'
|
|
a b i x y a_type b_type i_type x_type y_type
|
|
pan pan 1 0.3467901443380824 0.7268028627434533 string string int float float
|
|
eks pan 2 0.7586799647899636 0.5221511083334797 string string int float float
|
|
wye wye 3 0.20460330576630303 0.33831852551664776 string string int float float
|
|
eks wye 4 0.38139939387114097 0.13418874328430463 string string int float float
|
|
wye pan 5 0.5732889198020006 0.8636244699032729 string string int float float
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>Note that the value of the current field in the for-loop can be gotten either using the bound
|
|
variable <tt>value</tt>, or through a <b>computed field name</b> using square brackets as in <tt>$[key]</tt>.
|
|
|
|
<p/>Important note: to avoid inconsistent looping behavior in case you’re
|
|
setting new fields (and/or unsetting existing ones) while looping over the
|
|
record, <b>Miller makes a copy of the record before the loop: loop variables
|
|
are bound from the copy and all other reads/writes involve the record
|
|
itself</b>:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --from data/small --opprint put '
|
|
$sum1 = 0;
|
|
$sum2 = 0;
|
|
for (k,v in $*) {
|
|
if (isnumeric(v)) {
|
|
$sum1 +=v;
|
|
$sum2 += $[k];
|
|
}
|
|
}
|
|
'
|
|
a b i x y sum1 sum2
|
|
pan pan 1 0.3467901443380824 0.7268028627434533 2.073593 8.294372
|
|
eks pan 2 0.7586799647899636 0.5221511083334797 3.280831 13.123324
|
|
wye wye 3 0.20460330576630303 0.33831852551664776 3.542922 14.171687
|
|
eks wye 4 0.38139939387114097 0.13418874328430463 4.515588 18.062353
|
|
wye pan 5 0.5732889198020006 0.8636244699032729 6.436913 25.747654
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
It can be confusing to modify the stream record while iterating over a copy of it, so
|
|
instead you might find it simpler to use an out-of-stream variable in the loop and only update
|
|
the stream record after the loop:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --from data/small --opprint put '
|
|
@sum = 0;
|
|
for (k,v in $*) {
|
|
if (isnumeric(v)) {
|
|
@sum += $[k];
|
|
}
|
|
}
|
|
$sum = @sum
|
|
'
|
|
a b i x y sum
|
|
pan pan 1 0.3467901443380824 0.7268028627434533 2.073593
|
|
eks pan 2 0.7586799647899636 0.5221511083334797 3.280831
|
|
wye wye 3 0.20460330576630303 0.33831852551664776 3.542922
|
|
eks wye 4 0.38139939387114097 0.13418874328430463 4.515588
|
|
wye pan 5 0.5732889198020006 0.8636244699032729 6.436913
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/><b>For-loop over out-of-stream variable</b>: This is similar to looping
|
|
over the current stream record except for additional degrees of freedom: you
|
|
can start iterating on sub-hashmaps of an out-of-stream variable; you can loop
|
|
over nested keys; you can loop over all out-of-stream variables. As with
|
|
for-loops over stream records, the bound variables are bound to a copy of the
|
|
sub-hashmap as it was before the loop started. The sub-hashmap is specified by
|
|
square-bracketed indices after <tt>in</tt>, and additional deeper indices are
|
|
bound to loop key-variables. The terminal values are bound to the loop
|
|
value-variable whenever the keys are neither too shallow, nor too deep. Example
|
|
indexing is as follows:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
# Parentheses are optional for single key:
|
|
for (k1, v in @a["b"]["c"]) { ... }
|
|
for ((k1), v in @a["b"]["c"]) { ... }
|
|
# Parentheses are required for multiple keys:
|
|
for ((k1, k2), v in @a["b"]["c"]) { ... } # Loop over subhashmap of a variable
|
|
for ((k1, k2, k3), v in @a["b"]["c"]) { ... } # Ditto
|
|
for ((k1, k2, k3), v in @a { ... } # Loop over variable starting from basename
|
|
for ((k1, k2, k3), v in @* { ... } # Loop over all variables (k1 is bound to basename)
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>That’s confusing in the abstract, so a concrete example is in order.
|
|
Suppose the out-of-stream variable <tt>@myvar</tt> is populated as follows:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint --from data/small head -n 2 then put -q '
|
|
begin {
|
|
@myvar["nesting-is-too-shallow"] = 1;
|
|
@myvar["nesting-is"]["just-right"] = 2;
|
|
@myvar["nesting-is"]["also-just-right"] = 3;
|
|
@myvar["nesting"]["is"]["too-deep"] = 4;
|
|
}
|
|
end {
|
|
dump
|
|
}
|
|
'
|
|
{
|
|
"myvar": {
|
|
"nesting-is-too-shallow": 1,
|
|
"nesting-is": {
|
|
"just-right": 2,
|
|
"also-just-right": 3
|
|
},
|
|
"nesting": {
|
|
"is": {
|
|
"too-deep": 4
|
|
}
|
|
}
|
|
}
|
|
}
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/> Then the too-shallow parts — indexed by the basename <tt>myvar</tt>
|
|
and the index <tt>"nesting-is-too-shallow"</tt> — have depth two
|
|
(basename and one index specify a terminal value) and can be gotten as follows:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --from data/small head -n 2 then put -q '
|
|
begin {
|
|
@myvar["nesting-is-too-shallow"] = 1;
|
|
@myvar["nesting-is"]["just-right"] = 2;
|
|
@myvar["nesting-is"]["also-just-right"] = 3;
|
|
@myvar["nesting"]["is"]["too-deep"] = 4;
|
|
}
|
|
end {
|
|
for (k, v in @myvar) {
|
|
@terminal[k] = v
|
|
}
|
|
emit @terminal, "index1"
|
|
}
|
|
'
|
|
index1=nesting-is-too-shallow,terminal=1
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --from data/small head -n 2 then put -q '
|
|
begin {
|
|
@myvar["nesting-is-too-shallow"] = 1;
|
|
@myvar["nesting-is"]["just-right"] = 2;
|
|
@myvar["nesting-is"]["also-just-right"] = 3;
|
|
@myvar["nesting"]["is"]["too-deep"] = 4;
|
|
}
|
|
end {
|
|
for ((k1, k2), v in @*) {
|
|
@terminal[k1][k2] = v
|
|
}
|
|
emit @terminal, "basename", "index1"
|
|
}
|
|
'
|
|
basename=myvar,index1=nesting-is-too-shallow,terminal=1
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>Note that it would take more than these two indices to reach the deeper values in the hashmap so they
|
|
aren’t bound in either of these for-loops.
|
|
|
|
<p/>By contrast, the <tt>"just-right"</tt> parts have depth three (basename and
|
|
two indices specify a terminal value) and can be gotten at by any of the
|
|
following:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --from data/small head -n 2 then put -q '
|
|
begin {
|
|
@myvar["nesting-is-too-shallow"] = 1;
|
|
@myvar["nesting-is"]["just-right"] = 2;
|
|
@myvar["nesting-is"]["also-just-right"] = 3;
|
|
@myvar["nesting"]["is"]["too-deep"] = 4;
|
|
}
|
|
end {
|
|
for ((k1), v in @myvar["nesting-is"]) {
|
|
@terminal[k1] = v
|
|
}
|
|
emit @terminal, "index1"
|
|
}
|
|
'
|
|
index1=just-right,terminal=2
|
|
index1=also-just-right,terminal=3
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --from data/small head -n 2 then put -q '
|
|
begin {
|
|
@myvar["nesting-is-too-shallow"] = 1;
|
|
@myvar["nesting-is"]["just-right"] = 2;
|
|
@myvar["nesting-is"]["also-just-right"] = 3;
|
|
@myvar["nesting"]["is"]["too-deep"] = 4;
|
|
}
|
|
end {
|
|
for ((k1, k2), v in @myvar) {
|
|
@terminal[k1][k2] = v
|
|
}
|
|
emit @terminal, "index1", "index2"
|
|
}
|
|
'
|
|
index1=nesting-is,index2=just-right,terminal=2
|
|
index1=nesting-is,index2=also-just-right,terminal=3
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --from data/small head -n 2 then put -q '
|
|
begin {
|
|
@myvar["nesting-is-too-shallow"] = 1;
|
|
@myvar["nesting-is"]["just-right"] = 2;
|
|
@myvar["nesting-is"]["also-just-right"] = 3;
|
|
@myvar["nesting"]["is"]["too-deep"] = 4;
|
|
}
|
|
end {
|
|
for ((k1, k2, k3), v in @*) {
|
|
@terminal[k1][k2][k3] = v
|
|
}
|
|
emit @terminal, "basename", "index1", "index2"
|
|
}
|
|
'
|
|
basename=myvar,index1=nesting-is,index2=just-right,terminal=2
|
|
basename=myvar,index1=nesting-is,index2=also-just-right,terminal=3
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/> Note that three key levels are specified here: basename and two indices.
|
|
So these for-loops don’t produce the depth-two or depth-four entries in
|
|
the hashmap.
|
|
|
|
<p/><b>C-style triple-for loops</b> are supported as follows:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --from data/small --opprint put '
|
|
num suma = 0;
|
|
num sumb = 0;
|
|
for (num a = 1, num b = 1; a <= NR; a += 1, b *= 2) {
|
|
suma += a;
|
|
sumb += b;
|
|
}
|
|
$suma = suma;
|
|
$sumb = sumb;
|
|
'
|
|
a b i x y suma sumb
|
|
pan pan 1 0.3467901443380824 0.7268028627434533 1 1
|
|
eks pan 2 0.7586799647899636 0.5221511083334797 3 3
|
|
wye wye 3 0.20460330576630303 0.33831852551664776 6 7
|
|
eks wye 4 0.38139939387114097 0.13418874328430463 10 15
|
|
wye pan 5 0.5732889198020006 0.8636244699032729 15 31
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
Notes:
|
|
<ul>
|
|
|
|
<li/> In <tt>for (start; continuation; update) { body }</tt>, the start,
|
|
continuation, and update statements may be empty, single statements, or
|
|
multiple comma-separated statements. If the continuation is empty it defaults
|
|
to true.
|
|
|
|
<li/> In particular, you may use <tt>$</tt>-variables and/or
|
|
<tt>@</tt>-variables in the start, continuation, and/or update steps (as well
|
|
as the body, of course).
|
|
|
|
<li/> As with all for/if/while statements in Miller, the curly braces are
|
|
required even if the body is a single statement, or empty.
|
|
|
|
</ul>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="Begin/end_blocks"/><h3>Begin/end blocks</h3>
|
|
|
|
<p/>Miller supports an <tt>awk</tt>-like <tt>begin/end</tt> syntax. The
|
|
statements in the <tt>begin</tt> block are executed before any input records
|
|
are read; the statements in the <tt>end</tt> block are executed after the last
|
|
input record is read. (If you want to execute some statement at the start of
|
|
each file, not at the start of the first file as with <tt>begin</tt>, you might
|
|
use a pattern/action block of the form <tt>FNR == 1 { ... }</tt>.) All
|
|
statements outside of <tt>begin</tt> or <tt>end</tt> are, of course, executed
|
|
on every input record. Semicolons separate statements inside or outside of
|
|
begin/end blocks; semicolons are required between begin/end block bodies and
|
|
any subsequent statement. For example:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr put '
|
|
begin { @sum = 0 };
|
|
@x_sum += $x;
|
|
end { emit @x_sum }
|
|
' ../data/small
|
|
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
|
|
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
|
|
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
|
|
a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463
|
|
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
|
|
a=zee,b=pan,i=6,x=0.5271261600918548,y=0.49322128674835697
|
|
a=eks,b=zee,i=7,x=0.6117840605678454,y=0.1878849191181694
|
|
a=zee,b=wye,i=8,x=0.5985540091064224,y=0.976181385699006
|
|
a=hat,b=wye,i=9,x=0.03144187646093577,y=0.7495507603507059
|
|
a=pan,b=wye,i=10,x=0.5026260055412137,y=0.9526183602969864
|
|
x_sum=4.536294
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>Since uninitialized out-of-stream variables default to 0 for
|
|
addition/substraction and 1 for multiplication when they appear on expression
|
|
right-hand sides (as in <tt>awk</tt>), the above can be written more succinctly
|
|
as
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr put '
|
|
@x_sum += $x;
|
|
end { emit @x_sum }
|
|
' ../data/small
|
|
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
|
|
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
|
|
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
|
|
a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463
|
|
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
|
|
a=zee,b=pan,i=6,x=0.5271261600918548,y=0.49322128674835697
|
|
a=eks,b=zee,i=7,x=0.6117840605678454,y=0.1878849191181694
|
|
a=zee,b=wye,i=8,x=0.5985540091064224,y=0.976181385699006
|
|
a=hat,b=wye,i=9,x=0.03144187646093577,y=0.7495507603507059
|
|
a=pan,b=wye,i=10,x=0.5026260055412137,y=0.9526183602969864
|
|
x_sum=4.536294
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>The <b>put -q</b> option is a shorthand which suppresses printing of each
|
|
output record, with only <tt>emit</tt> statements being output. So to get only
|
|
summary outputs, one could write
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr put -q '
|
|
@x_sum += $x;
|
|
end { emit @x_sum }
|
|
' ../data/small
|
|
x_sum=4.536294
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>We can do similarly with multiple out-of-stream variables:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr put -q '
|
|
@x_count += 1;
|
|
@x_sum += $x;
|
|
end {
|
|
emit @x_count;
|
|
emit @x_sum;
|
|
}
|
|
' ../data/small
|
|
x_count=10
|
|
x_sum=4.536294
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
This is of course not much different than
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr stats1 -a count,sum -f x ../data/small
|
|
x_count=10,x_sum=4.536294
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>Note that it’s a syntax error for begin/end blocks to refer to field
|
|
names (beginning with <tt>$</tt>), since these execute outside the context of
|
|
input records.
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="Output_statements"/><h2>Output statements</h2>
|
|
|
|
<p/>You can <b>output</b> variable-values or expressions in <b>five ways</b>:
|
|
|
|
<ul>
|
|
|
|
<li/> <b>Assign</b> them to stream-record fields. For example,
|
|
<tt>$cumulative_sum = @sum</tt>. For another example, <tt>$nr = NR</tt> adds a
|
|
field named <tt>nr</tt> to each output record, containing the value of the
|
|
built-in variable <tt>NR</tt> as of when that record was ingested.
|
|
|
|
<li/> Use <b>emit</b>/<b>emitp</b>/<b>emitf</b> to send out-of-stream
|
|
variables’ current values to the output record stream, e.g. <tt>@sum +=
|
|
$x; emit @sum</tt> which produces an extra output record such as
|
|
<tt>sum=3.1648382</tt>.
|
|
|
|
<li/> Use the <b>dump</b> or <b>edump</b> keywords, which immediately print all
|
|
out-of-stream variables as a JSON data structure to the standard output or
|
|
standard error (respectively).
|
|
|
|
<li/> Use the <b>print</b> or <b>eprint</b> keywords which immediately print an
|
|
expression to standard output or standard error, respectively. Note that
|
|
<tt>dump</tt>, <tt>edump</tt>, <tt>print</tt>, and <tt>eprint</tt> don’t
|
|
output records which participate in <tt>then</tt>-chaining; rather,
|
|
they’re just immediate prints to stdout/stderr. The <tt>printn</tt> and
|
|
<tt>eprintn</tt> keywords are the same except that they don’t print final
|
|
newlines. Additionally, you can print to a specified file instead of stdout/stderr.
|
|
|
|
<li/> Use <b>tee</b> which formats the current stream record (not just an
|
|
arbitrary string as with <b>print</b>) to a specific file.
|
|
|
|
</ul>
|
|
|
|
<p/>For the first two options you are populating the output-records stream
|
|
which feeds into the next verb in a <tt>then</tt>-chain (if any), or which otherwise
|
|
is formatted for output using <tt>--o...</tt> flags.
|
|
|
|
<p/>For the last three options you are sending output directly to standard
|
|
output, standard error, or a file.
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="Emit_statements"/><h3>Emit statements</h3>
|
|
|
|
<p/>There are three variants: <tt>emitf</tt>, <tt>emit</tt>, and
|
|
<tt>emitp</tt>. Keep in mind that out-of-stream variables are a nested,
|
|
multi-level hashmap (directly viewable as JSON using <tt>dump</tt>), whereas
|
|
Miller output records are lists of single-level key-value pairs. The three emit
|
|
variants allow you to control how the multilevel hashmaps are flatten down to
|
|
output records.
|
|
|
|
<p/>Use <b>emitf</b> to output several out-of-stream variables side-by-side in the same output record.
|
|
For <tt>emitf</tt> these mustn’t have indexing using <tt>@name[...]</tt>. Example:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr put -q '@count += 1; @x_sum += $x; @y_sum += $y; end { emitf @count, @x_sum, @y_sum}' data/small
|
|
count=5,x_sum=2.264762,y_sum=2.585086
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>Use <b>emit</b> to output an out-of-stream variable. If it’s non-indexed you’ll get a simple key-value pair:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ cat data/small
|
|
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
|
|
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
|
|
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
|
|
a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463
|
|
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr put -q '@sum += $x; end { dump }' data/small
|
|
{
|
|
"sum": 2.264762
|
|
}
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr put -q '@sum += $x; end { emit @sum }' data/small
|
|
sum=2.264762
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>If it’s indexed then use as many names after <tt>emit</tt> as there are indices:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr put -q '@sum[$a] += $x; end { dump }' data/small
|
|
{
|
|
"sum": {
|
|
"pan": 0.346790,
|
|
"eks": 1.140079,
|
|
"wye": 0.777892
|
|
}
|
|
}
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr put -q '@sum[$a] += $x; end { emit @sum, "a" }' data/small
|
|
a=pan,sum=0.346790
|
|
a=eks,sum=1.140079
|
|
a=wye,sum=0.777892
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr put -q '@sum[$a][$b] += $x; end { dump }' data/small
|
|
{
|
|
"sum": {
|
|
"pan": {
|
|
"pan": 0.346790
|
|
},
|
|
"eks": {
|
|
"pan": 0.758680,
|
|
"wye": 0.381399
|
|
},
|
|
"wye": {
|
|
"wye": 0.204603,
|
|
"pan": 0.573289
|
|
}
|
|
}
|
|
}
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr put -q '@sum[$a][$b] += $x; end { emit @sum, "a", "b" }' data/small
|
|
a=pan,b=pan,sum=0.346790
|
|
a=eks,b=pan,sum=0.758680
|
|
a=eks,b=wye,sum=0.381399
|
|
a=wye,b=wye,sum=0.204603
|
|
a=wye,b=pan,sum=0.573289
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr put -q '@sum[$a][$b][$i] += $x; end { dump }' data/small
|
|
{
|
|
"sum": {
|
|
"pan": {
|
|
"pan": {
|
|
"1": 0.346790
|
|
}
|
|
},
|
|
"eks": {
|
|
"pan": {
|
|
"2": 0.758680
|
|
},
|
|
"wye": {
|
|
"4": 0.381399
|
|
}
|
|
},
|
|
"wye": {
|
|
"wye": {
|
|
"3": 0.204603
|
|
},
|
|
"pan": {
|
|
"5": 0.573289
|
|
}
|
|
}
|
|
}
|
|
}
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr put -q '@sum[$a][$b][$i] += $x; end { emit @sum, "a", "b", "i" }' data/small
|
|
a=pan,b=pan,i=1,sum=0.346790
|
|
a=eks,b=pan,i=2,sum=0.758680
|
|
a=eks,b=wye,i=4,sum=0.381399
|
|
a=wye,b=wye,i=3,sum=0.204603
|
|
a=wye,b=pan,i=5,sum=0.573289
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>Now for <b>emitp</b>: if you have as many names following <tt>emit</tt> as
|
|
there are levels in the out-of-stream variable’s hashmap, then <tt>emit</tt> and <tt>emitp</tt> do the same
|
|
thing. Where they differ is when you don’t specify as many names as there are hashmap levels. In this
|
|
case, Miller needs to flatten multiple map indices down to output-record keys: <tt>emitp</tt> includes full
|
|
prefixing (hence the <tt>p</tt> in <tt>emitp</tt>) while <tt>emit</tt> takes the deepest hashmap key as the
|
|
output-record key:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr put -q '@sum[$a][$b] += $x; end { dump }' data/small
|
|
{
|
|
"sum": {
|
|
"pan": {
|
|
"pan": 0.346790
|
|
},
|
|
"eks": {
|
|
"pan": 0.758680,
|
|
"wye": 0.381399
|
|
},
|
|
"wye": {
|
|
"wye": 0.204603,
|
|
"pan": 0.573289
|
|
}
|
|
}
|
|
}
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr put -q '@sum[$a][$b] += $x; end { emit @sum, "a" }' data/small
|
|
a=pan,pan=0.346790
|
|
a=eks,pan=0.758680,wye=0.381399
|
|
a=wye,wye=0.204603,pan=0.573289
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr put -q '@sum[$a][$b] += $x; end { emit @sum }' data/small
|
|
pan=0.346790
|
|
pan=0.758680,wye=0.381399
|
|
wye=0.204603,pan=0.573289
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr put -q '@sum[$a][$b] += $x; end { emitp @sum, "a" }' data/small
|
|
a=pan,sum:pan=0.346790
|
|
a=eks,sum:pan=0.758680,sum:wye=0.381399
|
|
a=wye,sum:wye=0.204603,sum:pan=0.573289
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr put -q '@sum[$a][$b] += $x; end { emitp @sum }' data/small
|
|
sum:pan:pan=0.346790,sum:eks:pan=0.758680,sum:eks:wye=0.381399,sum:wye:wye=0.204603,sum:wye:pan=0.573289
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --oxtab put -q '@sum[$a][$b] += $x; end { emitp @sum }' data/small
|
|
sum:pan:pan 0.346790
|
|
sum:eks:pan 0.758680
|
|
sum:eks:wye 0.381399
|
|
sum:wye:wye 0.204603
|
|
sum:wye:pan 0.573289
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>Use <b>--oflatsep</b> to specify the character which joins multilevel
|
|
keys for <tt>emitp</tt> (it defaults to a colon):
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr put -q --oflatsep / '@sum[$a][$b] += $x; end { emitp @sum, "a" }' data/small
|
|
a=pan,sum/pan=0.346790
|
|
a=eks,sum/pan=0.758680,sum/wye=0.381399
|
|
a=wye,sum/wye=0.204603,sum/pan=0.573289
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr put -q --oflatsep / '@sum[$a][$b] += $x; end { emitp @sum }' data/small
|
|
sum/pan/pan=0.346790,sum/eks/pan=0.758680,sum/eks/wye=0.381399,sum/wye/wye=0.204603,sum/wye/pan=0.573289
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --oxtab put -q --oflatsep / '@sum[$a][$b] += $x; end { emitp @sum }' data/small
|
|
sum/pan/pan 0.346790
|
|
sum/eks/pan 0.758680
|
|
sum/eks/wye 0.381399
|
|
sum/wye/wye 0.204603
|
|
sum/wye/pan 0.573289
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="Multi-emit_statements"/><h3>Multi-emit statements</h3>
|
|
|
|
<p/>You can emit <b>multiple out-of-stream variables side-by-side</b> by including their names in parentheses:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --from data/medium --opprint put -q '
|
|
@x_count[$a][$b] += 1;
|
|
@x_sum[$a][$b] += $x;
|
|
end {
|
|
for ((a, b), _ in @x_count) {
|
|
@x_mean[a][b] = @x_sum[a][b] / @x_count[a][b]
|
|
}
|
|
emit (@x_sum, @x_count, @x_mean), "a", "b"
|
|
}
|
|
'
|
|
a b x_sum x_count x_mean
|
|
pan pan 219.185129 427 0.513314
|
|
pan wye 198.432931 395 0.502362
|
|
pan eks 216.075228 429 0.503672
|
|
pan hat 205.222776 417 0.492141
|
|
pan zee 205.097518 413 0.496604
|
|
eks pan 179.963030 371 0.485076
|
|
eks wye 196.945286 407 0.483895
|
|
eks zee 176.880365 357 0.495463
|
|
eks eks 215.916097 413 0.522799
|
|
eks hat 208.783171 417 0.500679
|
|
wye wye 185.295850 377 0.491501
|
|
wye pan 195.847900 392 0.499612
|
|
wye hat 212.033183 426 0.497730
|
|
wye zee 194.774048 385 0.505907
|
|
wye eks 204.812961 386 0.530604
|
|
zee pan 202.213804 389 0.519830
|
|
zee wye 233.991394 455 0.514267
|
|
zee eks 190.961778 391 0.488393
|
|
zee zee 206.640635 403 0.512756
|
|
zee hat 191.300006 409 0.467726
|
|
hat wye 208.883010 423 0.493813
|
|
hat zee 196.349450 385 0.509999
|
|
hat eks 189.006793 389 0.485879
|
|
hat hat 182.853532 381 0.479931
|
|
hat pan 168.553807 363 0.464336
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
What this does is walk through the first out-of-stream variable
|
|
(<tt>@x_sum</tt> in this example) as usual, then for each keylist found (e.g.
|
|
<tt>pan,wye</tt>), include the values for the remaining out-of-stream variables
|
|
(here, <tt>@x_count</tt> and <tt>@x_mean</tt>). You should use this when all
|
|
out-of-stream variables in the emit statement have the same shape and the same
|
|
keylists.
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="Emit-all_statements"/><h3>Emit-all statements</h3>
|
|
|
|
<p/>Use <b>emit all</b> (or <tt>emit @*</tt> which is synonumous) to output all
|
|
out-of-stream variables. You can use the following idiom to get various
|
|
accumulators output side-by-side (reminiscent of <tt>mlr stats1</tt>):
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --from data/small --opprint put -q '@v[$a][$b]["sum"] += $x; @v[$a][$b]["count"] += 1; end{emit @*,"a","b"}'
|
|
a b sum count
|
|
pan pan 0.346790 1
|
|
eks pan 0.758680 1
|
|
eks wye 0.381399 1
|
|
wye wye 0.204603 1
|
|
wye pan 0.573289 1
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --from data/small --opprint put -q '@sum[$a][$b] += $x; @count[$a][$b] += 1; end{emit @*,"a","b"}'
|
|
a b sum
|
|
pan pan 0.346790
|
|
eks pan 0.758680
|
|
eks wye 0.381399
|
|
wye wye 0.204603
|
|
wye pan 0.573289
|
|
|
|
a b count
|
|
pan pan 1
|
|
eks pan 1
|
|
eks wye 1
|
|
wye wye 1
|
|
wye pan 1
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --from data/small --opprint put -q '@sum[$a][$b] += $x; @count[$a][$b] += 1; end{emit (@sum, @count),"a","b"}'
|
|
a b sum count
|
|
pan pan 0.346790 1
|
|
eks pan 0.758680 1
|
|
eks wye 0.381399 1
|
|
wye wye 0.204603 1
|
|
wye pan 0.573289 1
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="Redirected-output_statements"/><h3>Redirected-output statements</h3>
|
|
|
|
The <b>tee</b>, <b>emitf</b>, <b>emitp</b>, <b>emit</b>, <b>print</b>, and
|
|
<b>dump</b> keywords all allow you to redirect output to one or more files or
|
|
pipe-to commands. The filenames/commands are strings which can be constructed
|
|
using record-dependent values, so you can do things like splitting a table into
|
|
multiple files, one for each account ID, and so on.
|
|
|
|
<p/> Details:
|
|
|
|
<ul>
|
|
|
|
<li/> <tt>mlr put</tt> sends the current record (possibly modified by the
|
|
<tt>put</tt> expression) to the output record stream. Records are then input to
|
|
the following verb in a <tt>then</tt>-chain (if any), else printed to standard
|
|
output (unless <tt>put -q</tt>). The <b>tee</b> keyword <i>additionally</i>
|
|
writes the output record to specified file(s) or pipe-to command, or
|
|
immediately to <tt>stdout</tt>/<tt>stderr</tt>.
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --help-keyword tee
|
|
tee: prints the current record to specified file.
|
|
This is an immediate print to the specified file (except for pprint format
|
|
which of course waits until the end of the input stream to format all output).
|
|
|
|
The > and >> are for write and append, as in the shell, but (as with awk) the
|
|
file-overwrite for > is on first write, not per record. The | is for piping to
|
|
a process which will process the data. There will be one open file for each
|
|
distinct file name (for > and >>) or one subordinate process for each distinct
|
|
value of the piped-to command (for |). Output-formatting flags are taken from
|
|
the main command line.
|
|
|
|
You can use any of the output-format command-line flags, e.g. --ocsv, --ofs,
|
|
etc., to control the format of the output. See also mlr -h.
|
|
|
|
Example: mlr --from f.dat put 'tee > "/tmp/data-".$a, $*'
|
|
Example: mlr --from f.dat put 'tee >> "/tmp/data-".$a.$b, $*'
|
|
Example: mlr --from f.dat put 'tee > stderr, $*'
|
|
Example: mlr --from f.dat put -q 'tee | "tr [a-z\] [A-Z\]", $*'
|
|
Example: mlr --from f.dat put -q 'tee | "tr [a-z\] [A-Z\] > /tmp/data-".$a, $*'
|
|
Example: mlr --from f.dat put -q 'tee | "gzip > /tmp/data-".$a.".gz", $*'
|
|
Example: mlr --from f.dat put -q --ojson 'tee | "gzip > /tmp/data-".$a.".gz", $*'
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<li/> <tt>mlr put</tt>’s <tt>emitf</tt>, <tt>emitp</tt>, and
|
|
<tt>emit</tt> send out-of-stream variables to the output record stream. These
|
|
are then input to the following verb in a <tt>then</tt>-chain (if any), else
|
|
printed to standard output. When redirected with <tt>></tt>,
|
|
<tt>>></tt>, or <tt>|</tt>, they <i>instead</i> write the out-of-stream
|
|
variable(s) to specified file(s) or pipe-to command, or immediately to
|
|
<tt>stdout</tt>/<tt>stderr</tt>.
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --help-keyword emitf
|
|
emitf: inserts non-indexed out-of-stream variable(s) side-by-side into the
|
|
output record stream.
|
|
|
|
With >, >>, or |, the data do not become part of the output record stream but
|
|
are instead redirected.
|
|
|
|
The > and >> are for write and append, as in the shell, but (as with awk) the
|
|
file-overwrite for > is on first write, not per record. The | is for piping to
|
|
a process which will process the data. There will be one open file for each
|
|
distinct file name (for > and >>) or one subordinate process for each distinct
|
|
value of the piped-to command (for |). Output-formatting flags are taken from
|
|
the main command line.
|
|
|
|
You can use any of the output-format command-line flags, e.g. --ocsv, --ofs,
|
|
etc., to control the format of the output if the output is redirected. See also mlr -h.
|
|
|
|
Example: mlr --from f.dat put '@a=$i;@b+=$x;@c+=$y; emitf @a'
|
|
Example: mlr --from f.dat put --oxtab '@a=$i;@b+=$x;@c+=$y; emitf > "tap-".$i.".dat", @a'
|
|
Example: mlr --from f.dat put '@a=$i;@b+=$x;@c+=$y; emitf @a, @b, @c'
|
|
Example: mlr --from f.dat put '@a=$i;@b+=$x;@c+=$y; emitf > "mytap.dat", @a, @b, @c'
|
|
Example: mlr --from f.dat put '@a=$i;@b+=$x;@c+=$y; emitf >> "mytap.dat", @a, @b, @c'
|
|
Example: mlr --from f.dat put '@a=$i;@b+=$x;@c+=$y; emitf > stderr, @a, @b, @c'
|
|
Example: mlr --from f.dat put '@a=$i;@b+=$x;@c+=$y; emitf | "grep somepattern", @a, @b, @c'
|
|
Example: mlr --from f.dat put '@a=$i;@b+=$x;@c+=$y; emitf | "grep somepattern > mytap.dat", @a, @b, @c'
|
|
|
|
Please see http://johnkerl.org/miller/doc for more information.
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --help-keyword emitp
|
|
emitp: inserts an out-of-stream variable into the output record stream.
|
|
Hashmap indices present in the data but not slotted by emitp arguments are
|
|
output concatenated with ":".
|
|
|
|
With >, >>, or |, the data do not become part of the output record stream but
|
|
are instead redirected.
|
|
|
|
The > and >> are for write and append, as in the shell, but (as with awk) the
|
|
file-overwrite for > is on first write, not per record. The | is for piping to
|
|
a process which will process the data. There will be one open file for each
|
|
distinct file name (for > and >>) or one subordinate process for each distinct
|
|
value of the piped-to command (for |). Output-formatting flags are taken from
|
|
the main command line.
|
|
|
|
You can use any of the output-format command-line flags, e.g. --ocsv, --ofs,
|
|
etc., to control the format of the output if the output is redirected. See also mlr -h.
|
|
|
|
Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emitp @sums'
|
|
Example: mlr --from f.dat put --opprint '@sums[$a][$b]+=$x; emitp > "tap-".$a.$b.".dat", @sums'
|
|
Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emitp @sums, "index1", "index2"'
|
|
Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emitp @*, "index1", "index2"'
|
|
Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emitp > "mytap.dat", @*, "index1", "index2"'
|
|
Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emitp >> "mytap.dat", @*, "index1", "index2"'
|
|
Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emitp | "gzip > mytap.dat.gz", @*, "index1", "index2"'
|
|
Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emitp > stderr, @*, "index1", "index2"'
|
|
Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emitp | "grep somepattern", @*, "index1", "index2"'
|
|
|
|
Please see http://johnkerl.org/miller/doc for more information.
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --help-keyword emit
|
|
emit: inserts an out-of-stream variable into the output record stream. Hashmap
|
|
indices present in the data but not slotted by emit arguments are not output.
|
|
|
|
With >, >>, or |, the data do not become part of the output record stream but
|
|
are instead redirected.
|
|
|
|
The > and >> are for write and append, as in the shell, but (as with awk) the
|
|
file-overwrite for > is on first write, not per record. The | is for piping to
|
|
a process which will process the data. There will be one open file for each
|
|
distinct file name (for > and >>) or one subordinate process for each distinct
|
|
value of the piped-to command (for |). Output-formatting flags are taken from
|
|
the main command line.
|
|
|
|
You can use any of the output-format command-line flags, e.g. --ocsv, --ofs,
|
|
etc., to control the format of the output if the output is redirected. See also mlr -h.
|
|
|
|
Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emit @sums'
|
|
Example: mlr --from f.dat put --ojson '@sums[$a][$b]+=$x; emit > "tap-".$a.$b.".dat", @sums'
|
|
Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emit @sums, "index1", "index2"'
|
|
Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emit @*, "index1", "index2"'
|
|
Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emit > "mytap.dat", @*, "index1", "index2"'
|
|
Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emit >> "mytap.dat", @*, "index1", "index2"'
|
|
Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emit | "gzip > mytap.dat.gz", @*, "index1", "index2"'
|
|
Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emit > stderr, @*, "index1", "index2"'
|
|
Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emit | "grep somepattern", @*, "index1", "index2"'
|
|
|
|
Please see http://johnkerl.org/miller/doc for more information.
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<li/> The <tt>print</tt> and <tt>dump</tt> keywords produce output immediately
|
|
to standard output, or to specified file(s) or pipe-to command if present.
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --help-keyword print
|
|
print: prints expression immediately to stdout.
|
|
Example: mlr --from f.dat put -q 'print "The sum of x and y is ".($x+$y)'
|
|
Example: mlr --from f.dat put -q 'for (k, v in $*) { print k . " => " . v }'
|
|
Example: mlr --from f.dat put '(NR % 1000 == 0) { print > stderr, "Checkpoint ".NR}'
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --help-keyword dump
|
|
dump: prints all currently defined out-of-stream variables immediately
|
|
to stdout as JSON.
|
|
|
|
With >, >>, or |, the data do not become part of the output record stream but
|
|
are instead redirected.
|
|
|
|
The > and >> are for write and append, as in the shell, but (as with awk) the
|
|
file-overwrite for > is on first write, not per record. The | is for piping to
|
|
a process which will process the data. There will be one open file for each
|
|
distinct file name (for > and >>) or one subordinate process for each distinct
|
|
value of the piped-to command (for |). Output-formatting flags are taken from
|
|
the main command line.
|
|
|
|
Example: mlr --from f.dat put -q '@v[NR]=$*; end { dump }'
|
|
Example: mlr --from f.dat put -q '@v[NR]=$*; end { dump > "mytap.dat"}'
|
|
Example: mlr --from f.dat put -q '@v[NR]=$*; end { dump >> "mytap.dat"}'
|
|
Example: mlr --from f.dat put -q '@v[NR]=$*; end { dump | "jq .[]"}'
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
</ul>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="Unset_statements"/><h2>Unset statements</h2>
|
|
|
|
<p/>You can clear a map key by assigning the empty string as its value: <tt>$x=""</tt> or <tt>@x=""</tt>.
|
|
Using <tt>unset</tt> you can remove the key entirely. Examples:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ cat data/small
|
|
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
|
|
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
|
|
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
|
|
a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463
|
|
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr put 'unset $x, $a' data/small
|
|
b=pan,i=1,y=0.7268028627434533
|
|
b=pan,i=2,y=0.5221511083334797
|
|
b=wye,i=3,y=0.33831852551664776
|
|
b=wye,i=4,y=0.13418874328430463
|
|
b=pan,i=5,y=0.8636244699032729
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>This can also be done, of course, using <tt>mlr cut -x</tt>. You can also clear out-of-stream variables, at the base name level, or at an indexed sublevel:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr put -q '@sum[$a][$b] += $x; end { dump; unset @sum; dump }' data/small
|
|
{
|
|
"sum": {
|
|
"pan": {
|
|
"pan": 0.346790
|
|
},
|
|
"eks": {
|
|
"pan": 0.758680,
|
|
"wye": 0.381399
|
|
},
|
|
"wye": {
|
|
"wye": 0.204603,
|
|
"pan": 0.573289
|
|
}
|
|
}
|
|
}
|
|
{
|
|
}
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr put -q '@sum[$a][$b] += $x; end { dump; unset @sum["eks"]; dump }' data/small
|
|
{
|
|
"sum": {
|
|
"pan": {
|
|
"pan": 0.346790
|
|
},
|
|
"eks": {
|
|
"pan": 0.758680,
|
|
"wye": 0.381399
|
|
},
|
|
"wye": {
|
|
"wye": 0.204603,
|
|
"pan": 0.573289
|
|
}
|
|
}
|
|
}
|
|
{
|
|
"sum": {
|
|
"pan": {
|
|
"pan": 0.346790
|
|
},
|
|
"wye": {
|
|
"wye": 0.204603,
|
|
"pan": 0.573289
|
|
}
|
|
}
|
|
}
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>If you use <tt>unset all</tt> (or <tt>unset @*</tt> which is synonymous), that will unset all out-of-stream
|
|
variables which have been defined up to that point.
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="Filter_statements"/><h2>Filter statements</h2>
|
|
|
|
<p/> You can use <tt>filter</tt> within <tt>put</tt>. In fact, the
|
|
following two are synonymous:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr filter 'NR==2 || NR==3' data/small
|
|
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
|
|
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr put 'filter NR==2 || NR==3' data/small
|
|
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
|
|
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>The former, of course, is much easier to type. But the latter allows you to define more complex expressions
|
|
for the filter, and/or do other things in addition to the filter:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr put '@running_sum += $x; filter @running_sum > 1.3' data/small
|
|
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
|
|
a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463
|
|
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr put '$z = $x * $y; filter $z > 0.3' data/small
|
|
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797,z=0.396146
|
|
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729,z=0.495106
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="Built-in_functions_for_filter_and_put"/><h2>Built-in functions for filter and put</h2>
|
|
|
|
<p/>Each function takes a specific number of arguments, as shown below, except
|
|
for functions marked as variadic such as <tt>min</tt> and <tt>max</tt>. (The
|
|
latter compute min and max of any number of numerical arguments.) There is no
|
|
notion of optional or default-on-absent arguments. All argument-passing is
|
|
positional rather than by name; arguments are passed by value, not by
|
|
reference.
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --help-all-functions
|
|
+ (class=arithmetic #args=2): Addition.
|
|
|
|
+ (class=arithmetic #args=1): Unary plus.
|
|
|
|
- (class=arithmetic #args=2): Subtraction.
|
|
|
|
- (class=arithmetic #args=1): Unary minus.
|
|
|
|
* (class=arithmetic #args=2): Multiplication.
|
|
|
|
/ (class=arithmetic #args=2): Division.
|
|
|
|
// (class=arithmetic #args=2): Integer division: rounds to negative (pythonic).
|
|
|
|
% (class=arithmetic #args=2): Remainder; never negative-valued (pythonic).
|
|
|
|
** (class=arithmetic #args=2): Exponentiation; same as pow, but as an infix
|
|
operator.
|
|
|
|
| (class=arithmetic #args=2): Bitwise OR.
|
|
|
|
^ (class=arithmetic #args=2): Bitwise XOR.
|
|
|
|
& (class=arithmetic #args=2): Bitwise AND.
|
|
|
|
~ (class=arithmetic #args=1): Bitwise NOT. Beware '$y=~$x' since =~ is the
|
|
regex-match operator: try '$y = ~$x'.
|
|
|
|
<< (class=arithmetic #args=2): Bitwise left-shift.
|
|
|
|
>> (class=arithmetic #args=2): Bitwise right-shift.
|
|
|
|
== (class=boolean #args=2): String/numeric equality. Mixing number and string
|
|
results in string compare.
|
|
|
|
!= (class=boolean #args=2): String/numeric inequality. Mixing number and string
|
|
results in string compare.
|
|
|
|
=~ (class=boolean #args=2): String (left-hand side) matches regex (right-hand
|
|
side), e.g. '$name =~ "^a.*b$"'.
|
|
|
|
!=~ (class=boolean #args=2): String (left-hand side) does not match regex
|
|
(right-hand side), e.g. '$name !=~ "^a.*b$"'.
|
|
|
|
> (class=boolean #args=2): String/numeric greater-than. Mixing number and string
|
|
results in string compare.
|
|
|
|
>= (class=boolean #args=2): String/numeric greater-than-or-equals. Mixing number
|
|
and string results in string compare.
|
|
|
|
< (class=boolean #args=2): String/numeric less-than. Mixing number and string
|
|
results in string compare.
|
|
|
|
<= (class=boolean #args=2): String/numeric less-than-or-equals. Mixing number
|
|
and string results in string compare.
|
|
|
|
&& (class=boolean #args=2): Logical AND.
|
|
|
|
|| (class=boolean #args=2): Logical OR.
|
|
|
|
^^ (class=boolean #args=2): Logical XOR.
|
|
|
|
! (class=boolean #args=1): Logical negation.
|
|
|
|
? : (class=boolean #args=3): Ternary operator.
|
|
|
|
. (class=string #args=2): String concatenation.
|
|
|
|
gsub (class=string #args=3): Example: '$name=gsub($name, "old", "new")'
|
|
(replace all).
|
|
|
|
strlen (class=string #args=1): String length.
|
|
|
|
sub (class=string #args=3): Example: '$name=sub($name, "old", "new")'
|
|
(replace once).
|
|
|
|
substr (class=string #args=3): substr(s,m,n) gives substring of s from 0-up position m to n
|
|
inclusive. Negative indices -len .. -1 alias to 0 .. len-1.
|
|
|
|
tolower (class=string #args=1): Convert string to lowercase.
|
|
|
|
toupper (class=string #args=1): Convert string to uppercase.
|
|
|
|
abs (class=math #args=1): Absolute value.
|
|
|
|
acos (class=math #args=1): Inverse trigonometric cosine.
|
|
|
|
acosh (class=math #args=1): Inverse hyperbolic cosine.
|
|
|
|
asin (class=math #args=1): Inverse trigonometric sine.
|
|
|
|
asinh (class=math #args=1): Inverse hyperbolic sine.
|
|
|
|
atan (class=math #args=1): One-argument arctangent.
|
|
|
|
atan2 (class=math #args=2): Two-argument arctangent.
|
|
|
|
atanh (class=math #args=1): Inverse hyperbolic tangent.
|
|
|
|
cbrt (class=math #args=1): Cube root.
|
|
|
|
ceil (class=math #args=1): Ceiling: nearest integer at or above.
|
|
|
|
cos (class=math #args=1): Trigonometric cosine.
|
|
|
|
cosh (class=math #args=1): Hyperbolic cosine.
|
|
|
|
erf (class=math #args=1): Error function.
|
|
|
|
erfc (class=math #args=1): Complementary error function.
|
|
|
|
exp (class=math #args=1): Exponential function e**x.
|
|
|
|
expm1 (class=math #args=1): e**x - 1.
|
|
|
|
floor (class=math #args=1): Floor: nearest integer at or below.
|
|
|
|
invqnorm (class=math #args=1): Inverse of normal cumulative distribution
|
|
function. Note that invqorm(urand()) is normally distributed.
|
|
|
|
log (class=math #args=1): Natural (base-e) logarithm.
|
|
|
|
log10 (class=math #args=1): Base-10 logarithm.
|
|
|
|
log1p (class=math #args=1): log(1-x).
|
|
|
|
logifit (class=math #args=3): Given m and b from logistic regression, compute
|
|
fit: $yhat=logifit($x,$m,$b).
|
|
|
|
madd (class=math #args=3): a + b mod m (integers)
|
|
|
|
max (class=math variadic): max of n numbers; null loses
|
|
|
|
mexp (class=math #args=3): a ** b mod m (integers)
|
|
|
|
min (class=math variadic): Min of n numbers; null loses
|
|
|
|
mmul (class=math #args=3): a * b mod m (integers)
|
|
|
|
msub (class=math #args=3): a - b mod m (integers)
|
|
|
|
pow (class=math #args=2): Exponentiation; same as **.
|
|
|
|
qnorm (class=math #args=1): Normal cumulative distribution function.
|
|
|
|
round (class=math #args=1): Round to nearest integer.
|
|
|
|
roundm (class=math #args=2): Round to nearest multiple of m: roundm($x,$m) is
|
|
the same as round($x/$m)*$m
|
|
|
|
sgn (class=math #args=1): +1 for positive input, 0 for zero input, -1 for
|
|
negative input.
|
|
|
|
sin (class=math #args=1): Trigonometric sine.
|
|
|
|
sinh (class=math #args=1): Hyperbolic sine.
|
|
|
|
sqrt (class=math #args=1): Square root.
|
|
|
|
tan (class=math #args=1): Trigonometric tangent.
|
|
|
|
tanh (class=math #args=1): Hyperbolic tangent.
|
|
|
|
urand (class=math #args=0): Floating-point numbers on the unit interval.
|
|
Int-valued example: '$n=floor(20+urand()*11)'.
|
|
|
|
urand32 (class=math #args=0): Integer uniformly distributed 0 and 2**32-1
|
|
inclusive.
|
|
|
|
urandint (class=math #args=2): Integer uniformly distributed between inclusive
|
|
integer endpoints.
|
|
|
|
dhms2fsec (class=time #args=1): Recovers floating-point seconds as in
|
|
dhms2fsec("5d18h53m20.250000s") = 500000.250000
|
|
|
|
dhms2sec (class=time #args=1): Recovers integer seconds as in
|
|
dhms2sec("5d18h53m20s") = 500000
|
|
|
|
fsec2dhms (class=time #args=1): Formats floating-point seconds as in
|
|
fsec2dhms(500000.25) = "5d18h53m20.250000s"
|
|
|
|
fsec2hms (class=time #args=1): Formats floating-point seconds as in
|
|
fsec2hms(5000.25) = "01:23:20.250000"
|
|
|
|
gmt2sec (class=time #args=1): Parses GMT timestamp as integer seconds since
|
|
the epoch.
|
|
|
|
hms2fsec (class=time #args=1): Recovers floating-point seconds as in
|
|
hms2fsec("01:23:20.250000") = 5000.250000
|
|
|
|
hms2sec (class=time #args=1): Recovers integer seconds as in
|
|
hms2sec("01:23:20") = 5000
|
|
|
|
sec2dhms (class=time #args=1): Formats integer seconds as in sec2dhms(500000)
|
|
= "5d18h53m20s"
|
|
|
|
sec2gmt (class=time #args=1): Formats seconds since epoch (integer part)
|
|
as GMT timestamp, e.g. sec2gmt(1440768801.7) = "2015-08-28T13:33:21Z".
|
|
Leaves non-numbers as-is.
|
|
|
|
sec2gmtdate (class=time #args=1): Formats seconds since epoch (integer part)
|
|
as GMT timestamp with year-month-date, e.g. sec2gmtdate(1440768801.7) = "2015-08-28".
|
|
Leaves non-numbers as-is.
|
|
|
|
sec2hms (class=time #args=1): Formats integer seconds as in
|
|
sec2hms(5000) = "01:23:20"
|
|
|
|
strftime (class=time #args=2): Formats seconds since epoch (integer part)
|
|
as timestamp, e.g.
|
|
strftime(1440768801.7,"%Y-%m-%dT%H:%M:%SZ") = "2015-08-28T13:33:21Z".
|
|
|
|
strptime (class=time #args=2): Parses timestamp as integer seconds since epoch,
|
|
e.g. strptime("2015-08-28T13:33:21Z","%Y-%m-%dT%H:%M:%SZ") = 1440768801.
|
|
|
|
systime (class=time #args=0): Floating-point seconds since the epoch,
|
|
e.g. 1440768801.748936.
|
|
|
|
isabsent (class=typing #args=1): False if field is present in input, false otherwise
|
|
|
|
isbool (class=typing #args=1): True if field is present with boolean value. Synonymous with isboolean.
|
|
|
|
isboolean (class=typing #args=1): True if field is present with boolean value. Synonymous with isbool.
|
|
|
|
isempty (class=typing #args=1): True if field is present in input with empty string value, false otherwise.
|
|
|
|
isemptymap (class=typing #args=1): True if argument is a map which is empty.
|
|
|
|
isfloat (class=typing #args=1): True if field is present with value inferred to be float
|
|
|
|
isint (class=typing #args=1): True if field is present with value inferred to be int
|
|
|
|
ismap (class=typing #args=1): True if argument is a map.
|
|
|
|
isnonemptymap (class=typing #args=1): True if argument is a map which is non-empty.
|
|
|
|
isnotempty (class=typing #args=1): False if field is present in input with empty value, false otherwise
|
|
|
|
isnotnull (class=typing #args=1): False if argument is null (empty or absent), true otherwise.
|
|
|
|
isnull (class=typing #args=1): True if argument is null (empty or absent), false otherwise.
|
|
|
|
isnumeric (class=typing #args=1): True if field is present with value inferred to be int or float
|
|
|
|
ispresent (class=typing #args=1): True if field is present in input, false otherwise.
|
|
|
|
isscalar (class=typing #args=1): True if argument is not a map.
|
|
|
|
isstring (class=typing #args=1): True if field is present with string (including empty-string) value
|
|
|
|
assert_notnull (class=typing #args=1): Returns argument if non-null (non-empty and non-absent), else throws an error.
|
|
|
|
assert_present (class=typing #args=1): Returns argument if present in input, else throws an error.
|
|
|
|
assert_empty (class=typing #args=1): Returns argument if present in input with empty value, else throws an error.
|
|
|
|
assert_notempty (class=typing #args=1): Returns argument if present in input with non-empty value, else throws an error.
|
|
|
|
assert_numeric (class=typing #args=1): Returns argument if present with int or float value, else throws an error.
|
|
|
|
assert_int (class=typing #args=1): Returns argument if present with int value, else throws an error.
|
|
|
|
assert_float (class=typing #args=1): Returns argument if present with float value, else throws an error.
|
|
|
|
assert_bool (class=typing #args=1): Returns argument if present with boolean value, else throws an error.
|
|
|
|
assert_boolean (class=typing #args=1): Returns argument if present with boolean value, else throws an error.
|
|
|
|
assert_string (class=typing #args=1): Returns argument if present with string (including empty-string) value, else throws an error.
|
|
|
|
boolean (class=conversion #args=1): Convert int/float/bool/string to boolean.
|
|
|
|
float (class=conversion #args=1): Convert int/float/bool/string to float.
|
|
|
|
fmtnum (class=conversion #args=2): Convert int/float/bool to string using
|
|
printf-style format string, e.g. '$s = fmtnum($n, "%06lld")'.
|
|
|
|
hexfmt (class=conversion #args=1): Convert int to string, e.g. 255 to "0xff".
|
|
|
|
int (class=conversion #args=1): Convert int/float/bool/string to int.
|
|
|
|
string (class=conversion #args=1): Convert int/float/bool/string to string.
|
|
|
|
typeof (class=conversion #args=1): Convert argument to type of argument (e.g.
|
|
MT_STRING). For debug.
|
|
|
|
depth (class=maps #args=1): xxx temp.
|
|
|
|
haskey (class=maps #args=2): xxx temp.
|
|
|
|
joink (class=maps #args=2): xxx temp.
|
|
|
|
joinkv (class=maps #args=3): xxx temp.
|
|
|
|
joinv (class=maps #args=2): xxx temp.
|
|
|
|
leafcount (class=maps #args=1): xxx temp.
|
|
|
|
length (class=maps #args=1): xxx temp.
|
|
|
|
mapdiff (class=maps variadic): xxx temp.
|
|
|
|
mapsum (class=maps variadic): xxx temp.
|
|
|
|
splitkv (class=maps #args=3): xxx temp.
|
|
|
|
splitnv (class=maps #args=2): xxx temp.
|
|
|
|
To set the seed for urand, you may specify decimal or hexadecimal 32-bit
|
|
numbers of the form "mlr --seed 123456789" or "mlr --seed 0xcafefeed".
|
|
Miller's built-in variables are NF, NR, FNR, FILENUM, and FILENAME (awk-like)
|
|
along with the mathematical constants PI and E.
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="User-defined_functions_and_subroutines"/><h2>User-defined functions and subroutines</h2>
|
|
|
|
<p/> As of Miller 5.0.0 you can define your own functions, as well as subroutines.
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="User-defined_functions"/><h3>User-defined functions</h3>
|
|
|
|
<p/>Here’s the obligatory example of a recursive function to compute the factorial function:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint --from data/small put '
|
|
func f(n) {
|
|
if (isnumeric(n)) {
|
|
if (n > 0) {
|
|
return n * f(n-1);
|
|
} else {
|
|
return 1;
|
|
}
|
|
}
|
|
# implicitly return absent-null if non-numeric
|
|
}
|
|
$ox = f($x + NR);
|
|
$oi = f($i);
|
|
'
|
|
a b i x y ox oi
|
|
pan pan 1 0.3467901443380824 0.7268028627434533 0.467054 1
|
|
eks pan 2 0.7586799647899636 0.5221511083334797 3.680838 2
|
|
wye wye 3 0.20460330576630303 0.33831852551664776 1.741251 6
|
|
eks wye 4 0.38139939387114097 0.13418874328430463 18.588349 24
|
|
wye pan 5 0.5732889198020006 0.8636244699032729 211.387310 120
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>Properties of user-defined functions:
|
|
|
|
<ul>
|
|
|
|
<li/> Function bodies start with <tt>func</tt> and a parameter list, defined
|
|
outside of <tt>begin</tt>, <tt>end</tt>, or other <tt>func</tt> or
|
|
<tt>subr</tt> blocks. (I.e. the Miller DSL has no nested functions.)
|
|
|
|
<li/> A function (uniqified by its name) may not be redefined: either by
|
|
redefining a user-defined function, or by redefining a built-in function.
|
|
However, functions and subroutines have separate namespaces: you can define a
|
|
subroutine <tt>log</tt> which does not clash with the mathematical <tt>log</tt>
|
|
function.
|
|
|
|
<li/> Functions may be defined either before or after use (there is an
|
|
object-binding/linkage step at startup). More specifically, functions may be
|
|
either recursive or mutually recursive. Functions may not call subroutines.
|
|
|
|
<li/> Functions may be defined and called either within <tt>mlr put</tt> or
|
|
<tt>mlr put</tt>.
|
|
|
|
<li/> Functions have read access to <tt>$</tt>-variables and
|
|
<tt>@</tt>-variables but may not modify them.
|
|
See also
|
|
<a href="cookbook.html#Memoization_with_out-of-stream_variables">this cookbook item</a> for an example.
|
|
|
|
<li/> Argument values may be reassigned: they are not read-only.
|
|
|
|
<li/> When a return value is not implicitly returned, this results in a return
|
|
value of absent-null. (In the example above, if there were records for which
|
|
the argument to <tt>f</tt> is non-numeric, the assignments would be skipped.)
|
|
See also the section on
|
|
<a href="#Null_data:_empty_and_absent">empty_and_absent null data</a>.
|
|
|
|
<li/> See the section on <a href="#Local_variables">local variables</a> for
|
|
information on scope and extent of arguments, as well as for information on the
|
|
use of local variables within functions.
|
|
|
|
<li/> See the section on <a href="Expressions_from_files">expressions from
|
|
files</a> for information on the use of <tt>-f</tt> and <tt>-e</tt> flags.
|
|
|
|
</ul>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="User-defined_subroutines"/><h3>User-defined subroutines</h3>
|
|
|
|
<p/>Example:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint --from data/small put -q '
|
|
begin {
|
|
@call_count = 0;
|
|
}
|
|
subr s(n) {
|
|
@call_count += 1;
|
|
if (isnumeric(n)) {
|
|
if (n > 1) {
|
|
call s(n-1);
|
|
} else {
|
|
print "numcalls=" . @call_count;
|
|
}
|
|
}
|
|
}
|
|
print "NR=" . NR;
|
|
call s(NR);
|
|
'
|
|
NR=1
|
|
numcalls=1
|
|
NR=2
|
|
numcalls=3
|
|
NR=3
|
|
numcalls=6
|
|
NR=4
|
|
numcalls=10
|
|
NR=5
|
|
numcalls=15
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>Properties of user-defined subroutines:
|
|
|
|
<ul>
|
|
|
|
<li/> Subroutine bodies start with <tt>subr</tt> and a parameter list, defined
|
|
outside of <tt>begin</tt>, <tt>end</tt>, or other <tt>func</tt> or
|
|
<tt>subr</tt> blocks. (I.e. the Miller DSL has no nested subroutines.)
|
|
|
|
<li/> A subroutine (uniqified by its name) may not be redefined.
|
|
However, functions and subroutines have separate namespaces: you can define a
|
|
subroutine <tt>log</tt> which does not clash with the mathematical <tt>log</tt>
|
|
function.
|
|
|
|
<li/> Subroutines may be defined either before or after use (there is an
|
|
object-binding/linkage step at startup). More specifically, subroutines may be
|
|
either recursive or mutually recursive. Subroutines may call functions.
|
|
|
|
<li/> Subroutines may be defined and called either within <tt>mlr put</tt> or
|
|
<tt>mlr put</tt>.
|
|
|
|
<li/> Subroutines have read/write access to <tt>$</tt>-variables and
|
|
<tt>@</tt>-variables.
|
|
|
|
<li/> Argument values may be reassigned: they are not read-only.
|
|
|
|
<li/> See the section on <a href="#Local_variables">local variables</a> for
|
|
information on scope and extent of arguments, as well as for information on the
|
|
use of local variables within functions.
|
|
|
|
<li/> See the section on <a href="Expressions_from_files">expressions from
|
|
files</a> for information on the use of <tt>-f</tt> and <tt>-e</tt> flags.
|
|
|
|
</ul>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="A_note_on_the_complexity_of_Miller’s_expression_language"/><h2>A note on the complexity of Miller’s expression language</h2>
|
|
|
|
<p/> One of Miller’s strengths is its brevity: it’s much quicker
|
|
— and less error-prone — to type <tt>mlr stats1 -a sum -f x,y -g
|
|
a,b</tt> than having to track summation variables as in <tt>awk</tt>, or using
|
|
Miller’s out-of-stream variables. And the more language features
|
|
Miller’s put-DSL has (for-loops, if-statements, nested control
|
|
structures, etc.) then the <i>less</i> powerful it begins to seem: because of
|
|
the other programming-language features it <i>doesn’t</i> have.
|
|
|
|
<p/> When I was originally prototyping Miller in 2015, the decision I had was
|
|
whether to hand-code in a low-level language like C or Rust, with my own
|
|
hand-rolled DSL, or whether to use a higher-level language (like Python or Lua
|
|
or Nim) and let the <tt>put</tt> statements be handled by the implementation
|
|
language’s own <tt>eval</tt>: the implementation language would take the
|
|
place of a DSL. Multiple performance experiments showed me I could get better
|
|
throughput using the former, and using C in particular — by a wide margin. So
|
|
Miller is C under the hood with a hand-rolled DSL.
|
|
|
|
<p/> I do want to keep focusing on what Miller is good at — concise notation,
|
|
low latency, and high throughput — and not add too much in terms of
|
|
high-level-language features to the DSL. That said, some sort of looping over
|
|
field names is a basic thing to want. As of 4.1.0 we have recursive
|
|
for/while/if structures on about the same complexity level as <tt>awk</tt>.
|
|
While I’m excited by these powerful language features, I hope to keep new
|
|
features beyond 4.1.0 focused on Miller’s sweet spot which is speed plus
|
|
simplicity.
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="then-chaining"/><h1>then-chaining</h1>
|
|
|
|
<p/>
|
|
In accord with the
|
|
<a href="http://en.wikipedia.org/wiki/Unix_philosophy">Unix philosophy</a>, you can pipe data into or out of
|
|
Miller. For example:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
mlr cut --complement -f os_version *.dat | mlr sort -f hostname,uptime
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/>
|
|
You can, if you like, instead simply chain commands together using the
|
|
<tt>then</tt> keyword:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
mlr cut --complement -f os_version then sort -f hostname,uptime *.dat
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
Here’s a performance comparison:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
% cat piped.sh
|
|
mlr cut -x -f i,y data/big | mlr sort -n y > /dev/null
|
|
|
|
% time sh piped.sh
|
|
real 0m2.828s
|
|
user 0m3.183s
|
|
sys 0m0.137s
|
|
|
|
|
|
% cat chained.sh
|
|
mlr cut -x -f i,y then sort -n y data/big > /dev/null
|
|
|
|
% time sh chained.sh
|
|
real 0m2.082s
|
|
user 0m1.933s
|
|
sys 0m0.137s
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
There are two reasons to use then-chaining: one is for performance, although I
|
|
don’t expect this to be a win in all cases. Using then-chaining avoids
|
|
redundant string-parsing and string-formatting at each pipeline step: instead
|
|
input records are parsed once, they are fed through each pipeline stage in
|
|
memory, and then output records are formatted once. On the other hand, Miller
|
|
is single-threaded, while modern systems are usually multi-processor, and when
|
|
streaming-data programs operate through pipes, each one can use a CPU. Rest
|
|
assured you get the same results either way.
|
|
|
|
<p/>The other reason to use then-chaining is for simplicity: you don’t
|
|
have re-type formatting flags (e.g. <tt>--csv --rs lf --fs tab</tt>) at every
|
|
pipeline stage.
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="Data_types"/><h1>Data types</h1>
|
|
|
|
<p/> Miller’s input and output are all string-oriented: there is (as of
|
|
August 2015 anyway) no support for binary record packing. In this sense,
|
|
everything is a string in and out of Miller. During processing, field names
|
|
are always strings, even if they have names like "3"; field values are usually
|
|
strings. Field values’ ability to be interpreted as a non-string type
|
|
only has meaning when comparison or function operations are done on them. And
|
|
it is an error condition if Miller encounters non-numeric (or otherwise
|
|
mistyped) data in a field in which it has been asked to do numeric (or
|
|
otherwise type-specific) operations.
|
|
|
|
<p/> Field values are treated as numeric for the following:
|
|
<ul>
|
|
<li/> Numeric sort: <tt>mlr sort -n</tt>, <tt>mlr sort -nr</tt>.
|
|
<li/> Statistics: <tt>mlr histogram</tt>, <tt>mlr stats1</tt>, <tt>mlr stats2</tt>.
|
|
<li/> Cross-record arithmetic: <tt>mlr step</tt>.
|
|
</ul>
|
|
|
|
<p/>For <tt>mlr put</tt> and <tt>mlr filter</tt>:
|
|
|
|
<ul>
|
|
|
|
<li/> Miller’s types for function processing are <b>null</b> (empty
|
|
string), <b>error</b>, <b>string</b>, <b>float</b> (double-precision),
|
|
<b>int</b> (64-bit signed), and <b>boolean</b>.
|
|
|
|
<li/> On input, string values representable as numbers, e.g. "3" or "3.1", are
|
|
treated as int or float, respectively. If a record has <tt>x=1,y=2</tt> then
|
|
<tt>mlr put '$z=$x+$y'</tt> will produce <tt>x=1,y=2,z=3</tt>, and <tt>mlr put
|
|
'$z=$x.$y'</tt> gives an error. To coerce back to string for processing, use
|
|
the <tt>string</tt> function: <tt>mlr put '$z=string($x).string($y)'</tt> will
|
|
produce <tt>x=1,y=2,z=12</tt>.
|
|
|
|
<li/> On input, string values representable as boolean (e.g. <tt>"true"</tt>,
|
|
<tt>"false"</tt>) are <i>not</i> automatically treated as boolean. (This is
|
|
because <tt>"true"</tt> and <tt>"false"</tt> are ordinary words, and auto
|
|
string-to-boolean on a column consisting of words would result in some strings
|
|
mixed with some booleans.) Use the <tt>boolean</tt> function to coerce: e.g.
|
|
giving the record <tt>x=1,y=2,w=false</tt> to <tt>mlr put '$z=($x<$y) ||
|
|
boolean($w)'</tt>.
|
|
|
|
<li/> Functions take types as described in <tt>mlr --help-all-functions</tt>:
|
|
for example, <tt>log10</tt> takes float input and produces float output,
|
|
<tt>gmt2sec</tt> maps string to int, and <tt>sec2gmt</tt> maps int to string.
|
|
|
|
<li/> All math functions described in <tt>mlr --help-all-functions</tt> take
|
|
integer as well as float input.
|
|
|
|
</ul>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="Null_data:_empty_and_absent"/><h1>Null data: empty and absent</h1>
|
|
|
|
<p/> One of Miller’s key features is its support for <b>heterogeneous</b>
|
|
data. For example, take <tt>mlr sort</tt>: if you try to sort on field
|
|
<tt>hostname</tt> when not all records in the data stream <i>have</i> a field
|
|
named <tt>hostname</tt>, it is not an error (although you could pre-filter the
|
|
data stream using <tt>mlr having-fields --at-least hostname then sort
|
|
...</tt>). Rather, records lacking one or more sort keys are simply output
|
|
contiguously by <tt>mlr sort</tt>.
|
|
|
|
<p/> Miller has two kinds of null data:
|
|
|
|
<ul>
|
|
|
|
<li/> <b>Empty</b>: a field name is present in a record (or in an out-of-stream
|
|
variable) with empty value: e.g. <tt>x=,y=2</tt> in the data input stream, or
|
|
assignment <tt>$x=""</tt> or <tt>@x=""</tt> in <tt>mlr put</tt>.
|
|
|
|
<li/> <b>Absent</b>: a field name is not present, e.g. input record is
|
|
<tt>x=1,y=2</tt> and a <tt>put</tt> or <tt>filter</tt> expression refers to
|
|
<tt>$z</tt>. Or, reading an out-of-stream variable which hasn’t been
|
|
assigned a value yet,
|
|
e.g. <tt>mlr put -q '@sum += $x'; end{emit @sum}'</tt> or <tt>mlr put -q
|
|
'@sum[$a][$b] += $x'; end{emit @sum, "a", "b"}'</tt>.
|
|
|
|
</ul>
|
|
|
|
<p/>You can test these programatically using the functions
|
|
<tt>isempty</tt>/<tt>isnotempty</tt>, <tt>isabsent</tt>/<tt>ispresent</tt>, and
|
|
<tt>isnull</tt>/<tt>isnotnull</tt>. For the last pair, note that null means
|
|
either empty or absent.
|
|
|
|
<p/>
|
|
Rules for null-handling:
|
|
|
|
<ul>
|
|
|
|
<li> Records with one or more empty sort-field values sort after records with
|
|
all sort-field values present:
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr cat data/sort-null.dat
|
|
a=3,b=2
|
|
a=1,b=8
|
|
a=,b=4
|
|
x=9,b=10
|
|
a=5,b=7
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr sort -n a data/sort-null.dat
|
|
a=1,b=8
|
|
a=3,b=2
|
|
a=5,b=7
|
|
a=,b=4
|
|
x=9,b=10
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr sort -nr a data/sort-null.dat
|
|
a=,b=4
|
|
a=5,b=7
|
|
a=3,b=2
|
|
a=1,b=8
|
|
x=9,b=10
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<li> Functions/operators which have one or more <i>empty</i> arguments produce empty output: e.g.
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ echo 'x=2,y=3' | mlr put '$a=$x+$y'
|
|
x=2,y=3,a=5
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ echo 'x=,y=3' | mlr put '$a=$x+$y'
|
|
x=,y=3,a=
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ echo 'x=,y=3' | mlr put '$a=log($x);$b=log($y)'
|
|
x=,y=3,a=,b=1.098612
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
with the exception that the <tt>min</tt> and <tt>max</tt> functions are
|
|
special: if one argument is non-null, it wins:
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ echo 'x=,y=3' | mlr put '$a=min($x,$y);$b=max($x,$y)'
|
|
x=,y=3,a=3,b=3
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<li> Functions of <i>absent</i> variables (e.g. <tt>mlr put '$y =
|
|
log10($nonesuch)'</tt>) evaluate to absent, and arithmetic/bitwise/boolean
|
|
operators with both operands being absent evaluate to absent.
|
|
Arithmetic operators with one absent operand return the other operand.
|
|
More specifically, absent values act like zero for addition/subtraction, and
|
|
one for multiplication: Furthermore, <b>any expression which evaluates to
|
|
absent is not stored in the output record</b>:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ echo 'x=2,y=3' | mlr put '$a=$u+$v; $b=$u+$y; $c=$x+$y'
|
|
x=2,y=3,b=3,c=5
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ echo 'x=2,y=3' | mlr put '$a=min($x,$v);$b=max($u,$y);$c=min($u,$v)'
|
|
x=2,y=3,a=2,b=3
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
</ul>
|
|
|
|
The reasoning is as follows:
|
|
|
|
<ul>
|
|
|
|
<li/> Empty values are explicit in the data so they should explicitly affect accumulations:
|
|
<tt>mlr put '@sum += $x'</tt>
|
|
should accumulate numeric <tt>x</tt> values into the sum but an empty
|
|
<tt>x</tt>, when encountered in the input data stream, should make the sum
|
|
non-numeric. To work around this you can use the
|
|
<tt>isnotnull</tt> function as follows:
|
|
<tt>mlr put 'isnotnull($x) { @sum += $x }'</tt>
|
|
|
|
<li/> Absent stream-record values should not break accumulations, since Miller
|
|
by design handles heterogenous data: the running <tt>@sum</tt> in
|
|
<tt>mlr put '@sum += $x'</tt>
|
|
should not be invalidated for records which have no <tt>x</tt>.
|
|
|
|
<li/> Absent out-of-stream-variable values are precisely what allow you to write
|
|
<tt>mlr put '@sum += $x'</tt>. Otherwise you would have to write
|
|
<tt>mlr put 'begin{@sum = 0}; @sum += $x'</tt> —
|
|
which is tolerable — but for
|
|
<tt>mlr put 'begin{...}; @sum[$a][$b] += $x'</tt>
|
|
you’d have to pre-initialize <tt>@sum</tt> for all values of <tt>$a</tt> and <tt>$b</tt> in your
|
|
input data stream, which is intolerable.
|
|
|
|
<li/> The penalty for the absent feature is that misspelled variables can be hard to find:
|
|
e.g. in <tt>mlr put 'begin{@sumx = 10}; ...; update @sumx somehow per-record; ...; end {@something = @sum * 2}'</tt>
|
|
the accumulator is spelt <tt>@sumx</tt> in the begin-block but <tt>@sum</tt> in the end-block, where since it
|
|
is absent, <tt>@sum*2</tt> evaluates to 2.
|
|
|
|
</ul>
|
|
|
|
<p/>Since absent plus absent is absent (and likewise for other operators),
|
|
accumulations such as <tt>@sum += $x</tt> work correctly on heterogenous data,
|
|
as do within-record formulas if both operands are absent. If one operand is
|
|
present, you may get behavior you don’t desire. To work around this
|
|
— namely, to set an output field only for records which have all the
|
|
inputs present — you can use a pattern-action block with
|
|
<tt>ispresent</tt>:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr cat data/het.dkvp
|
|
resource=/path/to/file,loadsec=0.45,ok=true
|
|
record_count=100,resource=/path/to/file
|
|
resource=/path/to/second/file,loadsec=0.32,ok=true
|
|
record_count=150,resource=/path/to/second/file
|
|
resource=/some/other/path,loadsec=0.97,ok=false
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr put 'ispresent($loadsec) { $loadmillis = $loadsec * 1000 }' data/het.dkvp
|
|
resource=/path/to/file,loadsec=0.45,ok=true,loadmillis=450.000000
|
|
record_count=100,resource=/path/to/file
|
|
resource=/path/to/second/file,loadsec=0.32,ok=true,loadmillis=320.000000
|
|
record_count=150,resource=/path/to/second/file
|
|
resource=/some/other/path,loadsec=0.97,ok=false,loadmillis=970.000000
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr put '$loadmillis = (ispresent($loadsec) ? $loadsec : 0.0) * 1000' data/het.dkvp
|
|
resource=/path/to/file,loadsec=0.45,ok=true,loadmillis=450.000000
|
|
record_count=100,resource=/path/to/file,loadmillis=0.000000
|
|
resource=/path/to/second/file,loadsec=0.32,ok=true,loadmillis=320.000000
|
|
record_count=150,resource=/path/to/second/file,loadmillis=0.000000
|
|
resource=/some/other/path,loadsec=0.97,ok=false,loadmillis=970.000000
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<p/> If you’re interested in a formal description of how empty and absent
|
|
fields participate in arithmetic, here’s a table for plus (other
|
|
arithmetic/boolean/bitwise operators are similar):
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --print-type-arithmetic-info
|
|
(+) | error absent empty string int float bool
|
|
------ + ------ ------ ------ ------ ------ ------ ------
|
|
error | error error error error error error error
|
|
absent | error absent absent error int float error
|
|
empty | error absent empty error empty empty error
|
|
string | error error error error error error error
|
|
int | error int empty error int float error
|
|
float | error float empty error float float error
|
|
bool | error error error error error error error
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="String_literals"/><h1>String literals</h1>
|
|
|
|
<p/>
|
|
You can use the following backslash escapes for strings such as between the double quotes in contexts such as
|
|
<tt>mlr filter '$name =~ "..."'</tt>,
|
|
<tt>mlr put '$name = $othername . "..."'</tt>,
|
|
<tt>mlr put '$name = sub($name, "...", "...")</tt>, etc.:
|
|
|
|
<ul>
|
|
<li/> <tt>\a</tt>: ASCII code 0x07 (alarm/bell)
|
|
<li/> <tt>\b</tt>: ASCII code 0x08 (backspace)
|
|
<li/> <tt>\f</tt>: ASCII code 0x0c (formfeed)
|
|
<li/> <tt>\n</tt>: ASCII code 0x0a (LF/linefeed/newline)
|
|
<li/> <tt>\r</tt>: ASCII code 0x0d (CR/carriage return)
|
|
<li/> <tt>\t</tt>: ASCII code 0x09 (tab)
|
|
<li/> <tt>\v</tt>: ASCII code 0x0b (vertical tab)
|
|
<li/> <tt>\\</tt>: backslash
|
|
<li/> <tt>\"</tt>: double quote
|
|
<li/> <tt>\123</tt>: Octal 123, etc. for <tt>\000</tt> up to <tt>\377</tt>
|
|
<li/> <tt>\x7f</tt>: Hexadecimal 7f, etc. for <tt>\x00</tt> up to <tt>\xff</tt>
|
|
</ul>
|
|
|
|
<p/>See also <a href="https://en.wikipedia.org/wiki/Escape_sequences_in_C">https://en.wikipedia.org/wiki/Escape_sequences_in_C</a>.
|
|
|
|
<p/>These replacements apply only to strings you key in for the DSL expressions for <tt>filter</tt> and <tt>put</tt>:
|
|
that is, if you type <tt>\t</tt> in a string literal for a <tt>filter</tt>/<tt>put</tt> expression, it will be turned into a tab character. If you want a backslash followed by a <tt>t</tt>, then please type <tt>\\t</tt>.
|
|
|
|
<p/>However, these replacements are not done automatically within your data stream. If you wish to make these
|
|
replacements, you can do, for example, for a field named <tt>field</tt>, <tt> mlr put '$field = gsub($field, "\\t",
|
|
"\t")'</tt>. If you need to make such a replacement for all fields in your data, you should probably simply use the
|
|
system <tt>sed</tt> command.
|
|
|
|
<a id="Regular_expressions"/><h1>Regular expressions</h1>
|
|
|
|
<p/>Miller lets you use regular expressions (of type POSIX.2) in the following contexts:
|
|
|
|
<ul>
|
|
|
|
<li/> In <tt>mlr filter</tt> with <tt>=~</tt> or <tt>!=~</tt>, e.g. <tt>mlr
|
|
filter '$url =~ "http.*com"'</tt>
|
|
|
|
<li/> In <tt>mlr put</tt> with <tt>sub</tt> or <tt>gsub</tt>, e.g. <tt>mlr put
|
|
'$url = sub($url, "http.*com", "")'</tt>
|
|
|
|
<li/> In <tt>mlr having-fields</tt>, e.g. <tt>mlr having-fields
|
|
--any-matching '^sda[0-9]'</tt>
|
|
|
|
<li/> In <tt>mlr cut</tt>, e.g. <tt>mlr cut -r -f '^status$,^sda[0-9]'</tt>
|
|
|
|
<li/> In <tt>mlr rename</tt>, e.g. <tt>mlr rename -r '^(sda[0-9]).*$,dev/\1'</tt>
|
|
|
|
<li/> In <tt>mlr grep</tt>, e.g. <tt>mlr --csv grep 00188555487 myfiles*.csv</tt>
|
|
|
|
</ul>
|
|
|
|
<p/>Points demonstrated by the above examples:
|
|
|
|
<ul>
|
|
|
|
<li/> There are no implicit start-of-string or end-of-string anchors; please
|
|
use <tt>^</tt> and/or <tt>$</tt> explicitly.
|
|
|
|
<li/> Miller regexes are wrapped with double quotes rather than slashes.
|
|
|
|
<li/> The <tt>i</tt> after the ending double quote indicates a case-insensitive
|
|
regex.
|
|
|
|
<li/> Capture groups are wrapped with <tt>(...)</tt> rather than
|
|
<tt>\(...\)</tt>; use <tt>\(</tt> and <tt>\)</tt> to match against parentheses.
|
|
|
|
</ul>
|
|
|
|
<p/>For <tt>filter</tt> and <tt>put</tt>, if the regular expression is a string
|
|
literal (the normal case), it is precompiled at process start and reused
|
|
thereafter, which is efficient. If the regular expression is a more complex
|
|
expression, including string concatenation using <tt>.</tt>, or a column name
|
|
(in which case you can take regular expressions from input data!), then regexes
|
|
are compiled on each record which works but is less efficient. As well, in this
|
|
case there is no way to specify case-insensitive matching.
|
|
|
|
<p/>Example:
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ cat data/regex-in-data.dat
|
|
name=jane,regex=^j.*e$
|
|
name=bill,regex=^b[ou]ll$
|
|
name=bull,regex=^b[ou]ll$
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr filter '$name =~ $regex' data/regex-in-data.dat
|
|
name=jane,regex=^j.*e$
|
|
name=bull,regex=^b[ou]ll$
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<a id="Regex_captures"/><h2>Regex captures</h2>
|
|
|
|
<p/>Regex captures of the form <tt>\0</tt> through <tt>\9</tt> are supported as
|
|
follows: <ul>
|
|
|
|
<li/> Captures have in-function context for <tt>sub</tt> and <tt>gsub</tt>.
|
|
For example, the first <tt>\1,\2</tt> pair belong to the first <tt>sub</tt> and
|
|
the second <tt>\1,\2</tt> pair belong to the second <tt>sub</tt>:
|
|
|
|
<p/>
|
|
<div class=pokipanel>
|
|
<pre>
|
|
mlr put '$b = sub($a, "(..)_(...)", "\2-\1"); $c = sub($a, "(..)_(.)(..)", ":\1:\2:\3")'
|
|
</pre>
|
|
</div>
|
|
|
|
<li/> Captures endure for the entirety of a <tt>put</tt> for the <tt>=~</tt>
|
|
and <tt>!=~</tt> operators. For example, here the <tt>\1,\2</tt> are set by the
|
|
<tt>=~</tt> operator and are used by both subsequent assignment statements:
|
|
|
|
<p/>
|
|
<div class=pokipanel>
|
|
<pre>
|
|
mlr put '$a =~ "(..)_(....); $b = "left_\1"; $c = "right_\2"'
|
|
</pre>
|
|
</div>
|
|
|
|
<li/>The captures are not retained across multiple puts. For example, here the
|
|
<tt>\1,\2</tt> won’t be expanded from the regex capture:
|
|
|
|
<p/>
|
|
<div class=pokipanel>
|
|
<pre>
|
|
mlr put '$a =~ "(..)_(....)' then {... something else ...} then put '$b = "left_\1"; $c = "right_\2"'
|
|
</pre>
|
|
</div>
|
|
|
|
<li/> Captures are ignored in <tt>filter</tt> for the <tt>=~</tt> and
|
|
<tt>!=~</tt> operators. For example, there is no mechanism provided to refer to
|
|
the first <tt>(..)</tt> as <tt>\1</tt> or to the second <tt>(....)</tt> as
|
|
<tt>\2</tt> in the following filter statement:
|
|
|
|
<p/>
|
|
<div class=pokipanel>
|
|
<pre>
|
|
mlr filter '$a =~ "(..)_(....)'
|
|
</pre>
|
|
</div>
|
|
|
|
<li/> Up to nine matches are supported: <tt>\1</tt> through <tt>\9</tt>, while
|
|
<tt>\0</tt> is the entire match string; <tt>\15</tt> is treated as <tt>\1</tt>
|
|
followed by an unrelated <tt>5</tt>.
|
|
</ul>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="Operator_precedence"/><h1>Operator precedence</h1>
|
|
|
|
<p/>Operators are listed in order of decreasing precedence, highest first.
|
|
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
Operators Associativity
|
|
--------- -------------
|
|
() left to right
|
|
** right to left
|
|
! ~ unary+ unary- & right to left
|
|
binary* / // % left to right
|
|
binary+ binary- . left to right
|
|
<< >> left to right
|
|
& left to right
|
|
^ left to right
|
|
| left to right
|
|
< <= > >= left to right
|
|
== != =~ !=~ left to right
|
|
&& left to right
|
|
^^ left to right
|
|
|| left to right
|
|
? : right to left
|
|
= N/A for Miller (there is no $a=$b=$c)
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="Operator_and_function_semantics"/><h1>Operator and function semantics</h1>
|
|
|
|
<ul>
|
|
|
|
<li/> Functions are in general pass-throughs straight to the system-standard C
|
|
library.
|
|
|
|
<li/> The <tt>min</tt> and <tt>max</tt> functions are different from other
|
|
multi-argument functions which return null if any of their inputs are null: for
|
|
<tt>min</tt> and <tt>max</tt>, by contrast, if one argument is null, the other
|
|
is returned.
|
|
|
|
<li/> Symmetrically with respect to the bitwise OR, XOR, and AND operators
|
|
<tt>|</tt>, <tt>^</tt>, <tt>&</tt>, Miller has logical operators
|
|
<tt>||</tt>, <tt>^^</tt>, <tt>&&</tt>: the logical XOR not existing in
|
|
C.
|
|
|
|
<li/> The exponentiation operator <tt>**</tt> is familiar from many languages.
|
|
|
|
<li/> The regex-match and regex-not-match operators <tt>=~</tt> and
|
|
<tt>!=~</tt> are similar to those in Ruby and Perl.
|
|
|
|
</ul>
|
|
|
|
<!-- ================================================================ -->
|
|
<a id="Arithmetic"/><h1>Arithmetic</h1>
|
|
|
|
<a id="Input_scanning"/><h2>Input scanning</h2>
|
|
|
|
<p/>Numbers in Miller are double-precision float or 64-bit signed integers.
|
|
Anything scannable as int, e.g <tt>123</tt> or <tt>0xabcd</tt>, is treated as
|
|
an integer; otherwise, input scannable as float (<tt>4.56</tt> or <tt>8e9</tt>)
|
|
is treated as float; everything else is a string.
|
|
|
|
<p/>If you want all numbers to be treated as floats, then you may use
|
|
<tt>float()</tt> in your filter/put expressions (e.g. replacing <tt>$c = $a *
|
|
$b</tt> with <tt>$c = float($a) * float($b)</tt>) — or, more simply, use
|
|
<tt>mlr filter -F</tt> and <tt>mlr put -F</tt> which forces all numeric input,
|
|
whether from expression literals or field values, to float. Likewise <tt>mlr
|
|
stats1 -F</tt> and <tt>mlr step -F</tt> force integerable accumulators (such as
|
|
<tt>count</tt>) to be done in floating-point.
|
|
|
|
<a id="Conversion_by_math_routines"/><h2>Conversion by math routines</h2>
|
|
|
|
<p/>For most math functions, integers are cast to float on input, and produce
|
|
float output: e.g. <tt>exp(0) = 1.0</tt> rather than <tt>1</tt>. The
|
|
following, however, produce integer output if their inputs are integers:
|
|
<tt>+</tt> <tt>-</tt> <tt>*</tt> <tt>/</tt> <tt>//</tt> <tt>%</tt> <tt>abs</tt>
|
|
<tt>ceil</tt> <tt>floor</tt> <tt>max</tt> <tt>min</tt> <tt>round</tt>
|
|
<tt>roundm</tt> <tt>sgn</tt>. As well, <tt>stats1 -a min</tt>, <tt>stats1 -a
|
|
max</tt>, <tt>stats1 -a sum</tt>, <tt>step -a delta</tt>, and <tt>step -a
|
|
rsum</tt> produce integer output if their inputs are integers.
|
|
|
|
<a id="Conversion_by_arithmetic_operators"/><h2>Conversion by arithmetic operators</h2>
|
|
|
|
<p/>The sum, difference, and product of integers is again integer, except for
|
|
when that would overflow a 64-bit integer at which point Miller converts the
|
|
result to float.
|
|
|
|
<p/>The short of it is that Miller does this transparently for you so you
|
|
needn’t think about it.
|
|
|
|
<p/>Implementation details of this, for the interested: integer adds and
|
|
subtracts overflow by at most one bit so it suffices to check sign-changes.
|
|
Thus, Miller allows you to add and subtract arbitrary 64-bit signed integers,
|
|
converting only to float precisely when the result is less than -2<sup>63</sup>
|
|
or greater than 2<sup>63</sup>-1. Multiplies, on the other hand, can overflow
|
|
by a word size and a sign-change technique does not suffice to detect overflow.
|
|
Instead Miller tests whether the floating-point product exceeds the
|
|
representable integer range. Now, 64-bit integers have 64-bit precision while
|
|
IEEE-doubles have only 52-bit mantissas — so, there are 53 bits including
|
|
implicit leading one. The following experiment explicitly demonstrates the
|
|
resolution at this range:
|
|
|
|
<div class=pokipanel>
|
|
<pre>
|
|
64-bit integer 64-bit integer Casted to double Back to 64-bit
|
|
in hex in decimal integer
|
|
0x7ffffffffffff9ff 9223372036854774271 9223372036854773760.000000 0x7ffffffffffff800
|
|
0x7ffffffffffffa00 9223372036854774272 9223372036854773760.000000 0x7ffffffffffff800
|
|
0x7ffffffffffffbff 9223372036854774783 9223372036854774784.000000 0x7ffffffffffffc00
|
|
0x7ffffffffffffc00 9223372036854774784 9223372036854774784.000000 0x7ffffffffffffc00
|
|
0x7ffffffffffffdff 9223372036854775295 9223372036854774784.000000 0x7ffffffffffffc00
|
|
0x7ffffffffffffe00 9223372036854775296 9223372036854775808.000000 0x8000000000000000
|
|
0x7ffffffffffffffe 9223372036854775806 9223372036854775808.000000 0x8000000000000000
|
|
0x7fffffffffffffff 9223372036854775807 9223372036854775808.000000 0x8000000000000000
|
|
</pre>
|
|
</div>
|
|
|
|
<p/>That is, one cannot check an integer product to see if it is precisely
|
|
greater than 2<sup>63</sup>-1 or less than -2<sup>63</sup> using either integer
|
|
arithmetic (it may have already overflowed) or using double-precision (due to
|
|
granularity). Instead Miller checks for overflow in 64-bit integer
|
|
multiplication by seeing whether the absolute value of the double-precision
|
|
product exceeds the largest representable IEEE double less than 2<sup>63</sup>,
|
|
which we see from the listing above is 9223372036854774784. (An alternative
|
|
would be to do all integer multiplies using handcrafted multi-word 128-bit
|
|
arithmetic. This approach is not taken.)
|
|
|
|
<a id="Pythonic_division"/><h2>Pythonic division</h2>
|
|
|
|
<p/>Division and remainder are
|
|
<a href="http://python-history.blogspot.com/2010/08/why-pythons-integer-division-floors.html">
|
|
pythonic</a>:
|
|
<ul>
|
|
<li/> Quotient of integers is floating-point: <tt>7/2</tt> is <tt>3.5</tt>.
|
|
<li/> Integer division is done with <tt>//</tt>: <tt>7/2</tt> is <tt>3</tt>.
|
|
This rounds toward the negative.
|
|
<li/> Remainders are non-negative.
|
|
</ul>
|
|
</div>
|
|
</td>
|
|
|
|
</table>
|
|
</body>
|
|
</html>
|