Mirrors/miller
1
0
Fork 0
mirror of https://github.com/johnkerl/miller.git synced 2026-08-02 04:22:59 +00:00
Code Issues Projects Releases Wiki Activity
miller/docs6/_build/html/reference.html
John Kerl 1aaec752af Start port of docs to Miller 6
2021-05-24 00:11:53 -04:00

1112 lines
No EOL
107 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Main reference &#8212; Miller 5.10.2 documentation</title>
<link rel="stylesheet" href="_static/classic.css" type="text/css" />
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<script id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script>
<script src="_static/jquery.js"></script>
<script src="_static/underscore.js"></script>
<script src="_static/doctools.js"></script>
<script src="_static/language_data.js"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="Verbs reference" href="reference-verbs.html" />
<link rel="prev" title="Mixing with other languages" href="data-sharing.html" />
</head><body>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="genindex.html" title="General Index"
accesskey="I">index</a></li>
<li class="right" >
<a href="reference-verbs.html" title="Verbs reference"
accesskey="N">next</a> |</li>
<li class="right" >
<a href="data-sharing.html" title="Mixing with other languages"
accesskey="P">previous</a> |</li>
<li class="nav-item nav-item-0"><a href="index.html">Miller 5.10.2 documentation</a> &#187;</li>
<li class="nav-item nav-item-this"><a href="">Main reference</a></li>
</ul>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body" role="main">
<div class="section" id="main-reference">
<h1>Main reference<a class="headerlink" href="#main-reference" title="Permalink to this headline"></a></h1>
<div class="section" id="command-overview">
<span id="reference-command-overview"></span><h2>Command overview<a class="headerlink" href="#command-overview" title="Permalink to this headline"></a></h2>
<p>Whereas the Unix toolkit is made of the separate executables <code class="docutils literal notranslate"><span class="pre">cat</span></code>, <code class="docutils literal notranslate"><span class="pre">tail</span></code>, <code class="docutils literal notranslate"><span class="pre">cut</span></code>,
<code class="docutils literal notranslate"><span class="pre">sort</span></code>, etc., Miller has subcommands, or <strong>verbs</strong>, invoked as follows:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">mlr</span> <span class="n">tac</span> <span class="o">*.</span><span class="n">dat</span>
<span class="n">mlr</span> <span class="n">cut</span> <span class="o">--</span><span class="n">complement</span> <span class="o">-</span><span class="n">f</span> <span class="n">os_version</span> <span class="o">*.</span><span class="n">dat</span>
<span class="n">mlr</span> <span class="n">sort</span> <span class="o">-</span><span class="n">f</span> <span class="n">hostname</span><span class="p">,</span><span class="n">uptime</span> <span class="o">*.</span><span class="n">dat</span>
</pre></div>
</div>
<p>These fall into categories as follows:</p>
<ul class="simple">
<li><p>Analogs of their Unix-toolkit namesakes, discussed below as well as in <a class="reference internal" href="feature-comparison.html"><span class="doc">Unix-toolkit context</span></a>: <a class="reference internal" href="reference-verbs.html#reference-verbs-cat"><span class="std std-ref">cat</span></a> <a class="reference internal" href="reference-verbs.html#reference-verbs-cut"><span class="std std-ref">cut</span></a> <a class="reference internal" href="reference-verbs.html#reference-verbs-grep"><span class="std std-ref">grep</span></a> <a class="reference internal" href="reference-verbs.html#reference-verbs-head"><span class="std std-ref">head</span></a> <a class="reference internal" href="reference-verbs.html#reference-verbs-join"><span class="std std-ref">join</span></a> <a class="reference internal" href="reference-verbs.html#reference-verbs-sort"><span class="std std-ref">sort</span></a> <a class="reference internal" href="reference-verbs.html#reference-verbs-tac"><span class="std std-ref">tac</span></a> <a class="reference internal" href="reference-verbs.html#reference-verbs-tail"><span class="std std-ref">tail</span></a> <a class="reference internal" href="reference-verbs.html#reference-verbs-top"><span class="std std-ref">top</span></a> <a class="reference internal" href="reference-verbs.html#reference-verbs-uniq"><span class="std std-ref">uniq</span></a>.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">awk</span></code>-like functionality: <a class="reference internal" href="reference-verbs.html#reference-verbs-filter"><span class="std std-ref">filter</span></a> <a class="reference internal" href="reference-verbs.html#reference-verbs-put"><span class="std std-ref">put</span></a> <a class="reference internal" href="reference-verbs.html#reference-verbs-sec2gmt"><span class="std std-ref">sec2gmt</span></a> <a class="reference internal" href="reference-verbs.html#reference-verbs-sec2gmtdate"><span class="std std-ref">sec2gmtdate</span></a> <a class="reference internal" href="reference-verbs.html#reference-verbs-step"><span class="std std-ref">step</span></a> <a class="reference internal" href="reference-verbs.html#reference-verbs-tee"><span class="std std-ref">tee</span></a>.</p></li>
<li><p>Statistically oriented: <a class="reference internal" href="reference-verbs.html#reference-verbs-bar"><span class="std std-ref">bar</span></a> <a class="reference internal" href="reference-verbs.html#reference-verbs-bootstrap"><span class="std std-ref">bootstrap</span></a> <a class="reference internal" href="reference-verbs.html#reference-verbs-decimate"><span class="std std-ref">decimate</span></a> <a class="reference internal" href="reference-verbs.html#reference-verbs-histogram"><span class="std std-ref">histogram</span></a> <a class="reference internal" href="reference-verbs.html#reference-verbs-least-frequent"><span class="std std-ref">least-frequent</span></a> <a class="reference internal" href="reference-verbs.html#reference-verbs-most-frequent"><span class="std std-ref">most-frequent</span></a> <a class="reference internal" href="reference-verbs.html#reference-verbs-sample"><span class="std std-ref">sample</span></a> <a class="reference internal" href="reference-verbs.html#reference-verbs-shuffle"><span class="std std-ref">shuffle</span></a> <a class="reference internal" href="reference-verbs.html#reference-verbs-stats1"><span class="std std-ref">stats1</span></a> <a class="reference internal" href="reference-verbs.html#reference-verbs-stats2"><span class="std std-ref">stats2</span></a>.</p></li>
<li><p>Particularly oriented toward <a class="reference internal" href="record-heterogeneity.html"><span class="doc">Record-heterogeneity</span></a>, although all Miller commands can handle heterogeneous records: <a class="reference internal" href="reference-verbs.html#reference-verbs-group-by"><span class="std std-ref">group-by</span></a> <a class="reference internal" href="reference-verbs.html#reference-verbs-group-like"><span class="std std-ref">group-like</span></a> <a class="reference internal" href="reference-verbs.html#reference-verbs-having-fields"><span class="std std-ref">having-fields</span></a>.</p></li>
<li><p>These draw from other sources (see also <a class="reference internal" href="originality.html"><span class="doc">How original is Miller?</span></a>): <a class="reference internal" href="reference-verbs.html#reference-verbs-count-distinct"><span class="std std-ref">count-distinct</span></a> is SQL-ish, and <a class="reference internal" href="reference-verbs.html#reference-verbs-rename"><span class="std std-ref">rename</span></a> can be done by <code class="docutils literal notranslate"><span class="pre">sed</span></code> (which does it faster: see <a class="reference internal" href="performance.html"><span class="doc">Performance</span></a>. Verbs: <a class="reference internal" href="reference-verbs.html#reference-verbs-check"><span class="std std-ref">check</span></a> <a class="reference internal" href="reference-verbs.html#reference-verbs-count-distinct"><span class="std std-ref">count-distinct</span></a> <a class="reference internal" href="reference-verbs.html#reference-verbs-label"><span class="std std-ref">label</span></a> <a class="reference internal" href="reference-verbs.html#reference-verbs-merge-fields"><span class="std std-ref">merge-fields</span></a> <a class="reference internal" href="reference-verbs.html#reference-verbs-nest"><span class="std std-ref">nest</span></a> <a class="reference internal" href="reference-verbs.html#reference-verbs-nothing"><span class="std std-ref">nothing</span></a> <a class="reference internal" href="reference-verbs.html#reference-verbs-regularize"><span class="std std-ref">regularize</span></a> <a class="reference internal" href="reference-verbs.html#reference-verbs-rename"><span class="std std-ref">rename</span></a> <a class="reference internal" href="reference-verbs.html#reference-verbs-reorder"><span class="std std-ref">reorder</span></a> <a class="reference internal" href="reference-verbs.html#reference-verbs-reshape"><span class="std std-ref">reshape</span></a> <a class="reference internal" href="reference-verbs.html#reference-verbs-seqgen"><span class="std std-ref">seqgen</span></a>.</p></li>
</ul>
</div>
<div class="section" id="i-o-options">
<h2>I/O options<a class="headerlink" href="#i-o-options" title="Permalink to this headline"></a></h2>
<div class="section" id="formats">
<h3>Formats<a class="headerlink" href="#formats" title="Permalink to this headline"></a></h3>
<p>Options:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="o">--</span><span class="n">dkvp</span> <span class="o">--</span><span class="n">idkvp</span> <span class="o">--</span><span class="n">odkvp</span>
<span class="o">--</span><span class="n">nidx</span> <span class="o">--</span><span class="n">inidx</span> <span class="o">--</span><span class="n">onidx</span>
<span class="o">--</span><span class="n">csv</span> <span class="o">--</span><span class="n">icsv</span> <span class="o">--</span><span class="n">ocsv</span>
<span class="o">--</span><span class="n">csvlite</span> <span class="o">--</span><span class="n">icsvlite</span> <span class="o">--</span><span class="n">ocsvlite</span>
<span class="o">--</span><span class="n">pprint</span> <span class="o">--</span><span class="n">ipprint</span> <span class="o">--</span><span class="n">opprint</span> <span class="o">--</span><span class="n">right</span>
<span class="o">--</span><span class="n">xtab</span> <span class="o">--</span><span class="n">ixtab</span> <span class="o">--</span><span class="n">oxtab</span>
<span class="o">--</span><span class="n">json</span> <span class="o">--</span><span class="n">ijson</span> <span class="o">--</span><span class="n">ojson</span>
</pre></div>
</div>
<p>These are as discussed in <a class="reference internal" href="file-formats.html"><span class="doc">File formats</span></a>, with the exception of <code class="docutils literal notranslate"><span class="pre">--right</span></code> which makes pretty-printed output right-aligned:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ mlr --opprint cat data/small
a b i x y
pan pan 1 0.3467901443380824 0.7268028627434533
eks pan 2 0.7586799647899636 0.5221511083334797
wye wye 3 0.20460330576630303 0.33831852551664776
eks wye 4 0.38139939387114097 0.13418874328430463
wye pan 5 0.5732889198020006 0.8636244699032729
</pre></div>
</div>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ mlr --opprint --right cat data/small
a b i x y
pan pan 1 0.3467901443380824 0.7268028627434533
eks pan 2 0.7586799647899636 0.5221511083334797
wye wye 3 0.20460330576630303 0.33831852551664776
eks wye 4 0.38139939387114097 0.13418874328430463
wye pan 5 0.5732889198020006 0.8636244699032729
</pre></div>
</div>
<p>Additional notes:</p>
<ul class="simple">
<li><p>Use <code class="docutils literal notranslate"><span class="pre">--csv</span></code>, <code class="docutils literal notranslate"><span class="pre">--pprint</span></code>, etc. when the input and output formats are the same.</p></li>
<li><p>Use <code class="docutils literal notranslate"><span class="pre">--icsv</span> <span class="pre">--opprint</span></code>, etc. when you want format conversion as part of what Miller does to your data.</p></li>
<li><p>DKVP (key-value-pair) format is the default for input and output. So, <code class="docutils literal notranslate"><span class="pre">--oxtab</span></code> is the same as <code class="docutils literal notranslate"><span class="pre">--idkvp</span> <span class="pre">--oxtab</span></code>.</p></li>
</ul>
<p><strong>Pro-tip:</strong> Please use either <strong>–format1</strong>, or <strong>–iformat1 –oformat2</strong>. If you use <strong>–format1 –oformat2</strong> then what happens is that flags are set up for input <em>and</em> output for format1, some of which are overwritten for output in format2. For technical reasons, having <code class="docutils literal notranslate"><span class="pre">--oformat2</span></code> clobber all the output-related effects of <code class="docutils literal notranslate"><span class="pre">--format1</span></code> also removes some flexibility from the command-line interface. See also <a class="reference external" href="https://github.com/johnkerl/miller/issues/180">https://github.com/johnkerl/miller/issues/180</a> and <a class="reference external" href="https://github.com/johnkerl/miller/issues/199">https://github.com/johnkerl/miller/issues/199</a>.</p>
</div>
<div class="section" id="in-place-mode">
<h3>In-place mode<a class="headerlink" href="#in-place-mode" title="Permalink to this headline"></a></h3>
<p>Use the <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">-I</span></code> flag to process files in-place. For example, <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">-I</span> <span class="pre">--csv</span> <span class="pre">cut</span> <span class="pre">-x</span> <span class="pre">-f</span> <span class="pre">unwanted_column_name</span> <span class="pre">mydata/*.csv</span></code> will remove <code class="docutils literal notranslate"><span class="pre">unwanted_column_name</span></code> from all your <code class="docutils literal notranslate"><span class="pre">*.csv</span></code> files in your <code class="docutils literal notranslate"><span class="pre">mydata/</span></code> subdirectory.</p>
<p>By default, Miller output goes to the screen (or you can redirect a file using <code class="docutils literal notranslate"><span class="pre">&gt;</span></code> or to another process using <code class="docutils literal notranslate"><span class="pre">|</span></code>). With <code class="docutils literal notranslate"><span class="pre">-I</span></code>, for each file name on the command line, output is written to a temporary file in the same directory. Miller writes its output into that temp file, which is then renamed over the original. Then, processing continues on the next file. Each file is processed in isolation: if the output format is CSV, CSV headers will be present in each output file; statistics are only over each file’s own records; and so on.</p>
<p>Please see <a class="reference internal" href="10min.html#min-choices-for-printing-to-files"><span class="std std-ref">Choices for printing to files</span></a> for examples.</p>
</div>
<div class="section" id="compression">
<h3>Compression<a class="headerlink" href="#compression" title="Permalink to this headline"></a></h3>
<p>Options:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="o">--</span><span class="n">prepipe</span> <span class="p">{</span><span class="n">command</span><span class="p">}</span>
</pre></div>
</div>
<p>The prepipe command is anything which reads from standard input and produces data acceptable to Miller. Nominally this allows you to use whichever decompression utilities you have installed on your system, on a per-file basis. If the command has flags, quote them: e.g. <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">--prepipe</span> <span class="pre">'zcat</span> <span class="pre">-cf'</span></code>. Examples:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span># These two produce the same output:
$ gunzip &lt; myfile1.csv.gz | mlr cut -f hostname,uptime
$ mlr --prepipe gunzip cut -f hostname,uptime myfile1.csv.gz
# With multiple input files you need --prepipe:
$ mlr --prepipe gunzip cut -f hostname,uptime myfile1.csv.gz myfile2.csv.gz
$ mlr --prepipe gunzip --idkvp --oxtab cut -f hostname,uptime myfile1.dat.gz myfile2.dat.gz
</pre></div>
</div>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span># Similar to the above, but with compressed output as well as input:
$ gunzip &lt; myfile1.csv.gz | mlr cut -f hostname,uptime | gzip &gt; outfile.csv.gz
$ mlr --prepipe gunzip cut -f hostname,uptime myfile1.csv.gz | gzip &gt; outfile.csv.gz
$ mlr --prepipe gunzip cut -f hostname,uptime myfile1.csv.gz myfile2.csv.gz | gzip &gt; outfile.csv.gz
</pre></div>
</div>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span># Similar to the above, but with different compression tools for input and output:
$ gunzip &lt; myfile1.csv.gz | mlr cut -f hostname,uptime | xz -z &gt; outfile.csv.xz
$ xz -cd &lt; myfile1.csv.xz | mlr cut -f hostname,uptime | gzip &gt; outfile.csv.xz
$ mlr --prepipe &#39;xz -cd&#39; cut -f hostname,uptime myfile1.csv.xz myfile2.csv.xz | xz -z &gt; outfile.csv.xz
</pre></div>
</div>
</div>
<div class="section" id="record-field-pair-separators">
<span id="reference-separators"></span><h3>Record/field/pair separators<a class="headerlink" href="#record-field-pair-separators" title="Permalink to this headline"></a></h3>
<p>Miller has record separators <code class="docutils literal notranslate"><span class="pre">IRS</span></code> and <code class="docutils literal notranslate"><span class="pre">ORS</span></code>, field separators <code class="docutils literal notranslate"><span class="pre">IFS</span></code> and <code class="docutils literal notranslate"><span class="pre">OFS</span></code>, and pair separators <code class="docutils literal notranslate"><span class="pre">IPS</span></code> and <code class="docutils literal notranslate"><span class="pre">OPS</span></code>. For example, in the DKVP line <code class="docutils literal notranslate"><span class="pre">a=1,b=2,c=3</span></code>, the record separator is newline, field separator is comma, and pair separator is the equals sign. These are the default values.</p>
<p>Options:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="o">--</span><span class="n">rs</span> <span class="o">--</span><span class="n">irs</span> <span class="o">--</span><span class="n">ors</span>
<span class="o">--</span><span class="n">fs</span> <span class="o">--</span><span class="n">ifs</span> <span class="o">--</span><span class="n">ofs</span> <span class="o">--</span><span class="n">repifs</span>
<span class="o">--</span><span class="n">ps</span> <span class="o">--</span><span class="n">ips</span> <span class="o">--</span><span class="n">ops</span>
</pre></div>
</div>
<ul class="simple">
<li><p>You can change a separator from input to output via e.g. <code class="docutils literal notranslate"><span class="pre">--ifs</span> <span class="pre">=</span> <span class="pre">--ofs</span> <span class="pre">:</span></code>. Or, you can specify that the same separator is to be used for input and output via e.g. <code class="docutils literal notranslate"><span class="pre">--fs</span> <span class="pre">:</span></code>.</p></li>
<li><p>The pair separator is only relevant to DKVP format.</p></li>
<li><p>Pretty-print and xtab formats ignore the separator arguments altogether.</p></li>
<li><p>The <code class="docutils literal notranslate"><span class="pre">--repifs</span></code> means that multiple successive occurrences of the field separator count as one. For example, in CSV data we often signify nulls by empty strings, e.g. <code class="docutils literal notranslate"><span class="pre">2,9,,,,,6,5,4</span></code>. On the other hand, if the field separator is a space, it might be more natural to parse <code class="docutils literal notranslate"><span class="pre">2</span> <span class="pre">4</span>&#160;&#160;&#160; <span class="pre">5</span></code> the same as <code class="docutils literal notranslate"><span class="pre">2</span> <span class="pre">4</span> <span class="pre">5</span></code>: <code class="docutils literal notranslate"><span class="pre">--repifs</span> <span class="pre">--ifs</span> <span class="pre">'</span> <span class="pre">'</span></code> lets this happen. In fact, the <code class="docutils literal notranslate"><span class="pre">--ipprint</span></code> option above is internally implemented in terms of <code class="docutils literal notranslate"><span class="pre">--repifs</span></code>.</p></li>
<li><p>Just write out the desired separator, e.g. <code class="docutils literal notranslate"><span class="pre">--ofs</span> <span class="pre">'|'</span></code>. But you may use the symbolic names <code class="docutils literal notranslate"><span class="pre">newline</span></code>, <code class="docutils literal notranslate"><span class="pre">space</span></code>, <code class="docutils literal notranslate"><span class="pre">tab</span></code>, <code class="docutils literal notranslate"><span class="pre">pipe</span></code>, or <code class="docutils literal notranslate"><span class="pre">semicolon</span></code> if you like.</p></li>
</ul>
</div>
<div class="section" id="number-formatting">
<h3>Number formatting<a class="headerlink" href="#number-formatting" title="Permalink to this headline"></a></h3>
<p>The command-line option <code class="docutils literal notranslate"><span class="pre">--ofmt</span> <span class="pre">{format</span> <span class="pre">string}</span></code> is the global number format for commands which generate numeric output, e.g. <code class="docutils literal notranslate"><span class="pre">stats1</span></code>, <code class="docutils literal notranslate"><span class="pre">stats2</span></code>, <code class="docutils literal notranslate"><span class="pre">histogram</span></code>, and <code class="docutils literal notranslate"><span class="pre">step</span></code>, as well as <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">put</span></code>. Examples:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="o">--</span><span class="n">ofmt</span> <span class="o">%.</span><span class="mi">9</span><span class="n">le</span> <span class="o">--</span><span class="n">ofmt</span> <span class="o">%.</span><span class="mi">6</span><span class="n">lf</span> <span class="o">--</span><span class="n">ofmt</span> <span class="o">%.</span><span class="mi">0</span><span class="n">lf</span>
</pre></div>
</div>
<p>These are just C <code class="docutils literal notranslate"><span class="pre">printf</span></code> formats applied to double-precision numbers. Please don’t use <code class="docutils literal notranslate"><span class="pre">%s</span></code> or <code class="docutils literal notranslate"><span class="pre">%d</span></code>. Additionally, if you use leading width (e.g. <code class="docutils literal notranslate"><span class="pre">%18.12lf</span></code>) then the output will contain embedded whitespace, which may not be what you want if you pipe the output to something else, particularly CSV. I use Miller’s pretty-print format (<code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">--opprint</span></code>) to column-align numerical data.</p>
<p>To apply formatting to a single field, overriding the global <code class="docutils literal notranslate"><span class="pre">ofmt</span></code>, use <code class="docutils literal notranslate"><span class="pre">fmtnum</span></code> function within <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">put</span></code>. For example:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ echo &#39;x=3.1,y=4.3&#39; | mlr put &#39;$z=fmtnum($x*$y,&quot;%08lf&quot;)&#39;
x=3.1,y=4.3,z=13.330000
</pre></div>
</div>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ echo &#39;x=0xffff,y=0xff&#39; | mlr put &#39;$z=fmtnum(int($x*$y),&quot;%08llx&quot;)&#39;
x=0xffff,y=0xff,z=00feff01
</pre></div>
</div>
<p>Input conversion from hexadecimal is done automatically on fields handled by <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">put</span></code> and <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">filter</span></code> as long as the field value begins with “0x”. To apply output conversion to hexadecimal on a single column, you may use <code class="docutils literal notranslate"><span class="pre">fmtnum</span></code>, or the keystroke-saving <code class="docutils literal notranslate"><span class="pre">hexfmt</span></code> function. Example:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ echo &#39;x=0xffff,y=0xff&#39; | mlr put &#39;$z=hexfmt($x*$y)&#39;
x=0xffff,y=0xff,z=0xfeff01
</pre></div>
</div>
</div>
</div>
<div class="section" id="data-transformations-verbs">
<h2>Data transformations (verbs)<a class="headerlink" href="#data-transformations-verbs" title="Permalink to this headline"></a></h2>
<p>Please see the separate page <a class="reference internal" href="reference-verbs.html"><span class="doc">Verbs reference</span></a>.</p>
</div>
<div class="section" id="expression-language-for-filter-and-put">
<h2>Expression language for filter and put<a class="headerlink" href="#expression-language-for-filter-and-put" title="Permalink to this headline"></a></h2>
<p>Please see the separate page <a class="reference internal" href="reference-dsl.html"><span class="doc">DSL reference</span></a>.</p>
</div>
<div class="section" id="then-chaining">
<h2>then-chaining<a class="headerlink" href="#then-chaining" title="Permalink to this headline"></a></h2>
<p>In accord with the <a class="reference external" href="http://en.wikipedia.org/wiki/Unix_philosophy">Unix philosophy</a>, you can pipe data into or out of Miller. For example:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">mlr</span> <span class="n">cut</span> <span class="o">--</span><span class="n">complement</span> <span class="o">-</span><span class="n">f</span> <span class="n">os_version</span> <span class="o">*.</span><span class="n">dat</span> <span class="o">|</span> <span class="n">mlr</span> <span class="n">sort</span> <span class="o">-</span><span class="n">f</span> <span class="n">hostname</span><span class="p">,</span><span class="n">uptime</span>
</pre></div>
</div>
<p>You can, if you like, instead simply chain commands together using the <code class="docutils literal notranslate"><span class="pre">then</span></code> keyword:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">mlr</span> <span class="n">cut</span> <span class="o">--</span><span class="n">complement</span> <span class="o">-</span><span class="n">f</span> <span class="n">os_version</span> <span class="n">then</span> <span class="n">sort</span> <span class="o">-</span><span class="n">f</span> <span class="n">hostname</span><span class="p">,</span><span class="n">uptime</span> <span class="o">*.</span><span class="n">dat</span>
</pre></div>
</div>
<p>(You can precede the very first verb with <code class="docutils literal notranslate"><span class="pre">then</span></code>, if you like, for symmetry.)</p>
<p>Here’s a performance comparison:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="o">%</span> <span class="n">cat</span> <span class="n">piped</span><span class="o">.</span><span class="n">sh</span>
<span class="n">mlr</span> <span class="n">cut</span> <span class="o">-</span><span class="n">x</span> <span class="o">-</span><span class="n">f</span> <span class="n">i</span><span class="p">,</span><span class="n">y</span> <span class="n">data</span><span class="o">/</span><span class="n">big</span> <span class="o">|</span> <span class="n">mlr</span> <span class="n">sort</span> <span class="o">-</span><span class="n">n</span> <span class="n">y</span> <span class="o">&gt;</span> <span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">null</span>
<span class="o">%</span> <span class="n">time</span> <span class="n">sh</span> <span class="n">piped</span><span class="o">.</span><span class="n">sh</span>
<span class="n">real</span> <span class="mi">0</span><span class="n">m2</span><span class="o">.</span><span class="mi">828</span><span class="n">s</span>
<span class="n">user</span> <span class="mi">0</span><span class="n">m3</span><span class="o">.</span><span class="mi">183</span><span class="n">s</span>
<span class="n">sys</span> <span class="mi">0</span><span class="n">m0</span><span class="o">.</span><span class="mi">137</span><span class="n">s</span>
<span class="o">%</span> <span class="n">cat</span> <span class="n">chained</span><span class="o">.</span><span class="n">sh</span>
<span class="n">mlr</span> <span class="n">cut</span> <span class="o">-</span><span class="n">x</span> <span class="o">-</span><span class="n">f</span> <span class="n">i</span><span class="p">,</span><span class="n">y</span> <span class="n">then</span> <span class="n">sort</span> <span class="o">-</span><span class="n">n</span> <span class="n">y</span> <span class="n">data</span><span class="o">/</span><span class="n">big</span> <span class="o">&gt;</span> <span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">null</span>
<span class="o">%</span> <span class="n">time</span> <span class="n">sh</span> <span class="n">chained</span><span class="o">.</span><span class="n">sh</span>
<span class="n">real</span> <span class="mi">0</span><span class="n">m2</span><span class="o">.</span><span class="mi">082</span><span class="n">s</span>
<span class="n">user</span> <span class="mi">0</span><span class="n">m1</span><span class="o">.</span><span class="mi">933</span><span class="n">s</span>
<span class="n">sys</span> <span class="mi">0</span><span class="n">m0</span><span class="o">.</span><span class="mi">137</span><span class="n">s</span>
</pre></div>
</div>
<p>There are two reasons to use then-chaining: one is for performance, although I don’t expect this to be a win in all cases. Using then-chaining avoids redundant string-parsing and string-formatting at each pipeline step: instead input records are parsed once, they are fed through each pipeline stage in memory, and then output records are formatted once. On the other hand, Miller is single-threaded, while modern systems are usually multi-processor, and when streaming-data programs operate through pipes, each one can use a CPU. Rest assured you get the same results either way.</p>
<p>The other reason to use then-chaining is for simplicity: you don’t have re-type formatting flags (e.g. <code class="docutils literal notranslate"><span class="pre">--csv</span> <span class="pre">--fs</span> <span class="pre">tab</span></code>) at every pipeline stage.</p>
</div>
<div class="section" id="auxiliary-commands">
<h2>Auxiliary commands<a class="headerlink" href="#auxiliary-commands" title="Permalink to this headline"></a></h2>
<p>There are a few nearly-standalone programs which have nothing to do with the rest of Miller, do not participate in record streams, and do not deal with file formats. They might as well be little standalone executables but they’re delivered within the main Miller executable for convenience.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ mlr aux-list
Available subcommands:
aux-list
lecat
termcvt
hex
unhex
netbsd-strptime
For more information, please invoke mlr {subcommand} --help
</pre></div>
</div>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ mlr lecat --help
Usage: mlr lecat [options] {zero or more file names}
Simply echoes input, but flags CR characters in red and LF characters in green.
If zero file names are supplied, standard input is read.
Options:
--mono: don&#39;t try to colorize the output
-h or --help: print this message
</pre></div>
</div>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ mlr termcvt --help
Usage: mlr termcvt [option] {zero or more file names}
Option (exactly one is required):
--cr2crlf
--lf2crlf
--crlf2cr
--crlf2lf
--cr2lf
--lf2cr
-I in-place processing (default is to write to stdout)
-h or --help: print this message
Zero file names means read from standard input.
Output is always to standard output; files are not written in-place.
</pre></div>
</div>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ mlr hex --help
Usage: mlr hex [options] {zero or more file names}
Simple hex-dump.
If zero file names are supplied, standard input is read.
Options:
-r: print only raw hex without leading offset indicators or trailing ASCII dump.
-h or --help: print this message
</pre></div>
</div>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ mlr unhex --help
Usage: mlr unhex [option] {zero or more file names}
Options:
-h or --help: print this message
Zero file names means read from standard input.
Output is always to standard output; files are not written in-place.
</pre></div>
</div>
<p>Examples:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ echo &#39;Hello, world!&#39; | mlr lecat --mono
Hello, world![LF]
</pre></div>
</div>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ echo &#39;Hello, world!&#39; | mlr termcvt --lf2crlf | mlr lecat --mono
Hello, world![CR][LF]
</pre></div>
</div>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ mlr hex data/budget.csv
00000000: 23 20 41 73 61 6e 61 20 2d 2d 20 68 65 72 65 20 |# Asana -- here |
00000010: 61 72 65 20 74 68 65 20 62 75 64 67 65 74 20 66 |are the budget f|
00000020: 69 67 75 72 65 73 20 79 6f 75 20 61 73 6b 65 64 |igures you asked|
00000030: 20 66 6f 72 21 0a 74 79 70 65 2c 71 75 61 6e 74 | for!.type,quant|
00000040: 69 74 79 0a 70 75 72 70 6c 65 2c 34 35 36 2e 37 |ity.purple,456.7|
00000050: 38 0a 67 72 65 65 6e 2c 36 37 38 2e 31 32 0a 6f |8.green,678.12.o|
00000060: 72 61 6e 67 65 2c 31 32 33 2e 34 35 0a |range,123.45.|
</pre></div>
</div>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ mlr hex -r data/budget.csv
23 20 41 73 61 6e 61 20 2d 2d 20 68 65 72 65 20
61 72 65 20 74 68 65 20 62 75 64 67 65 74 20 66
69 67 75 72 65 73 20 79 6f 75 20 61 73 6b 65 64
20 66 6f 72 21 0a 74 79 70 65 2c 71 75 61 6e 74
69 74 79 0a 70 75 72 70 6c 65 2c 34 35 36 2e 37
38 0a 67 72 65 65 6e 2c 36 37 38 2e 31 32 0a 6f
72 61 6e 67 65 2c 31 32 33 2e 34 35 0a
</pre></div>
</div>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ mlr hex -r data/budget.csv | sed &#39;s/20/2a/g&#39; | mlr unhex
#*Asana*--*here*are*the*budget*figures*you*asked*for!
type,quantity
purple,456.78
green,678.12
orange,123.45
</pre></div>
</div>
</div>
<div class="section" id="data-types">
<h2>Data types<a class="headerlink" href="#data-types" title="Permalink to this headline"></a></h2>
<p>Miller’s input and output are all string-oriented: there is (as of August 2015 anyway) no support for binary record packing. In this sense, everything is a string in and out of Miller. During processing, field names are always strings, even if they have names like “3”; field values are usually strings. Field values’ ability to be interpreted as a non-string type only has meaning when comparison or function operations are done on them. And it is an error condition if Miller encounters non-numeric (or otherwise mistyped) data in a field in which it has been asked to do numeric (or otherwise type-specific) operations.</p>
<p>Field values are treated as numeric for the following:</p>
<ul class="simple">
<li><p>Numeric sort: <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">sort</span> <span class="pre">-n</span></code>, <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">sort</span> <span class="pre">-nr</span></code>.</p></li>
<li><p>Statistics: <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">histogram</span></code>, <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">stats1</span></code>, <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">stats2</span></code>.</p></li>
<li><p>Cross-record arithmetic: <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">step</span></code>.</p></li>
</ul>
<p>For <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">put</span></code> and <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">filter</span></code>:</p>
<ul class="simple">
<li><p>Miller’s types for function processing are <strong>empty-null</strong> (empty string), <strong>absent-null</strong> (reads of unset right-hand sides, or fall-through non-explicit return values from user-defined functions), <strong>error</strong>, <strong>string</strong>, <strong>float</strong> (double-precision), <strong>int</strong> (64-bit signed), and <strong>boolean</strong>.</p></li>
<li><p>On input, string values representable as numbers, e.g. “3” or “3.1”, are treated as int or float, respectively. If a record has <code class="docutils literal notranslate"><span class="pre">x=1,y=2</span></code> then <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">put</span> <span class="pre">'$z=$x+$y'</span></code> will produce <code class="docutils literal notranslate"><span class="pre">x=1,y=2,z=3</span></code>, and <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">put</span> <span class="pre">'$z=$x.$y'</span></code> does not give an error simply because the dot operator has been generalized to stringify non-strings. To coerce back to string for processing, use the <code class="docutils literal notranslate"><span class="pre">string</span></code> function: <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">put</span> <span class="pre">'$z=string($x).string($y)'</span></code> will produce <code class="docutils literal notranslate"><span class="pre">x=1,y=2,z=12</span></code>.</p></li>
<li><p>On input, string values representable as boolean (e.g. <code class="docutils literal notranslate"><span class="pre">&quot;true&quot;</span></code>, <code class="docutils literal notranslate"><span class="pre">&quot;false&quot;</span></code>) are <em>not</em> automatically treated as boolean. (This is because <code class="docutils literal notranslate"><span class="pre">&quot;true&quot;</span></code> and <code class="docutils literal notranslate"><span class="pre">&quot;false&quot;</span></code> are ordinary words, and auto string-to-boolean on a column consisting of words would result in some strings mixed with some booleans.) Use the <code class="docutils literal notranslate"><span class="pre">boolean</span></code> function to coerce: e.g. giving the record <code class="docutils literal notranslate"><span class="pre">x=1,y=2,w=false</span></code> to <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">put</span> <span class="pre">'$z=($x&lt;$y)</span> <span class="pre">||</span> <span class="pre">boolean($w)'</span></code>.</p></li>
<li><p>Functions take types as described in <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">--help-all-functions</span></code>: for example, <code class="docutils literal notranslate"><span class="pre">log10</span></code> takes float input and produces float output, <code class="docutils literal notranslate"><span class="pre">gmt2sec</span></code> maps string to int, and <code class="docutils literal notranslate"><span class="pre">sec2gmt</span></code> maps int to string.</p></li>
<li><p>All math functions described in <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">--help-all-functions</span></code> take integer as well as float input.</p></li>
</ul>
</div>
<div class="section" id="null-data-empty-and-absent">
<span id="reference-null-data"></span><h2>Null data: empty and absent<a class="headerlink" href="#null-data-empty-and-absent" title="Permalink to this headline"></a></h2>
<p>One of Miller’s key features is its support for <strong>heterogeneous</strong> data. For example, take <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">sort</span></code>: if you try to sort on field <code class="docutils literal notranslate"><span class="pre">hostname</span></code> when not all records in the data stream <em>have</em> a field named <code class="docutils literal notranslate"><span class="pre">hostname</span></code>, it is not an error (although you could pre-filter the data stream using <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">having-fields</span> <span class="pre">--at-least</span> <span class="pre">hostname</span> <span class="pre">then</span> <span class="pre">sort</span> <span class="pre">...</span></code>). Rather, records lacking one or more sort keys are simply output contiguously by <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">sort</span></code>.</p>
<p>Miller has two kinds of null data:</p>
<ul class="simple">
<li><p><strong>Empty (key present, value empty)</strong>: a field name is present in a record (or in an out-of-stream variable) with empty value: e.g. <code class="docutils literal notranslate"><span class="pre">x=,y=2</span></code> in the data input stream, or assignment <code class="docutils literal notranslate"><span class="pre">$x=&quot;&quot;</span></code> or <code class="docutils literal notranslate"><span class="pre">&#64;x=&quot;&quot;</span></code> in <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">put</span></code>.</p></li>
<li><p><strong>Absent (key not present)</strong>: a field name is not present, e.g. input record is <code class="docutils literal notranslate"><span class="pre">x=1,y=2</span></code> and a <code class="docutils literal notranslate"><span class="pre">put</span></code> or <code class="docutils literal notranslate"><span class="pre">filter</span></code> expression refers to <code class="docutils literal notranslate"><span class="pre">$z</span></code>. Or, reading an out-of-stream variable which hasn’t been assigned a value yet, e.g. <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">put</span> <span class="pre">-q</span> <span class="pre">'&#64;sum</span> <span class="pre">+=</span> <span class="pre">$x;</span> <span class="pre">end{emit</span> <span class="pre">&#64;sum}'</span></code> or <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">put</span> <span class="pre">-q</span> <span class="pre">'&#64;sum[$a][$b]</span> <span class="pre">+=</span> <span class="pre">$x;</span> <span class="pre">end{emit</span> <span class="pre">&#64;sum,</span> <span class="pre">&quot;a&quot;,</span> <span class="pre">&quot;b&quot;}'</span></code>.</p></li>
</ul>
<p>You can test these programatically using the functions <code class="docutils literal notranslate"><span class="pre">is_empty</span></code>/<code class="docutils literal notranslate"><span class="pre">is_not_empty</span></code>, <code class="docutils literal notranslate"><span class="pre">is_absent</span></code>/<code class="docutils literal notranslate"><span class="pre">is_present</span></code>, and <code class="docutils literal notranslate"><span class="pre">is_null</span></code>/<code class="docutils literal notranslate"><span class="pre">is_not_null</span></code>. For the last pair, note that null means either empty or absent.</p>
<p>Rules for null-handling:</p>
<ul class="simple">
<li><p>Records with one or more empty sort-field values sort after records with all sort-field values present:</p></li>
</ul>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ mlr cat data/sort-null.dat
a=3,b=2
a=1,b=8
a=,b=4
x=9,b=10
a=5,b=7
</pre></div>
</div>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ mlr sort -n a data/sort-null.dat
a=1,b=8
a=3,b=2
a=5,b=7
a=,b=4
x=9,b=10
</pre></div>
</div>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ mlr sort -nr a data/sort-null.dat
a=,b=4
a=5,b=7
a=3,b=2
a=1,b=8
x=9,b=10
</pre></div>
</div>
<ul class="simple">
<li><p>Functions/operators which have one or more <em>empty</em> arguments produce empty output: e.g.</p></li>
</ul>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ echo &#39;x=2,y=3&#39; | mlr put &#39;$a=$x+$y&#39;
x=2,y=3,a=5
</pre></div>
</div>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ echo &#39;x=,y=3&#39; | mlr put &#39;$a=$x+$y&#39;
x=,y=3,a=
</pre></div>
</div>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ echo &#39;x=,y=3&#39; | mlr put &#39;$a=log($x);$b=log($y)&#39;
x=,y=3,a=,b=1.098612
</pre></div>
</div>
<p>with the exception that the <code class="docutils literal notranslate"><span class="pre">min</span></code> and <code class="docutils literal notranslate"><span class="pre">max</span></code> functions are special: if one argument is non-null, it wins:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ echo &#39;x=,y=3&#39; | mlr put &#39;$a=min($x,$y);$b=max($x,$y)&#39;
x=,y=3,a=3,b=3
</pre></div>
</div>
<ul class="simple">
<li><p>Functions of <em>absent</em> variables (e.g. <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">put</span> <span class="pre">'$y</span> <span class="pre">=</span> <span class="pre">log10($nonesuch)'</span></code>) evaluate to absent, and arithmetic/bitwise/boolean operators with both operands being absent evaluate to absent. Arithmetic operators with one absent operand return the other operand. More specifically, absent values act like zero for addition/subtraction, and one for multiplication: Furthermore, <strong>any expression which evaluates to absent is not stored in the left-hand side of an assignment statement</strong>:</p></li>
</ul>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ echo &#39;x=2,y=3&#39; | mlr put &#39;$a=$u+$v; $b=$u+$y; $c=$x+$y&#39;
x=2,y=3,b=3,c=5
</pre></div>
</div>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ echo &#39;x=2,y=3&#39; | mlr put &#39;$a=min($x,$v);$b=max($u,$y);$c=min($u,$v)&#39;
x=2,y=3,a=2,b=3
</pre></div>
</div>
<ul class="simple">
<li><p>Likewise, for assignment to maps, <strong>absent-valued keys or values result in a skipped assignment</strong>.</p></li>
</ul>
<p>The reasoning is as follows:</p>
<ul class="simple">
<li><p>Empty values are explicit in the data so they should explicitly affect accumulations: <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">put</span> <span class="pre">'&#64;sum</span> <span class="pre">+=</span> <span class="pre">$x'</span></code> should accumulate numeric <code class="docutils literal notranslate"><span class="pre">x</span></code> values into the sum but an empty <code class="docutils literal notranslate"><span class="pre">x</span></code>, when encountered in the input data stream, should make the sum non-numeric. To work around this you can use the <code class="docutils literal notranslate"><span class="pre">is_not_null</span></code> function as follows: <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">put</span> <span class="pre">'is_not_null($x)</span> <span class="pre">{</span> <span class="pre">&#64;sum</span> <span class="pre">+=</span> <span class="pre">$x</span> <span class="pre">}'</span></code></p></li>
<li><p>Absent stream-record values should not break accumulations, since Miller by design handles heterogenous data: the running <code class="docutils literal notranslate"><span class="pre">&#64;sum</span></code> in <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">put</span> <span class="pre">'&#64;sum</span> <span class="pre">+=</span> <span class="pre">$x'</span></code> should not be invalidated for records which have no <code class="docutils literal notranslate"><span class="pre">x</span></code>.</p></li>
<li><p>Absent out-of-stream-variable values are precisely what allow you to write <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">put</span> <span class="pre">'&#64;sum</span> <span class="pre">+=</span> <span class="pre">$x'</span></code>. Otherwise you would have to write <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">put</span> <span class="pre">'begin{&#64;sum</span> <span class="pre">=</span> <span class="pre">0};</span> <span class="pre">&#64;sum</span> <span class="pre">+=</span> <span class="pre">$x'</span></code> – which is tolerable – but for <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">put</span> <span class="pre">'begin{...};</span> <span class="pre">&#64;sum[$a][$b]</span> <span class="pre">+=</span> <span class="pre">$x'</span></code> you’d have to pre-initialize <code class="docutils literal notranslate"><span class="pre">&#64;sum</span></code> for all values of <code class="docutils literal notranslate"><span class="pre">$a</span></code> and <code class="docutils literal notranslate"><span class="pre">$b</span></code> in your input data stream, which is intolerable.</p></li>
<li><p>The penalty for the absent feature is that misspelled variables can be hard to find: e.g. in <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">put</span> <span class="pre">'begin{&#64;sumx</span> <span class="pre">=</span> <span class="pre">10};</span> <span class="pre">...;</span> <span class="pre">update</span> <span class="pre">&#64;sumx</span> <span class="pre">somehow</span> <span class="pre">per-record;</span> <span class="pre">...;</span> <span class="pre">end</span> <span class="pre">{&#64;something</span> <span class="pre">=</span> <span class="pre">&#64;sum</span> <span class="pre">*</span> <span class="pre">2}'</span></code> the accumulator is spelt <code class="docutils literal notranslate"><span class="pre">&#64;sumx</span></code> in the begin-block but <code class="docutils literal notranslate"><span class="pre">&#64;sum</span></code> in the end-block, where since it is absent, <code class="docutils literal notranslate"><span class="pre">&#64;sum*2</span></code> evaluates to 2. See also the section on <a class="reference internal" href="reference-dsl.html#reference-dsl-errors-and-transparency"><span class="std std-ref">Errors and transparency</span></a>.</p></li>
</ul>
<p>Since absent plus absent is absent (and likewise for other operators), accumulations such as <code class="docutils literal notranslate"><span class="pre">&#64;sum</span> <span class="pre">+=</span> <span class="pre">$x</span></code> work correctly on heterogenous data, as do within-record formulas if both operands are absent. If one operand is present, you may get behavior you don’t desire. To work around this – namely, to set an output field only for records which have all the inputs present – you can use a pattern-action block with <code class="docutils literal notranslate"><span class="pre">is_present</span></code>:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ mlr cat data/het.dkvp
resource=/path/to/file,loadsec=0.45,ok=true
record_count=100,resource=/path/to/file
resource=/path/to/second/file,loadsec=0.32,ok=true
record_count=150,resource=/path/to/second/file
resource=/some/other/path,loadsec=0.97,ok=false
</pre></div>
</div>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ mlr put &#39;is_present($loadsec) { $loadmillis = $loadsec * 1000 }&#39; data/het.dkvp
resource=/path/to/file,loadsec=0.45,ok=true,loadmillis=450.000000
record_count=100,resource=/path/to/file
resource=/path/to/second/file,loadsec=0.32,ok=true,loadmillis=320.000000
record_count=150,resource=/path/to/second/file
resource=/some/other/path,loadsec=0.97,ok=false,loadmillis=970.000000
</pre></div>
</div>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ mlr put &#39;$loadmillis = (is_present($loadsec) ? $loadsec : 0.0) * 1000&#39; data/het.dkvp
resource=/path/to/file,loadsec=0.45,ok=true,loadmillis=450.000000
record_count=100,resource=/path/to/file,loadmillis=0.000000
resource=/path/to/second/file,loadsec=0.32,ok=true,loadmillis=320.000000
record_count=150,resource=/path/to/second/file,loadmillis=0.000000
resource=/some/other/path,loadsec=0.97,ok=false,loadmillis=970.000000
</pre></div>
</div>
<p>If you’re interested in a formal description of how empty and absent fields participate in arithmetic, here’s a table for plus (other arithmetic/boolean/bitwise operators are similar):</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ mlr --print-type-arithmetic-info
(+) | error absent empty string int float bool
------ + ------ ------ ------ ------ ------ ------ ------
error | error error error error error error error
absent | error absent absent error int float error
empty | error absent empty error empty empty error
string | error error error error error error error
int | error int empty error int float error
float | error float empty error float float error
bool | error error error error error error error
</pre></div>
</div>
</div>
<div class="section" id="string-literals">
<h2>String literals<a class="headerlink" href="#string-literals" title="Permalink to this headline"></a></h2>
<p>You can use the following backslash escapes for strings such as between the double quotes in contexts such as <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">filter</span> <span class="pre">'$name</span> <span class="pre">=~</span> <span class="pre">&quot;...&quot;'</span></code>, <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">put</span> <span class="pre">'$name</span> <span class="pre">=</span> <span class="pre">$othername</span> <span class="pre">.</span> <span class="pre">&quot;...&quot;'</span></code>, <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">put</span> <span class="pre">'$name</span> <span class="pre">=</span> <span class="pre">sub($name,</span> <span class="pre">&quot;...&quot;,</span> <span class="pre">&quot;...&quot;)</span></code>, etc.:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">\a</span></code>: ASCII code 0x07 (alarm/bell)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">\b</span></code>: ASCII code 0x08 (backspace)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">\f</span></code>: ASCII code 0x0c (formfeed)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">\n</span></code>: ASCII code 0x0a (LF/linefeed/newline)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">\r</span></code>: ASCII code 0x0d (CR/carriage return)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">\t</span></code>: ASCII code 0x09 (tab)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">\v</span></code>: ASCII code 0x0b (vertical tab)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">\\</span></code>: backslash</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">\&quot;</span></code>: double quote</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">\123</span></code>: Octal 123, etc. for <code class="docutils literal notranslate"><span class="pre">\000</span></code> up to <code class="docutils literal notranslate"><span class="pre">\377</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">\x7f</span></code>: Hexadecimal 7f, etc. for <code class="docutils literal notranslate"><span class="pre">\x00</span></code> up to <code class="docutils literal notranslate"><span class="pre">\xff</span></code></p></li>
</ul>
<p>See also <a class="reference external" href="https://en.wikipedia.org/wiki/Escape_sequences_in_C">https://en.wikipedia.org/wiki/Escape_sequences_in_C</a>.</p>
<p>These replacements apply only to strings you key in for the DSL expressions for <code class="docutils literal notranslate"><span class="pre">filter</span></code> and <code class="docutils literal notranslate"><span class="pre">put</span></code>: that is, if you type <code class="docutils literal notranslate"><span class="pre">\t</span></code> in a string literal for a <code class="docutils literal notranslate"><span class="pre">filter</span></code>/<code class="docutils literal notranslate"><span class="pre">put</span></code> expression, it will be turned into a tab character. If you want a backslash followed by a <code class="docutils literal notranslate"><span class="pre">t</span></code>, then please type <code class="docutils literal notranslate"><span class="pre">\\t</span></code>.</p>
<p>However, these replacements are not done automatically within your data stream. If you wish to make these replacements, you can do, for example, for a field named <code class="docutils literal notranslate"><span class="pre">field</span></code>, <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">put</span> <span class="pre">'$field</span> <span class="pre">=</span> <span class="pre">gsub($field,</span> <span class="pre">&quot;\\t&quot;,</span> <span class="pre">&quot;\t&quot;)'</span></code>. If you need to make such a replacement for all fields in your data, you should probably simply use the system <code class="docutils literal notranslate"><span class="pre">sed</span></code> command.</p>
</div>
<div class="section" id="regular-expressions">
<h2>Regular expressions<a class="headerlink" href="#regular-expressions" title="Permalink to this headline"></a></h2>
<p>Miller lets you use regular expressions (of type POSIX.2) in the following contexts:</p>
<ul class="simple">
<li><p>In <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">filter</span></code> with <code class="docutils literal notranslate"><span class="pre">=~</span></code> or <code class="docutils literal notranslate"><span class="pre">!=~</span></code>, e.g. <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">filter</span> <span class="pre">'$url</span> <span class="pre">=~</span> <span class="pre">&quot;http.*com&quot;'</span></code></p></li>
<li><p>In <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">put</span></code> with <code class="docutils literal notranslate"><span class="pre">sub</span></code> or <code class="docutils literal notranslate"><span class="pre">gsub</span></code>, e.g. <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">put</span> <span class="pre">'$url</span> <span class="pre">=</span> <span class="pre">sub($url,</span> <span class="pre">&quot;http.*com&quot;,</span> <span class="pre">&quot;&quot;)'</span></code></p></li>
<li><p>In <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">having-fields</span></code>, e.g. <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">having-fields</span> <span class="pre">--any-matching</span> <span class="pre">'^sda[0-9]'</span></code></p></li>
<li><p>In <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">cut</span></code>, e.g. <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">cut</span> <span class="pre">-r</span> <span class="pre">-f</span> <span class="pre">'^status$,^sda[0-9]'</span></code></p></li>
<li><p>In <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">rename</span></code>, e.g. <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">rename</span> <span class="pre">-r</span> <span class="pre">'^(sda[0-9]).*$,dev/\1'</span></code></p></li>
<li><p>In <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">grep</span></code>, e.g. <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">--csv</span> <span class="pre">grep</span> <span class="pre">00188555487</span> <span class="pre">myfiles*.csv</span></code></p></li>
</ul>
<p>Points demonstrated by the above examples:</p>
<ul class="simple">
<li><p>There are no implicit start-of-string or end-of-string anchors; please use <code class="docutils literal notranslate"><span class="pre">^</span></code> and/or <code class="docutils literal notranslate"><span class="pre">$</span></code> explicitly.</p></li>
<li><p>Miller regexes are wrapped with double quotes rather than slashes.</p></li>
<li><p>The <code class="docutils literal notranslate"><span class="pre">i</span></code> after the ending double quote indicates a case-insensitive regex.</p></li>
<li><p>Capture groups are wrapped with <code class="docutils literal notranslate"><span class="pre">(...)</span></code> rather than <code class="docutils literal notranslate"><span class="pre">\(...\)</span></code>; use <code class="docutils literal notranslate"><span class="pre">\(</span></code> and <code class="docutils literal notranslate"><span class="pre">\)</span></code> to match against parentheses.</p></li>
</ul>
<p>For <code class="docutils literal notranslate"><span class="pre">filter</span></code> and <code class="docutils literal notranslate"><span class="pre">put</span></code>, if the regular expression is a string literal (the normal case), it is precompiled at process start and reused thereafter, which is efficient. If the regular expression is a more complex expression, including string concatenation using <code class="docutils literal notranslate"><span class="pre">.</span></code>, or a column name (in which case you can take regular expressions from input data!), then regexes are compiled on each record which works but is less efficient. As well, in this case there is no way to specify case-insensitive matching.</p>
<p>Example:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ cat data/regex-in-data.dat
name=jane,regex=^j.*e$
name=bill,regex=^b[ou]ll$
name=bull,regex=^b[ou]ll$
</pre></div>
</div>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ mlr filter &#39;$name =~ $regex&#39; data/regex-in-data.dat
name=jane,regex=^j.*e$
name=bull,regex=^b[ou]ll$
</pre></div>
</div>
<div class="section" id="regex-captures">
<h3>Regex captures<a class="headerlink" href="#regex-captures" title="Permalink to this headline"></a></h3>
<p>Regex captures of the form <code class="docutils literal notranslate"><span class="pre">\0</span></code> through <code class="docutils literal notranslate"><span class="pre">\9</span></code> are supported as</p>
<ul class="simple">
<li><p>Captures have in-function context for <code class="docutils literal notranslate"><span class="pre">sub</span></code> and <code class="docutils literal notranslate"><span class="pre">gsub</span></code>. For example, the first <code class="docutils literal notranslate"><span class="pre">\1,\2</span></code> pair belong to the first <code class="docutils literal notranslate"><span class="pre">sub</span></code> and the second <code class="docutils literal notranslate"><span class="pre">\1,\2</span></code> pair belong to the second <code class="docutils literal notranslate"><span class="pre">sub</span></code>:</p></li>
</ul>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">mlr</span> <span class="n">put</span> <span class="s1">&#39;$b = sub($a, &quot;(..)_(...)&quot;, &quot;</span><span class="se">\2</span><span class="s1">-</span><span class="se">\1</span><span class="s1">&quot;); $c = sub($a, &quot;(..)_(.)(..)&quot;, &quot;:</span><span class="se">\1</span><span class="s1">:</span><span class="se">\2</span><span class="s1">:</span><span class="se">\3</span><span class="s1">&quot;)&#39;</span>
</pre></div>
</div>
<ul class="simple">
<li><p>Captures endure for the entirety of a <code class="docutils literal notranslate"><span class="pre">put</span></code> for the <code class="docutils literal notranslate"><span class="pre">=~</span></code> and <code class="docutils literal notranslate"><span class="pre">!=~</span></code> operators. For example, here the <code class="docutils literal notranslate"><span class="pre">\1,\2</span></code> are set by the <code class="docutils literal notranslate"><span class="pre">=~</span></code> operator and are used by both subsequent assignment statements:</p></li>
</ul>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">mlr</span> <span class="n">put</span> <span class="s1">&#39;$a =~ &quot;(..)_(....); $b = &quot;left_</span><span class="se">\1</span><span class="s1">&quot;; $c = &quot;right_</span><span class="se">\2</span><span class="s1">&quot;&#39;</span>
</pre></div>
</div>
<ul class="simple">
<li><p>The captures are not retained across multiple puts. For example, here the <code class="docutils literal notranslate"><span class="pre">\1,\2</span></code> won’t be expanded from the regex capture:</p></li>
</ul>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">mlr</span> <span class="n">put</span> <span class="s1">&#39;$a =~ &quot;(..)_(....)&#39;</span> <span class="n">then</span> <span class="p">{</span><span class="o">...</span> <span class="n">something</span> <span class="k">else</span> <span class="o">...</span><span class="p">}</span> <span class="n">then</span> <span class="n">put</span> <span class="s1">&#39;$b = &quot;left_</span><span class="se">\1</span><span class="s1">&quot;; $c = &quot;right_</span><span class="se">\2</span><span class="s1">&quot;&#39;</span>
</pre></div>
</div>
<ul class="simple">
<li><p>Captures are ignored in <code class="docutils literal notranslate"><span class="pre">filter</span></code> for the <code class="docutils literal notranslate"><span class="pre">=~</span></code> and <code class="docutils literal notranslate"><span class="pre">!=~</span></code> operators. For example, there is no mechanism provided to refer to the first <code class="docutils literal notranslate"><span class="pre">(..)</span></code> as <code class="docutils literal notranslate"><span class="pre">\1</span></code> or to the second <code class="docutils literal notranslate"><span class="pre">(....)</span></code> as <code class="docutils literal notranslate"><span class="pre">\2</span></code> in the following filter statement:</p></li>
</ul>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">mlr</span> <span class="nb">filter</span> <span class="s1">&#39;$a =~ &quot;(..)_(....)&#39;</span>
</pre></div>
</div>
<ul class="simple">
<li><p>Up to nine matches are supported: <code class="docutils literal notranslate"><span class="pre">\1</span></code> through <code class="docutils literal notranslate"><span class="pre">\9</span></code>, while <code class="docutils literal notranslate"><span class="pre">\0</span></code> is the entire match string; <code class="docutils literal notranslate"><span class="pre">\15</span></code> is treated as <code class="docutils literal notranslate"><span class="pre">\1</span></code> followed by an unrelated <code class="docutils literal notranslate"><span class="pre">5</span></code>.</p></li>
</ul>
</div>
</div>
<div class="section" id="arithmetic">
<h2>Arithmetic<a class="headerlink" href="#arithmetic" title="Permalink to this headline"></a></h2>
<div class="section" id="input-scanning">
<h3>Input scanning<a class="headerlink" href="#input-scanning" title="Permalink to this headline"></a></h3>
<p>Numbers in Miller are double-precision float or 64-bit signed integers. Anything scannable as int, e.g <code class="docutils literal notranslate"><span class="pre">123</span></code> or <code class="docutils literal notranslate"><span class="pre">0xabcd</span></code>, is treated as an integer; otherwise, input scannable as float (<code class="docutils literal notranslate"><span class="pre">4.56</span></code> or <code class="docutils literal notranslate"><span class="pre">8e9</span></code>) is treated as float; everything else is a string.</p>
<p>If you want all numbers to be treated as floats, then you may use <code class="docutils literal notranslate"><span class="pre">float()</span></code> in your filter/put expressions (e.g. replacing <code class="docutils literal notranslate"><span class="pre">$c</span> <span class="pre">=</span> <span class="pre">$a</span> <span class="pre">*</span> <span class="pre">$b</span></code> with <code class="docutils literal notranslate"><span class="pre">$c</span> <span class="pre">=</span> <span class="pre">float($a)</span> <span class="pre">*</span> <span class="pre">float($b)</span></code>) – or, more simply, use <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">filter</span> <span class="pre">-F</span></code> and <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">put</span> <span class="pre">-F</span></code> which forces all numeric input, whether from expression literals or field values, to float. Likewise <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">stats1</span> <span class="pre">-F</span></code> and <code class="docutils literal notranslate"><span class="pre">mlr</span> <span class="pre">step</span> <span class="pre">-F</span></code> force integerable accumulators (such as <code class="docutils literal notranslate"><span class="pre">count</span></code>) to be done in floating-point.</p>
</div>
<div class="section" id="conversion-by-math-routines">
<h3>Conversion by math routines<a class="headerlink" href="#conversion-by-math-routines" title="Permalink to this headline"></a></h3>
<p>For most math functions, integers are cast to float on input, and produce float output: e.g. <code class="docutils literal notranslate"><span class="pre">exp(0)</span> <span class="pre">=</span> <span class="pre">1.0</span></code> rather than <code class="docutils literal notranslate"><span class="pre">1</span></code>. The following, however, produce integer output if their inputs are integers: <code class="docutils literal notranslate"><span class="pre">+</span></code> <code class="docutils literal notranslate"><span class="pre">-</span></code> <code class="docutils literal notranslate"><span class="pre">*</span></code> <code class="docutils literal notranslate"><span class="pre">/</span></code> <code class="docutils literal notranslate"><span class="pre">//</span></code> <code class="docutils literal notranslate"><span class="pre">%</span></code> <code class="docutils literal notranslate"><span class="pre">abs</span></code> <code class="docutils literal notranslate"><span class="pre">ceil</span></code> <code class="docutils literal notranslate"><span class="pre">floor</span></code> <code class="docutils literal notranslate"><span class="pre">max</span></code> <code class="docutils literal notranslate"><span class="pre">min</span></code> <code class="docutils literal notranslate"><span class="pre">round</span></code> <code class="docutils literal notranslate"><span class="pre">roundm</span></code> <code class="docutils literal notranslate"><span class="pre">sgn</span></code>. As well, <code class="docutils literal notranslate"><span class="pre">stats1</span> <span class="pre">-a</span> <span class="pre">min</span></code>, <code class="docutils literal notranslate"><span class="pre">stats1</span> <span class="pre">-a</span> <span class="pre">max</span></code>, <code class="docutils literal notranslate"><span class="pre">stats1</span> <span class="pre">-a</span> <span class="pre">sum</span></code>, <code class="docutils literal notranslate"><span class="pre">step</span> <span class="pre">-a</span> <span class="pre">delta</span></code>, and <code class="docutils literal notranslate"><span class="pre">step</span> <span class="pre">-a</span> <span class="pre">rsum</span></code> produce integer output if their inputs are integers.</p>
</div>
<div class="section" id="conversion-by-arithmetic-operators">
<h3>Conversion by arithmetic operators<a class="headerlink" href="#conversion-by-arithmetic-operators" title="Permalink to this headline"></a></h3>
<p>The sum, difference, and product of integers is again integer, except for when that would overflow a 64-bit integer at which point Miller converts the result to float.</p>
<p>The short of it is that Miller does this transparently for you so you needn’t think about it.</p>
<p>Implementation details of this, for the interested: integer adds and subtracts overflow by at most one bit so it suffices to check sign-changes. Thus, Miller allows you to add and subtract arbitrary 64-bit signed integers, converting only to float precisely when the result is less than -2<sup>63</sup> or greater than 2<sup>63</sup>-1. Multiplies, on the other hand, can overflow by a word size and a sign-change technique does not suffice to detect overflow. Instead Miller tests whether the floating-point product exceeds the representable integer range. Now, 64-bit integers have 64-bit precision while IEEE-doubles have only 52-bit mantissas – so, there are 53 bits including implicit leading one. The following experiment explicitly demonstrates the resolution at this range:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="mi">64</span><span class="o">-</span><span class="n">bit</span> <span class="n">integer</span> <span class="mi">64</span><span class="o">-</span><span class="n">bit</span> <span class="n">integer</span> <span class="n">Casted</span> <span class="n">to</span> <span class="n">double</span> <span class="n">Back</span> <span class="n">to</span> <span class="mi">64</span><span class="o">-</span><span class="n">bit</span>
<span class="ow">in</span> <span class="nb">hex</span> <span class="ow">in</span> <span class="n">decimal</span> <span class="n">integer</span>
<span class="mh">0x7ffffffffffff9ff</span> <span class="mi">9223372036854774271</span> <span class="mf">9223372036854773760.000000</span> <span class="mh">0x7ffffffffffff800</span>
<span class="mh">0x7ffffffffffffa00</span> <span class="mi">9223372036854774272</span> <span class="mf">9223372036854773760.000000</span> <span class="mh">0x7ffffffffffff800</span>
<span class="mh">0x7ffffffffffffbff</span> <span class="mi">9223372036854774783</span> <span class="mf">9223372036854774784.000000</span> <span class="mh">0x7ffffffffffffc00</span>
<span class="mh">0x7ffffffffffffc00</span> <span class="mi">9223372036854774784</span> <span class="mf">9223372036854774784.000000</span> <span class="mh">0x7ffffffffffffc00</span>
<span class="mh">0x7ffffffffffffdff</span> <span class="mi">9223372036854775295</span> <span class="mf">9223372036854774784.000000</span> <span class="mh">0x7ffffffffffffc00</span>
<span class="mh">0x7ffffffffffffe00</span> <span class="mi">9223372036854775296</span> <span class="mf">9223372036854775808.000000</span> <span class="mh">0x8000000000000000</span>
<span class="mh">0x7ffffffffffffffe</span> <span class="mi">9223372036854775806</span> <span class="mf">9223372036854775808.000000</span> <span class="mh">0x8000000000000000</span>
<span class="mh">0x7fffffffffffffff</span> <span class="mi">9223372036854775807</span> <span class="mf">9223372036854775808.000000</span> <span class="mh">0x8000000000000000</span>
</pre></div>
</div>
<p>That is, one cannot check an integer product to see if it is precisely greater than 2<sup>63</sup>-1 or less than -2<sup>63</sup> using either integer arithmetic (it may have already overflowed) or using double-precision (due to granularity). Instead Miller checks for overflow in 64-bit integer multiplication by seeing whether the absolute value of the double-precision product exceeds the largest representable IEEE double less than 2<sup>63</sup>, which we see from the listing above is 9223372036854774784. (An alternative would be to do all integer multiplies using handcrafted multi-word 128-bit arithmetic. This approach is not taken.)</p>
</div>
<div class="section" id="pythonic-division">
<h3>Pythonic division<a class="headerlink" href="#pythonic-division" title="Permalink to this headline"></a></h3>
<p>Division and remainder are <a class="reference external" href="http://python-history.blogspot.com/2010/08/why-pythons-integer-division-floors.html">pythonic</a>:</p>
<ul class="simple">
<li><p>Quotient of integers is floating-point: <code class="docutils literal notranslate"><span class="pre">7/2</span></code> is <code class="docutils literal notranslate"><span class="pre">3.5</span></code>.</p></li>
<li><p>Integer division is done with <code class="docutils literal notranslate"><span class="pre">//</span></code>: <code class="docutils literal notranslate"><span class="pre">7//2</span></code> is <code class="docutils literal notranslate"><span class="pre">3</span></code>. This rounds toward the negative.</p></li>
<li><p>Remainders are non-negative.</p></li>
</ul>
</div>
</div>
<div class="section" id="on-line-help">
<h2>On-line help<a class="headerlink" href="#on-line-help" title="Permalink to this headline"></a></h2>
<p>Examples:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ mlr --help
Usage: mlr [I/O options] {verb} [verb-dependent options ...] {zero or more file names}
Command-line-syntax examples:
mlr --csv cut -f hostname,uptime mydata.csv
mlr --tsv --rs lf filter &#39;$status != &quot;down&quot; &amp;&amp; $upsec &gt;= 10000&#39; *.tsv
mlr --nidx put &#39;$sum = $7 &lt; 0.0 ? 3.5 : $7 + 2.1*$8&#39; *.dat
grep -v &#39;^#&#39; /etc/group | mlr --ifs : --nidx --opprint label group,pass,gid,member then sort -f group
mlr join -j account_id -f accounts.dat then group-by account_name balances.dat
mlr --json put &#39;$attr = sub($attr, &quot;([0-9]+)_([0-9]+)_.*&quot;, &quot;\1:\2&quot;)&#39; data/*.json
mlr stats1 -a min,mean,max,p10,p50,p90 -f flag,u,v data/*
mlr stats2 -a linreg-pca -f u,v -g shape data/*
mlr put -q &#39;@sum[$a][$b] += $x; end {emit @sum, &quot;a&quot;, &quot;b&quot;}&#39; data/*
mlr --from estimates.tbl put &#39;
for (k,v in $*) {
if (is_numeric(v) &amp;&amp; k =~ &quot;^[t-z].*$&quot;) {
$sum += v; $count += 1
}
}
$mean = $sum / $count # no assignment if count unset&#39;
mlr --from infile.dat put -f analyze.mlr
mlr --from infile.dat put &#39;tee &gt; &quot;./taps/data-&quot;.$a.&quot;-&quot;.$b, $*&#39;
mlr --from infile.dat put &#39;tee | &quot;gzip &gt; ./taps/data-&quot;.$a.&quot;-&quot;.$b.&quot;.gz&quot;, $*&#39;
mlr --from infile.dat put -q &#39;@v=$*; dump | &quot;jq .[]&quot;&#39;
mlr --from infile.dat put &#39;(NR % 1000 == 0) { print &gt; stderr, &quot;Checkpoint &quot;.NR}&#39;
Data-format examples:
DKVP: delimited key-value pairs (Miller default format)
+---------------------+
| apple=1,bat=2,cog=3 | Record 1: &quot;apple&quot; =&gt; &quot;1&quot;, &quot;bat&quot; =&gt; &quot;2&quot;, &quot;cog&quot; =&gt; &quot;3&quot;
| dish=7,egg=8,flint | Record 2: &quot;dish&quot; =&gt; &quot;7&quot;, &quot;egg&quot; =&gt; &quot;8&quot;, &quot;3&quot; =&gt; &quot;flint&quot;
+---------------------+
NIDX: implicitly numerically indexed (Unix-toolkit style)
+---------------------+
| the quick brown | Record 1: &quot;1&quot; =&gt; &quot;the&quot;, &quot;2&quot; =&gt; &quot;quick&quot;, &quot;3&quot; =&gt; &quot;brown&quot;
| fox jumped | Record 2: &quot;1&quot; =&gt; &quot;fox&quot;, &quot;2&quot; =&gt; &quot;jumped&quot;
+---------------------+
CSV/CSV-lite: comma-separated values with separate header line
+---------------------+
| apple,bat,cog |
| 1,2,3 | Record 1: &quot;apple =&gt; &quot;1&quot;, &quot;bat&quot; =&gt; &quot;2&quot;, &quot;cog&quot; =&gt; &quot;3&quot;
| 4,5,6 | Record 2: &quot;apple&quot; =&gt; &quot;4&quot;, &quot;bat&quot; =&gt; &quot;5&quot;, &quot;cog&quot; =&gt; &quot;6&quot;
+---------------------+
Tabular JSON: nested objects are supported, although arrays within them are not:
+---------------------+
| { |
| &quot;apple&quot;: 1, | Record 1: &quot;apple&quot; =&gt; &quot;1&quot;, &quot;bat&quot; =&gt; &quot;2&quot;, &quot;cog&quot; =&gt; &quot;3&quot;
| &quot;bat&quot;: 2, |
| &quot;cog&quot;: 3 |
| } |
| { |
| &quot;dish&quot;: { | Record 2: &quot;dish:egg&quot; =&gt; &quot;7&quot;, &quot;dish:flint&quot; =&gt; &quot;8&quot;, &quot;garlic&quot; =&gt; &quot;&quot;
| &quot;egg&quot;: 7, |
| &quot;flint&quot;: 8 |
| }, |
| &quot;garlic&quot;: &quot;&quot; |
| } |
+---------------------+
PPRINT: pretty-printed tabular
+---------------------+
| apple bat cog |
| 1 2 3 | Record 1: &quot;apple =&gt; &quot;1&quot;, &quot;bat&quot; =&gt; &quot;2&quot;, &quot;cog&quot; =&gt; &quot;3&quot;
| 4 5 6 | Record 2: &quot;apple&quot; =&gt; &quot;4&quot;, &quot;bat&quot; =&gt; &quot;5&quot;, &quot;cog&quot; =&gt; &quot;6&quot;
+---------------------+
XTAB: pretty-printed transposed tabular
+---------------------+
| apple 1 | Record 1: &quot;apple&quot; =&gt; &quot;1&quot;, &quot;bat&quot; =&gt; &quot;2&quot;, &quot;cog&quot; =&gt; &quot;3&quot;
| bat 2 |
| cog 3 |
| |
| dish 7 | Record 2: &quot;dish&quot; =&gt; &quot;7&quot;, &quot;egg&quot; =&gt; &quot;8&quot;
| egg 8 |
+---------------------+
Markdown tabular (supported for output only):
+-----------------------+
| | apple | bat | cog | |
| | --- | --- | --- | |
| | 1 | 2 | 3 | | Record 1: &quot;apple =&gt; &quot;1&quot;, &quot;bat&quot; =&gt; &quot;2&quot;, &quot;cog&quot; =&gt; &quot;3&quot;
| | 4 | 5 | 6 | | Record 2: &quot;apple&quot; =&gt; &quot;4&quot;, &quot;bat&quot; =&gt; &quot;5&quot;, &quot;cog&quot; =&gt; &quot;6&quot;
+-----------------------+
Help options:
-h or --help Show this message.
--version Show the software version.
{verb name} --help Show verb-specific help.
--help-all-verbs Show help on all verbs.
-l or --list-all-verbs List only verb names.
-L List only verb names, one per line.
-f or --help-all-functions Show help on all built-in functions.
-F Show a bare listing of built-in functions by name.
-k or --help-all-keywords Show help on all keywords.
-K Show a bare listing of keywords by name.
Customization via .mlrrc:
You can set up personal defaults via a $HOME/.mlrrc and/or ./.mlrrc.
For example, if you usually process CSV, then you can put &quot;--csv&quot; in your .mlrrc file
and that will be the default input/output format unless otherwise specified on the command line.
The .mlrrc file format is one &quot;--flag&quot; or &quot;--option value&quot; per line, with the leading &quot;--&quot; optional.
Hash-style comments and blank lines are ignored.
Sample .mlrrc:
# Input and output formats are CSV by default (unless otherwise specified
# on the mlr command line):
csv
# These are no-ops for CSV, but when I do use JSON output, I want these
# pretty-printing options to be used:
jvstack
jlistwrap
How to specify location of .mlrrc:
* If $MLRRC is set:
o If its value is &quot;__none__&quot; then no .mlrrc files are processed.
o Otherwise, its value (as a filename) is loaded and processed. If there are syntax
errors, they abort mlr with a usage message (as if you had mistyped something on the
command line). If the file can&#39;t be loaded at all, though, it is silently skipped.
o Any .mlrrc in your home directory or current directory is ignored whenever $MLRRC is
set in the environment.
* Otherwise:
o If $HOME/.mlrrc exists, it&#39;s then processed as above.
o If ./.mlrrc exists, it&#39;s then also processed as above.
(I.e. current-directory .mlrrc defaults are stacked over home-directory .mlrrc defaults.)
See also:
https://johnkerl.org/miller/doc/customization.html
Verbs:
altkv bar bootstrap cat check clean-whitespace count count-distinct
count-similar cut decimate fill-down fill-empty filter format-values
fraction grep group-by group-like having-fields head histogram join label
least-frequent merge-fields most-frequent nest nothing put regularize
remove-empty-columns rename reorder repeat reshape sample sec2gmt
sec2gmtdate seqgen shuffle skip-trivial-records sort sort-within-records
stats1 stats2 step tac tail tee top uniq unsparsify
Functions for the filter and put verbs:
+ + - - * / // .+ .+ .- .- .* ./ .// % ** | ^ &amp; ~ &lt;&lt; &gt;&gt; bitcount == != =~
!=~ &gt; &gt;= &lt; &lt;= &amp;&amp; || ^^ ! ? : . gsub regextract regextract_or_else strlen sub
ssub substr tolower toupper truncate capitalize lstrip rstrip strip
collapse_whitespace clean_whitespace system abs acos acosh asin asinh atan
atan2 atanh cbrt ceil cos cosh erf erfc exp expm1 floor invqnorm log log10
log1p logifit madd max mexp min mmul msub pow qnorm round roundm sgn sin
sinh sqrt tan tanh urand urandrange urand32 urandint dhms2fsec dhms2sec
fsec2dhms fsec2hms gmt2sec localtime2sec hms2fsec hms2sec sec2dhms sec2gmt
sec2gmt sec2gmtdate sec2localtime sec2localtime sec2localdate sec2hms
strftime strftime_local strptime strptime_local systime is_absent is_bool
is_boolean is_empty is_empty_map is_float is_int is_map is_nonempty_map
is_not_empty is_not_map is_not_null is_null is_numeric is_present is_string
asserting_absent asserting_bool asserting_boolean asserting_empty
asserting_empty_map asserting_float asserting_int asserting_map
asserting_nonempty_map asserting_not_empty asserting_not_map
asserting_not_null asserting_null asserting_numeric asserting_present
asserting_string boolean float fmtnum hexfmt int string typeof depth haskey
joink joinkv joinv leafcount length mapdiff mapexcept mapselect mapsum
splitkv splitkvx splitnv splitnvx
Please use &quot;mlr --help-function {function name}&quot; for function-specific help.
Data-format options, for input, output, or both:
--idkvp --odkvp --dkvp Delimited key-value pairs, e.g &quot;a=1,b=2&quot;
(this is Miller&#39;s default format).
--inidx --onidx --nidx Implicitly-integer-indexed fields
(Unix-toolkit style).
-T Synonymous with &quot;--nidx --fs tab&quot;.
--icsv --ocsv --csv Comma-separated value (or tab-separated
with --fs tab, etc.)
--itsv --otsv --tsv Keystroke-savers for &quot;--icsv --ifs tab&quot;,
&quot;--ocsv --ofs tab&quot;, &quot;--csv --fs tab&quot;.
--iasv --oasv --asv Similar but using ASCII FS 0x1f and RS 0x1e
--iusv --ousv --usv Similar but using Unicode FS U+241F (UTF-8 0xe2909f)
and RS U+241E (UTF-8 0xe2909e)
--icsvlite --ocsvlite --csvlite Comma-separated value (or tab-separated
with --fs tab, etc.). The &#39;lite&#39; CSV does not handle
RFC-CSV double-quoting rules; is slightly faster;
and handles heterogeneity in the input stream via
empty newline followed by new header line. See also
http://johnkerl.org/miller/doc/file-formats.html#CSV/TSV/etc.
--itsvlite --otsvlite --tsvlite Keystroke-savers for &quot;--icsvlite --ifs tab&quot;,
&quot;--ocsvlite --ofs tab&quot;, &quot;--csvlite --fs tab&quot;.
-t Synonymous with --tsvlite.
--iasvlite --oasvlite --asvlite Similar to --itsvlite et al. but using ASCII FS 0x1f and RS 0x1e
--iusvlite --ousvlite --usvlite Similar to --itsvlite et al. but using Unicode FS U+241F (UTF-8 0xe2909f)
and RS U+241E (UTF-8 0xe2909e)
--ipprint --opprint --pprint Pretty-printed tabular (produces no
output until all input is in).
--right Right-justifies all fields for PPRINT output.
--barred Prints a border around PPRINT output
(only available for output).
--omd Markdown-tabular (only available for output).
--ixtab --oxtab --xtab Pretty-printed vertical-tabular.
--xvright Right-justifies values for XTAB format.
--ijson --ojson --json JSON tabular: sequence or list of one-level
maps: {...}{...} or [{...},{...}].
--json-map-arrays-on-input JSON arrays are unmillerable. --json-map-arrays-on-input
--json-skip-arrays-on-input is the default: arrays are converted to integer-indexed
--json-fatal-arrays-on-input maps. The other two options cause them to be skipped, or
to be treated as errors. Please use the jq tool for full
JSON (pre)processing.
--jvstack Put one key-value pair per line for JSON
output.
--jsonx --ojsonx Keystroke-savers for --json --jvstack
--jsonx --ojsonx and --ojson --jvstack, respectively.
--jlistwrap Wrap JSON output in outermost [ ].
--jknquoteint Do not quote non-string map keys in JSON output.
--jvquoteall Quote map values in JSON output, even if they&#39;re
numeric.
--jflatsep {string} Separator for flattening multi-level JSON keys,
e.g. &#39;{&quot;a&quot;:{&quot;b&quot;:3}}&#39; becomes a:b =&gt; 3 for
non-JSON formats. Defaults to :.
-p is a keystroke-saver for --nidx --fs space --repifs
Examples: --csv for CSV-formatted input and output; --idkvp --opprint for
DKVP-formatted input and pretty-printed output.
Please use --iformat1 --oformat2 rather than --format1 --oformat2.
The latter sets up input and output flags for format1, not all of which
are overridden in all cases by setting output format to format2.
Comments in data:
--skip-comments Ignore commented lines (prefixed by &quot;#&quot;)
within the input.
--skip-comments-with {string} Ignore commented lines within input, with
specified prefix.
--pass-comments Immediately print commented lines (prefixed by &quot;#&quot;)
within the input.
--pass-comments-with {string} Immediately print commented lines within input, with
specified prefix.
Notes:
* Comments are only honored at the start of a line.
* In the absence of any of the above four options, comments are data like
any other text.
* When pass-comments is used, comment lines are written to standard output
immediately upon being read; they are not part of the record stream.
Results may be counterintuitive. A suggestion is to place comments at the
start of data files.
Format-conversion keystroke-saver options, for input, output, or both:
As keystroke-savers for format-conversion you may use the following:
--c2t --c2d --c2n --c2j --c2x --c2p --c2m
--t2c --t2d --t2n --t2j --t2x --t2p --t2m
--d2c --d2t --d2n --d2j --d2x --d2p --d2m
--n2c --n2t --n2d --n2j --n2x --n2p --n2m
--j2c --j2t --j2d --j2n --j2x --j2p --j2m
--x2c --x2t --x2d --x2n --x2j --x2p --x2m
--p2c --p2t --p2d --p2n --p2j --p2x --p2m
The letters c t d n j x p m refer to formats CSV, TSV, DKVP, NIDX, JSON, XTAB,
PPRINT, and markdown, respectively. Note that markdown format is available for
output only.
Compressed-data options:
--prepipe {command} This allows Miller to handle compressed inputs. You can do
without this for single input files, e.g. &quot;gunzip &lt; myfile.csv.gz | mlr ...&quot;.
However, when multiple input files are present, between-file separations are
lost; also, the FILENAME variable doesn&#39;t iterate. Using --prepipe you can
specify an action to be taken on each input file. This pre-pipe command must
be able to read from standard input; it will be invoked with
{command} &lt; {filename}.
Examples:
mlr --prepipe &#39;gunzip&#39;
mlr --prepipe &#39;zcat -cf&#39;
mlr --prepipe &#39;xz -cd&#39;
mlr --prepipe cat
mlr --prepipe-gunzip
mlr --prepipe-zcat
Note that this feature is quite general and is not limited to decompression
utilities. You can use it to apply per-file filters of your choice.
For output compression (or other) utilities, simply pipe the output:
mlr ... | {your compression command}
There are shorthands --prepipe-zcat and --prepipe-gunzip which are
valid in .mlrrc files. The --prepipe flag is not valid in .mlrrc
files since that would put execution of the prepipe command under
control of the .mlrrc file.
Separator options, for input, output, or both:
--rs --irs --ors Record separators, e.g. &#39;lf&#39; or &#39;\r\n&#39;
--fs --ifs --ofs --repifs Field separators, e.g. comma
--ps --ips --ops Pair separators, e.g. equals sign
Notes about line endings:
* Default line endings (--irs and --ors) are &quot;auto&quot; which means autodetect from
the input file format, as long as the input file(s) have lines ending in either
LF (also known as linefeed, &#39;\n&#39;, 0x0a, Unix-style) or CRLF (also known as
carriage-return/linefeed pairs, &#39;\r\n&#39;, 0x0d 0x0a, Windows style).
* If both irs and ors are auto (which is the default) then LF input will lead to LF
output and CRLF input will lead to CRLF output, regardless of the platform you&#39;re
running on.
* The line-ending autodetector triggers on the first line ending detected in the input
stream. E.g. if you specify a CRLF-terminated file on the command line followed by an
LF-terminated file then autodetected line endings will be CRLF.
* If you use --ors {something else} with (default or explicitly specified) --irs auto
then line endings are autodetected on input and set to what you specify on output.
* If you use --irs {something else} with (default or explicitly specified) --ors auto
then the output line endings used are LF on Unix/Linux/BSD/MacOSX, and CRLF on Windows.
Notes about all other separators:
* IPS/OPS are only used for DKVP and XTAB formats, since only in these formats
do key-value pairs appear juxtaposed.
* IRS/ORS are ignored for XTAB format. Nominally IFS and OFS are newlines;
XTAB records are separated by two or more consecutive IFS/OFS -- i.e.
a blank line. Everything above about --irs/--ors/--rs auto becomes --ifs/--ofs/--fs
auto for XTAB format. (XTAB&#39;s default IFS/OFS are &quot;auto&quot;.)
* OFS must be single-character for PPRINT format. This is because it is used
with repetition for alignment; multi-character separators would make
alignment impossible.
* OPS may be multi-character for XTAB format, in which case alignment is
disabled.
* TSV is simply CSV using tab as field separator (&quot;--fs tab&quot;).
* FS/PS are ignored for markdown format; RS is used.
* All FS and PS options are ignored for JSON format, since they are not relevant
to the JSON format.
* You can specify separators in any of the following ways, shown by example:
- Type them out, quoting as necessary for shell escapes, e.g.
&quot;--fs &#39;|&#39; --ips :&quot;
- C-style escape sequences, e.g. &quot;--rs &#39;\r\n&#39; --fs &#39;\t&#39;&quot;.
- To avoid backslashing, you can use any of the following names:
cr crcr newline lf lflf crlf crlfcrlf tab space comma pipe slash colon semicolon equals
* Default separators by format:
File format RS FS PS
gen N/A (N/A) (N/A)
dkvp auto , =
json auto (N/A) (N/A)
nidx auto space (N/A)
csv auto , (N/A)
csvlite auto , (N/A)
markdown auto (N/A) (N/A)
pprint auto space (N/A)
xtab (N/A) auto space
Relevant to CSV/CSV-lite input only:
--implicit-csv-header Use 1,2,3,... as field labels, rather than from line 1
of input files. Tip: combine with &quot;label&quot; to recreate
missing headers.
--no-implicit-csv-header Do not use --implicit-csv-header. This is the default
anyway -- the main use is for the flags to &#39;mlr join&#39; if you have
main file(s) which are headerless but you want to join in on
a file which does have a CSV header. Then you could use
&#39;mlr --csv --implicit-csv-header join --no-implicit-csv-header
-l your-join-in-with-header.csv ... your-headerless.csv&#39;
--allow-ragged-csv-input|--ragged If a data line has fewer fields than the header line,
fill remaining keys with empty string. If a data line has more
fields than the header line, use integer field labels as in
the implicit-header case.
--headerless-csv-output Print only CSV data lines.
-N Keystroke-saver for --implicit-csv-header --headerless-csv-output.
Double-quoting for CSV output:
--quote-all Wrap all fields in double quotes
--quote-none Do not wrap any fields in double quotes, even if they have
OFS or ORS in them
--quote-minimal Wrap fields in double quotes only if they have OFS or ORS
in them (default)
--quote-numeric Wrap fields in double quotes only if they have numbers
in them
--quote-original Wrap fields in double quotes if and only if they were
quoted on input. This isn&#39;t sticky for computed fields:
e.g. if fields a and b were quoted on input and you do
&quot;put &#39;$c = $a . $b&#39;&quot; then field c won&#39;t inherit a or b&#39;s
was-quoted-on-input flag.
Numerical formatting:
--ofmt {format} E.g. %.18lf, %.0lf. Please use sprintf-style codes for
double-precision. Applies to verbs which compute new
values, e.g. put, stats1, stats2. See also the fmtnum
function within mlr put (mlr --help-all-functions).
Defaults to %lf.
Other options:
--seed {n} with n of the form 12345678 or 0xcafefeed. For put/filter
urand()/urandint()/urand32().
--nr-progress-mod {m}, with m a positive integer: print filename and record
count to stderr every m input records.
--from {filename} Use this to specify an input file before the verb(s),
rather than after. May be used more than once. Example:
&quot;mlr --from a.dat --from b.dat cat&quot; is the same as
&quot;mlr cat a.dat b.dat&quot;.
-n Process no input files, nor standard input either. Useful
for mlr put with begin/end statements only. (Same as --from
/dev/null.) Also useful in &quot;mlr -n put -v &#39;...&#39;&quot; for
analyzing abstract syntax trees (if that&#39;s your thing).
-I Process files in-place. For each file name on the command
line, output is written to a temp file in the same
directory, which is then renamed over the original. Each
file is processed in isolation: if the output format is
CSV, CSV headers will be present in each output file;
statistics are only over each file&#39;s own records; and so on.
Then-chaining:
Output of one verb may be chained as input to another using &quot;then&quot;, e.g.
mlr stats1 -a min,mean,max -f flag,u,v -g color then sort -f color
Auxiliary commands:
Miller has a few otherwise-standalone executables packaged within it.
They do not participate in any other parts of Miller.
Available subcommands:
aux-list
lecat
termcvt
hex
unhex
netbsd-strptime
For more information, please invoke mlr {subcommand} --help
For more information please see http://johnkerl.org/miller/doc and/or
http://github.com/johnkerl/miller. This is Miller version v5.10.2-dev.
</pre></div>
</div>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ mlr sort --help
Usage: mlr sort {flags}
Flags:
-f {comma-separated field names} Lexical ascending
-n {comma-separated field names} Numerical ascending; nulls sort last
-nf {comma-separated field names} Same as -n
-r {comma-separated field names} Lexical descending
-nr {comma-separated field names} Numerical descending; nulls sort first
Sorts records primarily by the first specified field, secondarily by the second
field, and so on. (Any records not having all specified sort keys will appear
at the end of the output, in the order they were encountered, regardless of the
specified sort order.) The sort is stable: records that compare equal will sort
in the order they were encountered in the input record stream.
Example:
mlr sort -f a,b -nr x,y,z
which is the same as:
mlr sort -f a -f b -nr x -nr y -nr z
</pre></div>
</div>
</div>
</div>
<div class="clearer"></div>
</div>
</div>
</div>
<div class="sphinxsidebar" role="navigation" aria-label="main navigation">
<div class="sphinxsidebarwrapper">
<h3><a href="index.html">Table of Contents</a></h3>
<ul>
<li><a class="reference internal" href="#">Main reference</a><ul>
<li><a class="reference internal" href="#command-overview">Command overview</a></li>
<li><a class="reference internal" href="#i-o-options">I/O options</a><ul>
<li><a class="reference internal" href="#formats">Formats</a></li>
<li><a class="reference internal" href="#in-place-mode">In-place mode</a></li>
<li><a class="reference internal" href="#compression">Compression</a></li>
<li><a class="reference internal" href="#record-field-pair-separators">Record/field/pair separators</a></li>
<li><a class="reference internal" href="#number-formatting">Number formatting</a></li>
</ul>
</li>
<li><a class="reference internal" href="#data-transformations-verbs">Data transformations (verbs)</a></li>
<li><a class="reference internal" href="#expression-language-for-filter-and-put">Expression language for filter and put</a></li>
<li><a class="reference internal" href="#then-chaining">then-chaining</a></li>
<li><a class="reference internal" href="#auxiliary-commands">Auxiliary commands</a></li>
<li><a class="reference internal" href="#data-types">Data types</a></li>
<li><a class="reference internal" href="#null-data-empty-and-absent">Null data: empty and absent</a></li>
<li><a class="reference internal" href="#string-literals">String literals</a></li>
<li><a class="reference internal" href="#regular-expressions">Regular expressions</a><ul>
<li><a class="reference internal" href="#regex-captures">Regex captures</a></li>
</ul>
</li>
<li><a class="reference internal" href="#arithmetic">Arithmetic</a><ul>
<li><a class="reference internal" href="#input-scanning">Input scanning</a></li>
<li><a class="reference internal" href="#conversion-by-math-routines">Conversion by math routines</a></li>
<li><a class="reference internal" href="#conversion-by-arithmetic-operators">Conversion by arithmetic operators</a></li>
<li><a class="reference internal" href="#pythonic-division">Pythonic division</a></li>
</ul>
</li>
<li><a class="reference internal" href="#on-line-help">On-line help</a></li>
</ul>
</li>
</ul>
<h4>Previous topic</h4>
<p class="topless"><a href="data-sharing.html"
title="previous chapter">Mixing with other languages</a></p>
<h4>Next topic</h4>
<p class="topless"><a href="reference-verbs.html"
title="next chapter">Verbs reference</a></p>
<div role="note" aria-label="source link">
<h3>This Page</h3>
<ul class="this-page-menu">
<li><a href="_sources/reference.rst.txt"
rel="nofollow">Show Source</a></li>
</ul>
</div>
<div id="searchbox" style="display: none" role="search">
<h3 id="searchlabel">Quick search</h3>
<div class="searchformwrapper">
<form class="search" action="search.html" method="get">
<input type="text" name="q" aria-labelledby="searchlabel" />
<input type="submit" value="Go" />
</form>
</div>
</div>
<script>$('#searchbox').show(0);</script>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="genindex.html" title="General Index"
>index</a></li>
<li class="right" >
<a href="reference-verbs.html" title="Verbs reference"
>next</a> |</li>
<li class="right" >
<a href="data-sharing.html" title="Mixing with other languages"
>previous</a> |</li>
<li class="nav-item nav-item-0"><a href="index.html">Miller 5.10.2 documentation</a> &#187;</li>
<li class="nav-item nav-item-this"><a href="">Main reference</a></li>
</ul>
</div>
<div class="footer" role="contentinfo">
&#169; Copyright 2020, John Kerl.
Created using <a href="https://www.sphinx-doc.org/">Sphinx</a> 3.2.1.
</div>
</body>
</html>
Reference in a new issue View git blame Copy permalink