mirror of
https://github.com/johnkerl/miller.git
synced 2026-08-03 21:13:09 +00:00
reference-doc reorg
This commit is contained in:
John Kerl
parent
76e7c3bcc8
commit
0614117131
5 changed files with 515 additions and 451 deletions
|
|
@ -9,6 +9,7 @@ BUGFIXES
|
|||
TOP-OF-LIST SUMMARY
|
||||
|
||||
! entirely rework null-data section @ mld; manify?/mldify! mlr --print-type-arithmetic-info; typeof
|
||||
-> merge absent & uninit ?!?
|
||||
! valgrinds
|
||||
! do absent-identity for logical, bitwise, and exp? UTs for all regardless of what changes/doesn't.
|
||||
* doc warn: the same implicit-identity feature that makes oosvars usable (emph in nested maps) makes it confusing for
|
||||
|
|
|
|||
|
|
@ -83,85 +83,6 @@ POKI_RUN_COMMAND{{mlr --help}}HERE
|
|||
|
||||
POKI_RUN_COMMAND{{mlr sort --help}}HERE
|
||||
|
||||
<!-- ================================================================ -->
|
||||
<h1>Data types</h1>
|
||||
|
||||
<p/> Miller’s input and output are all string-oriented: there is (as of
|
||||
August 2015 anyway) no support for binary record packing. In this sense,
|
||||
everything is a string in and out of Miller. During processing, field names
|
||||
are always strings, even if they have names like "3"; field values are usually
|
||||
strings. Field values’ ability to be interpreted as a non-string type
|
||||
only has meaning when comparison or function operations are done on them. And
|
||||
it is an error condition if Miller encounters non-numeric (or otherwise
|
||||
mistyped) data in a field in which it has been asked to do numeric (or
|
||||
otherwise type-specific) operations.
|
||||
|
||||
<p/> Field values are treated as numeric for the following:
|
||||
<ul>
|
||||
<li/> Numeric sort: <tt>mlr sort -n</tt>, <tt>mlr sort -nr</tt>.
|
||||
<li/> Statistics: <tt>mlr histogram</tt>, <tt>mlr stats1</tt>, <tt>mlr stats2</tt>.
|
||||
<li/> Cross-record arithmetic: <tt>mlr step</tt>.
|
||||
</ul>
|
||||
|
||||
<p/>For <tt>mlr put</tt> and <tt>mlr filter</tt>:
|
||||
|
||||
<ul>
|
||||
|
||||
<li/> Miller’s types for function processing are <b>null</b> (empty
|
||||
string), <b>error</b>, <b>string</b>, <b>float</b> (double-precision),
|
||||
<b>int</b> (64-bit signed), and <b>boolean</b>.
|
||||
|
||||
<li/> On input, string values representable as numbers, e.g. "3" or "3.1", are
|
||||
treated as int or float, respectively. If a record has <tt>x=1,y=2</tt> then
|
||||
<tt>mlr put '$z=$x+$y'</tt> will produce <tt>x=1,y=2,z=3</tt>, and <tt>mlr put
|
||||
'$z=$x.$y'</tt> gives an error. To coerce back to string for processing, use
|
||||
the <tt>string</tt> function: <tt>mlr put '$z=string($x).string($y)'</tt> will
|
||||
produce <tt>x=1,y=2,z=12</tt>.
|
||||
|
||||
<li/> On input, string values representable as boolean (e.g. <tt>"true"</tt>,
|
||||
<tt>"false"</tt>) are <i>not</i> automatically treated as boolean. (This is
|
||||
because <tt>"true"</tt> and <tt>"false"</tt> are ordinary words, and auto
|
||||
string-to-boolean on a column consisting of words would result in some strings
|
||||
mixed with some booleans.) Use the <tt>boolean</tt> function to coerce: e.g.
|
||||
giving the record <tt>x=1,y=2,w=false</tt> to <tt>mlr put '$z=($x<$y) ||
|
||||
boolean($w)'</tt>.
|
||||
|
||||
<li/> Functions take types as described in <tt>mlr --help-all-functions</tt>:
|
||||
for example, <tt>log10</tt> takes float input and produces float output,
|
||||
<tt>gmt2sec</tt> maps string to int, and <tt>sec2gmt</tt> maps int to string.
|
||||
|
||||
<li/> All math functions described in <tt>mlr --help-all-functions</tt> take
|
||||
integer as well as float input.
|
||||
|
||||
</ul>
|
||||
|
||||
<!-- ================================================================ -->
|
||||
<h1>Null data</h1>
|
||||
|
||||
<p/> One of Miller’s key features is its support for <b>heterogeneous</b> data.
|
||||
Accordingly, if you try to sort on field <tt>hostname</tt> when not all records in the data
|
||||
stream <i>have</i> a field named <tt>hostname</tt>, it is not an error (although you could
|
||||
pre-filter the data stream using <tt>mlr having-fields --at-least hostname then sort ...</tt>).
|
||||
Rather, records lacking one or more sort keys are simply output contiguously by <tt>mlr sort</tt>.
|
||||
|
||||
<p/> Field values may also be null by being
|
||||
specified with present key but empty value: e.g. sending <tt>x=,y=2</tt> to <tt>mlr put '$z=$x+$y'</tt>.
|
||||
|
||||
<p/>
|
||||
Rules for null-handling:
|
||||
<ul>
|
||||
<li> Records with one or more null sort-field values sort after records with all sort-field values present:
|
||||
POKI_RUN_COMMAND{{mlr --opprint cat data/sort-null.dat}}HERE
|
||||
POKI_RUN_COMMAND{{mlr --opprint sort -n a data/sort-null.dat}}HERE
|
||||
POKI_RUN_COMMAND{{mlr --opprint sort -nr a data/sort-null.dat}}HERE
|
||||
<li> Functions which have one or more null arguments produce null output: e.g.
|
||||
POKI_RUN_COMMAND{{echo 'x=2,y=3' | mlr put '$a=$x+$y'}}HERE
|
||||
POKI_RUN_COMMAND{{echo 'x=,y=3' | mlr put '$a=$x+$y'}}HERE
|
||||
POKI_RUN_COMMAND{{echo 'x=,y=3' | mlr put '$a=log($x);$b=log($y)'}}HERE
|
||||
<li> The <tt>min</tt> and <tt>max</tt> functions are special: if one argument is non-null, it wins:
|
||||
POKI_RUN_COMMAND{{echo 'x=,y=3' | mlr put '$a=min($x,$y);$b=max($x,$y)'}}HERE
|
||||
</ul>
|
||||
|
||||
<!-- ================================================================ -->
|
||||
<h1>I/O options</h1>
|
||||
|
||||
|
|
@ -1294,146 +1215,114 @@ pipeline stage.
|
|||
POKI_RUN_COMMAND{{mlr --help-all-functions}}HERE
|
||||
|
||||
<!-- ================================================================ -->
|
||||
<h1>Operator precedence</h1>
|
||||
<h1>Data types</h1>
|
||||
|
||||
<p/>Operators are listed in order of decreasing precedence, highest first.
|
||||
<p/> Miller’s input and output are all string-oriented: there is (as of
|
||||
August 2015 anyway) no support for binary record packing. In this sense,
|
||||
everything is a string in and out of Miller. During processing, field names
|
||||
are always strings, even if they have names like "3"; field values are usually
|
||||
strings. Field values’ ability to be interpreted as a non-string type
|
||||
only has meaning when comparison or function operations are done on them. And
|
||||
it is an error condition if Miller encounters non-numeric (or otherwise
|
||||
mistyped) data in a field in which it has been asked to do numeric (or
|
||||
otherwise type-specific) operations.
|
||||
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
<pre>
|
||||
Operators Associativity
|
||||
--------- -------------
|
||||
() left to right
|
||||
** right to left
|
||||
! ~ unary+ unary- & right to left
|
||||
binary* / // % left to right
|
||||
binary+ binary- . left to right
|
||||
<< >> left to right
|
||||
& left to right
|
||||
^ left to right
|
||||
| left to right
|
||||
< <= > >= left to right
|
||||
== != =~ !=~ left to right
|
||||
&& left to right
|
||||
^^ left to right
|
||||
|| left to right
|
||||
? : right to left
|
||||
= N/A for Miller (there is no $a=$b=$c)
|
||||
</pre>
|
||||
</div>
|
||||
<p/>
|
||||
<p/> Field values are treated as numeric for the following:
|
||||
<ul>
|
||||
<li/> Numeric sort: <tt>mlr sort -n</tt>, <tt>mlr sort -nr</tt>.
|
||||
<li/> Statistics: <tt>mlr histogram</tt>, <tt>mlr stats1</tt>, <tt>mlr stats2</tt>.
|
||||
<li/> Cross-record arithmetic: <tt>mlr step</tt>.
|
||||
</ul>
|
||||
|
||||
<!-- ================================================================ -->
|
||||
<h1>Operator and function semantics</h1>
|
||||
<p/>For <tt>mlr put</tt> and <tt>mlr filter</tt>:
|
||||
|
||||
<ul>
|
||||
|
||||
<li/> Functions are in general pass-throughs straight to the system-standard C
|
||||
library.
|
||||
<li/> Miller’s types for function processing are <b>null</b> (empty
|
||||
string), <b>error</b>, <b>string</b>, <b>float</b> (double-precision),
|
||||
<b>int</b> (64-bit signed), and <b>boolean</b>.
|
||||
|
||||
<li/> The <tt>min</tt> and <tt>max</tt> functions are different from other
|
||||
multi-argument functions which return null if any of their inputs are null: for
|
||||
<tt>min</tt> and <tt>max</tt>, by contrast, if one argument is null, the other
|
||||
is returned.
|
||||
<li/> On input, string values representable as numbers, e.g. "3" or "3.1", are
|
||||
treated as int or float, respectively. If a record has <tt>x=1,y=2</tt> then
|
||||
<tt>mlr put '$z=$x+$y'</tt> will produce <tt>x=1,y=2,z=3</tt>, and <tt>mlr put
|
||||
'$z=$x.$y'</tt> gives an error. To coerce back to string for processing, use
|
||||
the <tt>string</tt> function: <tt>mlr put '$z=string($x).string($y)'</tt> will
|
||||
produce <tt>x=1,y=2,z=12</tt>.
|
||||
|
||||
<li/> Symmetrically with respect to the bitwise OR, XOR, and AND operators
|
||||
<tt>|</tt>, <tt>^</tt>, <tt>&</tt>, Miller has logical operators
|
||||
<tt>||</tt>, <tt>^^</tt>, <tt>&&</tt>: the logical XOR not existing in
|
||||
C.
|
||||
<li/> On input, string values representable as boolean (e.g. <tt>"true"</tt>,
|
||||
<tt>"false"</tt>) are <i>not</i> automatically treated as boolean. (This is
|
||||
because <tt>"true"</tt> and <tt>"false"</tt> are ordinary words, and auto
|
||||
string-to-boolean on a column consisting of words would result in some strings
|
||||
mixed with some booleans.) Use the <tt>boolean</tt> function to coerce: e.g.
|
||||
giving the record <tt>x=1,y=2,w=false</tt> to <tt>mlr put '$z=($x<$y) ||
|
||||
boolean($w)'</tt>.
|
||||
|
||||
<li/> The exponentiation operator <tt>**</tt> is familiar from many languages.
|
||||
<li/> Functions take types as described in <tt>mlr --help-all-functions</tt>:
|
||||
for example, <tt>log10</tt> takes float input and produces float output,
|
||||
<tt>gmt2sec</tt> maps string to int, and <tt>sec2gmt</tt> maps int to string.
|
||||
|
||||
<li/> The regex-match and regex-not-match operators <tt>=~</tt> and
|
||||
<tt>!=~</tt> are similar to those in Ruby and Perl.
|
||||
<li/> All math functions described in <tt>mlr --help-all-functions</tt> take
|
||||
integer as well as float input.
|
||||
|
||||
</ul>
|
||||
|
||||
<!-- ================================================================ -->
|
||||
<h1>Arithmetic</h1>
|
||||
<h1>Null data</h1>
|
||||
|
||||
<h2>Input scanning</h2>
|
||||
<p/> One of Miller’s key features is its support for <b>heterogeneous</b> data.
|
||||
Accordingly, if you try to sort on field <tt>hostname</tt> when not all records in the data
|
||||
stream <i>have</i> a field named <tt>hostname</tt>, it is not an error (although you could
|
||||
pre-filter the data stream using <tt>mlr having-fields --at-least hostname then sort ...</tt>).
|
||||
Rather, records lacking one or more sort keys are simply output contiguously by <tt>mlr sort</tt>.
|
||||
|
||||
<p/>Numbers in Miller are double-precision float or 64-bit signed integers.
|
||||
Anything scannable as int, e.g <tt>123</tt> or <tt>0xabcd</tt>, is treated as
|
||||
an integer; otherwise, input scannable as float (<tt>4.56</tt> or <tt>8e9</tt>)
|
||||
is treated as float; everything else is a string.
|
||||
<p/> Miller has three kinds of null data:
|
||||
|
||||
<p/>If you want all numbers to be treated as floats, then you may use
|
||||
<tt>float()</tt> in your filter/put expressions (e.g. replacing <tt>$c = $a *
|
||||
$b</tt> with <tt>$c = float($a) * float($b)</tt>) — or, more simply, use
|
||||
<tt>mlr filter -F</tt> and <tt>mlr put -F</tt> which forces all numeric input,
|
||||
whether from expression literals or field values, to float. Likewise <tt>mlr
|
||||
stats1 -F</tt> and <tt>mlr step -F</tt> force integerable accumulators (such as
|
||||
<tt>count</tt>) to be done in floating-point.
|
||||
<ul>
|
||||
|
||||
<h2>Conversion by math routines</h2>
|
||||
<li/> <b>Void</b>: a field name is present in a record (or in an out-of-stream
|
||||
variable) with empty value: e.g. <tt>x=,y=2</tt> in the data input stream, or
|
||||
assignment <tt>$x=""</tt> or <tt>@x=""</tt> in <tt>mlr put</tt>.
|
||||
|
||||
<p/>For most math functions, integers are cast to float on input, and produce
|
||||
float output: e.g. <tt>exp(0) = 1.0</tt> rather than <tt>1</tt>. The
|
||||
following, however, produce integer output if their inputs are integers:
|
||||
<tt>+</tt> <tt>-</tt> <tt>*</tt> <tt>/</tt> <tt>//</tt> <tt>%</tt> <tt>abs</tt>
|
||||
<tt>ceil</tt> <tt>floor</tt> <tt>max</tt> <tt>min</tt> <tt>round</tt>
|
||||
<tt>roundm</tt> <tt>sgn</tt>. As well, <tt>stats1 -a min</tt>, <tt>stats1 -a
|
||||
max</tt>, <tt>stats1 -a sum</tt>, <tt>step -a delta</tt>, and <tt>step -a
|
||||
rsum</tt> produce integer output if their inputs are integers.
|
||||
<li/> <b>Absent</b>: a field name is not present, e.g. input record is
|
||||
<tt>x=1,y=2</tt> and a <tt>put</tt> or <tt>filter</tt> expression refers to
|
||||
<tt>$z</tt>.
|
||||
|
||||
<h2>Conversion by arithmetic operators</h2>
|
||||
<li/> <b>Uninit</b>: an out-of-stream variable is read before being assigned
|
||||
to, e.g. <tt>mlr put -q '@sum += $x'; end{emit @sum}'</tt> or <tt>mlr put -q
|
||||
'@sum[$a][$b] += $x'; end{emit @sum, "a", "b"}'</tt>. Without uninit, for the
|
||||
former you’d need to type <tt>mlr put -q 'begin{@sum = 0}; @sum += $x';
|
||||
end{emit @sum}'</tt> (which is tolerable), but for the latter you’d need
|
||||
to know all values of <tt>$a</tt> and <tt>$b</tt> before reading your input
|
||||
data, which would be intolerable.
|
||||
|
||||
<p/>The sum, difference, and product of integers is again integer, except for
|
||||
when that would overflow a 64-bit integer at which point Miller converts the
|
||||
result to float.
|
||||
</ul>
|
||||
|
||||
<p/>The short of it is that Miller does this transparently for you so you
|
||||
needn’t think about it.
|
||||
<!--
|
||||
<p/>xxx more examples.
|
||||
|
||||
<p/>Implementation details of this, for the interested: integer adds and
|
||||
subtracts overflow by at most one bit so it suffices to check sign-changes.
|
||||
Thus, Miller allows you to add and subtract arbitrary 64-bit signed integers,
|
||||
converting only to float precisely when the result is less than -2<sup>63</sup>
|
||||
or greater than 2<sup>63</sup>-1. Multiplies, on the other hand, can overflow
|
||||
by a word size and a sign-change technique does not suffice to detect overflow.
|
||||
Instead Miller tests whether the floating-point product exceeds the
|
||||
representable integer range. Now, 64-bit integers have 64-bit precision while
|
||||
IEEE-doubles have only 52-bit mantissas — so, there are 53 bits including
|
||||
implicit leading one. The following experiment explicitly demonstrates the
|
||||
resolution at this range:
|
||||
<p/>xxx keep typing up from here.
|
||||
-->
|
||||
|
||||
<div class=pokipanel>
|
||||
<pre>
|
||||
64-bit integer 64-bit integer Casted to double Back to 64-bit
|
||||
in hex in decimal integer
|
||||
0x7ffffffffffff9ff 9223372036854774271 9223372036854773760.000000 0x7ffffffffffff800
|
||||
0x7ffffffffffffa00 9223372036854774272 9223372036854773760.000000 0x7ffffffffffff800
|
||||
0x7ffffffffffffbff 9223372036854774783 9223372036854774784.000000 0x7ffffffffffffc00
|
||||
0x7ffffffffffffc00 9223372036854774784 9223372036854774784.000000 0x7ffffffffffffc00
|
||||
0x7ffffffffffffdff 9223372036854775295 9223372036854774784.000000 0x7ffffffffffffc00
|
||||
0x7ffffffffffffe00 9223372036854775296 9223372036854775808.000000 0x8000000000000000
|
||||
0x7ffffffffffffffe 9223372036854775806 9223372036854775808.000000 0x8000000000000000
|
||||
0x7fffffffffffffff 9223372036854775807 9223372036854775808.000000 0x8000000000000000
|
||||
</pre>
|
||||
</div>
|
||||
<p/>
|
||||
Rules for null-handling:
|
||||
<ul>
|
||||
<li> Records with one or more null sort-field values sort after records with all sort-field values present:
|
||||
POKI_RUN_COMMAND{{mlr cat data/sort-null.dat}}HERE
|
||||
POKI_RUN_COMMAND{{mlr sort -n a data/sort-null.dat}}HERE
|
||||
POKI_RUN_COMMAND{{mlr sort -nr a data/sort-null.dat}}HERE
|
||||
<li> Functions which have one or more null arguments produce null output: e.g.
|
||||
POKI_RUN_COMMAND{{echo 'x=2,y=3' | mlr put '$a=$x+$y'}}HERE
|
||||
POKI_RUN_COMMAND{{echo 'x=,y=3' | mlr put '$a=$x+$y'}}HERE
|
||||
POKI_RUN_COMMAND{{echo 'x=,y=3' | mlr put '$a=log($x);$b=log($y)'}}HERE
|
||||
<li> The <tt>min</tt> and <tt>max</tt> functions are special: if one argument is non-null, it wins:
|
||||
POKI_RUN_COMMAND{{echo 'x=,y=3' | mlr put '$a=min($x,$y);$b=max($x,$y)'}}HERE
|
||||
</ul>
|
||||
|
||||
<p/>That is, one cannot check an integer product to see if it is precisely
|
||||
greater than 2<sup>63</sup>-1 or less than -2<sup>63</sup> using either integer
|
||||
arithmetic (it may have already overflowed) or using double-precision (due to
|
||||
granularity). Instead Miller checks for overflow in 64-bit integer
|
||||
multiplication by seeing whether the absolute value of the double-precision
|
||||
product exceeds the largest representable IEEE double less than 2<sup>63</sup>,
|
||||
which we see from the listing above is 9223372036854774784. (An alternative
|
||||
would be to do all integer multiplies using handcrafted multi-word 128-bit
|
||||
arithmetic. This approach is not taken.)
|
||||
<!--
|
||||
<p/>xxx
|
||||
|
||||
<h2>Pythonic division</h2>
|
||||
|
||||
<p/>Division and remainder are
|
||||
<a href="http://python-history.blogspot.com/2010/08/why-pythons-integer-division-floors.html">
|
||||
pythonic</a>:
|
||||
<ul>
|
||||
<li/> Quotient of integers is floating-point: <tt>7/2</tt> is <tt>3.5</tt>.
|
||||
<li/> Integer division is done with <tt>//</tt>: <tt>7/2</tt> is <tt>3</tt>.
|
||||
This rounds toward the negative.
|
||||
<li/> Remainders are non-negative.
|
||||
</ul>
|
||||
PxOKI_RUN_COMMAND{{mlr -x-print-type-arithmetic-info}}HERE
|
||||
-->
|
||||
|
||||
<!-- ================================================================ -->
|
||||
<h1>String literals</h1>
|
||||
|
|
@ -1574,3 +1463,145 @@ mlr filter '$a =~ "(..)_(....)'
|
|||
<tt>\0</tt> is the entire match string; <tt>\15</tt> is treated as <tt>\1</tt>
|
||||
followed by an unrelated <tt>5</tt>.
|
||||
</ul>
|
||||
|
||||
<!-- ================================================================ -->
|
||||
<h1>Operator precedence</h1>
|
||||
|
||||
<p/>Operators are listed in order of decreasing precedence, highest first.
|
||||
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
<pre>
|
||||
Operators Associativity
|
||||
--------- -------------
|
||||
() left to right
|
||||
** right to left
|
||||
! ~ unary+ unary- & right to left
|
||||
binary* / // % left to right
|
||||
binary+ binary- . left to right
|
||||
<< >> left to right
|
||||
& left to right
|
||||
^ left to right
|
||||
| left to right
|
||||
< <= > >= left to right
|
||||
== != =~ !=~ left to right
|
||||
&& left to right
|
||||
^^ left to right
|
||||
|| left to right
|
||||
? : right to left
|
||||
= N/A for Miller (there is no $a=$b=$c)
|
||||
</pre>
|
||||
</div>
|
||||
<p/>
|
||||
|
||||
<!-- ================================================================ -->
|
||||
<h1>Operator and function semantics</h1>
|
||||
|
||||
<ul>
|
||||
|
||||
<li/> Functions are in general pass-throughs straight to the system-standard C
|
||||
library.
|
||||
|
||||
<li/> The <tt>min</tt> and <tt>max</tt> functions are different from other
|
||||
multi-argument functions which return null if any of their inputs are null: for
|
||||
<tt>min</tt> and <tt>max</tt>, by contrast, if one argument is null, the other
|
||||
is returned.
|
||||
|
||||
<li/> Symmetrically with respect to the bitwise OR, XOR, and AND operators
|
||||
<tt>|</tt>, <tt>^</tt>, <tt>&</tt>, Miller has logical operators
|
||||
<tt>||</tt>, <tt>^^</tt>, <tt>&&</tt>: the logical XOR not existing in
|
||||
C.
|
||||
|
||||
<li/> The exponentiation operator <tt>**</tt> is familiar from many languages.
|
||||
|
||||
<li/> The regex-match and regex-not-match operators <tt>=~</tt> and
|
||||
<tt>!=~</tt> are similar to those in Ruby and Perl.
|
||||
|
||||
</ul>
|
||||
|
||||
<!-- ================================================================ -->
|
||||
<h1>Arithmetic</h1>
|
||||
|
||||
<h2>Input scanning</h2>
|
||||
|
||||
<p/>Numbers in Miller are double-precision float or 64-bit signed integers.
|
||||
Anything scannable as int, e.g <tt>123</tt> or <tt>0xabcd</tt>, is treated as
|
||||
an integer; otherwise, input scannable as float (<tt>4.56</tt> or <tt>8e9</tt>)
|
||||
is treated as float; everything else is a string.
|
||||
|
||||
<p/>If you want all numbers to be treated as floats, then you may use
|
||||
<tt>float()</tt> in your filter/put expressions (e.g. replacing <tt>$c = $a *
|
||||
$b</tt> with <tt>$c = float($a) * float($b)</tt>) — or, more simply, use
|
||||
<tt>mlr filter -F</tt> and <tt>mlr put -F</tt> which forces all numeric input,
|
||||
whether from expression literals or field values, to float. Likewise <tt>mlr
|
||||
stats1 -F</tt> and <tt>mlr step -F</tt> force integerable accumulators (such as
|
||||
<tt>count</tt>) to be done in floating-point.
|
||||
|
||||
<h2>Conversion by math routines</h2>
|
||||
|
||||
<p/>For most math functions, integers are cast to float on input, and produce
|
||||
float output: e.g. <tt>exp(0) = 1.0</tt> rather than <tt>1</tt>. The
|
||||
following, however, produce integer output if their inputs are integers:
|
||||
<tt>+</tt> <tt>-</tt> <tt>*</tt> <tt>/</tt> <tt>//</tt> <tt>%</tt> <tt>abs</tt>
|
||||
<tt>ceil</tt> <tt>floor</tt> <tt>max</tt> <tt>min</tt> <tt>round</tt>
|
||||
<tt>roundm</tt> <tt>sgn</tt>. As well, <tt>stats1 -a min</tt>, <tt>stats1 -a
|
||||
max</tt>, <tt>stats1 -a sum</tt>, <tt>step -a delta</tt>, and <tt>step -a
|
||||
rsum</tt> produce integer output if their inputs are integers.
|
||||
|
||||
<h2>Conversion by arithmetic operators</h2>
|
||||
|
||||
<p/>The sum, difference, and product of integers is again integer, except for
|
||||
when that would overflow a 64-bit integer at which point Miller converts the
|
||||
result to float.
|
||||
|
||||
<p/>The short of it is that Miller does this transparently for you so you
|
||||
needn’t think about it.
|
||||
|
||||
<p/>Implementation details of this, for the interested: integer adds and
|
||||
subtracts overflow by at most one bit so it suffices to check sign-changes.
|
||||
Thus, Miller allows you to add and subtract arbitrary 64-bit signed integers,
|
||||
converting only to float precisely when the result is less than -2<sup>63</sup>
|
||||
or greater than 2<sup>63</sup>-1. Multiplies, on the other hand, can overflow
|
||||
by a word size and a sign-change technique does not suffice to detect overflow.
|
||||
Instead Miller tests whether the floating-point product exceeds the
|
||||
representable integer range. Now, 64-bit integers have 64-bit precision while
|
||||
IEEE-doubles have only 52-bit mantissas — so, there are 53 bits including
|
||||
implicit leading one. The following experiment explicitly demonstrates the
|
||||
resolution at this range:
|
||||
|
||||
<div class=pokipanel>
|
||||
<pre>
|
||||
64-bit integer 64-bit integer Casted to double Back to 64-bit
|
||||
in hex in decimal integer
|
||||
0x7ffffffffffff9ff 9223372036854774271 9223372036854773760.000000 0x7ffffffffffff800
|
||||
0x7ffffffffffffa00 9223372036854774272 9223372036854773760.000000 0x7ffffffffffff800
|
||||
0x7ffffffffffffbff 9223372036854774783 9223372036854774784.000000 0x7ffffffffffffc00
|
||||
0x7ffffffffffffc00 9223372036854774784 9223372036854774784.000000 0x7ffffffffffffc00
|
||||
0x7ffffffffffffdff 9223372036854775295 9223372036854774784.000000 0x7ffffffffffffc00
|
||||
0x7ffffffffffffe00 9223372036854775296 9223372036854775808.000000 0x8000000000000000
|
||||
0x7ffffffffffffffe 9223372036854775806 9223372036854775808.000000 0x8000000000000000
|
||||
0x7fffffffffffffff 9223372036854775807 9223372036854775808.000000 0x8000000000000000
|
||||
</pre>
|
||||
</div>
|
||||
|
||||
<p/>That is, one cannot check an integer product to see if it is precisely
|
||||
greater than 2<sup>63</sup>-1 or less than -2<sup>63</sup> using either integer
|
||||
arithmetic (it may have already overflowed) or using double-precision (due to
|
||||
granularity). Instead Miller checks for overflow in 64-bit integer
|
||||
multiplication by seeing whether the absolute value of the double-precision
|
||||
product exceeds the largest representable IEEE double less than 2<sup>63</sup>,
|
||||
which we see from the listing above is 9223372036854774784. (An alternative
|
||||
would be to do all integer multiplies using handcrafted multi-word 128-bit
|
||||
arithmetic. This approach is not taken.)
|
||||
|
||||
<h2>Pythonic division</h2>
|
||||
|
||||
<p/>Division and remainder are
|
||||
<a href="http://python-history.blogspot.com/2010/08/why-pythons-integer-division-floors.html">
|
||||
pythonic</a>:
|
||||
<ul>
|
||||
<li/> Quotient of integers is floating-point: <tt>7/2</tt> is <tt>3.5</tt>.
|
||||
<li/> Integer division is done with <tt>//</tt>: <tt>7/2</tt> is <tt>3</tt>.
|
||||
This rounds toward the negative.
|
||||
<li/> Remainders are non-negative.
|
||||
</ul>
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
a=3,b=2
|
||||
a=1,b=8
|
||||
a=,b=4
|
||||
x=9,b=10
|
||||
a=5,b=7
|
||||
|
|
|
|||
|
|
@ -2,12 +2,12 @@
|
|||
.\" Title: mlr
|
||||
.\" Author: [see the "AUTHOR" section]
|
||||
.\" Generator: ./mkman.rb
|
||||
.\" Date: 2016-04-12
|
||||
.\" Date: 2016-04-13
|
||||
.\" Manual: \ \&
|
||||
.\" Source: \ \&
|
||||
.\" Language: English
|
||||
.\"
|
||||
.TH "MILLER" "1" "2016-04-12" "\ \&" "\ \&"
|
||||
.TH "MILLER" "1" "2016-04-13" "\ \&" "\ \&"
|
||||
.\" -----------------------------------------------------------------
|
||||
.\" * Portability definitions
|
||||
.\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
|
|
|||
|
|
@ -141,8 +141,6 @@ Miller commands were run with pretty-print-tabular output format.
|
|||
<center><b>Contents:</b></center>
|
||||
• <a href="#Command_overview">Command overview</a><br/>
|
||||
• <a href="#On-line_help">On-line help</a><br/>
|
||||
• <a href="#Data_types">Data types</a><br/>
|
||||
• <a href="#Null_data">Null data</a><br/>
|
||||
• <a href="#I/O_options">I/O options</a><br/>
|
||||
• <a href="#Formats">Formats</a><br/>
|
||||
• <a href="#Compression">Compression</a><br/>
|
||||
|
|
@ -200,6 +198,11 @@ Miller commands were run with pretty-print-tabular output format.
|
|||
• <a href="#uniq">uniq</a><br/>
|
||||
• <a href="#then-chaining">then-chaining</a><br/>
|
||||
• <a href="#Functions_for_filter_and_put">Functions for filter and put</a><br/>
|
||||
• <a href="#Data_types">Data types</a><br/>
|
||||
• <a href="#Null_data">Null data</a><br/>
|
||||
• <a href="#String_literals">String literals</a><br/>
|
||||
• <a href="#Regular_expressions">Regular expressions</a><br/>
|
||||
• <a href="#Regex_captures">Regex captures</a><br/>
|
||||
• <a href="#Operator_precedence">Operator precedence</a><br/>
|
||||
• <a href="#Operator_and_function_semantics">Operator and function semantics</a><br/>
|
||||
• <a href="#Arithmetic">Arithmetic</a><br/>
|
||||
|
|
@ -207,9 +210,6 @@ Miller commands were run with pretty-print-tabular output format.
|
|||
• <a href="#Conversion_by_math_routines">Conversion by math routines</a><br/>
|
||||
• <a href="#Conversion_by_arithmetic_operators">Conversion by arithmetic operators</a><br/>
|
||||
• <a href="#Pythonic_division">Pythonic division</a><br/>
|
||||
• <a href="#String_literals">String literals</a><br/>
|
||||
• <a href="#Regular_expressions">Regular expressions</a><br/>
|
||||
• <a href="#Regex_captures">Regex captures</a><br/>
|
||||
</div>
|
||||
<p/>
|
||||
|
||||
|
|
@ -553,146 +553,6 @@ which is the same as:
|
|||
</div>
|
||||
<p/>
|
||||
|
||||
<!-- ================================================================ -->
|
||||
<a id="Data_types"/><h1>Data types</h1>
|
||||
|
||||
<p/> Miller’s input and output are all string-oriented: there is (as of
|
||||
August 2015 anyway) no support for binary record packing. In this sense,
|
||||
everything is a string in and out of Miller. During processing, field names
|
||||
are always strings, even if they have names like "3"; field values are usually
|
||||
strings. Field values’ ability to be interpreted as a non-string type
|
||||
only has meaning when comparison or function operations are done on them. And
|
||||
it is an error condition if Miller encounters non-numeric (or otherwise
|
||||
mistyped) data in a field in which it has been asked to do numeric (or
|
||||
otherwise type-specific) operations.
|
||||
|
||||
<p/> Field values are treated as numeric for the following:
|
||||
<ul>
|
||||
<li/> Numeric sort: <tt>mlr sort -n</tt>, <tt>mlr sort -nr</tt>.
|
||||
<li/> Statistics: <tt>mlr histogram</tt>, <tt>mlr stats1</tt>, <tt>mlr stats2</tt>.
|
||||
<li/> Cross-record arithmetic: <tt>mlr step</tt>.
|
||||
</ul>
|
||||
|
||||
<p/>For <tt>mlr put</tt> and <tt>mlr filter</tt>:
|
||||
|
||||
<ul>
|
||||
|
||||
<li/> Miller’s types for function processing are <b>null</b> (empty
|
||||
string), <b>error</b>, <b>string</b>, <b>float</b> (double-precision),
|
||||
<b>int</b> (64-bit signed), and <b>boolean</b>.
|
||||
|
||||
<li/> On input, string values representable as numbers, e.g. "3" or "3.1", are
|
||||
treated as int or float, respectively. If a record has <tt>x=1,y=2</tt> then
|
||||
<tt>mlr put '$z=$x+$y'</tt> will produce <tt>x=1,y=2,z=3</tt>, and <tt>mlr put
|
||||
'$z=$x.$y'</tt> gives an error. To coerce back to string for processing, use
|
||||
the <tt>string</tt> function: <tt>mlr put '$z=string($x).string($y)'</tt> will
|
||||
produce <tt>x=1,y=2,z=12</tt>.
|
||||
|
||||
<li/> On input, string values representable as boolean (e.g. <tt>"true"</tt>,
|
||||
<tt>"false"</tt>) are <i>not</i> automatically treated as boolean. (This is
|
||||
because <tt>"true"</tt> and <tt>"false"</tt> are ordinary words, and auto
|
||||
string-to-boolean on a column consisting of words would result in some strings
|
||||
mixed with some booleans.) Use the <tt>boolean</tt> function to coerce: e.g.
|
||||
giving the record <tt>x=1,y=2,w=false</tt> to <tt>mlr put '$z=($x<$y) ||
|
||||
boolean($w)'</tt>.
|
||||
|
||||
<li/> Functions take types as described in <tt>mlr --help-all-functions</tt>:
|
||||
for example, <tt>log10</tt> takes float input and produces float output,
|
||||
<tt>gmt2sec</tt> maps string to int, and <tt>sec2gmt</tt> maps int to string.
|
||||
|
||||
<li/> All math functions described in <tt>mlr --help-all-functions</tt> take
|
||||
integer as well as float input.
|
||||
|
||||
</ul>
|
||||
|
||||
<!-- ================================================================ -->
|
||||
<a id="Null_data"/><h1>Null data</h1>
|
||||
|
||||
<p/> One of Miller’s key features is its support for <b>heterogeneous</b> data.
|
||||
Accordingly, if you try to sort on field <tt>hostname</tt> when not all records in the data
|
||||
stream <i>have</i> a field named <tt>hostname</tt>, it is not an error (although you could
|
||||
pre-filter the data stream using <tt>mlr having-fields --at-least hostname then sort ...</tt>).
|
||||
Rather, records lacking one or more sort keys are simply output contiguously by <tt>mlr sort</tt>.
|
||||
|
||||
<p/> Field values may also be null by being
|
||||
specified with present key but empty value: e.g. sending <tt>x=,y=2</tt> to <tt>mlr put '$z=$x+$y'</tt>.
|
||||
|
||||
<p/>
|
||||
Rules for null-handling:
|
||||
<ul>
|
||||
<li> Records with one or more null sort-field values sort after records with all sort-field values present:
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
<pre>
|
||||
$ mlr --opprint cat data/sort-null.dat
|
||||
a b
|
||||
3 2
|
||||
1 8
|
||||
- 4
|
||||
5 7
|
||||
</pre>
|
||||
</div>
|
||||
<p/>
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
<pre>
|
||||
$ mlr --opprint sort -n a data/sort-null.dat
|
||||
a b
|
||||
1 8
|
||||
3 2
|
||||
5 7
|
||||
- 4
|
||||
</pre>
|
||||
</div>
|
||||
<p/>
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
<pre>
|
||||
$ mlr --opprint sort -nr a data/sort-null.dat
|
||||
a b
|
||||
- 4
|
||||
5 7
|
||||
3 2
|
||||
1 8
|
||||
</pre>
|
||||
</div>
|
||||
<p/>
|
||||
<li> Functions which have one or more null arguments produce null output: e.g.
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
<pre>
|
||||
$ echo 'x=2,y=3' | mlr put '$a=$x+$y'
|
||||
x=2,y=3,a=5
|
||||
</pre>
|
||||
</div>
|
||||
<p/>
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
<pre>
|
||||
$ echo 'x=,y=3' | mlr put '$a=$x+$y'
|
||||
x=,y=3,a=
|
||||
</pre>
|
||||
</div>
|
||||
<p/>
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
<pre>
|
||||
$ echo 'x=,y=3' | mlr put '$a=log($x);$b=log($y)'
|
||||
x=,y=3,a=,b=1.098612
|
||||
</pre>
|
||||
</div>
|
||||
<p/>
|
||||
<li> The <tt>min</tt> and <tt>max</tt> functions are special: if one argument is non-null, it wins:
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
<pre>
|
||||
$ echo 'x=,y=3' | mlr put '$a=min($x,$y);$b=max($x,$y)'
|
||||
x=,y=3,a=3,b=3
|
||||
</pre>
|
||||
</div>
|
||||
<p/>
|
||||
</ul>
|
||||
|
||||
<!-- ================================================================ -->
|
||||
<a id="I/O_options"/><h1>I/O options</h1>
|
||||
|
||||
|
|
@ -4845,146 +4705,175 @@ along with the mathematical constants PI and E.
|
|||
<p/>
|
||||
|
||||
<!-- ================================================================ -->
|
||||
<a id="Operator_precedence"/><h1>Operator precedence</h1>
|
||||
<a id="Data_types"/><h1>Data types</h1>
|
||||
|
||||
<p/>Operators are listed in order of decreasing precedence, highest first.
|
||||
<p/> Miller’s input and output are all string-oriented: there is (as of
|
||||
August 2015 anyway) no support for binary record packing. In this sense,
|
||||
everything is a string in and out of Miller. During processing, field names
|
||||
are always strings, even if they have names like "3"; field values are usually
|
||||
strings. Field values’ ability to be interpreted as a non-string type
|
||||
only has meaning when comparison or function operations are done on them. And
|
||||
it is an error condition if Miller encounters non-numeric (or otherwise
|
||||
mistyped) data in a field in which it has been asked to do numeric (or
|
||||
otherwise type-specific) operations.
|
||||
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
<pre>
|
||||
Operators Associativity
|
||||
--------- -------------
|
||||
() left to right
|
||||
** right to left
|
||||
! ~ unary+ unary- & right to left
|
||||
binary* / // % left to right
|
||||
binary+ binary- . left to right
|
||||
<< >> left to right
|
||||
& left to right
|
||||
^ left to right
|
||||
| left to right
|
||||
< <= > >= left to right
|
||||
== != =~ !=~ left to right
|
||||
&& left to right
|
||||
^^ left to right
|
||||
|| left to right
|
||||
? : right to left
|
||||
= N/A for Miller (there is no $a=$b=$c)
|
||||
</pre>
|
||||
</div>
|
||||
<p/>
|
||||
<p/> Field values are treated as numeric for the following:
|
||||
<ul>
|
||||
<li/> Numeric sort: <tt>mlr sort -n</tt>, <tt>mlr sort -nr</tt>.
|
||||
<li/> Statistics: <tt>mlr histogram</tt>, <tt>mlr stats1</tt>, <tt>mlr stats2</tt>.
|
||||
<li/> Cross-record arithmetic: <tt>mlr step</tt>.
|
||||
</ul>
|
||||
|
||||
<!-- ================================================================ -->
|
||||
<a id="Operator_and_function_semantics"/><h1>Operator and function semantics</h1>
|
||||
<p/>For <tt>mlr put</tt> and <tt>mlr filter</tt>:
|
||||
|
||||
<ul>
|
||||
|
||||
<li/> Functions are in general pass-throughs straight to the system-standard C
|
||||
library.
|
||||
<li/> Miller’s types for function processing are <b>null</b> (empty
|
||||
string), <b>error</b>, <b>string</b>, <b>float</b> (double-precision),
|
||||
<b>int</b> (64-bit signed), and <b>boolean</b>.
|
||||
|
||||
<li/> The <tt>min</tt> and <tt>max</tt> functions are different from other
|
||||
multi-argument functions which return null if any of their inputs are null: for
|
||||
<tt>min</tt> and <tt>max</tt>, by contrast, if one argument is null, the other
|
||||
is returned.
|
||||
<li/> On input, string values representable as numbers, e.g. "3" or "3.1", are
|
||||
treated as int or float, respectively. If a record has <tt>x=1,y=2</tt> then
|
||||
<tt>mlr put '$z=$x+$y'</tt> will produce <tt>x=1,y=2,z=3</tt>, and <tt>mlr put
|
||||
'$z=$x.$y'</tt> gives an error. To coerce back to string for processing, use
|
||||
the <tt>string</tt> function: <tt>mlr put '$z=string($x).string($y)'</tt> will
|
||||
produce <tt>x=1,y=2,z=12</tt>.
|
||||
|
||||
<li/> Symmetrically with respect to the bitwise OR, XOR, and AND operators
|
||||
<tt>|</tt>, <tt>^</tt>, <tt>&</tt>, Miller has logical operators
|
||||
<tt>||</tt>, <tt>^^</tt>, <tt>&&</tt>: the logical XOR not existing in
|
||||
C.
|
||||
<li/> On input, string values representable as boolean (e.g. <tt>"true"</tt>,
|
||||
<tt>"false"</tt>) are <i>not</i> automatically treated as boolean. (This is
|
||||
because <tt>"true"</tt> and <tt>"false"</tt> are ordinary words, and auto
|
||||
string-to-boolean on a column consisting of words would result in some strings
|
||||
mixed with some booleans.) Use the <tt>boolean</tt> function to coerce: e.g.
|
||||
giving the record <tt>x=1,y=2,w=false</tt> to <tt>mlr put '$z=($x<$y) ||
|
||||
boolean($w)'</tt>.
|
||||
|
||||
<li/> The exponentiation operator <tt>**</tt> is familiar from many languages.
|
||||
<li/> Functions take types as described in <tt>mlr --help-all-functions</tt>:
|
||||
for example, <tt>log10</tt> takes float input and produces float output,
|
||||
<tt>gmt2sec</tt> maps string to int, and <tt>sec2gmt</tt> maps int to string.
|
||||
|
||||
<li/> The regex-match and regex-not-match operators <tt>=~</tt> and
|
||||
<tt>!=~</tt> are similar to those in Ruby and Perl.
|
||||
<li/> All math functions described in <tt>mlr --help-all-functions</tt> take
|
||||
integer as well as float input.
|
||||
|
||||
</ul>
|
||||
|
||||
<!-- ================================================================ -->
|
||||
<a id="Arithmetic"/><h1>Arithmetic</h1>
|
||||
<a id="Null_data"/><h1>Null data</h1>
|
||||
|
||||
<a id="Input_scanning"/><h2>Input scanning</h2>
|
||||
<p/> One of Miller’s key features is its support for <b>heterogeneous</b> data.
|
||||
Accordingly, if you try to sort on field <tt>hostname</tt> when not all records in the data
|
||||
stream <i>have</i> a field named <tt>hostname</tt>, it is not an error (although you could
|
||||
pre-filter the data stream using <tt>mlr having-fields --at-least hostname then sort ...</tt>).
|
||||
Rather, records lacking one or more sort keys are simply output contiguously by <tt>mlr sort</tt>.
|
||||
|
||||
<p/>Numbers in Miller are double-precision float or 64-bit signed integers.
|
||||
Anything scannable as int, e.g <tt>123</tt> or <tt>0xabcd</tt>, is treated as
|
||||
an integer; otherwise, input scannable as float (<tt>4.56</tt> or <tt>8e9</tt>)
|
||||
is treated as float; everything else is a string.
|
||||
<p/> Miller has three kinds of null data:
|
||||
|
||||
<p/>If you want all numbers to be treated as floats, then you may use
|
||||
<tt>float()</tt> in your filter/put expressions (e.g. replacing <tt>$c = $a *
|
||||
$b</tt> with <tt>$c = float($a) * float($b)</tt>) — or, more simply, use
|
||||
<tt>mlr filter -F</tt> and <tt>mlr put -F</tt> which forces all numeric input,
|
||||
whether from expression literals or field values, to float. Likewise <tt>mlr
|
||||
stats1 -F</tt> and <tt>mlr step -F</tt> force integerable accumulators (such as
|
||||
<tt>count</tt>) to be done in floating-point.
|
||||
<ul>
|
||||
|
||||
<a id="Conversion_by_math_routines"/><h2>Conversion by math routines</h2>
|
||||
<li/> <b>Void</b>: a field name is present in a record (or in an out-of-stream
|
||||
variable) with empty value: e.g. <tt>x=,y=2</tt> in the data input stream, or
|
||||
assignment <tt>$x=""</tt> or <tt>@x=""</tt> in <tt>mlr put</tt>.
|
||||
|
||||
<p/>For most math functions, integers are cast to float on input, and produce
|
||||
float output: e.g. <tt>exp(0) = 1.0</tt> rather than <tt>1</tt>. The
|
||||
following, however, produce integer output if their inputs are integers:
|
||||
<tt>+</tt> <tt>-</tt> <tt>*</tt> <tt>/</tt> <tt>//</tt> <tt>%</tt> <tt>abs</tt>
|
||||
<tt>ceil</tt> <tt>floor</tt> <tt>max</tt> <tt>min</tt> <tt>round</tt>
|
||||
<tt>roundm</tt> <tt>sgn</tt>. As well, <tt>stats1 -a min</tt>, <tt>stats1 -a
|
||||
max</tt>, <tt>stats1 -a sum</tt>, <tt>step -a delta</tt>, and <tt>step -a
|
||||
rsum</tt> produce integer output if their inputs are integers.
|
||||
<li/> <b>Absent</b>: a field name is not present, e.g. input record is
|
||||
<tt>x=1,y=2</tt> and a <tt>put</tt> or <tt>filter</tt> expression refers to
|
||||
<tt>$z</tt>.
|
||||
|
||||
<a id="Conversion_by_arithmetic_operators"/><h2>Conversion by arithmetic operators</h2>
|
||||
<li/> <b>Uninit</b>: an out-of-stream variable is read before being assigned
|
||||
to, e.g. <tt>mlr put -q '@sum += $x'; end{emit @sum}'</tt> or <tt>mlr put -q
|
||||
'@sum[$a][$b] += $x'; end{emit @sum, "a", "b"}'</tt>. Without uninit, for the
|
||||
former you’d need to type <tt>mlr put -q 'begin{@sum = 0}; @sum += $x';
|
||||
end{emit @sum}'</tt> (which is tolerable), but for the latter you’d need
|
||||
to know all values of <tt>$a</tt> and <tt>$b</tt> before reading your input
|
||||
data, which would be intolerable.
|
||||
|
||||
<p/>The sum, difference, and product of integers is again integer, except for
|
||||
when that would overflow a 64-bit integer at which point Miller converts the
|
||||
result to float.
|
||||
</ul>
|
||||
|
||||
<p/>The short of it is that Miller does this transparently for you so you
|
||||
needn’t think about it.
|
||||
<!--
|
||||
<p/>xxx more examples.
|
||||
|
||||
<p/>Implementation details of this, for the interested: integer adds and
|
||||
subtracts overflow by at most one bit so it suffices to check sign-changes.
|
||||
Thus, Miller allows you to add and subtract arbitrary 64-bit signed integers,
|
||||
converting only to float precisely when the result is less than -2<sup>63</sup>
|
||||
or greater than 2<sup>63</sup>-1. Multiplies, on the other hand, can overflow
|
||||
by a word size and a sign-change technique does not suffice to detect overflow.
|
||||
Instead Miller tests whether the floating-point product exceeds the
|
||||
representable integer range. Now, 64-bit integers have 64-bit precision while
|
||||
IEEE-doubles have only 52-bit mantissas — so, there are 53 bits including
|
||||
implicit leading one. The following experiment explicitly demonstrates the
|
||||
resolution at this range:
|
||||
<p/>xxx keep typing up from here.
|
||||
-->
|
||||
|
||||
<div class=pokipanel>
|
||||
<p/>
|
||||
Rules for null-handling:
|
||||
<ul>
|
||||
<li> Records with one or more null sort-field values sort after records with all sort-field values present:
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
<pre>
|
||||
64-bit integer 64-bit integer Casted to double Back to 64-bit
|
||||
in hex in decimal integer
|
||||
0x7ffffffffffff9ff 9223372036854774271 9223372036854773760.000000 0x7ffffffffffff800
|
||||
0x7ffffffffffffa00 9223372036854774272 9223372036854773760.000000 0x7ffffffffffff800
|
||||
0x7ffffffffffffbff 9223372036854774783 9223372036854774784.000000 0x7ffffffffffffc00
|
||||
0x7ffffffffffffc00 9223372036854774784 9223372036854774784.000000 0x7ffffffffffffc00
|
||||
0x7ffffffffffffdff 9223372036854775295 9223372036854774784.000000 0x7ffffffffffffc00
|
||||
0x7ffffffffffffe00 9223372036854775296 9223372036854775808.000000 0x8000000000000000
|
||||
0x7ffffffffffffffe 9223372036854775806 9223372036854775808.000000 0x8000000000000000
|
||||
0x7fffffffffffffff 9223372036854775807 9223372036854775808.000000 0x8000000000000000
|
||||
$ mlr cat data/sort-null.dat
|
||||
a=3,b=2
|
||||
a=1,b=8
|
||||
a=,b=4
|
||||
x=9,b=10
|
||||
a=5,b=7
|
||||
</pre>
|
||||
</div>
|
||||
<p/>
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
<pre>
|
||||
$ mlr sort -n a data/sort-null.dat
|
||||
a=1,b=8
|
||||
a=3,b=2
|
||||
a=5,b=7
|
||||
a=,b=4
|
||||
x=9,b=10
|
||||
</pre>
|
||||
</div>
|
||||
<p/>
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
<pre>
|
||||
$ mlr sort -nr a data/sort-null.dat
|
||||
a=,b=4
|
||||
a=5,b=7
|
||||
a=3,b=2
|
||||
a=1,b=8
|
||||
x=9,b=10
|
||||
</pre>
|
||||
</div>
|
||||
<p/>
|
||||
<li> Functions which have one or more null arguments produce null output: e.g.
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
<pre>
|
||||
$ echo 'x=2,y=3' | mlr put '$a=$x+$y'
|
||||
x=2,y=3,a=5
|
||||
</pre>
|
||||
</div>
|
||||
<p/>
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
<pre>
|
||||
$ echo 'x=,y=3' | mlr put '$a=$x+$y'
|
||||
x=,y=3,a=
|
||||
</pre>
|
||||
</div>
|
||||
<p/>
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
<pre>
|
||||
$ echo 'x=,y=3' | mlr put '$a=log($x);$b=log($y)'
|
||||
x=,y=3,a=,b=1.098612
|
||||
</pre>
|
||||
</div>
|
||||
<p/>
|
||||
<li> The <tt>min</tt> and <tt>max</tt> functions are special: if one argument is non-null, it wins:
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
<pre>
|
||||
$ echo 'x=,y=3' | mlr put '$a=min($x,$y);$b=max($x,$y)'
|
||||
x=,y=3,a=3,b=3
|
||||
</pre>
|
||||
</div>
|
||||
<p/>
|
||||
</ul>
|
||||
|
||||
<p/>That is, one cannot check an integer product to see if it is precisely
|
||||
greater than 2<sup>63</sup>-1 or less than -2<sup>63</sup> using either integer
|
||||
arithmetic (it may have already overflowed) or using double-precision (due to
|
||||
granularity). Instead Miller checks for overflow in 64-bit integer
|
||||
multiplication by seeing whether the absolute value of the double-precision
|
||||
product exceeds the largest representable IEEE double less than 2<sup>63</sup>,
|
||||
which we see from the listing above is 9223372036854774784. (An alternative
|
||||
would be to do all integer multiplies using handcrafted multi-word 128-bit
|
||||
arithmetic. This approach is not taken.)
|
||||
<!--
|
||||
<p/>xxx
|
||||
|
||||
<a id="Pythonic_division"/><h2>Pythonic division</h2>
|
||||
|
||||
<p/>Division and remainder are
|
||||
<a href="http://python-history.blogspot.com/2010/08/why-pythons-integer-division-floors.html">
|
||||
pythonic</a>:
|
||||
<ul>
|
||||
<li/> Quotient of integers is floating-point: <tt>7/2</tt> is <tt>3.5</tt>.
|
||||
<li/> Integer division is done with <tt>//</tt>: <tt>7/2</tt> is <tt>3</tt>.
|
||||
This rounds toward the negative.
|
||||
<li/> Remainders are non-negative.
|
||||
</ul>
|
||||
PxOKI_RUN_COMMAND{{mlr -x-print-type-arithmetic-info}}HERE
|
||||
-->
|
||||
|
||||
<!-- ================================================================ -->
|
||||
<a id="String_literals"/><h1>String literals</h1>
|
||||
|
|
@ -5142,6 +5031,148 @@ mlr filter '$a =~ "(..)_(....)'
|
|||
<tt>\0</tt> is the entire match string; <tt>\15</tt> is treated as <tt>\1</tt>
|
||||
followed by an unrelated <tt>5</tt>.
|
||||
</ul>
|
||||
|
||||
<!-- ================================================================ -->
|
||||
<a id="Operator_precedence"/><h1>Operator precedence</h1>
|
||||
|
||||
<p/>Operators are listed in order of decreasing precedence, highest first.
|
||||
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
<pre>
|
||||
Operators Associativity
|
||||
--------- -------------
|
||||
() left to right
|
||||
** right to left
|
||||
! ~ unary+ unary- & right to left
|
||||
binary* / // % left to right
|
||||
binary+ binary- . left to right
|
||||
<< >> left to right
|
||||
& left to right
|
||||
^ left to right
|
||||
| left to right
|
||||
< <= > >= left to right
|
||||
== != =~ !=~ left to right
|
||||
&& left to right
|
||||
^^ left to right
|
||||
|| left to right
|
||||
? : right to left
|
||||
= N/A for Miller (there is no $a=$b=$c)
|
||||
</pre>
|
||||
</div>
|
||||
<p/>
|
||||
|
||||
<!-- ================================================================ -->
|
||||
<a id="Operator_and_function_semantics"/><h1>Operator and function semantics</h1>
|
||||
|
||||
<ul>
|
||||
|
||||
<li/> Functions are in general pass-throughs straight to the system-standard C
|
||||
library.
|
||||
|
||||
<li/> The <tt>min</tt> and <tt>max</tt> functions are different from other
|
||||
multi-argument functions which return null if any of their inputs are null: for
|
||||
<tt>min</tt> and <tt>max</tt>, by contrast, if one argument is null, the other
|
||||
is returned.
|
||||
|
||||
<li/> Symmetrically with respect to the bitwise OR, XOR, and AND operators
|
||||
<tt>|</tt>, <tt>^</tt>, <tt>&</tt>, Miller has logical operators
|
||||
<tt>||</tt>, <tt>^^</tt>, <tt>&&</tt>: the logical XOR not existing in
|
||||
C.
|
||||
|
||||
<li/> The exponentiation operator <tt>**</tt> is familiar from many languages.
|
||||
|
||||
<li/> The regex-match and regex-not-match operators <tt>=~</tt> and
|
||||
<tt>!=~</tt> are similar to those in Ruby and Perl.
|
||||
|
||||
</ul>
|
||||
|
||||
<!-- ================================================================ -->
|
||||
<a id="Arithmetic"/><h1>Arithmetic</h1>
|
||||
|
||||
<a id="Input_scanning"/><h2>Input scanning</h2>
|
||||
|
||||
<p/>Numbers in Miller are double-precision float or 64-bit signed integers.
|
||||
Anything scannable as int, e.g <tt>123</tt> or <tt>0xabcd</tt>, is treated as
|
||||
an integer; otherwise, input scannable as float (<tt>4.56</tt> or <tt>8e9</tt>)
|
||||
is treated as float; everything else is a string.
|
||||
|
||||
<p/>If you want all numbers to be treated as floats, then you may use
|
||||
<tt>float()</tt> in your filter/put expressions (e.g. replacing <tt>$c = $a *
|
||||
$b</tt> with <tt>$c = float($a) * float($b)</tt>) — or, more simply, use
|
||||
<tt>mlr filter -F</tt> and <tt>mlr put -F</tt> which forces all numeric input,
|
||||
whether from expression literals or field values, to float. Likewise <tt>mlr
|
||||
stats1 -F</tt> and <tt>mlr step -F</tt> force integerable accumulators (such as
|
||||
<tt>count</tt>) to be done in floating-point.
|
||||
|
||||
<a id="Conversion_by_math_routines"/><h2>Conversion by math routines</h2>
|
||||
|
||||
<p/>For most math functions, integers are cast to float on input, and produce
|
||||
float output: e.g. <tt>exp(0) = 1.0</tt> rather than <tt>1</tt>. The
|
||||
following, however, produce integer output if their inputs are integers:
|
||||
<tt>+</tt> <tt>-</tt> <tt>*</tt> <tt>/</tt> <tt>//</tt> <tt>%</tt> <tt>abs</tt>
|
||||
<tt>ceil</tt> <tt>floor</tt> <tt>max</tt> <tt>min</tt> <tt>round</tt>
|
||||
<tt>roundm</tt> <tt>sgn</tt>. As well, <tt>stats1 -a min</tt>, <tt>stats1 -a
|
||||
max</tt>, <tt>stats1 -a sum</tt>, <tt>step -a delta</tt>, and <tt>step -a
|
||||
rsum</tt> produce integer output if their inputs are integers.
|
||||
|
||||
<a id="Conversion_by_arithmetic_operators"/><h2>Conversion by arithmetic operators</h2>
|
||||
|
||||
<p/>The sum, difference, and product of integers is again integer, except for
|
||||
when that would overflow a 64-bit integer at which point Miller converts the
|
||||
result to float.
|
||||
|
||||
<p/>The short of it is that Miller does this transparently for you so you
|
||||
needn’t think about it.
|
||||
|
||||
<p/>Implementation details of this, for the interested: integer adds and
|
||||
subtracts overflow by at most one bit so it suffices to check sign-changes.
|
||||
Thus, Miller allows you to add and subtract arbitrary 64-bit signed integers,
|
||||
converting only to float precisely when the result is less than -2<sup>63</sup>
|
||||
or greater than 2<sup>63</sup>-1. Multiplies, on the other hand, can overflow
|
||||
by a word size and a sign-change technique does not suffice to detect overflow.
|
||||
Instead Miller tests whether the floating-point product exceeds the
|
||||
representable integer range. Now, 64-bit integers have 64-bit precision while
|
||||
IEEE-doubles have only 52-bit mantissas — so, there are 53 bits including
|
||||
implicit leading one. The following experiment explicitly demonstrates the
|
||||
resolution at this range:
|
||||
|
||||
<div class=pokipanel>
|
||||
<pre>
|
||||
64-bit integer 64-bit integer Casted to double Back to 64-bit
|
||||
in hex in decimal integer
|
||||
0x7ffffffffffff9ff 9223372036854774271 9223372036854773760.000000 0x7ffffffffffff800
|
||||
0x7ffffffffffffa00 9223372036854774272 9223372036854773760.000000 0x7ffffffffffff800
|
||||
0x7ffffffffffffbff 9223372036854774783 9223372036854774784.000000 0x7ffffffffffffc00
|
||||
0x7ffffffffffffc00 9223372036854774784 9223372036854774784.000000 0x7ffffffffffffc00
|
||||
0x7ffffffffffffdff 9223372036854775295 9223372036854774784.000000 0x7ffffffffffffc00
|
||||
0x7ffffffffffffe00 9223372036854775296 9223372036854775808.000000 0x8000000000000000
|
||||
0x7ffffffffffffffe 9223372036854775806 9223372036854775808.000000 0x8000000000000000
|
||||
0x7fffffffffffffff 9223372036854775807 9223372036854775808.000000 0x8000000000000000
|
||||
</pre>
|
||||
</div>
|
||||
|
||||
<p/>That is, one cannot check an integer product to see if it is precisely
|
||||
greater than 2<sup>63</sup>-1 or less than -2<sup>63</sup> using either integer
|
||||
arithmetic (it may have already overflowed) or using double-precision (due to
|
||||
granularity). Instead Miller checks for overflow in 64-bit integer
|
||||
multiplication by seeing whether the absolute value of the double-precision
|
||||
product exceeds the largest representable IEEE double less than 2<sup>63</sup>,
|
||||
which we see from the listing above is 9223372036854774784. (An alternative
|
||||
would be to do all integer multiplies using handcrafted multi-word 128-bit
|
||||
arithmetic. This approach is not taken.)
|
||||
|
||||
<a id="Pythonic_division"/><h2>Pythonic division</h2>
|
||||
|
||||
<p/>Division and remainder are
|
||||
<a href="http://python-history.blogspot.com/2010/08/why-pythons-integer-division-floors.html">
|
||||
pythonic</a>:
|
||||
<ul>
|
||||
<li/> Quotient of integers is floating-point: <tt>7/2</tt> is <tt>3.5</tt>.
|
||||
<li/> Integer division is done with <tt>//</tt>: <tt>7/2</tt> is <tt>3</tt>.
|
||||
This rounds toward the negative.
|
||||
<li/> Remainders are non-negative.
|
||||
</ul>
|
||||
</div>
|
||||
</td>
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue