misc. doc neatens

This commit is contained in:
John Kerl 2016-12-10 13:31:45 -05:00
parent 180610a495
commit 115e1b9dec
7 changed files with 200 additions and 145 deletions

View file

@ -22,7 +22,6 @@ OTHER:
* mlr paste (sjackman)
* don't dump full help on unrecognized flag. WAY too much info. just print short, w/ info on how to get long.
* dump @records after @records[NR]=$*: json "1" rather than 1 indices are b04k3d
! 'Error code 1' -> clearer on regdiff fail
@ -41,13 +40,15 @@ MAPVAR CHECKLIST:
! this should throw an error: mlr --from s put 'int a=1;var b=a[2]'
? on-line help to separate web page, as w/ manpage output?
* emittable/dumpable/etc md_rhs -> more specific fcn-call.
* modify isx()-predicate-doc @ mlr -f to acknowledge maps. also, UT.
* death knell for mv-only evaluators: this is a syntax error:
mlr put -q 'o = haskey(NR==4 ? {"a": NF} : {"b": NF}, "a"); print "NR=".NR.",haskeya=".o'
* modify isx() predicate doc @ mlr -f to acknowledge maps. also, UT.
* debt-reduction
! naming conventions to clarify transfer/copy semantics
- copy/ref of keys vs. values in mlhmmv, local_stack, and callees -- ?!?
@ -83,8 +84,6 @@ MAPVAR CHECKLIST:
* bool_t:uchar typedef & perf check
* mv_error retvals vs. exitthrows vs. absentskips ... bleah
* man mlr section on maps
o things which are always maps:
- maplit
@ -96,6 +95,7 @@ MAPVAR CHECKLIST:
- func retval
- func args
o what funcs are available
o scalar contexts vs. map contexts
o keywords print, dump, emit, for-kv, for-k
o copy semantics

View file

@ -210,7 +210,7 @@ may be easy to grep/sed out the data into a simpler text form — this is a
general text-processing problem.
<p/>Miller does support tabular data represented in JSON: please see
POKI_PUT_LINK_FOR_PAGE(file-formats.html)HERE. See als <a
POKI_PUT_LINK_FOR_PAGE(file-formats.html)HERE. See also <a
href="http://stedolan.github.io/jq/">jq</a> for a truly powerful, JSON-specific
tool.

View file

@ -40,23 +40,33 @@ $3</tt> (adapted to name-based indexing), as are the variables <tt>FS</tt>,
functions, if you like) are stream-based: each of them maps a stream of records
into another stream of records.
<li/> Unlike <tt>awk</tt>, Miller doesn&rsquo;t allow you to define new functions.
Its domain-specific languages are limited to the <tt>filter</tt> and
<tt>put</tt> syntax. Futher programmability comes from chaining with
<tt>then</tt>.
<li/> Like <tt>awk</tt>, Miller (as of v5.0.0) allows you to define new
functions within its <tt>put</tt> and <tt>filter</tt> expression language.
Further programmability comes from chaining with <tt>then</tt>.
<li/> Unlike with <tt>awk</tt>, all variables are stream variables and all
functions are stream functions. This means <tt>NF</tt>, <tt>NR</tt>, etc.
change from one line to another, <tt>$x</tt> is a label for field <tt>x</tt> in
the current record, and the input to <tt>sqrt($x)</tt> changes from one record
to the next. Miller doesn&rsquo;t let you set <tt>sum=0</tt> before
records are read, then update that sum on each record, then print its value at the
end. (However, do see <tt>mlr step -a rsum</tt> in the
POKI_PUT_LINK_FOR_PAGE(reference.html)HERE) page.)
<li/> As with <tt>awk</tt>, <tt>$</tt>-variables are stream variables and all
verbs (such as <tt>cut</tt>, <tt>stats1</tt>, <tt>put</tt>, etc.) as well as
<tt>put</tt>/<tt>filter</tt> statements operate on streams. This means that
you define actions to be done on each record and then stream your data through
those actions. The built-in variables <tt>NF</tt>, <tt>NR</tt>, etc. change
from one line to another, <tt>$x</tt> is a label for field <tt>x</tt> in the
current record, and the input to <tt>sqrt($x)</tt> changes from one record to
the next. The expression language for the <tt>put</tt> and <tt>filter</tt>
verbs additionally allows you to define <tt>begin {...}</tt> and
<tt>end {...}</tt> blocks for actions to be taken before and after records are
processed, respectively.
<li/> Miller is faster than <tt>awk</tt>, <tt>cut</tt>, and so on (depending on
platform; see also POKI_PUT_LINK_FOR_PAGE(performance.html)HERE). In
particular, Miller&rsquo;s DSL syntax is parsed into C control structures at
<li/> As with <tt>awk</tt>, Miller&rsquo;s <tt>put</tt>/<tt>filter</tt>
language lets you set <tt>@sum=0</tt> before records are read, then update that
sum on each record, then print its value at the end. Unlike <tt>awk</tt>,
Miller makes syntactically explicit the difference between variables with
extent across all records (names starting with <tt>@</tt>, such as
<tt>@sum</tt>) and variables which are local to the current expression (names
starting without <tt>@</tt>, such as <tt>sum</tt>).
<li/> Miller can be faster than <tt>awk</tt>, <tt>cut</tt>, and so on,
depending on platform; see also POKI_PUT_LINK_FOR_PAGE(performance.html)HERE).
In particular, Miller&rsquo;s DSL syntax is parsed into C control structures at
startup time, with the bulk data-stream processing all done in C.
</ul>

View file

@ -1661,7 +1661,7 @@ notion of optional or default-on-absent arguments. All argument-passing is
positional rather than by name; arguments are passed by value, not by
reference.
POKI_RUN_COMMAND{{mlr --help-all-functions}}HERE
POKI_RUN_COMMAND{{mlr --help-all-functions | fmt -80}}HERE
<!-- ================================================================ -->
<h2>User-defined functions and subroutines</h2>

View file

@ -605,33 +605,33 @@ $ mlr --ofmt '%.9lf' --opprint seqgen --start 1 --stop 28 then put '
' then put '$seconds=systime()' then step -a delta -f seconds then cut -x -f seconds
i o fcount seconds_delta
1 1 1 0
2 2 3 0.000026941
3 3 5 0.000013113
2 2 3 0.000025034
3 3 5 0.000012875
4 5 9 0.000015974
5 8 15 0.000020981
6 13 25 0.000032902
5 8 15 0.000022173
6 13 25 0.000031948
7 21 41 0.000046015
8 34 67 0.000071049
9 55 109 0.000110865
10 89 177 0.000177145
11 144 287 0.000280857
12 233 465 0.000481129
13 377 753 0.000726938
14 610 1219 0.001166105
15 987 1973 0.001888037
16 1597 3193 0.003076792
17 2584 5167 0.007180214
18 4181 8361 0.009042978
19 6765 13529 0.012531996
20 10946 21891 0.020021915
21 17711 35421 0.039761066
22 28657 57313 0.057097912
23 46368 92735 0.089259148
24 75025 150049 0.147849798
25 121393 242785 0.231703997
26 196418 392835 0.374926090
27 317811 635621 0.696024895
28 514229 1028457 1.066029072
8 34 67 0.000069857
9 55 109 0.000111103
10 89 177 0.000175953
11 144 287 0.000282049
12 233 465 0.000449896
13 377 753 0.000727177
14 610 1219 0.001166821
15 987 1973 0.001890182
16 1597 3193 0.003028870
17 2584 5167 0.005067110
18 4181 8361 0.009028912
19 6765 13529 0.013338089
20 10946 21891 0.021298885
21 17711 35421 0.034317017
22 28657 57313 0.061784983
23 46368 92735 0.095272064
24 75025 150049 0.139719009
25 121393 242785 0.233756065
26 196418 392835 0.360270977
27 317811 635621 0.585737944
28 514229 1028457 0.969987869
</pre>
</div>
<p/>
@ -663,33 +663,33 @@ $ mlr --ofmt '%.9lf' --opprint seqgen --start 1 --stop 28 then put '
' then put '$seconds=systime()' then step -a delta -f seconds then cut -x -f seconds
i o fcount seconds_delta
1 1 1 0
2 2 3 0.000030041
3 3 3 0.000013828
4 5 3 0.000013113
5 8 3 0.000011921
6 13 3 0.000010967
7 21 3 0.000012159
8 34 3 0.000010967
2 2 3 0.000032902
3 3 3 0.000012875
4 5 3 0.000012159
5 8 3 0.000012875
6 13 3 0.000013113
7 21 3 0.000010967
8 34 3 0.000010014
9 55 3 0.000010967
10 89 3 0.000011921
10 89 3 0.000010967
11 144 3 0.000010967
12 233 3 0.000015020
13 377 3 0.000012159
14 610 3 0.000011921
13 377 3 0.000010967
14 610 3 0.000011206
15 987 3 0.000010967
16 1597 3 0.000010967
16 1597 3 0.000010014
17 2584 3 0.000010967
18 4181 3 0.000011206
19 6765 3 0.000010967
20 10946 3 0.000011921
21 17711 3 0.000010014
22 28657 3 0.000014067
23 46368 3 0.000013828
24 75025 3 0.000011206
18 4181 3 0.000015974
19 6765 3 0.000014067
20 10946 3 0.000010014
21 17711 3 0.000010967
22 28657 3 0.000013828
23 46368 3 0.000015974
24 75025 3 0.000013113
25 121393 3 0.000010967
26 196418 3 0.000011921
26 196418 3 0.000010967
27 317811 3 0.000010967
28 514229 3 0.000010967
28 514229 3 0.000010014
</pre>
</div>
<p/>

View file

@ -224,23 +224,33 @@ $3</tt> (adapted to name-based indexing), as are the variables <tt>FS</tt>,
functions, if you like) are stream-based: each of them maps a stream of records
into another stream of records.
<li/> Unlike <tt>awk</tt>, Miller doesn&rsquo;t allow you to define new functions.
Its domain-specific languages are limited to the <tt>filter</tt> and
<tt>put</tt> syntax. Futher programmability comes from chaining with
<tt>then</tt>.
<li/> Like <tt>awk</tt>, Miller (as of v5.0.0) allows you to define new
functions within its <tt>put</tt> and <tt>filter</tt> expression language.
Further programmability comes from chaining with <tt>then</tt>.
<li/> Unlike with <tt>awk</tt>, all variables are stream variables and all
functions are stream functions. This means <tt>NF</tt>, <tt>NR</tt>, etc.
change from one line to another, <tt>$x</tt> is a label for field <tt>x</tt> in
the current record, and the input to <tt>sqrt($x)</tt> changes from one record
to the next. Miller doesn&rsquo;t let you set <tt>sum=0</tt> before
records are read, then update that sum on each record, then print its value at the
end. (However, do see <tt>mlr step -a rsum</tt> in the
<a href="reference.html">Reference</a>) page.)
<li/> As with <tt>awk</tt>, <tt>$</tt>-variables are stream variables and all
verbs (such as <tt>cut</tt>, <tt>stats1</tt>, <tt>put</tt>, etc.) as well as
<tt>put</tt>/<tt>filter</tt> statements operate on streams. This means that
you define actions to be done on each record and then stream your data through
those actions. The built-in variables <tt>NF</tt>, <tt>NR</tt>, etc. change
from one line to another, <tt>$x</tt> is a label for field <tt>x</tt> in the
current record, and the input to <tt>sqrt($x)</tt> changes from one record to
the next. The expression language for the <tt>put</tt> and <tt>filter</tt>
verbs additionally allows you to define <tt>begin {...}</tt> and
<tt>end {...}</tt> blocks for actions to be taken before and after records are
processed, respectively.
<li/> Miller is faster than <tt>awk</tt>, <tt>cut</tt>, and so on (depending on
platform; see also <a href="performance.html">Performance</a>). In
particular, Miller&rsquo;s DSL syntax is parsed into C control structures at
<li/> As with <tt>awk</tt>, Miller&rsquo;s <tt>put</tt>/<tt>filter</tt>
language lets you set <tt>@sum=0</tt> before records are read, then update that
sum on each record, then print its value at the end. Unlike <tt>awk</tt>,
Miller makes syntactically explicit the difference between variables with
extent across all records (names starting with <tt>@</tt>, such as
<tt>@sum</tt>) and variables which are local to the current expression (names
starting without <tt>@</tt>, such as <tt>sum</tt>).
<li/> Miller can be faster than <tt>awk</tt>, <tt>cut</tt>, and so on,
depending on platform; see also <a href="performance.html">Performance</a>).
In particular, Miller&rsquo;s DSL syntax is parsed into C control structures at
startup time, with the bulk data-stream processing all done in C.
</ul>

View file

@ -6322,7 +6322,7 @@ reference.
<p/>
<div class="pokipanel">
<pre>
$ mlr --help-all-functions
$ mlr --help-all-functions | fmt -80
+ (class=arithmetic #args=2): Addition.
+ (class=arithmetic #args=1): Unary plus.
@ -6391,16 +6391,16 @@ and string results in string compare.
. (class=string #args=2): String concatenation.
gsub (class=string #args=3): Example: '$name=gsub($name, "old", "new")'
(replace all).
gsub (class=string #args=3): Example: '$name=gsub($name, "old", "new")' (replace
all).
strlen (class=string #args=1): String length.
sub (class=string #args=3): Example: '$name=sub($name, "old", "new")'
(replace once).
sub (class=string #args=3): Example: '$name=sub($name, "old", "new")' (replace
once).
substr (class=string #args=3): substr(s,m,n) gives substring of s from 0-up position m to n
inclusive. Negative indices -len .. -1 alias to 0 .. len-1.
substr (class=string #args=3): substr(s,m,n) gives substring of s from 0-up
position m to n inclusive. Negative indices -len .. -1 alias to 0 .. len-1.
tolower (class=string #args=1): Convert string to lowercase.
@ -6507,90 +6507,113 @@ fsec2dhms(500000.25) = "5d18h53m20.250000s"
fsec2hms (class=time #args=1): Formats floating-point seconds as in
fsec2hms(5000.25) = "01:23:20.250000"
gmt2sec (class=time #args=1): Parses GMT timestamp as integer seconds since
the epoch.
gmt2sec (class=time #args=1): Parses GMT timestamp as integer seconds since the
epoch.
hms2fsec (class=time #args=1): Recovers floating-point seconds as in
hms2fsec("01:23:20.250000") = 5000.250000
hms2sec (class=time #args=1): Recovers integer seconds as in
hms2sec("01:23:20") = 5000
hms2sec (class=time #args=1): Recovers integer seconds as in hms2sec("01:23:20")
= 5000
sec2dhms (class=time #args=1): Formats integer seconds as in sec2dhms(500000)
= "5d18h53m20s"
sec2dhms (class=time #args=1): Formats integer seconds as in sec2dhms(500000) =
"5d18h53m20s"
sec2gmt (class=time #args=1): Formats seconds since epoch (integer part)
as GMT timestamp, e.g. sec2gmt(1440768801.7) = "2015-08-28T13:33:21Z".
Leaves non-numbers as-is.
sec2gmt (class=time #args=1): Formats seconds since epoch (integer part) as GMT
timestamp, e.g. sec2gmt(1440768801.7) = "2015-08-28T13:33:21Z". Leaves
non-numbers as-is.
sec2gmtdate (class=time #args=1): Formats seconds since epoch (integer part)
as GMT timestamp with year-month-date, e.g. sec2gmtdate(1440768801.7) = "2015-08-28".
Leaves non-numbers as-is.
sec2gmtdate (class=time #args=1): Formats seconds since epoch (integer part) as
GMT timestamp with year-month-date, e.g. sec2gmtdate(1440768801.7) =
"2015-08-28". Leaves non-numbers as-is.
sec2hms (class=time #args=1): Formats integer seconds as in
sec2hms(5000) = "01:23:20"
sec2hms (class=time #args=1): Formats integer seconds as in sec2hms(5000) =
"01:23:20"
strftime (class=time #args=2): Formats seconds since epoch (integer part)
as timestamp, e.g.
strftime(1440768801.7,"%Y-%m-%dT%H:%M:%SZ") = "2015-08-28T13:33:21Z".
strftime (class=time #args=2): Formats seconds since epoch (integer part) as
timestamp, e.g. strftime(1440768801.7,"%Y-%m-%dT%H:%M:%SZ") =
"2015-08-28T13:33:21Z".
strptime (class=time #args=2): Parses timestamp as integer seconds since epoch,
e.g. strptime("2015-08-28T13:33:21Z","%Y-%m-%dT%H:%M:%SZ") = 1440768801.
systime (class=time #args=0): Floating-point seconds since the epoch,
e.g. 1440768801.748936.
systime (class=time #args=0): Floating-point seconds since the epoch, e.g.
1440768801.748936.
isabsent (class=typing #args=1): False if field is present in input, false otherwise
isabsent (class=typing #args=1): False if field is present in input, false
otherwise
isbool (class=typing #args=1): True if field is present with boolean value. Synonymous with isboolean.
isbool (class=typing #args=1): True if field is present with boolean value.
Synonymous with isboolean.
isboolean (class=typing #args=1): True if field is present with boolean value. Synonymous with isbool.
isboolean (class=typing #args=1): True if field is present with boolean value.
Synonymous with isbool.
isempty (class=typing #args=1): True if field is present in input with empty string value, false otherwise.
isempty (class=typing #args=1): True if field is present in input with empty
string value, false otherwise.
isemptymap (class=typing #args=1): True if argument is a map which is empty.
isfloat (class=typing #args=1): True if field is present with value inferred to be float
isfloat (class=typing #args=1): True if field is present with value inferred to
be float
isint (class=typing #args=1): True if field is present with value inferred to be int
isint (class=typing #args=1): True if field is present with value inferred to be
int
ismap (class=typing #args=1): True if argument is a map.
isnonemptymap (class=typing #args=1): True if argument is a map which is non-empty.
isnonemptymap (class=typing #args=1): True if argument is a map which is
non-empty.
isnotempty (class=typing #args=1): False if field is present in input with empty value, false otherwise
isnotempty (class=typing #args=1): False if field is present in input with empty
value, false otherwise
isnotnull (class=typing #args=1): False if argument is null (empty or absent), true otherwise.
isnotnull (class=typing #args=1): False if argument is null (empty or absent),
true otherwise.
isnull (class=typing #args=1): True if argument is null (empty or absent), false otherwise.
isnull (class=typing #args=1): True if argument is null (empty or absent), false
otherwise.
isnumeric (class=typing #args=1): True if field is present with value inferred to be int or float
isnumeric (class=typing #args=1): True if field is present with value inferred
to be int or float
ispresent (class=typing #args=1): True if field is present in input, false otherwise.
ispresent (class=typing #args=1): True if field is present in input, false
otherwise.
isscalar (class=typing #args=1): True if argument is not a map.
isstring (class=typing #args=1): True if field is present with string (including empty-string) value
isstring (class=typing #args=1): True if field is present with string (including
empty-string) value
assert_notnull (class=typing #args=1): Returns argument if non-null (non-empty and non-absent), else throws an error.
assert_notnull (class=typing #args=1): Returns argument if non-null (non-empty
and non-absent), else throws an error.
assert_present (class=typing #args=1): Returns argument if present in input, else throws an error.
assert_present (class=typing #args=1): Returns argument if present in input,
else throws an error.
assert_empty (class=typing #args=1): Returns argument if present in input with empty value, else throws an error.
assert_empty (class=typing #args=1): Returns argument if present in input with
empty value, else throws an error.
assert_notempty (class=typing #args=1): Returns argument if present in input with non-empty value, else throws an error.
assert_notempty (class=typing #args=1): Returns argument if present in input
with non-empty value, else throws an error.
assert_numeric (class=typing #args=1): Returns argument if present with int or float value, else throws an error.
assert_numeric (class=typing #args=1): Returns argument if present with int or
float value, else throws an error.
assert_int (class=typing #args=1): Returns argument if present with int value, else throws an error.
assert_int (class=typing #args=1): Returns argument if present with int value,
else throws an error.
assert_float (class=typing #args=1): Returns argument if present with float value, else throws an error.
assert_float (class=typing #args=1): Returns argument if present with float
value, else throws an error.
assert_bool (class=typing #args=1): Returns argument if present with boolean value, else throws an error.
assert_bool (class=typing #args=1): Returns argument if present with boolean
value, else throws an error.
assert_boolean (class=typing #args=1): Returns argument if present with boolean value, else throws an error.
assert_boolean (class=typing #args=1): Returns argument if present with boolean
value, else throws an error.
assert_string (class=typing #args=1): Returns argument if present with string (including empty-string) value, else throws an error.
assert_string (class=typing #args=1): Returns argument if present with string
(including empty-string) value, else throws an error.
boolean (class=conversion #args=1): Convert int/float/bool/string to boolean.
@ -6608,32 +6631,44 @@ string (class=conversion #args=1): Convert int/float/bool/string to string.
typeof (class=conversion #args=1): Convert argument to type of argument (e.g.
MT_STRING). For debug.
depth (class=maps #args=1): Prints maximum depth of hashmap: ''. Scalars have depth 0.
depth (class=maps #args=1): Prints maximum depth of hashmap: ''. Scalars have
depth 0.
haskey (class=maps #args=2): True/false if map has/hasn't key, e.g. 'haskey($*, "a")' or 'haskey(mymap, mykey)'. Error if 1st argument is not a map.
haskey (class=maps #args=2): True/false if map has/hasn't key, e.g. 'haskey($*,
"a")' or 'haskey(mymap, mykey)'. Error if 1st argument is not a map.
joink (class=maps #args=2): Makes string from map keys. E.g. 'joink($*, ",")'.
joinkv (class=maps #args=3): Makes string from map key-value pairs. E.g. 'joinkv(@v[2], "=", ",")'
joinkv (class=maps #args=3): Makes string from map key-value pairs. E.g.
'joinkv(@v[2], "=", ",")'
joinv (class=maps #args=2): Makes string from map keys. E.g. 'joinv(mymap, ",")'.
joinv (class=maps #args=2): Makes string from map keys. E.g. 'joinv(mymap,
",")'.
leafcount (class=maps #args=1): Counts total number of terminal values in hashmap. For single-level maps, same as length.
leafcount (class=maps #args=1): Counts total number of terminal values in
hashmap. For single-level maps, same as length.
length (class=maps #args=1): Counts number of top-level entries in hashmap. Scalars have length 1.
length (class=maps #args=1): Counts number of top-level entries in hashmap.
Scalars have length 1.
mapdiff (class=maps variadic): With 0 args, returns empty map. With 1 arg, returns copy of arg. With 2 or more, returns copy of arg 1 with all keys from any of remaining argument maps removed.
mapdiff (class=maps variadic): With 0 args, returns empty map. With 1 arg,
returns copy of arg. With 2 or more, returns copy of arg 1 with all keys from
any of remaining argument maps removed.
mapsum (class=maps variadic): With 0 args, returns empty map. With &gt;= 1 arg, returns a map with key-value pairs from all arguments. Rightmost collisions win, e.g. 'mapsum({1:2,3,4},{1:5})' is '{1:5,3:4}'.
mapsum (class=maps variadic): With 0 args, returns empty map. With &gt;= 1 arg,
returns a map with key-value pairs from all arguments. Rightmost collisions win,
e.g. 'mapsum({1:2,3,4},{1:5})' is '{1:5,3:4}'.
splitkv (class=maps #args=3): Splits string by separators into map. E.g. 'splitkv("a=1,b=2,c=3", "=", ",")' gives '{"a" : 1, "b" : 2, "c" : 3}'.
splitkv (class=maps #args=3): Splits string by separators into map. E.g.
'splitkv("a=1,b=2,c=3", "=", ",")' gives '{"a" : 1, "b" : 2, "c" : 3}'.
splitnv (class=maps #args=2): Splits string by separator into integer-indexed map. E.g. 'splitnv("a,b,c" , ",")' gives '{1 : "a", 2 : "b", 3 : "c"}'.
splitnv (class=maps #args=2): Splits string by separator into integer-indexed
map. E.g. 'splitnv("a,b,c" , ",")' gives '{1 : "a", 2 : "b", 3 : "c"}'.
To set the seed for urand, you may specify decimal or hexadecimal 32-bit
numbers of the form "mlr --seed 123456789" or "mlr --seed 0xcafefeed".
Miller's built-in variables are NF, NR, FNR, FILENUM, and FILENAME (awk-like)
along with the mathematical constants PI and E.
To set the seed for urand, you may specify decimal or hexadecimal 32-bit numbers
of the form "mlr --seed 123456789" or "mlr --seed 0xcafefeed". Miller's
built-in variables are NF, NR, FNR, FILENUM, and FILENAME (awk-like) along with
the mathematical constants PI and E.
</pre>
</div>
<p/>