diff --git a/c/todo.txt b/c/todo.txt index 112278ecf..c3bf085b5 100644 --- a/c/todo.txt +++ b/c/todo.txt @@ -22,7 +22,6 @@ OTHER: * mlr paste (sjackman) -* don't dump full help on unrecognized flag. WAY too much info. just print short, w/ info on how to get long. * dump @records after @records[NR]=$*: json "1" rather than 1 indices are b04k3d ! 'Error code 1' -> clearer on regdiff fail @@ -41,13 +40,15 @@ MAPVAR CHECKLIST: ! this should throw an error: mlr --from s put 'int a=1;var b=a[2]' +? on-line help to separate web page, as w/ manpage output? + * emittable/dumpable/etc md_rhs -> more specific fcn-call. +* modify isx()-predicate-doc @ mlr -f to acknowledge maps. also, UT. + * death knell for mv-only evaluators: this is a syntax error: mlr put -q 'o = haskey(NR==4 ? {"a": NF} : {"b": NF}, "a"); print "NR=".NR.",haskeya=".o' -* modify isx() predicate doc @ mlr -f to acknowledge maps. also, UT. - * debt-reduction ! naming conventions to clarify transfer/copy semantics - copy/ref of keys vs. values in mlhmmv, local_stack, and callees -- ?!? @@ -83,8 +84,6 @@ MAPVAR CHECKLIST: * bool_t:uchar typedef & perf check -* mv_error retvals vs. exitthrows vs. absentskips ... bleah - * man mlr section on maps o things which are always maps: - maplit @@ -96,6 +95,7 @@ MAPVAR CHECKLIST: - func retval - func args o what funcs are available + o scalar contexts vs. map contexts o keywords print, dump, emit, for-kv, for-k o copy semantics diff --git a/doc/content-for-faq.html b/doc/content-for-faq.html index acb0b0c5d..91286116e 100644 --- a/doc/content-for-faq.html +++ b/doc/content-for-faq.html @@ -210,7 +210,7 @@ may be easy to grep/sed out the data into a simpler text form — this is a general text-processing problem.

Miller does support tabular data represented in JSON: please see -POKI_PUT_LINK_FOR_PAGE(file-formats.html)HERE. See als jq for a truly powerful, JSON-specific tool. diff --git a/doc/content-for-feature-comparison.html b/doc/content-for-feature-comparison.html index 4fd343590..9c3de3c3b 100644 --- a/doc/content-for-feature-comparison.html +++ b/doc/content-for-feature-comparison.html @@ -40,23 +40,33 @@ $3 (adapted to name-based indexing), as are the variables FS, functions, if you like) are stream-based: each of them maps a stream of records into another stream of records. -

  • Unlike awk, Miller doesn’t allow you to define new functions. -Its domain-specific languages are limited to the filter and -put syntax. Futher programmability comes from chaining with -then. +
  • Like awk, Miller (as of v5.0.0) allows you to define new +functions within its put and filter expression language. +Further programmability comes from chaining with then. -
  • Unlike with awk, all variables are stream variables and all -functions are stream functions. This means NF, NR, etc. -change from one line to another, $x is a label for field x in -the current record, and the input to sqrt($x) changes from one record -to the next. Miller doesn’t let you set sum=0 before -records are read, then update that sum on each record, then print its value at the -end. (However, do see mlr step -a rsum in the -POKI_PUT_LINK_FOR_PAGE(reference.html)HERE) page.) +
  • As with awk, $-variables are stream variables and all +verbs (such as cut, stats1, put, etc.) as well as +put/filter statements operate on streams. This means that +you define actions to be done on each record and then stream your data through +those actions. The built-in variables NF, NR, etc. change +from one line to another, $x is a label for field x in the +current record, and the input to sqrt($x) changes from one record to +the next. The expression language for the put and filter +verbs additionally allows you to define begin {...} and +end {...} blocks for actions to be taken before and after records are +processed, respectively. -
  • Miller is faster than awk, cut, and so on (depending on -platform; see also POKI_PUT_LINK_FOR_PAGE(performance.html)HERE). In -particular, Miller’s DSL syntax is parsed into C control structures at +
  • As with awk, Miller’s put/filter +language lets you set @sum=0 before records are read, then update that +sum on each record, then print its value at the end. Unlike awk, +Miller makes syntactically explicit the difference between variables with +extent across all records (names starting with @, such as +@sum) and variables which are local to the current expression (names +starting without @, such as sum). + +
  • Miller can be faster than awk, cut, and so on, +depending on platform; see also POKI_PUT_LINK_FOR_PAGE(performance.html)HERE). +In particular, Miller’s DSL syntax is parsed into C control structures at startup time, with the bulk data-stream processing all done in C. diff --git a/doc/content-for-reference.html b/doc/content-for-reference.html index dd42ce806..4f7d0ffaf 100644 --- a/doc/content-for-reference.html +++ b/doc/content-for-reference.html @@ -1661,7 +1661,7 @@ notion of optional or default-on-absent arguments. All argument-passing is positional rather than by name; arguments are passed by value, not by reference. -POKI_RUN_COMMAND{{mlr --help-all-functions}}HERE +POKI_RUN_COMMAND{{mlr --help-all-functions | fmt -80}}HERE

    User-defined functions and subroutines

    diff --git a/doc/cookbook.html b/doc/cookbook.html index 43041fc46..3ddf77137 100644 --- a/doc/cookbook.html +++ b/doc/cookbook.html @@ -605,33 +605,33 @@ $ mlr --ofmt '%.9lf' --opprint seqgen --start 1 --stop 28 then put ' ' then put '$seconds=systime()' then step -a delta -f seconds then cut -x -f seconds i o fcount seconds_delta 1 1 1 0 -2 2 3 0.000026941 -3 3 5 0.000013113 +2 2 3 0.000025034 +3 3 5 0.000012875 4 5 9 0.000015974 -5 8 15 0.000020981 -6 13 25 0.000032902 +5 8 15 0.000022173 +6 13 25 0.000031948 7 21 41 0.000046015 -8 34 67 0.000071049 -9 55 109 0.000110865 -10 89 177 0.000177145 -11 144 287 0.000280857 -12 233 465 0.000481129 -13 377 753 0.000726938 -14 610 1219 0.001166105 -15 987 1973 0.001888037 -16 1597 3193 0.003076792 -17 2584 5167 0.007180214 -18 4181 8361 0.009042978 -19 6765 13529 0.012531996 -20 10946 21891 0.020021915 -21 17711 35421 0.039761066 -22 28657 57313 0.057097912 -23 46368 92735 0.089259148 -24 75025 150049 0.147849798 -25 121393 242785 0.231703997 -26 196418 392835 0.374926090 -27 317811 635621 0.696024895 -28 514229 1028457 1.066029072 +8 34 67 0.000069857 +9 55 109 0.000111103 +10 89 177 0.000175953 +11 144 287 0.000282049 +12 233 465 0.000449896 +13 377 753 0.000727177 +14 610 1219 0.001166821 +15 987 1973 0.001890182 +16 1597 3193 0.003028870 +17 2584 5167 0.005067110 +18 4181 8361 0.009028912 +19 6765 13529 0.013338089 +20 10946 21891 0.021298885 +21 17711 35421 0.034317017 +22 28657 57313 0.061784983 +23 46368 92735 0.095272064 +24 75025 150049 0.139719009 +25 121393 242785 0.233756065 +26 196418 392835 0.360270977 +27 317811 635621 0.585737944 +28 514229 1028457 0.969987869

    @@ -663,33 +663,33 @@ $ mlr --ofmt '%.9lf' --opprint seqgen --start 1 --stop 28 then put ' ' then put '$seconds=systime()' then step -a delta -f seconds then cut -x -f seconds i o fcount seconds_delta 1 1 1 0 -2 2 3 0.000030041 -3 3 3 0.000013828 -4 5 3 0.000013113 -5 8 3 0.000011921 -6 13 3 0.000010967 -7 21 3 0.000012159 -8 34 3 0.000010967 +2 2 3 0.000032902 +3 3 3 0.000012875 +4 5 3 0.000012159 +5 8 3 0.000012875 +6 13 3 0.000013113 +7 21 3 0.000010967 +8 34 3 0.000010014 9 55 3 0.000010967 -10 89 3 0.000011921 +10 89 3 0.000010967 11 144 3 0.000010967 12 233 3 0.000015020 -13 377 3 0.000012159 -14 610 3 0.000011921 +13 377 3 0.000010967 +14 610 3 0.000011206 15 987 3 0.000010967 -16 1597 3 0.000010967 +16 1597 3 0.000010014 17 2584 3 0.000010967 -18 4181 3 0.000011206 -19 6765 3 0.000010967 -20 10946 3 0.000011921 -21 17711 3 0.000010014 -22 28657 3 0.000014067 -23 46368 3 0.000013828 -24 75025 3 0.000011206 +18 4181 3 0.000015974 +19 6765 3 0.000014067 +20 10946 3 0.000010014 +21 17711 3 0.000010967 +22 28657 3 0.000013828 +23 46368 3 0.000015974 +24 75025 3 0.000013113 25 121393 3 0.000010967 -26 196418 3 0.000011921 +26 196418 3 0.000010967 27 317811 3 0.000010967 -28 514229 3 0.000010967 +28 514229 3 0.000010014

    diff --git a/doc/feature-comparison.html b/doc/feature-comparison.html index 255952bf8..fd8497753 100644 --- a/doc/feature-comparison.html +++ b/doc/feature-comparison.html @@ -224,23 +224,33 @@ $3 (adapted to name-based indexing), as are the variables FS, functions, if you like) are stream-based: each of them maps a stream of records into another stream of records. -

  • Unlike awk, Miller doesn’t allow you to define new functions. -Its domain-specific languages are limited to the filter and -put syntax. Futher programmability comes from chaining with -then. +
  • Like awk, Miller (as of v5.0.0) allows you to define new +functions within its put and filter expression language. +Further programmability comes from chaining with then. -
  • Unlike with awk, all variables are stream variables and all -functions are stream functions. This means NF, NR, etc. -change from one line to another, $x is a label for field x in -the current record, and the input to sqrt($x) changes from one record -to the next. Miller doesn’t let you set sum=0 before -records are read, then update that sum on each record, then print its value at the -end. (However, do see mlr step -a rsum in the -Reference) page.) +
  • As with awk, $-variables are stream variables and all +verbs (such as cut, stats1, put, etc.) as well as +put/filter statements operate on streams. This means that +you define actions to be done on each record and then stream your data through +those actions. The built-in variables NF, NR, etc. change +from one line to another, $x is a label for field x in the +current record, and the input to sqrt($x) changes from one record to +the next. The expression language for the put and filter +verbs additionally allows you to define begin {...} and +end {...} blocks for actions to be taken before and after records are +processed, respectively. -
  • Miller is faster than awk, cut, and so on (depending on -platform; see also Performance). In -particular, Miller’s DSL syntax is parsed into C control structures at +
  • As with awk, Miller’s put/filter +language lets you set @sum=0 before records are read, then update that +sum on each record, then print its value at the end. Unlike awk, +Miller makes syntactically explicit the difference between variables with +extent across all records (names starting with @, such as +@sum) and variables which are local to the current expression (names +starting without @, such as sum). + +
  • Miller can be faster than awk, cut, and so on, +depending on platform; see also Performance). +In particular, Miller’s DSL syntax is parsed into C control structures at startup time, with the bulk data-stream processing all done in C. diff --git a/doc/reference.html b/doc/reference.html index 6066ce00e..5a0fea85c 100644 --- a/doc/reference.html +++ b/doc/reference.html @@ -6322,7 +6322,7 @@ reference. 

    -$ mlr --help-all-functions
+$ mlr --help-all-functions | fmt -80
 + (class=arithmetic #args=2): Addition.
 
 + (class=arithmetic #args=1): Unary plus.
@@ -6391,16 +6391,16 @@ and string results in string compare.
 
 . (class=string #args=2): String concatenation.
 
-gsub (class=string #args=3): Example: '$name=gsub($name, "old", "new")'
-(replace all).
+gsub (class=string #args=3): Example: '$name=gsub($name, "old", "new")' (replace
+all).
 
 strlen (class=string #args=1): String length.
 
-sub (class=string #args=3): Example: '$name=sub($name, "old", "new")'
-(replace once).
+sub (class=string #args=3): Example: '$name=sub($name, "old", "new")' (replace
+once).
 
-substr (class=string #args=3): substr(s,m,n) gives substring of s from 0-up position m to n
-inclusive. Negative indices -len .. -1 alias to 0 .. len-1.
+substr (class=string #args=3): substr(s,m,n) gives substring of s from 0-up
+position m to n inclusive. Negative indices -len .. -1 alias to 0 .. len-1.
 
 tolower (class=string #args=1): Convert string to lowercase.
 
@@ -6507,90 +6507,113 @@ fsec2dhms(500000.25) = "5d18h53m20.250000s"
 fsec2hms (class=time #args=1): Formats floating-point seconds as in
 fsec2hms(5000.25) = "01:23:20.250000"
 
-gmt2sec (class=time #args=1): Parses GMT timestamp as integer seconds since
-the epoch.
+gmt2sec (class=time #args=1): Parses GMT timestamp as integer seconds since the
+epoch.
 
 hms2fsec (class=time #args=1): Recovers floating-point seconds as in
 hms2fsec("01:23:20.250000") = 5000.250000
 
-hms2sec (class=time #args=1): Recovers integer seconds as in
-hms2sec("01:23:20") = 5000
+hms2sec (class=time #args=1): Recovers integer seconds as in hms2sec("01:23:20")
+= 5000
 
-sec2dhms (class=time #args=1): Formats integer seconds as in sec2dhms(500000)
-= "5d18h53m20s"
+sec2dhms (class=time #args=1): Formats integer seconds as in sec2dhms(500000) =
+"5d18h53m20s"
 
-sec2gmt (class=time #args=1): Formats seconds since epoch (integer part)
-as GMT timestamp, e.g. sec2gmt(1440768801.7) = "2015-08-28T13:33:21Z".
-Leaves non-numbers as-is.
+sec2gmt (class=time #args=1): Formats seconds since epoch (integer part) as GMT
+timestamp, e.g. sec2gmt(1440768801.7) = "2015-08-28T13:33:21Z".  Leaves
+non-numbers as-is.
 
-sec2gmtdate (class=time #args=1): Formats seconds since epoch (integer part)
-as GMT timestamp with year-month-date, e.g. sec2gmtdate(1440768801.7) = "2015-08-28".
-Leaves non-numbers as-is.
+sec2gmtdate (class=time #args=1): Formats seconds since epoch (integer part) as
+GMT timestamp with year-month-date, e.g. sec2gmtdate(1440768801.7) =
+"2015-08-28".  Leaves non-numbers as-is.
 
-sec2hms (class=time #args=1): Formats integer seconds as in
-sec2hms(5000) = "01:23:20"
+sec2hms (class=time #args=1): Formats integer seconds as in sec2hms(5000) =
+"01:23:20"
 
-strftime (class=time #args=2): Formats seconds since epoch (integer part)
-as timestamp, e.g.
-strftime(1440768801.7,"%Y-%m-%dT%H:%M:%SZ") = "2015-08-28T13:33:21Z".
+strftime (class=time #args=2): Formats seconds since epoch (integer part) as
+timestamp, e.g.  strftime(1440768801.7,"%Y-%m-%dT%H:%M:%SZ") =
+"2015-08-28T13:33:21Z".
 
 strptime (class=time #args=2): Parses timestamp as integer seconds since epoch,
 e.g. strptime("2015-08-28T13:33:21Z","%Y-%m-%dT%H:%M:%SZ") = 1440768801.
 
-systime (class=time #args=0): Floating-point seconds since the epoch,
-e.g. 1440768801.748936.
+systime (class=time #args=0): Floating-point seconds since the epoch, e.g.
+1440768801.748936.
 
-isabsent (class=typing #args=1): False if field is present in input, false otherwise
+isabsent (class=typing #args=1): False if field is present in input, false
+otherwise
 
-isbool (class=typing #args=1): True if field is present with boolean value. Synonymous with isboolean.
+isbool (class=typing #args=1): True if field is present with boolean value.
+Synonymous with isboolean.
 
-isboolean (class=typing #args=1): True if field is present with boolean value. Synonymous with isbool.
+isboolean (class=typing #args=1): True if field is present with boolean value.
+Synonymous with isbool.
 
-isempty (class=typing #args=1): True if field is present in input with empty string value, false otherwise.
+isempty (class=typing #args=1): True if field is present in input with empty
+string value, false otherwise.
 
 isemptymap (class=typing #args=1): True if argument is a map which is empty.
 
-isfloat (class=typing #args=1): True if field is present with value inferred to be float
+isfloat (class=typing #args=1): True if field is present with value inferred to
+be float
 
-isint (class=typing #args=1): True if field is present with value inferred to be int
+isint (class=typing #args=1): True if field is present with value inferred to be
+int
 
 ismap (class=typing #args=1): True if argument is a map.
 
-isnonemptymap (class=typing #args=1): True if argument is a map which is non-empty.
+isnonemptymap (class=typing #args=1): True if argument is a map which is
+non-empty.
 
-isnotempty (class=typing #args=1): False if field is present in input with empty value, false otherwise
+isnotempty (class=typing #args=1): False if field is present in input with empty
+value, false otherwise
 
-isnotnull (class=typing #args=1): False if argument is null (empty or absent), true otherwise.
+isnotnull (class=typing #args=1): False if argument is null (empty or absent),
+true otherwise.
 
-isnull (class=typing #args=1): True if argument is null (empty or absent), false otherwise.
+isnull (class=typing #args=1): True if argument is null (empty or absent), false
+otherwise.
 
-isnumeric (class=typing #args=1): True if field is present with value inferred to be int or float
+isnumeric (class=typing #args=1): True if field is present with value inferred
+to be int or float
 
-ispresent (class=typing #args=1): True if field is present in input, false otherwise.
+ispresent (class=typing #args=1): True if field is present in input, false
+otherwise.
 
 isscalar (class=typing #args=1): True if argument is not a map.
 
-isstring (class=typing #args=1): True if field is present with string (including empty-string) value
+isstring (class=typing #args=1): True if field is present with string (including
+empty-string) value
 
-assert_notnull (class=typing #args=1): Returns argument if non-null (non-empty and non-absent), else throws an error.
+assert_notnull (class=typing #args=1): Returns argument if non-null (non-empty
+and non-absent), else throws an error.
 
-assert_present (class=typing #args=1): Returns argument if present in input, else throws an error.
+assert_present (class=typing #args=1): Returns argument if present in input,
+else throws an error.
 
-assert_empty (class=typing #args=1): Returns argument if present in input with empty value, else throws an error.
+assert_empty (class=typing #args=1): Returns argument if present in input with
+empty value, else throws an error.
 
-assert_notempty (class=typing #args=1): Returns argument if present in input with non-empty value, else throws an error.
+assert_notempty (class=typing #args=1): Returns argument if present in input
+with non-empty value, else throws an error.
 
-assert_numeric (class=typing #args=1): Returns argument if present with int or float value, else throws an error.
+assert_numeric (class=typing #args=1): Returns argument if present with int or
+float value, else throws an error.
 
-assert_int (class=typing #args=1): Returns argument if present with int value, else throws an error.
+assert_int (class=typing #args=1): Returns argument if present with int value,
+else throws an error.
 
-assert_float (class=typing #args=1): Returns argument if present with float value, else throws an error.
+assert_float (class=typing #args=1): Returns argument if present with float
+value, else throws an error.
 
-assert_bool (class=typing #args=1): Returns argument if present with boolean value, else throws an error.
+assert_bool (class=typing #args=1): Returns argument if present with boolean
+value, else throws an error.
 
-assert_boolean (class=typing #args=1): Returns argument if present with boolean value, else throws an error.
+assert_boolean (class=typing #args=1): Returns argument if present with boolean
+value, else throws an error.
 
-assert_string (class=typing #args=1): Returns argument if present with string (including empty-string) value, else throws an error.
+assert_string (class=typing #args=1): Returns argument if present with string
+(including empty-string) value, else throws an error.
 
 boolean (class=conversion #args=1): Convert int/float/bool/string to boolean.
 
@@ -6608,32 +6631,44 @@ string (class=conversion #args=1): Convert int/float/bool/string to string.
 typeof (class=conversion #args=1): Convert argument to type of argument (e.g.
 MT_STRING). For debug.
 
-depth (class=maps #args=1): Prints maximum depth of hashmap: ''. Scalars have depth 0.
+depth (class=maps #args=1): Prints maximum depth of hashmap: ''. Scalars have
+depth 0.
 
-haskey (class=maps #args=2): True/false if map has/hasn't key, e.g. 'haskey($*, "a")' or 'haskey(mymap, mykey)'. Error if 1st argument is not a map.
+haskey (class=maps #args=2): True/false if map has/hasn't key, e.g. 'haskey($*,
+"a")' or 'haskey(mymap, mykey)'. Error if 1st argument is not a map.
 
 joink (class=maps #args=2): Makes string from map keys. E.g. 'joink($*, ",")'.
 
-joinkv (class=maps #args=3): Makes string from map key-value pairs. E.g. 'joinkv(@v[2], "=", ",")'
+joinkv (class=maps #args=3): Makes string from map key-value pairs. E.g.
+'joinkv(@v[2], "=", ",")'
 
-joinv (class=maps #args=2): Makes string from map keys. E.g. 'joinv(mymap, ",")'.
+joinv (class=maps #args=2): Makes string from map keys. E.g. 'joinv(mymap,
+",")'.
 
-leafcount (class=maps #args=1): Counts total number of terminal values in hashmap. For single-level maps, same as length.
+leafcount (class=maps #args=1): Counts total number of terminal values in
+hashmap. For single-level maps, same as length.
 
-length (class=maps #args=1): Counts number of top-level entries in hashmap. Scalars have length 1.
+length (class=maps #args=1): Counts number of top-level entries in hashmap.
+Scalars have length 1.
 
-mapdiff (class=maps variadic): With 0 args, returns empty map. With 1 arg, returns copy of arg. With 2 or more, returns copy of arg 1 with all keys from any of remaining argument maps removed.
+mapdiff (class=maps variadic): With 0 args, returns empty map. With 1 arg,
+returns copy of arg. With 2 or more, returns copy of arg 1 with all keys from
+any of remaining argument maps removed.
 
-mapsum (class=maps variadic): With 0 args, returns empty map. With >= 1 arg, returns a map with key-value pairs from all arguments. Rightmost collisions win, e.g. 'mapsum({1:2,3,4},{1:5})' is '{1:5,3:4}'.
+mapsum (class=maps variadic): With 0 args, returns empty map. With >= 1 arg,
+returns a map with key-value pairs from all arguments. Rightmost collisions win,
+e.g. 'mapsum({1:2,3,4},{1:5})' is '{1:5,3:4}'.
 
-splitkv (class=maps #args=3): Splits string by separators into map. E.g. 'splitkv("a=1,b=2,c=3", "=", ",")' gives '{"a" : 1, "b" : 2, "c" : 3}'.
+splitkv (class=maps #args=3): Splits string by separators into map. E.g.
+'splitkv("a=1,b=2,c=3", "=", ",")' gives '{"a" : 1, "b" : 2, "c" : 3}'.
 
-splitnv (class=maps #args=2): Splits string by separator into integer-indexed map. E.g. 'splitnv("a,b,c" , ",")' gives '{1 : "a", 2 : "b", 3 : "c"}'.
+splitnv (class=maps #args=2): Splits string by separator into integer-indexed
+map. E.g. 'splitnv("a,b,c" , ",")' gives '{1 : "a", 2 : "b", 3 : "c"}'.
 
-To set the seed for urand, you may specify decimal or hexadecimal 32-bit
-numbers of the form "mlr --seed 123456789" or "mlr --seed 0xcafefeed".
-Miller's built-in variables are NF, NR, FNR, FILENUM, and FILENAME (awk-like)
-along with the mathematical constants PI and E.
+To set the seed for urand, you may specify decimal or hexadecimal 32-bit numbers
+of the form "mlr --seed 123456789" or "mlr --seed 0xcafefeed".  Miller's
+built-in variables are NF, NR, FNR, FILENUM, and FILENAME (awk-like) along with
+the mathematical constants PI and E.