Mirrors/miller
1
0
Fork 0
mirror of https://github.com/johnkerl/miller.git synced 2026-07-17 16:38:54 +00:00
Code Issues Projects Releases Wiki Activity

Add a first-class bytes type to the DSL, with b"..." literals and base64/hex codecs (#2122)

Browse source 
* Add MT_BYTES mlrval type: foundation and disposition tables

First step toward a first-class bytes type in the DSL (#1231).
Adds MT_BYTES (payload []byte, rendered as lowercase hex in all output
formats, JSON-encoded as a hex string), extends every disposition
matrix/vector with the new row/column -- real cells for comparison,
sorting, and dot-concat of bytes with bytes; type-error stubs
elsewhere -- and adds sweep tests asserting no table has nil cells,
since Go zero-fills short array literals when MT_DIM grows.

Bytes values are not yet constructible from the DSL; b"..." literals
and constructor/codec functions follow in subsequent commits.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Add b"..." bytes-literal syntax to the DSL

Adds a bytes_literal token to the grammar (regenerating the PGPG lexer
and parser) and a BytesLiteralNode in the CST which evaluates to an
MT_BYTES mlrval. Escape handling reuses UnbackslashStringLiteral,
which is already byte-oriented: b"\xff" is the single byte 0xff.
Unlike string literals, bytes literals never participate in
regex-capture replacement. A bare identifier b is unaffected.

Part of #1231.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Add bytes DSL functions: conversions, codecs, and bytes-aware built-ins

- bytes(x) converts strings to bytes; string(b) reinterprets raw bytes
  as UTF-8 text (the reverse)
- base64_decode now always returns bytes (superseding the interim
  string-or-hex behavior); base64_encode accepts string or bytes
- New hex_encode/hex_decode functions
- is_bytes and asserting_bytes predicates
- md5/sha1/sha256/sha512 accept bytes, hashing the raw payload
- strlen of bytes is the byte count; substr/substr0/substr1 on bytes
  slice by byte position and return bytes

The Cyrillic-LDAP scenario from #1231 now works without exec
workarounds: string(base64_decode($x)) recovers the text, and binary
payloads survive undamaged as bytes.

Closes #1231.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Add bytes-type docs and regression cases

Documents the bytes type on the data-types page, regenerates the
function-reference/man-page material, and adds regression coverage:
literal escape forms, operators (concat/compare/slice/sort and
type errors), conversions and codec round-trips, and CSV-to-JSON
output rendering of bytes fields.

Part of #1231.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Reposition MT_BYTES to sort adjacent to MT_STRING in the type enum

MT_BYTES was appended after MT_ABSENT for index stability; move it
right after MT_STRING instead, since that's where it conceptually
belongs and where it already sorts in the cmp disposition matrices.
Mechanically re-derive all ~40 disposition tables in pkg/bifs and
pkg/mlrval accordingly.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix windows CI

* fix merge

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
John Kerl 2026-07-03 11:58:44 -04:00 • committed by GitHub
parent 91eaff1341
commit 12c96298b9
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
58 changed files with 643579 additions and 624651 deletions
Download patch file Download diff file Expand all files Collapse all files

106
docs/src/manpage.md
View file

@ -236,34 +236,35 @@ This is simply a copy of what you should see on running `man mlr` at a command p
1mFUNCTION LIST0m
abs acos acosh antimode any append apply arrayify asin asinh asserting_absent
asserting_array asserting_bool asserting_boolean asserting_empty
asserting_empty_map asserting_error asserting_float asserting_int
asserting_map asserting_nonempty_map asserting_not_array asserting_not_empty
asserting_not_map asserting_not_null asserting_null asserting_numeric
asserting_present asserting_string atan atan2 atanh bitcount boolean
capitalize cbrt ceil clean_whitespace collapse_whitespace concat contains cos
cosh count depth dhms2fsec dhms2sec distinct_count erf erfc every exec exp
expm1 flatten float floor fmtifnum fmtnum fold format fsec2dhms fsec2hms
get_keys get_values gmt2localtime gmt2nsec gmt2sec gssub gsub haskey hasvalue
hexfmt hms2fsec hms2sec hostname index int invqnorm is_absent is_array is_bool
is_boolean is_empty is_empty_map is_error is_float is_int is_map is_nan
is_nonempty_map is_not_array is_not_empty is_not_map is_not_null is_null
is_numeric is_present is_string joink joinkv joinv json_parse json_stringify
kurtosis latin1_to_utf8 leafcount leftpad length localtime2gmt localtime2nsec
localtime2sec log log10 log1p logifit lstrip madd mapdiff mapexcept mapselect
mapsum max maxlen md5 mean meaneb median mexp min minlen mmul mode msub next
nsec2gmt nsec2gmtdate nsec2localdate nsec2localtime null_count os percentile
percentiles pow qnorm reduce regextract regextract_or_else rightpad round
roundm rstrip sec2dhms sec2gmt sec2gmtdate sec2hms sec2localdate sec2localtime
select sgn sha1 sha256 sha512 sin sinh skewness sort sort_collection splita
splitax splitkv splitkvx splitnv splitnvx sqrt ssub stat stddev strfntime
strfntime_local strftime strftime_local string strip strlen strmatch strmatchx
strpntime strpntime_local strptime strptime_local sub substr substr0 substr1
sum sum2 sum3 sum4 sysntime system systime systimeint tan tanh tolower toupper
truncate typeof unflatten unformat unformatx upntime uptime urand urand32
urandelement urandint urandrange utf8_to_latin1 variance version ! != !=~ % &
&& * ** + - . .* .+ .- ./ / // &lt; &lt;&lt; &lt;= &lt;=&gt; == =~ &gt; &gt;= &gt;&gt; &gt;&gt;&gt; ?: ?? ??? ^ ^^ |
|| ~
asserting_array asserting_bool asserting_boolean asserting_bytes
asserting_empty asserting_empty_map asserting_error asserting_float
asserting_int asserting_map asserting_nonempty_map asserting_not_array
asserting_not_empty asserting_not_map asserting_not_null asserting_null
asserting_numeric asserting_present asserting_string atan atan2 atanh
base64_decode base64_encode bitcount boolean bytes capitalize cbrt ceil
clean_whitespace collapse_whitespace concat contains cos cosh count depth
dhms2fsec dhms2sec distinct_count erf erfc every exec exp expm1 flatten float
floor fmtifnum fmtnum fold format fsec2dhms fsec2hms get_keys get_values
gmt2localtime gmt2nsec gmt2sec gssub gsub haskey hasvalue hex_decode
hex_encode hexfmt hms2fsec hms2sec hostname index int invqnorm is_absent
is_array is_bool is_boolean is_bytes is_empty is_empty_map is_error is_float
is_int is_map is_nan is_nonempty_map is_not_array is_not_empty is_not_map
is_not_null is_null is_numeric is_present is_string joink joinkv joinv
json_parse json_stringify kurtosis latin1_to_utf8 leafcount leftpad length
localtime2gmt localtime2nsec localtime2sec log log10 log1p logifit lstrip madd
mapdiff mapexcept mapselect mapsum max maxlen md5 mean meaneb median mexp min
minlen mmul mode msub next nsec2gmt nsec2gmtdate nsec2localdate nsec2localtime
null_count os percentile percentiles pow qnorm reduce regextract
regextract_or_else rightpad round roundm rstrip sec2dhms sec2gmt sec2gmtdate
sec2hms sec2localdate sec2localtime select sgn sha1 sha256 sha512 sin sinh
skewness sort sort_collection splita splitax splitkv splitkvx splitnv splitnvx
sqrt ssub stat stddev strfntime strfntime_local strftime strftime_local string
strip strlen strmatch strmatchx strpntime strpntime_local strptime
strptime_local sub substr substr0 substr1 sum sum2 sum3 sum4 sysntime system
systime systimeint tan tanh tolower toupper truncate typeof unflatten unformat
unformatx upntime uptime urand urand32 urandelement urandint urandrange
utf8_to_latin1 variance version ! != !=~ % & && * ** + - . .* .+ .- ./ / // &lt;
&lt;&lt; &lt;= &lt;=&gt; == =~ &gt; &gt;= &gt;&gt; &gt;&gt;&gt; ?: ?? ??? ^ ^^ | || ~
1mCOMMENTS-IN-DATA FLAGS0m
Miller lets you put comments in your data, such as
@ -2393,6 +2394,9 @@ This is simply a copy of what you should see on running `man mlr` at a command p
1masserting_boolean0m
(class=typing #args=1) Aborts with an error if is_boolean on the argument returns false, else returns its argument.
1masserting_bytes0m
(class=typing #args=1) Aborts with an error if is_bytes on the argument returns false, else returns its argument.
1masserting_empty0m
(class=typing #args=1) Aborts with an error if is_empty on the argument returns false, else returns its argument.
@ -2447,12 +2451,30 @@ This is simply a copy of what you should see on running `man mlr` at a command p
1matanh0m
(class=math #args=1) Inverse hyperbolic tangent.
1mbase64_decode0m
(class=string #args=1) Decodes a base64-encoded string to a bytes value; use string() to interpret the result as UTF-8 text. Returns error if the input is not valid base64.
Examples:
base64_decode("aGVsbG8=") gives the bytes 68656c6c6f
string(base64_decode("aGVsbG8=")) gives "hello"
1mbase64_encode0m
(class=string #args=1) Encodes a string or bytes value using standard base64 encoding.
Examples:
base64_encode("hello") gives "aGVsbG8="
base64_encode(b"\xff") gives "/w=="
1mbitcount0m
(class=arithmetic #args=1) Count of 1-bits.
1mboolean0m
(class=conversion #args=1) Convert int/float/bool/string to boolean.
1mbytes0m
(class=conversion #args=1) Convert string to bytes; bytes values pass through as-is. See string() for the reverse.
Examples:
bytes("hi") gives the bytes 6869
string(bytes("hi")) gives "hi"
1mcapitalize0m
(class=string #args=1) Convert string's first character to uppercase.
@ -2626,6 +2648,17 @@ This is simply a copy of what you should see on running `man mlr` at a command p
1mhasvalue0m
(class=collections #args=2) True/false if map/array has/hasn't value, e.g. 'hasvalue($*, "a")' or 'hasvalue(myarray, myvalue)', returns true if the value is present in the collection. Error if 1st argument is not a map or array.
1mhex_decode0m
(class=string #args=1) Decodes a hex-encoded string to a bytes value; use string() to interpret the result as UTF-8 text. Returns error if the input is not valid hex.
Examples:
hex_decode("6869") gives the bytes 6869
string(hex_decode("6869")) gives "hi"
1mhex_encode0m
(class=string #args=1) Encodes a string or bytes value as lowercase hex.
Example:
hex_encode("hi") gives "6869"
1mhexfmt0m
(class=conversion #args=1) Convert int to hex string, e.g. 255 to "0xff".
@ -2673,6 +2706,9 @@ This is simply a copy of what you should see on running `man mlr` at a command p
1mis_boolean0m
(class=typing #args=1) True if field is present with boolean value. Synonymous with is_bool.
1mis_bytes0m
(class=typing #args=1) True if field is present with bytes value.
1mis_empty0m
(class=typing #args=1) True if field is present in input with empty string value, false otherwise.
@ -2826,7 +2862,7 @@ This is simply a copy of what you should see on running `man mlr` at a command p
maxlen(["ao", "alto"]) is 4
1mmd50m
(class=hashing #args=1) MD5 hash.
(class=hashing #args=1) MD5 hash of a string or bytes value.
1mmean0m
(class=stats #args=1) Returns the arithmetic mean of values in an array or map. Returns empty string AKA void for empty array/map; returns error for non-array/non-map types.
@ -3047,13 +3083,13 @@ This is simply a copy of what you should see on running `man mlr` at a command p
(class=math #args=1) +1, 0, -1 for positive, zero, negative input respectively.
1msha10m
(class=hashing #args=1) SHA1 hash.
(class=hashing #args=1) SHA1 hash of a string or bytes value.
1msha2560m
(class=hashing #args=1) SHA256 hash.
(class=hashing #args=1) SHA256 hash of a string or bytes value.
1msha5120m
(class=hashing #args=1) SHA512 hash.
(class=hashing #args=1) SHA512 hash of a string or bytes value.
1msin0m
(class=math #args=1) Trigonometric sine.
@ -3169,13 +3205,13 @@ This is simply a copy of what you should see on running `man mlr` at a command p
strftime_local(1440768801.7, "%Y-%m-%d %H:%M:%3S %z", "Asia/Istanbul") = "2015-08-28 16:33:21.700 +0300"
1mstring0m
(class=conversion #args=1) Convert int/float/bool/string/array/map to string.
(class=conversion #args=1) Convert int/float/bool/string/array/map to string. For bytes values, reinterprets the raw bytes as a (UTF-8) string -- the reverse of bytes().
1mstrip0m
(class=string #args=1) Strip leading and trailing whitespace from string.
1mstrlen0m
(class=string #args=1) String length.
(class=string #args=1) String length in UTF-8 characters, or bytes-value length in bytes.
1mstrmatch0m
(class=string #args=2) Boolean yes/no for whether the stringable first argument matches the regular-expression second argument. No regex captures are provided; please see `strmatch`.
@ -3863,5 +3899,5 @@ This is simply a copy of what you should see on running `man mlr` at a command p
MIME Type for Comma-Separated Values (CSV) Files, the Miller docsite
https://miller.readthedocs.io
2026-06-28 4mMILLER24m(1)
2026-07-01 4mMILLER24m(1)
</pre>

106
docs/src/manpage.txt
View file

@ -215,34 +215,35 @@
1mFUNCTION LIST0m
abs acos acosh antimode any append apply arrayify asin asinh asserting_absent
asserting_array asserting_bool asserting_boolean asserting_empty
asserting_empty_map asserting_error asserting_float asserting_int
asserting_map asserting_nonempty_map asserting_not_array asserting_not_empty
asserting_not_map asserting_not_null asserting_null asserting_numeric
asserting_present asserting_string atan atan2 atanh bitcount boolean
capitalize cbrt ceil clean_whitespace collapse_whitespace concat contains cos
cosh count depth dhms2fsec dhms2sec distinct_count erf erfc every exec exp
expm1 flatten float floor fmtifnum fmtnum fold format fsec2dhms fsec2hms
get_keys get_values gmt2localtime gmt2nsec gmt2sec gssub gsub haskey hasvalue
hexfmt hms2fsec hms2sec hostname index int invqnorm is_absent is_array is_bool
is_boolean is_empty is_empty_map is_error is_float is_int is_map is_nan
is_nonempty_map is_not_array is_not_empty is_not_map is_not_null is_null
is_numeric is_present is_string joink joinkv joinv json_parse json_stringify
kurtosis latin1_to_utf8 leafcount leftpad length localtime2gmt localtime2nsec
localtime2sec log log10 log1p logifit lstrip madd mapdiff mapexcept mapselect
mapsum max maxlen md5 mean meaneb median mexp min minlen mmul mode msub next
nsec2gmt nsec2gmtdate nsec2localdate nsec2localtime null_count os percentile
percentiles pow qnorm reduce regextract regextract_or_else rightpad round
roundm rstrip sec2dhms sec2gmt sec2gmtdate sec2hms sec2localdate sec2localtime
select sgn sha1 sha256 sha512 sin sinh skewness sort sort_collection splita
splitax splitkv splitkvx splitnv splitnvx sqrt ssub stat stddev strfntime
strfntime_local strftime strftime_local string strip strlen strmatch strmatchx
strpntime strpntime_local strptime strptime_local sub substr substr0 substr1
sum sum2 sum3 sum4 sysntime system systime systimeint tan tanh tolower toupper
truncate typeof unflatten unformat unformatx upntime uptime urand urand32
urandelement urandint urandrange utf8_to_latin1 variance version ! != !=~ % &
&& * ** + - . .* .+ .- ./ / // < << <= <=> == =~ > >= >> >>> ?: ?? ??? ^ ^^ |
|| ~
asserting_array asserting_bool asserting_boolean asserting_bytes
asserting_empty asserting_empty_map asserting_error asserting_float
asserting_int asserting_map asserting_nonempty_map asserting_not_array
asserting_not_empty asserting_not_map asserting_not_null asserting_null
asserting_numeric asserting_present asserting_string atan atan2 atanh
base64_decode base64_encode bitcount boolean bytes capitalize cbrt ceil
clean_whitespace collapse_whitespace concat contains cos cosh count depth
dhms2fsec dhms2sec distinct_count erf erfc every exec exp expm1 flatten float
floor fmtifnum fmtnum fold format fsec2dhms fsec2hms get_keys get_values
gmt2localtime gmt2nsec gmt2sec gssub gsub haskey hasvalue hex_decode
hex_encode hexfmt hms2fsec hms2sec hostname index int invqnorm is_absent
is_array is_bool is_boolean is_bytes is_empty is_empty_map is_error is_float
is_int is_map is_nan is_nonempty_map is_not_array is_not_empty is_not_map
is_not_null is_null is_numeric is_present is_string joink joinkv joinv
json_parse json_stringify kurtosis latin1_to_utf8 leafcount leftpad length
localtime2gmt localtime2nsec localtime2sec log log10 log1p logifit lstrip madd
mapdiff mapexcept mapselect mapsum max maxlen md5 mean meaneb median mexp min
minlen mmul mode msub next nsec2gmt nsec2gmtdate nsec2localdate nsec2localtime
null_count os percentile percentiles pow qnorm reduce regextract
regextract_or_else rightpad round roundm rstrip sec2dhms sec2gmt sec2gmtdate
sec2hms sec2localdate sec2localtime select sgn sha1 sha256 sha512 sin sinh
skewness sort sort_collection splita splitax splitkv splitkvx splitnv splitnvx
sqrt ssub stat stddev strfntime strfntime_local strftime strftime_local string
strip strlen strmatch strmatchx strpntime strpntime_local strptime
strptime_local sub substr substr0 substr1 sum sum2 sum3 sum4 sysntime system
systime systimeint tan tanh tolower toupper truncate typeof unflatten unformat
unformatx upntime uptime urand urand32 urandelement urandint urandrange
utf8_to_latin1 variance version ! != !=~ % & && * ** + - . .* .+ .- ./ / // <
<< <= <=> == =~ > >= >> >>> ?: ?? ??? ^ ^^ | || ~
1mCOMMENTS-IN-DATA FLAGS0m
Miller lets you put comments in your data, such as
@ -2372,6 +2373,9 @@
1masserting_boolean0m
(class=typing #args=1) Aborts with an error if is_boolean on the argument returns false, else returns its argument.
1masserting_bytes0m
(class=typing #args=1) Aborts with an error if is_bytes on the argument returns false, else returns its argument.
1masserting_empty0m
(class=typing #args=1) Aborts with an error if is_empty on the argument returns false, else returns its argument.
@ -2426,12 +2430,30 @@
1matanh0m
(class=math #args=1) Inverse hyperbolic tangent.
1mbase64_decode0m
(class=string #args=1) Decodes a base64-encoded string to a bytes value; use string() to interpret the result as UTF-8 text. Returns error if the input is not valid base64.
Examples:
base64_decode("aGVsbG8=") gives the bytes 68656c6c6f
string(base64_decode("aGVsbG8=")) gives "hello"
1mbase64_encode0m
(class=string #args=1) Encodes a string or bytes value using standard base64 encoding.
Examples:
base64_encode("hello") gives "aGVsbG8="
base64_encode(b"\xff") gives "/w=="
1mbitcount0m
(class=arithmetic #args=1) Count of 1-bits.
1mboolean0m
(class=conversion #args=1) Convert int/float/bool/string to boolean.
1mbytes0m
(class=conversion #args=1) Convert string to bytes; bytes values pass through as-is. See string() for the reverse.
Examples:
bytes("hi") gives the bytes 6869
string(bytes("hi")) gives "hi"
1mcapitalize0m
(class=string #args=1) Convert string's first character to uppercase.
@ -2605,6 +2627,17 @@
1mhasvalue0m
(class=collections #args=2) True/false if map/array has/hasn't value, e.g. 'hasvalue($*, "a")' or 'hasvalue(myarray, myvalue)', returns true if the value is present in the collection. Error if 1st argument is not a map or array.
1mhex_decode0m
(class=string #args=1) Decodes a hex-encoded string to a bytes value; use string() to interpret the result as UTF-8 text. Returns error if the input is not valid hex.
Examples:
hex_decode("6869") gives the bytes 6869
string(hex_decode("6869")) gives "hi"
1mhex_encode0m
(class=string #args=1) Encodes a string or bytes value as lowercase hex.
Example:
hex_encode("hi") gives "6869"
1mhexfmt0m
(class=conversion #args=1) Convert int to hex string, e.g. 255 to "0xff".
@ -2652,6 +2685,9 @@
1mis_boolean0m
(class=typing #args=1) True if field is present with boolean value. Synonymous with is_bool.
1mis_bytes0m
(class=typing #args=1) True if field is present with bytes value.
1mis_empty0m
(class=typing #args=1) True if field is present in input with empty string value, false otherwise.
@ -2805,7 +2841,7 @@
maxlen(["ao", "alto"]) is 4
1mmd50m
(class=hashing #args=1) MD5 hash.
(class=hashing #args=1) MD5 hash of a string or bytes value.
1mmean0m
(class=stats #args=1) Returns the arithmetic mean of values in an array or map. Returns empty string AKA void for empty array/map; returns error for non-array/non-map types.
@ -3026,13 +3062,13 @@
(class=math #args=1) +1, 0, -1 for positive, zero, negative input respectively.
1msha10m
(class=hashing #args=1) SHA1 hash.
(class=hashing #args=1) SHA1 hash of a string or bytes value.
1msha2560m
(class=hashing #args=1) SHA256 hash.
(class=hashing #args=1) SHA256 hash of a string or bytes value.
1msha5120m
(class=hashing #args=1) SHA512 hash.
(class=hashing #args=1) SHA512 hash of a string or bytes value.
1msin0m
(class=math #args=1) Trigonometric sine.
@ -3148,13 +3184,13 @@
strftime_local(1440768801.7, "%Y-%m-%d %H:%M:%3S %z", "Asia/Istanbul") = "2015-08-28 16:33:21.700 +0300"
1mstring0m
(class=conversion #args=1) Convert int/float/bool/string/array/map to string.
(class=conversion #args=1) Convert int/float/bool/string/array/map to string. For bytes values, reinterprets the raw bytes as a (UTF-8) string -- the reverse of bytes().
1mstrip0m
(class=string #args=1) Strip leading and trailing whitespace from string.
1mstrlen0m
(class=string #args=1) String length.
(class=string #args=1) String length in UTF-8 characters, or bytes-value length in bytes.
1mstrmatch0m
(class=string #args=2) Boolean yes/no for whether the stringable first argument matches the regular-expression second argument. No regex captures are provided; please see `strmatch`.
@ -3842,4 +3878,4 @@
MIME Type for Comma-Separated Values (CSV) Files, the Miller docsite
https://miller.readthedocs.io
2026-06-28 4mMILLER24m(1)
2026-07-01 4mMILLER24m(1)

74
docs/src/reference-dsl-builtin-functions.md
View file

@ -55,15 +55,15 @@ Operators are listed here along with functions. In this case, the argument count
* [**Arithmetic functions**](#arithmetic-functions): [bitcount](#bitcount), [madd](#madd), [mexp](#mexp), [mmul](#mmul), [msub](#msub), [pow](#pow), [%](#percent), [&](#bitwise-and), [\*](#times), [\**](#exponentiation), [\+](#plus), [\-](#minus), [\.\*](#dot-times), [\.\+](#dot-plus), [\.\-](#dot-minus), [\./](#dot-slash), [/](#slash), [//](#slash-slash), [<<](#lsh), [>>](#srsh), [>>>](#ursh), [^](#bitwise-xor), [\|](#bitwise-or), [~](#bitwise-not).
* [**Boolean functions**](#boolean-functions): [\!](#exclamation-point), [\!=](#exclamation-point-equals), [!=~](#regnotmatch), [&&](#logical-and), [<](#less-than), [<=](#less-than-or-equals), [<=>](#<=>), [==](#double-equals), [=~](#regmatch), [>](#greater-than), [>=](#greater-than-or-equals), [?:](#question-mark-colon), [??](#absent-coalesce), [???](#absent-empty-coalesce), [^^](#logical-xor), [\|\|](#logical-or).
* [**Collections functions**](#collections-functions): [append](#append), [arrayify](#arrayify), [concat](#concat), [depth](#depth), [flatten](#flatten), [get_keys](#get_keys), [get_values](#get_values), [haskey](#haskey), [hasvalue](#hasvalue), [json_parse](#json_parse), [json_stringify](#json_stringify), [leafcount](#leafcount), [length](#length), [mapdiff](#mapdiff), [mapexcept](#mapexcept), [mapselect](#mapselect), [mapsum](#mapsum), [unflatten](#unflatten).
* [**Conversion functions**](#conversion-functions): [boolean](#boolean), [float](#float), [fmtifnum](#fmtifnum), [fmtnum](#fmtnum), [hexfmt](#hexfmt), [int](#int), [joink](#joink), [joinkv](#joinkv), [joinv](#joinv), [splita](#splita), [splitax](#splitax), [splitkv](#splitkv), [splitkvx](#splitkvx), [splitnv](#splitnv), [splitnvx](#splitnvx), [string](#string).
* [**Conversion functions**](#conversion-functions): [boolean](#boolean), [bytes](#bytes), [float](#float), [fmtifnum](#fmtifnum), [fmtnum](#fmtnum), [hexfmt](#hexfmt), [int](#int), [joink](#joink), [joinkv](#joinkv), [joinv](#joinv), [splita](#splita), [splitax](#splitax), [splitkv](#splitkv), [splitkvx](#splitkvx), [splitnv](#splitnv), [splitnvx](#splitnvx), [string](#string).
* [**Hashing functions**](#hashing-functions): [md5](#md5), [sha1](#sha1), [sha256](#sha256), [sha512](#sha512).
* [**Higher-order-functions functions**](#higher-order-functions-functions): [any](#any), [apply](#apply), [every](#every), [fold](#fold), [reduce](#reduce), [select](#select), [sort](#sort).
* [**Math functions**](#math-functions): [abs](#abs), [acos](#acos), [acosh](#acosh), [asin](#asin), [asinh](#asinh), [atan](#atan), [atan2](#atan2), [atanh](#atanh), [cbrt](#cbrt), [ceil](#ceil), [cos](#cos), [cosh](#cosh), [erf](#erf), [erfc](#erfc), [exp](#exp), [expm1](#expm1), [floor](#floor), [invqnorm](#invqnorm), [log](#log), [log10](#log10), [log1p](#log1p), [logifit](#logifit), [max](#max), [min](#min), [qnorm](#qnorm), [round](#round), [roundm](#roundm), [sgn](#sgn), [sin](#sin), [sinh](#sinh), [sqrt](#sqrt), [tan](#tan), [tanh](#tanh), [urand](#urand), [urand32](#urand32), [urandelement](#urandelement), [urandint](#urandint), [urandrange](#urandrange).
* [**Stats functions**](#stats-functions): [antimode](#antimode), [count](#count), [distinct_count](#distinct_count), [kurtosis](#kurtosis), [maxlen](#maxlen), [mean](#mean), [meaneb](#meaneb), [median](#median), [minlen](#minlen), [mode](#mode), [null_count](#null_count), [percentile](#percentile), [percentiles](#percentiles), [skewness](#skewness), [sort_collection](#sort_collection), [stddev](#stddev), [sum](#sum), [sum2](#sum2), [sum3](#sum3), [sum4](#sum4), [variance](#variance).
* [**String functions**](#string-functions): [capitalize](#capitalize), [clean_whitespace](#clean_whitespace), [collapse_whitespace](#collapse_whitespace), [contains](#contains), [format](#format), [gssub](#gssub), [gsub](#gsub), [index](#index), [latin1_to_utf8](#latin1_to_utf8), [leftpad](#leftpad), [lstrip](#lstrip), [regextract](#regextract), [regextract_or_else](#regextract_or_else), [rightpad](#rightpad), [rstrip](#rstrip), [ssub](#ssub), [strip](#strip), [strlen](#strlen), [strmatch](#strmatch), [strmatchx](#strmatchx), [sub](#sub), [substr](#substr), [substr0](#substr0), [substr1](#substr1), [tolower](#tolower), [toupper](#toupper), [truncate](#truncate), [unformat](#unformat), [unformatx](#unformatx), [utf8_to_latin1](#utf8_to_latin1), [\.](#dot).
* [**String functions**](#string-functions): [base64_decode](#base64_decode), [base64_encode](#base64_encode), [capitalize](#capitalize), [clean_whitespace](#clean_whitespace), [collapse_whitespace](#collapse_whitespace), [contains](#contains), [format](#format), [gssub](#gssub), [gsub](#gsub), [hex_decode](#hex_decode), [hex_encode](#hex_encode), [index](#index), [latin1_to_utf8](#latin1_to_utf8), [leftpad](#leftpad), [lstrip](#lstrip), [regextract](#regextract), [regextract_or_else](#regextract_or_else), [rightpad](#rightpad), [rstrip](#rstrip), [ssub](#ssub), [strip](#strip), [strlen](#strlen), [strmatch](#strmatch), [strmatchx](#strmatchx), [sub](#sub), [substr](#substr), [substr0](#substr0), [substr1](#substr1), [tolower](#tolower), [toupper](#toupper), [truncate](#truncate), [unformat](#unformat), [unformatx](#unformatx), [utf8_to_latin1](#utf8_to_latin1), [\.](#dot).
* [**System functions**](#system-functions): [exec](#exec), [hostname](#hostname), [next](#next), [os](#os), [stat](#stat), [system](#system), [version](#version).
* [**Time functions**](#time-functions): [dhms2fsec](#dhms2fsec), [dhms2sec](#dhms2sec), [fsec2dhms](#fsec2dhms), [fsec2hms](#fsec2hms), [gmt2localtime](#gmt2localtime), [gmt2nsec](#gmt2nsec), [gmt2sec](#gmt2sec), [hms2fsec](#hms2fsec), [hms2sec](#hms2sec), [localtime2gmt](#localtime2gmt), [localtime2nsec](#localtime2nsec), [localtime2sec](#localtime2sec), [nsec2gmt](#nsec2gmt), [nsec2gmtdate](#nsec2gmtdate), [nsec2localdate](#nsec2localdate), [nsec2localtime](#nsec2localtime), [sec2dhms](#sec2dhms), [sec2gmt](#sec2gmt), [sec2gmtdate](#sec2gmtdate), [sec2hms](#sec2hms), [sec2localdate](#sec2localdate), [sec2localtime](#sec2localtime), [strfntime](#strfntime), [strfntime_local](#strfntime_local), [strftime](#strftime), [strftime_local](#strftime_local), [strpntime](#strpntime), [strpntime_local](#strpntime_local), [strptime](#strptime), [strptime_local](#strptime_local), [sysntime](#sysntime), [systime](#systime), [systimeint](#systimeint), [upntime](#upntime), [uptime](#uptime).
* [**Typing functions**](#typing-functions): [asserting_absent](#asserting_absent), [asserting_array](#asserting_array), [asserting_bool](#asserting_bool), [asserting_boolean](#asserting_boolean), [asserting_empty](#asserting_empty), [asserting_empty_map](#asserting_empty_map), [asserting_error](#asserting_error), [asserting_float](#asserting_float), [asserting_int](#asserting_int), [asserting_map](#asserting_map), [asserting_nonempty_map](#asserting_nonempty_map), [asserting_not_array](#asserting_not_array), [asserting_not_empty](#asserting_not_empty), [asserting_not_map](#asserting_not_map), [asserting_not_null](#asserting_not_null), [asserting_null](#asserting_null), [asserting_numeric](#asserting_numeric), [asserting_present](#asserting_present), [asserting_string](#asserting_string), [is_absent](#is_absent), [is_array](#is_array), [is_bool](#is_bool), [is_boolean](#is_boolean), [is_empty](#is_empty), [is_empty_map](#is_empty_map), [is_error](#is_error), [is_float](#is_float), [is_int](#is_int), [is_map](#is_map), [is_nan](#is_nan), [is_nonempty_map](#is_nonempty_map), [is_not_array](#is_not_array), [is_not_empty](#is_not_empty), [is_not_map](#is_not_map), [is_not_null](#is_not_null), [is_null](#is_null), [is_numeric](#is_numeric), [is_present](#is_present), [is_string](#is_string), [typeof](#typeof).
* [**Typing functions**](#typing-functions): [asserting_absent](#asserting_absent), [asserting_array](#asserting_array), [asserting_bool](#asserting_bool), [asserting_boolean](#asserting_boolean), [asserting_bytes](#asserting_bytes), [asserting_empty](#asserting_empty), [asserting_empty_map](#asserting_empty_map), [asserting_error](#asserting_error), [asserting_float](#asserting_float), [asserting_int](#asserting_int), [asserting_map](#asserting_map), [asserting_nonempty_map](#asserting_nonempty_map), [asserting_not_array](#asserting_not_array), [asserting_not_empty](#asserting_not_empty), [asserting_not_map](#asserting_not_map), [asserting_not_null](#asserting_not_null), [asserting_null](#asserting_null), [asserting_numeric](#asserting_numeric), [asserting_present](#asserting_present), [asserting_string](#asserting_string), [is_absent](#is_absent), [is_array](#is_array), [is_bool](#is_bool), [is_boolean](#is_boolean), [is_bytes](#is_bytes), [is_empty](#is_empty), [is_empty_map](#is_empty_map), [is_error](#is_error), [is_float](#is_float), [is_int](#is_int), [is_map](#is_map), [is_nan](#is_nan), [is_nonempty_map](#is_nonempty_map), [is_not_array](#is_not_array), [is_not_empty](#is_not_empty), [is_not_map](#is_not_map), [is_not_null](#is_not_null), [is_null](#is_null), [is_numeric](#is_numeric), [is_present](#is_present), [is_string](#is_string), [typeof](#typeof).
## Arithmetic functions
@ -507,6 +507,15 @@ boolean (class=conversion #args=1) Convert int/float/bool/string to boolean.
</pre>
### bytes
<pre class="pre-non-highlight-non-pair">
bytes (class=conversion #args=1) Convert string to bytes; bytes values pass through as-is. See string() for the reverse.
Examples:
bytes("hi") gives the bytes 6869
string(bytes("hi")) gives "hi"
</pre>
### float
<pre class="pre-non-highlight-non-pair">
float (class=conversion #args=1) Convert int/float/bool/string to float.
@ -633,7 +642,7 @@ splitnvx("3,4,5", ",") = {"1":"3","2":"4","3":"5"}
### string
<pre class="pre-non-highlight-non-pair">
string (class=conversion #args=1) Convert int/float/bool/string/array/map to string.
string (class=conversion #args=1) Convert int/float/bool/string/array/map to string. For bytes values, reinterprets the raw bytes as a (UTF-8) string -- the reverse of bytes().
</pre>
## Hashing functions
@ -641,25 +650,25 @@ string (class=conversion #args=1) Convert int/float/bool/string/array/map to st
### md5
<pre class="pre-non-highlight-non-pair">
md5 (class=hashing #args=1) MD5 hash.
md5 (class=hashing #args=1) MD5 hash of a string or bytes value.
</pre>
### sha1
<pre class="pre-non-highlight-non-pair">
sha1 (class=hashing #args=1) SHA1 hash.
sha1 (class=hashing #args=1) SHA1 hash of a string or bytes value.
</pre>
### sha256
<pre class="pre-non-highlight-non-pair">
sha256 (class=hashing #args=1) SHA256 hash.
sha256 (class=hashing #args=1) SHA256 hash of a string or bytes value.
</pre>
### sha512
<pre class="pre-non-highlight-non-pair">
sha512 (class=hashing #args=1) SHA512 hash.
sha512 (class=hashing #args=1) SHA512 hash of a string or bytes value.
</pre>
## Higher-order-functions functions
@ -1197,6 +1206,24 @@ variance([4,5,9,10,11]) is 9.7
## String functions
### base64_decode
<pre class="pre-non-highlight-non-pair">
base64_decode (class=string #args=1) Decodes a base64-encoded string to a bytes value; use string() to interpret the result as UTF-8 text. Returns error if the input is not valid base64.
Examples:
base64_decode("aGVsbG8=") gives the bytes 68656c6c6f
string(base64_decode("aGVsbG8=")) gives "hello"
</pre>
### base64_encode
<pre class="pre-non-highlight-non-pair">
base64_encode (class=string #args=1) Encodes a string or bytes value using standard base64 encoding.
Examples:
base64_encode("hello") gives "aGVsbG8="
base64_encode(b"\xff") gives "/w=="
</pre>
### capitalize
<pre class="pre-non-highlight-non-pair">
capitalize (class=string #args=1) Convert string's first character to uppercase.
@ -1257,6 +1284,23 @@ gsub("prefix4529:suffix8567", "(....ix)([0-9]+)", "[\1 : \2]") gives "[prefix :
</pre>
### hex_decode
<pre class="pre-non-highlight-non-pair">
hex_decode (class=string #args=1) Decodes a hex-encoded string to a bytes value; use string() to interpret the result as UTF-8 text. Returns error if the input is not valid hex.
Examples:
hex_decode("6869") gives the bytes 6869
string(hex_decode("6869")) gives "hi"
</pre>
### hex_encode
<pre class="pre-non-highlight-non-pair">
hex_encode (class=string #args=1) Encodes a string or bytes value as lowercase hex.
Example:
hex_encode("hi") gives "6869"
</pre>
### index
<pre class="pre-non-highlight-non-pair">
index (class=string #args=2) Returns the index (1-based) of the second argument within the first. Returns -1 if the second argument isn't a substring of the first. Stringifies non-string scalar inputs; raises an error if either argument is an array or map. Uses UTF-8 encoding to count characters, not bytes.
@ -1344,7 +1388,7 @@ strip (class=string #args=1) Strip leading and trailing whitespace from string.
### strlen
<pre class="pre-non-highlight-non-pair">
strlen (class=string #args=1) String length.
strlen (class=string #args=1) String length in UTF-8 characters, or bytes-value length in bytes.
</pre>
@ -1846,6 +1890,12 @@ asserting_boolean (class=typing #args=1) Aborts with an error if is_boolean on
</pre>
### asserting_bytes
<pre class="pre-non-highlight-non-pair">
asserting_bytes (class=typing #args=1) Aborts with an error if is_bytes on the argument returns false, else returns its argument.
</pre>
### asserting_empty
<pre class="pre-non-highlight-non-pair">
asserting_empty (class=typing #args=1) Aborts with an error if is_empty on the argument returns false, else returns its argument.
@ -1960,6 +2010,12 @@ is_boolean (class=typing #args=1) True if field is present with boolean value.
</pre>
### is_bytes
<pre class="pre-non-highlight-non-pair">
is_bytes (class=typing #args=1) True if field is present with bytes value.
</pre>
### is_empty
<pre class="pre-non-highlight-non-pair">
is_empty (class=typing #args=1) True if field is present in input with empty string value, false otherwise.

2
docs/src/reference-dsl-variables.md
View file

@ -654,6 +654,7 @@ is_absent
is_array
is_bool
is_boolean
is_bytes
is_empty
is_empty_map
is_error
@ -680,6 +681,7 @@ asserting_absent
asserting_array
asserting_bool
asserting_boolean
asserting_bytes
asserting_empty
asserting_empty_map
asserting_error

1
docs/src/reference-main-data-types.md
View file

@ -25,6 +25,7 @@ Miller's types are:
* **float** and **int**: such as `1.2` and `3`: double-precision and 64-bit signed, respectively. See the section on [arithmetic operators and math-related library functions](reference-dsl-builtin-functions.md#math-functions) as well as the [Arithmetic](reference-main-arithmetic.md) page.
* dates/times are not a separate data type; Miller uses ints for [seconds since the epoch](https://en.wikipedia.org/wiki/Unix_time) and strings for formatted date/times. See the [DSL datetime/timezone functions page](reference-dsl-time.md) for more information.
* **boolean**: literals `true` and `false`; results of `==`, `<`, `>`, etc. See the section on [boolean operators](reference-dsl-builtin-functions.md#boolean-functions).
* **bytes**: raw binary data, such as from [base64_decode](reference-dsl-builtin-functions.md#base64_decode) or [hex_decode](reference-dsl-builtin-functions.md#hex_decode), or literals like `b"\x01\xff"`. Unlike strings, these are never interpreted as UTF-8 text: `strlen` is the byte count, `substr` slices by byte position, and the `.` operator concatenates bytes with bytes. On output (CSV, JSON, etc.) bytes are rendered as lowercase hex. Use [`bytes()`](reference-dsl-builtin-functions.md#bytes) and [`string()`](reference-dsl-builtin-functions.md#string) to convert to/from strings, and [`md5`](reference-dsl-builtin-functions.md#md5)/`sha1`/`sha256`/`sha512` to hash raw payloads.
* Collections:
* **map**: such as `{"a":1,"b":[2,3,4]}`, supporting key-indexing, preservation of insertion order, [library functions](reference-dsl-builtin-functions.md#collections-functions), etc. See the [Maps](reference-main-maps.md) page.
* **array**: such as `["a", 2, true]`, supporting one-up indexing and slicing, [library functions](reference-dsl-builtin-functions.md#collections-functions), etc. See the [Arrays](reference-main-arrays.md) page.

1
docs/src/reference-main-data-types.md.in
View file

@ -9,6 +9,7 @@ Miller's types are:
* **float** and **int**: such as `1.2` and `3`: double-precision and 64-bit signed, respectively. See the section on [arithmetic operators and math-related library functions](reference-dsl-builtin-functions.md#math-functions) as well as the [Arithmetic](reference-main-arithmetic.md) page.
* dates/times are not a separate data type; Miller uses ints for [seconds since the epoch](https://en.wikipedia.org/wiki/Unix_time) and strings for formatted date/times. See the [DSL datetime/timezone functions page](reference-dsl-time.md) for more information.
* **boolean**: literals `true` and `false`; results of `==`, `<`, `>`, etc. See the section on [boolean operators](reference-dsl-builtin-functions.md#boolean-functions).
* **bytes**: raw binary data, such as from [base64_decode](reference-dsl-builtin-functions.md#base64_decode) or [hex_decode](reference-dsl-builtin-functions.md#hex_decode), or literals like `b"\x01\xff"`. Unlike strings, these are never interpreted as UTF-8 text: `strlen` is the byte count, `substr` slices by byte position, and the `.` operator concatenates bytes with bytes. On output (CSV, JSON, etc.) bytes are rendered as lowercase hex. Use [`bytes()`](reference-dsl-builtin-functions.md#bytes) and [`string()`](reference-dsl-builtin-functions.md#string) to convert to/from strings, and [`md5`](reference-dsl-builtin-functions.md#md5)/`sha1`/`sha256`/`sha512` to hash raw payloads.
* Collections:
* **map**: such as `{"a":1,"b":[2,3,4]}`, supporting key-indexing, preservation of insertion order, [library functions](reference-dsl-builtin-functions.md#collections-functions), etc. See the [Maps](reference-main-maps.md) page.
* **array**: such as `["a", 2, true]`, supporting one-up indexing and slicing, [library functions](reference-dsl-builtin-functions.md#collections-functions), etc. See the [Arrays](reference-main-arrays.md) page.

1
docs/src/reference-main-null-data.md
View file

@ -38,6 +38,7 @@ is_absent
is_array
is_bool
is_boolean
is_bytes
is_empty
is_empty_map
is_error

106
man/manpage.txt
View file

@ -215,34 +215,35 @@
1mFUNCTION LIST0m
abs acos acosh antimode any append apply arrayify asin asinh asserting_absent
asserting_array asserting_bool asserting_boolean asserting_empty
asserting_empty_map asserting_error asserting_float asserting_int
asserting_map asserting_nonempty_map asserting_not_array asserting_not_empty
asserting_not_map asserting_not_null asserting_null asserting_numeric
asserting_present asserting_string atan atan2 atanh bitcount boolean
capitalize cbrt ceil clean_whitespace collapse_whitespace concat contains cos
cosh count depth dhms2fsec dhms2sec distinct_count erf erfc every exec exp
expm1 flatten float floor fmtifnum fmtnum fold format fsec2dhms fsec2hms
get_keys get_values gmt2localtime gmt2nsec gmt2sec gssub gsub haskey hasvalue
hexfmt hms2fsec hms2sec hostname index int invqnorm is_absent is_array is_bool
is_boolean is_empty is_empty_map is_error is_float is_int is_map is_nan
is_nonempty_map is_not_array is_not_empty is_not_map is_not_null is_null
is_numeric is_present is_string joink joinkv joinv json_parse json_stringify
kurtosis latin1_to_utf8 leafcount leftpad length localtime2gmt localtime2nsec
localtime2sec log log10 log1p logifit lstrip madd mapdiff mapexcept mapselect
mapsum max maxlen md5 mean meaneb median mexp min minlen mmul mode msub next
nsec2gmt nsec2gmtdate nsec2localdate nsec2localtime null_count os percentile
percentiles pow qnorm reduce regextract regextract_or_else rightpad round
roundm rstrip sec2dhms sec2gmt sec2gmtdate sec2hms sec2localdate sec2localtime
select sgn sha1 sha256 sha512 sin sinh skewness sort sort_collection splita
splitax splitkv splitkvx splitnv splitnvx sqrt ssub stat stddev strfntime
strfntime_local strftime strftime_local string strip strlen strmatch strmatchx
strpntime strpntime_local strptime strptime_local sub substr substr0 substr1
sum sum2 sum3 sum4 sysntime system systime systimeint tan tanh tolower toupper
truncate typeof unflatten unformat unformatx upntime uptime urand urand32
urandelement urandint urandrange utf8_to_latin1 variance version ! != !=~ % &
&& * ** + - . .* .+ .- ./ / // < << <= <=> == =~ > >= >> >>> ?: ?? ??? ^ ^^ |
|| ~
asserting_array asserting_bool asserting_boolean asserting_bytes
asserting_empty asserting_empty_map asserting_error asserting_float
asserting_int asserting_map asserting_nonempty_map asserting_not_array
asserting_not_empty asserting_not_map asserting_not_null asserting_null
asserting_numeric asserting_present asserting_string atan atan2 atanh
base64_decode base64_encode bitcount boolean bytes capitalize cbrt ceil
clean_whitespace collapse_whitespace concat contains cos cosh count depth
dhms2fsec dhms2sec distinct_count erf erfc every exec exp expm1 flatten float
floor fmtifnum fmtnum fold format fsec2dhms fsec2hms get_keys get_values
gmt2localtime gmt2nsec gmt2sec gssub gsub haskey hasvalue hex_decode
hex_encode hexfmt hms2fsec hms2sec hostname index int invqnorm is_absent
is_array is_bool is_boolean is_bytes is_empty is_empty_map is_error is_float
is_int is_map is_nan is_nonempty_map is_not_array is_not_empty is_not_map
is_not_null is_null is_numeric is_present is_string joink joinkv joinv
json_parse json_stringify kurtosis latin1_to_utf8 leafcount leftpad length
localtime2gmt localtime2nsec localtime2sec log log10 log1p logifit lstrip madd
mapdiff mapexcept mapselect mapsum max maxlen md5 mean meaneb median mexp min
minlen mmul mode msub next nsec2gmt nsec2gmtdate nsec2localdate nsec2localtime
null_count os percentile percentiles pow qnorm reduce regextract
regextract_or_else rightpad round roundm rstrip sec2dhms sec2gmt sec2gmtdate
sec2hms sec2localdate sec2localtime select sgn sha1 sha256 sha512 sin sinh
skewness sort sort_collection splita splitax splitkv splitkvx splitnv splitnvx
sqrt ssub stat stddev strfntime strfntime_local strftime strftime_local string
strip strlen strmatch strmatchx strpntime strpntime_local strptime
strptime_local sub substr substr0 substr1 sum sum2 sum3 sum4 sysntime system
systime systimeint tan tanh tolower toupper truncate typeof unflatten unformat
unformatx upntime uptime urand urand32 urandelement urandint urandrange
utf8_to_latin1 variance version ! != !=~ % & && * ** + - . .* .+ .- ./ / // <
<< <= <=> == =~ > >= >> >>> ?: ?? ??? ^ ^^ | || ~
1mCOMMENTS-IN-DATA FLAGS0m
Miller lets you put comments in your data, such as
@ -2372,6 +2373,9 @@
1masserting_boolean0m
(class=typing #args=1) Aborts with an error if is_boolean on the argument returns false, else returns its argument.
1masserting_bytes0m
(class=typing #args=1) Aborts with an error if is_bytes on the argument returns false, else returns its argument.
1masserting_empty0m
(class=typing #args=1) Aborts with an error if is_empty on the argument returns false, else returns its argument.
@ -2426,12 +2430,30 @@
1matanh0m
(class=math #args=1) Inverse hyperbolic tangent.
1mbase64_decode0m
(class=string #args=1) Decodes a base64-encoded string to a bytes value; use string() to interpret the result as UTF-8 text. Returns error if the input is not valid base64.
Examples:
base64_decode("aGVsbG8=") gives the bytes 68656c6c6f
string(base64_decode("aGVsbG8=")) gives "hello"
1mbase64_encode0m
(class=string #args=1) Encodes a string or bytes value using standard base64 encoding.
Examples:
base64_encode("hello") gives "aGVsbG8="
base64_encode(b"\xff") gives "/w=="
1mbitcount0m
(class=arithmetic #args=1) Count of 1-bits.
1mboolean0m
(class=conversion #args=1) Convert int/float/bool/string to boolean.
1mbytes0m
(class=conversion #args=1) Convert string to bytes; bytes values pass through as-is. See string() for the reverse.
Examples:
bytes("hi") gives the bytes 6869
string(bytes("hi")) gives "hi"
1mcapitalize0m
(class=string #args=1) Convert string's first character to uppercase.
@ -2605,6 +2627,17 @@
1mhasvalue0m
(class=collections #args=2) True/false if map/array has/hasn't value, e.g. 'hasvalue($*, "a")' or 'hasvalue(myarray, myvalue)', returns true if the value is present in the collection. Error if 1st argument is not a map or array.
1mhex_decode0m
(class=string #args=1) Decodes a hex-encoded string to a bytes value; use string() to interpret the result as UTF-8 text. Returns error if the input is not valid hex.
Examples:
hex_decode("6869") gives the bytes 6869
string(hex_decode("6869")) gives "hi"
1mhex_encode0m
(class=string #args=1) Encodes a string or bytes value as lowercase hex.
Example:
hex_encode("hi") gives "6869"
1mhexfmt0m
(class=conversion #args=1) Convert int to hex string, e.g. 255 to "0xff".
@ -2652,6 +2685,9 @@
1mis_boolean0m
(class=typing #args=1) True if field is present with boolean value. Synonymous with is_bool.
1mis_bytes0m
(class=typing #args=1) True if field is present with bytes value.
1mis_empty0m
(class=typing #args=1) True if field is present in input with empty string value, false otherwise.
@ -2805,7 +2841,7 @@
maxlen(["ao", "alto"]) is 4
1mmd50m
(class=hashing #args=1) MD5 hash.
(class=hashing #args=1) MD5 hash of a string or bytes value.
1mmean0m
(class=stats #args=1) Returns the arithmetic mean of values in an array or map. Returns empty string AKA void for empty array/map; returns error for non-array/non-map types.
@ -3026,13 +3062,13 @@
(class=math #args=1) +1, 0, -1 for positive, zero, negative input respectively.
1msha10m
(class=hashing #args=1) SHA1 hash.
(class=hashing #args=1) SHA1 hash of a string or bytes value.
1msha2560m
(class=hashing #args=1) SHA256 hash.
(class=hashing #args=1) SHA256 hash of a string or bytes value.
1msha5120m
(class=hashing #args=1) SHA512 hash.
(class=hashing #args=1) SHA512 hash of a string or bytes value.
1msin0m
(class=math #args=1) Trigonometric sine.
@ -3148,13 +3184,13 @@
strftime_local(1440768801.7, "%Y-%m-%d %H:%M:%3S %z", "Asia/Istanbul") = "2015-08-28 16:33:21.700 +0300"
1mstring0m
(class=conversion #args=1) Convert int/float/bool/string/array/map to string.
(class=conversion #args=1) Convert int/float/bool/string/array/map to string. For bytes values, reinterprets the raw bytes as a (UTF-8) string -- the reverse of bytes().
1mstrip0m
(class=string #args=1) Strip leading and trailing whitespace from string.
1mstrlen0m
(class=string #args=1) String length.
(class=string #args=1) String length in UTF-8 characters, or bytes-value length in bytes.
1mstrmatch0m
(class=string #args=2) Boolean yes/no for whether the stringable first argument matches the regular-expression second argument. No regex captures are provided; please see `strmatch`.
@ -3842,4 +3878,4 @@
MIME Type for Comma-Separated Values (CSV) Files, the Miller docsite
https://miller.readthedocs.io
2026-06-28 4mMILLER24m(1)
2026-07-01 4mMILLER24m(1)

150
man/mlr.1
View file

@ -2,12 +2,12 @@
.\" Title: mlr
.\" Author: [see the "AUTHOR" section]
.\" Generator: ./mkman.rb
.\" Date: 2026-06-28
.\" Date: 2026-07-01
.\" Manual: \ \&
.\" Source: \ \&
.\" Language: English
.\"
.TH "MILLER" "1" "2026-06-28" "\ \&" "\ \&"
.TH "MILLER" "1" "2026-07-01" "\ \&" "\ \&"
.\" -----------------------------------------------------------------
.\" * Portability definitions
.\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@ -263,34 +263,35 @@ unflatten uniq unspace unsparsify
.\}
.nf
abs acos acosh antimode any append apply arrayify asin asinh asserting_absent
asserting_array asserting_bool asserting_boolean asserting_empty
asserting_empty_map asserting_error asserting_float asserting_int
asserting_map asserting_nonempty_map asserting_not_array asserting_not_empty
asserting_not_map asserting_not_null asserting_null asserting_numeric
asserting_present asserting_string atan atan2 atanh bitcount boolean
capitalize cbrt ceil clean_whitespace collapse_whitespace concat contains cos
cosh count depth dhms2fsec dhms2sec distinct_count erf erfc every exec exp
expm1 flatten float floor fmtifnum fmtnum fold format fsec2dhms fsec2hms
get_keys get_values gmt2localtime gmt2nsec gmt2sec gssub gsub haskey hasvalue
hexfmt hms2fsec hms2sec hostname index int invqnorm is_absent is_array is_bool
is_boolean is_empty is_empty_map is_error is_float is_int is_map is_nan
is_nonempty_map is_not_array is_not_empty is_not_map is_not_null is_null
is_numeric is_present is_string joink joinkv joinv json_parse json_stringify
kurtosis latin1_to_utf8 leafcount leftpad length localtime2gmt localtime2nsec
localtime2sec log log10 log1p logifit lstrip madd mapdiff mapexcept mapselect
mapsum max maxlen md5 mean meaneb median mexp min minlen mmul mode msub next
nsec2gmt nsec2gmtdate nsec2localdate nsec2localtime null_count os percentile
percentiles pow qnorm reduce regextract regextract_or_else rightpad round
roundm rstrip sec2dhms sec2gmt sec2gmtdate sec2hms sec2localdate sec2localtime
select sgn sha1 sha256 sha512 sin sinh skewness sort sort_collection splita
splitax splitkv splitkvx splitnv splitnvx sqrt ssub stat stddev strfntime
strfntime_local strftime strftime_local string strip strlen strmatch strmatchx
strpntime strpntime_local strptime strptime_local sub substr substr0 substr1
sum sum2 sum3 sum4 sysntime system systime systimeint tan tanh tolower toupper
truncate typeof unflatten unformat unformatx upntime uptime urand urand32
urandelement urandint urandrange utf8_to_latin1 variance version ! != !=~ % &
&& * ** + - . .* .+ .- ./ / // < << <= <=> == =~ > >= >> >>> ?: ?? ??? ^ ^^ |
|| ~
asserting_array asserting_bool asserting_boolean asserting_bytes
asserting_empty asserting_empty_map asserting_error asserting_float
asserting_int asserting_map asserting_nonempty_map asserting_not_array
asserting_not_empty asserting_not_map asserting_not_null asserting_null
asserting_numeric asserting_present asserting_string atan atan2 atanh
base64_decode base64_encode bitcount boolean bytes capitalize cbrt ceil
clean_whitespace collapse_whitespace concat contains cos cosh count depth
dhms2fsec dhms2sec distinct_count erf erfc every exec exp expm1 flatten float
floor fmtifnum fmtnum fold format fsec2dhms fsec2hms get_keys get_values
gmt2localtime gmt2nsec gmt2sec gssub gsub haskey hasvalue hex_decode
hex_encode hexfmt hms2fsec hms2sec hostname index int invqnorm is_absent
is_array is_bool is_boolean is_bytes is_empty is_empty_map is_error is_float
is_int is_map is_nan is_nonempty_map is_not_array is_not_empty is_not_map
is_not_null is_null is_numeric is_present is_string joink joinkv joinv
json_parse json_stringify kurtosis latin1_to_utf8 leafcount leftpad length
localtime2gmt localtime2nsec localtime2sec log log10 log1p logifit lstrip madd
mapdiff mapexcept mapselect mapsum max maxlen md5 mean meaneb median mexp min
minlen mmul mode msub next nsec2gmt nsec2gmtdate nsec2localdate nsec2localtime
null_count os percentile percentiles pow qnorm reduce regextract
regextract_or_else rightpad round roundm rstrip sec2dhms sec2gmt sec2gmtdate
sec2hms sec2localdate sec2localtime select sgn sha1 sha256 sha512 sin sinh
skewness sort sort_collection splita splitax splitkv splitkvx splitnv splitnvx
sqrt ssub stat stddev strfntime strfntime_local strftime strftime_local string
strip strlen strmatch strmatchx strpntime strpntime_local strptime
strptime_local sub substr substr0 substr1 sum sum2 sum3 sum4 sysntime system
systime systimeint tan tanh tolower toupper truncate typeof unflatten unformat
unformatx upntime uptime urand urand32 urandelement urandint urandrange
utf8_to_latin1 variance version ! != !=~ % & && * ** + - . .* .+ .- ./ / // <
<< <= <=> == =~ > >= >> >>> ?: ?? ??? ^ ^^ | || ~
.fi
.if n \{\
.RE
@ -3068,6 +3069,15 @@ Map example: apply({"a":1, "b":3, "c":5}, func(k,v) {return {toupper(k): v ** 2}
.fi
.if n \{\
.RE
.SS "asserting_bytes"
.if n \{\
.RS 0
.\}
.nf
(class=typing #args=1) Aborts with an error if is_bytes on the argument returns false, else returns its argument.
.fi
.if n \{\
.RE
.SS "asserting_empty"
.if n \{\
.RS 0
@ -3230,6 +3240,30 @@ Map example: apply({"a":1, "b":3, "c":5}, func(k,v) {return {toupper(k): v ** 2}
.fi
.if n \{\
.RE
.SS "base64_decode"
.if n \{\
.RS 0
.\}
.nf
(class=string #args=1) Decodes a base64-encoded string to a bytes value; use string() to interpret the result as UTF-8 text. Returns error if the input is not valid base64.
Examples:
base64_decode("aGVsbG8=") gives the bytes 68656c6c6f
string(base64_decode("aGVsbG8=")) gives "hello"
.fi
.if n \{\
.RE
.SS "base64_encode"
.if n \{\
.RS 0
.\}
.nf
(class=string #args=1) Encodes a string or bytes value using standard base64 encoding.
Examples:
base64_encode("hello") gives "aGVsbG8="
base64_encode(b"\exff") gives "/w=="
.fi
.if n \{\
.RE
.SS "bitcount"
.if n \{\
.RS 0
@ -3248,6 +3282,18 @@ Map example: apply({"a":1, "b":3, "c":5}, func(k,v) {return {toupper(k): v ** 2}
.fi
.if n \{\
.RE
.SS "bytes"
.if n \{\
.RS 0
.\}
.nf
(class=conversion #args=1) Convert string to bytes; bytes values pass through as-is. See string() for the reverse.
Examples:
bytes("hi") gives the bytes 6869
string(bytes("hi")) gives "hi"
.fi
.if n \{\
.RE
.SS "capitalize"
.if n \{\
.RS 0
@ -3649,6 +3695,29 @@ gsub("prefix4529:suffix8567", "(....ix)([0-9]+)", "[\e1 : \e2]") gives "[prefix
.fi
.if n \{\
.RE
.SS "hex_decode"
.if n \{\
.RS 0
.\}
.nf
(class=string #args=1) Decodes a hex-encoded string to a bytes value; use string() to interpret the result as UTF-8 text. Returns error if the input is not valid hex.
Examples:
hex_decode("6869") gives the bytes 6869
string(hex_decode("6869")) gives "hi"
.fi
.if n \{\
.RE
.SS "hex_encode"
.if n \{\
.RS 0
.\}
.nf
(class=string #args=1) Encodes a string or bytes value as lowercase hex.
Example:
hex_encode("hi") gives "6869"
.fi
.if n \{\
.RE
.SS "hexfmt"
.if n \{\
.RS 0
@ -3762,6 +3831,15 @@ int(string(345), 16) gives decimal 837
.fi
.if n \{\
.RE
.SS "is_bytes"
.if n \{\
.RS 0
.\}
.nf
(class=typing #args=1) True if field is present with bytes value.
.fi
.if n \{\
.RE
.SS "is_empty"
.if n \{\
.RS 0
@ -4165,7 +4243,7 @@ maxlen(["año", "alto"]) is 4
.RS 0
.\}
.nf
(class=hashing #args=1) MD5 hash.
(class=hashing #args=1) MD5 hash of a string or bytes value.
.fi
.if n \{\
.RE
@ -4602,7 +4680,7 @@ Map example: select({"a":1, "b":3, "c":5}, func(k,v) {return v >= 3}) returns {"
.RS 0
.\}
.nf
(class=hashing #args=1) SHA1 hash.
(class=hashing #args=1) SHA1 hash of a string or bytes value.
.fi
.if n \{\
.RE
@ -4611,7 +4689,7 @@ Map example: select({"a":1, "b":3, "c":5}, func(k,v) {return v >= 3}) returns {"
.RS 0
.\}
.nf
(class=hashing #args=1) SHA256 hash.
(class=hashing #args=1) SHA256 hash of a string or bytes value.
.fi
.if n \{\
.RE
@ -4620,7 +4698,7 @@ Map example: select({"a":1, "b":3, "c":5}, func(k,v) {return v >= 3}) returns {"
.RS 0
.\}
.nf
(class=hashing #args=1) SHA512 hash.
(class=hashing #args=1) SHA512 hash of a string or bytes value.
.fi
.if n \{\
.RE
@ -4856,7 +4934,7 @@ strftime_local(1440768801.7, "%Y-%m-%d %H:%M:%3S %z", "Asia/Istanbul") = "2015-0
.RS 0
.\}
.nf
(class=conversion #args=1) Convert int/float/bool/string/array/map to string.
(class=conversion #args=1) Convert int/float/bool/string/array/map to string. For bytes values, reinterprets the raw bytes as a (UTF-8) string -- the reverse of bytes().
.fi
.if n \{\
.RE
@ -4874,7 +4952,7 @@ strftime_local(1440768801.7, "%Y-%m-%d %H:%M:%3S %z", "Asia/Istanbul") = "2015-0
.RS 0
.\}
.nf
(class=string #args=1) String length.
(class=string #args=1) String length in UTF-8 characters, or bytes-value length in bytes.
.fi
.if n \{\
.RE

304
pkg/bifs/arithmetic.go
View file

@ -20,6 +20,7 @@ var upos_dispositions = [mlrval.MT_DIM]UnaryFunc{
/*BOOL */ upos_te,
/*VOID */ _zero1,
/*STRING */ upos_te,
/*BYTES */ upos_te,
/*ARRAY */ _absn1,
/*MAP */ _absn1,
/*FUNC */ upos_te,
@ -52,6 +53,7 @@ var uneg_dispositions = [mlrval.MT_DIM]UnaryFunc{
/*BOOL */ uneg_te,
/*VOID */ _zero1,
/*STRING */ uneg_te,
/*BYTES */ uneg_te,
/*ARRAY */ _absn1,
/*MAP */ _absn1,
/*FUNC */ uneg_te,
@ -106,18 +108,19 @@ func plste(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
}
var plus_dispositions = [mlrval.MT_DIM][mlrval.MT_DIM]BinaryFunc{
// . INT FLOAT BOOL VOID STRING ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {plus_n_ii, plus_f_if, plste, _1___, plste, _absn, _absn, plste, plste, _1___, _1___},
/*FLOAT */ {plus_f_fi, plus_f_ff, plste, _1___, plste, _absn, _absn, plste, plste, _1___, _1___},
/*BOOL */ {plste, plste, plste, plste, plste, _absn, _absn, plste, plste, plste, plste},
/*VOID */ {_2___, _2___, plste, _void, plste, _absn, _absn, plste, plste, plste, _absn},
/*STRING */ {plste, plste, plste, plste, plste, _absn, _absn, plste, plste, plste, plste},
/*ARRAY */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, plste, _absn, _absn, _absn},
/*MAP */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, plste, _absn, _absn, _absn},
/*FUNC */ {plste, plste, plste, plste, plste, plste, plste, plste, plste, plste, plste},
/*ERROR */ {plste, plste, plste, plste, plste, _absn, _absn, plste, plste, plste, plste},
/*NULL */ {_2___, _2___, plste, plste, plste, _absn, _absn, plste, plste, _null, _absn},
/*ABSENT */ {_2___, _2___, plste, _absn, plste, _absn, _absn, plste, plste, _absn, _absn},
// . INT FLOAT BOOL VOID STRING BYTES ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {plus_n_ii, plus_f_if, plste, _1___, plste, plste, _absn, _absn, plste, plste, _1___, _1___},
/*FLOAT */ {plus_f_fi, plus_f_ff, plste, _1___, plste, plste, _absn, _absn, plste, plste, _1___, _1___},
/*BOOL */ {plste, plste, plste, plste, plste, plste, _absn, _absn, plste, plste, plste, plste},
/*VOID */ {_2___, _2___, plste, _void, plste, plste, _absn, _absn, plste, plste, plste, _absn},
/*STRING */ {plste, plste, plste, plste, plste, plste, _absn, _absn, plste, plste, plste, plste},
/*BYTES */ {plste, plste, plste, plste, plste, plste, plste, plste, plste, plste, plste, plste},
/*ARRAY */ {_absn, _absn, _absn, _absn, _absn, plste, _absn, _absn, plste, _absn, _absn, _absn},
/*MAP */ {_absn, _absn, _absn, _absn, _absn, plste, _absn, _absn, plste, _absn, _absn, _absn},
/*FUNC */ {plste, plste, plste, plste, plste, plste, plste, plste, plste, plste, plste, plste},
/*ERROR */ {plste, plste, plste, plste, plste, plste, _absn, _absn, plste, plste, plste, plste},
/*NULL */ {_2___, _2___, plste, plste, plste, plste, _absn, _absn, plste, plste, _null, _absn},
/*ABSENT */ {_2___, _2___, plste, _absn, plste, plste, _absn, _absn, plste, plste, _absn, _absn},
}
func BIF_plus_binary(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
@ -166,18 +169,19 @@ func mnste(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
}
var minus_dispositions = [mlrval.MT_DIM][mlrval.MT_DIM]BinaryFunc{
// . INT FLOAT BOOL VOID STRING ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {minus_n_ii, minus_f_if, mnste, _1___, mnste, _absn, _absn, mnste, mnste, _1___, _1___},
/*FLOAT */ {minus_f_fi, minus_f_ff, mnste, _1___, mnste, _absn, _absn, mnste, mnste, _1___, _1___},
/*BOOL */ {mnste, mnste, mnste, mnste, mnste, _absn, _absn, mnste, mnste, mnste, mnste},
/*VOID */ {_n2__, _n2__, mnste, _void, mnste, _absn, _absn, mnste, mnste, mnste, _absn},
/*STRING */ {mnste, mnste, mnste, mnste, mnste, _absn, _absn, mnste, mnste, mnste, mnste},
/*ARRAY */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, mnste, _absn, _absn, _absn},
/*MAP */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, mnste, _absn, _absn, _absn},
/*FUNC */ {mnste, mnste, mnste, mnste, mnste, mnste, mnste, mnste, mnste, mnste, mnste},
/*ERROR */ {mnste, mnste, mnste, mnste, mnste, _absn, _absn, mnste, mnste, mnste, mnste},
/*NULL */ {_2___, _2___, mnste, mnste, mnste, _absn, _absn, mnste, mnste, _null, _absn},
/*ABSENT */ {_2___, _2___, mnste, _absn, mnste, _absn, _absn, mnste, mnste, _absn, _absn},
// . INT FLOAT BOOL VOID STRING BYTES ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {minus_n_ii, minus_f_if, mnste, _1___, mnste, mnste, _absn, _absn, mnste, mnste, _1___, _1___},
/*FLOAT */ {minus_f_fi, minus_f_ff, mnste, _1___, mnste, mnste, _absn, _absn, mnste, mnste, _1___, _1___},
/*BOOL */ {mnste, mnste, mnste, mnste, mnste, mnste, _absn, _absn, mnste, mnste, mnste, mnste},
/*VOID */ {_n2__, _n2__, mnste, _void, mnste, mnste, _absn, _absn, mnste, mnste, mnste, _absn},
/*STRING */ {mnste, mnste, mnste, mnste, mnste, mnste, _absn, _absn, mnste, mnste, mnste, mnste},
/*BYTES */ {mnste, mnste, mnste, mnste, mnste, mnste, mnste, mnste, mnste, mnste, mnste, mnste},
/*ARRAY */ {_absn, _absn, _absn, _absn, _absn, mnste, _absn, _absn, mnste, _absn, _absn, _absn},
/*MAP */ {_absn, _absn, _absn, _absn, _absn, mnste, _absn, _absn, mnste, _absn, _absn, _absn},
/*FUNC */ {mnste, mnste, mnste, mnste, mnste, mnste, mnste, mnste, mnste, mnste, mnste, mnste},
/*ERROR */ {mnste, mnste, mnste, mnste, mnste, mnste, _absn, _absn, mnste, mnste, mnste, mnste},
/*NULL */ {_2___, _2___, mnste, mnste, mnste, mnste, _absn, _absn, mnste, mnste, _null, _absn},
/*ABSENT */ {_2___, _2___, mnste, _absn, mnste, mnste, _absn, _absn, mnste, mnste, _absn, _absn},
}
func BIF_minus_binary(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
@ -242,18 +246,19 @@ func tmste(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
}
var times_dispositions = [mlrval.MT_DIM][mlrval.MT_DIM]BinaryFunc{
// . INT FLOAT BOOL VOID STRING ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {times_n_ii, times_f_if, tmste, _1___, tmste, _absn, _absn, tmste, tmste, _1___, _1___},
/*FLOAT */ {times_f_fi, times_f_ff, tmste, _1___, tmste, _absn, _absn, tmste, tmste, _1___, _1___},
/*BOOL */ {tmste, tmste, tmste, tmste, tmste, _absn, _absn, tmste, tmste, tmste, tmste},
/*VOID */ {_2___, _2___, tmste, _void, tmste, _absn, _absn, tmste, tmste, tmste, _absn},
/*STRING */ {tmste, tmste, tmste, tmste, tmste, _absn, _absn, tmste, tmste, tmste, tmste},
/*ARRAY */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, tmste, _absn, _absn, _absn},
/*MAP */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, tmste, _absn, _absn, _absn},
/*FUNC */ {tmste, tmste, tmste, tmste, tmste, tmste, tmste, tmste, tmste, tmste, tmste},
/*ERROR */ {tmste, tmste, tmste, tmste, tmste, _absn, _absn, tmste, tmste, tmste, tmste},
/*NULL */ {_2___, _2___, tmste, tmste, tmste, _absn, _absn, tmste, tmste, _null, _absn},
/*ABSENT */ {_2___, _2___, tmste, _absn, tmste, _absn, _absn, tmste, tmste, _absn, _absn},
// . INT FLOAT BOOL VOID STRING BYTES ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {times_n_ii, times_f_if, tmste, _1___, tmste, tmste, _absn, _absn, tmste, tmste, _1___, _1___},
/*FLOAT */ {times_f_fi, times_f_ff, tmste, _1___, tmste, tmste, _absn, _absn, tmste, tmste, _1___, _1___},
/*BOOL */ {tmste, tmste, tmste, tmste, tmste, tmste, _absn, _absn, tmste, tmste, tmste, tmste},
/*VOID */ {_2___, _2___, tmste, _void, tmste, tmste, _absn, _absn, tmste, tmste, tmste, _absn},
/*STRING */ {tmste, tmste, tmste, tmste, tmste, tmste, _absn, _absn, tmste, tmste, tmste, tmste},
/*BYTES */ {tmste, tmste, tmste, tmste, tmste, tmste, tmste, tmste, tmste, tmste, tmste, tmste},
/*ARRAY */ {_absn, _absn, _absn, _absn, _absn, tmste, _absn, _absn, tmste, _absn, _absn, _absn},
/*MAP */ {_absn, _absn, _absn, _absn, _absn, tmste, _absn, _absn, tmste, _absn, _absn, _absn},
/*FUNC */ {tmste, tmste, tmste, tmste, tmste, tmste, tmste, tmste, tmste, tmste, tmste, tmste},
/*ERROR */ {tmste, tmste, tmste, tmste, tmste, tmste, _absn, _absn, tmste, tmste, tmste, tmste},
/*NULL */ {_2___, _2___, tmste, tmste, tmste, tmste, _absn, _absn, tmste, tmste, _null, _absn},
/*ABSENT */ {_2___, _2___, tmste, _absn, tmste, tmste, _absn, _absn, tmste, tmste, _absn, _absn},
}
func BIF_times(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
@ -307,18 +312,19 @@ func dvdte(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
}
var divide_dispositions = [mlrval.MT_DIM][mlrval.MT_DIM]BinaryFunc{
// . INT FLOAT BOOL VOID STRING ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {divide_n_ii, divide_f_if, dvdte, _void, dvdte, _absn, _absn, dvdte, dvdte, _1___, _1___},
/*FLOAT */ {divide_f_fi, divide_f_ff, dvdte, _void, dvdte, _absn, _absn, dvdte, dvdte, _1___, _1___},
/*BOOL */ {dvdte, dvdte, dvdte, dvdte, dvdte, _absn, _absn, dvdte, dvdte, dvdte, dvdte},
/*VOID */ {_void, _void, dvdte, _void, dvdte, _absn, _absn, dvdte, dvdte, dvdte, _absn},
/*STRING */ {dvdte, dvdte, dvdte, dvdte, dvdte, _absn, _absn, dvdte, dvdte, dvdte, dvdte},
/*ARRAY */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, dvdte, _absn, _absn, _absn},
/*MAP */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, dvdte, _absn, _absn, _absn},
/*FUNC */ {dvdte, dvdte, dvdte, dvdte, dvdte, dvdte, dvdte, dvdte, dvdte, dvdte, dvdte},
/*ERROR */ {dvdte, dvdte, dvdte, dvdte, dvdte, _absn, _absn, dvdte, dvdte, dvdte, dvdte},
/*NULL */ {_i0__, _f0__, dvdte, dvdte, dvdte, _absn, _absn, dvdte, dvdte, dvdte, _absn},
/*ABSENT */ {_i0__, _f0__, dvdte, _absn, dvdte, _absn, _absn, dvdte, dvdte, _absn, _absn},
// . INT FLOAT BOOL VOID STRING BYTES ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {divide_n_ii, divide_f_if, dvdte, _void, dvdte, dvdte, _absn, _absn, dvdte, dvdte, _1___, _1___},
/*FLOAT */ {divide_f_fi, divide_f_ff, dvdte, _void, dvdte, dvdte, _absn, _absn, dvdte, dvdte, _1___, _1___},
/*BOOL */ {dvdte, dvdte, dvdte, dvdte, dvdte, dvdte, _absn, _absn, dvdte, dvdte, dvdte, dvdte},
/*VOID */ {_void, _void, dvdte, _void, dvdte, dvdte, _absn, _absn, dvdte, dvdte, dvdte, _absn},
/*STRING */ {dvdte, dvdte, dvdte, dvdte, dvdte, dvdte, _absn, _absn, dvdte, dvdte, dvdte, dvdte},
/*BYTES */ {dvdte, dvdte, dvdte, dvdte, dvdte, dvdte, dvdte, dvdte, dvdte, dvdte, dvdte, dvdte},
/*ARRAY */ {_absn, _absn, _absn, _absn, _absn, dvdte, _absn, _absn, dvdte, _absn, _absn, _absn},
/*MAP */ {_absn, _absn, _absn, _absn, _absn, dvdte, _absn, _absn, dvdte, _absn, _absn, _absn},
/*FUNC */ {dvdte, dvdte, dvdte, dvdte, dvdte, dvdte, dvdte, dvdte, dvdte, dvdte, dvdte, dvdte},
/*ERROR */ {dvdte, dvdte, dvdte, dvdte, dvdte, dvdte, _absn, _absn, dvdte, dvdte, dvdte, dvdte},
/*NULL */ {_i0__, _f0__, dvdte, dvdte, dvdte, dvdte, _absn, _absn, dvdte, dvdte, dvdte, _absn},
/*ABSENT */ {_i0__, _f0__, dvdte, _absn, dvdte, dvdte, _absn, _absn, dvdte, dvdte, _absn, _absn},
}
func BIF_divide(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
@ -371,18 +377,19 @@ func idvte(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
}
var int_divide_dispositions = [mlrval.MT_DIM][mlrval.MT_DIM]BinaryFunc{
// . INT FLOAT BOOL VOID STRING ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {int_divide_n_ii, int_divide_f_if, idvte, _void, idvte, _absn, _absn, idvte, idvte, idvte, _1___},
/*FLOAT */ {int_divide_f_fi, int_divide_f_ff, idvte, _void, idvte, _absn, _absn, idvte, idvte, idvte, _1___},
/*BOOL */ {idvte, idvte, idvte, idvte, idvte, _absn, _absn, idvte, idvte, idvte, idvte},
/*VOID */ {_void, _void, idvte, _void, idvte, _absn, _absn, idvte, idvte, idvte, _absn},
/*STRING */ {idvte, idvte, idvte, idvte, idvte, _absn, _absn, idvte, idvte, idvte, idvte},
/*ARRAY */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, idvte, _absn, _absn, _absn},
/*MAP */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, idvte, _absn, _absn, _absn},
/*FUNC */ {idvte, idvte, idvte, idvte, idvte, idvte, idvte, idvte, idvte, idvte, idvte},
/*ERROR */ {idvte, idvte, idvte, idvte, idvte, _absn, _absn, idvte, idvte, idvte, idvte},
/*NULL */ {idvte, idvte, idvte, idvte, idvte, _absn, _absn, idvte, idvte, idvte, _absn},
/*ABSENT */ {_i0__, _f0__, idvte, _absn, idvte, _absn, _absn, idvte, idvte, _absn, _absn},
// . INT FLOAT BOOL VOID STRING BYTES ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {int_divide_n_ii, int_divide_f_if, idvte, _void, idvte, idvte, _absn, _absn, idvte, idvte, idvte, _1___},
/*FLOAT */ {int_divide_f_fi, int_divide_f_ff, idvte, _void, idvte, idvte, _absn, _absn, idvte, idvte, idvte, _1___},
/*BOOL */ {idvte, idvte, idvte, idvte, idvte, idvte, _absn, _absn, idvte, idvte, idvte, idvte},
/*VOID */ {_void, _void, idvte, _void, idvte, idvte, _absn, _absn, idvte, idvte, idvte, _absn},
/*STRING */ {idvte, idvte, idvte, idvte, idvte, idvte, _absn, _absn, idvte, idvte, idvte, idvte},
/*BYTES */ {idvte, idvte, idvte, idvte, idvte, idvte, idvte, idvte, idvte, idvte, idvte, idvte},
/*ARRAY */ {_absn, _absn, _absn, _absn, _absn, idvte, _absn, _absn, idvte, _absn, _absn, _absn},
/*MAP */ {_absn, _absn, _absn, _absn, _absn, idvte, _absn, _absn, idvte, _absn, _absn, _absn},
/*FUNC */ {idvte, idvte, idvte, idvte, idvte, idvte, idvte, idvte, idvte, idvte, idvte, idvte},
/*ERROR */ {idvte, idvte, idvte, idvte, idvte, idvte, _absn, _absn, idvte, idvte, idvte, idvte},
/*NULL */ {idvte, idvte, idvte, idvte, idvte, idvte, _absn, _absn, idvte, idvte, idvte, _absn},
/*ABSENT */ {_i0__, _f0__, idvte, _absn, idvte, idvte, _absn, _absn, idvte, idvte, _absn, _absn},
}
func BIF_int_divide(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
@ -410,18 +417,19 @@ func dplte(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
}
var dot_plus_dispositions = [mlrval.MT_DIM][mlrval.MT_DIM]BinaryFunc{
// . INT FLOAT BOOL VOID STRING ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {dotplus_i_ii, dotplus_f_if, dplte, _1___, dplte, _absn, _absn, dplte, dplte, _1___, _1___},
/*FLOAT */ {dotplus_f_fi, dotplus_f_ff, dplte, _1___, dplte, _absn, _absn, dplte, dplte, _1___, _1___},
/*BOOL */ {dplte, dplte, dplte, dplte, dplte, _absn, _absn, dplte, dplte, dplte, dplte},
/*VOID */ {_2___, _2___, dplte, _void, dplte, _absn, _absn, dplte, dplte, dplte, _absn},
/*STRING */ {dplte, dplte, dplte, dplte, dplte, _absn, _absn, dplte, dplte, dplte, dplte},
/*ARRAY */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, dplte, _absn, _absn, _absn},
/*MAP */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, dplte, _absn, _absn, _absn},
/*FUNC */ {dplte, dplte, dplte, dplte, dplte, dplte, dplte, dplte, dplte, dplte, dplte},
/*ERROR */ {dplte, dplte, dplte, dplte, dplte, _absn, _absn, dplte, dplte, dplte, dplte},
/*NULL */ {_2___, _2___, dplte, dplte, dplte, _absn, _absn, dplte, dplte, _null, _absn},
/*ABSENT */ {_2___, _2___, dplte, _absn, dplte, _absn, _absn, dplte, dplte, _absn, _absn},
// . INT FLOAT BOOL VOID STRING BYTES ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {dotplus_i_ii, dotplus_f_if, dplte, _1___, dplte, dplte, _absn, _absn, dplte, dplte, _1___, _1___},
/*FLOAT */ {dotplus_f_fi, dotplus_f_ff, dplte, _1___, dplte, dplte, _absn, _absn, dplte, dplte, _1___, _1___},
/*BOOL */ {dplte, dplte, dplte, dplte, dplte, dplte, _absn, _absn, dplte, dplte, dplte, dplte},
/*VOID */ {_2___, _2___, dplte, _void, dplte, dplte, _absn, _absn, dplte, dplte, dplte, _absn},
/*STRING */ {dplte, dplte, dplte, dplte, dplte, dplte, _absn, _absn, dplte, dplte, dplte, dplte},
/*BYTES */ {dplte, dplte, dplte, dplte, dplte, dplte, dplte, dplte, dplte, dplte, dplte, dplte},
/*ARRAY */ {_absn, _absn, _absn, _absn, _absn, dplte, _absn, _absn, dplte, _absn, _absn, _absn},
/*MAP */ {_absn, _absn, _absn, _absn, _absn, dplte, _absn, _absn, dplte, _absn, _absn, _absn},
/*FUNC */ {dplte, dplte, dplte, dplte, dplte, dplte, dplte, dplte, dplte, dplte, dplte, dplte},
/*ERROR */ {dplte, dplte, dplte, dplte, dplte, dplte, _absn, _absn, dplte, dplte, dplte, dplte},
/*NULL */ {_2___, _2___, dplte, dplte, dplte, dplte, _absn, _absn, dplte, dplte, _null, _absn},
/*ABSENT */ {_2___, _2___, dplte, _absn, dplte, dplte, _absn, _absn, dplte, dplte, _absn, _absn},
}
func BIF_dot_plus(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
@ -449,18 +457,19 @@ func dmnte(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
}
var dotminus_dispositions = [mlrval.MT_DIM][mlrval.MT_DIM]BinaryFunc{
// . INT FLOAT BOOL VOID STRING ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {dotminus_i_ii, dotminus_f_if, dmnte, _1___, dmnte, _absn, _absn, dmnte, dmnte, _1___, _1___},
/*FLOAT */ {dotminus_f_fi, dotminus_f_ff, dmnte, _1___, dmnte, _absn, _absn, dmnte, dmnte, _1___, _1___},
/*BOOL */ {dmnte, dmnte, dmnte, dmnte, dmnte, _absn, _absn, dmnte, dmnte, dmnte, dmnte},
/*VOID */ {_n2__, _n2__, dmnte, _void, dmnte, _absn, _absn, dmnte, dmnte, dmnte, _absn},
/*STRING */ {dmnte, dmnte, dmnte, dmnte, dmnte, _absn, _absn, dmnte, dmnte, dmnte, dmnte},
/*ARRAY */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, dmnte, _absn, _absn, _absn},
/*MAP */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, dmnte, _absn, _absn, _absn},
/*FUNC */ {dmnte, dmnte, dmnte, dmnte, dmnte, dmnte, dmnte, dmnte, dmnte, dmnte, dmnte},
/*ERROR */ {dmnte, dmnte, dmnte, dmnte, dmnte, _absn, _absn, dmnte, dmnte, dmnte, dmnte},
/*NULL */ {_n2__, _n2__, dmnte, dmnte, dmnte, _absn, _absn, dmnte, dmnte, _null, _absn},
/*ABSENT */ {_n2__, _n2__, dmnte, _absn, dmnte, _absn, _absn, dmnte, dmnte, _absn, _absn},
// . INT FLOAT BOOL VOID STRING BYTES ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {dotminus_i_ii, dotminus_f_if, dmnte, _1___, dmnte, dmnte, _absn, _absn, dmnte, dmnte, _1___, _1___},
/*FLOAT */ {dotminus_f_fi, dotminus_f_ff, dmnte, _1___, dmnte, dmnte, _absn, _absn, dmnte, dmnte, _1___, _1___},
/*BOOL */ {dmnte, dmnte, dmnte, dmnte, dmnte, dmnte, _absn, _absn, dmnte, dmnte, dmnte, dmnte},
/*VOID */ {_n2__, _n2__, dmnte, _void, dmnte, dmnte, _absn, _absn, dmnte, dmnte, dmnte, _absn},
/*STRING */ {dmnte, dmnte, dmnte, dmnte, dmnte, dmnte, _absn, _absn, dmnte, dmnte, dmnte, dmnte},
/*BYTES */ {dmnte, dmnte, dmnte, dmnte, dmnte, dmnte, dmnte, dmnte, dmnte, dmnte, dmnte, dmnte},
/*ARRAY */ {_absn, _absn, _absn, _absn, _absn, dmnte, _absn, _absn, dmnte, _absn, _absn, _absn},
/*MAP */ {_absn, _absn, _absn, _absn, _absn, dmnte, _absn, _absn, dmnte, _absn, _absn, _absn},
/*FUNC */ {dmnte, dmnte, dmnte, dmnte, dmnte, dmnte, dmnte, dmnte, dmnte, dmnte, dmnte, dmnte},
/*ERROR */ {dmnte, dmnte, dmnte, dmnte, dmnte, dmnte, _absn, _absn, dmnte, dmnte, dmnte, dmnte},
/*NULL */ {_n2__, _n2__, dmnte, dmnte, dmnte, dmnte, _absn, _absn, dmnte, dmnte, _null, _absn},
/*ABSENT */ {_n2__, _n2__, dmnte, _absn, dmnte, dmnte, _absn, _absn, dmnte, dmnte, _absn, _absn},
}
func BIF_dot_minus(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
@ -488,18 +497,19 @@ func dttte(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
}
var dottimes_dispositions = [mlrval.MT_DIM][mlrval.MT_DIM]BinaryFunc{
// . INT FLOAT BOOL VOID STRING ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {dottimes_i_ii, dottimes_f_if, dttte, _1___, dttte, _absn, _absn, dttte, dttte, _1___, _1___},
/*FLOAT */ {dottimes_f_fi, dottimes_f_ff, dttte, _1___, dttte, _absn, _absn, dttte, dttte, _1___, _1___},
/*BOOL */ {dttte, dttte, dttte, dttte, dttte, _absn, _absn, dttte, dttte, dttte, dttte},
/*VOID */ {_n2__, _n2__, dttte, _void, dttte, _absn, _absn, dttte, dttte, dttte, _absn},
/*STRING */ {dttte, dttte, dttte, dttte, dttte, _absn, _absn, dttte, dttte, dttte, dttte},
/*ARRAY */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, dttte, _absn, _absn, _absn},
/*MAP */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, dttte, _absn, _absn, _absn},
/*FUNC */ {dttte, dttte, dttte, dttte, dttte, dttte, dttte, dttte, dttte, dttte, dttte},
/*ERROR */ {dttte, dttte, dttte, dttte, dttte, _absn, _absn, dttte, dttte, dttte, dttte},
/*NULL */ {_2___, _2___, dttte, dttte, dttte, _absn, _absn, dttte, dttte, dttte, _absn},
/*ABSENT */ {_2___, _2___, dttte, _absn, dttte, _absn, _absn, dttte, dttte, _absn, _absn},
// . INT FLOAT BOOL VOID STRING BYTES ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {dottimes_i_ii, dottimes_f_if, dttte, _1___, dttte, dttte, _absn, _absn, dttte, dttte, _1___, _1___},
/*FLOAT */ {dottimes_f_fi, dottimes_f_ff, dttte, _1___, dttte, dttte, _absn, _absn, dttte, dttte, _1___, _1___},
/*BOOL */ {dttte, dttte, dttte, dttte, dttte, dttte, _absn, _absn, dttte, dttte, dttte, dttte},
/*VOID */ {_n2__, _n2__, dttte, _void, dttte, dttte, _absn, _absn, dttte, dttte, dttte, _absn},
/*STRING */ {dttte, dttte, dttte, dttte, dttte, dttte, _absn, _absn, dttte, dttte, dttte, dttte},
/*BYTES */ {dttte, dttte, dttte, dttte, dttte, dttte, dttte, dttte, dttte, dttte, dttte, dttte},
/*ARRAY */ {_absn, _absn, _absn, _absn, _absn, dttte, _absn, _absn, dttte, _absn, _absn, _absn},
/*MAP */ {_absn, _absn, _absn, _absn, _absn, dttte, _absn, _absn, dttte, _absn, _absn, _absn},
/*FUNC */ {dttte, dttte, dttte, dttte, dttte, dttte, dttte, dttte, dttte, dttte, dttte, dttte},
/*ERROR */ {dttte, dttte, dttte, dttte, dttte, dttte, _absn, _absn, dttte, dttte, dttte, dttte},
/*NULL */ {_2___, _2___, dttte, dttte, dttte, dttte, _absn, _absn, dttte, dttte, dttte, _absn},
/*ABSENT */ {_2___, _2___, dttte, _absn, dttte, dttte, _absn, _absn, dttte, dttte, _absn, _absn},
}
func BIF_dot_times(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
@ -527,18 +537,19 @@ func ddvte(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
}
var dotdivide_dispositions = [mlrval.MT_DIM][mlrval.MT_DIM]BinaryFunc{
// . INT FLOAT BOOL VOID STRING ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {dotdivide_i_ii, dotdivide_f_if, ddvte, _void, ddvte, _absn, _absn, ddvte, ddvte, ddvte, _1___},
/*FLOAT */ {dotdivide_f_fi, dotdivide_f_ff, ddvte, _void, ddvte, _absn, _absn, ddvte, ddvte, ddvte, _1___},
/*BOOL */ {ddvte, ddvte, ddvte, ddvte, ddvte, _absn, _absn, ddvte, ddvte, ddvte, ddvte},
/*VOID */ {_void, _void, ddvte, _void, ddvte, _absn, _absn, ddvte, ddvte, ddvte, _absn},
/*STRING */ {ddvte, ddvte, ddvte, ddvte, ddvte, _absn, _absn, ddvte, ddvte, ddvte, ddvte},
/*ARRAY */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, ddvte, _absn, _absn, _absn},
/*MAP */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, ddvte, _absn, _absn, _absn},
/*FUNC */ {ddvte, ddvte, ddvte, ddvte, ddvte, ddvte, ddvte, ddvte, ddvte, ddvte, ddvte},
/*ERROR */ {ddvte, ddvte, ddvte, ddvte, ddvte, _absn, _absn, ddvte, ddvte, ddvte, ddvte},
/*NULL */ {ddvte, ddvte, ddvte, ddvte, ddvte, _absn, _absn, ddvte, ddvte, ddvte, _absn},
/*ABSENT */ {_2___, _2___, ddvte, _absn, ddvte, _absn, _absn, ddvte, ddvte, _absn, _absn},
// . INT FLOAT BOOL VOID STRING BYTES ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {dotdivide_i_ii, dotdivide_f_if, ddvte, _void, ddvte, ddvte, _absn, _absn, ddvte, ddvte, ddvte, _1___},
/*FLOAT */ {dotdivide_f_fi, dotdivide_f_ff, ddvte, _void, ddvte, ddvte, _absn, _absn, ddvte, ddvte, ddvte, _1___},
/*BOOL */ {ddvte, ddvte, ddvte, ddvte, ddvte, ddvte, _absn, _absn, ddvte, ddvte, ddvte, ddvte},
/*VOID */ {_void, _void, ddvte, _void, ddvte, ddvte, _absn, _absn, ddvte, ddvte, ddvte, _absn},
/*STRING */ {ddvte, ddvte, ddvte, ddvte, ddvte, ddvte, _absn, _absn, ddvte, ddvte, ddvte, ddvte},
/*BYTES */ {ddvte, ddvte, ddvte, ddvte, ddvte, ddvte, ddvte, ddvte, ddvte, ddvte, ddvte, ddvte},
/*ARRAY */ {_absn, _absn, _absn, _absn, _absn, ddvte, _absn, _absn, ddvte, _absn, _absn, _absn},
/*MAP */ {_absn, _absn, _absn, _absn, _absn, ddvte, _absn, _absn, ddvte, _absn, _absn, _absn},
/*FUNC */ {ddvte, ddvte, ddvte, ddvte, ddvte, ddvte, ddvte, ddvte, ddvte, ddvte, ddvte, ddvte},
/*ERROR */ {ddvte, ddvte, ddvte, ddvte, ddvte, ddvte, _absn, _absn, ddvte, ddvte, ddvte, ddvte},
/*NULL */ {ddvte, ddvte, ddvte, ddvte, ddvte, ddvte, _absn, _absn, ddvte, ddvte, ddvte, _absn},
/*ABSENT */ {_2___, _2___, ddvte, _absn, ddvte, ddvte, _absn, _absn, ddvte, ddvte, _absn, _absn},
}
func BIF_dot_divide(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
@ -594,18 +605,19 @@ func modte(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
}
var modulus_dispositions = [mlrval.MT_DIM][mlrval.MT_DIM]BinaryFunc{
// . INT FLOAT BOOL VOID STRING ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {modulus_i_ii, modulus_f_if, modte, _void, modte, _absn, _absn, modte, modte, modte, _1___},
/*FLOAT */ {modulus_f_fi, modulus_f_ff, modte, _void, modte, _absn, _absn, modte, modte, modte, _1___},
/*BOOL */ {modte, modte, modte, modte, modte, _absn, _absn, modte, modte, modte, modte},
/*VOID */ {_void, _void, modte, _void, modte, _absn, _absn, modte, modte, modte, _absn},
/*STRING */ {modte, modte, modte, modte, modte, _absn, _absn, modte, modte, modte, modte},
/*ARRAY */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, modte, _absn, _absn, _absn},
/*MAP */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, modte, _absn, _absn, _absn},
/*FUNC */ {modte, modte, modte, modte, modte, modte, modte, modte, modte, modte, modte},
/*ERROR */ {modte, modte, modte, modte, modte, _absn, _absn, modte, modte, modte, modte},
/*NULL */ {modte, modte, modte, modte, modte, _absn, _absn, modte, modte, modte, _absn},
/*ABSENT */ {_i0__, _f0__, modte, _absn, modte, _absn, _absn, modte, modte, _absn, _absn},
// . INT FLOAT BOOL VOID STRING BYTES ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {modulus_i_ii, modulus_f_if, modte, _void, modte, modte, _absn, _absn, modte, modte, modte, _1___},
/*FLOAT */ {modulus_f_fi, modulus_f_ff, modte, _void, modte, modte, _absn, _absn, modte, modte, modte, _1___},
/*BOOL */ {modte, modte, modte, modte, modte, modte, _absn, _absn, modte, modte, modte, modte},
/*VOID */ {_void, _void, modte, _void, modte, modte, _absn, _absn, modte, modte, modte, _absn},
/*STRING */ {modte, modte, modte, modte, modte, modte, _absn, _absn, modte, modte, modte, modte},
/*BYTES */ {modte, modte, modte, modte, modte, modte, modte, modte, modte, modte, modte, modte},
/*ARRAY */ {_absn, _absn, _absn, _absn, _absn, modte, _absn, _absn, modte, _absn, _absn, _absn},
/*MAP */ {_absn, _absn, _absn, _absn, _absn, modte, _absn, _absn, modte, _absn, _absn, _absn},
/*FUNC */ {modte, modte, modte, modte, modte, modte, modte, modte, modte, modte, modte, modte},
/*ERROR */ {modte, modte, modte, modte, modte, modte, _absn, _absn, modte, modte, modte, modte},
/*NULL */ {modte, modte, modte, modte, modte, modte, _absn, _absn, modte, modte, modte, _absn},
/*ABSENT */ {_i0__, _f0__, modte, _absn, modte, modte, _absn, _absn, modte, modte, _absn, _absn},
}
func BIF_modulus(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
@ -769,18 +781,19 @@ func min_te(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
}
var min_dispositions = [mlrval.MT_DIM][mlrval.MT_DIM]BinaryFunc{
// . INT FLOAT BOOL VOID STRING ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {min_i_ii, min_f_if, _1___, _1___, _1___, _absn, _absn, min_te, min_te, _1___, _1___},
/*FLOAT */ {min_f_fi, min_f_ff, _1___, _1___, _1___, _absn, _absn, min_te, min_te, _1___, _1___},
/*BOOL */ {_2___, _2___, min_b_bb, _1___, _1___, _absn, _absn, min_te, min_te, _1___, _1___},
/*VOID */ {_2___, _2___, _2___, _void, _void, _absn, _absn, min_te, min_te, _1___, _1___},
/*STRING */ {_2___, _2___, _2___, _void, min_s_ss, _absn, _absn, min_te, min_te, _1___, _1___},
/*ARRAY */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, min_te, _absn, _absn, _absn},
/*MAP */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, min_te, _absn, _absn, _absn},
/*FUNC */ {min_te, min_te, min_te, min_te, min_te, min_te, min_te, min_te, min_te, min_te, min_te},
/*ERROR */ {min_te, min_te, min_te, min_te, min_te, _absn, _absn, min_te, min_te, min_te, min_te},
/*NULL */ {_2___, _2___, _2___, _2___, _2___, _absn, _absn, min_te, min_te, _null, _null},
/*ABSENT */ {_2___, _2___, _2___, _2___, _2___, _absn, _absn, min_te, min_te, _null, _absn},
// . INT FLOAT BOOL VOID STRING BYTES ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {min_i_ii, min_f_if, _1___, _1___, _1___, min_te, _absn, _absn, min_te, min_te, _1___, _1___},
/*FLOAT */ {min_f_fi, min_f_ff, _1___, _1___, _1___, min_te, _absn, _absn, min_te, min_te, _1___, _1___},
/*BOOL */ {_2___, _2___, min_b_bb, _1___, _1___, min_te, _absn, _absn, min_te, min_te, _1___, _1___},
/*VOID */ {_2___, _2___, _2___, _void, _void, min_te, _absn, _absn, min_te, min_te, _1___, _1___},
/*STRING */ {_2___, _2___, _2___, _void, min_s_ss, min_te, _absn, _absn, min_te, min_te, _1___, _1___},
/*BYTES */ {min_te, min_te, min_te, min_te, min_te, min_te, min_te, min_te, min_te, min_te, min_te, min_te},
/*ARRAY */ {_absn, _absn, _absn, _absn, _absn, min_te, _absn, _absn, min_te, _absn, _absn, _absn},
/*MAP */ {_absn, _absn, _absn, _absn, _absn, min_te, _absn, _absn, min_te, _absn, _absn, _absn},
/*FUNC */ {min_te, min_te, min_te, min_te, min_te, min_te, min_te, min_te, min_te, min_te, min_te, min_te},
/*ERROR */ {min_te, min_te, min_te, min_te, min_te, min_te, _absn, _absn, min_te, min_te, min_te, min_te},
/*NULL */ {_2___, _2___, _2___, _2___, _2___, min_te, _absn, _absn, min_te, min_te, _null, _null},
/*ABSENT */ {_2___, _2___, _2___, _2___, _2___, min_te, _absn, _absn, min_te, min_te, _null, _absn},
}
// BIF_min_binary is not a direct DSL function. It's a helper here,
@ -839,6 +852,7 @@ func init() {
/*BOOL */ _1u___,
/*VOID */ _1u___,
/*STRING */ _1u___,
/*BYTES */ min_unary_te,
/*ARRAY */ bif_min_unary_array,
/*MAP */ bif_min_unary_map,
/*FUNC */ min_unary_te,
@ -934,18 +948,19 @@ func max_te(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
}
var max_dispositions = [mlrval.MT_DIM][mlrval.MT_DIM]BinaryFunc{
// . INT FLOAT BOOL VOID STRING ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {max_i_ii, max_f_if, _2___, _2___, _2___, _absn, _absn, max_te, max_te, _null, _1___},
/*FLOAT */ {max_f_fi, max_f_ff, _2___, _2___, _2___, _absn, _absn, max_te, max_te, _null, _1___},
/*BOOL */ {_1___, _1___, max_b_bb, _2___, _2___, _absn, _absn, max_te, max_te, _null, _1___},
/*VOID */ {_1___, _1___, _1___, _void, _2___, _absn, _absn, max_te, max_te, _null, _1___},
/*STRING */ {_1___, _1___, _1___, _1___, max_s_ss, _absn, _absn, max_te, max_te, _null, _1___},
/*ARRAY */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, max_te, _absn, _absn, _absn},
/*MAP */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, max_te, _absn, _absn, _absn},
/*FUNC */ {max_te, max_te, max_te, max_te, max_te, max_te, max_te, max_te, max_te, max_te, max_te},
/*ERROR */ {max_te, max_te, max_te, max_te, max_te, _absn, _absn, max_te, max_te, _null, max_te},
/*NULL */ {_null, _null, _null, _null, _null, _absn, _absn, max_te, _null, _null, _absn},
/*ABSENT */ {_2___, _2___, _2___, _2___, _2___, _absn, _absn, max_te, max_te, _absn, _absn},
// . INT FLOAT BOOL VOID STRING BYTES ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {max_i_ii, max_f_if, _2___, _2___, _2___, max_te, _absn, _absn, max_te, max_te, _null, _1___},
/*FLOAT */ {max_f_fi, max_f_ff, _2___, _2___, _2___, max_te, _absn, _absn, max_te, max_te, _null, _1___},
/*BOOL */ {_1___, _1___, max_b_bb, _2___, _2___, max_te, _absn, _absn, max_te, max_te, _null, _1___},
/*VOID */ {_1___, _1___, _1___, _void, _2___, max_te, _absn, _absn, max_te, max_te, _null, _1___},
/*STRING */ {_1___, _1___, _1___, _1___, max_s_ss, max_te, _absn, _absn, max_te, max_te, _null, _1___},
/*BYTES */ {max_te, max_te, max_te, max_te, max_te, max_te, max_te, max_te, max_te, max_te, max_te, max_te},
/*ARRAY */ {_absn, _absn, _absn, _absn, _absn, max_te, _absn, _absn, max_te, _absn, _absn, _absn},
/*MAP */ {_absn, _absn, _absn, _absn, _absn, max_te, _absn, _absn, max_te, _absn, _absn, _absn},
/*FUNC */ {max_te, max_te, max_te, max_te, max_te, max_te, max_te, max_te, max_te, max_te, max_te, max_te},
/*ERROR */ {max_te, max_te, max_te, max_te, max_te, max_te, _absn, _absn, max_te, max_te, _null, max_te},
/*NULL */ {_null, _null, _null, _null, _null, max_te, _absn, _absn, max_te, _null, _null, _absn},
/*ABSENT */ {_2___, _2___, _2___, _2___, _2___, max_te, _absn, _absn, max_te, max_te, _absn, _absn},
}
// BIF_max_binary is not a direct DSL function. It's a helper here,
@ -1004,6 +1019,7 @@ func init() {
/*BOOL */ _1u___,
/*VOID */ _1u___,
/*STRING */ _1u___,
/*BYTES */ max_unary_te,
/*ARRAY */ bif_max_unary_array,
/*MAP */ bif_max_unary_map,
/*FUNC */ max_unary_te,

32
pkg/bifs/base64.go Normal file
View file

@ -0,0 +1,32 @@
package bifs
import (
"encoding/base64"
"github.com/johnkerl/miller/v6/pkg/mlrval"
)
func BIF_base64_encode(input1 *mlrval.Mlrval) *mlrval.Mlrval {
if input1.IsBytes() {
return mlrval.FromString(
base64.StdEncoding.EncodeToString(input1.AcquireBytesValue()),
)
}
if !input1.IsStringOrVoid() {
return mlrval.FromNotStringError("base64_encode", input1)
}
return mlrval.FromString(
base64.StdEncoding.EncodeToString([]byte(input1.AcquireStringValue())),
)
}
func BIF_base64_decode(input1 *mlrval.Mlrval) *mlrval.Mlrval {
if !input1.IsStringOrVoid() {
return mlrval.FromNotStringError("base64_decode", input1)
}
decoded, err := base64.StdEncoding.DecodeString(input1.AcquireStringValue())
if err != nil {
return mlrval.FromError(err)
}
return mlrval.FromBytes(decoded)
}

127
pkg/bifs/base64_test.go Normal file
View file

@ -0,0 +1,127 @@
package bifs
import (
"testing"
"github.com/stretchr/testify/assert"
"github.com/johnkerl/miller/v6/pkg/mlrval"
)
func TestBIF_base64_encode(t *testing.T) {
input1 := mlrval.FromDeferredType("")
output := BIF_base64_encode(input1)
stringval, ok := output.GetStringValue()
assert.True(t, ok)
assert.Equal(t, "", stringval)
input1 = mlrval.FromDeferredType("hello")
output = BIF_base64_encode(input1)
stringval, ok = output.GetStringValue()
assert.True(t, ok)
assert.Equal(t, "aGVsbG8=", stringval)
// Bytes input encodes the raw bytes
output = BIF_base64_encode(mlrval.FromBytes([]byte{0xff}))
stringval, ok = output.GetStringValue()
assert.True(t, ok)
assert.Equal(t, "/w==", stringval)
}
func TestBIF_base64_decode(t *testing.T) {
// Decode always yields bytes
output := BIF_base64_decode(mlrval.FromDeferredType("aGVsbG8="))
assert.True(t, output.IsBytes())
assert.Equal(t, []byte("hello"), output.AcquireBytesValue())
// string() recovers the text for UTF-8 payloads
assert.Equal(t, "hello", BIF_string(output).String())
// Non-UTF-8 payloads round-trip losslessly
output = BIF_base64_decode(mlrval.FromDeferredType("/w=="))
assert.True(t, output.IsBytes())
assert.Equal(t, []byte{0xff}, output.AcquireBytesValue())
assert.Equal(t, "ff", output.String())
// Malformed base64 input results in an error mlrval
output = BIF_base64_decode(mlrval.FromDeferredType("not!valid!base64"))
assert.True(t, output.IsError())
}
func TestBIF_base64_round_trip(t *testing.T) {
input1 := mlrval.FromBytes([]byte{0x00, 0x01, 0xfe, 0xff})
output := BIF_base64_decode(BIF_base64_encode(input1))
assert.True(t, output.IsBytes())
assert.Equal(t, []byte{0x00, 0x01, 0xfe, 0xff}, output.AcquireBytesValue())
}
func TestBIF_hex_encode_decode(t *testing.T) {
output := BIF_hex_encode(mlrval.FromBytes([]byte{0xde, 0xad, 0xbe, 0xef}))
stringval, ok := output.GetStringValue()
assert.True(t, ok)
assert.Equal(t, "deadbeef", stringval)
output = BIF_hex_encode(mlrval.FromDeferredType("hi"))
stringval, ok = output.GetStringValue()
assert.True(t, ok)
assert.Equal(t, "6869", stringval)
output = BIF_hex_decode(mlrval.FromDeferredType("deadbeef"))
assert.True(t, output.IsBytes())
assert.Equal(t, []byte{0xde, 0xad, 0xbe, 0xef}, output.AcquireBytesValue())
// Odd length and non-hex characters are errors
assert.True(t, BIF_hex_decode(mlrval.FromDeferredType("abc")).IsError())
assert.True(t, BIF_hex_decode(mlrval.FromDeferredType("zz")).IsError())
}
func TestBIF_bytes_conversions(t *testing.T) {
// bytes(string) → bytes; string(bytes) is the inverse
b := BIF_bytes(mlrval.FromString("héllo"))
assert.True(t, b.IsBytes())
assert.Equal(t, []byte("héllo"), b.AcquireBytesValue())
assert.Equal(t, "héllo", BIF_string(b).String())
// bytes(bytes) is the identity
assert.Equal(t, b, BIF_bytes(b))
// bytes of a number is a type error
assert.True(t, BIF_bytes(mlrval.FromInt(5)).IsError())
}
func TestBIF_bytes_strlen_substr(t *testing.T) {
// héllo is 5 runes, 6 bytes
s := mlrval.FromString("héllo")
b := BIF_bytes(s)
assert.Equal(t, "5", BIF_strlen(s).String())
assert.Equal(t, "6", BIF_strlen(b).String())
// Byte-oriented slicing
payload := mlrval.FromBytes([]byte{0x00, 0x01, 0x02, 0x03})
sliced := BIF_substr_0_up(payload, mlrval.FromInt(1), mlrval.FromInt(2))
assert.True(t, sliced.IsBytes())
assert.Equal(t, []byte{0x01, 0x02}, sliced.AcquireBytesValue())
sliced = BIF_substr_1_up(payload, mlrval.FromInt(1), mlrval.FromInt(2))
assert.True(t, sliced.IsBytes())
assert.Equal(t, []byte{0x00, 0x01}, sliced.AcquireBytesValue())
// Empty slice preserves the bytes type
sliced = BIF_substr_0_up(payload, mlrval.FromInt(3), mlrval.FromInt(1))
assert.True(t, sliced.IsBytes())
assert.Equal(t, 0, len(sliced.AcquireBytesValue()))
}
func TestBIF_bytes_hashing(t *testing.T) {
// Hash of bytes equals hash of the same payload as a string
fromString := BIF_md5(mlrval.FromDeferredType("miller"))
fromBytes := BIF_md5(BIF_bytes(mlrval.FromString("miller")))
assert.Equal(t, fromString.String(), fromBytes.String())
assert.Equal(t, "f0af962ddbc82430e947390b2f3f6e49", fromBytes.String())
// Non-UTF-8 bytes hash without corruption
output := BIF_sha256(mlrval.FromBytes([]byte{0xff}))
stringval, ok := output.GetStringValue()
assert.True(t, ok)
assert.Equal(t, "a8100ae6aa1940d0b663bb31cd466142ebbdbd5187131b92d93818987832eb89", stringval)
}

152
pkg/bifs/bits.go
View file

@ -20,6 +20,7 @@ var bitwise_not_dispositions = [mlrval.MT_DIM]UnaryFunc{
/*BOOL */ bitwise_not_te,
/*VOID */ _void1,
/*STRING */ bitwise_not_te,
/*BYTES */ bitwise_not_te,
/*ARRAY */ _absn1,
/*MAP */ _absn1,
/*FUNC */ bitwise_not_te,
@ -63,6 +64,7 @@ var bitcount_dispositions = [mlrval.MT_DIM]UnaryFunc{
/*BOOL */ bitcount_te,
/*VOID */ _void1,
/*STRING */ bitcount_te,
/*BYTES */ bitcount_te,
/*ARRAY */ _absn1,
/*MAP */ _absn1,
/*FUNC */ bitcount_te,
@ -86,18 +88,19 @@ func bwandte(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
}
var bitwise_and_dispositions = [mlrval.MT_DIM][mlrval.MT_DIM]BinaryFunc{
// . INT FLOAT BOOL VOID STRING ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {bitwise_and_i_ii, bwandte, bwandte, _void, bwandte, _absn, _absn, bwandte, bwandte, bwandte, _1___},
/*FLOAT */ {bwandte, bwandte, bwandte, _void, bwandte, _absn, _absn, bwandte, bwandte, bwandte, bwandte},
/*BOOL */ {bwandte, bwandte, bwandte, bwandte, bwandte, _absn, _absn, bwandte, bwandte, bwandte, bwandte},
/*VOID */ {_void, _void, bwandte, _void, bwandte, _absn, _absn, bwandte, bwandte, bwandte, _absn},
/*STRING */ {bwandte, bwandte, bwandte, bwandte, bwandte, _absn, _absn, bwandte, bwandte, bwandte, bwandte},
/*ARRAY */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, bwandte, _absn, bwandte, _absn},
/*MAP */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, bwandte, _absn, bwandte, _absn},
/*FUNC */ {bwandte, bwandte, bwandte, bwandte, bwandte, bwandte, bwandte, bwandte, bwandte, bwandte, bwandte},
/*ERROR */ {bwandte, bwandte, bwandte, bwandte, bwandte, _absn, _absn, bwandte, bwandte, bwandte, bwandte},
/*NULL */ {bwandte, bwandte, bwandte, bwandte, bwandte, bwandte, bwandte, bwandte, bwandte, bwandte, _absn},
/*ABSENT */ {_2___, bwandte, bwandte, _absn, bwandte, _absn, _absn, bwandte, bwandte, _absn, _absn},
// . INT FLOAT BOOL VOID STRING BYTES ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {bitwise_and_i_ii, bwandte, bwandte, _void, bwandte, bwandte, _absn, _absn, bwandte, bwandte, bwandte, _1___},
/*FLOAT */ {bwandte, bwandte, bwandte, _void, bwandte, bwandte, _absn, _absn, bwandte, bwandte, bwandte, bwandte},
/*BOOL */ {bwandte, bwandte, bwandte, bwandte, bwandte, bwandte, _absn, _absn, bwandte, bwandte, bwandte, bwandte},
/*VOID */ {_void, _void, bwandte, _void, bwandte, bwandte, _absn, _absn, bwandte, bwandte, bwandte, _absn},
/*STRING */ {bwandte, bwandte, bwandte, bwandte, bwandte, bwandte, _absn, _absn, bwandte, bwandte, bwandte, bwandte},
/*BYTES */ {bwandte, bwandte, bwandte, bwandte, bwandte, bwandte, bwandte, bwandte, bwandte, bwandte, bwandte, bwandte},
/*ARRAY */ {_absn, _absn, _absn, _absn, _absn, bwandte, _absn, _absn, bwandte, _absn, bwandte, _absn},
/*MAP */ {_absn, _absn, _absn, _absn, _absn, bwandte, _absn, _absn, bwandte, _absn, bwandte, _absn},
/*FUNC */ {bwandte, bwandte, bwandte, bwandte, bwandte, bwandte, bwandte, bwandte, bwandte, bwandte, bwandte, bwandte},
/*ERROR */ {bwandte, bwandte, bwandte, bwandte, bwandte, bwandte, _absn, _absn, bwandte, bwandte, bwandte, bwandte},
/*NULL */ {bwandte, bwandte, bwandte, bwandte, bwandte, bwandte, bwandte, bwandte, bwandte, bwandte, bwandte, _absn},
/*ABSENT */ {_2___, bwandte, bwandte, _absn, bwandte, bwandte, _absn, _absn, bwandte, bwandte, _absn, _absn},
}
func BIF_bitwise_and(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
@ -115,18 +118,19 @@ func bworte(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
}
var bitwise_or_dispositions = [mlrval.MT_DIM][mlrval.MT_DIM]BinaryFunc{
// . INT FLOAT BOOL VOID STRING ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {bitwise_or_i_ii, bworte, bworte, _void, bworte, _absn, _absn, bworte, bworte, bworte, _1___},
/*FLOAT */ {bworte, bworte, bworte, _void, bworte, _absn, _absn, bworte, bworte, bworte, bworte},
/*BOOL */ {bworte, bworte, bworte, bworte, bworte, _absn, _absn, bworte, bworte, bworte, bworte},
/*VOID */ {_void, _void, bworte, _void, bworte, _absn, _absn, bworte, bworte, bworte, _absn},
/*STRING */ {bworte, bworte, bworte, bworte, bworte, _absn, _absn, bworte, bworte, bworte, bworte},
/*ARRAY */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, bworte, _absn, bworte, _absn},
/*MAP */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, bworte, _absn, bworte, _absn},
/*FUNC */ {bworte, bworte, bworte, bworte, bworte, bworte, bworte, bworte, bworte, bworte, bworte},
/*ERROR */ {bworte, bworte, bworte, bworte, bworte, _absn, _absn, bworte, bworte, bworte, bworte},
/*NULL */ {bworte, bworte, bworte, bworte, bworte, bworte, bworte, bworte, bworte, bworte, _absn},
/*ABSENT */ {_2___, bworte, bworte, _absn, bworte, _absn, _absn, bworte, bworte, _absn, _absn},
// . INT FLOAT BOOL VOID STRING BYTES ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {bitwise_or_i_ii, bworte, bworte, _void, bworte, bworte, _absn, _absn, bworte, bworte, bworte, _1___},
/*FLOAT */ {bworte, bworte, bworte, _void, bworte, bworte, _absn, _absn, bworte, bworte, bworte, bworte},
/*BOOL */ {bworte, bworte, bworte, bworte, bworte, bworte, _absn, _absn, bworte, bworte, bworte, bworte},
/*VOID */ {_void, _void, bworte, _void, bworte, bworte, _absn, _absn, bworte, bworte, bworte, _absn},
/*STRING */ {bworte, bworte, bworte, bworte, bworte, bworte, _absn, _absn, bworte, bworte, bworte, bworte},
/*BYTES */ {bworte, bworte, bworte, bworte, bworte, bworte, bworte, bworte, bworte, bworte, bworte, bworte},
/*ARRAY */ {_absn, _absn, _absn, _absn, _absn, bworte, _absn, _absn, bworte, _absn, bworte, _absn},
/*MAP */ {_absn, _absn, _absn, _absn, _absn, bworte, _absn, _absn, bworte, _absn, bworte, _absn},
/*FUNC */ {bworte, bworte, bworte, bworte, bworte, bworte, bworte, bworte, bworte, bworte, bworte, bworte},
/*ERROR */ {bworte, bworte, bworte, bworte, bworte, bworte, _absn, _absn, bworte, bworte, bworte, bworte},
/*NULL */ {bworte, bworte, bworte, bworte, bworte, bworte, bworte, bworte, bworte, bworte, bworte, _absn},
/*ABSENT */ {_2___, bworte, bworte, _absn, bworte, bworte, _absn, _absn, bworte, bworte, _absn, _absn},
}
func BIF_bitwise_or(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
@ -144,18 +148,19 @@ func bwxorte(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
}
var bitwise_xor_dispositions = [mlrval.MT_DIM][mlrval.MT_DIM]BinaryFunc{
// . INT FLOAT BOOL VOID STRING ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {bitwise_xor_i_ii, bwxorte, bwxorte, _void, bwxorte, _absn, _absn, bwxorte, bwxorte, bwxorte, _1___},
/*FLOAT */ {bwxorte, bwxorte, bwxorte, _void, bwxorte, _absn, _absn, bwxorte, bwxorte, bwxorte, bwxorte},
/*BOOL */ {bwxorte, bwxorte, bwxorte, bwxorte, bwxorte, _absn, _absn, bwxorte, bwxorte, bwxorte, bwxorte},
/*VOID */ {_void, _void, bwxorte, _void, bwxorte, _absn, _absn, bwxorte, bwxorte, bwxorte, _absn},
/*STRING */ {bwxorte, bwxorte, bwxorte, bwxorte, bwxorte, _absn, _absn, bwxorte, bwxorte, bwxorte, bwxorte},
/*ARRAY */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, bwxorte, _absn, _absn, _absn},
/*MAP */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, bwxorte, _absn, _absn, _absn},
/*FUNC */ {bwxorte, bwxorte, bwxorte, bwxorte, bwxorte, bwxorte, bwxorte, bwxorte, bwxorte, bwxorte, bwxorte},
/*ERROR */ {bwxorte, bwxorte, bwxorte, bwxorte, bwxorte, _absn, _absn, bwxorte, bwxorte, bwxorte, bwxorte},
/*NULL */ {bwxorte, bwxorte, bwxorte, bwxorte, bwxorte, bwxorte, bwxorte, bwxorte, bwxorte, bwxorte, _absn},
/*ABSENT */ {_2___, bwxorte, bwxorte, _absn, bwxorte, _absn, _absn, bwxorte, bwxorte, _absn, _absn},
// . INT FLOAT BOOL VOID STRING BYTES ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {bitwise_xor_i_ii, bwxorte, bwxorte, _void, bwxorte, bwxorte, _absn, _absn, bwxorte, bwxorte, bwxorte, _1___},
/*FLOAT */ {bwxorte, bwxorte, bwxorte, _void, bwxorte, bwxorte, _absn, _absn, bwxorte, bwxorte, bwxorte, bwxorte},
/*BOOL */ {bwxorte, bwxorte, bwxorte, bwxorte, bwxorte, bwxorte, _absn, _absn, bwxorte, bwxorte, bwxorte, bwxorte},
/*VOID */ {_void, _void, bwxorte, _void, bwxorte, bwxorte, _absn, _absn, bwxorte, bwxorte, bwxorte, _absn},
/*STRING */ {bwxorte, bwxorte, bwxorte, bwxorte, bwxorte, bwxorte, _absn, _absn, bwxorte, bwxorte, bwxorte, bwxorte},
/*BYTES */ {bwxorte, bwxorte, bwxorte, bwxorte, bwxorte, bwxorte, bwxorte, bwxorte, bwxorte, bwxorte, bwxorte, bwxorte},
/*ARRAY */ {_absn, _absn, _absn, _absn, _absn, bwxorte, _absn, _absn, bwxorte, _absn, _absn, _absn},
/*MAP */ {_absn, _absn, _absn, _absn, _absn, bwxorte, _absn, _absn, bwxorte, _absn, _absn, _absn},
/*FUNC */ {bwxorte, bwxorte, bwxorte, bwxorte, bwxorte, bwxorte, bwxorte, bwxorte, bwxorte, bwxorte, bwxorte, bwxorte},
/*ERROR */ {bwxorte, bwxorte, bwxorte, bwxorte, bwxorte, bwxorte, _absn, _absn, bwxorte, bwxorte, bwxorte, bwxorte},
/*NULL */ {bwxorte, bwxorte, bwxorte, bwxorte, bwxorte, bwxorte, bwxorte, bwxorte, bwxorte, bwxorte, bwxorte, _absn},
/*ABSENT */ {_2___, bwxorte, bwxorte, _absn, bwxorte, bwxorte, _absn, _absn, bwxorte, bwxorte, _absn, _absn},
}
func BIF_bitwise_xor(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
@ -173,18 +178,19 @@ func lshfte(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
}
var left_shift_dispositions = [mlrval.MT_DIM][mlrval.MT_DIM]BinaryFunc{
// . INT FLOAT BOOL VOID STRING ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {lsh_i_ii, lshfte, lshfte, _void, lshfte, _absn, _absn, lshfte, lshfte, lshfte, _1___},
/*FLOAT */ {lshfte, lshfte, lshfte, _void, lshfte, _absn, _absn, lshfte, lshfte, lshfte, lshfte},
/*BOOL */ {lshfte, lshfte, lshfte, lshfte, lshfte, _absn, _absn, lshfte, lshfte, lshfte, lshfte},
/*VOID */ {_void, _void, lshfte, _void, lshfte, _absn, _absn, lshfte, lshfte, lshfte, _absn},
/*STRING */ {lshfte, lshfte, lshfte, lshfte, lshfte, _absn, _absn, lshfte, lshfte, lshfte, lshfte},
/*ARRAY */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, lshfte, _absn, lshfte, _absn},
/*MAP */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, lshfte, _absn, lshfte, _absn},
/*FUNC */ {lshfte, lshfte, lshfte, lshfte, lshfte, lshfte, lshfte, lshfte, lshfte, lshfte, lshfte},
/*ERROR */ {lshfte, lshfte, lshfte, lshfte, lshfte, _absn, _absn, lshfte, lshfte, lshfte, lshfte},
/*NULL */ {lshfte, lshfte, lshfte, lshfte, lshfte, lshfte, lshfte, lshfte, lshfte, lshfte, _absn},
/*ABSENT */ {_2___, lshfte, lshfte, _absn, lshfte, _absn, _absn, lshfte, lshfte, _absn, _absn},
// . INT FLOAT BOOL VOID STRING BYTES ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {lsh_i_ii, lshfte, lshfte, _void, lshfte, lshfte, _absn, _absn, lshfte, lshfte, lshfte, _1___},
/*FLOAT */ {lshfte, lshfte, lshfte, _void, lshfte, lshfte, _absn, _absn, lshfte, lshfte, lshfte, lshfte},
/*BOOL */ {lshfte, lshfte, lshfte, lshfte, lshfte, lshfte, _absn, _absn, lshfte, lshfte, lshfte, lshfte},
/*VOID */ {_void, _void, lshfte, _void, lshfte, lshfte, _absn, _absn, lshfte, lshfte, lshfte, _absn},
/*STRING */ {lshfte, lshfte, lshfte, lshfte, lshfte, lshfte, _absn, _absn, lshfte, lshfte, lshfte, lshfte},
/*BYTES */ {lshfte, lshfte, lshfte, lshfte, lshfte, lshfte, lshfte, lshfte, lshfte, lshfte, lshfte, lshfte},
/*ARRAY */ {_absn, _absn, _absn, _absn, _absn, lshfte, _absn, _absn, lshfte, _absn, lshfte, _absn},
/*MAP */ {_absn, _absn, _absn, _absn, _absn, lshfte, _absn, _absn, lshfte, _absn, lshfte, _absn},
/*FUNC */ {lshfte, lshfte, lshfte, lshfte, lshfte, lshfte, lshfte, lshfte, lshfte, lshfte, lshfte, lshfte},
/*ERROR */ {lshfte, lshfte, lshfte, lshfte, lshfte, lshfte, _absn, _absn, lshfte, lshfte, lshfte, lshfte},
/*NULL */ {lshfte, lshfte, lshfte, lshfte, lshfte, lshfte, lshfte, lshfte, lshfte, lshfte, lshfte, _absn},
/*ABSENT */ {_2___, lshfte, lshfte, _absn, lshfte, lshfte, _absn, _absn, lshfte, lshfte, _absn, _absn},
}
func BIF_left_shift(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
@ -202,18 +208,19 @@ func srste(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
}
var signed_right_shift_dispositions = [mlrval.MT_DIM][mlrval.MT_DIM]BinaryFunc{
// . INT FLOAT BOOL VOID STRING ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {srsh_i_ii, srste, srste, _void, srste, _absn, _absn, srste, srste, srste, _1___},
/*FLOAT */ {srste, srste, srste, _void, srste, _absn, _absn, srste, srste, srste, srste},
/*BOOL */ {srste, srste, srste, srste, srste, _absn, _absn, srste, srste, srste, srste},
/*VOID */ {_void, _void, srste, _void, srste, _absn, _absn, srste, srste, srste, _absn},
/*STRING */ {srste, srste, srste, srste, srste, _absn, _absn, srste, srste, srste, srste},
/*ARRAY */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, srste, _absn, srste, _absn},
/*MAP */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, srste, _absn, srste, _absn},
/*FUNC */ {srste, srste, srste, srste, srste, srste, srste, srste, srste, srste, srste},
/*ERROR */ {srste, srste, srste, srste, srste, _absn, _absn, srste, srste, srste, srste},
/*NULL */ {srste, srste, srste, srste, srste, srste, srste, srste, srste, srste, _absn},
/*ABSENT */ {_2___, srste, srste, _absn, srste, _absn, _absn, srste, srste, _absn, _absn},
// . INT FLOAT BOOL VOID STRING BYTES ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {srsh_i_ii, srste, srste, _void, srste, srste, _absn, _absn, srste, srste, srste, _1___},
/*FLOAT */ {srste, srste, srste, _void, srste, srste, _absn, _absn, srste, srste, srste, srste},
/*BOOL */ {srste, srste, srste, srste, srste, srste, _absn, _absn, srste, srste, srste, srste},
/*VOID */ {_void, _void, srste, _void, srste, srste, _absn, _absn, srste, srste, srste, _absn},
/*STRING */ {srste, srste, srste, srste, srste, srste, _absn, _absn, srste, srste, srste, srste},
/*BYTES */ {srste, srste, srste, srste, srste, srste, srste, srste, srste, srste, srste, srste},
/*ARRAY */ {_absn, _absn, _absn, _absn, _absn, srste, _absn, _absn, srste, _absn, srste, _absn},
/*MAP */ {_absn, _absn, _absn, _absn, _absn, srste, _absn, _absn, srste, _absn, srste, _absn},
/*FUNC */ {srste, srste, srste, srste, srste, srste, srste, srste, srste, srste, srste, srste},
/*ERROR */ {srste, srste, srste, srste, srste, srste, _absn, _absn, srste, srste, srste, srste},
/*NULL */ {srste, srste, srste, srste, srste, srste, srste, srste, srste, srste, srste, _absn},
/*ABSENT */ {_2___, srste, srste, _absn, srste, srste, _absn, _absn, srste, srste, _absn, _absn},
}
func BIF_signed_right_shift(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
@ -234,18 +241,19 @@ func rste(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
}
var unsigned_right_shift_dispositions = [mlrval.MT_DIM][mlrval.MT_DIM]BinaryFunc{
// . INT FLOAT BOOL VOID STRING ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {ursh_i_ii, rste, rste, _void, rste, _absn, _absn, rste, rste, rste, _1___},
/*FLOAT */ {rste, rste, rste, _void, rste, _absn, _absn, rste, rste, rste, rste},
/*BOOL */ {rste, rste, rste, rste, rste, _absn, _absn, rste, rste, rste, rste},
/*VOID */ {_void, _void, rste, _void, rste, _absn, _absn, rste, rste, rste, _absn},
/*STRING */ {rste, rste, rste, rste, rste, _absn, _absn, rste, rste, rste, rste},
/*ARRAY */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, rste, _absn, rste, _absn},
/*MAP */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, rste, _absn, rste, _absn},
/*FUNC */ {rste, rste, rste, rste, rste, rste, rste, rste, rste, rste, rste},
/*ERROR */ {rste, rste, rste, rste, rste, _absn, _absn, rste, rste, rste, rste},
/*NULL */ {rste, rste, rste, rste, rste, rste, rste, rste, rste, rste, _absn},
/*ABSENT */ {_2___, rste, rste, _absn, rste, _absn, _absn, rste, rste, _absn, _absn},
// . INT FLOAT BOOL VOID STRING BYTES ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {ursh_i_ii, rste, rste, _void, rste, rste, _absn, _absn, rste, rste, rste, _1___},
/*FLOAT */ {rste, rste, rste, _void, rste, rste, _absn, _absn, rste, rste, rste, rste},
/*BOOL */ {rste, rste, rste, rste, rste, rste, _absn, _absn, rste, rste, rste, rste},
/*VOID */ {_void, _void, rste, _void, rste, rste, _absn, _absn, rste, rste, rste, _absn},
/*STRING */ {rste, rste, rste, rste, rste, rste, _absn, _absn, rste, rste, rste, rste},
/*BYTES */ {rste, rste, rste, rste, rste, rste, rste, rste, rste, rste, rste, rste},
/*ARRAY */ {_absn, _absn, _absn, _absn, _absn, rste, _absn, _absn, rste, _absn, rste, _absn},
/*MAP */ {_absn, _absn, _absn, _absn, _absn, rste, _absn, _absn, rste, _absn, rste, _absn},
/*FUNC */ {rste, rste, rste, rste, rste, rste, rste, rste, rste, rste, rste, rste},
/*ERROR */ {rste, rste, rste, rste, rste, rste, _absn, _absn, rste, rste, rste, rste},
/*NULL */ {rste, rste, rste, rste, rste, rste, rste, rste, rste, rste, rste, _absn},
/*ABSENT */ {_2___, rste, rste, _absn, rste, rste, _absn, _absn, rste, rste, _absn, _absn},
}
func BIF_unsigned_right_shift(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {

201
pkg/bifs/cmp.go
View file

@ -3,6 +3,8 @@
package bifs
import (
"bytes"
"github.com/johnkerl/miller/v6/pkg/lib"
"github.com/johnkerl/miller/v6/pkg/mlrval"
)
@ -260,6 +262,30 @@ func ne_b_aa(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
return mlrval.FromBool(!output.AcquireBoolValue())
}
// - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
// 'y' is for bytes ('b' is for boolean)
func eq_b_yy(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
return mlrval.FromBool(bytes.Equal(input1.AcquireBytesValue(), input2.AcquireBytesValue()))
}
func ne_b_yy(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
return mlrval.FromBool(!bytes.Equal(input1.AcquireBytesValue(), input2.AcquireBytesValue()))
}
func gt_b_yy(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
return mlrval.FromBool(bytes.Compare(input1.AcquireBytesValue(), input2.AcquireBytesValue()) > 0)
}
func ge_b_yy(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
return mlrval.FromBool(bytes.Compare(input1.AcquireBytesValue(), input2.AcquireBytesValue()) >= 0)
}
func lt_b_yy(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
return mlrval.FromBool(bytes.Compare(input1.AcquireBytesValue(), input2.AcquireBytesValue()) < 0)
}
func le_b_yy(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
return mlrval.FromBool(bytes.Compare(input1.AcquireBytesValue(), input2.AcquireBytesValue()) <= 0)
}
func cmp_b_yy(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
return mlrval.FromInt(int64(bytes.Compare(input1.AcquireBytesValue(), input2.AcquireBytesValue())))
}
// - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
func eq_b_mm(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
return mlrval.FromBool(input1.AcquireMapValue().Equals(input2.AcquireMapValue()))
@ -278,18 +304,19 @@ func eqte(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
func init() {
eq_dispositions = [mlrval.MT_DIM][mlrval.MT_DIM]BinaryFunc{
// . INT FLOAT BOOL VOID STRING ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {eq_b_ii, eq_b_if, _fals, eq_b_xs, eq_b_xs, _fals, _fals, eqte, eqte, _fals, _absn},
/*FLOAT */ {eq_b_fi, eq_b_ff, _fals, eq_b_xs, eq_b_xs, _fals, _fals, eqte, eqte, _fals, _absn},
/*BOOL */ {_fals, _fals, eq_b_bb, _fals, _fals, _fals, _fals, eqte, eqte, _fals, _absn},
/*VOID */ {eq_b_sx, eq_b_sx, _fals, eq_b_ss, eq_b_ss, _fals, _fals, eqte, eqte, _fals, _absn},
/*STRING */ {eq_b_sx, eq_b_sx, _fals, eq_b_ss, eq_b_ss, _fals, _fals, eqte, eqte, _fals, _absn},
/*ARRAY */ {_fals, _fals, _fals, _fals, _fals, eq_b_aa, _fals, eqte, eqte, _fals, _absn},
/*MAP */ {_fals, _fals, _fals, _fals, _fals, _fals, eq_b_mm, eqte, eqte, _fals, _absn},
/*FUNC */ {eqte, eqte, eqte, eqte, eqte, eqte, eqte, eqte, eqte, eqte, eqte},
/*ERROR */ {eqte, eqte, eqte, eqte, eqte, eqte, eqte, eqte, eqte, eqte, eqte},
/*NULL */ {_fals, _fals, _fals, _fals, _fals, _fals, _fals, eqte, eqte, _true, _absn},
/*ABSENT */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, eqte, eqte, _absn, _absn},
// . INT FLOAT BOOL VOID STRING BYTES ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {eq_b_ii, eq_b_if, _fals, eq_b_xs, eq_b_xs, _fals, _fals, _fals, eqte, eqte, _fals, _absn},
/*FLOAT */ {eq_b_fi, eq_b_ff, _fals, eq_b_xs, eq_b_xs, _fals, _fals, _fals, eqte, eqte, _fals, _absn},
/*BOOL */ {_fals, _fals, eq_b_bb, _fals, _fals, _fals, _fals, _fals, eqte, eqte, _fals, _absn},
/*VOID */ {eq_b_sx, eq_b_sx, _fals, eq_b_ss, eq_b_ss, _fals, _fals, _fals, eqte, eqte, _fals, _absn},
/*STRING */ {eq_b_sx, eq_b_sx, _fals, eq_b_ss, eq_b_ss, _fals, _fals, _fals, eqte, eqte, _fals, _absn},
/*BYTES */ {_fals, _fals, _fals, _fals, _fals, eq_b_yy, _fals, _fals, eqte, eqte, _fals, _absn},
/*ARRAY */ {_fals, _fals, _fals, _fals, _fals, _fals, eq_b_aa, _fals, eqte, eqte, _fals, _absn},
/*MAP */ {_fals, _fals, _fals, _fals, _fals, _fals, _fals, eq_b_mm, eqte, eqte, _fals, _absn},
/*FUNC */ {eqte, eqte, eqte, eqte, eqte, eqte, eqte, eqte, eqte, eqte, eqte, eqte},
/*ERROR */ {eqte, eqte, eqte, eqte, eqte, eqte, eqte, eqte, eqte, eqte, eqte, eqte},
/*NULL */ {_fals, _fals, _fals, _fals, _fals, _fals, _fals, _fals, eqte, eqte, _true, _absn},
/*ABSENT */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, _absn, eqte, eqte, _absn, _absn},
}
}
@ -298,18 +325,19 @@ func nete(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
}
var ne_dispositions = [mlrval.MT_DIM][mlrval.MT_DIM]BinaryFunc{
// . INT FLOAT BOOL VOID STRING ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {ne_b_ii, ne_b_if, _true, ne_b_xs, ne_b_xs, _true, _true, nete, nete, _true, _absn},
/*FLOAT */ {ne_b_fi, ne_b_ff, _true, ne_b_xs, ne_b_xs, _true, _true, nete, nete, _true, _absn},
/*BOOL */ {_true, _true, ne_b_bb, _true, _true, _true, _true, nete, nete, _true, _absn},
/*VOID */ {ne_b_sx, ne_b_sx, _true, ne_b_ss, ne_b_ss, _true, _true, nete, nete, _true, _absn},
/*STRING */ {ne_b_sx, ne_b_sx, _true, ne_b_ss, ne_b_ss, _true, _true, nete, nete, _true, _absn},
/*ARRAY */ {_true, _true, _true, _true, _true, ne_b_aa, _true, nete, nete, _true, _absn},
/*MAP */ {_true, _true, _true, _true, _true, _true, ne_b_mm, nete, nete, _true, _absn},
/*FUNC */ {nete, nete, nete, nete, nete, nete, nete, nete, nete, nete, nete},
/*ERROR */ {nete, nete, nete, nete, nete, nete, nete, nete, nete, nete, nete},
/*NULL */ {_true, _true, _true, _true, _true, _true, _true, nete, nete, _fals, _absn},
/*ABSENT */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, nete, nete, _absn, _absn},
// . INT FLOAT BOOL VOID STRING BYTES ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {ne_b_ii, ne_b_if, _true, ne_b_xs, ne_b_xs, _true, _true, _true, nete, nete, _true, _absn},
/*FLOAT */ {ne_b_fi, ne_b_ff, _true, ne_b_xs, ne_b_xs, _true, _true, _true, nete, nete, _true, _absn},
/*BOOL */ {_true, _true, ne_b_bb, _true, _true, _true, _true, _true, nete, nete, _true, _absn},
/*VOID */ {ne_b_sx, ne_b_sx, _true, ne_b_ss, ne_b_ss, _true, _true, _true, nete, nete, _true, _absn},
/*STRING */ {ne_b_sx, ne_b_sx, _true, ne_b_ss, ne_b_ss, _true, _true, _true, nete, nete, _true, _absn},
/*BYTES */ {_true, _true, _true, _true, _true, ne_b_yy, _true, _true, nete, nete, _true, _absn},
/*ARRAY */ {_true, _true, _true, _true, _true, _true, ne_b_aa, _true, nete, nete, _true, _absn},
/*MAP */ {_true, _true, _true, _true, _true, _true, _true, ne_b_mm, nete, nete, _true, _absn},
/*FUNC */ {nete, nete, nete, nete, nete, nete, nete, nete, nete, nete, nete, nete},
/*ERROR */ {nete, nete, nete, nete, nete, nete, nete, nete, nete, nete, nete, nete},
/*NULL */ {_true, _true, _true, _true, _true, _true, _true, _true, nete, nete, _fals, _absn},
/*ABSENT */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, _absn, nete, nete, _absn, _absn},
}
func gtte(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
@ -317,18 +345,19 @@ func gtte(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
}
var gt_dispositions = [mlrval.MT_DIM][mlrval.MT_DIM]BinaryFunc{
// . INT FLOAT BOOL VOID STRING ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {gt_b_ii, gt_b_if, _fals, gt_b_xs, gt_b_xs, _fals, _fals, gtte, gtte, _fals, _absn},
/*FLOAT */ {gt_b_fi, gt_b_ff, _fals, gt_b_xs, gt_b_xs, _fals, _fals, gtte, gtte, _fals, _absn},
/*BOOL */ {_fals, _fals, gt_b_bb, _fals, _fals, _fals, _fals, gtte, gtte, _fals, _absn},
/*VOID */ {gt_b_sx, gt_b_sx, _fals, gt_b_ss, gt_b_ss, _fals, _fals, gtte, gtte, _fals, _absn},
/*STRING */ {gt_b_sx, gt_b_sx, _fals, gt_b_ss, gt_b_ss, _fals, _fals, gtte, gtte, _fals, _absn},
/*ARRAY */ {_fals, _fals, _fals, _fals, _fals, gtte, _fals, gtte, gtte, _fals, _absn},
/*MAP */ {_fals, _fals, _fals, _fals, _fals, _fals, gtte, gtte, gtte, _fals, _absn},
/*FUNC */ {gtte, gtte, gtte, gtte, gtte, gtte, gtte, gtte, gtte, gtte, gtte},
/*ERROR */ {gtte, gtte, gtte, gtte, gtte, gtte, gtte, gtte, gtte, _fals, gtte},
/*NULL */ {_true, _true, _true, _true, _true, _absn, _absn, gtte, _true, _fals, _fals},
/*ABSENT */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, gtte, gtte, _true, _absn},
// . INT FLOAT BOOL VOID STRING BYTES ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {gt_b_ii, gt_b_if, _fals, gt_b_xs, gt_b_xs, _fals, _fals, _fals, gtte, gtte, _fals, _absn},
/*FLOAT */ {gt_b_fi, gt_b_ff, _fals, gt_b_xs, gt_b_xs, _fals, _fals, _fals, gtte, gtte, _fals, _absn},
/*BOOL */ {_fals, _fals, gt_b_bb, _fals, _fals, _fals, _fals, _fals, gtte, gtte, _fals, _absn},
/*VOID */ {gt_b_sx, gt_b_sx, _fals, gt_b_ss, gt_b_ss, _fals, _fals, _fals, gtte, gtte, _fals, _absn},
/*STRING */ {gt_b_sx, gt_b_sx, _fals, gt_b_ss, gt_b_ss, _fals, _fals, _fals, gtte, gtte, _fals, _absn},
/*BYTES */ {_fals, _fals, _fals, _fals, _fals, gt_b_yy, _fals, _fals, gtte, gtte, _fals, _absn},
/*ARRAY */ {_fals, _fals, _fals, _fals, _fals, _fals, gtte, _fals, gtte, gtte, _fals, _absn},
/*MAP */ {_fals, _fals, _fals, _fals, _fals, _fals, _fals, gtte, gtte, gtte, _fals, _absn},
/*FUNC */ {gtte, gtte, gtte, gtte, gtte, gtte, gtte, gtte, gtte, gtte, gtte, gtte},
/*ERROR */ {gtte, gtte, gtte, gtte, gtte, gtte, gtte, gtte, gtte, gtte, _fals, gtte},
/*NULL */ {_true, _true, _true, _true, _true, _true, _absn, _absn, gtte, _true, _fals, _fals},
/*ABSENT */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, _absn, gtte, gtte, _true, _absn},
}
func gete(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
@ -336,18 +365,19 @@ func gete(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
}
var ge_dispositions = [mlrval.MT_DIM][mlrval.MT_DIM]BinaryFunc{
// . INT FLOAT BOOL VOID STRING ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {ge_b_ii, ge_b_if, _fals, ge_b_xs, ge_b_xs, _fals, _fals, gete, gete, _fals, _absn},
/*FLOAT */ {ge_b_fi, ge_b_ff, _fals, ge_b_xs, ge_b_xs, _fals, _fals, gete, gete, _fals, _absn},
/*BOOL */ {_fals, _fals, ge_b_bb, _fals, _fals, _fals, _fals, gete, gete, _fals, _absn},
/*VOID */ {ge_b_sx, ge_b_sx, _fals, ge_b_ss, ge_b_ss, _fals, _fals, gete, gete, _fals, _absn},
/*STRING */ {ge_b_sx, ge_b_sx, _fals, ge_b_ss, ge_b_ss, _fals, _fals, gete, gete, _fals, _absn},
/*ARRAY */ {_fals, _fals, _fals, _fals, _fals, gete, _fals, gete, gete, _fals, _absn},
/*MAP */ {_fals, _fals, _fals, _fals, _fals, _fals, gete, gete, gete, _fals, _absn},
/*FUNC */ {gete, gete, gete, gete, gete, gete, gete, gete, gete, gete, gete},
/*ERROR */ {gete, gete, gete, gete, gete, gete, gete, gete, gete, _fals, gete},
/*NULL */ {_true, _true, _true, _true, _true, _absn, _absn, gete, _true, _true, _fals},
/*ABSENT */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, gete, gete, _true, _absn},
// . INT FLOAT BOOL VOID STRING BYTES ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {ge_b_ii, ge_b_if, _fals, ge_b_xs, ge_b_xs, _fals, _fals, _fals, gete, gete, _fals, _absn},
/*FLOAT */ {ge_b_fi, ge_b_ff, _fals, ge_b_xs, ge_b_xs, _fals, _fals, _fals, gete, gete, _fals, _absn},
/*BOOL */ {_fals, _fals, ge_b_bb, _fals, _fals, _fals, _fals, _fals, gete, gete, _fals, _absn},
/*VOID */ {ge_b_sx, ge_b_sx, _fals, ge_b_ss, ge_b_ss, _fals, _fals, _fals, gete, gete, _fals, _absn},
/*STRING */ {ge_b_sx, ge_b_sx, _fals, ge_b_ss, ge_b_ss, _fals, _fals, _fals, gete, gete, _fals, _absn},
/*BYTES */ {_fals, _fals, _fals, _fals, _fals, ge_b_yy, _fals, _fals, gete, gete, _fals, _absn},
/*ARRAY */ {_fals, _fals, _fals, _fals, _fals, _fals, gete, _fals, gete, gete, _fals, _absn},
/*MAP */ {_fals, _fals, _fals, _fals, _fals, _fals, _fals, gete, gete, gete, _fals, _absn},
/*FUNC */ {gete, gete, gete, gete, gete, gete, gete, gete, gete, gete, gete, gete},
/*ERROR */ {gete, gete, gete, gete, gete, gete, gete, gete, gete, gete, _fals, gete},
/*NULL */ {_true, _true, _true, _true, _true, _true, _absn, _absn, gete, _true, _true, _fals},
/*ABSENT */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, _absn, gete, gete, _true, _absn},
}
func ltte(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
@ -355,18 +385,19 @@ func ltte(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
}
var lt_dispositions = [mlrval.MT_DIM][mlrval.MT_DIM]BinaryFunc{
// . INT FLOAT BOOL VOID STRING ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {lt_b_ii, lt_b_if, _fals, lt_b_xs, lt_b_xs, _fals, _fals, ltte, ltte, _true, _absn},
/*FLOAT */ {lt_b_fi, lt_b_ff, _fals, lt_b_xs, lt_b_xs, _fals, _fals, ltte, ltte, _true, _absn},
/*BOOL */ {_fals, _fals, lt_b_bb, _fals, _fals, _fals, _fals, ltte, ltte, _true, _absn},
/*VOID */ {lt_b_sx, lt_b_sx, _fals, lt_b_ss, lt_b_ss, _fals, _fals, ltte, ltte, _true, _absn},
/*STRING */ {lt_b_sx, lt_b_sx, _fals, lt_b_ss, lt_b_ss, _fals, _fals, ltte, ltte, _true, _absn},
/*ARRAY */ {_fals, _fals, _fals, _fals, _fals, ltte, _fals, ltte, ltte, _absn, _absn},
/*MAP */ {_fals, _fals, _fals, _fals, _fals, _fals, ltte, ltte, ltte, _absn, _absn},
/*FUNC */ {ltte, ltte, ltte, ltte, ltte, ltte, ltte, ltte, ltte, ltte, ltte},
/*ERROR */ {ltte, ltte, ltte, ltte, ltte, ltte, ltte, ltte, ltte, _true, ltte},
/*NULL */ {_fals, _fals, _fals, _fals, _fals, _absn, _absn, ltte, _fals, _fals, _true},
/*ABSENT */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, ltte, ltte, _fals, _absn},
// . INT FLOAT BOOL VOID STRING BYTES ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {lt_b_ii, lt_b_if, _fals, lt_b_xs, lt_b_xs, _fals, _fals, _fals, ltte, ltte, _true, _absn},
/*FLOAT */ {lt_b_fi, lt_b_ff, _fals, lt_b_xs, lt_b_xs, _fals, _fals, _fals, ltte, ltte, _true, _absn},
/*BOOL */ {_fals, _fals, lt_b_bb, _fals, _fals, _fals, _fals, _fals, ltte, ltte, _true, _absn},
/*VOID */ {lt_b_sx, lt_b_sx, _fals, lt_b_ss, lt_b_ss, _fals, _fals, _fals, ltte, ltte, _true, _absn},
/*STRING */ {lt_b_sx, lt_b_sx, _fals, lt_b_ss, lt_b_ss, _fals, _fals, _fals, ltte, ltte, _true, _absn},
/*BYTES */ {_fals, _fals, _fals, _fals, _fals, lt_b_yy, _fals, _fals, ltte, ltte, _fals, _absn},
/*ARRAY */ {_fals, _fals, _fals, _fals, _fals, _fals, ltte, _fals, ltte, ltte, _absn, _absn},
/*MAP */ {_fals, _fals, _fals, _fals, _fals, _fals, _fals, ltte, ltte, ltte, _absn, _absn},
/*FUNC */ {ltte, ltte, ltte, ltte, ltte, ltte, ltte, ltte, ltte, ltte, ltte, ltte},
/*ERROR */ {ltte, ltte, ltte, ltte, ltte, ltte, ltte, ltte, ltte, ltte, _true, ltte},
/*NULL */ {_fals, _fals, _fals, _fals, _fals, _fals, _absn, _absn, ltte, _fals, _fals, _true},
/*ABSENT */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, _absn, ltte, ltte, _fals, _absn},
}
func lete(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
@ -374,18 +405,19 @@ func lete(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
}
var le_dispositions = [mlrval.MT_DIM][mlrval.MT_DIM]BinaryFunc{
// . INT FLOAT BOOL VOID STRING ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {le_b_ii, le_b_if, _fals, le_b_xs, le_b_xs, _fals, _fals, lete, lete, _true, _absn},
/*FLOAT */ {le_b_fi, le_b_ff, _fals, le_b_xs, le_b_xs, _fals, _fals, lete, lete, _true, _absn},
/*BOOL */ {_fals, _fals, le_b_bb, _fals, _fals, _fals, _fals, lete, lete, _true, _absn},
/*VOID */ {le_b_sx, le_b_sx, _fals, le_b_ss, le_b_ss, _fals, _fals, lete, lete, _true, _absn},
/*STRING */ {le_b_sx, le_b_sx, _fals, le_b_ss, le_b_ss, _fals, _fals, lete, lete, _true, _absn},
/*ARRAY */ {_fals, _fals, _fals, _fals, _fals, lete, _fals, lete, lete, _absn, _absn},
/*MAP */ {_fals, _fals, _fals, _fals, _fals, _fals, lete, lete, lete, _absn, _absn},
/*FUNC */ {lete, lete, lete, lete, lete, lete, lete, lete, lete, lete, lete},
/*ERROR */ {lete, lete, lete, lete, lete, lete, lete, lete, lete, _true, lete},
/*NULL */ {_fals, _fals, _fals, _fals, _fals, _absn, _absn, lete, _fals, _true, _true},
/*ABSENT */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, lete, lete, _fals, _absn},
// . INT FLOAT BOOL VOID STRING BYTES ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {le_b_ii, le_b_if, _fals, le_b_xs, le_b_xs, _fals, _fals, _fals, lete, lete, _true, _absn},
/*FLOAT */ {le_b_fi, le_b_ff, _fals, le_b_xs, le_b_xs, _fals, _fals, _fals, lete, lete, _true, _absn},
/*BOOL */ {_fals, _fals, le_b_bb, _fals, _fals, _fals, _fals, _fals, lete, lete, _true, _absn},
/*VOID */ {le_b_sx, le_b_sx, _fals, le_b_ss, le_b_ss, _fals, _fals, _fals, lete, lete, _true, _absn},
/*STRING */ {le_b_sx, le_b_sx, _fals, le_b_ss, le_b_ss, _fals, _fals, _fals, lete, lete, _true, _absn},
/*BYTES */ {_fals, _fals, _fals, _fals, _fals, le_b_yy, _fals, _fals, lete, lete, _fals, _absn},
/*ARRAY */ {_fals, _fals, _fals, _fals, _fals, _fals, lete, _fals, lete, lete, _absn, _absn},
/*MAP */ {_fals, _fals, _fals, _fals, _fals, _fals, _fals, lete, lete, lete, _absn, _absn},
/*FUNC */ {lete, lete, lete, lete, lete, lete, lete, lete, lete, lete, lete, lete},
/*ERROR */ {lete, lete, lete, lete, lete, lete, lete, lete, lete, lete, _true, lete},
/*NULL */ {_fals, _fals, _fals, _fals, _fals, _fals, _absn, _absn, lete, _fals, _true, _true},
/*ABSENT */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, _absn, lete, lete, _fals, _absn},
}
func cmpte(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
@ -393,18 +425,19 @@ func cmpte(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
}
var cmp_dispositions = [mlrval.MT_DIM][mlrval.MT_DIM]BinaryFunc{
// . INT FLOAT BOOL VOID STRING ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {cmp_b_ii, cmp_b_if, _less, cmp_b_xs, cmp_b_xs, _less, _less, cmpte, cmpte, _true, _absn},
/*FLOAT */ {cmp_b_fi, cmp_b_ff, _less, cmp_b_xs, cmp_b_xs, _less, _less, cmpte, cmpte, _true, _absn},
/*BOOL */ {_more, _more, cmp_b_bb, _less, _less, _less, _less, cmpte, cmpte, _true, _absn},
/*VOID */ {cmp_b_sx, cmp_b_sx, _more, cmp_b_ss, cmp_b_ss, _less, _less, cmpte, cmpte, _true, _absn},
/*STRING */ {cmp_b_sx, cmp_b_sx, _more, cmp_b_ss, cmp_b_ss, _less, _less, cmpte, cmpte, _true, _absn},
/*ARRAY */ {_more, _more, _more, _more, _more, cmpte, _less, cmpte, cmpte, _absn, _absn},
/*MAP */ {_more, _more, _more, _more, _more, _more, cmpte, cmpte, cmpte, _absn, _absn},
/*FUNC */ {cmpte, cmpte, cmpte, cmpte, cmpte, cmpte, cmpte, cmpte, cmpte, cmpte, cmpte},
/*ERROR */ {cmpte, cmpte, cmpte, cmpte, cmpte, cmpte, cmpte, cmpte, cmpte, _true, cmpte},
/*NULL */ {_more, _more, _more, _more, _more, _absn, _absn, cmpte, _more, _same, _true},
/*ABSENT */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, cmpte, cmpte, _more, _absn},
// . INT FLOAT BOOL VOID STRING BYTES ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {cmp_b_ii, cmp_b_if, _less, cmp_b_xs, cmp_b_xs, _less, _less, _less, cmpte, cmpte, _true, _absn},
/*FLOAT */ {cmp_b_fi, cmp_b_ff, _less, cmp_b_xs, cmp_b_xs, _less, _less, _less, cmpte, cmpte, _true, _absn},
/*BOOL */ {_more, _more, cmp_b_bb, _less, _less, _less, _less, _less, cmpte, cmpte, _true, _absn},
/*VOID */ {cmp_b_sx, cmp_b_sx, _more, cmp_b_ss, cmp_b_ss, _less, _less, _less, cmpte, cmpte, _true, _absn},
/*STRING */ {cmp_b_sx, cmp_b_sx, _more, cmp_b_ss, cmp_b_ss, _less, _less, _less, cmpte, cmpte, _true, _absn},
/*BYTES */ {_more, _more, _more, _more, _more, cmp_b_yy, _less, _less, cmpte, cmpte, _less, _absn},
/*ARRAY */ {_more, _more, _more, _more, _more, _more, cmpte, _less, cmpte, cmpte, _absn, _absn},
/*MAP */ {_more, _more, _more, _more, _more, _more, _more, cmpte, cmpte, cmpte, _absn, _absn},
/*FUNC */ {cmpte, cmpte, cmpte, cmpte, cmpte, cmpte, cmpte, cmpte, cmpte, cmpte, cmpte, cmpte},
/*ERROR */ {cmpte, cmpte, cmpte, cmpte, cmpte, cmpte, cmpte, cmpte, cmpte, cmpte, _true, cmpte},
/*NULL */ {_more, _more, _more, _more, _more, _more, _absn, _absn, cmpte, _more, _same, _true},
/*ABSENT */ {_absn, _absn, _absn, _absn, _absn, _absn, _absn, _absn, cmpte, cmpte, _more, _absn},
}
func BIF_equals(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {

2
pkg/bifs/collections.go
View file

@ -74,6 +74,7 @@ func init() {
/*BOOL */ depth_from_scalar,
/*VOID */ depth_from_scalar,
/*STRING */ depth_from_scalar,
/*BYTES */ depth_te,
/*ARRAY */ depth_from_array,
/*MAP */ depth_from_map,
/*FUNC */ depth_te,
@ -145,6 +146,7 @@ var leafcount_dispositions = [mlrval.MT_DIM]UnaryFunc{
/*BOOL */ leafcount_from_scalar,
/*VOID */ leafcount_from_scalar,
/*STRING */ leafcount_from_scalar,
/*BYTES */ leafcount_te,
/*ARRAY */ leafcount_from_array,
/*MAP */ leafcount_from_map,
/*FUNC */ leafcount_te,

107
pkg/bifs/dispositions_test.go Normal file
View file

@ -0,0 +1,107 @@
package bifs
import (
"testing"
"github.com/johnkerl/miller/v6/pkg/mlrval"
)
// Disposition matrices/vectors are positional array literals sized by
// mlrval.MT_DIM. When a new mlrval type is added and MT_DIM is bumped, Go
// zero-fills the new slots in any table that hasn't been extended, leaving
// nil function pointers which panic at dispatch time rather than at compile
// time. These sweeps turn that hazard into a test failure.
func TestNoNilCellsInUnaryDispositionVectors(t *testing.T) {
vectors := map[string][mlrval.MT_DIM]UnaryFunc{
"upos_dispositions": upos_dispositions,
"uneg_dispositions": uneg_dispositions,
"min_unary_dispositions": min_unary_dispositions,
"max_unary_dispositions": max_unary_dispositions,
"bitwise_not_dispositions": bitwise_not_dispositions,
"bitcount_dispositions": bitcount_dispositions,
"depth_dispositions": depth_dispositions,
"leafcount_dispositions": leafcount_dispositions,
"to_int_dispositions": to_int_dispositions,
"to_float_dispositions": to_float_dispositions,
"to_boolean_dispositions": to_boolean_dispositions,
}
for name, vector := range vectors {
for i := 0; i < int(mlrval.MT_DIM); i++ {
if vector[i] == nil {
t.Errorf("nil cell in %s[%d]", name, i)
}
}
}
}
func TestNoNilCellsInUnaryMathLibDispositionVectors(t *testing.T) {
vectors := map[string][mlrval.MT_DIM]mathLibUnaryFuncWrapper{
"mudispo": mudispo,
"imudispo": imudispo,
}
for name, vector := range vectors {
for i := 0; i < int(mlrval.MT_DIM); i++ {
if vector[i] == nil {
t.Errorf("nil cell in %s[%d]", name, i)
}
}
}
}
func TestNoNilCellsInBinaryDispositionVectors(t *testing.T) {
vectors := map[string][mlrval.MT_DIM]BinaryFunc{
"to_int_with_base_dispositions": to_int_with_base_dispositions,
}
for name, vector := range vectors {
for i := 0; i < int(mlrval.MT_DIM); i++ {
if vector[i] == nil {
t.Errorf("nil cell in %s[%d]", name, i)
}
}
}
}
func TestNoNilCellsInBinaryDispositionMatrices(t *testing.T) {
matrices := map[string][mlrval.MT_DIM][mlrval.MT_DIM]BinaryFunc{
"plus_dispositions": plus_dispositions,
"minus_dispositions": minus_dispositions,
"times_dispositions": times_dispositions,
"divide_dispositions": divide_dispositions,
"int_divide_dispositions": int_divide_dispositions,
"dot_plus_dispositions": dot_plus_dispositions,
"dotminus_dispositions": dotminus_dispositions,
"dottimes_dispositions": dottimes_dispositions,
"dotdivide_dispositions": dotdivide_dispositions,
"modulus_dispositions": modulus_dispositions,
"min_dispositions": min_dispositions,
"max_dispositions": max_dispositions,
"bitwise_and_dispositions": bitwise_and_dispositions,
"bitwise_or_dispositions": bitwise_or_dispositions,
"bitwise_xor_dispositions": bitwise_xor_dispositions,
"left_shift_dispositions": left_shift_dispositions,
"signed_right_shift_dispositions": signed_right_shift_dispositions,
"unsigned_right_shift_dispositions": unsigned_right_shift_dispositions,
"eq_dispositions": eq_dispositions,
"ne_dispositions": ne_dispositions,
"gt_dispositions": gt_dispositions,
"ge_dispositions": ge_dispositions,
"lt_dispositions": lt_dispositions,
"le_dispositions": le_dispositions,
"cmp_dispositions": cmp_dispositions,
"pow_dispositions": pow_dispositions,
"atan2_dispositions": atan2_dispositions,
"roundm_dispositions": roundm_dispositions,
"dot_dispositions": dot_dispositions,
"fmtnum_dispositions": fmtnum_dispositions,
}
for name, matrix := range matrices {
for i := 0; i < int(mlrval.MT_DIM); i++ {
for j := 0; j < int(mlrval.MT_DIM); j++ {
if matrix[i][j] == nil {
t.Errorf("nil cell in %s[%d][%d]", name, i, j)
}
}
}
}
}

62
pkg/bifs/hashing.go
View file

@ -10,50 +10,46 @@ import (
"github.com/johnkerl/miller/v6/pkg/mlrval"
)
func BIF_md5(input1 *mlrval.Mlrval) *mlrval.Mlrval {
if !input1.IsStringOrVoid() {
return mlrval.FromNotStringError("md5", input1)
// hashableBytes returns the raw payload to hash: for strings, their UTF-8
// bytes; for bytes values, the bytes themselves.
func hashableBytes(funcname string, input1 *mlrval.Mlrval) ([]byte, *mlrval.Mlrval) {
if input1.IsBytes() {
return input1.AcquireBytesValue(), nil
}
return mlrval.FromString(
fmt.Sprintf(
"%x",
md5.Sum([]byte(input1.AcquireStringValue())),
),
)
if !input1.IsStringOrVoid() {
return nil, mlrval.FromNotStringError(funcname, input1)
}
return []byte(input1.AcquireStringValue()), nil
}
func BIF_md5(input1 *mlrval.Mlrval) *mlrval.Mlrval {
payload, errval := hashableBytes("md5", input1)
if errval != nil {
return errval
}
return mlrval.FromString(fmt.Sprintf("%x", md5.Sum(payload)))
}
func BIF_sha1(input1 *mlrval.Mlrval) *mlrval.Mlrval {
if !input1.IsStringOrVoid() {
return mlrval.FromNotStringError("sha1", input1)
payload, errval := hashableBytes("sha1", input1)
if errval != nil {
return errval
}
return mlrval.FromString(
fmt.Sprintf(
"%x",
sha1.Sum([]byte(input1.AcquireStringValue())),
),
)
return mlrval.FromString(fmt.Sprintf("%x", sha1.Sum(payload)))
}
func BIF_sha256(input1 *mlrval.Mlrval) *mlrval.Mlrval {
if !input1.IsStringOrVoid() {
return mlrval.FromNotStringError("sha256", input1)
payload, errval := hashableBytes("sha256", input1)
if errval != nil {
return errval
}
return mlrval.FromString(
fmt.Sprintf(
"%x",
sha256.Sum256([]byte(input1.AcquireStringValue())),
),
)
return mlrval.FromString(fmt.Sprintf("%x", sha256.Sum256(payload)))
}
func BIF_sha512(input1 *mlrval.Mlrval) *mlrval.Mlrval {
if !input1.IsStringOrVoid() {
return mlrval.FromNotStringError("sha512", input1)
payload, errval := hashableBytes("sha512", input1)
if errval != nil {
return errval
}
return mlrval.FromString(
fmt.Sprintf(
"%x",
sha512.Sum512([]byte(input1.AcquireStringValue())),
),
)
return mlrval.FromString(fmt.Sprintf("%x", sha512.Sum512(payload)))
}

32
pkg/bifs/hex.go Normal file
View file

@ -0,0 +1,32 @@
package bifs
import (
"encoding/hex"
"github.com/johnkerl/miller/v6/pkg/mlrval"
)
func BIF_hex_encode(input1 *mlrval.Mlrval) *mlrval.Mlrval {
if input1.IsBytes() {
return mlrval.FromString(
hex.EncodeToString(input1.AcquireBytesValue()),
)
}
if !input1.IsStringOrVoid() {
return mlrval.FromNotStringError("hex_encode", input1)
}
return mlrval.FromString(
hex.EncodeToString([]byte(input1.AcquireStringValue())),
)
}
func BIF_hex_decode(input1 *mlrval.Mlrval) *mlrval.Mlrval {
if !input1.IsStringOrVoid() {
return mlrval.FromNotStringError("hex_decode", input1)
}
decoded, err := hex.DecodeString(input1.AcquireStringValue())
if err != nil {
return mlrval.FromError(err)
}
return mlrval.FromBytes(decoded)
}

77
pkg/bifs/mathlib.go
View file

@ -46,6 +46,7 @@ var mudispo = [mlrval.MT_DIM]mathLibUnaryFuncWrapper{
/*BOOL */ _math_unary_erro1,
/*VOID */ _math_unary_void1,
/*STRING */ _math_unary_erro1,
/*BYTES */ _math_unary_erro1,
/*ARRAY */ _math_unary_absn1,
/*MAP */ _math_unary_absn1,
/*FUNC */ _math_unary_erro1,
@ -131,6 +132,7 @@ var imudispo = [mlrval.MT_DIM]mathLibUnaryFuncWrapper{
/*BOOL */ _math_unary_erro1,
/*VOID */ _math_unary_void1,
/*STRING */ _math_unary_erro1,
/*BYTES */ _math_unary_erro1,
/*ARRAY */ _math_unary_absn1,
/*MAP */ _math_unary_absn1,
/*FUNC */ _math_unary_erro1,
@ -183,18 +185,19 @@ func powte(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
}
var pow_dispositions = [mlrval.MT_DIM][mlrval.MT_DIM]BinaryFunc{
// . INT FLOAT BOOL VOID STRING ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {pow_f_ii, pow_f_if, powte, _void, powte, powte, powte, powte, powte, powte, _1___},
/*FLOAT */ {pow_f_fi, pow_f_ff, powte, _void, powte, powte, powte, powte, powte, powte, _1___},
/*BOOL */ {powte, powte, powte, powte, powte, powte, powte, powte, powte, powte, _absn},
/*VOID */ {_void, _void, powte, _void, powte, powte, powte, powte, powte, powte, _absn},
/*STRING */ {powte, powte, powte, powte, powte, powte, powte, powte, powte, powte, _absn},
/*ARRAY */ {powte, powte, powte, powte, powte, powte, powte, powte, powte, powte, _absn},
/*MAP */ {powte, powte, powte, powte, powte, powte, powte, powte, powte, powte, _absn},
/*FUNC */ {powte, powte, powte, powte, powte, powte, powte, powte, powte, powte, _absn},
/*ERROR */ {powte, powte, powte, powte, powte, powte, powte, powte, powte, powte, _absn},
/*NULL */ {powte, powte, powte, powte, powte, powte, powte, powte, powte, powte, _absn},
/*ABSENT */ {_i0__, _f0__, _absn, _absn, _absn, _absn, _absn, _absn, _absn, _absn, _absn},
// . INT FLOAT BOOL VOID STRING BYTES ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {pow_f_ii, pow_f_if, powte, _void, powte, powte, powte, powte, powte, powte, powte, _1___},
/*FLOAT */ {pow_f_fi, pow_f_ff, powte, _void, powte, powte, powte, powte, powte, powte, powte, _1___},
/*BOOL */ {powte, powte, powte, powte, powte, powte, powte, powte, powte, powte, powte, _absn},
/*VOID */ {_void, _void, powte, _void, powte, powte, powte, powte, powte, powte, powte, _absn},
/*STRING */ {powte, powte, powte, powte, powte, powte, powte, powte, powte, powte, powte, _absn},
/*BYTES */ {powte, powte, powte, powte, powte, powte, powte, powte, powte, powte, powte, _absn},
/*ARRAY */ {powte, powte, powte, powte, powte, powte, powte, powte, powte, powte, powte, _absn},
/*MAP */ {powte, powte, powte, powte, powte, powte, powte, powte, powte, powte, powte, _absn},
/*FUNC */ {powte, powte, powte, powte, powte, powte, powte, powte, powte, powte, powte, _absn},
/*ERROR */ {powte, powte, powte, powte, powte, powte, powte, powte, powte, powte, powte, _absn},
/*NULL */ {powte, powte, powte, powte, powte, powte, powte, powte, powte, powte, powte, _absn},
/*ABSENT */ {_i0__, _f0__, _absn, _absn, _absn, _absn, _absn, _absn, _absn, _absn, _absn, _absn},
}
func BIF_pow(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
@ -219,18 +222,19 @@ func atan2te(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
}
var atan2_dispositions = [mlrval.MT_DIM][mlrval.MT_DIM]BinaryFunc{
// . INT FLOAT BOOL VOID STRING ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {atan2_f_ii, atan2_f_if, atan2te, _void, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, _1___},
/*FLOAT */ {atan2_f_fi, atan2_f_ff, atan2te, _void, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, _1___},
/*BOOL */ {atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, _absn},
/*VOID */ {_void, _void, atan2te, _void, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, _absn},
/*STRING */ {atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, _absn},
/*ARRAY */ {atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, _absn},
/*MAP */ {atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, _absn},
/*FUNC */ {atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, _absn},
/*ERROR */ {atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, _absn},
/*NULL */ {atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, _absn},
/*ABSENT */ {_i0__, _f0__, atan2te, _absn, _absn, _absn, _absn, _absn, _absn, _absn, _absn},
// . INT FLOAT BOOL VOID STRING BYTES ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {atan2_f_ii, atan2_f_if, atan2te, _void, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, _1___},
/*FLOAT */ {atan2_f_fi, atan2_f_ff, atan2te, _void, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, _1___},
/*BOOL */ {atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, _absn},
/*VOID */ {_void, _void, atan2te, _void, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, _absn},
/*STRING */ {atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, _absn},
/*BYTES */ {atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, _absn},
/*ARRAY */ {atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, _absn},
/*MAP */ {atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, _absn},
/*FUNC */ {atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, _absn},
/*ERROR */ {atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, _absn},
/*NULL */ {atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, atan2te, _absn},
/*ABSENT */ {_i0__, _f0__, atan2te, _absn, _absn, _absn, _absn, _absn, _absn, _absn, _absn, _absn},
}
func BIF_atan2(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
@ -259,18 +263,19 @@ func rdmte(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
}
var roundm_dispositions = [mlrval.MT_DIM][mlrval.MT_DIM]BinaryFunc{
// . INT FLOAT BOOL VOID STRING ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {roundm_f_ii, roundm_f_if, rdmte, _void, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, _1___},
/*FLOAT */ {roundm_f_fi, roundm_f_ff, rdmte, _void, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, _1___},
/*BOOL */ {rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, _absn},
/*VOID */ {_void, _void, rdmte, _void, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, _absn},
/*STRING */ {rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, _absn},
/*ARRAY */ {rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, _absn},
/*MAP */ {rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, _absn},
/*FUNC */ {rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, _absn},
/*ERROR */ {rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, _absn},
/*NULL */ {rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, _absn},
/*ABSENT */ {_i0__, _f0__, _absn, _absn, _absn, _absn, _absn, _absn, _absn, _absn, _absn},
// . INT FLOAT BOOL VOID STRING BYTES ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {roundm_f_ii, roundm_f_if, rdmte, _void, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, _1___},
/*FLOAT */ {roundm_f_fi, roundm_f_ff, rdmte, _void, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, _1___},
/*BOOL */ {rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, _absn},
/*VOID */ {_void, _void, rdmte, _void, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, _absn},
/*STRING */ {rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, _absn},
/*BYTES */ {rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, _absn},
/*ARRAY */ {rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, _absn},
/*MAP */ {rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, _absn},
/*FUNC */ {rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, _absn},
/*ERROR */ {rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, _absn},
/*NULL */ {rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, rdmte, _absn},
/*ABSENT */ {_i0__, _f0__, _absn, _absn, _absn, _absn, _absn, _absn, _absn, _absn, _absn, _absn},
}
func BIF_roundm(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {

95
pkg/bifs/strings.go
View file

@ -12,6 +12,10 @@ import (
)
func BIF_strlen(input1 *mlrval.Mlrval) *mlrval.Mlrval {
// Strings have UTF-8 rune-count length; bytes have byte-count length.
if input1.IsBytes() {
return mlrval.FromInt(int64(len(input1.AcquireBytesValue())))
}
if !input1.IsStringOrVoid() {
return mlrval.FromTypeErrorUnary("strlen", input1)
}
@ -19,6 +23,12 @@ func BIF_strlen(input1 *mlrval.Mlrval) *mlrval.Mlrval {
}
func BIF_string(input1 *mlrval.Mlrval) *mlrval.Mlrval {
// For bytes, reinterpret the raw bytes as a string -- the inverse of
// bytes(s) -- rather than using the hex String() rendering. This is what
// makes string(base64_decode("aGVsbG8=")) yield "hello".
if input1.IsBytes() {
return mlrval.FromString(string(input1.AcquireBytesValue()))
}
return mlrval.FromString(input1.String())
}
@ -43,19 +53,32 @@ func dot_te(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
return mlrval.FromTypeErrorBinary(".", input1, input2)
}
// Concatenation of two bytes values is a bytes value; dotting bytes with
// anything else is a type error, since silently reinterpreting binary data as
// text (or vice versa) is exactly what the bytes type exists to prevent.
func dot_y_yy(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
a := input1.AcquireBytesValue()
b := input2.AcquireBytesValue()
c := make([]byte, 0, len(a)+len(b))
c = append(c, a...)
c = append(c, b...)
return mlrval.FromBytes(c)
}
var dot_dispositions = [mlrval.MT_DIM][mlrval.MT_DIM]BinaryFunc{
// . INT FLOAT BOOL VOID STRING ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {dot_s_xx, dot_s_xx, dot_s_xx, _s1__, dot_s_xx, dot_te, dot_te, dot_te, dot_te, _1___, _s1__},
/*FLOAT */ {dot_s_xx, dot_s_xx, dot_s_xx, _s1__, dot_s_xx, dot_te, dot_te, dot_te, dot_te, _1___, _s1__},
/*BOOL */ {dot_s_xx, dot_s_xx, dot_s_xx, _s1__, dot_s_xx, dot_te, dot_te, dot_te, dot_te, _1___, _s1__},
/*VOID */ {_s2__, _s2__, _s2__, _void, _2___, _absn, _absn, dot_te, dot_te, _void, _void},
/*STRING */ {dot_s_xx, dot_s_xx, dot_s_xx, _1___, dot_s_xx, dot_te, dot_te, dot_te, dot_te, _1___, _1___},
/*ARRAY */ {dot_te, dot_te, dot_te, dot_te, dot_te, dot_te, dot_te, dot_te, dot_te, dot_te, dot_te},
/*MAP */ {dot_te, dot_te, dot_te, dot_te, dot_te, dot_te, dot_te, dot_te, dot_te, dot_te, dot_te},
/*FUNC */ {dot_te, dot_te, dot_te, dot_te, dot_te, dot_te, dot_te, dot_te, dot_te, dot_te, dot_te},
/*ERROR */ {dot_te, dot_te, dot_te, dot_te, dot_te, _absn, _absn, dot_te, dot_te, dot_te, dot_te},
/*NULL */ {_s2__, _s2__, _s2__, _void, _2___, _absn, _absn, dot_te, dot_te, _null, _null},
/*ABSENT */ {_s2__, _s2__, _s2__, _void, _2___, _absn, _absn, dot_te, dot_te, _null, _absn},
// . INT FLOAT BOOL VOID STRING BYTES ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {dot_s_xx, dot_s_xx, dot_s_xx, _s1__, dot_s_xx, dot_te, dot_te, dot_te, dot_te, dot_te, _1___, _s1__},
/*FLOAT */ {dot_s_xx, dot_s_xx, dot_s_xx, _s1__, dot_s_xx, dot_te, dot_te, dot_te, dot_te, dot_te, _1___, _s1__},
/*BOOL */ {dot_s_xx, dot_s_xx, dot_s_xx, _s1__, dot_s_xx, dot_te, dot_te, dot_te, dot_te, dot_te, _1___, _s1__},
/*VOID */ {_s2__, _s2__, _s2__, _void, _2___, _2___, _absn, _absn, dot_te, dot_te, _void, _void},
/*STRING */ {dot_s_xx, dot_s_xx, dot_s_xx, _1___, dot_s_xx, dot_te, dot_te, dot_te, dot_te, dot_te, _1___, _1___},
/*BYTES */ {dot_te, dot_te, dot_te, _1___, dot_te, dot_y_yy, dot_te, dot_te, dot_te, dot_te, _1___, _1___},
/*ARRAY */ {dot_te, dot_te, dot_te, dot_te, dot_te, dot_te, dot_te, dot_te, dot_te, dot_te, dot_te, dot_te},
/*MAP */ {dot_te, dot_te, dot_te, dot_te, dot_te, dot_te, dot_te, dot_te, dot_te, dot_te, dot_te, dot_te},
/*FUNC */ {dot_te, dot_te, dot_te, dot_te, dot_te, dot_te, dot_te, dot_te, dot_te, dot_te, dot_te, dot_te},
/*ERROR */ {dot_te, dot_te, dot_te, dot_te, dot_te, dot_te, _absn, _absn, dot_te, dot_te, dot_te, dot_te},
/*NULL */ {_s2__, _s2__, _s2__, _void, _2___, _2___, _absn, _absn, dot_te, dot_te, _null, _null},
/*ABSENT */ {_s2__, _s2__, _s2__, _void, _2___, _2___, _absn, _absn, dot_te, dot_te, _null, _absn},
}
func BIF_dot(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
@ -72,6 +95,9 @@ func BIF_substr_1_up(input1, input2, input3 *mlrval.Mlrval) *mlrval.Mlrval {
if input1.IsError() {
return mlrval.FromTypeErrorUnary("substr1", input1)
}
if input1.IsBytes() {
return bytes_substr(input1, input2, input3, false)
}
sinput := input1.String()
// Handle UTF-8 correctly: len(input1.AcquireStringValue()) will count bytes, not runes.
@ -103,6 +129,9 @@ func BIF_substr_0_up(input1, input2, input3 *mlrval.Mlrval) *mlrval.Mlrval {
if input1.IsError() {
return mlrval.FromTypeErrorUnary("substr0", input1)
}
if input1.IsBytes() {
return bytes_substr(input1, input2, input3, true)
}
sinput := input1.String()
// Handle UTF-8 correctly: len(input1.AcquireStringValue()) will count bytes, not runes.
@ -124,6 +153,23 @@ func BIF_substr_0_up(input1, input2, input3 *mlrval.Mlrval) *mlrval.Mlrval {
return mlrval.FromString(string(runes[lowerZindex : upperZindex+1]))
}
// bytes_substr slices bytes by byte position, not rune position. An empty
// slice yields empty bytes rather than VOID, preserving the type.
func bytes_substr(input1, input2, input3 *mlrval.Mlrval, zeroUp bool) *mlrval.Mlrval {
b := input1.AcquireBytesValue()
sliceIsEmpty, absentOrError, lowerZindex, upperZindex := MillerSliceAccess(input2, input3, len(b), zeroUp)
if sliceIsEmpty {
return mlrval.FromBytes([]byte{})
}
if absentOrError != nil {
return absentOrError
}
return mlrval.FromBytes(append([]byte(nil), b[lowerZindex:upperZindex+1]...))
}
// index(string, substring) returns the index of substring within string (if found), or -1 if not
// found.
@ -511,18 +557,19 @@ func fmtnum_te(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {
}
var fmtnum_dispositions = [mlrval.MT_DIM][mlrval.MT_DIM]BinaryFunc{
// . INT FLOAT BOOL VOID STRING ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_is, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, _absn},
/*FLOAT */ {fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_fs, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, _absn},
/*BOOL */ {fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_bs, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, _absn},
/*VOID */ {fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, _absn},
/*STRING */ {fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, _absn},
/*ARRAY */ {fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, _absn},
/*MAP */ {fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, _absn},
/*FUNC */ {fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te},
/*ERROR */ {fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te},
/*NULL */ {fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, _absn},
/*ABSENT */ {_absn, _absn, fmtnum_te, _absn, _absn, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, _absn, _absn},
// . INT FLOAT BOOL VOID STRING BYTES ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_is, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, _absn},
/*FLOAT */ {fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_fs, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, _absn},
/*BOOL */ {fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_bs, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, _absn},
/*VOID */ {fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, _absn},
/*STRING */ {fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, _absn},
/*BYTES */ {fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, _absn},
/*ARRAY */ {fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, _absn},
/*MAP */ {fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, _absn},
/*FUNC */ {fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te},
/*ERROR */ {fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te},
/*NULL */ {fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, _absn},
/*ABSENT */ {_absn, _absn, fmtnum_te, _absn, _absn, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, fmtnum_te, _absn, _absn},
}
func BIF_fmtnum(input1, input2 *mlrval.Mlrval) *mlrval.Mlrval {

37
pkg/bifs/types.go
View file

@ -50,6 +50,7 @@ var to_int_dispositions = [mlrval.MT_DIM]UnaryFunc{
/*BOOL */ bool_to_int,
/*VOID */ _void1,
/*STRING */ string_to_int,
/*BYTES */ to_int_te,
/*ARRAY */ to_int_te,
/*MAP */ to_int_te,
/*FUNC */ to_int_te,
@ -102,6 +103,7 @@ var to_int_with_base_dispositions = [mlrval.MT_DIM]BinaryFunc{
/*BOOL */ bool_to_int_with_base,
/*VOID */ _void,
/*STRING */ string_to_int_with_base,
/*BYTES */ to_int_with_base_te,
/*ARRAY */ to_int_with_base_te,
/*MAP */ to_int_with_base_te,
/*FUNC */ to_int_with_base_te,
@ -153,6 +155,7 @@ var to_float_dispositions = [mlrval.MT_DIM]UnaryFunc{
/*BOOL */ bool_to_float,
/*VOID */ _void1,
/*STRING */ string_to_float,
/*BYTES */ to_float_te,
/*ARRAY */ to_float_te,
/*MAP */ to_float_te,
/*FUNC */ to_float_te,
@ -198,6 +201,7 @@ var to_boolean_dispositions = [mlrval.MT_DIM]UnaryFunc{
/*BOOL */ _1u___,
/*VOID */ _void1,
/*STRING */ string_to_boolean,
/*BYTES */ to_boolean_te,
/*ARRAY */ to_boolean_te,
/*MAP */ to_boolean_te,
/*FUNC */ to_boolean_te,
@ -210,6 +214,33 @@ func BIF_boolean(input1 *mlrval.Mlrval) *mlrval.Mlrval {
return to_boolean_dispositions[input1.Type()](input1)
}
func string_to_bytes(input1 *mlrval.Mlrval) *mlrval.Mlrval {
return mlrval.FromBytes([]byte(input1.AcquireStringValue()))
}
func to_bytes_te(input1 *mlrval.Mlrval) *mlrval.Mlrval {
return mlrval.FromTypeErrorUnary("bytes", input1)
}
var to_bytes_dispositions = [mlrval.MT_DIM]UnaryFunc{
/*INT */ to_bytes_te,
/*FLOAT */ to_bytes_te,
/*BOOL */ to_bytes_te,
/*VOID */ string_to_bytes,
/*STRING */ string_to_bytes,
/*BYTES */ _1u___,
/*ARRAY */ to_bytes_te,
/*MAP */ to_bytes_te,
/*FUNC */ to_bytes_te,
/*ERROR */ to_bytes_te,
/*NULL */ _null1,
/*ABSENT */ _absn1,
}
func BIF_bytes(input1 *mlrval.Mlrval) *mlrval.Mlrval {
return to_bytes_dispositions[input1.Type()](input1)
}
func BIF_is_absent(input1 *mlrval.Mlrval) *mlrval.Mlrval {
return mlrval.FromBool(input1.IsAbsent())
}
@ -236,6 +267,9 @@ func BIF_is_empty(input1 *mlrval.Mlrval) *mlrval.Mlrval {
func BIF_is_emptymap(input1 *mlrval.Mlrval) *mlrval.Mlrval {
return mlrval.FromBool(input1.IsMap() && input1.AcquireMapValue().IsEmpty())
}
func BIF_is_bytes(input1 *mlrval.Mlrval) *mlrval.Mlrval {
return mlrval.FromBool(input1.IsBytes())
}
func BIF_is_float(input1 *mlrval.Mlrval) *mlrval.Mlrval {
return mlrval.FromBool(input1.IsFloat())
}
@ -324,6 +358,9 @@ func BIF_asserting_bool(input1 *mlrval.Mlrval, context *types.Context) *mlrval.M
func BIF_asserting_boolean(input1 *mlrval.Mlrval, context *types.Context) *mlrval.Mlrval {
return assertingCommon(input1, BIF_is_boolean(input1), "is_boolean", context)
}
func BIF_asserting_bytes(input1 *mlrval.Mlrval, context *types.Context) *mlrval.Mlrval {
return assertingCommon(input1, BIF_is_bytes(input1), "is_bytes", context)
}
func BIF_asserting_empty(input1 *mlrval.Mlrval, context *types.Context) *mlrval.Mlrval {
return assertingCommon(input1, BIF_is_empty(input1), "is_empty", context)
}

1
pkg/dsl/ast_types.go
View file

@ -8,6 +8,7 @@ type TNodeType string
const (
NodeTypeStringLiteral TNodeType = "string literal"
NodeTypeBytesLiteral TNodeType = "bytes literal"
NodeTypeRegex TNodeType = "regular expression" // not in the BNF -- written during CST pre-build pass
NodeTypeRegexCaseInsensitive TNodeType = "case-insensitive regular expression" // E.g. "a.*b"i -- note the trailing 'i'
NodeTypeIntLiteral TNodeType = "int literal"

1
pkg/dsl/cst/ast_types.go
View file

@ -82,6 +82,7 @@ const (
NodeTypeIntLiteral = "int_literal"
NodeTypeFloatLiteral = "float_literal"
NodeTypeStringLiteral = "string_literal"
NodeTypeBytesLiteral = "bytes_literal"
NodeTypeBoolLiteral = "bool_literal"
NodeTypeNullLiteral = "null_literal"
NodeTypeMultiIndex = "MultiIndex"

80
pkg/dsl/cst/builtin_function_manager.go
View file

@ -429,6 +429,49 @@ used within subsequent DSL statements. See also "Regular expressions" at ` + lib
binaryFunc: bifs.BIF_dot,
},
{
name: "base64_decode",
class: FUNC_CLASS_STRING,
help: `Decodes a base64-encoded string to a bytes value; use string() to interpret the result as UTF-8 text. Returns error if the input is not valid base64.`,
unaryFunc: bifs.BIF_base64_decode,
examples: []string{
`base64_decode("aGVsbG8=") gives the bytes 68656c6c6f`,
`string(base64_decode("aGVsbG8=")) gives "hello"`,
},
},
{
name: "base64_encode",
class: FUNC_CLASS_STRING,
help: `Encodes a string or bytes value using standard base64 encoding.`,
unaryFunc: bifs.BIF_base64_encode,
examples: []string{
`base64_encode("hello") gives "aGVsbG8="`,
`base64_encode(b"\xff") gives "/w=="`,
},
},
{
name: "hex_decode",
class: FUNC_CLASS_STRING,
help: `Decodes a hex-encoded string to a bytes value; use string() to interpret the result as UTF-8 text. Returns error if the input is not valid hex.`,
unaryFunc: bifs.BIF_hex_decode,
examples: []string{
`hex_decode("6869") gives the bytes 6869`,
`string(hex_decode("6869")) gives "hi"`,
},
},
{
name: "hex_encode",
class: FUNC_CLASS_STRING,
help: `Encodes a string or bytes value as lowercase hex.`,
unaryFunc: bifs.BIF_hex_encode,
examples: []string{
`hex_encode("hi") gives "6869"`,
},
},
{
name: "capitalize",
class: FUNC_CLASS_STRING,
@ -498,7 +541,7 @@ argument) doesn't match the pattern (second argument).`,
{
name: "strlen",
class: FUNC_CLASS_STRING,
help: "String length.",
help: "String length in UTF-8 characters, or bytes-value length in bytes.",
unaryFunc: bifs.BIF_strlen,
},
@ -717,25 +760,25 @@ If argument is array or map, recurses into it.`,
{
name: "md5",
class: FUNC_CLASS_HASHING,
help: `MD5 hash.`,
help: `MD5 hash of a string or bytes value.`,
unaryFunc: bifs.BIF_md5,
},
{
name: "sha1",
class: FUNC_CLASS_HASHING,
help: `SHA1 hash.`,
help: `SHA1 hash of a string or bytes value.`,
unaryFunc: bifs.BIF_sha1,
},
{
name: "sha256",
class: FUNC_CLASS_HASHING,
help: `SHA256 hash.`,
help: `SHA256 hash of a string or bytes value.`,
unaryFunc: bifs.BIF_sha256,
},
{
name: "sha512",
class: FUNC_CLASS_HASHING,
help: `SHA512 hash.`,
help: `SHA512 hash of a string or bytes value.`,
unaryFunc: bifs.BIF_sha512,
},
@ -1723,6 +1766,13 @@ See also strftime_local.`,
unaryFunc: bifs.BIF_is_boolean,
},
{
name: "is_bytes",
class: FUNC_CLASS_TYPING,
help: "True if field is present with bytes value.",
unaryFunc: bifs.BIF_is_bytes,
},
{
name: "is_empty",
class: FUNC_CLASS_TYPING,
@ -1864,6 +1914,13 @@ Note that NaN has the property that NaN != NaN, so you need 'is_nan(x)' rather t
unaryFuncWithContext: bifs.BIF_asserting_boolean,
},
{
name: "asserting_bytes",
class: FUNC_CLASS_TYPING,
help: `Aborts with an error if is_bytes on the argument returns false, else returns its argument.`,
unaryFuncWithContext: bifs.BIF_asserting_bytes,
},
{
name: "asserting_error",
class: FUNC_CLASS_TYPING,
@ -1986,6 +2043,17 @@ Note that NaN has the property that NaN != NaN, so you need 'is_nan(x)' rather t
unaryFunc: bifs.BIF_boolean,
},
{
name: "bytes",
class: FUNC_CLASS_CONVERSION,
help: "Convert string to bytes; bytes values pass through as-is. See string() for the reverse.",
unaryFunc: bifs.BIF_bytes,
examples: []string{
`bytes("hi") gives the bytes 6869`,
`string(bytes("hi")) gives "hi"`,
},
},
{
name: "float",
class: FUNC_CLASS_CONVERSION,
@ -2155,7 +2223,7 @@ argument is separator to split on.`,
{
name: "string",
class: FUNC_CLASS_CONVERSION,
help: "Convert int/float/bool/string/array/map to string.",
help: "Convert int/float/bool/string/array/map to string. For bytes values, reinterprets the raw bytes as a (UTF-8) string -- the reverse of bytes().",
unaryFunc: bifs.BIF_string,
},

3
pkg/dsl/cst/evaluable.go
View file

@ -79,7 +79,8 @@ func (root *RootNode) BuildEvaluableNode(astNode *asts.ASTNode) (IEvaluable, err
asts.NodeType(NodeTypeBracedOosvarValue), asts.NodeType(NodeTypeFullOosvar),
asts.NodeType(NodeTypeLocalVariable),
asts.NodeType(NodeTypeIntLiteral), asts.NodeType(NodeTypeFloatLiteral),
asts.NodeType(NodeTypeStringLiteral), asts.NodeType(NodeTypeBoolLiteral),
asts.NodeType(NodeTypeStringLiteral), asts.NodeType(NodeTypeBytesLiteral),
asts.NodeType(NodeTypeBoolLiteral),
asts.NodeType(NodeTypeNullLiteral), asts.NodeType(NodeTypeRegex):
return root.BuildLeafNode(astNode)
}

32
pkg/dsl/cst/leaves.go
View file

@ -42,6 +42,9 @@ func (root *RootNode) BuildLeafNode(
case asts.NodeType(NodeTypeStringLiteral):
return root.BuildStringLiteralNode(sval), nil
case asts.NodeType(NodeTypeBytesLiteral):
return root.BuildBytesLiteralNode(sval), nil
case asts.NodeType(NodeTypeRegex):
// During the BNF parse all string literals -- "foo" or "(..)_(...)"
// regexes etc -- are marked as asts.NodeType(NodeTypeStringLiteral). However, a
@ -103,6 +106,8 @@ func (root *RootNode) BuildLeafNode(
return root.BuildLocalVariableNode(sval), nil
case "string_literal":
return root.BuildStringLiteralNode(sval), nil
case "bytes_literal":
return root.BuildBytesLiteralNode(sval), nil
case "int_literal":
return root.BuildIntLiteralNode(sval), nil
case "float_literal":
@ -330,6 +335,33 @@ func (node *StringLiteralNode) Evaluate(
return node.literal
}
type BytesLiteralNode struct {
literal *mlrval.Mlrval
}
func (root *RootNode) BuildBytesLiteralNode(literal string) IEvaluable {
// The PGPG lexer produces a bytes_literal token with the leading 'b' and
// surrounding quotes in the lexeme. Strip them so b"a" becomes a.
if len(literal) >= 1 && literal[0] == 'b' {
literal = literal[1:]
}
if len(literal) >= 2 && literal[0] == '"' && literal[len(literal)-1] == '"' {
literal = literal[1 : len(literal)-1]
}
// UnbackslashStringLiteral is byte-oriented: "\xff" becomes the single
// byte 0xff, not the UTF-8 encoding of U+00FF. Unlike string literals,
// bytes literals never participate in regex-capture replacement.
return &BytesLiteralNode{
literal: mlrval.FromBytes([]byte(lib.UnbackslashStringLiteral(literal))),
}
}
func (node *BytesLiteralNode) Evaluate(
state *runtime.State,
) *mlrval.Mlrval {
return node.literal
}
// As noted above, in things like
//
// if ($x =~ "(..)_(...)" {

17
pkg/mlrval/dispositions_test.go Normal file
View file

@ -0,0 +1,17 @@
package mlrval
import (
"testing"
)
// See pkg/bifs/dispositions_test.go: guards against nil cells left by
// zero-fill when MT_DIM grows.
func TestNoNilCellsInCmpDispositions(t *testing.T) {
for i := 0; i < int(MT_DIM); i++ {
for j := 0; j < int(MT_DIM); j++ {
if cmp_dispositions[i][j] == nil {
t.Errorf("nil cell in cmp_dispositions[%d][%d]", i, j)
}
}
}
}

61
pkg/mlrval/mlrval_bytes_test.go Normal file
View file

@ -0,0 +1,61 @@
package mlrval
import (
"testing"
"github.com/stretchr/testify/assert"
)
func TestFromBytes(t *testing.T) {
mv := FromBytes([]byte{0xff, 0x01})
assert.Equal(t, MT_BYTES, mv.Type())
assert.Equal(t, "bytes", mv.GetTypeName())
assert.True(t, mv.IsBytes())
assert.Equal(t, []byte{0xff, 0x01}, mv.AcquireBytesValue())
empty := FromBytes([]byte{})
assert.Equal(t, MT_BYTES, empty.Type())
assert.Equal(t, 0, len(empty.AcquireBytesValue()))
}
func TestBytesStringIsHex(t *testing.T) {
mv := FromBytes([]byte{0xff, 0x01})
assert.Equal(t, "ff01", mv.String())
assert.Equal(t, "", FromBytes([]byte{}).String())
// StringMaybeQuoted: bytes are unquoted, like numbers
assert.Equal(t, "ff01", FromBytes([]byte{0xff, 0x01}).StringMaybeQuoted())
}
func TestBytesJSON(t *testing.T) {
mv := FromBytes([]byte{0xff, 0x01})
s, err := mv.FormatAsJSON(JSON_SINGLE_LINE, false)
assert.Nil(t, err)
assert.Equal(t, `"ff01"`, s)
}
func TestBytesCopyIsIndependent(t *testing.T) {
original := []byte{1, 2, 3}
mv := FromBytes(original)
other := mv.Copy()
original[0] = 99
assert.Equal(t, []byte{99, 2, 3}, mv.AcquireBytesValue())
assert.Equal(t, []byte{1, 2, 3}, other.AcquireBytesValue())
}
func TestBytesCmp(t *testing.T) {
a := FromBytes([]byte{0x01})
b := FromBytes([]byte{0x02})
assert.Equal(t, -1, Cmp(a, b))
assert.Equal(t, 1, Cmp(b, a))
assert.Equal(t, 0, Cmp(a, FromBytes([]byte{0x01})))
assert.True(t, Equals(a, FromBytes([]byte{0x01})))
assert.False(t, Equals(a, b))
// Mixed-type sort order: string < bytes < array
assert.Equal(t, -1, Cmp(FromString("zzz"), a))
assert.Equal(t, 1, Cmp(a, FromString("zzz")))
assert.Equal(t, -1, Cmp(a, FromArray([]*Mlrval{})))
assert.Equal(t, 1, Cmp(FromArray([]*Mlrval{}), a))
}

30
pkg/mlrval/mlrval_cmp.go
View file

@ -12,6 +12,8 @@
package mlrval
import (
"bytes"
"github.com/johnkerl/miller/v6/pkg/lib"
)
@ -113,6 +115,9 @@ func cmp_b_ff(input1, input2 *Mlrval) int {
func cmp_b_bb(input1, input2 *Mlrval) int {
return int_cmp(int64(lib.BoolToInt(input1.intf.(bool))), int64(lib.BoolToInt(input2.intf.(bool))))
}
func cmp_b_yy(input1, input2 *Mlrval) int {
return bytes.Compare(input1.intf.([]byte), input2.intf.([]byte))
}
// TODO: cmp on array & map
//func eq_b_aa(input1, input2 *Mlrval) bool {
@ -139,16 +144,17 @@ func cmp_b_bb(input1, input2 *Mlrval) int {
//}
var cmp_dispositions = [MT_DIM][MT_DIM]CmpFuncInt{
// . INT FLOAT BOOL VOID STRING ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {cmp_b_ii, cmp_b_if, _less, _less, _less, _less, _less, _less, _less, _less, _less},
/*FLOAT */ {cmp_b_fi, cmp_b_ff, _less, _less, _less, _less, _less, _less, _less, _less, _less},
/*BOOL */ {_more, _more, cmp_b_bb, _less, _less, _less, _less, _less, _less, _less, _less},
/*VOID */ {_more, _more, _more, cmp_b_ss, cmp_b_ss, _less, _less, _less, _less, _less, _less},
/*STRING */ {_more, _more, _more, cmp_b_ss, cmp_b_ss, _less, _less, _less, _less, _less, _less},
/*ARRAY */ {_more, _more, _more, _more, _more, _same, _less, _less, _less, _less, _less},
/*MAP */ {_more, _more, _more, _more, _more, _more, _same, _less, _less, _less, _less},
/*func */ {_more, _more, _more, _more, _more, _more, _more, _same, _less, _less, _less},
/*ERROR */ {_more, _more, _more, _more, _more, _more, _more, _more, _same, _less, _less},
/*NULL */ {_more, _more, _more, _more, _more, _more, _more, _more, _more, _same, _less},
/*ABSENT */ {_more, _more, _more, _more, _more, _more, _more, _more, _more, _more, _same},
// . INT FLOAT BOOL VOID STRING BYTES ARRAY MAP FUNC ERROR NULL ABSENT
/*INT */ {cmp_b_ii, cmp_b_if, _less, _less, _less, _less, _less, _less, _less, _less, _less, _less},
/*FLOAT */ {cmp_b_fi, cmp_b_ff, _less, _less, _less, _less, _less, _less, _less, _less, _less, _less},
/*BOOL */ {_more, _more, cmp_b_bb, _less, _less, _less, _less, _less, _less, _less, _less, _less},
/*VOID */ {_more, _more, _more, cmp_b_ss, cmp_b_ss, _less, _less, _less, _less, _less, _less, _less},
/*STRING */ {_more, _more, _more, cmp_b_ss, cmp_b_ss, _less, _less, _less, _less, _less, _less, _less},
/*BYTES */ {_more, _more, _more, _more, _more, cmp_b_yy, _less, _less, _less, _less, _less, _less},
/*ARRAY */ {_more, _more, _more, _more, _more, _more, _same, _less, _less, _less, _less, _less},
/*MAP */ {_more, _more, _more, _more, _more, _more, _more, _same, _less, _less, _less, _less},
/*func */ {_more, _more, _more, _more, _more, _more, _more, _more, _same, _less, _less, _less},
/*ERROR */ {_more, _more, _more, _more, _more, _more, _more, _more, _more, _same, _less, _less},
/*NULL */ {_more, _more, _more, _more, _more, _more, _more, _more, _more, _more, _same, _less},
/*ABSENT */ {_more, _more, _more, _more, _more, _more, _more, _more, _more, _more, _more, _same},
}

2
pkg/mlrval/mlrval_copy.go
View file

@ -8,6 +8,8 @@ func (mv *Mlrval) Copy() *Mlrval {
other.intf = mv.intf.(*Mlrmap).Copy()
case MT_ARRAY:
other.intf = CopyMlrvalArray(mv.intf.([]*Mlrval))
case MT_BYTES:
other.intf = append([]byte(nil), mv.intf.([]byte)...)
}
return &other
}

5
pkg/mlrval/mlrval_get.go
View file

@ -149,6 +149,11 @@ func (mv *Mlrval) AcquireBoolValue() bool {
return mv.intf.(bool)
}
func (mv *Mlrval) AcquireBytesValue() []byte {
lib.InternalCodingErrorIf(mv.mvtype != MT_BYTES)
return mv.intf.([]byte)
}
func (mv *Mlrval) AcquireArrayValue() []*Mlrval {
lib.InternalCodingErrorIf(mv.mvtype != MT_ARRAY)
return mv.intf.([]*Mlrval)

5
pkg/mlrval/mlrval_is.go
View file

@ -89,6 +89,11 @@ func (mv *Mlrval) IsStringOrInt() bool {
return t == MT_STRING || t == MT_VOID || t == MT_INT
}
func (mv *Mlrval) IsBytes() bool {
// Bytes are never from deferred type, so no JIT-infer is needed.
return mv.mvtype == MT_BYTES
}
func (mv *Mlrval) IsInt() bool {
return mv.Type() == MT_INT
}

9
pkg/mlrval/mlrval_json.go
View file

@ -284,6 +284,8 @@ func (mv *Mlrval) marshalJSONAux(
return mv.marshalJSONVoid(outputIsStdout)
case MT_STRING:
return mv.marshalJSONString(outputIsStdout)
case MT_BYTES:
return mv.marshalJSONBytes(outputIsStdout)
case MT_INT:
return mv.marshalJSONInt(outputIsStdout)
case MT_FLOAT:
@ -337,6 +339,13 @@ func (mv *Mlrval) marshalJSONString(outputIsStdout bool) (string, error) {
return colorizer.MaybeColorizeValue(millerJSONEncodeString(mv.printrep), outputIsStdout), nil
}
// Bytes are JSON-encoded as their hex representation, quoted as a JSON
// string. Hex is quote-safe ASCII so no further escaping is needed.
func (mv *Mlrval) marshalJSONBytes(outputIsStdout bool) (string, error) {
lib.InternalCodingErrorIf(mv.mvtype != MT_BYTES)
return colorizer.MaybeColorizeValue(`"`+mv.String()+`"`, outputIsStdout), nil
}
// Wraps with double-quotes and escape-encoded JSON-special characters.
//
// Per https://www.json.org/json-en.html:

8
pkg/mlrval/mlrval_new.go
View file

@ -176,6 +176,14 @@ func (mv *Mlrval) SetFromString(input string) *Mlrval {
return mv
}
func FromBytes(input []byte) *Mlrval {
return &Mlrval{
mvtype: MT_BYTES,
printrepValid: false,
intf: input,
}
}
func FromInt(input int64) *Mlrval {
return &Mlrval{
mvtype: MT_INT,

4
pkg/mlrval/mlrval_output.go
View file

@ -1,6 +1,7 @@
package mlrval
import (
"encoding/hex"
"fmt"
"os"
"reflect"
@ -91,6 +92,9 @@ func (mv *Mlrval) setPrintRep() {
mv.printrep = "false"
}
case MT_BYTES:
mv.printrep = hex.EncodeToString(mv.intf.([]byte))
case MT_ARRAY:
bytes, err := mv.FormatAsJSON(JSON_MULTILINE, false)
// maybe just InternalCodingErrorIf(err != nil)

27
pkg/mlrval/mlrval_type.go
View file

@ -105,14 +105,21 @@ const (
MT_STRING MVType = 4
// intf is []byte. Bytes values are immutable once constructed; Copy()
// clones the slice defensively. Placed right after MT_STRING since bytes
// sort adjacent to strings in the mixed-type sort order (see the cmp
// disposition matrices, which are the actual source of truth for that
// ordering).
MT_BYTES MVType = 5
// intf is []*Mlrval
MT_ARRAY MVType = 5
MT_ARRAY MVType = 6
// intf is *Mlrmap
MT_MAP MVType = 6
MT_MAP MVType = 7
// intf is interface{} -- resolved in the cst package to avoid circular dependencies
MT_FUNC MVType = 7
MT_FUNC MVType = 8
// E.g. error encountered in one eval & it propagates up the AST at
// evaluation time. Various runtime errors, such as file-not-found, result
@ -120,22 +127,22 @@ const (
// are intended to result in "(error)"-valued output rather than a crash.
// This is analogous to the way that IEEE-754 arithmetic carries around
// Inf and NaN through computation chains.
MT_ERROR MVType = 8
MT_ERROR MVType = 9
// Used only for JSON null, and for 'empty' slots when an array is
// auto-extended by assigning to an index having a gap from the last index.
// E.g. x=[1,2,3] then x[5]=5; now x[4] is null
MT_NULL MVType = 9
MT_NULL MVType = 10
// Key not present in input record, e.g. 'foo = $nosuchkey'
MT_ABSENT MVType = 10
MT_ABSENT MVType = 11
// Not a type -- this is a dimension for disposition vectors and
// disposition matrices. For example, when we want to add two mlrvals,
// instead of if/elsing or switching on the types of both operands, we
// instead jump directly to a type-specific function in a matrix of
// function pointers which is MT_DIM x MT_DIM.
MT_DIM MVType = 11
MT_DIM MVType = 12
)
var TYPE_NAMES = [MT_DIM]string{
@ -144,6 +151,7 @@ var TYPE_NAMES = [MT_DIM]string{
"bool",
"empty", // For backward compatibility with the C impl: this is user-visible
"string",
"bytes",
"array",
"map",
"funct",
@ -162,6 +170,7 @@ const MT_TYPE_MASK_BOOL = 1 << MT_BOOL
const MT_TYPE_MASK_STRING = (1 << MT_STRING) | (1 << MT_VOID)
const MT_TYPE_MASK_ARRAY = 1 << MT_ARRAY
const MT_TYPE_MASK_MAP = 1 << MT_MAP
const MT_TYPE_MASK_BYTES = 1 << MT_BYTES
const MT_TYPE_MASK_VAR = (1 << MT_INT) |
(1 << MT_FLOAT) |
(1 << MT_BOOL) |
@ -169,7 +178,8 @@ const MT_TYPE_MASK_VAR = (1 << MT_INT) |
(1 << MT_NULL) |
(1 << MT_STRING) |
(1 << MT_ARRAY) |
(1 << MT_MAP)
(1 << MT_MAP) |
(1 << MT_BYTES)
const MT_TYPE_MASK_FUNC = 1 << MT_FUNC
// Not exposed in userspace
@ -184,6 +194,7 @@ var typeNameToMaskMap = map[string]int{
"str": MT_TYPE_MASK_STRING,
"arr": MT_TYPE_MASK_ARRAY,
"map": MT_TYPE_MASK_MAP,
"bytes": MT_TYPE_MASK_BYTES,
"funct": MT_TYPE_MASK_FUNC,
"var": MT_TYPE_MASK_VAR,
"any": MT_TYPE_MASK_ANY,

2
pkg/mlrval/mlrval_yaml.go
View file

@ -208,6 +208,8 @@ func mlrvalToYAMLNode(mv *Mlrval) (*yaml.Node, error) {
case MT_MAP:
m := mv.GetMap()
return MlrmapToYAMLNative(m)
case MT_BYTES:
return encodeScalarNode(mv.String())
case MT_ERROR, MT_PENDING:
return encodeScalarNode(mv.String())
default:

22993
pkg/parsing/gen/lexer.json
View file

File diff suppressed because it is too large Load diff

914212
pkg/parsing/gen/parser.json
View file

File diff suppressed because it is too large Load diff

13594
pkg/parsing/lexer/lexer.go
View file

File diff suppressed because it is too large Load diff

5
pkg/parsing/mlr.bnf
View file

@ -95,6 +95,8 @@ const_M_PI ::= 'M' '_' 'P' 'I' ;
const_M_E ::= 'M' '_' 'E' ;
# %%%panic%%% - special token for short-circuit tests (causes panic when evaluated)
panic ::= '%' '%' '%' 'p' 'a' 'n' 'i' 'c' '%' '%' '%' ;
# Bytes literals b"..." (must precede non_sigil_name since 'b' is an idchar)
bytes_literal ::= 'b' '"' { _string_char | _escape } '"' ;
non_sigil_name ::= _leading_idchar { _idchar } ;
_hex ::= '0'-'9' | 'A'-'F' | 'a'-'f' ;
@ -652,6 +654,7 @@ MlrvalOrFunction ::=
| FloatLiteral
| RegexCaseInsensitive
| StringLiteral
| BytesLiteral
| BoolLiteral
| NullLiteral
| PanicLiteral
@ -702,6 +705,8 @@ RegexCaseInsensitive ::=
string_literal non_sigil_name -> { "parent": 0, "children": [0], "type": "RegexCaseInsensitive" } ;
StringLiteral::=
string_literal -> { "parent": 0, "children": [], "type": "string_literal" } ;
BytesLiteral ::=
bytes_literal -> { "parent": 0, "children": [], "type": "bytes_literal" } ;
BoolLiteral ::=
literal_true -> { "parent": 0, "children": [], "type": "bool_literal" }
| literal_false -> { "parent": 0, "children": [], "type": "bool_literal" }

315233
pkg/parsing/parser/parser.go
View file

File diff suppressed because it is too large Load diff

1
test/cases/dsl-bytes-conversions/0001/cmd Normal file
View file

@ -0,0 +1 @@
mlr repl -s -q < ./${CASEDIR}/input

0
test/cases/dsl-bytes-conversions/0001/experr Normal file
View file

22
test/cases/dsl-bytes-conversions/0001/expout Normal file
View file

@ -0,0 +1,22 @@
"bytes"
6869
"hi"
bytes: unacceptable type int with value 5
true
false
"aGVsbG8="
68656c6c6f
"bytes"
"hello"
illegal base64 data at input byte 0
"/w=="
ff
"привет"
"6869"
6869
"hi"
encoding/hex: invalid byte: U+0078 'x'
true
"da39a3ee5e6b4b0d3255bfef95601890afd80709"
"a8100ae6aa1940d0b663bb31cd466142ebbdbd5187131b92d93818987832eb89"
1

22
test/cases/dsl-bytes-conversions/0001/input Normal file
View file

@ -0,0 +1,22 @@
typeof(bytes("hi"))
bytes("hi")
string(bytes("hi"))
bytes(5)
is_bytes(bytes("hi"))
is_bytes("hi")
base64_encode("hello")
base64_decode("aGVsbG8=")
typeof(base64_decode("aGVsbG8="))
string(base64_decode("aGVsbG8="))
base64_decode("abc")
base64_encode(b"\xff")
base64_decode("/w==")
string(base64_decode("0L/RgNC40LLQtdGC"))
hex_encode("hi")
hex_decode("6869")
string(hex_decode("6869"))
hex_decode("xyz")
md5(b"miller") == md5("miller")
sha1(bytes(""))
sha256(b"\xff")
strlen(base64_decode("/w=="))

1
test/cases/dsl-bytes-conversions/non-windows/0002/cmd Normal file
View file

@ -0,0 +1 @@
mlr --icsv --ojson put '$decoded = base64_decode($enc); $text = string($decoded); $nested = {"payload": $decoded}' ./${CASEDIR}/input

0
test/cases/dsl-bytes-conversions/non-windows/0002/experr Normal file
View file

20
test/cases/dsl-bytes-conversions/non-windows/0002/expout Normal file
View file

@ -0,0 +1,20 @@
[
{
"id": 1,
"enc": "aGVsbG8=",
"decoded": "68656c6c6f",
"text": "hello",
"nested": {
"payload": "68656c6c6f"
}
},
{
"id": 2,
"enc": "/wE=",
"decoded": "ff01",
"text": "ÿ\u0001",
"nested": {
"payload": "ff01"
}
}
]

3
test/cases/dsl-bytes-conversions/non-windows/0002/input Normal file
View file

@ -0,0 +1,3 @@
id,enc
1,aGVsbG8=
2,/wE=

1
test/cases/dsl-bytes-literals/0001/cmd Normal file
View file

@ -0,0 +1 @@
mlr repl -s -q < ./${CASEDIR}/input

0
test/cases/dsl-bytes-literals/0001/experr Normal file
View file

12
test/cases/dsl-bytes-literals/0001/expout Normal file
View file

@ -0,0 +1,12 @@
"bytes"
"bytes"
0
ff01
2
616263
414243
0a095c22
414243
68c3a96c6c6f
6
e29da6

12
test/cases/dsl-bytes-literals/0001/input Normal file
View file

@ -0,0 +1,12 @@
typeof(b"")
typeof(b"abc")
strlen(b"")
b"\xff\x01"
strlen(b"\xff\x01")
b"abc"
b"\x41\x42\x43"
b"\n\t\\\""
b"\101\102\103"
b"héllo"
strlen(b"héllo")
b"❦"

1
test/cases/dsl-bytes-ops/0001/cmd Normal file
View file

@ -0,0 +1 @@
mlr repl -s -q < ./${CASEDIR}/input

0
test/cases/dsl-bytes-ops/0001/experr Normal file
View file

18
test/cases/dsl-bytes-ops/0001/expout Normal file
View file

@ -0,0 +1,18 @@
0102
true
false
true
true
false
-1
false
0102
0102
0001
"bytes"
[3, "zz", "01", "02"]
+: unacceptable types bytes, int with values 01, 1
&: unacceptable types bytes, bytes with values 01, 02
fmtnum: unacceptable types bytes, string with values 01, "%d"
.: unacceptable types bytes, string with values 01, "abc"

18
test/cases/dsl-bytes-ops/0001/input Normal file
View file

@ -0,0 +1,18 @@
b"\x01" . b"\x02"
b"" . b""
b"\x01\x02" == b"\x01\x02"
b"\x01" == b"\x02"
b"\x01" != b"\x02"
b"\x01" < b"\x02"
b"\x02" <= b"\x01"
b"\x01" <=> b"\x02"
b"\x01" == "x"
substr(b"\x00\x01\x02\x03", 1, 2)
substr0(b"\x00\x01\x02\x03", 1, 2)
substr1(b"\x00\x01\x02\x03", 1, 2)
typeof(substr(b"\x00\x01", 1, 0))
sort([b"\x02", b"\x01", "zz", 3])
b"\x01" + 1
b"\x01" & b"\x02"
fmtnum(b"\x01", "%d")
b"\x01" . "abc"