miller/pkg/mlrval/mlrval_is.go
John Kerl 12c96298b9
Add a first-class bytes type to the DSL, with b"..." literals and base64/hex codecs (#2122)
* Add MT_BYTES mlrval type: foundation and disposition tables

First step toward a first-class bytes type in the DSL (#1231).
Adds MT_BYTES (payload []byte, rendered as lowercase hex in all output
formats, JSON-encoded as a hex string), extends every disposition
matrix/vector with the new row/column -- real cells for comparison,
sorting, and dot-concat of bytes with bytes; type-error stubs
elsewhere -- and adds sweep tests asserting no table has nil cells,
since Go zero-fills short array literals when MT_DIM grows.

Bytes values are not yet constructible from the DSL; b"..." literals
and constructor/codec functions follow in subsequent commits.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Add b"..." bytes-literal syntax to the DSL

Adds a bytes_literal token to the grammar (regenerating the PGPG lexer
and parser) and a BytesLiteralNode in the CST which evaluates to an
MT_BYTES mlrval. Escape handling reuses UnbackslashStringLiteral,
which is already byte-oriented: b"\xff" is the single byte 0xff.
Unlike string literals, bytes literals never participate in
regex-capture replacement. A bare identifier b is unaffected.

Part of #1231.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Add bytes DSL functions: conversions, codecs, and bytes-aware built-ins

- bytes(x) converts strings to bytes; string(b) reinterprets raw bytes
  as UTF-8 text (the reverse)
- base64_decode now always returns bytes (superseding the interim
  string-or-hex behavior); base64_encode accepts string or bytes
- New hex_encode/hex_decode functions
- is_bytes and asserting_bytes predicates
- md5/sha1/sha256/sha512 accept bytes, hashing the raw payload
- strlen of bytes is the byte count; substr/substr0/substr1 on bytes
  slice by byte position and return bytes

The Cyrillic-LDAP scenario from #1231 now works without exec
workarounds: string(base64_decode($x)) recovers the text, and binary
payloads survive undamaged as bytes.

Closes #1231.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Add bytes-type docs and regression cases

Documents the bytes type on the data-types page, regenerates the
function-reference/man-page material, and adds regression coverage:
literal escape forms, operators (concat/compare/slice/sort and
type errors), conversions and codec round-trips, and CSV-to-JSON
output rendering of bytes fields.

Part of #1231.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Reposition MT_BYTES to sort adjacent to MT_STRING in the type enum

MT_BYTES was appended after MT_ABSENT for index stability; move it
right after MT_STRING instead, since that's where it conceptually
belongs and where it already sorts in the cmp disposition matrices.
Mechanically re-derive all ~40 disposition tables in pkg/bifs and
pkg/mlrval accordingly.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix windows CI

* fix merge

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 11:58:44 -04:00

148 lines
3.6 KiB
Go

package mlrval
import (
"github.com/johnkerl/miller/v6/pkg/lib"
)
// It's essential that we use mv.Type() not mv.mvtype since types are
// JIT-computed on first access for most data-file values. See type.go for more
// information.
func (mv *Mlrval) IsLegit() bool {
t := mv.Type()
return MT_INT <= t && t < MT_ERROR
}
// TODO: comment no JIT-infer here -- absent is non-inferrable and we needn't take the expense of JIT.
func (mv *Mlrval) IsErrorOrAbsent() bool {
t := mv.mvtype
return t == MT_ERROR || t == MT_ABSENT
}
func (mv *Mlrval) IsError() bool {
return mv.Type() == MT_ERROR
}
func (mv *Mlrval) GetError() (bool, error) {
if mv.Type() == MT_ERROR {
return true, mv.err
}
return false, nil
}
// TODO: comment no JIT-infer here -- absent is non-inferrable and we needn't take the expense of JIT.
func (mv *Mlrval) IsAbsent() bool {
return mv.mvtype == MT_ABSENT
}
// TODO: comment no JIT-infer here -- NULL is non-inferrable and we needn't take the expense of JIT.
// This is a literal in JSON files, or else explicitly set to NULL.
func (mv *Mlrval) IsNull() bool {
return mv.mvtype == MT_NULL
}
func (mv *Mlrval) IsVoid() bool {
if mv.mvtype == MT_VOID {
return true
}
if mv.mvtype == MT_PENDING && mv.printrep == "" {
lib.InternalCodingErrorIf(!mv.printrepValid)
return true
}
return false
}
func (mv *Mlrval) IsErrorOrVoid() bool {
return mv.IsError() || mv.IsVoid()
}
// * Error is non-empty
// * Absent is non-empty (shouldn't have been assigned in the first place; error should be surfaced)
// * Void is empty
// * Empty string is empty
// * Int/float/bool/array/map are all non-empty
func (mv *Mlrval) IsEmptyString() bool {
if mv.mvtype == MT_VOID {
return true
}
if mv.mvtype == MT_STRING && mv.printrep == "" {
return true
}
if mv.mvtype == MT_PENDING && mv.printrep == "" {
lib.InternalCodingErrorIf(!mv.printrepValid)
return true
}
return false
}
func (mv *Mlrval) IsString() bool {
return mv.Type() == MT_STRING
}
func (mv *Mlrval) IsStringOrVoid() bool {
t := mv.Type()
return t == MT_STRING || t == MT_VOID
}
func (mv *Mlrval) IsStringOrInt() bool {
t := mv.Type()
return t == MT_STRING || t == MT_VOID || t == MT_INT
}
func (mv *Mlrval) IsBytes() bool {
// Bytes are never from deferred type, so no JIT-infer is needed.
return mv.mvtype == MT_BYTES
}
func (mv *Mlrval) IsInt() bool {
return mv.Type() == MT_INT
}
func (mv *Mlrval) IsFloat() bool {
return mv.Type() == MT_FLOAT
}
func (mv *Mlrval) IsNumeric() bool {
t := mv.Type()
return t == MT_INT || t == MT_FLOAT
}
func (mv *Mlrval) IsIntZero() bool {
return mv.Type() == MT_INT && mv.intf.(int64) == 0
}
func (mv *Mlrval) IsBool() bool {
return mv.Type() == MT_BOOL
}
func (mv *Mlrval) IsTrue() bool {
return mv.Type() == MT_BOOL && mv.intf.(bool)
}
func (mv *Mlrval) IsFalse() bool {
return mv.Type() == MT_BOOL && !mv.intf.(bool)
}
func (mv *Mlrval) IsArray() bool {
// TODO: comment non-deferrable type -- don't force a (potentially
// expensive in bulk) JIT-infer of other types
// return mv.Type() == MT_ARRAY
return mv.mvtype == MT_ARRAY
}
func (mv *Mlrval) IsMap() bool {
// TODO: comment non-deferrable type -- don't force a (potentially
// expensive in bulk) JIT-infer of other types
// return mv.Type() == MT_ARRAY
return mv.mvtype == MT_MAP
}
func (mv *Mlrval) IsArrayOrMap() bool {
// TODO: comment why not
// In flatten we don't want to type-infer things that don't need to be jitted.
// Arrays & maps are never from deferred type.
// t := mv.Type()
t := mv.mvtype
return t == MT_ARRAY || t == MT_MAP
}
func (mv *Mlrval) IsFunction() bool {
return mv.mvtype == MT_FUNC
}