mirror of
https://github.com/johnkerl/miller.git
synced 2026-07-18 00:45:47 +00:00
Replace GOCC parser-generator with PGPG (#2015)
* Porting * Update some tests that depend on AST-print output * iterating on GOCC -> PGPG * iterating * iterating * iterating * iterating * iterating * Modify expout files that need only AST-print-syntax updates * iterating * iterating * iterating * iterating * iterating * iterating * iterating * iterating * iterating * iterating * iterating * sync test cases from mlr-6.17.0, except AST-prints ... * sync test cases from mlr-6.17.0, except AST-prints ... * Fix lashed emit for $* and @* * Fix --1 and ++1 chained unary ops * Emit with function callsite * test cases * dot operator * M_PI and M_E * Iterating on lashed emit cases * error-wording differences * rm some should-fail files * Fix issue with leading semicolon * trailing comma in func params; most AST-print deltas * error-wording delta * fix unset all * AST-print deltas * go mod tidy: forced `go 1.25` to `go 1.25.0` * Depend on PGPG v1.0.0 * Fix Windows CI failure * Remove cmd/experiments/dsl_parser * GOCC -> PGPG * neaten * Fix regex issue found in doc gen
This commit is contained in:
John Kerl
GitHub
parent
40f3a72de2
commit
af1adf80ad
732 changed files with 924001 additions and 1122928 deletions
|
|
@ -94,15 +94,12 @@ So, in broad overview, the key packages are:
|
|||
### Dependencies
|
||||
|
||||
* Miller dependencies are all in the Go standard library, except two:
|
||||
* GOCC lexer/parser code-generator from [github.com/goccmack/gocc](https://github.com/goccmack/gocc):
|
||||
* Forked at [github.com/johnkerl/gocc](github.com/johnkerl/gocc).
|
||||
* This package defines the grammar for Miller's domain-specific language (DSL) for the Miller `put` and `filter` verbs. And, GOCC is a joy to use. :)
|
||||
* It is used on the terms of its open-source license.
|
||||
* PGPG lexer/parser code-generator from [github.com/johnkerl/pgpg](https://github.com/johnkerl/pgpg):
|
||||
* This package defines the grammar for Miller's domain-specific language (DSL) for the Miller `put` and `filter` verbs.
|
||||
* [golang.org/x/term](https://pkg.go.dev/golang.org/x/term):
|
||||
* Just a one-line Miller callsite for is-a-terminal checking for the [Miller REPL](./pkg/terminals/repl/README.md).
|
||||
* It is used on the terms of its open-source license.
|
||||
* See also [./go.mod](go.mod). Setup:
|
||||
* `go get github.com/johnkerl/gocc`
|
||||
* `go get golang.org/x/term`
|
||||
|
||||
### Miller per se
|
||||
|
|
@ -120,9 +117,9 @@ So, in broad overview, the key packages are:
|
|||
* [pkg/input](./pkg/input) is as above -- one record-reader type per supported input file format, and a factory method.
|
||||
* [pkg/output](./pkg/output) is as above -- one record-writer type per supported output file format, and a factory method.
|
||||
* [pkg/transformers](./pkg/transformers) contains the abstract record-transformer interface datatype, as well as the Go-channel chaining mechanism for piping one transformer into the next. It also contains all the concrete record-transformers such as `cat`, `tac`, `sort`, `put`, and so on.
|
||||
* [pkg/parsing](./pkg/parsing) contains a single source file, `mlr.bnf`, which is the lexical/semantic grammar file for the Miller `put`/`filter` DSL using the GOCC framework. All subdirectories of `pkg/parsing/` are autogen code created by GOCC's processing of `mlr.bnf`. If you need to edit `mlr.bnf`, please use [tools/build-dsl](./tools/build-dsl) to autogenerate Go code from it (using the GOCC tool). (This takes several minutes to run.) See also [tools/format-go-in-bnf](./tools/format-go-in-bnf) (which reads `stdin` and writes `stdout`) for automated formatting of the Go bits.
|
||||
* [pkg/dsl](./pkg/dsl) contains [`ast_types.go`](pkg/dsl/ast_types.go) which is the abstract syntax tree datatype shared between GOCC and Miller. I didn't use a `pkg/dsl/ast` naming convention, although that would have been nice, in order to avoid a Go package-dependency cycle.
|
||||
* [pkg/dsl/cst](./pkg/dsl/cst) is the concrete syntax tree, constructed from an AST produced by GOCC. The CST is what is actually executed on every input record when you do things like `$z = $x * 0.3 * $y`. Please see the [pkg/dsl/cst/README.md](./pkg/dsl/cst/README.md) for more information.
|
||||
* [pkg/parsing](./pkg/parsing) contains a single source file, `mlr.bnf`, which is the lexical/semantic grammar file for the Miller `put`/`filter` DSL using the PGPG framework. All subdirectories of `pkg/parsing/` are autogen code created by PGPG's processing of `mlr.bnf`. If you need to edit `mlr.bnf`, please use [tools/build-dsl](./tools/build-dsl) to autogenerate Go code from it (using the PGPG tool). (This takes several minutes to run.) See also [tools/format-go-in-bnf](./tools/format-go-in-bnf) (which reads `stdin` and writes `stdout`) for automated formatting of the Go bits.
|
||||
* [pkg/dsl](./pkg/dsl) contains [`ast_types.go`](pkg/dsl/ast_types.go) which is the abstract syntax tree datatype shared between PGPG and Miller. I didn't use a `pkg/dsl/ast` naming convention, although that would have been nice, in order to avoid a Go package-dependency cycle.
|
||||
* [pkg/dsl/cst](./pkg/dsl/cst) is the concrete syntax tree, constructed from an AST produced by PGPG. The CST is what is actually executed on every input record when you do things like `$z = $x * 0.3 * $y`. Please see the [pkg/dsl/cst/README.md](./pkg/dsl/cst/README.md) for more information.
|
||||
|
||||
## Nil-record conventions
|
||||
|
||||
|
|
|
|||
|
|
@ -1 +0,0 @@
|
|||
map \f :w<C-m>:!clear;echo Building ...; echo; build; echo; main<C-m>
|
||||
|
|
@ -1 +0,0 @@
|
|||
Running the GOCC parser-generator for the full Miller grammar takes a few minutes. That's a bit painful for experimentation; hence this.
|
||||
|
|
@ -1,62 +0,0 @@
|
|||
#!/bin/bash
|
||||
|
||||
# ----------------------------------------------------------------
|
||||
# Setup
|
||||
us=$(basename $0)
|
||||
set -euo pipefail
|
||||
|
||||
verbose="false"
|
||||
|
||||
if [ $# -eq 1 ]; then
|
||||
bnf="$1"
|
||||
elif [ $# -eq 2 ]; then
|
||||
if [ "$1" = "-v" ]; then
|
||||
verbose="true"
|
||||
else
|
||||
echo "Usage: $0 {.bnf file}" 1>&2
|
||||
exit 1
|
||||
fi
|
||||
bnf="$2"
|
||||
else
|
||||
echo "Usage: $0 {.bnf file}" 1>&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
dir=src
|
||||
mkdir -p $dir
|
||||
|
||||
# ----------------------------------------------------------------
|
||||
# Run the parser-generator
|
||||
|
||||
# Build the bin/gocc executable (use my fork for performance):
|
||||
# get github.com/goccmack/gocc
|
||||
go get github.com/johnkerl/gocc
|
||||
bingocc="$GOPATH/bin/gocc"
|
||||
|
||||
if [ ! -x "$bingocc" ]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
rm -f $dir/*.txt
|
||||
if [ "$verbose" = "true" ]; then
|
||||
lr1="$dir/LR1_conflicts.txt"
|
||||
# The -o specifies the package name within the autogen
|
||||
$bingocc -v -o $dir $bnf || expand -2 $lr1
|
||||
else
|
||||
$bingocc -o $dir $bnf
|
||||
fi
|
||||
|
||||
echo "Parser-autogen OK"
|
||||
|
||||
# Code-gen directories:
|
||||
# $dir/errors/
|
||||
# $dir/lexer/
|
||||
# $dir/parser/
|
||||
# $dir/token/
|
||||
# $dir/util/
|
||||
|
||||
# ----------------------------------------------------------------
|
||||
# Compile the main and the parser-autogen
|
||||
|
||||
go build main.go
|
||||
echo "Compile OK"
|
||||
|
|
@ -1,5 +0,0 @@
|
|||
module one
|
||||
|
||||
go 1.24
|
||||
|
||||
toolchain go1.24.5
|
||||
|
|
@ -1,74 +0,0 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
|
||||
"one/src/lexer"
|
||||
"one/src/parser"
|
||||
)
|
||||
|
||||
func parseOne(input string) {
|
||||
theLexer := lexer.NewLexer([]byte(input))
|
||||
theParser := parser.NewParser()
|
||||
_, err := theParser.Parse(theLexer)
|
||||
|
||||
green := "\033[32;01m"
|
||||
red := "\033[31;01m"
|
||||
textdefault := "\033[0m"
|
||||
|
||||
if err != nil {
|
||||
//fmt.Println(err)
|
||||
fmt.Printf("%sFail%s %s\n", red, textdefault, input)
|
||||
} else {
|
||||
fmt.Printf("%sOK%s %s\n", green, textdefault, input)
|
||||
}
|
||||
}
|
||||
|
||||
func main() {
|
||||
if len(os.Args) == 1 {
|
||||
|
||||
fmt.Println("EXPECT OK")
|
||||
goods := []string{
|
||||
"",
|
||||
";",
|
||||
";;",
|
||||
"x",
|
||||
"x;x",
|
||||
"x;x;x",
|
||||
"x;x;x;x",
|
||||
"x;",
|
||||
"x;;",
|
||||
";x",
|
||||
";;x",
|
||||
"x ; {}",
|
||||
"{} ; x",
|
||||
"{} x",
|
||||
"{ x }",
|
||||
"{ x; x }",
|
||||
"x; { x; x }",
|
||||
"{ x; x } x",
|
||||
"{ x; x } ; x",
|
||||
"{};{}",
|
||||
"{} {}",
|
||||
}
|
||||
for _, input := range goods {
|
||||
parseOne(input)
|
||||
}
|
||||
|
||||
fmt.Println()
|
||||
fmt.Println("EXPECT FAIL")
|
||||
bads := []string{
|
||||
"x x",
|
||||
"x {}",
|
||||
}
|
||||
for _, input := range bads {
|
||||
parseOne(input)
|
||||
}
|
||||
|
||||
} else {
|
||||
for _, arg := range os.Args[1:] {
|
||||
parseOne(arg)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -1,9 +0,0 @@
|
|||
#!/bin/bash
|
||||
|
||||
if [ $# -lt 1 ]; then
|
||||
echo "Usage: $0 {.bnf file} [expressions]" 1>&2
|
||||
exit 1
|
||||
fi
|
||||
bnf="$1"
|
||||
shift
|
||||
./build $bnf && echo && main "$@"
|
||||
|
|
@ -1,29 +0,0 @@
|
|||
!whitespace : ' ' | '\t' | '\n' | '\r' ;
|
||||
|
||||
Root
|
||||
: StatementBlock
|
||||
;
|
||||
|
||||
StatementBlock
|
||||
: ";"
|
||||
| BracelessStatement
|
||||
| ";" BracelessStatement
|
||||
| BracefulStatement
|
||||
| BracelessStatement ";" NonEmptyStatementBlock
|
||||
| BracefulStatement NonEmptyStatementBlock
|
||||
;
|
||||
|
||||
NonEmptyStatementBlock
|
||||
: BracelessStatement
|
||||
| BracelessStatement ";" StatementBlock
|
||||
| BracefulStatement
|
||||
| BracefulStatement StatementBlock
|
||||
;
|
||||
|
||||
BracelessStatement
|
||||
: "x"
|
||||
;
|
||||
|
||||
BracefulStatement
|
||||
: "{" StatementBlock "}"
|
||||
;
|
||||
|
|
@ -1,25 +0,0 @@
|
|||
!whitespace : ' ' | '\t' | '\n' | '\r' ;
|
||||
|
||||
Root
|
||||
: StatementBlock
|
||||
;
|
||||
|
||||
StatementBlock
|
||||
: empty
|
||||
| ";" StatementBlock
|
||||
| BracelessStatement
|
||||
| BracefulStatement
|
||||
| BracelessStatement ";" StatementBlock
|
||||
| BracefulStatement ";" BracefulStatement
|
||||
| BracefulStatement BracefulStatement
|
||||
| BracefulStatement ";" BracelessStatement
|
||||
| BracefulStatement BracelessStatement
|
||||
;
|
||||
|
||||
BracelessStatement
|
||||
: "x"
|
||||
;
|
||||
|
||||
BracefulStatement
|
||||
: "{" StatementBlock "}"
|
||||
;
|
||||
|
|
@ -1,42 +0,0 @@
|
|||
// ================================================================
|
||||
// LEXER
|
||||
|
||||
_string_literal_element
|
||||
: 'A'-'Z' | 'a'-'z' | '0'-'9'
|
||||
| ' ' | '!' | '#' | '$' | '%' | '&' | '\'' | '\\'
|
||||
| '(' | ')' | '*' | '+' | ',' | '-' | '.' | '/'
|
||||
| ':' | ';' | '<' | '=' | '>' | '?' | '@' | '['
|
||||
| ']' | '^' | '_' | '`' | '{' | '|' | '}' | '~'
|
||||
| ( '\\' '"' )
|
||||
| ( '\\' '[' )
|
||||
| ( '\\' ']' )
|
||||
| ( '\\' 'b' )
|
||||
| ( '\\' 'f' )
|
||||
| ( '\\' 'n' )
|
||||
| ( '\\' 'r' )
|
||||
| ( '\\' 't' )
|
||||
| ( '\\' '\\' )
|
||||
| ( '\\' '0' )
|
||||
| ( '\\' 'x' )
|
||||
| '\u0100'-'\U0010FFFF'
|
||||
;
|
||||
string_literal : '"' {_string_literal_element} '"' ;
|
||||
|
||||
// ================================================================
|
||||
// IMPORT
|
||||
|
||||
<< import "two/src/dsl" >>
|
||||
|
||||
// ================================================================
|
||||
// PARSER
|
||||
|
||||
// ----------------------------------------------------------------
|
||||
Root
|
||||
: StringLiteral
|
||||
<< dsl.NewASTWithErrorReturn($0) >>
|
||||
;
|
||||
|
||||
StringLiteral
|
||||
: string_literal
|
||||
<< dsl.NewASTNodeStripDoubleQuotePair($0, dsl.NodeTypeStringLiteral) >>
|
||||
;
|
||||
|
|
@ -1,81 +0,0 @@
|
|||
#!/bin/bash
|
||||
|
||||
# ----------------------------------------------------------------
|
||||
# Setup
|
||||
us=$(basename $0)
|
||||
set -euo pipefail
|
||||
|
||||
verbose="false"
|
||||
|
||||
if [ $# -eq 1 ]; then
|
||||
bnf="$1"
|
||||
elif [ $# -eq 2 ]; then
|
||||
if [ "$1" = "-v" ]; then
|
||||
verbose="true"
|
||||
else
|
||||
echo "Usage: $0 {.bnf file}" 1>&2
|
||||
exit 1
|
||||
fi
|
||||
bnf="$2"
|
||||
else
|
||||
echo "Usage: $0 {.bnf file}" 1>&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
dir=src/parsing
|
||||
mkdir -p $dir
|
||||
|
||||
# ----------------------------------------------------------------
|
||||
# Run the parser-generator
|
||||
|
||||
# Build the bin/gocc executable (use my fork for performance):
|
||||
# go get github.com/goccmack/gocc
|
||||
go get github.com/johnkerl/gocc
|
||||
bingocc="$GOPATH/bin/gocc"
|
||||
if [ ! -x "$bingocc" ]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
rm -f $dir/*.txt
|
||||
if [ "$verbose" = "true" ]; then
|
||||
lr1="$dir/LR1_conflicts.txt"
|
||||
# The -o specifies the package name within the autogen
|
||||
$bingocc -v -o $dir $bnf || expand -2 $lr1
|
||||
else
|
||||
$bingocc -o $dir $bnf
|
||||
fi
|
||||
|
||||
echo "Parser-autogen OK"
|
||||
|
||||
# Code-gen directories:
|
||||
# $dir/errors/
|
||||
# $dir/lexer/
|
||||
# $dir/parser/
|
||||
# $dir/token/
|
||||
# $dir/util/
|
||||
|
||||
# ----------------------------------------------------------------
|
||||
# Override GOCC codegen with customized error handling
|
||||
cp ../../../go/src/parsing/errors.go.template src/parsing/errors/errors.go
|
||||
sed -i .bak 's:miller/src:two/src:' src/parsing/errors/errors.go
|
||||
|
||||
# ----------------------------------------------------------------
|
||||
# Copy AST files from the main Miller tree
|
||||
|
||||
rm -rf ./src/lib/
|
||||
rm -rf ./src/dsl/
|
||||
|
||||
mkdir -p ./src/lib/
|
||||
mkdir -p ./src/dsl/
|
||||
|
||||
cp ../../../go/src/lib/*.go ./src/lib/
|
||||
cp ../../../go/src/dsl/ast*.go ./src/dsl/
|
||||
|
||||
# Different path to autogen between main Miller tree and here
|
||||
sed -i .bak 's:miller/src:two/src:' src/dsl/ast*go
|
||||
|
||||
# ----------------------------------------------------------------
|
||||
# Compile the main and the parser-autogen
|
||||
|
||||
go build main.go
|
||||
echo "Compile OK"
|
||||
|
|
@ -1,147 +0,0 @@
|
|||
// ================================================================
|
||||
// LEXER
|
||||
|
||||
!whitespace : ' ' | '\t' | '\n' | '\r' ;
|
||||
!comment : '#' {.} '\n' ;
|
||||
|
||||
_letter : 'a'-'z' | 'A'-'Z' ;
|
||||
_decdig : '0'-'9' ;
|
||||
_idchar : _letter | _decdig | '_' ;
|
||||
|
||||
emit : 'e' 'm' 'i' 't' ;
|
||||
|
||||
// ================================================================
|
||||
// IMPORT
|
||||
|
||||
<< import "miller/src/dsl" >>
|
||||
|
||||
// ================================================================
|
||||
// PARSER
|
||||
|
||||
// ----------------------------------------------------------------
|
||||
Root
|
||||
: EmitStatement
|
||||
<< dsl.NewASTWithErrorReturn($0) >>
|
||||
;
|
||||
|
||||
// ----------------------------------------------------------------
|
||||
// Examples:
|
||||
// emit @a
|
||||
// emit (@a, @b)
|
||||
// emit @a, "x", "y"
|
||||
// emit (@a, @b), "x", "y"
|
||||
// First argument (single or in parentheses) must be non-indexed
|
||||
// oosvar/localvar/fieldname, so we can use their names as keys in the emitted
|
||||
// record. These restrictions are enforced in the CST logic, to keep this
|
||||
// parser/AST logic simpler.
|
||||
EmitStatement
|
||||
: emit Rvalue
|
||||
<< dsl.NewASTNodeWithErrorReturn(
|
||||
$0,
|
||||
dsl.NodeTypeEmitStatement,
|
||||
[]interface{}{
|
||||
$1,
|
||||
}
|
||||
) >>
|
||||
|
||||
| emit Rvalue "," EmitArgs
|
||||
<< dsl.NewASTNodeWithErrorReturn(
|
||||
$0,
|
||||
dsl.NodeTypeEmitStatement,
|
||||
[]interface{}{
|
||||
$1,
|
||||
$3,
|
||||
}
|
||||
) >>
|
||||
|
||||
| emit "(" EmitArgs ")"
|
||||
<< dsl.NewASTNodeWithErrorReturn(
|
||||
$0,
|
||||
dsl.NodeTypeEmitStatement,
|
||||
[]interface{}{
|
||||
$2,
|
||||
}
|
||||
) >>
|
||||
|
||||
| emit "(" EmitArgs ")" "," EmitArgs
|
||||
<< dsl.NewASTNodeWithErrorReturn(
|
||||
$0,
|
||||
dsl.NodeTypeEmitStatement,
|
||||
[]interface{}{
|
||||
$2,
|
||||
$5,
|
||||
}
|
||||
) >>
|
||||
;
|
||||
|
||||
// ----------------------------------------------------------------
|
||||
EmitArgs
|
||||
: Rvalue
|
||||
<< dsl.NewASTNodeWithErrorReturn(
|
||||
nil,
|
||||
dsl.NodeTypeFunctionCallsite,
|
||||
[]interface{}{
|
||||
$0,
|
||||
}
|
||||
) >>
|
||||
|
||||
| Rvalue "," EmitArgs
|
||||
<< dsl.WithChildPrepended(
|
||||
$2,
|
||||
$0,
|
||||
) >>
|
||||
;
|
||||
|
||||
// ----------------------------------------------------------------
|
||||
FcnArgs
|
||||
: Rvalue
|
||||
<< dsl.NewASTNodeWithErrorReturn(
|
||||
nil,
|
||||
dsl.NodeTypeFunctionCallsite,
|
||||
[]interface{}{
|
||||
$0,
|
||||
}
|
||||
) >>
|
||||
|
||||
// Allow trailing final comma, especially for multiline statements
|
||||
| Rvalue ","
|
||||
<< dsl.NewASTNodeWithErrorReturn(
|
||||
nil,
|
||||
dsl.NodeTypeFunctionCallsite,
|
||||
[]interface{}{
|
||||
$0,
|
||||
}
|
||||
) >>
|
||||
|
||||
// Allow trailing final comma, especially for multiline statements
|
||||
| Rvalue "," FcnArgs
|
||||
<< dsl.WithChildPrepended(
|
||||
$2,
|
||||
$0,
|
||||
) >>
|
||||
;
|
||||
|
||||
|
||||
// ----------------------------------------------------------------
|
||||
Rvalue
|
||||
: Literal
|
||||
| "(" Literal ")"
|
||||
;
|
||||
|
||||
Literal
|
||||
: "x" << dsl.NewASTNodeWithErrorReturn(
|
||||
$0,
|
||||
dsl.NodeTypeStringLiteral,
|
||||
[]interface{}{},
|
||||
) >>
|
||||
| "y" << dsl.NewASTNodeWithErrorReturn(
|
||||
$0,
|
||||
dsl.NodeTypeStringLiteral,
|
||||
[]interface{}{},
|
||||
) >>
|
||||
| "z" << dsl.NewASTNodeWithErrorReturn(
|
||||
$0,
|
||||
dsl.NodeTypeStringLiteral,
|
||||
[]interface{}{},
|
||||
) >>
|
||||
;
|
||||
|
|
@ -1,176 +0,0 @@
|
|||
// ================================================================
|
||||
// LEXER
|
||||
|
||||
!whitespace : ' ' | '\t' | '\n' | '\r' ;
|
||||
!comment : '#' {.} '\n' ;
|
||||
|
||||
_letter : 'a'-'z' | 'A'-'Z' ;
|
||||
_decdig : '0'-'9' ;
|
||||
_idchar : _letter | _decdig | '_' ;
|
||||
|
||||
emit : 'e' 'm' 'i' 't' ;
|
||||
|
||||
// ================================================================
|
||||
// IMPORT
|
||||
|
||||
<< import "two/src/dsl" >>
|
||||
|
||||
// ================================================================
|
||||
// PARSER
|
||||
|
||||
// ----------------------------------------------------------------
|
||||
Root
|
||||
: EmitStatement
|
||||
<< dsl.NewASTWithErrorReturn($0) >>
|
||||
;
|
||||
|
||||
// ----------------------------------------------------------------
|
||||
// Examples:
|
||||
// emit @a
|
||||
// emit (@a, @b)
|
||||
// emit @a, "x", "y"
|
||||
// emit (@a, @b), "x", "y"
|
||||
// First argument (single or in parentheses) must be non-indexed
|
||||
// oosvar/localvar/fieldname, so we can use their names as keys in the emitted
|
||||
// record. These restrictions are enforced in the CST logic, to keep this
|
||||
// parser/AST logic simpler.
|
||||
|
||||
EmitStatement
|
||||
: emit Emittable
|
||||
<<
|
||||
dsl.NewASTNodeWithErrorReturn(
|
||||
$0,
|
||||
dsl.NodeTypeEmitStatement,
|
||||
[]interface{}{
|
||||
$1,
|
||||
},
|
||||
)
|
||||
>>
|
||||
|
||||
| emit "(" EmittableList ")"
|
||||
<< dsl.NewASTNodeWithErrorReturn(
|
||||
$0,
|
||||
dsl.NodeTypeEmitStatement,
|
||||
[]interface{}{
|
||||
$2,
|
||||
},
|
||||
) >>
|
||||
|
||||
| emit Emittable "," EmitKeys
|
||||
<< dsl.NewASTNodeWithErrorReturn(
|
||||
$0,
|
||||
dsl.NodeTypeEmitStatement,
|
||||
[]interface{}{
|
||||
$1,
|
||||
$3,
|
||||
},
|
||||
) >>
|
||||
|
||||
| emit "(" EmittableList ")" "," EmitKeys
|
||||
<< dsl.NewASTNodeWithErrorReturn(
|
||||
$0,
|
||||
dsl.NodeTypeEmitStatement,
|
||||
[]interface{}{
|
||||
$2,
|
||||
$5,
|
||||
},
|
||||
) >>
|
||||
;
|
||||
|
||||
// - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
|
||||
EmittableList
|
||||
: Emittable
|
||||
<< dsl.NewASTNodeWithErrorReturn(
|
||||
nil,
|
||||
dsl.NodeTypeEmittableList,
|
||||
[]interface{}{
|
||||
$0,
|
||||
}
|
||||
) >>
|
||||
|
||||
// Allow trailing final comma, especially for multiline statements
|
||||
| Emittable "," EmittableList
|
||||
<< dsl.WithChildPrepended(
|
||||
$2,
|
||||
$0,
|
||||
) >>
|
||||
;
|
||||
|
||||
Emittable
|
||||
: Literal
|
||||
;
|
||||
|
||||
// - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
|
||||
EmitKeys
|
||||
|
||||
: Rvalue
|
||||
<< dsl.NewASTNodeWithErrorReturn(
|
||||
nil,
|
||||
dsl.NodeTypeEmitKeys,
|
||||
[]interface{}{
|
||||
$0,
|
||||
},
|
||||
) >>
|
||||
|
||||
| Rvalue "," EmitKeys
|
||||
<< dsl.WithChildPrepended(
|
||||
$2,
|
||||
$0,
|
||||
) >>
|
||||
;
|
||||
|
||||
|
||||
// ----------------------------------------------------------------
|
||||
Rvalue
|
||||
: Literal
|
||||
<< dsl.NewASTNodeWithErrorReturn(
|
||||
nil,
|
||||
dsl.NodeTypeStringLiteral,
|
||||
[]interface{}{
|
||||
$0,
|
||||
},
|
||||
) >>
|
||||
| "(" Literal ")"
|
||||
<< dsl.NewASTNodeWithErrorReturn(
|
||||
nil,
|
||||
dsl.NodeTypeStringLiteral,
|
||||
[]interface{}{
|
||||
$1,
|
||||
},
|
||||
) >>
|
||||
| "[" Literal "]"
|
||||
<< dsl.NewASTNodeWithErrorReturn(
|
||||
nil,
|
||||
dsl.NodeTypeStringLiteral,
|
||||
[]interface{}{
|
||||
$1,
|
||||
},
|
||||
)) >>
|
||||
| "[" Literal "," Literal "]"
|
||||
<< dsl.NewASTNodeWithErrorReturn(
|
||||
nil,
|
||||
dsl.NodeTypeStringLiteral,
|
||||
[]interface{}{
|
||||
$1,
|
||||
$2,
|
||||
},
|
||||
)) >>
|
||||
;
|
||||
|
||||
Literal
|
||||
: "x" << dsl.NewASTNodeWithErrorReturn(
|
||||
$0,
|
||||
dsl.NodeTypeStringLiteral,
|
||||
[]interface{}{},
|
||||
) >>
|
||||
| "y" << dsl.NewASTNodeWithErrorReturn(
|
||||
$0,
|
||||
dsl.NodeTypeStringLiteral,
|
||||
[]interface{}{},
|
||||
) >>
|
||||
| "z" << dsl.NewASTNodeWithErrorReturn(
|
||||
$0,
|
||||
dsl.NodeTypeStringLiteral,
|
||||
[]interface{}{},
|
||||
)) >>
|
||||
;
|
||||
|
|
@ -1,5 +0,0 @@
|
|||
module two
|
||||
|
||||
go 1.24
|
||||
|
||||
toolchain go1.24.5
|
||||
|
|
@ -1,123 +0,0 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
|
||||
"two/src/dsl"
|
||||
|
||||
"two/src/parsing/lexer"
|
||||
"two/src/parsing/parser"
|
||||
)
|
||||
|
||||
const GREEN = "\033[32;01m"
|
||||
const RED = "\033[31;01m"
|
||||
const TEXTDEFAULT = "\033[0m"
|
||||
|
||||
func parseOne(input string, printError bool) bool {
|
||||
theLexer := lexer.NewLexer([]byte(input))
|
||||
theParser := parser.NewParser()
|
||||
iast, err := theParser.Parse(theLexer)
|
||||
if err == nil {
|
||||
fmt.Printf("%sOK%s %s\n", GREEN, TEXTDEFAULT, input)
|
||||
iast.(*dsl.AST).Print()
|
||||
fmt.Println()
|
||||
return true
|
||||
}
|
||||
if printError {
|
||||
fmt.Println(err)
|
||||
}
|
||||
fmt.Printf("%sFail%s %s\n", RED, TEXTDEFAULT, input)
|
||||
fmt.Println()
|
||||
return false
|
||||
}
|
||||
|
||||
func main() {
|
||||
printError := false
|
||||
args := os.Args[1:] // os.Args[0] is program name in Go
|
||||
if len(args) >= 1 && args[0] == "-v" {
|
||||
printError = true
|
||||
args = args[1:]
|
||||
}
|
||||
|
||||
if len(args) == 0 {
|
||||
ok := true
|
||||
|
||||
fmt.Println("----------------------------------------------------------------")
|
||||
fmt.Println("EXPECT OK")
|
||||
goods := []string{
|
||||
"",
|
||||
";",
|
||||
";;",
|
||||
";;;",
|
||||
"x",
|
||||
"x;x",
|
||||
"x;x;x",
|
||||
"x;x;x;x",
|
||||
"x;",
|
||||
"x;;",
|
||||
";x",
|
||||
";;x",
|
||||
"x ; {}",
|
||||
"{} ; x",
|
||||
"{} x",
|
||||
"{ x }",
|
||||
"{ x; x }",
|
||||
"x; { x; x }",
|
||||
"{ x; x } x",
|
||||
"{ x; x } ; x",
|
||||
"{};{}",
|
||||
"{} {}",
|
||||
"{} {};",
|
||||
"{};{};",
|
||||
"{} {} {}",
|
||||
"{};{}; {}",
|
||||
"{} ; ; {}",
|
||||
"x; x;",
|
||||
"{x; x;}",
|
||||
"{x; x;} x",
|
||||
"{x; x;} x;",
|
||||
"{x; x;} {};",
|
||||
"{} x; {}",
|
||||
"{} x; {};",
|
||||
"{} x; x; {}",
|
||||
"{} x; x; {};",
|
||||
"{} x; x; x; {}",
|
||||
"{} x; x; x; {};",
|
||||
"{} {} ;;; {;;} x; x; x; x; x; {}",
|
||||
}
|
||||
for _, input := range goods {
|
||||
if parseOne(input, printError) == false {
|
||||
ok = false
|
||||
}
|
||||
}
|
||||
|
||||
fmt.Println()
|
||||
fmt.Println("----------------------------------------------------------------")
|
||||
fmt.Println("EXPECT FAIL")
|
||||
bads := []string{
|
||||
"x x",
|
||||
"x {}",
|
||||
}
|
||||
for _, input := range bads {
|
||||
if parseOne(input, printError) == true {
|
||||
ok = false
|
||||
}
|
||||
}
|
||||
|
||||
fmt.Println()
|
||||
fmt.Println("----------------------------------------------------------------")
|
||||
if ok {
|
||||
fmt.Printf("%sALL AS EXPECTED%s\n", GREEN, TEXTDEFAULT)
|
||||
} else {
|
||||
fmt.Printf("%sNOT ALL AS EXPECTED%s\n", RED, TEXTDEFAULT)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
} else {
|
||||
for _, arg := range args {
|
||||
parseOne(arg, printError)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -1,15 +0,0 @@
|
|||
#!/bin/bash
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# ~/.ctags needs to look like
|
||||
# --langdef=Go
|
||||
# --langmap=Go:.go
|
||||
# --regex-Go=/func([ \t]+\([^)]+\))?[ \t]+([a-zA-Z0-9_]+)/\2/d,func/
|
||||
# --regex-Go=/var[ \t]+([a-zA-Z_][a-zA-Z0-9_]+)/\1/d,var/
|
||||
# --regex-Go=/type[ \t]+([a-zA-Z_][a-zA-Z0-9_]+)/\1/d,type/
|
||||
#
|
||||
# See also https://stackoverflow.com/questions/8204367/ctag-database-for-go
|
||||
|
||||
ctags -f gosource.tags -R `pwd`
|
||||
mv gosource.tags tags
|
||||
|
|
@ -1,9 +0,0 @@
|
|||
#!/bin/bash
|
||||
|
||||
if [ $# -lt 1 ]; then
|
||||
echo "Usage: $0 {.bnf file} [expressions]" 1>&2
|
||||
exit 1
|
||||
fi
|
||||
bnf="$1"
|
||||
shift
|
||||
./build $bnf && echo && main "$@"
|
||||
|
|
@ -1,127 +0,0 @@
|
|||
// ================================================================
|
||||
// LEXER
|
||||
|
||||
!whitespace : ' ' | '\t' | '\n' | '\r' ;
|
||||
|
||||
// ================================================================
|
||||
// IMPORT
|
||||
|
||||
<< import "two/src/dsl" >>
|
||||
|
||||
// ================================================================
|
||||
// PARSER
|
||||
|
||||
// ----------------------------------------------------------------
|
||||
Root
|
||||
: StatementBlock
|
||||
<< dsl.NewASTWithErrorReturn($0) >>
|
||||
;
|
||||
|
||||
// ================================================================
|
||||
StatementBlock
|
||||
|
||||
// Empty statement. This allows for 'mlr put ""', as well as repeated semicolons.
|
||||
: empty
|
||||
<< dsl.NewASTNodeWithErrorReturn(
|
||||
nil,
|
||||
dsl.NodeTypeStatementBlock,
|
||||
[]interface{}{},
|
||||
) >>
|
||||
|
||||
| NonEmptyStatementBlock
|
||||
<< dsl.WithErrorReturn($0) >>
|
||||
;
|
||||
|
||||
// ----------------------------------------------------------------
|
||||
NonEmptyStatementBlock
|
||||
// ---------------------- Terminal rules
|
||||
|
||||
// Things not ending in a curly brace, like assignments -- and also do-while.
|
||||
: BracelessStatement
|
||||
<< dsl.NewASTNodeWithErrorReturn(
|
||||
nil,
|
||||
dsl.NodeTypeStatementBlock,
|
||||
[]interface{}{
|
||||
$0,
|
||||
},
|
||||
) >>
|
||||
|
||||
// Things ending in a curly brace, like for/do/while, begin/end, and pattern-action blocks
|
||||
| BracefulStatement
|
||||
<< dsl.NewASTNodeWithErrorReturn(
|
||||
nil,
|
||||
dsl.NodeTypeStatementBlock,
|
||||
[]interface{}{
|
||||
$0,
|
||||
},
|
||||
) >>
|
||||
|
||||
// ---------------------- Recursive rules
|
||||
|
||||
// So statements can start with a semicolon
|
||||
| ";" StatementBlock
|
||||
<< dsl.NewASTNodeWithErrorReturn(
|
||||
|
||||
// Normal case for sequential statements like '$x=1; $y=2'
|
||||
| BracelessStatement ";" StatementBlock
|
||||
<<dsl.WithChildPrepended($2, $0) >>
|
||||
|
||||
// For 'begin {...} ; $x=1'
|
||||
| BracefulStatement ";" StatementBlock
|
||||
<<dsl.WithChildPrepended($2, $0) >>
|
||||
|
||||
// These are for things like 'begin {...} begin {...} ...' -- where people
|
||||
// shouldn't have to put semicolons after the closing curly braces.
|
||||
//
|
||||
// We get LR-1 conflicts with the following, so we need a pair of more
|
||||
// explicit lookahead-by-more production rules instead. (By using ternaries
|
||||
// we are effectively getting lookahead-by-two.)
|
||||
//
|
||||
// | BracefulStatement StatementBlock
|
||||
// <<dsl.WithChildPrepended($1, $0) >>
|
||||
|
||||
// E.g. 'begin {...} begin {...} $x=1'
|
||||
| BracefulStatement BracefulStatement StatementBlock
|
||||
<<dsl.WithTwoChildrenPreprended($2, $0, $1) >>
|
||||
|
||||
// E.g. 'begin {...} $x=1'
|
||||
| BracefulStatement BracelessStatement
|
||||
<< dsl.NewASTNodeWithErrorReturn(
|
||||
nil,
|
||||
dsl.NodeTypeStatementBlock,
|
||||
[]interface{}{
|
||||
$0,
|
||||
$1,
|
||||
},
|
||||
) >>
|
||||
|
||||
// E.g. 'begin {...} $x=1 ;'
|
||||
| BracefulStatement BracelessStatement ";"
|
||||
<< dsl.NewASTNodeWithErrorReturn(
|
||||
nil,
|
||||
dsl.NodeTypeStatementBlock,
|
||||
[]interface{}{
|
||||
$0,
|
||||
$1,
|
||||
},
|
||||
) >>
|
||||
|
||||
| BracefulStatement BracelessStatement ";" NonEmptyStatementBlock
|
||||
<<dsl.WithTwoChildrenPreprended($3, $0, $1) >>
|
||||
;
|
||||
|
||||
// ----------------------------------------------------------------
|
||||
BracelessStatement
|
||||
: "x"
|
||||
<< dsl.NewASTNodeWithErrorReturn(
|
||||
$0,
|
||||
dsl.NodeTypeBareBoolean,
|
||||
[]interface{}{},
|
||||
) >>
|
||||
;
|
||||
|
||||
// ----------------------------------------------------------------
|
||||
BracefulStatement
|
||||
: "{" StatementBlock "}"
|
||||
<< dsl.NewASTNodeWithErrorReturn(
|
||||
;
|
||||
|
|
@ -1,302 +0,0 @@
|
|||
// ================================================================
|
||||
// LEXER
|
||||
|
||||
!whitespace : ' ' | '\t' | '\n' | '\r' ;
|
||||
!comment : '#' {.} '\n' ;
|
||||
|
||||
_letter : 'a'-'z' | 'A'-'Z' ;
|
||||
_decdig : '0'-'9' ;
|
||||
_idchar : _letter | _decdig | '_' ;
|
||||
|
||||
emitf : 'e' 'm' 'i' 't' 'f';
|
||||
emit : 'e' 'm' 'i' 't' ;
|
||||
|
||||
stdout : 's' 't' 'd' 'o' 'u' 't' ;
|
||||
stderr : 's' 't' 'd' 'e' 'r' 'r' ;
|
||||
|
||||
// ================================================================
|
||||
// IMPORT
|
||||
|
||||
<< import "two/src/dsl" >>
|
||||
|
||||
// ================================================================
|
||||
// PARSER
|
||||
|
||||
// ----------------------------------------------------------------
|
||||
Root
|
||||
: EmitFStatement
|
||||
<< dsl.NewASTWithErrorReturn($0) >>
|
||||
| EmitStatement
|
||||
<< dsl.NewASTWithErrorReturn($0) >>
|
||||
;
|
||||
|
||||
// ----------------------------------------------------------------
|
||||
Redirector
|
||||
: ">" RedirectTarget
|
||||
<< dsl.NewASTNodeWithErrorReturn(
|
||||
$0,
|
||||
dsl.NodeTypeRedirectWrite,
|
||||
[]interface{}{
|
||||
$1,
|
||||
}
|
||||
) >>
|
||||
| ">>" RedirectTarget
|
||||
<< dsl.NewASTNodeWithErrorReturn(
|
||||
$0,
|
||||
dsl.NodeTypeRedirectAppend,
|
||||
[]interface{}{
|
||||
$1,
|
||||
}
|
||||
) >>
|
||||
| "|" RedirectTarget
|
||||
<< dsl.NewASTNodeWithErrorReturn(
|
||||
$0,
|
||||
dsl.NodeTypeRedirectPipe,
|
||||
[]interface{}{
|
||||
$1,
|
||||
}
|
||||
) >>
|
||||
;
|
||||
|
||||
RedirectTarget
|
||||
: stdout
|
||||
<< dsl.NewASTNodeWithErrorReturn(
|
||||
$0,
|
||||
dsl.NodeTypeRedirectTargetStdout,
|
||||
[]interface{}{}
|
||||
) >>
|
||||
| stderr
|
||||
<< dsl.NewASTNodeWithErrorReturn(
|
||||
$0,
|
||||
dsl.NodeTypeRedirectTargetStderr,
|
||||
[]interface{}{}
|
||||
) >>
|
||||
| Rvalue
|
||||
;
|
||||
|
||||
// ----------------------------------------------------------------
|
||||
EmitStatement
|
||||
: emit Emittable
|
||||
<<
|
||||
dsl.NewASTNodeWithErrorReturn(
|
||||
$0,
|
||||
dsl.NodeTypeEmitStatement,
|
||||
[]interface{}{
|
||||
$1,
|
||||
dsl.NewASTNodeTerminal(nil, dsl.NodeTypeNoOp),
|
||||
dsl.NewASTNodeTerminal(nil, dsl.NodeTypeNoOp),
|
||||
}
|
||||
))
|
||||
>>
|
||||
|
||||
| emit Redirector "," Emittable
|
||||
<<
|
||||
dsl.NewASTNodeWithErrorReturn(
|
||||
$0,
|
||||
dsl.NodeTypeEmitStatement,
|
||||
[]interface{}{
|
||||
$3,
|
||||
dsl.NewASTNodeTerminal(nil, dsl.NodeTypeNoOp),
|
||||
$1,
|
||||
}
|
||||
))
|
||||
>>
|
||||
|
||||
| emit "(" EmittableList ")"
|
||||
<<
|
||||
dsl.NewASTNodeWithErrorReturn(
|
||||
$0,
|
||||
dsl.NodeTypeEmitStatement,
|
||||
[]interface{}{
|
||||
$2,
|
||||
dsl.NewASTNodeTerminal(nil, dsl.NodeTypeNoOp),
|
||||
dsl.NewASTNodeTerminal(nil, dsl.NodeTypeNoOp),
|
||||
}
|
||||
))
|
||||
>>
|
||||
|
||||
| emit Redirector "," "(" EmittableList ")"
|
||||
<<
|
||||
dsl.NewASTNodeWithErrorReturn(
|
||||
$0,
|
||||
dsl.NodeTypeEmitStatement,
|
||||
[]interface{}{
|
||||
$4,
|
||||
dsl.NewASTNodeTerminal(nil, dsl.NodeTypeNoOp),
|
||||
$1,
|
||||
}
|
||||
))
|
||||
>>
|
||||
|
||||
| emit Emittable "," EmitKeys
|
||||
<<
|
||||
dsl.NewASTNodeWithErrorReturn(
|
||||
$0,
|
||||
dsl.NodeTypeEmitStatement,
|
||||
[]interface{}{
|
||||
$1,
|
||||
$3,
|
||||
dsl.NewASTNodeTerminal(nil, dsl.NodeTypeNoOp),
|
||||
}
|
||||
))
|
||||
>>
|
||||
|
||||
| emit Redirector "," Emittable "," EmitKeys
|
||||
<<
|
||||
dsl.NewASTNodeWithErrorReturn(
|
||||
$0,
|
||||
dsl.NodeTypeEmitStatement,
|
||||
[]interface{}{
|
||||
$3,
|
||||
$5,
|
||||
$1,
|
||||
}
|
||||
))
|
||||
>>
|
||||
|
||||
| emit "(" EmittableList ")" "," EmitKeys
|
||||
<<
|
||||
dsl.NewASTNodeWithErrorReturn(
|
||||
$0,
|
||||
dsl.NodeTypeEmitStatement,
|
||||
[]interface{}{
|
||||
$2,
|
||||
$5,
|
||||
dsl.NewASTNodeTerminal(nil, dsl.NodeTypeNoOp),
|
||||
}
|
||||
))
|
||||
>>
|
||||
|
||||
| emit Redirector "," "(" EmittableList ")" "," EmitKeys
|
||||
<<
|
||||
dsl.NewASTNodeWithErrorReturn(
|
||||
$0,
|
||||
dsl.NodeTypeEmitStatement,
|
||||
[]interface{}{
|
||||
$4,
|
||||
$7,
|
||||
$1,
|
||||
}
|
||||
))
|
||||
>>
|
||||
;
|
||||
|
||||
// ----------------------------------------------------------------
|
||||
// Examples:
|
||||
// emitf @a
|
||||
// emitf @a, b, $c
|
||||
// Each argument must be a non-indexed oosvar/localvar/fieldname, so we can use
|
||||
// their names as keys in the emitted record.
|
||||
EmitFStatement
|
||||
|
||||
: emitf EmittableList
|
||||
<< dsl.WithChildrenAdopted(
|
||||
dsl.NewASTNodeTerminal(
|
||||
$0,
|
||||
dsl.NodeTypeEmitFStatement,
|
||||
),
|
||||
$1,
|
||||
) >>
|
||||
|
||||
| emitf Redirector "," EmittableList
|
||||
// TODO
|
||||
|
||||
;
|
||||
|
||||
// ----------------------------------------------------------------
|
||||
EmittableList
|
||||
|
||||
: Emittable
|
||||
<< dsl.NewASTNodeWithErrorReturn(
|
||||
nil,
|
||||
dsl.NodeTypeEmittableList,
|
||||
[]interface{}{
|
||||
$0,
|
||||
}
|
||||
) >>
|
||||
|
||||
// Allow trailing final comma, especially for multiline statements
|
||||
| Emittable "," EmittableList
|
||||
<< dsl.WithChildPrepended(
|
||||
$2,
|
||||
$0,
|
||||
) >>
|
||||
;
|
||||
|
||||
Emittable
|
||||
: Literal
|
||||
;
|
||||
|
||||
// ----------------------------------------------------------------
|
||||
EmitKeys
|
||||
|
||||
: Rvalue
|
||||
<< dsl.NewASTNodeWithErrorReturn(
|
||||
nil,
|
||||
dsl.NodeTypeEmitKeys,
|
||||
[]interface{}{
|
||||
$0,
|
||||
}
|
||||
) >>
|
||||
|
||||
| Rvalue "," EmitKeys
|
||||
<< dsl.WithChildPrepended(
|
||||
$2,
|
||||
$0,
|
||||
) >>
|
||||
;
|
||||
|
||||
// ----------------------------------------------------------------
|
||||
Rvalue
|
||||
: Literal
|
||||
<< dsl.NewASTNodeWithErrorReturn(
|
||||
nil,
|
||||
dsl.NodeTypeStringLiteral,
|
||||
[]interface{}{
|
||||
$0,
|
||||
}
|
||||
) >>
|
||||
| "(" Literal ")"
|
||||
<< dsl.NewASTNodeWithErrorReturn(
|
||||
nil,
|
||||
dsl.NodeTypeStringLiteral,
|
||||
[]interface{}{
|
||||
$1,
|
||||
}
|
||||
) >>
|
||||
| "[" Literal "]"
|
||||
<< dsl.NewASTNodeWithErrorReturn(
|
||||
nil,
|
||||
dsl.NodeTypeStringLiteral,
|
||||
[]interface{}{
|
||||
$1,
|
||||
}
|
||||
) >>
|
||||
| "[" Literal "," Literal "]"
|
||||
<< dsl.NewASTNodeWithErrorReturn(
|
||||
nil,
|
||||
dsl.NodeTypeStringLiteral,
|
||||
[]interface{}{
|
||||
$1,
|
||||
$2,
|
||||
}
|
||||
) >>
|
||||
;
|
||||
|
||||
Literal
|
||||
: "x" << dsl.NewASTNodeWithErrorReturn(
|
||||
$0,
|
||||
dsl.NodeTypeStringLiteral,
|
||||
[]interface{}{}
|
||||
) >>
|
||||
| "y" << dsl.NewASTNodeWithErrorReturn(
|
||||
$0,
|
||||
dsl.NodeTypeStringLiteral,
|
||||
[]interface{}{}
|
||||
) >>
|
||||
| "z" << dsl.NewASTNodeWithErrorReturn(
|
||||
$0,
|
||||
dsl.NodeTypeStringLiteral,
|
||||
[]interface{}{}
|
||||
) >>
|
||||
;
|
||||
|
|
@ -3820,5 +3820,5 @@ This is simply a copy of what you should see on running `man mlr` at a command p
|
|||
MIME Type for Comma-Separated Values (CSV) Files, the Miller docsite
|
||||
https://miller.readthedocs.io
|
||||
|
||||
2026-03-06 4mMILLER24m(1)
|
||||
2026-03-16 4mMILLER24m(1)
|
||||
</pre>
|
||||
|
|
|
|||
|
|
@ -3799,4 +3799,4 @@
|
|||
MIME Type for Comma-Separated Values (CSV) Files, the Miller docsite
|
||||
https://miller.readthedocs.io
|
||||
|
||||
2026-03-06 4mMILLER24m(1)
|
||||
2026-03-16 4mMILLER24m(1)
|
||||
|
|
|
|||
|
|
@ -64,12 +64,7 @@ error.
|
|||
</pre>
|
||||
<pre class="pre-non-highlight-in-pair">
|
||||
mlr: cannot parse DSL expression.
|
||||
mlr: Parse error on token "$y" at line 6 column 3.
|
||||
Please check for missing semicolon.
|
||||
Expected one of:
|
||||
␚ ; > >> | ? || ^^ && =~ !=~ == != <=> >= < <= ^ & << >>> + - .+ .- *
|
||||
/ // % .* ./ .// . ?? ??? ** [
|
||||
|
||||
mlr: parse error: unexpected field_name ("$y")
|
||||
</pre>
|
||||
|
||||
## elif
|
||||
|
|
|
|||
|
|
@ -776,12 +776,7 @@ As of September 2021, immediately invoked function expressions (IIFEs) are not p
|
|||
</pre>
|
||||
<pre class="pre-non-highlight-in-pair">
|
||||
mlr: cannot parse DSL expression.
|
||||
mlr: Parse error on token "(" at line 4 column 35.
|
||||
Please check for missing semicolon.
|
||||
Expected one of:
|
||||
; } > >> | ? || ^^ && =~ !=~ == != <=> >= < <= ^ & << >>> + - .+ .- * /
|
||||
// % .* ./ .// . ?? ??? **
|
||||
|
||||
mlr: parse error: unexpected lparen ("(")
|
||||
</pre>
|
||||
|
||||
but this does:
|
||||
|
|
|
|||
|
|
@ -457,7 +457,7 @@ to files.
|
|||
Keep in mind that out-of-stream variables are a nested, multi-level [map](reference-main-maps.md) (directly viewable as JSON using `dump`), while Miller record values are as well during processing -- but records may be flattened down for output to tabular formats. See the page [Flatten/unflatten: JSON vs. tabular formats](flatten-unflatten.md) for more information.
|
||||
|
||||
* You can use `emit1` to emit any map-valued expression, including `$*`, map-valued out-of-stream variables, the entire out-of-stream-variable collection `@*`, map-valued local variables, map literals, or map-valued function return values.
|
||||
* For `emit`, `emitp`, and `emitf`, you can emit map-valued local variables, map-valued field attributes (with `$`), map-va out-of-stream variables (with `@`), `$*`, `@*`, or map literals (with outermost `{...}`) -- but not arbitrary expressions which evaluate to map (such as function return values).
|
||||
* For `emit`, `emitp`, and `emitf`, you can emit map-valued local variables, map-valued field attributes (with `$`), map-valued out-of-stream variables (with `@`), `$*`, `@*`, map literals (with outermost `{...}`), or map-valued function return values (e.g. `emit f($a, $b)`).
|
||||
|
||||
The reason for this is partly historical and partly technical. As we'll see below, you can do lots of syntactical things with `emit`, `emitp`, and `emitf`, including printing them side-by-side, indexing them, redirecting the output to files, etc. What this means syntactically is that Miller's parser needs to handle all sorts of commas, parentheses, and so on:
|
||||
|
||||
|
|
|
|||
|
|
@ -209,7 +209,7 @@ to files.
|
|||
Keep in mind that out-of-stream variables are a nested, multi-level [map](reference-main-maps.md) (directly viewable as JSON using `dump`), while Miller record values are as well during processing -- but records may be flattened down for output to tabular formats. See the page [Flatten/unflatten: JSON vs. tabular formats](flatten-unflatten.md) for more information.
|
||||
|
||||
* You can use `emit1` to emit any map-valued expression, including `$*`, map-valued out-of-stream variables, the entire out-of-stream-variable collection `@*`, map-valued local variables, map literals, or map-valued function return values.
|
||||
* For `emit`, `emitp`, and `emitf`, you can emit map-valued local variables, map-valued field attributes (with `$`), map-va out-of-stream variables (with `@`), `$*`, `@*`, or map literals (with outermost `{...}`) -- but not arbitrary expressions which evaluate to map (such as function return values).
|
||||
* For `emit`, `emitp`, and `emitf`, you can emit map-valued local variables, map-valued field attributes (with `$`), map-valued out-of-stream variables (with `@`), `$*`, `@*`, map literals (with outermost `{...}`), or map-valued function return values (e.g. `emit f($a, $b)`).
|
||||
|
||||
The reason for this is partly historical and partly technical. As we'll see below, you can do lots of syntactical things with `emit`, `emitp`, and `emitf`, including printing them side-by-side, indexing them, redirecting the output to files, etc. What this means syntactically is that Miller's parser needs to handle all sorts of commas, parentheses, and so on:
|
||||
|
||||
|
|
|
|||
|
|
@ -383,7 +383,8 @@ ASCII character classes:
|
|||
[[:word:]] word characters (== [0-9A-Za-z_])
|
||||
[[:xdigit:]] hex digit (== [0-9A-Fa-f])
|
||||
|
||||
Unicode character classes are those in unicode.Categories and unicode.Scripts.
|
||||
Unicode character classes are those in unicode.Categories,
|
||||
unicode.CategoryAliases, and unicode.Scripts.
|
||||
|
||||
func IsWordChar(r rune) bool
|
||||
type EmptyOp uint8
|
||||
|
|
|
|||
1
go.mod
1
go.mod
|
|
@ -19,6 +19,7 @@ go 1.25.0
|
|||
require (
|
||||
github.com/facette/natsort v0.0.0-20181210072756-2cd4dd1e2dcb
|
||||
github.com/johnkerl/lumin v1.0.0
|
||||
github.com/johnkerl/pgpg/go v1.0.0
|
||||
github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51
|
||||
github.com/klauspost/compress v1.18.4
|
||||
github.com/kshedden/statmodel v0.0.0-20210519035403-ee97d3e48df1
|
||||
|
|
|
|||
2
go.sum
2
go.sum
|
|
@ -15,6 +15,8 @@ github.com/google/pprof v0.0.0-20211214055906-6f57359322fd/go.mod h1:KgnwoLYCZ8I
|
|||
github.com/ianlancetaylor/demangle v0.0.0-20210905161508-09a460cdf81d/go.mod h1:aYm2/VgdVmcIU8iMfdMvDMsRAQjcfZSKFby6HOFvi/w=
|
||||
github.com/johnkerl/lumin v1.0.0 h1:CV34cHZOJ92Y02RbQ0rd4gA0C06Qck9q8blOyaPoWpU=
|
||||
github.com/johnkerl/lumin v1.0.0/go.mod h1:eLf5AdQOaLvzZ2zVy4REr/DSeEwG+CZreHwNLICqv9E=
|
||||
github.com/johnkerl/pgpg/go v1.0.0 h1:JG3Xiu9uatZmouiKvqZV2Ql6QmqV9Vw0t/GD/Dn+oj8=
|
||||
github.com/johnkerl/pgpg/go v1.0.0/go.mod h1:Jh0kkya72bexXymdoZwWJPu6bEOTPLDa5V+gBkKVWL0=
|
||||
github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 h1:Z9n2FFNUXsshfwJMBgNA0RU6/i7WVaAegv3PtuIHPMs=
|
||||
github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51/go.mod h1:CzGEWj7cYgsdH8dAjBGEr58BoE7ScuLd+fwFZ44+/x8=
|
||||
github.com/klauspost/compress v1.18.4 h1:RPhnKRAQ4Fh8zU2FY/6ZFDwTVTxgJ/EMydqSTzE9a2c=
|
||||
|
|
|
|||
|
|
@ -3799,4 +3799,4 @@
|
|||
MIME Type for Comma-Separated Values (CSV) Files, the Miller docsite
|
||||
https://miller.readthedocs.io
|
||||
|
||||
2026-03-06 4mMILLER24m(1)
|
||||
2026-03-16 4mMILLER24m(1)
|
||||
|
|
|
|||
|
|
@ -2,12 +2,12 @@
|
|||
.\" Title: mlr
|
||||
.\" Author: [see the "AUTHOR" section]
|
||||
.\" Generator: ./mkman.rb
|
||||
.\" Date: 2026-03-06
|
||||
.\" Date: 2026-03-16
|
||||
.\" Manual: \ \&
|
||||
.\" Source: \ \&
|
||||
.\" Language: English
|
||||
.\"
|
||||
.TH "MILLER" "1" "2026-03-06" "\ \&" "\ \&"
|
||||
.TH "MILLER" "1" "2026-03-16" "\ \&" "\ \&"
|
||||
.\" -----------------------------------------------------------------
|
||||
.\" * Portability definitions
|
||||
.\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ Parsing a Miller DSL (domain-specific language) expression goes through three re
|
|||
* Abstract syntax tree (AST)
|
||||
* Concrete syntax tree (AST)
|
||||
|
||||
The job of the GOCC parser is to turn the DSL string into an AST.
|
||||
The job of the PGPG parser is to turn the DSL string into an AST.
|
||||
|
||||
The job of the CST builder is to turn the AST into a CST.
|
||||
|
||||
|
|
|
|||
|
|
@ -1,198 +0,0 @@
|
|||
// AST-build methods, for use by callbacks within the GOCC/BNF Miller
|
||||
// DSL grammar in mlr.bnf.
|
||||
|
||||
package dsl
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
"github.com/johnkerl/miller/v6/pkg/lib"
|
||||
"github.com/johnkerl/miller/v6/pkg/parsing/token"
|
||||
)
|
||||
|
||||
// This is for the GOCC/BNF parser, which produces an AST
|
||||
func NewASTWithErrorReturn(iroot interface{}) (*AST, error) {
|
||||
return &AST{
|
||||
RootNode: iroot.(*ASTNode),
|
||||
}, nil
|
||||
}
|
||||
|
||||
func NewASTNodeTerminal(itok interface{}, nodeType TNodeType) *ASTNode {
|
||||
var tok *token.Token = nil
|
||||
if itok != nil {
|
||||
tok = itok.(*token.Token)
|
||||
}
|
||||
return &ASTNode{
|
||||
Token: tok,
|
||||
Type: nodeType,
|
||||
Children: nil,
|
||||
}
|
||||
}
|
||||
|
||||
// NewASTNode is the ASTNode constructor. If children is non-nil and length 0, a
|
||||
// zary node is created. (Example: a function call with zero arguments.) If
|
||||
// children is nil, a terminal node is created. (Example: a string or integer
|
||||
// literal.)
|
||||
func NewASTNode(
|
||||
itok interface{},
|
||||
nodeType TNodeType,
|
||||
children []interface{},
|
||||
) *ASTNode {
|
||||
|
||||
var tok *token.Token = nil
|
||||
if itok != nil {
|
||||
tok = itok.(*token.Token)
|
||||
}
|
||||
|
||||
node := &ASTNode{
|
||||
Token: tok,
|
||||
Type: nodeType,
|
||||
Children: nil,
|
||||
}
|
||||
|
||||
if children == nil {
|
||||
return node
|
||||
}
|
||||
|
||||
n := len(children)
|
||||
node.Children = make([]*ASTNode, n)
|
||||
for i, child := range children {
|
||||
node.Children[i] = child.(*ASTNode)
|
||||
}
|
||||
return node
|
||||
}
|
||||
|
||||
// Pass-through expressions in the grammar sometimes need to be turned from
|
||||
// (ASTNode) to (ASTNode, error). This is for GOCC.
|
||||
func WithErrorReturn(iparent interface{}) (*ASTNode, error) {
|
||||
return iparent.(*ASTNode), nil
|
||||
}
|
||||
|
||||
func NewASTNodeWithErrorReturn(
|
||||
itok interface{},
|
||||
nodeType TNodeType,
|
||||
children []interface{},
|
||||
) (*ASTNode, error) {
|
||||
return WithErrorReturn(NewASTNode(itok, nodeType, children))
|
||||
}
|
||||
|
||||
func NewASTNodeTerminalWithErrorReturn(itok interface{}, nodeType TNodeType) (*ASTNode, error) {
|
||||
return WithErrorReturn(NewASTNode(itok, nodeType, nil))
|
||||
}
|
||||
|
||||
// Strips the leading '$' from field names, or '@' from oosvar names. Not done
|
||||
// in the parser itself due to LR-1 conflicts.
|
||||
func NewASTNodeStripDollarOrAtSign(itok interface{}, nodeType TNodeType) (*ASTNode, error) {
|
||||
oldToken := itok.(*token.Token)
|
||||
lib.InternalCodingErrorIf(len(oldToken.Lit) < 2)
|
||||
lib.InternalCodingErrorIf(oldToken.Lit[0] != '$' && oldToken.Lit[0] != '@')
|
||||
newToken := &token.Token{
|
||||
Type: oldToken.Type,
|
||||
Lit: oldToken.Lit[1:],
|
||||
Pos: oldToken.Pos,
|
||||
}
|
||||
return NewASTNodeTerminal(newToken, nodeType), nil
|
||||
}
|
||||
|
||||
// Strips the leading '${' and trailing '}' from braced field names, or '@{'
|
||||
// and '}' from oosvar names. Not done in the parser itself due to LR-1
|
||||
// conflicts.
|
||||
func NewASTNodeStripDollarOrAtSignAndCurlyBraces(
|
||||
itok interface{},
|
||||
nodeType TNodeType,
|
||||
) (*ASTNode, error) {
|
||||
oldToken := itok.(*token.Token)
|
||||
n := len(oldToken.Lit)
|
||||
lib.InternalCodingErrorIf(n < 4)
|
||||
lib.InternalCodingErrorIf(oldToken.Lit[0] != '$' && oldToken.Lit[0] != '@')
|
||||
lib.InternalCodingErrorIf(oldToken.Lit[1] != '{')
|
||||
lib.InternalCodingErrorIf(oldToken.Lit[n-1] != '}')
|
||||
newToken := &token.Token{
|
||||
Type: oldToken.Type,
|
||||
Lit: oldToken.Lit[2 : n-1],
|
||||
Pos: oldToken.Pos,
|
||||
}
|
||||
return NewASTNodeTerminal(newToken, nodeType), nil
|
||||
}
|
||||
|
||||
// Likewise for the leading/trailing double quotes on string literals. Also,
|
||||
// since string literals can have backslash-escaped double-quotes like
|
||||
// "...\"...\"...", we also unbackslash here.
|
||||
func NewASTNodeStripDoubleQuotePair(
|
||||
itok interface{},
|
||||
nodeType TNodeType,
|
||||
) (*ASTNode, error) {
|
||||
oldToken := itok.(*token.Token)
|
||||
n := len(oldToken.Lit)
|
||||
contents := string(oldToken.Lit[1 : n-1])
|
||||
|
||||
newToken := &token.Token{
|
||||
Type: oldToken.Type,
|
||||
Lit: []byte(contents),
|
||||
Pos: oldToken.Pos,
|
||||
}
|
||||
return NewASTNodeTerminal(newToken, nodeType), nil
|
||||
}
|
||||
|
||||
func WithChildPrepended(iparent interface{}, ichild interface{}) (*ASTNode, error) {
|
||||
parent := iparent.(*ASTNode)
|
||||
child := ichild.(*ASTNode)
|
||||
if parent.Children == nil {
|
||||
parent.Children = []*ASTNode{child}
|
||||
} else {
|
||||
parent.Children = append([]*ASTNode{child}, parent.Children...)
|
||||
}
|
||||
return parent, nil
|
||||
}
|
||||
|
||||
func WithTwoChildrenPreprended(iparent interface{}, ichildA, ichildB interface{}) (*ASTNode, error) {
|
||||
parent := iparent.(*ASTNode)
|
||||
childA := ichildA.(*ASTNode)
|
||||
childB := ichildB.(*ASTNode)
|
||||
if parent.Children == nil {
|
||||
parent.Children = []*ASTNode{childA, childB}
|
||||
} else {
|
||||
parent.Children = append([]*ASTNode{childA, childB}, parent.Children...)
|
||||
}
|
||||
return parent, nil
|
||||
}
|
||||
|
||||
func WithChildAppended(iparent interface{}, child interface{}) (*ASTNode, error) {
|
||||
parent := iparent.(*ASTNode)
|
||||
if parent.Children == nil {
|
||||
parent.Children = []*ASTNode{child.(*ASTNode)}
|
||||
} else {
|
||||
parent.Children = append(parent.Children, child.(*ASTNode))
|
||||
}
|
||||
return parent, nil
|
||||
}
|
||||
|
||||
func WithChildrenAdopted(iparent interface{}, ichild interface{}) (*ASTNode, error) {
|
||||
parent := iparent.(*ASTNode)
|
||||
child := ichild.(*ASTNode)
|
||||
parent.Children = child.Children
|
||||
child.Children = nil
|
||||
return parent, nil
|
||||
}
|
||||
|
||||
func (node *ASTNode) CheckArity(
|
||||
arity int,
|
||||
) error {
|
||||
if len(node.Children) != arity {
|
||||
return fmt.Errorf("expected AST node arity %d, got %d", arity, len(node.Children))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Tokens are produced by GOCC. However there is an exception: for the ternary
|
||||
// operator I want the AST to have a "?:" token, which GOCC doesn't produce
|
||||
// since nothing is actually spelled like that in the DSL.
|
||||
func NewASTToken(iliteral interface{}, iclonee interface{}) *token.Token {
|
||||
literal := iliteral.(string)
|
||||
clonee := iclonee.(*token.Token)
|
||||
return &token.Token{
|
||||
Type: clonee.Type,
|
||||
Lit: []byte(literal),
|
||||
Pos: clonee.Pos,
|
||||
}
|
||||
}
|
||||
|
|
@ -1,312 +0,0 @@
|
|||
// Print routines for AST and ASTNode
|
||||
|
||||
package dsl
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Print is indent-style multiline print.
|
||||
// Example, given parse of '$y = 2 * $x + 1':
|
||||
//
|
||||
// * statement block
|
||||
// * assignment "="
|
||||
// * direct field value "y"
|
||||
// * operator "+"
|
||||
// * operator "*"
|
||||
// * int literal "2"
|
||||
// * direct field value "x"
|
||||
// * int literal "1"
|
||||
|
||||
func (ast *AST) Print() {
|
||||
ast.RootNode.Print()
|
||||
}
|
||||
|
||||
// PrintParex is parenthesized-expression print.
|
||||
// Example, given parse of '$y = 2 * $x + 1':
|
||||
//
|
||||
// (statement-block
|
||||
// (=
|
||||
// $y
|
||||
// (+
|
||||
// (* 2 $x)
|
||||
// 1
|
||||
// )
|
||||
// )
|
||||
// )
|
||||
|
||||
func (ast *AST) PrintParex() {
|
||||
ast.RootNode.PrintParex()
|
||||
}
|
||||
|
||||
// PrintParexOneLine is parenthesized-expression print, all on one line.
|
||||
// Example, given parse of '$y = 2 * $x + 1':
|
||||
//
|
||||
// (statement-block (= $y (+ (* 2 $x) 1)))
|
||||
|
||||
func (ast *AST) PrintParexOneLine() {
|
||||
ast.RootNode.PrintParexOneLine()
|
||||
}
|
||||
|
||||
// Print is indent-style multiline print.
|
||||
func (node *ASTNode) Print() {
|
||||
node.printAux(0)
|
||||
}
|
||||
|
||||
// printAux is a recursion-helper for Print.
|
||||
func (node *ASTNode) printAux(depth int) {
|
||||
// Indent
|
||||
for range depth {
|
||||
fmt.Print(" ")
|
||||
}
|
||||
|
||||
// Token text (if non-nil) and token type
|
||||
tok := node.Token
|
||||
fmt.Print("* " + node.Type)
|
||||
if tok != nil {
|
||||
fmt.Printf(" \"%s\"", string(tok.Lit))
|
||||
}
|
||||
fmt.Println()
|
||||
|
||||
// Children, indented one level further
|
||||
for _, child := range node.Children {
|
||||
child.printAux(depth + 1)
|
||||
}
|
||||
}
|
||||
|
||||
// PrintParex is parenthesized-expression print.
|
||||
func (node *ASTNode) PrintParex() {
|
||||
node.printParexAux(0)
|
||||
}
|
||||
|
||||
// printParexAux is a recursion-helper for PrintParex.
|
||||
func (node *ASTNode) printParexAux(depth int) {
|
||||
if node.IsLeaf() {
|
||||
for range depth {
|
||||
fmt.Print(" ")
|
||||
}
|
||||
fmt.Println(node.Text())
|
||||
|
||||
} else if node.ChildrenAreAllLeaves() {
|
||||
// E.g. (= sum 0) or (+ 1 2)
|
||||
for range depth {
|
||||
fmt.Print(" ")
|
||||
}
|
||||
fmt.Print("(")
|
||||
fmt.Print(node.Text())
|
||||
|
||||
for _, child := range node.Children {
|
||||
fmt.Print(" ")
|
||||
fmt.Print(child.Text())
|
||||
}
|
||||
fmt.Println(")")
|
||||
|
||||
} else {
|
||||
// Parent and opening parenthesis on first line
|
||||
for range depth {
|
||||
fmt.Print(" ")
|
||||
}
|
||||
fmt.Print("(")
|
||||
fmt.Println(node.Text())
|
||||
|
||||
// Children on their own lines
|
||||
for _, child := range node.Children {
|
||||
child.printParexAux(depth + 1)
|
||||
}
|
||||
|
||||
// Closing parenthesis on last line
|
||||
for range depth {
|
||||
fmt.Print(" ")
|
||||
}
|
||||
fmt.Println(")")
|
||||
}
|
||||
}
|
||||
|
||||
// PrintParexOneLine is parenthesized-expression print, all on one line.
|
||||
func (node *ASTNode) PrintParexOneLine() {
|
||||
node.printParexOneLineAux()
|
||||
fmt.Println()
|
||||
}
|
||||
|
||||
// printParexOneLineAux is a recursion-helper for PrintParexOneLine.
|
||||
func (node *ASTNode) printParexOneLineAux() {
|
||||
if node.IsLeaf() {
|
||||
fmt.Print(node.Text())
|
||||
} else {
|
||||
fmt.Print("(")
|
||||
fmt.Print(node.Text())
|
||||
for _, child := range node.Children {
|
||||
fmt.Print(" ")
|
||||
child.printParexOneLineAux()
|
||||
}
|
||||
fmt.Print(")")
|
||||
}
|
||||
}
|
||||
|
||||
// IsLeaf determines if an AST node is a leaf node.
|
||||
func (node *ASTNode) IsLeaf() bool {
|
||||
return len(node.Children) == 0
|
||||
}
|
||||
|
||||
// ChildrenAreAllLeaves determines if an AST node's children are all leaf nodes.
|
||||
func (node *ASTNode) ChildrenAreAllLeaves() bool {
|
||||
for _, child := range node.Children {
|
||||
if !child.IsLeaf() {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// Text makes a human-readable, whitespace-free name for an AST node. Some
|
||||
// nodes have non-nil tokens; other, nil. And token-types can have spaces in
|
||||
// them. In this method we use custom mappings to always get a whitespace-free
|
||||
// representation of the content of a single AST node.
|
||||
func (node *ASTNode) Text() string {
|
||||
tokenText := ""
|
||||
if node.Token != nil {
|
||||
tokenText = string(node.Token.Lit)
|
||||
}
|
||||
|
||||
switch node.Type {
|
||||
|
||||
case NodeTypeStringLiteral:
|
||||
return "\"" + strings.ReplaceAll(tokenText, "\"", "\\\"") + "\""
|
||||
case NodeTypeIntLiteral:
|
||||
return tokenText
|
||||
case NodeTypeFloatLiteral:
|
||||
return tokenText
|
||||
case NodeTypeBoolLiteral:
|
||||
return tokenText
|
||||
case NodeTypeNullLiteral:
|
||||
return tokenText
|
||||
case NodeTypeArrayLiteral:
|
||||
return tokenText
|
||||
case NodeTypeMapLiteral:
|
||||
return tokenText
|
||||
case NodeTypeMapLiteralKeyValuePair:
|
||||
return tokenText
|
||||
|
||||
case NodeTypeArrayOrMapIndexAccess:
|
||||
return "[]"
|
||||
case NodeTypeArraySliceAccess:
|
||||
return "[:]"
|
||||
case NodeTypeArraySliceEmptyLowerIndex:
|
||||
return "array-slice-empty-lower-index"
|
||||
case NodeTypeArraySliceEmptyUpperIndex:
|
||||
return "array-slice-empty-upper-index"
|
||||
case NodeTypeContextVariable:
|
||||
return tokenText
|
||||
case NodeTypeConstant:
|
||||
return tokenText
|
||||
case NodeTypeEnvironmentVariable:
|
||||
return "ENV[\"" + tokenText + "\"]"
|
||||
|
||||
case NodeTypeDirectFieldValue:
|
||||
return "$" + tokenText
|
||||
case NodeTypeIndirectFieldValue:
|
||||
return "$[" + tokenText + "]"
|
||||
case NodeTypeFullSrec:
|
||||
return tokenText
|
||||
case NodeTypeDirectOosvarValue:
|
||||
return "@" + tokenText
|
||||
case NodeTypeIndirectOosvarValue:
|
||||
return "@[" + tokenText + "]"
|
||||
case NodeTypeFullOosvar:
|
||||
return tokenText
|
||||
case NodeTypeLocalVariable:
|
||||
return tokenText
|
||||
case NodeTypeTypedecl:
|
||||
return tokenText
|
||||
|
||||
case NodeTypeStatementBlock:
|
||||
return "statement-block"
|
||||
case NodeTypeAssignment:
|
||||
return tokenText
|
||||
case NodeTypeUnset:
|
||||
return tokenText
|
||||
|
||||
case NodeTypeBareBoolean:
|
||||
return "bare-boolean"
|
||||
case NodeTypeFilterStatement:
|
||||
return tokenText
|
||||
case NodeTypeEmit1Statement:
|
||||
return tokenText
|
||||
case NodeTypeEmitStatement:
|
||||
return tokenText
|
||||
case NodeTypeEmitPStatement:
|
||||
return tokenText
|
||||
case NodeTypeEmitFStatement:
|
||||
return tokenText
|
||||
case NodeTypeDumpStatement:
|
||||
return tokenText
|
||||
case NodeTypeEdumpStatement:
|
||||
return tokenText
|
||||
case NodeTypePrintStatement:
|
||||
return tokenText
|
||||
case NodeTypeEprintStatement:
|
||||
return tokenText
|
||||
case NodeTypePrintnStatement:
|
||||
return tokenText
|
||||
case NodeTypeEprintnStatement:
|
||||
return tokenText
|
||||
|
||||
case NodeTypeNoOp:
|
||||
return "no-op"
|
||||
|
||||
case NodeTypeOperator:
|
||||
return tokenText
|
||||
case NodeTypeDotOperator:
|
||||
return tokenText
|
||||
case NodeTypeFunctionCallsite:
|
||||
return tokenText
|
||||
|
||||
case NodeTypeBeginBlock:
|
||||
return "begin"
|
||||
case NodeTypeEndBlock:
|
||||
return "end"
|
||||
case NodeTypeIfChain:
|
||||
return "if-chain"
|
||||
case NodeTypeIfItem:
|
||||
return tokenText
|
||||
case NodeTypeCondBlock:
|
||||
return "cond"
|
||||
case NodeTypeWhileLoop:
|
||||
return tokenText
|
||||
case NodeTypeDoWhileLoop:
|
||||
return tokenText
|
||||
case NodeTypeForLoopOneVariable:
|
||||
return tokenText
|
||||
case NodeTypeForLoopTwoVariable:
|
||||
return tokenText
|
||||
case NodeTypeForLoopMultivariable:
|
||||
return tokenText
|
||||
case NodeTypeTripleForLoop:
|
||||
return tokenText
|
||||
case NodeTypeBreak:
|
||||
return tokenText
|
||||
case NodeTypeContinue:
|
||||
return tokenText
|
||||
|
||||
case NodeTypeNamedFunctionDefinition:
|
||||
return "named-udf"
|
||||
case NodeTypeUnnamedFunctionDefinition:
|
||||
return "unnamed-udf"
|
||||
case NodeTypeSubroutineDefinition:
|
||||
return "subr"
|
||||
case NodeTypeParameterList:
|
||||
return "parameters"
|
||||
case NodeTypeParameter:
|
||||
return "parameter"
|
||||
case NodeTypeParameterName:
|
||||
return tokenText
|
||||
case NodeTypeReturn:
|
||||
return tokenText
|
||||
|
||||
case NodeTypePanic:
|
||||
return tokenText
|
||||
|
||||
}
|
||||
return "[ERROR]"
|
||||
}
|
||||
|
|
@ -1,21 +1,9 @@
|
|||
// AST and ASTNode data structures for the Miller DSL parser
|
||||
// TNodeType constants for the Miller DSL parser.
|
||||
// The CST builder uses asts.ASTNode from github.com/johnkerl/pgpg;
|
||||
// these types identify node kinds when converting PGPG ASTs to CST nodes.
|
||||
|
||||
package dsl
|
||||
|
||||
import (
|
||||
"github.com/johnkerl/miller/v6/pkg/parsing/token"
|
||||
)
|
||||
|
||||
type AST struct {
|
||||
RootNode *ASTNode
|
||||
}
|
||||
|
||||
type ASTNode struct {
|
||||
Token *token.Token // Nil for tokenless/structural nodes
|
||||
Type TNodeType
|
||||
Children []*ASTNode
|
||||
}
|
||||
|
||||
type TNodeType string
|
||||
|
||||
const (
|
||||
|
|
|
|||
|
|
@ -3,16 +3,18 @@
|
|||
package cst
|
||||
|
||||
import (
|
||||
"github.com/johnkerl/miller/v6/pkg/dsl"
|
||||
"fmt"
|
||||
|
||||
"github.com/johnkerl/miller/v6/pkg/lib"
|
||||
"github.com/johnkerl/miller/v6/pkg/runtime"
|
||||
"github.com/johnkerl/pgpg/go/lib/pkg/asts"
|
||||
)
|
||||
|
||||
func (root *RootNode) BuildAssignmentNode(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
) (*AssignmentNode, error) {
|
||||
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeAssignment)
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeAssignment))
|
||||
err := astNode.CheckArity(2)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
|
|
@ -37,6 +39,99 @@ func (root *RootNode) BuildAssignmentNode(
|
|||
}, nil
|
||||
}
|
||||
|
||||
func (root *RootNode) BuildCompoundAssignmentNode(
|
||||
astNode *asts.ASTNode,
|
||||
) (*AssignmentNode, error) {
|
||||
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeCompoundAssignment))
|
||||
err := astNode.CheckArity(3)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
lhsASTNode := astNode.Children[0]
|
||||
opASTNode := astNode.Children[1]
|
||||
rhsASTNode := astNode.Children[2]
|
||||
|
||||
compoundOp := tokenLit(opASTNode)
|
||||
baseOp := compoundOpToBaseOp(compoundOp)
|
||||
if baseOp == "" {
|
||||
return nil, fmt.Errorf("unknown compound assignment operator: %s", compoundOp)
|
||||
}
|
||||
|
||||
lvalueNode, err := root.BuildAssignableNode(lhsASTNode)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
lvalueAsRvalue, err := root.BuildEvaluableNode(lhsASTNode)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
rvalueNode, err := root.BuildEvaluableNode(rhsASTNode)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
compoundRvalueNode, err := root.buildBinaryOperatorFromEvaluables(baseOp, lvalueAsRvalue, rvalueNode, rhsASTNode)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return &AssignmentNode{
|
||||
lvalueNode: lvalueNode,
|
||||
rvalueNode: compoundRvalueNode,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// compoundOpToBaseOp maps compound assignment operators to their base operator.
|
||||
// E.g. "+=" -> "+", "**=" -> "**".
|
||||
func compoundOpToBaseOp(compound string) string {
|
||||
switch compound {
|
||||
case "||=":
|
||||
return "||"
|
||||
case "^^=":
|
||||
return "^^"
|
||||
case "&&=":
|
||||
return "&&"
|
||||
case "??=":
|
||||
return "??"
|
||||
case "???=":
|
||||
return "???"
|
||||
case "|=":
|
||||
return "|"
|
||||
case "&=":
|
||||
return "&"
|
||||
case "^=":
|
||||
return "^"
|
||||
case "<<=":
|
||||
return "<<"
|
||||
case ">>=":
|
||||
return ">>"
|
||||
case ">>>=":
|
||||
return ">>>"
|
||||
case "+=":
|
||||
return "+"
|
||||
case ".=":
|
||||
return "."
|
||||
case "-=":
|
||||
return "-"
|
||||
case "*=":
|
||||
return "*"
|
||||
case "/=":
|
||||
return "/"
|
||||
case "//=":
|
||||
return "//"
|
||||
case "%=":
|
||||
return "%"
|
||||
case "**=":
|
||||
return "**"
|
||||
default:
|
||||
return ""
|
||||
}
|
||||
}
|
||||
|
||||
type AssignmentNode struct {
|
||||
lvalueNode IAssignable
|
||||
rvalueNode IEvaluable
|
||||
|
|
@ -56,17 +151,24 @@ func (node *AssignmentNode) Execute(
|
|||
}
|
||||
|
||||
func (root *RootNode) BuildUnsetNode(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
) (*UnsetNode, error) {
|
||||
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeUnset)
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeUnset))
|
||||
|
||||
lvalueNodes := make([]IAssignable, len(astNode.Children))
|
||||
|
||||
for i, lhsASTNode := range astNode.Children {
|
||||
lvalueNode, err := root.BuildAssignableNode(lhsASTNode)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
var lvalueNode IAssignable
|
||||
var err error
|
||||
// "all" is a synonym for @* (full oosvar) in unset context.
|
||||
if lhsASTNode.Type == asts.NodeType(NodeTypeLocalVariable) && tokenLit(lhsASTNode) == "all" {
|
||||
lvalueNode = NewFullOosvarLvalueNode()
|
||||
} else {
|
||||
lvalueNode, err = root.BuildAssignableNode(lhsASTNode)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
lvalueNodes[i] = lvalueNode
|
||||
}
|
||||
|
|
|
|||
153
pkg/dsl/cst/ast_types.go
Normal file
153
pkg/dsl/cst/ast_types.go
Normal file
|
|
@ -0,0 +1,153 @@
|
|||
// PGPG AST node type constants. These match the "type" field values
|
||||
// produced by the mlr.bnf grammar. The CST uses these when
|
||||
// comparing asts.ASTNode.Type (asts.NodeType is string).
|
||||
|
||||
package cst
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
"github.com/johnkerl/pgpg/go/lib/pkg/asts"
|
||||
"github.com/johnkerl/pgpg/go/lib/pkg/tokens"
|
||||
)
|
||||
|
||||
// PGPG grammar node types (camelCase from grammar hints)
|
||||
const (
|
||||
NodeTypeStatementBlock = "StatementBlock"
|
||||
NodeTypeStatementBlockInBraces = "StatementBlockInBraces"
|
||||
NodeTypeAssignment = "Assignment"
|
||||
NodeTypeCompoundAssignment = "CompoundAssignment"
|
||||
NodeTypeUnset = "Unset"
|
||||
NodeTypeBareBoolean = "BareBoolean"
|
||||
NodeTypeFilterStatement = "FilterStatement"
|
||||
NodeTypeRedirectWrite = "RedirectWrite"
|
||||
NodeTypeRedirectAppend = "RedirectAppend"
|
||||
NodeTypeRedirectPipe = "RedirectPipe"
|
||||
NodeTypeRedirectTargetStdout = "RedirectTargetStdout"
|
||||
NodeTypeRedirectTargetStderr = "RedirectTargetStderr"
|
||||
NodeTypeRedirectTargetRvalue = "RedirectTargetRvalue"
|
||||
NodeTypePrintStatement = "PrintStatement"
|
||||
NodeTypePrintnStatement = "PrintnStatement"
|
||||
NodeTypeEprintStatement = "EprintStatement"
|
||||
NodeTypeEprintnStatement = "EprintnStatement"
|
||||
NodeTypeDumpStatement = "DumpStatement"
|
||||
NodeTypeEdumpStatement = "EdumpStatement"
|
||||
NodeTypeTeeStatement = "TeeStatement"
|
||||
NodeTypeEmit1Statement = "Emit1Statement"
|
||||
NodeTypeEmitStatement = "EmitStatement"
|
||||
NodeTypeEmitPStatement = "EmitPStatement"
|
||||
NodeTypeEmitFStatement = "EmitFStatement"
|
||||
NodeTypeBeginBlock = "BeginBlock"
|
||||
NodeTypeEndBlock = "EndBlock"
|
||||
NodeTypeCondBlock = "CondBlock"
|
||||
NodeTypeIfChain = "IfChain"
|
||||
NodeTypeIfItem = "IfItem"
|
||||
NodeTypeWhileLoop = "WhileLoop"
|
||||
NodeTypeDoWhileLoop = "DoWhileLoop"
|
||||
NodeTypeForLoopOneVariable = "ForLoopOneVariable"
|
||||
NodeTypeForLoopTwoVariable = "ForLoopTwoVariable"
|
||||
NodeTypeForLoopMultivariable = "ForLoopMultivariable"
|
||||
NodeTypeTripleForLoop = "TripleForLoop"
|
||||
NodeTypeBreakStatement = "BreakStatement"
|
||||
NodeTypeContinueStatement = "ContinueStatement"
|
||||
NodeTypeReturnStatement = "ReturnStatement"
|
||||
NodeTypeSubroutineCallsite = "SubroutineCallsite"
|
||||
NodeTypeNamedFunctionDefinition = "NamedFunctionDefinition"
|
||||
NodeTypeSubroutineDefinition = "SubroutineDefinition"
|
||||
NodeTypeUnnamedFunctionDefinition = "UnnamedFunctionDefinition"
|
||||
NodeTypeParameterList = "ParameterList"
|
||||
NodeTypeParameter = "Parameter"
|
||||
NodeTypeDirectFieldValue = "DirectFieldValue"
|
||||
NodeTypeIndirectFieldValue = "IndirectFieldValue"
|
||||
NodeTypeBracedFieldValue = "BracedFieldValue"
|
||||
NodeTypeFullSrec = "FullSrec"
|
||||
NodeTypeDirectOosvarValue = "DirectOosvarValue"
|
||||
NodeTypeIndirectOosvarValue = "IndirectOosvarValue"
|
||||
NodeTypeBracedOosvarValue = "BracedOosvarValue"
|
||||
NodeTypeFullOosvar = "FullOosvar"
|
||||
NodeTypeLocalVariable = "LocalVariable"
|
||||
NodeTypeOperator = "Operator"
|
||||
NodeTypeDotOperator = "DotOperator"
|
||||
NodeTypeFunctionCallsite = "FunctionCallsite"
|
||||
NodeTypeArrayLiteral = "ArrayLiteral"
|
||||
NodeTypeMapLiteral = "MapLiteral"
|
||||
NodeTypeMapLiteralKeyValuePair = "MapLiteralKeyValuePair"
|
||||
NodeTypeArrayOrMapIndexAccess = "ArrayOrMapIndexAccess"
|
||||
NodeTypeArraySliceLoHi = "ArraySliceLoHi"
|
||||
NodeTypeArraySliceHiOnly = "ArraySliceHiOnly"
|
||||
NodeTypeArraySliceLoOnly = "ArraySliceLoOnly"
|
||||
NodeTypeArraySliceFull = "ArraySliceFull"
|
||||
NodeTypeParenthesized = "Parenthesized"
|
||||
NodeTypeFcnArgs = "FcnArgs"
|
||||
NodeTypeIntLiteral = "int_literal"
|
||||
NodeTypeFloatLiteral = "float_literal"
|
||||
NodeTypeStringLiteral = "string_literal"
|
||||
NodeTypeBoolLiteral = "bool_literal"
|
||||
NodeTypeNullLiteral = "null_literal"
|
||||
NodeTypeMultiIndex = "MultiIndex"
|
||||
NodeTypeTypedecl = "Typedecl"
|
||||
// Synthetic nodes for $[[n]] / $[[[n]]] (positional indexing - not in PGPG grammar)
|
||||
NodeTypePositionalFieldName = "PositionalFieldName"
|
||||
NodeTypePositionalFieldValue = "PositionalFieldValue"
|
||||
// For array slice empty bounds: [lo:], [:hi], [:]
|
||||
NodeTypeArraySliceEmptyLowerIndex = "ArraySliceEmptyLowerIndex"
|
||||
NodeTypeArraySliceEmptyUpperIndex = "ArraySliceEmptyUpperIndex"
|
||||
)
|
||||
|
||||
// Injected during regexProtectPrePass (not from grammar)
|
||||
const NodeTypeRegex = "Regex"
|
||||
|
||||
// NoOp: used when optional redirect/expressions are absent (PGPG may not produce these)
|
||||
const NodeTypeNoOp = "NoOp"
|
||||
|
||||
// NodeTypePanic: special token for testing short-circuit (not in grammar)
|
||||
const NodeTypePanic = "Panic"
|
||||
|
||||
// Types not in mlr.bnf (positional indexing, ENV, context vars excluded).
|
||||
// Added so CST compiles; extend grammar to support.
|
||||
const (
|
||||
NodeTypeArraySliceAccess = "ArraySliceAccess" // alias for any of ArraySliceLoHi etc
|
||||
NodeTypeArrayOrMapPositionalNameAccess = "ArrayOrMapPositionalNameAccess"
|
||||
NodeTypeArrayOrMapPositionalValueAccess = "ArrayOrMapPositionalValueAccess"
|
||||
NodeTypeEmitKeys = "EmitKeys"
|
||||
NodeTypeEnvironmentVariable = "EnvironmentVariable"
|
||||
NodeTypeRegexCaseInsensitive = "RegexCaseInsensitive"
|
||||
NodeTypeContextVariable = "ContextVariable"
|
||||
NodeTypeConstant = "Constant"
|
||||
)
|
||||
|
||||
// tokenLit returns the lexeme text for an AST node's token, or "" if nil.
|
||||
func tokenLit(node *asts.ASTNode) string {
|
||||
if node == nil || node.Token == nil {
|
||||
return ""
|
||||
}
|
||||
return node.Token.LexemeText()
|
||||
}
|
||||
|
||||
// tokenLitStripDollarOrAt returns the lexeme with leading $ or @ stripped.
|
||||
// Used for DirectFieldValue ($n -> n) and DirectOosvarValue (@x -> x).
|
||||
func tokenLitStripDollarOrAt(node *asts.ASTNode) string {
|
||||
s := tokenLit(node)
|
||||
if len(s) >= 1 && (s[0] == '$' || s[0] == '@') {
|
||||
return s[1:]
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// tokenLitStripBraced returns the lexeme with ${ / @{ and } stripped.
|
||||
// Used for BracedFieldValue (${foo} -> foo) and BracedOosvarValue (@{bar} -> bar).
|
||||
func tokenLitStripBraced(node *asts.ASTNode) string {
|
||||
s := tokenLit(node)
|
||||
if len(s) >= 4 && (s[0] == '$' || s[0] == '@') && s[1] == '{' && s[len(s)-1] == '}' {
|
||||
return s[2 : len(s)-1]
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// pgpgTokenToLocationInfo formats location info for error messages.
|
||||
func pgpgTokenToLocationInfo(tok *tokens.Token) string {
|
||||
if tok == nil {
|
||||
return ""
|
||||
}
|
||||
return fmt.Sprintf(" at DSL expression line %d column %d", tok.Location.LineNumber, tok.Location.ColumnNumber)
|
||||
}
|
||||
|
|
@ -6,16 +6,17 @@ package cst
|
|||
import (
|
||||
"fmt"
|
||||
|
||||
"github.com/johnkerl/miller/v6/pkg/dsl"
|
||||
"github.com/johnkerl/miller/v6/pkg/lib"
|
||||
"github.com/johnkerl/miller/v6/pkg/runtime"
|
||||
|
||||
"github.com/johnkerl/pgpg/go/lib/pkg/asts"
|
||||
)
|
||||
|
||||
type BreakNode struct {
|
||||
}
|
||||
|
||||
func (root *RootNode) BuildBreakNode(astNode *dsl.ASTNode) (*BreakNode, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeBreak)
|
||||
func (root *RootNode) BuildBreakNode(astNode *asts.ASTNode) (*BreakNode, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeBreakStatement))
|
||||
lib.InternalCodingErrorIf(astNode.Children == nil)
|
||||
lib.InternalCodingErrorIf(len(astNode.Children) != 0)
|
||||
|
||||
|
|
@ -32,8 +33,8 @@ func (node *BreakNode) Execute(state *runtime.State) (*BlockExitPayload, error)
|
|||
type ContinueNode struct {
|
||||
}
|
||||
|
||||
func (root *RootNode) BuildContinueNode(astNode *dsl.ASTNode) (*ContinueNode, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeContinue)
|
||||
func (root *RootNode) BuildContinueNode(astNode *asts.ASTNode) (*ContinueNode, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeContinueStatement))
|
||||
lib.InternalCodingErrorIf(astNode.Children == nil)
|
||||
lib.InternalCodingErrorIf(len(astNode.Children) != 0)
|
||||
|
||||
|
|
@ -51,8 +52,8 @@ type ReturnNode struct {
|
|||
returnValueExpression IEvaluable
|
||||
}
|
||||
|
||||
func (root *RootNode) BuildReturnNode(astNode *dsl.ASTNode) (*ReturnNode, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeReturn)
|
||||
func (root *RootNode) BuildReturnNode(astNode *asts.ASTNode) (*ReturnNode, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeReturnStatement))
|
||||
lib.InternalCodingErrorIf(astNode.Children == nil)
|
||||
if len(astNode.Children) == 0 {
|
||||
return &ReturnNode{returnValueExpression: nil}, nil
|
||||
|
|
|
|||
|
|
@ -4,9 +4,10 @@
|
|||
package cst
|
||||
|
||||
import (
|
||||
"github.com/johnkerl/miller/v6/pkg/dsl"
|
||||
"github.com/johnkerl/miller/v6/pkg/lib"
|
||||
"github.com/johnkerl/miller/v6/pkg/runtime"
|
||||
|
||||
"github.com/johnkerl/pgpg/go/lib/pkg/asts"
|
||||
)
|
||||
|
||||
func NewStatementBlockNode() *StatementBlockNode {
|
||||
|
|
@ -20,12 +21,12 @@ func (node *StatementBlockNode) AppendStatementNode(executable IExecutable) {
|
|||
}
|
||||
|
||||
func (root *RootNode) BuildStatementBlockNodeFromBeginOrEnd(
|
||||
astBeginOrEndNode *dsl.ASTNode,
|
||||
astBeginOrEndNode *asts.ASTNode,
|
||||
) (*StatementBlockNode, error) {
|
||||
|
||||
lib.InternalCodingErrorIf(
|
||||
astBeginOrEndNode.Type != dsl.NodeTypeBeginBlock &&
|
||||
astBeginOrEndNode.Type != dsl.NodeTypeEndBlock,
|
||||
astBeginOrEndNode.Type != asts.NodeType(NodeTypeBeginBlock) &&
|
||||
astBeginOrEndNode.Type != asts.NodeType(NodeTypeEndBlock),
|
||||
)
|
||||
lib.InternalCodingErrorIf(astBeginOrEndNode.Children == nil)
|
||||
// TODO: change the BNF to make it always 1 in the AST
|
||||
|
|
@ -58,7 +59,13 @@ func (root *RootNode) BuildStatementBlockNodeFromBeginOrEnd(
|
|||
// * IntLiteral "4"
|
||||
|
||||
astStatementBlockNode := astBeginOrEndNode.Children[0]
|
||||
lib.InternalCodingErrorIf(astStatementBlockNode.Type != dsl.NodeTypeStatementBlock)
|
||||
// PGPG: BeginBlock/EndBlock have StatementBlockInBraces as child.
|
||||
// With "parent":1,"children":[1], StatementBlockInBraces.Children[0] is the StatementBlock.
|
||||
// Unwrap so we pass StatementBlock to BuildStatementBlockNode.
|
||||
if astStatementBlockNode.Type == asts.NodeType(NodeTypeStatementBlockInBraces) {
|
||||
lib.InternalCodingErrorIf(astStatementBlockNode.Children == nil || len(astStatementBlockNode.Children) < 1)
|
||||
astStatementBlockNode = astStatementBlockNode.Children[0]
|
||||
}
|
||||
statementBlockNode, err := root.BuildStatementBlockNode(astStatementBlockNode)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
|
|
@ -67,10 +74,15 @@ func (root *RootNode) BuildStatementBlockNodeFromBeginOrEnd(
|
|||
}
|
||||
|
||||
func (root *RootNode) BuildStatementBlockNode(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
) (*StatementBlockNode, error) {
|
||||
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeStatementBlock)
|
||||
// PGPG: StatementBlockInBraces has "children":[1] so its child is StatementBlock; unwrap.
|
||||
if astNode.Type == asts.NodeType(NodeTypeStatementBlockInBraces) &&
|
||||
astNode.Children != nil && len(astNode.Children) == 1 &&
|
||||
astNode.Children[0].Type == asts.NodeType(NodeTypeStatementBlock) {
|
||||
astNode = astNode.Children[0]
|
||||
}
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeStatementBlock))
|
||||
|
||||
statementBlockNode := NewStatementBlockNode()
|
||||
|
||||
|
|
|
|||
|
|
@ -7,10 +7,10 @@ import (
|
|||
"fmt"
|
||||
|
||||
"github.com/johnkerl/miller/v6/pkg/bifs"
|
||||
"github.com/johnkerl/miller/v6/pkg/dsl"
|
||||
"github.com/johnkerl/miller/v6/pkg/lib"
|
||||
"github.com/johnkerl/miller/v6/pkg/mlrval"
|
||||
"github.com/johnkerl/miller/v6/pkg/runtime"
|
||||
"github.com/johnkerl/pgpg/go/lib/pkg/asts"
|
||||
)
|
||||
|
||||
// BIF_next is for mlr script: reads next record from input stream.
|
||||
|
|
@ -31,16 +31,21 @@ func BIF_next(state *runtime.State) *mlrval.Mlrval {
|
|||
}
|
||||
|
||||
func (root *RootNode) BuildBuiltinFunctionCallsiteNode(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
) (IEvaluable, error) {
|
||||
lib.InternalCodingErrorIf(
|
||||
astNode.Type != dsl.NodeTypeFunctionCallsite &&
|
||||
astNode.Type != dsl.NodeTypeOperator,
|
||||
astNode.Type != asts.NodeType(NodeTypeFunctionCallsite) &&
|
||||
astNode.Type != asts.NodeType(NodeTypeOperator),
|
||||
)
|
||||
lib.InternalCodingErrorIf(astNode.Token == nil)
|
||||
lib.InternalCodingErrorIf(astNode.Children == nil)
|
||||
|
||||
functionName := string(astNode.Token.Lit)
|
||||
functionName := tokenLit(astNode)
|
||||
|
||||
// PGPG produces ternary with "?" token; builtin is registered as "?:"
|
||||
if functionName == "?" && len(astNode.Children) == 3 {
|
||||
functionName = "?:"
|
||||
}
|
||||
|
||||
builtinFunctionInfo := BuiltinFunctionManagerInstance.LookUp(functionName)
|
||||
if builtinFunctionInfo != nil {
|
||||
|
|
@ -79,7 +84,7 @@ func (root *RootNode) BuildBuiltinFunctionCallsiteNode(
|
|||
}
|
||||
|
||||
func (root *RootNode) BuildMultipleArityFunctionCallsiteNode(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
builtinFunctionInfo *BuiltinFunctionInfo,
|
||||
) (IEvaluable, error) {
|
||||
callsiteArity := len(astNode.Children)
|
||||
|
|
@ -104,7 +109,7 @@ type ZaryFunctionCallsiteNode struct {
|
|||
}
|
||||
|
||||
func BuildZaryFunctionCallsiteNode(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
builtinFunctionInfo *BuiltinFunctionInfo,
|
||||
) (IEvaluable, error) {
|
||||
callsiteArity := len(astNode.Children)
|
||||
|
|
@ -125,7 +130,7 @@ func BuildZaryFunctionCallsiteNode(
|
|||
}
|
||||
|
||||
func BuildZaryFunctionWithStateCallsiteNode(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
builtinFunctionInfo *BuiltinFunctionInfo,
|
||||
) (IEvaluable, error) {
|
||||
callsiteArity := len(astNode.Children)
|
||||
|
|
@ -167,7 +172,7 @@ type UnaryFunctionCallsiteNode struct {
|
|||
}
|
||||
|
||||
func (root *RootNode) BuildUnaryFunctionCallsiteNode(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
builtinFunctionInfo *BuiltinFunctionInfo,
|
||||
) (IEvaluable, error) {
|
||||
callsiteArity := len(astNode.Children)
|
||||
|
|
@ -205,7 +210,7 @@ type UnaryFunctionWithContextCallsiteNode struct {
|
|||
}
|
||||
|
||||
func (root *RootNode) BuildUnaryFunctionWithContextCallsiteNode(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
builtinFunctionInfo *BuiltinFunctionInfo,
|
||||
) (IEvaluable, error) {
|
||||
callsiteArity := len(astNode.Children)
|
||||
|
|
@ -244,7 +249,7 @@ type BinaryFunctionCallsiteNode struct {
|
|||
}
|
||||
|
||||
func (root *RootNode) BuildBinaryFunctionCallsiteNode(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
builtinFunctionInfo *BuiltinFunctionInfo,
|
||||
) (IEvaluable, error) {
|
||||
callsiteArity := len(astNode.Children)
|
||||
|
|
@ -301,6 +306,56 @@ func (root *RootNode) BuildBinaryFunctionCallsiteNode(
|
|||
}, nil
|
||||
}
|
||||
|
||||
// buildBinaryOperatorFromEvaluables builds an IEvaluable for (a baseOp b) from
|
||||
// evaluables, without going through the AST. Used by compound assignment (e.g.
|
||||
// @sum += $x -> @sum = @sum + $x). rhsASTNode is used for the dot operator's
|
||||
// string2 (map key) when baseOp is ".".
|
||||
func (root *RootNode) buildBinaryOperatorFromEvaluables(
|
||||
baseOp string,
|
||||
evaluable1, evaluable2 IEvaluable,
|
||||
rhsASTNode *asts.ASTNode,
|
||||
) (IEvaluable, error) {
|
||||
if baseOp == "." {
|
||||
return &DotCallsiteNode{
|
||||
evaluable1: evaluable1,
|
||||
evaluable2: evaluable2,
|
||||
string2: tokenLit(rhsASTNode),
|
||||
}, nil
|
||||
}
|
||||
|
||||
builtinFunctionInfo := BuiltinFunctionManagerInstance.LookUp(baseOp)
|
||||
if builtinFunctionInfo == nil {
|
||||
return nil, fmt.Errorf("compound assignment: unknown operator %q", baseOp)
|
||||
}
|
||||
if builtinFunctionInfo.binaryFunc != nil {
|
||||
if baseOp == "&&" {
|
||||
return BuildLogicalANDOperatorNode(evaluable1, evaluable2), nil
|
||||
}
|
||||
if baseOp == "||" {
|
||||
return BuildLogicalOROperatorNode(evaluable1, evaluable2), nil
|
||||
}
|
||||
if baseOp == "??" {
|
||||
return BuildAbsentCoalesceOperatorNode(evaluable1, evaluable2), nil
|
||||
}
|
||||
if baseOp == "???" {
|
||||
return BuildEmptyCoalesceOperatorNode(evaluable1, evaluable2), nil
|
||||
}
|
||||
return &BinaryFunctionCallsiteNode{
|
||||
binaryFunc: builtinFunctionInfo.binaryFunc,
|
||||
evaluable1: evaluable1,
|
||||
evaluable2: evaluable2,
|
||||
}, nil
|
||||
}
|
||||
if builtinFunctionInfo.binaryFuncWithState != nil {
|
||||
return &BinaryFunctionWithStateCallsiteNode{
|
||||
binaryFuncWithState: builtinFunctionInfo.binaryFuncWithState,
|
||||
evaluable1: evaluable1,
|
||||
evaluable2: evaluable2,
|
||||
}, nil
|
||||
}
|
||||
return nil, fmt.Errorf("compound assignment: operator %q not implemented", baseOp)
|
||||
}
|
||||
|
||||
func (node *BinaryFunctionCallsiteNode) Evaluate(
|
||||
state *runtime.State,
|
||||
) *mlrval.Mlrval {
|
||||
|
|
@ -317,7 +372,7 @@ type BinaryFunctionWithStateCallsiteNode struct {
|
|||
}
|
||||
|
||||
func (root *RootNode) BuildBinaryFunctionWithStateCallsiteNode(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
builtinFunctionInfo *BuiltinFunctionInfo,
|
||||
) (IEvaluable, error) {
|
||||
callsiteArity := len(astNode.Children)
|
||||
|
|
@ -366,7 +421,7 @@ type TernaryFunctionWithStateCallsiteNode struct {
|
|||
}
|
||||
|
||||
func (root *RootNode) BuildTernaryFunctionWithStateCallsiteNode(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
builtinFunctionInfo *BuiltinFunctionInfo,
|
||||
) (IEvaluable, error) {
|
||||
callsiteArity := len(astNode.Children)
|
||||
|
|
@ -451,7 +506,7 @@ type RegexCaptureBinaryFunctionCallsiteNode struct {
|
|||
}
|
||||
|
||||
func (root *RootNode) BuildRegexCaptureBinaryFunctionCallsiteNode(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
builtinFunctionInfo *BuiltinFunctionInfo,
|
||||
) (IEvaluable, error) {
|
||||
callsiteArity := len(astNode.Children)
|
||||
|
|
@ -503,7 +558,7 @@ type DotCallsiteNode struct {
|
|||
}
|
||||
|
||||
func (root *RootNode) BuildDotCallsiteNode(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
) (IEvaluable, error) {
|
||||
callsiteArity := len(astNode.Children)
|
||||
expectedArity := 2
|
||||
|
|
@ -529,7 +584,7 @@ func (root *RootNode) BuildDotCallsiteNode(
|
|||
return &DotCallsiteNode{
|
||||
evaluable1: evaluable1,
|
||||
evaluable2: evaluable2,
|
||||
string2: string(astNode.Children[1].Token.Lit),
|
||||
string2: tokenLit(astNode.Children[1]),
|
||||
}, nil
|
||||
}
|
||||
|
||||
|
|
@ -562,7 +617,7 @@ type TernaryFunctionCallsiteNode struct {
|
|||
}
|
||||
|
||||
func (root *RootNode) BuildTernaryFunctionCallsiteNode(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
builtinFunctionInfo *BuiltinFunctionInfo,
|
||||
) (IEvaluable, error) {
|
||||
callsiteArity := len(astNode.Children)
|
||||
|
|
@ -623,7 +678,7 @@ type VariadicFunctionCallsiteNode struct {
|
|||
}
|
||||
|
||||
func (root *RootNode) BuildVariadicFunctionCallsiteNode(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
builtinFunctionInfo *BuiltinFunctionInfo,
|
||||
) (IEvaluable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Children == nil)
|
||||
|
|
@ -680,7 +735,7 @@ type VariadicFunctionWithStateCallsiteNode struct {
|
|||
}
|
||||
|
||||
func (root *RootNode) BuildVariadicFunctionWithStateCallsiteNode(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
builtinFunctionInfo *BuiltinFunctionInfo,
|
||||
) (IEvaluable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Children == nil)
|
||||
|
|
|
|||
|
|
@ -7,10 +7,10 @@ import (
|
|||
"fmt"
|
||||
|
||||
"github.com/johnkerl/miller/v6/pkg/bifs"
|
||||
"github.com/johnkerl/miller/v6/pkg/dsl"
|
||||
"github.com/johnkerl/miller/v6/pkg/lib"
|
||||
"github.com/johnkerl/miller/v6/pkg/mlrval"
|
||||
"github.com/johnkerl/miller/v6/pkg/runtime"
|
||||
"github.com/johnkerl/pgpg/go/lib/pkg/asts"
|
||||
)
|
||||
|
||||
type ArrayLiteralNode struct {
|
||||
|
|
@ -18,9 +18,9 @@ type ArrayLiteralNode struct {
|
|||
}
|
||||
|
||||
func (node *RootNode) BuildArrayLiteralNode(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
) (IEvaluable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeArrayLiteral)
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeArrayLiteral))
|
||||
// An empty array should have non-nil zero-length children, not nil
|
||||
// children
|
||||
lib.InternalCodingErrorIf(astNode.Children == nil)
|
||||
|
|
@ -56,9 +56,9 @@ type ArrayOrMapIndexAccessNode struct {
|
|||
}
|
||||
|
||||
func (node *RootNode) BuildArrayOrMapIndexAccessNode(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
) (IEvaluable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeArrayOrMapIndexAccess)
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeArrayOrMapIndexAccess))
|
||||
lib.InternalCodingErrorIf(len(astNode.Children) != 2)
|
||||
|
||||
baseASTNode := astNode.Children[0]
|
||||
|
|
@ -198,30 +198,81 @@ type ArraySliceAccessNode struct {
|
|||
}
|
||||
|
||||
func (node *RootNode) BuildArraySliceAccessNode(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
) (IEvaluable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeArraySliceAccess)
|
||||
lib.InternalCodingErrorIf(len(astNode.Children) != 3)
|
||||
|
||||
baseASTNode := astNode.Children[0]
|
||||
lowerIndexASTNode := astNode.Children[1]
|
||||
upperIndexASTNode := astNode.Children[2]
|
||||
// PGPG produces ArraySliceLoHi (3 children), ArraySliceHiOnly (2), ArraySliceLoOnly (2), ArraySliceFull (1)
|
||||
switch astNode.Type {
|
||||
case asts.NodeType(NodeTypeArraySliceLoHi):
|
||||
// [lo:hi]
|
||||
lib.InternalCodingErrorIf(len(astNode.Children) != 3)
|
||||
return node.buildArraySliceAccessNodeFromChildren(astNode.Children[0], astNode.Children[1], astNode.Children[2])
|
||||
case asts.NodeType(NodeTypeArraySliceHiOnly):
|
||||
// [:hi] - lower defaults to 1
|
||||
lib.InternalCodingErrorIf(len(astNode.Children) != 2)
|
||||
synthLower, _ := node.BuildArraySliceEmptyLowerIndexNode(asts.NewASTNode(nil, asts.NodeType(NodeTypeArraySliceEmptyLowerIndex), nil))
|
||||
baseEval, err := node.BuildEvaluableNode(astNode.Children[0])
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
upperEval, err := node.BuildEvaluableNode(astNode.Children[1])
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return node.buildArraySliceAccessFromEvaluables(baseEval, synthLower, upperEval)
|
||||
case asts.NodeType(NodeTypeArraySliceLoOnly):
|
||||
// [lo:] - upper defaults to n
|
||||
lib.InternalCodingErrorIf(len(astNode.Children) != 2)
|
||||
synthUpper, _ := node.BuildArraySliceEmptyUpperIndexNode(asts.NewASTNode(nil, asts.NodeType(NodeTypeArraySliceEmptyUpperIndex), nil))
|
||||
baseEval, err := node.BuildEvaluableNode(astNode.Children[0])
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
lowerEval, err := node.BuildEvaluableNode(astNode.Children[1])
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return node.buildArraySliceAccessFromEvaluables(baseEval, lowerEval, synthUpper)
|
||||
case asts.NodeType(NodeTypeArraySliceFull):
|
||||
// [:]
|
||||
lib.InternalCodingErrorIf(len(astNode.Children) != 1)
|
||||
synthLower, _ := node.BuildArraySliceEmptyLowerIndexNode(asts.NewASTNode(nil, asts.NodeType(NodeTypeArraySliceEmptyLowerIndex), nil))
|
||||
synthUpper, _ := node.BuildArraySliceEmptyUpperIndexNode(asts.NewASTNode(nil, asts.NodeType(NodeTypeArraySliceEmptyUpperIndex), nil))
|
||||
baseEval, err := node.BuildEvaluableNode(astNode.Children[0])
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return node.buildArraySliceAccessFromEvaluables(baseEval, synthLower, synthUpper)
|
||||
default:
|
||||
lib.InternalCodingErrorIf(true)
|
||||
return nil, nil
|
||||
}
|
||||
}
|
||||
|
||||
func (node *RootNode) buildArraySliceAccessNodeFromChildren(
|
||||
baseASTNode *asts.ASTNode,
|
||||
lowerIndexASTNode *asts.ASTNode,
|
||||
upperIndexASTNode *asts.ASTNode,
|
||||
) (IEvaluable, error) {
|
||||
baseEvaluable, err := node.BuildEvaluableNode(baseASTNode)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
lowerIndexEvaluable, err := node.BuildEvaluableNode(lowerIndexASTNode)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
upperIndexEvaluable, err := node.BuildEvaluableNode(upperIndexASTNode)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return node.buildArraySliceAccessFromEvaluables(baseEvaluable, lowerIndexEvaluable, upperIndexEvaluable)
|
||||
}
|
||||
|
||||
func (node *RootNode) buildArraySliceAccessFromEvaluables(
|
||||
baseEvaluable IEvaluable,
|
||||
lowerIndexEvaluable IEvaluable,
|
||||
upperIndexEvaluable IEvaluable,
|
||||
) (IEvaluable, error) {
|
||||
return &ArraySliceAccessNode{
|
||||
baseEvaluable: baseEvaluable,
|
||||
lowerIndexEvaluable: lowerIndexEvaluable,
|
||||
|
|
@ -287,9 +338,9 @@ type PositionalFieldNameNode struct {
|
|||
}
|
||||
|
||||
func (node *RootNode) BuildPositionalFieldNameNode(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
) (IEvaluable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypePositionalFieldName)
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypePositionalFieldName))
|
||||
lib.InternalCodingErrorIf(len(astNode.Children) != 1)
|
||||
|
||||
indexASTNode := astNode.Children[0]
|
||||
|
|
@ -333,9 +384,9 @@ type PositionalFieldValueNode struct {
|
|||
}
|
||||
|
||||
func (node *RootNode) BuildPositionalFieldValueNode(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
) (IEvaluable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypePositionalFieldValue)
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypePositionalFieldValue))
|
||||
lib.InternalCodingErrorIf(len(astNode.Children) != 1)
|
||||
|
||||
indexASTNode := astNode.Children[0]
|
||||
|
|
@ -379,9 +430,9 @@ type ArrayOrMapPositionalNameAccessNode struct {
|
|||
}
|
||||
|
||||
func (node *RootNode) BuildArrayOrMapPositionalNameAccessNode(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
) (IEvaluable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeArrayOrMapPositionalNameAccess)
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeArrayOrMapPositionalNameAccess))
|
||||
lib.InternalCodingErrorIf(len(astNode.Children) != 2)
|
||||
|
||||
baseASTNode := astNode.Children[0]
|
||||
|
|
@ -455,9 +506,9 @@ type ArrayOrMapPositionalValueAccessNode struct {
|
|||
}
|
||||
|
||||
func (node *RootNode) BuildArrayOrMapPositionalValueAccessNode(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
) (IEvaluable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeArrayOrMapPositionalValueAccess)
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeArrayOrMapPositionalValueAccess))
|
||||
lib.InternalCodingErrorIf(len(astNode.Children) != 2)
|
||||
|
||||
baseASTNode := astNode.Children[0]
|
||||
|
|
@ -546,9 +597,9 @@ type MapLiteralNode struct {
|
|||
}
|
||||
|
||||
func (node *RootNode) BuildMapLiteralNode(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
) (IEvaluable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeMapLiteral)
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeMapLiteral))
|
||||
// An empty array should have non-nil zero-length children, not nil
|
||||
// children
|
||||
lib.InternalCodingErrorIf(astNode.Children == nil)
|
||||
|
|
|
|||
|
|
@ -6,23 +6,24 @@ package cst
|
|||
import (
|
||||
"fmt"
|
||||
|
||||
"github.com/johnkerl/miller/v6/pkg/dsl"
|
||||
"github.com/johnkerl/miller/v6/pkg/lib"
|
||||
"github.com/johnkerl/miller/v6/pkg/mlrval"
|
||||
"github.com/johnkerl/miller/v6/pkg/parsing/token"
|
||||
"github.com/johnkerl/miller/v6/pkg/runtime"
|
||||
|
||||
"github.com/johnkerl/pgpg/go/lib/pkg/asts"
|
||||
"github.com/johnkerl/pgpg/go/lib/pkg/tokens"
|
||||
)
|
||||
|
||||
type CondBlockNode struct {
|
||||
conditionNode IEvaluable
|
||||
conditionToken *token.Token
|
||||
conditionToken *tokens.Token
|
||||
statementBlockNode *StatementBlockNode
|
||||
}
|
||||
|
||||
// Sample AST:
|
||||
|
||||
func (root *RootNode) BuildCondBlockNode(astNode *dsl.ASTNode) (*CondBlockNode, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeCondBlock)
|
||||
func (root *RootNode) BuildCondBlockNode(astNode *asts.ASTNode) (*CondBlockNode, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeCondBlock))
|
||||
lib.InternalCodingErrorIf(len(astNode.Children) != 2)
|
||||
|
||||
conditionNode, err := root.BuildEvaluableNode(astNode.Children[0])
|
||||
|
|
@ -58,7 +59,7 @@ func (node *CondBlockNode) Execute(
|
|||
} else if !isBool {
|
||||
return nil, fmt.Errorf(
|
||||
"conditional expression did not evaluate to boolean%s",
|
||||
dsl.TokenToLocationInfo(node.conditionToken),
|
||||
pgpgTokenToLocationInfo(node.conditionToken),
|
||||
)
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -19,11 +19,11 @@ import (
|
|||
"os"
|
||||
"strings"
|
||||
|
||||
"github.com/johnkerl/miller/v6/pkg/dsl"
|
||||
"github.com/johnkerl/miller/v6/pkg/lib"
|
||||
"github.com/johnkerl/miller/v6/pkg/output"
|
||||
"github.com/johnkerl/miller/v6/pkg/runtime"
|
||||
"github.com/johnkerl/miller/v6/pkg/types"
|
||||
"github.com/johnkerl/pgpg/go/lib/pkg/asts"
|
||||
)
|
||||
|
||||
type tDumpToRedirectFunc func(
|
||||
|
|
@ -38,16 +38,16 @@ type DumpStatementNode struct {
|
|||
outputHandlerManager output.OutputHandlerManager // for file/pipe targets
|
||||
}
|
||||
|
||||
func (root *RootNode) BuildDumpStatementNode(astNode *dsl.ASTNode) (IExecutable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeDumpStatement)
|
||||
func (root *RootNode) BuildDumpStatementNode(astNode *asts.ASTNode) (IExecutable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeDumpStatement))
|
||||
return root.buildDumpxStatementNode(
|
||||
astNode,
|
||||
os.Stdout,
|
||||
)
|
||||
}
|
||||
|
||||
func (root *RootNode) BuildEdumpStatementNode(astNode *dsl.ASTNode) (IExecutable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeEdumpStatement)
|
||||
func (root *RootNode) BuildEdumpStatementNode(astNode *asts.ASTNode) (IExecutable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeEdumpStatement))
|
||||
return root.buildDumpxStatementNode(
|
||||
astNode,
|
||||
os.Stderr,
|
||||
|
|
@ -57,24 +57,55 @@ func (root *RootNode) BuildEdumpStatementNode(astNode *dsl.ASTNode) (IExecutable
|
|||
// Common code for building dump/edump nodes
|
||||
|
||||
func (root *RootNode) buildDumpxStatementNode(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
defaultOutputStream *os.File,
|
||||
) (IExecutable, error) {
|
||||
lib.InternalCodingErrorIf(len(astNode.Children) != 2)
|
||||
expressionsNode := astNode.Children[0]
|
||||
redirectorNode := astNode.Children[1]
|
||||
// Normalize PGPG AST to 2-child layout (expressions, redirector).
|
||||
// PGPG produces: 0 children (dump), 1 child (dump s or dump > file), 2+ children (dump s,t or dump > file, s).
|
||||
// With "dump Redirector comma FcnArgs", children are [Redirector, FcnArgs].
|
||||
// With "dump FcnArgs" (with_adopted_grandchildren), children are the Rvalues directly.
|
||||
var expressionsNode, redirectorNode *asts.ASTNode
|
||||
isRedirector := func(n *asts.ASTNode) bool {
|
||||
return n.Type == asts.NodeType(NodeTypeRedirectWrite) ||
|
||||
n.Type == asts.NodeType(NodeTypeRedirectAppend) ||
|
||||
n.Type == asts.NodeType(NodeTypeRedirectPipe)
|
||||
}
|
||||
switch len(astNode.Children) {
|
||||
case 0:
|
||||
expressionsNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeNoOp), nil)
|
||||
redirectorNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeNoOp), nil)
|
||||
case 1:
|
||||
if isRedirector(astNode.Children[0]) {
|
||||
expressionsNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeNoOp), nil)
|
||||
redirectorNode = astNode.Children[0]
|
||||
} else {
|
||||
expressionsNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeFcnArgs), astNode.Children)
|
||||
redirectorNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeNoOp), nil)
|
||||
}
|
||||
default:
|
||||
if isRedirector(astNode.Children[0]) {
|
||||
// dump Redirector comma FcnArgs -> [Redirector, FcnArgs]
|
||||
expressionsNode = astNode.Children[1]
|
||||
redirectorNode = astNode.Children[0]
|
||||
} else {
|
||||
// dump FcnArgs with_adopted_grandchildren -> [Rvalue, Rvalue, ...]
|
||||
expressionsNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeFcnArgs), astNode.Children)
|
||||
redirectorNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeNoOp), nil)
|
||||
}
|
||||
}
|
||||
|
||||
// - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
|
||||
// Things to be dumped, e.g. $a and $b in 'dump > "foo.dat", $a, $b'.
|
||||
|
||||
var expressionEvaluables []IEvaluable = nil
|
||||
|
||||
if expressionsNode.Type == dsl.NodeTypeNoOp {
|
||||
if expressionsNode.Type == asts.NodeType(NodeTypeNoOp) {
|
||||
// Just 'dump' without 'dump $something'
|
||||
expressionEvaluables = make([]IEvaluable, 1)
|
||||
expressionEvaluable := root.BuildFullOosvarRvalueNode()
|
||||
expressionEvaluables[0] = expressionEvaluable
|
||||
} else if expressionsNode.Type == dsl.NodeTypeFunctionCallsite {
|
||||
} else if expressionsNode.Type == asts.NodeType(NodeTypeFunctionCallsite) ||
|
||||
expressionsNode.Type == asts.NodeType(NodeTypeFcnArgs) {
|
||||
expressionEvaluables = make([]IEvaluable, len(expressionsNode.Children))
|
||||
for i, childNode := range expressionsNode.Children {
|
||||
expressionEvaluable, err := root.BuildEvaluableNode(childNode)
|
||||
|
|
@ -97,7 +128,7 @@ func (root *RootNode) buildDumpxStatementNode(
|
|||
outputHandlerManager: nil,
|
||||
}
|
||||
|
||||
if redirectorNode.Type == dsl.NodeTypeNoOp {
|
||||
if redirectorNode.Type == asts.NodeType(NodeTypeNoOp) {
|
||||
// No > >> or | was provided.
|
||||
if defaultOutputStream == os.Stdout {
|
||||
retval.dumpToRedirectFunc = retval.dumpToStdout
|
||||
|
|
@ -113,23 +144,27 @@ func (root *RootNode) buildDumpxStatementNode(
|
|||
redirectorTargetNode := redirectorNode.Children[0]
|
||||
var err error
|
||||
|
||||
if redirectorTargetNode.Type == dsl.NodeTypeRedirectTargetStdout {
|
||||
if redirectorTargetNode.Type == asts.NodeType(NodeTypeRedirectTargetStdout) {
|
||||
retval.dumpToRedirectFunc = retval.dumpToStdout
|
||||
} else if redirectorTargetNode.Type == dsl.NodeTypeRedirectTargetStderr {
|
||||
} else if redirectorTargetNode.Type == asts.NodeType(NodeTypeRedirectTargetStderr) {
|
||||
retval.dumpToRedirectFunc = retval.dumpToStderr
|
||||
} else {
|
||||
retval.dumpToRedirectFunc = retval.dumpToFileOrPipe
|
||||
|
||||
retval.redirectorTargetEvaluable, err = root.BuildEvaluableNode(redirectorTargetNode)
|
||||
targetNode := redirectorTargetNode
|
||||
if redirectorTargetNode.Type == asts.NodeType(NodeTypeRedirectTargetRvalue) &&
|
||||
redirectorTargetNode.Children != nil && len(redirectorTargetNode.Children) > 0 {
|
||||
targetNode = redirectorTargetNode.Children[0]
|
||||
}
|
||||
retval.redirectorTargetEvaluable, err = root.BuildEvaluableNode(targetNode)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if redirectorNode.Type == dsl.NodeTypeRedirectWrite {
|
||||
if redirectorNode.Type == asts.NodeType(NodeTypeRedirectWrite) {
|
||||
retval.outputHandlerManager = output.NewFileWritetHandlerManager(root.recordWriterOptions)
|
||||
} else if redirectorNode.Type == dsl.NodeTypeRedirectAppend {
|
||||
} else if redirectorNode.Type == asts.NodeType(NodeTypeRedirectAppend) {
|
||||
retval.outputHandlerManager = output.NewFileAppendHandlerManager(root.recordWriterOptions)
|
||||
} else if redirectorNode.Type == dsl.NodeTypeRedirectPipe {
|
||||
} else if redirectorNode.Type == asts.NodeType(NodeTypeRedirectPipe) {
|
||||
retval.outputHandlerManager = output.NewPipeWriteHandlerManager(root.recordWriterOptions)
|
||||
} else {
|
||||
return nil, fmt.Errorf("unhandled redirector node type %s", string(redirectorNode.Type))
|
||||
|
|
|
|||
|
|
@ -20,23 +20,23 @@ package cst
|
|||
import (
|
||||
"fmt"
|
||||
|
||||
"github.com/johnkerl/miller/v6/pkg/dsl"
|
||||
"github.com/johnkerl/miller/v6/pkg/lib"
|
||||
"github.com/johnkerl/miller/v6/pkg/runtime"
|
||||
"github.com/johnkerl/miller/v6/pkg/types"
|
||||
"github.com/johnkerl/pgpg/go/lib/pkg/asts"
|
||||
)
|
||||
|
||||
type Emit1StatementNode struct {
|
||||
evaluable IEvaluable
|
||||
}
|
||||
|
||||
func (root *RootNode) BuildEmit1StatementNode(astNode *dsl.ASTNode) (IExecutable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeEmit1Statement)
|
||||
func (root *RootNode) BuildEmit1StatementNode(astNode *asts.ASTNode) (IExecutable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeEmit1Statement))
|
||||
return root.buildEmit1StatementNode(astNode, false)
|
||||
}
|
||||
|
||||
func (root *RootNode) buildEmit1StatementNode(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
isEmitP bool,
|
||||
) (IExecutable, error) {
|
||||
lib.InternalCodingErrorIf(len(astNode.Children) != 1)
|
||||
|
|
|
|||
|
|
@ -40,12 +40,12 @@ import (
|
|||
"fmt"
|
||||
|
||||
"github.com/johnkerl/miller/v6/pkg/cli"
|
||||
"github.com/johnkerl/miller/v6/pkg/dsl"
|
||||
"github.com/johnkerl/miller/v6/pkg/lib"
|
||||
"github.com/johnkerl/miller/v6/pkg/mlrval"
|
||||
"github.com/johnkerl/miller/v6/pkg/output"
|
||||
"github.com/johnkerl/miller/v6/pkg/runtime"
|
||||
"github.com/johnkerl/miller/v6/pkg/types"
|
||||
"github.com/johnkerl/pgpg/go/lib/pkg/asts"
|
||||
)
|
||||
|
||||
// Shared by emit and emitp
|
||||
|
|
@ -94,40 +94,184 @@ type EmitXStatementNode struct {
|
|||
flatsep string
|
||||
}
|
||||
|
||||
func (root *RootNode) BuildEmitStatementNode(astNode *dsl.ASTNode) (IExecutable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeEmitStatement)
|
||||
func (root *RootNode) BuildEmitStatementNode(astNode *asts.ASTNode) (IExecutable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeEmitStatement))
|
||||
return root.buildEmitXStatementNode(astNode, false)
|
||||
}
|
||||
func (root *RootNode) BuildEmitPStatementNode(astNode *dsl.ASTNode) (IExecutable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeEmitPStatement)
|
||||
func (root *RootNode) BuildEmitPStatementNode(astNode *asts.ASTNode) (IExecutable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeEmitPStatement))
|
||||
return root.buildEmitXStatementNode(astNode, true)
|
||||
}
|
||||
|
||||
var EMITX_NAMED_NODE_TYPES = map[dsl.TNodeType]bool{
|
||||
dsl.NodeTypeLocalVariable: true,
|
||||
dsl.NodeTypeDirectOosvarValue: true,
|
||||
dsl.NodeTypeIndirectOosvarValue: true,
|
||||
dsl.NodeTypeDirectFieldValue: true,
|
||||
dsl.NodeTypeIndirectFieldValue: true,
|
||||
func allChildrenAreNamedNodes(children []*asts.ASTNode) bool {
|
||||
for _, c := range children {
|
||||
if !EMITX_NAMED_NODE_TYPES[c.Type] && !EMITX_NAMELESS_NODE_TYPES[c.Type] {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return len(children) > 0
|
||||
}
|
||||
|
||||
var EMITX_NAMELESS_NODE_TYPES = map[dsl.TNodeType]bool{
|
||||
dsl.NodeTypeFullSrec: true,
|
||||
dsl.NodeTypeFullOosvar: true,
|
||||
dsl.NodeTypeMapLiteral: true,
|
||||
var EMITX_NAMED_NODE_TYPES = map[asts.NodeType]bool{
|
||||
asts.NodeType(NodeTypeLocalVariable): true,
|
||||
asts.NodeType(NodeTypeDirectOosvarValue): true,
|
||||
asts.NodeType(NodeTypeIndirectOosvarValue): true,
|
||||
asts.NodeType(NodeTypeBracedOosvarValue): true, // @{variable.name}
|
||||
asts.NodeType(NodeTypeDirectFieldValue): true,
|
||||
asts.NodeType(NodeTypeIndirectFieldValue): true,
|
||||
asts.NodeType(NodeTypeArrayOrMapIndexAccess): true, // $x[1], @a[111], etc.
|
||||
asts.NodeType(NodeTypeDotOperator): true, // $a.$b (string concat or map access)
|
||||
asts.NodeType(NodeTypeFunctionCallsite): true,
|
||||
}
|
||||
|
||||
var EMITX_NAMELESS_NODE_TYPES = map[asts.NodeType]bool{
|
||||
asts.NodeType(NodeTypeFullSrec): true,
|
||||
asts.NodeType(NodeTypeFullOosvar): true,
|
||||
asts.NodeType(NodeTypeMapLiteral): true,
|
||||
}
|
||||
|
||||
// emitKeyName extracts the key name for emit/emitp output. Strips leading $ or @
|
||||
// and for braced forms strips ${ } or @{ } so that @sum emits as "sum", ${x+y} as "x+y".
|
||||
func emitKeyName(childNode *asts.ASTNode) string {
|
||||
// Walk to base for ArrayOrMapIndexAccess/DotOperator (e.g. @v[1][1] -> "v")
|
||||
walker := childNode
|
||||
for walker != nil &&
|
||||
(walker.Type == asts.NodeType(NodeTypeArrayOrMapIndexAccess) ||
|
||||
walker.Type == asts.NodeType(NodeTypeDotOperator)) &&
|
||||
walker.Children != nil && len(walker.Children) > 0 {
|
||||
walker = walker.Children[0]
|
||||
}
|
||||
if walker != nil {
|
||||
childNode = walker
|
||||
}
|
||||
|
||||
var s string
|
||||
if childNode.Type == asts.NodeType(NodeTypeBracedFieldValue) ||
|
||||
childNode.Type == asts.NodeType(NodeTypeBracedOosvarValue) {
|
||||
s = tokenLitStripBraced(childNode)
|
||||
} else {
|
||||
s = tokenLitStripDollarOrAt(childNode)
|
||||
}
|
||||
if s == "" && childNode.Children != nil && len(childNode.Children) > 0 {
|
||||
s = tokenLitStripDollarOrAt(childNode.Children[0])
|
||||
}
|
||||
if s == "" {
|
||||
s = tokenLit(childNode)
|
||||
}
|
||||
if len(s) >= 1 && (s[0] == '$' || s[0] == '@') {
|
||||
return s[1:]
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// EMIT AND EMITP
|
||||
|
||||
func (root *RootNode) buildEmitXStatementNode(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
isEmitP bool,
|
||||
) (IExecutable, error) {
|
||||
lib.InternalCodingErrorIf(len(astNode.Children) != 3)
|
||||
|
||||
emittablesNode := astNode.Children[0]
|
||||
keysNode := astNode.Children[1]
|
||||
redirectorNode := astNode.Children[2]
|
||||
// Normalize PGPG AST to 3-child layout: emittables, keys, redirector.
|
||||
// PGPG produces: 1 child (emitp @s), 2 children (emitp > file, @s OR emitp @s, "key"),
|
||||
// or 3+ adopted children (emitp @s, "k1", "k2").
|
||||
var emittablesNode, keysNode, redirectorNode *asts.ASTNode
|
||||
isRedirector := func(n *asts.ASTNode) bool {
|
||||
return n.Type == asts.NodeType(NodeTypeRedirectWrite) ||
|
||||
n.Type == asts.NodeType(NodeTypeRedirectAppend) ||
|
||||
n.Type == asts.NodeType(NodeTypeRedirectPipe)
|
||||
}
|
||||
switch len(astNode.Children) {
|
||||
case 1:
|
||||
child := astNode.Children[0]
|
||||
if child.Type == asts.NodeType(NodeTypeFcnArgs) && child.Children != nil && len(child.Children) >= 1 {
|
||||
// PGPG gave us FcnArgs directly: [emittable], [emittable, emittable, ...] (lashed), or
|
||||
// [emittable, key1, key2, ...]
|
||||
if len(child.Children) == 1 {
|
||||
emittablesNode = child
|
||||
keysNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeNoOp), nil)
|
||||
} else if len(child.Children) >= 2 &&
|
||||
(EMITX_NAMED_NODE_TYPES[child.Children[0].Type] || EMITX_NAMELESS_NODE_TYPES[child.Children[0].Type]) &&
|
||||
!EMITX_NAMED_NODE_TYPES[child.Children[1].Type] {
|
||||
// First is emittable, rest are index keys
|
||||
emittablesNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeFcnArgs), []*asts.ASTNode{child.Children[0]})
|
||||
keysNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeEmitKeys), child.Children[1:])
|
||||
} else {
|
||||
emittablesNode = child
|
||||
keysNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeNoOp), nil)
|
||||
}
|
||||
redirectorNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeNoOp), nil)
|
||||
} else {
|
||||
// PGPG: kw_emitp FcnArgs with with_adopted_grandchildren -> single child is emittable
|
||||
emittablesNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeFcnArgs), []*asts.ASTNode{child})
|
||||
keysNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeNoOp), nil)
|
||||
redirectorNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeNoOp), nil)
|
||||
}
|
||||
case 2:
|
||||
if isRedirector(astNode.Children[0]) {
|
||||
// PGPG: kw_emit Redirector comma FcnArgs -> children: [Redirector, FcnArgs]
|
||||
// FcnArgs may be [emittable], [emittable, emittable, ...] (lashed), or [emittable, key1, key2, ...]
|
||||
fcnArgs := astNode.Children[1]
|
||||
if fcnArgs.Type == asts.NodeType(NodeTypeFcnArgs) && fcnArgs.Children != nil && len(fcnArgs.Children) >= 2 &&
|
||||
(EMITX_NAMED_NODE_TYPES[fcnArgs.Children[0].Type] || EMITX_NAMELESS_NODE_TYPES[fcnArgs.Children[0].Type]) &&
|
||||
!EMITX_NAMED_NODE_TYPES[fcnArgs.Children[1].Type] {
|
||||
// First is emittable, rest are index keys (string literals)
|
||||
emittablesNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeFcnArgs), []*asts.ASTNode{fcnArgs.Children[0]})
|
||||
keysNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeEmitKeys), fcnArgs.Children[1:])
|
||||
} else {
|
||||
emittablesNode = fcnArgs
|
||||
keysNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeNoOp), nil)
|
||||
}
|
||||
redirectorNode = astNode.Children[0]
|
||||
} else if astNode.Children[0].Type == asts.NodeType(NodeTypeFcnArgs) &&
|
||||
astNode.Children[0].Children != nil && len(astNode.Children[0].Children) >= 2 &&
|
||||
astNode.Children[1].Type == asts.NodeType(NodeTypeFcnArgs) {
|
||||
// PGPG: kw_emitp FcnArgsParen comma FcnArgs -> [emittables, keys]
|
||||
emittablesNode = astNode.Children[0]
|
||||
keysNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeEmitKeys), astNode.Children[1].Children)
|
||||
redirectorNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeNoOp), nil)
|
||||
} else if EMITX_NAMED_NODE_TYPES[astNode.Children[0].Type] && EMITX_NAMED_NODE_TYPES[astNode.Children[1].Type] {
|
||||
// PGPG: kw_emitp FcnArgsParen -> lashed emitp (@a, @b), both are emittables
|
||||
emittablesNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeFcnArgs), astNode.Children)
|
||||
keysNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeNoOp), nil)
|
||||
redirectorNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeNoOp), nil)
|
||||
} else {
|
||||
// PGPG: kw_emitp FcnArgs with adoption -> [emittable, key]; first is emittable, rest are keys
|
||||
emittablesNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeFcnArgs), []*asts.ASTNode{astNode.Children[0]})
|
||||
keysNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeEmitKeys), []*asts.ASTNode{astNode.Children[1]})
|
||||
redirectorNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeNoOp), nil)
|
||||
}
|
||||
default:
|
||||
if isRedirector(astNode.Children[0]) {
|
||||
// len 3: emit Redirector comma FcnArgsParen comma FcnArgs -> [Redirector, emittables, keys]
|
||||
// len 2: emit Redirector comma FcnArgs -> [Redirector, FcnArgs]; split if FcnArgs=[emittable, keys...]
|
||||
if len(astNode.Children) >= 3 && astNode.Children[2].Type == asts.NodeType(NodeTypeFcnArgs) &&
|
||||
astNode.Children[2].Children != nil {
|
||||
emittablesNode = astNode.Children[1]
|
||||
keysNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeEmitKeys), astNode.Children[2].Children)
|
||||
} else {
|
||||
fcnArgs := astNode.Children[1]
|
||||
if fcnArgs.Type == asts.NodeType(NodeTypeFcnArgs) && fcnArgs.Children != nil && len(fcnArgs.Children) >= 2 &&
|
||||
(EMITX_NAMED_NODE_TYPES[fcnArgs.Children[0].Type] || EMITX_NAMELESS_NODE_TYPES[fcnArgs.Children[0].Type]) &&
|
||||
!EMITX_NAMED_NODE_TYPES[fcnArgs.Children[1].Type] {
|
||||
emittablesNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeFcnArgs), []*asts.ASTNode{fcnArgs.Children[0]})
|
||||
keysNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeEmitKeys), fcnArgs.Children[1:])
|
||||
} else {
|
||||
emittablesNode = fcnArgs
|
||||
keysNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeNoOp), nil)
|
||||
}
|
||||
}
|
||||
redirectorNode = astNode.Children[0]
|
||||
} else if allChildrenAreNamedNodes(astNode.Children) {
|
||||
// PGPG: kw_emitp FcnArgsParen with 3+ args -> lashed emitp (@a, @b, @c)
|
||||
emittablesNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeFcnArgs), astNode.Children)
|
||||
keysNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeNoOp), nil)
|
||||
redirectorNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeNoOp), nil)
|
||||
} else {
|
||||
// emitp FcnArgs with adoption -> [emittable, key1, key2, ...]
|
||||
emittablesNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeFcnArgs), []*asts.ASTNode{astNode.Children[0]})
|
||||
keysNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeEmitKeys), astNode.Children[1:])
|
||||
redirectorNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeNoOp), nil)
|
||||
}
|
||||
}
|
||||
|
||||
retval := &EmitXStatementNode{
|
||||
isEmitP: isEmitP,
|
||||
|
|
@ -144,10 +288,18 @@ func (root *RootNode) buildEmitXStatementNode(
|
|||
lib.InternalCodingErrorIf(len(emittablesNode.Children) < 1)
|
||||
if len(emittablesNode.Children) == 1 {
|
||||
childNode := emittablesNode.Children[0]
|
||||
// Unwrap Parenthesized: emitp (@a) parses as Parenthesized containing @a
|
||||
if childNode.Type == asts.NodeType(NodeTypeParenthesized) &&
|
||||
childNode.Children != nil && len(childNode.Children) == 1 {
|
||||
childNode = childNode.Children[0]
|
||||
}
|
||||
|
||||
if EMITX_NAMED_NODE_TYPES[childNode.Type] {
|
||||
if childNode.Type == asts.NodeType(NodeTypeLocalVariable) && tokenLit(childNode) == "all" {
|
||||
// "emit all" / "emitp all" means emit all out-of-stream variables (same as @*)
|
||||
retval.topLevelEvaluableMap = root.BuildFullOosvarRvalueNode()
|
||||
} else if EMITX_NAMED_NODE_TYPES[childNode.Type] {
|
||||
retval.topLevelNameList = make([]string, 1)
|
||||
retval.topLevelNameList[0] = string(childNode.Token.Lit)
|
||||
retval.topLevelNameList[0] = emitKeyName(childNode)
|
||||
|
||||
retval.topLevelEvaluableList = make([]IEvaluable, 1)
|
||||
evaluable, err := root.BuildEvaluableNode(childNode)
|
||||
|
|
@ -173,9 +325,9 @@ func (root *RootNode) buildEmitXStatementNode(
|
|||
} else {
|
||||
retval.isLashed = true
|
||||
for _, childNode := range emittablesNode.Children {
|
||||
if !EMITX_NAMED_NODE_TYPES[childNode.Type] {
|
||||
if !EMITX_NAMED_NODE_TYPES[childNode.Type] && !EMITX_NAMELESS_NODE_TYPES[childNode.Type] {
|
||||
return nil, fmt.Errorf(
|
||||
"lashed-emit node types must be local variables, field names, or oosvars; got %s",
|
||||
"lashed-emit node types must be local variables, field names, oosvars, or maps; got %s",
|
||||
childNode.Type,
|
||||
)
|
||||
}
|
||||
|
|
@ -184,7 +336,7 @@ func (root *RootNode) buildEmitXStatementNode(
|
|||
retval.topLevelNameList = make([]string, len(emittablesNode.Children))
|
||||
retval.topLevelEvaluableList = make([]IEvaluable, len(emittablesNode.Children))
|
||||
for i, childNode := range emittablesNode.Children {
|
||||
retval.topLevelNameList[i] = string(childNode.Token.Lit)
|
||||
retval.topLevelNameList[i] = emitKeyName(childNode)
|
||||
evaluable, err := root.BuildEvaluableNode(childNode)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
|
|
@ -197,8 +349,8 @@ func (root *RootNode) buildEmitXStatementNode(
|
|||
// Indices (if any) on the emittables
|
||||
|
||||
isIndexed := false
|
||||
if keysNode.Type != dsl.NodeTypeNoOp { // There are "x","y" present
|
||||
lib.InternalCodingErrorIf(keysNode.Type != dsl.NodeTypeEmitKeys)
|
||||
if keysNode.Type != asts.NodeType(NodeTypeNoOp) { // There are "x","y" present
|
||||
lib.InternalCodingErrorIf(keysNode.Type != asts.NodeType(NodeTypeEmitKeys))
|
||||
isIndexed = true
|
||||
numKeys := len(keysNode.Children)
|
||||
retval.indexEvaluables = make([]IEvaluable, numKeys)
|
||||
|
|
@ -232,7 +384,7 @@ func (root *RootNode) buildEmitXStatementNode(
|
|||
// - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
|
||||
// Redirections and redirection targets (the thing after > >> |, if any).
|
||||
|
||||
if redirectorNode.Type == dsl.NodeTypeNoOp {
|
||||
if redirectorNode.Type == asts.NodeType(NodeTypeNoOp) {
|
||||
// No > >> or | was provided.
|
||||
retval.emitToRedirectFunc = retval.emitRecordToRecordStream
|
||||
} else {
|
||||
|
|
@ -242,27 +394,31 @@ func (root *RootNode) buildEmitXStatementNode(
|
|||
redirectorTargetNode := redirectorNode.Children[0]
|
||||
var err error
|
||||
|
||||
if redirectorTargetNode.Type == dsl.NodeTypeRedirectTargetStdout {
|
||||
if redirectorTargetNode.Type == asts.NodeType(NodeTypeRedirectTargetStdout) {
|
||||
retval.emitToRedirectFunc = retval.emitRecordToFileOrPipe
|
||||
retval.outputHandlerManager = output.NewStdoutWriteHandlerManager(root.recordWriterOptions)
|
||||
retval.redirectorTargetEvaluable = root.BuildStringLiteralNode("(stdout)")
|
||||
} else if redirectorTargetNode.Type == dsl.NodeTypeRedirectTargetStderr {
|
||||
} else if redirectorTargetNode.Type == asts.NodeType(NodeTypeRedirectTargetStderr) {
|
||||
retval.emitToRedirectFunc = retval.emitRecordToFileOrPipe
|
||||
retval.outputHandlerManager = output.NewStderrWriteHandlerManager(root.recordWriterOptions)
|
||||
retval.redirectorTargetEvaluable = root.BuildStringLiteralNode("(stderr)")
|
||||
} else {
|
||||
retval.emitToRedirectFunc = retval.emitRecordToFileOrPipe
|
||||
|
||||
retval.redirectorTargetEvaluable, err = root.BuildEvaluableNode(redirectorTargetNode)
|
||||
targetNode := redirectorTargetNode
|
||||
if redirectorTargetNode.Type == asts.NodeType(NodeTypeRedirectTargetRvalue) &&
|
||||
redirectorTargetNode.Children != nil && len(redirectorTargetNode.Children) > 0 {
|
||||
targetNode = redirectorTargetNode.Children[0]
|
||||
}
|
||||
retval.redirectorTargetEvaluable, err = root.BuildEvaluableNode(targetNode)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if redirectorNode.Type == dsl.NodeTypeRedirectWrite {
|
||||
if redirectorNode.Type == asts.NodeType(NodeTypeRedirectWrite) {
|
||||
retval.outputHandlerManager = output.NewFileWritetHandlerManager(root.recordWriterOptions)
|
||||
} else if redirectorNode.Type == dsl.NodeTypeRedirectAppend {
|
||||
} else if redirectorNode.Type == asts.NodeType(NodeTypeRedirectAppend) {
|
||||
retval.outputHandlerManager = output.NewFileAppendHandlerManager(root.recordWriterOptions)
|
||||
} else if redirectorNode.Type == dsl.NodeTypeRedirectPipe {
|
||||
} else if redirectorNode.Type == asts.NodeType(NodeTypeRedirectPipe) {
|
||||
retval.outputHandlerManager = output.NewPipeWriteHandlerManager(root.recordWriterOptions)
|
||||
} else {
|
||||
return nil, fmt.Errorf("unhandled redirector node type %s", string(redirectorNode.Type))
|
||||
|
|
|
|||
|
|
@ -6,12 +6,12 @@ package cst
|
|||
import (
|
||||
"fmt"
|
||||
|
||||
"github.com/johnkerl/miller/v6/pkg/dsl"
|
||||
"github.com/johnkerl/miller/v6/pkg/lib"
|
||||
"github.com/johnkerl/miller/v6/pkg/mlrval"
|
||||
"github.com/johnkerl/miller/v6/pkg/output"
|
||||
"github.com/johnkerl/miller/v6/pkg/runtime"
|
||||
"github.com/johnkerl/miller/v6/pkg/types"
|
||||
"github.com/johnkerl/pgpg/go/lib/pkg/asts"
|
||||
)
|
||||
|
||||
// Examples:
|
||||
|
|
@ -48,11 +48,27 @@ type EmitFStatementNode struct {
|
|||
// * direct oosvar value "c"
|
||||
// * no-op
|
||||
|
||||
func (root *RootNode) BuildEmitFStatementNode(astNode *dsl.ASTNode) (IExecutable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeEmitFStatement)
|
||||
lib.InternalCodingErrorIf(len(astNode.Children) != 2)
|
||||
expressionsNode := astNode.Children[0]
|
||||
redirectorNode := astNode.Children[1]
|
||||
func (root *RootNode) BuildEmitFStatementNode(astNode *asts.ASTNode) (IExecutable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeEmitFStatement))
|
||||
|
||||
// Normalize PGPG AST to 2-child layout (expressions, redirector).
|
||||
// PGPG produces: 1+ children (emitf @a,@b,@c from with_adopted_grandchildren)
|
||||
// or 2 children (emitf > file, @a,@b from children [Redirector, FcnArgs]).
|
||||
var expressionsNode, redirectorNode *asts.ASTNode
|
||||
switch {
|
||||
case len(astNode.Children) >= 2 && astNode.Children[0].Type == asts.NodeType(NodeTypeRedirectWrite) ||
|
||||
astNode.Children[0].Type == asts.NodeType(NodeTypeRedirectAppend) ||
|
||||
astNode.Children[0].Type == asts.NodeType(NodeTypeRedirectPipe):
|
||||
// Redirector variant: [Redirector, FcnArgs]
|
||||
expressionsNode = astNode.Children[1]
|
||||
redirectorNode = astNode.Children[0]
|
||||
case len(astNode.Children) >= 1:
|
||||
// No redirector: all children are emittables from FcnArgs
|
||||
expressionsNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeFcnArgs), astNode.Children)
|
||||
redirectorNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeNoOp), nil)
|
||||
default:
|
||||
lib.InternalCodingErrorIf(true) // unexpected child count
|
||||
}
|
||||
|
||||
// - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
|
||||
// Things to be emitted, e.g. @a and @b in 'emitf > "foo.dat", @a, @b'.
|
||||
|
|
@ -82,7 +98,7 @@ func (root *RootNode) BuildEmitFStatementNode(astNode *dsl.ASTNode) (IExecutable
|
|||
// - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
|
||||
// Redirection targets (the thing after > >> |, if any).
|
||||
|
||||
if redirectorNode.Type == dsl.NodeTypeNoOp {
|
||||
if redirectorNode.Type == asts.NodeType(NodeTypeNoOp) {
|
||||
// No > >> or | was provided.
|
||||
retval.emitfToRedirectFunc = retval.emitfToRecordStream
|
||||
} else {
|
||||
|
|
@ -92,27 +108,31 @@ func (root *RootNode) BuildEmitFStatementNode(astNode *dsl.ASTNode) (IExecutable
|
|||
redirectorTargetNode := redirectorNode.Children[0]
|
||||
var err error
|
||||
|
||||
if redirectorTargetNode.Type == dsl.NodeTypeRedirectTargetStdout {
|
||||
if redirectorTargetNode.Type == asts.NodeType(NodeTypeRedirectTargetStdout) {
|
||||
retval.emitfToRedirectFunc = retval.emitfToFileOrPipe
|
||||
retval.outputHandlerManager = output.NewStdoutWriteHandlerManager(root.recordWriterOptions)
|
||||
retval.redirectorTargetEvaluable = root.BuildStringLiteralNode("(stdout)")
|
||||
} else if redirectorTargetNode.Type == dsl.NodeTypeRedirectTargetStderr {
|
||||
} else if redirectorTargetNode.Type == asts.NodeType(NodeTypeRedirectTargetStderr) {
|
||||
retval.emitfToRedirectFunc = retval.emitfToFileOrPipe
|
||||
retval.outputHandlerManager = output.NewStderrWriteHandlerManager(root.recordWriterOptions)
|
||||
retval.redirectorTargetEvaluable = root.BuildStringLiteralNode("(stderr)")
|
||||
} else {
|
||||
retval.emitfToRedirectFunc = retval.emitfToFileOrPipe
|
||||
|
||||
retval.redirectorTargetEvaluable, err = root.BuildEvaluableNode(redirectorTargetNode)
|
||||
targetNode := redirectorTargetNode
|
||||
if redirectorTargetNode.Type == asts.NodeType(NodeTypeRedirectTargetRvalue) &&
|
||||
redirectorTargetNode.Children != nil && len(redirectorTargetNode.Children) > 0 {
|
||||
targetNode = redirectorTargetNode.Children[0]
|
||||
}
|
||||
retval.redirectorTargetEvaluable, err = root.BuildEvaluableNode(targetNode)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if redirectorNode.Type == dsl.NodeTypeRedirectWrite {
|
||||
if redirectorNode.Type == asts.NodeType(NodeTypeRedirectWrite) {
|
||||
retval.outputHandlerManager = output.NewFileWritetHandlerManager(root.recordWriterOptions)
|
||||
} else if redirectorNode.Type == dsl.NodeTypeRedirectAppend {
|
||||
} else if redirectorNode.Type == asts.NodeType(NodeTypeRedirectAppend) {
|
||||
retval.outputHandlerManager = output.NewFileAppendHandlerManager(root.recordWriterOptions)
|
||||
} else if redirectorNode.Type == dsl.NodeTypeRedirectPipe {
|
||||
} else if redirectorNode.Type == asts.NodeType(NodeTypeRedirectPipe) {
|
||||
retval.outputHandlerManager = output.NewPipeWriteHandlerManager(root.recordWriterOptions)
|
||||
} else {
|
||||
return nil, fmt.Errorf("unhandled redirector node type %s", string(redirectorNode.Type))
|
||||
|
|
@ -150,13 +170,13 @@ func (node *EmitFStatementNode) Execute(state *runtime.State) (*BlockExitPayload
|
|||
//
|
||||
// TODO: support indirects like 'emitf @[x."_sum"]'
|
||||
|
||||
func getNameFromNamedNode(astNode *dsl.ASTNode, description string) (string, error) {
|
||||
if astNode.Type == dsl.NodeTypeDirectOosvarValue {
|
||||
return string(astNode.Token.Lit), nil
|
||||
} else if astNode.Type == dsl.NodeTypeLocalVariable {
|
||||
return string(astNode.Token.Lit), nil
|
||||
} else if astNode.Type == dsl.NodeTypeDirectFieldValue {
|
||||
return string(astNode.Token.Lit), nil
|
||||
func getNameFromNamedNode(astNode *asts.ASTNode, description string) (string, error) {
|
||||
if astNode.Type == asts.NodeType(NodeTypeDirectOosvarValue) {
|
||||
return tokenLitStripDollarOrAt(astNode), nil
|
||||
} else if astNode.Type == asts.NodeType(NodeTypeLocalVariable) {
|
||||
return tokenLit(astNode), nil
|
||||
} else if astNode.Type == asts.NodeType(NodeTypeDirectFieldValue) {
|
||||
return tokenLitStripDollarOrAt(astNode), nil
|
||||
}
|
||||
return "", fmt.Errorf(`can't get name of node type "%s" for %s`, string(astNode.Type), description)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -8,21 +8,55 @@ package cst
|
|||
import (
|
||||
"os"
|
||||
|
||||
"github.com/johnkerl/miller/v6/pkg/dsl"
|
||||
"github.com/johnkerl/miller/v6/pkg/lib"
|
||||
"github.com/johnkerl/miller/v6/pkg/mlrval"
|
||||
"github.com/johnkerl/miller/v6/pkg/runtime"
|
||||
"github.com/johnkerl/pgpg/go/lib/pkg/asts"
|
||||
)
|
||||
|
||||
type EnvironmentVariableNode struct {
|
||||
nameEvaluable IEvaluable
|
||||
}
|
||||
|
||||
func (root *RootNode) BuildEnvironmentVariableNode(astNode *dsl.ASTNode) (*EnvironmentVariableNode, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeEnvironmentVariable)
|
||||
func (root *RootNode) BuildEnvironmentVariableNode(astNode *asts.ASTNode) (*EnvironmentVariableNode, error) {
|
||||
lib.InternalCodingErrorWithMessageIf(
|
||||
astNode.Type != asts.NodeType(NodeTypeEnvironmentVariable) && string(astNode.Type) != NodeTypeEnvironmentVariable,
|
||||
"expected EnvironmentVariable node",
|
||||
)
|
||||
lib.InternalCodingErrorIf(astNode.Children == nil)
|
||||
lib.InternalCodingErrorIf(len(astNode.Children) != 1)
|
||||
nameEvaluable, err := root.BuildEvaluableNode(astNode.Children[0])
|
||||
child := astNode.Children[0]
|
||||
// Unwrap Parenthesized if present: ENV[("FOO")] or similar.
|
||||
if string(child.Type) == NodeTypeParenthesized && child.Children != nil && len(child.Children) == 1 {
|
||||
child = child.Children[0]
|
||||
}
|
||||
// ENV.FOO: child is non_sigil_name (identifier FOO) -> use as env var name directly
|
||||
if string(child.Type) == "non_sigil_name" || string(child.Type) == NodeTypeLocalVariable {
|
||||
sval := tokenLit(child)
|
||||
if sval != "" {
|
||||
return &EnvironmentVariableNode{
|
||||
nameEvaluable: root.BuildStringLiteralNode(sval),
|
||||
}, nil
|
||||
}
|
||||
}
|
||||
// PGPG may put the string literal's text on the node or its child; extract it directly
|
||||
// so we get the correct env var name even when BuildEvaluableNode would yield empty.
|
||||
if string(child.Type) == NodeTypeStringLiteral || string(child.Type) == "string_literal" {
|
||||
sval := tokenLit(child)
|
||||
if sval == "" && child.Children != nil && len(child.Children) == 1 {
|
||||
sval = tokenLit(child.Children[0])
|
||||
}
|
||||
if sval != "" {
|
||||
// PGPG lexer may include surrounding quotes in the lexeme; strip them.
|
||||
if len(sval) >= 2 && sval[0] == '"' && sval[len(sval)-1] == '"' {
|
||||
sval = sval[1 : len(sval)-1]
|
||||
}
|
||||
return &EnvironmentVariableNode{
|
||||
nameEvaluable: root.BuildStringLiteralNode(lib.UnbackslashStringLiteral(sval)),
|
||||
}, nil
|
||||
}
|
||||
}
|
||||
nameEvaluable, err := root.BuildEvaluableNode(child)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
|
|
|||
|
|
@ -8,64 +8,103 @@ import (
|
|||
"fmt"
|
||||
"os"
|
||||
|
||||
"github.com/johnkerl/miller/v6/pkg/dsl"
|
||||
"github.com/johnkerl/miller/v6/pkg/lib"
|
||||
"github.com/johnkerl/miller/v6/pkg/mlrval"
|
||||
"github.com/johnkerl/miller/v6/pkg/runtime"
|
||||
|
||||
"github.com/johnkerl/pgpg/go/lib/pkg/asts"
|
||||
)
|
||||
|
||||
func (root *RootNode) BuildEvaluableNode(astNode *dsl.ASTNode) (IEvaluable, error) {
|
||||
|
||||
if astNode.Children == nil {
|
||||
return root.BuildLeafNode(astNode)
|
||||
func (root *RootNode) BuildEvaluableNode(astNode *asts.ASTNode) (IEvaluable, error) {
|
||||
// Try BuildLeafNode first for terminals
|
||||
if astNode.Children == nil || len(astNode.Children) == 0 {
|
||||
if leaf, err := root.BuildLeafNode(astNode); err == nil {
|
||||
return leaf, nil
|
||||
}
|
||||
// Fall through to switch for leaf types BuildLeafNode doesn't know
|
||||
}
|
||||
|
||||
switch astNode.Type {
|
||||
|
||||
case dsl.NodeTypeArrayLiteral: // [...]
|
||||
case asts.NodeType(NodeTypeArrayLiteral): // [...]
|
||||
return root.BuildArrayLiteralNode(astNode)
|
||||
|
||||
case dsl.NodeTypeMapLiteral: // {...}
|
||||
case asts.NodeType(NodeTypeMapLiteral): // {...}
|
||||
return root.BuildMapLiteralNode(astNode)
|
||||
|
||||
case dsl.NodeTypeArrayOrMapIndexAccess: // x[...]
|
||||
case asts.NodeType(NodeTypeArrayOrMapIndexAccess): // x[...]
|
||||
return root.BuildArrayOrMapIndexAccessNode(astNode)
|
||||
|
||||
case dsl.NodeTypeArraySliceAccess: // myarray[lo:hi]
|
||||
case asts.NodeType(NodeTypeArraySliceLoHi), asts.NodeType(NodeTypeArraySliceHiOnly),
|
||||
asts.NodeType(NodeTypeArraySliceLoOnly), asts.NodeType(NodeTypeArraySliceFull): // myarray[lo:hi]
|
||||
return root.BuildArraySliceAccessNode(astNode)
|
||||
|
||||
case dsl.NodeTypeIndirectFieldValue: // $[...] (includes $[[n]] and $[[[n]]])
|
||||
case asts.NodeType(NodeTypeIndirectFieldValue): // $[...] (includes $[[n]] and $[[[n]]])
|
||||
return root.BuildIndirectFieldValueNode(astNode)
|
||||
case dsl.NodeTypeIndirectOosvarValue: // $[...]
|
||||
case asts.NodeType(NodeTypeIndirectOosvarValue): // $[...]
|
||||
return root.BuildIndirectOosvarValueNode(astNode)
|
||||
|
||||
case dsl.NodeTypeEnvironmentVariable: // ENV["NAME"]
|
||||
case asts.NodeType(NodeTypeEnvironmentVariable): // ENV["NAME"]
|
||||
return root.BuildEnvironmentVariableNode(astNode)
|
||||
|
||||
// Operators are just functions with infix syntax so we treat them like
|
||||
// functions in the CST. (The distinction between infix syntax, e.g.
|
||||
// '1+2', and prefix syntax, e.g. 'plus(1,2)' disappears post-parse -- both
|
||||
// parse to the same-shape AST.)
|
||||
case dsl.NodeTypeOperator:
|
||||
case asts.NodeType(NodeTypeOperator):
|
||||
return root.BuildFunctionCallsiteNode(astNode)
|
||||
case dsl.NodeTypeFunctionCallsite:
|
||||
case asts.NodeType(NodeTypeFunctionCallsite):
|
||||
return root.BuildFunctionCallsiteNode(astNode)
|
||||
|
||||
// The dot operator is a little different from other operators since it's
|
||||
// type-dependent: for strings/int/bools etc it's just concatenation of
|
||||
// string representations, but if the left-hand side is a map, it's a
|
||||
// key-lookup with an unquoted literal on the right. E.g. mymap.foo is the
|
||||
// same as mymap["foo"].
|
||||
case dsl.NodeTypeDotOperator:
|
||||
// The dot operator is a little different from other operators since it's
|
||||
// type-dependent: for strings/int/bools etc it's just concatenation of
|
||||
// string representations, but if the left-hand side is a map, it's a
|
||||
// key-lookup with an unquoted literal on the right. E.g. mymap.foo is the
|
||||
// same as mymap["foo"].
|
||||
case asts.NodeType(NodeTypeDotOperator):
|
||||
return root.BuildDotCallsiteNode(astNode)
|
||||
|
||||
case asts.NodeType(NodeTypeParenthesized):
|
||||
// (expr) — unwrap and build the inner expression
|
||||
lib.InternalCodingErrorIf(astNode.Children == nil || len(astNode.Children) != 1)
|
||||
return root.BuildEvaluableNode(astNode.Children[0])
|
||||
|
||||
// Function literals like 'func (a,b) { return b - a }'
|
||||
case dsl.NodeTypeUnnamedFunctionDefinition:
|
||||
case asts.NodeType(NodeTypeUnnamedFunctionDefinition):
|
||||
return root.BuildUnnamedUDFNode(astNode)
|
||||
|
||||
// Leaf/terminal types (PGPG may give them non-nil empty Children)
|
||||
case asts.NodeType(NodeTypeDirectFieldValue), asts.NodeType(NodeTypeBracedFieldValue),
|
||||
asts.NodeType(NodeTypeFullSrec), asts.NodeType(NodeTypeDirectOosvarValue),
|
||||
asts.NodeType(NodeTypeBracedOosvarValue), asts.NodeType(NodeTypeFullOosvar),
|
||||
asts.NodeType(NodeTypeLocalVariable),
|
||||
asts.NodeType(NodeTypeIntLiteral), asts.NodeType(NodeTypeFloatLiteral),
|
||||
asts.NodeType(NodeTypeStringLiteral), asts.NodeType(NodeTypeBoolLiteral),
|
||||
asts.NodeType(NodeTypeNullLiteral), asts.NodeType(NodeTypeRegex):
|
||||
return root.BuildLeafNode(astNode)
|
||||
}
|
||||
|
||||
// Parenthesized: (expr) — unwrap and build the inner expression.
|
||||
// Use string comparison in case asts.NodeType differs from astNode.Type's type.
|
||||
if string(astNode.Type) == NodeTypeParenthesized && astNode.Children != nil && len(astNode.Children) == 1 {
|
||||
return root.BuildEvaluableNode(astNode.Children[0])
|
||||
}
|
||||
|
||||
// EnvironmentVariable: ENV["FOO"] or ENV.FOO
|
||||
if string(astNode.Type) == NodeTypeEnvironmentVariable && astNode.Children != nil && len(astNode.Children) == 1 {
|
||||
return root.BuildEnvironmentVariableNode(astNode)
|
||||
}
|
||||
|
||||
// Fallback: try BuildLeafNode for unhandled types (e.g. DirectFieldValue, IntLiteral).
|
||||
// Only for leaf-like nodes (0 or 1 child); nodes with 2+ children are not leaves.
|
||||
if astNode.Children == nil || len(astNode.Children) <= 1 {
|
||||
if leaf, err := root.BuildLeafNode(astNode); err == nil {
|
||||
return leaf, nil
|
||||
}
|
||||
}
|
||||
|
||||
return nil, fmt.Errorf(
|
||||
"at CST BuildEvaluableNode: unhandled AST node type %s", string(astNode.Type),
|
||||
"at CST BuildEvaluableNode: unhandled AST node type %q (len=%d)", string(astNode.Type), len(astNode.Children),
|
||||
)
|
||||
}
|
||||
|
||||
|
|
@ -74,30 +113,24 @@ type IndirectFieldValueNode struct {
|
|||
}
|
||||
|
||||
func (root *RootNode) BuildIndirectFieldValueNode(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
) (IEvaluable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeIndirectFieldValue)
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeIndirectFieldValue))
|
||||
lib.InternalCodingErrorIf(astNode.Children == nil)
|
||||
lib.InternalCodingErrorIf(len(astNode.Children) != 1)
|
||||
|
||||
child := astNode.Children[0]
|
||||
if child.Type == dsl.NodeTypeArrayLiteral && len(child.Children) == 1 {
|
||||
if child.Type == asts.NodeType(NodeTypeArrayLiteral) && len(child.Children) == 1 {
|
||||
inner := child.Children[0]
|
||||
if inner.Type == dsl.NodeTypeArrayLiteral && len(inner.Children) == 1 {
|
||||
if inner.Type == asts.NodeType(NodeTypeArrayLiteral) && len(inner.Children) == 1 {
|
||||
// $[[[n]]] → positional field value
|
||||
indexASTNode := inner.Children[0]
|
||||
syntheticAST := &dsl.ASTNode{
|
||||
Type: dsl.NodeTypePositionalFieldValue,
|
||||
Children: []*dsl.ASTNode{indexASTNode},
|
||||
}
|
||||
syntheticAST := asts.NewASTNode(nil, asts.NodeType(NodeTypePositionalFieldValue), []*asts.ASTNode{indexASTNode})
|
||||
return root.BuildPositionalFieldValueNode(syntheticAST)
|
||||
}
|
||||
// $[[n]] → positional field name
|
||||
indexASTNode := inner
|
||||
syntheticAST := &dsl.ASTNode{
|
||||
Type: dsl.NodeTypePositionalFieldName,
|
||||
Children: []*dsl.ASTNode{indexASTNode},
|
||||
}
|
||||
syntheticAST := asts.NewASTNode(nil, asts.NodeType(NodeTypePositionalFieldName), []*asts.ASTNode{indexASTNode})
|
||||
return root.BuildPositionalFieldNameNode(syntheticAST)
|
||||
}
|
||||
|
||||
|
|
@ -145,9 +178,9 @@ type IndirectOosvarValueNode struct {
|
|||
}
|
||||
|
||||
func (root *RootNode) BuildIndirectOosvarValueNode(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
) (*IndirectOosvarValueNode, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeIndirectOosvarValue)
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeIndirectOosvarValue))
|
||||
lib.InternalCodingErrorIf(astNode.Children == nil)
|
||||
lib.InternalCodingErrorIf(len(astNode.Children) != 1)
|
||||
oosvarNameEvaluable, err := root.BuildEvaluableNode(astNode.Children[0])
|
||||
|
|
|
|||
|
|
@ -17,9 +17,9 @@
|
|||
package cst
|
||||
|
||||
import (
|
||||
"github.com/johnkerl/miller/v6/pkg/dsl"
|
||||
"github.com/johnkerl/miller/v6/pkg/lib"
|
||||
"github.com/johnkerl/miller/v6/pkg/runtime"
|
||||
"github.com/johnkerl/pgpg/go/lib/pkg/asts"
|
||||
)
|
||||
|
||||
// BareBooleanStatementNode is for implicit filter statements such as mlr
|
||||
|
|
@ -31,8 +31,8 @@ type BareBooleanStatementNode struct {
|
|||
writeToState bool
|
||||
}
|
||||
|
||||
func (root *RootNode) BuildBareBooleanStatementNode(astNode *dsl.ASTNode) (IExecutable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeBareBoolean)
|
||||
func (root *RootNode) BuildBareBooleanStatementNode(astNode *asts.ASTNode) (IExecutable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeBareBoolean))
|
||||
lib.InternalCodingErrorIf(len(astNode.Children) != 1)
|
||||
|
||||
bareBooleanEvaluable, err := root.BuildEvaluableNode(astNode.Children[0])
|
||||
|
|
@ -65,8 +65,8 @@ type FilterStatementNode struct {
|
|||
filterEvaluable IEvaluable
|
||||
}
|
||||
|
||||
func (root *RootNode) BuildFilterStatementNode(astNode *dsl.ASTNode) (IExecutable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeFilterStatement)
|
||||
func (root *RootNode) BuildFilterStatementNode(astNode *asts.ASTNode) (IExecutable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeFilterStatement))
|
||||
lib.InternalCodingErrorIf(len(astNode.Children) != 1)
|
||||
|
||||
filterEvaluable, err := root.BuildEvaluableNode(astNode.Children[0])
|
||||
|
|
|
|||
|
|
@ -5,11 +5,11 @@ package cst
|
|||
import (
|
||||
"fmt"
|
||||
|
||||
"github.com/johnkerl/miller/v6/pkg/dsl"
|
||||
"github.com/johnkerl/miller/v6/pkg/lib"
|
||||
"github.com/johnkerl/miller/v6/pkg/mlrval"
|
||||
"github.com/johnkerl/miller/v6/pkg/parsing/token"
|
||||
"github.com/johnkerl/miller/v6/pkg/runtime"
|
||||
"github.com/johnkerl/pgpg/go/lib/pkg/asts"
|
||||
"github.com/johnkerl/pgpg/go/lib/pkg/tokens"
|
||||
)
|
||||
|
||||
// Sample AST:
|
||||
|
|
@ -65,17 +65,17 @@ func NewForLoopOneVariableNode(
|
|||
// * LocalVariable "k"
|
||||
// * LocalVariable "v"
|
||||
|
||||
func (root *RootNode) BuildForLoopOneVariableNode(astNode *dsl.ASTNode) (*ForLoopOneVariableNode, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeForLoopOneVariable)
|
||||
func (root *RootNode) BuildForLoopOneVariableNode(astNode *asts.ASTNode) (*ForLoopOneVariableNode, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeForLoopOneVariable))
|
||||
lib.InternalCodingErrorIf(len(astNode.Children) != 3)
|
||||
|
||||
variableASTNode := astNode.Children[0]
|
||||
indexableASTNode := astNode.Children[1]
|
||||
blockASTNode := astNode.Children[2]
|
||||
|
||||
lib.InternalCodingErrorIf(variableASTNode.Type != dsl.NodeTypeLocalVariable)
|
||||
lib.InternalCodingErrorIf(variableASTNode.Type != asts.NodeType(NodeTypeLocalVariable))
|
||||
lib.InternalCodingErrorIf(variableASTNode.Token == nil)
|
||||
variableName := string(variableASTNode.Token.Lit)
|
||||
variableName := tokenLit(variableASTNode)
|
||||
|
||||
// TODO: error if loop-over node isn't map/array (inasmuch as can be
|
||||
// detected at CST-build time)
|
||||
|
|
@ -237,8 +237,8 @@ func NewForLoopTwoVariableNode(
|
|||
// * LocalVariable "k"
|
||||
// * LocalVariable "v"
|
||||
|
||||
func (root *RootNode) BuildForLoopTwoVariableNode(astNode *dsl.ASTNode) (*ForLoopTwoVariableNode, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeForLoopTwoVariable)
|
||||
func (root *RootNode) BuildForLoopTwoVariableNode(astNode *asts.ASTNode) (*ForLoopTwoVariableNode, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeForLoopTwoVariable))
|
||||
lib.InternalCodingErrorIf(len(astNode.Children) != 4)
|
||||
|
||||
keyVariableASTNode := astNode.Children[0]
|
||||
|
|
@ -246,14 +246,14 @@ func (root *RootNode) BuildForLoopTwoVariableNode(astNode *dsl.ASTNode) (*ForLoo
|
|||
indexableASTNode := astNode.Children[2]
|
||||
blockASTNode := astNode.Children[3]
|
||||
|
||||
lib.InternalCodingErrorIf(keyVariableASTNode.Type != dsl.NodeTypeLocalVariable)
|
||||
lib.InternalCodingErrorIf(keyVariableASTNode.Type != asts.NodeType(NodeTypeLocalVariable))
|
||||
lib.InternalCodingErrorIf(keyVariableASTNode.Token == nil)
|
||||
keyVariableName := string(keyVariableASTNode.Token.Lit)
|
||||
keyVariableName := tokenLit(keyVariableASTNode)
|
||||
keyIndexVariable := runtime.NewStackVariable(keyVariableName)
|
||||
|
||||
lib.InternalCodingErrorIf(valueVariableASTNode.Type != dsl.NodeTypeLocalVariable)
|
||||
lib.InternalCodingErrorIf(valueVariableASTNode.Type != asts.NodeType(NodeTypeLocalVariable))
|
||||
lib.InternalCodingErrorIf(valueVariableASTNode.Token == nil)
|
||||
valueVariableName := string(valueVariableASTNode.Token.Lit)
|
||||
valueVariableName := tokenLit(valueVariableASTNode)
|
||||
valueIndexVariable := runtime.NewStackVariable(valueVariableName)
|
||||
|
||||
// TODO: error if loop-over node isn't map/array (inasmuch as can be
|
||||
|
|
@ -425,9 +425,9 @@ func NewForLoopMultivariableNode(
|
|||
// * statement block
|
||||
|
||||
func (root *RootNode) BuildForLoopMultivariableNode(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
) (*ForLoopMultivariableNode, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeForLoopMultivariable)
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeForLoopMultivariable))
|
||||
lib.InternalCodingErrorIf(len(astNode.Children) != 4)
|
||||
|
||||
keyVariablesASTNode := astNode.Children[0]
|
||||
|
|
@ -435,18 +435,20 @@ func (root *RootNode) BuildForLoopMultivariableNode(
|
|||
indexableASTNode := astNode.Children[2]
|
||||
blockASTNode := astNode.Children[3]
|
||||
|
||||
lib.InternalCodingErrorIf(keyVariablesASTNode.Type != dsl.NodeTypeParameterList)
|
||||
// PGPG produces MultiIndex; legacy produced ParameterList. Both have LocalVariable children.
|
||||
lib.InternalCodingErrorIf(keyVariablesASTNode.Type != asts.NodeType(NodeTypeParameterList) &&
|
||||
keyVariablesASTNode.Type != asts.NodeType(NodeTypeMultiIndex))
|
||||
lib.InternalCodingErrorIf(keyVariablesASTNode.Children == nil)
|
||||
keyIndexVariables := make([]*runtime.StackVariable, len(keyVariablesASTNode.Children))
|
||||
for i, keyVariableASTNode := range keyVariablesASTNode.Children {
|
||||
lib.InternalCodingErrorIf(keyVariableASTNode.Token == nil)
|
||||
keyIndexVariableName := string(keyVariableASTNode.Token.Lit)
|
||||
keyIndexVariableName := tokenLit(keyVariableASTNode)
|
||||
keyIndexVariables[i] = runtime.NewStackVariable(keyIndexVariableName)
|
||||
}
|
||||
|
||||
lib.InternalCodingErrorIf(valueVariableASTNode.Type != dsl.NodeTypeLocalVariable)
|
||||
lib.InternalCodingErrorIf(valueVariableASTNode.Type != asts.NodeType(NodeTypeLocalVariable))
|
||||
lib.InternalCodingErrorIf(valueVariableASTNode.Token == nil)
|
||||
valueVariableName := string(valueVariableASTNode.Token.Lit)
|
||||
valueVariableName := tokenLit(valueVariableASTNode)
|
||||
valueIndexVariable := runtime.NewStackVariable(valueVariableName)
|
||||
|
||||
// TODO: error if loop-over node isn't map/array (inasmuch as can be
|
||||
|
|
@ -693,7 +695,7 @@ type TripleForLoopNode struct {
|
|||
startBlockNode *StatementBlockNode
|
||||
precontinuationAssignments []IExecutable
|
||||
continuationExpressionNode IEvaluable
|
||||
continuationExpressionToken *token.Token
|
||||
continuationExpressionToken *tokens.Token
|
||||
updateBlockNode *StatementBlockNode
|
||||
bodyBlockNode *StatementBlockNode
|
||||
}
|
||||
|
|
@ -702,7 +704,7 @@ func NewTripleForLoopNode(
|
|||
startBlockNode *StatementBlockNode,
|
||||
precontinuationAssignments []IExecutable,
|
||||
continuationExpressionNode IEvaluable,
|
||||
continuationExpressionToken *token.Token,
|
||||
continuationExpressionToken *tokens.Token,
|
||||
updateBlockNode *StatementBlockNode,
|
||||
bodyBlockNode *StatementBlockNode,
|
||||
) *TripleForLoopNode {
|
||||
|
|
@ -756,8 +758,8 @@ func NewTripleForLoopNode(
|
|||
// * DirectFieldValue "i"
|
||||
// * LocalVariable "i"
|
||||
|
||||
func (root *RootNode) BuildTripleForLoopNode(astNode *dsl.ASTNode) (*TripleForLoopNode, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeTripleForLoop)
|
||||
func (root *RootNode) BuildTripleForLoopNode(astNode *asts.ASTNode) (*TripleForLoopNode, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeTripleForLoop))
|
||||
lib.InternalCodingErrorIf(len(astNode.Children) != 4)
|
||||
|
||||
startBlockASTNode := astNode.Children[0]
|
||||
|
|
@ -765,10 +767,16 @@ func (root *RootNode) BuildTripleForLoopNode(astNode *dsl.ASTNode) (*TripleForLo
|
|||
updateBlockASTNode := astNode.Children[2]
|
||||
bodyBlockASTNode := astNode.Children[3]
|
||||
|
||||
lib.InternalCodingErrorIf(startBlockASTNode.Type != dsl.NodeTypeStatementBlock)
|
||||
lib.InternalCodingErrorIf(continuationExpressionASTNode.Type != dsl.NodeTypeStatementBlock)
|
||||
lib.InternalCodingErrorIf(updateBlockASTNode.Type != dsl.NodeTypeStatementBlock)
|
||||
lib.InternalCodingErrorIf(bodyBlockASTNode.Type != dsl.NodeTypeStatementBlock)
|
||||
// PGPG: body is StatementBlockInBraces; unwrap to get StatementBlock
|
||||
if bodyBlockASTNode.Type == asts.NodeType(NodeTypeStatementBlockInBraces) &&
|
||||
len(bodyBlockASTNode.Children) == 1 {
|
||||
bodyBlockASTNode = bodyBlockASTNode.Children[0]
|
||||
}
|
||||
|
||||
lib.InternalCodingErrorIf(startBlockASTNode.Type != asts.NodeType(NodeTypeStatementBlock))
|
||||
lib.InternalCodingErrorIf(continuationExpressionASTNode.Type != asts.NodeType(NodeTypeStatementBlock))
|
||||
lib.InternalCodingErrorIf(updateBlockASTNode.Type != asts.NodeType(NodeTypeStatementBlock))
|
||||
lib.InternalCodingErrorIf(bodyBlockASTNode.Type != asts.NodeType(NodeTypeStatementBlock))
|
||||
|
||||
startBlockNode, err := root.BuildStatementBlockNode(startBlockASTNode)
|
||||
if err != nil {
|
||||
|
|
@ -780,20 +788,26 @@ func (root *RootNode) BuildTripleForLoopNode(astNode *dsl.ASTNode) (*TripleForLo
|
|||
// for (int i = 0; c += 1, i < 10; i += 1) { ... }
|
||||
var precontinuationAssignments []IExecutable = nil
|
||||
var continuationExpressionNode IEvaluable = nil
|
||||
var continuationExpressionToken *token.Token = nil
|
||||
var continuationExpressionToken *tokens.Token = nil
|
||||
if len(continuationExpressionASTNode.Children) > 0 { // empty is true
|
||||
n := len(continuationExpressionASTNode.Children)
|
||||
if n > 1 {
|
||||
precontinuationAssignments = make([]IExecutable, n-1)
|
||||
for i := 0; i < n-1; i++ {
|
||||
if continuationExpressionASTNode.Children[i].Type != dsl.NodeTypeAssignment {
|
||||
childType := continuationExpressionASTNode.Children[i].Type
|
||||
if childType != asts.NodeType(NodeTypeAssignment) &&
|
||||
childType != asts.NodeType(NodeTypeCompoundAssignment) {
|
||||
return nil, fmt.Errorf(
|
||||
"the non-final triple-for continuation statements must be assignments",
|
||||
)
|
||||
}
|
||||
precontinuationAssignment, err := root.BuildAssignmentNode(
|
||||
continuationExpressionASTNode.Children[i],
|
||||
)
|
||||
var precontinuationAssignment IExecutable
|
||||
var err error
|
||||
if childType == asts.NodeType(NodeTypeCompoundAssignment) {
|
||||
precontinuationAssignment, err = root.BuildCompoundAssignmentNode(continuationExpressionASTNode.Children[i])
|
||||
} else {
|
||||
precontinuationAssignment, err = root.BuildAssignmentNode(continuationExpressionASTNode.Children[i])
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
|
@ -802,7 +816,7 @@ func (root *RootNode) BuildTripleForLoopNode(astNode *dsl.ASTNode) (*TripleForLo
|
|||
}
|
||||
|
||||
bareBooleanASTNode := continuationExpressionASTNode.Children[n-1]
|
||||
if bareBooleanASTNode.Type != dsl.NodeTypeBareBoolean {
|
||||
if bareBooleanASTNode.Type != asts.NodeType(NodeTypeBareBoolean) {
|
||||
if n == 1 {
|
||||
return nil, fmt.Errorf(
|
||||
"the triple-for continuation statement must be a bare boolean",
|
||||
|
|
@ -881,7 +895,7 @@ func (node *TripleForLoopNode) Execute(state *runtime.State) (*BlockExitPayload,
|
|||
if !isBool {
|
||||
return nil, fmt.Errorf(
|
||||
"for-loop continuation did not evaluate to boolean%s",
|
||||
dsl.TokenToLocationInfo(node.continuationExpressionToken),
|
||||
pgpgTokenToLocationInfo(node.continuationExpressionToken),
|
||||
)
|
||||
}
|
||||
if !boolValue {
|
||||
|
|
|
|||
|
|
@ -7,8 +7,8 @@
|
|||
package cst
|
||||
|
||||
import (
|
||||
"github.com/johnkerl/miller/v6/pkg/dsl"
|
||||
"github.com/johnkerl/miller/v6/pkg/lib"
|
||||
"github.com/johnkerl/pgpg/go/lib/pkg/asts"
|
||||
)
|
||||
|
||||
// Function lookup:
|
||||
|
|
@ -21,15 +21,20 @@ import (
|
|||
// o On a next pass, we will walk that list resolving against all encountered
|
||||
// UDF definitions. (It will be an error then if it's still unresolvable.)
|
||||
|
||||
func (root *RootNode) BuildFunctionCallsiteNode(astNode *dsl.ASTNode) (IEvaluable, error) {
|
||||
func (root *RootNode) BuildFunctionCallsiteNode(astNode *asts.ASTNode) (IEvaluable, error) {
|
||||
lib.InternalCodingErrorIf(
|
||||
astNode.Type != dsl.NodeTypeFunctionCallsite &&
|
||||
astNode.Type != dsl.NodeTypeOperator,
|
||||
astNode.Type != asts.NodeType(NodeTypeFunctionCallsite) &&
|
||||
astNode.Type != asts.NodeType(NodeTypeOperator),
|
||||
)
|
||||
lib.InternalCodingErrorIf(astNode.Token == nil)
|
||||
lib.InternalCodingErrorIf(astNode.Children == nil)
|
||||
|
||||
functionName := string(astNode.Token.Lit)
|
||||
functionName := tokenLit(astNode)
|
||||
|
||||
// PGPG produces ternary (cond ? a : b) with "?" token; builtin is registered as "?:"
|
||||
if functionName == "?" && len(astNode.Children) == 3 {
|
||||
functionName = "?:"
|
||||
}
|
||||
|
||||
// - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
|
||||
// Special-case the dot operator, which is:
|
||||
|
|
|
|||
|
|
@ -5,11 +5,11 @@ package cst
|
|||
import (
|
||||
"fmt"
|
||||
|
||||
"github.com/johnkerl/miller/v6/pkg/dsl"
|
||||
"github.com/johnkerl/miller/v6/pkg/lib"
|
||||
"github.com/johnkerl/miller/v6/pkg/mlrval"
|
||||
"github.com/johnkerl/miller/v6/pkg/parsing/token"
|
||||
"github.com/johnkerl/miller/v6/pkg/runtime"
|
||||
"github.com/johnkerl/pgpg/go/lib/pkg/asts"
|
||||
"github.com/johnkerl/pgpg/go/lib/pkg/tokens"
|
||||
)
|
||||
|
||||
type IfChainNode struct {
|
||||
|
|
@ -26,7 +26,7 @@ func NewIfChainNode(ifItems []*IfItem) *IfChainNode {
|
|||
// statement-block part {...}. For "else", the conditional is nil.
|
||||
type IfItem struct {
|
||||
conditionNode IEvaluable
|
||||
conditionToken *token.Token
|
||||
conditionToken *tokens.Token
|
||||
statementBlockNode *StatementBlockNode
|
||||
}
|
||||
|
||||
|
|
@ -67,16 +67,16 @@ type IfItem struct {
|
|||
// * DirectFieldValue "z"
|
||||
// * IntLiteral "900"
|
||||
|
||||
func (root *RootNode) BuildIfChainNode(astNode *dsl.ASTNode) (*IfChainNode, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeIfChain)
|
||||
func (root *RootNode) BuildIfChainNode(astNode *asts.ASTNode) (*IfChainNode, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeIfChain))
|
||||
|
||||
ifItems := []*IfItem{}
|
||||
|
||||
astChildren := astNode.Children
|
||||
|
||||
for _, astChild := range astChildren {
|
||||
lib.InternalCodingErrorIf(astChild.Type != dsl.NodeTypeIfItem)
|
||||
token := string(astChild.Token.Lit) // "if", "elif", "else"
|
||||
lib.InternalCodingErrorIf(astChild.Type != asts.NodeType(NodeTypeIfItem))
|
||||
token := tokenLit(astChild) // "if", "elif", "else"
|
||||
if token == "if" || token == "elif" {
|
||||
lib.InternalCodingErrorIf(len(astChild.Children) != 2)
|
||||
conditionNode, err := root.BuildEvaluableNode(astChild.Children[0])
|
||||
|
|
@ -125,7 +125,7 @@ func (node *IfChainNode) Execute(state *runtime.State) (*BlockExitPayload, error
|
|||
if !isBool {
|
||||
return nil, fmt.Errorf(
|
||||
"conditional expression did not evaluate to boolean%s",
|
||||
dsl.TokenToLocationInfo(ifItem.conditionToken),
|
||||
pgpgTokenToLocationInfo(ifItem.conditionToken),
|
||||
)
|
||||
}
|
||||
if boolValue {
|
||||
|
|
|
|||
|
|
@ -5,78 +5,127 @@ package cst
|
|||
import (
|
||||
"fmt"
|
||||
"math"
|
||||
"strings"
|
||||
|
||||
"github.com/johnkerl/miller/v6/pkg/dsl"
|
||||
"github.com/johnkerl/miller/v6/pkg/lib"
|
||||
"github.com/johnkerl/miller/v6/pkg/mlrval"
|
||||
"github.com/johnkerl/miller/v6/pkg/runtime"
|
||||
"github.com/johnkerl/pgpg/go/lib/pkg/asts"
|
||||
)
|
||||
|
||||
func (root *RootNode) BuildLeafNode(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
) (IEvaluable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Children != nil)
|
||||
sval := string(astNode.Token.Lit)
|
||||
// The BNF uses empty slice for terminals. It may also produce reduced literal nodes
|
||||
// (float_literal, int_literal) with one child (the terminal).
|
||||
lib.InternalCodingErrorIf(astNode.Children != nil && len(astNode.Children) > 1)
|
||||
sval := tokenLit(astNode)
|
||||
if sval == "" && astNode.Children != nil && len(astNode.Children) == 1 {
|
||||
sval = tokenLit(astNode.Children[0])
|
||||
}
|
||||
|
||||
switch astNode.Type {
|
||||
|
||||
case dsl.NodeTypeDirectFieldValue:
|
||||
return root.BuildDirectFieldRvalueNode(sval), nil
|
||||
case dsl.NodeTypeFullSrec:
|
||||
case asts.NodeType(NodeTypeDirectFieldValue):
|
||||
return root.BuildDirectFieldRvalueNode(tokenLitStripDollarOrAt(astNode)), nil
|
||||
case asts.NodeType(NodeTypeFullSrec):
|
||||
return root.BuildFullSrecRvalueNode(), nil
|
||||
|
||||
case dsl.NodeTypeDirectOosvarValue:
|
||||
return root.BuildDirectOosvarRvalueNode(sval), nil
|
||||
case dsl.NodeTypeFullOosvar:
|
||||
case asts.NodeType(NodeTypeDirectOosvarValue):
|
||||
return root.BuildDirectOosvarRvalueNode(tokenLitStripDollarOrAt(astNode)), nil
|
||||
case asts.NodeType(NodeTypeFullOosvar):
|
||||
return root.BuildFullOosvarRvalueNode(), nil
|
||||
|
||||
case dsl.NodeTypeLocalVariable:
|
||||
case asts.NodeType(NodeTypeLocalVariable):
|
||||
return root.BuildLocalVariableNode(sval), nil
|
||||
|
||||
case dsl.NodeTypeStringLiteral:
|
||||
case asts.NodeType(NodeTypeStringLiteral):
|
||||
return root.BuildStringLiteralNode(sval), nil
|
||||
|
||||
case dsl.NodeTypeRegex:
|
||||
case asts.NodeType(NodeTypeRegex):
|
||||
// During the BNF parse all string literals -- "foo" or "(..)_(...)"
|
||||
// regexes etc -- are marked as dsl.NodeTypeStringLiteral. However, a
|
||||
// regexes etc -- are marked as asts.NodeType(NodeTypeStringLiteral). However, a
|
||||
// CST-build pre-pass relabels second argument to sub/gsub etc -- any
|
||||
// known regex positions -- from dsl.NodeTypeStringLiteral to
|
||||
// dsl.NodeTypeRegex. The RegexLiteralNode is responsible for
|
||||
// known regex positions -- from asts.NodeType(NodeTypeStringLiteral) to
|
||||
// asts.NodeType(NodeTypeRegex). The RegexLiteralNode is responsible for
|
||||
// handling backslash sequences for regex literals differently from
|
||||
// those for non-regex string literals.
|
||||
return root.BuildRegexLiteralNode(sval), nil
|
||||
|
||||
case dsl.NodeTypeRegexCaseInsensitive:
|
||||
// StringLiteral nodes like '"abc"' entered by the user come in from
|
||||
// the AST as 'abc', with double quotes removed. Case-insensitive
|
||||
// regexes like '"a.*b"i' come in with initial '"' and final '"i'
|
||||
// intact. We let the sub/regextract/etc functions deal with this.
|
||||
// (The alternative would be to make a separate Mlrval type separate
|
||||
// from string.)
|
||||
case asts.NodeType(NodeTypeRegexCaseInsensitive):
|
||||
// PGPG: RegexCaseInsensitive has string_literal child. CompileMillerRegex
|
||||
// expects "a.*b"i format for case-insensitive; append "i" if not present.
|
||||
if sval == "" && astNode.Children != nil && len(astNode.Children) > 0 {
|
||||
sval = tokenLit(astNode.Children[0])
|
||||
}
|
||||
if sval != "" && !strings.HasSuffix(sval, "i") {
|
||||
sval = sval + "i"
|
||||
}
|
||||
return root.BuildRegexLiteralNode(sval), nil
|
||||
|
||||
case dsl.NodeTypeIntLiteral:
|
||||
case asts.NodeType(NodeTypeIntLiteral):
|
||||
return root.BuildIntLiteralNode(sval), nil
|
||||
case dsl.NodeTypeFloatLiteral:
|
||||
case asts.NodeType(NodeTypeFloatLiteral):
|
||||
return root.BuildFloatLiteralNode(sval), nil
|
||||
case dsl.NodeTypeBoolLiteral:
|
||||
case asts.NodeType(NodeTypeBoolLiteral):
|
||||
return root.BuildBoolLiteralNode(sval), nil
|
||||
case dsl.NodeTypeNullLiteral:
|
||||
case asts.NodeType(NodeTypeNullLiteral):
|
||||
return root.BuildNullLiteralNode(), nil
|
||||
case dsl.NodeTypeContextVariable:
|
||||
case asts.NodeType(NodeTypeContextVariable):
|
||||
return root.BuildContextVariableNode(astNode)
|
||||
case dsl.NodeTypeConstant:
|
||||
case asts.NodeType(NodeTypeConstant):
|
||||
return root.BuildConstantNode(astNode)
|
||||
|
||||
case dsl.NodeTypeArraySliceEmptyLowerIndex:
|
||||
case asts.NodeType(NodeTypeArraySliceEmptyLowerIndex):
|
||||
return root.BuildArraySliceEmptyLowerIndexNode(astNode)
|
||||
case dsl.NodeTypeArraySliceEmptyUpperIndex:
|
||||
case asts.NodeType(NodeTypeArraySliceEmptyUpperIndex):
|
||||
return root.BuildArraySliceEmptyUpperIndexNode(astNode)
|
||||
|
||||
case dsl.NodeTypePanic:
|
||||
case asts.NodeType(NodeTypePanic):
|
||||
return root.BuildPanicNode(astNode)
|
||||
}
|
||||
|
||||
// PGPG may use different type strings; try string comparison for known leaf types
|
||||
switch string(astNode.Type) {
|
||||
case "DirectFieldValue":
|
||||
return root.BuildDirectFieldRvalueNode(tokenLitStripDollarOrAt(astNode)), nil
|
||||
case "FullSrec":
|
||||
return root.BuildFullSrecRvalueNode(), nil
|
||||
case "DirectOosvarValue":
|
||||
return root.BuildDirectOosvarRvalueNode(tokenLitStripDollarOrAt(astNode)), nil
|
||||
case "BracedFieldValue":
|
||||
return root.BuildDirectFieldRvalueNode(tokenLitStripBraced(astNode)), nil
|
||||
case "FullOosvar":
|
||||
return root.BuildFullOosvarRvalueNode(), nil
|
||||
case "BracedOosvarValue":
|
||||
return root.BuildDirectOosvarRvalueNode(tokenLitStripBraced(astNode)), nil
|
||||
case "LocalVariable":
|
||||
return root.BuildLocalVariableNode(sval), nil
|
||||
case "string_literal":
|
||||
return root.BuildStringLiteralNode(sval), nil
|
||||
case "int_literal":
|
||||
return root.BuildIntLiteralNode(sval), nil
|
||||
case "float_literal":
|
||||
return root.BuildFloatLiteralNode(sval), nil
|
||||
case "bool_literal":
|
||||
return root.BuildBoolLiteralNode(sval), nil
|
||||
case "null_literal":
|
||||
return root.BuildNullLiteralNode(), nil
|
||||
case "RegexCaseInsensitive", "regex_case_insensitive":
|
||||
if sval == "" && astNode.Children != nil && len(astNode.Children) > 0 {
|
||||
sval = tokenLit(astNode.Children[0])
|
||||
}
|
||||
if sval != "" && !strings.HasSuffix(sval, "i") {
|
||||
sval = sval + "i"
|
||||
}
|
||||
return root.BuildRegexLiteralNode(sval), nil
|
||||
}
|
||||
|
||||
// PGPG produces ctx_NF, ctx_NR, etc. as terminal types; treat as ContextVariable
|
||||
if strings.HasPrefix(string(astNode.Type), "ctx_") {
|
||||
return root.BuildContextVariableNode(astNode)
|
||||
}
|
||||
|
||||
return nil, fmt.Errorf("at CST BuildLeafNode: unhandled AST node %s", string(astNode.Type))
|
||||
}
|
||||
|
||||
|
|
@ -203,9 +252,9 @@ func (node *LocalVariableNode) Evaluate(
|
|||
}
|
||||
|
||||
// During the BNF parse all string literals -- "foo" or "(..)_(...)" regexes
|
||||
// etc -- are marked as dsl.NodeTypeStringLiteral. However, a CST-build
|
||||
// etc -- are marked as asts.NodeType(NodeTypeStringLiteral). However, a CST-build
|
||||
// pre-pass relabels second argument to sub/gsub etc -- any known regex
|
||||
// positions -- from dsl.NodeTypeStringLiteral to dsl.NodeTypeRegex. The
|
||||
// positions -- from asts.NodeType(NodeTypeStringLiteral) to asts.NodeType(NodeTypeRegex). The
|
||||
// RegexLiteralNode is responsible for handling backslash sequences for
|
||||
// regex literals differently from those for non-regex string literals.
|
||||
|
||||
|
|
@ -252,6 +301,11 @@ type RegexCaptureReplacementNode struct {
|
|||
}
|
||||
|
||||
func (root *RootNode) BuildStringLiteralNode(literal string) IEvaluable {
|
||||
// The PGPG lexer produces string_literal token with surrounding quotes in the lexeme.
|
||||
// Strip them so "a" becomes a.
|
||||
if len(literal) >= 2 && literal[0] == '"' && literal[len(literal)-1] == '"' {
|
||||
literal = literal[1 : len(literal)-1]
|
||||
}
|
||||
// Convert "\t" to tab character, "\"" to double-quote character, etc.
|
||||
// This is intentionally done for StringLiteralNode but not for
|
||||
// RegexLiteralNode. See also https://github.com/johnkerl/miller/issues/297.
|
||||
|
|
@ -379,9 +433,9 @@ func (node *MlrvalLiteralNode) Evaluate(
|
|||
return node.literal
|
||||
}
|
||||
|
||||
func (root *RootNode) BuildContextVariableNode(astNode *dsl.ASTNode) (IEvaluable, error) {
|
||||
func (root *RootNode) BuildContextVariableNode(astNode *asts.ASTNode) (IEvaluable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Token == nil)
|
||||
sval := string(astNode.Token.Lit)
|
||||
sval := tokenLit(astNode)
|
||||
|
||||
switch sval {
|
||||
|
||||
|
|
@ -562,9 +616,9 @@ func (node *FLATSEPNode) Evaluate(
|
|||
return mlrval.FromString(state.Options.WriterOptions.FLATSEP)
|
||||
}
|
||||
|
||||
func (root *RootNode) BuildConstantNode(astNode *dsl.ASTNode) (IEvaluable, error) {
|
||||
func (root *RootNode) BuildConstantNode(astNode *asts.ASTNode) (IEvaluable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Token == nil)
|
||||
sval := string(astNode.Token.Lit)
|
||||
sval := tokenLit(astNode)
|
||||
|
||||
switch sval {
|
||||
|
||||
|
|
@ -608,9 +662,9 @@ type LiteralOneNode struct {
|
|||
// In array slices like 'myarray[:4]', the lower index is always 1 since Miller
|
||||
// user-space indices are 1-up.
|
||||
func (root *RootNode) BuildArraySliceEmptyLowerIndexNode(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
) (*LiteralOneNode, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeArraySliceEmptyLowerIndex)
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeArraySliceEmptyLowerIndex))
|
||||
return &LiteralOneNode{}, nil
|
||||
}
|
||||
func (node *LiteralOneNode) Evaluate(
|
||||
|
|
@ -627,9 +681,9 @@ type LiteralEmptyStringNode struct {
|
|||
// we don't have access to the array length in this AST node so we return ""
|
||||
// so the slice-index CST node can compute it.
|
||||
func (root *RootNode) BuildArraySliceEmptyUpperIndexNode(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
) (*LiteralEmptyStringNode, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeArraySliceEmptyUpperIndex)
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeArraySliceEmptyUpperIndex))
|
||||
return &LiteralEmptyStringNode{}, nil
|
||||
}
|
||||
func (node *LiteralEmptyStringNode) Evaluate(
|
||||
|
|
@ -645,7 +699,7 @@ func (node *LiteralEmptyStringNode) Evaluate(
|
|||
type PanicNode struct {
|
||||
}
|
||||
|
||||
func (root *RootNode) BuildPanicNode(astNode *dsl.ASTNode) (*PanicNode, error) {
|
||||
func (root *RootNode) BuildPanicNode(astNode *asts.ASTNode) (*PanicNode, error) {
|
||||
return &PanicNode{}, nil
|
||||
}
|
||||
func (node *PanicNode) Evaluate(
|
||||
|
|
|
|||
|
|
@ -8,43 +8,46 @@ import (
|
|||
"fmt"
|
||||
"os"
|
||||
|
||||
"github.com/johnkerl/miller/v6/pkg/dsl"
|
||||
"github.com/johnkerl/miller/v6/pkg/lib"
|
||||
"github.com/johnkerl/miller/v6/pkg/mlrval"
|
||||
"github.com/johnkerl/miller/v6/pkg/runtime"
|
||||
"github.com/johnkerl/pgpg/go/lib/pkg/asts"
|
||||
)
|
||||
|
||||
func (root *RootNode) BuildAssignableNode(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
) (IAssignable, error) {
|
||||
|
||||
switch astNode.Type {
|
||||
|
||||
case dsl.NodeTypeDirectFieldValue:
|
||||
case asts.NodeType(NodeTypeDirectFieldValue), asts.NodeType(NodeTypeBracedFieldValue):
|
||||
return root.BuildDirectFieldValueLvalueNode(astNode)
|
||||
case dsl.NodeTypeIndirectFieldValue:
|
||||
case asts.NodeType(NodeTypeIndirectFieldValue):
|
||||
return root.BuildIndirectFieldValueLvalueNode(astNode)
|
||||
|
||||
case dsl.NodeTypeFullSrec:
|
||||
case asts.NodeType(NodeTypeFullSrec):
|
||||
return root.BuildFullSrecLvalueNode(astNode)
|
||||
|
||||
case dsl.NodeTypeDirectOosvarValue:
|
||||
case asts.NodeType(NodeTypeDirectOosvarValue), asts.NodeType(NodeTypeBracedOosvarValue):
|
||||
return root.BuildDirectOosvarValueLvalueNode(astNode)
|
||||
case dsl.NodeTypeIndirectOosvarValue:
|
||||
case asts.NodeType(NodeTypeIndirectOosvarValue):
|
||||
return root.BuildIndirectOosvarValueLvalueNode(astNode)
|
||||
case dsl.NodeTypeFullOosvar:
|
||||
case asts.NodeType(NodeTypeFullOosvar):
|
||||
return root.BuildFullOosvarLvalueNode(astNode)
|
||||
case dsl.NodeTypeLocalVariable:
|
||||
case asts.NodeType(NodeTypeLocalVariable):
|
||||
return root.BuildLocalVariableLvalueNode(astNode)
|
||||
|
||||
case dsl.NodeTypeArrayOrMapIndexAccess:
|
||||
case asts.NodeType(NodeTypeArrayOrMapIndexAccess):
|
||||
return root.BuildIndexedLvalueNode(astNode)
|
||||
|
||||
case dsl.NodeTypeDotOperator:
|
||||
case asts.NodeType(NodeTypeDotOperator):
|
||||
return root.BuildIndexedLvalueNode(astNode)
|
||||
|
||||
case dsl.NodeTypeEnvironmentVariable:
|
||||
case asts.NodeType(NodeTypeEnvironmentVariable):
|
||||
return root.BuildEnvironmentVariableLvalueNode(astNode)
|
||||
|
||||
case asts.NodeType("TypedeclLocalVariable"):
|
||||
return root.BuildTypedeclLocalVariableLvalueNode(astNode)
|
||||
}
|
||||
|
||||
return nil, errors.New(
|
||||
|
|
@ -57,11 +60,18 @@ type DirectFieldValueLvalueNode struct {
|
|||
}
|
||||
|
||||
func (root *RootNode) BuildDirectFieldValueLvalueNode(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
) (IAssignable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeDirectFieldValue)
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeDirectFieldValue) &&
|
||||
astNode.Type != asts.NodeType(NodeTypeBracedFieldValue))
|
||||
|
||||
lhsFieldName := mlrval.FromString(string(astNode.Token.Lit))
|
||||
var name string
|
||||
if astNode.Type == asts.NodeType(NodeTypeBracedFieldValue) {
|
||||
name = tokenLitStripBraced(astNode)
|
||||
} else {
|
||||
name = tokenLitStripDollarOrAt(astNode)
|
||||
}
|
||||
lhsFieldName := mlrval.FromString(name)
|
||||
return NewDirectFieldValueLvalueNode(lhsFieldName), nil
|
||||
}
|
||||
|
||||
|
|
@ -144,30 +154,24 @@ type IndirectFieldValueLvalueNode struct {
|
|||
}
|
||||
|
||||
func (root *RootNode) BuildIndirectFieldValueLvalueNode(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
) (IAssignable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeIndirectFieldValue)
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeIndirectFieldValue))
|
||||
lib.InternalCodingErrorIf(astNode == nil)
|
||||
lib.InternalCodingErrorIf(len(astNode.Children) != 1)
|
||||
|
||||
child := astNode.Children[0]
|
||||
if child.Type == dsl.NodeTypeArrayLiteral && len(child.Children) == 1 {
|
||||
if child.Type == asts.NodeType(NodeTypeArrayLiteral) && len(child.Children) == 1 {
|
||||
inner := child.Children[0]
|
||||
if inner.Type == dsl.NodeTypeArrayLiteral && len(inner.Children) == 1 {
|
||||
if inner.Type == asts.NodeType(NodeTypeArrayLiteral) && len(inner.Children) == 1 {
|
||||
// $[[[n]]] → positional field value lvalue
|
||||
indexASTNode := inner.Children[0]
|
||||
syntheticAST := &dsl.ASTNode{
|
||||
Type: dsl.NodeTypePositionalFieldValue,
|
||||
Children: []*dsl.ASTNode{indexASTNode},
|
||||
}
|
||||
syntheticAST := asts.NewASTNode(nil, asts.NodeType(NodeTypePositionalFieldValue), []*asts.ASTNode{indexASTNode})
|
||||
return root.BuildPositionalFieldValueLvalueNode(syntheticAST)
|
||||
}
|
||||
// $[[n]] → positional field name lvalue
|
||||
indexASTNode := inner
|
||||
syntheticAST := &dsl.ASTNode{
|
||||
Type: dsl.NodeTypePositionalFieldName,
|
||||
Children: []*dsl.ASTNode{indexASTNode},
|
||||
}
|
||||
syntheticAST := asts.NewASTNode(nil, asts.NodeType(NodeTypePositionalFieldName), []*asts.ASTNode{indexASTNode})
|
||||
return root.BuildPositionalFieldNameLvalueNode(syntheticAST)
|
||||
}
|
||||
|
||||
|
|
@ -264,9 +268,9 @@ type PositionalFieldNameLvalueNode struct {
|
|||
}
|
||||
|
||||
func (root *RootNode) BuildPositionalFieldNameLvalueNode(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
) (IAssignable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypePositionalFieldName)
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypePositionalFieldName))
|
||||
lib.InternalCodingErrorIf(astNode == nil)
|
||||
lib.InternalCodingErrorIf(len(astNode.Children) != 1)
|
||||
lhsFieldIndexExpression, err := root.BuildEvaluableNode(astNode.Children[0])
|
||||
|
|
@ -372,9 +376,9 @@ type PositionalFieldValueLvalueNode struct {
|
|||
}
|
||||
|
||||
func (root *RootNode) BuildPositionalFieldValueLvalueNode(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
) (IAssignable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypePositionalFieldValue)
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypePositionalFieldValue))
|
||||
lib.InternalCodingErrorIf(astNode == nil)
|
||||
lib.InternalCodingErrorIf(len(astNode.Children) != 1)
|
||||
lhsFieldIndexExpression, err := root.BuildEvaluableNode(astNode.Children[0])
|
||||
|
|
@ -485,10 +489,11 @@ func (node *PositionalFieldValueLvalueNode) UnassignIndexed(
|
|||
type FullSrecLvalueNode struct {
|
||||
}
|
||||
|
||||
func (root *RootNode) BuildFullSrecLvalueNode(astNode *dsl.ASTNode) (IAssignable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeFullSrec)
|
||||
func (root *RootNode) BuildFullSrecLvalueNode(astNode *asts.ASTNode) (IAssignable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeFullSrec))
|
||||
lib.InternalCodingErrorIf(astNode == nil)
|
||||
lib.InternalCodingErrorIf(astNode.Children != nil)
|
||||
// PGPG FullSrec has empty list as children
|
||||
lib.InternalCodingErrorIf(astNode.Children != nil && len(astNode.Children) > 0)
|
||||
return NewFullSrecLvalueNode(), nil
|
||||
}
|
||||
|
||||
|
|
@ -559,10 +564,17 @@ type DirectOosvarValueLvalueNode struct {
|
|||
lhsOosvarName *mlrval.Mlrval
|
||||
}
|
||||
|
||||
func (root *RootNode) BuildDirectOosvarValueLvalueNode(astNode *dsl.ASTNode) (IAssignable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeDirectOosvarValue)
|
||||
func (root *RootNode) BuildDirectOosvarValueLvalueNode(astNode *asts.ASTNode) (IAssignable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeDirectOosvarValue) &&
|
||||
astNode.Type != asts.NodeType(NodeTypeBracedOosvarValue))
|
||||
|
||||
lhsOosvarName := mlrval.FromString(string(astNode.Token.Lit))
|
||||
var name string
|
||||
if astNode.Type == asts.NodeType(NodeTypeBracedOosvarValue) {
|
||||
name = tokenLitStripBraced(astNode)
|
||||
} else {
|
||||
name = tokenLitStripDollarOrAt(astNode)
|
||||
}
|
||||
lhsOosvarName := mlrval.FromString(name)
|
||||
return NewDirectOosvarValueLvalueNode(lhsOosvarName), nil
|
||||
}
|
||||
|
||||
|
|
@ -624,9 +636,9 @@ type IndirectOosvarValueLvalueNode struct {
|
|||
}
|
||||
|
||||
func (root *RootNode) BuildIndirectOosvarValueLvalueNode(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
) (IAssignable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeIndirectOosvarValue)
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeIndirectOosvarValue))
|
||||
lib.InternalCodingErrorIf(astNode == nil)
|
||||
lib.InternalCodingErrorIf(len(astNode.Children) != 1)
|
||||
|
||||
|
|
@ -702,10 +714,10 @@ func (node *IndirectOosvarValueLvalueNode) UnassignIndexed(
|
|||
type FullOosvarLvalueNode struct {
|
||||
}
|
||||
|
||||
func (root *RootNode) BuildFullOosvarLvalueNode(astNode *dsl.ASTNode) (IAssignable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeFullOosvar)
|
||||
func (root *RootNode) BuildFullOosvarLvalueNode(astNode *asts.ASTNode) (IAssignable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeFullOosvar))
|
||||
lib.InternalCodingErrorIf(astNode == nil)
|
||||
lib.InternalCodingErrorIf(astNode.Children != nil)
|
||||
lib.InternalCodingErrorIf(astNode.Children != nil && len(astNode.Children) > 0)
|
||||
return NewFullOosvarLvalueNode(), nil
|
||||
}
|
||||
|
||||
|
|
@ -767,25 +779,34 @@ type LocalVariableLvalueNode struct {
|
|||
defineTypedAtScope bool
|
||||
}
|
||||
|
||||
func (root *RootNode) BuildLocalVariableLvalueNode(astNode *dsl.ASTNode) (IAssignable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeLocalVariable)
|
||||
func (root *RootNode) BuildTypedeclLocalVariableLvalueNode(astNode *asts.ASTNode) (IAssignable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Children == nil || len(astNode.Children) < 2)
|
||||
typeNode := astNode.Children[0]
|
||||
varNode := astNode.Children[1]
|
||||
// PGPG Typedecl produces kw_int, kw_bool, etc. (no Typedecl wrapper node)
|
||||
lib.InternalCodingErrorIf(varNode.Type != asts.NodeType(NodeTypeLocalVariable))
|
||||
typeName := tokenLit(typeNode)
|
||||
variableName := tokenLit(varNode)
|
||||
return NewLocalVariableLvalueNode(
|
||||
runtime.NewStackVariable(variableName),
|
||||
typeName,
|
||||
true, // defineTypedAtScope
|
||||
), nil
|
||||
}
|
||||
|
||||
func (root *RootNode) BuildLocalVariableLvalueNode(astNode *asts.ASTNode) (IAssignable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeLocalVariable))
|
||||
|
||||
// TODO require type mask in strict mode
|
||||
|
||||
variableName := string(astNode.Token.Lit)
|
||||
variableName := tokenLit(astNode)
|
||||
typeName := "any"
|
||||
defineTypedAtScope := false
|
||||
if astNode.Children == nil { // untyped, like 'x = 3'
|
||||
if root.strictMode {
|
||||
return nil, fmt.Errorf(
|
||||
`need typedecl such as "var", "str", "num", etc. for variable "%s" in strict mode`,
|
||||
variableName,
|
||||
)
|
||||
}
|
||||
} else { // typed, like 'num x = 3'
|
||||
// PGPG: LocalVariable is terminal (children nil or empty). Miller had typed params with Children[0]=Typedecl.
|
||||
if astNode.Children != nil && len(astNode.Children) > 0 { // typed, like 'num x = 3'
|
||||
typeNode := astNode.Children[0]
|
||||
lib.InternalCodingErrorIf(typeNode.Type != dsl.NodeTypeTypedecl)
|
||||
typeName = string(typeNode.Token.Lit)
|
||||
lib.InternalCodingErrorIf(typeNode.Type != asts.NodeType(NodeTypeTypedecl))
|
||||
typeName = tokenLit(typeNode)
|
||||
defineTypedAtScope = true
|
||||
}
|
||||
return NewLocalVariableLvalueNode(
|
||||
|
|
@ -870,8 +891,8 @@ type IndexedLvalueNode struct {
|
|||
|
||||
// Either 'mymap["attr"]' or 'mymap.attr'. Furthermore they can be mixed as in
|
||||
// 'mymap["foo"].bar' or 'mymap.foo["bar"]'.
|
||||
func (root *RootNode) BuildIndexedLvalueNode(astNode *dsl.ASTNode) (IAssignable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeArrayOrMapIndexAccess && astNode.Type != dsl.NodeTypeDotOperator)
|
||||
func (root *RootNode) BuildIndexedLvalueNode(astNode *asts.ASTNode) (IAssignable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeArrayOrMapIndexAccess) && astNode.Type != asts.NodeType(NodeTypeDotOperator))
|
||||
lib.InternalCodingErrorIf(astNode == nil)
|
||||
|
||||
var baseLvalue IAssignable = nil
|
||||
|
|
@ -895,7 +916,7 @@ func (root *RootNode) BuildIndexedLvalueNode(astNode *dsl.ASTNode) (IAssignable,
|
|||
// then the base Lvalue.
|
||||
walkerNode := astNode
|
||||
for {
|
||||
if walkerNode.Type == dsl.NodeTypeArrayOrMapIndexAccess {
|
||||
if walkerNode.Type == asts.NodeType(NodeTypeArrayOrMapIndexAccess) {
|
||||
lib.InternalCodingErrorIf(walkerNode == nil)
|
||||
lib.InternalCodingErrorIf(len(walkerNode.Children) != 2)
|
||||
indexEvaluable, err := root.BuildEvaluableNode(walkerNode.Children[1])
|
||||
|
|
@ -904,10 +925,10 @@ func (root *RootNode) BuildIndexedLvalueNode(astNode *dsl.ASTNode) (IAssignable,
|
|||
}
|
||||
indexEvaluables = append([]IEvaluable{indexEvaluable}, indexEvaluables...)
|
||||
walkerNode = walkerNode.Children[0]
|
||||
} else if walkerNode.Type == dsl.NodeTypeDotOperator {
|
||||
} else if walkerNode.Type == asts.NodeType(NodeTypeDotOperator) {
|
||||
lib.InternalCodingErrorIf(walkerNode == nil)
|
||||
lib.InternalCodingErrorIf(len(walkerNode.Children) != 2)
|
||||
indexEvaluable := root.BuildStringLiteralNode(string(walkerNode.Children[1].Token.Lit))
|
||||
indexEvaluable := root.BuildStringLiteralNode(tokenLit(walkerNode.Children[1]))
|
||||
indexEvaluables = append([]IEvaluable{indexEvaluable}, indexEvaluables...)
|
||||
|
||||
walkerNode = walkerNode.Children[0]
|
||||
|
|
@ -990,15 +1011,23 @@ type EnvironmentVariableLvalueNode struct {
|
|||
nameExpression IEvaluable
|
||||
}
|
||||
|
||||
func (root *RootNode) BuildEnvironmentVariableLvalueNode(astNode *dsl.ASTNode) (IAssignable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeEnvironmentVariable)
|
||||
func (root *RootNode) BuildEnvironmentVariableLvalueNode(astNode *asts.ASTNode) (IAssignable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeEnvironmentVariable))
|
||||
lib.InternalCodingErrorIf(astNode == nil)
|
||||
lib.InternalCodingErrorIf(len(astNode.Children) != 1)
|
||||
nameExpression, err := root.BuildEvaluableNode(astNode.Children[0])
|
||||
if err != nil {
|
||||
return nil, err
|
||||
child := astNode.Children[0]
|
||||
var nameExpression IEvaluable
|
||||
// ENV.FOO: child is non_sigil_name; BuildEvaluableNode doesn't handle bare identifiers
|
||||
if string(child.Type) == "non_sigil_name" {
|
||||
sval := tokenLit(child)
|
||||
nameExpression = root.BuildStringLiteralNode(sval)
|
||||
} else {
|
||||
var err error
|
||||
nameExpression, err = root.BuildEvaluableNode(child)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
return NewEnvironmentVariableLvalueNode(nameExpression), nil
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -7,11 +7,11 @@ import (
|
|||
"fmt"
|
||||
"os"
|
||||
|
||||
"github.com/johnkerl/miller/v6/pkg/dsl"
|
||||
"github.com/johnkerl/miller/v6/pkg/lib"
|
||||
"github.com/johnkerl/miller/v6/pkg/output"
|
||||
"github.com/johnkerl/miller/v6/pkg/runtime"
|
||||
"github.com/johnkerl/miller/v6/pkg/types"
|
||||
"github.com/johnkerl/pgpg/go/lib/pkg/asts"
|
||||
)
|
||||
|
||||
// Example ASTs:
|
||||
|
|
@ -158,8 +158,8 @@ type PrintStatementNode struct {
|
|||
outputHandlerManager output.OutputHandlerManager // for file/pipe targets
|
||||
}
|
||||
|
||||
func (root *RootNode) BuildPrintStatementNode(astNode *dsl.ASTNode) (IExecutable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypePrintStatement)
|
||||
func (root *RootNode) BuildPrintStatementNode(astNode *asts.ASTNode) (IExecutable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypePrintStatement))
|
||||
return root.buildPrintxStatementNode(
|
||||
astNode,
|
||||
os.Stdout,
|
||||
|
|
@ -167,8 +167,8 @@ func (root *RootNode) BuildPrintStatementNode(astNode *dsl.ASTNode) (IExecutable
|
|||
)
|
||||
}
|
||||
|
||||
func (root *RootNode) BuildPrintnStatementNode(astNode *dsl.ASTNode) (IExecutable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypePrintnStatement)
|
||||
func (root *RootNode) BuildPrintnStatementNode(astNode *asts.ASTNode) (IExecutable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypePrintnStatement))
|
||||
return root.buildPrintxStatementNode(
|
||||
astNode,
|
||||
os.Stdout,
|
||||
|
|
@ -176,8 +176,8 @@ func (root *RootNode) BuildPrintnStatementNode(astNode *dsl.ASTNode) (IExecutabl
|
|||
)
|
||||
}
|
||||
|
||||
func (root *RootNode) BuildEprintStatementNode(astNode *dsl.ASTNode) (IExecutable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeEprintStatement)
|
||||
func (root *RootNode) BuildEprintStatementNode(astNode *asts.ASTNode) (IExecutable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeEprintStatement))
|
||||
return root.buildPrintxStatementNode(
|
||||
astNode,
|
||||
os.Stderr,
|
||||
|
|
@ -185,8 +185,8 @@ func (root *RootNode) BuildEprintStatementNode(astNode *dsl.ASTNode) (IExecutabl
|
|||
)
|
||||
}
|
||||
|
||||
func (root *RootNode) BuildEprintnStatementNode(astNode *dsl.ASTNode) (IExecutable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeEprintnStatement)
|
||||
func (root *RootNode) BuildEprintnStatementNode(astNode *asts.ASTNode) (IExecutable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeEprintnStatement))
|
||||
return root.buildPrintxStatementNode(
|
||||
astNode,
|
||||
os.Stderr,
|
||||
|
|
@ -197,25 +197,56 @@ func (root *RootNode) BuildEprintnStatementNode(astNode *dsl.ASTNode) (IExecutab
|
|||
// Common code for building print/eprint/printn/eprintn nodes
|
||||
|
||||
func (root *RootNode) buildPrintxStatementNode(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
defaultOutputStream *os.File,
|
||||
terminator string,
|
||||
) (IExecutable, error) {
|
||||
lib.InternalCodingErrorIf(len(astNode.Children) != 2)
|
||||
expressionsNode := astNode.Children[0]
|
||||
redirectorNode := astNode.Children[1]
|
||||
// Normalize PGPG AST to 2-child layout (expressions, redirector).
|
||||
// PGPG produces: 0 children (print), 1 child (print s or print > file), 2+ children (print s,t or print > file, s).
|
||||
// With "print Redirector comma FcnArgs", children are [Redirector, FcnArgs].
|
||||
// With "print FcnArgs" (with_adopted_grandchildren), children are the Rvalues directly.
|
||||
var expressionsNode, redirectorNode *asts.ASTNode
|
||||
isRedirector := func(n *asts.ASTNode) bool {
|
||||
return n.Type == asts.NodeType(NodeTypeRedirectWrite) ||
|
||||
n.Type == asts.NodeType(NodeTypeRedirectAppend) ||
|
||||
n.Type == asts.NodeType(NodeTypeRedirectPipe)
|
||||
}
|
||||
switch len(astNode.Children) {
|
||||
case 0:
|
||||
expressionsNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeNoOp), nil)
|
||||
redirectorNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeNoOp), nil)
|
||||
case 1:
|
||||
if isRedirector(astNode.Children[0]) {
|
||||
expressionsNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeNoOp), nil)
|
||||
redirectorNode = astNode.Children[0]
|
||||
} else {
|
||||
expressionsNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeFcnArgs), astNode.Children)
|
||||
redirectorNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeNoOp), nil)
|
||||
}
|
||||
default:
|
||||
if isRedirector(astNode.Children[0]) {
|
||||
// print Redirector comma FcnArgs -> [Redirector, FcnArgs]
|
||||
expressionsNode = astNode.Children[1]
|
||||
redirectorNode = astNode.Children[0]
|
||||
} else {
|
||||
// print FcnArgs with_adopted_grandchildren -> [Rvalue, Rvalue, ...]
|
||||
expressionsNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeFcnArgs), astNode.Children)
|
||||
redirectorNode = asts.NewASTNode(nil, asts.NodeType(NodeTypeNoOp), nil)
|
||||
}
|
||||
}
|
||||
|
||||
// - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
|
||||
// Things to be printed, e.g. $a and $b in 'print > "foo.dat", $a, $b'.
|
||||
|
||||
var expressionEvaluables []IEvaluable = nil
|
||||
|
||||
if expressionsNode.Type == dsl.NodeTypeNoOp {
|
||||
if expressionsNode.Type == asts.NodeType(NodeTypeNoOp) {
|
||||
// Just 'print' without 'print $something'
|
||||
expressionEvaluables = make([]IEvaluable, 1)
|
||||
expressionEvaluable := root.BuildStringLiteralNode("")
|
||||
expressionEvaluables[0] = expressionEvaluable
|
||||
} else if expressionsNode.Type == dsl.NodeTypeFunctionCallsite {
|
||||
} else if expressionsNode.Type == asts.NodeType(NodeTypeFunctionCallsite) ||
|
||||
expressionsNode.Type == asts.NodeType(NodeTypeFcnArgs) {
|
||||
expressionEvaluables = make([]IEvaluable, len(expressionsNode.Children))
|
||||
for i, childNode := range expressionsNode.Children {
|
||||
expressionEvaluable, err := root.BuildEvaluableNode(childNode)
|
||||
|
|
@ -239,7 +270,7 @@ func (root *RootNode) buildPrintxStatementNode(
|
|||
outputHandlerManager: nil,
|
||||
}
|
||||
|
||||
if redirectorNode.Type == dsl.NodeTypeNoOp {
|
||||
if redirectorNode.Type == asts.NodeType(NodeTypeNoOp) {
|
||||
// No > >> or | was provided.
|
||||
if defaultOutputStream == os.Stdout {
|
||||
retval.printToRedirectFunc = retval.printToStdout
|
||||
|
|
@ -255,23 +286,28 @@ func (root *RootNode) buildPrintxStatementNode(
|
|||
redirectorTargetNode := redirectorNode.Children[0]
|
||||
var err error
|
||||
|
||||
if redirectorTargetNode.Type == dsl.NodeTypeRedirectTargetStdout {
|
||||
if redirectorTargetNode.Type == asts.NodeType(NodeTypeRedirectTargetStdout) {
|
||||
retval.printToRedirectFunc = retval.printToStdout
|
||||
} else if redirectorTargetNode.Type == dsl.NodeTypeRedirectTargetStderr {
|
||||
} else if redirectorTargetNode.Type == asts.NodeType(NodeTypeRedirectTargetStderr) {
|
||||
retval.printToRedirectFunc = retval.printToStderr
|
||||
} else {
|
||||
retval.printToRedirectFunc = retval.printToFileOrPipe
|
||||
|
||||
retval.redirectorTargetEvaluable, err = root.BuildEvaluableNode(redirectorTargetNode)
|
||||
// RedirectTargetRvalue wraps Rvalue; unwrap to build the expression
|
||||
targetNode := redirectorTargetNode
|
||||
if redirectorTargetNode.Type == asts.NodeType(NodeTypeRedirectTargetRvalue) &&
|
||||
redirectorTargetNode.Children != nil && len(redirectorTargetNode.Children) > 0 {
|
||||
targetNode = redirectorTargetNode.Children[0]
|
||||
}
|
||||
retval.redirectorTargetEvaluable, err = root.BuildEvaluableNode(targetNode)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if redirectorNode.Type == dsl.NodeTypeRedirectWrite {
|
||||
if redirectorNode.Type == asts.NodeType(NodeTypeRedirectWrite) {
|
||||
retval.outputHandlerManager = output.NewFileWritetHandlerManager(root.recordWriterOptions)
|
||||
} else if redirectorNode.Type == dsl.NodeTypeRedirectAppend {
|
||||
} else if redirectorNode.Type == asts.NodeType(NodeTypeRedirectAppend) {
|
||||
retval.outputHandlerManager = output.NewFileAppendHandlerManager(root.recordWriterOptions)
|
||||
} else if redirectorNode.Type == dsl.NodeTypeRedirectPipe {
|
||||
} else if redirectorNode.Type == asts.NodeType(NodeTypeRedirectPipe) {
|
||||
retval.outputHandlerManager = output.NewPipeWriteHandlerManager(root.recordWriterOptions)
|
||||
} else {
|
||||
return nil, fmt.Errorf("unhandled redirector node type %s", string(redirectorNode.Type))
|
||||
|
|
|
|||
|
|
@ -10,12 +10,13 @@ import (
|
|||
"strings"
|
||||
|
||||
"github.com/johnkerl/miller/v6/pkg/cli"
|
||||
"github.com/johnkerl/miller/v6/pkg/dsl"
|
||||
"github.com/johnkerl/miller/v6/pkg/mlrval"
|
||||
"github.com/johnkerl/miller/v6/pkg/output"
|
||||
"github.com/johnkerl/miller/v6/pkg/parsing/lexer"
|
||||
"github.com/johnkerl/miller/v6/pkg/parsing/parser"
|
||||
"github.com/johnkerl/miller/v6/pkg/runtime"
|
||||
|
||||
"github.com/johnkerl/pgpg/go/lib/pkg/asts"
|
||||
)
|
||||
|
||||
// NewEmptyRoot sets up an empty CST, before ingesting any DSL strings. For
|
||||
|
|
@ -59,7 +60,7 @@ func (root *RootNode) WithStrictMode(strictMode bool) *RootNode {
|
|||
// ASTBuildVisitorFunc is a callback, used by RootNode's Build method, which
|
||||
// CST-builder callsites can use to visit parse-to-AST result of multi-string
|
||||
// DSL inputs. Nominal use: mlr put -v, mlr put -d, etc.
|
||||
type ASTBuildVisitorFunc func(dslString string, astNode *dsl.AST)
|
||||
type ASTBuildVisitorFunc func(dslString string, astNode *asts.AST)
|
||||
|
||||
// Used by DSL -> AST -> CST callsites including mlr put, mlr filter, and mlr
|
||||
// repl. The RootNode must be separately instantiated (e.g. NewEmptyRoot())
|
||||
|
|
@ -102,7 +103,7 @@ func (root *RootNode) Build(
|
|||
return hadWarnings, err
|
||||
}
|
||||
|
||||
func buildASTFromStringWithMessage(dslString string) (*dsl.AST, error) {
|
||||
func buildASTFromStringWithMessage(dslString string) (*asts.AST, error) {
|
||||
astRootNode, err := buildASTFromString(dslString)
|
||||
if err != nil {
|
||||
// Leave this out until we get better control over the error-messaging.
|
||||
|
|
@ -113,13 +114,13 @@ func buildASTFromStringWithMessage(dslString string) (*dsl.AST, error) {
|
|||
return astRootNode, nil
|
||||
}
|
||||
|
||||
func buildASTFromString(dslString string) (*dsl.AST, error) {
|
||||
func buildASTFromString(dslString string) (*asts.AST, error) {
|
||||
// For non-Windows, already stripped by the shell; helpful here for Windows.
|
||||
if strings.HasPrefix(dslString, "'") && strings.HasSuffix(dslString, "'") {
|
||||
dslString = dslString[1 : len(dslString)-1]
|
||||
}
|
||||
|
||||
// The comment-stripping lex expression in Miller's GOCC grammar matches from '#' to '\n' ... in
|
||||
// The comment-stripping lex expression in Miller's grammar matches from '#' to '\n' ... in
|
||||
// the case where there is a '#', then comment text, then end-of-string without any final
|
||||
// newline, the comment text does _not_ get stripped out and is a parse error.
|
||||
// It's simplest to ensure, here, that DSL-expression strings have a final newline.
|
||||
|
|
@ -128,13 +129,8 @@ func buildASTFromString(dslString string) (*dsl.AST, error) {
|
|||
}
|
||||
|
||||
theLexer := lexer.NewLexer([]byte(dslString))
|
||||
theParser := parser.NewParser()
|
||||
interfaceAST, err := theParser.Parse(theLexer)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
astRootNode := interfaceAST.(*dsl.AST)
|
||||
return astRootNode, nil
|
||||
theParser := parser.NewMlrParser()
|
||||
return theParser.Parse(theLexer, "")
|
||||
}
|
||||
|
||||
// If the user has multiple put -f / put -e pieces, we can AST-parse each
|
||||
|
|
@ -142,7 +138,7 @@ func buildASTFromString(dslString string) (*dsl.AST, error) {
|
|||
// until after they're all ingested -- e.g. first piece calls a function which
|
||||
// the second defines, or mutual recursion across pieces, etc.
|
||||
func (root *RootNode) IngestAST(
|
||||
ast *dsl.AST,
|
||||
ast *asts.AST,
|
||||
// False for non-REPL use. Also false for bulk-load REPL use. True for
|
||||
// interactive REPL statements which are intended to be executed once
|
||||
// (immediately) but not retained.
|
||||
|
|
@ -231,37 +227,33 @@ func (root *RootNode) Resolve() error {
|
|||
// * string literal " "
|
||||
// * string literal "TAB"
|
||||
|
||||
func (root *RootNode) regexProtectPrePass(ast *dsl.AST) {
|
||||
func (root *RootNode) regexProtectPrePass(ast *asts.AST) {
|
||||
root.regexProtectPrePassAux(ast.RootNode)
|
||||
}
|
||||
|
||||
func (root *RootNode) regexProtectPrePassAux(astNode *dsl.ASTNode) {
|
||||
func (root *RootNode) regexProtectPrePassAux(astNode *asts.ASTNode) {
|
||||
|
||||
if len(astNode.Children) == 0 {
|
||||
return
|
||||
}
|
||||
|
||||
isCallsiteOfInterest := false
|
||||
if astNode.Type == dsl.NodeTypeOperator {
|
||||
if astNode.Token != nil {
|
||||
nodeName := string(astNode.Token.Lit)
|
||||
if nodeName == "=~" || nodeName == "!=~" {
|
||||
isCallsiteOfInterest = true
|
||||
}
|
||||
if astNode.Type == asts.NodeType(NodeTypeOperator) {
|
||||
nodeName := tokenLit(astNode)
|
||||
if nodeName == "=~" || nodeName == "!=~" {
|
||||
isCallsiteOfInterest = true
|
||||
}
|
||||
} else if astNode.Type == dsl.NodeTypeFunctionCallsite {
|
||||
if astNode.Token != nil {
|
||||
nodeName := string(astNode.Token.Lit)
|
||||
if nodeName == "sub" || nodeName == "gsub" || nodeName == "regextract" || nodeName == "regextract_or_else" {
|
||||
isCallsiteOfInterest = true
|
||||
}
|
||||
} else if astNode.Type == asts.NodeType(NodeTypeFunctionCallsite) {
|
||||
nodeName := tokenLit(astNode)
|
||||
if nodeName == "sub" || nodeName == "gsub" || nodeName == "regextract" || nodeName == "regextract_or_else" {
|
||||
isCallsiteOfInterest = true
|
||||
}
|
||||
}
|
||||
|
||||
for i, astChild := range astNode.Children {
|
||||
if isCallsiteOfInterest && i == 1 {
|
||||
if astChild.Type == dsl.NodeTypeStringLiteral {
|
||||
astChild.Type = dsl.NodeTypeRegex
|
||||
if astChild.Type == asts.NodeType(NodeTypeStringLiteral) {
|
||||
astChild.Type = asts.NodeType(NodeTypeRegex)
|
||||
}
|
||||
}
|
||||
root.regexProtectPrePassAux(astChild)
|
||||
|
|
@ -273,9 +265,9 @@ func (root *RootNode) regexProtectPrePassAux(astNode *dsl.ASTNode) {
|
|||
// functions may be called before they are defined, so a follow-up pass will
|
||||
// need to resolve those callsites.
|
||||
|
||||
func (root *RootNode) buildMainPass(ast *dsl.AST, isReplImmediate bool) error {
|
||||
func (root *RootNode) buildMainPass(ast *asts.AST, isReplImmediate bool) error {
|
||||
|
||||
if ast.RootNode.Type != dsl.NodeTypeStatementBlock {
|
||||
if ast.RootNode.Type != asts.NodeType(NodeTypeStatementBlock) {
|
||||
return fmt.Errorf("at CST root build: non-statement-block AST root node unhandled")
|
||||
}
|
||||
astChildren := ast.RootNode.Children
|
||||
|
|
@ -304,25 +296,25 @@ func (root *RootNode) buildMainPass(ast *dsl.AST, isReplImmediate bool) error {
|
|||
|
||||
for _, astChild := range astChildren {
|
||||
|
||||
if astChild.Type == dsl.NodeTypeNamedFunctionDefinition {
|
||||
if astChild.Type == asts.NodeType(NodeTypeNamedFunctionDefinition) {
|
||||
err := root.BuildAndInstallUDF(astChild)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
} else if astChild.Type == dsl.NodeTypeSubroutineDefinition {
|
||||
} else if astChild.Type == asts.NodeType(NodeTypeSubroutineDefinition) {
|
||||
err := root.BuildAndInstallUDS(astChild)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
} else if astChild.Type == dsl.NodeTypeBeginBlock || astChild.Type == dsl.NodeTypeEndBlock {
|
||||
} else if astChild.Type == asts.NodeType(NodeTypeBeginBlock) || astChild.Type == asts.NodeType(NodeTypeEndBlock) {
|
||||
statementBlockNode, err := root.BuildStatementBlockNodeFromBeginOrEnd(astChild)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if astChild.Type == dsl.NodeTypeBeginBlock {
|
||||
if astChild.Type == asts.NodeType(NodeTypeBeginBlock) {
|
||||
root.beginBlocks = append(root.beginBlocks, statementBlockNode)
|
||||
} else {
|
||||
root.endBlocks = append(root.endBlocks, statementBlockNode)
|
||||
|
|
|
|||
|
|
@ -6,24 +6,30 @@ package cst
|
|||
import (
|
||||
"fmt"
|
||||
|
||||
"github.com/johnkerl/miller/v6/pkg/dsl"
|
||||
"github.com/johnkerl/pgpg/go/lib/pkg/asts"
|
||||
)
|
||||
|
||||
func (root *RootNode) BuildStatementNode(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
) (IExecutable, error) {
|
||||
|
||||
var statement IExecutable = nil
|
||||
var err error
|
||||
switch astNode.Type {
|
||||
|
||||
case dsl.NodeTypeAssignment:
|
||||
case asts.NodeType(NodeTypeAssignment):
|
||||
statement, err = root.BuildAssignmentNode(astNode)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
case dsl.NodeTypeUnset:
|
||||
case asts.NodeType(NodeTypeCompoundAssignment):
|
||||
statement, err = root.BuildCompoundAssignmentNode(astNode)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
case asts.NodeType(NodeTypeUnset):
|
||||
statement, err = root.BuildUnsetNode(astNode)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
|
|
@ -32,71 +38,71 @@ func (root *RootNode) BuildStatementNode(
|
|||
// E.g. 'NR > 10' without if or '{...}' body. For put, these are no-ops
|
||||
// except side-effects (like regex-captures); for filter, they set the
|
||||
// filter condition only if they're the last statement in the main block.
|
||||
case dsl.NodeTypeBareBoolean:
|
||||
case asts.NodeType(NodeTypeBareBoolean):
|
||||
return root.BuildBareBooleanStatementNode(astNode)
|
||||
// E.g. 'filter NR > 10'.
|
||||
case dsl.NodeTypeFilterStatement:
|
||||
case asts.NodeType(NodeTypeFilterStatement):
|
||||
return root.BuildFilterStatementNode(astNode)
|
||||
|
||||
case dsl.NodeTypePrintStatement:
|
||||
case asts.NodeType(NodeTypePrintStatement):
|
||||
return root.BuildPrintStatementNode(astNode)
|
||||
case dsl.NodeTypePrintnStatement:
|
||||
case asts.NodeType(NodeTypePrintnStatement):
|
||||
return root.BuildPrintnStatementNode(astNode)
|
||||
case dsl.NodeTypeEprintStatement:
|
||||
case asts.NodeType(NodeTypeEprintStatement):
|
||||
return root.BuildEprintStatementNode(astNode)
|
||||
case dsl.NodeTypeEprintnStatement:
|
||||
case asts.NodeType(NodeTypeEprintnStatement):
|
||||
return root.BuildEprintnStatementNode(astNode)
|
||||
|
||||
case dsl.NodeTypeDumpStatement:
|
||||
case asts.NodeType(NodeTypeDumpStatement):
|
||||
return root.BuildDumpStatementNode(astNode)
|
||||
case dsl.NodeTypeEdumpStatement:
|
||||
case asts.NodeType(NodeTypeEdumpStatement):
|
||||
return root.BuildEdumpStatementNode(astNode)
|
||||
|
||||
case dsl.NodeTypeTeeStatement:
|
||||
case asts.NodeType(NodeTypeTeeStatement):
|
||||
return root.BuildTeeStatementNode(astNode)
|
||||
case dsl.NodeTypeEmit1Statement:
|
||||
case asts.NodeType(NodeTypeEmit1Statement):
|
||||
return root.BuildEmit1StatementNode(astNode)
|
||||
case dsl.NodeTypeEmitStatement:
|
||||
case asts.NodeType(NodeTypeEmitStatement):
|
||||
return root.BuildEmitStatementNode(astNode)
|
||||
case dsl.NodeTypeEmitFStatement:
|
||||
case asts.NodeType(NodeTypeEmitFStatement):
|
||||
return root.BuildEmitFStatementNode(astNode)
|
||||
case dsl.NodeTypeEmitPStatement:
|
||||
case asts.NodeType(NodeTypeEmitPStatement):
|
||||
return root.BuildEmitPStatementNode(astNode)
|
||||
|
||||
case dsl.NodeTypeBeginBlock:
|
||||
case asts.NodeType(NodeTypeBeginBlock):
|
||||
return nil, fmt.Errorf("begin blocks may only be declared at top level")
|
||||
case dsl.NodeTypeEndBlock:
|
||||
case asts.NodeType(NodeTypeEndBlock):
|
||||
return nil, fmt.Errorf("end blocks may only be declared at top level")
|
||||
|
||||
case dsl.NodeTypeIfChain:
|
||||
case asts.NodeType(NodeTypeIfChain):
|
||||
return root.BuildIfChainNode(astNode)
|
||||
case dsl.NodeTypeCondBlock:
|
||||
case asts.NodeType(NodeTypeCondBlock):
|
||||
return root.BuildCondBlockNode(astNode)
|
||||
case dsl.NodeTypeWhileLoop:
|
||||
case asts.NodeType(NodeTypeWhileLoop):
|
||||
return root.BuildWhileLoopNode(astNode)
|
||||
case dsl.NodeTypeDoWhileLoop:
|
||||
case asts.NodeType(NodeTypeDoWhileLoop):
|
||||
return root.BuildDoWhileLoopNode(astNode)
|
||||
case dsl.NodeTypeForLoopOneVariable:
|
||||
case asts.NodeType(NodeTypeForLoopOneVariable):
|
||||
return root.BuildForLoopOneVariableNode(astNode)
|
||||
case dsl.NodeTypeForLoopTwoVariable:
|
||||
case asts.NodeType(NodeTypeForLoopTwoVariable):
|
||||
return root.BuildForLoopTwoVariableNode(astNode)
|
||||
case dsl.NodeTypeForLoopMultivariable:
|
||||
case asts.NodeType(NodeTypeForLoopMultivariable):
|
||||
return root.BuildForLoopMultivariableNode(astNode)
|
||||
case dsl.NodeTypeTripleForLoop:
|
||||
case asts.NodeType(NodeTypeTripleForLoop):
|
||||
return root.BuildTripleForLoopNode(astNode)
|
||||
|
||||
case dsl.NodeTypeNamedFunctionDefinition:
|
||||
case asts.NodeType(NodeTypeNamedFunctionDefinition):
|
||||
return nil, fmt.Errorf("functions may only be declared at top level")
|
||||
case dsl.NodeTypeSubroutineDefinition:
|
||||
case asts.NodeType(NodeTypeSubroutineDefinition):
|
||||
return nil, fmt.Errorf("subroutines may only be declared at top level")
|
||||
case dsl.NodeTypeSubroutineCallsite:
|
||||
case asts.NodeType(NodeTypeSubroutineCallsite):
|
||||
return root.BuildSubroutineCallsiteNode(astNode)
|
||||
|
||||
case dsl.NodeTypeBreak:
|
||||
case asts.NodeType(NodeTypeBreakStatement):
|
||||
return root.BuildBreakNode(astNode)
|
||||
case dsl.NodeTypeContinue:
|
||||
case asts.NodeType(NodeTypeContinueStatement):
|
||||
return root.BuildContinueNode(astNode)
|
||||
case dsl.NodeTypeReturn:
|
||||
case asts.NodeType(NodeTypeReturnStatement):
|
||||
return root.BuildReturnNode(astNode)
|
||||
|
||||
default:
|
||||
|
|
|
|||
|
|
@ -7,8 +7,8 @@
|
|||
package cst
|
||||
|
||||
import (
|
||||
"github.com/johnkerl/miller/v6/pkg/dsl"
|
||||
"github.com/johnkerl/miller/v6/pkg/lib"
|
||||
"github.com/johnkerl/pgpg/go/lib/pkg/asts"
|
||||
)
|
||||
|
||||
// Subroutine lookup:
|
||||
|
|
@ -22,15 +22,15 @@ import (
|
|||
// o On a next pass, we will walk that list resolving against all encountered
|
||||
// UDS definitions. (It will be an error then if it's still unresolvable.)
|
||||
|
||||
func (root *RootNode) BuildSubroutineCallsiteNode(astNode *dsl.ASTNode) (IExecutable, error) {
|
||||
func (root *RootNode) BuildSubroutineCallsiteNode(astNode *asts.ASTNode) (IExecutable, error) {
|
||||
lib.InternalCodingErrorIf(
|
||||
astNode.Type != dsl.NodeTypeSubroutineCallsite &&
|
||||
astNode.Type != dsl.NodeTypeOperator,
|
||||
astNode.Type != asts.NodeType(NodeTypeSubroutineCallsite) &&
|
||||
astNode.Type != asts.NodeType(NodeTypeOperator),
|
||||
)
|
||||
lib.InternalCodingErrorIf(astNode.Token == nil)
|
||||
lib.InternalCodingErrorIf(astNode.Children == nil)
|
||||
|
||||
subroutineName := string(astNode.Token.Lit)
|
||||
subroutineName := tokenLit(astNode)
|
||||
|
||||
// - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
|
||||
// Look for a user-defined subroutine with the given name.
|
||||
|
|
|
|||
|
|
@ -5,12 +5,12 @@ package cst
|
|||
import (
|
||||
"fmt"
|
||||
|
||||
"github.com/johnkerl/miller/v6/pkg/dsl"
|
||||
"github.com/johnkerl/miller/v6/pkg/lib"
|
||||
"github.com/johnkerl/miller/v6/pkg/mlrval"
|
||||
"github.com/johnkerl/miller/v6/pkg/output"
|
||||
"github.com/johnkerl/miller/v6/pkg/runtime"
|
||||
"github.com/johnkerl/miller/v6/pkg/types"
|
||||
"github.com/johnkerl/pgpg/go/lib/pkg/asts"
|
||||
)
|
||||
|
||||
// Examples:
|
||||
|
|
@ -64,16 +64,17 @@ type TeeStatementNode struct {
|
|||
outputHandlerManager output.OutputHandlerManager // for file/pipe targets
|
||||
}
|
||||
|
||||
func (root *RootNode) BuildTeeStatementNode(astNode *dsl.ASTNode) (IExecutable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeTeeStatement)
|
||||
func (root *RootNode) BuildTeeStatementNode(astNode *asts.ASTNode) (IExecutable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeTeeStatement))
|
||||
lib.InternalCodingErrorIf(len(astNode.Children) != 2)
|
||||
expressionNode := astNode.Children[0]
|
||||
redirectorNode := astNode.Children[1]
|
||||
// PGPG: kw_tee Redirector comma FullSrec -> children [1,3] = [Redirector, FullSrec]
|
||||
redirectorNode := astNode.Children[0]
|
||||
expressionNode := astNode.Children[1]
|
||||
|
||||
// - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
|
||||
// Expression to be teed, which is $*.
|
||||
|
||||
lib.InternalCodingErrorIf(expressionNode.Type != dsl.NodeTypeFullSrec)
|
||||
lib.InternalCodingErrorIf(expressionNode.Type != asts.NodeType(NodeTypeFullSrec))
|
||||
expressionEvaluable := root.BuildFullSrecRvalueNode()
|
||||
|
||||
// - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
|
||||
|
|
@ -92,27 +93,31 @@ func (root *RootNode) BuildTeeStatementNode(astNode *dsl.ASTNode) (IExecutable,
|
|||
redirectorTargetNode := redirectorNode.Children[0]
|
||||
var err error
|
||||
|
||||
if redirectorTargetNode.Type == dsl.NodeTypeRedirectTargetStdout {
|
||||
if redirectorTargetNode.Type == asts.NodeType(NodeTypeRedirectTargetStdout) {
|
||||
retval.teeToRedirectFunc = retval.teeToFileOrPipe
|
||||
retval.outputHandlerManager = output.NewStdoutWriteHandlerManager(root.recordWriterOptions)
|
||||
retval.redirectorTargetEvaluable = root.BuildStringLiteralNode("(stdout)")
|
||||
} else if redirectorTargetNode.Type == dsl.NodeTypeRedirectTargetStderr {
|
||||
} else if redirectorTargetNode.Type == asts.NodeType(NodeTypeRedirectTargetStderr) {
|
||||
retval.teeToRedirectFunc = retval.teeToFileOrPipe
|
||||
retval.outputHandlerManager = output.NewStderrWriteHandlerManager(root.recordWriterOptions)
|
||||
retval.redirectorTargetEvaluable = root.BuildStringLiteralNode("(stderr)")
|
||||
} else {
|
||||
retval.teeToRedirectFunc = retval.teeToFileOrPipe
|
||||
|
||||
retval.redirectorTargetEvaluable, err = root.BuildEvaluableNode(redirectorTargetNode)
|
||||
targetNode := redirectorTargetNode
|
||||
if redirectorTargetNode.Type == asts.NodeType(NodeTypeRedirectTargetRvalue) &&
|
||||
redirectorTargetNode.Children != nil && len(redirectorTargetNode.Children) > 0 {
|
||||
targetNode = redirectorTargetNode.Children[0]
|
||||
}
|
||||
retval.redirectorTargetEvaluable, err = root.BuildEvaluableNode(targetNode)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if redirectorNode.Type == dsl.NodeTypeRedirectWrite {
|
||||
if redirectorNode.Type == asts.NodeType(NodeTypeRedirectWrite) {
|
||||
retval.outputHandlerManager = output.NewFileWritetHandlerManager(root.recordWriterOptions)
|
||||
} else if redirectorNode.Type == dsl.NodeTypeRedirectAppend {
|
||||
} else if redirectorNode.Type == asts.NodeType(NodeTypeRedirectAppend) {
|
||||
retval.outputHandlerManager = output.NewFileAppendHandlerManager(root.recordWriterOptions)
|
||||
} else if redirectorNode.Type == dsl.NodeTypeRedirectPipe {
|
||||
} else if redirectorNode.Type == asts.NodeType(NodeTypeRedirectPipe) {
|
||||
retval.outputHandlerManager = output.NewPipeWriteHandlerManager(root.recordWriterOptions)
|
||||
} else {
|
||||
return nil, fmt.Errorf("unhandled redirector node type %s", string(redirectorNode.Type))
|
||||
|
|
|
|||
|
|
@ -4,10 +4,10 @@ package cst
|
|||
|
||||
import (
|
||||
"github.com/johnkerl/miller/v6/pkg/cli"
|
||||
"github.com/johnkerl/miller/v6/pkg/dsl"
|
||||
"github.com/johnkerl/miller/v6/pkg/mlrval"
|
||||
"github.com/johnkerl/miller/v6/pkg/output"
|
||||
"github.com/johnkerl/miller/v6/pkg/runtime"
|
||||
"github.com/johnkerl/pgpg/go/lib/pkg/asts"
|
||||
)
|
||||
|
||||
// DSLInstanceType is for minor differences in DSL handling between mlr put,
|
||||
|
|
@ -50,7 +50,7 @@ type RootNode struct {
|
|||
|
||||
// Many functions have this signature. This type-alias is for function-name
|
||||
// lookup tables.
|
||||
type NodeBuilder func(astNode *dsl.ASTNode) (IEvaluable, error)
|
||||
type NodeBuilder func(astNode *asts.ASTNode) (IEvaluable, error)
|
||||
|
||||
// This is for all statements and statement blocks within the CST.
|
||||
type IExecutable interface {
|
||||
|
|
|
|||
|
|
@ -6,11 +6,11 @@ import (
|
|||
"fmt"
|
||||
"os"
|
||||
|
||||
"github.com/johnkerl/miller/v6/pkg/dsl"
|
||||
"github.com/johnkerl/miller/v6/pkg/lib"
|
||||
"github.com/johnkerl/miller/v6/pkg/mlrval"
|
||||
"github.com/johnkerl/miller/v6/pkg/runtime"
|
||||
"github.com/johnkerl/miller/v6/pkg/types"
|
||||
"github.com/johnkerl/pgpg/go/lib/pkg/asts"
|
||||
)
|
||||
|
||||
type UDF struct {
|
||||
|
|
@ -391,12 +391,12 @@ func (mgr *UDFManager) LookUpDisregardingArity(functionName string) *UDF {
|
|||
// * DirectFieldValue "x"
|
||||
|
||||
// BuildAndInstallUDF is for named UDFs, like `func f(a, b) { return b - a}'.
|
||||
func (root *RootNode) BuildAndInstallUDF(astNode *dsl.ASTNode) error {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeNamedFunctionDefinition)
|
||||
func (root *RootNode) BuildAndInstallUDF(astNode *asts.ASTNode) error {
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeNamedFunctionDefinition))
|
||||
lib.InternalCodingErrorIf(astNode.Children == nil)
|
||||
lib.InternalCodingErrorIf(len(astNode.Children) != 2 && len(astNode.Children) != 3)
|
||||
|
||||
functionName := string(astNode.Token.Lit)
|
||||
functionName := tokenLit(astNode)
|
||||
|
||||
if BuiltinFunctionManagerInstance.LookUp(functionName) != nil {
|
||||
return fmt.Errorf(
|
||||
|
|
@ -438,8 +438,8 @@ type UnnamedUDFNode struct {
|
|||
udfAsMlrval *mlrval.Mlrval
|
||||
}
|
||||
|
||||
func (root *RootNode) BuildUnnamedUDFNode(astNode *dsl.ASTNode) (IEvaluable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeUnnamedFunctionDefinition)
|
||||
func (root *RootNode) BuildUnnamedUDFNode(astNode *asts.ASTNode) (IEvaluable, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeUnnamedFunctionDefinition))
|
||||
|
||||
name := genFunctionLiteralName()
|
||||
|
||||
|
|
@ -459,28 +459,82 @@ func (node *UnnamedUDFNode) Evaluate(state *runtime.State) *mlrval.Mlrval {
|
|||
return node.udfAsMlrval
|
||||
}
|
||||
|
||||
// typedDeclNodeTypeToName maps PGPG type-keyword node types to TypeNameToMask names.
|
||||
func typedDeclNodeTypeToName(nodeType string) string {
|
||||
switch nodeType {
|
||||
case "kw_int", "int":
|
||||
return "int"
|
||||
case "kw_float", "float":
|
||||
return "float"
|
||||
case "kw_num":
|
||||
return "num"
|
||||
case "kw_bool", "bool":
|
||||
return "bool"
|
||||
case "kw_str", "str":
|
||||
return "str"
|
||||
case "kw_arr", "arr":
|
||||
return "arr"
|
||||
case "kw_map", "map":
|
||||
return "map"
|
||||
case "kw_var", "var":
|
||||
return "var"
|
||||
case "kw_funct", "funct":
|
||||
return "funct"
|
||||
case "Typedecl":
|
||||
return "any"
|
||||
}
|
||||
return "any"
|
||||
}
|
||||
|
||||
// flattenParameterList recurses through nested ParameterList nodes and returns
|
||||
// a flat list of Parameter or LocalVariable nodes. PGPG produces (params (params (Parameter) (Parameter))).
|
||||
func flattenParameterList(nodes []*asts.ASTNode) []*asts.ASTNode {
|
||||
var out []*asts.ASTNode
|
||||
for _, n := range nodes {
|
||||
if n == nil {
|
||||
continue
|
||||
}
|
||||
if string(n.Type) == NodeTypeParameterList || n.Type == asts.NodeType(NodeTypeParameterList) {
|
||||
if n.Children != nil {
|
||||
out = append(out, flattenParameterList(n.Children)...)
|
||||
}
|
||||
continue
|
||||
}
|
||||
out = append(out, n)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// BuildUDF is for named UDFs, like `func f(a, b) { return b - a}', or,
|
||||
// unnamed UDFs like `func (a, b) { return b - a }'.
|
||||
func (root *RootNode) BuildUDF(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
functionName string,
|
||||
isFunctionLiteral bool,
|
||||
) (*UDF, error) {
|
||||
lib.InternalCodingErrorIf(
|
||||
(astNode.Type != dsl.NodeTypeNamedFunctionDefinition) &&
|
||||
(astNode.Type != dsl.NodeTypeUnnamedFunctionDefinition))
|
||||
(astNode.Type != asts.NodeType(NodeTypeNamedFunctionDefinition)) &&
|
||||
(astNode.Type != asts.NodeType(NodeTypeUnnamedFunctionDefinition)))
|
||||
|
||||
lib.InternalCodingErrorIf(astNode.Children == nil)
|
||||
lib.InternalCodingErrorIf(len(astNode.Children) != 2 && len(astNode.Children) != 3)
|
||||
|
||||
parameterListASTNode := astNode.Children[0]
|
||||
functionBodyASTNode := astNode.Children[1]
|
||||
|
||||
var functionBodyASTNode *asts.ASTNode
|
||||
returnValueTypeName := "any"
|
||||
if len(astNode.Children) == 3 {
|
||||
typeNode := astNode.Children[2]
|
||||
lib.InternalCodingErrorIf(typeNode.Type != dsl.NodeTypeTypedecl)
|
||||
returnValueTypeName = string(typeNode.Token.Lit)
|
||||
// PGPG: children [2, 4, 5] = [FuncParams, Typedecl, StatementBlockInBraces]
|
||||
typeNode := astNode.Children[1]
|
||||
functionBodyASTNode = astNode.Children[2]
|
||||
returnValueTypeName = tokenLit(typeNode)
|
||||
if returnValueTypeName == "" && typeNode.Children != nil && len(typeNode.Children) > 0 {
|
||||
returnValueTypeName = tokenLit(typeNode.Children[0])
|
||||
}
|
||||
if returnValueTypeName == "" {
|
||||
returnValueTypeName = typedDeclNodeTypeToName(string(typeNode.Type))
|
||||
}
|
||||
} else {
|
||||
functionBodyASTNode = astNode.Children[1]
|
||||
}
|
||||
typeGatedReturnValue, err := types.NewTypeGatedMlrvalName(
|
||||
"function return value",
|
||||
|
|
@ -490,24 +544,60 @@ func (root *RootNode) BuildUDF(
|
|||
return nil, err
|
||||
}
|
||||
|
||||
lib.InternalCodingErrorIf(parameterListASTNode.Type != dsl.NodeTypeParameterList)
|
||||
// PGPG may use different type strings; accept ParameterList by structure
|
||||
if string(parameterListASTNode.Type) != NodeTypeParameterList && parameterListASTNode.Type != asts.NodeType(NodeTypeParameterList) {
|
||||
lib.InternalCodingErrorWithMessageIf(true, "expected ParameterList node")
|
||||
}
|
||||
lib.InternalCodingErrorIf(parameterListASTNode.Children == nil)
|
||||
arity := len(parameterListASTNode.Children)
|
||||
typeGatedParameterNames := make([]*types.TypeGatedMlrvalName, arity)
|
||||
for i, parameterASTNode := range parameterListASTNode.Children {
|
||||
lib.InternalCodingErrorIf(parameterASTNode.Type != dsl.NodeTypeParameter)
|
||||
lib.InternalCodingErrorIf(parameterASTNode.Children == nil)
|
||||
lib.InternalCodingErrorIf(len(parameterASTNode.Children) != 1)
|
||||
typeGatedParameterNameASTNode := parameterASTNode.Children[0]
|
||||
|
||||
lib.InternalCodingErrorIf(typeGatedParameterNameASTNode.Type != dsl.NodeTypeParameterName)
|
||||
variableName := string(typeGatedParameterNameASTNode.Token.Lit)
|
||||
typeName := "any"
|
||||
if typeGatedParameterNameASTNode.Children != nil { // typed parameter like 'num x'
|
||||
lib.InternalCodingErrorIf(len(typeGatedParameterNameASTNode.Children) != 1)
|
||||
typeNode := typeGatedParameterNameASTNode.Children[0]
|
||||
lib.InternalCodingErrorIf(typeNode.Type != dsl.NodeTypeTypedecl)
|
||||
typeName = string(typeNode.Token.Lit)
|
||||
// PGPG produces nested ParameterList (params contains params contains Parameter, Parameter).
|
||||
// Flatten to a list of Parameter or LocalVariable nodes.
|
||||
flatParams := flattenParameterList(parameterListASTNode.Children)
|
||||
arity := len(flatParams)
|
||||
typeGatedParameterNames := make([]*types.TypeGatedMlrvalName, arity)
|
||||
for i, parameterASTNode := range flatParams {
|
||||
// PGPG: Parameter has one child (LocalVariable), or two (Typedecl, LocalVariable) for typed params.
|
||||
var variableName string
|
||||
var typeName string = "any"
|
||||
if parameterASTNode.Children != nil && len(parameterASTNode.Children) == 2 {
|
||||
// Typedecl LocalVariable -> [Typedecl, LocalVariable]
|
||||
typeNode := parameterASTNode.Children[0]
|
||||
nameNode := parameterASTNode.Children[1]
|
||||
typeName = tokenLit(typeNode)
|
||||
if typeName == "" && typeNode.Children != nil && len(typeNode.Children) > 0 {
|
||||
typeName = tokenLit(typeNode.Children[0])
|
||||
}
|
||||
if typeName == "" {
|
||||
// PGPG may produce type keywords (kw_int etc) without token on node; use node type
|
||||
typeName = typedDeclNodeTypeToName(string(typeNode.Type))
|
||||
}
|
||||
if string(nameNode.Type) == NodeTypeLocalVariable || nameNode.Type == asts.NodeType(NodeTypeLocalVariable) {
|
||||
variableName = tokenLit(nameNode)
|
||||
}
|
||||
} else if parameterASTNode.Children != nil && len(parameterASTNode.Children) == 1 {
|
||||
typeGatedParameterNameASTNode := parameterASTNode.Children[0]
|
||||
if string(typeGatedParameterNameASTNode.Type) != NodeTypeLocalVariable &&
|
||||
typeGatedParameterNameASTNode.Type != asts.NodeType(NodeTypeLocalVariable) {
|
||||
lib.InternalCodingErrorWithMessageIf(true, "expected LocalVariable as parameter name")
|
||||
}
|
||||
variableName = tokenLit(typeGatedParameterNameASTNode)
|
||||
} else if parameterASTNode.Children == nil || len(parameterASTNode.Children) == 0 {
|
||||
// Direct LocalVariable (e.g. from with_prepended_children flattening)
|
||||
if string(parameterASTNode.Type) != NodeTypeLocalVariable &&
|
||||
parameterASTNode.Type != asts.NodeType(NodeTypeLocalVariable) {
|
||||
lib.InternalCodingErrorWithMessageIf(true, "expected Parameter or LocalVariable")
|
||||
}
|
||||
variableName = tokenLit(parameterASTNode)
|
||||
} else {
|
||||
// PGPG may produce Parameter with multiple children; try first child, then node itself
|
||||
typeGatedParameterNameASTNode := parameterASTNode.Children[0]
|
||||
variableName = tokenLit(typeGatedParameterNameASTNode)
|
||||
if variableName == "" {
|
||||
variableName = tokenLit(parameterASTNode)
|
||||
}
|
||||
if variableName == "" {
|
||||
lib.InternalCodingErrorWithMessageIf(true, "expected Parameter node with one child")
|
||||
}
|
||||
}
|
||||
typeGatedParameterName, err := types.NewTypeGatedMlrvalName(
|
||||
variableName,
|
||||
|
|
|
|||
|
|
@ -5,11 +5,11 @@ package cst
|
|||
import (
|
||||
"fmt"
|
||||
|
||||
"github.com/johnkerl/miller/v6/pkg/dsl"
|
||||
"github.com/johnkerl/miller/v6/pkg/lib"
|
||||
"github.com/johnkerl/miller/v6/pkg/mlrval"
|
||||
"github.com/johnkerl/miller/v6/pkg/runtime"
|
||||
"github.com/johnkerl/miller/v6/pkg/types"
|
||||
"github.com/johnkerl/pgpg/go/lib/pkg/asts"
|
||||
)
|
||||
|
||||
type UDS struct {
|
||||
|
|
@ -228,12 +228,12 @@ func (mgr *UDSManager) ExistsByName(name string) bool {
|
|||
// * SubroutineCallsite "f"
|
||||
// * DirectFieldValue "x"
|
||||
|
||||
func (root *RootNode) BuildAndInstallUDS(astNode *dsl.ASTNode) error {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeSubroutineDefinition)
|
||||
func (root *RootNode) BuildAndInstallUDS(astNode *asts.ASTNode) error {
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeSubroutineDefinition))
|
||||
lib.InternalCodingErrorIf(astNode.Children == nil)
|
||||
lib.InternalCodingErrorIf(len(astNode.Children) != 2 && len(astNode.Children) != 3)
|
||||
|
||||
subroutineName := string(astNode.Token.Lit)
|
||||
subroutineName := tokenLit(astNode)
|
||||
|
||||
if !root.allowUDFUDSRedefinitions {
|
||||
if root.udsManager.ExistsByName(subroutineName) {
|
||||
|
|
@ -247,24 +247,34 @@ func (root *RootNode) BuildAndInstallUDS(astNode *dsl.ASTNode) error {
|
|||
parameterListASTNode := astNode.Children[0]
|
||||
subroutineBodyASTNode := astNode.Children[1]
|
||||
|
||||
lib.InternalCodingErrorIf(parameterListASTNode.Type != dsl.NodeTypeParameterList)
|
||||
lib.InternalCodingErrorIf(parameterListASTNode.Type != asts.NodeType(NodeTypeParameterList))
|
||||
lib.InternalCodingErrorIf(parameterListASTNode.Children == nil)
|
||||
arity := len(parameterListASTNode.Children)
|
||||
flatParams := flattenParameterList(parameterListASTNode.Children)
|
||||
arity := len(flatParams)
|
||||
typeGatedParameterNames := make([]*types.TypeGatedMlrvalName, arity)
|
||||
for i, parameterASTNode := range parameterListASTNode.Children {
|
||||
lib.InternalCodingErrorIf(parameterASTNode.Type != dsl.NodeTypeParameter)
|
||||
lib.InternalCodingErrorIf(parameterASTNode.Children == nil)
|
||||
lib.InternalCodingErrorIf(len(parameterASTNode.Children) != 1)
|
||||
typeGatedParameterNameASTNode := parameterASTNode.Children[0]
|
||||
|
||||
lib.InternalCodingErrorIf(typeGatedParameterNameASTNode.Type != dsl.NodeTypeParameterName)
|
||||
variableName := string(typeGatedParameterNameASTNode.Token.Lit)
|
||||
typeName := "any"
|
||||
if typeGatedParameterNameASTNode.Children != nil { // typed parameter like 'num x'
|
||||
lib.InternalCodingErrorIf(len(typeGatedParameterNameASTNode.Children) != 1)
|
||||
typeNode := typeGatedParameterNameASTNode.Children[0]
|
||||
lib.InternalCodingErrorIf(typeNode.Type != dsl.NodeTypeTypedecl)
|
||||
typeName = string(typeNode.Token.Lit)
|
||||
for i, parameterASTNode := range flatParams {
|
||||
var variableName string
|
||||
var typeName string = "any"
|
||||
if parameterASTNode.Children != nil && len(parameterASTNode.Children) == 2 {
|
||||
// Typedecl LocalVariable -> [Typedecl, LocalVariable]
|
||||
typeNode := parameterASTNode.Children[0]
|
||||
nameNode := parameterASTNode.Children[1]
|
||||
typeName = tokenLit(typeNode)
|
||||
if typeName == "" && typeNode.Children != nil && len(typeNode.Children) > 0 {
|
||||
typeName = tokenLit(typeNode.Children[0])
|
||||
}
|
||||
if typeName == "" {
|
||||
typeName = typedDeclNodeTypeToName(string(typeNode.Type))
|
||||
}
|
||||
if string(nameNode.Type) == NodeTypeLocalVariable || nameNode.Type == asts.NodeType(NodeTypeLocalVariable) {
|
||||
variableName = tokenLit(nameNode)
|
||||
}
|
||||
} else if parameterASTNode.Children != nil && len(parameterASTNode.Children) == 1 {
|
||||
typeGatedParameterNameASTNode := parameterASTNode.Children[0]
|
||||
lib.InternalCodingErrorIf(typeGatedParameterNameASTNode.Type != asts.NodeType(NodeTypeLocalVariable))
|
||||
variableName = tokenLit(typeGatedParameterNameASTNode)
|
||||
} else {
|
||||
lib.InternalCodingErrorWithMessageIf(true, "expected Parameter with 1 or 2 children")
|
||||
}
|
||||
typeGatedParameterName, err := types.NewTypeGatedMlrvalName(
|
||||
variableName,
|
||||
|
|
@ -273,7 +283,6 @@ func (root *RootNode) BuildAndInstallUDS(astNode *dsl.ASTNode) error {
|
|||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
typeGatedParameterNames[i] = typeGatedParameterName
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -7,12 +7,13 @@ package cst
|
|||
import (
|
||||
"fmt"
|
||||
|
||||
"github.com/johnkerl/miller/v6/pkg/dsl"
|
||||
"github.com/johnkerl/miller/v6/pkg/lib"
|
||||
|
||||
"github.com/johnkerl/pgpg/go/lib/pkg/asts"
|
||||
)
|
||||
|
||||
func ValidateAST(
|
||||
ast *dsl.AST,
|
||||
ast *asts.AST,
|
||||
dslInstanceType DSLInstanceType, // mlr put, mlr filter, mlr repl
|
||||
) error {
|
||||
atTopLevel := true
|
||||
|
|
@ -54,7 +55,7 @@ func ValidateAST(
|
|||
}
|
||||
|
||||
func validateASTAux(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
dslInstanceType DSLInstanceType, // mlr put, mlr filter, mlr repl
|
||||
atTopLevel bool,
|
||||
inLoop bool,
|
||||
|
|
@ -73,7 +74,7 @@ func validateASTAux(
|
|||
nextLevelIsAssignmentLHS := isAssignmentLHS
|
||||
nextLevelIsUnset := isUnset
|
||||
|
||||
if astNode.Type == dsl.NodeTypeFilterStatement {
|
||||
if astNode.Type == asts.NodeType(NodeTypeFilterStatement) {
|
||||
if dslInstanceType == DSLInstanceTypeFilter {
|
||||
return fmt.Errorf(
|
||||
`filter expressions must not also contain the "filter" keyword`,
|
||||
|
|
@ -82,42 +83,42 @@ func validateASTAux(
|
|||
}
|
||||
|
||||
// Check: begin/end/func/subr must be at top-level
|
||||
if astNode.Type == dsl.NodeTypeBeginBlock {
|
||||
if astNode.Type == asts.NodeType(NodeTypeBeginBlock) {
|
||||
if !atTopLevel {
|
||||
return fmt.Errorf(
|
||||
"begin blocks can only be at top level",
|
||||
)
|
||||
}
|
||||
nextLevelInBeginOrEnd = true
|
||||
} else if astNode.Type == dsl.NodeTypeEndBlock {
|
||||
} else if astNode.Type == asts.NodeType(NodeTypeEndBlock) {
|
||||
if !atTopLevel {
|
||||
return fmt.Errorf(
|
||||
"end blocks can only be at top level",
|
||||
)
|
||||
}
|
||||
nextLevelInBeginOrEnd = true
|
||||
} else if astNode.Type == dsl.NodeTypeNamedFunctionDefinition {
|
||||
} else if astNode.Type == asts.NodeType(NodeTypeNamedFunctionDefinition) {
|
||||
if !atTopLevel {
|
||||
return fmt.Errorf(
|
||||
"func blocks can only be at top level",
|
||||
)
|
||||
}
|
||||
nextLevelInUDF = true
|
||||
} else if astNode.Type == dsl.NodeTypeUnnamedFunctionDefinition {
|
||||
} else if astNode.Type == asts.NodeType(NodeTypeUnnamedFunctionDefinition) {
|
||||
nextLevelInUDF = true
|
||||
} else if astNode.Type == dsl.NodeTypeSubroutineDefinition {
|
||||
} else if astNode.Type == asts.NodeType(NodeTypeSubroutineDefinition) {
|
||||
if !atTopLevel {
|
||||
return fmt.Errorf(
|
||||
"subr blocks can only be at top level",
|
||||
)
|
||||
}
|
||||
nextLevelInUDS = true
|
||||
} else if astNode.Type == dsl.NodeTypeForLoopTwoVariable {
|
||||
} else if astNode.Type == asts.NodeType(NodeTypeForLoopTwoVariable) {
|
||||
err := validateForLoopTwoVariableUniqueNames(astNode)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
} else if astNode.Type == dsl.NodeTypeForLoopMultivariable {
|
||||
} else if astNode.Type == asts.NodeType(NodeTypeForLoopMultivariable) {
|
||||
err := validateForLoopMultivariableUniqueNames(astNode)
|
||||
if err != nil {
|
||||
return err
|
||||
|
|
@ -126,9 +127,9 @@ func validateASTAux(
|
|||
|
||||
// Check: $-anything cannot be in begin/end
|
||||
if inBeginOrEnd {
|
||||
if astNode.Type == dsl.NodeTypeDirectFieldValue ||
|
||||
astNode.Type == dsl.NodeTypeIndirectFieldValue ||
|
||||
astNode.Type == dsl.NodeTypeFullSrec {
|
||||
if astNode.Type == asts.NodeType(NodeTypeDirectFieldValue) ||
|
||||
astNode.Type == asts.NodeType(NodeTypeIndirectFieldValue) ||
|
||||
astNode.Type == asts.NodeType(NodeTypeFullSrec) {
|
||||
return fmt.Errorf(
|
||||
"begin/end blocks cannot refer to records via $x, $*, etc",
|
||||
)
|
||||
|
|
@ -137,7 +138,7 @@ func validateASTAux(
|
|||
|
||||
// Check: break/continue outside of loop
|
||||
if !inLoop {
|
||||
if astNode.Type == dsl.NodeTypeBreak {
|
||||
if astNode.Type == asts.NodeType(NodeTypeBreakStatement) {
|
||||
return fmt.Errorf(
|
||||
"break statements are only valid within for/do/while loops",
|
||||
)
|
||||
|
|
@ -145,25 +146,25 @@ func validateASTAux(
|
|||
}
|
||||
|
||||
if !inLoop {
|
||||
if astNode.Type == dsl.NodeTypeContinue {
|
||||
if astNode.Type == asts.NodeType(NodeTypeContinueStatement) {
|
||||
return fmt.Errorf(
|
||||
"break statements are only valid within for/do/while loops",
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
if astNode.Type == dsl.NodeTypeWhileLoop ||
|
||||
astNode.Type == dsl.NodeTypeDoWhileLoop ||
|
||||
astNode.Type == dsl.NodeTypeForLoopOneVariable ||
|
||||
astNode.Type == dsl.NodeTypeForLoopTwoVariable ||
|
||||
astNode.Type == dsl.NodeTypeForLoopMultivariable ||
|
||||
astNode.Type == dsl.NodeTypeTripleForLoop {
|
||||
if astNode.Type == asts.NodeType(NodeTypeWhileLoop) ||
|
||||
astNode.Type == asts.NodeType(NodeTypeDoWhileLoop) ||
|
||||
astNode.Type == asts.NodeType(NodeTypeForLoopOneVariable) ||
|
||||
astNode.Type == asts.NodeType(NodeTypeForLoopTwoVariable) ||
|
||||
astNode.Type == asts.NodeType(NodeTypeForLoopMultivariable) ||
|
||||
astNode.Type == asts.NodeType(NodeTypeTripleForLoop) {
|
||||
nextLevelInLoop = true
|
||||
}
|
||||
|
||||
// Check: return outside of func/subr
|
||||
if !inUDF && !inUDS {
|
||||
if astNode.Type == dsl.NodeTypeReturn {
|
||||
if astNode.Type == asts.NodeType(NodeTypeReturnStatement) {
|
||||
return fmt.Errorf(
|
||||
"return statements are only valid within func/subr blocks",
|
||||
)
|
||||
|
|
@ -171,7 +172,7 @@ func validateASTAux(
|
|||
}
|
||||
|
||||
// Check: enforce return-value iff in a function; return-void iff in a subroutine
|
||||
if astNode.Type == dsl.NodeTypeReturn {
|
||||
if astNode.Type == asts.NodeType(NodeTypeReturnStatement) {
|
||||
if inUDF {
|
||||
if len(astNode.Children) != 1 {
|
||||
return fmt.Errorf(
|
||||
|
|
@ -190,7 +191,7 @@ func validateASTAux(
|
|||
|
||||
// Check: prohibit NR etc at LHS; 1+2=3+4; etc
|
||||
if isAssignmentLHS {
|
||||
ok := VALID_LHS_NODE_TYPES[astNode.Type]
|
||||
ok := VALID_LHS_NODE_TYPES[string(astNode.Type)]
|
||||
if !ok {
|
||||
return fmt.Errorf(
|
||||
"%s is not valid on the left-hand side of an assignment",
|
||||
|
|
@ -201,7 +202,7 @@ func validateASTAux(
|
|||
|
||||
// Check: prohibit NR etc at LHS; 1+2=3+4; etc
|
||||
if isUnset {
|
||||
ok := VALID_LHS_NODE_TYPES[astNode.Type]
|
||||
ok := VALID_LHS_NODE_TYPES[string(astNode.Type)]
|
||||
if !ok {
|
||||
return fmt.Errorf(
|
||||
"%s is not valid for unset statement",
|
||||
|
|
@ -214,8 +215,8 @@ func validateASTAux(
|
|||
// Treewalk
|
||||
|
||||
for i, astChild := range astNode.Children {
|
||||
nextLevelIsAssignmentLHS = astNode.Type == dsl.NodeTypeAssignment && i == 0
|
||||
nextLevelIsUnset = astNode.Type == dsl.NodeTypeUnset
|
||||
nextLevelIsAssignmentLHS = (astNode.Type == asts.NodeType(NodeTypeAssignment) || astNode.Type == asts.NodeType(NodeTypeCompoundAssignment)) && i == 0
|
||||
nextLevelIsUnset = astNode.Type == asts.NodeType(NodeTypeUnset)
|
||||
err := validateASTAux(
|
||||
astChild,
|
||||
dslInstanceType,
|
||||
|
|
@ -245,15 +246,15 @@ func validateASTAux(
|
|||
// * full record "$*"
|
||||
// * statement block
|
||||
|
||||
func validateForLoopTwoVariableUniqueNames(astNode *dsl.ASTNode) error {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeForLoopTwoVariable)
|
||||
func validateForLoopTwoVariableUniqueNames(astNode *asts.ASTNode) error {
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeForLoopTwoVariable))
|
||||
lib.InternalCodingErrorIf(len(astNode.Children) != 4)
|
||||
keyVarNode := astNode.Children[0]
|
||||
valVarNode := astNode.Children[1]
|
||||
lib.InternalCodingErrorIf(keyVarNode.Type != dsl.NodeTypeLocalVariable)
|
||||
lib.InternalCodingErrorIf(valVarNode.Type != dsl.NodeTypeLocalVariable)
|
||||
keyVarName := string(keyVarNode.Token.Lit)
|
||||
valVarName := string(valVarNode.Token.Lit)
|
||||
lib.InternalCodingErrorIf(keyVarNode.Type != asts.NodeType(NodeTypeLocalVariable))
|
||||
lib.InternalCodingErrorIf(valVarNode.Type != asts.NodeType(NodeTypeLocalVariable))
|
||||
keyVarName := tokenLit(keyVarNode)
|
||||
valVarName := tokenLit(valVarNode)
|
||||
if keyVarName == valVarName {
|
||||
return fmt.Errorf("redefinition of variable %s in the same scope", keyVarName)
|
||||
}
|
||||
|
|
@ -270,18 +271,20 @@ func validateForLoopTwoVariableUniqueNames(astNode *dsl.ASTNode) error {
|
|||
// - local variable "a"
|
||||
// - full record "$*"
|
||||
// - statement block
|
||||
func validateForLoopMultivariableUniqueNames(astNode *dsl.ASTNode) error {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeForLoopMultivariable)
|
||||
func validateForLoopMultivariableUniqueNames(astNode *asts.ASTNode) error {
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeForLoopMultivariable))
|
||||
keyVarsNode := astNode.Children[0]
|
||||
valVarNode := astNode.Children[1]
|
||||
lib.InternalCodingErrorIf(keyVarsNode.Type != dsl.NodeTypeParameterList)
|
||||
lib.InternalCodingErrorIf(valVarNode.Type != dsl.NodeTypeLocalVariable)
|
||||
// PGPG produces MultiIndex; legacy produced ParameterList. Both have LocalVariable children.
|
||||
lib.InternalCodingErrorIf(keyVarsNode.Type != asts.NodeType(NodeTypeParameterList) &&
|
||||
keyVarsNode.Type != asts.NodeType(NodeTypeMultiIndex))
|
||||
lib.InternalCodingErrorIf(valVarNode.Type != asts.NodeType(NodeTypeLocalVariable))
|
||||
|
||||
seen := make(map[string]bool)
|
||||
|
||||
for _, keyVarNode := range keyVarsNode.Children {
|
||||
lib.InternalCodingErrorIf(keyVarNode.Type != dsl.NodeTypeLocalVariable)
|
||||
name := string(keyVarNode.Token.Lit)
|
||||
lib.InternalCodingErrorIf(keyVarNode.Type != asts.NodeType(NodeTypeLocalVariable))
|
||||
name := tokenLit(keyVarNode)
|
||||
_, present := seen[name]
|
||||
if present {
|
||||
return fmt.Errorf("redefinition of variable %s in the same scope", name)
|
||||
|
|
@ -289,7 +292,7 @@ func validateForLoopMultivariableUniqueNames(astNode *dsl.ASTNode) error {
|
|||
seen[name] = true
|
||||
}
|
||||
|
||||
valVarName := string(valVarNode.Token.Lit)
|
||||
valVarName := tokenLit(valVarNode)
|
||||
if seen[valVarName] {
|
||||
return fmt.Errorf("redefinition of variable %s in the same scope", valVarName)
|
||||
}
|
||||
|
|
@ -297,16 +300,22 @@ func validateForLoopMultivariableUniqueNames(astNode *dsl.ASTNode) error {
|
|||
return nil
|
||||
}
|
||||
|
||||
var VALID_LHS_NODE_TYPES = map[dsl.TNodeType]bool{
|
||||
dsl.NodeTypeArrayOrMapIndexAccess: true,
|
||||
dsl.NodeTypeDotOperator: true,
|
||||
dsl.NodeTypeArraySliceAccess: true,
|
||||
dsl.NodeTypeDirectFieldValue: true,
|
||||
dsl.NodeTypeDirectOosvarValue: true,
|
||||
dsl.NodeTypeEnvironmentVariable: true,
|
||||
dsl.NodeTypeFullOosvar: true,
|
||||
dsl.NodeTypeFullSrec: true,
|
||||
dsl.NodeTypeIndirectFieldValue: true, // includes $[[n]] and $[[[n]]]
|
||||
dsl.NodeTypeIndirectOosvarValue: true,
|
||||
dsl.NodeTypeLocalVariable: true,
|
||||
var VALID_LHS_NODE_TYPES = map[string]bool{
|
||||
NodeTypeArrayOrMapIndexAccess: true,
|
||||
NodeTypeDotOperator: true,
|
||||
NodeTypeArraySliceLoHi: true,
|
||||
NodeTypeArraySliceHiOnly: true,
|
||||
NodeTypeArraySliceLoOnly: true,
|
||||
NodeTypeArraySliceFull: true,
|
||||
NodeTypeDirectFieldValue: true,
|
||||
NodeTypeBracedFieldValue: true, // ${foo}, ${x+y}
|
||||
NodeTypeEnvironmentVariable: true, // ENV["FOO"] = "bar"
|
||||
NodeTypeDirectOosvarValue: true,
|
||||
NodeTypeBracedOosvarValue: true, // @{foo}, @{variable.name}
|
||||
NodeTypeFullOosvar: true,
|
||||
NodeTypeFullSrec: true,
|
||||
NodeTypeIndirectFieldValue: true, // includes $[[n]] and $[[[n]]]
|
||||
NodeTypeIndirectOosvarValue: true,
|
||||
NodeTypeLocalVariable: true,
|
||||
"TypedeclLocalVariable": true, // int a = 5; etc
|
||||
}
|
||||
|
|
|
|||
|
|
@ -9,13 +9,14 @@ import (
|
|||
"fmt"
|
||||
"os"
|
||||
|
||||
"github.com/johnkerl/miller/v6/pkg/dsl"
|
||||
"github.com/johnkerl/miller/v6/pkg/lib"
|
||||
|
||||
"github.com/johnkerl/pgpg/go/lib/pkg/asts"
|
||||
)
|
||||
|
||||
// Returns true if there are no warnings.
|
||||
func WarnOnAST(
|
||||
ast *dsl.AST,
|
||||
ast *asts.AST,
|
||||
) bool {
|
||||
variableNamesWrittenTo := make(map[string]bool)
|
||||
inAssignment := false
|
||||
|
|
@ -80,7 +81,7 @@ func WarnOnAST(
|
|||
|
||||
// Returns true if there are no warnings
|
||||
func warnOnASTAux(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
variableNamesWrittenTo map[string]bool,
|
||||
inAssignment bool,
|
||||
) bool {
|
||||
|
|
@ -89,8 +90,8 @@ func warnOnASTAux(
|
|||
|
||||
// Check local-variable references, and see if they're reads or writes
|
||||
// based on the AST parenting of this node.
|
||||
if astNode.Type == dsl.NodeTypeLocalVariable {
|
||||
variableName := string(astNode.Token.Lit)
|
||||
if astNode.Type == asts.NodeType(NodeTypeLocalVariable) {
|
||||
variableName := tokenLit(astNode)
|
||||
if inAssignment {
|
||||
variableNamesWrittenTo[variableName] = true
|
||||
} else {
|
||||
|
|
@ -99,27 +100,27 @@ func warnOnASTAux(
|
|||
os.Stderr,
|
||||
"Variable name %s might not have been assigned yet%s.\n",
|
||||
variableName,
|
||||
dsl.TokenToLocationInfo(astNode.Token),
|
||||
pgpgTokenToLocationInfo(astNode.Token),
|
||||
)
|
||||
ok = false
|
||||
}
|
||||
}
|
||||
|
||||
} else if astNode.Type == dsl.NodeTypeBeginBlock {
|
||||
} else if astNode.Type == asts.NodeType(NodeTypeBeginBlock) {
|
||||
// Locals are confined to begin/end blocks and func/subr blocks.
|
||||
// Reset for this part of the treewalk.
|
||||
variableNamesWrittenTo = make(map[string]bool)
|
||||
} else if astNode.Type == dsl.NodeTypeEndBlock {
|
||||
} else if astNode.Type == asts.NodeType(NodeTypeEndBlock) {
|
||||
// Locals are confined to begin/end blocks and func/subr blocks.
|
||||
// Reset for this part of the treewalk.
|
||||
variableNamesWrittenTo = make(map[string]bool)
|
||||
|
||||
} else if astNode.Type == dsl.NodeTypeNamedFunctionDefinition {
|
||||
} else if astNode.Type == asts.NodeType(NodeTypeNamedFunctionDefinition) {
|
||||
// Locals are confined to begin/end blocks and func/subr blocks. Reset
|
||||
// for this part of the treewalk, except mark the parameters as
|
||||
// defined.
|
||||
variableNamesWrittenTo = noteParametersForWarnings(astNode)
|
||||
} else if astNode.Type == dsl.NodeTypeSubroutineDefinition {
|
||||
} else if astNode.Type == asts.NodeType(NodeTypeSubroutineDefinition) {
|
||||
// Locals are confined to begin/end blocks and func/subr blocks. Reset
|
||||
// for this part of the treewalk, except mark the parameters as
|
||||
// defined.
|
||||
|
|
@ -131,21 +132,21 @@ func warnOnASTAux(
|
|||
for i, astChild := range astNode.Children {
|
||||
childInAssignment := inAssignment
|
||||
|
||||
if astNode.Type == dsl.NodeTypeAssignment && i == 0 {
|
||||
if (astNode.Type == asts.NodeType(NodeTypeAssignment) || astNode.Type == asts.NodeType(NodeTypeCompoundAssignment)) && i == 0 {
|
||||
// LHS of assignment statements
|
||||
childInAssignment = true
|
||||
} else if astNode.Type == dsl.NodeTypeForLoopOneVariable && i == 0 {
|
||||
} else if astNode.Type == asts.NodeType(NodeTypeForLoopOneVariable) && i == 0 {
|
||||
// The 'k' in 'for (k in $*)'
|
||||
childInAssignment = true
|
||||
} else if astNode.Type == dsl.NodeTypeForLoopTwoVariable && (i == 0 || i == 1) {
|
||||
} else if astNode.Type == asts.NodeType(NodeTypeForLoopTwoVariable) && (i == 0 || i == 1) {
|
||||
// The 'k' and 'v' in 'for (k,v in $*)'
|
||||
childInAssignment = true
|
||||
} else if astNode.Type == dsl.NodeTypeForLoopMultivariable && (i == 0 || i == 1) {
|
||||
} else if astNode.Type == asts.NodeType(NodeTypeForLoopMultivariable) && (i == 0 || i == 1) {
|
||||
// The 'k1', 'k2', and 'v' in 'for ((k1,k2),v in $*)'
|
||||
childInAssignment = true
|
||||
} else if astNode.Type == dsl.NodeTypeParameterList {
|
||||
} else if astNode.Type == asts.NodeType(NodeTypeParameterList) {
|
||||
childInAssignment = true
|
||||
} else if inAssignment && astNode.Type == dsl.NodeTypeArrayOrMapIndexAccess {
|
||||
} else if inAssignment && astNode.Type == asts.NodeType(NodeTypeArrayOrMapIndexAccess) {
|
||||
// In 'z[i] = 1', the 'i' is a read and the 'z' is a write.
|
||||
//
|
||||
// mlr --from r put -v -W 'z[i] = 1'
|
||||
|
|
@ -177,31 +178,45 @@ func warnOnASTAux(
|
|||
return ok
|
||||
}
|
||||
|
||||
// collectParameterNodes flattens ParameterList (which may nest in PGPG grammar) into Parameter nodes.
|
||||
func collectParameterNodes(parameterListNode *asts.ASTNode) []*asts.ASTNode {
|
||||
var out []*asts.ASTNode
|
||||
for _, ch := range parameterListNode.Children {
|
||||
if ch.Type == asts.NodeType(NodeTypeParameterList) {
|
||||
out = append(out, collectParameterNodes(ch)...)
|
||||
} else if ch.Type == asts.NodeType(NodeTypeParameter) {
|
||||
out = append(out, ch)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Given a func/subr block, find the names of its parameters. All the
|
||||
// lib.InternalCodingErrorIf parts are shape-assertions to make sure this code
|
||||
// is in sync with the BNF grammar which builds the AST from a Miller-DSL
|
||||
// source string.
|
||||
//
|
||||
// PGPG: Parameter has one child which is LocalVariable (not ParameterName).
|
||||
func noteParametersForWarnings(
|
||||
astNode *dsl.ASTNode,
|
||||
astNode *asts.ASTNode,
|
||||
) map[string]bool {
|
||||
|
||||
variableNamesWrittenTo := make(map[string]bool)
|
||||
|
||||
lib.InternalCodingErrorIf(
|
||||
astNode.Type != dsl.NodeTypeNamedFunctionDefinition &&
|
||||
astNode.Type != dsl.NodeTypeSubroutineDefinition)
|
||||
astNode.Type != asts.NodeType(NodeTypeNamedFunctionDefinition) &&
|
||||
astNode.Type != asts.NodeType(NodeTypeSubroutineDefinition))
|
||||
lib.InternalCodingErrorIf(len(astNode.Children) < 1)
|
||||
parameterListNode := astNode.Children[0]
|
||||
|
||||
lib.InternalCodingErrorIf(parameterListNode.Type != dsl.NodeTypeParameterList)
|
||||
lib.InternalCodingErrorIf(parameterListNode.Type != asts.NodeType(NodeTypeParameterList))
|
||||
|
||||
for _, parameterNode := range parameterListNode.Children {
|
||||
lib.InternalCodingErrorIf(parameterNode.Type != dsl.NodeTypeParameter)
|
||||
lib.InternalCodingErrorIf(len(parameterNode.Children) != 1)
|
||||
parameterNameNode := parameterNode.Children[0]
|
||||
lib.InternalCodingErrorIf(parameterNameNode.Type != dsl.NodeTypeParameterName)
|
||||
parameterName := string(parameterNameNode.Token.Lit)
|
||||
variableNamesWrittenTo[parameterName] = true
|
||||
// PGPG: ParameterList can nest (FuncParams wraps FuncOrSubrParameterList, which builds list recursively).
|
||||
for _, ch := range collectParameterNodes(parameterListNode) {
|
||||
lib.InternalCodingErrorIf(len(ch.Children) != 1)
|
||||
paramNameNode := ch.Children[0]
|
||||
lib.InternalCodingErrorIf(paramNameNode.Type != asts.NodeType(NodeTypeLocalVariable))
|
||||
variableNamesWrittenTo[tokenLit(paramNameNode)] = true
|
||||
}
|
||||
|
||||
return variableNamesWrittenTo
|
||||
|
|
|
|||
|
|
@ -5,21 +5,21 @@ package cst
|
|||
import (
|
||||
"fmt"
|
||||
|
||||
"github.com/johnkerl/miller/v6/pkg/dsl"
|
||||
"github.com/johnkerl/miller/v6/pkg/lib"
|
||||
"github.com/johnkerl/miller/v6/pkg/parsing/token"
|
||||
"github.com/johnkerl/miller/v6/pkg/runtime"
|
||||
"github.com/johnkerl/pgpg/go/lib/pkg/asts"
|
||||
"github.com/johnkerl/pgpg/go/lib/pkg/tokens"
|
||||
)
|
||||
|
||||
type WhileLoopNode struct {
|
||||
conditionNode IEvaluable
|
||||
conditionToken *token.Token
|
||||
conditionToken *tokens.Token
|
||||
statementBlockNode *StatementBlockNode
|
||||
}
|
||||
|
||||
func NewWhileLoopNode(
|
||||
conditionNode IEvaluable,
|
||||
conditionToken *token.Token,
|
||||
conditionToken *tokens.Token,
|
||||
statementBlockNode *StatementBlockNode,
|
||||
) *WhileLoopNode {
|
||||
return &WhileLoopNode{
|
||||
|
|
@ -29,8 +29,8 @@ func NewWhileLoopNode(
|
|||
}
|
||||
}
|
||||
|
||||
func (root *RootNode) BuildWhileLoopNode(astNode *dsl.ASTNode) (*WhileLoopNode, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeWhileLoop)
|
||||
func (root *RootNode) BuildWhileLoopNode(astNode *asts.ASTNode) (*WhileLoopNode, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeWhileLoop))
|
||||
lib.InternalCodingErrorIf(len(astNode.Children) != 2)
|
||||
|
||||
conditionNode, err := root.BuildEvaluableNode(astNode.Children[0])
|
||||
|
|
@ -57,7 +57,7 @@ func (node *WhileLoopNode) Execute(state *runtime.State) (*BlockExitPayload, err
|
|||
if !isBool {
|
||||
return nil, fmt.Errorf(
|
||||
"conditional expression did not evaluate to boolean%s",
|
||||
dsl.TokenToLocationInfo(node.conditionToken),
|
||||
pgpgTokenToLocationInfo(node.conditionToken),
|
||||
)
|
||||
}
|
||||
if !boolValue {
|
||||
|
|
@ -89,13 +89,13 @@ func (node *WhileLoopNode) Execute(state *runtime.State) (*BlockExitPayload, err
|
|||
type DoWhileLoopNode struct {
|
||||
statementBlockNode *StatementBlockNode
|
||||
conditionNode IEvaluable
|
||||
conditionToken *token.Token
|
||||
conditionToken *tokens.Token
|
||||
}
|
||||
|
||||
func NewDoWhileLoopNode(
|
||||
statementBlockNode *StatementBlockNode,
|
||||
conditionNode IEvaluable,
|
||||
conditionToken *token.Token,
|
||||
conditionToken *tokens.Token,
|
||||
) *DoWhileLoopNode {
|
||||
return &DoWhileLoopNode{
|
||||
statementBlockNode: statementBlockNode,
|
||||
|
|
@ -104,8 +104,8 @@ func NewDoWhileLoopNode(
|
|||
}
|
||||
}
|
||||
|
||||
func (root *RootNode) BuildDoWhileLoopNode(astNode *dsl.ASTNode) (*DoWhileLoopNode, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != dsl.NodeTypeDoWhileLoop)
|
||||
func (root *RootNode) BuildDoWhileLoopNode(astNode *asts.ASTNode) (*DoWhileLoopNode, error) {
|
||||
lib.InternalCodingErrorIf(astNode.Type != asts.NodeType(NodeTypeDoWhileLoop))
|
||||
lib.InternalCodingErrorIf(len(astNode.Children) != 2)
|
||||
|
||||
statementBlockNode, err := root.BuildStatementBlockNode(astNode.Children[0])
|
||||
|
|
@ -152,7 +152,7 @@ func (node *DoWhileLoopNode) Execute(state *runtime.State) (*BlockExitPayload, e
|
|||
if !isBool {
|
||||
return nil, fmt.Errorf(
|
||||
"conditional expression did not evaluate to boolean%s",
|
||||
dsl.TokenToLocationInfo(node.conditionToken),
|
||||
pgpgTokenToLocationInfo(node.conditionToken),
|
||||
)
|
||||
}
|
||||
if !boolValue {
|
||||
|
|
|
|||
|
|
@ -1,16 +0,0 @@
|
|||
package dsl
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
"github.com/johnkerl/miller/v6/pkg/parsing/token"
|
||||
)
|
||||
|
||||
// TokenToLocationInfo is used to track runtime errors back to source-code locations in DSL
|
||||
// expressions, so we can have more informative error messages.
|
||||
func TokenToLocationInfo(sourceToken *token.Token) string {
|
||||
if sourceToken == nil {
|
||||
return ""
|
||||
}
|
||||
return fmt.Sprintf(" at DSL expression line %d column %d", sourceToken.Pos.Line, sourceToken.Pos.Column)
|
||||
}
|
||||
|
|
@ -1,26 +1,11 @@
|
|||
This directory contains a single source file, `mlr.bnf`, which is the
|
||||
lexical/semantic grammar file for the Miller `put`/`filter` DSL
|
||||
(domain-specific language) using the GOCC framework. (In a classical Lex/Yacc
|
||||
framework, there would be separate `mlr.l` and `mlr.y` files; using GOCC, there
|
||||
is a single `mlr.bnf` file.)
|
||||
This directory contains a single source file, `mlr.bnf`, which is the lexical/semantic grammar file
|
||||
for the Miller `put`/`filter` DSL (domain-specific language) using the [PGPG
|
||||
framework](https://github.com/johnkerl/pgpg). (In a classical Lex/Yacc framework, there would be
|
||||
separate `mlr.l` and `mlr.y` files; using PGPG, there is a single `mlr.bnf` file.)
|
||||
|
||||
All subdirectories of `pkg/parsing/` are autogen code created by GOCC's
|
||||
processing of `mlr.bnf`. They are nonetheless committed to source control,
|
||||
since running GOCC takes quite a bit longer than the `go build` does, and the
|
||||
BNF file doesn't often change. (_BNF_ is for _Backus-Naur Form_ which is the
|
||||
phrasing of the grammar file that GOCC support.) See the top-level `miller/go`
|
||||
build scripts for how to rerun GOCC. As of this writing, it's
|
||||
All subdirectories of `pkg/parsing/` are autogen code created by PGPG's processing of `mlr.bnf`.
|
||||
They are nonetheless committed to source control, since running PGPG takes a bit longer than the `go
|
||||
build` does, and the BNF file doesn't often change. (_BNF_ is for _Backus-Naur Form_ which is the
|
||||
phrasing of the grammar file that PGPG support.)
|
||||
|
||||
```
|
||||
gocc -o pkg/parsing pkg/parsing/mlr.bnf
|
||||
````
|
||||
|
||||
as invoked from the repo base directory -- however you should instead use
|
||||
|
||||
```
|
||||
tools/build-dsl
|
||||
```
|
||||
|
||||
Making changes to `mlr.bnf` requires several minutes to re-run GOCC. For
|
||||
experimental changes, please see the
|
||||
[cmd/experiments/dsl_parser](../../../cmd/experiments/dsl_parser) directory.
|
||||
Run [tools/build-dsl](../../tools/build-dsl) from the repo root to regenerate the lexer and parser.
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
// Package parsing contains the single source file mlr.bnf, which is the
|
||||
// lexical/semantic grammar file for the Miller put/filter DSL using the GOCC
|
||||
// framework. Go source files are autogenerated by GOCC.
|
||||
// lexical/semantic grammar file for the Miller put/filter DSL using the PGPG
|
||||
// framework. Go source files are autogenerated by PGPG.
|
||||
package parsing
|
||||
|
|
|
|||
|
|
@ -1,100 +0,0 @@
|
|||
// ================================================================
|
||||
// During the Miller build, after GOCC codegen and Go compile, this is copied
|
||||
// over the top of GOCC codegen so that we can customize handling of error
|
||||
// messages.
|
||||
//
|
||||
// Source: pkg/parsing/errors.go.template
|
||||
// Destination: pkg/parsing/errors/errors.go
|
||||
// ================================================================
|
||||
|
||||
package errors
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"github.com/johnkerl/miller/v6/pkg/parsing/token"
|
||||
)
|
||||
|
||||
type ErrorSymbol interface {
|
||||
}
|
||||
|
||||
type Error struct {
|
||||
Err error
|
||||
ErrorToken *token.Token
|
||||
ErrorSymbols []ErrorSymbol
|
||||
ExpectedTokens []string
|
||||
StackTop int
|
||||
}
|
||||
|
||||
func (e *Error) String() string {
|
||||
w := new(strings.Builder)
|
||||
fmt.Fprintf(w, "Error")
|
||||
if e.Err != nil {
|
||||
fmt.Fprintf(w, " %s\n", e.Err)
|
||||
} else {
|
||||
fmt.Fprintf(w, "\n")
|
||||
}
|
||||
fmt.Fprintf(w, "Token: type=%d, lit=%s\n", e.ErrorToken.Type, e.ErrorToken.Lit)
|
||||
fmt.Fprintf(w, "Pos: offset=%d, line=%d, column=%d\n", e.ErrorToken.Pos.Offset, e.ErrorToken.Pos.Line, e.ErrorToken.Pos.Column)
|
||||
fmt.Fprintf(w, "Expected one of: ")
|
||||
for _, sym := range e.ExpectedTokens {
|
||||
fmt.Fprintf(w, "%s ", sym)
|
||||
}
|
||||
fmt.Fprintf(w, "ErrorSymbol:\n")
|
||||
for _, sym := range e.ErrorSymbols {
|
||||
fmt.Fprintf(w, "%v\n", sym)
|
||||
}
|
||||
return w.String()
|
||||
}
|
||||
|
||||
func (e *Error) Error() string {
|
||||
w := new(strings.Builder)
|
||||
fmt.Fprintf(
|
||||
w,
|
||||
"Parse error on token \"%s\" at line %d column %d.\n",
|
||||
string(e.ErrorToken.Lit),
|
||||
e.ErrorToken.Pos.Line,
|
||||
e.ErrorToken.Pos.Column,
|
||||
)
|
||||
if e.Err != nil {
|
||||
fmt.Fprintf(w, "%+v\n", e.Err)
|
||||
} else {
|
||||
suggestSemicolons := false
|
||||
for _, expected := range e.ExpectedTokens {
|
||||
if expected == ";" {
|
||||
suggestSemicolons = true
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
if suggestSemicolons {
|
||||
fmt.Fprintf(w, "Please check for missing semicolon.\n")
|
||||
}
|
||||
|
||||
fmt.Fprintf(w, "Expected one of:\n")
|
||||
|
||||
//for _, expected := range e.ExpectedTokens {
|
||||
// fmt.Fprintf(w, "%s ", expected)
|
||||
//}
|
||||
|
||||
// Print a carriage return every so often, in case there are many possible
|
||||
// next tokens.
|
||||
line := ""
|
||||
for _, expected := range e.ExpectedTokens {
|
||||
if line != "" {
|
||||
line += " "
|
||||
}
|
||||
line += expected
|
||||
if len(line) > 70 {
|
||||
fmt.Fprintf(w, " %s\n", line)
|
||||
line = ""
|
||||
}
|
||||
}
|
||||
if line != "" {
|
||||
fmt.Fprintf(w, " %s\n", line)
|
||||
}
|
||||
|
||||
}
|
||||
return w.String()
|
||||
}
|
||||
|
|
@ -1,3 +0,0 @@
|
|||
// Package errors contains source code autogenerated by GOCC from the Miller DSL's
|
||||
// grammar file pkg/parsing/mlr.bnf.
|
||||
package errors
|
||||
|
|
@ -1,95 +0,0 @@
|
|||
// ================================================================
|
||||
// During the Miller build, after GOCC codegen and Go compile, this is copied
|
||||
// over the top of GOCC codegen so that we can customize handling of error
|
||||
// messages.
|
||||
//
|
||||
// Source: pkg/parsing/errors.go.template
|
||||
// Destination: pkg/parsing/errors/errors.go
|
||||
// ================================================================
|
||||
|
||||
package errors
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"slices"
|
||||
"strings"
|
||||
|
||||
"github.com/johnkerl/miller/v6/pkg/parsing/token"
|
||||
)
|
||||
|
||||
type ErrorSymbol interface {
|
||||
}
|
||||
|
||||
type Error struct {
|
||||
Err error
|
||||
ErrorToken *token.Token
|
||||
ErrorSymbols []ErrorSymbol
|
||||
ExpectedTokens []string
|
||||
StackTop int
|
||||
}
|
||||
|
||||
func (e *Error) String() string {
|
||||
w := new(strings.Builder)
|
||||
fmt.Fprintf(w, "Error")
|
||||
if e.Err != nil {
|
||||
fmt.Fprintf(w, " %s\n", e.Err)
|
||||
} else {
|
||||
fmt.Fprintf(w, "\n")
|
||||
}
|
||||
fmt.Fprintf(w, "Token: type=%d, lit=%s\n", e.ErrorToken.Type, e.ErrorToken.Lit)
|
||||
fmt.Fprintf(w, "Pos: offset=%d, line=%d, column=%d\n", e.ErrorToken.Pos.Offset, e.ErrorToken.Pos.Line, e.ErrorToken.Pos.Column)
|
||||
fmt.Fprintf(w, "Expected one of: ")
|
||||
for _, sym := range e.ExpectedTokens {
|
||||
fmt.Fprintf(w, "%s ", sym)
|
||||
}
|
||||
fmt.Fprintf(w, "ErrorSymbol:\n")
|
||||
for _, sym := range e.ErrorSymbols {
|
||||
fmt.Fprintf(w, "%v\n", sym)
|
||||
}
|
||||
return w.String()
|
||||
}
|
||||
|
||||
func (e *Error) Error() string {
|
||||
w := new(strings.Builder)
|
||||
fmt.Fprintf(
|
||||
w,
|
||||
"Parse error on token \"%s\" at line %d column %d.\n",
|
||||
string(e.ErrorToken.Lit),
|
||||
e.ErrorToken.Pos.Line,
|
||||
e.ErrorToken.Pos.Column,
|
||||
)
|
||||
if e.Err != nil {
|
||||
fmt.Fprintf(w, "%+v\n", e.Err)
|
||||
} else {
|
||||
suggestSemicolons := slices.Contains(e.ExpectedTokens, ";")
|
||||
|
||||
if suggestSemicolons {
|
||||
fmt.Fprintf(w, "Please check for missing semicolon.\n")
|
||||
}
|
||||
|
||||
fmt.Fprintf(w, "Expected one of:\n")
|
||||
|
||||
//for _, expected := range e.ExpectedTokens {
|
||||
// fmt.Fprintf(w, "%s ", expected)
|
||||
//}
|
||||
|
||||
// Print a carriage return every so often, in case there are many possible
|
||||
// next tokens.
|
||||
line := ""
|
||||
for _, expected := range e.ExpectedTokens {
|
||||
if line != "" {
|
||||
line += " "
|
||||
}
|
||||
line += expected
|
||||
if len(line) > 70 {
|
||||
fmt.Fprintf(w, " %s\n", line)
|
||||
line = ""
|
||||
}
|
||||
}
|
||||
if line != "" {
|
||||
fmt.Fprintf(w, " %s\n", line)
|
||||
}
|
||||
|
||||
}
|
||||
return w.String()
|
||||
}
|
||||
39154
pkg/parsing/gen/lexer.json
Normal file
39154
pkg/parsing/gen/lexer.json
Normal file
File diff suppressed because it is too large
Load diff
662022
pkg/parsing/gen/parser.json
Normal file
662022
pkg/parsing/gen/parser.json
Normal file
File diff suppressed because it is too large
Load diff
File diff suppressed because it is too large
Load diff
16
pkg/parsing/lexer/adapter.go
Normal file
16
pkg/parsing/lexer/adapter.go
Normal file
|
|
@ -0,0 +1,16 @@
|
|||
// adapter.go provides NewLexer([]byte) for compatibility with pkg/dsl/cst/root.go.
|
||||
// The PGPG-generated lexer uses NewMlrLexer(io.Reader); this adapter accepts []byte.
|
||||
|
||||
package lexer
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
|
||||
liblexers "github.com/johnkerl/pgpg/go/lib/pkg/lexers"
|
||||
)
|
||||
|
||||
// NewLexer creates a lexer from a byte slice. This matches the API expected by
|
||||
// pkg/dsl/cst/root.go (same signature as the former PGPG lexer).
|
||||
func NewLexer(src []byte) liblexers.AbstractLexer {
|
||||
return NewMlrLexer(bytes.NewReader(src))
|
||||
}
|
||||
|
|
@ -1,3 +1,3 @@
|
|||
// Package lexer contains source code autogenerated by GOCC from the Miller DSL's
|
||||
// Package lexer contains source code autogenerated by PGPG from the Miller DSL's
|
||||
// grammar file pkg/parsing/mlr.bnf.
|
||||
package lexer
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
File diff suppressed because it is too large
Load diff
4153
pkg/parsing/mlr.bnf
4153
pkg/parsing/mlr.bnf
File diff suppressed because it is too large
Load diff
|
|
@ -1,51 +0,0 @@
|
|||
// Code generated by gocc; DO NOT EDIT.
|
||||
|
||||
package parser
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
)
|
||||
|
||||
type action interface {
|
||||
act()
|
||||
String() string
|
||||
}
|
||||
|
||||
type (
|
||||
accept bool
|
||||
shift int // value is next state index
|
||||
reduce int // value is production index
|
||||
)
|
||||
|
||||
func (this accept) act() {}
|
||||
func (this shift) act() {}
|
||||
func (this reduce) act() {}
|
||||
|
||||
func (this accept) Equal(that action) bool {
|
||||
if _, ok := that.(accept); ok {
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (this reduce) Equal(that action) bool {
|
||||
that1, ok := that.(reduce)
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
return this == that1
|
||||
}
|
||||
|
||||
func (this shift) Equal(that action) bool {
|
||||
that1, ok := that.(shift)
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
return this == that1
|
||||
}
|
||||
|
||||
func (this accept) String() string { return "accept(0)" }
|
||||
func (this shift) String() string { return fmt.Sprintf("shift:%d", this) }
|
||||
func (this reduce) String() string {
|
||||
return fmt.Sprintf("reduce:%d(%s)", this, productionsTable[this].String)
|
||||
}
|
||||
File diff suppressed because it is too large
Load diff
|
|
@ -1,7 +0,0 @@
|
|||
// Code generated by gocc; DO NOT EDIT.
|
||||
|
||||
package parser
|
||||
|
||||
// Parser-specific user-defined and entirely-optional context,
|
||||
// accessible as '$Context' in SDT actions.
|
||||
type Context interface{}
|
||||
|
|
@ -1,3 +1,3 @@
|
|||
// Package parser contains source code autogenerated by GOCC from the Miller DSL's
|
||||
// Package parser contains source code autogenerated by PGPG from the Miller DSL's
|
||||
// grammar file pkg/parsing/mlr.bnf.
|
||||
package parser
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
206625
pkg/parsing/parser/parser.go
206625
pkg/parsing/parser/parser.go
File diff suppressed because it is too large
Load diff
File diff suppressed because it is too large
Load diff
|
|
@ -1,14 +0,0 @@
|
|||
// Code generated by gocc; DO NOT EDIT.
|
||||
|
||||
package token
|
||||
|
||||
// Context allows user-defined data to be associated with the
|
||||
// lexer/scanner to be associated with each token that lexer
|
||||
// produces.
|
||||
type Context interface{}
|
||||
|
||||
// Sourcer is a Context interface which presents a Source() method
|
||||
// identifying e.g the filename for the current code.
|
||||
type Sourcer interface {
|
||||
Source() string
|
||||
}
|
||||
|
|
@ -1,3 +0,0 @@
|
|||
// Package token contains source code autogenerated by GOCC from the Miller DSL's
|
||||
// grammar file pkg/parsing/mlr.bnf.
|
||||
package token
|
||||
|
|
@ -1,425 +0,0 @@
|
|||
// Code generated by gocc; DO NOT EDIT.
|
||||
|
||||
package token
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"fmt"
|
||||
"strconv"
|
||||
"unicode/utf8"
|
||||
)
|
||||
|
||||
type Token struct {
|
||||
Type
|
||||
Lit []byte
|
||||
Pos
|
||||
}
|
||||
|
||||
type Type int
|
||||
|
||||
const (
|
||||
INVALID Type = iota
|
||||
EOF
|
||||
)
|
||||
|
||||
type Pos struct {
|
||||
Offset int
|
||||
Line int
|
||||
Column int
|
||||
Context Context
|
||||
}
|
||||
|
||||
func (p Pos) String() string {
|
||||
// If the context provides a filename, provide a human-readable File:Line:Column representation.
|
||||
switch src := p.Context.(type) {
|
||||
case Sourcer:
|
||||
return fmt.Sprintf("%s:%d:%d", src.Source(), p.Line, p.Column)
|
||||
default:
|
||||
return fmt.Sprintf("Pos(offset=%d, line=%d, column=%d)", p.Offset, p.Line, p.Column)
|
||||
}
|
||||
}
|
||||
|
||||
type TokenMap struct {
|
||||
typeMap []string
|
||||
idMap map[string]Type
|
||||
}
|
||||
|
||||
func (m TokenMap) Id(tok Type) string {
|
||||
if int(tok) < len(m.typeMap) {
|
||||
return m.typeMap[tok]
|
||||
}
|
||||
return "unknown"
|
||||
}
|
||||
|
||||
func (m TokenMap) Type(tok string) Type {
|
||||
if typ, exist := m.idMap[tok]; exist {
|
||||
return typ
|
||||
}
|
||||
return INVALID
|
||||
}
|
||||
|
||||
func (m TokenMap) TokenString(tok *Token) string {
|
||||
return fmt.Sprintf("%s(%d,%s)", m.Id(tok.Type), tok.Type, tok.Lit)
|
||||
}
|
||||
|
||||
func (m TokenMap) StringType(typ Type) string {
|
||||
return fmt.Sprintf("%s(%d)", m.Id(typ), typ)
|
||||
}
|
||||
|
||||
// Equals returns returns true if the token Type and Lit are matches.
|
||||
func (t *Token) Equals(rhs interface{}) bool {
|
||||
switch rhsT := rhs.(type) {
|
||||
case *Token:
|
||||
return t == rhsT || (t.Type == rhsT.Type && bytes.Equal(t.Lit, rhsT.Lit))
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
// CharLiteralValue returns the string value of the char literal.
|
||||
func (t *Token) CharLiteralValue() string {
|
||||
return string(t.Lit[1 : len(t.Lit)-1])
|
||||
}
|
||||
|
||||
// Float32Value returns the float32 value of the token or an error if the token literal does not
|
||||
// denote a valid float32.
|
||||
func (t *Token) Float32Value() (float32, error) {
|
||||
if v, err := strconv.ParseFloat(string(t.Lit), 32); err != nil {
|
||||
return 0, err
|
||||
} else {
|
||||
return float32(v), nil
|
||||
}
|
||||
}
|
||||
|
||||
// Float64Value returns the float64 value of the token or an error if the token literal does not
|
||||
// denote a valid float64.
|
||||
func (t *Token) Float64Value() (float64, error) {
|
||||
return strconv.ParseFloat(string(t.Lit), 64)
|
||||
}
|
||||
|
||||
// IDValue returns the string representation of an identifier token.
|
||||
func (t *Token) IDValue() string {
|
||||
return string(t.Lit)
|
||||
}
|
||||
|
||||
// Int32Value returns the int32 value of the token or an error if the token literal does not
|
||||
// denote a valid float64.
|
||||
func (t *Token) Int32Value() (int32, error) {
|
||||
if v, err := strconv.ParseInt(string(t.Lit), 10, 64); err != nil {
|
||||
return 0, err
|
||||
} else {
|
||||
return int32(v), nil
|
||||
}
|
||||
}
|
||||
|
||||
// Int64Value returns the int64 value of the token or an error if the token literal does not
|
||||
// denote a valid float64.
|
||||
func (t *Token) Int64Value() (int64, error) {
|
||||
return strconv.ParseInt(string(t.Lit), 10, 64)
|
||||
}
|
||||
|
||||
// UTF8Rune decodes the UTF8 rune in the token literal. It returns utf8.RuneError if
|
||||
// the token literal contains an invalid rune.
|
||||
func (t *Token) UTF8Rune() (rune, error) {
|
||||
r, _ := utf8.DecodeRune(t.Lit)
|
||||
if r == utf8.RuneError {
|
||||
err := fmt.Errorf("Invalid rune")
|
||||
return r, err
|
||||
}
|
||||
return r, nil
|
||||
}
|
||||
|
||||
// StringValue returns the string value of the token literal.
|
||||
func (t *Token) StringValue() string {
|
||||
return string(t.Lit[1 : len(t.Lit)-1])
|
||||
}
|
||||
|
||||
var TokMap = TokenMap{
|
||||
typeMap: []string{
|
||||
"INVALID",
|
||||
"␚",
|
||||
"empty",
|
||||
";",
|
||||
"{",
|
||||
"}",
|
||||
"=",
|
||||
"unset",
|
||||
"filter",
|
||||
">",
|
||||
">>",
|
||||
"|",
|
||||
"stdout",
|
||||
"stderr",
|
||||
"print",
|
||||
",",
|
||||
"printn",
|
||||
"eprint",
|
||||
"eprintn",
|
||||
"dump",
|
||||
"edump",
|
||||
"tee",
|
||||
"emitf",
|
||||
"emit1",
|
||||
"emit",
|
||||
"(",
|
||||
")",
|
||||
"emitp",
|
||||
"field_name",
|
||||
"$[",
|
||||
"]",
|
||||
"braced_field_name",
|
||||
"full_srec",
|
||||
"oosvar_name",
|
||||
"@[",
|
||||
"braced_oosvar_name",
|
||||
"full_oosvar",
|
||||
"all",
|
||||
"non_sigil_name",
|
||||
"arr",
|
||||
"bool",
|
||||
"float",
|
||||
"int",
|
||||
"map",
|
||||
"num",
|
||||
"str",
|
||||
"var",
|
||||
"funct",
|
||||
"||=",
|
||||
"^^=",
|
||||
"&&=",
|
||||
"??=",
|
||||
"???=",
|
||||
"|=",
|
||||
"&=",
|
||||
"^=",
|
||||
"<<=",
|
||||
">>=",
|
||||
">>>=",
|
||||
"+=",
|
||||
".=",
|
||||
"-=",
|
||||
"*=",
|
||||
"/=",
|
||||
"//=",
|
||||
"%=",
|
||||
"**=",
|
||||
"?",
|
||||
":",
|
||||
"||",
|
||||
"^^",
|
||||
"&&",
|
||||
"=~",
|
||||
"!=~",
|
||||
"==",
|
||||
"!=",
|
||||
"<=>",
|
||||
">=",
|
||||
"<",
|
||||
"<=",
|
||||
"^",
|
||||
"&",
|
||||
"<<",
|
||||
">>>",
|
||||
"+",
|
||||
"-",
|
||||
".+",
|
||||
".-",
|
||||
"*",
|
||||
"/",
|
||||
"//",
|
||||
"%",
|
||||
".*",
|
||||
"./",
|
||||
".//",
|
||||
".",
|
||||
"!",
|
||||
"~",
|
||||
"??",
|
||||
"???",
|
||||
"**",
|
||||
"string_literal",
|
||||
"regex_case_insensitive",
|
||||
"int_literal",
|
||||
"float_literal",
|
||||
"boolean_literal",
|
||||
"null_literal",
|
||||
"inf_literal",
|
||||
"nan_literal",
|
||||
"const_M_PI",
|
||||
"const_M_E",
|
||||
"panic",
|
||||
"[",
|
||||
"ctx_IPS",
|
||||
"ctx_IFS",
|
||||
"ctx_IRS",
|
||||
"ctx_OPS",
|
||||
"ctx_OFS",
|
||||
"ctx_ORS",
|
||||
"ctx_FLATSEP",
|
||||
"ctx_NF",
|
||||
"ctx_NR",
|
||||
"ctx_FNR",
|
||||
"ctx_FILENAME",
|
||||
"ctx_FILENUM",
|
||||
"env",
|
||||
"call",
|
||||
"begin",
|
||||
"end",
|
||||
"if",
|
||||
"elif",
|
||||
"else",
|
||||
"while",
|
||||
"do",
|
||||
"for",
|
||||
"in",
|
||||
"break",
|
||||
"continue",
|
||||
"func",
|
||||
"subr",
|
||||
"return",
|
||||
},
|
||||
|
||||
idMap: map[string]Type{
|
||||
"INVALID": 0,
|
||||
"␚": 1,
|
||||
"empty": 2,
|
||||
";": 3,
|
||||
"{": 4,
|
||||
"}": 5,
|
||||
"=": 6,
|
||||
"unset": 7,
|
||||
"filter": 8,
|
||||
">": 9,
|
||||
">>": 10,
|
||||
"|": 11,
|
||||
"stdout": 12,
|
||||
"stderr": 13,
|
||||
"print": 14,
|
||||
",": 15,
|
||||
"printn": 16,
|
||||
"eprint": 17,
|
||||
"eprintn": 18,
|
||||
"dump": 19,
|
||||
"edump": 20,
|
||||
"tee": 21,
|
||||
"emitf": 22,
|
||||
"emit1": 23,
|
||||
"emit": 24,
|
||||
"(": 25,
|
||||
")": 26,
|
||||
"emitp": 27,
|
||||
"field_name": 28,
|
||||
"$[": 29,
|
||||
"]": 30,
|
||||
"braced_field_name": 31,
|
||||
"full_srec": 32,
|
||||
"oosvar_name": 33,
|
||||
"@[": 34,
|
||||
"braced_oosvar_name": 35,
|
||||
"full_oosvar": 36,
|
||||
"all": 37,
|
||||
"non_sigil_name": 38,
|
||||
"arr": 39,
|
||||
"bool": 40,
|
||||
"float": 41,
|
||||
"int": 42,
|
||||
"map": 43,
|
||||
"num": 44,
|
||||
"str": 45,
|
||||
"var": 46,
|
||||
"funct": 47,
|
||||
"||=": 48,
|
||||
"^^=": 49,
|
||||
"&&=": 50,
|
||||
"??=": 51,
|
||||
"???=": 52,
|
||||
"|=": 53,
|
||||
"&=": 54,
|
||||
"^=": 55,
|
||||
"<<=": 56,
|
||||
">>=": 57,
|
||||
">>>=": 58,
|
||||
"+=": 59,
|
||||
".=": 60,
|
||||
"-=": 61,
|
||||
"*=": 62,
|
||||
"/=": 63,
|
||||
"//=": 64,
|
||||
"%=": 65,
|
||||
"**=": 66,
|
||||
"?": 67,
|
||||
":": 68,
|
||||
"||": 69,
|
||||
"^^": 70,
|
||||
"&&": 71,
|
||||
"=~": 72,
|
||||
"!=~": 73,
|
||||
"==": 74,
|
||||
"!=": 75,
|
||||
"<=>": 76,
|
||||
">=": 77,
|
||||
"<": 78,
|
||||
"<=": 79,
|
||||
"^": 80,
|
||||
"&": 81,
|
||||
"<<": 82,
|
||||
">>>": 83,
|
||||
"+": 84,
|
||||
"-": 85,
|
||||
".+": 86,
|
||||
".-": 87,
|
||||
"*": 88,
|
||||
"/": 89,
|
||||
"//": 90,
|
||||
"%": 91,
|
||||
".*": 92,
|
||||
"./": 93,
|
||||
".//": 94,
|
||||
".": 95,
|
||||
"!": 96,
|
||||
"~": 97,
|
||||
"??": 98,
|
||||
"???": 99,
|
||||
"**": 100,
|
||||
"string_literal": 101,
|
||||
"regex_case_insensitive": 102,
|
||||
"int_literal": 103,
|
||||
"float_literal": 104,
|
||||
"boolean_literal": 105,
|
||||
"null_literal": 106,
|
||||
"inf_literal": 107,
|
||||
"nan_literal": 108,
|
||||
"const_M_PI": 109,
|
||||
"const_M_E": 110,
|
||||
"panic": 111,
|
||||
"[": 112,
|
||||
"ctx_IPS": 113,
|
||||
"ctx_IFS": 114,
|
||||
"ctx_IRS": 115,
|
||||
"ctx_OPS": 116,
|
||||
"ctx_OFS": 117,
|
||||
"ctx_ORS": 118,
|
||||
"ctx_FLATSEP": 119,
|
||||
"ctx_NF": 120,
|
||||
"ctx_NR": 121,
|
||||
"ctx_FNR": 122,
|
||||
"ctx_FILENAME": 123,
|
||||
"ctx_FILENUM": 124,
|
||||
"env": 125,
|
||||
"call": 126,
|
||||
"begin": 127,
|
||||
"end": 128,
|
||||
"if": 129,
|
||||
"elif": 130,
|
||||
"else": 131,
|
||||
"while": 132,
|
||||
"do": 133,
|
||||
"for": 134,
|
||||
"in": 135,
|
||||
"break": 136,
|
||||
"continue": 137,
|
||||
"func": 138,
|
||||
"subr": 139,
|
||||
"return": 140,
|
||||
},
|
||||
}
|
||||
|
|
@ -1,3 +0,0 @@
|
|||
// Package util contains source code autogenerated by GOCC from the Miller DSL's
|
||||
// grammar file pkg/parsing/mlr.bnf.
|
||||
package util
|
||||
|
|
@ -1,101 +0,0 @@
|
|||
// Code generated by gocc; DO NOT EDIT.
|
||||
|
||||
package util
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strconv"
|
||||
"unicode"
|
||||
"unicode/utf8"
|
||||
)
|
||||
|
||||
// Interface.
|
||||
|
||||
// RuneValue will convert the literal value of a scanned token to a rune.
|
||||
func RuneValue(lit []byte) rune {
|
||||
if lit[1] == '\\' {
|
||||
return escapeCharVal(lit)
|
||||
}
|
||||
r, size := utf8.DecodeRune(lit[1:])
|
||||
if size != len(lit)-2 {
|
||||
panic(fmt.Sprintf("Error decoding rune. Lit: %s, rune: %d, size%d\n", lit, r, size))
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
// UintValue will attempt to parse a byte-slice as a signed base-10 64-bit integer.
|
||||
func IntValue(lit []byte) (int64, error) {
|
||||
return strconv.ParseInt(string(lit), 10, 64)
|
||||
}
|
||||
|
||||
// UintValue will attempt to parse a byte-slice as an unsigned base-10 64-bit integer.
|
||||
func UintValue(lit []byte) (uint64, error) {
|
||||
return strconv.ParseUint(string(lit), 10, 64)
|
||||
}
|
||||
|
||||
// Helpers.
|
||||
func escapeCharVal(lit []byte) rune {
|
||||
var i, base, max uint32
|
||||
offset := 2
|
||||
switch lit[offset] {
|
||||
case 'a':
|
||||
return '\a'
|
||||
case 'b':
|
||||
return '\b'
|
||||
case 'f':
|
||||
return '\f'
|
||||
case 'n':
|
||||
return '\n'
|
||||
case 'r':
|
||||
return '\r'
|
||||
case 't':
|
||||
return '\t'
|
||||
case 'v':
|
||||
return '\v'
|
||||
case '\\':
|
||||
return '\\'
|
||||
case '\'':
|
||||
return '\''
|
||||
case '0', '1', '2', '3', '4', '5', '6', '7':
|
||||
i, base, max = 3, 8, 255
|
||||
case 'x':
|
||||
i, base, max = 2, 16, 255
|
||||
offset++
|
||||
case 'u':
|
||||
i, base, max = 4, 16, unicode.MaxRune
|
||||
offset++
|
||||
case 'U':
|
||||
i, base, max = 8, 16, unicode.MaxRune
|
||||
offset++
|
||||
default:
|
||||
panic(fmt.Sprintf("Error decoding character literal: %s\n", lit))
|
||||
}
|
||||
|
||||
var x uint32
|
||||
for ; i > 0 && offset < len(lit)-1; i-- {
|
||||
ch, size := utf8.DecodeRune(lit[offset:])
|
||||
offset += size
|
||||
d := uint32(digitVal(ch))
|
||||
if d >= base {
|
||||
panic(fmt.Sprintf("charVal(%s): illegal character (%c) in escape sequence. size=%d, offset=%d", lit, ch, size, offset))
|
||||
}
|
||||
x = x*base + d
|
||||
}
|
||||
if x > max || 0xD800 <= x && x < 0xE000 {
|
||||
panic(fmt.Sprintf("Error decoding escape char value. Lit:%s, offset:%d, escape sequence is invalid Unicode code point\n", lit, offset))
|
||||
}
|
||||
|
||||
return rune(x)
|
||||
}
|
||||
|
||||
func digitVal(ch rune) int {
|
||||
switch {
|
||||
case '0' <= ch && ch <= '9':
|
||||
return int(ch) - '0'
|
||||
case 'a' <= ch && ch <= 'f':
|
||||
return int(ch) - 'a' + 10
|
||||
case 'A' <= ch && ch <= 'F':
|
||||
return int(ch) - 'A' + 10
|
||||
}
|
||||
return 16 // larger than any legal digit val
|
||||
}
|
||||
|
|
@ -1,39 +0,0 @@
|
|||
// Code generated by gocc; DO NOT EDIT.
|
||||
|
||||
package util
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
)
|
||||
|
||||
func RuneToString(r rune) string {
|
||||
if r >= 0x20 && r < 0x7f {
|
||||
return fmt.Sprintf("'%c'", r)
|
||||
}
|
||||
switch r {
|
||||
case 0x07:
|
||||
return "'\\a'"
|
||||
case 0x08:
|
||||
return "'\\b'"
|
||||
case 0x0C:
|
||||
return "'\\f'"
|
||||
case 0x0A:
|
||||
return "'\\n'"
|
||||
case 0x0D:
|
||||
return "'\\r'"
|
||||
case 0x09:
|
||||
return "'\\t'"
|
||||
case 0x0b:
|
||||
return "'\\v'"
|
||||
case 0x5c:
|
||||
return "'\\\\\\'"
|
||||
case 0x27:
|
||||
return "'\\''"
|
||||
case 0x22:
|
||||
return "'\\\"'"
|
||||
}
|
||||
if r < 0x10000 {
|
||||
return fmt.Sprintf("\\u%04x", r)
|
||||
}
|
||||
return fmt.Sprintf("\\U%08x", r)
|
||||
}
|
||||
|
|
@ -21,9 +21,9 @@ import (
|
|||
"fmt"
|
||||
"strings"
|
||||
|
||||
"github.com/johnkerl/miller/v6/pkg/dsl"
|
||||
"github.com/johnkerl/miller/v6/pkg/dsl/cst"
|
||||
"github.com/johnkerl/miller/v6/pkg/mlrval"
|
||||
"github.com/johnkerl/pgpg/go/lib/pkg/asts"
|
||||
)
|
||||
|
||||
func (repl *Repl) handleDSLStringImmediate(dslString string, doWarnings bool) error {
|
||||
|
|
@ -51,7 +51,7 @@ func (repl *Repl) handleDSLStringAux(
|
|||
cst.DSLInstanceTypeREPL,
|
||||
isReplImmediate,
|
||||
doWarnings,
|
||||
func(dslString string, astNode *dsl.AST) {
|
||||
func(dslString string, astNode *asts.AST) {
|
||||
if repl.astPrintMode == ASTPrintParex {
|
||||
astNode.PrintParex()
|
||||
} else if repl.astPrintMode == ASTPrintParexOneLine {
|
||||
|
|
|
|||
|
|
@ -9,11 +9,11 @@ import (
|
|||
"strings"
|
||||
|
||||
"github.com/johnkerl/miller/v6/pkg/colorizer"
|
||||
"github.com/johnkerl/miller/v6/pkg/dsl"
|
||||
"github.com/johnkerl/miller/v6/pkg/dsl/cst"
|
||||
"github.com/johnkerl/miller/v6/pkg/lib"
|
||||
"github.com/johnkerl/miller/v6/pkg/mlrval"
|
||||
"github.com/johnkerl/miller/v6/pkg/types"
|
||||
"github.com/johnkerl/pgpg/go/lib/pkg/asts"
|
||||
)
|
||||
|
||||
// Types for the lookup table.
|
||||
|
|
@ -462,7 +462,7 @@ func handleSkipOrProcessUntil(repl *Repl, dslString string, processingNotSkippin
|
|||
cst.DSLInstanceTypeREPL,
|
||||
true, // isReplImmediate
|
||||
repl.doWarnings,
|
||||
func(dslString string, astNode *dsl.AST) {
|
||||
func(dslString string, astNode *asts.AST) {
|
||||
if repl.astPrintMode == ASTPrintParex {
|
||||
astNode.PrintParex()
|
||||
} else if repl.astPrintMode == ASTPrintParexOneLine {
|
||||
|
|
|
|||
|
|
@ -6,12 +6,12 @@ import (
|
|||
"strings"
|
||||
|
||||
"github.com/johnkerl/miller/v6/pkg/cli"
|
||||
"github.com/johnkerl/miller/v6/pkg/dsl"
|
||||
"github.com/johnkerl/miller/v6/pkg/dsl/cst"
|
||||
"github.com/johnkerl/miller/v6/pkg/lib"
|
||||
"github.com/johnkerl/miller/v6/pkg/mlrval"
|
||||
"github.com/johnkerl/miller/v6/pkg/runtime"
|
||||
"github.com/johnkerl/miller/v6/pkg/types"
|
||||
"github.com/johnkerl/pgpg/go/lib/pkg/asts"
|
||||
)
|
||||
|
||||
const verbNamePut = "put"
|
||||
|
|
@ -425,7 +425,7 @@ func NewTransformerPut(
|
|||
false, // isReplImmediate
|
||||
doWarnings,
|
||||
|
||||
func(dslString string, astNode *dsl.AST) {
|
||||
func(dslString string, astNode *asts.AST) {
|
||||
|
||||
if echoDSLString {
|
||||
fmt.Println("DSL EXPRESSION:")
|
||||
|
|
|
|||
|
|
@ -2,12 +2,14 @@ DSL EXPRESSION:
|
|||
begin{begin{@x=1}}
|
||||
|
||||
AST:
|
||||
* statement block
|
||||
* begin block
|
||||
* statement block
|
||||
* begin block
|
||||
* statement block
|
||||
* assignment "="
|
||||
* direct oosvar value "x"
|
||||
* int literal "1"
|
||||
"block" [tt:block] [nt:StatementBlock]
|
||||
"begin" [tt:kw_begin] [nt:BeginBlock]
|
||||
"block" [tt:block] [nt:StatementBlockInBraces]
|
||||
"block" [tt:block] [nt:StatementBlock]
|
||||
"begin" [tt:kw_begin] [nt:BeginBlock]
|
||||
"block" [tt:block] [nt:StatementBlockInBraces]
|
||||
"block" [tt:block] [nt:StatementBlock]
|
||||
"=" [tt:equals] [nt:Assignment]
|
||||
"@x" [tt:oosvar_name] [nt:DirectOosvarValue]
|
||||
"1" [tt:int_literal] [nt:int_literal]
|
||||
|
||||
|
|
|
|||
|
|
@ -2,13 +2,15 @@ DSL EXPRESSION:
|
|||
true{begin{@x=1}}
|
||||
|
||||
AST:
|
||||
* statement block
|
||||
* cond block
|
||||
* bool literal "true"
|
||||
* statement block
|
||||
* begin block
|
||||
* statement block
|
||||
* assignment "="
|
||||
* direct oosvar value "x"
|
||||
* int literal "1"
|
||||
"block" [tt:block] [nt:StatementBlock]
|
||||
"true" [tt:literal_true] [nt:CondBlock]
|
||||
"true" [tt:literal_true] [nt:bool_literal]
|
||||
"block" [tt:block] [nt:StatementBlockInBraces]
|
||||
"block" [tt:block] [nt:StatementBlock]
|
||||
"begin" [tt:kw_begin] [nt:BeginBlock]
|
||||
"block" [tt:block] [nt:StatementBlockInBraces]
|
||||
"block" [tt:block] [nt:StatementBlock]
|
||||
"=" [tt:equals] [nt:Assignment]
|
||||
"@x" [tt:oosvar_name] [nt:DirectOosvarValue]
|
||||
"1" [tt:int_literal] [nt:int_literal]
|
||||
|
||||
|
|
|
|||
|
|
@ -2,12 +2,14 @@ DSL EXPRESSION:
|
|||
end{end{@x=1}}
|
||||
|
||||
AST:
|
||||
* statement block
|
||||
* end block
|
||||
* statement block
|
||||
* end block
|
||||
* statement block
|
||||
* assignment "="
|
||||
* direct oosvar value "x"
|
||||
* int literal "1"
|
||||
"block" [tt:block] [nt:StatementBlock]
|
||||
"end" [tt:kw_end] [nt:EndBlock]
|
||||
"block" [tt:block] [nt:StatementBlockInBraces]
|
||||
"block" [tt:block] [nt:StatementBlock]
|
||||
"end" [tt:kw_end] [nt:EndBlock]
|
||||
"block" [tt:block] [nt:StatementBlockInBraces]
|
||||
"block" [tt:block] [nt:StatementBlock]
|
||||
"=" [tt:equals] [nt:Assignment]
|
||||
"@x" [tt:oosvar_name] [nt:DirectOosvarValue]
|
||||
"1" [tt:int_literal] [nt:int_literal]
|
||||
|
||||
|
|
|
|||
|
|
@ -2,13 +2,15 @@ DSL EXPRESSION:
|
|||
true{end{@x=1}}
|
||||
|
||||
AST:
|
||||
* statement block
|
||||
* cond block
|
||||
* bool literal "true"
|
||||
* statement block
|
||||
* end block
|
||||
* statement block
|
||||
* assignment "="
|
||||
* direct oosvar value "x"
|
||||
* int literal "1"
|
||||
"block" [tt:block] [nt:StatementBlock]
|
||||
"true" [tt:literal_true] [nt:CondBlock]
|
||||
"true" [tt:literal_true] [nt:bool_literal]
|
||||
"block" [tt:block] [nt:StatementBlockInBraces]
|
||||
"block" [tt:block] [nt:StatementBlock]
|
||||
"end" [tt:kw_end] [nt:EndBlock]
|
||||
"block" [tt:block] [nt:StatementBlockInBraces]
|
||||
"block" [tt:block] [nt:StatementBlock]
|
||||
"=" [tt:equals] [nt:Assignment]
|
||||
"@x" [tt:oosvar_name] [nt:DirectOosvarValue]
|
||||
"1" [tt:int_literal] [nt:int_literal]
|
||||
|
||||
|
|
|
|||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Add table
Add a link
Reference in a new issue