Mirrors/miller
1
0
Fork 0
mirror of https://github.com/johnkerl/miller.git synced 2026-08-02 20:42:52 +00:00
Code Issues Projects Releases Wiki Activity

Merge pull request #411 from johnkerl/go-mod

Browse source 
Use Go modules
This commit is contained in:
John Kerl 2021-02-14 10:51:10 -05:00 • committed by GitHub
commit 9f28822f68
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
969 changed files with 582 additions and 229784 deletions
Download patch file Download diff file Expand all files Collapse all files

60
go/README.md
View file

@ -81,10 +81,10 @@ During the coding of Miller, I've been guided by the following:
* Names of files, variables, functions, etc. should be fully spelled out (e.g. `NewEvaluableLeafNode`), except for a small number of most-used names where a longer name would cause unnecessary line-wraps (e.g. `Mlrval` instead of `MillerValue` since this appears very very often).
* Code should not be too clever. This includes some reasonable amounts of code duplication from time to time, to keep things inline, rather than lasagna code.
* Things should be transparent. For example, `mlr -n put -v '$y = 3 + 0.1 * $x'` shows you the abstract syntax tree derived from the DSL expression.
* Comments should be robust with respect to reasonably anticipated changes. For example, one package should cross-link to another in its comments, but I try to avoid mentioning specific filenames too much in the comments and README files since these may change over time. I make an exception for stable points such as [mlr.go](./mlr.go), [mlr.bnf](./src/miller/parsing/mlr.bnf), [stream.go](./src/miller/stream/stream.go), etc.
* Comments should be robust with respect to reasonably anticipated changes. For example, one package should cross-link to another in its comments, but I try to avoid mentioning specific filenames too much in the comments and README files since these may change over time. I make an exception for stable points such as [mlr.go](./mlr.go), [mlr.bnf](./src/parsing/mlr.bnf), [stream.go](./src/stream/stream.go), etc.
* *Miller should be pleasant to write.*
* It should be quick to answer the question *Did I just break anything?* -- hence the `build` and `reg_test/run` regression scripts.
* It should be quick to find out what to do next as you iteratively develop -- see for example [cst/README.md](https://github.com/johnkerl/miller/blob/master/go/src/miller/dsl/cst/README.md).
* It should be quick to find out what to do next as you iteratively develop -- see for example [cst/README.md](https://github.com/johnkerl/miller/blob/master/go/src/dsl/cst/README.md).
* *The language should be an asset, not a liability.*
* One of the reasons I chose Go is that (personally anyway) I find it to be reasonably efficient, well-supported with standard libraries, straightforward, and fun. I hope you enjoy it as much as I have.
@ -103,40 +103,44 @@ sequence of key-value pairs. The basic **stream** operation is:
So, in broad overview, the key packages are:
* [src/miller/stream](./src/miller/stream) -- connect input -> transforms -> output via Go channels
* [src/miller/input](./src/miller/input) -- read input records
* [src/miller/transforming](./src/miller/transforming) -- transform input records to output records
* [src/miller/output](./src/miller/output) -- write output records
* [src/stream](./src/stream) -- connect input -> transforms -> output via Go channels
* [src/input](./src/input) -- read input records
* [src/transforming](./src/transforming) -- transform input records to output records
* [src/output](./src/output) -- write output records
* The rest are details to support this.
## Directory-structure details
### Dependencies
* Miller dependencies are all in the Go standard library, except a local one:
* `src/github.com/goccmack`
* GOCC lexer/parser code-generator from [github.com/goccmack/gocc](https://github.com/goccmack/gocc):
* Miller dependencies are all in the Go standard library, except two:
* GOCC lexer/parser code-generator from [github.com/goccmack/gocc](https://github.com/goccmack/gocc):
* This package defines the grammar for Miller's domain-specific language (DSL) for the Miller `put` and `filter` verbs. And, GOCC is a joy to use. :)
* Note on the path: `go get github.com/goccmack/gocc` uses this directory path, and is nice enough to also create `bin/gocc` for me -- so I thought I would just let it continue to do that by using that local path. :)
* I kept this locally so I could source-control it along with Miller and guarantee its stability. It is used on the terms of its open-source license.
* It is used on the terms of its open-source license.
* [golang.org/x/term](https://pkg.go.dev/golang.org/x/term):
* Just a one-line Miller callsite for is-a-terminal checking for the [Miller REPL](https://github.com/johnkerl/miller/blob/go-mod/go/src/auxents/repl/README.md).
* It is used on the terms of its open-source license.
* See also [./go.mod](go.mod). Setup:
* `go get github.com/goccmack/gocc`
* `go get golang.org/x/term`
### Miller per se
* The main entry point is [mlr.go](./mlr.go); everything else in [src/miller](./src/miller).
* [src/miller/lib](./src/miller/lib):
* Implementation of the [`Mlrval`](./src/miller/types/mlrval.go) datatype which includes string/int/float/boolean/void/absent/error types. These are used for record values, as well as expression/variable values in the Miller `put`/`filter` DSL. See also below for more details.
* [`Mlrmap`](./src/miller/types/mlrmap.go) is the sequence of key-value pairs which represents a Miller record. The key-lookup mechanism is optimized for Miller read/write usage patterns -- please see [mlrmap.go](./src/miller/types/mlrmap.go) for more details.
* [`context`](./src/miller/types/context.go) supports AWK-like variables such as `FILENAME`, `NF`, `NR`, and so on.
* [src/miller/cli](./src/miller/cli) is the flag-parsing logic for supporting Miller's command-line interface. When you type something like `mlr --icsv --ojson put '$sum = $a + $b' then filter '$sum > 1000' myfile.csv`, it's the CLI parser which makes it possible for Miller to construct a CSV record-reader, a transformer-chain of `put` then `filter`, and a JSON record-writer.
* [src/miller/cliutil](./src/miller/cliutil) contains datatypes for the CLI-parser, which was split out to avoid a Go package-import cycle.
* [src/miller/stream](./src/miller/stream) is as above -- it uses Go channels to pipe together file-reads, to record-reading/parsing, to a chain of record-transformers, to record-writing/formatting, to terminal standard output.
* [src/miller/input](./src/miller/input) is as above -- one record-reader type per supported input file format, and a factory method.
* [src/miller/output](./src/miller/output) is as above -- one record-writer type per supported output file format, and a factory method.
* [src/miller/transforming](./src/miller/transforming) contains the abstract record-transformer interface datatype, as well as the Go-channel chaining mechanism for piping one transformer into the next.
* [src/miller/transformers](./src/miller/transformers) is all the concrete record-transformers such as `cat`, `tac`, `sort`, `put`, and so on. I put it here, not in `transforming`, so all files in `transformers` would be of the same type.
* [src/miller/parsing](./src/miller/parsing) contains a single source file, `mlr.bnf`, which is the lexical/semantic grammar file for the Miller `put`/`filter` DSL using the GOCC framework. All subdirectories of `src/miller/parsing/` are autogen code created by GOCC's processing of `mlr.bnf`.
* [src/miller/dsl](./src/miller/dsl) contains [`ast_types.go`](src/miller/dsl/ast_types.go) which is the abstract syntax tree datatype shared between GOCC and Miller. I didn't use a `src/miller/dsl/ast` naming convention, although that would have been nice, in order to avoid a Go package-dependency cycle.
* [src/miller/dsl/cst](./src/miller/dsl/cst) is the concrete syntax tree, constructed from an AST produced by GOCC. The CST is what is actually executed on every input record when you do things like `$z = $x * 0.3 * $y`. Please see the [src/miller/dsl/cst/README.md](./src/miller/dsl/cst/README.md) for more information.
* The main entry point is [mlr.go](./mlr.go); everything else in [src](./src).
* [src/lib](./src/lib):
* Implementation of the [`Mlrval`](./src/types/mlrval.go) datatype which includes string/int/float/boolean/void/absent/error types. These are used for record values, as well as expression/variable values in the Miller `put`/`filter` DSL. See also below for more details.
* [`Mlrmap`](./src/types/mlrmap.go) is the sequence of key-value pairs which represents a Miller record. The key-lookup mechanism is optimized for Miller read/write usage patterns -- please see [mlrmap.go](./src/types/mlrmap.go) for more details.
* [`context`](./src/types/context.go) supports AWK-like variables such as `FILENAME`, `NF`, `NR`, and so on.
* [src/cli](./src/cli) is the flag-parsing logic for supporting Miller's command-line interface. When you type something like `mlr --icsv --ojson put '$sum = $a + $b' then filter '$sum > 1000' myfile.csv`, it's the CLI parser which makes it possible for Miller to construct a CSV record-reader, a transformer-chain of `put` then `filter`, and a JSON record-writer.
* [src/cliutil](./src/cliutil) contains datatypes for the CLI-parser, which was split out to avoid a Go package-import cycle.
* [src/stream](./src/stream) is as above -- it uses Go channels to pipe together file-reads, to record-reading/parsing, to a chain of record-transformers, to record-writing/formatting, to terminal standard output.
* [src/input](./src/input) is as above -- one record-reader type per supported input file format, and a factory method.
* [src/output](./src/output) is as above -- one record-writer type per supported output file format, and a factory method.
* [src/transforming](./src/transforming) contains the abstract record-transformer interface datatype, as well as the Go-channel chaining mechanism for piping one transformer into the next.
* [src/transformers](./src/transformers) is all the concrete record-transformers such as `cat`, `tac`, `sort`, `put`, and so on. I put it here, not in `transforming`, so all files in `transformers` would be of the same type.
* [src/parsing](./src/parsing) contains a single source file, `mlr.bnf`, which is the lexical/semantic grammar file for the Miller `put`/`filter` DSL using the GOCC framework. All subdirectories of `src/parsing/` are autogen code created by GOCC's processing of `mlr.bnf`.
* [src/dsl](./src/dsl) contains [`ast_types.go`](src/dsl/ast_types.go) which is the abstract syntax tree datatype shared between GOCC and Miller. I didn't use a `src/dsl/ast` naming convention, although that would have been nice, in order to avoid a Go package-dependency cycle.
* [src/dsl/cst](./src/dsl/cst) is the concrete syntax tree, constructed from an AST produced by GOCC. The CST is what is actually executed on every input record when you do things like `$z = $x * 0.3 * $y`. Please see the [src/dsl/cst/README.md](./src/dsl/cst/README.md) for more information.
## Nil-record conventions
@ -168,7 +172,7 @@ nil through the reader/transformer/writer sequence.
## More about mlrvals
[`Mlrval`](./src/miller/types/mlrval.go) is the datatype of record values, as well as expression/variable values in the Miller `put`/`filter` DSL. It includes string/int/float/boolean/void/absent/error types, not unlike PHP's `zval`.
[`Mlrval`](./src/types/mlrval.go) is the datatype of record values, as well as expression/variable values in the Miller `put`/`filter` DSL. It includes string/int/float/boolean/void/absent/error types, not unlike PHP's `zval`.
* Miller's `absent` type is like Javascript's `undefined` -- it's for times when there is no such key, as in a DSL expression `$out = $foo` when the input record is `$x=3,y=4` -- there is no `$foo` so `$foo` has `absent` type. Nothing is written to the `$out` field in this case. See also [here](http://johnkerl.org/miller/doc/reference.html#Null_data:_empty_and_absent) for more information.
* Miller's `void` type is like Javascript's `null` -- it's for times when there is a key with no value, as in `$out = $x` when the input record is `$x=,$y=4`. This is an overlap with `string` type, since a void value looks like an empty string. I've gone back and forth on this (including when I was writing the C implementation) -- whether to retain `void` as a distinct type from empty-string, or not. I ended up keeping it as it made the `Mlrval` logic easier to understand.
@ -176,7 +180,7 @@ nil through the reader/transformer/writer sequence.
* Miller's number handling makes auto-overflow from int to float transparent, while preserving the possibility of 64-bit bitwise arithmetic.
* This is different from JavaScript, which has only double-precision floats and thus no support for 64-bit numbers (note however that there is now [`BigInt`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/BigInt)).
* This is also different from C and Go, wherein casts are necessary -- without which int arithmetic overflows.
* See also [here](http://johnkerl.org/miller/doc/reference.html#Arithmetic) for the semantics of Miller arithmetic, which the [`Mlrval`](./src/miller/types/mlrval.go) class implements.
* See also [here](http://johnkerl.org/miller/doc/reference.html#Arithmetic) for the semantics of Miller arithmetic, which the [`Mlrval`](./src/types/mlrval.go) class implements.
## Software-testing methodology

38
go/build-dsl
View file

@ -2,6 +2,13 @@
# ================================================================
# Reads the Miller DSL grammar file and generates Go code.
#
# This is not run on every build / commit / etc.
#
# It's intended to be run manually by the developer, as needed when mlr.bnf
# changes for example.
#
# Resulting auto-generated .go files should then be checked into source control.
# ================================================================
set -euo pipefail
@ -13,27 +20,30 @@ if [ $# -eq 1 ]; then
fi
fi
export GOPATH=$(pwd)
# Build the bin/gocc executable:
go get github.com/goccmack/gocc
#go get github.com/goccmack/gocc
go get github.com/johnkerl/gocc
bingocc="$GOPATH/bin/gocc"
if [ ! -x "$bingocc" ]; then
exit 1
fi
rm -f src/miller/parsing/*.txt
rm -f src/parsing/*.txt
if [ "$verbose" = "true" ]; then
lr1="src/miller/parsing/LR1_conflicts.txt"
bin/gocc -v -o src/miller/parsing src/miller/parsing/mlr.bnf || expand -2 $lr1
lr1="src/parsing/LR1_conflicts.txt"
$bingocc -v -o ./src/parsing -p miller/src/parsing src/parsing/mlr.bnf || expand -2 $lr1
else
bin/gocc -o src/miller/parsing src/miller/parsing/mlr.bnf
$bingocc -o ./src/parsing -p miller/src/parsing src/parsing/mlr.bnf
fi
# Code-gen directories:
# src/miller/parsing/errors/
# src/miller/parsing/lexer/
# src/miller/parsing/parser/
# src/miller/parsing/token/
# src/miller/parsing/util/
# src/parsing/errors/
# src/parsing/lexer/
# src/parsing/parser/
# src/parsing/token/
# src/parsing/util/
# Override GOCC codegen with customized error handling
cp src/miller/parsing/errors.go.template src/miller/parsing/errors/errors.go
cp src/parsing/errors.go.template src/parsing/errors/errors.go
for x in src/miller/parsing/*/*.go; do gofmt -w $x; done
for x in src/parsing/*/*.go; do gofmt -w $x; done

4
go/build-go
View file

@ -2,10 +2,8 @@
set -euo pipefail
export GOPATH=$(pwd)
# Override GOCC codegen with customized error handling
cp src/miller/parsing/errors.go.template src/miller/parsing/errors/errors.go
cp src/parsing/errors.go.template src/parsing/errors/errors.go
go build mlr.go
cp mlr mlrgo

9
go/go.mod Normal file
View file

@ -0,0 +1,9 @@
module miller
go 1.15
require (
github.com/goccmack/gocc v0.0.0-20210201103733-1bd198f09019 // indirect
github.com/johnkerl/gocc v0.0.0-20210214145925-018e125fd1ee // indirect
golang.org/x/term v0.0.0-20201210144234-2321bbc49cbf
)

39
go/go.sum Normal file
View file

@ -0,0 +1,39 @@
github.com/goccmack/gocc v0.0.0-20210201103733-1bd198f09019 h1:frVuW4LszXD/AjX9dC1y4I+GwTjZMl6153O2mk/lmHo=
github.com/goccmack/gocc v0.0.0-20210201103733-1bd198f09019/go.mod h1:ZjwYs2UwOh6NiluQ1XCoKNLVECdeFf33rrGEZ+IpNq4=
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/johnkerl/gocc v0.0.0-20210214145651-3650cede3074 h1:8+mZMwac/qKah1hHAuw1U5I7mqN+uRF+oVZ4ismxcJU=
github.com/johnkerl/gocc v0.0.0-20210214145651-3650cede3074/go.mod h1:6oAcMpl/qp57NgiufLc2bISrDa4q12XciKeOCUqdeoU=
github.com/johnkerl/gocc v0.0.0-20210214145925-018e125fd1ee h1:UCSd1mw2XP4Hv2iVlLjU6Om+DZfBc2XMJ1GqHorQUSc=
github.com/johnkerl/gocc v0.0.0-20210214145925-018e125fd1ee/go.mod h1:6oAcMpl/qp57NgiufLc2bISrDa4q12XciKeOCUqdeoU=
github.com/kisielk/errcheck v1.2.0/go.mod h1:/BMXB+zMLi60iA8Vv6Ksmxu/1UDYcXs4uQLJ+jE2L00=
github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/mod v0.3.0 h1:RM4zey1++hCTbCVQfnWeKs9/IEsaBLA8vTkd0WVtmH4=
golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210119212857-b64e53b001e4 h1:myAQVi0cGEoqQVR5POX+8RR2mrocKqNN1hmeMqhX27k=
golang.org/x/sys v0.0.0-20210119212857-b64e53b001e4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/term v0.0.0-20201210144234-2321bbc49cbf h1:MZ2shdL+ZM/XzY3ZGOnh4Nlpnxz5GSOhOmtHo3iPU6M=
golang.org/x/term v0.0.0-20201210144234-2321bbc49cbf/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20181030221726-6c7e314b6563/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.1.0/go.mod h1:xkSsbof2nBLbhDlRMhhhyNLN/zl3eTqcnHD5viDpcZ0=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=

2
go/mktags
View file

@ -11,5 +11,5 @@ set -euo pipefail
#
# See also https://stackoverflow.com/questions/8204367/ctag-database-for-go
ctags -f gosource.tags -R `pwd`/src/miller
ctags -f gosource.tags -R `pwd`/src
mv gosource.tags tags

6
go/mlr.go
View file

@ -6,9 +6,9 @@ import (
"runtime"
"runtime/pprof"
"miller/auxents"
"miller/cli"
"miller/stream"
"miller/src/auxents"
"miller/src/cli"
"miller/src/stream"
)
// ----------------------------------------------------------------

12
go/parser-experiments/one/build
View file

@ -27,18 +27,22 @@ mkdir -p $dir
# ----------------------------------------------------------------
# Run the parser-generator
export GOPATH=$(pwd)
# Build the bin/gocc executable:
go get github.com/goccmack/gocc
#go get github.com/goccmack/gocc
go get github.com/johnkerl/gocc
bingocc="$GOPATH/bin/gocc"
if [ ! -x "$bingocc" ]; then
exit 1
fi
rm -f $dir/*.txt
if [ "$verbose" = "true" ]; then
lr1="$dir/LR1_conflicts.txt"
# The -o specifies the package name within the autogen
bin/gocc -v -o $dir $bnf || expand -2 $lr1
$bingocc -v -o $dir $bnf || expand -2 $lr1
else
bin/gocc -o $dir $bnf
$bingocc -o $dir $bnf
fi
echo "Parser-autogen OK"

4
go/parser-experiments/one/main.go
View file

@ -4,8 +4,8 @@ import (
"fmt"
"os"
"experimental/lexer"
"experimental/parser"
"miller/src/parsing/lexer"
"miller/src/parsing/parser"
)
func parseOne(input string) {

28
go/parser-experiments/two/build
View file

@ -27,18 +27,22 @@ mkdir -p $dir
# ----------------------------------------------------------------
# Run the parser-generator
export GOPATH=$(pwd)
# Build the bin/gocc executable:
go get github.com/goccmack/gocc
#go get github.com/goccmack/gocc
go get github.com/johnkerl/gocc
bingocc="$GOPATH/bin/gocc"
if [ ! -x "$bingocc" ]; then
exit 1
fi
rm -f $dir/*.txt
if [ "$verbose" = "true" ]; then
lr1="$dir/LR1_conflicts.txt"
# The -o specifies the package name within the autogen
bin/gocc -v -o $dir $bnf || expand -2 $lr1
$bingocc -v -o $dir $bnf || expand -2 $lr1
else
bin/gocc -o $dir $bnf
$bingocc -o $dir $bnf
fi
echo "Parser-autogen OK"
@ -52,23 +56,23 @@ echo "Parser-autogen OK"
# ----------------------------------------------------------------
# Override GOCC codegen with customized error handling
cp ../../src/miller/parsing/errors.go.template src/experimental/errors/errors.go
cp ../../src/parsing/errors.go.template src/experimental/errors/errors.go
sed -i .bak 's:miller/parsing:experimental:' src/experimental/errors/errors.go
# ----------------------------------------------------------------
# Copy AST files from the main Miller tree
rm -rf ./src/miller/lib/
rm -rf ./src/miller/dsl/
rm -rf ./src/lib/
rm -rf ./src/dsl/
mkdir -p ./src/miller/lib/
mkdir -p ./src/miller/dsl/
mkdir -p ./src/lib/
mkdir -p ./src/dsl/
cp ../../src/miller/lib/*.go ./src/miller/lib/
cp ../../src/miller/dsl/ast*.go ./src/miller/dsl/
cp ../../src/lib/*.go ./src/lib/
cp ../../src/dsl/ast*.go ./src/dsl/
# Different path to autogen between main Miller tree and here
sed -i .bak 's:miller/parsing:experimental:' src/miller/dsl/ast*go
sed -i .bak 's:miller/parsing:experimental:' src/dsl/ast*go
# ----------------------------------------------------------------
# Compile the main and the parser-autogen

6
go/parser-experiments/two/main.go
View file

@ -4,10 +4,10 @@ import (
"fmt"
"os"
"miller/dsl"
"miller/src/dsl"
"experimental/lexer"
"experimental/parser"
"miller/src/parsing/lexer"
"miller/src/parsing/parser"
)
const GREEN = "\033[32;01m"

7
go/src/README.md
View file

@ -1,6 +1 @@
* Miller dependencies are all in the Go standard library, except a local one:
* `src/github.com/goccmack`
* GOCC lexer/parser code-generator from [github.com/goccmack/gocc](https://github.com/goccmack/gocc):
* This package defines the grammar for Miller's domain-specific language (DSL) for the Miller `put` and `filter` verbs. And, GOCC is a joy to use. :)
* Note on the path: `go get github.com/goccmack/gocc` uses this directory path, and is nice enough to also create `bin/gocc` for me -- so I thought I would just let it continue to do that by using that local path. :)
* I kept this locally so I could source-control it along with Miller and guarantee its stability. It is used on the terms of its open-source license.
Please see [go/README.md](https://github.com/johnkerl/miller/blob/master/go/README.md) for an overview; please see each subdirectory for details about it.

2
go/src/miller/auxents/auxents.go → go/src/auxents/auxents.go
View file

@ -10,7 +10,7 @@ import (
"os"
"path"
"miller/auxents/repl"
"miller/src/auxents/repl"
)
// ----------------------------------------------------------------

0
go/src/miller/auxents/hex.go → go/src/auxents/hex.go
View file

0
go/src/miller/auxents/lecat.go → go/src/auxents/lecat.go
View file

0
go/src/miller/auxents/repl/README.md → go/src/auxents/repl/README.md
View file

8
go/src/miller/auxents/repl/ast.go → go/src/auxents/repl/ast.go
View file

@ -8,10 +8,10 @@ import (
"fmt"
"os"
"miller/dsl"
"miller/lib"
"miller/parsing/lexer"
"miller/parsing/parser"
"miller/src/dsl"
"miller/src/lib"
"miller/src/parsing/lexer"
"miller/src/parsing/parser"
)
// ----------------------------------------------------------------

2
go/src/miller/auxents/repl/dsl.go → go/src/auxents/repl/dsl.go
View file

@ -23,7 +23,7 @@ import (
"fmt"
"strings"
"miller/types"
"miller/src/types"
)
// ----------------------------------------------------------------

4
go/src/miller/auxents/repl/entry.go → go/src/auxents/repl/entry.go
View file

@ -27,8 +27,8 @@ import (
"path"
"strings"
"miller/cli"
"miller/cliutil"
"miller/src/cli"
"miller/src/cliutil"
)
// ================================================================

2
go/src/miller/auxents/repl/prompt.go → go/src/auxents/repl/prompt.go
View file

@ -10,7 +10,7 @@ import (
"golang.org/x/term"
"miller/version"
"miller/src/version"
)
const ENV_PRIMARY_PROMPT = "MLR_REPL_PS1"

12
go/src/miller/auxents/repl/session.go → go/src/auxents/repl/session.go
View file

@ -26,12 +26,12 @@ import (
"strings"
"syscall"
"miller/cliutil"
"miller/dsl/cst"
"miller/input"
"miller/output"
"miller/runtime"
"miller/types"
"miller/src/cliutil"
"miller/src/dsl/cst"
"miller/src/input"
"miller/src/output"
"miller/src/runtime"
"miller/src/types"
)
// ----------------------------------------------------------------

12
go/src/miller/auxents/repl/types.go → go/src/auxents/repl/types.go
View file

@ -7,12 +7,12 @@ package repl
import (
"os"
"miller/cliutil"
"miller/dsl/cst"
"miller/input"
"miller/output"
"miller/runtime"
"miller/types"
"miller/src/cliutil"
"miller/src/dsl/cst"
"miller/src/input"
"miller/src/output"
"miller/src/runtime"
"miller/src/types"
)
// ================================================================

6
go/src/miller/auxents/repl/verbs.go → go/src/auxents/repl/verbs.go
View file

@ -10,9 +10,9 @@ import (
"os"
"strings"
"miller/dsl/cst"
"miller/lib"
"miller/types"
"miller/src/dsl/cst"
"miller/src/lib"
"miller/src/types"
)
// ----------------------------------------------------------------

0
go/src/miller/auxents/termcvt.go → go/src/auxents/termcvt.go
View file

0
go/src/miller/auxents/unhex.go → go/src/auxents/unhex.go
View file

5
go/src/cli/README.md Normal file
View file

@ -0,0 +1,5 @@
Logic for parsing the Miller command line.
* `src/cli` is the flag-parsing logic for supporting Miller's command-line interface. When you type something like `mlr --icsv --ojson put '$sum = $a + $b' then filter '$sum > 1000' myfile.csv`, it's the CLI parser which makes it possible for Miller to construct a CSV record-reader, a transformer chain of `put` then `filter`, and a JSON record-writer.
* `src/cliutil` contains datatypes for the CLI-parser, which was split out to avoid a Go package-import cycle.
* I don't use the Go [`flag`](https://golang.org/pkg/flag/) package here, although I do use it within the transformers' subcommand flag-handling. The `flag` package is quite fine; Miller's command-line processing is multi-purpose between serving CLI needs per se as well as for manpage/docfile generation, and I found it simplest to roll my own command-line handling here.

0
go/src/miller/cli/mlrcli.go → go/src/cli/mlrcli.go
View file

12
go/src/miller/cli/mlrcli_parse.go → go/src/cli/mlrcli_parse.go
View file

@ -8,12 +8,12 @@ import (
"regexp"
"strings"
"miller/cliutil"
"miller/dsl/cst"
"miller/lib"
"miller/transformers"
"miller/transforming"
"miller/version"
"miller/src/cliutil"
"miller/src/dsl/cst"
"miller/src/lib"
"miller/src/transformers"
"miller/src/transforming"
"miller/src/version"
)
// ----------------------------------------------------------------

6
go/src/miller/cli/mlrcli_transformers.go → go/src/cli/mlrcli_transformers.go
View file

@ -4,9 +4,9 @@ import (
"fmt"
"os"
"miller/lib"
"miller/transformers"
"miller/transforming"
"miller/src/lib"
"miller/src/transformers"
"miller/src/transforming"
)
// ----------------------------------------------------------------

4
go/src/miller/cli/mlrcli_usage.go → go/src/cli/mlrcli_usage.go
View file

@ -4,8 +4,8 @@ import (
"fmt"
"os"
"miller/lib"
"miller/version"
"miller/src/lib"
"miller/src/version"
)
// ----------------------------------------------------------------

4
go/src/cliutil/README.md Normal file
View file

@ -0,0 +1,4 @@
Datatypes for parsing the Miller command line.
* `src/cli` is the flag-parsing logic for supporting Miller's command-line interface. When you type something like `mlr --icsv --ojson put '$sum = $a + $b' then filter '$sum > 1000' myfile.csv`, it's the CLI parser which makes it possible for Miller to construct a CSV record-reader, a transformer chain of `put` then `filter`, and a JSON record-writer.
* `src/cliutil` contains datatypes for the CLI-parser, which was split out to avoid a Go package-import cycle.

2
go/src/miller/cliutil/mlrcli_util.go → go/src/cliutil/mlrcli_util.go
View file

@ -4,7 +4,7 @@ import (
"fmt"
"os"
"miller/lib"
"miller/src/lib"
)
// For flags with values, e.g. ["-n" "10"], while we're looking at the "-n"

2
go/src/miller/cliutil/option_parse.go → go/src/cliutil/option_parse.go
View file

@ -10,7 +10,7 @@ import (
"fmt"
"os"
"miller/lib"
"miller/src/lib"
)
const ASV_FS = "\x1f"

0
go/src/miller/cliutil/option_types.go → go/src/cliutil/option_types.go
View file

2
go/src/miller/cliutil/verb_utils.go → go/src/cliutil/verb_utils.go
View file

@ -9,7 +9,7 @@ import (
"os"
"strconv"
"miller/lib"
"miller/src/lib"
)
// For flags with values, e.g. ["-n" "10"], while we're looking at the "-n" this let us see if the "10" slot exists.

2
go/src/miller/dsl/README.md → go/src/dsl/README.md
View file

@ -97,5 +97,5 @@ tree is executed once on every data record.
# Source directories/files
* The AST logic is in `./ast*.go`. I didn't use a `src/miller/dsl/ast` naming convention, although that would have been nice, in order to avoid a Go package-dependency cycle.
* The AST logic is in `./ast*.go`. I didn't use a `src/dsl/ast` naming convention, although that would have been nice, in order to avoid a Go package-dependency cycle.
* The CST logic is in [`./cst`](./cst). Please see [cst/README.md](./cst/README.md) for more information.

4
go/src/miller/dsl/ast_build.go → go/src/dsl/ast_build.go
View file

@ -10,8 +10,8 @@ import (
"fmt"
"strings"
"miller/lib"
"miller/parsing/token"
"miller/src/lib"
"miller/src/parsing/token"
)
// ----------------------------------------------------------------

0
go/src/miller/dsl/ast_print.go → go/src/dsl/ast_print.go
View file

6
go/src/miller/dsl/ast_test.go → go/src/dsl/ast_test.go
View file

@ -6,9 +6,9 @@ import (
"fmt"
"testing"
"miller/dsl"
"miller/parsing/lexer"
"miller/parsing/parser"
"miller/src/dsl"
"miller/src/parsing/lexer"
"miller/src/parsing/parser"
)
func testSingle(sourceString []byte) (*dsl.AST, error) {

2
go/src/miller/dsl/ast_types.go → go/src/dsl/ast_types.go
View file

@ -5,7 +5,7 @@
package dsl
import (
"miller/parsing/token"
"miller/src/parsing/token"
)
// ----------------------------------------------------------------

4
go/src/miller/dsl/cst/README.md → go/src/dsl/cst/README.md
View file

@ -1,4 +1,4 @@
See [go/src/miller/dsl/README.md](https://github.com/johnkerl/miller/blob/master/go/src/miller/dsl/README.md) for more information about Miller's use of abstract syntax trees (ASTs) and concrete syntax trees (CSTs) within the Miller `put`/`filter` domain-specific language (DSL).
See [go/src/dsl/README.md](https://github.com/johnkerl/miller/blob/master/go/src/dsl/README.md) for more information about Miller's use of abstract syntax trees (ASTs) and concrete syntax trees (CSTs) within the Miller `put`/`filter` domain-specific language (DSL).
## Files
@ -11,7 +11,7 @@ See [go/src/miller/dsl/README.md](https://github.com/johnkerl/miller/blob/master
Go is a strongly typed language, but the AST is polymorphic. This results in if/else or switch statements as an AST is walked.
Also, when we modify code, there can be changes in the [BNF grammar](../../parsing/mlr.bnf) not yet reflected in the [AST](../../src/miller/dsl/ast_types.go). Likewise, there can be AST changes not yet reflected here. (Example: you are partway through adding a new binary operator to the grammar.)
Also, when we modify code, there can be changes in the [BNF grammar](../../parsing/mlr.bnf) not yet reflected in the [AST](../../src/dsl/ast_types.go). Likewise, there can be AST changes not yet reflected here. (Example: you are partway through adding a new binary operator to the grammar.)
As a result, throughout the code, there are error checks which may seem redundant but which are in place to make incremental development more pleasant and robust.

6
go/src/miller/dsl/cst/assignments.go → go/src/dsl/cst/assignments.go
View file

@ -5,9 +5,9 @@
package cst
import (
"miller/dsl"
"miller/lib"
"miller/runtime"
"miller/src/dsl"
"miller/src/lib"
"miller/src/runtime"
)
// ================================================================

6
go/src/miller/dsl/cst/block-exit.go → go/src/dsl/cst/block-exit.go
View file

@ -8,9 +8,9 @@ package cst
import (
"errors"
"miller/dsl"
"miller/lib"
"miller/runtime"
"miller/src/dsl"
"miller/src/lib"
"miller/src/runtime"
)
// ----------------------------------------------------------------

6
go/src/miller/dsl/cst/blocks.go → go/src/dsl/cst/blocks.go
View file

@ -6,9 +6,9 @@
package cst
import (
"miller/dsl"
"miller/lib"
"miller/runtime"
"miller/src/dsl"
"miller/src/lib"
"miller/src/runtime"
)
// ----------------------------------------------------------------

4
go/src/miller/dsl/cst/builtin_function_manager.go → go/src/dsl/cst/builtin_function_manager.go
View file

@ -11,8 +11,8 @@ import (
"os"
"strings"
"miller/lib"
"miller/types"
"miller/src/lib"
"miller/src/types"
)
type TFunctionClass string

8
go/src/miller/dsl/cst/builtin_functions.go → go/src/dsl/cst/builtin_functions.go
View file

@ -8,10 +8,10 @@ import (
"errors"
"fmt"
"miller/dsl"
"miller/lib"
"miller/runtime"
"miller/types"
"miller/src/dsl"
"miller/src/lib"
"miller/src/runtime"
"miller/src/types"
)
// ----------------------------------------------------------------

8
go/src/miller/dsl/cst/collections.go → go/src/dsl/cst/collections.go
View file

@ -6,10 +6,10 @@
package cst
import (
"miller/dsl"
"miller/lib"
"miller/runtime"
"miller/types"
"miller/src/dsl"
"miller/src/lib"
"miller/src/runtime"
"miller/src/types"
)
// ----------------------------------------------------------------

8
go/src/miller/dsl/cst/cond.go → go/src/dsl/cst/cond.go
View file

@ -8,10 +8,10 @@ package cst
import (
"errors"
"miller/dsl"
"miller/lib"
"miller/runtime"
"miller/types"
"miller/src/dsl"
"miller/src/lib"
"miller/src/runtime"
"miller/src/types"
)
type CondBlockNode struct {

10
go/src/miller/dsl/cst/dump.go → go/src/dsl/cst/dump.go
View file

@ -22,11 +22,11 @@ import (
"os"
"strings"
"miller/dsl"
"miller/lib"
"miller/output"
"miller/runtime"
"miller/types"
"miller/src/dsl"
"miller/src/lib"
"miller/src/output"
"miller/src/runtime"
"miller/src/types"
)
// ================================================================

10
go/src/miller/dsl/cst/emit_emitp.go → go/src/dsl/cst/emit_emitp.go
View file

@ -34,11 +34,11 @@ import (
"errors"
"fmt"
"miller/dsl"
"miller/lib"
"miller/output"
"miller/runtime"
"miller/types"
"miller/src/dsl"
"miller/src/lib"
"miller/src/output"
"miller/src/runtime"
"miller/src/types"
)
// ================================================================

10
go/src/miller/dsl/cst/emitf.go → go/src/dsl/cst/emitf.go
View file

@ -9,11 +9,11 @@ import (
"errors"
"fmt"
"miller/dsl"
"miller/lib"
"miller/output"
"miller/runtime"
"miller/types"
"miller/src/dsl"
"miller/src/lib"
"miller/src/output"
"miller/src/runtime"
"miller/src/types"
)
// ================================================================

8
go/src/miller/dsl/cst/env.go → go/src/dsl/cst/env.go
View file

@ -10,10 +10,10 @@ package cst
import (
"os"
"miller/dsl"
"miller/lib"
"miller/runtime"
"miller/types"
"miller/src/dsl"
"miller/src/lib"
"miller/src/runtime"
"miller/src/types"
)
type EnvironmentVariableNode struct {

8
go/src/miller/dsl/cst/evaluable.go → go/src/dsl/cst/evaluable.go
View file

@ -11,10 +11,10 @@ import (
"fmt"
"os"
"miller/dsl"
"miller/lib"
"miller/runtime"
"miller/types"
"miller/src/dsl"
"miller/src/lib"
"miller/src/runtime"
"miller/src/types"
)
// ----------------------------------------------------------------

6
go/src/miller/dsl/cst/filter.go → go/src/dsl/cst/filter.go
View file

@ -5,9 +5,9 @@
package cst
import (
"miller/dsl"
"miller/lib"
"miller/runtime"
"miller/src/dsl"
"miller/src/lib"
"miller/src/runtime"
)
// ----------------------------------------------------------------

8
go/src/miller/dsl/cst/for.go → go/src/dsl/cst/for.go
View file

@ -7,10 +7,10 @@ package cst
import (
"errors"
"miller/dsl"
"miller/lib"
"miller/runtime"
"miller/types"
"miller/src/dsl"
"miller/src/lib"
"miller/src/runtime"
"miller/src/types"
)
// ----------------------------------------------------------------

4
go/src/miller/dsl/cst/functions.go → go/src/dsl/cst/functions.go
View file

@ -9,8 +9,8 @@
package cst
import (
"miller/dsl"
"miller/lib"
"miller/src/dsl"
"miller/src/lib"
)
// ----------------------------------------------------------------

8
go/src/miller/dsl/cst/if.go → go/src/dsl/cst/if.go
View file

@ -7,10 +7,10 @@ package cst
import (
"errors"
"miller/dsl"
"miller/lib"
"miller/runtime"
"miller/types"
"miller/src/dsl"
"miller/src/lib"
"miller/src/runtime"
"miller/src/types"
)
// ----------------------------------------------------------------

8
go/src/miller/dsl/cst/leaves.go → go/src/dsl/cst/leaves.go
View file

@ -8,10 +8,10 @@ import (
"errors"
"math"
"miller/dsl"
"miller/lib"
"miller/runtime"
"miller/types"
"miller/src/dsl"
"miller/src/lib"
"miller/src/runtime"
"miller/src/types"
)
// ----------------------------------------------------------------

8
go/src/miller/dsl/cst/lvalues.go → go/src/dsl/cst/lvalues.go
View file

@ -10,10 +10,10 @@ import (
"fmt"
"os"
"miller/dsl"
"miller/lib"
"miller/runtime"
"miller/types"
"miller/src/dsl"
"miller/src/lib"
"miller/src/runtime"
"miller/src/types"
)
// ----------------------------------------------------------------

10
go/src/miller/dsl/cst/print.go → go/src/dsl/cst/print.go
View file

@ -10,11 +10,11 @@ import (
"fmt"
"os"
"miller/dsl"
"miller/lib"
"miller/output"
"miller/runtime"
"miller/types"
"miller/src/dsl"
"miller/src/lib"
"miller/src/output"
"miller/src/runtime"
"miller/src/types"
)
// ----------------------------------------------------------------

12
go/src/miller/dsl/cst/root.go → go/src/dsl/cst/root.go
View file

@ -11,12 +11,12 @@ import (
"fmt"
"os"
"miller/cliutil"
"miller/dsl"
"miller/lib"
"miller/output"
"miller/runtime"
"miller/types"
"miller/src/cliutil"
"miller/src/dsl"
"miller/src/lib"
"miller/src/output"
"miller/src/runtime"
"miller/src/types"
)
// ----------------------------------------------------------------

2
go/src/miller/dsl/cst/signature.go → go/src/dsl/cst/signature.go
View file

@ -6,7 +6,7 @@
package cst
import (
"miller/types"
"miller/src/types"
)
// ----------------------------------------------------------------

2
go/src/miller/dsl/cst/statements.go → go/src/dsl/cst/statements.go
View file

@ -8,7 +8,7 @@ package cst
import (
"errors"
"miller/dsl"
"miller/src/dsl"
)
// ----------------------------------------------------------------

4
go/src/miller/dsl/cst/subroutines.go → go/src/dsl/cst/subroutines.go
View file

@ -9,8 +9,8 @@
package cst
import (
"miller/dsl"
"miller/lib"
"miller/src/dsl"
"miller/src/lib"
)
// ----------------------------------------------------------------

10
go/src/miller/dsl/cst/tee.go → go/src/dsl/cst/tee.go
View file

@ -8,11 +8,11 @@ import (
"errors"
"fmt"
"miller/dsl"
"miller/lib"
"miller/output"
"miller/runtime"
"miller/types"
"miller/src/dsl"
"miller/src/lib"
"miller/src/output"
"miller/src/runtime"
"miller/src/types"
)
// ----------------------------------------------------------------

8
go/src/miller/dsl/cst/types.go → go/src/dsl/cst/types.go
View file

@ -7,10 +7,10 @@ package cst
import (
"container/list"
"miller/cliutil"
"miller/dsl"
"miller/runtime"
"miller/types"
"miller/src/cliutil"
"miller/src/dsl"
"miller/src/runtime"
"miller/src/types"
)
// ----------------------------------------------------------------

8
go/src/miller/dsl/cst/udf.go → go/src/dsl/cst/udf.go
View file

@ -9,10 +9,10 @@ import (
"fmt"
"os"
"miller/dsl"
"miller/lib"
"miller/runtime"
"miller/types"
"miller/src/dsl"
"miller/src/lib"
"miller/src/runtime"
"miller/src/types"
)
// ----------------------------------------------------------------

8
go/src/miller/dsl/cst/uds.go → go/src/dsl/cst/uds.go
View file

@ -8,10 +8,10 @@ import (
"errors"
"fmt"
"miller/dsl"
"miller/lib"
"miller/runtime"
"miller/types"
"miller/src/dsl"
"miller/src/lib"
"miller/src/runtime"
"miller/src/types"
)
// ----------------------------------------------------------------

4
go/src/miller/dsl/cst/validate.go → go/src/dsl/cst/validate.go
View file

@ -10,8 +10,8 @@ import (
"errors"
"fmt"
"miller/dsl"
"miller/lib"
"miller/src/dsl"
"miller/src/lib"
)
// ----------------------------------------------------------------

6
go/src/miller/dsl/cst/while.go → go/src/dsl/cst/while.go
View file

@ -7,9 +7,9 @@ package cst
import (
"errors"
"miller/dsl"
"miller/lib"
"miller/runtime"
"miller/src/dsl"
"miller/src/lib"
"miller/src/runtime"
)
// ================================================================

1
go/src/github.com/goccmack/gocc/.errcheck-ignore
View file

@ -1 +0,0 @@
fmt.Fprintf

3
go/src/github.com/goccmack/gocc/.gitignore vendored
View file

@ -1,3 +0,0 @@
gocc
sm_*
.vscode

6
go/src/github.com/goccmack/gocc/.travis.yml
View file

@ -1,6 +0,0 @@
language: go
go:
- 1.11.x
- 1.12.x

14
go/src/github.com/goccmack/gocc/LICENSE
View file

@ -1,14 +0,0 @@
Copyright 2012 Vastech SA (PTY) LTD
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

197
go/src/github.com/goccmack/gocc/README.md
View file

@ -1,197 +0,0 @@
# New
Have a look at [https://github.com/goccmack/gogll](https://github.com/goccmack/gogll) for scannerless GLL parser generation.
# Gocc
[![Build Status](https://travis-ci.org/goccmack/gocc.svg?branch=master)](https://travis-ci.org/goccmack/gocc)
## Introduction
Gocc is a compiler kit for Go written in Go.
Gocc generates lexers and parsers or stand-alone DFAs or parsers from a BNF.
Lexers are DFAs, which recognise regular languages. Gocc lexers accept UTF-8 input.
Gocc parsers are PDAs, which recognise LR-1 languages. Optional LR1 conflict handling automatically resolves shift / reduce and reduce / reduce conflicts.
Generating a lexer and parser starts with creating a bnf file. Action expressions embedded in the BNF allows the user to specify semantic actions for syntax productions.
For complex applications the user typically uses an abstract syntax tree (AST) to represent the derivation of the input. The user provides a set of functions to construct the AST, which are called from the action expressions specified in the BNF.
See the [README](example/bools/README) for an included example.
[User Guide (PDF): Learn You a gocc for Great Good](https://raw.githubusercontent.com/goccmack/gocc/master/doc/gocc_user_guide.pdf) (gocc3 user guide will be published shortly)
## Installation
* First download and Install Go From http://golang.org/
* Setup your GOPATH environment variable.
* Next in your command line run: go get github.com/goccmack/gocc (go get will git clone gocc into GOPATH/src/github.com/goccmack/gocc and run go install)
* Alternatively clone the source: https://github.com/goccmack/gocc . Followed by go install github.com/goccmack/gocc
* Finally make sure that the bin folder where the gocc binary is located is in your PATH environment variable.
## Getting Started
Once installed start by creating your BNF in a package folder.
For example GOPATH/src/foo/bar.bnf:
```
/* Lexical Part */
id : 'a'-'z' {'a'-'z'} ;
!whitespace : ' ' | '\t' | '\n' | '\r' ;
/* Syntax Part */
<< import "foo/ast" >>
Hello: "hello" id << ast.NewWorld($1) >> ;
```
Next to use gocc, run:
```sh
cd $GOPATH/src/foo
gocc bar.bnf
```
This will generate a scanner, parser and token package inside GOPATH/src/foo Following times you might only want to run gocc without the scanner flag, since you might want to start making the scanner your own. Gocc is after all only a parser generator even if the default scanner is quite useful.
Next create ast.go file at $GOPATH/src/foo/ast with the following contents:
```go
package ast
import (
"foo/token"
)
type Attrib interface {}
type World struct {
Name string
}
func NewWorld(id Attrib) (*World, error) {
return &World{string(id.(*token.Token).Lit)}, nil
}
func (this *World) String() string {
return "hello " + this.Name
}
```
Finally we want to parse a string into the ast, so let us write a test at $GOPATH/src/foo/test/parse_test.go with the following contents:
```go
package test
import (
"foo/ast"
"foo/lexer"
"foo/parser"
"testing"
)
func TestWorld(t *testing.T) {
input := []byte(`hello gocc`)
lex := lexer.NewLexer(input)
p := parser.NewParser()
st, err := p.Parse(lex)
if err != nil {
panic(err)
}
w, ok := st.(*ast.World)
if !ok {
t.Fatalf("This is not a world")
}
if w.Name != `gocc` {
t.Fatalf("Wrong world %v", w.Name)
}
}
```
Finally run the test:
```sh
cd $GOPATH/src/foo/test
go test -v
```
You have now created your first grammar with gocc. This should now be relatively easy to change into the grammar you actually want to create or an existing LR1 grammar you would like to parse.
## BNF
The Gocc BNF is specified [here](spec/gocc2.ebnf)
An example bnf with action expressions can be found [here](example/bools/example.bnf)
## Action Expressions and AST
An action expression is specified as "<", "<", goccExpressionList , ">", ">" . The goccExpressionList is equivalent to a [goExpressionList](https://golang.org/ref/spec#ExpressionList). This expression list should return an Attrib and an error. Where Attrib is:
```go
type Attrib interface {}
```
Also parsed elements of the corresponding bnf rule can be represented in the expressionList as "$", digit.
Some action expression examples:
```
<< $0, nil >>
<< ast.NewFoo($1) >>
<< ast.NewBar($3, $1) >>
<< ast.TRUE, nil >>
```
Contants, functions, etc. that are returned or called should be programmed by the user in his ast (Abstract Syntax Tree) package. The ast package requires that you define your own Attrib interface as shown above. All parameters passed to functions will be of this type.
Some example of functions:
```go
func NewFoo(a Attrib) (*Foo, error) { ... }
func NewBar(a, b Attrib) (*Bar, error) { ... }
```
An example of an ast can be found [here](example/bools/ast/ast.go)
## Release Notes for gocc 2.1
### Changes
1. no_lexer option added to suppress generation of lexer. See the user guide.
2. Unreachable code removed from generated code.
### Bugs fixed:
1. gocc 2.1 does not support string_lit symbols with the same value as production names of the BNF. E.g. (t2.bnf):
```
A : "a" | "A" ;
```
string_lit "A" is not allowed.
Previously gocc silently ignored the conflicting string_lit. Now it generates an ugly panic:
```
$ gocc t2.bnf
panic: string_lit "A" conflicts with production name A
```
This issue will be properly resolved in a future release.
## Users
These projects use gocc:
* [gogo](https://github.com/shivansh/gogo) - [BNF file](https://github.com/shivansh/gogo/blob/master/src/lang.bnf) - a Go to MIPS compiler written in Go
* [gonum/gonum](https://github.com/gonum/gonum) - [BNF file](https://github.com/gonum/gonum/blob/master/graph/formats/dot/internal/dot.bnf) - DOT decoder (part of the graph library of Gonum)
* [llir/llvm](https://github.com/llir/llvm) - [BNF file](https://github.com/llir/llvm/blob/master/asm/internal/ll.bnf) - LLVM IR library in pure Go
* [mewmew/uc](https://github.com/mewmew/uc) - [BNF file](https://github.com/mewmew/uc/blob/master/gocc/uc.bnf) - A compiler for the µC language
* [gographviz](https://github.com/awalterschulze/gographviz) - [BNF file](https://github.com/awalterschulze/gographviz/blob/master/dot.bnf) - Parses the Graphviz DOT language in golang
* [katydid/relapse](http://katydid.github.io/) - [BNF file](https://github.com/katydid/katydid/blob/master/relapse/bnf/all.bnf) - Encoding agnostic validation language

4
go/src/github.com/goccmack/gocc/doc/.gitignore vendored
View file

@ -1,4 +0,0 @@
*.aux
*.log
*.out
*.toc

BIN
go/src/github.com/goccmack/gocc/doc/gocc.jpg
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 190 KiB

BIN
go/src/github.com/goccmack/gocc/doc/gocc_user_guide.pdf
View file

Binary file not shown.

1299
go/src/github.com/goccmack/gocc/doc/gocc_user_guide.tex
View file

File diff suppressed because it is too large Load diff

BIN
go/src/github.com/goccmack/gocc/doc/hl_design.pdf
View file

Binary file not shown.

1
go/src/github.com/goccmack/gocc/doc/makedoc.sh
View file

@ -1 +0,0 @@
pdflatex gocc_user_guide && pdflatex gocc_user_guide && pdflatex gocc_user_guide

0
go/src/github.com/goccmack/gocc/doc/target_grammar.tex
View file

1
go/src/github.com/goccmack/gocc/example/.gitignore vendored
View file

@ -1 +0,0 @@
sm_*

24
go/src/github.com/goccmack/gocc/example/astx/ast.bnf
View file

@ -1,24 +0,0 @@
/* Lexical elements */
_letter : 'a'-'z' | 'A'-'Z' ;
_digit : '0'-'9' ;
_idchar : _letter | _digit | '_' ;
id : (_letter | '_') {_idchar} ;
!whitespace : ' ' | '\t' | '\n' | '\r' ;
/* Syntax elements */
<< import "github.com/goccmack/gocc/example/astx/ast" >>
StmtList :
Stmt << ast.NewStmtList($0) >>
| StmtList Stmt << ast.AppendStmt($0, $1) >>
;
Stmt :
id << ast.NewStmt($0) >>
;

22
go/src/github.com/goccmack/gocc/example/astx/ast/ast.go
View file

@ -1,22 +0,0 @@
package ast
import (
"github.com/goccmack/gocc/example/astx/token"
)
type (
StmtList []Stmt
Stmt string
)
func NewStmtList(stmt interface{}) (StmtList, error) {
return StmtList{stmt.(Stmt)}, nil
}
func AppendStmt(stmtList, stmt interface{}) (StmtList, error) {
return append(stmtList.(StmtList), stmt.(Stmt)), nil
}
func NewStmt(stmtList interface{}) (Stmt, error) {
return Stmt(stmtList.(*token.Token).Lit), nil
}

38
go/src/github.com/goccmack/gocc/example/astx/ast_test.go
View file

@ -1,38 +0,0 @@
package astx
import (
"fmt"
"testing"
"github.com/goccmack/gocc/example/astx/ast"
"github.com/goccmack/gocc/example/astx/lexer"
"github.com/goccmack/gocc/example/astx/parser"
)
func TestPass(t *testing.T) {
sml, err := test([]byte("a b c d e f"))
if err != nil {
t.Fatal(err.Error())
}
fmt.Printf("output: %s\n", sml)
}
func TestFail(t *testing.T) {
_, err := test([]byte("a b ; d e f"))
if err == nil {
t.Fatal("expected parse error")
} else {
fmt.Printf("Parsing failed as expected: %v\n", err)
}
}
func test(src []byte) (astree ast.StmtList, err error) {
fmt.Printf("input: %s\n", src)
s := lexer.NewLexer(src)
p := parser.NewParser()
a, err := p.Parse(s)
if err == nil {
astree = a.(ast.StmtList)
}
return
}

1
go/src/github.com/goccmack/gocc/example/astx/doc.go
View file

@ -1 +0,0 @@
package astx

56
go/src/github.com/goccmack/gocc/example/astx/errors/errors.go
View file

@ -1,56 +0,0 @@
// Code generated by gocc; DO NOT EDIT.
package errors
import (
"fmt"
"strings"
"github.com/goccmack/gocc/example/astx/token"
)
type ErrorSymbol interface {
}
type Error struct {
Err error
ErrorToken *token.Token
ErrorSymbols []ErrorSymbol
ExpectedTokens []string
StackTop int
}
func (e *Error) String() string {
w := new(strings.Builder)
fmt.Fprintf(w, "Error")
if e.Err != nil {
fmt.Fprintf(w, " %s\n", e.Err)
} else {
fmt.Fprintf(w, "\n")
}
fmt.Fprintf(w, "Token: type=%d, lit=%s\n", e.ErrorToken.Type, e.ErrorToken.Lit)
fmt.Fprintf(w, "Pos: offset=%d, line=%d, column=%d\n", e.ErrorToken.Pos.Offset, e.ErrorToken.Pos.Line, e.ErrorToken.Pos.Column)
fmt.Fprintf(w, "Expected one of: ")
for _, sym := range e.ExpectedTokens {
fmt.Fprintf(w, "%s ", sym)
}
fmt.Fprintf(w, "ErrorSymbol:\n")
for _, sym := range e.ErrorSymbols {
fmt.Fprintf(w, "%v\n", sym)
}
return w.String()
}
func (e *Error) Error() string {
w := new(strings.Builder)
fmt.Fprintf(w, "Error in S%d: %s, %s", e.StackTop, token.TokMap.TokenString(e.ErrorToken), e.ErrorToken.Pos.String())
if e.Err != nil {
fmt.Fprintf(w, ": %+v", e.Err)
} else {
fmt.Fprintf(w, ", expected one of: ")
for _, expected := range e.ExpectedTokens {
fmt.Fprintf(w, "%s ", expected)
}
}
return w.String()
}

51
go/src/github.com/goccmack/gocc/example/astx/lexer/acttab.go
View file

@ -1,51 +0,0 @@
// Code generated by gocc; DO NOT EDIT.
package lexer
import (
"fmt"
"github.com/goccmack/gocc/example/astx/token"
)
type ActionTable [NumStates]ActionRow
type ActionRow struct {
Accept token.Type
Ignore string
}
func (a ActionRow) String() string {
return fmt.Sprintf("Accept=%d, Ignore=%s", a.Accept, a.Ignore)
}
var ActTab = ActionTable{
ActionRow{ // S0
Accept: 0,
Ignore: "",
},
ActionRow{ // S1
Accept: -1,
Ignore: "!whitespace",
},
ActionRow{ // S2
Accept: 2,
Ignore: "",
},
ActionRow{ // S3
Accept: 2,
Ignore: "",
},
ActionRow{ // S4
Accept: 2,
Ignore: "",
},
ActionRow{ // S5
Accept: 2,
Ignore: "",
},
ActionRow{ // S6
Accept: 2,
Ignore: "",
},
}

126
go/src/github.com/goccmack/gocc/example/astx/lexer/lexer.go
View file

@ -1,126 +0,0 @@
// Code generated by gocc; DO NOT EDIT.
package lexer
import (
"io/ioutil"
"unicode/utf8"
"github.com/goccmack/gocc/example/astx/token"
)
const (
NoState = -1
NumStates = 7
NumSymbols = 10
)
type Lexer struct {
src []byte
pos int
line int
column int
}
func NewLexer(src []byte) *Lexer {
lexer := &Lexer{
src: src,
pos: 0,
line: 1,
column: 1,
}
return lexer
}
func NewLexerFile(fpath string) (*Lexer, error) {
src, err := ioutil.ReadFile(fpath)
if err != nil {
return nil, err
}
return NewLexer(src), nil
}
func (l *Lexer) Scan() (tok *token.Token) {
tok = new(token.Token)
if l.pos >= len(l.src) {
tok.Type = token.EOF
tok.Pos.Offset, tok.Pos.Line, tok.Pos.Column = l.pos, l.line, l.column
return
}
start, startLine, startColumn, end := l.pos, l.line, l.column, 0
tok.Type = token.INVALID
state, rune1, size := 0, rune(-1), 0
for state != -1 {
if l.pos >= len(l.src) {
rune1 = -1
} else {
rune1, size = utf8.DecodeRune(l.src[l.pos:])
l.pos += size
}
nextState := -1
if rune1 != -1 {
nextState = TransTab[state](rune1)
}
state = nextState
if state != -1 {
switch rune1 {
case '\n':
l.line++
l.column = 1
case '\r':
l.column = 1
case '\t':
l.column += 4
default:
l.column++
}
switch {
case ActTab[state].Accept != -1:
tok.Type = ActTab[state].Accept
end = l.pos
case ActTab[state].Ignore != "":
start, startLine, startColumn = l.pos, l.line, l.column
state = 0
if start >= len(l.src) {
tok.Type = token.EOF
}
}
} else {
if tok.Type == token.INVALID {
end = l.pos
}
}
}
if end > start {
l.pos = end
tok.Lit = l.src[start:end]
} else {
tok.Lit = []byte{}
}
tok.Pos.Offset, tok.Pos.Line, tok.Pos.Column = start, startLine, startColumn
return
}
func (l *Lexer) Reset() {
l.pos = 0
}
/*
Lexer symbols:
0: '_'
1: '_'
2: ' '
3: '\t'
4: '\n'
5: '\r'
6: 'a'-'z'
7: 'A'-'Z'
8: '0'-'9'
9: .
*/

109
go/src/github.com/goccmack/gocc/example/astx/lexer/transitiontable.go
View file

@ -1,109 +0,0 @@
// Code generated by gocc; DO NOT EDIT.
package lexer
/*
Let s be the current state
Let r be the current input rune
transitionTable[s](r) returns the next state.
*/
type TransitionTable [NumStates]func(rune) int
var TransTab = TransitionTable{
// S0
func(r rune) int {
switch {
case r == 9: // ['\t','\t']
return 1
case r == 10: // ['\n','\n']
return 1
case r == 13: // ['\r','\r']
return 1
case r == 32: // [' ',' ']
return 1
case 65 <= r && r <= 90: // ['A','Z']
return 2
case r == 95: // ['_','_']
return 3
case 97 <= r && r <= 122: // ['a','z']
return 2
}
return NoState
},
// S1
func(r rune) int {
switch {
}
return NoState
},
// S2
func(r rune) int {
switch {
case 48 <= r && r <= 57: // ['0','9']
return 4
case 65 <= r && r <= 90: // ['A','Z']
return 5
case r == 95: // ['_','_']
return 6
case 97 <= r && r <= 122: // ['a','z']
return 5
}
return NoState
},
// S3
func(r rune) int {
switch {
case 48 <= r && r <= 57: // ['0','9']
return 4
case 65 <= r && r <= 90: // ['A','Z']
return 5
case r == 95: // ['_','_']
return 6
case 97 <= r && r <= 122: // ['a','z']
return 5
}
return NoState
},
// S4
func(r rune) int {
switch {
case 48 <= r && r <= 57: // ['0','9']
return 4
case 65 <= r && r <= 90: // ['A','Z']
return 5
case r == 95: // ['_','_']
return 6
case 97 <= r && r <= 122: // ['a','z']
return 5
}
return NoState
},
// S5
func(r rune) int {
switch {
case 48 <= r && r <= 57: // ['0','9']
return 4
case 65 <= r && r <= 90: // ['A','Z']
return 5
case r == 95: // ['_','_']
return 6
case 97 <= r && r <= 122: // ['a','z']
return 5
}
return NoState
},
// S6
func(r rune) int {
switch {
case 48 <= r && r <= 57: // ['0','9']
return 4
case 65 <= r && r <= 90: // ['A','Z']
return 5
case r == 95: // ['_','_']
return 6
case 97 <= r && r <= 122: // ['a','z']
return 5
}
return NoState
},
}

54
go/src/github.com/goccmack/gocc/example/astx/parser/actiontable.go
View file

@ -1,54 +0,0 @@
// Code generated by gocc; DO NOT EDIT.
package parser
type (
actionTable [numStates]actionRow
actionRow struct {
canRecover bool
actions [numSymbols]action
}
)
var actionTab = actionTable{
actionRow{ // S0
canRecover: false,
actions: [numSymbols]action{
nil, // INVALID
nil, // $
shift(3), // id
},
},
actionRow{ // S1
canRecover: false,
actions: [numSymbols]action{
nil, // INVALID
accept(true), // $
shift(3), // id
},
},
actionRow{ // S2
canRecover: false,
actions: [numSymbols]action{
nil, // INVALID
reduce(1), // $, reduce: StmtList
reduce(1), // id, reduce: StmtList
},
},
actionRow{ // S3
canRecover: false,
actions: [numSymbols]action{
nil, // INVALID
reduce(3), // $, reduce: Stmt
reduce(3), // id, reduce: Stmt
},
},
actionRow{ // S4
canRecover: false,
actions: [numSymbols]action{
nil, // INVALID
reduce(2), // $, reduce: StmtList
reduce(2), // id, reduce: StmtList
},
},
}

38
go/src/github.com/goccmack/gocc/example/astx/parser/gototable.go
View file

@ -1,38 +0,0 @@
// Code generated by gocc; DO NOT EDIT.
package parser
const numNTSymbols = 3
type (
gotoTable [numStates]gotoRow
gotoRow [numNTSymbols]int
)
var gotoTab = gotoTable{
gotoRow{ // S0
-1, // S'
1, // StmtList
2, // Stmt
},
gotoRow{ // S1
-1, // S'
-1, // StmtList
4, // Stmt
},
gotoRow{ // S2
-1, // S'
-1, // StmtList
-1, // Stmt
},
gotoRow{ // S3
-1, // S'
-1, // StmtList
-1, // Stmt
},
gotoRow{ // S4
-1, // S'
-1, // StmtList
-1, // Stmt
},
}

216
go/src/github.com/goccmack/gocc/example/astx/parser/parser.go
View file

@ -1,216 +0,0 @@
// Code generated by gocc; DO NOT EDIT.
package parser
import (
"fmt"
"strings"
parseError "github.com/goccmack/gocc/example/astx/errors"
"github.com/goccmack/gocc/example/astx/token"
)
const (
numProductions = 4
numStates = 5
numSymbols = 6
)
// Stack
type stack struct {
state []int
attrib []Attrib
}
const iNITIAL_STACK_SIZE = 100
func newStack() *stack {
return &stack{
state: make([]int, 0, iNITIAL_STACK_SIZE),
attrib: make([]Attrib, 0, iNITIAL_STACK_SIZE),
}
}
func (s *stack) reset() {
s.state = s.state[:0]
s.attrib = s.attrib[:0]
}
func (s *stack) push(state int, a Attrib) {
s.state = append(s.state, state)
s.attrib = append(s.attrib, a)
}
func (s *stack) top() int {
return s.state[len(s.state)-1]
}
func (s *stack) peek(pos int) int {
return s.state[pos]
}
func (s *stack) topIndex() int {
return len(s.state) - 1
}
func (s *stack) popN(items int) []Attrib {
lo, hi := len(s.state)-items, len(s.state)
attrib := s.attrib[lo:hi]
s.state = s.state[:lo]
s.attrib = s.attrib[:lo]
return attrib
}
func (s *stack) String() string {
w := new(strings.Builder)
fmt.Fprintf(w, "stack:\n")
for i, st := range s.state {
fmt.Fprintf(w, "\t%d: %d , ", i, st)
if s.attrib[i] == nil {
fmt.Fprintf(w, "nil")
} else {
switch attr := s.attrib[i].(type) {
case *token.Token:
fmt.Fprintf(w, "%s", attr.Lit)
default:
fmt.Fprintf(w, "%v", attr)
}
}
fmt.Fprintf(w, "\n")
}
return w.String()
}
// Parser
type Parser struct {
stack *stack
nextToken *token.Token
pos int
}
type Scanner interface {
Scan() (tok *token.Token)
}
func NewParser() *Parser {
p := &Parser{stack: newStack()}
p.Reset()
return p
}
func (p *Parser) Reset() {
p.stack.reset()
p.stack.push(0, nil)
}
func (p *Parser) Error(err error, scanner Scanner) (recovered bool, errorAttrib *parseError.Error) {
errorAttrib = &parseError.Error{
Err: err,
ErrorToken: p.nextToken,
ErrorSymbols: p.popNonRecoveryStates(),
ExpectedTokens: make([]string, 0, 8),
}
for t, action := range actionTab[p.stack.top()].actions {
if action != nil {
errorAttrib.ExpectedTokens = append(errorAttrib.ExpectedTokens, token.TokMap.Id(token.Type(t)))
}
}
if action := actionTab[p.stack.top()].actions[token.TokMap.Type("error")]; action != nil {
p.stack.push(int(action.(shift)), errorAttrib) // action can only be shift
} else {
return
}
if action := actionTab[p.stack.top()].actions[p.nextToken.Type]; action != nil {
recovered = true
}
for !recovered && p.nextToken.Type != token.EOF {
p.nextToken = scanner.Scan()
if action := actionTab[p.stack.top()].actions[p.nextToken.Type]; action != nil {
recovered = true
}
}
return
}
func (p *Parser) popNonRecoveryStates() (removedAttribs []parseError.ErrorSymbol) {
if rs, ok := p.firstRecoveryState(); ok {
errorSymbols := p.stack.popN(p.stack.topIndex() - rs)
removedAttribs = make([]parseError.ErrorSymbol, len(errorSymbols))
for i, e := range errorSymbols {
removedAttribs[i] = e
}
} else {
removedAttribs = []parseError.ErrorSymbol{}
}
return
}
// recoveryState points to the highest state on the stack, which can recover
func (p *Parser) firstRecoveryState() (recoveryState int, canRecover bool) {
recoveryState, canRecover = p.stack.topIndex(), actionTab[p.stack.top()].canRecover
for recoveryState > 0 && !canRecover {
recoveryState--
canRecover = actionTab[p.stack.peek(recoveryState)].canRecover
}
return
}
func (p *Parser) newError(err error) error {
e := &parseError.Error{
Err: err,
StackTop: p.stack.top(),
ErrorToken: p.nextToken,
}
actRow := actionTab[p.stack.top()]
for i, t := range actRow.actions {
if t != nil {
e.ExpectedTokens = append(e.ExpectedTokens, token.TokMap.Id(token.Type(i)))
}
}
return e
}
func (p *Parser) Parse(scanner Scanner) (res interface{}, err error) {
p.Reset()
p.nextToken = scanner.Scan()
for acc := false; !acc; {
action := actionTab[p.stack.top()].actions[p.nextToken.Type]
if action == nil {
if recovered, errAttrib := p.Error(nil, scanner); !recovered {
p.nextToken = errAttrib.ErrorToken
return nil, p.newError(nil)
}
if action = actionTab[p.stack.top()].actions[p.nextToken.Type]; action == nil {
panic("Error recovery led to invalid action")
}
}
switch act := action.(type) {
case accept:
res = p.stack.popN(1)[0]
acc = true
case shift:
p.stack.push(int(act), p.nextToken)
p.nextToken = scanner.Scan()
case reduce:
prod := productionsTable[int(act)]
attrib, err := prod.ReduceFunc(p.stack.popN(prod.NumSymbols))
if err != nil {
return nil, p.newError(err)
} else {
p.stack.push(gotoTab[p.stack.top()][prod.NTType], attrib)
}
default:
panic("unknown action: " + action.String())
}
}
return res, nil
}

63
go/src/github.com/goccmack/gocc/example/astx/parser/productionstable.go
View file

@ -1,63 +0,0 @@
// Code generated by gocc; DO NOT EDIT.
package parser
import "github.com/goccmack/gocc/example/astx/ast"
type (
//TODO: change type and variable names to be consistent with other tables
ProdTab [numProductions]ProdTabEntry
ProdTabEntry struct {
String string
Id string
NTType int
Index int
NumSymbols int
ReduceFunc func([]Attrib) (Attrib, error)
}
Attrib interface {
}
)
var productionsTable = ProdTab{
ProdTabEntry{
String: `S' : StmtList << >>`,
Id: "S'",
NTType: 0,
Index: 0,
NumSymbols: 1,
ReduceFunc: func(X []Attrib) (Attrib, error) {
return X[0], nil
},
},
ProdTabEntry{
String: `StmtList : Stmt << ast.NewStmtList(X[0]) >>`,
Id: "StmtList",
NTType: 1,
Index: 1,
NumSymbols: 1,
ReduceFunc: func(X []Attrib) (Attrib, error) {
return ast.NewStmtList(X[0])
},
},
ProdTabEntry{
String: `StmtList : StmtList Stmt << ast.AppendStmt(X[0], X[1]) >>`,
Id: "StmtList",
NTType: 1,
Index: 2,
NumSymbols: 2,
ReduceFunc: func(X []Attrib) (Attrib, error) {
return ast.AppendStmt(X[0], X[1])
},
},
ProdTabEntry{
String: `Stmt : id << ast.NewStmt(X[0]) >>`,
Id: "Stmt",
NTType: 2,
Index: 3,
NumSymbols: 1,
ReduceFunc: func(X []Attrib) (Attrib, error) {
return ast.NewStmt(X[0])
},
},
}

72
go/src/github.com/goccmack/gocc/example/astx/token/token.go
View file

@ -1,72 +0,0 @@
// Code generated by gocc; DO NOT EDIT.
package token
import (
"fmt"
)
type Token struct {
Type
Lit []byte
Pos
}
type Type int
const (
INVALID Type = iota
EOF
)
type Pos struct {
Offset int
Line int
Column int
}
func (p Pos) String() string {
return fmt.Sprintf("Pos(offset=%d, line=%d, column=%d)", p.Offset, p.Line, p.Column)
}
type TokenMap struct {
typeMap []string
idMap map[string]Type
}
func (m TokenMap) Id(tok Type) string {
if int(tok) < len(m.typeMap) {
return m.typeMap[tok]
}
return "unknown"
}
func (m TokenMap) Type(tok string) Type {
if typ, exist := m.idMap[tok]; exist {
return typ
}
return INVALID
}
func (m TokenMap) TokenString(tok *Token) string {
//TODO: refactor to print pos & token string properly
return fmt.Sprintf("%s(%d,%s)", m.Id(tok.Type), tok.Type, tok.Lit)
}
func (m TokenMap) StringType(typ Type) string {
return fmt.Sprintf("%s(%d)", m.Id(typ), typ)
}
var TokMap = TokenMap{
typeMap: []string{
"INVALID",
"$",
"id",
},
idMap: map[string]Type{
"INVALID": 0,
"$": 1,
"id": 2,
},
}

12
go/src/github.com/goccmack/gocc/example/bools/README
View file

@ -1,12 +0,0 @@
Generating a parser starts with creating a bnf file.
We have created an example.bnf file to illustrate this.
Running gen.sh will generate the parser, scanner and token code
needed to parse the language.
The ast (abstract syntax tree) package needs to created by the user.
This package should contain the corresponding function calls
required by the syntax directed translation rules.
See the example/ast package we have created.
Parsing a string with the generated code and evaluating the ast is
shown in example/example_test.go with some corresponding tests.

149
go/src/github.com/goccmack/gocc/example/bools/ast/ast.go
View file

@ -1,149 +0,0 @@
//Copyright 2012 Vastech SA (PTY) LTD
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package ast
import (
"strconv"
"strings"
"github.com/goccmack/gocc/example/bools/token"
"github.com/goccmack/gocc/example/bools/util"
)
type Attrib interface{}
type Val interface {
Attrib
Eval() bool
String() string
}
type Op int
func (o Op) String() string {
switch o {
case OR:
return "|"
case AND:
return "&"
}
return ""
}
const (
NOOP = Op(0)
OR = Op(1)
AND = Op(2)
)
type BoolExpr struct {
A Val
B Val
Op Op
}
func NewBoolAndExpr(a, b Attrib) (*BoolExpr, error) {
return &BoolExpr{a.(Val), b.(Val), AND}, nil
}
func NewBoolOrExpr(a, b Attrib) (*BoolExpr, error) {
return &BoolExpr{a.(Val), b.(Val), OR}, nil
}
func NewBoolGroupExpr(a Attrib) (*BoolExpr, error) {
return &BoolExpr{a.(Val), nil, NOOP}, nil
}
func (this *BoolExpr) Eval() bool {
switch this.Op {
case OR:
return this.A.Eval() || this.B.Eval()
case AND:
return this.A.Eval() && this.B.Eval()
}
return this.A.Eval()
}
func (this *BoolExpr) String() string {
return this.A.String() + " " + this.Op.String() + " " + this.B.String()
}
type val bool
func (v val) Eval() bool {
return bool(v)
}
func (v val) String() string {
if v {
return "true"
}
return "false"
}
var (
TRUE = val(true)
FALSE = val(false)
)
type LessThanExpr struct {
A int64
B int64
}
func NewLessThanExpr(a, b Attrib) (*LessThanExpr, error) {
aint, err := util.IntValue(a.(*token.Token).Lit)
if err != nil {
return nil, err
}
bint, err := util.IntValue(b.(*token.Token).Lit)
if err != nil {
return nil, err
}
return &LessThanExpr{aint, bint}, nil
}
func (this *LessThanExpr) Eval() bool {
return this.A < this.B
}
func (this *LessThanExpr) String() string {
return strconv.FormatInt(this.A, 10) + " < " + strconv.FormatInt(this.B, 10)
}
type SubStringExpr struct {
A string
B string
}
func NewSubStringExpr(a, b Attrib) (*SubStringExpr, error) {
astr, err := strconv.Unquote(string(a.(*token.Token).Lit))
if err != nil {
return nil, err
}
bstr, err := strconv.Unquote(string(b.(*token.Token).Lit))
if err != nil {
return nil, err
}
return &SubStringExpr{astr, bstr}, nil
}
func (this *SubStringExpr) Eval() bool {
return strings.Contains(this.B, this.A)
}
func (this *SubStringExpr) String() string {
return this.A + " in " + this.B
}

1
go/src/github.com/goccmack/gocc/example/bools/doc.go
View file

@ -1 +0,0 @@
package example

56
go/src/github.com/goccmack/gocc/example/bools/errors/errors.go
View file

@ -1,56 +0,0 @@
// Code generated by gocc; DO NOT EDIT.
package errors
import (
"fmt"
"strings"
"github.com/goccmack/gocc/example/bools/token"
)
type ErrorSymbol interface {
}
type Error struct {
Err error
ErrorToken *token.Token
ErrorSymbols []ErrorSymbol
ExpectedTokens []string
StackTop int
}
func (e *Error) String() string {
w := new(strings.Builder)
fmt.Fprintf(w, "Error")
if e.Err != nil {
fmt.Fprintf(w, " %s\n", e.Err)
} else {
fmt.Fprintf(w, "\n")
}
fmt.Fprintf(w, "Token: type=%d, lit=%s\n", e.ErrorToken.Type, e.ErrorToken.Lit)
fmt.Fprintf(w, "Pos: offset=%d, line=%d, column=%d\n", e.ErrorToken.Pos.Offset, e.ErrorToken.Pos.Line, e.ErrorToken.Pos.Column)
fmt.Fprintf(w, "Expected one of: ")
for _, sym := range e.ExpectedTokens {
fmt.Fprintf(w, "%s ", sym)
}
fmt.Fprintf(w, "ErrorSymbol:\n")
for _, sym := range e.ErrorSymbols {
fmt.Fprintf(w, "%v\n", sym)
}
return w.String()
}
func (e *Error) Error() string {
w := new(strings.Builder)
fmt.Fprintf(w, "Error in S%d: %s, %s", e.StackTop, token.TokMap.TokenString(e.ErrorToken), e.ErrorToken.Pos.String())
if e.Err != nil {
fmt.Fprintf(w, ": %+v", e.Err)
} else {
fmt.Fprintf(w, ", expected one of: ")
for _, expected := range e.ExpectedTokens {
fmt.Fprintf(w, "%s ", expected)
}
}
return w.String()
}

Some files were not shown because too many files have changed in this diff Show more