Replace GOCC parser-generator with PGPG (#2015)

* Porting

* Update some tests that depend on AST-print output

* iterating on GOCC -> PGPG

* iterating

* iterating

* iterating

* iterating

* iterating

* Modify expout files that need only AST-print-syntax updates

* iterating

* iterating

* iterating

* iterating

* iterating

* iterating

* iterating

* iterating

* iterating

* iterating

* iterating

* sync test cases from mlr-6.17.0, except AST-prints ...

* sync test cases from mlr-6.17.0, except AST-prints ...

* Fix lashed emit for $* and @*

* Fix --1 and ++1 chained unary ops

* Emit with function callsite

* test cases

* dot operator

* M_PI and M_E

* Iterating on lashed emit cases

* error-wording differences

* rm some should-fail files

* Fix issue with leading semicolon

* trailing comma in func params; most AST-print deltas

* error-wording delta

* fix unset all

* AST-print deltas

* go mod tidy: forced `go 1.25` to `go 1.25.0`

* Depend on PGPG v1.0.0

* Fix Windows CI failure

* Remove cmd/experiments/dsl_parser

* GOCC -> PGPG

* neaten

* Fix regex issue found in doc gen

tools/build-dsl

@@ -1,61 +1,59 @@
 #!/bin/bash
-
 # ================================================================
-# Reads the Miller DSL grammar file and generates Go code.
+# Generate lexer and parser from mlr.bnf using PGPG.
 #
-# This is not run on every build / commit / etc.
-#
-# It's intended to be run manually by the developer, as needed when mlr.bnf
-# changes for example.
-#
-# Resulting auto-generated .go files should then be checked into source control.
-#
-# With verbose, *.txt files are created with information about LR1 conflicts
-# etc.  Please don't commit them.
-#
-# As of mid-2021 this takes easily 5-10 minutes to run.
+# Run from the Miller repo root. Output goes to pkg/parsing/lexer/
+# and pkg/parsing/parser/.
 # ================================================================
 
 set -euo pipefail
 
-verbose="true"
-if [ $# -eq 1 ]; then
-  if [ "$1" == "-v" ]; then
-    verbose="true"
-  elif [ "$1" == "-q" ]; then
-    verbose="true"
-  fi
-fi
+cd "$(dirname "$0")/.."
+bnf="pkg/parsing/mlr.bnf"
 
-# Build the bin/gocc executable (use my fork for performance):
-go get     github.com/johnkerl/gocc
-go install github.com/johnkerl/gocc
-go mod tidy
-bingocc="$HOME/go/bin/gocc"
-if [ ! -x "$bingocc" ]; then
+if [ ! -f "$bnf" ]; then
+  echo "build-dsl: grammar file not found: $bnf"
   exit 1
 fi
 
-rm -f pkg/parsing/*.txt
-if [ "$verbose" = "true" ]; then
-  lr1="pkg/parsing/LR1_conflicts.txt"
-  $bingocc -v -o ./pkg/parsing -p mlr/pkg/parsing pkg/parsing/mlr.bnf || expand -2 $lr1
-else
-  $bingocc -o ./pkg/parsing -p mlr/pkg/parsing pkg/parsing/mlr.bnf
-fi
+# Intermediate JSON files (not checked in)
+mkdir -p pkg/parsing/gen
+lexjson="pkg/parsing/gen/lexer.json"
+parsejson="pkg/parsing/gen/parser.json"
 
-# Code-gen directories:
-#   pkg/parsing/errors/
-#   pkg/parsing/lexer/
-#   pkg/parsing/parser/
-#   pkg/parsing/token/
-#   pkg/parsing/util/
+echo "build-dsl: generating lexer tables..."
+go run github.com/johnkerl/pgpg/go/generators/cmd/lexgen-tables \
+  -o "$lexjson" \
+  "$bnf"
 
-# Override GOCC codegen with customized error handling
-cp pkg/parsing/errors.go.template pkg/parsing/errors/errors.go
+echo "build-dsl: generating lexer code..."
+go run github.com/johnkerl/pgpg/go/generators/cmd/lexgen-code \
+  -o pkg/parsing/lexer/lexer.go \
+  -package lexer \
+  -type MlrLexer \
+  "$lexjson"
 
-# We might need a manual replace of os.ReadFile by ioutil.ReadFile in autogen code.  Note we don't
-# use latest-and-greatest Go compiler version in our go.mod since we want to build on Centos which
-# can be trailing-edge in that regard.
+echo "build-dsl: generating parser tables..."
+go run github.com/johnkerl/pgpg/go/generators/cmd/parsegen-tables \
+  -o "$parsejson" \
+  "$bnf"
 
-for x in pkg/parsing/*/*.go; do gofmt -w $x; done
+echo "build-dsl: generating parser code..."
+go run github.com/johnkerl/pgpg/go/generators/cmd/parsegen-code \
+  -o pkg/parsing/parser/parser.go \
+  -package parser \
+  -type MlrParser \
+  "$parsejson"
+
+# Remove PGPPG-generated files that are no longer needed
+rm -f pkg/parsing/lexer/acttab.go \
+      pkg/parsing/lexer/transitiontable.go \
+      pkg/parsing/parser/action.go \
+      pkg/parsing/parser/actiontable.go \
+      pkg/parsing/parser/context.go \
+      pkg/parsing/parser/gototable.go \
+      pkg/parsing/parser/productionstable.go \
+  2>/dev/null || true
+
+gofmt -w pkg/parsing/lexer/lexer.go pkg/parsing/parser/parser.go
+echo "build-dsl: done."