Format Go bits within mlr.bnf (#1955)

* step

* step

* step

* step

* step

* step

* step

* whitespace

* programmatic reformat

* BNF-formatter automation

* run tools/build-dsl

README-dev.md

@@ -120,7 +120,7 @@ So, in broad overview, the key packages are:
 * [pkg/input](./pkg/input) is as above -- one record-reader type per supported input file format, and a factory method.
 * [pkg/output](./pkg/output) is as above -- one record-writer type per supported output file format, and a factory method.
 * [pkg/transformers](./pkg/transformers) contains the abstract record-transformer interface datatype, as well as the Go-channel chaining mechanism for piping one transformer into the next. It also contains all the concrete record-transformers such as `cat`, `tac`, `sort`, `put`, and so on.
-* [pkg/parsing](./pkg/parsing) contains a single source file, `mlr.bnf`, which is the lexical/semantic grammar file for the Miller `put`/`filter` DSL using the GOCC framework. All subdirectories of `pkg/parsing/` are autogen code created by GOCC's processing of `mlr.bnf`. If you need to edit `mlr.bnf`, please use [tools/build-dsl](./tools/build-dsl) to autogenerate Go code from it (using the GOCC tool). (This takes several minutes to run.)
+* [pkg/parsing](./pkg/parsing) contains a single source file, `mlr.bnf`, which is the lexical/semantic grammar file for the Miller `put`/`filter` DSL using the GOCC framework. All subdirectories of `pkg/parsing/` are autogen code created by GOCC's processing of `mlr.bnf`. If you need to edit `mlr.bnf`, please use [tools/build-dsl](./tools/build-dsl) to autogenerate Go code from it (using the GOCC tool). (This takes several minutes to run.) See also [tools/format-go-in-bnf](./tools/format-go-in-bnf) (which reads `stdin` and writes `stdout`) for automated formatting of the Go bits.
 * [pkg/dsl](./pkg/dsl) contains [`ast_types.go`](pkg/dsl/ast_types.go) which is the abstract syntax tree datatype shared between GOCC and Miller. I didn't use a `pkg/dsl/ast` naming convention, although that would have been nice, in order to avoid a Go package-dependency cycle.
 * [pkg/dsl/cst](./pkg/dsl/cst) is the concrete syntax tree, constructed from an AST produced by GOCC. The CST is what is actually executed on every input record when you do things like `$z = $x * 0.3 * $y`. Please see the [pkg/dsl/cst/README.md](./pkg/dsl/cst/README.md) for more information.
 

tools/format-go-in-bnf

@@ -0,0 +1,65 @@
+#!/usr/bin/env python
+
+# ================================================================
+# Run pkg/parsing/mlr.bnf through this filter to reformat the Go code.
+# This only finds the Go code if the "<<" and the ">>" are on blank
+# lines before and after.
+# ================================================================
+
+import sys
+import re
+import subprocess
+
+in_dsl = False
+dsl_lines = []
+
+while True:
+    line = sys.stdin.readline()
+    if line == '':
+        break
+    line = line.rstrip()
+
+    if re.match("^ *<<$", line):
+        in_dsl = True
+        print(line)
+
+    elif re.match("^ *>>$", line):
+        in_dsl = False
+
+        in_block = "\n".join(dsl_lines)
+        in_block = re.sub(r'\$', 'DOLLAR_', in_block)
+
+        result = subprocess.run(
+            ['gofmt'],
+            input=in_block,
+            capture_output=True,
+            text=True,
+        )
+
+        if result.returncode == 0:
+            out_block = result.stdout
+            out_block = re.sub('DOLLAR_', '$', out_block)
+            out_block = re.sub('\t','  ', out_block)
+            out_lines = out_block.split("\n")
+            for out_line in out_lines:
+                out_line = re.sub(r"\s*$", "", out_line)
+                if out_line != "":
+                    print("    " + out_line)
+        else:
+            print()
+            print(result.stdout)
+            print(result.stderr)
+            print()
+            print("Exiting!")
+            print()
+            sys.exit(1)
+
+        print(line)
+        dsl_lines = []
+
+    elif in_dsl:
+        dsl_lines.append(line)
+
+    else:
+        #print("OUTSIDE")
+        print(line)