# Miller regression tests

There are a few files unit-tested with Go's `testing` package -- a few dozen cases total.

The vast majority of Miller tests, though -- thousands of cases -- are tested by running scripted invocations of `mlr` with various flags and inputs, comparing against expected output, and checking the exit code back to the shell.

## How to run the regression tests, in brief

*Note: while this `README.md` file is within the `go/reg-test/` subdirectory, all paths in this file are written from the perspective of the user being cd'ed into the `go/` directory, i.e. this directory's parent directory.*

* `rr` in the Miller `go/` subdirectory is an alias-like script for `reg-test/run`
* `reg-test/run --help` for help
* Without `-v`, this runs all cases with a pass/fail indication per case, and an overall pass/fail indication -- overall pass only if all cases pass. With `-v`, individual invocation lines are shown, as well as `diff` output comparing actual to expected results.

## Items for the duration of the Go port

* `rr -o` runs only cases where the output was generated by the C version of `mlr`: case-files starting with names `case-c-`.
* `rr -n` runs cases which were either written for the Go version, or which have been verified to work with it: case-files not starting with names `case-c-`. (As the port progresses, the `case-c-` and `case-go-` prefixes will be removed and replaced with simply `case-`.)
* `rr -c` runs the C version of Miller from the local checkout; `rr -g` (which is the default) runs the Go version of Miller from the local checkout.

## Examples

All new/ported cases:


```
$ reg-test/run -n
Using mlr executable ./reg-test/../../go/mlr
PASS  ./reg-test/cases/case-altkv.sh
PASS  ./reg-test/cases/case-bootstrap.sh
PASS  ./reg-test/cases/case-cat.sh
PASS  ./reg-test/cases/case-env.sh
PASS  ./reg-test/cases/case-go-chain.sh
...
PASS  ./reg-test/cases/case-sample.sh
PASS  ./reg-test/cases/case-shuffle.sh
PASS  ./reg-test/cases/case-subr.sh

NUMBER OF MILLER INVOCATIONS 807
NUMBER OF CASES PASSED 84
NUMBER OF CASES FAILED 0

PASS
```

All cases (as of November 2020 -- Go port in progress, not all C cases succeeding yet as many things are not ported):

```
$ reg-test/run
Using mlr executable ./reg-test/../../go/mlr
PASS  ./reg-test/cases/case-altkv.sh
PASS  ./reg-test/cases/case-bootstrap.sh
FAIL  ./reg-test/cases/case-c-auxents.sh
PASS  ./reg-test/cases/case-c-awkish-conds.sh
FAIL  ./reg-test/cases/case-c-bar.sh
...
PASS  ./reg-test/cases/case-go-skip-trivial-records.sh
PASS  ./reg-test/cases/case-go-sort-within-records.sh
PASS  ./reg-test/cases/case-go-sort.sh
PASS  ./reg-test/cases/case-go-tail.sh
PASS  ./reg-test/cases/case-go-unsparsify.sh
PASS  ./reg-test/cases/case-min-max-types.sh
PASS  ./reg-test/cases/case-no-filter-in-filter.sh
PASS  ./reg-test/cases/case-remove-empty-columns.sh
PASS  ./reg-test/cases/case-sample.sh
PASS  ./reg-test/cases/case-shuffle.sh
PASS  ./reg-test/cases/case-subr.sh

NUMBER OF MILLER INVOCATIONS 3767
NUMBER OF CASES PASSED 127
NUMBER OF CASES FAILED 91

FAIL
```

Single case, with verbosity:

```
$ reg-test/run -v reg-test/cases/case-cat.sh
Using mlr executable reg-test/../../go/mlr

----------------------------------------------------------------
BEGIN reg-test/cases/case-cat.sh

mlr cat reg-test/input/abixy
mlr cat /dev/null
mlr cat -n reg-test/input/abixy-het
mlr cat -N counter reg-test/input/abixy-het
mlr cat -g a,b reg-test/input/abixy-het
mlr cat -g a,b reg-test/input/abixy-het
mlr cat -g a,b -n reg-test/input/abixy-het
mlr cat -g a,b -N counter reg-test/input/abixy-het
mlr cat
mlr cat
mlr --opprint cat reg-test/input/s.dkvp
mlr --opprint cat -n reg-test/input/s.dkvp
mlr --opprint cat -n -g a reg-test/input/s.dkvp
mlr --opprint cat -n -g a,b reg-test/input/s.dkvp

num_invocations_attempted  14
num_invocations_passed     14
num_invocations_failed     0

diff -a -I ^mlr -I ^Miller: -I ^cat reg-test/expected/case-cat.sh.out output-reg-test/case-cat.sh.out

PASS  reg-test/cases/case-cat.sh
----------------------------------------------------------------

NUMBER OF MILLER INVOCATIONS 14
NUMBER OF CASES PASSED 1
NUMBER OF CASES FAILED 0

PASS
```


## More details

* `reg-test/cases/case*.sh` files consist of "invocations" of Miller.

* Each case `./reg-test/cases/case-foo.sh` has expected output `./reg-test/expected/case-foo.sh.out` and `./reg-test/expected/case-foo.sh.err`, along with actual outputs `output-reg-test/case-c-cat.sh.out` and `output-reg-test/case-c-cat.sh.err`.
* The `reg-test/run` script loops over all of the `case-*.sh` files and executes them via
  sourcing them with the Bash `.` operator.
* Each has lines of the form `run_mlr ...` or `mlr_expect_fail ...`. Those functions are defined in `reg_test/run`. They take `mlr` command-lines as arguments. They simply invoke `mlr` with the specified arguments, along with logic to check/count the shell exit codes, save off output for comparison to expected, etc.
* Each case-file can fail in the following ways:
  * Zero invocations were attempted.
  * A given `run_mlr ...` invocation exits with non-zero when it should exit with zero.
  * A given `mlr_expect_fail ...` invocation exits with zero when it should exit with non-zero.
  * The output of the invocations in the case's actual-output file differs from the case's expected-output file.

Example single case-file:

```
$ cat reg-test/cases/case-cat.sh
run_mlr cat $indir/abixy
run_mlr cat /dev/null

run_mlr cat -n $indir/abixy-het
run_mlr cat -N counter $indir/abixy-het

run_mlr cat -g a,b $indir/abixy-het
run_mlr cat -g a,b $indir/abixy-het

run_mlr cat -g a,b -n $indir/abixy-het
run_mlr cat -g a,b -N counter $indir/abixy-het

run_mlr cat <<EOF
a,b,c,d,e,f
EOF
run_mlr cat <<EOF
a,b,c,d,e,f,g
EOF

run_mlr --opprint cat           $indir/s.dkvp
run_mlr --opprint cat -n        $indir/s.dkvp
run_mlr --opprint cat -n -g a   $indir/s.dkvp
run_mlr --opprint cat -n -g a,b $indir/s.dkvp
```

## Debugging failures of existing cases

* If a case fails, you can run it by itself with `-v` if you like: e.g. `./reg-test/run -v reg-test/cases/case-cat.sh`.
* Also `-C 1` or `-C 5`, etc. (note the space) to control number of context lines in the diff output.
* You can also run `rr -s ...` to view the output without diffing against expected results.

Example with `-s`:

```
$ rr -s $cases/case-cat.sh
Using mlr executable ./reg-test/../../go/mlr


mlr cat ./reg-test/input/abixy
a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463
a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729
a=zee,b=pan,i=6,x=0.5271261600918548,y=0.49322128674835697
a=eks,b=zee,i=7,x=0.6117840605678454,y=0.1878849191181694
a=zee,b=wye,i=8,x=0.5985540091064224,y=0.976181385699006
a=hat,b=wye,i=9,x=0.03144187646093577,y=0.7495507603507059
a=pan,b=wye,i=10,x=0.5026260055412137,y=0.9526183602969864

mlr cat /dev/null

...

mlr --opprint cat -n -g a,b ./reg-test/input/s.dkvp
n a   b   i x                   y
1 pan pan 1 0.3467901443380824  0.7268028627434533
1 eks pan 2 0.7586799647899636  0.5221511083334797
1 wye wye 3 0.20460330576630303 0.33831852551664776
1 eks wye 4 0.38139939387114097 0.13418874328430463
```

## Creating new cases

* Edit `reg-test/cases/case-new-name-goes-here.sh`. Note that the `reg-test/cases` directory path, the filename starting with `case-`, and the filename ending with `.sh` are all required.
* Run `reg-test/run reg-test/cases/case-new-name-goes-here.sh`
* That will create `output-regtest/case-new-name-goes-here.sh.out`
* If this all looks OK, `accept-case case-new-name-goes-here.sh` which will copy actual output to `reg-test/expected/case-new-name-goes-here.sh.out`
* Add the `case*sh` and the expected-output file to source control.
