diff --git a/.gitignore b/.gitignore index c6985b12e..f2ef8ce28 100644 --- a/.gitignore +++ b/.gitignore @@ -17,3 +17,5 @@ data/nmc?.* docs/src/polyglot-dkvp-io/__pycache__ docs/site/ + +.cursor diff --git a/README-dev.md b/README-dev.md index 82a8e6aa0..452f55a8f 100644 --- a/README-dev.md +++ b/README-dev.md @@ -111,8 +111,8 @@ So, in broad overview, the key packages are: * [pkg/entrypoint](./pkg/entrypoint): All the usual contents of `main()` are here, for ease of testing. * [pkg/platform](./pkg/platform): Platform-dependent code, which as of early 2021 is the command-line parser. Handling single quotes and double quotes is different on Windows unless particular care is taken, which is what this package does. * [pkg/lib](./pkg/lib): - * Implementation of the [`Mlrval`](./pkg/types/mlrval.go) datatype which includes string/int/float/boolean/void/absent/error types. These are used for record values, as well as expression/variable values in the Miller `put`/`filter` DSL. See also below for more details. - * [`Mlrmap`](./pkg/types/mlrmap.go) is the sequence of key-value pairs which represents a Miller record. The key-lookup mechanism is optimized for Miller read/write usage patterns -- please see [mlrmap.go](./pkg/types/mlrmap.go) for more details. + * Implementation of the [`Mlrval`](./pkg/mlrval/) datatype which includes string/int/float/boolean/void/absent/error types. These are used for record values, as well as expression/variable values in the Miller `put`/`filter` DSL. See also below for more details. + * [`Mlrmap`](./pkg/mlrval/mlrmap.go) is the sequence of key-value pairs which represents a Miller record. The key-lookup mechanism is optimized for Miller read/write usage patterns -- please see [mlrmap.go](./pkg/mlrval/mlrmap.go) for more details. * [`context`](./pkg/types/context.go) supports AWK-like variables such as `FILENAME`, `NF`, `NR`, and so on. * [pkg/cli](./pkg/cli) is the flag-parsing logic for supporting Miller's command-line interface. When you type something like `mlr --icsv --ojson put '$sum = $a + $b' then filter '$sum > 1000' myfile.csv`, it's the CLI parser which makes it possible for Miller to construct a CSV record-reader, a transformer-chain of `put` then `filter`, and a JSON record-writer. * [pkg/climain](./pkg/climain) contains a layer which invokes `pkg/cli`, which was split out to avoid a Go package-import cycle. @@ -154,7 +154,7 @@ nil through the reader/transformer/writer sequence. ## More about mlrvals -[`Mlrval`](./pkg/types/mlrval.go) is the datatype of record values, as well as expression/variable values in the Miller `put`/`filter` DSL. It includes string/int/float/boolean/void/absent/error types, not unlike PHP's `zval`. +[`Mlrval`](./pkg/mlrval/) is the datatype of record values, as well as expression/variable values in the Miller `put`/`filter` DSL. It includes string/int/float/boolean/void/absent/error types, not unlike PHP's `zval`. * Miller's `absent` type is like Javascript's `undefined` -- it's for times when there is no such key, as in a DSL expression `$out = $foo` when the input record is `$x=3,y=4` -- there is no `$foo` so `$foo` has `absent` type. Nothing is written to the `$out` field in this case. See also [here](https://miller.readthedocs.io/en/latest/reference-main-null-data) for more information. * Miller's `void` type is like Javascript's `null` -- it's for times when there is a key with no value, as in `$out = $x` when the input record is `$x=,$y=4`. This is an overlap with `string` type, since a void value looks like an empty string. I've gone back and forth on this (including when I was writing the C implementation) -- whether to retain `void` as a distinct type from empty-string, or not. I ended up keeping it as it made the `Mlrval` logic easier to understand. @@ -162,7 +162,7 @@ nil through the reader/transformer/writer sequence. * Miller's number handling makes auto-overflow from int to float transparent, while preserving the possibility of 64-bit bitwise arithmetic. * This is different from JavaScript, which has only double-precision floats and thus no support for 64-bit numbers (note however that there is now [`BigInt`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/BigInt)). * This is also different from C and Go, wherein casts are necessary -- without which int arithmetic overflows. - * See also [here](https://miller.readthedocs.io/en/latest/reference-main-arithmetic) for the semantics of Miller arithmetic, which the [`Mlrval`](./pkg/types/mlrval.go) class implements. + * See also [here](https://miller.readthedocs.io/en/latest/reference-main-arithmetic) for the semantics of Miller arithmetic, which the [`Mlrval`](./pkg/mlrval/) package implements. ## Performance optimizations diff --git a/README-profiling.md b/README-profiling.md index 5c067614e..8279ded8b 100644 --- a/README-profiling.md +++ b/README-profiling.md @@ -9,7 +9,7 @@ Run the profiler: ``` -mlr --cpuprofile cpu.pprof --csv put -f scripts/chain-1.mlr ~/tmp/big.csv > /dev/null +mlr --cpuprofile cpu.pprof --csv put -f scripts/perf/chain-1.mlr ~/data/big.csv > /dev/null ``` (or whatever command-line flags for Miller). @@ -39,13 +39,12 @@ Note that you can drill into subcomponents of the flame graph: Scripts: -* [./scripts/make-big-files](./scripts/make-big-files) -- Create million-record data files in various formats. -* [./scripts/chain-cmps.sh](./scripts/chain-cmps.sh) -- Run a few processing scenarios on the million-record CSV file. - * [./scripts/chain-1.mlr](./scripts/chain-1.mlr) -- An example `mlr put` used by the previous script -* [./scripts/time-big-files](./scripts/time-big-files) -- Runs `mlr cat` for million-record files of various file formats. Catting files isn't intrinsically interesting but it shows how input and output processing vary over file formats. - * [./scripts/time-big-file](./scripts/time-big-file) -- Helper script for the former. -* [./scripts/chain-lengths.sh](./scripts/chain-lengths.sh) -- Run longer and longer chains of `scripts/chain1.mlr`, showing how Miller handles multicore and concurrency. -* [./scripts/make-data-stream](./scripts/make-data-stream) -- Create an endless stream of data to be piped into Miller for steady-state load-testing: e.g. `scripts/make-data-stream | mlr ...` then look at `htop` in another window. +* [./scripts/perf/prep-perf-data.sh](./scripts/perf/prep-perf-data.sh) -- Create million-record data files in various formats. +* [./scripts/perf/time-verbs.py](./scripts/perf/time-verbs.py) -- Run a few processing scenarios on the million-record CSV file. + * One batch of cases times various Miller verbs. + * A second batch of cases runs `mlr cat` for million-record files of various file formats. Catting files isn't intrinsically interesting but it shows how input and output processing vary over file formats. + * A third batch of cases runs longer and longer chains of `scripts/perf/chain-1.mlr`, showing how Miller handles multicore and concurrency. +* [./scripts/perf/make-data-stream](./scripts/perf/make-data-stream) -- Create an endless stream of data to be piped into Miller for steady-state load-testing: e.g. `scripts/perf/make-data-stream | mlr ...` then look at `htop` in another window. Notes: @@ -55,9 +54,9 @@ Notes: # How to vary compiler versions -* [./scripts/compiler-versions-install](./scripts/compiler-versions-install) -* [./scripts/compiler-versions-build](./scripts/compiler-versions-build) -* [./scripts/compiler-versions-time](./scripts/compiler-versions-time) +* [./scripts/perf/compiler-versions-install](./scripts/compiler-versions-install) +* [./scripts/perf/compiler-versions-build](./scripts/compiler-versions-build) +* [./scripts/perf/compiler-versions-time](./scripts/compiler-versions-time) # How to control garbage collection diff --git a/docs/src/new-in-miller-6.md b/docs/src/new-in-miller-6.md index 86a52a40d..93698d0a4 100644 --- a/docs/src/new-in-miller-6.md +++ b/docs/src/new-in-miller-6.md @@ -306,96 +306,4 @@ Please see the [section on emit statements](reference-dsl-output-statements.md#e ## Performance benchmarks -For performance testing, the [example.csv](https://github.com/johnkerl/miller/blob/main/docs/src/example.csv) file -[was expanded](https://github.com/johnkerl/miller/blob/main/scripts/make-big-files) into a million-line CSV file, -then converted to DKVP, JSON, etc. - -Notes: - -* These benchmarks were run on two laptops: a commodity Mac laptop with four CPUs, on MacOS Monterey, using `go1.16.5 darwin/amd64`, and a commodity Linux Lenovo with eight CPUs, on Ubuntu 21.10, using `go1.17.5 linux/amd64`. -* Interestingly, I noted a significant slowdown -- for this particular Linux laptop on low battery -- for the Go version but not the C version. Perhaps multicore interacts with power-saving mode. -* As of late 2021, Miller has been benchmarked using Go compiler versions 1.15.15, 1.16.12, 1.17.5, and 1.18beta1, with no significant performance changes attributable to compiler versions. - -For the [first benchmark](https://github.com/johnkerl/miller/blob/main/scripts/chain-cmps.sh), the format is CSV and the operations are varied: - -**Mac** - -| Operation | Miller 5 | Miller 6 | Speedup | -| --- | --- | --- | --- | -| CSV check | 1.541 | 1.216 | 1.27x | -| CSV cat | 2.403 | 1.430 | 1.68x | -| CSV tail | 1.526 | 1.222 | 1.25x | -| CSV tac | 2.785 | 3.122 | 0.89x | -| CSV sort -f shape | 2.996 | 3.139 | 0.95x | -| CSV sort -n quantity | 4.895 | 5.200 | 0.94x | -| CSV stats1 | 2.955 | 1.865 | 1.58x | -| CSV put expressions | 5.642 | 2.577 | 2.19x | - -**Linux** - -| Operation | Miller 5 | Miller 6 | Speedup | -| --- | --- | --- | --- | -| CSV check | 0.680 | 1.104 | 0.62x | -| CSV cat | 1.066 | 1.231 | 0.87x | -| CSV tail | 0.691 | 1.130 | 0.61x | -| CSV tac | 1.648 | 2.620 | 0.63x | -| CSV sort -f shape | 2.087 | 2.953 | 0.71x | -| CSV sort -n quantity | 5.588 | 5.337 | 1.05x | -| CSV stats1 | 2.376 | 1.751 | 1.36x | -| CSV put expressions | 4.520 | 2.091 | 2.16x | - -For the [second benchmark](https://github.com/johnkerl/miller/blob/main/scripts/time-big-files), we have `mlr cat` of those files, varying file types, with processing times shown. Catting out files as-is isn't a particularly useful operation in itself, but it gives an idea of how processing time depends on file format: - -**Mac** - -| Format | Miller 5 | Miller 6 | Speedup | -| --- | --- | --- | --- | -| CSV | 2.393 | 1.493 | 1.60x | -| CSV-lite | 1.644 | 1.351 | 1.22x | -| DKVP | 2.418 | 1.920 | 1.26x | -| NIDX | 1.053 | 0.958 | 1.10x | -| XTAB | 4.978 | 2.003 | 2.49x | -| JSON | 10.966 | 10.569 | 1.04x | - -**Linux** - -| Format | Miller 5 | Miller 6 | Speedup | -| --- | --- | --- | --- | -| CSV | 1.069 | 1.157 | 0.92x | -| CSV-lite | 0.640 | 1.187 | 0.54x | -| DKVP | 1.017 | 1.853 | 0.55x | -| NIDX | 0.623 | 1.398 | 0.45x | -| XTAB | 2.159 | 1.893 | 1.14x | -| JSON | 5.077 | 10.445 | 0.49x | - -For the [third benchmark](https://github.com/johnkerl/miller/blob/main/scripts/chain-lengths.sh), we have longer and longer then-chains: `mlr put ...`, then `mlr put ... then put ...`, etc. -- deepening the then-chain from one to six: - -**Mac** - -| Chain length | Miller 5 | Miller 6 | Speedup | -| --- | --- | --- | --- | -| 1 | 5.709 | 2.567 | 2.22x | -| 2 | 8.926 | 3.110 | 2.87x | -| 3 | 11.915 | 3.712 | 3.21x | -| 4 | 15.093 | 4.391 | 3.44x | -| 5 | 18.209 | 5.090 | 3.58x | -| 6 | 21.109 | 6.032 | 3.50x | - -**Linux** - -| Chain length | Miller 5 | Miller 6 | Speedup | -| --- | --- | --- | --- | -| 1 | 4.732 | 2.106 | 2.25x | -| 2 | 8.103 | 2.992 | 2.71x | -| 3 | 11.42 | 3.4743 | 3.29x | -| 4 | 14.904 | 3.859 | 3.86x | -| 5 | 18.128 | 4.1563 | 4.36x | -| 6 | 21.827 | 4.512 | 4.84x | - -Notes: - -* CSV processing is particularly improved in Miller 6. -* Record I/O is improved across the board, except that JSON continues to be a CPU-intensive format. Miller 6 JSON throughput is the same on Mac and Linux; Miller 5 did better than Miller 6 but only on Linux, not Mac. -* Miller 6's `sort` merits more performance analysis. -* Even single-verb processing with `put` and `stats1` is significantly faster on both platforms. -* Longer then-chains benefit even more from Miller 6's [multicore approach](cpu.md). +See the [performance page](performance.md). diff --git a/docs/src/new-in-miller-6.md.in b/docs/src/new-in-miller-6.md.in index 2da9d3feb..9a3dca63c 100644 --- a/docs/src/new-in-miller-6.md.in +++ b/docs/src/new-in-miller-6.md.in @@ -255,96 +255,4 @@ Please see the [section on emit statements](reference-dsl-output-statements.md#e ## Performance benchmarks -For performance testing, the [example.csv](https://github.com/johnkerl/miller/blob/main/docs/src/example.csv) file -[was expanded](https://github.com/johnkerl/miller/blob/main/scripts/make-big-files) into a million-line CSV file, -then converted to DKVP, JSON, etc. - -Notes: - -* These benchmarks were run on two laptops: a commodity Mac laptop with four CPUs, on MacOS Monterey, using `go1.16.5 darwin/amd64`, and a commodity Linux Lenovo with eight CPUs, on Ubuntu 21.10, using `go1.17.5 linux/amd64`. -* Interestingly, I noted a significant slowdown -- for this particular Linux laptop on low battery -- for the Go version but not the C version. Perhaps multicore interacts with power-saving mode. -* As of late 2021, Miller has been benchmarked using Go compiler versions 1.15.15, 1.16.12, 1.17.5, and 1.18beta1, with no significant performance changes attributable to compiler versions. - -For the [first benchmark](https://github.com/johnkerl/miller/blob/main/scripts/chain-cmps.sh), the format is CSV and the operations are varied: - -**Mac** - -| Operation | Miller 5 | Miller 6 | Speedup | -| --- | --- | --- | --- | -| CSV check | 1.541 | 1.216 | 1.27x | -| CSV cat | 2.403 | 1.430 | 1.68x | -| CSV tail | 1.526 | 1.222 | 1.25x | -| CSV tac | 2.785 | 3.122 | 0.89x | -| CSV sort -f shape | 2.996 | 3.139 | 0.95x | -| CSV sort -n quantity | 4.895 | 5.200 | 0.94x | -| CSV stats1 | 2.955 | 1.865 | 1.58x | -| CSV put expressions | 5.642 | 2.577 | 2.19x | - -**Linux** - -| Operation | Miller 5 | Miller 6 | Speedup | -| --- | --- | --- | --- | -| CSV check | 0.680 | 1.104 | 0.62x | -| CSV cat | 1.066 | 1.231 | 0.87x | -| CSV tail | 0.691 | 1.130 | 0.61x | -| CSV tac | 1.648 | 2.620 | 0.63x | -| CSV sort -f shape | 2.087 | 2.953 | 0.71x | -| CSV sort -n quantity | 5.588 | 5.337 | 1.05x | -| CSV stats1 | 2.376 | 1.751 | 1.36x | -| CSV put expressions | 4.520 | 2.091 | 2.16x | - -For the [second benchmark](https://github.com/johnkerl/miller/blob/main/scripts/time-big-files), we have `mlr cat` of those files, varying file types, with processing times shown. Catting out files as-is isn't a particularly useful operation in itself, but it gives an idea of how processing time depends on file format: - -**Mac** - -| Format | Miller 5 | Miller 6 | Speedup | -| --- | --- | --- | --- | -| CSV | 2.393 | 1.493 | 1.60x | -| CSV-lite | 1.644 | 1.351 | 1.22x | -| DKVP | 2.418 | 1.920 | 1.26x | -| NIDX | 1.053 | 0.958 | 1.10x | -| XTAB | 4.978 | 2.003 | 2.49x | -| JSON | 10.966 | 10.569 | 1.04x | - -**Linux** - -| Format | Miller 5 | Miller 6 | Speedup | -| --- | --- | --- | --- | -| CSV | 1.069 | 1.157 | 0.92x | -| CSV-lite | 0.640 | 1.187 | 0.54x | -| DKVP | 1.017 | 1.853 | 0.55x | -| NIDX | 0.623 | 1.398 | 0.45x | -| XTAB | 2.159 | 1.893 | 1.14x | -| JSON | 5.077 | 10.445 | 0.49x | - -For the [third benchmark](https://github.com/johnkerl/miller/blob/main/scripts/chain-lengths.sh), we have longer and longer then-chains: `mlr put ...`, then `mlr put ... then put ...`, etc. -- deepening the then-chain from one to six: - -**Mac** - -| Chain length | Miller 5 | Miller 6 | Speedup | -| --- | --- | --- | --- | -| 1 | 5.709 | 2.567 | 2.22x | -| 2 | 8.926 | 3.110 | 2.87x | -| 3 | 11.915 | 3.712 | 3.21x | -| 4 | 15.093 | 4.391 | 3.44x | -| 5 | 18.209 | 5.090 | 3.58x | -| 6 | 21.109 | 6.032 | 3.50x | - -**Linux** - -| Chain length | Miller 5 | Miller 6 | Speedup | -| --- | --- | --- | --- | -| 1 | 4.732 | 2.106 | 2.25x | -| 2 | 8.103 | 2.992 | 2.71x | -| 3 | 11.42 | 3.4743 | 3.29x | -| 4 | 14.904 | 3.859 | 3.86x | -| 5 | 18.128 | 4.1563 | 4.36x | -| 6 | 21.827 | 4.512 | 4.84x | - -Notes: - -* CSV processing is particularly improved in Miller 6. -* Record I/O is improved across the board, except that JSON continues to be a CPU-intensive format. Miller 6 JSON throughput is the same on Mac and Linux; Miller 5 did better than Miller 6 but only on Linux, not Mac. -* Miller 6's `sort` merits more performance analysis. -* Even single-verb processing with `put` and `stats1` is significantly faster on both platforms. -* Longer then-chains benefit even more from Miller 6's [multicore approach](cpu.md). +See the [performance page](performance.md). diff --git a/docs/src/perf/2026-02-22-cats.png b/docs/src/perf/2026-02-22-cats.png new file mode 100644 index 000000000..e3ca40b79 Binary files /dev/null and b/docs/src/perf/2026-02-22-cats.png differ diff --git a/docs/src/perf/2026-02-22-chains.png b/docs/src/perf/2026-02-22-chains.png new file mode 100644 index 000000000..7efe785aa Binary files /dev/null and b/docs/src/perf/2026-02-22-chains.png differ diff --git a/docs/src/perf/2026-02-22-verbs.png b/docs/src/perf/2026-02-22-verbs.png new file mode 100644 index 000000000..8c7ff4ff1 Binary files /dev/null and b/docs/src/perf/2026-02-22-verbs.png differ diff --git a/docs/src/performance.md b/docs/src/performance.md index 0449a6fbc..ed7e00b0b 100644 --- a/docs/src/performance.md +++ b/docs/src/performance.md @@ -16,8 +16,6 @@ Quick links: # Performance -See also the [performance-benchmarks section](new-in-miller-6.md#performance-benchmarks). - ## Disclaimer In a previous version of this page, I compared Miller to some items in the Unix toolkit in terms of run time. But such comparisons are very much not apples-to-apples: @@ -34,7 +32,38 @@ In a previous version of this page, I compared Miller to some items in the Unix Miller can do many kinds of processing on key-value-pair data using elapsed time roughly of the same order of magnitude as items in the Unix toolkit can handle positionally indexed data. Specific results vary widely by platform, implementation details, and multi-core use (or not). Lastly, specific special-purpose non-record-aware processing will run far faster if implemented in `grep`, `sed`, etc. -## Some examples +## Performance benchmarks + +For performance testing, the [example.csv](https://github.com/johnkerl/miller/blob/main/docs/src/example.csv) file +[was expanded](https://github.com/johnkerl/miller/blob/main/scripts/perf/prep-perf-data.sh) into a million-line CSV file, +then converted to DKVP, JSON, etc. + +Notes: + +* These benchmarks were run in early 2026 on MacBook Air laptop with an M1 processor. +* As of late 2021, Miller was benchmarked using Go compiler versions 1.15.15, 1.16.12, 1.17.5, and 1.18beta1, with no significant performance changes attributable to compiler versions. + +For the [first benchmark](https://github.com/johnkerl/miller/blob/main/scripts/perf/time-verbs.py), the format is CSV and the operations were varied: + +![perf/2026-02-22-verbs.png](perf/2026-02-22-verbs.png) + +For the [second benchmark](https://github.com/johnkerl/miller/blob/main/scripts/perf/time-verbs.py), we have `mlr cat` of those files, varying file types, with processing times shown. Catting out files as-is isn't a particularly useful operation in itself, but it gives an idea of how processing time depends on file format: + +![perf/2026-02-22-cats.png](perf/2026-02-22-cats.png) + +For the [third benchmark](https://github.com/johnkerl/miller/blob/main/scripts/perf/time-verbs.py), we have longer and longer then-chains: `mlr put ...`, then `mlr put ... then put ...`, etc. -- deepening the then-chain from one to four: + +![perf/2026-02-22-chains.png](perf/2026-02-22-chains.png) + +Notes: + +* CSV processing was particularly improved in Miller 6. +* Record I/O was improved across the board, except that JSON continues to be a CPU-intensive format. Miller 6 JSON throughput was the same on Mac and Linux; Miller 5 did better than Miller 6 but only on Linux, not Mac. +* Miller 6.0.0's initial issues with `sort` and JSON processing have been resolved. +* Even single-verb processing with `put` and `stats1` was significantly faster on both platforms. +* Longer then-chains benefit even more from Miller 6's [multicore approach](cpu.md). + +## Decompression options This is some data from [https://community.opencellid.org](https://community.opencellid.org): approximately 40 million records, 1.2GB compressed, 2.9GB uncompressed: diff --git a/docs/src/performance.md.in b/docs/src/performance.md.in index edc2fa143..aa4c7e99d 100644 --- a/docs/src/performance.md.in +++ b/docs/src/performance.md.in @@ -1,7 +1,5 @@ # Performance -See also the [performance-benchmarks section](new-in-miller-6.md#performance-benchmarks). - ## Disclaimer In a previous version of this page, I compared Miller to some items in the Unix toolkit in terms of run time. But such comparisons are very much not apples-to-apples: @@ -18,7 +16,38 @@ In a previous version of this page, I compared Miller to some items in the Unix Miller can do many kinds of processing on key-value-pair data using elapsed time roughly of the same order of magnitude as items in the Unix toolkit can handle positionally indexed data. Specific results vary widely by platform, implementation details, and multi-core use (or not). Lastly, specific special-purpose non-record-aware processing will run far faster if implemented in `grep`, `sed`, etc. -## Some examples +## Performance benchmarks + +For performance testing, the [example.csv](https://github.com/johnkerl/miller/blob/main/docs/src/example.csv) file +[was expanded](https://github.com/johnkerl/miller/blob/main/scripts/perf/prep-perf-data.sh) into a million-line CSV file, +then converted to DKVP, JSON, etc. + +Notes: + +* These benchmarks were run in early 2026 on MacBook Air laptop with an M1 processor. +* As of late 2021, Miller was benchmarked using Go compiler versions 1.15.15, 1.16.12, 1.17.5, and 1.18beta1, with no significant performance changes attributable to compiler versions. + +For the [first benchmark](https://github.com/johnkerl/miller/blob/main/scripts/perf/time-verbs.py), the format is CSV and the operations were varied: + +![perf/2026-02-22-verbs.png](perf/2026-02-22-verbs.png) + +For the [second benchmark](https://github.com/johnkerl/miller/blob/main/scripts/perf/time-verbs.py), we have `mlr cat` of those files, varying file types, with processing times shown. Catting out files as-is isn't a particularly useful operation in itself, but it gives an idea of how processing time depends on file format: + +![perf/2026-02-22-cats.png](perf/2026-02-22-cats.png) + +For the [third benchmark](https://github.com/johnkerl/miller/blob/main/scripts/perf/time-verbs.py), we have longer and longer then-chains: `mlr put ...`, then `mlr put ... then put ...`, etc. -- deepening the then-chain from one to four: + +![perf/2026-02-22-chains.png](perf/2026-02-22-chains.png) + +Notes: + +* CSV processing was particularly improved in Miller 6. +* Record I/O was improved across the board, except that JSON continues to be a CPU-intensive format. Miller 6 JSON throughput was the same on Mac and Linux; Miller 5 did better than Miller 6 but only on Linux, not Mac. +* Miller 6.0.0's initial issues with `sort` and JSON processing have been resolved. +* Even single-verb processing with `put` and `stats1` was significantly faster on both platforms. +* Longer then-chains benefit even more from Miller 6's [multicore approach](cpu.md). + +## Decompression options This is some data from [https://community.opencellid.org](https://community.opencellid.org): approximately 40 million records, 1.2GB compressed, 2.9GB uncompressed: diff --git a/pkg/dsl/cst/README.md b/pkg/dsl/cst/README.md index d1936faf9..08f815a1b 100644 --- a/pkg/dsl/cst/README.md +++ b/pkg/dsl/cst/README.md @@ -11,7 +11,7 @@ See [../dsl/README.md](../README.md) for more information about Miller's use of Go is a strongly typed language, but the AST is polymorphic. This results in if/else or switch statements as an AST is walked. -Also, when we modify code, there can be changes in the [BNF grammar](../../parsing/mlr.bnf) not yet reflected in the [AST](../../pkg/dsl/ast_types.go). Likewise, there can be AST changes not yet reflected here. (Example: you are partway through adding a new binary operator to the grammar.) +Also, when we modify code, there can be changes in the [BNF grammar](../../parsing/mlr.bnf) not yet reflected in the [AST](../ast_types.go). Likewise, there can be AST changes not yet reflected here. (Example: you are partway through adding a new binary operator to the grammar.) As a result, throughout the code, there are error checks which may seem redundant but which are in place to make incremental development more pleasant and robust. diff --git a/pkg/parsing/README.md b/pkg/parsing/README.md index 161e5c75d..bec012621 100644 --- a/pkg/parsing/README.md +++ b/pkg/parsing/README.md @@ -23,4 +23,4 @@ tools/build-dsl Making changes to `mlr.bnf` requires several minutes to re-run GOCC. For experimental changes, please see the -[experiments](../../../experiments/dsl-parser) directory. +[cmd/experiments/dsl_parser](../../../cmd/experiments/dsl_parser) directory. diff --git a/pkg/terminals/repl/README.md b/pkg/terminals/repl/README.md index 5fca6d415..1777a7767 100644 --- a/pkg/terminals/repl/README.md +++ b/pkg/terminals/repl/README.md @@ -106,6 +106,5 @@ One subtlety is that non-assignment expressions like `NR < 10` are filter statem * [entry.go](./entry.go) -- shell command-line entry point to the Miller repl command line. E.g. handles `mlr repl --json` which is typed at the shell prompt, and starts a command-line session at the Miller REPL prompt. * [session.go](./session.go) -- constructs a `Repl` object and ingests command lines, dispatching them either to the DSL (e.g. `$z = $x + $y`) or to the non-DSL verb handler (e.g. `:open foo.dat` or `help`). * [prompt.go](./prompt.go) -- Handling for default and customized banners/prompts for the Miller REPL. -* [dsl.go](./dsl.go) -- Handler for taking DSL statements typed in interactively by the user, parsing them to an AST, building a CST from the AST, and executing the CST. -* [ast.go](./ast.go) -- Interface between the REPL and the DSL-to-AST parser. +* [dsl.go](./dsl.go) -- Handler for taking DSL statements typed in interactively by the user, parsing them to an AST (via the [parsing](../../parsing/) package), building a CST from the AST, and executing the CST. * [verbs.go](./verbs.go) -- Handlers for non-DSL statements like `:open foo.dat` or `:help`. diff --git a/pkg/types/README.md b/pkg/types/README.md index 42eda57eb..1e9613a40 100644 --- a/pkg/types/README.md +++ b/pkg/types/README.md @@ -1,8 +1,8 @@ -This contains the implementation of the [`types.Mlrval`](./mlrval.go) datatype which is used for record values, as well as expression/variable values in the Miller `put`/`filter` DSL. +This package contains [`types.Context`](./context.go) and related stream types. The [`Mlrval`](../mlrval/) and [`Mlrmap`](../mlrval/mlrmap.go) types used for record values and expression/variable values in the Miller `put`/`filter` DSL are implemented in the [pkg/mlrval](../mlrval/) package. ## Mlrval -The [`types.Mlrval`](./mlrval.go) structure includes **string, int, float, boolean, array-of-mlrval, map-string-to-mlrval, void, absent, and error** types as well as type-conversion logic for various operators. +The [`Mlrval`](../mlrval/) structure (in package `mlrval`) includes **string, int, float, boolean, array-of-mlrval, map-string-to-mlrval, void, absent, and error** types as well as type-conversion logic for various operators. * Miller's `absent` type is like Javascript's `undefined` -- it's for times when there is no such key, as in a DSL expression `$out = $foo` when the input record is `$x=3,y=4` -- there is no `$foo` so `$foo` has `absent` type. Nothing is written to the `$out` field in this case. See also [here](https://miller.readthedocs.io/en/latest/reference-main-null-data) for more information. * Miller's `void` type is like Javascript's `null` -- it's for times when there is a key with no value, as in `$out = $x` when the input record is `$x=,$y=4`. This is an overlap with `string` type, since a void value looks like an empty string. I've gone back and forth on this (including when I was writing the C implementation) -- whether to retain `void` as a distinct type from empty-string, or not. I ended up keeping it as it made the `Mlrval` logic easier to understand. @@ -20,7 +20,7 @@ The [`types.Mlrval`](./mlrval.go) structure includes **string, int, float, boole # Mlrmap -[`types.Mlrmap`](./mlrmap.go) is the sequence of key-value pairs which represents a Miller record. The key-lookup mechanism is optimized for Miller read/write usage patterns -- please see `mlrmap.go` for more details. +[`Mlrmap`](../mlrval/mlrmap.go) is the sequence of key-value pairs which represents a Miller record (implemented in package `mlrval`). The key-lookup mechanism is optimized for Miller read/write usage patterns -- please see [mlrmap.go](../mlrval/mlrmap.go) for more details. It's also an ordered map structure, with string keys and Mlrval values. This is used within Mlrval itself. @@ -31,7 +31,7 @@ It's also an ordered map structure, with string keys and Mlrval values. This is # A note on JSON * The code for JSON I/O is mixed between `Mlrval` and `Mlrmap. This is unsurprising since JSON is a mutually recursive data structure -- arrays can contain maps and vice versa. -* JSON has non-collection types (string, int, float, etc) as well as collection types (array and object). Support for objects is principally in [./mlrmap_json.go](mlrmap_json.go); support for non-collection types as well as arrays is in [./mlrval_json.go](mlrval_json.go). +* JSON has non-collection types (string, int, float, etc) as well as collection types (array and object). Support for objects is principally in [mlrmap_json.go](../mlrval/mlrmap_json.go); support for non-collection types as well as arrays is in [mlrval_json.go](../mlrval/mlrval_json.go). * Both multi-line and single-line formats are supported. * Callsites for JSON output are record-writing (e.g. `--ojson`), the `dump` and `print` DSL routines, and the `json_stringify` DSL function. * The choice between single-line and multi-line for JSON record-writing is controlled by `--jvstack` and `--no-jvstack`, the former (multiline) being the default. diff --git a/scripts/c2m b/scripts/c2m deleted file mode 100755 index b54c2623c..000000000 --- a/scripts/c2m +++ /dev/null @@ -1,6 +0,0 @@ -for x in "$@"; do - echo $x - sed "s:'.*':-f \${CASEDIR}/mlr:" $x/cmd > $x/cmd.tmp - sed "s:[^']*'::" $x/cmd | sed "s:'.*::" | left > $x/mlr - mv $x/cmd.tmp $x/cmd -done diff --git a/scripts/chain-cmps.sh b/scripts/chain-cmps.sh deleted file mode 100755 index 186c9d5bb..000000000 --- a/scripts/chain-cmps.sh +++ /dev/null @@ -1,28 +0,0 @@ -mlrs="mlr" -if [ $# -ge 1 ]; then - mlrs="$@" -fi - -#reps="1" -reps="1 2 3" - -echo; for mlr in $mlrs; do for k in $reps; do justtime $mlr --csv --from ~/tmp/big.csv check > /dev/null; done; done -echo; for mlr in $mlrs; do for k in $reps; do justtime $mlr --csv --from ~/tmp/big.csv cat > /dev/null; done; done -echo; for mlr in $mlrs; do for k in $reps; do justtime $mlr --csv --from ~/tmp/big.csv tail > /dev/null; done; done -echo; for mlr in $mlrs; do for k in $reps; do justtime $mlr --csv --from ~/tmp/big.csv tac > /dev/null; done; done - -echo; for mlr in $mlrs; do for k in $reps; do - justtime $mlr --csv --from ~/tmp/big.csv sort -f shape > /dev/null; -done; done - -echo; for mlr in $mlrs; do for k in $reps; do - justtime $mlr --csv --from ~/tmp/big.csv sort -n quantity > /dev/null; -done; done - -echo; for mlr in $mlrs; do for k in $reps; do - justtime $mlr --c2p stats1 -a min,mean,max -f quantity,rate -g shape ~/tmp/big.csv > /dev/null -done; done - -echo; for mlr in $mlrs; do for k in $reps; do - justtime $mlr --csv --from ~/tmp/big.csv put -f scripts/chain-1.mlr > /dev/null -done; done diff --git a/scripts/chain-lengths.sh b/scripts/chain-lengths.sh deleted file mode 100755 index a584ed738..000000000 --- a/scripts/chain-lengths.sh +++ /dev/null @@ -1,71 +0,0 @@ -mlrs="mlr" -if [ $# -ge 1 ]; then - mlrs="$@" -fi - -#reps="1" -reps="1 2 3" -#reps="1 2 3 4 5 6 7 8 9 10" - -echo; for mlr in $mlrs; do - for k in $reps; do - justtime $mlr --csv --from ~/tmp/big.csv \ - then put -f scripts/chain-1.mlr \ - > /dev/null - done -done - -echo; for mlr in $mlrs; do - for k in $reps; do - justtime $mlr --csv --from ~/tmp/big.csv \ - then put -f scripts/chain-1.mlr \ - then put -f scripts/chain-1.mlr \ - > /dev/null - done -done - -echo; for mlr in $mlrs; do - for k in $reps; do - justtime $mlr --csv --from ~/tmp/big.csv \ - then put -f scripts/chain-1.mlr \ - then put -f scripts/chain-1.mlr \ - then put -f scripts/chain-1.mlr \ - > /dev/null - done -done - -echo; for mlr in $mlrs; do - for k in $reps; do - justtime $mlr --csv --from ~/tmp/big.csv \ - then put -f scripts/chain-1.mlr \ - then put -f scripts/chain-1.mlr \ - then put -f scripts/chain-1.mlr \ - then put -f scripts/chain-1.mlr \ - > /dev/null - done -done - -echo; for mlr in $mlrs; do - for k in $reps; do - justtime $mlr --csv --from ~/tmp/big.csv \ - then put -f scripts/chain-1.mlr \ - then put -f scripts/chain-1.mlr \ - then put -f scripts/chain-1.mlr \ - then put -f scripts/chain-1.mlr \ - then put -f scripts/chain-1.mlr \ - > /dev/null - done -done - -echo; for mlr in $mlrs; do - for k in $reps; do - justtime $mlr --csv --from ~/tmp/big.csv \ - then put -f scripts/chain-1.mlr \ - then put -f scripts/chain-1.mlr \ - then put -f scripts/chain-1.mlr \ - then put -f scripts/chain-1.mlr \ - then put -f scripts/chain-1.mlr \ - then put -f scripts/chain-1.mlr \ - > /dev/null - done -done diff --git a/scripts/compiler-versions-time b/scripts/compiler-versions-time deleted file mode 100755 index 03ed64965..000000000 --- a/scripts/compiler-versions-time +++ /dev/null @@ -1,9 +0,0 @@ -#!/bin/sh - -for mlr in mlr5 mlr-go1.1*; do justtime $mlr --csv check ~/tmp/big.csv > /dev/null; done -echo - -for mlr in mlr5 mlr-go1.1*; do justtime $mlr --csv cat ~/tmp/big.csv > /dev/null; done -echo - -for mlr in mlr5 mlr-go1.1*; do justtime $mlr --csv --from ~/tmp/big.csv put -f ./scripts/chain-1.mlr > /dev/null; done diff --git a/scripts/early-multi-language-timings/new-timings.sh b/scripts/early-multi-language-timings/new-timings.sh deleted file mode 100644 index ec9345a07..000000000 --- a/scripts/early-multi-language-timings/new-timings.sh +++ /dev/null @@ -1,9 +0,0 @@ -justtime ./catc < ~/tmp/big > /dev/null -justtime ./catc0 < ~/tmp/big > /dev/null -justtime ./catm < ~/tmp/big > /dev/null -justtime ./catrust < ~/tmp/big > /dev/null -justtime ./catrust2 < ~/tmp/big > /dev/null -justtime ./ff1 < ~/tmp/big > /dev/null -justtime ./ff2 < ~/tmp/big > /dev/null -justtime ./ff3 < ~/tmp/big > /dev/null -justtime ./catgo < ~/tmp/big > /dev/null diff --git a/scripts/make-big-files b/scripts/make-big-files deleted file mode 100755 index e23d2fcb3..000000000 --- a/scripts/make-big-files +++ /dev/null @@ -1,23 +0,0 @@ -#!/bin/bash - -set -x - -mkdir ~/tmp/ - -mlr --csv \ - repeat -n 100000 \ - then shuffle \ - then put ' - begin{@index=1} - $k = NR; - @index += urandint(2,10); - $index=@index; - $quantity=fmtnum(urandrange(50,100),"%.4f"); - $rate=fmtnum(urandrange(1,10),"%.4f"); - ' \ -docs/src/example.csv > ~/tmp/big.csv - -mlr --c2d cat ~/tmp/big.csv > ~/tmp/big.dkvp -mlr --c2j cat ~/tmp/big.csv > ~/tmp/big.json -mlr --c2n cat ~/tmp/big.csv > ~/tmp/big.nidx -mlr --c2x cat ~/tmp/big.csv > ~/tmp/big.xtab diff --git a/scripts/chain-1.mlr b/scripts/perf/chain-1.mlr similarity index 100% rename from scripts/chain-1.mlr rename to scripts/perf/chain-1.mlr diff --git a/scripts/compiler-versions-build b/scripts/perf/compiler-versions-build similarity index 100% rename from scripts/compiler-versions-build rename to scripts/perf/compiler-versions-build diff --git a/scripts/compiler-versions-install b/scripts/perf/compiler-versions-install similarity index 100% rename from scripts/compiler-versions-install rename to scripts/perf/compiler-versions-install diff --git a/scripts/perf/compiler-versions-time b/scripts/perf/compiler-versions-time new file mode 100755 index 000000000..026fc288c --- /dev/null +++ b/scripts/perf/compiler-versions-time @@ -0,0 +1,9 @@ +#!/bin/sh + +for mlr in mlr5 mlr-go1.1*; do justtime $mlr --csv check ~/data/big.csv > /dev/null; done +echo + +for mlr in mlr5 mlr-go1.1*; do justtime $mlr --csv cat ~/data/big.csv > /dev/null; done +echo + +for mlr in mlr5 mlr-go1.1*; do justtime $mlr --csv --from ~/data/big.csv put -f ./scripts/chain-1.mlr > /dev/null; done diff --git a/scripts/early-multi-language-timings/Makefile.no-autoconfig b/scripts/perf/early-multi-language-timings/Makefile.no-autoconfig similarity index 100% rename from scripts/early-multi-language-timings/Makefile.no-autoconfig rename to scripts/perf/early-multi-language-timings/Makefile.no-autoconfig diff --git a/scripts/early-multi-language-timings/catc.c b/scripts/perf/early-multi-language-timings/catc.c similarity index 100% rename from scripts/early-multi-language-timings/catc.c rename to scripts/perf/early-multi-language-timings/catc.c diff --git a/scripts/early-multi-language-timings/catc.c.txt b/scripts/perf/early-multi-language-timings/catc.c.txt similarity index 100% rename from scripts/early-multi-language-timings/catc.c.txt rename to scripts/perf/early-multi-language-timings/catc.c.txt diff --git a/scripts/early-multi-language-timings/catc0.c b/scripts/perf/early-multi-language-timings/catc0.c similarity index 100% rename from scripts/early-multi-language-timings/catc0.c rename to scripts/perf/early-multi-language-timings/catc0.c diff --git a/scripts/early-multi-language-timings/catc0.c.txt b/scripts/perf/early-multi-language-timings/catc0.c.txt similarity index 100% rename from scripts/early-multi-language-timings/catc0.c.txt rename to scripts/perf/early-multi-language-timings/catc0.c.txt diff --git a/scripts/early-multi-language-timings/catd.d b/scripts/perf/early-multi-language-timings/catd.d similarity index 100% rename from scripts/early-multi-language-timings/catd.d rename to scripts/perf/early-multi-language-timings/catd.d diff --git a/scripts/early-multi-language-timings/catd.d.txt b/scripts/perf/early-multi-language-timings/catd.d.txt similarity index 100% rename from scripts/early-multi-language-timings/catd.d.txt rename to scripts/perf/early-multi-language-timings/catd.d.txt diff --git a/scripts/early-multi-language-timings/catgo.go b/scripts/perf/early-multi-language-timings/catgo.go similarity index 100% rename from scripts/early-multi-language-timings/catgo.go rename to scripts/perf/early-multi-language-timings/catgo.go diff --git a/scripts/early-multi-language-timings/catgo.go.txt b/scripts/perf/early-multi-language-timings/catgo.go.txt similarity index 100% rename from scripts/early-multi-language-timings/catgo.go.txt rename to scripts/perf/early-multi-language-timings/catgo.go.txt diff --git a/scripts/early-multi-language-timings/catgo2.go b/scripts/perf/early-multi-language-timings/catgo2.go similarity index 100% rename from scripts/early-multi-language-timings/catgo2.go rename to scripts/perf/early-multi-language-timings/catgo2.go diff --git a/scripts/early-multi-language-timings/catgo2.go.txt b/scripts/perf/early-multi-language-timings/catgo2.go.txt similarity index 100% rename from scripts/early-multi-language-timings/catgo2.go.txt rename to scripts/perf/early-multi-language-timings/catgo2.go.txt diff --git a/scripts/early-multi-language-timings/catm.c b/scripts/perf/early-multi-language-timings/catm.c similarity index 100% rename from scripts/early-multi-language-timings/catm.c rename to scripts/perf/early-multi-language-timings/catm.c diff --git a/scripts/early-multi-language-timings/catm2.c b/scripts/perf/early-multi-language-timings/catm2.c similarity index 100% rename from scripts/early-multi-language-timings/catm2.c rename to scripts/perf/early-multi-language-timings/catm2.c diff --git a/scripts/early-multi-language-timings/catnim.nim b/scripts/perf/early-multi-language-timings/catnim.nim similarity index 100% rename from scripts/early-multi-language-timings/catnim.nim rename to scripts/perf/early-multi-language-timings/catnim.nim diff --git a/scripts/early-multi-language-timings/catrust.rs b/scripts/perf/early-multi-language-timings/catrust.rs similarity index 100% rename from scripts/early-multi-language-timings/catrust.rs rename to scripts/perf/early-multi-language-timings/catrust.rs diff --git a/scripts/early-multi-language-timings/catrust.rs.txt b/scripts/perf/early-multi-language-timings/catrust.rs.txt similarity index 100% rename from scripts/early-multi-language-timings/catrust.rs.txt rename to scripts/perf/early-multi-language-timings/catrust.rs.txt diff --git a/scripts/early-multi-language-timings/catrust2.rs b/scripts/perf/early-multi-language-timings/catrust2.rs similarity index 100% rename from scripts/early-multi-language-timings/catrust2.rs rename to scripts/perf/early-multi-language-timings/catrust2.rs diff --git a/scripts/early-multi-language-timings/cutd.d b/scripts/perf/early-multi-language-timings/cutd.d similarity index 100% rename from scripts/early-multi-language-timings/cutd.d rename to scripts/perf/early-multi-language-timings/cutd.d diff --git a/scripts/early-multi-language-timings/cutd.d.txt b/scripts/perf/early-multi-language-timings/cutd.d.txt similarity index 100% rename from scripts/early-multi-language-timings/cutd.d.txt rename to scripts/perf/early-multi-language-timings/cutd.d.txt diff --git a/scripts/early-multi-language-timings/cutgo.go b/scripts/perf/early-multi-language-timings/cutgo.go similarity index 100% rename from scripts/early-multi-language-timings/cutgo.go rename to scripts/perf/early-multi-language-timings/cutgo.go diff --git a/scripts/early-multi-language-timings/cutgo.go.txt b/scripts/perf/early-multi-language-timings/cutgo.go.txt similarity index 100% rename from scripts/early-multi-language-timings/cutgo.go.txt rename to scripts/perf/early-multi-language-timings/cutgo.go.txt diff --git a/scripts/early-multi-language-timings/cutnim.nim b/scripts/perf/early-multi-language-timings/cutnim.nim similarity index 100% rename from scripts/early-multi-language-timings/cutnim.nim rename to scripts/perf/early-multi-language-timings/cutnim.nim diff --git a/scripts/early-multi-language-timings/do-timings.rb b/scripts/perf/early-multi-language-timings/do-timings.rb similarity index 100% rename from scripts/early-multi-language-timings/do-timings.rb rename to scripts/perf/early-multi-language-timings/do-timings.rb diff --git a/scripts/early-multi-language-timings/ff1.rs b/scripts/perf/early-multi-language-timings/ff1.rs similarity index 89% rename from scripts/early-multi-language-timings/ff1.rs rename to scripts/perf/early-multi-language-timings/ff1.rs index cda208f0b..a78c18e4b 100644 --- a/scripts/early-multi-language-timings/ff1.rs +++ b/scripts/perf/early-multi-language-timings/ff1.rs @@ -20,5 +20,5 @@ fn read_lines_buf_reader(path: &str) -> io::Result<()> { } fn main() { - let _ = read_lines_buf_reader("/Users/kerl/tmp/big"); + let _ = read_lines_buf_reader("/Users/kerl/data/big"); } diff --git a/scripts/early-multi-language-timings/ff2.rs b/scripts/perf/early-multi-language-timings/ff2.rs similarity index 90% rename from scripts/early-multi-language-timings/ff2.rs rename to scripts/perf/early-multi-language-timings/ff2.rs index c83baecbc..87d010fff 100644 --- a/scripts/early-multi-language-timings/ff2.rs +++ b/scripts/perf/early-multi-language-timings/ff2.rs @@ -17,5 +17,5 @@ fn read_lines_manual(path: &str) -> io::Result<()> { } fn main() { - let _ = read_lines_manual("/Users/kerl/tmp/big"); + let _ = read_lines_manual("/Users/kerl/data/big"); } diff --git a/scripts/early-multi-language-timings/ff3.rs b/scripts/perf/early-multi-language-timings/ff3.rs similarity index 88% rename from scripts/early-multi-language-timings/ff3.rs rename to scripts/perf/early-multi-language-timings/ff3.rs index 835df7c40..2fecfe5da 100644 --- a/scripts/early-multi-language-timings/ff3.rs +++ b/scripts/perf/early-multi-language-timings/ff3.rs @@ -17,5 +17,5 @@ fn read_lines_custom_buffer(path: &str) -> io::Result<()> { } fn main() { - let _ = read_lines_custom_buffer("/Users/kerl/tmp/big"); + let _ = read_lines_custom_buffer("/Users/kerl/data/big"); } diff --git a/scripts/early-multi-language-timings/ff4.rs b/scripts/perf/early-multi-language-timings/ff4.rs similarity index 89% rename from scripts/early-multi-language-timings/ff4.rs rename to scripts/perf/early-multi-language-timings/ff4.rs index 4ee3224ee..418c13186 100644 --- a/scripts/early-multi-language-timings/ff4.rs +++ b/scripts/perf/early-multi-language-timings/ff4.rs @@ -20,5 +20,5 @@ fn read_lines_memmap(path: &str) -> io::Result<()> { } fn main() { - let _ = read_lines_custom_buffer("/Users/kerl/tmp/big"); + let _ = read_lines_custom_buffer("/Users/kerl/data/big"); } diff --git a/scripts/early-multi-language-timings/mand/mand.c b/scripts/perf/early-multi-language-timings/mand/mand.c similarity index 100% rename from scripts/early-multi-language-timings/mand/mand.c rename to scripts/perf/early-multi-language-timings/mand/mand.c diff --git a/scripts/early-multi-language-timings/mand/mand.mlr b/scripts/perf/early-multi-language-timings/mand/mand.mlr similarity index 100% rename from scripts/early-multi-language-timings/mand/mand.mlr rename to scripts/perf/early-multi-language-timings/mand/mand.mlr diff --git a/scripts/early-multi-language-timings/mand/mand.py b/scripts/perf/early-multi-language-timings/mand/mand.py similarity index 100% rename from scripts/early-multi-language-timings/mand/mand.py rename to scripts/perf/early-multi-language-timings/mand/mand.py diff --git a/scripts/early-multi-language-timings/mand/mand.rb b/scripts/perf/early-multi-language-timings/mand/mand.rb similarity index 100% rename from scripts/early-multi-language-timings/mand/mand.rb rename to scripts/perf/early-multi-language-timings/mand/mand.rb diff --git a/scripts/early-multi-language-timings/mand/mandg.go b/scripts/perf/early-multi-language-timings/mand/mandg.go similarity index 100% rename from scripts/early-multi-language-timings/mand/mandg.go rename to scripts/perf/early-multi-language-timings/mand/mandg.go diff --git a/scripts/early-multi-language-timings/mand/timings.txt b/scripts/perf/early-multi-language-timings/mand/timings.txt similarity index 100% rename from scripts/early-multi-language-timings/mand/timings.txt rename to scripts/perf/early-multi-language-timings/mand/timings.txt diff --git a/scripts/early-multi-language-timings/mkplots.sh b/scripts/perf/early-multi-language-timings/mkplots.sh similarity index 100% rename from scripts/early-multi-language-timings/mkplots.sh rename to scripts/perf/early-multi-language-timings/mkplots.sh diff --git a/scripts/perf/early-multi-language-timings/new-timings.sh b/scripts/perf/early-multi-language-timings/new-timings.sh new file mode 100644 index 000000000..f5b2d8479 --- /dev/null +++ b/scripts/perf/early-multi-language-timings/new-timings.sh @@ -0,0 +1,9 @@ +justtime ./catc < ~/data/big > /dev/null +justtime ./catc0 < ~/data/big > /dev/null +justtime ./catm < ~/data/big > /dev/null +justtime ./catrust < ~/data/big > /dev/null +justtime ./catrust2 < ~/data/big > /dev/null +justtime ./ff1 < ~/data/big > /dev/null +justtime ./ff2 < ~/data/big > /dev/null +justtime ./ff3 < ~/data/big > /dev/null +justtime ./catgo < ~/data/big > /dev/null diff --git a/scripts/early-multi-language-timings/new-timings.txt b/scripts/perf/early-multi-language-timings/new-timings.txt similarity index 86% rename from scripts/early-multi-language-timings/new-timings.txt rename to scripts/perf/early-multi-language-timings/new-timings.txt index bd095e95e..816444580 100644 --- a/scripts/early-multi-language-timings/new-timings.txt +++ b/scripts/perf/early-multi-language-timings/new-timings.txt @@ -20,5 +20,5 @@ TIME IN SECONDS 9.691 -- ./catc TIME IN SECONDS 12.975 -- ./catrust TIME IN SECONDS 52.282 -- ./catrust2 -$ justtime mlr cat ~/tmp/big > /dev/null -TIME IN SECONDS 79.106 -- mlr cat /Users/kerl/tmp/big +$ justtime mlr cat ~/data/big > /dev/null +TIME IN SECONDS 79.106 -- mlr cat /Users/kerl/data/big diff --git a/scripts/early-multi-language-timings/nimcat.nim.txt b/scripts/perf/early-multi-language-timings/nimcat.nim.txt similarity index 100% rename from scripts/early-multi-language-timings/nimcat.nim.txt rename to scripts/perf/early-multi-language-timings/nimcat.nim.txt diff --git a/scripts/early-multi-language-timings/nimcut.nim.txt b/scripts/perf/early-multi-language-timings/nimcut.nim.txt similarity index 100% rename from scripts/early-multi-language-timings/nimcut.nim.txt rename to scripts/perf/early-multi-language-timings/nimcut.nim.txt diff --git a/scripts/early-multi-language-timings/nimlc.nim b/scripts/perf/early-multi-language-timings/nimlc.nim similarity index 100% rename from scripts/early-multi-language-timings/nimlc.nim rename to scripts/perf/early-multi-language-timings/nimlc.nim diff --git a/scripts/early-multi-language-timings/nimwc.nim b/scripts/perf/early-multi-language-timings/nimwc.nim similarity index 100% rename from scripts/early-multi-language-timings/nimwc.nim rename to scripts/perf/early-multi-language-timings/nimwc.nim diff --git a/scripts/early-multi-language-timings/t1s.rb b/scripts/perf/early-multi-language-timings/t1s.rb similarity index 100% rename from scripts/early-multi-language-timings/t1s.rb rename to scripts/perf/early-multi-language-timings/t1s.rb diff --git a/scripts/early-multi-language-timings/timings.txt b/scripts/perf/early-multi-language-timings/timings.txt similarity index 100% rename from scripts/early-multi-language-timings/timings.txt rename to scripts/perf/early-multi-language-timings/timings.txt diff --git a/scripts/early-multi-language-timings/tn.rb b/scripts/perf/early-multi-language-timings/tn.rb similarity index 100% rename from scripts/early-multi-language-timings/tn.rb rename to scripts/perf/early-multi-language-timings/tn.rb diff --git a/scripts/make-data-stream b/scripts/perf/make-data-stream similarity index 98% rename from scripts/make-data-stream rename to scripts/perf/make-data-stream index 0e8c8485b..67a61d97f 100755 --- a/scripts/make-data-stream +++ b/scripts/perf/make-data-stream @@ -1,3 +1,5 @@ +#!/bin/bash + stop=1000000000 profile="" @@ -29,4 +31,3 @@ mlr \ then rename i,k \ then cut -xf index \ then filter '$rate > 2' - diff --git a/scripts/perf/prep-perf-data.sh b/scripts/perf/prep-perf-data.sh new file mode 100755 index 000000000..623517f36 --- /dev/null +++ b/scripts/perf/prep-perf-data.sh @@ -0,0 +1,33 @@ +#!/bin/bash + +# Invoke from Miller repo base. + +set -x + +indir=./test/input +outdir=~/data +mkdir -p $outdir + +mlr --csv \ + --from docs/src/example.csv \ + repeat -n 100000 \ + then shuffle \ + then put ' + begin{@index=1} + $k = NR; + @index += urandint(2,10); + $index=@index; + $quantity=fmtnum(urandrange(50,100),"%.4f"); + $rate=fmtnum(urandrange(1,10),"%.4f"); + ' \ +> $outdir/big.csv + +mlr --csv head -n 1000 $outdir/big.csv > $outdir/small.csv +mlr --csv head -n 10000 $outdir/big.csv > $outdir/medium.csv + +for kind in small medium big; do + mlr --c2d cat $outdir/$kind.csv > $outdir/$kind.dkvp + mlr --c2n cat $outdir/$kind.csv > $outdir/$kind.nidx + mlr --c2x cat $outdir/$kind.csv > $outdir/$kind.xtab + mlr --c2j cat $outdir/$kind.csv > $outdir/$kind.json +done diff --git a/scripts/perf/time-verbs.py b/scripts/perf/time-verbs.py new file mode 100755 index 000000000..0dc31e6f4 --- /dev/null +++ b/scripts/perf/time-verbs.py @@ -0,0 +1,77 @@ +#!/usr/bin/env python + +import os +import sys +import time +import subprocess + +# ================================================================ +def main(): + mlrs = [ "mlr" ] + # + kind = "big" + #kind = "medium" + #kind = "small" + in_csv = f"$HOME/data/{kind}.csv" + in_csvlite = f"$HOME/data/{kind}.csv" + in_dkvp = f"$HOME/data/{kind}.dkvp" + in_nidx = f"$HOME/data/{kind}.nidx" + in_xtab = f"$HOME/data/{kind}.xtab" + in_json = f"$HOME/data/{kind}.json" + # + cases = [ + ["check", f"--csv --from {in_csv} check"], + ["cat", f"--csv --from {in_csv} cat"], + ["tail", f"--csv --from {in_csv} tail"], + ["tac", f"--csv --from {in_csv} tac"], + ["sort-f", f"--csv --from {in_csv} sort -f shape"], + ["sort-n", f"--csv --from {in_csv} sort -n quantity"], + ["stats1", f"--csv --from {in_csv} stats1 -a min,mean,max -f quantity,rate -g shape"], + None, + ["chain-1", f"--csv --from {in_csv} put -f scripts/perf/chain-1.mlr"], + ["chain-2", f"--csv --from {in_csv} put -f scripts/perf/chain-1.mlr then put -f scripts/perf/chain-1.mlr"], + ["chain-3", f"--csv --from {in_csv} put -f scripts/perf/chain-1.mlr then put -f scripts/perf/chain-1.mlr then put -f scripts/perf/chain-1.mlr"], + ["chain-4", f"--csv --from {in_csv} put -f scripts/perf/chain-1.mlr then put -f scripts/perf/chain-1.mlr then put -f scripts/perf/chain-1.mlr then put -f scripts/perf/chain-1.mlr"], + None, + ["cat-csv", f"--csv --from {in_csv} cat"], + ["cat-csvlite", f"--csvlite --from {in_csvlite} cat"], + ["cat-dkvp", f"--dkvp --from {in_dkvp} cat"], + ["cat-nidx", f"--nidx --from {in_nidx} cat"], + ["cat-xtab", f"--xtab --from {in_xtab} cat"], + ["cat-json", f"--json --from {in_json} cat"], + ] + # + nreps = 5 + + if len(sys.argv) > 1: + mlrs = sys.argv[1:] + + + for case in cases: + if case is None: + print() + continue + for mlr in mlrs: + desc, args = case + avg = time_runs(desc, mlr, args, nreps) + print(f"desc={desc},version={mlr},seconds={avg:.3f}") + +# ---------------------------------------------------------------- +def time_runs(desc, mlr, args, nreps): + args = os.path.expandvars(args) + cmd = f"{mlr} {args}" + times = [time_run(mlr, args) for _ in range(nreps)] + avg = sum(times) / len(times) + return avg + +# ---------------------------------------------------------------- +def time_run(mlr, args): + cmd = f"{mlr} {args} > /dev/null" + start = time.perf_counter() + subprocess.run(cmd, shell=True) + elapsed = time.perf_counter() - start + return elapsed + +# ================================================================ +if __name__ == "__main__": + main() diff --git a/scripts/perf/timings-2026-02-22.dat b/scripts/perf/timings-2026-02-22.dat new file mode 100644 index 000000000..2a7b97af9 --- /dev/null +++ b/scripts/perf/timings-2026-02-22.dat @@ -0,0 +1,85 @@ +desc=check,version=mlr-5.10.3,seconds=0.839 +desc=check,version=mlr-6.0.0,seconds=0.726 +desc=check,version=mlr-6.8.0,seconds=0.674 +desc=check,version=mlr-6.16.0,seconds=0.814 +desc=check,version=mlr-6.17.0,seconds=0.761 +desc=cat,version=mlr-5.10.3,seconds=1.439 +desc=cat,version=mlr-6.0.0,seconds=0.812 +desc=cat,version=mlr-6.8.0,seconds=0.763 +desc=cat,version=mlr-6.16.0,seconds=0.864 +desc=cat,version=mlr-6.17.0,seconds=0.820 +desc=tail,version=mlr-5.10.3,seconds=0.904 +desc=tail,version=mlr-6.0.0,seconds=0.757 +desc=tail,version=mlr-6.8.0,seconds=0.694 +desc=tail,version=mlr-6.16.0,seconds=0.845 +desc=tail,version=mlr-6.17.0,seconds=0.782 +desc=tac,version=mlr-5.10.3,seconds=1.603 +desc=tac,version=mlr-6.0.0,seconds=1.807 +desc=tac,version=mlr-6.8.0,seconds=1.943 +desc=tac,version=mlr-6.16.0,seconds=1.997 +desc=tac,version=mlr-6.17.0,seconds=1.275 +desc=sort-f,version=mlr-5.10.3,seconds=1.719 +desc=sort-f,version=mlr-6.0.0,seconds=1.939 +desc=sort-f,version=mlr-6.8.0,seconds=1.768 +desc=sort-f,version=mlr-6.16.0,seconds=1.716 +desc=sort-f,version=mlr-6.17.0,seconds=1.374 +desc=sort-n,version=mlr-5.10.3,seconds=2.914 +desc=sort-n,version=mlr-6.0.0,seconds=5.992 +desc=sort-n,version=mlr-6.8.0,seconds=2.782 +desc=sort-n,version=mlr-6.16.0,seconds=2.781 +desc=sort-n,version=mlr-6.17.0,seconds=2.491 +desc=stats1,version=mlr-5.10.3,seconds=1.827 +desc=stats1,version=mlr-6.0.0,seconds=1.077 +desc=stats1,version=mlr-6.8.0,seconds=1.095 +desc=stats1,version=mlr-6.16.0,seconds=1.112 +desc=stats1,version=mlr-6.17.0,seconds=0.903 +desc=chain-1,version=mlr-5.10.3,seconds=3.782 +desc=chain-1,version=mlr-6.0.0,seconds=1.382 +desc=chain-1,version=mlr-6.8.0,seconds=1.413 +desc=chain-1,version=mlr-6.16.0,seconds=1.637 +desc=chain-1,version=mlr-6.17.0,seconds=1.603 +desc=chain-2,version=mlr-5.10.3,seconds=5.959 +desc=chain-2,version=mlr-6.0.0,seconds=1.658 +desc=chain-2,version=mlr-6.8.0,seconds=1.628 +desc=chain-2,version=mlr-6.16.0,seconds=1.771 +desc=chain-2,version=mlr-6.17.0,seconds=1.709 +desc=chain-3,version=mlr-5.10.3,seconds=7.869 +desc=chain-3,version=mlr-6.0.0,seconds=1.982 +desc=chain-3,version=mlr-6.8.0,seconds=1.915 +desc=chain-3,version=mlr-6.16.0,seconds=2.015 +desc=chain-3,version=mlr-6.17.0,seconds=1.926 +desc=chain-4,version=mlr-5.10.3,seconds=10.188 +desc=chain-4,version=mlr-6.0.0,seconds=2.236 +desc=chain-4,version=mlr-6.8.0,seconds=2.162 +desc=chain-4,version=mlr-6.16.0,seconds=2.277 +desc=chain-4,version=mlr-6.17.0,seconds=2.156 +desc=cat-csv,version=mlr-5.10.3,seconds=1.525 +desc=cat-csv,version=mlr-6.0.0,seconds=0.865 +desc=cat-csv,version=mlr-6.8.0,seconds=0.805 +desc=cat-csv,version=mlr-6.16.0,seconds=0.898 +desc=cat-csv,version=mlr-6.17.0,seconds=0.840 +desc=cat-csvlite,version=mlr-5.10.3,seconds=0.950 +desc=cat-csvlite,version=mlr-6.0.0,seconds=0.939 +desc=cat-csvlite,version=mlr-6.8.0,seconds=0.892 +desc=cat-csvlite,version=mlr-6.16.0,seconds=1.008 +desc=cat-csvlite,version=mlr-6.17.0,seconds=0.944 +desc=cat-dkvp,version=mlr-5.10.3,seconds=1.526 +desc=cat-dkvp,version=mlr-6.0.0,seconds=1.216 +desc=cat-dkvp,version=mlr-6.8.0,seconds=1.186 +desc=cat-dkvp,version=mlr-6.16.0,seconds=1.257 +desc=cat-dkvp,version=mlr-6.17.0,seconds=1.207 +desc=cat-nidx,version=mlr-5.10.3,seconds=0.915 +desc=cat-nidx,version=mlr-6.0.0,seconds=0.940 +desc=cat-nidx,version=mlr-6.8.0,seconds=2.675 +desc=cat-nidx,version=mlr-6.16.0,seconds=2.770 +desc=cat-nidx,version=mlr-6.17.0,seconds=3.177 +desc=cat-xtab,version=mlr-5.10.3,seconds=3.159 +desc=cat-xtab,version=mlr-6.0.0,seconds=1.125 +desc=cat-xtab,version=mlr-6.8.0,seconds=1.166 +desc=cat-xtab,version=mlr-6.16.0,seconds=1.228 +desc=cat-xtab,version=mlr-6.17.0,seconds=1.118 +desc=cat-json,version=mlr-5.10.3,seconds=6.319 +desc=cat-json,version=mlr-6.0.0,seconds=16.032 +desc=cat-json,version=mlr-6.8.0,seconds=9.723 +desc=cat-json,version=mlr-6.16.0,seconds=8.637 +desc=cat-json,version=mlr-6.17.0,seconds=8.663 diff --git a/scripts/perf/timings-2026-02-22.sh b/scripts/perf/timings-2026-02-22.sh new file mode 100644 index 000000000..b354238f7 --- /dev/null +++ b/scripts/perf/timings-2026-02-22.sh @@ -0,0 +1,16 @@ +datafile=timings-2026-02-22.dat + +mlr --d2p --from $datafile \ + grep cat then reshape -s desc,seconds \ + | sed '1s/^/#/' \ + | pgr -cat -flabels -lul -lp -ms 5 -o cats.png & + +mlr --d2p --from $datafile \ + grep chain then reshape -s desc,seconds \ + | sed '1s/^/#/' \ + | pgr -cat -flabels -lul -lp -ms 5 -o chains.png & + +mlr --d2p --from $datafile \ + grep -v cat then grep -v chain then reshape -s desc,seconds \ + | sed '1s/^/#/' \ + | pgr -cat -flabels -lul -lp -ms 5 -o verbs.png & diff --git a/scripts/time-big-file b/scripts/time-big-file deleted file mode 100755 index fef1e07ca..000000000 --- a/scripts/time-big-file +++ /dev/null @@ -1,18 +0,0 @@ -#!/bin/bash - -mlr="mlr" -suffix="dkvp" - -iflag="" -if [ $# -ge 1 ]; then - iflag="--$1" - if [ "$iflag" = "--csvlite" ]; then - suffix="csv" - else - suffix=$1 - fi -fi -if [ $# -eq 2 ]; then - mlr="$2" -fi -justtime $mlr $iflag cat ~/tmp/big.$suffix > /dev/null diff --git a/scripts/time-big-files b/scripts/time-big-files deleted file mode 100755 index 7f761c794..000000000 --- a/scripts/time-big-files +++ /dev/null @@ -1,20 +0,0 @@ -#!/bin/bash - -ourdir=$(dirname $0) - -mlrs="mlr" -if [ $# -ge 1 ]; then - mlrs="$@" -fi -#mlrs="mlr5 ./mlr" - -#reps="1" -reps="1 2 3" - -echo; for mlr in $mlrs; do for k in $reps; do $ourdir/time-big-file csv $mlr; done; done -echo; for mlr in $mlrs; do for k in $reps; do $ourdir/time-big-file csvlite $mlr; done; done -echo; for mlr in $mlrs; do for k in $reps; do $ourdir/time-big-file dkvp $mlr; done; done -echo; for mlr in $mlrs; do for k in $reps; do $ourdir/time-big-file nidx $mlr; done; done -echo; for mlr in $mlrs; do for k in $reps; do $ourdir/time-big-file xtab $mlr; done; done -echo; for mlr in $mlrs; do for k in $reps; do $ourdir/time-big-file json $mlr; done; done -