diff --git a/docs/src/perf/timings-2026-06-20-cats.png b/docs/src/perf/timings-2026-06-20-cats.png new file mode 100644 index 000000000..660ba0411 Binary files /dev/null and b/docs/src/perf/timings-2026-06-20-cats.png differ diff --git a/docs/src/perf/timings-2026-06-20-chains.png b/docs/src/perf/timings-2026-06-20-chains.png new file mode 100644 index 000000000..3d719d3d0 Binary files /dev/null and b/docs/src/perf/timings-2026-06-20-chains.png differ diff --git a/docs/src/perf/timings-2026-06-20-verbs.png b/docs/src/perf/timings-2026-06-20-verbs.png new file mode 100644 index 000000000..32a0c9010 Binary files /dev/null and b/docs/src/perf/timings-2026-06-20-verbs.png differ diff --git a/docs/src/performance.md b/docs/src/performance.md index f8d8fc53c..2240afc10 100644 --- a/docs/src/performance.md +++ b/docs/src/performance.md @@ -83,6 +83,10 @@ The table below compares wall-clock time before and after these changes, using t | `put` UDF-heavy | 2.81s | 1.71s | -39% | | `put` map-returning UDF | 2.72s | 1.92s | -29% | +![perf/2026-06-20-verbs.png](perf/2026-02-22-verbs.png) +![perf/2026-06-20-cats.png](perf/2026-06-20-cats.png) +![perf/2026-06-20-chains.png](perf/2026-06-20-chains.png) + Peak resident set size (RSS) for representative workloads: | Workload | Before | After | Change | diff --git a/docs/src/performance.md.in b/docs/src/performance.md.in index a414dd7fc..349618072 100644 --- a/docs/src/performance.md.in +++ b/docs/src/performance.md.in @@ -67,6 +67,10 @@ The table below compares wall-clock time before and after these changes, using t | `put` UDF-heavy | 2.81s | 1.71s | -39% | | `put` map-returning UDF | 2.72s | 1.92s | -29% | +![perf/2026-06-20-verbs.png](perf/2026-02-22-verbs.png) +![perf/2026-06-20-cats.png](perf/2026-06-20-cats.png) +![perf/2026-06-20-chains.png](perf/2026-06-20-chains.png) + Peak resident set size (RSS) for representative workloads: | Workload | Before | After | Change | diff --git a/scripts/perf/README.md b/scripts/perf/README.md new file mode 100644 index 000000000..6288211ad --- /dev/null +++ b/scripts/perf/README.md @@ -0,0 +1,78 @@ +# Miller Performance Benchmarks + +Scripts for reproducing the performance graphs at +https://miller.readthedocs.io/en/latest/performance/ + +## Quick start + +``` +# Once — if ~/data/big.csv doesn't already exist (run from repo root): +bash scripts/perf/prep-perf-data.sh + +# Collect timings and render graphs (run from scripts/perf/): +cd scripts/perf +bash run-perf.sh ~/bin/mlr-6.18.1 ~/bin/mlr-6.19.0 +``` + +That runs `time-verbs.py` with both executables (5 reps each, ~18 cases), +saves `timings-YYYY-MM-DD.dat`, then calls `plot-timings.sh` to produce three +PNGs: `-verbs.png`, `-chains.png`, `-cats.png`. + +## Prerequisites + +- Python 3 +- [`pgr`](https://github.com/johnkerl/pgr) (used by `plot-timings.sh` to render PNG graphs) +- One or more Miller executables to compare + +## Step-by-step + +### Step 1 — Generate test data (one-time) + +Run from the Miller repo root: + +``` +bash scripts/perf/prep-perf-data.sh +``` + +This creates `~/data/big.csv` (~1 million rows) and derived files in DKVP, +NIDX, XTAB, and JSON formats, plus `small.csv` and `medium.csv` subsets. + +### Step 2 — Collect timings + +Run from `scripts/perf/`, passing one or more executables to compare: + +``` +python time-verbs.py ~/bin/mlr-6.18.1 ~/bin/mlr-6.19.0 \ + > timings-$(date +%Y-%m-%d).dat +``` + +Output is DKVP, one record per (case, executable): `desc=...,version=...,seconds=...` + +Five reps are averaged per case. Cases cover: + +- **Verbs**: `check`, `cat`, `tail`, `tac`, `sort -f`, `sort -n`, `stats1` +- **Then-chains**: one to four chained `put -f chain-1.mlr` steps +- **Formats**: `cat` across CSV, CSVLITE, DKVP, NIDX, XTAB, JSON + +### Step 3 — Plot + +``` +bash plot-timings.sh timings-YYYY-MM-DD.dat +``` + +Produces three PNGs alongside the dat file: +- `*-verbs.png` — verb timings +- `*-chains.png` — then-chain depth +- `*-cats.png` — format comparison + +## Files + +| File | Purpose | +|------|---------| +| `prep-perf-data.sh` | Generate `~/data/big.csv` and format variants (one-time) | +| `time-verbs.py` | Time each verb/format/chain case; outputs DKVP | +| `chain-1.mlr` | DSL script used for then-chain benchmarks | +| `run-perf.sh` | Wrapper: collect timings + plot | +| `plot-timings.sh` | Read a `.dat` file and render PNGs via `pgr` | +| `timings-2026-02-22.sh` | Original one-off plot script (hardcoded filename) | +| `timings-2026-02-22.dat` | Timing data from the February 2026 run | diff --git a/scripts/perf/plot-timings.sh b/scripts/perf/plot-timings.sh new file mode 100755 index 000000000..3f7f5aeaa --- /dev/null +++ b/scripts/perf/plot-timings.sh @@ -0,0 +1,30 @@ +#!/bin/bash +# Usage: plot-timings.sh timings.dat +# Requires: mlr, pgr +# Run from the Miller repo root. + +set -e + +datafile="${1:?Usage: $0 timings.dat}" +stem="${datafile%.dat}" + +mlr --d2p --from "$datafile" \ + grep cat then reshape -s desc,seconds \ + | sed '1s/^/#/' \ + | pgr -cat -ymin 0 -flabels -lur -lp -ms 5 -o "${stem}-cats.png" & + +mlr --d2p --from "$datafile" \ + grep chain then reshape -s desc,seconds \ + | sed '1s/^/#/' \ + | pgr -cat -ymin 0 -flabels -lur -lp -ms 5 -o "${stem}-chains.png" & + +mlr --d2p --from "$datafile" \ + grep -v cat then grep -v chain then reshape -s desc,seconds \ + | sed '1s/^/#/' \ + | pgr -cat -ymin 0 -flabels -lur -lp -ms 5 -o "${stem}-verbs.png" & + +wait +echo "Plots written:" +echo " ${stem}-cats.png" +echo " ${stem}-chains.png" +echo " ${stem}-verbs.png" diff --git a/scripts/perf/run-perf.sh b/scripts/perf/run-perf.sh new file mode 100755 index 000000000..bf25912d4 --- /dev/null +++ b/scripts/perf/run-perf.sh @@ -0,0 +1,22 @@ +#!/bin/bash +# Usage: run-perf.sh mlr-executable [mlr-executable ...] +# Run from scripts/perf/. + +set -e + +dir="$(cd "$(dirname "$0")" && pwd)" + +if [ $# -eq 0 ]; then + echo "Usage: $0 mlr-executable [mlr-executable ...]" >&2 + echo "Example: $0 ~/bin/mlr-6.18.1 ~/bin/mlr-6.19.0" >&2 + exit 1 +fi + +datfile="timings-$(date +%Y-%m-%d).dat" + +echo "Collecting timings into $datfile ..." +python "$dir/time-verbs.py" "$@" | tee "$datfile" + +echo "" +echo "Generating plots ..." +bash "$dir/plot-timings.sh" "$datfile" diff --git a/scripts/perf/time-verbs.py b/scripts/perf/time-verbs.py index 0dc31e6f4..5e4921c03 100755 --- a/scripts/perf/time-verbs.py +++ b/scripts/perf/time-verbs.py @@ -5,6 +5,9 @@ import sys import time import subprocess +script_dir = os.path.dirname(os.path.abspath(__file__)) +chain_mlr = os.path.join(script_dir, "chain-1.mlr") + # ================================================================ def main(): mlrs = [ "mlr" ] @@ -28,10 +31,10 @@ def main(): ["sort-n", f"--csv --from {in_csv} sort -n quantity"], ["stats1", f"--csv --from {in_csv} stats1 -a min,mean,max -f quantity,rate -g shape"], None, - ["chain-1", f"--csv --from {in_csv} put -f scripts/perf/chain-1.mlr"], - ["chain-2", f"--csv --from {in_csv} put -f scripts/perf/chain-1.mlr then put -f scripts/perf/chain-1.mlr"], - ["chain-3", f"--csv --from {in_csv} put -f scripts/perf/chain-1.mlr then put -f scripts/perf/chain-1.mlr then put -f scripts/perf/chain-1.mlr"], - ["chain-4", f"--csv --from {in_csv} put -f scripts/perf/chain-1.mlr then put -f scripts/perf/chain-1.mlr then put -f scripts/perf/chain-1.mlr then put -f scripts/perf/chain-1.mlr"], + ["chain-1", f"--csv --from {in_csv} put -f {chain_mlr}"], + ["chain-2", f"--csv --from {in_csv} put -f {chain_mlr} then put -f {chain_mlr}"], + ["chain-3", f"--csv --from {in_csv} put -f {chain_mlr} then put -f {chain_mlr} then put -f {chain_mlr}"], + ["chain-4", f"--csv --from {in_csv} put -f {chain_mlr} then put -f {chain_mlr} then put -f {chain_mlr} then put -f {chain_mlr}"], None, ["cat-csv", f"--csv --from {in_csv} cat"], ["cat-csvlite", f"--csvlite --from {in_csvlite} cat"], diff --git a/scripts/perf/timings-2026-06-20.dat b/scripts/perf/timings-2026-06-20.dat new file mode 100644 index 000000000..0d9b8ffef --- /dev/null +++ b/scripts/perf/timings-2026-06-20.dat @@ -0,0 +1,36 @@ +desc=check,version=mlr-6.18.1,seconds=0.614 +desc=check,version=mlr-6.19.0,seconds=0.217 +desc=cat,version=mlr-6.18.1,seconds=0.620 +desc=cat,version=mlr-6.19.0,seconds=0.223 +desc=tail,version=mlr-6.18.1,seconds=0.606 +desc=tail,version=mlr-6.19.0,seconds=0.210 +desc=tac,version=mlr-6.18.1,seconds=1.066 +desc=tac,version=mlr-6.19.0,seconds=0.463 +desc=sort-f,version=mlr-6.18.1,seconds=1.120 +desc=sort-f,version=mlr-6.19.0,seconds=0.521 +desc=sort-n,version=mlr-6.18.1,seconds=2.696 +desc=sort-n,version=mlr-6.19.0,seconds=2.008 +desc=stats1,version=mlr-6.18.1,seconds=0.678 +desc=stats1,version=mlr-6.19.0,seconds=0.549 + +desc=chain-1,version=mlr-6.18.1,seconds=1.052 +desc=chain-1,version=mlr-6.19.0,seconds=0.719 +desc=chain-2,version=mlr-6.18.1,seconds=1.096 +desc=chain-2,version=mlr-6.19.0,seconds=0.759 +desc=chain-3,version=mlr-6.18.1,seconds=1.188 +desc=chain-3,version=mlr-6.19.0,seconds=0.881 +desc=chain-4,version=mlr-6.18.1,seconds=1.357 +desc=chain-4,version=mlr-6.19.0,seconds=0.959 + +desc=cat-csv,version=mlr-6.18.1,seconds=0.638 +desc=cat-csv,version=mlr-6.19.0,seconds=0.233 +desc=cat-csvlite,version=mlr-6.18.1,seconds=0.703 +desc=cat-csvlite,version=mlr-6.19.0,seconds=0.289 +desc=cat-dkvp,version=mlr-6.18.1,seconds=0.949 +desc=cat-dkvp,version=mlr-6.19.0,seconds=0.462 +desc=cat-nidx,version=mlr-6.18.1,seconds=2.058 +desc=cat-nidx,version=mlr-6.19.0,seconds=1.559 +desc=cat-xtab,version=mlr-6.18.1,seconds=0.767 +desc=cat-xtab,version=mlr-6.19.0,seconds=1.305 +desc=cat-json,version=mlr-6.18.1,seconds=7.890 +desc=cat-json,version=mlr-6.19.0,seconds=7.512