diff --git a/plans/asof-joins.md b/plans/asof-joins.md new file mode 100644 index 000000000..08820cdbb --- /dev/null +++ b/plans/asof-joins.md @@ -0,0 +1,442 @@ +# Plan: asof joins (new verb `join-asof`) + +## Context + +[Issue #173](https://github.com/johnkerl/miller/issues/173) opens with a +request for arbitrary join conditions (`join -c '$start_time <= $time && +$end_time > $time'`), but the motivating use case — and every follow-up +comment on the thread — is specifically an **asof join**: for each record in +a stream (e.g. trades), attach fields from the reference-table record (e.g. +currency exchange rates) whose key is the closest one at-or-before the +stream record's key. The thread accumulates prior art as other tools grew +this exact feature: + +- R data.table: *rolling joins* (`roll = TRUE`) +- pandas: [`merge_asof`](https://pandas.pydata.org/docs/reference/api/pandas.merge_asof.html) + (with `direction=backward|forward|nearest`, `tolerance`, + `allow_exact_matches`, `by=` equality keys) +- DuckDB 0.8.0+: SQL `ASOF JOIN` +- ClickHouse: `ASOF JOIN` +- Polars: `join_asof` + +This plan proposes a dedicated verb, **`join-asof`**, with semantics modeled +on pandas `merge_asof` (the most fully-featured of the group), rather than +either (a) bolting asof flags onto the already-complex `join` verb +(`pkg/transformers/join.go` is 739 lines with two streaming modes whose +flag semantics — `-s`/`-u`, doubly-streaming — don't transfer cleanly to +asof matching), or (b) implementing the original generalized `-c` +DSL-condition join (see "Non-goals" below). + +The canonical example from the issue: + +``` +# rates.csv # trades.csv # desired output of +time,rate time # backward asof join +1,1.0 1 time rate +4,1.1 2 1 1.0 +5,1.5 3 2 1.0 +7,2.0 4 3 1.0 +9,1.8 5 4 1.1 +10,1.7 6 5 1.5 +11,1.9 7 6 1.5 + 8 7 2.0 + 9 8 2.0 + 10 9 1.8 + 10 1.7 +``` + +## Naming + +`join-asof` rather than `asof-join`: + +- Sorts adjacent to `join` in the alphabetized verb list (`mlr help verbs`, + the docs page, `aaa_transformer_table.go`), which aids discovery. +- Matches the operation-then-modifier order of polars `join_asof` and + pandas `merge_asof`. + +## Left/right terminology + +Follow `mlr join`'s existing convention: the `-f` file is the **left** +(lookup/reference) table, ingested into memory; the main record stream on +the Miller argument list is the **right** (probe) side. Each right record +pairs with **at most one** left record. + +Note this is the *opposite* of pandas' argument naming — in +`pd.merge_asof(trades, rates)` the probe side (trades) is "left" — but +consistency with `mlr join` (where `-f` = left, held in memory) matters +more here than consistency with pandas. Docs should include a sentence +calling this out for pandas users. + +For the canonical example: `rates.csv` goes in `-f`; `trades.csv` is the +main stream. + +## Proposed CLI signature + +``` +Usage: mlr join-asof [options] +Joins each record of the main input stream with the record from the left +file whose key is nearest, per the selected direction: the most recent +at-or-before key (backward, the default), the earliest at-or-after key +(forward), or the closest in either direction (nearest). This is the +"asof join" of pandas, polars, DuckDB, and ClickHouse, and the "rolling +join" of R data.table -- typically used to align timestamped data (e.g. +trades vs. sparse exchange-rate quotes). +``` + +Flags (OptionSpec table, following `joinOptions` in +`pkg/transformers/join.go`): + +| Flag | Arg | Type | Description | +|---|---|---|---| +| `-f` | `{left file name}` | filename | Left/lookup file name for the join. Required. | +| `-j` | `{name}` | string | Asof-key field name for output; also the left and right key field name unless overridden by `-l`/`-r`. Required. Exactly one field name (not a comma-separated list). | +| `-l` | `{name}` | string | Asof-key field name in the left file; defaults to the `-j` value. | +| `-r` | `{name}` | string | Asof-key field name in the right (main-stream) records; defaults to the `-j` value. | +| `-g` | `{a,b,c}` | csv-list | Optional comma-separated equality-match field names (pandas `by=`): records pair only when these fields match exactly, with the asof rule applied within each group. | +| `--lg` | `{a,b,c}` | csv-list | Equality-match field names in the left file; defaults to `-g` values. | +| `--rg` | `{a,b,c}` | csv-list | Equality-match field names in the right records; defaults to `-g` values. | +| `--dir` | `{d}` | string | Match direction: `backward` (left key <= right key; default), `forward` (left key >= right key), or `nearest` (smallest absolute distance; ties resolve backward). | +| `--strict` | | bool | Exclude exact key matches (pandas `allow_exact_matches=False`): backward becomes strictly-less-than, forward strictly-greater-than, nearest excludes equal keys. | +| `--tolerance` | `{n}` | string | Maximum allowed absolute difference between right and left keys; pairs farther apart than this are treated as unmatched. Requires numeric keys (see semantics below). | +| `--asof-key-field` | `{name}` | string | If supplied, also emit the *matched left record's* key value into paired output records under this field name. Useful with `nearest`/`tolerance` to see what actually matched. | +| `--lk`, `--left-keep-field-names` | `{a,b,c}` | csv-list | Keep only these fields from the left file (asof-key and `-g` fields automatically included), as in `join --lk`. | +| `--lp` | `{text}` | string | Prefix for non-key output field names from the left file, as in `join --lp`. | +| `--rp` | `{text}` | string | Prefix for non-key output field names from the right records, as in `join --rp`. | +| `--np` | | bool | Do not emit paired records. | +| `--ul` | | bool | Emit unpaired records from the left file (those never matched by any right record), after end of stream. | +| `--ur` | | bool | Emit unpaired records from the right stream (those with no qualifying left match, e.g. before the first quote, beyond tolerance, or missing key fields). | +| `--prepipe`, `--prepipex` | `{command}` | string | As in `join`: shell command to prepipe the left file through. | + +Plus the same left-file format-override passthrough as `join` (`-i`, +`--icsv`, `--ijson`, `--irs/--ifs/--ips`, `--implicit-csv-header`, etc.), +handled by the same `cli.FLAG_TABLE.Parse` fallthrough in the parse loop. + +Deliberate differences from `join`: + +- No `-s`/`-u` flags. There is only one mode (see "Streaming model"): the + left file is ingested and sorted in memory; the right stream is fully + streaming and **need not be sorted** (unlike pandas/polars, which require + both sides sorted). A doubly-streaming sorted mode is deferred (see + Non-goals). +- `-j`/`-l`/`-r` take a single field name, not a list — the asof key is one + field by construction. Multi-field equality keys go in `-g`. + +### Example invocations + +``` +# Canonical example from the issue: +mlr --csv join-asof -f rates.csv -j time trades.csv + +# Different key names on each side; keep only the rate column from the left: +mlr --csv join-asof -f rates.csv -l quote_time -r trade_time -j time --lk rate trades.csv + +# Multi-currency: exact match on currency, asof on time, within 60 seconds: +mlr --csv join-asof -f rates.csv -j time -g currency --tolerance 60 --ur trades.csv + +# Forward-looking match, excluding same-timestamp quotes: +mlr --csv join-asof -f rates.csv -j time --dir forward --strict trades.csv +``` + +## Semantics + +### Matching rule + +For each right record: + +1. Extract the `--rg` field values; if any is absent, the record is + right-unpaired. Look up the left-record bucket with equal `--lg` values + (whole-record-string equality on the joined group values, exactly as + `join` buckets by `GetSelectedValuesJoined`). No bucket → right-unpaired. +2. Extract the `-r` asof-key value; absent → right-unpaired. +3. Binary-search the bucket's key-sorted left records for the candidate + per `--dir`: + - `backward`: largest left key `<=` right key (`<` under `--strict`). + - `forward`: smallest left key `>=` right key (`>` under `--strict`). + - `nearest`: whichever of the backward and forward candidates has the + smaller absolute key distance; on an exact tie, the backward + (earlier) candidate wins. Under `--strict`, equal-key candidates are + excluded first. +4. If `--tolerance {n}` is given, reject the candidate when + `abs(rightkey - leftkey) > n`. Both keys must be numeric for the + difference to be computable; if either is not, the candidate is + rejected (record becomes right-unpaired). `n` itself must parse as a + non-negative number — CLI error otherwise. +5. No qualifying candidate → right-unpaired. Otherwise emit the merged + record (unless `--np`) and mark that specific left record as paired + (for `--ul` accounting). + +### Key ordering / comparison + +Keys are compared with Miller's natural-ordering comparator +(`mlrval.NaturalAscendingComparator`, `pkg/mlrval/mlrval_sort.go` — the +same collation as `sort -t`): numeric values compare numerically +(int/float mixed OK), strings compare lexically, and the ordering is a +total order across mixed types. This makes both integer/float epoch times +and ISO-8601 timestamp strings (which sort lexically = chronologically) +work without any type flags. `nearest`'s distance comparison and +`--tolerance` additionally need numeric subtraction, hence the numeric +requirement in step 4; `backward`/`forward` without tolerance work fine on +string keys. + +### Duplicate left keys + +If several left records in a bucket share the key selected by the +direction rule, the **last one in left-file input order** wins (pandas +behavior). Implementation: stable-sort the bucket by key, binary-search +for the rightmost record with the chosen backward key / leftmost with the +chosen forward key as appropriate — for backward the rightmost duplicate, +for forward also the *last* duplicate of the smallest qualifying key, per +pandas. Output is always at most one record per right record. + +### Output record layout + +Mirroring `join`'s `formAndEmitPairs`, a paired output record contains, in +order: + +1. The asof-key field, named by `-j`, carrying the **right** record's key + value (the probe time — matching pandas/DuckDB output and the issue's + desired output). +2. The `-g` equality fields under their `-g` output names (values equal on + both sides by construction). +3. If `--asof-key-field {name}` was given: the matched left record's key + value under `{name}`. +4. Left-record fields not already added (i.e. not the left key, not `--lg` + fields), each prefixed with `--lp` if given, filtered by `--lk` if + given. +5. Right-record fields not already added, each prefixed with `--rp`. + +Unpaired records (under `--ul`/`--ur`) get the same key-rename-to-output- +names and prefixing treatment as `join`'s `transformUnpairedRecord`, so +column names stay consistent for downstream `unsparsify` etc. + +For the canonical example this yields exactly `time,rate` per the issue. + +### Heterogeneity and edge cases + +- Left records missing the `-l` key or any `--lg` field: never matchable; + emitted only under `--ul`. +- Empty left file: every right record is unpaired. +- Left file need **not** be pre-sorted; it is sorted internally after + ingest. Right stream order is irrelevant to correctness (each record is + matched independently), and input order of right records is preserved + in the output. +- `--ul` output ordering: as in `join`'s half-streaming mode, left + unpairables are emitted at end of stream, in left-file order. +- All emit flags unset (`--np` without `--ul`/`--ur`): CLI error, same + message pattern as `join`. + +## Streaming model and performance + +Single mode, analogous to `join`'s default half-streaming mode: + +- Ingest the entire left file at construction/first-record time (reusing + `join`'s left-file reader logic), bucketing by `--lg` values. +- Per bucket: stable-sort records by key once (`sort.SliceStable`, + `O(L log L)`), storing a parallel slice of key mlrvals. +- Per right record: hash lookup of bucket + binary search (`sort.Search`), + `O(log L)`. The right stream is never buffered. + +Memory is proportional to the left file only — appropriate since the +lookup table (rates) is typically much smaller than the stream (trades). + +## Implementation sketch + +1. **New file `pkg/transformers/join_asof.go`** (~450 lines), containing: + - `verbNameJoinAsof = "join-asof"`, `joinAsofOptions []OptionSpec`, + `JoinAsofSetup TransformerSetup`, usage func, ParseCLI func — + structured exactly like `join.go`'s. + - `tJoinAsofOptions` struct (leftFileName, key/group field names, + direction enum, strict bool, tolerance *mlrval, prefixes, keep-set, + emit flags, prepipe, `joinFlagOptions cli.TOptions`). + - `TransformerJoinAsof` with lazy first-record ingest (same + `tr.ingested` pattern as `transformHalfStreaming`), bucket map + `map[string]*asofBucket` where + + ```go + type asofBucket struct { + recordsAndContexts []*types.RecordAndContext // stable-sorted by key + keys []*mlrval.Mlrval // parallel slice + paired []bool // per-record, for --ul + } + ``` + + - Match/merge logic per the semantics above. The merge loop is a + single-left-record specialization of `join.go`'s `formAndEmitPairs`. +2. **Shared left-file ingestion.** `join.go`'s `ingestLeftFile` + (`join.go:505-587`) is coupled to `TransformerJoin`'s bucketing. + Extract the reader-plumbing core (create reader, channels, drain to a + `[]*types.RecordAndContext`, including the error-channel-before-EOS + check) into a helper in `pkg/transformers/utils/` (e.g. + `utils.IngestFileToSlice(fileName string, readerOpts + *cli.TReaderOptions) ([]*types.RecordAndContext, error)`); both `join` + and `join-asof` then do their own bucketing over the slice. If the + refactor proves riskier than expected, fall back to a private copy in + `join_asof.go` and file a follow-up to unify. +3. **Registration**: add `JoinAsofSetup` to `STANDARD_TRANSFORMERS` in + `pkg/transformers/aaa_transformer_table.go` immediately after + `JoinSetup` (line 43) — the table is alphabetical. `mlr help verbs`, + `mlr --usage-all-verbs`, and manpage content are generated from this + table and the usage func, so they pick the verb up automatically. +4. **CLI validation** in ParseCLI: require `-f` and `-j`; reject + comma-containing `-j`/`-l`/`-r` values with a message pointing to `-g` + for equality keys; validate `--dir` ∈ {backward, forward, nearest}; + validate `--tolerance` parses as a non-negative number; require at + least one emit flag effective; same `-l`/`-r` (and `--lg`/`--rg`) + defaulting rules as `join`'s. + +## Test cases + +### Test inputs + +New files under `test/input/`: + +- `asof-rates.csv` — the issue's 7-row rates table. +- `asof-trades.csv` — the issue's 10-row trades table. +- `asof-rates-multi.csv` — rates with a `currency` column (e.g. CAD and + EUR interleaved, distinct gap structures per currency) for `-g` tests. +- `asof-trades-multi.csv` — trades with a `currency` column, including one + currency absent from the rates file. +- `asof-rates-dup.csv` — rates with a repeated `time` value (two quotes at + t=5) for the duplicate-key rule. +- `asof-rates-iso.csv` / `asof-trades-iso.csv` — ISO-8601 string + timestamps. +- `asof-rates.json` — JSON twin of `asof-rates.csv` for the mixed-format + case. + +### Regression cases (`test/cases/verb-join-asof/00NN/{cmd,expout,experr}`) + +Happy paths: + +1. **Backward default** (the issue's canonical example): + `mlr --icsv --opprint join-asof -f test/input/asof-rates.csv -j time + test/input/asof-trades.csv` — expout is exactly the issue's desired + table: + + ``` + time rate + 1 1.0 + 2 1.0 + 3 1.0 + 4 1.1 + 5 1.5 + 6 1.5 + 7 2.0 + 8 2.0 + 9 1.8 + 10 1.7 + ``` + +2. **Forward**: `--dir forward` — expected rates + `1.0, 1.1, 1.1, 1.1, 1.5, 2.0, 2.0, 1.8, 1.8, 1.7` for trades 1–10. +3. **Nearest with tie-break**: `--dir nearest` — trades 6 and 8 are + equidistant between quotes and must take the backward value + (`1.5, 2.0` respectively); full expected column + `1.0, 1.0, 1.1, 1.1, 1.5, 1.5, 2.0, 2.0, 1.8, 1.7`. +4. **Strict backward**: `--strict --ur` — trade 1 has no strictly-earlier + quote and comes out unpaired; trade 4 gets 1.0 (not 1.1), trade 5 gets + 1.1, etc. +5. **Tolerance**: `--tolerance 1 --ur` — trade 3 (distance 2 from quote + at t=1) is unpaired; all others pair. +6. **Group-by**: `-g currency` on the multi-currency inputs — asof applies + within each currency; trades in the currency absent from rates are + unpaired (verify both with and without `--ur`). +7. **Duplicate left keys**: rates-dup input, trade at t=5 and t=6 — both + must take the *last* t=5 quote in file order. +8. **String keys**: ISO-8601 inputs, backward join — verifies lexical + collation path. +9. **Key-name overrides**: `-j time -l quote_time -r trade_time` (rename + variants of the inputs, or reuse via `rename` in the cmd pipeline) — + output key column is named `time`. +10. **Prefixes and collisions**: both sides carrying a same-named non-key + field, with `--lp l_ --rp r_`, and again with no prefixes (last-put + wins per `join` semantics — pin whichever behavior `formAndEmitPairs` + ordering produces). +11. **`--lk`**: keep only a subset of left fields; asof-key auto-included. +12. **`--asof-key-field matched_time`**: paired records carry the left + key; combined with `nearest` so the matched key differs from the probe + key. +13. **`--np --ul --ur`**: only-unpaired output, both sides, checking + end-of-stream `--ul` emission order. +14. **`--ul` marking granularity**: with the canonical inputs, quote at + t=11 is never matched and must appear under `--ul`; all other quotes + must not. +15. **Right records missing the key field**: heterogeneous dkvp input + where some records lack `time` — unpaired, surfaced only with `--ur`. +16. **Empty left file**: all right records unpaired; no crash. +17. **Mixed formats**: `--ijson` after the verb for a JSON left file with + CSV main input (mirroring `test/cases/verb-join-mixed-format`). +18. **Unsorted left file**: rates file shuffled on disk — output identical + to case 1 (internal sort). +19. **In-chain use**: `mlr ... cat then join-asof ...` to confirm normal + then-chain behavior. + +Error cases (expout empty, experr pinned): + +20. Missing `-f`; missing `-j`. +21. `-j a,b` (comma in asof key) — error mentioning `-g`. +22. `--dir sideways` — error listing valid directions. +23. `--tolerance banana` and `--tolerance -1` — parse/negativity errors. +24. `--np` alone (no emit flags effective). + +### Unit tests + +- `pkg/transformers/join_asof_test.go` (or in `utils` if the search is + factored there): table-driven tests for the candidate-selection function + — direction × strict × tolerance × duplicate-keys × empty-bucket × + string-keys, isolated from record plumbing. This is where the fiddly + binary-search boundary conditions (first element, last element, + all-equal keys) get exhaustive coverage. +- A case in `aaa_transformer_json_test.go`-style if applicable for CLI + parse of the new verb (match how neighboring verbs are covered). + +## Documentation + +- `docs/src/reference-verbs.md.in`: new `## join-asof` section directly + after `## join` (~line 616), with GENMD-RUN-COMMAND blocks for + `mlr join-asof --help` and worked examples. Example data files + `docs/src/data/asof-rates.csv` and `docs/src/data/asof-trades.csv` + (docs GENMD commands run relative to `docs/src`). Include: the + canonical backward example, `--dir nearest`, `-g currency`, and + `--tolerance` with `--ur`. +- Cross-references: add a pointer from the `## join` docs section ("for + nearest-key rather than equal-key matching, see join-asof"), and a + sentence in `join`'s usage string is *not* needed — keep usage strings + self-contained. +- A short paragraph for pandas/SQL users naming the equivalents + (`merge_asof`, `ASOF JOIN`, rolling joins) — these are the search terms + people arrive with, per the issue thread — and the left/right naming + caveat from "Left/right terminology" above. +- Rebuild with `make -C docs/src forcebuild`; `make dev` regenerates the + manpage and verb listings. + +## Non-goals (this pass) + +- **Generalized DSL join conditions** (`join -c ''`, the issue's + original framing). Arbitrary conditions defeat both hashing and binary + search, forcing an O(N·M) cross-product scan with DSL evaluation per + pair, and require wiring a CST evaluator with a two-record binding + convention (left vs. right field namespacing) into a verb. Every + concrete use case in the thread is served by asof semantics. If demand + remains after `join-asof` ships, that's a separate plan; the CLI surface + proposed here doesn't foreclose it. +- **Doubly-streaming sorted mode** (`join -s` analog) for left files too + large for memory. The bucket-keeper machinery doesn't transfer directly + (asof needs lookback across bucket boundaries); defer until someone + asks. +- **DSL-level asof lookup function** (e.g. a map-backed + `asof_lookup(map, key)` BIF). Possibly useful someday; out of scope. +- **Interpolation** (e.g. linear interpolation between bracketing quotes, + merge-ordered-style). data.table and pandas keep this separate from + asof joins; so should Miller. + +## Verification + +1. `make build`; manual run of the canonical example and a spot-check of + `--dir nearest` / `--tolerance` against pandas `merge_asof` output on + the same data. +2. `make check` (unit + regression, including the new + `test/cases/verb-join-asof/` cases). +3. `make dev` — confirms docs regeneration picks up the new verb and no + existing goldens shift (in particular, `mlr help verbs` output and + manpage goldens will change and need regenerating in the same commit). +4. `make lint` before pushing.