diff --git a/plans/recutils.md b/plans/recutils.md new file mode 100644 index 000000000..0fa703070 --- /dev/null +++ b/plans/recutils.md @@ -0,0 +1,199 @@ +# Plan: add GNU recutils (`.rec`) as a Miller I/O format + +## Context + +[Issue #378](https://github.com/johnkerl/miller/issues/378) requests support for +[GNU recutils](https://www.gnu.org/software/recutils/manual/index.html), a +human-editable, text-based record format: records are `FieldName: Value` lines, +records are separated by blank lines, and Miller's own docs already describe XTAB +as "perhaps most useful for looking at very wide... data" in a similar spirit. The +issue reporter noted the format's kinship with Miller's philosophy and with XTAB, +but explicitly left the integration decision to the maintainer. + +Investigation found that Miller already has an even closer structural precedent +than XTAB: **DCF (Debian control file)** format +(`pkg/input/record_reader_dcf.go`, `pkg/output/record_writer_dcf.go`), which is +also `Key: value` lines with blank-line-separated paragraphs and continuation-line +folding. DCF's registration footprint (factory switches, `separators.go` maps, +`option_parse.go` flags, docs, regression tests) is the direct template for +wiring up recutils. DCF's *parsing*, however, delegates to an external +Debian-specific library and uses RFC822-style (leading-space) continuation — +wrong syntax for recutils (which uses `+`-prefixed and backslash-newline +continuation) — so the recutils reader needs to be hand-rolled, using **XTAB's** +blank-line stanza-scanning goroutine (`channelizedStanzaScanner` in +`pkg/input/record_reader_xtab.go`) as the structural template instead. + +Scope decision: recutils' "record descriptor" (`%rec:`-prefixed) mechanism gives +the format a schema/constraint/type-checking layer (mandatory fields, types, +auto-increment keys, foreign-key-style references) approaching a lightweight +database. That layer is **out of scope** — Miller is a schema-less stream +processor. `%rec`-prefixed records are parsed with the exact same generic +logic as any other record (their fields, e.g. `%rec`, `%mandatory`, `%type`, +just become ordinary Miller fields with no special interpretation). This +mirrors how the existing DCF reader implements Debian control-file *syntax* +without any package-relationship *semantics*. + +Two naming/behavior questions were resolved with the user: +- **Format name / flags**: `recutils`, not the shorter `rec` — flags are + `--irecutils`, `--orecutils`, `--recutils`, matching the issue's own + terminology for discoverability. (Not added to the `--c2j`-style + format-conversion keystroke-saver matrix, following DCF's own precedent of + being excluded from that 10x10 table.) +- **Malformed-line handling**: hard error (no leniency flag) for a data line + missing the `": "` separator, or a `+`-continuation line with no preceding + field in the stanza. This matches recutils' own spec (the separator is + exactly colon-space) and keeps v1 scope small. + +## Format semantics to implement + +- Records = stanzas separated by one or more blank lines (leading/trailing + blank lines ignored) — identical boundary logic to XTAB's. +- Each field is one line, `FieldName: Value`; split on the **first** + occurrence of `": "` only (a value may legitimately contain `: ` later in + the string). +- Two continuation mechanisms, both processed *before* colon-splitting, in + this order: + 1. **Backslash-newline**: a line whose last character is `\` is joined + directly (no separator) with the next physical line. + 2. **`+`-continuation**: a line starting with `+` continues the + *previous field's* value with an embedded `\n`; a single leading space + after the `+` is stripped (`+ text` → continuation text `text`); a bare + `+` embeds an empty line. +- Comment lines (`#`-prefixed) reuse Miller's existing generic + `readerOptions.CommentHandling`/`CommentString` machinery — same + before-stanza-logic prefix check XTAB already does. No new option fields + needed. +- `%rec`-prefixed descriptor records: no special handling (see Scope decision + above) — parsed like any other record. +- PS is fixed at `": "`, RS is fixed at blank-line, FS is not applicable — + none of these are configurable via `--ifs`/`--ips`/etc., matching DCF's + "N/A" treatment. **No new fields needed in `TReaderOptions`/`TWriterOptions`** + (`pkg/cli/option_types.go`) — this format needs zero format-specific option + state, same as DCF. +- Known, documented limitation: a value whose last physical line ends in a + literal `\` is ambiguous on round-trip (indistinguishable from an + in-progress backslash-continuation) — this is an inherent ambiguity in the + recutils format itself (no in-value backslash-escaping exists), not a gap + specific to this implementation. Document it in `file-formats.md.in` and + pin the (lossy) behavior with a regression test rather than inventing + non-standard escaping. + +## Implementation + +### Reader: `pkg/input/record_reader_rec.go` (new) + +Copy the outer `Read`/`processHandle` boilerplate from +`record_reader_dcf.go` (open stdin/files, batch onto `readerChannel`, respect +`downstreamDoneChannel`). For stanza-scanning, copy-and-adapt +`channelizedStanzaScanner` from `record_reader_xtab.go` — same +`tStanza{dataLines, commentLines}` blank-line-boundary/comment-handling logic, +but call `NewLineReader(handle, "\n")` with a fixed `"\n"` (not +`readerOptions.IFS`, since RS isn't configurable here unlike XTAB). + +Replace XTAB's whitespace-run `pairSplitter` with a new +`recordFromRECLines(stanza []string) (*mlrval.Mlrmap, error)` implementing the +three-pass continuation/split algorithm above: +1. Backslash-join pass over raw lines. +2. `+`-continuation fold pass (error if a `+` line has no preceding field yet + in the stanza). +3. First-`": "` split pass per resulting logical line (error if no `": "` is + found). + +Both error cases propagate via `errorChannel`, consistent with how other +readers fail on structurally malformed input (e.g. CSV's ragged-row default). + +### Writer: `pkg/output/record_writer_rec.go` (new) + +Copy `record_writer_dcf.go` near-verbatim with two changes: +1. Drop DCF's hardcoded array-field comma-joining (`dcfValueString`) — use + plain `mv.String()` instead, like XTAB's writer. recutils has no special + list-valued fields; nested/array values go through Miller's standard + auto-flatten mechanism, since recutils is a non-nestable format (no change + needed to `pkg/cli/flatten_unflatten.go`'s `isNestable()` — the existing + default already does the right thing, same as XTAB). +2. Fold embedded `\n` in a value using `"+ "`-prefixed continuation lines + (recutils convention), not DCF's single-leading-space convention. + +Blank line written after every record (no `onFirst` guard needed, matching +DCF's writer). + +### Registration (mirror every DCF touch point) + +- `pkg/input/record_reader_factory.go`: add `case "recutils":` → + `NewRecordReaderREC`. +- `pkg/output/record_writer_factory.go`: add `case "recutils":` → + `NewRecordWriterREC`. +- `pkg/cli/separators.go`: add `"recutils"` entries to all four maps + (`defaultFSes`, `defaultPSes`, `defaultRSes`, `defaultAllowRepeatIFSes`), + identical shape to the existing `"dcf"` entries (`"N/A"` / `"N/A"` / `"N/A"` + / `false`). This is also the master legal-format-name list + (`GetFileFormatNames()`), so this step alone makes `-i recutils`, `-o + recutils`, `--io recutils` work. +- `pkg/cli/option_parse.go` (`FileFormatFlagSection`): add `--irecutils`, + `--orecutils`, `--recutils` flags, copied from the `--idcf`/`--odcf`/`--dcf` + blocks (~lines 888, 1142, 1340). Update + `FormatConversionKeystrokeSaverPrintInfo`'s info string to add a sentence + analogous to the existing DCF one ("recutils is also supported (use + --recutils for recutils in and out)"); do **not** add recutils to the + 10x10 keystroke-saver matrix. +- No changes needed to `pkg/cli/option_types.go` or + `pkg/cli/flatten_unflatten.go` (see above). + +### Docs + +- `docs/src/file-formats.md.in`: new `## recutils` section, modeled on the + DCF section, placed just before "## Data-conversion keystroke-savers". + Include a `cat data/sample.rec` block and an + `mlr -i recutils -o json cat data/sample.rec` block. Note the `+` + continuation and the trailing-backslash round-trip limitation. +- `docs/src/data/sample.rec` (new): a small, illustrative example (e.g. + address-book style, echoing GNU recutils' own canonical manual example), + exercising at least one `+`-continuation. +- `docs/src/reference-main-separators.md.in`: new table row, shaped like the + DCF row but with the PS cell reading `Always ": "; not alterable` + (colon-*space*, not bare colon — malformed otherwise per the hard-error + decision above) and RS reading `N/A; records separated by blank lines`. +- `docs/src/glossary.md.in` (~line 467): add "recutils" to the bracketed list + of non-line-oriented formats (`CSV, TSV, JSON, YAML, DCF, recutils, and + others`). +- `docs/src/reference-main-flag-list.md.in` is auto-generated from + `option_parse.go` — no manual edit, just rebuild via + `make -C docs/src forcebuild`. + +### Tests + +- `pkg/input/record_reader_rec_test.go` (new): unit tests for + `recordFromRECLines` covering backslash-join, `+`-continuation (including + bare `+`), colon-in-value-after-first-occurrence, `+` with no preceding + field → error, missing `": "` → error, and comments interleaved with + continuation lines (comments are stripped by the scanner before the + continuation passes ever see them). +- `test/cases/io-recutils/` (new), modeled on `test/cases/io-dcf/`: + - `0001`: `csv → recutils` conversion smoke test. + - `0002`: `recutils → recutils` round trip. + - `0003`: `recutils → json` conversion. + - `0004`: `+`-continuation and comment handling exercised at the CLI level + (the one genuinely new behavior vs. DCF, which has no continuation + syntax). +- `test/input/test.rec` (new fixture): multi-record file including a + `+`-continued field and a comment line, reused across the cases above (like + `test/input/test.dcf` is reused across DCF's cases). + +## Verification + +1. `make build` then manual smoke tests: + - `mlr --icsv --orecutils cat somefile.csv` — confirm blank-line-separated + `Key: value` output. + - `mlr --irecutils --ojson cat docs/src/data/sample.rec` — confirm nested + JSON output round-trips field values correctly, including the + `+`-continued one. + - `mlr --irecutils --orecutils cat docs/src/data/sample.rec` — round trip + should reproduce equivalent records (modulo the documented + trailing-backslash limitation). + - Feed a malformed file (line with no `": "`, or a leading `+` with no + prior field) and confirm a clear error, not a crash or silent + misparse. +2. `mlr regtest -p test/cases/io-recutils` to populate `expout`/`experr`, + then review the diffs by eye before committing them. +3. `make dev` (fmt, build, unit + regression tests, docs rebuild) and + `make lint` — both required before pushing per this repo's CLAUDE.md.