miller/plans/recutils.md

Plan: add GNU recutils (.rec) as a Miller I/O format

Context

Issue #378 requests support for GNU recutils, a human-editable, text-based record format: records are FieldName: Value lines, records are separated by blank lines, and Miller's own docs already describe XTAB as "perhaps most useful for looking at very wide... data" in a similar spirit. The issue reporter noted the format's kinship with Miller's philosophy and with XTAB, but explicitly left the integration decision to the maintainer.

Investigation found that Miller already has an even closer structural precedent than XTAB: DCF (Debian control file) format (pkg/input/record_reader_dcf.go, pkg/output/record_writer_dcf.go), which is also Key: value lines with blank-line-separated paragraphs and continuation-line folding. DCF's registration footprint (factory switches, separators.go maps, option_parse.go flags, docs, regression tests) is the direct template for wiring up recutils. DCF's parsing, however, delegates to an external Debian-specific library and uses RFC822-style (leading-space) continuation — wrong syntax for recutils (which uses +-prefixed and backslash-newline continuation) — so the recutils reader needs to be hand-rolled, using XTAB's blank-line stanza-scanning goroutine (channelizedStanzaScanner in pkg/input/record_reader_xtab.go) as the structural template instead.

Scope decision: recutils' "record descriptor" (%rec:-prefixed) mechanism gives the format a schema/constraint/type-checking layer (mandatory fields, types, auto-increment keys, foreign-key-style references) approaching a lightweight database. That layer is out of scope — Miller is a schema-less stream processor. %rec-prefixed records are parsed with the exact same generic logic as any other record (their fields, e.g. %rec, %mandatory, %type, just become ordinary Miller fields with no special interpretation). This mirrors how the existing DCF reader implements Debian control-file syntax without any package-relationship semantics.

Two naming/behavior questions were resolved with the user:

• Format name / flags: recutils, not the shorter rec — flags are --irecutils, --orecutils, --recutils, matching the issue's own terminology for discoverability. (Not added to the --c2j-style format-conversion keystroke-saver matrix, following DCF's own precedent of being excluded from that 10x10 table.)
• Malformed-line handling: hard error (no leniency flag) for a data line missing the ": " separator, or a +-continuation line with no preceding field in the stanza. This matches recutils' own spec (the separator is exactly colon-space) and keeps v1 scope small.

Format semantics to implement

• Records = stanzas separated by one or more blank lines (leading/trailing blank lines ignored) — identical boundary logic to XTAB's.
• Each field is one line, FieldName: Value; split on the first occurrence of ": " only (a value may legitimately contain : later in the string).
• Two continuation mechanisms, both processed before colon-splitting, in this order:
  1. Backslash-newline: a line whose last character is \ is joined directly (no separator) with the next physical line.
  2. +-continuation: a line starting with + continues the previous field's value with an embedded \n; a single leading space after the + is stripped (+ text → continuation text text); a bare + embeds an empty line.
• Comment lines (#-prefixed) reuse Miller's existing generic readerOptions.CommentHandling/CommentString machinery — same before-stanza-logic prefix check XTAB already does. No new option fields needed.
• %rec-prefixed descriptor records: no special handling (see Scope decision above) — parsed like any other record.
• PS is fixed at ": ", RS is fixed at blank-line, FS is not applicable — none of these are configurable via --ifs/--ips/etc., matching DCF's "N/A" treatment. No new fields needed in TReaderOptions/TWriterOptions (pkg/cli/option_types.go) — this format needs zero format-specific option state, same as DCF.
• Known, documented limitation: a value whose last physical line ends in a literal \ is ambiguous on round-trip (indistinguishable from an in-progress backslash-continuation) — this is an inherent ambiguity in the recutils format itself (no in-value backslash-escaping exists), not a gap specific to this implementation. Document it in file-formats.md.in and pin the (lossy) behavior with a regression test rather than inventing non-standard escaping.

Implementation

Reader: pkg/input/record_reader_rec.go (new)

Copy the outer Read/processHandle boilerplate from record_reader_dcf.go (open stdin/files, batch onto readerChannel, respect downstreamDoneChannel). For stanza-scanning, copy-and-adapt channelizedStanzaScanner from record_reader_xtab.go — same tStanza{dataLines, commentLines} blank-line-boundary/comment-handling logic, but call NewLineReader(handle, "\n") with a fixed "\n" (not readerOptions.IFS, since RS isn't configurable here unlike XTAB).

Replace XTAB's whitespace-run pairSplitter with a new recordFromRECLines(stanza []string) (*mlrval.Mlrmap, error) implementing the three-pass continuation/split algorithm above:

1. Backslash-join pass over raw lines.
2. +-continuation fold pass (error if a + line has no preceding field yet in the stanza).
3. First-": " split pass per resulting logical line (error if no ": " is found).

Both error cases propagate via errorChannel, consistent with how other readers fail on structurally malformed input (e.g. CSV's ragged-row default).

Writer: pkg/output/record_writer_rec.go (new)

Copy record_writer_dcf.go near-verbatim with two changes:

1. Drop DCF's hardcoded array-field comma-joining (dcfValueString) — use plain mv.String() instead, like XTAB's writer. recutils has no special list-valued fields; nested/array values go through Miller's standard auto-flatten mechanism, since recutils is a non-nestable format (no change needed to pkg/cli/flatten_unflatten.go's isNestable() — the existing default already does the right thing, same as XTAB).
2. Fold embedded \n in a value using "+ "-prefixed continuation lines (recutils convention), not DCF's single-leading-space convention.

Blank line written after every record (no onFirst guard needed, matching DCF's writer).

Registration (mirror every DCF touch point)

• pkg/input/record_reader_factory.go: add case "recutils": → NewRecordReaderREC.
• pkg/output/record_writer_factory.go: add case "recutils": → NewRecordWriterREC.
• pkg/cli/separators.go: add "recutils" entries to all four maps (defaultFSes, defaultPSes, defaultRSes, defaultAllowRepeatIFSes), identical shape to the existing "dcf" entries ("N/A" / "N/A" / "N/A" / false). This is also the master legal-format-name list (GetFileFormatNames()), so this step alone makes -i recutils, -o recutils, --io recutils work.
• pkg/cli/option_parse.go (FileFormatFlagSection): add --irecutils, --orecutils, --recutils flags, copied from the --idcf/--odcf/--dcf blocks (~lines 888, 1142, 1340). Update FormatConversionKeystrokeSaverPrintInfo's info string to add a sentence analogous to the existing DCF one ("recutils is also supported (use --recutils for recutils in and out)"); do not add recutils to the 10x10 keystroke-saver matrix.
• No changes needed to pkg/cli/option_types.go or pkg/cli/flatten_unflatten.go (see above).

Docs

• docs/src/file-formats.md.in: new ## recutils section, modeled on the DCF section, placed just before "## Data-conversion keystroke-savers". Include a cat data/sample.rec block and an mlr -i recutils -o json cat data/sample.rec block. Note the + continuation and the trailing-backslash round-trip limitation.
• docs/src/data/sample.rec (new): a small, illustrative example (e.g. address-book style, echoing GNU recutils' own canonical manual example), exercising at least one +-continuation.
• docs/src/reference-main-separators.md.in: new table row, shaped like the DCF row but with the PS cell reading Always ": "; not alterable (colon-space, not bare colon — malformed otherwise per the hard-error decision above) and RS reading N/A; records separated by blank lines.
• docs/src/glossary.md.in (~line 467): add "recutils" to the bracketed list of non-line-oriented formats (CSV, TSV, JSON, YAML, DCF, recutils, and others).
• docs/src/reference-main-flag-list.md.in is auto-generated from option_parse.go — no manual edit, just rebuild via make -C docs/src forcebuild.

Tests

• pkg/input/record_reader_rec_test.go (new): unit tests for recordFromRECLines covering backslash-join, +-continuation (including bare +), colon-in-value-after-first-occurrence, + with no preceding field → error, missing ": " → error, and comments interleaved with continuation lines (comments are stripped by the scanner before the continuation passes ever see them).
• test/cases/io-recutils/ (new), modeled on test/cases/io-dcf/:
  • 0001: csv → recutils conversion smoke test.
  • 0002: recutils → recutils round trip.
  • 0003: recutils → json conversion.
  • 0004: +-continuation and comment handling exercised at the CLI level (the one genuinely new behavior vs. DCF, which has no continuation syntax).
• test/input/test.rec (new fixture): multi-record file including a +-continued field and a comment line, reused across the cases above (like test/input/test.dcf is reused across DCF's cases).

Verification

1. make build then manual smoke tests:
   • mlr --icsv --orecutils cat somefile.csv — confirm blank-line-separated Key: value output.
   • mlr --irecutils --ojson cat docs/src/data/sample.rec — confirm nested JSON output round-trips field values correctly, including the +-continued one.
   • mlr --irecutils --orecutils cat docs/src/data/sample.rec — round trip should reproduce equivalent records (modulo the documented trailing-backslash limitation).
   • Feed a malformed file (line with no ": ", or a leading + with no prior field) and confirm a clear error, not a crash or silent misparse.
2. mlr regtest -p test/cases/io-recutils to populate expout/experr, then review the diffs by eye before committing them.
3. make dev (fmt, build, unit + regression tests, docs rebuild) and make lint — both required before pushing per this repo's CLAUDE.md.