diff --git a/docs/src/ai-support.md b/docs/src/ai-support.md index 24f996707..4ce12c9e2 100644 --- a/docs/src/ai-support.md +++ b/docs/src/ai-support.md @@ -64,7 +64,7 @@ one-line summary (here trimmed, and then counted, using Miller itself): 
 [
 {
-  "count": 667
+  "count": 670
 }
 ]
diff --git a/docs/src/data/sample.rec b/docs/src/data/sample.rec new file mode 100644 index 000000000..4e0b70495 --- /dev/null +++ b/docs/src/data/sample.rec @@ -0,0 +1,11 @@ +# A simple contacts database. + +Name: Mr. Foo +Email: foo@example.com +Phone: 555-1234 +Notes: Likes long walks ++ on the beach. + +Name: Mrs. Bar +Email: bar@example.com +Phone: 555-5678 diff --git a/docs/src/file-formats.md b/docs/src/file-formats.md index 2074138c6..c7349ce15 100644 --- a/docs/src/file-formats.md +++ b/docs/src/file-formats.md @@ -948,6 +948,56 @@ Description: Another package. ] +## recutils + +[GNU recutils](https://www.gnu.org/software/recutils/manual/index.html) is a text-based format for record-oriented databases. Records are `FieldName: Value` lines, one field per line, with records separated by one or more blank lines: + +
+cat data/sample.rec
+
+
+# A simple contacts database.
+
+Name: Mr. Foo
+Email: foo@example.com
+Phone: 555-1234
+Notes: Likes long walks
++ on the beach.
+
+Name: Mrs. Bar
+Email: bar@example.com
+Phone: 555-5678
+
+ +A field's value can be continued onto following lines by prefixing each continuation line with `+`; the continuation is joined onto the value with an embedded newline. (A trailing `\` at the very end of a line is also honored, recutils-style, to join two physical lines into one logical line with no embedded newline.) + +Since `#`-prefixed lines are, like every other Miller format, treated as data unless `--skip-comments` or `--pass-comments` is given (see [Comments in data](#comments-in-data)), reading the file above needs `--skip-comments`: + +
+mlr --skip-comments -i recutils -o json cat data/sample.rec
+
+
+[
+{
+  "Name": "Mr. Foo",
+  "Email": "foo@example.com",
+  "Phone": "555-1234",
+  "Notes": "Likes long walks\non the beach."
+},
+{
+  "Name": "Mrs. Bar",
+  "Email": "bar@example.com",
+  "Phone": "555-5678"
+}
+]
+
+ +Miller has no notion of recutils' record-descriptor/schema records (lines starting with `%rec:` which declare field types, mandatory fields, keys, and so on) -- those are read and written as ordinary records, with no special interpretation, since Miller is a schema-less stream processor. + +Use `--irecutils`/`--orecutils`/`--recutils` (or `-i recutils`/`-o recutils`) for recutils input/output/both, analogously to `--idcf`/`--odcf`/`--dcf`. + +Note: a field value whose last line ends in a literal `\` is ambiguous with an in-progress backslash-continuation on write/re-read, since recutils has no in-value backslash-escaping mechanism. This is a limitation of the recutils format itself, not specific to Miller. + ## Data-conversion keystroke-savers While you can do format conversion using `mlr --icsv --ojson cat myfile.csv`, there are also keystroke-savers for this purpose, such as `mlr --c2j cat myfile.csv`. For a complete list: @@ -960,6 +1010,7 @@ FORMAT-CONVERSION KEYSTROKE-SAVER FLAGS As keystroke-savers for format-conversion you may use the following. The letters c, t, j, l, d, n, x, p, m, and y refer to formats CSV, TSV, JSON, JSON Lines, DKVP, NIDX, XTAB, PPRINT, markdown, and YAML, respectively. DCF is also supported (use --dcf for DCF in and out). +GNU recutils is also supported (use --recutils for recutils in and out). | In\out | CSV | TSV | JSON | JSONL | DKVP | NIDX | XTAB | PPRINT | Markdown | YAML | +----------+----------+----------+----------+-------+-------+-------+-------+--------+----------+--------+ diff --git a/docs/src/file-formats.md.in b/docs/src/file-formats.md.in index 4a821710a..1599fafc5 100644 --- a/docs/src/file-formats.md.in +++ b/docs/src/file-formats.md.in @@ -462,6 +462,28 @@ GENMD-RUN-COMMAND mlr -i dcf -o json cat data/sample.dcf GENMD-EOF +## recutils + +[GNU recutils](https://www.gnu.org/software/recutils/manual/index.html) is a text-based format for record-oriented databases. Records are `FieldName: Value` lines, one field per line, with records separated by one or more blank lines: + +GENMD-RUN-COMMAND +cat data/sample.rec +GENMD-EOF + +A field's value can be continued onto following lines by prefixing each continuation line with `+`; the continuation is joined onto the value with an embedded newline. (A trailing `\` at the very end of a line is also honored, recutils-style, to join two physical lines into one logical line with no embedded newline.) + +Since `#`-prefixed lines are, like every other Miller format, treated as data unless `--skip-comments` or `--pass-comments` is given (see [Comments in data](#comments-in-data)), reading the file above needs `--skip-comments`: + +GENMD-RUN-COMMAND +mlr --skip-comments -i recutils -o json cat data/sample.rec +GENMD-EOF + +Miller has no notion of recutils' record-descriptor/schema records (lines starting with `%rec:` which declare field types, mandatory fields, keys, and so on) -- those are read and written as ordinary records, with no special interpretation, since Miller is a schema-less stream processor. + +Use `--irecutils`/`--orecutils`/`--recutils` (or `-i recutils`/`-o recutils`) for recutils input/output/both, analogously to `--idcf`/`--odcf`/`--dcf`. + +Note: a field value whose last line ends in a literal `\` is ambiguous with an in-progress backslash-continuation on write/re-read, since recutils has no in-value backslash-escaping mechanism. This is a limitation of the recutils format itself, not specific to Miller. + ## Data-conversion keystroke-savers While you can do format conversion using `mlr --icsv --ojson cat myfile.csv`, there are also keystroke-savers for this purpose, such as `mlr --c2j cat myfile.csv`. For a complete list: diff --git a/docs/src/glossary.md b/docs/src/glossary.md index 9dce6e454..392ee0c95 100644 --- a/docs/src/glossary.md +++ b/docs/src/glossary.md @@ -480,7 +480,7 @@ trying to define a [local variable](#local-variable) `if = 3` will result in a p A subsequence of a text file in between line-ending symbols such as the special linefeed character. Tools in the [Unix toolkit](#unix-toolkit) generally operate on lines; Miller is designed to do that (using the [NIDX format flags](file-formats.md#nidx-index-numbered-toolkit-style)), as well -as non-line-oriented formats such as [CSV, TSV, JSON, YAML, DCF, and others](file-formats.md). +as non-line-oriented formats such as [CSV, TSV, JSON, YAML, DCF, recutils, and others](file-formats.md). ## local variable diff --git a/docs/src/glossary.md.in b/docs/src/glossary.md.in index 6bcb5c7af..57afcdf2e 100644 --- a/docs/src/glossary.md.in +++ b/docs/src/glossary.md.in @@ -464,7 +464,7 @@ trying to define a [local variable](#local-variable) `if = 3` will result in a p A subsequence of a text file in between line-ending symbols such as the special linefeed character. Tools in the [Unix toolkit](#unix-toolkit) generally operate on lines; Miller is designed to do that (using the [NIDX format flags](file-formats.md#nidx-index-numbered-toolkit-style)), as well -as non-line-oriented formats such as [CSV, TSV, JSON, YAML, DCF, and others](file-formats.md). +as non-line-oriented formats such as [CSV, TSV, JSON, YAML, DCF, recutils, and others](file-formats.md). ## local variable diff --git a/docs/src/manpage.md b/docs/src/manpage.md index 0b643ea59..6b1d1b71a 100644 --- a/docs/src/manpage.md +++ b/docs/src/manpage.md @@ -441,6 +441,7 @@ This is simply a copy of what you should see on running `man mlr` at a command p --io {format name} Use format name for input and output data. For example: `--io csv` is the same as `--csv`. --ipprint Use PPRINT format for input data. + --irecutils Use GNU recutils (.rec) format for input data. --itsv Use TSV format for input data. --itsvlite Use TSV-lite format for input data. --iusv or --iusvlite Use USV format for input data. @@ -461,12 +462,15 @@ This is simply a copy of what you should see on running `man mlr` at a command p --omd or --omarkdown Use markdown-tabular format for output data. --onidx Use NIDX format for output data. --opprint Use PPRINT format for output data. + --orecutils Use GNU recutils (.rec) format for output data. --otsv Use TSV format for output data. --otsvlite Use TSV-lite format for output data. --ousv or --ousvlite Use USV format for output data. --oxtab Use XTAB format for output data. --oyaml Use YAML format for output data. --pprint or --p2p Use PPRINT format for input and output data. + --recutils Use GNU recutils (.rec) format for input and output + data. --tsv or -t or --t2t Use TSV format for input and output data. --tsvlite Use TSV-lite format for input and output data. --usv or --usvlite Use USV format for input and output data. @@ -503,6 +507,7 @@ This is simply a copy of what you should see on running `man mlr` at a command p As keystroke-savers for format-conversion you may use the following. The letters c, t, j, l, d, n, x, p, m, and y refer to formats CSV, TSV, JSON, JSON Lines, DKVP, NIDX, XTAB, PPRINT, markdown, and YAML, respectively. DCF is also supported (use --dcf for DCF in and out). + GNU recutils is also supported (use --recutils for recutils in and out). | In\out | CSV | TSV | JSON | JSONL | DKVP | NIDX | XTAB | PPRINT | Markdown | YAML | +----------+----------+----------+----------+-------+-------+-------+-------+--------+----------+--------+ @@ -928,6 +933,7 @@ This is simply a copy of what you should see on running `man mlr` at a command p markdown " " N/A "\n" nidx " " N/A "\n" pprint " " N/A "\n" + recutils N/A N/A N/A tsv " " N/A "\n" xtab "\n" " " "\n\n" yaml N/A N/A N/A diff --git a/docs/src/manpage.txt b/docs/src/manpage.txt index 5cde2e3fb..3415b2b84 100644 --- a/docs/src/manpage.txt +++ b/docs/src/manpage.txt @@ -420,6 +420,7 @@ --io {format name} Use format name for input and output data. For example: `--io csv` is the same as `--csv`. --ipprint Use PPRINT format for input data. + --irecutils Use GNU recutils (.rec) format for input data. --itsv Use TSV format for input data. --itsvlite Use TSV-lite format for input data. --iusv or --iusvlite Use USV format for input data. @@ -440,12 +441,15 @@ --omd or --omarkdown Use markdown-tabular format for output data. --onidx Use NIDX format for output data. --opprint Use PPRINT format for output data. + --orecutils Use GNU recutils (.rec) format for output data. --otsv Use TSV format for output data. --otsvlite Use TSV-lite format for output data. --ousv or --ousvlite Use USV format for output data. --oxtab Use XTAB format for output data. --oyaml Use YAML format for output data. --pprint or --p2p Use PPRINT format for input and output data. + --recutils Use GNU recutils (.rec) format for input and output + data. --tsv or -t or --t2t Use TSV format for input and output data. --tsvlite Use TSV-lite format for input and output data. --usv or --usvlite Use USV format for input and output data. @@ -482,6 +486,7 @@ As keystroke-savers for format-conversion you may use the following. The letters c, t, j, l, d, n, x, p, m, and y refer to formats CSV, TSV, JSON, JSON Lines, DKVP, NIDX, XTAB, PPRINT, markdown, and YAML, respectively. DCF is also supported (use --dcf for DCF in and out). + GNU recutils is also supported (use --recutils for recutils in and out). | In\out | CSV | TSV | JSON | JSONL | DKVP | NIDX | XTAB | PPRINT | Markdown | YAML | +----------+----------+----------+----------+-------+-------+-------+-------+--------+----------+--------+ @@ -907,6 +912,7 @@ markdown " " N/A "\n" nidx " " N/A "\n" pprint " " N/A "\n" + recutils N/A N/A N/A tsv " " N/A "\n" xtab "\n" " " "\n\n" yaml N/A N/A N/A diff --git a/docs/src/reference-main-flag-list.md b/docs/src/reference-main-flag-list.md index 17b0de7df..cf4566fa7 100644 --- a/docs/src/reference-main-flag-list.md +++ b/docs/src/reference-main-flag-list.md @@ -174,6 +174,7 @@ are overridden in all cases by setting output format to `format2`. * `--inidx`: Use NIDX format for input data. * `--io {format name}`: Use format name for input and output data. For example: `--io csv` is the same as `--csv`. * `--ipprint`: Use PPRINT format for input data. +* `--irecutils`: Use GNU recutils (.rec) format for input data. * `--itsv`: Use TSV format for input data. * `--itsvlite`: Use TSV-lite format for input data. * `--iusv or --iusvlite`: Use USV format for input data. @@ -193,12 +194,14 @@ are overridden in all cases by setting output format to `format2`. * `--omd or --omarkdown`: Use markdown-tabular format for output data. * `--onidx`: Use NIDX format for output data. * `--opprint`: Use PPRINT format for output data. +* `--orecutils`: Use GNU recutils (.rec) format for output data. * `--otsv`: Use TSV format for output data. * `--otsvlite`: Use TSV-lite format for output data. * `--ousv or --ousvlite`: Use USV format for output data. * `--oxtab`: Use XTAB format for output data. * `--oyaml`: Use YAML format for output data. * `--pprint or --p2p`: Use PPRINT format for input and output data. +* `--recutils`: Use GNU recutils (.rec) format for input and output data. * `--tsv or -t or --t2t`: Use TSV format for input and output data. * `--tsvlite`: Use TSV-lite format for input and output data. * `--usv or --usvlite`: Use USV format for input and output data. @@ -527,6 +530,7 @@ Notes about all other separators: markdown " " N/A "\n" nidx " " N/A "\n" pprint " " N/A "\n" + recutils N/A N/A N/A tsv " " N/A "\n" xtab "\n" " " "\n\n" yaml N/A N/A N/A diff --git a/docs/src/reference-main-separators.md b/docs/src/reference-main-separators.md index f133e8a04..2976a09ff 100644 --- a/docs/src/reference-main-separators.md +++ b/docs/src/reference-main-separators.md @@ -277,6 +277,7 @@ Notes: | [**JSON**](file-formats.md#json) | N/A; records are between `{` and `}` | Always `,`; not alterable | Always `:`; not alterable | | [**YAML**](file-formats.md#yaml) | N/A; documents separated by `---` or single array | N/A; not alterable | Always `:`; not alterable | | [**DCF**](file-formats.md#dcf-debian-control-file) | N/A; paragraphs separated by blank lines | N/A; not alterable | Always `:`; not alterable | +| [**recutils**](file-formats.md#recutils) | N/A; records separated by blank lines | N/A; not alterable | Always `: `; not alterable | | [**DKVP**](file-formats.md#dkvp-key-value-pairs) | Default `\n` | Default `,` | Default `=` | | [**DKVPX**](file-formats.md#dkvpx-key-value-pairs-with-csv-style-quoting) | Default `\n` | Default `,`; must be single-character for input | Default `=`; must be single-character for input | | [**NIDX**](file-formats.md#nidx-index-numbered-toolkit-style) | Default `\n` | Default space | None | diff --git a/docs/src/reference-main-separators.md.in b/docs/src/reference-main-separators.md.in index 940d24bef..19d0b5c1d 100644 --- a/docs/src/reference-main-separators.md.in +++ b/docs/src/reference-main-separators.md.in @@ -167,6 +167,7 @@ Notes: | [**JSON**](file-formats.md#json) | N/A; records are between `{` and `}` | Always `,`; not alterable | Always `:`; not alterable | | [**YAML**](file-formats.md#yaml) | N/A; documents separated by `---` or single array | N/A; not alterable | Always `:`; not alterable | | [**DCF**](file-formats.md#dcf-debian-control-file) | N/A; paragraphs separated by blank lines | N/A; not alterable | Always `:`; not alterable | +| [**recutils**](file-formats.md#recutils) | N/A; records separated by blank lines | N/A; not alterable | Always `: `; not alterable | | [**DKVP**](file-formats.md#dkvp-key-value-pairs) | Default `\n` | Default `,` | Default `=` | | [**DKVPX**](file-formats.md#dkvpx-key-value-pairs-with-csv-style-quoting) | Default `\n` | Default `,`; must be single-character for input | Default `=`; must be single-character for input | | [**NIDX**](file-formats.md#nidx-index-numbered-toolkit-style) | Default `\n` | Default space | None | diff --git a/man/manpage.txt b/man/manpage.txt index 5cde2e3fb..3415b2b84 100644 --- a/man/manpage.txt +++ b/man/manpage.txt @@ -420,6 +420,7 @@ --io {format name} Use format name for input and output data. For example: `--io csv` is the same as `--csv`. --ipprint Use PPRINT format for input data. + --irecutils Use GNU recutils (.rec) format for input data. --itsv Use TSV format for input data. --itsvlite Use TSV-lite format for input data. --iusv or --iusvlite Use USV format for input data. @@ -440,12 +441,15 @@ --omd or --omarkdown Use markdown-tabular format for output data. --onidx Use NIDX format for output data. --opprint Use PPRINT format for output data. + --orecutils Use GNU recutils (.rec) format for output data. --otsv Use TSV format for output data. --otsvlite Use TSV-lite format for output data. --ousv or --ousvlite Use USV format for output data. --oxtab Use XTAB format for output data. --oyaml Use YAML format for output data. --pprint or --p2p Use PPRINT format for input and output data. + --recutils Use GNU recutils (.rec) format for input and output + data. --tsv or -t or --t2t Use TSV format for input and output data. --tsvlite Use TSV-lite format for input and output data. --usv or --usvlite Use USV format for input and output data. @@ -482,6 +486,7 @@ As keystroke-savers for format-conversion you may use the following. The letters c, t, j, l, d, n, x, p, m, and y refer to formats CSV, TSV, JSON, JSON Lines, DKVP, NIDX, XTAB, PPRINT, markdown, and YAML, respectively. DCF is also supported (use --dcf for DCF in and out). + GNU recutils is also supported (use --recutils for recutils in and out). | In\out | CSV | TSV | JSON | JSONL | DKVP | NIDX | XTAB | PPRINT | Markdown | YAML | +----------+----------+----------+----------+-------+-------+-------+-------+--------+----------+--------+ @@ -907,6 +912,7 @@ markdown " " N/A "\n" nidx " " N/A "\n" pprint " " N/A "\n" + recutils N/A N/A N/A tsv " " N/A "\n" xtab "\n" " " "\n\n" yaml N/A N/A N/A diff --git a/man/mlr.1 b/man/mlr.1 index a7b882e2a..621f3d71a 100644 --- a/man/mlr.1 +++ b/man/mlr.1 @@ -508,6 +508,7 @@ are overridden in all cases by setting output format to `format2`. --io {format name} Use format name for input and output data. For example: `--io csv` is the same as `--csv`. --ipprint Use PPRINT format for input data. +--irecutils Use GNU recutils (.rec) format for input data. --itsv Use TSV format for input data. --itsvlite Use TSV-lite format for input data. --iusv or --iusvlite Use USV format for input data. @@ -528,12 +529,15 @@ are overridden in all cases by setting output format to `format2`. --omd or --omarkdown Use markdown-tabular format for output data. --onidx Use NIDX format for output data. --opprint Use PPRINT format for output data. +--orecutils Use GNU recutils (.rec) format for output data. --otsv Use TSV format for output data. --otsvlite Use TSV-lite format for output data. --ousv or --ousvlite Use USV format for output data. --oxtab Use XTAB format for output data. --oyaml Use YAML format for output data. --pprint or --p2p Use PPRINT format for input and output data. +--recutils Use GNU recutils (.rec) format for input and output + data. --tsv or -t or --t2t Use TSV format for input and output data. --tsvlite Use TSV-lite format for input and output data. --usv or --usvlite Use USV format for input and output data. @@ -586,6 +590,7 @@ See the flatten/unflatten doc page https://miller.readthedocs.io/en/latest/flatt As keystroke-savers for format-conversion you may use the following. The letters c, t, j, l, d, n, x, p, m, and y refer to formats CSV, TSV, JSON, JSON Lines, DKVP, NIDX, XTAB, PPRINT, markdown, and YAML, respectively. DCF is also supported (use --dcf for DCF in and out). +GNU recutils is also supported (use --recutils for recutils in and out). | In\eout | CSV | TSV | JSON | JSONL | DKVP | NIDX | XTAB | PPRINT | Markdown | YAML | +----------+----------+----------+----------+-------+-------+-------+-------+--------+----------+--------+ @@ -1075,6 +1080,7 @@ Notes about all other separators: markdown " " N/A "\en" nidx " " N/A "\en" pprint " " N/A "\en" + recutils N/A N/A N/A tsv " " N/A "\en" xtab "\en" " " "\en\en" yaml N/A N/A N/A diff --git a/pkg/cli/option_parse.go b/pkg/cli/option_parse.go index 5cf197194..e9083bc95 100644 --- a/pkg/cli/option_parse.go +++ b/pkg/cli/option_parse.go @@ -893,6 +893,15 @@ var FileFormatFlagSection = FlagSection{ }, }, + { + name: "--irecutils", + help: "Use GNU recutils (.rec) format for input data.", + parser: func(args []string, argc int, pargi *int, options *TOptions) { + options.ReaderOptions.InputFileFormat = "recutils" + *pargi += 1 + }, + }, + { name: "--inidx", help: "Use NIDX format for input data.", @@ -1147,6 +1156,15 @@ var FileFormatFlagSection = FlagSection{ }, }, + { + name: "--orecutils", + help: "Use GNU recutils (.rec) format for output data.", + parser: func(args []string, argc int, pargi *int, options *TOptions) { + options.WriterOptions.OutputFileFormat = "recutils" + *pargi += 1 + }, + }, + { name: "--onidx", help: "Use NIDX format for output data.", @@ -1346,6 +1364,16 @@ var FileFormatFlagSection = FlagSection{ }, }, + { + name: "--recutils", + help: "Use GNU recutils (.rec) format for input and output data.", + parser: func(args []string, argc int, pargi *int, options *TOptions) { + options.ReaderOptions.InputFileFormat = "recutils" + options.WriterOptions.OutputFileFormat = "recutils" + *pargi += 1 + }, + }, + { name: "--nidx", help: "Use NIDX format for input and output data.", @@ -1398,6 +1426,7 @@ func FormatConversionKeystrokeSaverPrintInfo() { fmt.Println(`As keystroke-savers for format-conversion you may use the following. The letters c, t, j, l, d, n, x, p, m, and y refer to formats CSV, TSV, JSON, JSON Lines, DKVP, NIDX, XTAB, PPRINT, markdown, and YAML, respectively. DCF is also supported (use --dcf for DCF in and out). +GNU recutils is also supported (use --recutils for recutils in and out). | In\out | CSV | TSV | JSON | JSONL | DKVP | NIDX | XTAB | PPRINT | Markdown | YAML | +----------+----------+----------+----------+-------+-------+-------+-------+--------+----------+--------+ diff --git a/pkg/cli/separators.go b/pkg/cli/separators.go index 0d7e26251..0c0b63d66 100644 --- a/pkg/cli/separators.go +++ b/pkg/cli/separators.go @@ -92,6 +92,7 @@ var defaultFSes = map[string]string{ "json": "N/A", // not alterable; not parameterizable in JSON format "yaml": "N/A", "dcf": "N/A", + "recutils": "N/A", "nidx": " ", "markdown": " ", "pprint": " ", @@ -129,6 +130,7 @@ var defaultPSes = map[string]string{ "json": "N/A", // not alterable; not parameterizable in JSON format "yaml": "N/A", "dcf": "N/A", + "recutils": "N/A", "markdown": "N/A", "nidx": "N/A", "pprint": "N/A", @@ -145,6 +147,7 @@ var defaultRSes = map[string]string{ "json": "N/A", // not alterable; not parameterizable in JSON format "yaml": "N/A", "dcf": "N/A", + "recutils": "N/A", "markdown": "\n", "nidx": "\n", "pprint": "\n", @@ -161,6 +164,7 @@ var defaultAllowRepeatIFSes = map[string]bool{ "json": false, "yaml": false, "dcf": false, + "recutils": false, "markdown": false, "nidx": false, "pprint": true, diff --git a/pkg/input/record_reader_factory.go b/pkg/input/record_reader_factory.go index c8b2edd12..20da954b0 100644 --- a/pkg/input/record_reader_factory.go +++ b/pkg/input/record_reader_factory.go @@ -34,6 +34,8 @@ func Create(readerOptions *cli.TReaderOptions, recordsPerBatch int64) (IRecordRe return NewRecordReaderXTAB(readerOptions, recordsPerBatch) case "dcf": return NewRecordReaderDCF(readerOptions, recordsPerBatch) + case "recutils": + return NewRecordReaderREC(readerOptions, recordsPerBatch) case "gen": return NewPseudoReaderGen(readerOptions, recordsPerBatch) default: diff --git a/pkg/input/record_reader_rec.go b/pkg/input/record_reader_rec.go new file mode 100644 index 000000000..f8d05db7e --- /dev/null +++ b/pkg/input/record_reader_rec.go @@ -0,0 +1,251 @@ +package input + +import ( + "fmt" + "io" + "strings" + + "github.com/johnkerl/miller/v6/pkg/cli" + "github.com/johnkerl/miller/v6/pkg/lib" + "github.com/johnkerl/miller/v6/pkg/mlrval" + "github.com/johnkerl/miller/v6/pkg/types" +) + +// RecordReaderREC reads GNU recutils (.rec) format: records are stanzas of +// "FieldName: Value" lines separated by one or more blank lines, with two +// continuation-line mechanisms (trailing-backslash logical-line joining, and +// "+"-prefixed embedded-newline continuation) and "#" comments. Record +// descriptors ("%rec: ..." stanzas) are not given any special +// schema/constraint interpretation -- they are parsed like any other record, +// since Miller has no schema-enforcement concept. +type RecordReaderREC struct { + readerOptions *cli.TReaderOptions + recordsPerBatch int64 // distinct from readerOptions.RecordsPerBatch for join/repl + // recordArena batch-allocates field entries/values; reset per getRecordBatch. + recordArena *mlrval.RecordArena + + // Note: recutils uses blank lines to delimit records, not a + // configurable IRS; and ": " to delimit fields, not a configurable IPS. +} + +func NewRecordReaderREC( + readerOptions *cli.TReaderOptions, + recordsPerBatch int64, +) (*RecordReaderREC, error) { + return &RecordReaderREC{ + readerOptions: readerOptions, + recordsPerBatch: recordsPerBatch, + recordArena: mlrval.NewRecordArena(64), + }, nil +} + +func (reader *RecordReaderREC) Read( + filenames []string, + context types.Context, + readerChannel chan<- []*types.RecordAndContext, // list of *types.RecordAndContext + errorChannel chan error, + downstreamDoneChannel <-chan bool, // for mlr head +) { + if filenames != nil { // nil for mlr -n + if len(filenames) == 0 { // read from stdin + handle, err := lib.OpenStdin( + reader.readerOptions.Prepipe, + reader.readerOptions.PrepipeIsRaw, + reader.readerOptions.FileInputEncoding, + ) + if err != nil { + errorChannel <- err + } else { + reader.processHandle(handle, "(stdin)", &context, readerChannel, errorChannel, downstreamDoneChannel) + } + } else { + for _, filename := range filenames { + handle, err := lib.OpenFileForRead( + filename, + reader.readerOptions.Prepipe, + reader.readerOptions.PrepipeIsRaw, + reader.readerOptions.FileInputEncoding, + ) + if err != nil { + errorChannel <- err + } else { + reader.processHandle(handle, filename, &context, readerChannel, errorChannel, downstreamDoneChannel) + _ = handle.Close() + } + } + } + } + readerChannel <- types.NewEndOfStreamMarkerList(&context) +} + +func (reader *RecordReaderREC) processHandle( + handle io.Reader, + filename string, + context *types.Context, + readerChannel chan<- []*types.RecordAndContext, // list of *types.RecordAndContext + errorChannel chan error, + downstreamDoneChannel <-chan bool, // for mlr head +) { + context.UpdateForStartOfFile(filename) + recordsPerBatch := reader.recordsPerBatch + + // recutils records are blank-line-delimited stanzas, same boundary + // logic as XTAB; RS is fixed at "\n" (unlike XTAB, it's not + // configurable via IFS). + lineReader := NewLineReader(handle, "\n") + + stanzasChannel := make(chan []*tStanza, recordsPerBatch) + go channelizedStanzaScanner(lineReader, reader.readerOptions, stanzasChannel, downstreamDoneChannel, + recordsPerBatch) + + for { + recordsAndContexts, eof := reader.getRecordBatch(stanzasChannel, context, errorChannel) + if len(recordsAndContexts) > 0 { + readerChannel <- recordsAndContexts + } + if eof { + break + } + } +} + +func (reader *RecordReaderREC) getRecordBatch( + stanzasChannel <-chan []*tStanza, + context *types.Context, + errorChannel chan error, +) ( + recordsAndContexts []*types.RecordAndContext, + eof bool, +) { + recordsAndContexts = []*types.RecordAndContext{} + + stanzas, more := <-stanzasChannel + if !more { + return recordsAndContexts, true + } + + reader.recordArena = mlrval.NewRecordArena(len(stanzas) * 8) + + for _, stanza := range stanzas { + + if len(stanza.commentLines) > 0 { + for _, line := range stanza.commentLines { + recordsAndContexts = append(recordsAndContexts, types.NewOutputString(line+"\n", context)) + } + } + + if len(stanza.dataLines) > 0 { + record, err := reader.recordFromRECLines(stanza.dataLines) + if err != nil { + errorChannel <- err + return + } + context.UpdateForInputRecord() + recordsAndContexts = append(recordsAndContexts, types.NewRecordAndContext(record, context)) + } + } + + return recordsAndContexts, false +} + +// recordFromRECLines converts one blank-line-delimited stanza's raw lines +// into a Miller record. This happens in three passes, in this order: +// +// 1. Backslash-newline logical-line joining: a line ending in a literal "\" +// is joined directly (no separator inserted) with the next physical +// line. +// 2. "+"-continuation folding: a line starting with "+" continues the +// immediately preceding field's value with an embedded "\n"; a single +// leading space after the "+" is stripped. +// 3. Splitting each remaining logical line on the first occurrence of +// ": " into a field name and value. +// +// Malformed input (a "+"-continuation with no preceding field, or a line +// with no ": " separator) is a hard error -- recutils' own spec requires +// exactly ": " as the field separator, and there is no leniency flag for +// this format (unlike e.g. CSV's ragged-input handling). +func (reader *RecordReaderREC) recordFromRECLines( + stanza []string, +) (*mlrval.Mlrmap, error) { + record := reader.recordArena.NewRecord() + dedupeFieldNames := reader.readerOptions.DedupeFieldNames + + joinedLines := joinRECBackslashContinuations(stanza) + + fieldLines, err := foldRECPlusContinuations(joinedLines) + if err != nil { + return nil, err + } + + for _, line := range fieldLines { + key, value, found := strings.Cut(line, ": ") + if !found { + return nil, fmt.Errorf( + "mlr: recutils: missing \": \" field separator in line %q", + line, + ) + } + reader.recordArena.PutDeferred(record, key, value, dedupeFieldNames) + } + + return record, nil +} + +// joinRECBackslashContinuations implements recutils' backslash-newline +// logical-line continuation: a line whose last character is "\" is joined +// directly with the next physical line, with the backslash removed and no +// separator inserted. +func joinRECBackslashContinuations(lines []string) []string { + joined := make([]string, 0, len(lines)) + var pending strings.Builder + havePending := false + + for _, line := range lines { + if trimmed, ok := strings.CutSuffix(line, `\`); ok { + pending.WriteString(trimmed) + havePending = true + continue + } + if havePending { + pending.WriteString(line) + joined = append(joined, pending.String()) + pending.Reset() + havePending = false + } else { + joined = append(joined, line) + } + } + // A stanza-final line ending in "\" has nothing left to join with; keep + // its (backslash-stripped) content as-is. + if havePending { + joined = append(joined, pending.String()) + } + + return joined +} + +// foldRECPlusContinuations implements recutils' "+"-continuation: a line +// starting with "+" continues the immediately preceding field's value with +// an embedded "\n". A single leading space after the "+" is stripped (a +// bare "+" folds in an empty continuation line). +func foldRECPlusContinuations(lines []string) ([]string, error) { + folded := make([]string, 0, len(lines)) + + for _, line := range lines { + if strings.HasPrefix(line, "+") { + if len(folded) == 0 { + return nil, fmt.Errorf( + "mlr: recutils: continuation line %q has no preceding field in this record", + line, + ) + } + continuation := strings.TrimPrefix(line, "+") + continuation = strings.TrimPrefix(continuation, " ") + folded[len(folded)-1] = folded[len(folded)-1] + "\n" + continuation + } else { + folded = append(folded, line) + } + } + + return folded, nil +} diff --git a/pkg/input/record_reader_rec_test.go b/pkg/input/record_reader_rec_test.go new file mode 100644 index 000000000..b8c0d722a --- /dev/null +++ b/pkg/input/record_reader_rec_test.go @@ -0,0 +1,163 @@ +package input + +import ( + "testing" + + "github.com/stretchr/testify/assert" + + "github.com/johnkerl/miller/v6/pkg/cli" +) + +func newTestRECReader(t *testing.T) *RecordReaderREC { + readerOptions := cli.DefaultReaderOptions() + err := cli.FinalizeReaderOptions(&readerOptions) + assert.Nil(t, err) + reader, err := NewRecordReaderREC(&readerOptions, 1) + assert.NotNil(t, reader) + assert.Nil(t, err) + return reader +} + +func TestRecordFromRECLinesBasic(t *testing.T) { + reader := newTestRECReader(t) + + record, err := reader.recordFromRECLines([]string{ + "name: John Doe", + "email: jdoe@example.com", + }) + assert.Nil(t, err) + assert.NotNil(t, record) + assert.Equal(t, int64(2), record.FieldCount) + assert.Equal(t, "name", record.Head.Key) + assert.Equal(t, "John Doe", record.Head.Value.String()) + assert.Equal(t, "email", record.Head.Next.Key) + assert.Equal(t, "jdoe@example.com", record.Head.Next.Value.String()) +} + +func TestRecordFromRECLinesPlusContinuation(t *testing.T) { + reader := newTestRECReader(t) + + record, err := reader.recordFromRECLines([]string{ + "notes: line one", + "+ line two", + "+line three", + }) + assert.Nil(t, err) + assert.NotNil(t, record) + assert.Equal(t, int64(1), record.FieldCount) + assert.Equal(t, "notes", record.Head.Key) + assert.Equal(t, "line one\nline two\nline three", record.Head.Value.String()) +} + +func TestRecordFromRECLinesBarePlusContinuation(t *testing.T) { + reader := newTestRECReader(t) + + record, err := reader.recordFromRECLines([]string{ + "notes: line one", + "+", + "+ line three", + }) + assert.Nil(t, err) + assert.NotNil(t, record) + assert.Equal(t, "line one\n\nline three", record.Head.Value.String()) +} + +func TestRecordFromRECLinesBackslashContinuation(t *testing.T) { + reader := newTestRECReader(t) + + record, err := reader.recordFromRECLines([]string{ + `name: hello \`, + `world`, + }) + assert.Nil(t, err) + assert.NotNil(t, record) + assert.Equal(t, int64(1), record.FieldCount) + assert.Equal(t, "hello world", record.Head.Value.String()) +} + +func TestRecordFromRECLinesColonInValue(t *testing.T) { + reader := newTestRECReader(t) + + record, err := reader.recordFromRECLines([]string{ + "url: http://example.com: see notes", + }) + assert.Nil(t, err) + assert.NotNil(t, record) + assert.Equal(t, "url", record.Head.Key) + assert.Equal(t, "http://example.com: see notes", record.Head.Value.String()) +} + +func TestRecordFromRECLinesDedupeFieldNames(t *testing.T) { + reader := newTestRECReader(t) + + record, err := reader.recordFromRECLines([]string{ + "a: 1", + "b: 2", + "b: 3", + }) + assert.Nil(t, err) + assert.NotNil(t, record) + assert.Equal(t, int64(3), record.FieldCount) + assert.Equal(t, "a", record.Head.Key) + assert.Equal(t, "b", record.Head.Next.Key) + assert.Equal(t, "b_2", record.Head.Next.Next.Key) +} + +func TestRecordFromRECLinesOrphanPlusIsError(t *testing.T) { + reader := newTestRECReader(t) + + record, err := reader.recordFromRECLines([]string{ + "+ oops", + "name: x", + }) + assert.Nil(t, record) + assert.NotNil(t, err) +} + +func TestRecordFromRECLinesMissingColonSpaceIsError(t *testing.T) { + reader := newTestRECReader(t) + + record, err := reader.recordFromRECLines([]string{ + "this line has no colon-space separator", + }) + assert.Nil(t, record) + assert.NotNil(t, err) +} + +func TestRecordFromRECLinesMissingColonSpaceBareColonIsError(t *testing.T) { + reader := newTestRECReader(t) + + // Per the recutils spec, the separator is exactly ": " (colon-space) -- + // a bare colon with no following space does not count. + record, err := reader.recordFromRECLines([]string{ + "Foo:bar", + }) + assert.Nil(t, record) + assert.NotNil(t, err) +} + +func TestJoinRECBackslashContinuations(t *testing.T) { + assert.Equal(t, + []string{"ab"}, + joinRECBackslashContinuations([]string{`a\`, "b"}), + ) + assert.Equal(t, + []string{"a", "b"}, + joinRECBackslashContinuations([]string{"a", "b"}), + ) + // A trailing backslash with nothing left to join to keeps its + // (backslash-stripped) content as-is. + assert.Equal(t, + []string{"a"}, + joinRECBackslashContinuations([]string{`a\`}), + ) +} + +func TestFoldRECPlusContinuations(t *testing.T) { + folded, err := foldRECPlusContinuations([]string{"a: 1", "+ 2", "+3"}) + assert.Nil(t, err) + assert.Equal(t, []string{"a: 1\n2\n3"}, folded) + + _, err = foldRECPlusContinuations([]string{"+ orphan"}) + assert.NotNil(t, err) +} diff --git a/pkg/output/record_writer_factory.go b/pkg/output/record_writer_factory.go index 7530774c6..6e8ab44fc 100644 --- a/pkg/output/record_writer_factory.go +++ b/pkg/output/record_writer_factory.go @@ -24,6 +24,8 @@ func Create(writerOptions *cli.TWriterOptions) (IRecordWriter, error) { return NewRecordWriterYAML(writerOptions) case "dcf": return NewRecordWriterDCF(writerOptions) + case "recutils": + return NewRecordWriterREC(writerOptions) case "md": return NewRecordWriterMarkdown(writerOptions) case "markdown": diff --git a/pkg/output/record_writer_rec.go b/pkg/output/record_writer_rec.go new file mode 100644 index 000000000..ff0e9b98c --- /dev/null +++ b/pkg/output/record_writer_rec.go @@ -0,0 +1,68 @@ +package output + +import ( + "bufio" + "strings" + + "github.com/johnkerl/miller/v6/pkg/cli" + "github.com/johnkerl/miller/v6/pkg/colorizer" + "github.com/johnkerl/miller/v6/pkg/mlrval" + "github.com/johnkerl/miller/v6/pkg/types" +) + +// RecordWriterREC writes GNU recutils (.rec) format: "FieldName: Value" +// lines, one record per blank-line-separated stanza. Unlike DCF, recutils +// has no hardcoded list-valued field names -- array/map values are handled +// by Miller's generic auto-flatten mechanism upstream of this writer. +type RecordWriterREC struct { + writerOptions *cli.TWriterOptions +} + +func NewRecordWriterREC(writerOptions *cli.TWriterOptions) (*RecordWriterREC, error) { + return &RecordWriterREC{ + writerOptions: writerOptions, + }, nil +} + +func (writer *RecordWriterREC) Write( + outrec *mlrval.Mlrmap, + _ *types.Context, + bufferedOutputStream *bufio.Writer, + outputIsStdout bool, +) error { + if outrec == nil { + return nil + } + if outrec.IsEmpty() { + bufferedOutputStream.WriteString("\n") + return nil + } + + for pe := outrec.Head; pe != nil; pe = pe.Next { + valueStr := pe.Value.String() + keyStr := colorizer.MaybeColorizeKey(pe.Key, outputIsStdout) + valueStr = colorizer.MaybeColorizeValue(valueStr, outputIsStdout) + writeRECField(bufferedOutputStream, keyStr, valueStr) + } + bufferedOutputStream.WriteString("\n") + return nil +} + +// writeRECField writes one "Key: value" line, folding embedded newlines in +// the value into recutils continuation lines: each continuation line is +// prefixed with "+ ". +func writeRECField(b *bufio.Writer, key, value string) { + lines := strings.Split(value, "\n") + for i, line := range lines { + if i == 0 { + b.WriteString(key) + b.WriteString(": ") + b.WriteString(line) + b.WriteString("\n") + } else { + b.WriteString("+ ") + b.WriteString(line) + b.WriteString("\n") + } + } +} diff --git a/test/cases/io-recutils/0001/cmd b/test/cases/io-recutils/0001/cmd new file mode 100644 index 000000000..106814d4e --- /dev/null +++ b/test/cases/io-recutils/0001/cmd @@ -0,0 +1 @@ +mlr --itsv --rs lf --orecutils cat test/input/simple.tsv diff --git a/test/cases/io-recutils/0001/experr b/test/cases/io-recutils/0001/experr new file mode 100644 index 000000000..e69de29bb diff --git a/test/cases/io-recutils/0001/expout b/test/cases/io-recutils/0001/expout new file mode 100644 index 000000000..88e43f4a0 --- /dev/null +++ b/test/cases/io-recutils/0001/expout @@ -0,0 +1,30 @@ +a: pan +b: pan +i: 1 +x: 2 +y: 0.98994500 + +a: eks +b: pan +i: 2 +x: 1 +y: 0.77515900 + +a: wye +b: wye +i: 3 +x: 1 +y: 0.76164200 + +a: eks +b: wye +i: 4 +x: 5 +y: 0.32293400 + +a: wye +b: pan +i: 5 +x: 5 +y: 0.44828300 + diff --git a/test/cases/io-recutils/0002/cmd b/test/cases/io-recutils/0002/cmd new file mode 100644 index 000000000..04b1f340e --- /dev/null +++ b/test/cases/io-recutils/0002/cmd @@ -0,0 +1 @@ +mlr --skip-comments --irecutils --orecutils cat test/input/test.rec diff --git a/test/cases/io-recutils/0002/experr b/test/cases/io-recutils/0002/experr new file mode 100644 index 000000000..e69de29bb diff --git a/test/cases/io-recutils/0002/expout b/test/cases/io-recutils/0002/expout new file mode 100644 index 000000000..cfab53d67 --- /dev/null +++ b/test/cases/io-recutils/0002/expout @@ -0,0 +1,9 @@ +Package: foo +Version: 1.00000000 +Description: A test package ++ with a continued description. + +Package: bar +Version: 2.00000000 +Description: Another package. + diff --git a/test/cases/io-recutils/0003/cmd b/test/cases/io-recutils/0003/cmd new file mode 100644 index 000000000..30a21b80e --- /dev/null +++ b/test/cases/io-recutils/0003/cmd @@ -0,0 +1 @@ +mlr --skip-comments --irecutils --ojson cat test/input/test.rec diff --git a/test/cases/io-recutils/0003/experr b/test/cases/io-recutils/0003/experr new file mode 100644 index 000000000..e69de29bb diff --git a/test/cases/io-recutils/0003/expout b/test/cases/io-recutils/0003/expout new file mode 100644 index 000000000..449fce3be --- /dev/null +++ b/test/cases/io-recutils/0003/expout @@ -0,0 +1,12 @@ +[ +{ + "Package": "foo", + "Version": 1.00000000, + "Description": "A test package\nwith a continued description." +}, +{ + "Package": "bar", + "Version": 2.00000000, + "Description": "Another package." +} +] diff --git a/test/cases/io-recutils/0004/cmd b/test/cases/io-recutils/0004/cmd new file mode 100644 index 000000000..5eb5f9ae1 --- /dev/null +++ b/test/cases/io-recutils/0004/cmd @@ -0,0 +1 @@ +mlr --pass-comments --irecutils --ojson cat test/input/test.rec diff --git a/test/cases/io-recutils/0004/experr b/test/cases/io-recutils/0004/experr new file mode 100644 index 000000000..e69de29bb diff --git a/test/cases/io-recutils/0004/expout b/test/cases/io-recutils/0004/expout new file mode 100644 index 000000000..f4e9d3c88 --- /dev/null +++ b/test/cases/io-recutils/0004/expout @@ -0,0 +1,13 @@ +# Test fixture for recutils I/O +[ +{ + "Package": "foo", + "Version": 1.00000000, + "Description": "A test package\nwith a continued description." +}, +{ + "Package": "bar", + "Version": 2.00000000, + "Description": "Another package." +} +] diff --git a/test/cases/io-recutils/0005/cmd b/test/cases/io-recutils/0005/cmd new file mode 100644 index 000000000..634be7afb --- /dev/null +++ b/test/cases/io-recutils/0005/cmd @@ -0,0 +1 @@ +mlr --irecutils --ojson cat ${CASEDIR}/input diff --git a/test/cases/io-recutils/0005/experr b/test/cases/io-recutils/0005/experr new file mode 100644 index 000000000..5a598ac2c --- /dev/null +++ b/test/cases/io-recutils/0005/experr @@ -0,0 +1 @@ +mlr: recutils: continuation line "+ orphaned continuation with no preceding field in this record" has no preceding field in this record diff --git a/test/cases/io-recutils/0005/expout b/test/cases/io-recutils/0005/expout new file mode 100644 index 000000000..d25b98737 --- /dev/null +++ b/test/cases/io-recutils/0005/expout @@ -0,0 +1,5 @@ +[ +{ + "name": "ok" +} +] diff --git a/test/cases/io-recutils/0005/input b/test/cases/io-recutils/0005/input new file mode 100644 index 000000000..bc10a0897 --- /dev/null +++ b/test/cases/io-recutils/0005/input @@ -0,0 +1,3 @@ +name: ok + ++ orphaned continuation with no preceding field in this record diff --git a/test/cases/io-recutils/0005/should-fail b/test/cases/io-recutils/0005/should-fail new file mode 100644 index 000000000..e69de29bb diff --git a/test/input/test.rec b/test/input/test.rec new file mode 100644 index 000000000..7379e6882 --- /dev/null +++ b/test/input/test.rec @@ -0,0 +1,9 @@ +# Test fixture for recutils I/O +Package: foo +Version: 1.0 +Description: A test package ++ with a continued description. + +Package: bar +Version: 2.0 +Description: Another package.