From 6a38408bb3488945db59c5391f442bdcaf434f8d Mon Sep 17 00:00:00 2001 From: John Kerl Date: Wed, 15 Jul 2026 10:15:33 -0400 Subject: [PATCH] Treat YAML like JSON for auto-flatten/auto-unflatten (#2196) * Treat YAML like JSON for auto-flatten/auto-unflatten (#2195) Auto-flatten/unflatten decisions only checked for "json", so nested data written to YAML was needlessly flattened and non-JSON data converted to YAML was never unflattened into arrays/maps. Both formats support native nesting, so both should be treated the same way. Co-Authored-By: Claude Sonnet 5 * Clarify why DCF is excluded from auto-flatten separately from JSON/YAML DCF isn't a nestable format like JSON/YAML -- it just has its own hardcoded comma-list serialization for a fixed set of field names that generic key-spreading flatten would clobber. Split the comment so that distinction isn't lost. Co-Authored-By: Claude Sonnet 5 * Refer to JSON/YAML throughout the flatten-unflatten doc The summary paragraph calling out YAML as an aside did not read naturally next to prose that otherwise only said JSON. Fold YAML into the headings and body text directly instead. Co-Authored-By: Claude Sonnet 5 --------- Co-authored-by: Claude Sonnet 5 --- docs/src/flatten-unflatten.md | 39 ++++++++++++++-------------- docs/src/flatten-unflatten.md.in | 39 ++++++++++++++-------------- docs/src/manpage.md | 17 ++++++------ docs/src/manpage.txt | 17 ++++++------ docs/src/reference-main-flag-list.md | 6 ++--- man/manpage.txt | 17 ++++++------ man/mlr.1 | 19 +++++++------- pkg/cli/flatten_unflatten.go | 27 ++++++++++++++----- pkg/cli/option_parse.go | 6 ++--- test/cases/io-yaml-io/0005/cmd | 1 + test/cases/io-yaml-io/0005/experr | 0 test/cases/io-yaml-io/0005/expout | 8 ++++++ test/cases/io-yaml-io/0005/input | 3 +++ test/cases/io-yaml-io/0006/cmd | 1 + test/cases/io-yaml-io/0006/experr | 0 test/cases/io-yaml-io/0006/expout | 4 +++ test/cases/io-yaml-io/0006/input | 3 +++ test/cases/io-yaml-io/0007/cmd | 1 + test/cases/io-yaml-io/0007/experr | 0 test/cases/io-yaml-io/0007/expout | 3 +++ test/cases/io-yaml-io/0007/input | 8 ++++++ 21 files changed, 136 insertions(+), 83 deletions(-) create mode 100644 test/cases/io-yaml-io/0005/cmd create mode 100644 test/cases/io-yaml-io/0005/experr create mode 100644 test/cases/io-yaml-io/0005/expout create mode 100644 test/cases/io-yaml-io/0005/input create mode 100644 test/cases/io-yaml-io/0006/cmd create mode 100644 test/cases/io-yaml-io/0006/experr create mode 100644 test/cases/io-yaml-io/0006/expout create mode 100644 test/cases/io-yaml-io/0006/input create mode 100644 test/cases/io-yaml-io/0007/cmd create mode 100644 test/cases/io-yaml-io/0007/experr create mode 100644 test/cases/io-yaml-io/0007/expout create mode 100644 test/cases/io-yaml-io/0007/input diff --git a/docs/src/flatten-unflatten.md b/docs/src/flatten-unflatten.md index c0cb6fe6b..d876dea09 100644 --- a/docs/src/flatten-unflatten.md +++ b/docs/src/flatten-unflatten.md @@ -14,11 +14,11 @@ Quick links: Release docs -# Flatten/unflatten: converting between JSON and tabular formats +# Flatten/unflatten: converting between JSON/YAML and tabular formats Miller has long supported reading and writing multiple [file -formats](file-formats.md) including CSV and JSON, as well as converting back -and forth between them. Two things new in [Miller 6](new-in-miller-6-md), +formats](file-formats.md) including CSV, JSON, and YAML, as well as converting +back and forth between them. Two things new in [Miller 6](new-in-miller-6-md), though, are that [arrays are now fully supported](reference-main-arrays.md), and that [record values are typed](new-in-miller-6.md#improved-numeric-conversion) throughout Miller's processing chain from input through [verbs](reference-verbs.md) @@ -26,9 +26,10 @@ to output -- which includes improved handling for [maps](reference-main-maps.md) [arrays](reference-main-arrays.md) as record values. This raises the question, though, of how to handle maps and arrays as record values. -For [JSON files](file-formats.md#json), this is easy -- JSON is a nested format where values -can be maps or arrays, which can contain other maps or arrays, and so on, with the nesting -happily indicated by curly braces: +For [JSON](file-formats.md#json) or [YAML](file-formats.md#yaml) files, this is easy -- +both are nested formats where values can be maps or arrays, which can contain other maps +or arrays, and so on, with the nesting happily indicated by curly braces (JSON) or +indentation (YAML):
 cat data/map-values.json
@@ -60,11 +61,11 @@ happily indicated by curly braces:
 
 How can we represent these in CSV files?
 
-Miller's [non-JSON formats](file-formats.md), such as CSV, are all non-nested -- a
-cell in a CSV row can't contain another entire row. As we'll see in this
-section, there are two main ways to **flatten** nested data structures down to
-individual CSV cells -- either by _key-spreading_ (which is the default), or by
-_JSON-stringifying_:
+Miller's non-JSON/YAML [file formats](file-formats.md), such as CSV, are all
+non-nested -- a cell in a CSV row can't contain another entire row. As we'll
+see in this section, there are two main ways to **flatten** nested data
+structures down to individual CSV cells -- either by _key-spreading_ (which
+is the default), or by _JSON-stringifying_:
 
 * **Key-spreading** is when the single map-valued field
 `b={"x": 2, "y": 3}` spreads into multiple fields `b.x=2,b.y=3`;
@@ -73,7 +74,7 @@ _JSON-stringifying_:
 Miller intends to provide intuitive default behavior for these conversions, while also
 providing you with more control when you need it.
 
-## Converting maps between JSON and non-JSON
+## Converting maps between JSON/YAML and non-JSON/YAML
 
 Let's first look at the default behavior with map-valued fields. Miller's
 default behavior is to spread the map values into multiple keys -- using
@@ -152,7 +153,7 @@ a b.s.w b.s.x b.t.y b.t.z
 6 7     8     9     10
 
-**Unflattening** is simply the reverse -- from non-JSON back to JSON: +**Unflattening** is simply the reverse -- from non-JSON/YAML back to JSON or YAML:
 cat data/map-values.json
@@ -199,7 +200,7 @@ a,b.x,b.y
 ]
 
-## Converting arrays between JSON and non-JSON +## Converting arrays between JSON/YAML and non-JSON/YAML If the input data contains arrays, these are also flattened similarly: the [1-up array indices](reference-main-arrays.md#1-up-indexing) `1,2,3,...` become string keys @@ -257,7 +258,7 @@ In the nested-data examples shown here, nested map values are shown containing maps, and nested array values are shown containing arrays -- of course (even though not shown here) nested map values can contain arrays, and vice versa. -**Unflattening** arrays is, again, simply the reverse -- from non-JSON back to JSON: +**Unflattening** arrays is, again, simply the reverse -- from non-JSON/YAML back to JSON or YAML:
 cat data/array-values.json
@@ -352,7 +353,7 @@ a.1,a.3,a.5
 
 An additional heuristic is that if a field name starts with a `.`, ends with
 a `.`, or has two or more consecutive `.` characters, no attempt is made
-to unflatten it on conversion from non-JSON to JSON.
+to unflatten it on conversion from non-JSON/YAML to JSON or YAML.
 
 
 cat data/flatten-dots.csv
@@ -403,13 +404,13 @@ unflattening (if the defaults aren't working for us in a particular situation),
 let's first look a little into how they're implemented.
 
 * There are two [verbs](reference-verbs.md) called [flatten](reference-verbs.md#flatten) and [unflatten](reference-verbs.md#unflatten).
-* When the output format is not JSON, if you've specified `mlr ... cat then sort ...` (some [chain](reference-main-then-chaining.md) of verbs) then Miller appends, in effect, `then flatten` to the end of the chain.
+* When the output format is not JSON or YAML, if you've specified `mlr ... cat then sort ...` (some [chain](reference-main-then-chaining.md) of verbs) then Miller appends, in effect, `then flatten` to the end of the chain.
     * This behavior is on by default but it can be suppressed using the `--no-auto-flatten` [flag](reference-main-flag-list.md#flatten-unflatten-flags).
-* When the output format is JSON and the input format is not JSON, then (similarly) appends, in effect, `then unflatten`   to the end of the chain.
+* When the output format is JSON or YAML and the input format is neither, then (similarly) Miller appends, in effect, `then unflatten` to the end of the chain.
     * This behavior is on by default but it can be suppressed using the `--no-auto-unflatten` [flag](reference-main-flag-list.md#flatten-unflatten-flags).
 
 Note in particular that auto-flatten happens even when the input format and the
-output format are both non-JSON, e.g. even for CSV-to-CSV processing. This is
+output format are both non-JSON/non-YAML, e.g. even for CSV-to-CSV processing. This is
 because
 [map](reference-main-maps.md)-valued/[array](reference-main-arrays.md)-valued
 fields can be produced using [DSL statements](miller-programming-language.md):
diff --git a/docs/src/flatten-unflatten.md.in b/docs/src/flatten-unflatten.md.in
index 951ea1f58..f61d64aa3 100644
--- a/docs/src/flatten-unflatten.md.in
+++ b/docs/src/flatten-unflatten.md.in
@@ -1,8 +1,8 @@
-# Flatten/unflatten: converting between JSON and tabular formats
+# Flatten/unflatten: converting between JSON/YAML and tabular formats
 
 Miller has long supported reading and writing multiple [file
-formats](file-formats.md) including CSV and JSON, as well as converting back
-and forth between them. Two things new in [Miller 6](new-in-miller-6-md),
+formats](file-formats.md) including CSV, JSON, and YAML, as well as converting
+back and forth between them. Two things new in [Miller 6](new-in-miller-6-md),
 though, are that [arrays are now fully supported](reference-main-arrays.md),
 and that [record values are typed](new-in-miller-6.md#improved-numeric-conversion)
 throughout Miller's processing chain from input through [verbs](reference-verbs.md)
@@ -10,9 +10,10 @@ to output -- which includes improved handling for [maps](reference-main-maps.md)
 [arrays](reference-main-arrays.md) as record values.
 
 This raises the question, though, of how to handle maps and arrays as record values.
-For [JSON files](file-formats.md#json), this is easy -- JSON is a nested format where values
-can be maps or arrays, which can contain other maps or arrays, and so on, with the nesting
-happily indicated by curly braces:
+For [JSON](file-formats.md#json) or [YAML](file-formats.md#yaml) files, this is easy --
+both are nested formats where values can be maps or arrays, which can contain other maps
+or arrays, and so on, with the nesting happily indicated by curly braces (JSON) or
+indentation (YAML):
 
 GENMD-RUN-COMMAND
 cat data/map-values.json
@@ -24,11 +25,11 @@ GENMD-EOF
 
 How can we represent these in CSV files?
 
-Miller's [non-JSON formats](file-formats.md), such as CSV, are all non-nested -- a
-cell in a CSV row can't contain another entire row. As we'll see in this
-section, there are two main ways to **flatten** nested data structures down to
-individual CSV cells -- either by _key-spreading_ (which is the default), or by
-_JSON-stringifying_:
+Miller's non-JSON/YAML [file formats](file-formats.md), such as CSV, are all
+non-nested -- a cell in a CSV row can't contain another entire row. As we'll
+see in this section, there are two main ways to **flatten** nested data
+structures down to individual CSV cells -- either by _key-spreading_ (which
+is the default), or by _JSON-stringifying_:
 
 * **Key-spreading** is when the single map-valued field
 `b={"x": 2, "y": 3}` spreads into multiple fields `b.x=2,b.y=3`;
@@ -37,7 +38,7 @@ _JSON-stringifying_:
 Miller intends to provide intuitive default behavior for these conversions, while also
 providing you with more control when you need it.
 
-## Converting maps between JSON and non-JSON
+## Converting maps between JSON/YAML and non-JSON/YAML
 
 Let's first look at the default behavior with map-valued fields. Miller's
 default behavior is to spread the map values into multiple keys -- using
@@ -76,7 +77,7 @@ GENMD-RUN-COMMAND
 mlr --ijson --opprint cat data/map-values-nested.json
 GENMD-EOF
 
-**Unflattening** is simply the reverse -- from non-JSON back to JSON:
+**Unflattening** is simply the reverse -- from non-JSON/YAML back to JSON or YAML:
 
 GENMD-RUN-COMMAND
 cat data/map-values.json
@@ -90,7 +91,7 @@ GENMD-RUN-COMMAND
 mlr --ijson --ocsv cat data/map-values.json | mlr --icsv --ojson cat
 GENMD-EOF
 
-## Converting arrays between JSON and non-JSON
+## Converting arrays between JSON/YAML and non-JSON/YAML
 
 If the input data contains arrays, these are also flattened similarly: the
 [1-up array indices](reference-main-arrays.md#1-up-indexing) `1,2,3,...` become string keys
@@ -118,7 +119,7 @@ In the nested-data examples shown here, nested map values are shown containing
 maps, and nested array values are shown containing arrays -- of course (even
 though not shown here) nested map values can contain arrays, and vice versa.
 
-**Unflattening** arrays is, again, simply the reverse -- from non-JSON back to JSON:
+**Unflattening** arrays is, again, simply the reverse -- from non-JSON/YAML back to JSON or YAML:
 
 GENMD-RUN-COMMAND
 cat data/array-values.json
@@ -160,7 +161,7 @@ GENMD-EOF
 
 An additional heuristic is that if a field name starts with a `.`, ends with
 a `.`, or has two or more consecutive `.` characters, no attempt is made
-to unflatten it on conversion from non-JSON to JSON.
+to unflatten it on conversion from non-JSON/YAML to JSON or YAML.
 
 GENMD-RUN-COMMAND
 cat data/flatten-dots.csv
@@ -181,13 +182,13 @@ unflattening (if the defaults aren't working for us in a particular situation),
 let's first look a little into how they're implemented.
 
 * There are two [verbs](reference-verbs.md) called [flatten](reference-verbs.md#flatten) and [unflatten](reference-verbs.md#unflatten).
-* When the output format is not JSON, if you've specified `mlr ... cat then sort ...` (some [chain](reference-main-then-chaining.md) of verbs) then Miller appends, in effect, `then flatten` to the end of the chain.
+* When the output format is not JSON or YAML, if you've specified `mlr ... cat then sort ...` (some [chain](reference-main-then-chaining.md) of verbs) then Miller appends, in effect, `then flatten` to the end of the chain.
     * This behavior is on by default but it can be suppressed using the `--no-auto-flatten` [flag](reference-main-flag-list.md#flatten-unflatten-flags).
-* When the output format is JSON and the input format is not JSON, then (similarly) appends, in effect, `then unflatten`   to the end of the chain.
+* When the output format is JSON or YAML and the input format is neither, then (similarly) Miller appends, in effect, `then unflatten` to the end of the chain.
     * This behavior is on by default but it can be suppressed using the `--no-auto-unflatten` [flag](reference-main-flag-list.md#flatten-unflatten-flags).
 
 Note in particular that auto-flatten happens even when the input format and the
-output format are both non-JSON, e.g. even for CSV-to-CSV processing. This is
+output format are both non-JSON/non-YAML, e.g. even for CSV-to-CSV processing. This is
 because
 [map](reference-main-maps.md)-valued/[array](reference-main-arrays.md)-valued
 fields can be produced using [DSL statements](miller-programming-language.md):
diff --git a/docs/src/manpage.md b/docs/src/manpage.md
index 5c216d509..0b643ea59 100644
--- a/docs/src/manpage.md
+++ b/docs/src/manpage.md
@@ -479,7 +479,7 @@ This is simply a copy of what you should see on running `man mlr` at a command p
                                 csv` is the same as `--ocsv`.
 
 1mFLATTEN-UNFLATTEN FLAGS0m
-       These flags control how Miller converts record values which are maps or arrays, when input is JSON and output is non-JSON (flattening) or input is non-JSON and output is JSON (unflattening).
+       These flags control how Miller converts record values which are maps or arrays, when input is JSON/YAML and output is not (flattening) or input is not JSON/YAML and output is JSON/YAML (unflattening).
 
        See the flatten/unflatten doc page https://miller.readthedocs.io/en/latest/flatten-unflatten for more information.
 
@@ -487,16 +487,17 @@ This is simply a copy of what you should see on running `man mlr` at a command p
                                 Separator for flattening multi-level JSON keys, e.g.
                                 `{"a":{"b":3}}` becomes `a:b => 3` for non-JSON
                                 formats. Defaults to `.`.
-       --no-auto-flatten        When output is non-JSON, suppress the default
+       --no-auto-flatten        When output is not JSON or YAML, suppress the default
                                 auto-flatten behavior. Default: if `$y = [7,8,9]`
                                 then this flattens to `y.1=7,y.2=8,y.3=9`, and
                                 similarly for maps. With `--no-auto-flatten`, instead
                                 we get `$y=[1, 2, 3]`.
-       --no-auto-unflatten      When input is non-JSON and output is JSON, suppress
-                                the default auto-unflatten behavior. Default: if the
-                                input has `y.1=7,y.2=8,y.3=9` then this unflattens to
-                                `$y=[7,8,9]`. With `--no-auto-flatten`, instead we
-                                get `${y.1}=7,${y.2}=8,${y.3}=9`.
+       --no-auto-unflatten      When input is not JSON or YAML and output is JSON or
+                                YAML, suppress the default auto-unflatten behavior.
+                                Default: if the input has `y.1=7,y.2=8,y.3=9` then
+                                this unflattens to `$y=[7,8,9]`. With
+                                `--no-auto-flatten`, instead we get
+                                `${y.1}=7,${y.2}=8,${y.3}=9`.
 
 1mFORMAT-CONVERSION KEYSTROKE-SAVER FLAGS0m
        As keystroke-savers for format-conversion you may use the following.
@@ -4159,5 +4160,5 @@ This is simply a copy of what you should see on running `man mlr` at a command p
        MIME Type for Comma-Separated Values (CSV) Files, the Miller docsite
        https://miller.readthedocs.io
 
-                                  2026-07-14                         4mMILLER24m(1)
+                                  2026-07-15                         4mMILLER24m(1)
 
diff --git a/docs/src/manpage.txt b/docs/src/manpage.txt index b235fea97..5cde2e3fb 100644 --- a/docs/src/manpage.txt +++ b/docs/src/manpage.txt @@ -458,7 +458,7 @@ csv` is the same as `--ocsv`. 1mFLATTEN-UNFLATTEN FLAGS0m - These flags control how Miller converts record values which are maps or arrays, when input is JSON and output is non-JSON (flattening) or input is non-JSON and output is JSON (unflattening). + These flags control how Miller converts record values which are maps or arrays, when input is JSON/YAML and output is not (flattening) or input is not JSON/YAML and output is JSON/YAML (unflattening). See the flatten/unflatten doc page https://miller.readthedocs.io/en/latest/flatten-unflatten for more information. @@ -466,16 +466,17 @@ Separator for flattening multi-level JSON keys, e.g. `{"a":{"b":3}}` becomes `a:b => 3` for non-JSON formats. Defaults to `.`. - --no-auto-flatten When output is non-JSON, suppress the default + --no-auto-flatten When output is not JSON or YAML, suppress the default auto-flatten behavior. Default: if `$y = [7,8,9]` then this flattens to `y.1=7,y.2=8,y.3=9`, and similarly for maps. With `--no-auto-flatten`, instead we get `$y=[1, 2, 3]`. - --no-auto-unflatten When input is non-JSON and output is JSON, suppress - the default auto-unflatten behavior. Default: if the - input has `y.1=7,y.2=8,y.3=9` then this unflattens to - `$y=[7,8,9]`. With `--no-auto-flatten`, instead we - get `${y.1}=7,${y.2}=8,${y.3}=9`. + --no-auto-unflatten When input is not JSON or YAML and output is JSON or + YAML, suppress the default auto-unflatten behavior. + Default: if the input has `y.1=7,y.2=8,y.3=9` then + this unflattens to `$y=[7,8,9]`. With + `--no-auto-flatten`, instead we get + `${y.1}=7,${y.2}=8,${y.3}=9`. 1mFORMAT-CONVERSION KEYSTROKE-SAVER FLAGS0m As keystroke-savers for format-conversion you may use the following. @@ -4138,4 +4139,4 @@ MIME Type for Comma-Separated Values (CSV) Files, the Miller docsite https://miller.readthedocs.io - 2026-07-14 4mMILLER24m(1) + 2026-07-15 4mMILLER24m(1) diff --git a/docs/src/reference-main-flag-list.md b/docs/src/reference-main-flag-list.md index d9b09cf0a..17b0de7df 100644 --- a/docs/src/reference-main-flag-list.md +++ b/docs/src/reference-main-flag-list.md @@ -210,7 +210,7 @@ are overridden in all cases by setting output format to `format2`. ## Flatten-unflatten flags -These flags control how Miller converts record values which are maps or arrays, when input is JSON and output is non-JSON (flattening) or input is non-JSON and output is JSON (unflattening). +These flags control how Miller converts record values which are maps or arrays, when input is JSON/YAML and output is not (flattening) or input is not JSON/YAML and output is JSON/YAML (unflattening). See the flatten/unflatten doc page https://miller.readthedocs.io/en/latest/flatten-unflatten for more information. @@ -218,8 +218,8 @@ See the flatten/unflatten doc page https://miller.readthedocs.io/en/latest/flatt **Flags:** * `--flatsep or --jflatsep {string}`: Separator for flattening multi-level JSON keys, e.g. `{"a":{"b":3}}` becomes `a:b => 3` for non-JSON formats. Defaults to `.`. -* `--no-auto-flatten`: When output is non-JSON, suppress the default auto-flatten behavior. Default: if `$y = [7,8,9]` then this flattens to `y.1=7,y.2=8,y.3=9`, and similarly for maps. With `--no-auto-flatten`, instead we get `$y=[1, 2, 3]`. -* `--no-auto-unflatten`: When input is non-JSON and output is JSON, suppress the default auto-unflatten behavior. Default: if the input has `y.1=7,y.2=8,y.3=9` then this unflattens to `$y=[7,8,9]`. With `--no-auto-flatten`, instead we get `${y.1}=7,${y.2}=8,${y.3}=9`. +* `--no-auto-flatten`: When output is not JSON or YAML, suppress the default auto-flatten behavior. Default: if `$y = [7,8,9]` then this flattens to `y.1=7,y.2=8,y.3=9`, and similarly for maps. With `--no-auto-flatten`, instead we get `$y=[1, 2, 3]`. +* `--no-auto-unflatten`: When input is not JSON or YAML and output is JSON or YAML, suppress the default auto-unflatten behavior. Default: if the input has `y.1=7,y.2=8,y.3=9` then this unflattens to `$y=[7,8,9]`. With `--no-auto-flatten`, instead we get `${y.1}=7,${y.2}=8,${y.3}=9`. ## Format-conversion keystroke-saver flags diff --git a/man/manpage.txt b/man/manpage.txt index b235fea97..5cde2e3fb 100644 --- a/man/manpage.txt +++ b/man/manpage.txt @@ -458,7 +458,7 @@ csv` is the same as `--ocsv`. 1mFLATTEN-UNFLATTEN FLAGS0m - These flags control how Miller converts record values which are maps or arrays, when input is JSON and output is non-JSON (flattening) or input is non-JSON and output is JSON (unflattening). + These flags control how Miller converts record values which are maps or arrays, when input is JSON/YAML and output is not (flattening) or input is not JSON/YAML and output is JSON/YAML (unflattening). See the flatten/unflatten doc page https://miller.readthedocs.io/en/latest/flatten-unflatten for more information. @@ -466,16 +466,17 @@ Separator for flattening multi-level JSON keys, e.g. `{"a":{"b":3}}` becomes `a:b => 3` for non-JSON formats. Defaults to `.`. - --no-auto-flatten When output is non-JSON, suppress the default + --no-auto-flatten When output is not JSON or YAML, suppress the default auto-flatten behavior. Default: if `$y = [7,8,9]` then this flattens to `y.1=7,y.2=8,y.3=9`, and similarly for maps. With `--no-auto-flatten`, instead we get `$y=[1, 2, 3]`. - --no-auto-unflatten When input is non-JSON and output is JSON, suppress - the default auto-unflatten behavior. Default: if the - input has `y.1=7,y.2=8,y.3=9` then this unflattens to - `$y=[7,8,9]`. With `--no-auto-flatten`, instead we - get `${y.1}=7,${y.2}=8,${y.3}=9`. + --no-auto-unflatten When input is not JSON or YAML and output is JSON or + YAML, suppress the default auto-unflatten behavior. + Default: if the input has `y.1=7,y.2=8,y.3=9` then + this unflattens to `$y=[7,8,9]`. With + `--no-auto-flatten`, instead we get + `${y.1}=7,${y.2}=8,${y.3}=9`. 1mFORMAT-CONVERSION KEYSTROKE-SAVER FLAGS0m As keystroke-savers for format-conversion you may use the following. @@ -4138,4 +4139,4 @@ MIME Type for Comma-Separated Values (CSV) Files, the Miller docsite https://miller.readthedocs.io - 2026-07-14 4mMILLER24m(1) + 2026-07-15 4mMILLER24m(1) diff --git a/man/mlr.1 b/man/mlr.1 index 3c5b28c52..a7b882e2a 100644 --- a/man/mlr.1 +++ b/man/mlr.1 @@ -2,12 +2,12 @@ .\" Title: mlr .\" Author: [see the "AUTHOR" section] .\" Generator: ./mkman.rb -.\" Date: 2026-07-14 +.\" Date: 2026-07-15 .\" Manual: \ \& .\" Source: \ \& .\" Language: English .\" -.TH "MILLER" "1" "2026-07-14" "\ \&" "\ \&" +.TH "MILLER" "1" "2026-07-15" "\ \&" "\ \&" .\" ----------------------------------------------------------------- .\" * Portability definitions .\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -554,7 +554,7 @@ are overridden in all cases by setting output format to `format2`. .RS 0 .\} .nf -These flags control how Miller converts record values which are maps or arrays, when input is JSON and output is non-JSON (flattening) or input is non-JSON and output is JSON (unflattening). +These flags control how Miller converts record values which are maps or arrays, when input is JSON/YAML and output is not (flattening) or input is not JSON/YAML and output is JSON/YAML (unflattening). See the flatten/unflatten doc page https://miller.readthedocs.io/en/latest/flatten-unflatten for more information. @@ -562,16 +562,17 @@ See the flatten/unflatten doc page https://miller.readthedocs.io/en/latest/flatt Separator for flattening multi-level JSON keys, e.g. `{"a":{"b":3}}` becomes `a:b => 3` for non-JSON formats. Defaults to `.`. ---no-auto-flatten When output is non-JSON, suppress the default +--no-auto-flatten When output is not JSON or YAML, suppress the default auto-flatten behavior. Default: if `$y = [7,8,9]` then this flattens to `y.1=7,y.2=8,y.3=9`, and similarly for maps. With `--no-auto-flatten`, instead we get `$y=[1, 2, 3]`. ---no-auto-unflatten When input is non-JSON and output is JSON, suppress - the default auto-unflatten behavior. Default: if the - input has `y.1=7,y.2=8,y.3=9` then this unflattens to - `$y=[7,8,9]`. With `--no-auto-flatten`, instead we - get `${y.1}=7,${y.2}=8,${y.3}=9`. +--no-auto-unflatten When input is not JSON or YAML and output is JSON or + YAML, suppress the default auto-unflatten behavior. + Default: if the input has `y.1=7,y.2=8,y.3=9` then + this unflattens to `$y=[7,8,9]`. With + `--no-auto-flatten`, instead we get + `${y.1}=7,${y.2}=8,${y.3}=9`. .fi .if n \{\ .RE diff --git a/pkg/cli/flatten_unflatten.go b/pkg/cli/flatten_unflatten.go index bd2593f2a..d256494bc 100644 --- a/pkg/cli/flatten_unflatten.go +++ b/pkg/cli/flatten_unflatten.go @@ -5,7 +5,7 @@ package cli // // PROBLEM TO BE SOLVED: // -// JSON has nested structures and CSV et al. do not. For example: +// JSON and YAML have nested structures and CSV et al. do not. For example: // { // "req" : { // "method": "GET", @@ -22,7 +22,9 @@ package cli // // APPROACH: // -// Use the Principle of Least Surprise (POLS). +// Use the Principle of Least Surprise (POLS). Below, "JSON" stands for any +// format capable of representing nested structures natively -- currently +// JSON, JSON Lines, and YAML. // // * If input is JSON and output is JSON: // o Records can be nested from record-read @@ -54,11 +56,24 @@ package cli // flatten, don't undo that by putting an unflatten right after. // +// isNestable returns true for formats which can represent nested/array +// structures natively, and thus don't need auto-flatten/auto-unflatten. +func isNestable(format string) bool { + return format == "json" || format == "jsonl" || format == "yaml" +} + func DecideFinalFlatten(writerOptions *TWriterOptions) bool { ofmt := writerOptions.OutputFileFormat if writerOptions.AutoFlatten { - // Preserve nested/array structure for formats that support it. - if ofmt != "json" && ofmt != "jsonl" && ofmt != "dcf" { + // JSON/YAML/JSON-Lines preserve nested/array structure natively, so + // they never need flattening. + // + // DCF is excluded for a different reason: it's not nestable, but it + // has its own hardcoded comma-list serialization for a fixed set of + // field names (Depends, Recommends, etc. -- see + // pkg/output/record_writer_dcf.go), which generic key-spreading + // flatten would clobber. + if !isNestable(ofmt) && ofmt != "dcf" { return true } } @@ -85,8 +100,8 @@ func DecideFinalUnflatten( ofmt := options.WriterOptions.OutputFileFormat if options.WriterOptions.AutoUnflatten { - if ifmt != "json" { - if ofmt == "json" { + if !isNestable(ifmt) { + if isNestable(ofmt) { return true } } diff --git a/pkg/cli/option_parse.go b/pkg/cli/option_parse.go index 08b707ff7..5cf197194 100644 --- a/pkg/cli/option_parse.go +++ b/pkg/cli/option_parse.go @@ -3300,7 +3300,7 @@ var OutputColorizationFlagSection = FlagSection{ // FLATTEN/UNFLATTEN FLAGS func FlattenUnflattenPrintInfo() { - fmt.Println("These flags control how Miller converts record values which are maps or arrays, when input is JSON and output is non-JSON (flattening) or input is non-JSON and output is JSON (unflattening).") + fmt.Println("These flags control how Miller converts record values which are maps or arrays, when input is JSON/YAML and output is not (flattening) or input is not JSON/YAML and output is JSON/YAML (unflattening).") fmt.Println() fmt.Println("See the flatten/unflatten doc page https://miller.readthedocs.io/en/latest/flatten-unflatten for more information.") } @@ -3326,7 +3326,7 @@ var FlattenUnflattenFlagSection = FlagSection{ { name: "--no-auto-flatten", - help: "When output is non-JSON, suppress the default auto-flatten behavior. Default: if `$y = [7,8,9]` then this flattens to `y.1=7,y.2=8,y.3=9`, and similarly for maps. With `--no-auto-flatten`, instead we get `$y=[1, 2, 3]`.", + help: "When output is not JSON or YAML, suppress the default auto-flatten behavior. Default: if `$y = [7,8,9]` then this flattens to `y.1=7,y.2=8,y.3=9`, and similarly for maps. With `--no-auto-flatten`, instead we get `$y=[1, 2, 3]`.", parser: func(args []string, argc int, pargi *int, options *TOptions) { options.WriterOptions.AutoFlatten = false *pargi += 1 @@ -3335,7 +3335,7 @@ var FlattenUnflattenFlagSection = FlagSection{ { name: "--no-auto-unflatten", - help: "When input is non-JSON and output is JSON, suppress the default auto-unflatten behavior. Default: if the input has `y.1=7,y.2=8,y.3=9` then this unflattens to `$y=[7,8,9]`. With `--no-auto-flatten`, instead we get `${y.1}=7,${y.2}=8,${y.3}=9`.", + help: "When input is not JSON or YAML and output is JSON or YAML, suppress the default auto-unflatten behavior. Default: if the input has `y.1=7,y.2=8,y.3=9` then this unflattens to `$y=[7,8,9]`. With `--no-auto-flatten`, instead we get `${y.1}=7,${y.2}=8,${y.3}=9`.", parser: func(args []string, argc int, pargi *int, options *TOptions) { options.WriterOptions.AutoUnflatten = false *pargi += 1 diff --git a/test/cases/io-yaml-io/0005/cmd b/test/cases/io-yaml-io/0005/cmd new file mode 100644 index 000000000..84ada0c8e --- /dev/null +++ b/test/cases/io-yaml-io/0005/cmd @@ -0,0 +1 @@ +mlr --icsv --oyaml cat ${CASEDIR}/input diff --git a/test/cases/io-yaml-io/0005/experr b/test/cases/io-yaml-io/0005/experr new file mode 100644 index 000000000..e69de29bb diff --git a/test/cases/io-yaml-io/0005/expout b/test/cases/io-yaml-io/0005/expout new file mode 100644 index 000000000..c40c130f9 --- /dev/null +++ b/test/cases/io-yaml-io/0005/expout @@ -0,0 +1,8 @@ +- a: 1 + b: + - 2 + - 3 +- a: 4 + b: + - 5 + - 6 diff --git a/test/cases/io-yaml-io/0005/input b/test/cases/io-yaml-io/0005/input new file mode 100644 index 000000000..d17ad1a6d --- /dev/null +++ b/test/cases/io-yaml-io/0005/input @@ -0,0 +1,3 @@ +a,b.1,b.2 +1,2,3 +4,5,6 diff --git a/test/cases/io-yaml-io/0006/cmd b/test/cases/io-yaml-io/0006/cmd new file mode 100644 index 000000000..5c51bf61a --- /dev/null +++ b/test/cases/io-yaml-io/0006/cmd @@ -0,0 +1 @@ +mlr --ijson --oyaml cat ${CASEDIR}/input diff --git a/test/cases/io-yaml-io/0006/experr b/test/cases/io-yaml-io/0006/experr new file mode 100644 index 000000000..e69de29bb diff --git a/test/cases/io-yaml-io/0006/expout b/test/cases/io-yaml-io/0006/expout new file mode 100644 index 000000000..9aeb886e1 --- /dev/null +++ b/test/cases/io-yaml-io/0006/expout @@ -0,0 +1,4 @@ +- a: 1 + req: + method: GET + path: api/check diff --git a/test/cases/io-yaml-io/0006/input b/test/cases/io-yaml-io/0006/input new file mode 100644 index 000000000..a9c61b51a --- /dev/null +++ b/test/cases/io-yaml-io/0006/input @@ -0,0 +1,3 @@ +[ +{"a": 1, "req": {"method": "GET", "path": "api/check"}} +] diff --git a/test/cases/io-yaml-io/0007/cmd b/test/cases/io-yaml-io/0007/cmd new file mode 100644 index 000000000..6fd32a30b --- /dev/null +++ b/test/cases/io-yaml-io/0007/cmd @@ -0,0 +1 @@ +mlr --iyaml --ocsv cat ${CASEDIR}/input diff --git a/test/cases/io-yaml-io/0007/experr b/test/cases/io-yaml-io/0007/experr new file mode 100644 index 000000000..e69de29bb diff --git a/test/cases/io-yaml-io/0007/expout b/test/cases/io-yaml-io/0007/expout new file mode 100644 index 000000000..d17ad1a6d --- /dev/null +++ b/test/cases/io-yaml-io/0007/expout @@ -0,0 +1,3 @@ +a,b.1,b.2 +1,2,3 +4,5,6 diff --git a/test/cases/io-yaml-io/0007/input b/test/cases/io-yaml-io/0007/input new file mode 100644 index 000000000..c40c130f9 --- /dev/null +++ b/test/cases/io-yaml-io/0007/input @@ -0,0 +1,8 @@ +- a: 1 + b: + - 2 + - 3 +- a: 4 + b: + - 5 + - 6