Mirrors/miller
1
0
Fork 0
mirror of https://github.com/johnkerl/miller.git synced 2026-07-21 02:23:20 +00:00
Code Issues Projects Releases Wiki Activity

Treat YAML like JSON for auto-flatten/auto-unflatten (#2195)

Browse source 
Auto-flatten/unflatten decisions only checked for "json", so nested
data written to YAML was needlessly flattened and non-JSON data
converted to YAML was never unflattened into arrays/maps. Both formats
support native nesting, so both should be treated the same way.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
John Kerl 2026-07-15 09:39:07 -04:00
parent 4c2625a5fe
commit 93d385697c
21 changed files with 106 additions and 50 deletions
Download patch file Download diff file Expand all files Collapse all files

12
docs/src/flatten-unflatten.md
View file

@ -73,6 +73,12 @@ _JSON-stringifying_:
Miller intends to provide intuitive default behavior for these conversions, while also
providing you with more control when you need it.
Everything below is written in terms of JSON, but it applies equally to
[YAML](file-formats.md#yaml) -- another nested format where values can be
maps or arrays. Auto-flattening happens whenever the output format is
neither JSON nor YAML; auto-unflattening happens whenever the input format
is neither JSON nor YAML but the output format is one or the other.
## Converting maps between JSON and non-JSON
Let's first look at the default behavior with map-valued fields. Miller's
@ -403,13 +409,13 @@ unflattening (if the defaults aren't working for us in a particular situation),
let's first look a little into how they're implemented.
* There are two [verbs](reference-verbs.md) called [flatten](reference-verbs.md#flatten) and [unflatten](reference-verbs.md#unflatten).
* When the output format is not JSON, if you've specified `mlr ... cat then sort ...` (some [chain](reference-main-then-chaining.md) of verbs) then Miller appends, in effect, `then flatten` to the end of the chain.
* When the output format is not JSON or YAML, if you've specified `mlr ... cat then sort ...` (some [chain](reference-main-then-chaining.md) of verbs) then Miller appends, in effect, `then flatten` to the end of the chain.
* This behavior is on by default but it can be suppressed using the `--no-auto-flatten` [flag](reference-main-flag-list.md#flatten-unflatten-flags).
* When the output format is JSON and the input format is not JSON, then (similarly) appends, in effect, `then unflatten` to the end of the chain.
* When the output format is JSON or YAML and the input format is neither, then (similarly) Miller appends, in effect, `then unflatten` to the end of the chain.
* This behavior is on by default but it can be suppressed using the `--no-auto-unflatten` [flag](reference-main-flag-list.md#flatten-unflatten-flags).
Note in particular that auto-flatten happens even when the input format and the
output format are both non-JSON, e.g. even for CSV-to-CSV processing. This is
output format are both non-JSON/non-YAML, e.g. even for CSV-to-CSV processing. This is
because
[map](reference-main-maps.md)-valued/[array](reference-main-arrays.md)-valued
fields can be produced using [DSL statements](miller-programming-language.md):

12
docs/src/flatten-unflatten.md.in
View file

@ -37,6 +37,12 @@ _JSON-stringifying_:
Miller intends to provide intuitive default behavior for these conversions, while also
providing you with more control when you need it.
Everything below is written in terms of JSON, but it applies equally to
[YAML](file-formats.md#yaml) -- another nested format where values can be
maps or arrays. Auto-flattening happens whenever the output format is
neither JSON nor YAML; auto-unflattening happens whenever the input format
is neither JSON nor YAML but the output format is one or the other.
## Converting maps between JSON and non-JSON
Let's first look at the default behavior with map-valued fields. Miller's
@ -181,13 +187,13 @@ unflattening (if the defaults aren't working for us in a particular situation),
let's first look a little into how they're implemented.
* There are two [verbs](reference-verbs.md) called [flatten](reference-verbs.md#flatten) and [unflatten](reference-verbs.md#unflatten).
* When the output format is not JSON, if you've specified `mlr ... cat then sort ...` (some [chain](reference-main-then-chaining.md) of verbs) then Miller appends, in effect, `then flatten` to the end of the chain.
* When the output format is not JSON or YAML, if you've specified `mlr ... cat then sort ...` (some [chain](reference-main-then-chaining.md) of verbs) then Miller appends, in effect, `then flatten` to the end of the chain.
* This behavior is on by default but it can be suppressed using the `--no-auto-flatten` [flag](reference-main-flag-list.md#flatten-unflatten-flags).
* When the output format is JSON and the input format is not JSON, then (similarly) appends, in effect, `then unflatten` to the end of the chain.
* When the output format is JSON or YAML and the input format is neither, then (similarly) Miller appends, in effect, `then unflatten` to the end of the chain.
* This behavior is on by default but it can be suppressed using the `--no-auto-unflatten` [flag](reference-main-flag-list.md#flatten-unflatten-flags).
Note in particular that auto-flatten happens even when the input format and the
output format are both non-JSON, e.g. even for CSV-to-CSV processing. This is
output format are both non-JSON/non-YAML, e.g. even for CSV-to-CSV processing. This is
because
[map](reference-main-maps.md)-valued/[array](reference-main-arrays.md)-valued
fields can be produced using [DSL statements](miller-programming-language.md):

17
docs/src/manpage.md
View file

@ -479,7 +479,7 @@ This is simply a copy of what you should see on running `man mlr` at a command p
csv` is the same as `--ocsv`.
1mFLATTEN-UNFLATTEN FLAGS0m
These flags control how Miller converts record values which are maps or arrays, when input is JSON and output is non-JSON (flattening) or input is non-JSON and output is JSON (unflattening).
These flags control how Miller converts record values which are maps or arrays, when input is JSON/YAML and output is not (flattening) or input is not JSON/YAML and output is JSON/YAML (unflattening).
See the flatten/unflatten doc page https://miller.readthedocs.io/en/latest/flatten-unflatten for more information.
@ -487,16 +487,17 @@ This is simply a copy of what you should see on running `man mlr` at a command p
Separator for flattening multi-level JSON keys, e.g.
`{"a":{"b":3}}` becomes `a:b =&gt; 3` for non-JSON
formats. Defaults to `.`.
--no-auto-flatten When output is non-JSON, suppress the default
--no-auto-flatten When output is not JSON or YAML, suppress the default
auto-flatten behavior. Default: if `$y = [7,8,9]`
then this flattens to `y.1=7,y.2=8,y.3=9`, and
similarly for maps. With `--no-auto-flatten`, instead
we get `$y=[1, 2, 3]`.
--no-auto-unflatten When input is non-JSON and output is JSON, suppress
the default auto-unflatten behavior. Default: if the
input has `y.1=7,y.2=8,y.3=9` then this unflattens to
`$y=[7,8,9]`. With `--no-auto-flatten`, instead we
get `${y.1}=7,${y.2}=8,${y.3}=9`.
--no-auto-unflatten When input is not JSON or YAML and output is JSON or
YAML, suppress the default auto-unflatten behavior.
Default: if the input has `y.1=7,y.2=8,y.3=9` then
this unflattens to `$y=[7,8,9]`. With
`--no-auto-flatten`, instead we get
`${y.1}=7,${y.2}=8,${y.3}=9`.
1mFORMAT-CONVERSION KEYSTROKE-SAVER FLAGS0m
As keystroke-savers for format-conversion you may use the following.
@ -4159,5 +4160,5 @@ This is simply a copy of what you should see on running `man mlr` at a command p
MIME Type for Comma-Separated Values (CSV) Files, the Miller docsite
https://miller.readthedocs.io
2026-07-14 4mMILLER24m(1)
2026-07-15 4mMILLER24m(1)
</pre>

17
docs/src/manpage.txt
View file

@ -458,7 +458,7 @@
csv` is the same as `--ocsv`.
1mFLATTEN-UNFLATTEN FLAGS0m
These flags control how Miller converts record values which are maps or arrays, when input is JSON and output is non-JSON (flattening) or input is non-JSON and output is JSON (unflattening).
These flags control how Miller converts record values which are maps or arrays, when input is JSON/YAML and output is not (flattening) or input is not JSON/YAML and output is JSON/YAML (unflattening).
See the flatten/unflatten doc page https://miller.readthedocs.io/en/latest/flatten-unflatten for more information.
@ -466,16 +466,17 @@
Separator for flattening multi-level JSON keys, e.g.
`{"a":{"b":3}}` becomes `a:b => 3` for non-JSON
formats. Defaults to `.`.
--no-auto-flatten When output is non-JSON, suppress the default
--no-auto-flatten When output is not JSON or YAML, suppress the default
auto-flatten behavior. Default: if `$y = [7,8,9]`
then this flattens to `y.1=7,y.2=8,y.3=9`, and
similarly for maps. With `--no-auto-flatten`, instead
we get `$y=[1, 2, 3]`.
--no-auto-unflatten When input is non-JSON and output is JSON, suppress
the default auto-unflatten behavior. Default: if the
input has `y.1=7,y.2=8,y.3=9` then this unflattens to
`$y=[7,8,9]`. With `--no-auto-flatten`, instead we
get `${y.1}=7,${y.2}=8,${y.3}=9`.
--no-auto-unflatten When input is not JSON or YAML and output is JSON or
YAML, suppress the default auto-unflatten behavior.
Default: if the input has `y.1=7,y.2=8,y.3=9` then
this unflattens to `$y=[7,8,9]`. With
`--no-auto-flatten`, instead we get
`${y.1}=7,${y.2}=8,${y.3}=9`.
1mFORMAT-CONVERSION KEYSTROKE-SAVER FLAGS0m
As keystroke-savers for format-conversion you may use the following.
@ -4138,4 +4139,4 @@
MIME Type for Comma-Separated Values (CSV) Files, the Miller docsite
https://miller.readthedocs.io
2026-07-14 4mMILLER24m(1)
2026-07-15 4mMILLER24m(1)

6
docs/src/reference-main-flag-list.md
View file

@ -210,7 +210,7 @@ are overridden in all cases by setting output format to `format2`.
## Flatten-unflatten flags
These flags control how Miller converts record values which are maps or arrays, when input is JSON and output is non-JSON (flattening) or input is non-JSON and output is JSON (unflattening).
These flags control how Miller converts record values which are maps or arrays, when input is JSON/YAML and output is not (flattening) or input is not JSON/YAML and output is JSON/YAML (unflattening).
See the flatten/unflatten doc page https://miller.readthedocs.io/en/latest/flatten-unflatten for more information.
@ -218,8 +218,8 @@ See the flatten/unflatten doc page https://miller.readthedocs.io/en/latest/flatt
**Flags:**
* `--flatsep or --jflatsep {string}`: Separator for flattening multi-level JSON keys, e.g. `{"a":{"b":3}}` becomes `a:b => 3` for non-JSON formats. Defaults to `.`.
* `--no-auto-flatten`: When output is non-JSON, suppress the default auto-flatten behavior. Default: if `$y = [7,8,9]` then this flattens to `y.1=7,y.2=8,y.3=9`, and similarly for maps. With `--no-auto-flatten`, instead we get `$y=[1, 2, 3]`.
* `--no-auto-unflatten`: When input is non-JSON and output is JSON, suppress the default auto-unflatten behavior. Default: if the input has `y.1=7,y.2=8,y.3=9` then this unflattens to `$y=[7,8,9]`. With `--no-auto-flatten`, instead we get `${y.1}=7,${y.2}=8,${y.3}=9`.
* `--no-auto-flatten`: When output is not JSON or YAML, suppress the default auto-flatten behavior. Default: if `$y = [7,8,9]` then this flattens to `y.1=7,y.2=8,y.3=9`, and similarly for maps. With `--no-auto-flatten`, instead we get `$y=[1, 2, 3]`.
* `--no-auto-unflatten`: When input is not JSON or YAML and output is JSON or YAML, suppress the default auto-unflatten behavior. Default: if the input has `y.1=7,y.2=8,y.3=9` then this unflattens to `$y=[7,8,9]`. With `--no-auto-flatten`, instead we get `${y.1}=7,${y.2}=8,${y.3}=9`.
## Format-conversion keystroke-saver flags

17
man/manpage.txt
View file

@ -458,7 +458,7 @@
csv` is the same as `--ocsv`.
1mFLATTEN-UNFLATTEN FLAGS0m
These flags control how Miller converts record values which are maps or arrays, when input is JSON and output is non-JSON (flattening) or input is non-JSON and output is JSON (unflattening).
These flags control how Miller converts record values which are maps or arrays, when input is JSON/YAML and output is not (flattening) or input is not JSON/YAML and output is JSON/YAML (unflattening).
See the flatten/unflatten doc page https://miller.readthedocs.io/en/latest/flatten-unflatten for more information.
@ -466,16 +466,17 @@
Separator for flattening multi-level JSON keys, e.g.
`{"a":{"b":3}}` becomes `a:b => 3` for non-JSON
formats. Defaults to `.`.
--no-auto-flatten When output is non-JSON, suppress the default
--no-auto-flatten When output is not JSON or YAML, suppress the default
auto-flatten behavior. Default: if `$y = [7,8,9]`
then this flattens to `y.1=7,y.2=8,y.3=9`, and
similarly for maps. With `--no-auto-flatten`, instead
we get `$y=[1, 2, 3]`.
--no-auto-unflatten When input is non-JSON and output is JSON, suppress
the default auto-unflatten behavior. Default: if the
input has `y.1=7,y.2=8,y.3=9` then this unflattens to
`$y=[7,8,9]`. With `--no-auto-flatten`, instead we
get `${y.1}=7,${y.2}=8,${y.3}=9`.
--no-auto-unflatten When input is not JSON or YAML and output is JSON or
YAML, suppress the default auto-unflatten behavior.
Default: if the input has `y.1=7,y.2=8,y.3=9` then
this unflattens to `$y=[7,8,9]`. With
`--no-auto-flatten`, instead we get
`${y.1}=7,${y.2}=8,${y.3}=9`.
1mFORMAT-CONVERSION KEYSTROKE-SAVER FLAGS0m
As keystroke-savers for format-conversion you may use the following.
@ -4138,4 +4139,4 @@
MIME Type for Comma-Separated Values (CSV) Files, the Miller docsite
https://miller.readthedocs.io
2026-07-14 4mMILLER24m(1)
2026-07-15 4mMILLER24m(1)

19
man/mlr.1
View file

@ -2,12 +2,12 @@
.\" Title: mlr
.\" Author: [see the "AUTHOR" section]
.\" Generator: ./mkman.rb
.\" Date: 2026-07-14
.\" Date: 2026-07-15
.\" Manual: \ \&
.\" Source: \ \&
.\" Language: English
.\"
.TH "MILLER" "1" "2026-07-14" "\ \&" "\ \&"
.TH "MILLER" "1" "2026-07-15" "\ \&" "\ \&"
.\" -----------------------------------------------------------------
.\" * Portability definitions
.\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@ -554,7 +554,7 @@ are overridden in all cases by setting output format to `format2`.
.RS 0
.\}
.nf
These flags control how Miller converts record values which are maps or arrays, when input is JSON and output is non-JSON (flattening) or input is non-JSON and output is JSON (unflattening).
These flags control how Miller converts record values which are maps or arrays, when input is JSON/YAML and output is not (flattening) or input is not JSON/YAML and output is JSON/YAML (unflattening).
See the flatten/unflatten doc page https://miller.readthedocs.io/en/latest/flatten-unflatten for more information.
@ -562,16 +562,17 @@ See the flatten/unflatten doc page https://miller.readthedocs.io/en/latest/flatt
Separator for flattening multi-level JSON keys, e.g.
`{"a":{"b":3}}` becomes `a:b => 3` for non-JSON
formats. Defaults to `.`.
--no-auto-flatten When output is non-JSON, suppress the default
--no-auto-flatten When output is not JSON or YAML, suppress the default
auto-flatten behavior. Default: if `$y = [7,8,9]`
then this flattens to `y.1=7,y.2=8,y.3=9`, and
similarly for maps. With `--no-auto-flatten`, instead
we get `$y=[1, 2, 3]`.
--no-auto-unflatten When input is non-JSON and output is JSON, suppress
the default auto-unflatten behavior. Default: if the
input has `y.1=7,y.2=8,y.3=9` then this unflattens to
`$y=[7,8,9]`. With `--no-auto-flatten`, instead we
get `${y.1}=7,${y.2}=8,${y.3}=9`.
--no-auto-unflatten When input is not JSON or YAML and output is JSON or
YAML, suppress the default auto-unflatten behavior.
Default: if the input has `y.1=7,y.2=8,y.3=9` then
this unflattens to `$y=[7,8,9]`. With
`--no-auto-flatten`, instead we get
`${y.1}=7,${y.2}=8,${y.3}=9`.
.fi
.if n \{\
.RE

18
pkg/cli/flatten_unflatten.go
View file

@ -5,7 +5,7 @@ package cli
//
// PROBLEM TO BE SOLVED:
//
// JSON has nested structures and CSV et al. do not. For example:
// JSON and YAML have nested structures and CSV et al. do not. For example:
// {
// "req" : {
// "method": "GET",
@ -22,7 +22,9 @@ package cli
//
// APPROACH:
//
// Use the Principle of Least Surprise (POLS).
// Use the Principle of Least Surprise (POLS). Below, "JSON" stands for any
// format capable of representing nested structures natively -- currently
// JSON, JSON Lines, and YAML.
//
// * If input is JSON and output is JSON:
// o Records can be nested from record-read
@ -54,11 +56,17 @@ package cli
// flatten, don't undo that by putting an unflatten right after.
//
// isNestable returns true for formats which can represent nested/array
// structures natively, and thus don't need auto-flatten/auto-unflatten.
func isNestable(format string) bool {
return format == "json" || format == "jsonl" || format == "yaml"
}
func DecideFinalFlatten(writerOptions *TWriterOptions) bool {
ofmt := writerOptions.OutputFileFormat
if writerOptions.AutoFlatten {
// Preserve nested/array structure for formats that support it.
if ofmt != "json" && ofmt != "jsonl" && ofmt != "dcf" {
if !isNestable(ofmt) && ofmt != "dcf" {
return true
}
}
@ -85,8 +93,8 @@ func DecideFinalUnflatten(
ofmt := options.WriterOptions.OutputFileFormat
if options.WriterOptions.AutoUnflatten {
if ifmt != "json" {
if ofmt == "json" {
if !isNestable(ifmt) {
if isNestable(ofmt) {
return true
}
}

6
pkg/cli/option_parse.go
View file

@ -3300,7 +3300,7 @@ var OutputColorizationFlagSection = FlagSection{
// FLATTEN/UNFLATTEN FLAGS
func FlattenUnflattenPrintInfo() {
fmt.Println("These flags control how Miller converts record values which are maps or arrays, when input is JSON and output is non-JSON (flattening) or input is non-JSON and output is JSON (unflattening).")
fmt.Println("These flags control how Miller converts record values which are maps or arrays, when input is JSON/YAML and output is not (flattening) or input is not JSON/YAML and output is JSON/YAML (unflattening).")
fmt.Println()
fmt.Println("See the flatten/unflatten doc page https://miller.readthedocs.io/en/latest/flatten-unflatten for more information.")
}
@ -3326,7 +3326,7 @@ var FlattenUnflattenFlagSection = FlagSection{
{
name: "--no-auto-flatten",
help: "When output is non-JSON, suppress the default auto-flatten behavior. Default: if `$y = [7,8,9]` then this flattens to `y.1=7,y.2=8,y.3=9`, and similarly for maps. With `--no-auto-flatten`, instead we get `$y=[1, 2, 3]`.",
help: "When output is not JSON or YAML, suppress the default auto-flatten behavior. Default: if `$y = [7,8,9]` then this flattens to `y.1=7,y.2=8,y.3=9`, and similarly for maps. With `--no-auto-flatten`, instead we get `$y=[1, 2, 3]`.",
parser: func(args []string, argc int, pargi *int, options *TOptions) {
options.WriterOptions.AutoFlatten = false
*pargi += 1
@ -3335,7 +3335,7 @@ var FlattenUnflattenFlagSection = FlagSection{
{
name: "--no-auto-unflatten",
help: "When input is non-JSON and output is JSON, suppress the default auto-unflatten behavior. Default: if the input has `y.1=7,y.2=8,y.3=9` then this unflattens to `$y=[7,8,9]`. With `--no-auto-flatten`, instead we get `${y.1}=7,${y.2}=8,${y.3}=9`.",
help: "When input is not JSON or YAML and output is JSON or YAML, suppress the default auto-unflatten behavior. Default: if the input has `y.1=7,y.2=8,y.3=9` then this unflattens to `$y=[7,8,9]`. With `--no-auto-flatten`, instead we get `${y.1}=7,${y.2}=8,${y.3}=9`.",
parser: func(args []string, argc int, pargi *int, options *TOptions) {
options.WriterOptions.AutoUnflatten = false
*pargi += 1

1
test/cases/io-yaml-io/0005/cmd Normal file
View file

@ -0,0 +1 @@
mlr --icsv --oyaml cat ${CASEDIR}/input

0
test/cases/io-yaml-io/0005/experr Normal file
View file

8
test/cases/io-yaml-io/0005/expout Normal file
View file

@ -0,0 +1,8 @@
- a: 1
b:
- 2
- 3
- a: 4
b:
- 5
- 6

3
test/cases/io-yaml-io/0005/input Normal file
View file

@ -0,0 +1,3 @@
a,b.1,b.2
1,2,3
4,5,6

1
test/cases/io-yaml-io/0006/cmd Normal file
View file

@ -0,0 +1 @@
mlr --ijson --oyaml cat ${CASEDIR}/input

0
test/cases/io-yaml-io/0006/experr Normal file
View file

4
test/cases/io-yaml-io/0006/expout Normal file
View file

@ -0,0 +1,4 @@
- a: 1
req:
method: GET
path: api/check

3
test/cases/io-yaml-io/0006/input Normal file
View file

@ -0,0 +1,3 @@
[
{"a": 1, "req": {"method": "GET", "path": "api/check"}}
]

1
test/cases/io-yaml-io/0007/cmd Normal file
View file

@ -0,0 +1 @@
mlr --iyaml --ocsv cat ${CASEDIR}/input

0
test/cases/io-yaml-io/0007/experr Normal file
View file

3
test/cases/io-yaml-io/0007/expout Normal file
View file

@ -0,0 +1,3 @@
a,b.1,b.2
1,2,3
4,5,6

8
test/cases/io-yaml-io/0007/input Normal file
View file

@ -0,0 +1,8 @@
- a: 1
b:
- 2
- 3
- a: 4
b:
- 5
- 6