diff --git a/docs/src/data/nested-body.json b/docs/src/data/nested-body.json new file mode 100644 index 000000000..618a3c13d --- /dev/null +++ b/docs/src/data/nested-body.json @@ -0,0 +1,4 @@ +[ + { "Body": { "meta": 5, "id": "abc" } }, + { "Body": { "meta": 6, "id": "def" } } +] diff --git a/docs/src/data/whois.json b/docs/src/data/whois.json new file mode 100644 index 000000000..f9b3079e9 --- /dev/null +++ b/docs/src/data/whois.json @@ -0,0 +1,11 @@ +{ + "domain": { + "id": "2138514_DOMAIN_COM-VRSN", + "domain": "google.com", + "extension": "com" + }, + "registrar": { + "id": "292", + "name": "MarkMonitor Inc." + } +} diff --git a/docs/src/flatten-unflatten.md b/docs/src/flatten-unflatten.md index 9706ddd1d..d63490cf0 100644 --- a/docs/src/flatten-unflatten.md +++ b/docs/src/flatten-unflatten.md @@ -530,3 +530,141 @@ See also the [JSON parse and stringify section](reference-main-data-types.md#json-parse-and-stringify) section for more on this -- for example, when Miller is producing SQL-query output from tables having one or more columns that contain JSON-encoded data. + +## Using verbs with nested data + +Miller's [verbs](reference-verbs.md) -- such as [cut](reference-verbs.md#cut), +[rename](reference-verbs.md#rename), [sort](reference-verbs.md#sort), and so on +-- refer to fields by _non-nested_ field names. To a verb, `domain.domain` is +simply a nine-character field name, not an instruction to look up the key +`domain` inside a map-valued field named `domain`. (Within the [Miller +programming language](miller-programming-language.md), by contrast, +`$domain.domain` _does_ mean nested-field access.) + +So, given nested JSON input like this: + +
+cat data/whois.json ++
+{
+ "domain": {
+ "id": "2138514_DOMAIN_COM-VRSN",
+ "domain": "google.com",
+ "extension": "com"
+ },
+ "registrar": {
+ "id": "292",
+ "name": "MarkMonitor Inc."
+ }
+}
+
+
+the following produces no data output, since the record contains a
+map-valued field named `domain` but nothing named `domain.domain`:
+
++mlr --ijson --ocsv cut -f domain.domain data/whois.json ++
+ + ++ +Note that even though the output format here is CSV, auto-flatten happens at +the _end_ of the [chain](reference-main-then-chaining.md) -- so the `cut` verb +still sees the nested data. + +The solution is to use the [flatten](reference-verbs.md#flatten) verb _before_ +the verb which needs the flattened field names. After the flatten, the record +really does contain a field named `domain.domain`, which `cut` can operate on: + +
+mlr --ijson --ocsv flatten then cut -f domain.domain data/whois.json ++
+domain.domain +google.com ++ +If the output format is JSON, auto-flatten and auto-unflatten don't happen +(there's no need to flatten JSON-to-JSON) -- so, to get nested output back, use +the [unflatten](reference-verbs.md#unflatten) verb after the verb which needed +the flattened field names: + +
+mlr --json flatten then cut -f domain.domain then unflatten data/whois.json ++
+[
+{
+ "domain": {
+ "domain": "google.com"
+ }
+}
+]
+
+
+The same technique applies to renaming a nested field. Given this input:
+
++cat data/nested-body.json ++
+[
+ { "Body": { "meta": 5, "id": "abc" } },
+ { "Body": { "meta": 6, "id": "def" } }
+]
+
+
+using `rename Body.meta,Body.renamed_meta` by itself is a no-op, since no field
+is literally named `Body.meta` -- but with flatten and unflatten around it, we
+get what we want:
+
++mlr --json flatten then rename Body.meta,Body.renamed_meta then unflatten data/nested-body.json ++
+[
+{
+ "Body": {
+ "renamed_meta": 5,
+ "id": "abc"
+ }
+},
+{
+ "Body": {
+ "renamed_meta": 6,
+ "id": "def"
+ }
+}
+]
+
+
+An alternative, without flatten/unflatten, is to use
+[put](reference-verbs.md#put) and [unset](reference-dsl-variables.md) with the
+DSL's nested-field access:
+
++mlr --json put '$Body.renamed_meta = $Body.meta; unset $Body.meta' data/nested-body.json ++
+[
+{
+ "Body": {
+ "id": "abc",
+ "renamed_meta": 5
+ }
+},
+{
+ "Body": {
+ "id": "def",
+ "renamed_meta": 6
+ }
+}
+]
+
+
+One difference between the two: flatten-rename-unflatten leaves the renamed
+field in its original position within the record, while the put-then-unset
+approach places the new field after the existing ones.
diff --git a/docs/src/flatten-unflatten.md.in b/docs/src/flatten-unflatten.md.in
index 951ea1f58..04407b1fe 100644
--- a/docs/src/flatten-unflatten.md.in
+++ b/docs/src/flatten-unflatten.md.in
@@ -241,3 +241,73 @@ See also the
[JSON parse and stringify section](reference-main-data-types.md#json-parse-and-stringify) section for
more on this -- for example, when Miller is producing SQL-query output from
tables having one or more columns that contain JSON-encoded data.
+
+## Using verbs with nested data
+
+Miller's [verbs](reference-verbs.md) -- such as [cut](reference-verbs.md#cut),
+[rename](reference-verbs.md#rename), [sort](reference-verbs.md#sort), and so on
+-- refer to fields by _non-nested_ field names. To a verb, `domain.domain` is
+simply a nine-character field name, not an instruction to look up the key
+`domain` inside a map-valued field named `domain`. (Within the [Miller
+programming language](miller-programming-language.md), by contrast,
+`$domain.domain` _does_ mean nested-field access.)
+
+So, given nested JSON input like this:
+
+GENMD-RUN-COMMAND
+cat data/whois.json
+GENMD-EOF
+
+the following produces no data output, since the record contains a
+map-valued field named `domain` but nothing named `domain.domain`:
+
+GENMD-RUN-COMMAND
+mlr --ijson --ocsv cut -f domain.domain data/whois.json
+GENMD-EOF
+
+Note that even though the output format here is CSV, auto-flatten happens at
+the _end_ of the [chain](reference-main-then-chaining.md) -- so the `cut` verb
+still sees the nested data.
+
+The solution is to use the [flatten](reference-verbs.md#flatten) verb _before_
+the verb which needs the flattened field names. After the flatten, the record
+really does contain a field named `domain.domain`, which `cut` can operate on:
+
+GENMD-RUN-COMMAND
+mlr --ijson --ocsv flatten then cut -f domain.domain data/whois.json
+GENMD-EOF
+
+If the output format is JSON, auto-flatten and auto-unflatten don't happen
+(there's no need to flatten JSON-to-JSON) -- so, to get nested output back, use
+the [unflatten](reference-verbs.md#unflatten) verb after the verb which needed
+the flattened field names:
+
+GENMD-RUN-COMMAND
+mlr --json flatten then cut -f domain.domain then unflatten data/whois.json
+GENMD-EOF
+
+The same technique applies to renaming a nested field. Given this input:
+
+GENMD-RUN-COMMAND
+cat data/nested-body.json
+GENMD-EOF
+
+using `rename Body.meta,Body.renamed_meta` by itself is a no-op, since no field
+is literally named `Body.meta` -- but with flatten and unflatten around it, we
+get what we want:
+
+GENMD-RUN-COMMAND
+mlr --json flatten then rename Body.meta,Body.renamed_meta then unflatten data/nested-body.json
+GENMD-EOF
+
+An alternative, without flatten/unflatten, is to use
+[put](reference-verbs.md#put) and [unset](reference-dsl-variables.md) with the
+DSL's nested-field access:
+
+GENMD-RUN-COMMAND
+mlr --json put '$Body.renamed_meta = $Body.meta; unset $Body.meta' data/nested-body.json
+GENMD-EOF
+
+One difference between the two: flatten-rename-unflatten leaves the renamed
+field in its original position within the record, while the put-then-unset
+approach places the new field after the existing ones.