diff --git a/docs/src/data/nested-body.json b/docs/src/data/nested-body.json new file mode 100644 index 000000000..618a3c13d --- /dev/null +++ b/docs/src/data/nested-body.json @@ -0,0 +1,4 @@ +[ + { "Body": { "meta": 5, "id": "abc" } }, + { "Body": { "meta": 6, "id": "def" } } +] diff --git a/docs/src/data/whois.json b/docs/src/data/whois.json new file mode 100644 index 000000000..f9b3079e9 --- /dev/null +++ b/docs/src/data/whois.json @@ -0,0 +1,11 @@ +{ + "domain": { + "id": "2138514_DOMAIN_COM-VRSN", + "domain": "google.com", + "extension": "com" + }, + "registrar": { + "id": "292", + "name": "MarkMonitor Inc." + } +} diff --git a/docs/src/flatten-unflatten.md b/docs/src/flatten-unflatten.md index 9706ddd1d..d63490cf0 100644 --- a/docs/src/flatten-unflatten.md +++ b/docs/src/flatten-unflatten.md @@ -530,3 +530,141 @@ See also the [JSON parse and stringify section](reference-main-data-types.md#json-parse-and-stringify) section for more on this -- for example, when Miller is producing SQL-query output from tables having one or more columns that contain JSON-encoded data. + +## Using verbs with nested data + +Miller's [verbs](reference-verbs.md) -- such as [cut](reference-verbs.md#cut), +[rename](reference-verbs.md#rename), [sort](reference-verbs.md#sort), and so on +-- refer to fields by _non-nested_ field names. To a verb, `domain.domain` is +simply a nine-character field name, not an instruction to look up the key +`domain` inside a map-valued field named `domain`. (Within the [Miller +programming language](miller-programming-language.md), by contrast, +`$domain.domain` _does_ mean nested-field access.) + +So, given nested JSON input like this: + +
+cat data/whois.json
+
+
+{
+  "domain": {
+    "id": "2138514_DOMAIN_COM-VRSN",
+    "domain": "google.com",
+    "extension": "com"
+  },
+  "registrar": {
+    "id": "292",
+    "name": "MarkMonitor Inc."
+  }
+}
+
+ +the following produces no data output, since the record contains a +map-valued field named `domain` but nothing named `domain.domain`: + +
+mlr --ijson --ocsv cut -f domain.domain data/whois.json
+
+
+
+
+
+ +Note that even though the output format here is CSV, auto-flatten happens at +the _end_ of the [chain](reference-main-then-chaining.md) -- so the `cut` verb +still sees the nested data. + +The solution is to use the [flatten](reference-verbs.md#flatten) verb _before_ +the verb which needs the flattened field names. After the flatten, the record +really does contain a field named `domain.domain`, which `cut` can operate on: + +
+mlr --ijson --ocsv flatten then cut -f domain.domain data/whois.json
+
+
+domain.domain
+google.com
+
+ +If the output format is JSON, auto-flatten and auto-unflatten don't happen +(there's no need to flatten JSON-to-JSON) -- so, to get nested output back, use +the [unflatten](reference-verbs.md#unflatten) verb after the verb which needed +the flattened field names: + +
+mlr --json flatten then cut -f domain.domain then unflatten data/whois.json
+
+
+[
+{
+  "domain": {
+    "domain": "google.com"
+  }
+}
+]
+
+ +The same technique applies to renaming a nested field. Given this input: + +
+cat data/nested-body.json
+
+
+[
+  { "Body": { "meta": 5,  "id": "abc" } },
+  { "Body": { "meta": 6,  "id": "def" } }
+]
+
+ +using `rename Body.meta,Body.renamed_meta` by itself is a no-op, since no field +is literally named `Body.meta` -- but with flatten and unflatten around it, we +get what we want: + +
+mlr --json flatten then rename Body.meta,Body.renamed_meta then unflatten data/nested-body.json
+
+
+[
+{
+  "Body": {
+    "renamed_meta": 5,
+    "id": "abc"
+  }
+},
+{
+  "Body": {
+    "renamed_meta": 6,
+    "id": "def"
+  }
+}
+]
+
+ +An alternative, without flatten/unflatten, is to use +[put](reference-verbs.md#put) and [unset](reference-dsl-variables.md) with the +DSL's nested-field access: + +
+mlr --json put '$Body.renamed_meta = $Body.meta; unset $Body.meta' data/nested-body.json
+
+
+[
+{
+  "Body": {
+    "id": "abc",
+    "renamed_meta": 5
+  }
+},
+{
+  "Body": {
+    "id": "def",
+    "renamed_meta": 6
+  }
+}
+]
+
+ +One difference between the two: flatten-rename-unflatten leaves the renamed +field in its original position within the record, while the put-then-unset +approach places the new field after the existing ones. diff --git a/docs/src/flatten-unflatten.md.in b/docs/src/flatten-unflatten.md.in index 951ea1f58..04407b1fe 100644 --- a/docs/src/flatten-unflatten.md.in +++ b/docs/src/flatten-unflatten.md.in @@ -241,3 +241,73 @@ See also the [JSON parse and stringify section](reference-main-data-types.md#json-parse-and-stringify) section for more on this -- for example, when Miller is producing SQL-query output from tables having one or more columns that contain JSON-encoded data. + +## Using verbs with nested data + +Miller's [verbs](reference-verbs.md) -- such as [cut](reference-verbs.md#cut), +[rename](reference-verbs.md#rename), [sort](reference-verbs.md#sort), and so on +-- refer to fields by _non-nested_ field names. To a verb, `domain.domain` is +simply a nine-character field name, not an instruction to look up the key +`domain` inside a map-valued field named `domain`. (Within the [Miller +programming language](miller-programming-language.md), by contrast, +`$domain.domain` _does_ mean nested-field access.) + +So, given nested JSON input like this: + +GENMD-RUN-COMMAND +cat data/whois.json +GENMD-EOF + +the following produces no data output, since the record contains a +map-valued field named `domain` but nothing named `domain.domain`: + +GENMD-RUN-COMMAND +mlr --ijson --ocsv cut -f domain.domain data/whois.json +GENMD-EOF + +Note that even though the output format here is CSV, auto-flatten happens at +the _end_ of the [chain](reference-main-then-chaining.md) -- so the `cut` verb +still sees the nested data. + +The solution is to use the [flatten](reference-verbs.md#flatten) verb _before_ +the verb which needs the flattened field names. After the flatten, the record +really does contain a field named `domain.domain`, which `cut` can operate on: + +GENMD-RUN-COMMAND +mlr --ijson --ocsv flatten then cut -f domain.domain data/whois.json +GENMD-EOF + +If the output format is JSON, auto-flatten and auto-unflatten don't happen +(there's no need to flatten JSON-to-JSON) -- so, to get nested output back, use +the [unflatten](reference-verbs.md#unflatten) verb after the verb which needed +the flattened field names: + +GENMD-RUN-COMMAND +mlr --json flatten then cut -f domain.domain then unflatten data/whois.json +GENMD-EOF + +The same technique applies to renaming a nested field. Given this input: + +GENMD-RUN-COMMAND +cat data/nested-body.json +GENMD-EOF + +using `rename Body.meta,Body.renamed_meta` by itself is a no-op, since no field +is literally named `Body.meta` -- but with flatten and unflatten around it, we +get what we want: + +GENMD-RUN-COMMAND +mlr --json flatten then rename Body.meta,Body.renamed_meta then unflatten data/nested-body.json +GENMD-EOF + +An alternative, without flatten/unflatten, is to use +[put](reference-verbs.md#put) and [unset](reference-dsl-variables.md) with the +DSL's nested-field access: + +GENMD-RUN-COMMAND +mlr --json put '$Body.renamed_meta = $Body.meta; unset $Body.meta' data/nested-body.json +GENMD-EOF + +One difference between the two: flatten-rename-unflatten leaves the renamed +field in its original position within the record, while the put-then-unset +approach places the new field after the existing ones.