+
+
Suppose you have some heterogeneous data like this:
+
+POKI_INCLUDE_ESCAPED(data/features.json)HERE
+
+
A reasonable question to ask is, how many occurrences of each field are
+there? And, what percentage of total row count has each of them? Since the
+denominator of the percentage is not known until the end, this is a two-pass
+algorithm:
+
+POKI_INCLUDE_ESCAPED(data/feature-count.mlr)HERE
+
+
Then
+POKI_RUN_COMMAND{{mlr --json put -q -f data/feature-count.mlr data/features.json}}HERE
+POKI_RUN_COMMAND{{mlr --ijson --opprint put -q -f data/feature-count.mlr data/features.json}}HERE
+
+
+
Suppose you have some heterogeneous data like this:
+
+
+
+
+{ "qoh": 29874, "rate": 1.68, "latency": 0.02 }
+{ "name": "alice", "uid": 572 }
+{ "qoh": 1227, "rate": 1.01, "latency": 0.07 }
+{ "qoh": 13458, "rate": 1.72, "latency": 0.04 }
+{ "qoh": 56782, "rate": 1.64, "latency": 0.02 }
+{ "qoh": 23512, "rate": 1.71, "latency": 0.03 }
+{ "qoh": 9876, "rate": 1.89, "latency": 0.08 }
+{ "name": "bill", "uid": 684 }
+{ "name": "chuck", "uid": 908 }
+{ "name": "dottie", "uid": 440 }
+{ "qoh": 0, "rate": 0.40, "latency": 0.01 }
+{ "qoh": 5438, "rate": 1.56, "latency": 0.17 }
+
+
+
+
+
A reasonable question to ask is, how many occurrences of each field are
+there? And, what percentage of total row count has each of them? Since the
+denominator of the percentage is not known until the end, this is a two-pass
+algorithm:
+
+
+
+
+for (key in $*) {
+ @key_counts[key] += 1;
+}
+@record_count += 1;
+
+end {
+ for (key in @key_counts) {
+ @key_fraction[key] = @key_counts[key] / @record_count
+ }
+ emit @record_count;
+ emit @key_counts, "key";
+ emit @key_fraction,"key"
+}
+
+
+
+
+
Then
+
+
+
+$ mlr --json put -q -f data/feature-count.mlr data/features.json
+{ "record_count": 12 }
+{ "key": "qoh", "key_counts": 8 }
+{ "key": "rate", "key_counts": 8 }
+{ "key": "latency", "key_counts": 8 }
+{ "key": "name", "key_counts": 4 }
+{ "key": "uid", "key_counts": 4 }
+{ "key": "qoh", "key_fraction": 0.666667 }
+{ "key": "rate", "key_fraction": 0.666667 }
+{ "key": "latency", "key_fraction": 0.666667 }
+{ "key": "name", "key_fraction": 0.333333 }
+{ "key": "uid", "key_fraction": 0.333333 }
+
+
+
+
+
+
+$ mlr --ijson --opprint put -q -f data/feature-count.mlr data/features.json
+record_count
+12
+
+key key_counts
+qoh 8
+rate 8
+latency 8
+name 4
+uid 4
+
+key key_fraction
+qoh 0.666667
+rate 0.666667
+latency 0.666667
+name 0.333333
+uid 0.333333
+
+
+
+