diff --git a/docs/src/data/sensor-humidity.csv b/docs/src/data/sensor-humidity.csv new file mode 100644 index 000000000..c22ff5d7d --- /dev/null +++ b/docs/src/data/sensor-humidity.csv @@ -0,0 +1,3 @@ +unixTime,minValue,averageValue,maxValue +1740000000,50.1,50.3,50.5 +1740000120,52.3,52.5,52.7 diff --git a/docs/src/data/sensor-pressure.csv b/docs/src/data/sensor-pressure.csv new file mode 100644 index 000000000..341104a14 --- /dev/null +++ b/docs/src/data/sensor-pressure.csv @@ -0,0 +1,4 @@ +unixTime,minValue,averageValue,maxValue +1740000000,1012.2,1012.3,1012.4 +1740000060,1011.8,1011.9,1012.0 +1740000120,1011.5,1011.6,1011.7 diff --git a/docs/src/data/sensor-temp.csv b/docs/src/data/sensor-temp.csv new file mode 100644 index 000000000..a07b02046 --- /dev/null +++ b/docs/src/data/sensor-temp.csv @@ -0,0 +1,4 @@ +unixTime,minValue,averageValue,maxValue +1740000000,37.2,37.4,37.6 +1740000060,37.5,37.6,37.7 +1740000120,37.8,37.9,38.0 diff --git a/docs/src/questions-about-joins.md b/docs/src/questions-about-joins.md index d7f7e70e2..c620e8190 100644 --- a/docs/src/questions-about-joins.md +++ b/docs/src/questions-about-joins.md @@ -277,6 +277,108 @@ id status name task 30 occupied Alice clean +## Merging several files on a common key when column names collide + +The previous example worked painlessly because the non-key column names -- `name` and `status` -- were different in each lookup file. Now suppose you have several files, each containing measurements of a _different_ quantity, but all with the _same_ column names -- say, one file each for temperature, humidity, and pressure, keyed by timestamp: + +
+cat data/sensor-temp.csv ++
+unixTime,minValue,averageValue,maxValue +1740000000,37.2,37.4,37.6 +1740000060,37.5,37.6,37.7 +1740000120,37.8,37.9,38.0 ++ +
+cat data/sensor-humidity.csv ++
+unixTime,minValue,averageValue,maxValue +1740000000,50.1,50.3,50.5 +1740000120,52.3,52.5,52.7 ++ +
+cat data/sensor-pressure.csv ++
+unixTime,minValue,averageValue,maxValue +1740000000,1012.2,1012.3,1012.4 +1740000060,1011.8,1011.9,1012.0 +1740000120,1011.5,1011.6,1011.7 ++ +(Note that the humidity file is missing a row for the middle timestamp.) + +If we merge these with a `then`-chain of `join` commands, as in the previous section, columns are lost: since every file's non-key columns have the same names, each join step overwrites the values from the step before, and only one file's values survive: + +
+mlr --icsv --opprint \ + join -j unixTime -f data/sensor-temp.csv \ + then join -j unixTime -f data/sensor-humidity.csv \ + data/sensor-pressure.csv ++
+unixTime minValue averageValue maxValue +1740000000 1012.2 1012.3 1012.4 +1740000120 1011.5 1011.6 1011.7 ++ +The fix is `join`'s `--lp` (left-prefix) option, which renames the non-key columns coming from each `-f` file so that nothing collides: + +
+mlr --icsv --opprint \ + join --lp temp: -j unixTime -f data/sensor-temp.csv \ + then join --lp humidity: -j unixTime -f data/sensor-humidity.csv \ + data/sensor-pressure.csv ++
+unixTime humidity:minValue humidity:averageValue humidity:maxValue temp:minValue temp:averageValue temp:maxValue minValue averageValue maxValue +1740000000 50.1 50.3 50.5 37.2 37.4 37.6 1012.2 1012.3 1012.4 +1740000120 52.3 52.5 52.7 37.8 37.9 38.0 1011.5 1011.6 1011.7 ++ +The columns from the file at the end of the command line -- here, the pressure file -- keep their unprefixed names. + +If you want just one value column per file, you can also use `join`'s `--lk` option to keep only that column from each `-f` file, then use [cut](reference-verbs.md#cut) and [label](reference-verbs.md#label) to arrange and rename the output columns: + +
+mlr --icsv --opprint \ + join --lp temp: --lk averageValue -j unixTime -f data/sensor-temp.csv \ + then join --lp humidity: --lk averageValue -j unixTime -f data/sensor-humidity.csv \ + then cut -o -f unixTime,temp:averageValue,humidity:averageValue,averageValue \ + then label unixTime,temperature,humidity,pressure \ + data/sensor-pressure.csv ++
+unixTime temperature humidity pressure +1740000000 37.4 50.3 1012.3 +1740000120 37.9 52.5 1011.6 ++ +Note that `join` gives inner-join semantics by default, so the timestamp missing from the humidity file has been dropped from the output. (This is also why the `paste` command is not a substitute for `join` here: `paste` matches rows by position, so a row missing from one file shifts that file's remaining values onto the wrong rows.) If you'd rather keep those rows, with empty cells where a file has no data, add `--ul --ur` to each join step, and [unsparsify](reference-verbs.md#unsparsify) afterward: + +
+mlr --icsv --opprint \ + join --ul --ur --lp temp: --lk averageValue -j unixTime -f data/sensor-temp.csv \ + then join --ul --ur --lp humidity: --lk averageValue -j unixTime -f data/sensor-humidity.csv \ + then unsparsify \ + then cut -o -f unixTime,temp:averageValue,humidity:averageValue,averageValue \ + then label unixTime,temperature,humidity,pressure \ + then sort -t unixTime \ + data/sensor-pressure.csv ++
+unixTime temperature humidity pressure +1740000000 37.4 50.3 1012.3 +1740000060 37.6 - 1011.9 +1740000120 37.9 52.5 1011.6 ++ +The `unsparsify` must come before the `cut`-and-`label` step, since `label` renames columns positionally. The `sort` is there because unpaired records may be emitted out of order relative to paired ones. + ## How to preprocess the left file of a join? The left file (the `-f` argument to `join`) is opened by the `join` verb itself, so it doesn't pass through the main record stream: a `then`-chain can preprocess the right file(s), but not the left one. diff --git a/docs/src/questions-about-joins.md.in b/docs/src/questions-about-joins.md.in index 368e6bda5..76328a2d7 100644 --- a/docs/src/questions-about-joins.md.in +++ b/docs/src/questions-about-joins.md.in @@ -130,6 +130,70 @@ mlr --icsv --opprint join -f multi-join/name-lookup.csv -j id \ multi-join/input.csv GENMD-EOF +## Merging several files on a common key when column names collide + +The previous example worked painlessly because the non-key column names -- `name` and `status` -- were different in each lookup file. Now suppose you have several files, each containing measurements of a _different_ quantity, but all with the _same_ column names -- say, one file each for temperature, humidity, and pressure, keyed by timestamp: + +GENMD-RUN-COMMAND +cat data/sensor-temp.csv +GENMD-EOF + +GENMD-RUN-COMMAND +cat data/sensor-humidity.csv +GENMD-EOF + +GENMD-RUN-COMMAND +cat data/sensor-pressure.csv +GENMD-EOF + +(Note that the humidity file is missing a row for the middle timestamp.) + +If we merge these with a `then`-chain of `join` commands, as in the previous section, columns are lost: since every file's non-key columns have the same names, each join step overwrites the values from the step before, and only one file's values survive: + +GENMD-RUN-COMMAND +mlr --icsv --opprint \ + join -j unixTime -f data/sensor-temp.csv \ + then join -j unixTime -f data/sensor-humidity.csv \ + data/sensor-pressure.csv +GENMD-EOF + +The fix is `join`'s `--lp` (left-prefix) option, which renames the non-key columns coming from each `-f` file so that nothing collides: + +GENMD-RUN-COMMAND +mlr --icsv --opprint \ + join --lp temp: -j unixTime -f data/sensor-temp.csv \ + then join --lp humidity: -j unixTime -f data/sensor-humidity.csv \ + data/sensor-pressure.csv +GENMD-EOF + +The columns from the file at the end of the command line -- here, the pressure file -- keep their unprefixed names. + +If you want just one value column per file, you can also use `join`'s `--lk` option to keep only that column from each `-f` file, then use [cut](reference-verbs.md#cut) and [label](reference-verbs.md#label) to arrange and rename the output columns: + +GENMD-RUN-COMMAND +mlr --icsv --opprint \ + join --lp temp: --lk averageValue -j unixTime -f data/sensor-temp.csv \ + then join --lp humidity: --lk averageValue -j unixTime -f data/sensor-humidity.csv \ + then cut -o -f unixTime,temp:averageValue,humidity:averageValue,averageValue \ + then label unixTime,temperature,humidity,pressure \ + data/sensor-pressure.csv +GENMD-EOF + +Note that `join` gives inner-join semantics by default, so the timestamp missing from the humidity file has been dropped from the output. (This is also why the `paste` command is not a substitute for `join` here: `paste` matches rows by position, so a row missing from one file shifts that file's remaining values onto the wrong rows.) If you'd rather keep those rows, with empty cells where a file has no data, add `--ul --ur` to each join step, and [unsparsify](reference-verbs.md#unsparsify) afterward: + +GENMD-RUN-COMMAND +mlr --icsv --opprint \ + join --ul --ur --lp temp: --lk averageValue -j unixTime -f data/sensor-temp.csv \ + then join --ul --ur --lp humidity: --lk averageValue -j unixTime -f data/sensor-humidity.csv \ + then unsparsify \ + then cut -o -f unixTime,temp:averageValue,humidity:averageValue,averageValue \ + then label unixTime,temperature,humidity,pressure \ + then sort -t unixTime \ + data/sensor-pressure.csv +GENMD-EOF + +The `unsparsify` must come before the `cut`-and-`label` step, since `label` renames columns positionally. The `sort` is there because unpaired records may be emitted out of order relative to paired ones. + ## How to preprocess the left file of a join? The left file (the `-f` argument to `join`) is opened by the `join` verb itself, so it doesn't pass through the main record stream: a `then`-chain can preprocess the right file(s), but not the left one.