Mirrors/miller
1
0
Fork 0
mirror of https://github.com/johnkerl/miller.git synced 2026-07-17 16:38:54 +00:00
Code Issues Projects Releases Wiki Activity

Docs: recipe for merging several files on a common key when column names collide (#2156)

Browse source 
Adds a section to the questions-about-joins page showing how to merge
N files side-by-side on a shared key when the files' non-key column
names are all the same: a then-chain of join verbs with per-file --lp
prefixes (plus --lk / cut / label to trim, and --ul --ur / unsparsify
for outer-join semantics). Addresses #1775.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
John Kerl 2026-07-06 17:08:02 -04:00 • committed by GitHub
parent c69411f1b4
commit 405be3542c
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
5 changed files with 177 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

3
docs/src/data/sensor-humidity.csv Normal file
View file

@ -0,0 +1,3 @@
unixTime,minValue,averageValue,maxValue
1740000000,50.1,50.3,50.5
1740000120,52.3,52.5,52.7
1 unixTime minValue averageValue maxValue
2 1740000000 50.1 50.3 50.5
3 1740000120 52.3 52.5 52.7

4
docs/src/data/sensor-pressure.csv Normal file
View file

@ -0,0 +1,4 @@
unixTime,minValue,averageValue,maxValue
1740000000,1012.2,1012.3,1012.4
1740000060,1011.8,1011.9,1012.0
1740000120,1011.5,1011.6,1011.7
1 unixTime minValue averageValue maxValue
2 1740000000 1012.2 1012.3 1012.4
3 1740000060 1011.8 1011.9 1012.0
4 1740000120 1011.5 1011.6 1011.7

4
docs/src/data/sensor-temp.csv Normal file
View file

@ -0,0 +1,4 @@
unixTime,minValue,averageValue,maxValue
1740000000,37.2,37.4,37.6
1740000060,37.5,37.6,37.7
1740000120,37.8,37.9,38.0
1 unixTime minValue averageValue maxValue
2 1740000000 37.2 37.4 37.6
3 1740000060 37.5 37.6 37.7
4 1740000120 37.8 37.9 38.0

102
docs/src/questions-about-joins.md
View file

@ -277,6 +277,108 @@ id status name task
30 occupied Alice clean
</pre>
## Merging several files on a common key when column names collide
The previous example worked painlessly because the non-key column names -- `name` and `status` -- were different in each lookup file. Now suppose you have several files, each containing measurements of a _different_ quantity, but all with the _same_ column names -- say, one file each for temperature, humidity, and pressure, keyed by timestamp:
<pre class="pre-highlight-in-pair">
<b>cat data/sensor-temp.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
unixTime,minValue,averageValue,maxValue
1740000000,37.2,37.4,37.6
1740000060,37.5,37.6,37.7
1740000120,37.8,37.9,38.0
</pre>
<pre class="pre-highlight-in-pair">
<b>cat data/sensor-humidity.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
unixTime,minValue,averageValue,maxValue
1740000000,50.1,50.3,50.5
1740000120,52.3,52.5,52.7
</pre>
<pre class="pre-highlight-in-pair">
<b>cat data/sensor-pressure.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
unixTime,minValue,averageValue,maxValue
1740000000,1012.2,1012.3,1012.4
1740000060,1011.8,1011.9,1012.0
1740000120,1011.5,1011.6,1011.7
</pre>
(Note that the humidity file is missing a row for the middle timestamp.)
If we merge these with a `then`-chain of `join` commands, as in the previous section, columns are lost: since every file's non-key columns have the same names, each join step overwrites the values from the step before, and only one file's values survive:
<pre class="pre-highlight-in-pair">
<b>mlr --icsv --opprint \</b>
<b> join -j unixTime -f data/sensor-temp.csv \</b>
<b> then join -j unixTime -f data/sensor-humidity.csv \</b>
<b> data/sensor-pressure.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
unixTime minValue averageValue maxValue
1740000000 1012.2 1012.3 1012.4
1740000120 1011.5 1011.6 1011.7
</pre>
The fix is `join`'s `--lp` (left-prefix) option, which renames the non-key columns coming from each `-f` file so that nothing collides:
<pre class="pre-highlight-in-pair">
<b>mlr --icsv --opprint \</b>
<b> join --lp temp: -j unixTime -f data/sensor-temp.csv \</b>
<b> then join --lp humidity: -j unixTime -f data/sensor-humidity.csv \</b>
<b> data/sensor-pressure.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
unixTime humidity:minValue humidity:averageValue humidity:maxValue temp:minValue temp:averageValue temp:maxValue minValue averageValue maxValue
1740000000 50.1 50.3 50.5 37.2 37.4 37.6 1012.2 1012.3 1012.4
1740000120 52.3 52.5 52.7 37.8 37.9 38.0 1011.5 1011.6 1011.7
</pre>
The columns from the file at the end of the command line -- here, the pressure file -- keep their unprefixed names.
If you want just one value column per file, you can also use `join`'s `--lk` option to keep only that column from each `-f` file, then use [cut](reference-verbs.md#cut) and [label](reference-verbs.md#label) to arrange and rename the output columns:
<pre class="pre-highlight-in-pair">
<b>mlr --icsv --opprint \</b>
<b> join --lp temp: --lk averageValue -j unixTime -f data/sensor-temp.csv \</b>
<b> then join --lp humidity: --lk averageValue -j unixTime -f data/sensor-humidity.csv \</b>
<b> then cut -o -f unixTime,temp:averageValue,humidity:averageValue,averageValue \</b>
<b> then label unixTime,temperature,humidity,pressure \</b>
<b> data/sensor-pressure.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
unixTime temperature humidity pressure
1740000000 37.4 50.3 1012.3
1740000120 37.9 52.5 1011.6
</pre>
Note that `join` gives inner-join semantics by default, so the timestamp missing from the humidity file has been dropped from the output. (This is also why the `paste` command is not a substitute for `join` here: `paste` matches rows by position, so a row missing from one file shifts that file's remaining values onto the wrong rows.) If you'd rather keep those rows, with empty cells where a file has no data, add `--ul --ur` to each join step, and [unsparsify](reference-verbs.md#unsparsify) afterward:
<pre class="pre-highlight-in-pair">
<b>mlr --icsv --opprint \</b>
<b> join --ul --ur --lp temp: --lk averageValue -j unixTime -f data/sensor-temp.csv \</b>
<b> then join --ul --ur --lp humidity: --lk averageValue -j unixTime -f data/sensor-humidity.csv \</b>
<b> then unsparsify \</b>
<b> then cut -o -f unixTime,temp:averageValue,humidity:averageValue,averageValue \</b>
<b> then label unixTime,temperature,humidity,pressure \</b>
<b> then sort -t unixTime \</b>
<b> data/sensor-pressure.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
unixTime temperature humidity pressure
1740000000 37.4 50.3 1012.3
1740000060 37.6 - 1011.9
1740000120 37.9 52.5 1011.6
</pre>
The `unsparsify` must come before the `cut`-and-`label` step, since `label` renames columns positionally. The `sort` is there because unpaired records may be emitted out of order relative to paired ones.
## How to preprocess the left file of a join?
The left file (the `-f` argument to `join`) is opened by the `join` verb itself, so it doesn't pass through the main record stream: a `then`-chain can preprocess the right file(s), but not the left one.

64
docs/src/questions-about-joins.md.in
View file

@ -130,6 +130,70 @@ mlr --icsv --opprint join -f multi-join/name-lookup.csv -j id \
multi-join/input.csv
GENMD-EOF
## Merging several files on a common key when column names collide
The previous example worked painlessly because the non-key column names -- `name` and `status` -- were different in each lookup file. Now suppose you have several files, each containing measurements of a _different_ quantity, but all with the _same_ column names -- say, one file each for temperature, humidity, and pressure, keyed by timestamp:
GENMD-RUN-COMMAND
cat data/sensor-temp.csv
GENMD-EOF
GENMD-RUN-COMMAND
cat data/sensor-humidity.csv
GENMD-EOF
GENMD-RUN-COMMAND
cat data/sensor-pressure.csv
GENMD-EOF
(Note that the humidity file is missing a row for the middle timestamp.)
If we merge these with a `then`-chain of `join` commands, as in the previous section, columns are lost: since every file's non-key columns have the same names, each join step overwrites the values from the step before, and only one file's values survive:
GENMD-RUN-COMMAND
mlr --icsv --opprint \
join -j unixTime -f data/sensor-temp.csv \
then join -j unixTime -f data/sensor-humidity.csv \
data/sensor-pressure.csv
GENMD-EOF
The fix is `join`'s `--lp` (left-prefix) option, which renames the non-key columns coming from each `-f` file so that nothing collides:
GENMD-RUN-COMMAND
mlr --icsv --opprint \
join --lp temp: -j unixTime -f data/sensor-temp.csv \
then join --lp humidity: -j unixTime -f data/sensor-humidity.csv \
data/sensor-pressure.csv
GENMD-EOF
The columns from the file at the end of the command line -- here, the pressure file -- keep their unprefixed names.
If you want just one value column per file, you can also use `join`'s `--lk` option to keep only that column from each `-f` file, then use [cut](reference-verbs.md#cut) and [label](reference-verbs.md#label) to arrange and rename the output columns:
GENMD-RUN-COMMAND
mlr --icsv --opprint \
join --lp temp: --lk averageValue -j unixTime -f data/sensor-temp.csv \
then join --lp humidity: --lk averageValue -j unixTime -f data/sensor-humidity.csv \
then cut -o -f unixTime,temp:averageValue,humidity:averageValue,averageValue \
then label unixTime,temperature,humidity,pressure \
data/sensor-pressure.csv
GENMD-EOF
Note that `join` gives inner-join semantics by default, so the timestamp missing from the humidity file has been dropped from the output. (This is also why the `paste` command is not a substitute for `join` here: `paste` matches rows by position, so a row missing from one file shifts that file's remaining values onto the wrong rows.) If you'd rather keep those rows, with empty cells where a file has no data, add `--ul --ur` to each join step, and [unsparsify](reference-verbs.md#unsparsify) afterward:
GENMD-RUN-COMMAND
mlr --icsv --opprint \
join --ul --ur --lp temp: --lk averageValue -j unixTime -f data/sensor-temp.csv \
then join --ul --ur --lp humidity: --lk averageValue -j unixTime -f data/sensor-humidity.csv \
then unsparsify \
then cut -o -f unixTime,temp:averageValue,humidity:averageValue,averageValue \
then label unixTime,temperature,humidity,pressure \
then sort -t unixTime \
data/sensor-pressure.csv
GENMD-EOF
The `unsparsify` must come before the `cut`-and-`label` step, since `label` renames columns positionally. The `sort` is there because unpaired records may be emitted out of order relative to paired ones.
## How to preprocess the left file of a join?
The left file (the `-f` argument to `join`) is opened by the `join` verb itself, so it doesn't pass through the main record stream: a `then`-chain can preprocess the right file(s), but not the left one.