diff --git a/docs/src/data/het/sales-2021.csv b/docs/src/data/het/sales-2021.csv new file mode 100644 index 000000000..e1476218f --- /dev/null +++ b/docs/src/data/het/sales-2021.csv @@ -0,0 +1,3 @@ +id,product,sales +1,pencil,100 +2,eraser,17 diff --git a/docs/src/data/het/sales-2022.csv b/docs/src/data/het/sales-2022.csv new file mode 100644 index 000000000..142bbcae2 --- /dev/null +++ b/docs/src/data/het/sales-2022.csv @@ -0,0 +1,3 @@ +id,product,color,sales +3,pencil,red,120 +4,eraser,green,32 diff --git a/docs/src/data/het/sales-2023.csv b/docs/src/data/het/sales-2023.csv new file mode 100644 index 000000000..1053c178c --- /dev/null +++ b/docs/src/data/het/sales-2023.csv @@ -0,0 +1,3 @@ +id,product,sales,returns +5,pencil,200,6 +6,notebook,41,2 diff --git a/docs/src/record-heterogeneity.md b/docs/src/record-heterogeneity.md index fe4126121..5daed3644 100644 --- a/docs/src/record-heterogeneity.md +++ b/docs/src/record-heterogeneity.md @@ -474,6 +474,95 @@ a,b,c 7,8,9,10 +### Combining multiple CSV files with different headers + +A common special case of the above: you have several CSV files whose column names overlap but +don't all match -- say, data exported year by year, where columns were added or reordered over +time -- and you want to concatenate them into a single rectangular table, with empty cells where a +file didn't have a given column. + +
+cat data/het/sales-2021.csv ++
+id,product,sales +1,pencil,100 +2,eraser,17 ++ +
+cat data/het/sales-2022.csv ++
+id,product,color,sales +3,pencil,red,120 +4,eraser,green,32 ++ +
+cat data/het/sales-2023.csv ++
+id,product,sales,returns +5,pencil,200,6 +6,notebook,41,2 ++ +Since each file's records take their schema from that file's header line, simply catting the +files together gives a schema change partway through the record stream -- which RFC-4180 CSV +output doesn't allow: + +
+mlr --csv cat data/het/sales-2021.csv data/het/sales-2022.csv data/het/sales-2023.csv ++
+id,product,sales +1,pencil,100 +2,eraser,17 +mlr: CSV schema change: first keys "id,product,sales"; current keys "id,product,color,sales" +mlr: exiting due to data error ++ +The solution is the [`unsparsify`](reference-verbs.md#unsparsify) verb, which we saw above: it +gives every record the union of all field names, filling in empty values (or, using +`--fill-with`, whatever you like) where a record lacks a field: + +
+mlr --csv unsparsify data/het/sales-2021.csv data/het/sales-2022.csv data/het/sales-2023.csv ++
+id,product,sales,color,returns +1,pencil,100,, +2,eraser,17,, +3,pencil,120,red, +4,eraser,32,green, +5,pencil,200,,6 +6,notebook,41,,2 ++ +Note that `unsparsify` (without `-f`) is +[non-streaming](streaming-and-memory.md): it can't know the full set of field names until it has +read all input, so it retains all records in memory before producing any output. If your files +are too large for that, but you know the complete list of field names up front, you can use +`unsparsify -f` which is streaming. Fields filled in by `-f` are appended to each record, so if +the input files order their columns differently, follow up with +[`regularize`](reference-verbs.md#regularize) or +[`sort-within-records`](reference-verbs.md#sort-within-records) (also both streaming) to give all +records the same column ordering: + +
+mlr --csv unsparsify -f id,product,sales,color,returns then regularize data/het/sales-2021.csv data/het/sales-2022.csv data/het/sales-2023.csv ++
+id,product,sales,color,returns +1,pencil,100,, +2,eraser,17,, +3,pencil,120,red, +4,eraser,32,green, +5,pencil,200,,6 +6,notebook,41,,2 ++ ## Processing heterogeneous data Above we saw how to make heterogeneous data homogeneous, and then how to print heterogeneous data. diff --git a/docs/src/record-heterogeneity.md.in b/docs/src/record-heterogeneity.md.in index e3c128b57..806784c99 100644 --- a/docs/src/record-heterogeneity.md.in +++ b/docs/src/record-heterogeneity.md.in @@ -205,6 +205,55 @@ GENMD-RUN-COMMAND mlr --csv --allow-ragged-csv-input cat data/het/ragged.csv GENMD-EOF +### Combining multiple CSV files with different headers + +A common special case of the above: you have several CSV files whose column names overlap but +don't all match -- say, data exported year by year, where columns were added or reordered over +time -- and you want to concatenate them into a single rectangular table, with empty cells where a +file didn't have a given column. + +GENMD-RUN-COMMAND +cat data/het/sales-2021.csv +GENMD-EOF + +GENMD-RUN-COMMAND +cat data/het/sales-2022.csv +GENMD-EOF + +GENMD-RUN-COMMAND +cat data/het/sales-2023.csv +GENMD-EOF + +Since each file's records take their schema from that file's header line, simply catting the +files together gives a schema change partway through the record stream -- which RFC-4180 CSV +output doesn't allow: + +GENMD-RUN-COMMAND-TOLERATING-ERROR +mlr --csv cat data/het/sales-2021.csv data/het/sales-2022.csv data/het/sales-2023.csv +GENMD-EOF + +The solution is the [`unsparsify`](reference-verbs.md#unsparsify) verb, which we saw above: it +gives every record the union of all field names, filling in empty values (or, using +`--fill-with`, whatever you like) where a record lacks a field: + +GENMD-RUN-COMMAND +mlr --csv unsparsify data/het/sales-2021.csv data/het/sales-2022.csv data/het/sales-2023.csv +GENMD-EOF + +Note that `unsparsify` (without `-f`) is +[non-streaming](streaming-and-memory.md): it can't know the full set of field names until it has +read all input, so it retains all records in memory before producing any output. If your files +are too large for that, but you know the complete list of field names up front, you can use +`unsparsify -f` which is streaming. Fields filled in by `-f` are appended to each record, so if +the input files order their columns differently, follow up with +[`regularize`](reference-verbs.md#regularize) or +[`sort-within-records`](reference-verbs.md#sort-within-records) (also both streaming) to give all +records the same column ordering: + +GENMD-RUN-COMMAND +mlr --csv unsparsify -f id,product,sales,color,returns then regularize data/het/sales-2021.csv data/het/sales-2022.csv data/het/sales-2023.csv +GENMD-EOF + ## Processing heterogeneous data Above we saw how to make heterogeneous data homogeneous, and then how to print heterogeneous data.