diff --git a/docs/src/data/lazy-quotes.csv b/docs/src/data/lazy-quotes.csv new file mode 100644 index 000000000..13074bd42 --- /dev/null +++ b/docs/src/data/lazy-quotes.csv @@ -0,0 +1,3 @@ +id,name,flag,city +1,"ACME CORP. INC,Q8,Rome +2,BETA,X,Milan diff --git a/docs/src/file-formats.md b/docs/src/file-formats.md index c39278211..685d1433b 100644 --- a/docs/src/file-formats.md +++ b/docs/src/file-formats.md @@ -237,6 +237,79 @@ CSV, TSV, CSV-lite, and TSV-lite have in common the `--implicit-csv-header` flag See also the [`--lazy-quotes` flag](reference-main-flag-list.md#csv-only-flags), which can help with CSV files that are not fully compliant with RFC-4180. +### Handling stray quote characters + +The [`--lazy-quotes` flag](reference-main-flag-list.md#csv-only-flags) makes two specific +relaxations to RFC-4180 parsing (following the semantics of the +[Go CSV library](https://pkg.go.dev/encoding/csv)): a quote may appear inside an *unquoted* +field, and a non-doubled quote may appear inside a *quoted* field. + +What it does **not** change is how quoted fields are delimited. A field whose first character +is a double quote is still a quoted field, and its contents extend -- across field separators +and even line endings -- until the next double quote. In particular, if a field has an +unmatched opening quote, everything up to the next quote character in the file (or the end +of the file) becomes part of that field. This matches the behavior of the Go CSV library +with `LazyQuotes`, as well as Python's `csv` module. For example: + +
+cat data/lazy-quotes.csv
+
+
+id,name,flag,city
+1,"ACME CORP. INC,Q8,Rome
+2,BETA,X,Milan
+
+ +Here the second field of the first data line has an opening quote with no matching close, +so without `--lazy-quotes` we get an error: + +
+mlr --icsv --ojson cat data/lazy-quotes.csv
+
+
+mlr: CSV header/data length mismatch 4 != 1 at filename data/lazy-quotes.csv row 2
+
+ +With `--lazy-quotes`, the quoted field silently absorbs the rest of the file -- including +the field separators and the newline -- since there is no closing quote anywhere: + +
+mlr --icsv --ojson --lazy-quotes --allow-ragged-csv-input cat data/lazy-quotes.csv
+
+
+[
+{
+  "id": 1,
+  "name": "ACME CORP. INC,Q8,Rome\n2,BETA,X,Milan\n"
+}
+]
+
+ +If quote characters in your data are really just ordinary data characters -- that is, the +file doesn't use RFC-4180-style quoting at all -- then [CSV-lite](file-formats.md#csvtsvasvusvetc) +is often a better choice, since it splits on the field separator without treating quotes +specially (the literal quote character is retained in the data): + +
+mlr --icsvlite --ojson cat data/lazy-quotes.csv
+
+
+[
+{
+  "id": 1,
+  "name": "\"ACME CORP. INC",
+  "flag": "Q8",
+  "city": "Rome"
+},
+{
+  "id": 2,
+  "name": "BETA",
+  "flag": "X",
+  "city": "Milan"
+}
+]
+
+ ### Troubleshooting CSV and JSON input Please see [this page](troubleshooting-csv-and-json-input.md). diff --git a/docs/src/file-formats.md.in b/docs/src/file-formats.md.in index 2322ff474..4a821710a 100644 --- a/docs/src/file-formats.md.in +++ b/docs/src/file-formats.md.in @@ -79,6 +79,47 @@ CSV, TSV, CSV-lite, and TSV-lite have in common the `--implicit-csv-header` flag See also the [`--lazy-quotes` flag](reference-main-flag-list.md#csv-only-flags), which can help with CSV files that are not fully compliant with RFC-4180. +### Handling stray quote characters + +The [`--lazy-quotes` flag](reference-main-flag-list.md#csv-only-flags) makes two specific +relaxations to RFC-4180 parsing (following the semantics of the +[Go CSV library](https://pkg.go.dev/encoding/csv)): a quote may appear inside an *unquoted* +field, and a non-doubled quote may appear inside a *quoted* field. + +What it does **not** change is how quoted fields are delimited. A field whose first character +is a double quote is still a quoted field, and its contents extend -- across field separators +and even line endings -- until the next double quote. In particular, if a field has an +unmatched opening quote, everything up to the next quote character in the file (or the end +of the file) becomes part of that field. This matches the behavior of the Go CSV library +with `LazyQuotes`, as well as Python's `csv` module. For example: + +GENMD-RUN-COMMAND +cat data/lazy-quotes.csv +GENMD-EOF + +Here the second field of the first data line has an opening quote with no matching close, +so without `--lazy-quotes` we get an error: + +GENMD-RUN-COMMAND-TOLERATING-ERROR +mlr --icsv --ojson cat data/lazy-quotes.csv +GENMD-EOF + +With `--lazy-quotes`, the quoted field silently absorbs the rest of the file -- including +the field separators and the newline -- since there is no closing quote anywhere: + +GENMD-RUN-COMMAND +mlr --icsv --ojson --lazy-quotes --allow-ragged-csv-input cat data/lazy-quotes.csv +GENMD-EOF + +If quote characters in your data are really just ordinary data characters -- that is, the +file doesn't use RFC-4180-style quoting at all -- then [CSV-lite](file-formats.md#csvtsvasvusvetc) +is often a better choice, since it splits on the field separator without treating quotes +specially (the literal quote character is retained in the data): + +GENMD-RUN-COMMAND +mlr --icsvlite --ojson cat data/lazy-quotes.csv +GENMD-EOF + ### Troubleshooting CSV and JSON input Please see [this page](troubleshooting-csv-and-json-input.md). diff --git a/docs/src/troubleshooting-csv-and-json-input.md b/docs/src/troubleshooting-csv-and-json-input.md index 449b4a854..49c3880ab 100644 --- a/docs/src/troubleshooting-csv-and-json-input.md +++ b/docs/src/troubleshooting-csv-and-json-input.md @@ -30,7 +30,9 @@ When Miller reports a parse error, it is often useful to first identify whether - For CSV generated by spreadsheets or hand-written scripts, `--csv-trim-leading-space` can help when fields look like `"foo", "bar"`, where the second field has a leading space before the quote. - For non-standard CSV that contains stray quote characters, `--lazy-quotes` can help Miller accept - input that is not fully RFC-4180 compliant. + input that is not fully RFC-4180 compliant. Note that a field with an unmatched _opening_ quote + still absorbs field separators and newlines until the next quote character or end of file; see + [Handling stray quote characters](file-formats.md#handling-stray-quote-characters). - For headerless CSV, add `--implicit-csv-header` (or `-N` when you also want headerless output) so fields are addressed as `$1`, `$2`, and so on. See also [CSV, with and without headers](csv-with-and-without-headers.md). diff --git a/docs/src/troubleshooting-csv-and-json-input.md.in b/docs/src/troubleshooting-csv-and-json-input.md.in index 1bcdd3074..543e7ab7f 100644 --- a/docs/src/troubleshooting-csv-and-json-input.md.in +++ b/docs/src/troubleshooting-csv-and-json-input.md.in @@ -14,7 +14,9 @@ When Miller reports a parse error, it is often useful to first identify whether - For CSV generated by spreadsheets or hand-written scripts, `--csv-trim-leading-space` can help when fields look like `"foo", "bar"`, where the second field has a leading space before the quote. - For non-standard CSV that contains stray quote characters, `--lazy-quotes` can help Miller accept - input that is not fully RFC-4180 compliant. + input that is not fully RFC-4180 compliant. Note that a field with an unmatched _opening_ quote + still absorbs field separators and newlines until the next quote character or end of file; see + [Handling stray quote characters](file-formats.md#handling-stray-quote-characters). - For headerless CSV, add `--implicit-csv-header` (or `-N` when you also want headerless output) so fields are addressed as `$1`, `$2`, and so on. See also [CSV, with and without headers](csv-with-and-without-headers.md).