diff --git a/docs/src/data/lazy-quotes.csv b/docs/src/data/lazy-quotes.csv new file mode 100644 index 000000000..13074bd42 --- /dev/null +++ b/docs/src/data/lazy-quotes.csv @@ -0,0 +1,3 @@ +id,name,flag,city +1,"ACME CORP. INC,Q8,Rome +2,BETA,X,Milan diff --git a/docs/src/file-formats.md b/docs/src/file-formats.md index c39278211..685d1433b 100644 --- a/docs/src/file-formats.md +++ b/docs/src/file-formats.md @@ -237,6 +237,79 @@ CSV, TSV, CSV-lite, and TSV-lite have in common the `--implicit-csv-header` flag See also the [`--lazy-quotes` flag](reference-main-flag-list.md#csv-only-flags), which can help with CSV files that are not fully compliant with RFC-4180. +### Handling stray quote characters + +The [`--lazy-quotes` flag](reference-main-flag-list.md#csv-only-flags) makes two specific +relaxations to RFC-4180 parsing (following the semantics of the +[Go CSV library](https://pkg.go.dev/encoding/csv)): a quote may appear inside an *unquoted* +field, and a non-doubled quote may appear inside a *quoted* field. + +What it does **not** change is how quoted fields are delimited. A field whose first character +is a double quote is still a quoted field, and its contents extend -- across field separators +and even line endings -- until the next double quote. In particular, if a field has an +unmatched opening quote, everything up to the next quote character in the file (or the end +of the file) becomes part of that field. This matches the behavior of the Go CSV library +with `LazyQuotes`, as well as Python's `csv` module. For example: + +
+cat data/lazy-quotes.csv ++
+id,name,flag,city +1,"ACME CORP. INC,Q8,Rome +2,BETA,X,Milan ++ +Here the second field of the first data line has an opening quote with no matching close, +so without `--lazy-quotes` we get an error: + +
+mlr --icsv --ojson cat data/lazy-quotes.csv ++
+mlr: CSV header/data length mismatch 4 != 1 at filename data/lazy-quotes.csv row 2 ++ +With `--lazy-quotes`, the quoted field silently absorbs the rest of the file -- including +the field separators and the newline -- since there is no closing quote anywhere: + +
+mlr --icsv --ojson --lazy-quotes --allow-ragged-csv-input cat data/lazy-quotes.csv ++
+[
+{
+ "id": 1,
+ "name": "ACME CORP. INC,Q8,Rome\n2,BETA,X,Milan\n"
+}
+]
+
+
+If quote characters in your data are really just ordinary data characters -- that is, the
+file doesn't use RFC-4180-style quoting at all -- then [CSV-lite](file-formats.md#csvtsvasvusvetc)
+is often a better choice, since it splits on the field separator without treating quotes
+specially (the literal quote character is retained in the data):
+
++mlr --icsvlite --ojson cat data/lazy-quotes.csv ++
+[
+{
+ "id": 1,
+ "name": "\"ACME CORP. INC",
+ "flag": "Q8",
+ "city": "Rome"
+},
+{
+ "id": 2,
+ "name": "BETA",
+ "flag": "X",
+ "city": "Milan"
+}
+]
+
+
### Troubleshooting CSV and JSON input
Please see [this page](troubleshooting-csv-and-json-input.md).
diff --git a/docs/src/file-formats.md.in b/docs/src/file-formats.md.in
index 2322ff474..4a821710a 100644
--- a/docs/src/file-formats.md.in
+++ b/docs/src/file-formats.md.in
@@ -79,6 +79,47 @@ CSV, TSV, CSV-lite, and TSV-lite have in common the `--implicit-csv-header` flag
See also the [`--lazy-quotes` flag](reference-main-flag-list.md#csv-only-flags), which can help with CSV files that are not fully compliant with RFC-4180.
+### Handling stray quote characters
+
+The [`--lazy-quotes` flag](reference-main-flag-list.md#csv-only-flags) makes two specific
+relaxations to RFC-4180 parsing (following the semantics of the
+[Go CSV library](https://pkg.go.dev/encoding/csv)): a quote may appear inside an *unquoted*
+field, and a non-doubled quote may appear inside a *quoted* field.
+
+What it does **not** change is how quoted fields are delimited. A field whose first character
+is a double quote is still a quoted field, and its contents extend -- across field separators
+and even line endings -- until the next double quote. In particular, if a field has an
+unmatched opening quote, everything up to the next quote character in the file (or the end
+of the file) becomes part of that field. This matches the behavior of the Go CSV library
+with `LazyQuotes`, as well as Python's `csv` module. For example:
+
+GENMD-RUN-COMMAND
+cat data/lazy-quotes.csv
+GENMD-EOF
+
+Here the second field of the first data line has an opening quote with no matching close,
+so without `--lazy-quotes` we get an error:
+
+GENMD-RUN-COMMAND-TOLERATING-ERROR
+mlr --icsv --ojson cat data/lazy-quotes.csv
+GENMD-EOF
+
+With `--lazy-quotes`, the quoted field silently absorbs the rest of the file -- including
+the field separators and the newline -- since there is no closing quote anywhere:
+
+GENMD-RUN-COMMAND
+mlr --icsv --ojson --lazy-quotes --allow-ragged-csv-input cat data/lazy-quotes.csv
+GENMD-EOF
+
+If quote characters in your data are really just ordinary data characters -- that is, the
+file doesn't use RFC-4180-style quoting at all -- then [CSV-lite](file-formats.md#csvtsvasvusvetc)
+is often a better choice, since it splits on the field separator without treating quotes
+specially (the literal quote character is retained in the data):
+
+GENMD-RUN-COMMAND
+mlr --icsvlite --ojson cat data/lazy-quotes.csv
+GENMD-EOF
+
### Troubleshooting CSV and JSON input
Please see [this page](troubleshooting-csv-and-json-input.md).
diff --git a/docs/src/troubleshooting-csv-and-json-input.md b/docs/src/troubleshooting-csv-and-json-input.md
index 449b4a854..49c3880ab 100644
--- a/docs/src/troubleshooting-csv-and-json-input.md
+++ b/docs/src/troubleshooting-csv-and-json-input.md
@@ -30,7 +30,9 @@ When Miller reports a parse error, it is often useful to first identify whether
- For CSV generated by spreadsheets or hand-written scripts, `--csv-trim-leading-space` can help
when fields look like `"foo", "bar"`, where the second field has a leading space before the quote.
- For non-standard CSV that contains stray quote characters, `--lazy-quotes` can help Miller accept
- input that is not fully RFC-4180 compliant.
+ input that is not fully RFC-4180 compliant. Note that a field with an unmatched _opening_ quote
+ still absorbs field separators and newlines until the next quote character or end of file; see
+ [Handling stray quote characters](file-formats.md#handling-stray-quote-characters).
- For headerless CSV, add `--implicit-csv-header` (or `-N` when you also want headerless output) so
fields are addressed as `$1`, `$2`, and so on. See also
[CSV, with and without headers](csv-with-and-without-headers.md).
diff --git a/docs/src/troubleshooting-csv-and-json-input.md.in b/docs/src/troubleshooting-csv-and-json-input.md.in
index 1bcdd3074..543e7ab7f 100644
--- a/docs/src/troubleshooting-csv-and-json-input.md.in
+++ b/docs/src/troubleshooting-csv-and-json-input.md.in
@@ -14,7 +14,9 @@ When Miller reports a parse error, it is often useful to first identify whether
- For CSV generated by spreadsheets or hand-written scripts, `--csv-trim-leading-space` can help
when fields look like `"foo", "bar"`, where the second field has a leading space before the quote.
- For non-standard CSV that contains stray quote characters, `--lazy-quotes` can help Miller accept
- input that is not fully RFC-4180 compliant.
+ input that is not fully RFC-4180 compliant. Note that a field with an unmatched _opening_ quote
+ still absorbs field separators and newlines until the next quote character or end of file; see
+ [Handling stray quote characters](file-formats.md#handling-stray-quote-characters).
- For headerless CSV, add `--implicit-csv-header` (or `-N` when you also want headerless output) so
fields are addressed as `$1`, `$2`, and so on. See also
[CSV, with and without headers](csv-with-and-without-headers.md).