Mirrors/miller
1
0
Fork 0
mirror of https://github.com/johnkerl/miller.git synced 2026-07-17 16:38:54 +00:00
Code Issues Projects Releases Wiki Activity

Document --lazy-quotes semantics for unmatched opening quotes (#999) (#2165)

Browse source 
The behavior reported in #999 matches Go's encoding/csv LazyQuotes
semantics (and Python's csv module): a field beginning with an unmatched
double quote is a quoted field whose contents extend across field
separators and newlines until the next quote character or end of file.

Add a docs section explaining what --lazy-quotes does and does not
relax, with a worked example, and point to CSV-lite as the alternative
when quote characters are ordinary data.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
John Kerl 2026-07-06 17:16:50 -04:00 • committed by GitHub
parent 405be3542c
commit e291280fde
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
5 changed files with 123 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

3
docs/src/data/lazy-quotes.csv Normal file
View file

@ -0,0 +1,3 @@
id,name,flag,city
1,"ACME CORP. INC,Q8,Rome
2,BETA,X,Milan
Can't render this file because it contains an unexpected character in line 3 and column 16.

73
docs/src/file-formats.md
View file

@ -237,6 +237,79 @@ CSV, TSV, CSV-lite, and TSV-lite have in common the `--implicit-csv-header` flag
See also the [`--lazy-quotes` flag](reference-main-flag-list.md#csv-only-flags), which can help with CSV files that are not fully compliant with RFC-4180.
### Handling stray quote characters
The [`--lazy-quotes` flag](reference-main-flag-list.md#csv-only-flags) makes two specific
relaxations to RFC-4180 parsing (following the semantics of the
[Go CSV library](https://pkg.go.dev/encoding/csv)): a quote may appear inside an *unquoted*
field, and a non-doubled quote may appear inside a *quoted* field.
What it does **not** change is how quoted fields are delimited. A field whose first character
is a double quote is still a quoted field, and its contents extend -- across field separators
and even line endings -- until the next double quote. In particular, if a field has an
unmatched opening quote, everything up to the next quote character in the file (or the end
of the file) becomes part of that field. This matches the behavior of the Go CSV library
with `LazyQuotes`, as well as Python's `csv` module. For example:
<pre class="pre-highlight-in-pair">
<b>cat data/lazy-quotes.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
id,name,flag,city
1,"ACME CORP. INC,Q8,Rome
2,BETA,X,Milan
</pre>
Here the second field of the first data line has an opening quote with no matching close,
so without `--lazy-quotes` we get an error:
<pre class="pre-highlight-in-pair">
<b>mlr --icsv --ojson cat data/lazy-quotes.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
mlr: CSV header/data length mismatch 4 != 1 at filename data/lazy-quotes.csv row 2
</pre>
With `--lazy-quotes`, the quoted field silently absorbs the rest of the file -- including
the field separators and the newline -- since there is no closing quote anywhere:
<pre class="pre-highlight-in-pair">
<b>mlr --icsv --ojson --lazy-quotes --allow-ragged-csv-input cat data/lazy-quotes.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
[
{
"id": 1,
"name": "ACME CORP. INC,Q8,Rome\n2,BETA,X,Milan\n"
}
]
</pre>
If quote characters in your data are really just ordinary data characters -- that is, the
file doesn't use RFC-4180-style quoting at all -- then [CSV-lite](file-formats.md#csvtsvasvusvetc)
is often a better choice, since it splits on the field separator without treating quotes
specially (the literal quote character is retained in the data):
<pre class="pre-highlight-in-pair">
<b>mlr --icsvlite --ojson cat data/lazy-quotes.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
[
{
"id": 1,
"name": "\"ACME CORP. INC",
"flag": "Q8",
"city": "Rome"
},
{
"id": 2,
"name": "BETA",
"flag": "X",
"city": "Milan"
}
]
</pre>
### Troubleshooting CSV and JSON input
Please see [this page](troubleshooting-csv-and-json-input.md).

41
docs/src/file-formats.md.in
View file

@ -79,6 +79,47 @@ CSV, TSV, CSV-lite, and TSV-lite have in common the `--implicit-csv-header` flag
See also the [`--lazy-quotes` flag](reference-main-flag-list.md#csv-only-flags), which can help with CSV files that are not fully compliant with RFC-4180.
### Handling stray quote characters
The [`--lazy-quotes` flag](reference-main-flag-list.md#csv-only-flags) makes two specific
relaxations to RFC-4180 parsing (following the semantics of the
[Go CSV library](https://pkg.go.dev/encoding/csv)): a quote may appear inside an *unquoted*
field, and a non-doubled quote may appear inside a *quoted* field.
What it does **not** change is how quoted fields are delimited. A field whose first character
is a double quote is still a quoted field, and its contents extend -- across field separators
and even line endings -- until the next double quote. In particular, if a field has an
unmatched opening quote, everything up to the next quote character in the file (or the end
of the file) becomes part of that field. This matches the behavior of the Go CSV library
with `LazyQuotes`, as well as Python's `csv` module. For example:
GENMD-RUN-COMMAND
cat data/lazy-quotes.csv
GENMD-EOF
Here the second field of the first data line has an opening quote with no matching close,
so without `--lazy-quotes` we get an error:
GENMD-RUN-COMMAND-TOLERATING-ERROR
mlr --icsv --ojson cat data/lazy-quotes.csv
GENMD-EOF
With `--lazy-quotes`, the quoted field silently absorbs the rest of the file -- including
the field separators and the newline -- since there is no closing quote anywhere:
GENMD-RUN-COMMAND
mlr --icsv --ojson --lazy-quotes --allow-ragged-csv-input cat data/lazy-quotes.csv
GENMD-EOF
If quote characters in your data are really just ordinary data characters -- that is, the
file doesn't use RFC-4180-style quoting at all -- then [CSV-lite](file-formats.md#csvtsvasvusvetc)
is often a better choice, since it splits on the field separator without treating quotes
specially (the literal quote character is retained in the data):
GENMD-RUN-COMMAND
mlr --icsvlite --ojson cat data/lazy-quotes.csv
GENMD-EOF
### Troubleshooting CSV and JSON input
Please see [this page](troubleshooting-csv-and-json-input.md).

4
docs/src/troubleshooting-csv-and-json-input.md
View file

@ -30,7 +30,9 @@ When Miller reports a parse error, it is often useful to first identify whether
- For CSV generated by spreadsheets or hand-written scripts, `--csv-trim-leading-space` can help
when fields look like `"foo", "bar"`, where the second field has a leading space before the quote.
- For non-standard CSV that contains stray quote characters, `--lazy-quotes` can help Miller accept
input that is not fully RFC-4180 compliant.
input that is not fully RFC-4180 compliant. Note that a field with an unmatched _opening_ quote
still absorbs field separators and newlines until the next quote character or end of file; see
[Handling stray quote characters](file-formats.md#handling-stray-quote-characters).
- For headerless CSV, add `--implicit-csv-header` (or `-N` when you also want headerless output) so
fields are addressed as `$1`, `$2`, and so on. See also
[CSV, with and without headers](csv-with-and-without-headers.md).

4
docs/src/troubleshooting-csv-and-json-input.md.in
View file

@ -14,7 +14,9 @@ When Miller reports a parse error, it is often useful to first identify whether
- For CSV generated by spreadsheets or hand-written scripts, `--csv-trim-leading-space` can help
when fields look like `"foo", "bar"`, where the second field has a leading space before the quote.
- For non-standard CSV that contains stray quote characters, `--lazy-quotes` can help Miller accept
input that is not fully RFC-4180 compliant.
input that is not fully RFC-4180 compliant. Note that a field with an unmatched _opening_ quote
still absorbs field separators and newlines until the next quote character or end of file; see
[Handling stray quote characters](file-formats.md#handling-stray-quote-characters).
- For headerless CSV, add `--implicit-csv-header` (or `-N` when you also want headerless output) so
fields are addressed as `$1`, `$2`, and so on. See also
[CSV, with and without headers](csv-with-and-without-headers.md).