Mirrors/miller
1
0
Fork 0
mirror of https://github.com/johnkerl/miller.git synced 2026-07-17 16:38:54 +00:00
Code Issues Projects Releases Wiki Activity

docs: add CSV and JSON troubleshooting tips (#2123)

Browse source 
Co-authored-by: John Kerl <kerl.john.r@gmail.com>
This commit is contained in:
Kral 2026-07-03 08:44:42 +08:00 • committed by GitHub
parent 5eafa1c9f9
commit e682484a2a
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 34 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

17
docs/src/file-formats.md
View file

@ -237,6 +237,23 @@ CSV, TSV, CSV-lite, and TSV-lite have in common the `--implicit-csv-header` flag
See also the [`--lazy-quotes` flag](reference-main-flag-list.md#csv-only-flags), which can help with CSV files that are not fully compliant with RFC-4180.
### Troubleshooting CSV and JSON input
When Miller reports a parse error, it is often useful to first identify whether the issue is in the file format, in the chosen input flag, or in the shell quoting around the Miller expression.
* Use the input-format flag which matches the data you are reading. For example, use `--icsv` for CSV input and `--ijson` for JSON input, even if you are writing a different format such as `--ojson` or `--ocsv`.
* If a CSV file has embedded commas or newlines inside fields, those fields need RFC-4180 double quotes. If the file simply splits on commas with no CSV escaping, try `--csvlite` or `--icsvlite` instead.
* If a CSV error mentions a header/data length mismatch, inspect the reported input line for an unquoted comma, a missing closing quote, or an unexpected extra delimiter.
* For CSV generated by spreadsheets or hand-written scripts, `--csv-trim-leading-space` can help when fields look like `"foo", "bar"`, where the second field has a leading space before the quote.
* For non-standard CSV that contains stray quote characters, `--lazy-quotes` can help Miller accept input that is not fully RFC-4180 compliant.
* For headerless CSV, add `--implicit-csv-header` (or `-N` when you also want headerless output) so fields are addressed as `$1`, `$2`, and so on.
* JSON input to Miller should be tabular: a single object, a stream of objects, or an array of objects. A top-level scalar such as `3`, or an array of scalars such as `[1,2,3]`, is valid JSON but not a Miller record stream.
* When converting nested JSON to CSV or other tabular formats, remember that Miller may flatten nested keys. Use `--flatsep` to choose a separator, or `--no-auto-flatten` when you want to preserve nested values in non-JSON output.
* Use `--ijsonl` for newline-delimited JSON objects (JSON Lines). Use `--ijson` for ordinary JSON arrays or single objects.
* In shell commands, wrap Miller DSL expressions in single quotes so the shell does not expand `$field` names before Miller sees them: `mlr --icsv put '$total = $price * $quantity' data.csv`.
* If the DSL expression itself needs a single quote, put the expression in a script file and use `-f script.mlr`, or use your shell's quoting rules carefully.
* To isolate format issues from DSL issues, start with a simple command such as `mlr --icsv head -n 3 file.csv` or `mlr --ijson head -n 3 file.json`, then add verbs one at a time.
## JSON
[JSON](https://json.org) is a format which supports scalars (numbers, strings,

17
docs/src/file-formats.md.in
View file

@ -79,6 +79,23 @@ CSV, TSV, CSV-lite, and TSV-lite have in common the `--implicit-csv-header` flag
See also the [`--lazy-quotes` flag](reference-main-flag-list.md#csv-only-flags), which can help with CSV files that are not fully compliant with RFC-4180.
### Troubleshooting CSV and JSON input
When Miller reports a parse error, it is often useful to first identify whether the issue is in the file format, in the chosen input flag, or in the shell quoting around the Miller expression.
* Use the input-format flag which matches the data you are reading. For example, use `--icsv` for CSV input and `--ijson` for JSON input, even if you are writing a different format such as `--ojson` or `--ocsv`.
* If a CSV file has embedded commas or newlines inside fields, those fields need RFC-4180 double quotes. If the file simply splits on commas with no CSV escaping, try `--csvlite` or `--icsvlite` instead.
* If a CSV error mentions a header/data length mismatch, inspect the reported input line for an unquoted comma, a missing closing quote, or an unexpected extra delimiter.
* For CSV generated by spreadsheets or hand-written scripts, `--csv-trim-leading-space` can help when fields look like `"foo", "bar"`, where the second field has a leading space before the quote.
* For non-standard CSV that contains stray quote characters, `--lazy-quotes` can help Miller accept input that is not fully RFC-4180 compliant.
* For headerless CSV, add `--implicit-csv-header` (or `-N` when you also want headerless output) so fields are addressed as `$1`, `$2`, and so on.
* JSON input to Miller should be tabular: a single object, a stream of objects, or an array of objects. A top-level scalar such as `3`, or an array of scalars such as `[1,2,3]`, is valid JSON but not a Miller record stream.
* When converting nested JSON to CSV or other tabular formats, remember that Miller may flatten nested keys. Use `--flatsep` to choose a separator, or `--no-auto-flatten` when you want to preserve nested values in non-JSON output.
* Use `--ijsonl` for newline-delimited JSON objects (JSON Lines). Use `--ijson` for ordinary JSON arrays or single objects.
* In shell commands, wrap Miller DSL expressions in single quotes so the shell does not expand `$field` names before Miller sees them: `mlr --icsv put '$total = $price * $quantity' data.csv`.
* If the DSL expression itself needs a single quote, put the expression in a script file and use `-f script.mlr`, or use your shell's quoting rules carefully.
* To isolate format issues from DSL issues, start with a simple command such as `mlr --icsv head -n 3 file.csv` or `mlr --ijson head -n 3 file.json`, then add verbs one at a time.
## JSON
[JSON](https://json.org) is a format which supports scalars (numbers, strings,