diff --git a/docs/src/file-formats.md b/docs/src/file-formats.md index cd24cb09a..efe35e6cb 100644 --- a/docs/src/file-formats.md +++ b/docs/src/file-formats.md @@ -237,6 +237,23 @@ CSV, TSV, CSV-lite, and TSV-lite have in common the `--implicit-csv-header` flag See also the [`--lazy-quotes` flag](reference-main-flag-list.md#csv-only-flags), which can help with CSV files that are not fully compliant with RFC-4180. +### Troubleshooting CSV and JSON input + +When Miller reports a parse error, it is often useful to first identify whether the issue is in the file format, in the chosen input flag, or in the shell quoting around the Miller expression. + +* Use the input-format flag which matches the data you are reading. For example, use `--icsv` for CSV input and `--ijson` for JSON input, even if you are writing a different format such as `--ojson` or `--ocsv`. +* If a CSV file has embedded commas or newlines inside fields, those fields need RFC-4180 double quotes. If the file simply splits on commas with no CSV escaping, try `--csvlite` or `--icsvlite` instead. +* If a CSV error mentions a header/data length mismatch, inspect the reported input line for an unquoted comma, a missing closing quote, or an unexpected extra delimiter. +* For CSV generated by spreadsheets or hand-written scripts, `--csv-trim-leading-space` can help when fields look like `"foo", "bar"`, where the second field has a leading space before the quote. +* For non-standard CSV that contains stray quote characters, `--lazy-quotes` can help Miller accept input that is not fully RFC-4180 compliant. +* For headerless CSV, add `--implicit-csv-header` (or `-N` when you also want headerless output) so fields are addressed as `$1`, `$2`, and so on. +* JSON input to Miller should be tabular: a single object, a stream of objects, or an array of objects. A top-level scalar such as `3`, or an array of scalars such as `[1,2,3]`, is valid JSON but not a Miller record stream. +* When converting nested JSON to CSV or other tabular formats, remember that Miller may flatten nested keys. Use `--flatsep` to choose a separator, or `--no-auto-flatten` when you want to preserve nested values in non-JSON output. +* Use `--ijsonl` for newline-delimited JSON objects (JSON Lines). Use `--ijson` for ordinary JSON arrays or single objects. +* In shell commands, wrap Miller DSL expressions in single quotes so the shell does not expand `$field` names before Miller sees them: `mlr --icsv put '$total = $price * $quantity' data.csv`. +* If the DSL expression itself needs a single quote, put the expression in a script file and use `-f script.mlr`, or use your shell's quoting rules carefully. +* To isolate format issues from DSL issues, start with a simple command such as `mlr --icsv head -n 3 file.csv` or `mlr --ijson head -n 3 file.json`, then add verbs one at a time. + ## JSON [JSON](https://json.org) is a format which supports scalars (numbers, strings, diff --git a/docs/src/file-formats.md.in b/docs/src/file-formats.md.in index 00f3dc908..e49327600 100644 --- a/docs/src/file-formats.md.in +++ b/docs/src/file-formats.md.in @@ -79,6 +79,23 @@ CSV, TSV, CSV-lite, and TSV-lite have in common the `--implicit-csv-header` flag See also the [`--lazy-quotes` flag](reference-main-flag-list.md#csv-only-flags), which can help with CSV files that are not fully compliant with RFC-4180. +### Troubleshooting CSV and JSON input + +When Miller reports a parse error, it is often useful to first identify whether the issue is in the file format, in the chosen input flag, or in the shell quoting around the Miller expression. + +* Use the input-format flag which matches the data you are reading. For example, use `--icsv` for CSV input and `--ijson` for JSON input, even if you are writing a different format such as `--ojson` or `--ocsv`. +* If a CSV file has embedded commas or newlines inside fields, those fields need RFC-4180 double quotes. If the file simply splits on commas with no CSV escaping, try `--csvlite` or `--icsvlite` instead. +* If a CSV error mentions a header/data length mismatch, inspect the reported input line for an unquoted comma, a missing closing quote, or an unexpected extra delimiter. +* For CSV generated by spreadsheets or hand-written scripts, `--csv-trim-leading-space` can help when fields look like `"foo", "bar"`, where the second field has a leading space before the quote. +* For non-standard CSV that contains stray quote characters, `--lazy-quotes` can help Miller accept input that is not fully RFC-4180 compliant. +* For headerless CSV, add `--implicit-csv-header` (or `-N` when you also want headerless output) so fields are addressed as `$1`, `$2`, and so on. +* JSON input to Miller should be tabular: a single object, a stream of objects, or an array of objects. A top-level scalar such as `3`, or an array of scalars such as `[1,2,3]`, is valid JSON but not a Miller record stream. +* When converting nested JSON to CSV or other tabular formats, remember that Miller may flatten nested keys. Use `--flatsep` to choose a separator, or `--no-auto-flatten` when you want to preserve nested values in non-JSON output. +* Use `--ijsonl` for newline-delimited JSON objects (JSON Lines). Use `--ijson` for ordinary JSON arrays or single objects. +* In shell commands, wrap Miller DSL expressions in single quotes so the shell does not expand `$field` names before Miller sees them: `mlr --icsv put '$total = $price * $quantity' data.csv`. +* If the DSL expression itself needs a single quote, put the expression in a script file and use `-f script.mlr`, or use your shell's quoting rules carefully. +* To isolate format issues from DSL issues, start with a simple command such as `mlr --icsv head -n 3 file.csv` or `mlr --ijson head -n 3 file.json`, then add verbs one at a time. + ## JSON [JSON](https://json.org) is a format which supports scalars (numbers, strings,