miller/docs/src/troubleshooting-csv-and-json-input.md

Quick links:   Flags   Verbs   Functions   Glossary   Release docs

# Troubleshooting CSV and JSON input

When Miller reports a parse error, it is often useful to first identify whether the issue is in the file format, in the chosen input flag, or in the shell quoting around the Miller expression.

• Use the input-format flag which matches the data you are reading. For example, use --icsv for CSV input and --ijson for JSON input, even if you are writing a different format such as --ojson or --ocsv.
• If a CSV file has embedded commas or newlines inside fields, those fields need RFC-4180 double quotes: you need --icsv. If the file simply splits on commas with no CSV escaping, try --csvlite or --icsvlite instead. See also File Formats.
• If a CSV error mentions a "header/data length mismatch", inspect the reported input line for an unquoted comma, a missing closing quote, or an unexpected extra delimiter.
• For CSV generated by spreadsheets or hand-written scripts, --csv-trim-leading-space can help when fields look like "foo", "bar", where the second field has a leading space before the quote.
• For non-standard CSV that contains stray quote characters, --lazy-quotes can help Miller accept input that is not fully RFC-4180 compliant. Note that a field with an unmatched opening quote still absorbs field separators and newlines until the next quote character or end of file; see Handling stray quote characters.
• For headerless CSV, add --implicit-csv-header (or -N when you also want headerless output) so fields are addressed as $1, $2, and so on. See also CSV, with and without headers.
• JSON input to Miller should be tabular: a single object, a stream of objects, or an array of objects. A top-level scalar such as 3, or an array of scalars such as [1,2,3], is valid JSON but not a Miller record. See also this page.
• When converting nested JSON to CSV or other tabular formats, remember that Miller may flatten nested keys. Use --flatsep to choose a separator, or --no-auto-flatten when you want to preserve nested values in non-JSON output. See also Flatten/unflatten: converting between JSON and tabular formats.
• Use --ijsonl for newline-delimited JSON objects (JSON Lines). Use --ijson for ordinary JSON arrays or single objects.
• In shell commands, wrap Miller DSL expressions in single quotes so the shell does not expand $field names before Miller sees them: mlr --icsv put '$total = $price * $quantity' data.csv.
• If the DSL expression itself needs a single quote, put the expression in a script file and use -f script.mlr, or use your shell's quoting rules carefully. See also Miller on Windows for some complexity issues on Windows.
• To isolate format issues from DSL issues, start with a simple command such as mlr --icsv head -n 3 file.csv or mlr --ijson head -n 3 file.json, then add verbs one at a time. See also Questions about then-chaining.