Page-refactor for CSV/JSON troubleshooting (#2128)

* new troubleshooting-csv-and-json-input.md.in page

* ../README-docs.md

* ../README-docs.md

README-docs.md

@@ -1,62 +1,94 @@
 # Miller docs
 
-## Why use Mkdocs
+## Why Mkdocs
 
-* Connects to https://miller.readthedocs.io so people can get their docmods onto the web instead of the self-hosted https://johnkerl.org/miller/doc. Thanks to @pabloab for the great advice!
-* More standard look and feel -- lots of people use readthedocs for other things so this should feel familiar.
-* We get a Search feature for free.
-* Mkdocs vs Sphinx: these are similar tools, but I find that I more easily get better desktop+mobile formatting using Mkdocs.
+- Connects to [https://miller.readthedocs.io](https://miller.readthedocs.io) so people can get their
+  docmods onto the web instead of the self-hosted
+  [https://johnkerl.org/miller/doc](https://johnkerl.org/miller/doc). Thanks to @pabloab for the
+  great advice!
+- More standard look and feel -- lots of people use readthedocs for other things so this should feel
+  familiar.
+- We get a Search feature for free.
+- Mkdocs vs Sphinx: these are similar tools, but I find that I more easily get better desktop+mobile
+  formatting using Mkdocs.
 
-## Contributing
+## Setup
 
-* You need `pip install mkdocs` (or `pip3 install mkdocs`) as well as `pip install mkdocs-material`.
-* The docs include lots of live code examples which will be invoked using `mlr` which must be somewhere in your `$PATH`.
-* Clone https://github.com/johnkerl/miller and cd into `docs/` within your clone.
-* Overview of flow:
-  * `docs/src` has `*.md.in` files containing markdown as well as directives for auto-generating code samples.
-  * A `genmds` script reads `docs/src/*.md.in` and writes `docs/src/*.md`.
-  * The `mkdocs build` tools reads `docs/src/*.md` and writes HTML files in `docs/site`.
-  * Running `make` within the `docs` directory handles both of those steps.
-  * TL;DR just `make docs` from the Miller base directory
-* Quick-editing loop:
-  * In one terminal, cd to the `docs` directory and leave `mkdocs serve` running.
-  * In another terminal, cd to the `docs/src` subdirectory and edit `*.md.in`.
-  * Run `genmds` to re-create all the `*.md` files, or `genmds foo.md.in` to just re-create the `foo.md.in` file you just edited, or (simplest) just `make` within the `docs/src` subdirectory.
-  * In your browser, visit http://127.0.0.1:8000
-  * This doesn't write HTML in `docs/site`; HTML is served up directly in the browser -- this is nice for previewing interactive edits.
-* For-publish editing loop:
-  * cd to the `src` subdirectory of `docs` and edit `*.md.in`.
-  * `make -C ..`
-  * This does write HTML in `docs/site`.
-  * In your browser, visit `file:///your/path/to/miller/docs/site/index.html`
-  * Link-checking:
-    * `sudo pip3 install git+https://github.com/linkchecker/linkchecker.git`
-    * `cd site` and `linkchecker .`
-* Submitting:
-  * Do the for-publish editing steps -- in particular, `docs/src/*.md.in` and `docs/src/*.md` are both checked in to source control.
-    * TL;DR edit `docs/src/foo.md.in` and run `make docs`
-    * If you don't want to do `pip install mkdocs` then feel free to put up a PR which edits a `foo.md.in` as well as its `foo.md`.
-  * `git add` your modified files (`*.md.in` as well as `*.md), `git commit`, `git push`, and submit a PR at https://github.com/johnkerl/miller.
+- `pip install mkdocs` (or `pip3 install mkdocs`) as well as `pip install mkdocs-material`.
+- The docs include lots of live code examples which are invoked using `mlr`, which must be somewhere
+  in your `$PATH`.
+- Clone [https://github.com/johnkerl/miller](https://github.com/johnkerl/miller) and cd into `docs/`
+  within your clone.
 
-## Notes
+## How the build works
 
-* Miller documents use the Oxford comma: not _red, yellow and green_, but rather _red, yellow, and green_.
-* CSS:
-  * I used the Mkdocs "material" theme which I like a lot. I customized `docs/src/extra.css` for Miller coloring/branding.
-* Live code:
-  * I didn't find a way to include non-Python live-code examples within Mkdocs so I adapted the pre-Mkdocs Miller-doc strategy which is to have a generator script read a template file (here, `foo.md.in`), run the marked lines, and generate the output file (`foo.md`). This is `genmds`.
-  * Edit the `*.md.in` files, not `*.md` directly.
-  * Within the `*.md.in` files are lines like `GENMD_RUN_COMMAND`. These will be run, and their output included, by `genmds` which calls the `genmds` script for you.
-* readthedocs:
-  * https://readthedocs.org/
-  * https://readthedocs.org/projects/miller/
-  * https://readthedocs.org/projects/miller/builds/
-  * https://miller.readthedocs.io/en/latest/
+- `docs/src` has `*.md.in` files containing markdown as well as directives for auto-generating code
+  samples. Edit these files, not `*.md` directly.
+- Within a `*.md.in` file, lines like `GENMD_RUN_COMMAND` mark a live code sample: the command gets
+  run, and its output included, when the file is processed.
+- The `genmds` script reads `docs/src/*.md.in` and writes `docs/src/*.md`.
+- `mkdocs build` reads `docs/src/*.md` and writes HTML files in `docs/site`.
+- Running `make` within the `docs` directory handles both of those steps.
+- TL;DR: just `make docs` from the Miller base directory.
 
-## readthedocs website
+## Everyday editing loop
 
-* Published to https://miller.readthedocs.io/en/latest on each commit to `main` in this repo
-* https://readthedocs.org/projects/miller/
-* https://readthedocs.org/api/v2/webhook/miller/134065/
-* https://readthedocs.org/projects/miller/builds/
-* https://readthedocs.org/
+- In one terminal, cd to the `docs` directory and leave `mkdocs serve` running.
+- In another terminal, cd to the `docs/src` subdirectory and edit `*.md.in`.
+- Run `genmds` to re-create all the `*.md` files, or `genmds foo.md.in` to just re-create the
+  `foo.md.in` file you just edited, or (simplest) just `make` within the `docs/src` subdirectory.
+- In your browser, visit [http://127.0.0.1:8000](http://127.0.0.1:8000)
+- This doesn't write HTML in `docs/site`; HTML is served up directly in the browser -- this is nice
+  for previewing interactive edits.
+
+## Adding a new page
+
+- Create `docs/src/foo.md.in`.
+- Add a nav entry in `docs/mkdocs.yml` pointing at `foo.md` -- the filename must match the basename
+  of your new `.md.in` exactly, or the built page won't be linked into the site.
+- First-time gotcha: `docs/src/Makefile`'s `genmds` target only rebuilds files already listed in
+  `$(wildcard *.md)`. Since `foo.md` doesn't exist yet, a plain `make` within `docs/src` won't
+  generate it. Two ways around this:
+  - Simplest: just use `make docs` from the Miller base directory (or `make -C docs/src forcebuild`)
+    -- these force-run `genmds` on every `*.md.in` unconditionally, new or not.
+  - Or, from `docs/src`, run `./genmds foo.md.in` directly, once, to create the initial `foo.md`.
+    After that it's picked up by ordinary `make` runs like any other page.
+- Don't work around this by pre-creating an empty `foo.md` yourself (e.g. via `touch`): if its mtime
+  ends up newer than `foo.md.in`, make's implicit `%.md: %.md.in` rule will see the target as up to
+  date and skip regenerating it, silently leaving it empty.
+- `git add` both `foo.md.in` and the generated `foo.md`.
+
+## Publishing build
+
+- cd to the `src` subdirectory of `docs` and edit `*.md.in`.
+- `make -C ..`
+- This does write HTML in `docs/site`.
+- In your browser, visit `file:///your/path/to/miller/docs/site/index.html`
+- Link-checking:
+  - `sudo pip3 install git+https://github.com/linkchecker/linkchecker.git`
+  - `cd site` and `linkchecker .`
+
+## Submitting
+
+- Do the publishing-build steps -- in particular, `docs/src/*.md.in` and `docs/src/*.md` are both
+  checked in to source control.
+  - TL;DR: edit `docs/src/foo.md.in` and run `make docs`.
+  - If you don't want to do `pip install mkdocs` then feel free to put up a PR which edits a
+    `foo.md.in` as well as its `foo.md`.
+- `git add` your modified files (`*.md.in` as well as `*.md`), `git commit`, `git push`, and submit
+  a PR at [https://github.com/johnkerl/miller](https://github.com/johnkerl/miller).
+
+## Style notes
+
+- Miller documents use the Oxford comma: not _red, yellow and green_, but rather _red, yellow, and
+  green_.
+- CSS: the Mkdocs "material" theme is used, customized via `docs/src/extra.css` for Miller
+  coloring/branding.
+
+## readthedocs
+
+- Published to [https://miller.readthedocs.io/en/latest](https://miller.readthedocs.io/en/latest) on each commit to `main` in this repo.
+- [https://readthedocs.org/](https://readthedocs.org/)
+- [https://readthedocs.org/projects/miller/](https://readthedocs.org/projects/miller/)
+- [https://readthedocs.org/projects/miller/builds/](https://readthedocs.org/projects/miller/builds/)
+- [https://readthedocs.org/api/v2/webhook/miller/134065/](https://readthedocs.org/api/v2/webhook/miller/134065/)

docs/README.md

@@ -0,0 +1 @@
+Please see [../README-docs.md](../README-docs.md).

docs/mkdocs.yml

@@ -44,6 +44,7 @@ nav:
     - "How to contribute": "contributing.md"
   - 'FAQs and examples':
     - "CSV, with and without headers": "csv-with-and-without-headers.md"
+    - "Troubleshooting CSV and JSON input": "troubleshooting-csv-and-json-input.md"
     - "Shapes of data": "shapes-of-data.md"
     - "Operating on all fields": "operating-on-all-fields.md"
     - "Operating on all records": "operating-on-all-records.md"

docs/src/file-formats.md

@@ -239,20 +239,7 @@ See also the [`--lazy-quotes` flag](reference-main-flag-list.md#csv-only-flags),
 
 ### Troubleshooting CSV and JSON input
 
-When Miller reports a parse error, it is often useful to first identify whether the issue is in the file format, in the chosen input flag, or in the shell quoting around the Miller expression.
-
-* Use the input-format flag which matches the data you are reading. For example, use `--icsv` for CSV input and `--ijson` for JSON input, even if you are writing a different format such as `--ojson` or `--ocsv`.
-* If a CSV file has embedded commas or newlines inside fields, those fields need RFC-4180 double quotes. If the file simply splits on commas with no CSV escaping, try `--csvlite` or `--icsvlite` instead.
-* If a CSV error mentions a header/data length mismatch, inspect the reported input line for an unquoted comma, a missing closing quote, or an unexpected extra delimiter.
-* For CSV generated by spreadsheets or hand-written scripts, `--csv-trim-leading-space` can help when fields look like `"foo", "bar"`, where the second field has a leading space before the quote.
-* For non-standard CSV that contains stray quote characters, `--lazy-quotes` can help Miller accept input that is not fully RFC-4180 compliant.
-* For headerless CSV, add `--implicit-csv-header` (or `-N` when you also want headerless output) so fields are addressed as `$1`, `$2`, and so on.
-* JSON input to Miller should be tabular: a single object, a stream of objects, or an array of objects. A top-level scalar such as `3`, or an array of scalars such as `[1,2,3]`, is valid JSON but not a Miller record stream.
-* When converting nested JSON to CSV or other tabular formats, remember that Miller may flatten nested keys. Use `--flatsep` to choose a separator, or `--no-auto-flatten` when you want to preserve nested values in non-JSON output.
-* Use `--ijsonl` for newline-delimited JSON objects (JSON Lines). Use `--ijson` for ordinary JSON arrays or single objects.
-* In shell commands, wrap Miller DSL expressions in single quotes so the shell does not expand `$field` names before Miller sees them: `mlr --icsv put '$total = $price * $quantity' data.csv`.
-* If the DSL expression itself needs a single quote, put the expression in a script file and use `-f script.mlr`, or use your shell's quoting rules carefully.
-* To isolate format issues from DSL issues, start with a simple command such as `mlr --icsv head -n 3 file.csv` or `mlr --ijson head -n 3 file.json`, then add verbs one at a time.
+Please see [xxx](xxx).
 
 ## JSON
 

docs/src/file-formats.md.in

@@ -81,20 +81,7 @@ See also the [`--lazy-quotes` flag](reference-main-flag-list.md#csv-only-flags),
 
 ### Troubleshooting CSV and JSON input
 
-When Miller reports a parse error, it is often useful to first identify whether the issue is in the file format, in the chosen input flag, or in the shell quoting around the Miller expression.
-
-* Use the input-format flag which matches the data you are reading. For example, use `--icsv` for CSV input and `--ijson` for JSON input, even if you are writing a different format such as `--ojson` or `--ocsv`.
-* If a CSV file has embedded commas or newlines inside fields, those fields need RFC-4180 double quotes. If the file simply splits on commas with no CSV escaping, try `--csvlite` or `--icsvlite` instead.
-* If a CSV error mentions a header/data length mismatch, inspect the reported input line for an unquoted comma, a missing closing quote, or an unexpected extra delimiter.
-* For CSV generated by spreadsheets or hand-written scripts, `--csv-trim-leading-space` can help when fields look like `"foo", "bar"`, where the second field has a leading space before the quote.
-* For non-standard CSV that contains stray quote characters, `--lazy-quotes` can help Miller accept input that is not fully RFC-4180 compliant.
-* For headerless CSV, add `--implicit-csv-header` (or `-N` when you also want headerless output) so fields are addressed as `$1`, `$2`, and so on.
-* JSON input to Miller should be tabular: a single object, a stream of objects, or an array of objects. A top-level scalar such as `3`, or an array of scalars such as `[1,2,3]`, is valid JSON but not a Miller record stream.
-* When converting nested JSON to CSV or other tabular formats, remember that Miller may flatten nested keys. Use `--flatsep` to choose a separator, or `--no-auto-flatten` when you want to preserve nested values in non-JSON output.
-* Use `--ijsonl` for newline-delimited JSON objects (JSON Lines). Use `--ijson` for ordinary JSON arrays or single objects.
-* In shell commands, wrap Miller DSL expressions in single quotes so the shell does not expand `$field` names before Miller sees them: `mlr --icsv put '$total = $price * $quantity' data.csv`.
-* If the DSL expression itself needs a single quote, put the expression in a script file and use `-f script.mlr`, or use your shell's quoting rules carefully.
-* To isolate format issues from DSL issues, start with a simple command such as `mlr --icsv head -n 3 file.csv` or `mlr --ijson head -n 3 file.json`, then add verbs one at a time.
+Please see [xxx](xxx).
 
 ## JSON
 

docs/src/troubleshooting-csv-and-json-input.md

@@ -0,0 +1,53 @@
+<!---  PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
+<div>
+<span class="quicklinks">
+Quick links:
+&nbsp;
+<a class="quicklink" href="../reference-main-flag-list/index.html">Flags</a>
+&nbsp;
+<a class="quicklink" href="../reference-verbs/index.html">Verbs</a>
+&nbsp;
+<a class="quicklink" href="../reference-dsl-builtin-functions/index.html">Functions</a>
+&nbsp;
+<a class="quicklink" href="../glossary/index.html">Glossary</a>
+&nbsp;
+<a class="quicklink" href="../release-docs/index.html">Release docs</a>
+</span>
+</div>
+# Troubleshooting CSV and JSON input
+
+When Miller reports a parse error, it is often useful to first identify whether the issue is in the
+**file format**, in the chosen **input flag**, or in the **shell quoting** around the Miller expression.
+
+- Use the input-format flag which matches the data you are reading. For example, use `--icsv` for
+  CSV input and `--ijson` for JSON input, even if you are writing a different format such as
+  `--ojson` or `--ocsv`.
+- If a CSV file has embedded commas or newlines inside fields, those fields need RFC-4180 double
+  quotes: you need `--icsv`. If the file simply splits on commas with no CSV escaping, try
+  `--csvlite` or `--icsvlite` instead. See also [File Formats](file-formats.md).
+- If a CSV error mentions a "header/data length mismatch", inspect the reported input line for an
+  unquoted comma, a missing closing quote, or an unexpected extra delimiter.
+- For CSV generated by spreadsheets or hand-written scripts, `--csv-trim-leading-space` can help
+  when fields look like `"foo", "bar"`, where the second field has a leading space before the quote.
+- For non-standard CSV that contains stray quote characters, `--lazy-quotes` can help Miller accept
+  input that is not fully RFC-4180 compliant.
+- For headerless CSV, add `--implicit-csv-header` (or `-N` when you also want headerless output) so
+  fields are addressed as `$1`, `$2`, and so on. See also
+  [CSV, with and without headers](csv-with-and-without-headers.md).
+- JSON input to Miller should be tabular: a single object, a stream of objects, or an array of
+  objects. A top-level scalar such as `3`, or an array of scalars such as `[1,2,3]`, is valid JSON
+  but not a Miller record. See also [this page](file-formats.md/#json).
+- When converting nested JSON to CSV or other tabular formats, remember that Miller may flatten
+  nested keys. Use `--flatsep` to choose a separator, or `--no-auto-flatten` when you want to
+  preserve nested values in non-JSON output. See also
+  [Flatten/unflatten: converting between JSON and tabular formats](flatten-unflatten.md).
+- Use `--ijsonl` for newline-delimited JSON objects (JSON Lines). Use `--ijson` for ordinary JSON
+  arrays or single objects.
+- In shell commands, wrap Miller DSL expressions in single quotes so the shell does not expand
+  `$field` names before Miller sees them: `mlr --icsv put '$total = $price * $quantity' data.csv`.
+- If the DSL expression itself needs a single quote, put the expression in a script file and use `-f
+  script.mlr`, or use your shell's quoting rules carefully. See also
+  [Miller on Windows](miller-on-windows.md/#differences) for some complexity issues on Windows.
+- To isolate format issues from DSL issues, start with a simple command such as `mlr --icsv head -n
+  3 file.csv` or `mlr --ijson head -n 3 file.json`, then add verbs one at a time. See also
+  [Questions about then-chaining](questions-about-then-chaining.md).

docs/src/troubleshooting-csv-and-json-input.md.in

@@ -0,0 +1,37 @@
+# Troubleshooting CSV and JSON input
+
+When Miller reports a parse error, it is often useful to first identify whether the issue is in the
+**file format**, in the chosen **input flag**, or in the **shell quoting** around the Miller expression.
+
+- Use the input-format flag which matches the data you are reading. For example, use `--icsv` for
+  CSV input and `--ijson` for JSON input, even if you are writing a different format such as
+  `--ojson` or `--ocsv`.
+- If a CSV file has embedded commas or newlines inside fields, those fields need RFC-4180 double
+  quotes: you need `--icsv`. If the file simply splits on commas with no CSV escaping, try
+  `--csvlite` or `--icsvlite` instead. See also [File Formats](file-formats.md).
+- If a CSV error mentions a "header/data length mismatch", inspect the reported input line for an
+  unquoted comma, a missing closing quote, or an unexpected extra delimiter.
+- For CSV generated by spreadsheets or hand-written scripts, `--csv-trim-leading-space` can help
+  when fields look like `"foo", "bar"`, where the second field has a leading space before the quote.
+- For non-standard CSV that contains stray quote characters, `--lazy-quotes` can help Miller accept
+  input that is not fully RFC-4180 compliant.
+- For headerless CSV, add `--implicit-csv-header` (or `-N` when you also want headerless output) so
+  fields are addressed as `$1`, `$2`, and so on. See also
+  [CSV, with and without headers](csv-with-and-without-headers.md).
+- JSON input to Miller should be tabular: a single object, a stream of objects, or an array of
+  objects. A top-level scalar such as `3`, or an array of scalars such as `[1,2,3]`, is valid JSON
+  but not a Miller record. See also [this page](file-formats.md/#json).
+- When converting nested JSON to CSV or other tabular formats, remember that Miller may flatten
+  nested keys. Use `--flatsep` to choose a separator, or `--no-auto-flatten` when you want to
+  preserve nested values in non-JSON output. See also
+  [Flatten/unflatten: converting between JSON and tabular formats](flatten-unflatten.md).
+- Use `--ijsonl` for newline-delimited JSON objects (JSON Lines). Use `--ijson` for ordinary JSON
+  arrays or single objects.
+- In shell commands, wrap Miller DSL expressions in single quotes so the shell does not expand
+  `$field` names before Miller sees them: `mlr --icsv put '$total = $price * $quantity' data.csv`.
+- If the DSL expression itself needs a single quote, put the expression in a script file and use `-f
+  script.mlr`, or use your shell's quoting rules carefully. See also
+  [Miller on Windows](miller-on-windows.md/#differences) for some complexity issues on Windows.
+- To isolate format issues from DSL issues, start with a simple command such as `mlr --icsv head -n
+  3 file.csv` or `mlr --ijson head -n 3 file.json`, then add verbs one at a time. See also
+  [Questions about then-chaining](questions-about-then-chaining.md).