From ca07014e94ff2c355363fd77c2f1a326f6c42e23 Mon Sep 17 00:00:00 2001 From: John Kerl Date: Fri, 5 Jan 2018 10:11:18 -0500 Subject: [PATCH] comment-handling documentation --- c/draft-release-notes.md | 5 +-- c/todo.txt | 6 +-- doc/content-for-file-formats.html | 19 +++++++- doc/data/budget.csv | 5 +++ doc/file-formats.html | 74 ++++++++++++++++++++++++++++++- 5 files changed, 100 insertions(+), 9 deletions(-) create mode 100644 doc/data/budget.csv diff --git a/c/draft-release-notes.md b/c/draft-release-notes.md index 4d83f965b..047bda161 100644 --- a/c/draft-release-notes.md +++ b/c/draft-release-notes.md @@ -1,9 +1,6 @@ ## Features: -* **Comment strings in data files:** `mlr --skip-comments` allows you to -filter out input lines starting with `#`, for all file formats. Likewise, `mlr ---skip-comments-with X` lets you specify the comment-string `X`. Comments are -only supported at start of data line. +* [**Comment strings in data files:**](http://johnkerl.org/miller-releases/miller-5.3.0/doc/file-formats.html#Comments_in_data) `mlr --skip-comments` allows you to filter out input lines starting with `#`, for all file formats. Likewise, `mlr --skip-comments-with X` lets you specify the comment-string `X`. Comments are only supported at start of data line. `mlr --pass-comments` and `mlr --pass-comments-with X` allow you to forward comments to program output as they are read. * The [**count-similar**](http://johnkerl.org/miller-releases/miller-5.3.0/doc/reference-verbs.html#count-similar) verb lets you compute cluster sizes by cluster labels. diff --git a/c/todo.txt b/c/todo.txt index 234eff55a..2d95f751f 100644 --- a/c/todo.txt +++ b/c/todo.txt @@ -16,10 +16,10 @@ BUGFIXES * comment-handling: o UT for CRLFs - o mlh - o mld + o UT for non-# + +! double-check line-number handling for csv/csvlite/other errors -* xxx note for trailing whitespace in json fixed * sql-setup doc somewhere ....... * bare ./configure ? diff --git a/doc/content-for-file-formats.html b/doc/content-for-file-formats.html index e8f488686..6d94563d5 100644 --- a/doc/content-for-file-formats.html +++ b/doc/content-for-file-formats.html @@ -10,7 +10,8 @@ POKI_PUT_TOC_HERE

Miller handles name-indexed data using several formats: some you probably know by name, such as CSV, TSV, and JSON — and other formats you’re -likely already seeing and using in your structured data. +likely already seeing and using in your structured data. Additionally, Miller gives +you the option of including comments within your data.

Examples

@@ -361,3 +362,19 @@ POKI_PUT_LINK_FOR_PAGE(reference.html#Record/field/pair_separators)HERE for more information about record/field/pair separators. +

Comments in data

+ +
+ +

You can include comments within your data files, and either have them ignored, or passed directly +through to the standard output as soon as they are encountered: + +POKI_RUN_COMMAND{{mlr --usage-comments-in-data}}HERE + +

Examples: + +POKI_RUN_COMMAND{{cat data/budget.csv}}HERE +POKI_RUN_COMMAND{{mlr --skip-comments --icsv --opprint sort -nr quantity data/budget.csv}}HERE +POKI_RUN_COMMAND{{mlr --pass-comments --icsv --opprint sort -nr quantity data/budget.csv}}HERE + +

diff --git a/doc/data/budget.csv b/doc/data/budget.csv new file mode 100644 index 000000000..db8efc622 --- /dev/null +++ b/doc/data/budget.csv @@ -0,0 +1,5 @@ +# Asana -- here are the budget figure you asked for! +type,quantity +purple,456.78 +green,678.12 +orange,123.45 diff --git a/doc/file-formats.html b/doc/file-formats.html index ad5440572..4373f2b41 100644 --- a/doc/file-formats.html +++ b/doc/file-formats.html @@ -204,6 +204,7 @@ Miller commands were run with pretty-print-tabular output format. • Markdown tabular
• Data-conversion keystroke-savers
• Autodetect of line endings
+• Comments in data

@@ -217,7 +218,8 @@ Miller commands were run with pretty-print-tabular output format.

Miller handles name-indexed data using several formats: some you probably know by name, such as CSV, TSV, and JSON — and other formats you’re -likely already seeing and using in your structured data. +likely already seeing and using in your structured data. Additionally, Miller gives +you the option of including comments within your data.

Examples

@@ -1004,6 +1006,76 @@ Unix/Linux/BSD/MacOSX, and CRLF on Windows. Reference for more information about record/field/pair separators. + +

Comments in data

+ +
+ +

You can include comments within your data files, and either have them ignored, or passed directly +through to the standard output as soon as they are encountered: + +

+

+
+$ mlr --usage-comments-in-data
+  --skip-comments                 Ignore commented lines (prefixed by "#")
+                                  within the input.
+  --skip-comments-with {string}   Ignore commented lines within input, with
+                                  specified prefix.
+  --pass-comments                 Immediately print commented lines (prefixed by "#")
+                                  within the input.
+  --pass-comments-with {string}   Immediately print commented lines within input, with
+                                  specified prefix.
+Notes:
+* Comments are only honored at the start of a line.
+* In the absence of any of the above four options, comments are data like
+  any other text.
+* When pass-comments is used, comment lines are written to standard output
+  immediately upon being read; they are not part of the record stream.
+  Results may be counterintuitive. A suggestion is to place comments at the
+  start of data files.
+
+
+

+ +

Examples: + +

+

+
+$ cat data/budget.csv
+# Asana -- here are the budget figure you asked for!
+type,quantity
+purple,456.78
+green,678.12
+orange,123.45
+
+
+

+

+

+
+$ mlr --skip-comments --icsv --opprint sort -nr quantity data/budget.csv
+type   quantity
+green  678.12
+purple 456.78
+orange 123.45
+
+
+

+

+

+
+$ mlr --pass-comments --icsv --opprint sort -nr quantity data/budget.csv
+# Asana -- here are the budget figure you asked for!
+type   quantity
+green  678.12
+purple 456.78
+orange 123.45
+
+
+

+