From 21067acde0759bc36731b9eb0ed079a2c8109a47 Mon Sep 17 00:00:00 2001 From: John Kerl Date: Sun, 2 Feb 2020 09:45:19 -0500 Subject: [PATCH] CSS neatens --- doc/10-min.html | 60 +++++++++++++++---------------- doc/build.html | 2 +- doc/content-for-10-min.html | 60 +++++++++++++++---------------- doc/content-for-build.html | 2 +- doc/content-for-faq.html | 8 ++--- doc/content-for-file-formats.html | 14 ++++---- doc/content-for-originality.html | 22 ++++++------ doc/content-for-why.html | 32 ++++++++--------- doc/css/miller.css | 4 +-- doc/faq.html | 8 ++--- doc/file-formats.html | 14 ++++---- doc/originality.html | 22 ++++++------ doc/why.html | 32 ++++++++--------- 13 files changed, 140 insertions(+), 140 deletions(-) diff --git a/doc/10-min.html b/doc/10-min.html index 4d2a9dff5..f872126a1 100644 --- a/doc/10-min.html +++ b/doc/10-min.html @@ -81,7 +81,7 @@ pageTracker._trackPageview();
-

Sample CSV data file: +

Sample CSV data file:

@@ -102,7 +102,7 @@ purple,square,0,91,72.3735,8.2430

-

mlr cat is like cat ... +

mlr cat is like cat ...

@@ -123,7 +123,7 @@ purple,square,0,91,72.3735,8.2430

-

... but it can also do format conversion (here, to pretty-printed tabular format): +

... but it can also do format conversion (here, to pretty-printed tabular format):

@@ -144,8 +144,8 @@ purple square 0 91 72.3735 8.2430

-

mlr head and mlr tail count records rather than lines. The CSV -header is included either way: +

mlr head and mlr tail count records rather than lines. The CSV +header is included either way:

@@ -172,8 +172,8 @@ purple,square,0,91,72.3735,8.2430

-

Sort primarily alphabetically on one field, then secondarily -numerically descending on another field: +

Sort primarily alphabetically on one field, then secondarily +numerically descending on another field:

@@ -194,7 +194,7 @@ yellow triangle 1 11 43.6498 9.8870

-

Use cut to retain only specified fields, in input-data order: +

Use cut to retain only specified fields, in input-data order:

@@ -215,7 +215,7 @@ square 0

-

Use cut -o to retain only specified fields, in your specified order: +

Use cut -o to retain only specified fields, in your specified order:

@@ -236,7 +236,7 @@ flag shape

-

Use cut -x to omit specified fields: +

Use cut -x to omit specified fields:

@@ -257,7 +257,7 @@ purple 91 72.3735 8.2430

-

Use filter to retain specified records: +

Use filter to retain specified records:

@@ -282,7 +282,7 @@ red circle 1 16 13.8103 2.9010

-

Use put to add/replace fields which are computed from other fields: +

Use put to add/replace fields which are computed from other fields:

@@ -303,10 +303,10 @@ purple square 0 91 72.3735 8.2430 8.779995 purple_square

-

Even though Miller’s main selling point is +

Even though Miller’s main selling point is name-indexing, sometimes you really want to refer to a field name by its positional index. Use $[[3]] to access the name of field 3 or -$[[[3]]] to access the value of field 3: +$[[[3]]] to access the value of field 3:

@@ -345,7 +345,7 @@ purple square NEW 91 72.3735 8.2430

-

JSON output: +

JSON output:

@@ -365,7 +365,7 @@ $ mlr --icsv --ojson put '$ratio = $quantity/$rate; $shape = toupper($shape)' ex

-

JSON output with vertical-formatting flags: +

JSON output with vertical-formatting flags:

@@ -393,11 +393,11 @@ $ mlr --icsv --ojson --jvstack --jlistwrap tail -n 2 example.csv

-

Use then to pipe commands together. Also, the +

Use then to pipe commands together. Also, the -g option for many Miller commands is for group-by: here, head -n 1 -g shape outputs the first record for each distinct value of the shape field. This means we’re finding the record with highest -index field for each distinct shape field: +index field for each distinct shape field:

@@ -411,9 +411,9 @@ purple triangle 0 65 80.1405 5.8240

-

Statistics can be computed with or without group-by field(s). Also, the first of these two +

Statistics can be computed with or without group-by field(s). Also, the first of these two examples uses --oxtab output format which is a nice alternative to --opprint when you -have lots of columns: +have lots of columns:

@@ -461,11 +461,11 @@ square purple 1 72.373500 72.373500 72.373500
-

Often we want to print output to the screen. Miller does this by default, as we’ve -seen in the previous examples. +

Often we want to print output to the screen. Miller does this by default, as we’ve +seen in the previous examples.

-

Sometimes we want to print output to another file: just use '> -outputfilenamegoeshere' at the end of your command: +

Sometimes we want to print output to another file: just use '> +outputfilenamegoeshere' at the end of your command:

@@ -490,7 +490,7 @@ purple square   0    91    72.3735  8.2430
 
-

Other times we just want our files to be changed in-place: just use 'mlr -I'. +

Other times we just want our files to be changed in-place: just use 'mlr -I'.

@@ -538,8 +538,8 @@ mlr -I --csv cut -x -f unwanted_column_name *.csv
 
 

If you like, you can first copy off your original data somewhere else, before doing in-place operations. -

Lastly, using tee within put, you can split your input data into separate files -per one or more field names: +

Lastly, using tee within put, you can split your input data into separate files +per one or more field names:

@@ -616,7 +616,7 @@ shape=square,flag=0,index=36
-

Data written this way are called DKVP, for delimited key-value pairs. +

Data written this way are called DKVP, for delimited key-value pairs.

We’ve also already seen other ways to write the same data: @@ -640,8 +640,8 @@ shape=square,flag=0,index=36 flag 1 "flag": 0, -

Anything we can do with CSV input data, we can do with any -other format input data. And you can read from one format, do any +

Anything we can do with CSV input data, we can do with any +other format input data.

And you can read from one format, do any record-processing, and output to the same format as the input, or to a different output format. diff --git a/doc/build.html b/doc/build.html index 94e4e25d2..bd692a514 100644 --- a/doc/build.html +++ b/doc/build.html @@ -353,7 +353,7 @@ other 21st-century compilers)

Optional external dependencies

-This documentation pageset is built using Poki: +This documentation pageset is built using Poki:
docs here, source code here. Note that http://johnkerl.org/miller/doc/index.html diff --git a/doc/content-for-10-min.html b/doc/content-for-10-min.html index 2e9ba2ad5..60138fc04 100644 --- a/doc/content-for-10-min.html +++ b/doc/content-for-10-min.html @@ -8,77 +8,77 @@ POKI_PUT_TOC_HERE
-

Sample CSV data file: +

Sample CSV data file:

POKI_RUN_COMMAND{{cat example.csv}}HERE -

mlr cat is like cat ... +

mlr cat is like cat ...

POKI_RUN_COMMAND{{mlr --csv cat example.csv}}HERE -

... but it can also do format conversion (here, to pretty-printed tabular format): +

... but it can also do format conversion (here, to pretty-printed tabular format):

POKI_RUN_COMMAND{{mlr --icsv --opprint cat example.csv}}HERE -

mlr head and mlr tail count records rather than lines. The CSV -header is included either way: +

mlr head and mlr tail count records rather than lines. The CSV +header is included either way:

POKI_RUN_COMMAND{{mlr --csv head -n 4 example.csv}}HERE POKI_RUN_COMMAND{{mlr --csv tail -n 4 example.csv}}HERE -

Sort primarily alphabetically on one field, then secondarily -numerically descending on another field: +

Sort primarily alphabetically on one field, then secondarily +numerically descending on another field:

POKI_RUN_COMMAND{{mlr --icsv --opprint sort -f shape -nr index example.csv}}HERE -

Use cut to retain only specified fields, in input-data order: +

Use cut to retain only specified fields, in input-data order:

POKI_RUN_COMMAND{{mlr --icsv --opprint cut -f flag,shape example.csv}}HERE -

Use cut -o to retain only specified fields, in your specified order: +

Use cut -o to retain only specified fields, in your specified order:

POKI_RUN_COMMAND{{mlr --icsv --opprint cut -o -f flag,shape example.csv}}HERE -

Use cut -x to omit specified fields: +

Use cut -x to omit specified fields:

POKI_RUN_COMMAND{{mlr --icsv --opprint cut -x -f flag,shape example.csv}}HERE -

Use filter to retain specified records: +

Use filter to retain specified records:

POKI_RUN_COMMAND{{mlr --icsv --opprint filter '$color == "red"' example.csv}}HERE POKI_RUN_COMMAND{{mlr --icsv --opprint filter '$color == "red" && $flag == 1' example.csv}}HERE -

Use put to add/replace fields which are computed from other fields: +

Use put to add/replace fields which are computed from other fields:

POKI_RUN_COMMAND{{mlr --icsv --opprint put '$ratio = $quantity / $rate; $color_shape = $color . "_" . $shape' example.csv}}HERE -

Even though Miller’s main selling point is +

Even though Miller’s main selling point is name-indexing, sometimes you really want to refer to a field name by its positional index. Use $[[3]] to access the name of field 3 or -$[[[3]]] to access the value of field 3: +$[[[3]]] to access the value of field 3:

POKI_RUN_COMMAND{{mlr --icsv --opprint put '$[[3]] = "NEW"' example.csv}}HERE POKI_RUN_COMMAND{{mlr --icsv --opprint put '$[[[3]]] = "NEW"' example.csv}}HERE -

JSON output: +

JSON output:

POKI_RUN_COMMAND{{mlr --icsv --ojson put '$ratio = $quantity/$rate; $shape = toupper($shape)' example.csv}}HERE -

JSON output with vertical-formatting flags: +

JSON output with vertical-formatting flags:

POKI_RUN_COMMAND{{mlr --icsv --ojson --jvstack --jlistwrap tail -n 2 example.csv}}HERE -

Use then to pipe commands together. Also, the +

Use then to pipe commands together. Also, the -g option for many Miller commands is for group-by: here, head -n 1 -g shape outputs the first record for each distinct value of the shape field. This means we’re finding the record with highest -index field for each distinct shape field: +index field for each distinct shape field:

POKI_RUN_COMMAND{{mlr --icsv --opprint sort -f shape -nr index then head -n 1 -g shape example.csv}}HERE -

Statistics can be computed with or without group-by field(s). Also, the first of these two +

Statistics can be computed with or without group-by field(s). Also, the first of these two examples uses --oxtab output format which is a nice alternative to --opprint when you -have lots of columns: +have lots of columns:

POKI_RUN_COMMAND{{mlr --icsv --oxtab --from example.csv stats1 -a p0,p10,p25,p50,p75,p90,p99,p100 -f rate}}HERE POKI_RUN_COMMAND{{mlr --icsv --opprint --from example.csv stats1 -a count,min,mean,max -f quantity -g shape}}HERE @@ -89,11 +89,11 @@ POKI_RUN_COMMAND{{mlr --icsv --opprint --from example.csv stats1 -a count,min,me
-

Often we want to print output to the screen. Miller does this by default, as we’ve -seen in the previous examples. +

Often we want to print output to the screen. Miller does this by default, as we’ve +seen in the previous examples.

-

Sometimes we want to print output to another file: just use '> -outputfilenamegoeshere' at the end of your command: +

Sometimes we want to print output to another file: just use '> +outputfilenamegoeshere' at the end of your command:

@@ -118,7 +118,7 @@ purple square   0    91    72.3735  8.2430
 
-

Other times we just want our files to be changed in-place: just use 'mlr -I'. +

Other times we just want our files to be changed in-place: just use 'mlr -I'.

@@ -166,8 +166,8 @@ mlr -I --csv cut -x -f unwanted_column_name *.csv
 
 

If you like, you can first copy off your original data somewhere else, before doing in-place operations. -

Lastly, using tee within put, you can split your input data into separate files -per one or more field names: +

Lastly, using tee within put, you can split your input data into separate files +per one or more field names:

POKI_RUN_COMMAND{{mlr --csv --from example.csv put -q 'tee > $shape.".csv", $*'}}HERE
@@ -207,7 +207,7 @@ shape=square,flag=0,index=36 -

Data written this way are called DKVP, for delimited key-value pairs. +

Data written this way are called DKVP, for delimited key-value pairs.

We’ve also already seen other ways to write the same data: @@ -231,8 +231,8 @@ shape=square,flag=0,index=36 flag 1 "flag": 0, -

Anything we can do with CSV input data, we can do with any -other format input data. And you can read from one format, do any +

Anything we can do with CSV input data, we can do with any +other format input data.

And you can read from one format, do any record-processing, and output to the same format as the input, or to a different output format. diff --git a/doc/content-for-build.html b/doc/content-for-build.html index 109148d97..04795755d 100644 --- a/doc/content-for-build.html +++ b/doc/content-for-build.html @@ -241,7 +241,7 @@ other 21st-century compilers)

Optional external dependencies

-This documentation pageset is built using Poki: +This documentation pageset is built using Poki: docs here, source code here. Note that http://johnkerl.org/miller/doc/index.html diff --git a/doc/content-for-faq.html b/doc/content-for-faq.html index 9e49f4cf3..3bce6a09b 100644 --- a/doc/content-for-faq.html +++ b/doc/content-for-faq.html @@ -373,14 +373,14 @@ POKI_RUN_COMMAND{{mlr --csv join --ul -j id -f data/color-codes.csv then unspars
-

Miller handles tabular data, which is a list of +

Miller handles tabular data, which is a list of records each having fields which are key-value pairs. Miller also doesn’t require that each record have the same field names (see also here). Regardless, tabular data is a -non-recursive data structure. +non-recursive data structure. -

XML, JSON, etc. are, by contrast, all recursive -or nested data structures. For example, in JSON +

XML, JSON, etc. are, by contrast, all recursive +or nested data structures. For example, in JSON you can represent a hash map whose values are lists of lists.

Now, you can put tabular data into these formats — since list-of-key-value-pairs diff --git a/doc/content-for-file-formats.html b/doc/content-for-file-formats.html index 7f01cc1cd..8024b1d17 100644 --- a/doc/content-for-file-formats.html +++ b/doc/content-for-file-formats.html @@ -179,7 +179,7 @@ POKI_RUN_COMMAND{{mlr --nidx --fs ' ' --repifs cut -f 2,3 data/mydata.txt}}HERE

JSON is a format which supports arbitrarily deep nesting of “objects” (hashmaps) and “arrays” (lists), while Miller -is a tool for handling tabular data only. This means +is a tool for handling tabular data only. This means Miller cannot (and should not) handle arbitrary JSON. (Check out jq.) @@ -187,8 +187,8 @@ href="http://stedolan.github.io/jq/">jq.)

Single-level JSON objects

-An array of single-level objects is, quite simply, -a table: +An array of single-level objects is, quite simply, +a table: POKI_RUN_COMMAND{{mlr --json head -n 2 then cut -f color,shape data/json-example-1.json}}HERE POKI_RUN_COMMAND{{mlr --json --jvstack head -n 2 then cut -f color,u,v data/json-example-1.json}}HERE @@ -196,7 +196,7 @@ POKI_RUN_COMMAND{{mlr --ijson --opprint stats1 -a mean,stddev,count -f u -g shap

Nested JSON objects

-

Additionally, Miller can tabularize nested objects by concatentating keys: +

Additionally, Miller can tabularize nested objects by concatentating keys: POKI_RUN_COMMAND{{mlr --json --jvstack head -n 2 data/json-example-2.json}}HERE POKI_RUN_COMMAND{{mlr --ijson --opprint head -n 4 data/json-example-2.json}}HERE @@ -358,15 +358,15 @@ POKI_RUN_COMMAND{{mlr --usage-format-conversion-keystroke-saver-options}}HERE

Default line endings (--irs and --ors) are 'auto' -which means autodetect from the input file format, as +which means autodetect from the input file format, as long as the input file(s) have lines ending in either LF (also known as linefeed, '\n', 0x0a, Unix-style) or CRLF (also known as carriage-return/linefeed pairs, '\r\n', 0x0d 0x0a, Windows style). -

If both IRS and ORS are auto (which is the default) then LF input will +

If both IRS and ORS are auto (which is the default) then LF input will lead to LF output and CRLF input will lead to CRLF output, regardless of the -platform you’re running on. +platform you’re running on.

The line-ending autodetector triggers on the first line ending detected in the input stream. E.g. if you specify a CRLF-terminated file on the command diff --git a/doc/content-for-originality.html b/doc/content-for-originality.html index 9aa33cd48..40a031eb1 100644 --- a/doc/content-for-originality.html +++ b/doc/content-for-originality.html @@ -6,7 +6,7 @@ online-analytical-processing culture. Other key participants include concept, Miller explicitly strives to imitate several existing tools:

-Unix toolkit: Intentional similarities as described in +Unix toolkit: Intentional similarities as described in POKI_PUT_LINK_FOR_PAGE(feature-comparison.html)HERE.

Recipes abound for command-line data analysis using the Unix toolkit. Here are just a couple of my favorites: @@ -16,7 +16,7 @@ POKI_PUT_LINK_FOR_PAGE(feature-comparison.html)HERE.

  • https://github.com/dbohdan/structured-text-tools -

    RecordStream: Miller owes particular inspiration +

    RecordStream: Miller owes particular inspiration to RecordStream. The key difference is that RecordStream is a Perl-based tool for manipulating JSON (including requiring it to separately manipulate other formats such as CSV into @@ -25,15 +25,15 @@ The similarities include the sort, stats1 (analog of RecordStream’s collate), and delta operations, as well as filter and put, and pretty-print formatting. -

    stats_m: A third source of lineage is my Python +

    stats_m: A third source of lineage is my Python stats_m module. This includes simple single-pass algorithms which form Miller’s stats1 and stats2 subcommands. -

    SQL: Fourthly, Miller’s group-by command +

    SQL: Fourthly, Miller’s group-by command name is from SQL, as is the term aggregate. -

    Added value: +

    Added value: Miller’s added values include:

    • Name-indexing, compared to the Unix toolkit’s positional indexing. @@ -43,17 +43,17 @@ Miller’s added values include:
    • Various file formats, and on-the-fly format conversion.
    -

    jq: Miller does for name-indexed text what +

    jq: Miller does for name-indexed text what jq does for JSON. If you’re not already familiar with jq, please check it out!. -

    What about similar tools? +

    What about similar tools? Here’s a comprehensive list: https://github.com/dbohdan/structured-text-tools. It doesn’t mention rows so here’s a plug for that as well. As it turns out, I learned about most of these after writing Miller. -

    What about DOTADIW? One of the key points of the +

    What about DOTADIW? One of the key points of the Unix philosophy is that a tool should do one thing and do it well. Hence sort and cut do just one thing. Why does Miller put awk-like @@ -70,7 +70,7 @@ be a tool which collects together format-aware record-stream processing into one place, with good reuse of Miller-internal library code for its various features. -

    Why not use Perl/Python/Ruby etc.? Maybe you +

    Why not use Perl/Python/Ruby etc.? Maybe you should. With those tools you’ll get far more expressive power, and sufficiently quick turnaround time for small-to-medium-sized data. Using Miller you’ll get something less than a complete programming language, @@ -88,8 +88,8 @@ If I’d gotten good enough performance from the latter I’d have done it without question and Miller would be far more flexible. But C won the performance criteria by a landslide so we have Miller in C with a custom DSL. -

    No, really, why one more command-line data-manipulation -tool? I wrote Miller because I was frustrated with tools like +

    No, really, why one more command-line data-manipulation +tool? I wrote Miller because I was frustrated with tools like grep, sed, and so on being line-aware without being format-aware. The single most poignant example I can think of is seeing people grep data lines out of their CSV files and sadly losing their header diff --git a/doc/content-for-why.html b/doc/content-for-why.html index f7fb3eb1e..5c342a34b 100644 --- a/doc/content-for-why.html +++ b/doc/content-for-why.html @@ -13,15 +13,15 @@ why I felt it necessary to build Miller, etc. Here are some answers.

    For background, I’m a software engineer, with a heavy devops bent and a non-trivial amount of data-engineering in my career. -Initially I wrote Miller mainly for myself: I’m +Initially I wrote Miller mainly for myself: I’m coder-friendly (being a coder); I’m Github-friendly; most of my data are well-structured or easily structurable (TSV-formatted SQL-query output, CSV files, log files, JSON data structures); I care about interoperability between all the various formats Miller supports (I’ve encountered them all); I do all my work on Linux or OSX. -

    But now there’s this neat little tool which seems to be -useful for people in various disciplines. I don’t even know +

    But now there’s this neat little tool which seems to be +useful for people in various disciplines. I don’t even know entirely who. I can click through Github starrers and read a bit about what they seem to do, but not everyone’s on Github (or stars things). I’ve gotten a lot of feature requests through Github — but @@ -50,7 +50,7 @@ it’s for? years of my career in the software industry I’ve found myself, and others, doing a lot of ad-hoc things which really were fundamentally the same except for format. So the number one thing about Miller is doing common -things while supporting multiple formats: (a) ingest a +things while supporting multiple formats: (a) ingest a list of records where a record is a list of key-value pairs (however represented in the input files); (b) transform that stream of records; (c) emit the transformed stream — either in the same format as input, or in a @@ -60,9 +60,9 @@ different format. something only for a single file format, I didn’t want to build something only for one problem domain. In my work doing software engineering, devops, data engineering, etc. I saw a lot of commonalities and I wanted to -solve as many problems simultaneously as possible. +solve as many problems simultaneously as possible. -

    Third: it had to be streaming. As time goes by +

    Third: it had to be streaming. As time goes by and we (some of us, sometimes) have machines with tens or hundreds of GB of RAM, it’s maybe less important, but I’m unhappy with tools which ingest all data, then do stuff, then emit all data. One reason is to be able to @@ -71,7 +71,7 @@ input which trickles in, e.g. you have some process emitting data now and then and you can pipe it to Miller and it will emit transformed records one at a time. -

    Fourth: it had to be fast. This precludes all +

    Fourth: it had to be fast. This precludes all sorts of very nice things written in Ruby, for example. I love Ruby as a very expressive language, and I have several very useful little utility scripts written in Ruby. But a few years ago I ported over some of my old @@ -83,8 +83,8 @@ in order to make it performant. I did simple experiments in several languages, and nothing was as fast as C, so I used C: see also here. -

    Fifth thing: I wanted Miller to be pipe-friendly and -interoperate with other command-line tools. Since the basic +

    Fifth thing: I wanted Miller to be pipe-friendly and +interoperate with other command-line tools. Since the basic paradigm is ingest records, transform records, emit records — where the input and output formats can be the same or different, and the transform can be complex, or just pass-through — this means you can use it to transform @@ -93,17 +93,17 @@ data-cleaning/prep/formatting and do all the "real" work in R, you can. If you just want a little glue script between other tools you can get that. And if you want to do non-trivial data-reduction in Miller you can. -

    Sixth thing: Must have comprehensive documentation and -unit-test. Since Miller handles a lot of formats and solves a lot +

    Sixth thing: Must have comprehensive documentation and +unit-test. Since Miller handles a lot of formats and solves a lot of problems, there’s a lot to test and a lot to keep working correctly as I add features or optimize. And I wanted it to be able to explain itself — not only through web docs like the one you’re reading but also through man mlr and mlr --help, mlr sort --help, etc. -

    Seventh thing: Must have a domain-specific -language (DSL) but also must let you do common things -without it. All those little verbs Miller has to help you +

    Seventh thing: Must have a domain-specific +language (DSL) but also must let you do common things +without it. All those little verbs Miller has to help you avoid having to write for-loops are great. I use them for keystroke-saving: mlr stats1 -a mean,stddev,min,max -f quantity, for example, without you having to write for-loops or define accumulator variables. @@ -112,8 +112,8 @@ you want to: mlr put '$distance = $rate * $time' or anything else y can think up. In Perl/AWK/etc. it’s all DSL. In xsv et al. it’s all verbs. In Miller I like having the combination. -

    Eighth thing: It’s an awful lot of fun to -write. In my experience I didn’t find any tools which do +

    Eighth thing: It’s an awful lot of fun to +write. In my experience I didn’t find any tools which do multi-format, streaming, efficient, multi-purpose, with DSL and non-DSL, so I wrote one. But I don’t guarantee it’s unique in the world. It fills a niche in the world (people use it) but it also fills a niche in my life. diff --git a/doc/css/miller.css b/doc/css/miller.css index 692e07743..12dc0abeb 100644 --- a/doc/css/miller.css +++ b/doc/css/miller.css @@ -14,10 +14,10 @@ body { background-color: white; } -maroon { +.maroon { color: maroon; } -boldmaroon { +.boldmaroon { font-weight: bold; color: maroon; } diff --git a/doc/faq.html b/doc/faq.html index f3479b577..02fae1956 100644 --- a/doc/faq.html +++ b/doc/faq.html @@ -995,14 +995,14 @@ id,code,color

    -

    Miller handles tabular data, which is a list of +

    Miller handles tabular data, which is a list of records each having fields which are key-value pairs. Miller also doesn’t require that each record have the same field names (see also here). Regardless, tabular data is a -non-recursive data structure. +non-recursive data structure. -

    XML, JSON, etc. are, by contrast, all recursive -or nested data structures. For example, in JSON +

    XML, JSON, etc. are, by contrast, all recursive +or nested data structures. For example, in JSON you can represent a hash map whose values are lists of lists.

    Now, you can put tabular data into these formats — since list-of-key-value-pairs diff --git a/doc/file-formats.html b/doc/file-formats.html index 939f1870b..0669a9c39 100644 --- a/doc/file-formats.html +++ b/doc/file-formats.html @@ -420,7 +420,7 @@ light

    JSON is a format which supports arbitrarily deep nesting of “objects” (hashmaps) and “arrays” (lists), while Miller -is a tool for handling tabular data only. This means +is a tool for handling tabular data only. This means Miller cannot (and should not) handle arbitrary JSON. (Check out jq.) @@ -428,8 +428,8 @@ href="http://stedolan.github.io/jq/">jq.)

    Single-level JSON objects

    -An array of single-level objects is, quite simply, -a table: +An array of single-level objects is, quite simply, +a table:

    @@ -471,7 +471,7 @@ circle 0.366013 0.209094 3

    Nested JSON objects

    -

    Additionally, Miller can tabularize nested objects by concatentating keys: +

    Additionally, Miller can tabularize nested objects by concatentating keys:

  • https://github.com/dbohdan/structured-text-tools -

    RecordStream: Miller owes particular inspiration +

    RecordStream: Miller owes particular inspiration to RecordStream. The key difference is that RecordStream is a Perl-based tool for manipulating JSON (including requiring it to separately manipulate other formats such as CSV into @@ -91,15 +91,15 @@ The similarities include the sort, stats1 (analog of RecordStream’s collate), and delta operations, as well as filter and put, and pretty-print formatting. -

    stats_m: A third source of lineage is my Python +

    stats_m: A third source of lineage is my Python stats_m module. This includes simple single-pass algorithms which form Miller’s stats1 and stats2 subcommands. -

    SQL: Fourthly, Miller’s group-by command +

    SQL: Fourthly, Miller’s group-by command name is from SQL, as is the term aggregate. -

    Added value: +

    Added value: Miller’s added values include:

    • Name-indexing, compared to the Unix toolkit’s positional indexing. @@ -109,17 +109,17 @@ Miller’s added values include:
    • Various file formats, and on-the-fly format conversion.
    -

    jq: Miller does for name-indexed text what +

    jq: Miller does for name-indexed text what jq does for JSON. If you’re not already familiar with jq, please check it out!. -

    What about similar tools? +

    What about similar tools? Here’s a comprehensive list: https://github.com/dbohdan/structured-text-tools. It doesn’t mention rows so here’s a plug for that as well. As it turns out, I learned about most of these after writing Miller. -

    What about DOTADIW? One of the key points of the +

    What about DOTADIW? One of the key points of the Unix philosophy is that a tool should do one thing and do it well. Hence sort and cut do just one thing. Why does Miller put awk-like @@ -136,7 +136,7 @@ be a tool which collects together format-aware record-stream processing into one place, with good reuse of Miller-internal library code for its various features. -

    Why not use Perl/Python/Ruby etc.? Maybe you +

    Why not use Perl/Python/Ruby etc.? Maybe you should. With those tools you’ll get far more expressive power, and sufficiently quick turnaround time for small-to-medium-sized data. Using Miller you’ll get something less than a complete programming language, @@ -154,8 +154,8 @@ If I’d gotten good enough performance from the latter I’d have done it without question and Miller would be far more flexible. But C won the performance criteria by a landslide so we have Miller in C with a custom DSL. -

    No, really, why one more command-line data-manipulation -tool? I wrote Miller because I was frustrated with tools like +

    No, really, why one more command-line data-manipulation +tool? I wrote Miller because I was frustrated with tools like grep, sed, and so on being line-aware without being format-aware. The single most poignant example I can think of is seeing people grep data lines out of their CSV files and sadly losing their header diff --git a/doc/why.html b/doc/why.html index 2bab78398..f29641141 100644 --- a/doc/why.html +++ b/doc/why.html @@ -84,15 +84,15 @@ why I felt it necessary to build Miller, etc. Here are some answers.

    For background, I’m a software engineer, with a heavy devops bent and a non-trivial amount of data-engineering in my career. -Initially I wrote Miller mainly for myself: I’m +Initially I wrote Miller mainly for myself: I’m coder-friendly (being a coder); I’m Github-friendly; most of my data are well-structured or easily structurable (TSV-formatted SQL-query output, CSV files, log files, JSON data structures); I care about interoperability between all the various formats Miller supports (I’ve encountered them all); I do all my work on Linux or OSX. -

    But now there’s this neat little tool which seems to be -useful for people in various disciplines. I don’t even know +

    But now there’s this neat little tool which seems to be +useful for people in various disciplines. I don’t even know entirely who. I can click through Github starrers and read a bit about what they seem to do, but not everyone’s on Github (or stars things). I’ve gotten a lot of feature requests through Github — but @@ -121,7 +121,7 @@ it’s for? years of my career in the software industry I’ve found myself, and others, doing a lot of ad-hoc things which really were fundamentally the same except for format. So the number one thing about Miller is doing common -things while supporting multiple formats: (a) ingest a +things while supporting multiple formats: (a) ingest a list of records where a record is a list of key-value pairs (however represented in the input files); (b) transform that stream of records; (c) emit the transformed stream — either in the same format as input, or in a @@ -131,9 +131,9 @@ different format. something only for a single file format, I didn’t want to build something only for one problem domain. In my work doing software engineering, devops, data engineering, etc. I saw a lot of commonalities and I wanted to -solve as many problems simultaneously as possible. +solve as many problems simultaneously as possible. -

    Third: it had to be streaming. As time goes by +

    Third: it had to be streaming. As time goes by and we (some of us, sometimes) have machines with tens or hundreds of GB of RAM, it’s maybe less important, but I’m unhappy with tools which ingest all data, then do stuff, then emit all data. One reason is to be able to @@ -142,7 +142,7 @@ input which trickles in, e.g. you have some process emitting data now and then and you can pipe it to Miller and it will emit transformed records one at a time. -

    Fourth: it had to be fast. This precludes all +

    Fourth: it had to be fast. This precludes all sorts of very nice things written in Ruby, for example. I love Ruby as a very expressive language, and I have several very useful little utility scripts written in Ruby. But a few years ago I ported over some of my old @@ -154,8 +154,8 @@ in order to make it performant. I did simple experiments in several languages, and nothing was as fast as C, so I used C: see also here. -

    Fifth thing: I wanted Miller to be pipe-friendly and -interoperate with other command-line tools. Since the basic +

    Fifth thing: I wanted Miller to be pipe-friendly and +interoperate with other command-line tools. Since the basic paradigm is ingest records, transform records, emit records — where the input and output formats can be the same or different, and the transform can be complex, or just pass-through — this means you can use it to transform @@ -164,17 +164,17 @@ data-cleaning/prep/formatting and do all the "real" work in R, you can. If you just want a little glue script between other tools you can get that. And if you want to do non-trivial data-reduction in Miller you can. -

    Sixth thing: Must have comprehensive documentation and -unit-test. Since Miller handles a lot of formats and solves a lot +

    Sixth thing: Must have comprehensive documentation and +unit-test. Since Miller handles a lot of formats and solves a lot of problems, there’s a lot to test and a lot to keep working correctly as I add features or optimize. And I wanted it to be able to explain itself — not only through web docs like the one you’re reading but also through man mlr and mlr --help, mlr sort --help, etc. -

    Seventh thing: Must have a domain-specific -language (DSL) but also must let you do common things -without it. All those little verbs Miller has to help you +

    Seventh thing: Must have a domain-specific +language (DSL) but also must let you do common things +without it. All those little verbs Miller has to help you avoid having to write for-loops are great. I use them for keystroke-saving: mlr stats1 -a mean,stddev,min,max -f quantity, for example, without you having to write for-loops or define accumulator variables. @@ -183,8 +183,8 @@ you want to: mlr put '$distance = $rate * $time' or anything else y can think up. In Perl/AWK/etc. it’s all DSL. In xsv et al. it’s all verbs. In Miller I like having the combination. -

    Eighth thing: It’s an awful lot of fun to -write. In my experience I didn’t find any tools which do +

    Eighth thing: It’s an awful lot of fun to +write. In my experience I didn’t find any tools which do multi-format, streaming, efficient, multi-purpose, with DSL and non-DSL, so I wrote one. But I don’t guarantee it’s unique in the world. It fills a niche in the world (people use it) but it also fills a niche in my life.