Even though Miller’s main selling point is
name-indexing, sometimes you really want to refer to a field name by its
positional index. Use $[[3]] to access the name of field 3 or
-$[[[3]]] to access the value of field 3:
+$[[[3]]] to access the value of field 3:
@@ -345,7 +345,7 @@ purple square NEW 91 72.3735 8.2430
- JSON output:
+
JSON output:
@@ -365,7 +365,7 @@ $ mlr --icsv --ojson put '$ratio = $quantity/$rate; $shape = toupper($shape)' ex
Use then to pipe commands together. Also, the
-g option for many Miller commands is for group-by: here, head -n
1 -g shape outputs the first record for each distinct value of the
shape field. This means we’re finding the record with highest
-index field for each distinct shape field:
- Statistics can be computed with or without group-by field(s). Also, the first of these two
+
Statistics can be computed with or without group-by field(s). Also, the first of these two
examples uses --oxtab output format which is a nice alternative to --opprint when you
-have lots of columns:
-Often we want to print output to the screen. Miller does this by default, as we’ve
-seen in the previous examples.
+
Often we want to print output to the screen. Miller does this by default, as we’ve
+seen in the previous examples.
- Sometimes we want to print output to another file: just use '>
-outputfilenamegoeshere' at the end of your command:
+ Sometimes we want to print output to another file: just use '>
+outputfilenamegoeshere' at the end of your command:
- Other times we just want our files to be changed in-place: just use 'mlr -I'.
+ Other times we just want our files to be changed in-place: just use 'mlr -I'.
@@ -538,8 +538,8 @@ mlr -I --csv cut -x -f unwanted_column_name *.csv
If you like, you can first copy off your original data somewhere else, before doing in-place operations.
- Lastly, using tee within put, you can split your input data into separate files
-per one or more field names:
+
Lastly, using tee within put, you can split your input data into separate files
+per one or more field names:
@@ -616,7 +616,7 @@ shape=square,flag=0,index=36
-Data written this way are called DKVP, for delimited key-value pairs.
+
Data written this way are called DKVP, for delimited key-value pairs.
We’ve also already seen other ways to write the same data:
@@ -640,8 +640,8 @@ shape=square,flag=0,index=36 flag 1 "flag": 0,
-Anything we can do with CSV input data, we can do with any
-other format input data. And you can read from one format, do any
+
Anything we can do with CSV input data, we can do with any
+other format input data.
And you can read from one format, do any
record-processing, and output to the same format as the input, or to a
different output format.
diff --git a/doc/build.html b/doc/build.html
index 94e4e25d2..bd692a514 100644
--- a/doc/build.html
+++ b/doc/build.html
@@ -353,7 +353,7 @@ other 21st-century compilers)
Optional external dependencies
-This documentation pageset is built using Poki:
+This documentation pageset is built using Poki:
docs here,
source code here.
Note that http://johnkerl.org/miller/doc/index.html
diff --git a/doc/content-for-10-min.html b/doc/content-for-10-min.html
index 2e9ba2ad5..60138fc04 100644
--- a/doc/content-for-10-min.html
+++ b/doc/content-for-10-min.html
@@ -8,77 +8,77 @@ POKI_PUT_TOC_HERE
- Sample CSV data file:
+
Sample CSV data file:
POKI_RUN_COMMAND{{cat example.csv}}HERE
-mlr cat is like cat ...
+
mlr cat is like cat ...
POKI_RUN_COMMAND{{mlr --csv cat example.csv}}HERE
-... but it can also do format conversion (here, to pretty-printed tabular format):
+
... but it can also do format conversion (here, to pretty-printed tabular format):
POKI_RUN_COMMAND{{mlr --icsv --opprint cat example.csv}}HERE
-mlr head and mlr tail count records rather than lines. The CSV
-header is included either way:
+
mlr head and mlr tail count records rather than lines. The CSV
+header is included either way:
POKI_RUN_COMMAND{{mlr --csv head -n 4 example.csv}}HERE
POKI_RUN_COMMAND{{mlr --csv tail -n 4 example.csv}}HERE
- Sort primarily alphabetically on one field, then secondarily
-numerically descending on another field:
+
Sort primarily alphabetically on one field, then secondarily
+numerically descending on another field:
POKI_RUN_COMMAND{{mlr --icsv --opprint sort -f shape -nr index example.csv}}HERE
- Use cut to retain only specified fields, in input-data order:
+
Use cut to retain only specified fields, in input-data order:
POKI_RUN_COMMAND{{mlr --icsv --opprint cut -f flag,shape example.csv}}HERE
- Use cut -o to retain only specified fields, in your specified order:
+
Use cut -o to retain only specified fields, in your specified order:
POKI_RUN_COMMAND{{mlr --icsv --opprint cut -o -f flag,shape example.csv}}HERE
- Use cut -x to omit specified fields:
+
Use cut -x to omit specified fields:
POKI_RUN_COMMAND{{mlr --icsv --opprint cut -x -f flag,shape example.csv}}HERE
- Use filter to retain specified records:
+
Use filter to retain specified records:
POKI_RUN_COMMAND{{mlr --icsv --opprint filter '$color == "red"' example.csv}}HERE
POKI_RUN_COMMAND{{mlr --icsv --opprint filter '$color == "red" && $flag == 1' example.csv}}HERE
- Use put to add/replace fields which are computed from other fields:
+
Use put to add/replace fields which are computed from other fields:
POKI_RUN_COMMAND{{mlr --icsv --opprint put '$ratio = $quantity / $rate; $color_shape = $color . "_" . $shape' example.csv}}HERE
- Even though Miller’s main selling point is
+
Even though Miller’s main selling point is
name-indexing, sometimes you really want to refer to a field name by its
positional index. Use $[[3]] to access the name of field 3 or
-$[[[3]]] to access the value of field 3:
+$[[[3]]] to access the value of field 3:
POKI_RUN_COMMAND{{mlr --icsv --opprint put '$[[3]] = "NEW"' example.csv}}HERE
POKI_RUN_COMMAND{{mlr --icsv --opprint put '$[[[3]]] = "NEW"' example.csv}}HERE
- JSON output:
+
JSON output:
POKI_RUN_COMMAND{{mlr --icsv --ojson put '$ratio = $quantity/$rate; $shape = toupper($shape)' example.csv}}HERE
- JSON output with vertical-formatting flags:
+
JSON output with vertical-formatting flags:
POKI_RUN_COMMAND{{mlr --icsv --ojson --jvstack --jlistwrap tail -n 2 example.csv}}HERE
- Use then to pipe commands together. Also, the
+
Use then to pipe commands together. Also, the
-g option for many Miller commands is for group-by: here, head -n
1 -g shape outputs the first record for each distinct value of the
shape field. This means we’re finding the record with highest
-index field for each distinct shape field:
+index field for each distinct shape field:
POKI_RUN_COMMAND{{mlr --icsv --opprint sort -f shape -nr index then head -n 1 -g shape example.csv}}HERE
- Statistics can be computed with or without group-by field(s). Also, the first of these two
+
Statistics can be computed with or without group-by field(s). Also, the first of these two
examples uses --oxtab output format which is a nice alternative to --opprint when you
-have lots of columns:
+have lots of columns:
POKI_RUN_COMMAND{{mlr --icsv --oxtab --from example.csv stats1 -a p0,p10,p25,p50,p75,p90,p99,p100 -f rate}}HERE
POKI_RUN_COMMAND{{mlr --icsv --opprint --from example.csv stats1 -a count,min,mean,max -f quantity -g shape}}HERE
@@ -89,11 +89,11 @@ POKI_RUN_COMMAND{{mlr --icsv --opprint --from example.csv stats1 -a count,min,me
-Often we want to print output to the screen. Miller does this by default, as we’ve
-seen in the previous examples.
+
Often we want to print output to the screen. Miller does this by default, as we’ve
+seen in the previous examples.
- Sometimes we want to print output to another file: just use '>
-outputfilenamegoeshere' at the end of your command:
+ Sometimes we want to print output to another file: just use '>
+outputfilenamegoeshere' at the end of your command:
- Other times we just want our files to be changed in-place: just use 'mlr -I'.
+ Other times we just want our files to be changed in-place: just use 'mlr -I'.
@@ -166,8 +166,8 @@ mlr -I --csv cut -x -f unwanted_column_name *.csv
If you like, you can first copy off your original data somewhere else, before doing in-place operations.
- Lastly, using tee within put, you can split your input data into separate files
-per one or more field names:
+
Lastly, using tee within put, you can split your input data into separate files
+per one or more field names:
POKI_RUN_COMMAND{{mlr --csv --from example.csv put -q 'tee > $shape.".csv", $*'}}HERE
@@ -207,7 +207,7 @@ shape=square,flag=0,index=36
-Data written this way are called DKVP, for delimited key-value pairs.
+
Data written this way are called DKVP, for delimited key-value pairs.
We’ve also already seen other ways to write the same data:
@@ -231,8 +231,8 @@ shape=square,flag=0,index=36 flag 1 "flag": 0,
-Anything we can do with CSV input data, we can do with any
-other format input data. And you can read from one format, do any
+
Anything we can do with CSV input data, we can do with any
+other format input data.
And you can read from one format, do any
record-processing, and output to the same format as the input, or to a
different output format.
diff --git a/doc/content-for-build.html b/doc/content-for-build.html
index 109148d97..04795755d 100644
--- a/doc/content-for-build.html
+++ b/doc/content-for-build.html
@@ -241,7 +241,7 @@ other 21st-century compilers)
Optional external dependencies
-This documentation pageset is built using Poki:
+This documentation pageset is built using Poki:
docs here,
source code here.
Note that http://johnkerl.org/miller/doc/index.html
diff --git a/doc/content-for-faq.html b/doc/content-for-faq.html
index 9e49f4cf3..3bce6a09b 100644
--- a/doc/content-for-faq.html
+++ b/doc/content-for-faq.html
@@ -373,14 +373,14 @@ POKI_RUN_COMMAND{{mlr --csv join --ul -j id -f data/color-codes.csv then unspars
-Miller handles tabular data, which is a list of
+Miller handles tabular data, which is a list of
records each having fields which are key-value pairs. Miller also doesn’t
require that each record have the same field names (see also here). Regardless, tabular data is a
-non-recursive data structure.
+non-recursive data structure.
- XML, JSON, etc. are, by contrast, all recursive
-or nested data structures. For example, in JSON
+ XML, JSON, etc. are, by contrast, all recursive
+or nested data structures. For example, in JSON
you can represent a hash map whose values are lists of lists.
Now, you can put tabular data into these formats — since list-of-key-value-pairs
diff --git a/doc/content-for-file-formats.html b/doc/content-for-file-formats.html
index 7f01cc1cd..8024b1d17 100644
--- a/doc/content-for-file-formats.html
+++ b/doc/content-for-file-formats.html
@@ -179,7 +179,7 @@ POKI_RUN_COMMAND{{mlr --nidx --fs ' ' --repifs cut -f 2,3 data/mydata.txt}}HERE
JSON is a format which supports arbitrarily deep nesting of
“objects” (hashmaps) and “arrays” (lists), while Miller
-is a tool for handling tabular data only. This means
+is a tool for handling tabular data only. This means
Miller cannot (and should not) handle arbitrary JSON. (Check out jq.)
@@ -187,8 +187,8 @@ href="http://stedolan.github.io/jq/">jq.)
Single-level JSON objects
-An array of single-level objects is, quite simply,
-a table:
+An array of single-level objects is, quite simply,
+a table:
POKI_RUN_COMMAND{{mlr --json head -n 2 then cut -f color,shape data/json-example-1.json}}HERE
POKI_RUN_COMMAND{{mlr --json --jvstack head -n 2 then cut -f color,u,v data/json-example-1.json}}HERE
@@ -196,7 +196,7 @@ POKI_RUN_COMMAND{{mlr --ijson --opprint stats1 -a mean,stddev,count -f u -g shap
Nested JSON objects
-Additionally, Miller can tabularize nested objects by concatentating keys:
+Additionally, Miller can tabularize nested objects by concatentating keys:
POKI_RUN_COMMAND{{mlr --json --jvstack head -n 2 data/json-example-2.json}}HERE
POKI_RUN_COMMAND{{mlr --ijson --opprint head -n 4 data/json-example-2.json}}HERE
@@ -358,15 +358,15 @@ POKI_RUN_COMMAND{{mlr --usage-format-conversion-keystroke-saver-options}}HERE
Default line endings (--irs and --ors) are 'auto'
-which means autodetect from the input file format, as
+which means autodetect from the input file format, as
long as the input file(s) have lines ending in either LF (also known as
linefeed, '\n', 0x0a, Unix-style) or CRLF (also known as
carriage-return/linefeed pairs, '\r\n', 0x0d 0x0a, Windows
style).
- If both IRS and ORS are auto (which is the default) then LF input will
+ If both IRS and ORS are auto (which is the default) then LF input will
lead to LF output and CRLF input will lead to CRLF output, regardless of the
-platform you’re running on.
+platform you’re running on.
The line-ending autodetector triggers on the first line ending detected in
the input stream. E.g. if you specify a CRLF-terminated file on the command
diff --git a/doc/content-for-originality.html b/doc/content-for-originality.html
index 9aa33cd48..40a031eb1 100644
--- a/doc/content-for-originality.html
+++ b/doc/content-for-originality.html
@@ -6,7 +6,7 @@ online-analytical-processing culture. Other key participants include
concept, Miller explicitly strives to imitate several existing tools:
-Unix toolkit: Intentional similarities as described in
+Unix toolkit: Intentional similarities as described in
POKI_PUT_LINK_FOR_PAGE(feature-comparison.html)HERE.
Recipes abound for command-line data analysis using the Unix toolkit. Here are just a couple of my favorites:
@@ -16,7 +16,7 @@ POKI_PUT_LINK_FOR_PAGE(feature-comparison.html)HERE.
https://github.com/dbohdan/structured-text-tools
- RecordStream: Miller owes particular inspiration
+ RecordStream: Miller owes particular inspiration
to RecordStream. The
key difference is that RecordStream is a Perl-based tool for manipulating JSON
(including requiring it to separately manipulate other formats such as CSV into
@@ -25,15 +25,15 @@ The similarities include the sort, stats1 (analog of
RecordStream’s collate), and delta operations, as well
as filter and put, and pretty-print formatting.
- stats_m: A third source of lineage is my Python
+ stats_m: A third source of lineage is my Python
stats_m
module. This includes simple single-pass algorithms which form Miller’s
stats1 and stats2 subcommands.
- SQL: Fourthly, Miller’s group-by command
+ SQL: Fourthly, Miller’s group-by command
name is from SQL, as is the term aggregate.
- Added value:
+ Added value:
Miller’s added values include:
Name-indexing, compared to the Unix toolkit’s positional indexing.
@@ -43,17 +43,17 @@ Miller’s added values include:
Various file formats, and on-the-fly format conversion.
-jq: Miller does for name-indexed text what
+jq: Miller does for name-indexed text what
jq does for JSON. If you’re
not already familiar with jq, please check it out!.
-What about similar tools?
+What about similar tools?
Here’s a comprehensive list:
https://github.com/dbohdan/structured-text-tools.
It doesn’t mention rows so here’s a plug for that as well.
As it turns out, I learned about most of these after writing Miller.
-What about DOTADIW? One of the key points of the
+What about DOTADIW? One of the key points of the
Unix philosophy is
that a tool should do one thing and do it well. Hence sort and
cut do just one thing. Why does Miller put awk-like
@@ -70,7 +70,7 @@ be a tool which collects together format-aware record-stream processing into
one place, with good reuse of Miller-internal library code for its various
features.
-Why not use Perl/Python/Ruby etc.? Maybe you
+Why not use Perl/Python/Ruby etc.? Maybe you
should. With those tools you’ll get far more expressive power, and
sufficiently quick turnaround time for small-to-medium-sized data. Using
Miller you’ll get something less than a complete programming language,
@@ -88,8 +88,8 @@ If I’d gotten good enough performance from the latter I’d have done
it without question and Miller would be far more flexible. But C won the
performance criteria by a landslide so we have Miller in C with a custom DSL.
- No, really, why one more command-line data-manipulation
-tool? I wrote Miller because I was frustrated with tools like
+ No, really, why one more command-line data-manipulation
+tool? I wrote Miller because I was frustrated with tools like
grep, sed, and so on being line-aware without being
format-aware. The single most poignant example I can think of is seeing
people grep data lines out of their CSV files and sadly losing their header
diff --git a/doc/content-for-why.html b/doc/content-for-why.html
index f7fb3eb1e..5c342a34b 100644
--- a/doc/content-for-why.html
+++ b/doc/content-for-why.html
@@ -13,15 +13,15 @@ why I felt it necessary to build Miller, etc. Here are some answers.
For background, I’m a software engineer, with a heavy devops bent
and a non-trivial amount of data-engineering in my career.
-Initially I wrote Miller mainly for myself: I’m
+Initially I wrote Miller mainly for myself: I’m
coder-friendly (being a coder); I’m Github-friendly; most of my data are
well-structured or easily structurable (TSV-formatted SQL-query output, CSV
files, log files, JSON data structures); I care about interoperability between
all the various formats Miller supports (I’ve encountered them all); I do
all my work on Linux or OSX.
- But now there’s this neat little tool which seems to be
-useful for people in various disciplines. I don’t even know
+ But now there’s this neat little tool which seems to be
+useful for people in various disciplines. I don’t even know
entirely who. I can click through Github starrers and read a bit about
what they seem to do, but not everyone’s on Github (or stars
things). I’ve gotten a lot of feature requests through Github — but
@@ -50,7 +50,7 @@ it’s for?
years of my career in the software industry I’ve found myself, and
others, doing a lot of ad-hoc things which really were fundamentally the same
except for format. So the number one thing about Miller is doing common
-things while supporting multiple formats: (a) ingest a
+things while supporting multiple formats: (a) ingest a
list of records where a record is a list of key-value pairs (however
represented in the input files); (b) transform that stream of records; (c) emit
the transformed stream — either in the same format as input, or in a
@@ -60,9 +60,9 @@ different format.
something only for a single file format, I didn’t want to build something
only for one problem domain. In my work doing software engineering, devops,
data engineering, etc. I saw a lot of commonalities and I wanted to
-solve as many problems simultaneously as possible.
+solve as many problems simultaneously as possible.
- Third: it had to be streaming. As time goes by
+ Third: it had to be streaming. As time goes by
and we (some of us, sometimes) have machines with tens or hundreds of GB of
RAM, it’s maybe less important, but I’m unhappy with tools which
ingest all data, then do stuff, then emit all data. One reason is to be able to
@@ -71,7 +71,7 @@ input which trickles in, e.g. you have some process emitting data now and then
and you can pipe it to Miller and it will emit transformed records one at a
time.
- Fourth: it had to be fast. This precludes all
+ Fourth: it had to be fast. This precludes all
sorts of very nice things written in Ruby, for example. I love Ruby as a very
expressive language, and I have several very useful little utility scripts
written in Ruby. But a few years ago I ported over some of my old
@@ -83,8 +83,8 @@ in order to make it performant. I did simple experiments in several languages,
and nothing was as fast as C, so I used C: see also here.
- Fifth thing: I wanted Miller to be pipe-friendly and
-interoperate with other command-line tools. Since the basic
+ Fifth thing: I wanted Miller to be pipe-friendly and
+interoperate with other command-line tools. Since the basic
paradigm is ingest records, transform records, emit records — where the
input and output formats can be the same or different, and the transform can be
complex, or just pass-through — this means you can use it to transform
@@ -93,17 +93,17 @@ data-cleaning/prep/formatting and do all the "real" work in R, you can. If you
just want a little glue script between other tools you can get that. And if you
want to do non-trivial data-reduction in Miller you can.
- Sixth thing: Must have comprehensive documentation and
-unit-test. Since Miller handles a lot of formats and solves a lot
+ Sixth thing: Must have comprehensive documentation and
+unit-test. Since Miller handles a lot of formats and solves a lot
of problems, there’s a lot to test and a lot to keep working correctly as
I add features or optimize. And I wanted it to be able to explain itself
— not only through web docs like the one you’re reading but also
through man mlr and mlr --help, mlr sort --help,
etc.
- Seventh thing: Must have a domain-specific
-language (DSL) but also must let you do common things
-without it. All those little verbs Miller has to help you
+ Seventh thing: Must have a domain-specific
+language (DSL) but also must let you do common things
+without it. All those little verbs Miller has to help you
avoid having to write for-loops are great. I use them for
keystroke-saving: mlr stats1 -a mean,stddev,min,max -f quantity, for
example, without you having to write for-loops or define accumulator variables.
@@ -112,8 +112,8 @@ you want to: mlr put '$distance = $rate * $time' or anything else y
can think up. In Perl/AWK/etc. it’s all DSL. In xsv et al. it’s
all verbs. In Miller I like having the combination.
- Eighth thing: It’s an awful lot of fun to
-write. In my experience I didn’t find any tools which do
+ Eighth thing: It’s an awful lot of fun to
+write. In my experience I didn’t find any tools which do
multi-format, streaming, efficient, multi-purpose, with DSL and non-DSL, so I
wrote one. But I don’t guarantee it’s unique in the world. It fills
a niche in the world (people use it) but it also fills a niche in my life.
diff --git a/doc/css/miller.css b/doc/css/miller.css
index 692e07743..12dc0abeb 100644
--- a/doc/css/miller.css
+++ b/doc/css/miller.css
@@ -14,10 +14,10 @@ body {
background-color: white;
}
-maroon {
+.maroon {
color: maroon;
}
-boldmaroon {
+.boldmaroon {
font-weight: bold;
color: maroon;
}
diff --git a/doc/faq.html b/doc/faq.html
index f3479b577..02fae1956 100644
--- a/doc/faq.html
+++ b/doc/faq.html
@@ -995,14 +995,14 @@ id,code,color
-Miller handles tabular data, which is a list of
+Miller handles tabular data, which is a list of
records each having fields which are key-value pairs. Miller also doesn’t
require that each record have the same field names (see also here). Regardless, tabular data is a
-non-recursive data structure.
+non-recursive data structure.
- XML, JSON, etc. are, by contrast, all recursive
-or nested data structures. For example, in JSON
+ XML, JSON, etc. are, by contrast, all recursive
+or nested data structures. For example, in JSON
you can represent a hash map whose values are lists of lists.
Now, you can put tabular data into these formats — since list-of-key-value-pairs
diff --git a/doc/file-formats.html b/doc/file-formats.html
index 939f1870b..0669a9c39 100644
--- a/doc/file-formats.html
+++ b/doc/file-formats.html
@@ -420,7 +420,7 @@ light
JSON is a format which supports arbitrarily deep nesting of
“objects” (hashmaps) and “arrays” (lists), while Miller
-is a tool for handling tabular data only. This means
+is a tool for handling tabular data only. This means
Miller cannot (and should not) handle arbitrary JSON. (Check out jq.)
@@ -428,8 +428,8 @@ href="http://stedolan.github.io/jq/">jq.)
Single-level JSON objects
-An array of single-level objects is, quite simply,
-a table:
+An array of single-level objects is, quite simply,
+a table:
@@ -471,7 +471,7 @@ circle 0.366013 0.209094 3
Nested JSON objects
-Additionally, Miller can tabularize nested objects by concatentating keys:
+Additionally, Miller can tabularize nested objects by concatentating keys:
@@ -878,15 +878,15 @@ output only.
Default line endings (--irs and --ors) are 'auto'
-which means autodetect from the input file format, as
+which means autodetect from the input file format, as
long as the input file(s) have lines ending in either LF (also known as
linefeed, '\n', 0x0a, Unix-style) or CRLF (also known as
carriage-return/linefeed pairs, '\r\n', 0x0d 0x0a, Windows
style).
- If both IRS and ORS are auto (which is the default) then LF input will
+ If both IRS and ORS are auto (which is the default) then LF input will
lead to LF output and CRLF input will lead to CRLF output, regardless of the
-platform you’re running on.
+platform you’re running on.
The line-ending autodetector triggers on the first line ending detected in
the input stream. E.g. if you specify a CRLF-terminated file on the command
diff --git a/doc/originality.html b/doc/originality.html
index 2150eed4f..d9c46af86 100644
--- a/doc/originality.html
+++ b/doc/originality.html
@@ -72,7 +72,7 @@ online-analytical-processing culture. Other key participants include
concept, Miller explicitly strives to imitate several existing tools:
-Unix toolkit: Intentional similarities as described in
+Unix toolkit: Intentional similarities as described in
Unix-toolkit context.
Recipes abound for command-line data analysis using the Unix toolkit. Here are just a couple of my favorites:
@@ -82,7 +82,7 @@ concept, Miller explicitly strives to imitate several existing tools:
https://github.com/dbohdan/structured-text-tools
- RecordStream: Miller owes particular inspiration
+ RecordStream: Miller owes particular inspiration
to RecordStream. The
key difference is that RecordStream is a Perl-based tool for manipulating JSON
(including requiring it to separately manipulate other formats such as CSV into
@@ -91,15 +91,15 @@ The similarities include the sort, stats1 (analog of
RecordStream’s collate), and delta operations, as well
as filter and put, and pretty-print formatting.
- stats_m: A third source of lineage is my Python
+ stats_m: A third source of lineage is my Python
stats_m
module. This includes simple single-pass algorithms which form Miller’s
stats1 and stats2 subcommands.
- SQL: Fourthly, Miller’s group-by command
+ SQL: Fourthly, Miller’s group-by command
name is from SQL, as is the term aggregate.
- Added value:
+ Added value:
Miller’s added values include:
Name-indexing, compared to the Unix toolkit’s positional indexing.
@@ -109,17 +109,17 @@ Miller’s added values include:
Various file formats, and on-the-fly format conversion.
-jq: Miller does for name-indexed text what
+jq: Miller does for name-indexed text what
jq does for JSON. If you’re
not already familiar with jq, please check it out!.
-What about similar tools?
+What about similar tools?
Here’s a comprehensive list:
https://github.com/dbohdan/structured-text-tools.
It doesn’t mention rows so here’s a plug for that as well.
As it turns out, I learned about most of these after writing Miller.
-What about DOTADIW? One of the key points of the
+What about DOTADIW? One of the key points of the
Unix philosophy is
that a tool should do one thing and do it well. Hence sort and
cut do just one thing. Why does Miller put awk-like
@@ -136,7 +136,7 @@ be a tool which collects together format-aware record-stream processing into
one place, with good reuse of Miller-internal library code for its various
features.
-Why not use Perl/Python/Ruby etc.? Maybe you
+Why not use Perl/Python/Ruby etc.? Maybe you
should. With those tools you’ll get far more expressive power, and
sufficiently quick turnaround time for small-to-medium-sized data. Using
Miller you’ll get something less than a complete programming language,
@@ -154,8 +154,8 @@ If I’d gotten good enough performance from the latter I’d have done
it without question and Miller would be far more flexible. But C won the
performance criteria by a landslide so we have Miller in C with a custom DSL.
- No, really, why one more command-line data-manipulation
-tool? I wrote Miller because I was frustrated with tools like
+ No, really, why one more command-line data-manipulation
+tool? I wrote Miller because I was frustrated with tools like
grep, sed, and so on being line-aware without being
format-aware. The single most poignant example I can think of is seeing
people grep data lines out of their CSV files and sadly losing their header
diff --git a/doc/why.html b/doc/why.html
index 2bab78398..f29641141 100644
--- a/doc/why.html
+++ b/doc/why.html
@@ -84,15 +84,15 @@ why I felt it necessary to build Miller, etc. Here are some answers.
For background, I’m a software engineer, with a heavy devops bent
and a non-trivial amount of data-engineering in my career.
-Initially I wrote Miller mainly for myself: I’m
+Initially I wrote Miller mainly for myself: I’m
coder-friendly (being a coder); I’m Github-friendly; most of my data are
well-structured or easily structurable (TSV-formatted SQL-query output, CSV
files, log files, JSON data structures); I care about interoperability between
all the various formats Miller supports (I’ve encountered them all); I do
all my work on Linux or OSX.
- But now there’s this neat little tool which seems to be
-useful for people in various disciplines. I don’t even know
+ But now there’s this neat little tool which seems to be
+useful for people in various disciplines. I don’t even know
entirely who. I can click through Github starrers and read a bit about
what they seem to do, but not everyone’s on Github (or stars
things). I’ve gotten a lot of feature requests through Github — but
@@ -121,7 +121,7 @@ it’s for?
years of my career in the software industry I’ve found myself, and
others, doing a lot of ad-hoc things which really were fundamentally the same
except for format. So the number one thing about Miller is doing common
-things while supporting multiple formats: (a) ingest a
+things while supporting multiple formats: (a) ingest a
list of records where a record is a list of key-value pairs (however
represented in the input files); (b) transform that stream of records; (c) emit
the transformed stream — either in the same format as input, or in a
@@ -131,9 +131,9 @@ different format.
something only for a single file format, I didn’t want to build something
only for one problem domain. In my work doing software engineering, devops,
data engineering, etc. I saw a lot of commonalities and I wanted to
-solve as many problems simultaneously as possible.
+solve as many problems simultaneously as possible.
- Third: it had to be streaming. As time goes by
+ Third: it had to be streaming. As time goes by
and we (some of us, sometimes) have machines with tens or hundreds of GB of
RAM, it’s maybe less important, but I’m unhappy with tools which
ingest all data, then do stuff, then emit all data. One reason is to be able to
@@ -142,7 +142,7 @@ input which trickles in, e.g. you have some process emitting data now and then
and you can pipe it to Miller and it will emit transformed records one at a
time.
- Fourth: it had to be fast. This precludes all
+ Fourth: it had to be fast. This precludes all
sorts of very nice things written in Ruby, for example. I love Ruby as a very
expressive language, and I have several very useful little utility scripts
written in Ruby. But a few years ago I ported over some of my old
@@ -154,8 +154,8 @@ in order to make it performant. I did simple experiments in several languages,
and nothing was as fast as C, so I used C: see also here.
- Fifth thing: I wanted Miller to be pipe-friendly and
-interoperate with other command-line tools. Since the basic
+ Fifth thing: I wanted Miller to be pipe-friendly and
+interoperate with other command-line tools. Since the basic
paradigm is ingest records, transform records, emit records — where the
input and output formats can be the same or different, and the transform can be
complex, or just pass-through — this means you can use it to transform
@@ -164,17 +164,17 @@ data-cleaning/prep/formatting and do all the "real" work in R, you can. If you
just want a little glue script between other tools you can get that. And if you
want to do non-trivial data-reduction in Miller you can.
- Sixth thing: Must have comprehensive documentation and
-unit-test. Since Miller handles a lot of formats and solves a lot
+ Sixth thing: Must have comprehensive documentation and
+unit-test. Since Miller handles a lot of formats and solves a lot
of problems, there’s a lot to test and a lot to keep working correctly as
I add features or optimize. And I wanted it to be able to explain itself
— not only through web docs like the one you’re reading but also
through man mlr and mlr --help, mlr sort --help,
etc.
- Seventh thing: Must have a domain-specific
-language (DSL) but also must let you do common things
-without it. All those little verbs Miller has to help you
+ Seventh thing: Must have a domain-specific
+language (DSL) but also must let you do common things
+without it. All those little verbs Miller has to help you
avoid having to write for-loops are great. I use them for
keystroke-saving: mlr stats1 -a mean,stddev,min,max -f quantity, for
example, without you having to write for-loops or define accumulator variables.
@@ -183,8 +183,8 @@ you want to: mlr put '$distance = $rate * $time' or anything else y
can think up. In Perl/AWK/etc. it’s all DSL. In xsv et al. it’s
all verbs. In Miller I like having the combination.
- Eighth thing: It’s an awful lot of fun to
-write. In my experience I didn’t find any tools which do
+ Eighth thing: It’s an awful lot of fun to
+write. In my experience I didn’t find any tools which do
multi-format, streaming, efficient, multi-purpose, with DSL and non-DSL, so I
wrote one. But I don’t guarantee it’s unique in the world. It fills
a niche in the world (people use it) but it also fills a niche in my life.