What’s new in Miller 6

See also the list of issues tagged with go-port.

Documentation improvements

Documentation (what you’re reading here) and on-line help (mlr --help) have been completely reworked.

In the initial release, the focus was convincing users already familiar with awk/grep/cut that Miller was a viable option. Over time it’s become clear that many users aren’t expert with these. The focus has shifted toward a higher quantity of more introductory/accessible material for command-line data processing.

Similarly, the FAQ/recipe material has been expanded to include more, and simpler, use-cases including resolved questions from https://github.com/johnkerl/miller/issues and https://github.com/johnkerl/miller/discussions. More complex/niche material has been pushed farther down. The long reference pages have been split up into separate pages.

Since CSV is overwhelmingly the most popular data format for Miller, it is now discussed first, and more examples use CSV.

JSON support, and arrays

Arrays are now supported in Miller’s put/filter programming language, as described at Reference: arrays. Also, array is now a keyword so this is no longer usable as a local-variable or UDF name.

JSON support is improved:

  • Direct support for arrays means that you can now use Miller to process more JSON files.

  • Streamable JSON parsing: Miller’s internal record-processing pipeline starts as soon as the first record is read (which was already the case for other file formats). This means that, unless records are wrapped with outermost [...], Miller now handles JSON in tail -f contexts like it does for other file formats.

  • Flatten/unflatten – TODO pick a name and link to a separate page/section

Improved Windows experience

Stronger support for Windows (with or without MSYS2), with a couple of exceptions. See Miller on Windows for more information.

Binaries are reliably available using GitHub Actions: see also Installation.

In-process support for compressed input

In addition to --prepipe gunzip, you can now use the --gzin flag. In fact, if your files end in .gz you don’t even need to do that – Miller will autodetect by file extension and automatically uncompress mlr --csv cat foo.csv.gz. Similarly for .z and .bz2 files. Please see section [TODO:linkify] for more information.

Output colorization

Miller uses separate, customizable colors for keys and values whenever the output is to a terminal. See Output colorization.

Improved numeric conversion

The most central part of Miller 6 is a deep refactor of how data values are parsed from file contents, how types are inferred, and how they’re converted back to text into output files.

This was all initiated by https://github.com/johnkerl/miller/issues/151.

In Miller 5 and below, all values were stored as strings, then only converted to int/float as-needed, for example when a particular field was referenced in the stats1 or put verbs. This led to awkwardnesses such as the -S and -F flags for put and filter.

In Miller 6, things parseable as int/float are treated as such from the moment the input data is read, and these are passed along through the verb chain. All values are typed from when they’re read, and their types are passed along. Meanwhile the original string representation of each value is also retained. If a numeric field isn’t modified during the processing chain, it’s printed out the way it arrived. Also, quoted values in JSON strings are flagged as being strings throughout the processing chain.

For example (see https://github.com/johnkerl/miller/issues/178) you can now do

 echo '{ "a": "0123" }' | mlr --json cat
 {
   "a": "0123"
 }

 echo '{ "x": 1.230, "y": 1.230000000 }' | mlr --json cat
 {
   "x": 1.230,
   "y": 1.230000000
 }

REPL

Miller now has a read-evaluate-print-loop (The REPL) where you can single-step through your data-file record, express arbitrary statements to converse with the data, etc.

New DSL functions / operators

Improved command-line parsing

Miller 6 has getoptish command-line parsing (https://github.com/johnkerl/miller/pull/467):

  • -xyz expands automatically to -x -y -z, so (for example) mlr cut -of shape,flag is the same as mlr cut -o -f shape,flag.

  • --foo=bar expands automatically to --foo bar, so (for example) mlr --ifs=comma is the same as mlr --ifs comma.

  • --mfrom, --load, --mload as described at [TODO:linkify].

Improved error messages for DSL parsing

For mlr put and mlr filter, parse-error messages now include location information:

mlr: cannot parse DSL expression.
Parse error on token ">" at line 63 columnn 7.

Developer-specific aspects