f637633420
* Tier-2 structured verb options: OptionSpec, initial migration (#2098) PR 3 of the AI-friendly roadmap (plans/plan-2098-llm.md). Infrastructure: - Add OptionSpec{Flag,Arg,Type,Desc,Repeatable,Values} to pkg/transformers/aaa_record_transformer.go alongside TransformerSetup. Type is one of: bool, string, int, float, csv-list, regex, filename, format, enum. For type=="enum", Values lists the valid choices. - Add Options []OptionSpec to TransformerSetup (nil = not yet migrated). - Emit Options in VerbInfoForJSON (omitempty so unmigrated verbs stay backward-compatible; agents check key presence for Tier-2 availability). UsageText is always present as the Tier-1 prose fallback. - Add VerbOptionsNilCheck() in aaa_verb_options_check.go: progress report of migrated vs. unmigrated verbs, analogous to FLAG_TABLE.NilCheck(). - Wire verb-options-nil-check into mlr help (internal/docgen section). Initial migration (5/70 verbs): - nothing: empty Options (no verb-specific options, explicitly migrated) - cat: -n (bool), -N (string), -g (csv-list), --filename, --filenum (bool) - head: -g (csv-list), -n (int) - tail: -g (csv-list), -n (int) - tee: -a, -p (bool) Tests: - 5 new unit tests in aaa_transformer_json_test.go covering migrated/ unmigrated paths, field population, JSON round-trip, and key-presence. - Regression test case 0003: mlr help verb-options-nil-check golden output. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * Migrate all 70 verbs to structured OptionSpec; bump catalog schema to v2 Completes the Tier-2 migration started in the previous commit. Every verb in TRANSFORMER_LOOKUP_TABLE now has a non-nil Options field. - Workflow-migrated all 65 remaining verbs. Each Setup var now carries Options: []OptionSpec{...} with Flag/Arg/Type/Desc fields. Verbs with no verb-specific options (altkv, check, group-like, nothing, etc.) use an empty slice to signal "migrated but no options." - Drop `omitempty` from VerbInfoForJSON.Options: empty slices were silently dropped, making migrated-no-option verbs indistinguishable from unmigrated ones in JSON. Without omitempty: null=unmigrated, []=migrated-no-options, [...]= migrated-with-options. Bump catalogSchemaVersion 1→2 for this shape change. - Replace the two "unmigrated-verb" unit tests (which used stats1 as an example) with TestAllVerbsFullyMigrated (asserts every verb has non-nil Options) and TestAllVerbsHaveOptionsKeyInJSON (asserts every migrated verb emits the "options" key in JSON). - Regenerate test/cases/cli-help/0003/expout: now reads "Verb options migration: 70/70 migrated." Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * remove a transitional helper * git rms * Render verb usage Options blocks from structured OptionSpec Each verb's usage message and its Tier-2 OptionSpec list previously duplicated the option text. New WriteVerbOptions (aaa_verb_usage.go) renders the "Options:" block from the specs: aligned flag column, descriptions word-wrapped at 80, uniform trailing -h|--help line. - OptionSpec gains Aliases (JSON "aliases") so long-form spellings like join's --lk|--left-keep-field-names survive in both outputs - All 70 verbs migrated; options literals hoisted to package-level vars (usage funcs can't reference their Setup var without a Go init cycle) - Hand-written per-option details the specs had condensed away are merged into Desc, enriching the JSON catalog - Non-option prose (examples, cross-references, dynamic accumulator listings) kept verbatim - Regenerated the six usage-embedding regression expectations and the two affected doc pages Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix pre-existing usage-text bugs surfaced by the OptionSpec migration - gap: usage said "One of -f or -g is required" but the parser takes -n or -g - seqgen: drop description line copy-pasted from cat ("Passes input records directly to output...") which contradicted "Discards the input record stream" - utf8-to-latin1: description read inverted ("from Latin-1 to UTF-8") - sec2gmtdate: usage said "../c/mlr" instead of "mlr" - top: document the accepted-but-undocumented --max flag - stats2: add linreg-pca to the -a enum values, matching the runtime accumulator table Regression expectations and docs regenerated accordingly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix check usage sentence order; stats1 usage blank line to usage stream - check: the description's second and third lines were swapped, reading "Consumes records without printing any output, / Useful for doing a well-formatted check on input data. / with the exception that warnings are printed to stderr." - stats1: a bare fmt.Println() in the usage func wrote its blank line to process stdout instead of the usage output stream Regression expectation and docs regenerated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> |
||
|---|---|---|
| .github | ||
| cmd | ||
| data | ||
| docs | ||
| man | ||
| pkg | ||
| plans | ||
| python | ||
| scripts | ||
| snap | ||
| test | ||
| tools | ||
| vim | ||
| .all-contributorsrc | ||
| .codespellignore | ||
| .gitattributes | ||
| .gitignore | ||
| .golangci.yml | ||
| .goreleaser.yml | ||
| .readthedocs.yaml | ||
| .vimrc | ||
| CLAUDE.md | ||
| configure | ||
| create-release-tarball | ||
| delve.txt | ||
| go.mod | ||
| go.sum | ||
| LICENSE.txt | ||
| Makefile | ||
| miller.spec | ||
| README-dev.md | ||
| README-docs.md | ||
| README-go-port.md | ||
| README-profiling.md | ||
| README-RPM.md | ||
| README-versions.md | ||
| README.md | ||
| regression_test.go | ||
| todo.txt | ||
What is Miller?
Miller is like awk, sed, cut, join, and sort for data formats such as CSV, TSV, JSON, JSON Lines, and positionally-indexed.
What can Miller do for me?
With Miller, you get to use named fields without needing to count positional indices, using familiar formats such as CSV, TSV, JSON, JSON Lines, and positionally-indexed. Then, on the fly, you can add new fields which are functions of existing fields, drop fields, sort, aggregate statistically, pretty-print, and more.
-
Miller operates on key-value-pair data while the familiar Unix tools operate on integer-indexed fields: if the natural data structure for the latter is the array, then Miller's natural data structure is the insertion-ordered hash map.
-
Miller handles a variety of data formats, including but not limited to the familiar CSV, TSV, and JSON/JSON Lines. (Miller can handle positionally-indexed data too!)
In the above image you can see how Miller embraces the common themes of key-value-pair data in a variety of data formats.
Getting started
- Miller in 10 minutes
- A Guide To Command-Line Data Manipulation
- A quick tutorial on Miller
- Miller Exercises
- Tools to manipulate CSV files from the Command Line
- www.togaware.com/linux/survivor/CSV_Files.html
- MLR for CSV manipulation
- Linux Magazine: Process structured text files with Miller
- Miller: Command Line CSV File Processing
- Miller - A Swiss Army Chainsaw for CSV Data, Data Science and Data Munging
- Pandas Killer: mlr, the Scientist
More documentation links
- Full documentation
- Miller's license is two-clause BSD
- Notes about issue-labeling in the Github repo
- Active issues
Installing
There's a good chance you can get Miller pre-built for your system:
| OS | Installation command |
|---|---|
| Linux | yum install millerapt-get install millersnap install miller |
| Mac | brew install millerport install miller |
| Windows | choco install millerwinget install Miller.Millerscoop install main/miller |
See also README-versions.md for a full list of package versions. Note that long-term-support (LtS) releases will likely be on older versions.
See also building from source.
Community
- Discussion forum: https://github.com/johnkerl/miller/discussions
- Feature requests / bug reports: https://github.com/johnkerl/miller/issues
- How to contribute: https://miller.readthedocs.io/en/latest/contributing/
Build status
Building from source
- First:
cd /where/you/want/to/put/the/sourcegit clone https://github.com/johnkerl/millercd miller
- With
make:- To build:
make. This takes just a few seconds and produces the Miller executable, which is./mlr(or.\mlr.exeon Windows). - To run tests:
make check. - To install:
make install. This installs the executable/usr/local/bin/mlrand manual page/usr/local/share/man/man1/mlr.1(so you can doman mlr). - You can do
./configure --prefix=/some/install/pathbeforemake installif you want to install somewhere other than/usr/local.
- To build:
- Without
make:- To build:
go build github.com/johnkerl/miller/v6/cmd/mlr. - To run tests:
go test github.com/johnkerl/miller/v6/pkg/...andmlr regtest. - To install:
go install github.com/johnkerl/miller/v6/cmd/mlr@latestwill install to GOPATH/bin/mlr.
- To build:
- See also the doc page on building from source.
- For more developer information please see README-dev.md.
For developers
License
Features
-
Miller is multi-purpose: it's useful for data cleaning, data reduction, statistical reporting, devops, system administration, log-file processing, format conversion, and database-query post-processing.
-
You can use Miller to snarf and munge log-file data, including selecting out relevant substreams, then produce CSV format and load that into all-in-memory/data-frame utilities for further statistical and/or graphical processing.
-
Miller complements data-analysis tools such as R, pandas, etc.: you can use Miller to clean and prepare your data. While you can do basic statistics entirely in Miller, its streaming-data feature and single-pass algorithms enable you to reduce very large data sets.
-
Miller complements SQL databases: you can slice, dice, and reformat data on the client side on its way into or out of a database. You can also reap some of the benefits of databases for quick, setup-free one-off tasks when you just need to query some data in disk files in a hurry.
-
Miller also goes beyond the classic Unix tools by stepping fully into our modern, no-SQL world: its essential record-heterogeneity property allows Miller to operate on data where records with different schema (field names) are interleaved.
-
Miller is streaming: most operations need only a single record in memory at a time, rather than ingesting all input before producing any output. For those operations which require deeper retention (
sort,tac,stats1), Miller retains only as much data as needed. This means that whenever functionally possible, you can operate on files which are larger than your system’s available RAM, and you can use Miller in tail -f contexts. -
Miller is pipe-friendly and interoperates with the Unix toolkit.
-
Miller's I/O formats include tabular pretty-printing, positionally indexed (Unix-toolkit style), CSV, TSV, JSON, JSON Lines, and others.
-
Miller does conversion between formats.
-
Miller's processing is format-aware: e.g. CSV
sortandtackeep header lines first. -
Miller has high-throughput performance on par with the Unix toolkit.
-
Miller is written in portable, modern Go, with zero runtime dependencies. You can download or compile a single binary,
scpit to a faraway machine, and expect it to work.
What people are saying about Miller
Today I discovered Miller—it's like jq but for CSV: https://t.co/pn5Ni241KM
— Adrien Trouillaud (@adrienjt) September 24, 2020
Also, "Miller complements data-analysis tools such as R, pandas, etc.: you can use Miller to clean and prepare your data." @GreatBlueC @nfmcclure
Underappreciated swiss-army command-line chainsaw.
— Dirk Eddelbuettel (@eddelbuettel) February 28, 2017
"Miller is like awk, sed, cut, join, and sort for [...] CSV, TSV, and [...] JSON." https://t.co/TrQqSUK3KK
Miller looks like a great command line tool for working with CSV data. Sed, awk, cut, join all rolled into one: http://t.co/9BBb6VCZ6Y
— Mike Loukides (@mikeloukides) August 16, 2015
Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV: http://t.co/1zPbfg6B2W - handy tool!
— Ilya Grigorik (@igrigorik) August 22, 2015
Btw, I think Miller is the best CLI tool to deal with CSV. I used to use this when I need to preprocess too big CSVs to load into R (now we have vroom, so such cases might be rare, though...)https://t.co/kUjrSSGJoT
— Hiroaki Yutani (@yutannihilat_en) April 21, 2020
Miller: a *format-aware* data munging tool By @__jo_ker__ to overcome limitations with *line-aware* workshorses like awk, sed et al https://t.co/LCyPkhYvt9
— Donny Daniel (@dnnydnl) September 9, 2018
The project website is a fantastic example of good software documentation!!
Holy holly data swiss army knife batman! How did no one suggest Miller https://t.co/JGQpmRAZLv for solving database cleaning / ETL issues to me before
— James Miller (@japanlawprof) June 12, 2018
Congrats to @__jo_ker__ for amazingly intuitive tool for critical data management tasks!#DataScienceandLaw #ComputationalLaw
🤯@__jo_ker__'s Miller easily reads, transforms, + writes all sorts of tabular data. It's standalone, fast, and built for streaming data (operating on one line at a time, so you can work on files larger than memory).
— Benjamin Wolfe (he/him) (@BenjaminWolfe) September 9, 2021
And the docs are dream. I've been reading them all morning! https://t.co/Be2pGPZK6t
Contributors ✨
Thanks to all the fine people who help make Miller better (emoji key):
This project follows the all-contributors specification. Contributions of any kind are welcome!