mirror of
https://github.com/johnkerl/miller.git
synced 2026-07-17 16:38:54 +00:00
README.md
This commit is contained in:
John Kerl
parent
f9afae491f
commit
063223e498
1 changed files with 72 additions and 72 deletions
144
README.md
144
README.md
|
|
@ -24,29 +24,6 @@ including but not limited to the familiar **CSV**, **TSV**, and **JSON**/**JSON
|
||||||
In the above image you can see how Miller embraces the common themes of
|
In the above image you can see how Miller embraces the common themes of
|
||||||
key-value-pair data in a variety of data formats.
|
key-value-pair data in a variety of data formats.
|
||||||
|
|
||||||
# Getting started
|
|
||||||
|
|
||||||
[](https://deepwiki.com/johnkerl/miller)
|
|
||||||
|
|
||||||
* [Miller in 10 minutes](https://miller.readthedocs.io/en/latest/10min)
|
|
||||||
* [A Guide To Command-Line Data Manipulation](https://www.smashingmagazine.com/2022/12/guide-command-line-data-manipulation-cli-miller)
|
|
||||||
* [A quick tutorial on Miller](https://www.ict4g.net/adolfo/notes/data-analysis/miller-quick-tutorial.html)
|
|
||||||
* [Miller Exercises](https://github.com/GuilloteauQ/miller-exercises)
|
|
||||||
* [Tools to manipulate CSV files from the Command Line](https://www.ict4g.net/adolfo/notes/data-analysis/tools-to-manipulate-csv.html)
|
|
||||||
* [www.togaware.com/linux/survivor/CSV_Files.html](https://www.togaware.com/linux/survivor/CSV_Files.html)
|
|
||||||
* [MLR for CSV manipulation](https://guillim.github.io/terminal/2018/06/19/MLR-for-CSV-manipulation.html)
|
|
||||||
* [Linux Magazine: Process structured text files with Miller](https://www.linux-magazine.com/Issues/2016/187/Miller)
|
|
||||||
* [Miller: Command Line CSV File Processing](https://onepointzero.app/posts/miller-command-line-csv-file-processing/)
|
|
||||||
* [Miller - A Swiss Army Chainsaw for CSV Data, Data Science and Data Munging](https://fuzzyblog.io/blog/data_science/2022/05/13/miller-a-swiss-army-chainsaw-for-csv-data-data-science-and-data-munging.html)
|
|
||||||
* [Pandas Killer: mlr, the Scientist](https://xvzftube.xyz/posts/pandas_killers/#mlr%3A-the-scientist)
|
|
||||||
|
|
||||||
# More documentation links
|
|
||||||
|
|
||||||
* [**Full documentation**](https://miller.readthedocs.io/)
|
|
||||||
* [Miller's license is two-clause BSD](https://github.com/johnkerl/miller/blob/main/LICENSE.txt)
|
|
||||||
* [Notes about issue-labeling in the Github repo](https://github.com/johnkerl/miller/wiki/Issue-labeling)
|
|
||||||
* [Active issues](https://github.com/johnkerl/miller/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc)
|
|
||||||
|
|
||||||
# Installing
|
# Installing
|
||||||
There's a good chance you can get Miller pre-built for your system:
|
There's a good chance you can get Miller pre-built for your system:
|
||||||
[](https://launchpad.net/ubuntu/+source/miller)
|
[](https://launchpad.net/ubuntu/+source/miller)
|
||||||
|
|
@ -75,6 +52,78 @@ See also [README-versions.md](./README-versions.md) for a full list of package v
|
||||||
|
|
||||||
See also [building from source](https://miller.readthedocs.io/en/latest/build.html).
|
See also [building from source](https://miller.readthedocs.io/en/latest/build.html).
|
||||||
|
|
||||||
|
# Features
|
||||||
|
|
||||||
|
* Miller is **multi-purpose**: it's useful for **data cleaning**,
|
||||||
|
**data reduction**, **statistical reporting**, **devops**, **system
|
||||||
|
administration**, **log-file processing**, **format conversion**, and
|
||||||
|
**database-query post-processing**.
|
||||||
|
|
||||||
|
* You can use Miller to snarf and munge **log-file data**, including selecting
|
||||||
|
out relevant substreams, then produce CSV format and load that into
|
||||||
|
all-in-memory/data-frame utilities for further statistical and/or graphical
|
||||||
|
processing.
|
||||||
|
|
||||||
|
* Miller complements **data-analysis tools** such as **R**, **pandas**, etc.:
|
||||||
|
you can use Miller to **clean** and **prepare** your data. While you can do
|
||||||
|
**basic statistics** entirely in Miller, its streaming-data feature and
|
||||||
|
single-pass algorithms enable you to **reduce very large data sets**.
|
||||||
|
|
||||||
|
* Miller complements SQL **databases**: you can slice, dice, and reformat data
|
||||||
|
on the client side on its way into or out of a database. You can also reap some
|
||||||
|
of the benefits of databases for quick, setup-free one-off tasks when you just
|
||||||
|
need to query some data in disk files in a hurry.
|
||||||
|
|
||||||
|
* Miller also goes beyond the classic Unix tools by stepping fully into our
|
||||||
|
modern, **no-SQL** world: its essential record-heterogeneity property allows
|
||||||
|
Miller to operate on data where records with different schema (field names) are
|
||||||
|
interleaved.
|
||||||
|
|
||||||
|
* Miller is **streaming**: most operations need only a single record in
|
||||||
|
memory at a time, rather than ingesting all input before producing any output.
|
||||||
|
For those operations which require deeper retention (`sort`, `tac`, `stats1`),
|
||||||
|
Miller retains only as much data as needed. This means that whenever
|
||||||
|
functionally possible, you can operate on files which are larger than your
|
||||||
|
system’s available RAM, and you can use Miller in **tail -f** contexts.
|
||||||
|
|
||||||
|
* Miller is **pipe-friendly** and interoperates with the Unix toolkit.
|
||||||
|
|
||||||
|
* Miller's I/O formats include **tabular pretty-printing**, **positionally
|
||||||
|
indexed** (Unix-toolkit style), CSV, TSV, JSON, JSON Lines, and others.
|
||||||
|
|
||||||
|
* Miller does **conversion** between formats.
|
||||||
|
|
||||||
|
* Miller's **processing is format-aware**: e.g. CSV `sort` and `tac` keep header lines first.
|
||||||
|
|
||||||
|
* Miller has high-throughput **performance** on par with the Unix toolkit.
|
||||||
|
|
||||||
|
* Miller is written in portable, modern Go, with **zero runtime dependencies**.
|
||||||
|
You can download or compile a single binary, `scp` it to a faraway machine,
|
||||||
|
and expect it to work.
|
||||||
|
|
||||||
|
# Getting started
|
||||||
|
|
||||||
|
[](https://deepwiki.com/johnkerl/miller)
|
||||||
|
|
||||||
|
* [Miller in 10 minutes](https://miller.readthedocs.io/en/latest/10min)
|
||||||
|
* [A Guide To Command-Line Data Manipulation](https://www.smashingmagazine.com/2022/12/guide-command-line-data-manipulation-cli-miller)
|
||||||
|
* [A quick tutorial on Miller](https://www.ict4g.net/adolfo/notes/data-analysis/miller-quick-tutorial.html)
|
||||||
|
* [Miller Exercises](https://github.com/GuilloteauQ/miller-exercises)
|
||||||
|
* [Tools to manipulate CSV files from the Command Line](https://www.ict4g.net/adolfo/notes/data-analysis/tools-to-manipulate-csv.html)
|
||||||
|
* [www.togaware.com/linux/survivor/CSV_Files.html](https://www.togaware.com/linux/survivor/CSV_Files.html)
|
||||||
|
* [MLR for CSV manipulation](https://guillim.github.io/terminal/2018/06/19/MLR-for-CSV-manipulation.html)
|
||||||
|
* [Linux Magazine: Process structured text files with Miller](https://www.linux-magazine.com/Issues/2016/187/Miller)
|
||||||
|
* [Miller: Command Line CSV File Processing](https://onepointzero.app/posts/miller-command-line-csv-file-processing/)
|
||||||
|
* [Miller - A Swiss Army Chainsaw for CSV Data, Data Science and Data Munging](https://fuzzyblog.io/blog/data_science/2022/05/13/miller-a-swiss-army-chainsaw-for-csv-data-data-science-and-data-munging.html)
|
||||||
|
* [Pandas Killer: mlr, the Scientist](https://xvzftube.xyz/posts/pandas_killers/#mlr%3A-the-scientist)
|
||||||
|
|
||||||
|
# More documentation links
|
||||||
|
|
||||||
|
* [**Full documentation**](https://miller.readthedocs.io/)
|
||||||
|
* [Miller's license is two-clause BSD](https://github.com/johnkerl/miller/blob/main/LICENSE.txt)
|
||||||
|
* [Notes about issue-labeling in the Github repo](https://github.com/johnkerl/miller/wiki/Issue-labeling)
|
||||||
|
* [Active issues](https://github.com/johnkerl/miller/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc)
|
||||||
|
|
||||||
# Community
|
# Community
|
||||||
|
|
||||||
[](https://github.com/johnkerl/miller/stargazers)
|
[](https://github.com/johnkerl/miller/stargazers)
|
||||||
|
|
@ -125,55 +174,6 @@ See also [building from source](https://miller.readthedocs.io/en/latest/build.ht
|
||||||
|
|
||||||
[License: BSD2](https://github.com/johnkerl/miller/blob/main/LICENSE.txt)
|
[License: BSD2](https://github.com/johnkerl/miller/blob/main/LICENSE.txt)
|
||||||
|
|
||||||
# Features
|
|
||||||
|
|
||||||
* Miller is **multi-purpose**: it's useful for **data cleaning**,
|
|
||||||
**data reduction**, **statistical reporting**, **devops**, **system
|
|
||||||
administration**, **log-file processing**, **format conversion**, and
|
|
||||||
**database-query post-processing**.
|
|
||||||
|
|
||||||
* You can use Miller to snarf and munge **log-file data**, including selecting
|
|
||||||
out relevant substreams, then produce CSV format and load that into
|
|
||||||
all-in-memory/data-frame utilities for further statistical and/or graphical
|
|
||||||
processing.
|
|
||||||
|
|
||||||
* Miller complements **data-analysis tools** such as **R**, **pandas**, etc.:
|
|
||||||
you can use Miller to **clean** and **prepare** your data. While you can do
|
|
||||||
**basic statistics** entirely in Miller, its streaming-data feature and
|
|
||||||
single-pass algorithms enable you to **reduce very large data sets**.
|
|
||||||
|
|
||||||
* Miller complements SQL **databases**: you can slice, dice, and reformat data
|
|
||||||
on the client side on its way into or out of a database. You can also reap some
|
|
||||||
of the benefits of databases for quick, setup-free one-off tasks when you just
|
|
||||||
need to query some data in disk files in a hurry.
|
|
||||||
|
|
||||||
* Miller also goes beyond the classic Unix tools by stepping fully into our
|
|
||||||
modern, **no-SQL** world: its essential record-heterogeneity property allows
|
|
||||||
Miller to operate on data where records with different schema (field names) are
|
|
||||||
interleaved.
|
|
||||||
|
|
||||||
* Miller is **streaming**: most operations need only a single record in
|
|
||||||
memory at a time, rather than ingesting all input before producing any output.
|
|
||||||
For those operations which require deeper retention (`sort`, `tac`, `stats1`),
|
|
||||||
Miller retains only as much data as needed. This means that whenever
|
|
||||||
functionally possible, you can operate on files which are larger than your
|
|
||||||
system’s available RAM, and you can use Miller in **tail -f** contexts.
|
|
||||||
|
|
||||||
* Miller is **pipe-friendly** and interoperates with the Unix toolkit.
|
|
||||||
|
|
||||||
* Miller's I/O formats include **tabular pretty-printing**, **positionally
|
|
||||||
indexed** (Unix-toolkit style), CSV, TSV, JSON, JSON Lines, and others.
|
|
||||||
|
|
||||||
* Miller does **conversion** between formats.
|
|
||||||
|
|
||||||
* Miller's **processing is format-aware**: e.g. CSV `sort` and `tac` keep header lines first.
|
|
||||||
|
|
||||||
* Miller has high-throughput **performance** on par with the Unix toolkit.
|
|
||||||
|
|
||||||
* Miller is written in portable, modern Go, with **zero runtime dependencies**.
|
|
||||||
You can download or compile a single binary, `scp` it to a faraway machine,
|
|
||||||
and expect it to work.
|
|
||||||
|
|
||||||
# What people are saying about Miller
|
# What people are saying about Miller
|
||||||
|
|
||||||
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">Today I discovered Miller—it's like jq but for CSV: <a href="https://t.co/pn5Ni241KM">https://t.co/pn5Ni241KM</a><br><br>Also, "Miller complements data-analysis tools such as R, pandas, etc.: you can use Miller to clean and prepare your data." <a href="https://twitter.com/GreatBlueC?ref_src=twsrc%5Etfw">@GreatBlueC</a> <a href="https://twitter.com/nfmcclure?ref_src=twsrc%5Etfw">@nfmcclure</a></p>— Adrien Trouillaud (@adrienjt) <a href="https://twitter.com/adrienjt/status/1308963056592891904?ref_src=twsrc%5Etfw">September 24, 2020</a></blockquote>
|
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">Today I discovered Miller—it's like jq but for CSV: <a href="https://t.co/pn5Ni241KM">https://t.co/pn5Ni241KM</a><br><br>Also, "Miller complements data-analysis tools such as R, pandas, etc.: you can use Miller to clean and prepare your data." <a href="https://twitter.com/GreatBlueC?ref_src=twsrc%5Etfw">@GreatBlueC</a> <a href="https://twitter.com/nfmcclure?ref_src=twsrc%5Etfw">@nfmcclure</a></p>— Adrien Trouillaud (@adrienjt) <a href="https://twitter.com/adrienjt/status/1308963056592891904?ref_src=twsrc%5Etfw">September 24, 2020</a></blockquote>
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue