mirror of
https://github.com/johnkerl/miller.git
synced 2026-07-22 15:37:59 +00:00
Merge remote-tracking branch 'origin/main' into issue-826-csv-upsert-recipe
# Conflicts: # docs/src/questions-about-joins.md # docs/src/questions-about-joins.md.in
This commit is contained in:
John Kerl
commit
32f16df2cb
370 changed files with 4190 additions and 444 deletions
7
.github/dependabot.yml
vendored
7
.github/dependabot.yml
vendored
|
|
@ -10,6 +10,13 @@ updates:
|
|||
directory: "/"
|
||||
schedule:
|
||||
interval: "daily"
|
||||
groups:
|
||||
# The codeql-analysis.yml workflow pins init/autobuild/analyze to the
|
||||
# same commit SHA, so they must be bumped together or CI breaks with a
|
||||
# CodeQL config-version mismatch.
|
||||
codeql-action:
|
||||
patterns:
|
||||
- "github/codeql-action/*"
|
||||
|
||||
# Maintain dependencies for Go
|
||||
- package-ecosystem: "gomod"
|
||||
|
|
|
|||
6
.github/workflows/codeql-analysis.yml
vendored
6
.github/workflows/codeql-analysis.yml
vendored
|
|
@ -40,7 +40,7 @@ jobs:
|
|||
|
||||
# Initializes the CodeQL tools for scanning.
|
||||
- name: Initialize CodeQL
|
||||
uses: github/codeql-action/init@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a
|
||||
uses: github/codeql-action/init@99df26d4f13ea111d4ec1a7dddef6063f76b97e9
|
||||
with:
|
||||
languages: ${{ matrix.language }}
|
||||
# If you wish to specify custom queries, you can do so here or in a config file.
|
||||
|
|
@ -51,7 +51,7 @@ jobs:
|
|||
# Autobuild attempts to build any compiled languages (C/C++, C#, or Java).
|
||||
# If this step fails, then you should remove it and run the build manually (see below)
|
||||
- name: Autobuild
|
||||
uses: github/codeql-action/autobuild@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a
|
||||
uses: github/codeql-action/autobuild@99df26d4f13ea111d4ec1a7dddef6063f76b97e9
|
||||
|
||||
# ℹ️ Command-line programs to run using the OS shell.
|
||||
# 📚 https://git.io/JvXDl
|
||||
|
|
@ -65,4 +65,4 @@ jobs:
|
|||
# make release
|
||||
|
||||
- name: Perform CodeQL Analysis
|
||||
uses: github/codeql-action/analyze@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a
|
||||
uses: github/codeql-action/analyze@99df26d4f13ea111d4ec1a7dddef6063f76b97e9
|
||||
|
|
|
|||
1
.gitignore
vendored
1
.gitignore
vendored
|
|
@ -17,6 +17,7 @@ data/nmc?.*
|
|||
|
||||
docs/src/polyglot-dkvp-io/__pycache__
|
||||
docs/site/
|
||||
docs/miller-docs.epub
|
||||
|
||||
.cursor
|
||||
.claude
|
||||
|
|
|
|||
|
|
@ -10,6 +10,17 @@ build:
|
|||
os: ubuntu-24.04
|
||||
tools:
|
||||
python: "3.12"
|
||||
apt_packages:
|
||||
- pandoc
|
||||
jobs:
|
||||
# Read the Docs' built-in `formats:` support (below) only produces
|
||||
# epub/PDF/htmlzip for Sphinx projects, not MkDocs ones. So for issue
|
||||
# #1835 we build the epub ourselves after the MkDocs HTML build; anything
|
||||
# placed in $READTHEDOCS_OUTPUT/epub/ is published on the project's
|
||||
# Downloads page and in the docs' flyout menu.
|
||||
post_build:
|
||||
- mkdir -p $READTHEDOCS_OUTPUT/epub
|
||||
- bash docs/build-epub.sh $READTHEDOCS_OUTPUT/epub/miller.epub
|
||||
|
||||
python:
|
||||
install:
|
||||
|
|
|
|||
36
CLAUDE.md
36
CLAUDE.md
|
|
@ -28,6 +28,24 @@ staticcheck -version
|
|||
|
||||
If this works, you can use `make staticcheck` without any further setup.
|
||||
|
||||
### Setting Up golangci-lint
|
||||
|
||||
CI runs `golangci-lint` (config in `.golangci.yml`) on every push and PR via
|
||||
`.github/workflows/golangci-lint.yml`. To run it locally, install the version
|
||||
matching CI (currently v2.12.2) per the
|
||||
[official instructions](https://golangci-lint.run/welcome/install/#local-installation), e.g.:
|
||||
|
||||
```bash
|
||||
curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(go env GOPATH)/bin v2.12.2
|
||||
```
|
||||
|
||||
This also installs to `~/go/bin/`, so the same `PATH` addition above covers it.
|
||||
|
||||
**Verify the installation:**
|
||||
```bash
|
||||
golangci-lint --version
|
||||
```
|
||||
|
||||
## Build & Test
|
||||
|
||||
### Building
|
||||
|
|
@ -48,8 +66,23 @@ make bench # Run benchmarks
|
|||
```bash
|
||||
make fmt # Format code with go fmt
|
||||
make staticcheck # Run static analysis (see Initial Setup section above)
|
||||
make lint # Run golangci-lint, same invocation as CI (see Initial Setup section above)
|
||||
```
|
||||
|
||||
`make lint` runs `golangci-lint run ./cmd/mlr ./pkg/...`, matching
|
||||
`.github/workflows/golangci-lint.yml` exactly, so a clean local run usually
|
||||
means CI's lint job will pass too. It is not part of `make check` or
|
||||
`make dev`, so run it explicitly before pushing.
|
||||
|
||||
**Note:** CI's lint job restores a persistent `golangci-lint` results cache
|
||||
shared across commits and PRs (via `golangci-lint-action`), which can
|
||||
occasionally serve stale `staticcheck` results (e.g. spurious `SA5011`
|
||||
nil-check warnings) unrelated to your diff. The job is `continue-on-error:
|
||||
true` for this reason, so it won't block merging. If CI lint fails but
|
||||
`make lint` is clean locally, it's likely a stale cache — rerun the job, or a
|
||||
maintainer can clear it: `gh cache list --key golangci-lint` then
|
||||
`gh cache delete <id>`.
|
||||
|
||||
### Full Developer Workflow
|
||||
```bash
|
||||
make dev # Format, build, test, generate docs (comprehensive check before pushing)
|
||||
|
|
@ -125,9 +158,10 @@ Documentation is built from `.md.in` template files that contain live code sampl
|
|||
Always run:
|
||||
```bash
|
||||
make dev
|
||||
make lint
|
||||
```
|
||||
|
||||
This ensures code formatting, builds successfully, passes all tests, and documentation is up to date.
|
||||
`make dev` ensures code formatting, builds successfully, passes all tests, and documentation is up to date. `make lint` is separate (not part of `make dev`/`make check`) and mirrors the CI lint job.
|
||||
|
||||
## Additional Resources
|
||||
|
||||
|
|
|
|||
7
Makefile
7
Makefile
|
|
@ -79,6 +79,11 @@ fmt format:
|
|||
staticcheck:
|
||||
staticcheck ./pkg/... ./cmd/mlr/...
|
||||
|
||||
# Needs first: https://golangci-lint.run/welcome/install/#local-installation
|
||||
# Config lives in .golangci.yml; same invocation as .github/workflows/golangci-lint.yml
|
||||
lint:
|
||||
golangci-lint run ./cmd/mlr ./pkg/...
|
||||
|
||||
# ----------------------------------------------------------------
|
||||
# For developers before pushing to GitHub.
|
||||
#
|
||||
|
|
@ -125,4 +130,4 @@ release_tarball release-tarball: build check
|
|||
|
||||
# ================================================================
|
||||
# Go does its own dependency management, outside of make.
|
||||
.PHONY: build mlr check unit_test regression_test bench fmt staticcheck dev docs man
|
||||
.PHONY: build mlr check unit_test regression_test bench fmt staticcheck lint dev docs man
|
||||
|
|
|
|||
|
|
@ -1,6 +1,7 @@
|
|||
# Quickstart for developers
|
||||
|
||||
* `make`, `make check`, `make docs`, etc: see [Makefile](Makefile) in the repo base directory.
|
||||
* Linting: `make lint` runs [golangci-lint](https://golangci-lint.run/) (config in [.golangci.yml](.golangci.yml)) with the same invocation used by [.github/workflows/golangci-lint.yml](.github/workflows/golangci-lint.yml), so a clean local run usually means CI's lint job will pass too. Install golangci-lint locally per the [official instructions](https://golangci-lint.run/welcome/install/#local-installation) (match the CI version, currently v2.12.2). It's a separate step from `make dev`/`make check` -- run it explicitly before pushing. `make staticcheck` (requires [staticcheck](https://staticcheck.io)) is a second, independent static-analysis pass. Note: CI's lint job caches `staticcheck` results across commits/PRs and can occasionally fail on a stale cache (spurious warnings unrelated to your diff) even when `make lint` is clean locally; the job is `continue-on-error: true` so this won't block merging, and a maintainer can clear the cache with `gh cache delete` if needed.
|
||||
* Software-testing methodology: see [./test/README.md](./test/README.md).
|
||||
* Source-code indexing: please see [https://sourcegraph.com/github.com/johnkerl/miller](https://sourcegraph.com/github.com/johnkerl/miller)
|
||||
* Godoc As of September 2021, `godoc` support is minimal: package-level synopses exist; most `func`/`const`/etc content lacks `godoc`-style comments. To view doc material, you can:
|
||||
|
|
|
|||
144
README.md
144
README.md
|
|
@ -24,29 +24,6 @@ including but not limited to the familiar **CSV**, **TSV**, and **JSON**/**JSON
|
|||
In the above image you can see how Miller embraces the common themes of
|
||||
key-value-pair data in a variety of data formats.
|
||||
|
||||
# Getting started
|
||||
|
||||
[](https://deepwiki.com/johnkerl/miller)
|
||||
|
||||
* [Miller in 10 minutes](https://miller.readthedocs.io/en/latest/10min)
|
||||
* [A Guide To Command-Line Data Manipulation](https://www.smashingmagazine.com/2022/12/guide-command-line-data-manipulation-cli-miller)
|
||||
* [A quick tutorial on Miller](https://www.ict4g.net/adolfo/notes/data-analysis/miller-quick-tutorial.html)
|
||||
* [Miller Exercises](https://github.com/GuilloteauQ/miller-exercises)
|
||||
* [Tools to manipulate CSV files from the Command Line](https://www.ict4g.net/adolfo/notes/data-analysis/tools-to-manipulate-csv.html)
|
||||
* [www.togaware.com/linux/survivor/CSV_Files.html](https://www.togaware.com/linux/survivor/CSV_Files.html)
|
||||
* [MLR for CSV manipulation](https://guillim.github.io/terminal/2018/06/19/MLR-for-CSV-manipulation.html)
|
||||
* [Linux Magazine: Process structured text files with Miller](https://www.linux-magazine.com/Issues/2016/187/Miller)
|
||||
* [Miller: Command Line CSV File Processing](https://onepointzero.app/posts/miller-command-line-csv-file-processing/)
|
||||
* [Miller - A Swiss Army Chainsaw for CSV Data, Data Science and Data Munging](https://fuzzyblog.io/blog/data_science/2022/05/13/miller-a-swiss-army-chainsaw-for-csv-data-data-science-and-data-munging.html)
|
||||
* [Pandas Killer: mlr, the Scientist](https://xvzftube.xyz/posts/pandas_killers/#mlr%3A-the-scientist)
|
||||
|
||||
# More documentation links
|
||||
|
||||
* [**Full documentation**](https://miller.readthedocs.io/)
|
||||
* [Miller's license is two-clause BSD](https://github.com/johnkerl/miller/blob/main/LICENSE.txt)
|
||||
* [Notes about issue-labeling in the Github repo](https://github.com/johnkerl/miller/wiki/Issue-labeling)
|
||||
* [Active issues](https://github.com/johnkerl/miller/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc)
|
||||
|
||||
# Installing
|
||||
There's a good chance you can get Miller pre-built for your system:
|
||||
[](https://launchpad.net/ubuntu/+source/miller)
|
||||
|
|
@ -75,6 +52,78 @@ See also [README-versions.md](./README-versions.md) for a full list of package v
|
|||
|
||||
See also [building from source](https://miller.readthedocs.io/en/latest/build.html).
|
||||
|
||||
# Features
|
||||
|
||||
* Miller is **multi-purpose**: it's useful for **data cleaning**,
|
||||
**data reduction**, **statistical reporting**, **devops**, **system
|
||||
administration**, **log-file processing**, **format conversion**, and
|
||||
**database-query post-processing**.
|
||||
|
||||
* You can use Miller to snarf and munge **log-file data**, including selecting
|
||||
out relevant substreams, then produce CSV format and load that into
|
||||
all-in-memory/data-frame utilities for further statistical and/or graphical
|
||||
processing.
|
||||
|
||||
* Miller complements **data-analysis tools** such as **R**, **pandas**, etc.:
|
||||
you can use Miller to **clean** and **prepare** your data. While you can do
|
||||
**basic statistics** entirely in Miller, its streaming-data feature and
|
||||
single-pass algorithms enable you to **reduce very large data sets**.
|
||||
|
||||
* Miller complements SQL **databases**: you can slice, dice, and reformat data
|
||||
on the client side on its way into or out of a database. You can also reap some
|
||||
of the benefits of databases for quick, setup-free one-off tasks when you just
|
||||
need to query some data in disk files in a hurry.
|
||||
|
||||
* Miller also goes beyond the classic Unix tools by stepping fully into our
|
||||
modern, **no-SQL** world: its essential record-heterogeneity property allows
|
||||
Miller to operate on data where records with different schema (field names) are
|
||||
interleaved.
|
||||
|
||||
* Miller is **streaming**: most operations need only a single record in
|
||||
memory at a time, rather than ingesting all input before producing any output.
|
||||
For those operations which require deeper retention (`sort`, `tac`, `stats1`),
|
||||
Miller retains only as much data as needed. This means that whenever
|
||||
functionally possible, you can operate on files which are larger than your
|
||||
system’s available RAM, and you can use Miller in **tail -f** contexts.
|
||||
|
||||
* Miller is **pipe-friendly** and interoperates with the Unix toolkit.
|
||||
|
||||
* Miller's I/O formats include **tabular pretty-printing**, **positionally
|
||||
indexed** (Unix-toolkit style), CSV, TSV, JSON, JSON Lines, and others.
|
||||
|
||||
* Miller does **conversion** between formats.
|
||||
|
||||
* Miller's **processing is format-aware**: e.g. CSV `sort` and `tac` keep header lines first.
|
||||
|
||||
* Miller has high-throughput **performance** on par with the Unix toolkit.
|
||||
|
||||
* Miller is written in portable, modern Go, with **zero runtime dependencies**.
|
||||
You can download or compile a single binary, `scp` it to a faraway machine,
|
||||
and expect it to work.
|
||||
|
||||
# Getting started
|
||||
|
||||
[](https://deepwiki.com/johnkerl/miller)
|
||||
|
||||
* [Miller in 10 minutes](https://miller.readthedocs.io/en/latest/10min)
|
||||
* [A Guide To Command-Line Data Manipulation](https://www.smashingmagazine.com/2022/12/guide-command-line-data-manipulation-cli-miller)
|
||||
* [A quick tutorial on Miller](https://www.ict4g.net/adolfo/notes/data-analysis/miller-quick-tutorial.html)
|
||||
* [Miller Exercises](https://github.com/GuilloteauQ/miller-exercises)
|
||||
* [Tools to manipulate CSV files from the Command Line](https://www.ict4g.net/adolfo/notes/data-analysis/tools-to-manipulate-csv.html)
|
||||
* [www.togaware.com/linux/survivor/CSV_Files.html](https://www.togaware.com/linux/survivor/CSV_Files.html)
|
||||
* [MLR for CSV manipulation](https://guillim.github.io/terminal/2018/06/19/MLR-for-CSV-manipulation.html)
|
||||
* [Linux Magazine: Process structured text files with Miller](https://www.linux-magazine.com/Issues/2016/187/Miller)
|
||||
* [Miller: Command Line CSV File Processing](https://onepointzero.app/posts/miller-command-line-csv-file-processing/)
|
||||
* [Miller - A Swiss Army Chainsaw for CSV Data, Data Science and Data Munging](https://fuzzyblog.io/blog/data_science/2022/05/13/miller-a-swiss-army-chainsaw-for-csv-data-data-science-and-data-munging.html)
|
||||
* [Pandas Killer: mlr, the Scientist](https://xvzftube.xyz/posts/pandas_killers/#mlr%3A-the-scientist)
|
||||
|
||||
# More documentation links
|
||||
|
||||
* [**Full documentation**](https://miller.readthedocs.io/)
|
||||
* [Miller's license is two-clause BSD](https://github.com/johnkerl/miller/blob/main/LICENSE.txt)
|
||||
* [Notes about issue-labeling in the Github repo](https://github.com/johnkerl/miller/wiki/Issue-labeling)
|
||||
* [Active issues](https://github.com/johnkerl/miller/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc)
|
||||
|
||||
# Community
|
||||
|
||||
[](https://github.com/johnkerl/miller/stargazers)
|
||||
|
|
@ -125,55 +174,6 @@ See also [building from source](https://miller.readthedocs.io/en/latest/build.ht
|
|||
|
||||
[License: BSD2](https://github.com/johnkerl/miller/blob/main/LICENSE.txt)
|
||||
|
||||
# Features
|
||||
|
||||
* Miller is **multi-purpose**: it's useful for **data cleaning**,
|
||||
**data reduction**, **statistical reporting**, **devops**, **system
|
||||
administration**, **log-file processing**, **format conversion**, and
|
||||
**database-query post-processing**.
|
||||
|
||||
* You can use Miller to snarf and munge **log-file data**, including selecting
|
||||
out relevant substreams, then produce CSV format and load that into
|
||||
all-in-memory/data-frame utilities for further statistical and/or graphical
|
||||
processing.
|
||||
|
||||
* Miller complements **data-analysis tools** such as **R**, **pandas**, etc.:
|
||||
you can use Miller to **clean** and **prepare** your data. While you can do
|
||||
**basic statistics** entirely in Miller, its streaming-data feature and
|
||||
single-pass algorithms enable you to **reduce very large data sets**.
|
||||
|
||||
* Miller complements SQL **databases**: you can slice, dice, and reformat data
|
||||
on the client side on its way into or out of a database. You can also reap some
|
||||
of the benefits of databases for quick, setup-free one-off tasks when you just
|
||||
need to query some data in disk files in a hurry.
|
||||
|
||||
* Miller also goes beyond the classic Unix tools by stepping fully into our
|
||||
modern, **no-SQL** world: its essential record-heterogeneity property allows
|
||||
Miller to operate on data where records with different schema (field names) are
|
||||
interleaved.
|
||||
|
||||
* Miller is **streaming**: most operations need only a single record in
|
||||
memory at a time, rather than ingesting all input before producing any output.
|
||||
For those operations which require deeper retention (`sort`, `tac`, `stats1`),
|
||||
Miller retains only as much data as needed. This means that whenever
|
||||
functionally possible, you can operate on files which are larger than your
|
||||
system’s available RAM, and you can use Miller in **tail -f** contexts.
|
||||
|
||||
* Miller is **pipe-friendly** and interoperates with the Unix toolkit.
|
||||
|
||||
* Miller's I/O formats include **tabular pretty-printing**, **positionally
|
||||
indexed** (Unix-toolkit style), CSV, TSV, JSON, JSON Lines, and others.
|
||||
|
||||
* Miller does **conversion** between formats.
|
||||
|
||||
* Miller's **processing is format-aware**: e.g. CSV `sort` and `tac` keep header lines first.
|
||||
|
||||
* Miller has high-throughput **performance** on par with the Unix toolkit.
|
||||
|
||||
* Miller is written in portable, modern Go, with **zero runtime dependencies**.
|
||||
You can download or compile a single binary, `scp` it to a faraway machine,
|
||||
and expect it to work.
|
||||
|
||||
# What people are saying about Miller
|
||||
|
||||
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">Today I discovered Miller—it's like jq but for CSV: <a href="https://t.co/pn5Ni241KM">https://t.co/pn5Ni241KM</a><br><br>Also, "Miller complements data-analysis tools such as R, pandas, etc.: you can use Miller to clean and prepare your data." <a href="https://twitter.com/GreatBlueC?ref_src=twsrc%5Etfw">@GreatBlueC</a> <a href="https://twitter.com/nfmcclure?ref_src=twsrc%5Etfw">@nfmcclure</a></p>— Adrien Trouillaud (@adrienjt) <a href="https://twitter.com/adrienjt/status/1308963056592891904?ref_src=twsrc%5Etfw">September 24, 2020</a></blockquote>
|
||||
|
|
|
|||
|
|
@ -3,4 +3,10 @@ build:
|
|||
make -C src genmds
|
||||
mkdocs build
|
||||
|
||||
.PHONY: build
|
||||
# Optional: build a single-file epub of the documentation for offline
|
||||
# reading. Requires pandoc. Not part of the default build; Read the Docs runs
|
||||
# this via .readthedocs.yaml (see docs/build-epub.sh for details).
|
||||
epub:
|
||||
./build-epub.sh
|
||||
|
||||
.PHONY: build epub
|
||||
|
|
|
|||
74
docs/build-epub.sh
Executable file
74
docs/build-epub.sh
Executable file
|
|
@ -0,0 +1,74 @@
|
|||
#!/bin/bash
|
||||
|
||||
# ================================================================
|
||||
# Builds a single-file epub of the Miller documentation from the generated
|
||||
# Markdown files in docs/src, for offline reading (issue #1835).
|
||||
#
|
||||
# Usage: docs/build-epub.sh [output-path]
|
||||
# Default output is ./miller-docs.epub.
|
||||
#
|
||||
# Requires pandoc (https://pandoc.org). This script is invoked on Read the
|
||||
# Docs (see .readthedocs.yaml) to publish an epub download alongside the HTML
|
||||
# docs; Read the Docs' built-in `formats:` support only produces epub/PDF for
|
||||
# Sphinx projects, not MkDocs ones, hence this script. It can also be run
|
||||
# locally by anyone with pandoc installed. It is not part of `make dev` or
|
||||
# any other default build path.
|
||||
# ================================================================
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
output="${1:-miller-docs.epub}"
|
||||
# Make the output path absolute, since we cd around below.
|
||||
case "$output" in
|
||||
/*) ;;
|
||||
*) output="$PWD/$output" ;;
|
||||
esac
|
||||
|
||||
docs_dir=$(cd "$(dirname "$0")" && pwd)
|
||||
src_dir="$docs_dir/src"
|
||||
|
||||
if ! command -v pandoc > /dev/null 2>&1; then
|
||||
echo "$0: pandoc not found; please install it (https://pandoc.org)." 1>&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Chapter ordering is the nav order in mkdocs.yml. Nav entries look like
|
||||
# - "Miller in 10 minutes": "10min.md"
|
||||
# and these are the only lines in mkdocs.yml ending with a quoted .md name.
|
||||
chapters=$(sed -n 's/^ *-.*: *"\(.*\.md\)" *$/\1/p' "$docs_dir/mkdocs.yml")
|
||||
if [ -z "$chapters" ]; then
|
||||
echo "$0: could not extract any nav entries from $docs_dir/mkdocs.yml." 1>&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
tmp_dir=$(mktemp -d)
|
||||
trap 'rm -rf "$tmp_dir"' EXIT
|
||||
|
||||
# Each generated page starts with a quicklinks navigation block -- raw HTML,
|
||||
# useful on the website but not in an epub -- which we strip here. It is the
|
||||
# only <div>...</div> pair at column zero in each page.
|
||||
inputs=()
|
||||
for chapter in $chapters; do
|
||||
if [ ! -f "$src_dir/$chapter" ]; then
|
||||
echo "$0: $src_dir/$chapter is listed in mkdocs.yml nav but does not exist." 1>&2
|
||||
echo "$0: perhaps you need to run: make -C $src_dir genmds" 1>&2
|
||||
exit 1
|
||||
fi
|
||||
sed '/^<div>$/,/^<\/div>$/d' "$src_dir/$chapter" > "$tmp_dir/$chapter"
|
||||
inputs+=("$tmp_dir/$chapter")
|
||||
done
|
||||
|
||||
# Run from src_dir so relative image paths (pix/*.png) resolve.
|
||||
cd "$src_dir"
|
||||
pandoc \
|
||||
--toc \
|
||||
--toc-depth=2 \
|
||||
--split-level=1 \
|
||||
--resource-path="$src_dir" \
|
||||
--metadata title="Miller Documentation" \
|
||||
--metadata author="John Kerl" \
|
||||
--metadata lang="en" \
|
||||
--output "$output" \
|
||||
"${inputs[@]}"
|
||||
|
||||
echo "Wrote $output"
|
||||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
@ -64,7 +64,7 @@ one-line summary (here trimmed, and then counted, using Miller itself):
|
|||
<pre class="pre-non-highlight-in-pair">
|
||||
[
|
||||
{
|
||||
"count": 661
|
||||
"count": 665
|
||||
}
|
||||
]
|
||||
</pre>
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
3
docs/src/data/het/sales-2021.csv
Normal file
3
docs/src/data/het/sales-2021.csv
Normal file
|
|
@ -0,0 +1,3 @@
|
|||
id,product,sales
|
||||
1,pencil,100
|
||||
2,eraser,17
|
||||
|
3
docs/src/data/het/sales-2022.csv
Normal file
3
docs/src/data/het/sales-2022.csv
Normal file
|
|
@ -0,0 +1,3 @@
|
|||
id,product,color,sales
|
||||
3,pencil,red,120
|
||||
4,eraser,green,32
|
||||
|
3
docs/src/data/het/sales-2023.csv
Normal file
3
docs/src/data/het/sales-2023.csv
Normal file
|
|
@ -0,0 +1,3 @@
|
|||
id,product,sales,returns
|
||||
5,pencil,200,6
|
||||
6,notebook,41,2
|
||||
|
3
docs/src/data/join-nest-left.csv
Normal file
3
docs/src/data/join-nest-left.csv
Normal file
|
|
@ -0,0 +1,3 @@
|
|||
id,color
|
||||
1;2,blue
|
||||
3,green
|
||||
|
3
docs/src/data/join-nest-right.csv
Normal file
3
docs/src/data/join-nest-right.csv
Normal file
|
|
@ -0,0 +1,3 @@
|
|||
id,shape
|
||||
1,circle
|
||||
2;3,square
|
||||
|
4
docs/src/data/join-x.csv
Normal file
4
docs/src/data/join-x.csv
Normal file
|
|
@ -0,0 +1,4 @@
|
|||
a,b,c
|
||||
a,t,1
|
||||
b,u,2
|
||||
c,v,3
|
||||
|
4
docs/src/data/join-y.csv
Normal file
4
docs/src/data/join-y.csv
Normal file
|
|
@ -0,0 +1,4 @@
|
|||
e,f,g
|
||||
a,t,3
|
||||
b,u,2
|
||||
d,w,1
|
||||
|
3
docs/src/data/lazy-quotes.csv
Normal file
3
docs/src/data/lazy-quotes.csv
Normal file
|
|
@ -0,0 +1,3 @@
|
|||
id,name,flag,city
|
||||
1,"ACME CORP. INC,Q8,Rome
|
||||
2,BETA,X,Milan
|
||||
|
Can't render this file because it contains an unexpected character in line 3 and column 16.
|
8
docs/src/data/rank-example.csv
Normal file
8
docs/src/data/rank-example.csv
Normal file
|
|
@ -0,0 +1,8 @@
|
|||
g,x
|
||||
a,10
|
||||
a,20
|
||||
a,20
|
||||
a,30
|
||||
b,5
|
||||
b,5
|
||||
b,9
|
||||
|
3
docs/src/data/sensor-humidity.csv
Normal file
3
docs/src/data/sensor-humidity.csv
Normal file
|
|
@ -0,0 +1,3 @@
|
|||
unixTime,minValue,averageValue,maxValue
|
||||
1740000000,50.1,50.3,50.5
|
||||
1740000120,52.3,52.5,52.7
|
||||
|
4
docs/src/data/sensor-pressure.csv
Normal file
4
docs/src/data/sensor-pressure.csv
Normal file
|
|
@ -0,0 +1,4 @@
|
|||
unixTime,minValue,averageValue,maxValue
|
||||
1740000000,1012.2,1012.3,1012.4
|
||||
1740000060,1011.8,1011.9,1012.0
|
||||
1740000120,1011.5,1011.6,1011.7
|
||||
|
4
docs/src/data/sensor-temp.csv
Normal file
4
docs/src/data/sensor-temp.csv
Normal file
|
|
@ -0,0 +1,4 @@
|
|||
unixTime,minValue,averageValue,maxValue
|
||||
1740000000,37.2,37.4,37.6
|
||||
1740000060,37.5,37.6,37.7
|
||||
1740000120,37.8,37.9,38.0
|
||||
|
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
@ -237,6 +237,79 @@ CSV, TSV, CSV-lite, and TSV-lite have in common the `--implicit-csv-header` flag
|
|||
|
||||
See also the [`--lazy-quotes` flag](reference-main-flag-list.md#csv-only-flags), which can help with CSV files that are not fully compliant with RFC-4180.
|
||||
|
||||
### Handling stray quote characters
|
||||
|
||||
The [`--lazy-quotes` flag](reference-main-flag-list.md#csv-only-flags) makes two specific
|
||||
relaxations to RFC-4180 parsing (following the semantics of the
|
||||
[Go CSV library](https://pkg.go.dev/encoding/csv)): a quote may appear inside an *unquoted*
|
||||
field, and a non-doubled quote may appear inside a *quoted* field.
|
||||
|
||||
What it does **not** change is how quoted fields are delimited. A field whose first character
|
||||
is a double quote is still a quoted field, and its contents extend -- across field separators
|
||||
and even line endings -- until the next double quote. In particular, if a field has an
|
||||
unmatched opening quote, everything up to the next quote character in the file (or the end
|
||||
of the file) becomes part of that field. This matches the behavior of the Go CSV library
|
||||
with `LazyQuotes`, as well as Python's `csv` module. For example:
|
||||
|
||||
<pre class="pre-highlight-in-pair">
|
||||
<b>cat data/lazy-quotes.csv</b>
|
||||
</pre>
|
||||
<pre class="pre-non-highlight-in-pair">
|
||||
id,name,flag,city
|
||||
1,"ACME CORP. INC,Q8,Rome
|
||||
2,BETA,X,Milan
|
||||
</pre>
|
||||
|
||||
Here the second field of the first data line has an opening quote with no matching close,
|
||||
so without `--lazy-quotes` we get an error:
|
||||
|
||||
<pre class="pre-highlight-in-pair">
|
||||
<b>mlr --icsv --ojson cat data/lazy-quotes.csv</b>
|
||||
</pre>
|
||||
<pre class="pre-non-highlight-in-pair">
|
||||
mlr: CSV header/data length mismatch 4 != 1 at filename data/lazy-quotes.csv row 2
|
||||
</pre>
|
||||
|
||||
With `--lazy-quotes`, the quoted field silently absorbs the rest of the file -- including
|
||||
the field separators and the newline -- since there is no closing quote anywhere:
|
||||
|
||||
<pre class="pre-highlight-in-pair">
|
||||
<b>mlr --icsv --ojson --lazy-quotes --allow-ragged-csv-input cat data/lazy-quotes.csv</b>
|
||||
</pre>
|
||||
<pre class="pre-non-highlight-in-pair">
|
||||
[
|
||||
{
|
||||
"id": 1,
|
||||
"name": "ACME CORP. INC,Q8,Rome\n2,BETA,X,Milan\n"
|
||||
}
|
||||
]
|
||||
</pre>
|
||||
|
||||
If quote characters in your data are really just ordinary data characters -- that is, the
|
||||
file doesn't use RFC-4180-style quoting at all -- then [CSV-lite](file-formats.md#csvtsvasvusvetc)
|
||||
is often a better choice, since it splits on the field separator without treating quotes
|
||||
specially (the literal quote character is retained in the data):
|
||||
|
||||
<pre class="pre-highlight-in-pair">
|
||||
<b>mlr --icsvlite --ojson cat data/lazy-quotes.csv</b>
|
||||
</pre>
|
||||
<pre class="pre-non-highlight-in-pair">
|
||||
[
|
||||
{
|
||||
"id": 1,
|
||||
"name": "\"ACME CORP. INC",
|
||||
"flag": "Q8",
|
||||
"city": "Rome"
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"name": "BETA",
|
||||
"flag": "X",
|
||||
"city": "Milan"
|
||||
}
|
||||
]
|
||||
</pre>
|
||||
|
||||
### Troubleshooting CSV and JSON input
|
||||
|
||||
Please see [this page](troubleshooting-csv-and-json-input.md).
|
||||
|
|
@ -531,6 +604,47 @@ Since Miller 6.11.0, you can use `--barred-input` with pprint input format:
|
|||
]
|
||||
</pre>
|
||||
|
||||
Use `--right` to right-align all cells, or `--right-align-numeric` to right-align only the cells
|
||||
having numeric values, leaving other cells left-aligned. Headers are right-aligned over columns
|
||||
whose values are all numeric, so that header and data share the same alignment:
|
||||
|
||||
<pre class="pre-highlight-in-pair">
|
||||
<b>mlr --icsv --opprint --right-align-numeric cat example.csv</b>
|
||||
</pre>
|
||||
<pre class="pre-non-highlight-in-pair">
|
||||
color shape flag k index quantity rate
|
||||
yellow triangle true 1 11 43.6498 9.8870
|
||||
red square true 2 15 79.2778 0.0130
|
||||
red circle true 3 16 13.8103 2.9010
|
||||
red square false 4 48 77.5542 7.4670
|
||||
purple triangle false 5 51 81.2290 8.5910
|
||||
red square false 6 64 77.1991 9.5310
|
||||
purple triangle false 7 65 80.1405 5.8240
|
||||
yellow circle true 8 73 63.9785 4.2370
|
||||
yellow circle true 9 87 63.5058 8.3350
|
||||
purple square false 10 91 72.3735 8.2430
|
||||
</pre>
|
||||
|
||||
<pre class="pre-highlight-in-pair">
|
||||
<b>mlr --icsv --opprint --barred --right-align-numeric cat example.csv</b>
|
||||
</pre>
|
||||
<pre class="pre-non-highlight-in-pair">
|
||||
+--------+----------+-------+----+-------+----------+--------+
|
||||
| color | shape | flag | k | index | quantity | rate |
|
||||
+--------+----------+-------+----+-------+----------+--------+
|
||||
| yellow | triangle | true | 1 | 11 | 43.6498 | 9.8870 |
|
||||
| red | square | true | 2 | 15 | 79.2778 | 0.0130 |
|
||||
| red | circle | true | 3 | 16 | 13.8103 | 2.9010 |
|
||||
| red | square | false | 4 | 48 | 77.5542 | 7.4670 |
|
||||
| purple | triangle | false | 5 | 51 | 81.2290 | 8.5910 |
|
||||
| red | square | false | 6 | 64 | 77.1991 | 9.5310 |
|
||||
| purple | triangle | false | 7 | 65 | 80.1405 | 5.8240 |
|
||||
| yellow | circle | true | 8 | 73 | 63.9785 | 4.2370 |
|
||||
| yellow | circle | true | 9 | 87 | 63.5058 | 8.3350 |
|
||||
| purple | square | false | 10 | 91 | 72.3735 | 8.2430 |
|
||||
+--------+----------+-------+----+-------+----------+--------+
|
||||
</pre>
|
||||
|
||||
## Markdown tabular
|
||||
|
||||
Markdown format looks like this:
|
||||
|
|
@ -580,6 +694,31 @@ do not need to pass `--md` in addition:
|
|||
<b>mlr --md-aligned cat data/small</b>
|
||||
</pre>
|
||||
|
||||
The `--right-align-numeric` flag also applies to markdown output: numeric columns get a
|
||||
right-alignment marker (`---:`) in the header-separator line, so they render right-aligned in
|
||||
Markdown viewers. With `--omd`, since output is streaming, the marker for each column is chosen
|
||||
from the first record of each same-schema group; with `--omd-aligned`, a column gets the marker
|
||||
when all its values are numeric, and its header and cell text are right-justified in the raw
|
||||
markdown as well:
|
||||
|
||||
<pre class="pre-highlight-in-pair">
|
||||
<b>mlr --icsv --omd-aligned --right-align-numeric cat example.csv</b>
|
||||
</pre>
|
||||
<pre class="pre-non-highlight-in-pair">
|
||||
| color | shape | flag | k | index | quantity | rate |
|
||||
| --- | --- | --- | ---: | ---: | ---: | ---: |
|
||||
| yellow | triangle | true | 1 | 11 | 43.6498 | 9.8870 |
|
||||
| red | square | true | 2 | 15 | 79.2778 | 0.0130 |
|
||||
| red | circle | true | 3 | 16 | 13.8103 | 2.9010 |
|
||||
| red | square | false | 4 | 48 | 77.5542 | 7.4670 |
|
||||
| purple | triangle | false | 5 | 51 | 81.2290 | 8.5910 |
|
||||
| red | square | false | 6 | 64 | 77.1991 | 9.5310 |
|
||||
| purple | triangle | false | 7 | 65 | 80.1405 | 5.8240 |
|
||||
| yellow | circle | true | 8 | 73 | 63.9785 | 4.2370 |
|
||||
| yellow | circle | true | 9 | 87 | 63.5058 | 8.3350 |
|
||||
| purple | square | false | 10 | 91 | 72.3735 | 8.2430 |
|
||||
</pre>
|
||||
|
||||
## XTAB: Vertical tabular
|
||||
|
||||
This is perhaps most useful for looking a very wide and/or multi-column data which causes line-wraps on the screen (but see also
|
||||
|
|
|
|||
|
|
@ -79,6 +79,47 @@ CSV, TSV, CSV-lite, and TSV-lite have in common the `--implicit-csv-header` flag
|
|||
|
||||
See also the [`--lazy-quotes` flag](reference-main-flag-list.md#csv-only-flags), which can help with CSV files that are not fully compliant with RFC-4180.
|
||||
|
||||
### Handling stray quote characters
|
||||
|
||||
The [`--lazy-quotes` flag](reference-main-flag-list.md#csv-only-flags) makes two specific
|
||||
relaxations to RFC-4180 parsing (following the semantics of the
|
||||
[Go CSV library](https://pkg.go.dev/encoding/csv)): a quote may appear inside an *unquoted*
|
||||
field, and a non-doubled quote may appear inside a *quoted* field.
|
||||
|
||||
What it does **not** change is how quoted fields are delimited. A field whose first character
|
||||
is a double quote is still a quoted field, and its contents extend -- across field separators
|
||||
and even line endings -- until the next double quote. In particular, if a field has an
|
||||
unmatched opening quote, everything up to the next quote character in the file (or the end
|
||||
of the file) becomes part of that field. This matches the behavior of the Go CSV library
|
||||
with `LazyQuotes`, as well as Python's `csv` module. For example:
|
||||
|
||||
GENMD-RUN-COMMAND
|
||||
cat data/lazy-quotes.csv
|
||||
GENMD-EOF
|
||||
|
||||
Here the second field of the first data line has an opening quote with no matching close,
|
||||
so without `--lazy-quotes` we get an error:
|
||||
|
||||
GENMD-RUN-COMMAND-TOLERATING-ERROR
|
||||
mlr --icsv --ojson cat data/lazy-quotes.csv
|
||||
GENMD-EOF
|
||||
|
||||
With `--lazy-quotes`, the quoted field silently absorbs the rest of the file -- including
|
||||
the field separators and the newline -- since there is no closing quote anywhere:
|
||||
|
||||
GENMD-RUN-COMMAND
|
||||
mlr --icsv --ojson --lazy-quotes --allow-ragged-csv-input cat data/lazy-quotes.csv
|
||||
GENMD-EOF
|
||||
|
||||
If quote characters in your data are really just ordinary data characters -- that is, the
|
||||
file doesn't use RFC-4180-style quoting at all -- then [CSV-lite](file-formats.md#csvtsvasvusvetc)
|
||||
is often a better choice, since it splits on the field separator without treating quotes
|
||||
specially (the literal quote character is retained in the data):
|
||||
|
||||
GENMD-RUN-COMMAND
|
||||
mlr --icsvlite --ojson cat data/lazy-quotes.csv
|
||||
GENMD-EOF
|
||||
|
||||
### Troubleshooting CSV and JSON input
|
||||
|
||||
Please see [this page](troubleshooting-csv-and-json-input.md).
|
||||
|
|
@ -212,6 +253,18 @@ GENMD-RUN-COMMAND
|
|||
mlr -o pprint --barred cat data/small | mlr -i pprint --barred-input -o json filter '$b == "pan"'
|
||||
GENMD-EOF
|
||||
|
||||
Use `--right` to right-align all cells, or `--right-align-numeric` to right-align only the cells
|
||||
having numeric values, leaving other cells left-aligned. Headers are right-aligned over columns
|
||||
whose values are all numeric, so that header and data share the same alignment:
|
||||
|
||||
GENMD-RUN-COMMAND
|
||||
mlr --icsv --opprint --right-align-numeric cat example.csv
|
||||
GENMD-EOF
|
||||
|
||||
GENMD-RUN-COMMAND
|
||||
mlr --icsv --opprint --barred --right-align-numeric cat example.csv
|
||||
GENMD-EOF
|
||||
|
||||
## Markdown tabular
|
||||
|
||||
Markdown format looks like this:
|
||||
|
|
@ -243,6 +296,17 @@ GENMD-RUN-COMMAND
|
|||
mlr --md-aligned cat data/small
|
||||
GENMD-EOF
|
||||
|
||||
The `--right-align-numeric` flag also applies to markdown output: numeric columns get a
|
||||
right-alignment marker (`---:`) in the header-separator line, so they render right-aligned in
|
||||
Markdown viewers. With `--omd`, since output is streaming, the marker for each column is chosen
|
||||
from the first record of each same-schema group; with `--omd-aligned`, a column gets the marker
|
||||
when all its values are numeric, and its header and cell text are right-justified in the raw
|
||||
markdown as well:
|
||||
|
||||
GENMD-RUN-COMMAND
|
||||
mlr --icsv --omd-aligned --right-align-numeric cat example.csv
|
||||
GENMD-EOF
|
||||
|
||||
## XTAB: Vertical tabular
|
||||
|
||||
This is perhaps most useful for looking a very wide and/or multi-column data which causes line-wraps on the screen (but see also
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -18,7 +18,7 @@ def main
|
|||
input_handle = $stdin
|
||||
output_handle = $stdout
|
||||
|
||||
output_handle.puts("<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->")
|
||||
output_handle.puts("<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->")
|
||||
|
||||
# The filename on the command line is used for link-generation; file contents
|
||||
# are read from standard input.
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
@ -228,11 +228,11 @@ This is simply a copy of what you should see on running `man mlr` at a command p
|
|||
count-similar cut decimate describe fill-down fill-empty filter flatten
|
||||
format-values fraction gap grep group-by group-like gsub having-fields head
|
||||
histogram json-parse json-stringify join label latin1-to-utf8 least-frequent
|
||||
merge-fields most-frequent nest nothing put regularize remove-empty-columns
|
||||
rename reorder repeat reshape sample sec2gmtdate sec2gmt seqgen shuffle
|
||||
skip-trivial-records sort sort-within-records sparsify split ssub stats1
|
||||
stats2 step sub summary surv tac tail tee template top utf8-to-latin1
|
||||
unflatten uniq unspace unsparsify
|
||||
merge-fields most-frequent nest nothing put rank regularize
|
||||
remove-empty-columns rename reorder repeat reshape sample sec2gmtdate sec2gmt
|
||||
seqgen shuffle skip-trivial-records sort sort-within-records sparkline
|
||||
sparsify split ssub stats1 stats2 step sub summary surv tac tail tee template
|
||||
top utf8-to-latin1 unflatten uniq unspace unsparsify
|
||||
|
||||
1mFUNCTION LIST0m
|
||||
abs acos acosh antimode any append apply arrayify asin asinh asserting_absent
|
||||
|
|
@ -257,14 +257,14 @@ This is simply a copy of what you should see on running `man mlr` at a command p
|
|||
null_count os percentile percentiles pow qnorm reduce regextract
|
||||
regextract_or_else rightpad round roundm rstrip sec2dhms sec2gmt sec2gmtdate
|
||||
sec2hms sec2localdate sec2localtime select sgn sha1 sha256 sha512 sin sinh
|
||||
skewness sort sort_collection splita splitax splitkv splitkvx splitnv splitnvx
|
||||
sqrt ssub stat stddev strfntime strfntime_local strftime strftime_local string
|
||||
strip strlen strmatch strmatchx strpntime strpntime_local strptime
|
||||
strptime_local sub substr substr0 substr1 sum sum2 sum3 sum4 sysntime system
|
||||
systime systimeint tan tanh tolower toupper truncate typeof unflatten unformat
|
||||
unformatx upntime uptime urand urand32 urandelement urandint urandrange
|
||||
utf8_to_latin1 variance version ! != !=~ % & && * ** + - . .* .+ .- ./ / // <
|
||||
<< <= <=> == =~ > >= >> >>> ?: ?? ??? ^ ^^ | || ~
|
||||
skewness sort sort_collection sparkline splita splitax splitkv splitkvx
|
||||
splitnv splitnvx sqrt ssub stat stddev strfntime strfntime_local strftime
|
||||
strftime_local string strip strlen strmatch strmatchx strpntime
|
||||
strpntime_local strptime strptime_local sub substr substr0 substr1 sum sum2
|
||||
sum3 sum4 sysntime system systime systimeint tan tanh tolower toupper truncate
|
||||
typeof unflatten unformat unformatx upntime uptime urand urand32 urandelement
|
||||
urandint urandrange utf8_to_latin1 variance version ! != !=~ % & && * ** + - .
|
||||
.* .+ .- ./ / // < << <= <=> == =~ > >= >> >>> ?: ?? ??? ^ ^^ | || ~
|
||||
|
||||
1mCOMMENTS-IN-DATA FLAGS0m
|
||||
Miller lets you put comments in your data, such as
|
||||
|
|
@ -801,6 +801,13 @@ This is simply a copy of what you should see on running `man mlr` at a command p
|
|||
right-align-multi-word
|
||||
--fw {string} Shortcut for --fixed left-align-multi-word
|
||||
--right Right-justifies all fields for PPRINT output.
|
||||
--right-align-numeric Right-justifies fields with numeric values for PPRINT
|
||||
output, leaving other fields left-justified. Headers
|
||||
are right-justified over columns whose values are all
|
||||
numeric, so that header and data share the same
|
||||
alignment. Also applies to markdown output, where
|
||||
numeric columns get right-alignment markers (`---:`)
|
||||
in the header-separator line.
|
||||
|
||||
1mPROFILING FLAGS0m
|
||||
These are flags for profiling Miller performance.
|
||||
|
|
@ -840,6 +847,9 @@ This is simply a copy of what you should see on running `man mlr` at a command p
|
|||
* Default line endings (`--irs` and `--ors`) are newline
|
||||
which is interpreted to accept carriage-return/newline files (e.g. on Windows)
|
||||
for input, and to produce platform-appropriate line endings on output.
|
||||
* For CSV, CSV-lite, TSV, and TSV-lite output, ORS may be either newline (the
|
||||
default) or carriage-return/newline: e.g. `--ors crlf` or `--ors '\r\n'`
|
||||
for RFC-4180-style line endings on any platform.
|
||||
|
||||
Notes about all other separators:
|
||||
|
||||
|
|
@ -1452,7 +1462,11 @@ This is simply a copy of what you should see on running `man mlr` at a command p
|
|||
--auto Automatically computes limits, ignoring --lo and --hi. Holds all
|
||||
values in memory before producing any output.
|
||||
-o {prefix} Prefix for output field name. Default: no prefix.
|
||||
-s Print a one-line Unicode sparkline per field instead of per-bin
|
||||
counts.
|
||||
-h|--help Show this message.
|
||||
With -s, output is one record per value-field, with a sparkline field
|
||||
instead of one record per bin.
|
||||
|
||||
1mjson-parse0m
|
||||
Usage: mlr json-parse [options]
|
||||
|
|
@ -1798,6 +1812,34 @@ This is simply a copy of what you should see on running `man mlr` at a command p
|
|||
|
||||
See also https://miller.readthedocs.io/reference-dsl for more context.
|
||||
|
||||
1mrank0m
|
||||
Usage: mlr rank [options]
|
||||
For each record's value in specified fields, computes the standard
|
||||
competition rank (1,2,2,4,...) of that value among all input records,
|
||||
optionally within groups.
|
||||
E.g. with input records x=10, x=20, x=20, and x=30, emits output records
|
||||
x=10,x_rank=1 x=20,x_rank=2 x=20,x_rank=2 and x=30,x_rank=4.
|
||||
|
||||
Note: by default this is a two-pass algorithm: on the first pass it retains
|
||||
input records and their values; on the second pass it computes ranks and
|
||||
emits output records, in original input order. This means it produces no
|
||||
output until all input is read, but gives correct ranks regardless of input
|
||||
order. Use --sorted for a single-pass streaming alternative.
|
||||
|
||||
Options:
|
||||
-f {a,b,c} Field name(s) to rank.
|
||||
-g {d,e,f} Optional group-by-field name(s).
|
||||
--sorted Promise that the input is already sorted by the field(s) being ranked
|
||||
(within each group, if -g is given). This computes rank in a single
|
||||
streaming pass and O(1) space, by comparing each record's value only
|
||||
to the immediately preceding one, rather than buffering all records
|
||||
to compute an order-independent rank. Produces wrong output if the
|
||||
input is not in fact sorted.
|
||||
-h|--help Show this message.
|
||||
Example: mlr rank -f x data/rank-example.csv
|
||||
Example: mlr rank -f x -g g data/rank-example.csv
|
||||
Example: mlr sort -f x then rank -f x --sorted data/rank-example.csv
|
||||
|
||||
1mregularize0m
|
||||
Usage: mlr regularize [options]
|
||||
Outputs records sorted lexically ascending by keys.
|
||||
|
|
@ -1840,8 +1882,10 @@ This is simply a copy of what you should see on running `man mlr` at a command p
|
|||
record start.
|
||||
-f {a,b,c} Field names to reorder.
|
||||
-r {a,b,c} Treat field names as regular expressions. Matched fields are moved to
|
||||
start or end in record order. Example: -r '^YYY,^XXX' puts all YYY-
|
||||
and XXX-prefixed fields first (in record order), then the rest.
|
||||
start or end, grouped by the order the regexes are given; within each
|
||||
group, fields keep their record order. Example: -r '^YYY,^XXX' puts
|
||||
all YYY-prefixed fields first, then all XXX-prefixed fields, then the
|
||||
rest.
|
||||
-b {x} Put field names specified with -f before field name specified by {x},
|
||||
if any. If {x} isn't present in a given record, the specified fields
|
||||
will not be moved.
|
||||
|
|
@ -1853,7 +1897,7 @@ This is simply a copy of what you should see on running `man mlr` at a command p
|
|||
Examples:
|
||||
mlr reorder -f a,b sends input record "d=4,b=2,a=1,c=3" to "a=1,b=2,d=4,c=3".
|
||||
mlr reorder -e -f a,b sends input record "d=4,b=2,a=1,c=3" to "d=4,c=3,a=1,b=2".
|
||||
mlr reorder -r '^YYY,^XXX' puts YYY- and XXX-prefixed fields first (record order), then rest.
|
||||
mlr reorder -r '^YYY,^XXX' puts YYY-prefixed fields first, then XXX-prefixed fields, then rest.
|
||||
|
||||
1mrepeat0m
|
||||
Usage: mlr repeat [options]
|
||||
|
|
@ -2071,6 +2115,16 @@ This is simply a copy of what you should see on running `man mlr` at a command p
|
|||
-n Sort field names naturally (e.g. 2 before 12). Combines with -f/-r.
|
||||
-h|--help Show this message.
|
||||
|
||||
1msparkline0m
|
||||
Usage: mlr sparkline [options]
|
||||
Reduces numeric field(s), across all records in input order, to a compact
|
||||
Unicode sparkline -- one block character per record -- for visualizing
|
||||
trends. Emits one output record per field. Holds all records in memory
|
||||
before producing any output.
|
||||
Options:
|
||||
-f {a,b,c} Field names to sparkline.
|
||||
-h|--help Show this message.
|
||||
|
||||
1msparsify0m
|
||||
Usage: mlr sparsify [options]
|
||||
Unsets fields for which the key is the empty string (or, optionally, another
|
||||
|
|
@ -2312,7 +2366,7 @@ This is simply a copy of what you should see on running `man mlr` at a command p
|
|||
Show summary statistics about the input data.
|
||||
|
||||
All summarizers:
|
||||
field_type string, int, etc. -- if a column has mixed types, all encountered types are printed
|
||||
field_type string, int, etc. -- if a column has mixed types, all encountered types are printed (see notes below)
|
||||
count +1 for every instance of the field across all records in the input record stream
|
||||
null_count count of field values either empty string or JSON null
|
||||
distinct_count count of distinct values for the field
|
||||
|
|
@ -2342,6 +2396,8 @@ This is simply a copy of what you should see on running `man mlr` at a command p
|
|||
* min, p25, median, p75, and max work for strings as well as numbers
|
||||
* Distinct-counts are computed on string representations -- so 4.1 and 4.10 are counted as distinct here.
|
||||
* If the mode is not unique in the input data, the first-encountered value is reported as the mode.
|
||||
* A field_type of "int-string", "empty-string", etc. means the column contains values of mixed types --
|
||||
all types encountered are printed, hyphen-joined, in the order first encountered.
|
||||
|
||||
Options:
|
||||
-a {mean,sum,etc.} Use only the specified summarizers.
|
||||
|
|
@ -2447,6 +2503,9 @@ This is simply a copy of what you should see on running `man mlr` at a command p
|
|||
count-distinct. For uniq, -f is a synonym for -g. Output fields are
|
||||
written in the order in which they are named with -g or -f, not in the
|
||||
order in which they appear in the input records.
|
||||
To deduplicate records by one or more fields while keeping all other
|
||||
fields, use head: e.g. "mlr head -n 1 -g hash" keeps the first record
|
||||
for each distinct value of the hash field, with all fields intact.
|
||||
|
||||
Options:
|
||||
-g {d,e,f} Group-by field names for uniq counts.
|
||||
|
|
@ -2738,11 +2797,13 @@ This is simply a copy of what you should see on running `man mlr` at a command p
|
|||
Map example: fold({"a":1, "b":3, "c": 5}, func(acck,accv,ek,ev) {return {"sum": accv+ev**2}}, {"sum":10000}) returns 10035.
|
||||
|
||||
1mformat0m
|
||||
(class=string #args=variadic) Using first argument as format string, interpolate remaining arguments in place of each "{}" in the format string. Too-few arguments are treated as the empty string; too-many arguments are discarded.
|
||||
(class=string #args=variadic) Using first argument as format string, interpolate remaining arguments in place of each "{}" in the format string. Also supports 1-based positional placeholders such as "{1}", which allow arguments to be reused and/or reordered. Plain "{}" placeholders consume arguments sequentially, independently of any positional placeholders. Too-few arguments are treated as the empty string, as are positional placeholders exceeding the number of arguments; too-many arguments are discarded. "{0}" is an error value, since positional placeholders are 1-based.
|
||||
Examples:
|
||||
format("{}:{}:{}", 1,2) gives "1:2:".
|
||||
format("{}:{}:{}", 1,2,3) gives "1:2:3".
|
||||
format("{}:{}:{}", 1,2,3,4) gives "1:2:3".
|
||||
format("{}:{}:{}", 1,2) gives "1:2:".
|
||||
format("{}:{}:{}", 1,2,3) gives "1:2:3".
|
||||
format("{}:{}:{}", 1,2,3,4) gives "1:2:3".
|
||||
format("{1}:{2}:{1}", "a","b") gives "a:b:a".
|
||||
format("{2}{}:{1}{}", 3,4) gives "43:34".
|
||||
|
||||
1mfsec2dhms0m
|
||||
(class=time #args=1) Formats floating-point seconds as in fsec2dhms(500000.25) = "5d18h53m20.250000s"
|
||||
|
|
@ -3265,6 +3326,12 @@ This is simply a copy of what you should see on running `man mlr` at a command p
|
|||
1msort_collection0m
|
||||
(class=stats #args=1) This is a helper function for the percentiles function; please see its online help for details.
|
||||
|
||||
1msparkline0m
|
||||
(class=stats #args=1) Returns a string of Unicode block characters (one of per element) representing the relative magnitudes of values in an array or map, for a compact ASCII/Unicode bar chart. Returns error for non-array/non-map types.
|
||||
Examples:
|
||||
sparkline([1,2,3,4,5,6,7,8]) is ""
|
||||
sparkline([3,3,3]) is ""
|
||||
|
||||
1msplita0m
|
||||
(class=conversion #args=2) Splits string into array with type inference. First argument is string to split; second is the separator to split on.
|
||||
Example:
|
||||
|
|
@ -4043,5 +4110,5 @@ This is simply a copy of what you should see on running `man mlr` at a command p
|
|||
MIME Type for Comma-Separated Values (CSV) Files, the Miller docsite
|
||||
https://miller.readthedocs.io
|
||||
|
||||
2026-07-05 4mMILLER24m(1)
|
||||
2026-07-08 4mMILLER24m(1)
|
||||
</pre>
|
||||
|
|
|
|||
|
|
@ -207,11 +207,11 @@
|
|||
count-similar cut decimate describe fill-down fill-empty filter flatten
|
||||
format-values fraction gap grep group-by group-like gsub having-fields head
|
||||
histogram json-parse json-stringify join label latin1-to-utf8 least-frequent
|
||||
merge-fields most-frequent nest nothing put regularize remove-empty-columns
|
||||
rename reorder repeat reshape sample sec2gmtdate sec2gmt seqgen shuffle
|
||||
skip-trivial-records sort sort-within-records sparsify split ssub stats1
|
||||
stats2 step sub summary surv tac tail tee template top utf8-to-latin1
|
||||
unflatten uniq unspace unsparsify
|
||||
merge-fields most-frequent nest nothing put rank regularize
|
||||
remove-empty-columns rename reorder repeat reshape sample sec2gmtdate sec2gmt
|
||||
seqgen shuffle skip-trivial-records sort sort-within-records sparkline
|
||||
sparsify split ssub stats1 stats2 step sub summary surv tac tail tee template
|
||||
top utf8-to-latin1 unflatten uniq unspace unsparsify
|
||||
|
||||
1mFUNCTION LIST0m
|
||||
abs acos acosh antimode any append apply arrayify asin asinh asserting_absent
|
||||
|
|
@ -236,14 +236,14 @@
|
|||
null_count os percentile percentiles pow qnorm reduce regextract
|
||||
regextract_or_else rightpad round roundm rstrip sec2dhms sec2gmt sec2gmtdate
|
||||
sec2hms sec2localdate sec2localtime select sgn sha1 sha256 sha512 sin sinh
|
||||
skewness sort sort_collection splita splitax splitkv splitkvx splitnv splitnvx
|
||||
sqrt ssub stat stddev strfntime strfntime_local strftime strftime_local string
|
||||
strip strlen strmatch strmatchx strpntime strpntime_local strptime
|
||||
strptime_local sub substr substr0 substr1 sum sum2 sum3 sum4 sysntime system
|
||||
systime systimeint tan tanh tolower toupper truncate typeof unflatten unformat
|
||||
unformatx upntime uptime urand urand32 urandelement urandint urandrange
|
||||
utf8_to_latin1 variance version ! != !=~ % & && * ** + - . .* .+ .- ./ / // <
|
||||
<< <= <=> == =~ > >= >> >>> ?: ?? ??? ^ ^^ | || ~
|
||||
skewness sort sort_collection sparkline splita splitax splitkv splitkvx
|
||||
splitnv splitnvx sqrt ssub stat stddev strfntime strfntime_local strftime
|
||||
strftime_local string strip strlen strmatch strmatchx strpntime
|
||||
strpntime_local strptime strptime_local sub substr substr0 substr1 sum sum2
|
||||
sum3 sum4 sysntime system systime systimeint tan tanh tolower toupper truncate
|
||||
typeof unflatten unformat unformatx upntime uptime urand urand32 urandelement
|
||||
urandint urandrange utf8_to_latin1 variance version ! != !=~ % & && * ** + - .
|
||||
.* .+ .- ./ / // < << <= <=> == =~ > >= >> >>> ?: ?? ??? ^ ^^ | || ~
|
||||
|
||||
1mCOMMENTS-IN-DATA FLAGS0m
|
||||
Miller lets you put comments in your data, such as
|
||||
|
|
@ -780,6 +780,13 @@
|
|||
right-align-multi-word
|
||||
--fw {string} Shortcut for --fixed left-align-multi-word
|
||||
--right Right-justifies all fields for PPRINT output.
|
||||
--right-align-numeric Right-justifies fields with numeric values for PPRINT
|
||||
output, leaving other fields left-justified. Headers
|
||||
are right-justified over columns whose values are all
|
||||
numeric, so that header and data share the same
|
||||
alignment. Also applies to markdown output, where
|
||||
numeric columns get right-alignment markers (`---:`)
|
||||
in the header-separator line.
|
||||
|
||||
1mPROFILING FLAGS0m
|
||||
These are flags for profiling Miller performance.
|
||||
|
|
@ -819,6 +826,9 @@
|
|||
* Default line endings (`--irs` and `--ors`) are newline
|
||||
which is interpreted to accept carriage-return/newline files (e.g. on Windows)
|
||||
for input, and to produce platform-appropriate line endings on output.
|
||||
* For CSV, CSV-lite, TSV, and TSV-lite output, ORS may be either newline (the
|
||||
default) or carriage-return/newline: e.g. `--ors crlf` or `--ors '\r\n'`
|
||||
for RFC-4180-style line endings on any platform.
|
||||
|
||||
Notes about all other separators:
|
||||
|
||||
|
|
@ -1431,7 +1441,11 @@
|
|||
--auto Automatically computes limits, ignoring --lo and --hi. Holds all
|
||||
values in memory before producing any output.
|
||||
-o {prefix} Prefix for output field name. Default: no prefix.
|
||||
-s Print a one-line Unicode sparkline per field instead of per-bin
|
||||
counts.
|
||||
-h|--help Show this message.
|
||||
With -s, output is one record per value-field, with a sparkline field
|
||||
instead of one record per bin.
|
||||
|
||||
1mjson-parse0m
|
||||
Usage: mlr json-parse [options]
|
||||
|
|
@ -1777,6 +1791,34 @@
|
|||
|
||||
See also https://miller.readthedocs.io/reference-dsl for more context.
|
||||
|
||||
1mrank0m
|
||||
Usage: mlr rank [options]
|
||||
For each record's value in specified fields, computes the standard
|
||||
competition rank (1,2,2,4,...) of that value among all input records,
|
||||
optionally within groups.
|
||||
E.g. with input records x=10, x=20, x=20, and x=30, emits output records
|
||||
x=10,x_rank=1 x=20,x_rank=2 x=20,x_rank=2 and x=30,x_rank=4.
|
||||
|
||||
Note: by default this is a two-pass algorithm: on the first pass it retains
|
||||
input records and their values; on the second pass it computes ranks and
|
||||
emits output records, in original input order. This means it produces no
|
||||
output until all input is read, but gives correct ranks regardless of input
|
||||
order. Use --sorted for a single-pass streaming alternative.
|
||||
|
||||
Options:
|
||||
-f {a,b,c} Field name(s) to rank.
|
||||
-g {d,e,f} Optional group-by-field name(s).
|
||||
--sorted Promise that the input is already sorted by the field(s) being ranked
|
||||
(within each group, if -g is given). This computes rank in a single
|
||||
streaming pass and O(1) space, by comparing each record's value only
|
||||
to the immediately preceding one, rather than buffering all records
|
||||
to compute an order-independent rank. Produces wrong output if the
|
||||
input is not in fact sorted.
|
||||
-h|--help Show this message.
|
||||
Example: mlr rank -f x data/rank-example.csv
|
||||
Example: mlr rank -f x -g g data/rank-example.csv
|
||||
Example: mlr sort -f x then rank -f x --sorted data/rank-example.csv
|
||||
|
||||
1mregularize0m
|
||||
Usage: mlr regularize [options]
|
||||
Outputs records sorted lexically ascending by keys.
|
||||
|
|
@ -1819,8 +1861,10 @@
|
|||
record start.
|
||||
-f {a,b,c} Field names to reorder.
|
||||
-r {a,b,c} Treat field names as regular expressions. Matched fields are moved to
|
||||
start or end in record order. Example: -r '^YYY,^XXX' puts all YYY-
|
||||
and XXX-prefixed fields first (in record order), then the rest.
|
||||
start or end, grouped by the order the regexes are given; within each
|
||||
group, fields keep their record order. Example: -r '^YYY,^XXX' puts
|
||||
all YYY-prefixed fields first, then all XXX-prefixed fields, then the
|
||||
rest.
|
||||
-b {x} Put field names specified with -f before field name specified by {x},
|
||||
if any. If {x} isn't present in a given record, the specified fields
|
||||
will not be moved.
|
||||
|
|
@ -1832,7 +1876,7 @@
|
|||
Examples:
|
||||
mlr reorder -f a,b sends input record "d=4,b=2,a=1,c=3" to "a=1,b=2,d=4,c=3".
|
||||
mlr reorder -e -f a,b sends input record "d=4,b=2,a=1,c=3" to "d=4,c=3,a=1,b=2".
|
||||
mlr reorder -r '^YYY,^XXX' puts YYY- and XXX-prefixed fields first (record order), then rest.
|
||||
mlr reorder -r '^YYY,^XXX' puts YYY-prefixed fields first, then XXX-prefixed fields, then rest.
|
||||
|
||||
1mrepeat0m
|
||||
Usage: mlr repeat [options]
|
||||
|
|
@ -2050,6 +2094,16 @@
|
|||
-n Sort field names naturally (e.g. 2 before 12). Combines with -f/-r.
|
||||
-h|--help Show this message.
|
||||
|
||||
1msparkline0m
|
||||
Usage: mlr sparkline [options]
|
||||
Reduces numeric field(s), across all records in input order, to a compact
|
||||
Unicode sparkline -- one block character per record -- for visualizing
|
||||
trends. Emits one output record per field. Holds all records in memory
|
||||
before producing any output.
|
||||
Options:
|
||||
-f {a,b,c} Field names to sparkline.
|
||||
-h|--help Show this message.
|
||||
|
||||
1msparsify0m
|
||||
Usage: mlr sparsify [options]
|
||||
Unsets fields for which the key is the empty string (or, optionally, another
|
||||
|
|
@ -2291,7 +2345,7 @@
|
|||
Show summary statistics about the input data.
|
||||
|
||||
All summarizers:
|
||||
field_type string, int, etc. -- if a column has mixed types, all encountered types are printed
|
||||
field_type string, int, etc. -- if a column has mixed types, all encountered types are printed (see notes below)
|
||||
count +1 for every instance of the field across all records in the input record stream
|
||||
null_count count of field values either empty string or JSON null
|
||||
distinct_count count of distinct values for the field
|
||||
|
|
@ -2321,6 +2375,8 @@
|
|||
* min, p25, median, p75, and max work for strings as well as numbers
|
||||
* Distinct-counts are computed on string representations -- so 4.1 and 4.10 are counted as distinct here.
|
||||
* If the mode is not unique in the input data, the first-encountered value is reported as the mode.
|
||||
* A field_type of "int-string", "empty-string", etc. means the column contains values of mixed types --
|
||||
all types encountered are printed, hyphen-joined, in the order first encountered.
|
||||
|
||||
Options:
|
||||
-a {mean,sum,etc.} Use only the specified summarizers.
|
||||
|
|
@ -2426,6 +2482,9 @@
|
|||
count-distinct. For uniq, -f is a synonym for -g. Output fields are
|
||||
written in the order in which they are named with -g or -f, not in the
|
||||
order in which they appear in the input records.
|
||||
To deduplicate records by one or more fields while keeping all other
|
||||
fields, use head: e.g. "mlr head -n 1 -g hash" keeps the first record
|
||||
for each distinct value of the hash field, with all fields intact.
|
||||
|
||||
Options:
|
||||
-g {d,e,f} Group-by field names for uniq counts.
|
||||
|
|
@ -2717,11 +2776,13 @@
|
|||
Map example: fold({"a":1, "b":3, "c": 5}, func(acck,accv,ek,ev) {return {"sum": accv+ev**2}}, {"sum":10000}) returns 10035.
|
||||
|
||||
1mformat0m
|
||||
(class=string #args=variadic) Using first argument as format string, interpolate remaining arguments in place of each "{}" in the format string. Too-few arguments are treated as the empty string; too-many arguments are discarded.
|
||||
(class=string #args=variadic) Using first argument as format string, interpolate remaining arguments in place of each "{}" in the format string. Also supports 1-based positional placeholders such as "{1}", which allow arguments to be reused and/or reordered. Plain "{}" placeholders consume arguments sequentially, independently of any positional placeholders. Too-few arguments are treated as the empty string, as are positional placeholders exceeding the number of arguments; too-many arguments are discarded. "{0}" is an error value, since positional placeholders are 1-based.
|
||||
Examples:
|
||||
format("{}:{}:{}", 1,2) gives "1:2:".
|
||||
format("{}:{}:{}", 1,2,3) gives "1:2:3".
|
||||
format("{}:{}:{}", 1,2,3,4) gives "1:2:3".
|
||||
format("{}:{}:{}", 1,2) gives "1:2:".
|
||||
format("{}:{}:{}", 1,2,3) gives "1:2:3".
|
||||
format("{}:{}:{}", 1,2,3,4) gives "1:2:3".
|
||||
format("{1}:{2}:{1}", "a","b") gives "a:b:a".
|
||||
format("{2}{}:{1}{}", 3,4) gives "43:34".
|
||||
|
||||
1mfsec2dhms0m
|
||||
(class=time #args=1) Formats floating-point seconds as in fsec2dhms(500000.25) = "5d18h53m20.250000s"
|
||||
|
|
@ -3244,6 +3305,12 @@
|
|||
1msort_collection0m
|
||||
(class=stats #args=1) This is a helper function for the percentiles function; please see its online help for details.
|
||||
|
||||
1msparkline0m
|
||||
(class=stats #args=1) Returns a string of Unicode block characters (one of per element) representing the relative magnitudes of values in an array or map, for a compact ASCII/Unicode bar chart. Returns error for non-array/non-map types.
|
||||
Examples:
|
||||
sparkline([1,2,3,4,5,6,7,8]) is ""
|
||||
sparkline([3,3,3]) is ""
|
||||
|
||||
1msplita0m
|
||||
(class=conversion #args=2) Splits string into array with type inference. First argument is string to split; second is the separator to split on.
|
||||
Example:
|
||||
|
|
@ -4022,4 +4089,4 @@
|
|||
MIME Type for Comma-Separated Values (CSV) Files, the Miller docsite
|
||||
https://miller.readthedocs.io
|
||||
|
||||
2026-07-05 4mMILLER24m(1)
|
||||
2026-07-08 4mMILLER24m(1)
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
@ -139,6 +139,84 @@ Thanks to @aborruso for the tip!
|
|||
|
||||
See also the [record-heterogeneity page](record-heterogeneity.md).
|
||||
|
||||
## Doing SQL-style left, right, inner, and full-outer joins
|
||||
|
||||
Miller's `join` verb is defined in terms of _paired_ and _unpaired_ records, rather than SQL-database terminology -- but you can get SQL-style joins using the `--ul` and `--ur` flags (which emit unpaired left-file and right-file records, respectively), along with [`unsparsify`](reference-verbs.md#unsparsify) to fill in empty cells for non-matches.
|
||||
|
||||
Suppose you have the following two data files, where we want to join on the left file's `a` field matching the right file's `e` field:
|
||||
|
||||
<pre class="pre-non-highlight-non-pair">
|
||||
a,b,c
|
||||
a,t,1
|
||||
b,u,2
|
||||
c,v,3
|
||||
</pre>
|
||||
|
||||
<pre class="pre-non-highlight-non-pair">
|
||||
e,f,g
|
||||
a,t,3
|
||||
b,u,2
|
||||
d,w,1
|
||||
</pre>
|
||||
|
||||
In all the following examples, the `-f` file (`data/join-x.csv`) is the left file, and the file in the main input stream (`data/join-y.csv`) is the right file. The flags `-j a -r e` say that the left file's `a` field is matched against the right file's `e` field, with the output join column named `a`.
|
||||
|
||||
**Inner join** -- only matching records -- is what Miller's `join` does by default, since only paired records are emitted:
|
||||
|
||||
<pre class="pre-highlight-in-pair">
|
||||
<b>mlr --icsv --ocsv join -j a -r e -f data/join-x.csv data/join-y.csv</b>
|
||||
</pre>
|
||||
<pre class="pre-non-highlight-in-pair">
|
||||
a,b,c,f,g
|
||||
a,t,1,t,3
|
||||
b,u,2,u,2
|
||||
</pre>
|
||||
|
||||
**Left join** keeps all records from the left file, with empty cells where the right file has no match. Use `--ul` to also emit unpaired left-file records, then `unsparsify` to square up the output:
|
||||
|
||||
<pre class="pre-highlight-in-pair">
|
||||
<b>mlr --icsv --ocsv join --ul -j a -r e -f data/join-x.csv \</b>
|
||||
<b> then unsparsify --fill-with "" \</b>
|
||||
<b> data/join-y.csv</b>
|
||||
</pre>
|
||||
<pre class="pre-non-highlight-in-pair">
|
||||
a,b,c,f,g
|
||||
a,t,1,t,3
|
||||
b,u,2,u,2
|
||||
c,v,3,,
|
||||
</pre>
|
||||
|
||||
**Right join** keeps all records from the right file. Use `--ur` to also emit unpaired right-file records:
|
||||
|
||||
<pre class="pre-highlight-in-pair">
|
||||
<b>mlr --icsv --ocsv join --ur -j a -r e -f data/join-x.csv \</b>
|
||||
<b> then unsparsify --fill-with "" \</b>
|
||||
<b> data/join-y.csv</b>
|
||||
</pre>
|
||||
<pre class="pre-non-highlight-in-pair">
|
||||
a,b,c,f,g
|
||||
a,t,1,t,3
|
||||
b,u,2,u,2
|
||||
d,,,w,1
|
||||
</pre>
|
||||
|
||||
**Full outer join** keeps all records from both files. Use both `--ul` and `--ur`:
|
||||
|
||||
<pre class="pre-highlight-in-pair">
|
||||
<b>mlr --icsv --ocsv join --ul --ur -j a -r e -f data/join-x.csv \</b>
|
||||
<b> then unsparsify --fill-with "" \</b>
|
||||
<b> data/join-y.csv</b>
|
||||
</pre>
|
||||
<pre class="pre-non-highlight-in-pair">
|
||||
a,b,c,f,g
|
||||
a,t,1,t,3
|
||||
b,u,2,u,2
|
||||
d,,,w,1
|
||||
c,v,3,,
|
||||
</pre>
|
||||
|
||||
Note that unpaired records are emitted after all paired records, so the output ordering may differ from what a SQL database would produce; you can pipe the output through [`sort`](reference-verbs.md#sort) if you need a particular ordering.
|
||||
|
||||
## Doing multiple joins
|
||||
|
||||
Suppose we have the following data:
|
||||
|
|
@ -199,6 +277,178 @@ id status name task
|
|||
30 occupied Alice clean
|
||||
</pre>
|
||||
|
||||
## Merging several files on a common key when column names collide
|
||||
|
||||
The previous example worked painlessly because the non-key column names -- `name` and `status` -- were different in each lookup file. Now suppose you have several files, each containing measurements of a _different_ quantity, but all with the _same_ column names -- say, one file each for temperature, humidity, and pressure, keyed by timestamp:
|
||||
|
||||
<pre class="pre-highlight-in-pair">
|
||||
<b>cat data/sensor-temp.csv</b>
|
||||
</pre>
|
||||
<pre class="pre-non-highlight-in-pair">
|
||||
unixTime,minValue,averageValue,maxValue
|
||||
1740000000,37.2,37.4,37.6
|
||||
1740000060,37.5,37.6,37.7
|
||||
1740000120,37.8,37.9,38.0
|
||||
</pre>
|
||||
|
||||
<pre class="pre-highlight-in-pair">
|
||||
<b>cat data/sensor-humidity.csv</b>
|
||||
</pre>
|
||||
<pre class="pre-non-highlight-in-pair">
|
||||
unixTime,minValue,averageValue,maxValue
|
||||
1740000000,50.1,50.3,50.5
|
||||
1740000120,52.3,52.5,52.7
|
||||
</pre>
|
||||
|
||||
<pre class="pre-highlight-in-pair">
|
||||
<b>cat data/sensor-pressure.csv</b>
|
||||
</pre>
|
||||
<pre class="pre-non-highlight-in-pair">
|
||||
unixTime,minValue,averageValue,maxValue
|
||||
1740000000,1012.2,1012.3,1012.4
|
||||
1740000060,1011.8,1011.9,1012.0
|
||||
1740000120,1011.5,1011.6,1011.7
|
||||
</pre>
|
||||
|
||||
(Note that the humidity file is missing a row for the middle timestamp.)
|
||||
|
||||
If we merge these with a `then`-chain of `join` commands, as in the previous section, columns are lost: since every file's non-key columns have the same names, each join step overwrites the values from the step before, and only one file's values survive:
|
||||
|
||||
<pre class="pre-highlight-in-pair">
|
||||
<b>mlr --icsv --opprint \</b>
|
||||
<b> join -j unixTime -f data/sensor-temp.csv \</b>
|
||||
<b> then join -j unixTime -f data/sensor-humidity.csv \</b>
|
||||
<b> data/sensor-pressure.csv</b>
|
||||
</pre>
|
||||
<pre class="pre-non-highlight-in-pair">
|
||||
unixTime minValue averageValue maxValue
|
||||
1740000000 1012.2 1012.3 1012.4
|
||||
1740000120 1011.5 1011.6 1011.7
|
||||
</pre>
|
||||
|
||||
The fix is `join`'s `--lp` (left-prefix) option, which renames the non-key columns coming from each `-f` file so that nothing collides:
|
||||
|
||||
<pre class="pre-highlight-in-pair">
|
||||
<b>mlr --icsv --opprint \</b>
|
||||
<b> join --lp temp: -j unixTime -f data/sensor-temp.csv \</b>
|
||||
<b> then join --lp humidity: -j unixTime -f data/sensor-humidity.csv \</b>
|
||||
<b> data/sensor-pressure.csv</b>
|
||||
</pre>
|
||||
<pre class="pre-non-highlight-in-pair">
|
||||
unixTime humidity:minValue humidity:averageValue humidity:maxValue temp:minValue temp:averageValue temp:maxValue minValue averageValue maxValue
|
||||
1740000000 50.1 50.3 50.5 37.2 37.4 37.6 1012.2 1012.3 1012.4
|
||||
1740000120 52.3 52.5 52.7 37.8 37.9 38.0 1011.5 1011.6 1011.7
|
||||
</pre>
|
||||
|
||||
The columns from the file at the end of the command line -- here, the pressure file -- keep their unprefixed names.
|
||||
|
||||
If you want just one value column per file, you can also use `join`'s `--lk` option to keep only that column from each `-f` file, then use [cut](reference-verbs.md#cut) and [label](reference-verbs.md#label) to arrange and rename the output columns:
|
||||
|
||||
<pre class="pre-highlight-in-pair">
|
||||
<b>mlr --icsv --opprint \</b>
|
||||
<b> join --lp temp: --lk averageValue -j unixTime -f data/sensor-temp.csv \</b>
|
||||
<b> then join --lp humidity: --lk averageValue -j unixTime -f data/sensor-humidity.csv \</b>
|
||||
<b> then cut -o -f unixTime,temp:averageValue,humidity:averageValue,averageValue \</b>
|
||||
<b> then label unixTime,temperature,humidity,pressure \</b>
|
||||
<b> data/sensor-pressure.csv</b>
|
||||
</pre>
|
||||
<pre class="pre-non-highlight-in-pair">
|
||||
unixTime temperature humidity pressure
|
||||
1740000000 37.4 50.3 1012.3
|
||||
1740000120 37.9 52.5 1011.6
|
||||
</pre>
|
||||
|
||||
Note that `join` gives inner-join semantics by default, so the timestamp missing from the humidity file has been dropped from the output. (This is also why the `paste` command is not a substitute for `join` here: `paste` matches rows by position, so a row missing from one file shifts that file's remaining values onto the wrong rows.) If you'd rather keep those rows, with empty cells where a file has no data, add `--ul --ur` to each join step, and [unsparsify](reference-verbs.md#unsparsify) afterward:
|
||||
|
||||
<pre class="pre-highlight-in-pair">
|
||||
<b>mlr --icsv --opprint \</b>
|
||||
<b> join --ul --ur --lp temp: --lk averageValue -j unixTime -f data/sensor-temp.csv \</b>
|
||||
<b> then join --ul --ur --lp humidity: --lk averageValue -j unixTime -f data/sensor-humidity.csv \</b>
|
||||
<b> then unsparsify \</b>
|
||||
<b> then cut -o -f unixTime,temp:averageValue,humidity:averageValue,averageValue \</b>
|
||||
<b> then label unixTime,temperature,humidity,pressure \</b>
|
||||
<b> then sort -t unixTime \</b>
|
||||
<b> data/sensor-pressure.csv</b>
|
||||
</pre>
|
||||
<pre class="pre-non-highlight-in-pair">
|
||||
unixTime temperature humidity pressure
|
||||
1740000000 37.4 50.3 1012.3
|
||||
1740000060 37.6 - 1011.9
|
||||
1740000120 37.9 52.5 1011.6
|
||||
</pre>
|
||||
|
||||
The `unsparsify` must come before the `cut`-and-`label` step, since `label` renames columns positionally. The `sort` is there because unpaired records may be emitted out of order relative to paired ones.
|
||||
|
||||
## How to preprocess the left file of a join?
|
||||
|
||||
The left file (the `-f` argument to `join`) is opened by the `join` verb itself, so it doesn't pass through the main record stream: a `then`-chain can preprocess the right file(s), but not the left one.
|
||||
|
||||
For example, suppose both of these files have multi-valued `id` fields which need [nest --explode](reference-verbs.md#nest) before joining:
|
||||
|
||||
<pre class="pre-highlight-in-pair">
|
||||
<b>cat data/join-nest-left.csv</b>
|
||||
</pre>
|
||||
<pre class="pre-non-highlight-in-pair">
|
||||
id,color
|
||||
1;2,blue
|
||||
3,green
|
||||
</pre>
|
||||
|
||||
<pre class="pre-highlight-in-pair">
|
||||
<b>cat data/join-nest-right.csv</b>
|
||||
</pre>
|
||||
<pre class="pre-non-highlight-in-pair">
|
||||
id,shape
|
||||
1,circle
|
||||
2;3,square
|
||||
</pre>
|
||||
|
||||
Using `nest` in a `then`-chain handles the right file, but the left file still has the unexploded `id` value `1;2`, so only `id=3` pairs up:
|
||||
|
||||
<pre class="pre-highlight-in-pair">
|
||||
<b>mlr --csv nest --evar ';' -f id \</b>
|
||||
<b> then join -j id -f data/join-nest-left.csv \</b>
|
||||
<b> data/join-nest-right.csv</b>
|
||||
</pre>
|
||||
<pre class="pre-non-highlight-in-pair">
|
||||
id,color,shape
|
||||
3,green,square
|
||||
</pre>
|
||||
|
||||
One way to preprocess the left file, without creating an intermediate file, is the main-level `--prepipe` flag. The left file inherits the main input options -- including `--prepipe` -- so the specified command is applied to the left file as well as to the right file(s):
|
||||
|
||||
<pre class="pre-highlight-in-pair">
|
||||
<b>mlr --csv --prepipe 'mlr --csv nest --evar ";" -f id' \</b>
|
||||
<b> join -j id -f data/join-nest-left.csv \</b>
|
||||
<b> data/join-nest-right.csv</b>
|
||||
</pre>
|
||||
<pre class="pre-non-highlight-in-pair">
|
||||
id,color,shape
|
||||
1,blue,circle
|
||||
2,blue,square
|
||||
3,green,square
|
||||
</pre>
|
||||
|
||||
Note that `--prepipe` applies the same command to _every_ input file -- which is just what's wanted here, since both files need the same `nest`.
|
||||
|
||||
Another way, if your shell supports it -- bash, zsh, and ksh do, although plain POSIX `sh` and Windows `cmd` do not -- is [process substitution](https://en.wikipedia.org/wiki/Process_substitution), which lets you preprocess the left file with any command at all, independently of the right file(s):
|
||||
|
||||
<pre class="pre-highlight-in-pair">
|
||||
<b>mlr --csv nest --evar ';' -f id \</b>
|
||||
<b> then join -j id -f <(mlr --csv nest --evar ';' -f id data/join-nest-left.csv) \</b>
|
||||
<b> data/join-nest-right.csv</b>
|
||||
</pre>
|
||||
<pre class="pre-non-highlight-in-pair">
|
||||
id,color,shape
|
||||
1,blue,circle
|
||||
2,blue,square
|
||||
3,green,square
|
||||
</pre>
|
||||
|
||||
Note that while `mlr join --help` lists verb-level `--prepipe` and `--prepipex` flags, as of Miller 6 these do not take effect for the left file -- please use one of the recipes above instead.
|
||||
|
||||
Thanks to @sonicdoe for the process-substitution tip!
|
||||
|
||||
## Updating a database file with new values from another
|
||||
|
||||
Suppose you have a durable "database" file, and you periodically receive a download
|
||||
|
|
|
|||
|
|
@ -60,6 +60,50 @@ Thanks to @aborruso for the tip!
|
|||
|
||||
See also the [record-heterogeneity page](record-heterogeneity.md).
|
||||
|
||||
## Doing SQL-style left, right, inner, and full-outer joins
|
||||
|
||||
Miller's `join` verb is defined in terms of _paired_ and _unpaired_ records, rather than SQL-database terminology -- but you can get SQL-style joins using the `--ul` and `--ur` flags (which emit unpaired left-file and right-file records, respectively), along with [`unsparsify`](reference-verbs.md#unsparsify) to fill in empty cells for non-matches.
|
||||
|
||||
Suppose you have the following two data files, where we want to join on the left file's `a` field matching the right file's `e` field:
|
||||
|
||||
GENMD-INCLUDE-ESCAPED(data/join-x.csv)
|
||||
|
||||
GENMD-INCLUDE-ESCAPED(data/join-y.csv)
|
||||
|
||||
In all the following examples, the `-f` file (`data/join-x.csv`) is the left file, and the file in the main input stream (`data/join-y.csv`) is the right file. The flags `-j a -r e` say that the left file's `a` field is matched against the right file's `e` field, with the output join column named `a`.
|
||||
|
||||
**Inner join** -- only matching records -- is what Miller's `join` does by default, since only paired records are emitted:
|
||||
|
||||
GENMD-RUN-COMMAND
|
||||
mlr --icsv --ocsv join -j a -r e -f data/join-x.csv data/join-y.csv
|
||||
GENMD-EOF
|
||||
|
||||
**Left join** keeps all records from the left file, with empty cells where the right file has no match. Use `--ul` to also emit unpaired left-file records, then `unsparsify` to square up the output:
|
||||
|
||||
GENMD-RUN-COMMAND
|
||||
mlr --icsv --ocsv join --ul -j a -r e -f data/join-x.csv \
|
||||
then unsparsify --fill-with "" \
|
||||
data/join-y.csv
|
||||
GENMD-EOF
|
||||
|
||||
**Right join** keeps all records from the right file. Use `--ur` to also emit unpaired right-file records:
|
||||
|
||||
GENMD-RUN-COMMAND
|
||||
mlr --icsv --ocsv join --ur -j a -r e -f data/join-x.csv \
|
||||
then unsparsify --fill-with "" \
|
||||
data/join-y.csv
|
||||
GENMD-EOF
|
||||
|
||||
**Full outer join** keeps all records from both files. Use both `--ul` and `--ur`:
|
||||
|
||||
GENMD-RUN-COMMAND
|
||||
mlr --icsv --ocsv join --ul --ur -j a -r e -f data/join-x.csv \
|
||||
then unsparsify --fill-with "" \
|
||||
data/join-y.csv
|
||||
GENMD-EOF
|
||||
|
||||
Note that unpaired records are emitted after all paired records, so the output ordering may differ from what a SQL database would produce; you can pipe the output through [`sort`](reference-verbs.md#sort) if you need a particular ordering.
|
||||
|
||||
## Doing multiple joins
|
||||
|
||||
Suppose we have the following data:
|
||||
|
|
@ -86,6 +130,118 @@ mlr --icsv --opprint join -f multi-join/name-lookup.csv -j id \
|
|||
multi-join/input.csv
|
||||
GENMD-EOF
|
||||
|
||||
## Merging several files on a common key when column names collide
|
||||
|
||||
The previous example worked painlessly because the non-key column names -- `name` and `status` -- were different in each lookup file. Now suppose you have several files, each containing measurements of a _different_ quantity, but all with the _same_ column names -- say, one file each for temperature, humidity, and pressure, keyed by timestamp:
|
||||
|
||||
GENMD-RUN-COMMAND
|
||||
cat data/sensor-temp.csv
|
||||
GENMD-EOF
|
||||
|
||||
GENMD-RUN-COMMAND
|
||||
cat data/sensor-humidity.csv
|
||||
GENMD-EOF
|
||||
|
||||
GENMD-RUN-COMMAND
|
||||
cat data/sensor-pressure.csv
|
||||
GENMD-EOF
|
||||
|
||||
(Note that the humidity file is missing a row for the middle timestamp.)
|
||||
|
||||
If we merge these with a `then`-chain of `join` commands, as in the previous section, columns are lost: since every file's non-key columns have the same names, each join step overwrites the values from the step before, and only one file's values survive:
|
||||
|
||||
GENMD-RUN-COMMAND
|
||||
mlr --icsv --opprint \
|
||||
join -j unixTime -f data/sensor-temp.csv \
|
||||
then join -j unixTime -f data/sensor-humidity.csv \
|
||||
data/sensor-pressure.csv
|
||||
GENMD-EOF
|
||||
|
||||
The fix is `join`'s `--lp` (left-prefix) option, which renames the non-key columns coming from each `-f` file so that nothing collides:
|
||||
|
||||
GENMD-RUN-COMMAND
|
||||
mlr --icsv --opprint \
|
||||
join --lp temp: -j unixTime -f data/sensor-temp.csv \
|
||||
then join --lp humidity: -j unixTime -f data/sensor-humidity.csv \
|
||||
data/sensor-pressure.csv
|
||||
GENMD-EOF
|
||||
|
||||
The columns from the file at the end of the command line -- here, the pressure file -- keep their unprefixed names.
|
||||
|
||||
If you want just one value column per file, you can also use `join`'s `--lk` option to keep only that column from each `-f` file, then use [cut](reference-verbs.md#cut) and [label](reference-verbs.md#label) to arrange and rename the output columns:
|
||||
|
||||
GENMD-RUN-COMMAND
|
||||
mlr --icsv --opprint \
|
||||
join --lp temp: --lk averageValue -j unixTime -f data/sensor-temp.csv \
|
||||
then join --lp humidity: --lk averageValue -j unixTime -f data/sensor-humidity.csv \
|
||||
then cut -o -f unixTime,temp:averageValue,humidity:averageValue,averageValue \
|
||||
then label unixTime,temperature,humidity,pressure \
|
||||
data/sensor-pressure.csv
|
||||
GENMD-EOF
|
||||
|
||||
Note that `join` gives inner-join semantics by default, so the timestamp missing from the humidity file has been dropped from the output. (This is also why the `paste` command is not a substitute for `join` here: `paste` matches rows by position, so a row missing from one file shifts that file's remaining values onto the wrong rows.) If you'd rather keep those rows, with empty cells where a file has no data, add `--ul --ur` to each join step, and [unsparsify](reference-verbs.md#unsparsify) afterward:
|
||||
|
||||
GENMD-RUN-COMMAND
|
||||
mlr --icsv --opprint \
|
||||
join --ul --ur --lp temp: --lk averageValue -j unixTime -f data/sensor-temp.csv \
|
||||
then join --ul --ur --lp humidity: --lk averageValue -j unixTime -f data/sensor-humidity.csv \
|
||||
then unsparsify \
|
||||
then cut -o -f unixTime,temp:averageValue,humidity:averageValue,averageValue \
|
||||
then label unixTime,temperature,humidity,pressure \
|
||||
then sort -t unixTime \
|
||||
data/sensor-pressure.csv
|
||||
GENMD-EOF
|
||||
|
||||
The `unsparsify` must come before the `cut`-and-`label` step, since `label` renames columns positionally. The `sort` is there because unpaired records may be emitted out of order relative to paired ones.
|
||||
|
||||
## How to preprocess the left file of a join?
|
||||
|
||||
The left file (the `-f` argument to `join`) is opened by the `join` verb itself, so it doesn't pass through the main record stream: a `then`-chain can preprocess the right file(s), but not the left one.
|
||||
|
||||
For example, suppose both of these files have multi-valued `id` fields which need [nest --explode](reference-verbs.md#nest) before joining:
|
||||
|
||||
GENMD-RUN-COMMAND
|
||||
cat data/join-nest-left.csv
|
||||
GENMD-EOF
|
||||
|
||||
GENMD-RUN-COMMAND
|
||||
cat data/join-nest-right.csv
|
||||
GENMD-EOF
|
||||
|
||||
Using `nest` in a `then`-chain handles the right file, but the left file still has the unexploded `id` value `1;2`, so only `id=3` pairs up:
|
||||
|
||||
GENMD-RUN-COMMAND
|
||||
mlr --csv nest --evar ';' -f id \
|
||||
then join -j id -f data/join-nest-left.csv \
|
||||
data/join-nest-right.csv
|
||||
GENMD-EOF
|
||||
|
||||
One way to preprocess the left file, without creating an intermediate file, is the main-level `--prepipe` flag. The left file inherits the main input options -- including `--prepipe` -- so the specified command is applied to the left file as well as to the right file(s):
|
||||
|
||||
GENMD-RUN-COMMAND
|
||||
mlr --csv --prepipe 'mlr --csv nest --evar ";" -f id' \
|
||||
join -j id -f data/join-nest-left.csv \
|
||||
data/join-nest-right.csv
|
||||
GENMD-EOF
|
||||
|
||||
Note that `--prepipe` applies the same command to _every_ input file -- which is just what's wanted here, since both files need the same `nest`.
|
||||
|
||||
Another way, if your shell supports it -- bash, zsh, and ksh do, although plain POSIX `sh` and Windows `cmd` do not -- is [process substitution](https://en.wikipedia.org/wiki/Process_substitution), which lets you preprocess the left file with any command at all, independently of the right file(s):
|
||||
|
||||
GENMD-CARDIFY-HIGHLIGHT-THREE
|
||||
mlr --csv nest --evar ';' -f id \
|
||||
then join -j id -f <(mlr --csv nest --evar ';' -f id data/join-nest-left.csv) \
|
||||
data/join-nest-right.csv
|
||||
id,color,shape
|
||||
1,blue,circle
|
||||
2,blue,square
|
||||
3,green,square
|
||||
GENMD-EOF
|
||||
|
||||
Note that while `mlr join --help` lists verb-level `--prepipe` and `--prepipex` flags, as of Miller 6 these do not take effect for the left file -- please use one of the recipes above instead.
|
||||
|
||||
Thanks to @sonicdoe for the process-substitution tip!
|
||||
|
||||
## Updating a database file with new values from another
|
||||
|
||||
Suppose you have a durable "database" file, and you periodically receive a download
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
@ -474,6 +474,95 @@ a,b,c
|
|||
7,8,9,10
|
||||
</pre>
|
||||
|
||||
### Combining multiple CSV files with different headers
|
||||
|
||||
A common special case of the above: you have several CSV files whose column names overlap but
|
||||
don't all match -- say, data exported year by year, where columns were added or reordered over
|
||||
time -- and you want to concatenate them into a single rectangular table, with empty cells where a
|
||||
file didn't have a given column.
|
||||
|
||||
<pre class="pre-highlight-in-pair">
|
||||
<b>cat data/het/sales-2021.csv</b>
|
||||
</pre>
|
||||
<pre class="pre-non-highlight-in-pair">
|
||||
id,product,sales
|
||||
1,pencil,100
|
||||
2,eraser,17
|
||||
</pre>
|
||||
|
||||
<pre class="pre-highlight-in-pair">
|
||||
<b>cat data/het/sales-2022.csv</b>
|
||||
</pre>
|
||||
<pre class="pre-non-highlight-in-pair">
|
||||
id,product,color,sales
|
||||
3,pencil,red,120
|
||||
4,eraser,green,32
|
||||
</pre>
|
||||
|
||||
<pre class="pre-highlight-in-pair">
|
||||
<b>cat data/het/sales-2023.csv</b>
|
||||
</pre>
|
||||
<pre class="pre-non-highlight-in-pair">
|
||||
id,product,sales,returns
|
||||
5,pencil,200,6
|
||||
6,notebook,41,2
|
||||
</pre>
|
||||
|
||||
Since each file's records take their schema from that file's header line, simply catting the
|
||||
files together gives a schema change partway through the record stream -- which RFC-4180 CSV
|
||||
output doesn't allow:
|
||||
|
||||
<pre class="pre-highlight-in-pair">
|
||||
<b>mlr --csv cat data/het/sales-2021.csv data/het/sales-2022.csv data/het/sales-2023.csv</b>
|
||||
</pre>
|
||||
<pre class="pre-non-highlight-in-pair">
|
||||
id,product,sales
|
||||
1,pencil,100
|
||||
2,eraser,17
|
||||
mlr: CSV schema change: first keys "id,product,sales"; current keys "id,product,color,sales"
|
||||
mlr: exiting due to data error
|
||||
</pre>
|
||||
|
||||
The solution is the [`unsparsify`](reference-verbs.md#unsparsify) verb, which we saw above: it
|
||||
gives every record the union of all field names, filling in empty values (or, using
|
||||
`--fill-with`, whatever you like) where a record lacks a field:
|
||||
|
||||
<pre class="pre-highlight-in-pair">
|
||||
<b>mlr --csv unsparsify data/het/sales-2021.csv data/het/sales-2022.csv data/het/sales-2023.csv</b>
|
||||
</pre>
|
||||
<pre class="pre-non-highlight-in-pair">
|
||||
id,product,sales,color,returns
|
||||
1,pencil,100,,
|
||||
2,eraser,17,,
|
||||
3,pencil,120,red,
|
||||
4,eraser,32,green,
|
||||
5,pencil,200,,6
|
||||
6,notebook,41,,2
|
||||
</pre>
|
||||
|
||||
Note that `unsparsify` (without `-f`) is
|
||||
[non-streaming](streaming-and-memory.md): it can't know the full set of field names until it has
|
||||
read all input, so it retains all records in memory before producing any output. If your files
|
||||
are too large for that, but you know the complete list of field names up front, you can use
|
||||
`unsparsify -f` which is streaming. Fields filled in by `-f` are appended to each record, so if
|
||||
the input files order their columns differently, follow up with
|
||||
[`regularize`](reference-verbs.md#regularize) or
|
||||
[`sort-within-records`](reference-verbs.md#sort-within-records) (also both streaming) to give all
|
||||
records the same column ordering:
|
||||
|
||||
<pre class="pre-highlight-in-pair">
|
||||
<b>mlr --csv unsparsify -f id,product,sales,color,returns then regularize data/het/sales-2021.csv data/het/sales-2022.csv data/het/sales-2023.csv</b>
|
||||
</pre>
|
||||
<pre class="pre-non-highlight-in-pair">
|
||||
id,product,sales,color,returns
|
||||
1,pencil,100,,
|
||||
2,eraser,17,,
|
||||
3,pencil,120,red,
|
||||
4,eraser,32,green,
|
||||
5,pencil,200,,6
|
||||
6,notebook,41,,2
|
||||
</pre>
|
||||
|
||||
## Processing heterogeneous data
|
||||
|
||||
Above we saw how to make heterogeneous data homogeneous, and then how to print heterogeneous data.
|
||||
|
|
|
|||
|
|
@ -205,6 +205,55 @@ GENMD-RUN-COMMAND
|
|||
mlr --csv --allow-ragged-csv-input cat data/het/ragged.csv
|
||||
GENMD-EOF
|
||||
|
||||
### Combining multiple CSV files with different headers
|
||||
|
||||
A common special case of the above: you have several CSV files whose column names overlap but
|
||||
don't all match -- say, data exported year by year, where columns were added or reordered over
|
||||
time -- and you want to concatenate them into a single rectangular table, with empty cells where a
|
||||
file didn't have a given column.
|
||||
|
||||
GENMD-RUN-COMMAND
|
||||
cat data/het/sales-2021.csv
|
||||
GENMD-EOF
|
||||
|
||||
GENMD-RUN-COMMAND
|
||||
cat data/het/sales-2022.csv
|
||||
GENMD-EOF
|
||||
|
||||
GENMD-RUN-COMMAND
|
||||
cat data/het/sales-2023.csv
|
||||
GENMD-EOF
|
||||
|
||||
Since each file's records take their schema from that file's header line, simply catting the
|
||||
files together gives a schema change partway through the record stream -- which RFC-4180 CSV
|
||||
output doesn't allow:
|
||||
|
||||
GENMD-RUN-COMMAND-TOLERATING-ERROR
|
||||
mlr --csv cat data/het/sales-2021.csv data/het/sales-2022.csv data/het/sales-2023.csv
|
||||
GENMD-EOF
|
||||
|
||||
The solution is the [`unsparsify`](reference-verbs.md#unsparsify) verb, which we saw above: it
|
||||
gives every record the union of all field names, filling in empty values (or, using
|
||||
`--fill-with`, whatever you like) where a record lacks a field:
|
||||
|
||||
GENMD-RUN-COMMAND
|
||||
mlr --csv unsparsify data/het/sales-2021.csv data/het/sales-2022.csv data/het/sales-2023.csv
|
||||
GENMD-EOF
|
||||
|
||||
Note that `unsparsify` (without `-f`) is
|
||||
[non-streaming](streaming-and-memory.md): it can't know the full set of field names until it has
|
||||
read all input, so it retains all records in memory before producing any output. If your files
|
||||
are too large for that, but you know the complete list of field names up front, you can use
|
||||
`unsparsify -f` which is streaming. Fields filled in by `-f` are appended to each record, so if
|
||||
the input files order their columns differently, follow up with
|
||||
[`regularize`](reference-verbs.md#regularize) or
|
||||
[`sort-within-records`](reference-verbs.md#sort-within-records) (also both streaming) to give all
|
||||
records the same column ordering:
|
||||
|
||||
GENMD-RUN-COMMAND
|
||||
mlr --csv unsparsify -f id,product,sales,color,returns then regularize data/het/sales-2021.csv data/het/sales-2022.csv data/het/sales-2023.csv
|
||||
GENMD-EOF
|
||||
|
||||
## Processing heterogeneous data
|
||||
|
||||
Above we saw how to make heterogeneous data homogeneous, and then how to print heterogeneous data.
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
@ -59,7 +59,7 @@ Operators are listed here along with functions. In this case, the argument count
|
|||
* [**Hashing functions**](#hashing-functions): [md5](#md5), [sha1](#sha1), [sha256](#sha256), [sha512](#sha512).
|
||||
* [**Higher-order-functions functions**](#higher-order-functions-functions): [any](#any), [apply](#apply), [every](#every), [fold](#fold), [reduce](#reduce), [select](#select), [sort](#sort).
|
||||
* [**Math functions**](#math-functions): [abs](#abs), [acos](#acos), [acosh](#acosh), [asin](#asin), [asinh](#asinh), [atan](#atan), [atan2](#atan2), [atanh](#atanh), [cbrt](#cbrt), [ceil](#ceil), [cos](#cos), [cosh](#cosh), [erf](#erf), [erfc](#erfc), [exp](#exp), [expm1](#expm1), [floor](#floor), [invqnorm](#invqnorm), [log](#log), [log10](#log10), [log1p](#log1p), [logifit](#logifit), [max](#max), [min](#min), [qnorm](#qnorm), [round](#round), [roundm](#roundm), [sgn](#sgn), [sin](#sin), [sinh](#sinh), [sqrt](#sqrt), [tan](#tan), [tanh](#tanh), [urand](#urand), [urand32](#urand32), [urandelement](#urandelement), [urandint](#urandint), [urandrange](#urandrange).
|
||||
* [**Stats functions**](#stats-functions): [antimode](#antimode), [count](#count), [distinct_count](#distinct_count), [kurtosis](#kurtosis), [maxlen](#maxlen), [mean](#mean), [meaneb](#meaneb), [median](#median), [minlen](#minlen), [mode](#mode), [null_count](#null_count), [percentile](#percentile), [percentiles](#percentiles), [skewness](#skewness), [sort_collection](#sort_collection), [stddev](#stddev), [sum](#sum), [sum2](#sum2), [sum3](#sum3), [sum4](#sum4), [variance](#variance).
|
||||
* [**Stats functions**](#stats-functions): [antimode](#antimode), [count](#count), [distinct_count](#distinct_count), [kurtosis](#kurtosis), [maxlen](#maxlen), [mean](#mean), [meaneb](#meaneb), [median](#median), [minlen](#minlen), [mode](#mode), [null_count](#null_count), [percentile](#percentile), [percentiles](#percentiles), [skewness](#skewness), [sort_collection](#sort_collection), [sparkline](#sparkline), [stddev](#stddev), [sum](#sum), [sum2](#sum2), [sum3](#sum3), [sum4](#sum4), [variance](#variance).
|
||||
* [**String functions**](#string-functions): [base64_decode](#base64_decode), [base64_encode](#base64_encode), [capitalize](#capitalize), [clean_whitespace](#clean_whitespace), [collapse_whitespace](#collapse_whitespace), [contains](#contains), [format](#format), [gssub](#gssub), [gsub](#gsub), [hex_decode](#hex_decode), [hex_encode](#hex_encode), [index](#index), [latin1_to_utf8](#latin1_to_utf8), [leftpad](#leftpad), [lstrip](#lstrip), [regextract](#regextract), [regextract_or_else](#regextract_or_else), [rightpad](#rightpad), [rstrip](#rstrip), [ssub](#ssub), [strip](#strip), [strlen](#strlen), [strmatch](#strmatch), [strmatchx](#strmatchx), [sub](#sub), [substr](#substr), [substr0](#substr0), [substr1](#substr1), [tolower](#tolower), [toupper](#toupper), [truncate](#truncate), [unformat](#unformat), [unformatx](#unformatx), [utf8_to_latin1](#utf8_to_latin1), [\.](#dot).
|
||||
* [**System functions**](#system-functions): [exec](#exec), [hostname](#hostname), [next](#next), [os](#os), [stat](#stat), [system](#system), [version](#version).
|
||||
* [**Time functions**](#time-functions): [dhms2fsec](#dhms2fsec), [dhms2sec](#dhms2sec), [fsec2dhms](#fsec2dhms), [fsec2hms](#fsec2hms), [gmt2localtime](#gmt2localtime), [gmt2nsec](#gmt2nsec), [gmt2sec](#gmt2sec), [hms2fsec](#hms2fsec), [hms2sec](#hms2sec), [localtime2gmt](#localtime2gmt), [localtime2nsec](#localtime2nsec), [localtime2sec](#localtime2sec), [nsec2gmt](#nsec2gmt), [nsec2gmtdate](#nsec2gmtdate), [nsec2localdate](#nsec2localdate), [nsec2localtime](#nsec2localtime), [sec2dhms](#sec2dhms), [sec2gmt](#sec2gmt), [sec2gmtdate](#sec2gmtdate), [sec2hms](#sec2hms), [sec2localdate](#sec2localdate), [sec2localtime](#sec2localtime), [strfntime](#strfntime), [strfntime_local](#strfntime_local), [strftime](#strftime), [strftime_local](#strftime_local), [strpntime](#strpntime), [strpntime_local](#strpntime_local), [strptime](#strptime), [strptime_local](#strptime_local), [sysntime](#sysntime), [systime](#systime), [systimeint](#systimeint), [upntime](#upntime), [uptime](#uptime).
|
||||
|
|
@ -1156,6 +1156,15 @@ sort_collection (class=stats #args=1) This is a helper function for the percent
|
|||
</pre>
|
||||
|
||||
|
||||
### sparkline
|
||||
<pre class="pre-non-highlight-non-pair">
|
||||
sparkline (class=stats #args=1) Returns a string of Unicode block characters (one of ▁▂▃▄▅▆▇█ per element) representing the relative magnitudes of values in an array or map, for a compact ASCII/Unicode bar chart. Returns error for non-array/non-map types.
|
||||
Examples:
|
||||
sparkline([1,2,3,4,5,6,7,8]) is "▁▂▃▄▅▆▇█"
|
||||
sparkline([3,3,3]) is "▁▁▁"
|
||||
</pre>
|
||||
|
||||
|
||||
### stddev
|
||||
<pre class="pre-non-highlight-non-pair">
|
||||
stddev (class=stats #args=1) Returns the sample standard deviation of values in an array or map. Returns empty string AKA void for array/map of length less than two; returns error for non-array/non-map types.
|
||||
|
|
@ -1256,11 +1265,13 @@ contains([1,2,3], 2) gives (error)
|
|||
|
||||
### format
|
||||
<pre class="pre-non-highlight-non-pair">
|
||||
format (class=string #args=variadic) Using first argument as format string, interpolate remaining arguments in place of each "{}" in the format string. Too-few arguments are treated as the empty string; too-many arguments are discarded.
|
||||
format (class=string #args=variadic) Using first argument as format string, interpolate remaining arguments in place of each "{}" in the format string. Also supports 1-based positional placeholders such as "{1}", which allow arguments to be reused and/or reordered. Plain "{}" placeholders consume arguments sequentially, independently of any positional placeholders. Too-few arguments are treated as the empty string, as are positional placeholders exceeding the number of arguments; too-many arguments are discarded. "{0}" is an error value, since positional placeholders are 1-based.
|
||||
Examples:
|
||||
format("{}:{}:{}", 1,2) gives "1:2:".
|
||||
format("{}:{}:{}", 1,2,3) gives "1:2:3".
|
||||
format("{}:{}:{}", 1,2,3,4) gives "1:2:3".
|
||||
format("{}:{}:{}", 1,2) gives "1:2:".
|
||||
format("{}:{}:{}", 1,2,3) gives "1:2:3".
|
||||
format("{}:{}:{}", 1,2,3,4) gives "1:2:3".
|
||||
format("{1}:{2}:{1}", "a","b") gives "a:b:a".
|
||||
format("{2}{}:{1}{}", 3,4) gives "43:34".
|
||||
</pre>
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
@ -298,17 +298,29 @@ Available format strings for `strptime`:
|
|||
| Pattern | Description |
|
||||
|---------|-------------|
|
||||
| `%%` | A literal '%' character. |
|
||||
| `%a` | Weekday as locale’s abbreviated name. |
|
||||
| `%A` | Weekday as locale’s full name. |
|
||||
| `%b` | Month as locale’s abbreviated name. |
|
||||
| `%B` | Month as locale’s full name. |
|
||||
| `%c` | Locale’s date and time representation. Equivalent to `%a %b %e %H:%M:%S %Y`. |
|
||||
| `%d` | Day of the month as a zero-padded decimal number. |
|
||||
| `%D` | Equivalent to `%m/%d/%y`. |
|
||||
| `%e` | Day of the month as a space-padded decimal number. |
|
||||
| `%f` | Microsecond as a decimal number, zero-padded on the left. |
|
||||
| `%F` | Equivalent to `%Y-%m-%d`. |
|
||||
| `%h` | Same as `%b`. |
|
||||
| `%H` | Hour (24-hour clock) as a zero-padded decimal number. |
|
||||
| `%I` | Hour (12-hour clock) as a zero-padded decimal number. |
|
||||
| `%j` | Three-digit day of year, like 004 or 363. |
|
||||
| `%m` | Month as a zero-padded decimal number. |
|
||||
| `%M` | Minute as a zero-padded decimal number. |
|
||||
| `%p` | Locale’s equivalent of either AM or PM. |
|
||||
| `%r` | Equivalent to `%I:%M:%S %p`. |
|
||||
| `%R` | Equivalent to `%H:%M`. |
|
||||
| `%S` | Second as a zero-padded decimal number. |
|
||||
| `%T` | Equivalent to `%H:%M:%S`. |
|
||||
| `%x` | Locale’s date representation. Equivalent to `%m/%d/%y`. |
|
||||
| `%X` | Locale’s time representation. Equivalent to `%H:%M:%S`. |
|
||||
| `%y` | Year without century as a zero-padded decimal number. |
|
||||
| `%Y` | Year with century as a decimal number. |
|
||||
| `%z` | UTC offset in the form +HHMM or -HHMM. |
|
||||
|
|
@ -481,6 +493,74 @@ sec2dhms (class=time #args=1) Formats integer seconds as in sec2dhms(500000) =
|
|||
sec2hms (class=time #args=1) Formats integer seconds as in sec2hms(5000) = "01:23:20"
|
||||
</pre>
|
||||
|
||||
## Timezone troubleshooting
|
||||
|
||||
Since Miller applies your `TZ` environment variable at startup, a timezone problem can make _any_
|
||||
Miller command fail immediately -- even one that does nothing with times, such as `mlr --csv cat
|
||||
example.csv`. The symptom is an error message like
|
||||
|
||||
<pre class="pre-non-highlight-non-pair">
|
||||
mlr: TZ environment variable appears malformed: "..."
|
||||
</pre>
|
||||
|
||||
or, from Miller 6.6.0 and earlier, the underlying error from the Go `time` library:
|
||||
|
||||
<pre class="pre-non-highlight-non-pair">
|
||||
mlr : time: invalid location name
|
||||
</pre>
|
||||
|
||||
There are two distinct causes.
|
||||
|
||||
**Cause 1: `TZ` is set to a file path.** The Go `time` library, which Miller uses, accepts only
|
||||
[IANA timezone names](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) such as
|
||||
`America/New_York` (plus the special values `UTC` and `Local`). Unlike the C library, it does not
|
||||
accept file paths such as `/usr/share/zoneinfo/US/Eastern`, nor the POSIX form with a leading colon
|
||||
such as `:/etc/localtime` -- even though those work fine with other tools on your system (and worked
|
||||
with Miller 5, which used the C library):
|
||||
|
||||
<pre class="pre-highlight-in-pair">
|
||||
<b>export TZ=/usr/share/zoneinfo/US/Eastern</b>
|
||||
<b>mlr -n put 'end { print strftime_local(0, "%Y-%m-%d %H:%M:%S %Z") }'</b>
|
||||
</pre>
|
||||
<pre class="pre-non-highlight-in-pair">
|
||||
mlr: TZ environment variable appears malformed: "/usr/share/zoneinfo/US/Eastern"
|
||||
</pre>
|
||||
|
||||
The remedy is to set `TZ` to the zone's name rather than its file path -- here, `export
|
||||
TZ=US/Eastern` -- or to unset it (`unset TZ`, or `export TZ=""`), in which case Miller uses the
|
||||
system's local-time setting.
|
||||
|
||||
**Cause 2: the timezone database is missing.** Go programs such as Miller normally read timezone
|
||||
definitions from a database on the host system, and (unlike C programs) don't fall back to any
|
||||
built-in copy. If that database is absent, even a perfectly valid name fails:
|
||||
|
||||
<pre class="pre-highlight-in-pair">
|
||||
<b>mlr --csv cat example.csv</b>
|
||||
</pre>
|
||||
<pre class="pre-non-highlight-in-pair">
|
||||
mlr: TZ environment variable appears malformed: "Asia/Taipei"
|
||||
</pre>
|
||||
|
||||
The database is normally present on Linux and macOS (at `/usr/share/zoneinfo`), but is often missing on:
|
||||
|
||||
* Windows -- unless [Go itself is installed](https://go.dev/doc/install), since the Go toolchain ships a copy of the database (you don't need to _use_ Go; merely installing it suffices).
|
||||
* Cygwin, unless a package providing a full timezone database is installed.
|
||||
* Minimal Linux environments such as Alpine-based or `scratch` Docker containers, where the database is an optional package.
|
||||
|
||||
Remedies, in decreasing order of thoroughness:
|
||||
|
||||
* Install the timezone database: `apk add tzdata` on Alpine, `apt-get install tzdata` on
|
||||
Debian/Ubuntu, and so on; on Windows, install Go as noted above.
|
||||
* Point the `ZONEINFO` environment variable at a zoneinfo directory or zip file -- the Go `time`
|
||||
library checks it before the default system locations. For example, a Go installation provides
|
||||
one at `$GOROOT/lib/time/zoneinfo.zip`.
|
||||
* Avoid timezone-name lookups altogether: leave `TZ` unset (or set to `""` or `UTC`, which need no
|
||||
database), use the UTC-oriented functions such as
|
||||
[sec2gmt](reference-dsl-builtin-functions.md#sec2gmt) and
|
||||
[strftime](reference-dsl-builtin-functions.md#strftime), and use numeric fixed offsets such as
|
||||
`+0200` (matched by `%z` in [strptime](reference-dsl-builtin-functions.md#strptime) format
|
||||
strings) rather than timezone names.
|
||||
|
||||
## References
|
||||
|
||||
* Non-Miller-specific list of formatting characters for `strftime` and `strptime`: [https://devhints.io/strftime](https://devhints.io/strftime)
|
||||
|
|
|
|||
|
|
@ -230,17 +230,29 @@ Available format strings for `strptime`:
|
|||
| Pattern | Description |
|
||||
|---------|-------------|
|
||||
| `%%` | A literal '%' character. |
|
||||
| `%a` | Weekday as locale’s abbreviated name. |
|
||||
| `%A` | Weekday as locale’s full name. |
|
||||
| `%b` | Month as locale’s abbreviated name. |
|
||||
| `%B` | Month as locale’s full name. |
|
||||
| `%c` | Locale’s date and time representation. Equivalent to `%a %b %e %H:%M:%S %Y`. |
|
||||
| `%d` | Day of the month as a zero-padded decimal number. |
|
||||
| `%D` | Equivalent to `%m/%d/%y`. |
|
||||
| `%e` | Day of the month as a space-padded decimal number. |
|
||||
| `%f` | Microsecond as a decimal number, zero-padded on the left. |
|
||||
| `%F` | Equivalent to `%Y-%m-%d`. |
|
||||
| `%h` | Same as `%b`. |
|
||||
| `%H` | Hour (24-hour clock) as a zero-padded decimal number. |
|
||||
| `%I` | Hour (12-hour clock) as a zero-padded decimal number. |
|
||||
| `%j` | Three-digit day of year, like 004 or 363. |
|
||||
| `%m` | Month as a zero-padded decimal number. |
|
||||
| `%M` | Minute as a zero-padded decimal number. |
|
||||
| `%p` | Locale’s equivalent of either AM or PM. |
|
||||
| `%r` | Equivalent to `%I:%M:%S %p`. |
|
||||
| `%R` | Equivalent to `%H:%M`. |
|
||||
| `%S` | Second as a zero-padded decimal number. |
|
||||
| `%T` | Equivalent to `%H:%M:%S`. |
|
||||
| `%x` | Locale’s date representation. Equivalent to `%m/%d/%y`. |
|
||||
| `%X` | Locale’s time representation. Equivalent to `%H:%M:%S`. |
|
||||
| `%y` | Year without century as a zero-padded decimal number. |
|
||||
| `%Y` | Year with century as a decimal number. |
|
||||
| `%z` | UTC offset in the form +HHMM or -HHMM. |
|
||||
|
|
@ -358,6 +370,69 @@ GENMD-RUN-COMMAND
|
|||
mlr -F | grep hms
|
||||
GENMD-EOF
|
||||
|
||||
## Timezone troubleshooting
|
||||
|
||||
Since Miller applies your `TZ` environment variable at startup, a timezone problem can make _any_
|
||||
Miller command fail immediately -- even one that does nothing with times, such as `mlr --csv cat
|
||||
example.csv`. The symptom is an error message like
|
||||
|
||||
<pre class="pre-non-highlight-non-pair">
|
||||
mlr: TZ environment variable appears malformed: "..."
|
||||
</pre>
|
||||
|
||||
or, from Miller 6.6.0 and earlier, the underlying error from the Go `time` library:
|
||||
|
||||
<pre class="pre-non-highlight-non-pair">
|
||||
mlr : time: invalid location name
|
||||
</pre>
|
||||
|
||||
There are two distinct causes.
|
||||
|
||||
**Cause 1: `TZ` is set to a file path.** The Go `time` library, which Miller uses, accepts only
|
||||
[IANA timezone names](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) such as
|
||||
`America/New_York` (plus the special values `UTC` and `Local`). Unlike the C library, it does not
|
||||
accept file paths such as `/usr/share/zoneinfo/US/Eastern`, nor the POSIX form with a leading colon
|
||||
such as `:/etc/localtime` -- even though those work fine with other tools on your system (and worked
|
||||
with Miller 5, which used the C library):
|
||||
|
||||
GENMD-RUN-COMMAND-TOLERATING-ERROR
|
||||
export TZ=/usr/share/zoneinfo/US/Eastern
|
||||
mlr -n put 'end { print strftime_local(0, "%Y-%m-%d %H:%M:%S %Z") }'
|
||||
GENMD-EOF
|
||||
|
||||
The remedy is to set `TZ` to the zone's name rather than its file path -- here, `export
|
||||
TZ=US/Eastern` -- or to unset it (`unset TZ`, or `export TZ=""`), in which case Miller uses the
|
||||
system's local-time setting.
|
||||
|
||||
**Cause 2: the timezone database is missing.** Go programs such as Miller normally read timezone
|
||||
definitions from a database on the host system, and (unlike C programs) don't fall back to any
|
||||
built-in copy. If that database is absent, even a perfectly valid name fails:
|
||||
|
||||
GENMD-CARDIFY-HIGHLIGHT-ONE
|
||||
mlr --csv cat example.csv
|
||||
mlr: TZ environment variable appears malformed: "Asia/Taipei"
|
||||
GENMD-EOF
|
||||
|
||||
The database is normally present on Linux and macOS (at `/usr/share/zoneinfo`), but is often missing on:
|
||||
|
||||
* Windows -- unless [Go itself is installed](https://go.dev/doc/install), since the Go toolchain ships a copy of the database (you don't need to _use_ Go; merely installing it suffices).
|
||||
* Cygwin, unless a package providing a full timezone database is installed.
|
||||
* Minimal Linux environments such as Alpine-based or `scratch` Docker containers, where the database is an optional package.
|
||||
|
||||
Remedies, in decreasing order of thoroughness:
|
||||
|
||||
* Install the timezone database: `apk add tzdata` on Alpine, `apt-get install tzdata` on
|
||||
Debian/Ubuntu, and so on; on Windows, install Go as noted above.
|
||||
* Point the `ZONEINFO` environment variable at a zoneinfo directory or zip file -- the Go `time`
|
||||
library checks it before the default system locations. For example, a Go installation provides
|
||||
one at `$GOROOT/lib/time/zoneinfo.zip`.
|
||||
* Avoid timezone-name lookups altogether: leave `TZ` unset (or set to `""` or `UTC`, which need no
|
||||
database), use the UTC-oriented functions such as
|
||||
[sec2gmt](reference-dsl-builtin-functions.md#sec2gmt) and
|
||||
[strftime](reference-dsl-builtin-functions.md#strftime), and use numeric fixed offsets such as
|
||||
`+0200` (matched by `%z` in [strptime](reference-dsl-builtin-functions.md#strptime) format
|
||||
strings) rather than timezone names.
|
||||
|
||||
## References
|
||||
|
||||
* Non-Miller-specific list of formatting characters for `strftime` and `strptime`: [https://devhints.io/strftime](https://devhints.io/strftime)
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
@ -415,6 +415,7 @@ These are flags which are applicable to PPRINT format.
|
|||
* `--fixed {string}`: Fixed width specification. One of 'widths:<col1-width>,<col2-width>,...', left-align, left-align-multi-word, right-align, right-align-multi-word
|
||||
* `--fw {string}`: Shortcut for --fixed left-align-multi-word
|
||||
* `--right`: Right-justifies all fields for PPRINT output.
|
||||
* `--right-align-numeric`: Right-justifies fields with numeric values for PPRINT output, leaving other fields left-justified. Headers are right-justified over columns whose values are all numeric, so that header and data share the same alignment. Also applies to markdown output, where numeric columns get right-alignment markers (`---:`) in the header-separator line.
|
||||
|
||||
## Profiling flags
|
||||
|
||||
|
|
@ -451,6 +452,9 @@ Notes about line endings:
|
|||
* Default line endings (`--irs` and `--ors`) are newline
|
||||
which is interpreted to accept carriage-return/newline files (e.g. on Windows)
|
||||
for input, and to produce platform-appropriate line endings on output.
|
||||
* For CSV, CSV-lite, TSV, and TSV-lite output, ORS may be either newline (the
|
||||
default) or carriage-return/newline: e.g. `--ors crlf` or `--ors '\r\n'`
|
||||
for RFC-4180-style line endings on any platform.
|
||||
|
||||
Notes about all other separators:
|
||||
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<!--- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
|
||||
<!-- PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. -->
|
||||
<div>
|
||||
<span class="quicklinks">
|
||||
Quick links:
|
||||
|
|
|
|||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Add table
Add a link
Reference in a new issue