Mirrors/miller
1
0
Fork 0
mirror of https://github.com/johnkerl/miller.git synced 2026-07-17 16:38:54 +00:00
Code Issues Projects Releases Wiki Activity

Tier-2 structured verb options: OptionSpec, initial migration (#2098) (#2111)

Browse source 
* Tier-2 structured verb options: OptionSpec, initial migration (#2098)

PR 3 of the AI-friendly roadmap (plans/plan-2098-llm.md).

Infrastructure:
- Add OptionSpec{Flag,Arg,Type,Desc,Repeatable,Values} to
  pkg/transformers/aaa_record_transformer.go alongside TransformerSetup.
  Type is one of: bool, string, int, float, csv-list, regex, filename,
  format, enum. For type=="enum", Values lists the valid choices.
- Add Options []OptionSpec to TransformerSetup (nil = not yet migrated).
- Emit Options in VerbInfoForJSON (omitempty so unmigrated verbs stay
  backward-compatible; agents check key presence for Tier-2 availability).
  UsageText is always present as the Tier-1 prose fallback.
- Add VerbOptionsNilCheck() in aaa_verb_options_check.go: progress report
  of migrated vs. unmigrated verbs, analogous to FLAG_TABLE.NilCheck().
- Wire verb-options-nil-check into mlr help (internal/docgen section).

Initial migration (5/70 verbs):
- nothing: empty Options (no verb-specific options, explicitly migrated)
- cat: -n (bool), -N (string), -g (csv-list), --filename, --filenum (bool)
- head: -g (csv-list), -n (int)
- tail: -g (csv-list), -n (int)
- tee: -a, -p (bool)

Tests:
- 5 new unit tests in aaa_transformer_json_test.go covering migrated/
  unmigrated paths, field population, JSON round-trip, and key-presence.
- Regression test case 0003: mlr help verb-options-nil-check golden output.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* Migrate all 70 verbs to structured OptionSpec; bump catalog schema to v2

Completes the Tier-2 migration started in the previous commit. Every verb
in TRANSFORMER_LOOKUP_TABLE now has a non-nil Options field.

- Workflow-migrated all 65 remaining verbs. Each Setup var now carries
  Options: []OptionSpec{...} with Flag/Arg/Type/Desc fields. Verbs with
  no verb-specific options (altkv, check, group-like, nothing, etc.) use
  an empty slice to signal "migrated but no options."
- Drop `omitempty` from VerbInfoForJSON.Options: empty slices were silently
  dropped, making migrated-no-option verbs indistinguishable from unmigrated
  ones in JSON. Without omitempty: null=unmigrated, []=migrated-no-options,
  [...]= migrated-with-options. Bump catalogSchemaVersion 1→2 for this shape
  change.
- Replace the two "unmigrated-verb" unit tests (which used stats1 as an
  example) with TestAllVerbsFullyMigrated (asserts every verb has non-nil
  Options) and TestAllVerbsHaveOptionsKeyInJSON (asserts every migrated
  verb emits the "options" key in JSON).
- Regenerate test/cases/cli-help/0003/expout: now reads
  "Verb options migration: 70/70 migrated."

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* remove a transitional helper

* git rms

* Render verb usage Options blocks from structured OptionSpec

Each verb's usage message and its Tier-2 OptionSpec list previously
duplicated the option text. New WriteVerbOptions (aaa_verb_usage.go)
renders the "Options:" block from the specs: aligned flag column,
descriptions word-wrapped at 80, uniform trailing -h|--help line.

- OptionSpec gains Aliases (JSON "aliases") so long-form spellings
  like join's --lk|--left-keep-field-names survive in both outputs
- All 70 verbs migrated; options literals hoisted to package-level
  vars (usage funcs can't reference their Setup var without a Go
  init cycle)
- Hand-written per-option details the specs had condensed away are
  merged into Desc, enriching the JSON catalog
- Non-option prose (examples, cross-references, dynamic accumulator
  listings) kept verbatim
- Regenerated the six usage-embedding regression expectations and
  the two affected doc pages

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Fix pre-existing usage-text bugs surfaced by the OptionSpec migration

- gap: usage said "One of -f or -g is required" but the parser takes
  -n or -g
- seqgen: drop description line copy-pasted from cat ("Passes input
  records directly to output...") which contradicted "Discards the
  input record stream"
- utf8-to-latin1: description read inverted ("from Latin-1 to UTF-8")
- sec2gmtdate: usage said "../c/mlr" instead of "mlr"
- top: document the accepted-but-undocumented --max flag
- stats2: add linreg-pca to the -a enum values, matching the runtime
  accumulator table

Regression expectations and docs regenerated accordingly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Fix check usage sentence order; stats1 usage blank line to usage stream

- check: the description's second and third lines were swapped,
  reading "Consumes records without printing any output, / Useful for
  doing a well-formatted check on input data. / with the exception
  that warnings are printed to stderr."
- stats1: a bare fmt.Println() in the usage func wrote its blank line
  to process stdout instead of the usage output stream

Regression expectation and docs regenerated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
John Kerl 2026-07-03 14:27:23 -04:00 • committed by GitHub
parent 12c96298b9
commit f637633420
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
79 changed files with 2131 additions and 1519 deletions
Download patch file Download diff file Expand all files Collapse all files

56
docs/src/online-help.md
View file

@ -124,13 +124,15 @@ To search by substring, use `mlr help find`:
</pre>
<pre class="pre-non-highlight-in-pair">
sec2gmtdate
Usage: ../c/mlr sec2gmtdate {comma-separated list of field names}
Usage: mlr sec2gmtdate {comma-separated list of field names}
Replaces a numeric field representing seconds since the epoch with the
corresponding GMT year-month-day timestamp; leaves non-numbers as-is.
This is nothing more than a keystroke-saver for the sec2gmtdate function:
../c/mlr sec2gmtdate time1,time2
mlr sec2gmtdate time1,time2
is the same as
../c/mlr put '$time1=sec2gmtdate($time1);$time2=sec2gmtdate($time2)'
mlr put '$time1=sec2gmtdate($time1);$time2=sec2gmtdate($time2)'
Options:
-h|--help Show this message.
sec2gmt
Usage: mlr sec2gmt [options] {comma-separated list of field names}
Replaces a numeric field representing seconds since the epoch with the
@ -140,10 +142,18 @@ more than a keystroke-saver for the sec2gmt function:
is the same as
mlr put '$time1 = sec2gmt($time1); $time2 = sec2gmt($time2)'
Options:
-1 through -9: format the seconds using 1..9 decimal places, respectively.
--millis Input numbers are treated as milliseconds since the epoch.
--micros Input numbers are treated as microseconds since the epoch.
--nanos Input numbers are treated as nanoseconds since the epoch.
-1 Format seconds with 1 decimal place.
-2 Format seconds with 2 decimal places.
-3 Format seconds with 3 decimal places.
-4 Format seconds with 4 decimal places.
-5 Format seconds with 5 decimal places.
-6 Format seconds with 6 decimal places.
-7 Format seconds with 7 decimal places.
-8 Format seconds with 8 decimal places.
-9 Format seconds with 9 decimal places.
--millis Input numbers are treated as milliseconds since the epoch.
--micros Input numbers are treated as microseconds since the epoch.
--nanos Input numbers are treated as nanoseconds since the epoch.
-h|--help Show this message.
gmt2localtime (class=time #args=1,2) Convert from a GMT-time string to a local-time string. Consulting $TZ unless second argument is supplied.
Examples:
@ -197,11 +207,12 @@ Usage: mlr cat [options]
Passes input records directly to output. Most useful for format conversion.
Options:
-n Prepend field "n" to each record with record-counter starting at 1.
-N {name} Prepend field {name} to each record with record-counter starting at 1.
-g {a,b,c} Optional group-by-field names for counters, e.g. a,b,c
-N {name} Prepend field {name} to each record with record-counter starting at
1.
-g {a,b,c} Optional group-by-field names for counters, e.g. a,b,c.
--filename Prepend current filename to each record.
--filenum Prepend current filenum (1-up) to each record.
-h|--help Show this message.
-h|--help Show this message.
</pre>
<pre class="pre-highlight-in-pair">
@ -227,17 +238,20 @@ specified sort order.) The sort is stable: records that compare equal will sort
in the order they were encountered in the input record stream.
Options:
-f {comma-separated field names} Lexical ascending
-r {comma-separated field names} Lexical descending
-c {comma-separated field names} Case-folded lexical ascending
-cr {comma-separated field names} Case-folded lexical descending
-n {comma-separated field names} Numerical ascending; nulls sort last
-nf {comma-separated field names} Same as -n
-nr {comma-separated field names} Numerical descending; nulls sort first
-t {comma-separated field names} Natural ascending
-b Move sort fields to start of record, as in reorder -b
-tr|-rt {comma-separated field names} Natural descending
-h|--help Show this message.
-f {a,b,c} Lexical ascending sort on the specified field names.
-r {a,b,c} Lexical descending sort on the specified field names.
-c {a,b,c} Case-folded lexical ascending sort on the specified field names.
-cr {a,b,c} Case-folded lexical descending sort on the specified field
names.
-n {a,b,c} Numerical ascending sort on the specified field names; nulls
sort last.
-nf {a,b,c} Same as -n.
-nr {a,b,c} Numerical descending sort on the specified field names; nulls
sort first.
-t {a,b,c} Natural ascending sort on the specified field names.
-b Move sort fields to start of record, as in reorder -b.
-tr|-rt {a,b,c} Natural descending sort on the specified field names.
-h|--help Show this message.
Example:
mlr sort -f a,b -nr x,y,z

995
docs/src/reference-verbs.md
View file

File diff suppressed because it is too large Load diff

2
pkg/terminals/help/entry_json.go
View file

@ -28,7 +28,7 @@ import (
// catalogSchemaVersion is bumped whenever the shape of the JSON catalog
// changes. Agents and tools can use this (together with mlr_version) as a
// cache key: re-fetch only when either value changes.
const catalogSchemaVersion = 1
const catalogSchemaVersion = 2
// CatalogForJSON is the top-level document emitted by `mlr help --as-json`
// with no further topic: the entire help catalog in one machine-readable

26
pkg/transformers/aaa_record_transformer.go
View file

@ -49,6 +49,27 @@ type TransformerParseCLIFunc func(
doConstruct bool, // false for first pass of CLI-parse, true for second pass
) (RecordTransformer, error)
// OptionSpec describes one verb option in a machine-readable form. It is the
// building block for the structured Tier-2 catalog (PR3 of the AI-friendly
// roadmap). Verbs migrate to this incrementally; verbs with Options == nil
// fall back to the prose UsageText in the JSON catalog.
//
// Type is one of: bool, string, int, float, csv-list, regex, filename,
// format, enum. For Type=="enum", Values contains every valid choice.
// Arg is the placeholder shown in usage text, e.g. "{n}" or "{a,b,c}";
// it is empty for bool flags. Repeatable marks flags that may appear
// multiple times on the command line. Aliases lists alternate spellings
// of Flag (e.g. "--sorted-input" for "-s").
type OptionSpec struct {
Flag string `json:"flag"`
Aliases []string `json:"aliases,omitempty"`
Arg string `json:"arg,omitempty"`
Type string `json:"type"`
Desc string `json:"desc"`
Repeatable bool `json:"repeatable,omitempty"`
Values []string `json:"values,omitempty"`
}
type TransformerSetup struct {
Verb string
UsageFunc TransformerUsageFunc
@ -59,6 +80,11 @@ type TransformerSetup struct {
// own. (The seqgen verb probably should have been designed as a zero-file
// record "reader" object, rather than a verb, alas.)
IgnoresInput bool
// Options is the structured option list for this verb. nil means the verb
// has not yet been migrated to Tier-2; agents fall back to UsageText.
// An explicitly empty slice means the verb accepts no options.
Options []OptionSpec
}
// HandleDefaultDownstreamDone is a utility function for most verbs other than

27
pkg/transformers/aaa_transformer_json.go
View file

@ -1,12 +1,10 @@
// Machine-readable (JSON) accessors over the verb (transformer) catalog, for
// `mlr help --as-json` and similar tooling.
//
// Tier-1 caveat: unlike functions and flags, verb options are not held in any
// structured form -- each verb hand-writes a UsageFunc that prints prose. So
// here we expose the verb name, a one-line summary, and the captured raw usage
// text. Structured per-verb options (flag/arg/type) are a planned follow-on
// (an optional Options field on TransformerSetup); when present they can be
// emitted alongside UsageText.
// Tier-1: every verb exposes Summary and UsageText (prose fallback).
// Tier-2: verbs that have been migrated populate Options on their
// TransformerSetup; those entries appear as a structured option list in the
// JSON and agents no longer need to scrape the prose.
package transformers
@ -17,14 +15,16 @@ import (
"strings"
)
// VerbInfoForJSON is the structured view of a single verb. Summary is the first
// non-"Usage:" line of the usage text; UsageText is the verb's full usage
// output verbatim (the Tier-1 fallback for not-yet-structured options).
// VerbInfoForJSON is the structured view of a single verb.
// - Options is non-nil for Tier-2 verbs that have been migrated; agents
// should prefer it when available.
// - UsageText is always present as the Tier-1 prose fallback.
type VerbInfoForJSON struct {
Name string `json:"name"`
Summary string `json:"summary"`
IgnoresInput bool `json:"ignores_input"`
UsageText string `json:"usage_text"`
Name string `json:"name"`
Summary string `json:"summary"`
IgnoresInput bool `json:"ignores_input"`
Options []OptionSpec `json:"options"`
UsageText string `json:"usage_text"`
}
// captureUsageFunc runs a verb's UsageFunc against a pipe and returns what it
@ -70,6 +70,7 @@ func makeVerbInfoForJSON(setup *TransformerSetup) *VerbInfoForJSON {
Name: setup.Verb,
Summary: summarizeUsageText(usageText),
IgnoresInput: setup.IgnoresInput,
Options: setup.Options, // nil for unmigrated verbs; omitted from JSON via omitempty
UsageText: strings.TrimRight(usageText, "\n"),
}
}

138
pkg/transformers/aaa_transformer_json_test.go Normal file
View file

@ -0,0 +1,138 @@
package transformers
import (
"encoding/json"
"testing"
)
// TestMigratedVerbsHaveOptions checks that verbs which have been given
// structured Options emit them in their JSON catalog entry (non-nil, non-empty
// for verbs that actually have options).
func TestMigratedVerbsHaveOptions(t *testing.T) {
cases := []struct {
verb string
wantNonNilOpts bool
wantMinOptCount int
}{
{"nothing", true, 0}, // explicitly migrated, no verb-specific options
{"cat", true, 5},
{"head", true, 2},
{"tail", true, 2},
{"tee", true, 2},
}
for _, tc := range cases {
info := GetVerbInfoForJSON(tc.verb)
if info == nil {
t.Errorf("verb %q not found in catalog", tc.verb)
continue
}
if tc.wantNonNilOpts && info.Options == nil {
t.Errorf("verb %q: Options is nil, want non-nil (migrated)", tc.verb)
continue
}
if len(info.Options) < tc.wantMinOptCount {
t.Errorf("verb %q: got %d options, want >= %d", tc.verb, len(info.Options), tc.wantMinOptCount)
}
}
}
// TestAllVerbsFullyMigrated asserts the full-migration invariant: every verb
// in the catalog has a non-nil Options slice and a non-empty UsageText.
// This is the Tier-2 completion check; it fails if a new verb is added without
// populating Options.
func TestAllVerbsFullyMigrated(t *testing.T) {
for i := range TRANSFORMER_LOOKUP_TABLE {
setup := &TRANSFORMER_LOOKUP_TABLE[i]
if setup.Options == nil {
t.Errorf("verb %q has nil Options (not yet migrated to Tier-2)", setup.Verb)
}
info := GetVerbInfoForJSON(setup.Verb)
if info == nil {
t.Errorf("verb %q: GetVerbInfoForJSON returned nil", setup.Verb)
continue
}
if info.UsageText == "" {
t.Errorf("verb %q: UsageText is empty", setup.Verb)
}
}
}
// TestOptionSpecFieldsPopulated verifies that migrated OptionSpec entries have
// the required fields filled in.
func TestOptionSpecFieldsPopulated(t *testing.T) {
info := GetVerbInfoForJSON("cat")
if info == nil {
t.Fatal("cat not found")
}
for _, opt := range info.Options {
if opt.Flag == "" {
t.Errorf("cat: OptionSpec with empty Flag: %+v", opt)
}
if opt.Type == "" {
t.Errorf("cat: OptionSpec %q has empty Type", opt.Flag)
}
if opt.Desc == "" {
t.Errorf("cat: OptionSpec %q has empty Desc", opt.Flag)
}
}
}
// TestOptionsRoundTripJSON verifies that OptionSpec survives JSON
// marshal/unmarshal with all fields intact.
func TestOptionsRoundTripJSON(t *testing.T) {
info := GetVerbInfoForJSON("cat")
if info == nil {
t.Fatal("cat not found")
}
b, err := json.Marshal(info)
if err != nil {
t.Fatalf("marshal: %v", err)
}
var roundTrip VerbInfoForJSON
if err := json.Unmarshal(b, &roundTrip); err != nil {
t.Fatalf("unmarshal: %v", err)
}
if len(roundTrip.Options) != len(info.Options) {
t.Errorf("options count: got %d, want %d", len(roundTrip.Options), len(info.Options))
}
for i, orig := range info.Options {
got := roundTrip.Options[i]
if got.Flag != orig.Flag || got.Type != orig.Type || got.Desc != orig.Desc {
t.Errorf("option[%d] mismatch: got %+v, want %+v", i, got, orig)
}
}
}
// TestAllVerbsHaveOptionsKeyInJSON verifies that every fully-migrated verb
// emits an "options" key in its JSON output (even when the list is empty),
// so agents can rely on key presence as the Tier-2 signal.
func TestAllVerbsHaveOptionsKeyInJSON(t *testing.T) {
for i := range TRANSFORMER_LOOKUP_TABLE {
setup := &TRANSFORMER_LOOKUP_TABLE[i]
if setup.Options == nil {
continue // not yet migrated; skip rather than double-fail
}
info := GetVerbInfoForJSON(setup.Verb)
if info == nil {
t.Errorf("verb %q: GetVerbInfoForJSON returned nil", setup.Verb)
continue
}
b, err := json.Marshal(info)
if err != nil {
t.Errorf("verb %q: marshal error: %v", setup.Verb, err)
continue
}
var raw map[string]any
if err := json.Unmarshal(b, &raw); err != nil {
t.Errorf("verb %q: unmarshal error: %v", setup.Verb, err)
continue
}
if _, ok := raw["options"]; !ok {
t.Errorf("verb %q: missing \"options\" key in JSON", setup.Verb)
}
}
}

73
pkg/transformers/aaa_verb_usage.go Normal file
View file

@ -0,0 +1,73 @@
// Renders the "Options:" block of a verb's usage message from its structured
// []OptionSpec, so that the prose usage text and the JSON catalog (Tier-2)
// stay in sync -- each verb's option list is written once, in its OptionSpec.
package transformers
import (
"fmt"
"os"
"strings"
)
// verbUsageLineWidth is the wrap width for generated option descriptions.
const verbUsageLineWidth = 80
const helpFlagHead = "-h|--help"
// WriteVerbOptions writes the "Options:" block for a verb's usage message
// from its structured option list. Flag heads are aligned into a single
// column; descriptions are word-wrapped with continuation lines indented to
// the description column. Since every verb supports -h|--help, that line is
// appended uniformly.
func WriteVerbOptions(o *os.File, options []OptionSpec) {
fmt.Fprintf(o, "Options:\n")
heads := make([]string, len(options))
maxHeadLen := len(helpFlagHead)
for i := range options {
opt := &options[i]
head := opt.Flag
if len(opt.Aliases) > 0 {
head = opt.Flag + "|" + strings.Join(opt.Aliases, "|")
}
if opt.Arg != "" {
head += " " + opt.Arg
}
heads[i] = head
if len(head) > maxHeadLen {
maxHeadLen = len(head)
}
}
descColumn := maxHeadLen + 1
for i := range options {
writeVerbOptionLine(o, heads[i], options[i].Desc, descColumn)
}
writeVerbOptionLine(o, helpFlagHead, "Show this message.", descColumn)
}
// writeVerbOptionLine writes one option's head and word-wrapped description,
// with continuation lines indented to descColumn.
func writeVerbOptionLine(o *os.File, head string, desc string, descColumn int) {
if desc == "" {
fmt.Fprintf(o, "%s\n", head)
return
}
indent := strings.Repeat(" ", descColumn)
line := fmt.Sprintf("%-*s", descColumn, head)
lineHasDesc := false
for word := range strings.FieldsSeq(desc) {
if lineHasDesc && len(line)+1+len(word) > verbUsageLineWidth {
fmt.Fprintf(o, "%s\n", line)
line = indent
lineHasDesc = false
}
if lineHasDesc {
line += " "
}
line += word
lineHasDesc = true
}
fmt.Fprintf(o, "%s\n", line)
}

6
pkg/transformers/altkv.go
View file

@ -13,11 +13,14 @@ import (
const verbNameAltkv = "altkv"
var altkvOptions = []OptionSpec{}
var AltkvSetup = TransformerSetup{
Verb: verbNameAltkv,
UsageFunc: transformerAltkvUsage,
ParseCLIFunc: transformerAltkvParseCLI,
IgnoresInput: false,
Options: altkvOptions,
}
func transformerAltkvUsage(
@ -25,8 +28,7 @@ func transformerAltkvUsage(
) {
fmt.Fprintf(o, "Usage: %s %s [options]\n", "mlr", verbNameAltkv)
fmt.Fprintf(o, "Given fields with values of the form a,b,c,d,e,f emits a=b,c=d,e=f pairs.\n")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
WriteVerbOptions(o, altkvOptions)
}
func transformerAltkvParseCLI(

24
pkg/transformers/bar.go
View file

@ -20,11 +20,23 @@ const barDefaultWidth = int64(40)
const verbNameBar = "bar"
var barOptions = []OptionSpec{
{Flag: "-f", Arg: "{a,b,c}", Type: "csv-list", Desc: "Field names to convert to bars."},
{Flag: "--lo", Arg: "{lo}", Type: "float", Desc: "Lower-limit value for min-width bar: default '0.000000'."},
{Flag: "--hi", Arg: "{hi}", Type: "float", Desc: "Upper-limit value for max-width bar: default '100.000000'."},
{Flag: "-w", Arg: "{n}", Type: "int", Desc: "Bar-field width: default '40'."},
{Flag: "--auto", Type: "bool", Desc: "Automatically computes limits, ignoring --lo and --hi. Holds all records in memory before producing any output."},
{Flag: "-c", Arg: "{character}", Type: "string", Desc: "Fill character: default '*'."},
{Flag: "-x", Arg: "{character}", Type: "string", Desc: "Out-of-bounds character: default '#'."},
{Flag: "-b", Arg: "{character}", Type: "string", Desc: "Blank character: default '.'."},
}
var BarSetup = TransformerSetup{
Verb: verbNameBar,
UsageFunc: transformerBarUsage,
ParseCLIFunc: transformerBarParseCLI,
IgnoresInput: false,
Options: barOptions,
}
func transformerBarUsage(
@ -33,19 +45,9 @@ func transformerBarUsage(
fmt.Fprintf(o, "Usage: %s %s [options]\n", "mlr", verbNameBar)
fmt.Fprintf(o, "Replaces a numeric field with a number of asterisks, allowing for cheesy\n")
fmt.Fprintf(o, "bar plots. These align best with --opprint or --oxtab output format.\n")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-f {a,b,c} Field names to convert to bars.\n")
fmt.Fprintf(o, "--lo {lo} Lower-limit value for min-width bar: default '%f'.\n", barDefaultLo)
fmt.Fprintf(o, "--hi {hi} Upper-limit value for max-width bar: default '%f'.\n", barDefaultHi)
fmt.Fprintf(o, "-w {n} Bar-field width: default '%d'.\n", barDefaultWidth)
fmt.Fprintf(o, "--auto Automatically computes limits, ignoring --lo and --hi.\n")
fmt.Fprintf(o, " Holds all records in memory before producing any output.\n")
fmt.Fprintf(o, "-c {character} Fill character: default '%s'.\n", barDefaultFillString)
fmt.Fprintf(o, "-x {character} Out-of-bounds character: default '%s'.\n", barDefaultOOBString)
fmt.Fprintf(o, "-b {character} Blank character: default '%s'.\n", barDefaultBlankString)
WriteVerbOptions(o, barOptions)
fmt.Fprintf(o, "Nominally the fill, out-of-bounds, and blank characters will be strings of length 1.\n")
fmt.Fprintf(o, "However you can make them all longer if you so desire.\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
}
func transformerBarParseCLI(

12
pkg/transformers/bootstrap.go
View file

@ -12,11 +12,16 @@ import (
const verbNameBootstrap = "bootstrap"
var bootstrapOptions = []OptionSpec{
{Flag: "-n", Arg: "{n}", Type: "int", Desc: "Number of samples to output. Defaults to number of input records. Must be non-negative."},
}
var BootstrapSetup = TransformerSetup{
Verb: verbNameBootstrap,
UsageFunc: transformerBootstrapUsage,
ParseCLIFunc: transformerBootstrapParseCLI,
IgnoresInput: false,
Options: bootstrapOptions,
}
func transformerBootstrapUsage(
@ -27,12 +32,7 @@ func transformerBootstrapUsage(
`Emits an n-sample, with replacement, of the input records.
See also %s sample and %s shuffle.
`, "mlr", "mlr")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o,
` -n Number of samples to output. Defaults to number of input records.
Must be non-negative.
`)
fmt.Fprintf(o, "-h|--help Show this message.\n")
WriteVerbOptions(o, bootstrapOptions)
}
func transformerBootstrapParseCLI(

21
pkg/transformers/case.go
View file

@ -16,11 +16,22 @@ import (
const verbNameCase = "case"
var caseOptions = []OptionSpec{
{Flag: "-k", Type: "bool", Desc: "Case only keys, not keys and values."},
{Flag: "-v", Type: "bool", Desc: "Case only values, not keys and values."},
{Flag: "-f", Arg: "{a,b,c}", Type: "csv-list", Desc: "Specify which field names to case (default: all)."},
{Flag: "-u", Type: "bool", Desc: "Convert to uppercase."},
{Flag: "-l", Type: "bool", Desc: "Convert to lowercase."},
{Flag: "-s", Type: "bool", Desc: "Convert to sentence case (capitalize first letter)."},
{Flag: "-t", Type: "bool", Desc: "Convert to title case (capitalize words)."},
}
var CaseSetup = TransformerSetup{
Verb: verbNameCase,
UsageFunc: transformerCaseUsage,
ParseCLIFunc: transformerCaseParseCLI,
IgnoresInput: false,
Options: caseOptions,
}
const (
@ -36,15 +47,7 @@ func transformerCaseUsage(
) {
fmt.Fprintf(o, "Usage: %s %s [options]\n", "mlr", verbNameCase)
fmt.Fprintf(o, "Uppercases strings in record keys and/or values.\n")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-k Case only keys, not keys and values.\n")
fmt.Fprintf(o, "-v Case only values, not keys and values.\n")
fmt.Fprintf(o, "-f {a,b,c} Specify which field names to case (default: all)\n")
fmt.Fprintf(o, "-u Convert to uppercase\n")
fmt.Fprintf(o, "-l Convert to lowercase\n")
fmt.Fprintf(o, "-s Convert to sentence case (capitalize first letter)\n")
fmt.Fprintf(o, "-t Convert to title case (capitalize words)\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
WriteVerbOptions(o, caseOptions)
}
func transformerCaseParseCLI(

17
pkg/transformers/cat.go
View file

@ -12,11 +12,20 @@ import (
const verbNameCat = "cat"
var catOptions = []OptionSpec{
{Flag: "-n", Type: "bool", Desc: "Prepend field \"n\" to each record with record-counter starting at 1."},
{Flag: "-N", Arg: "{name}", Type: "string", Desc: "Prepend field {name} to each record with record-counter starting at 1."},
{Flag: "-g", Arg: "{a,b,c}", Type: "csv-list", Desc: "Optional group-by-field names for counters, e.g. a,b,c."},
{Flag: "--filename", Type: "bool", Desc: "Prepend current filename to each record."},
{Flag: "--filenum", Type: "bool", Desc: "Prepend current filenum (1-up) to each record."},
}
var CatSetup = TransformerSetup{
Verb: verbNameCat,
UsageFunc: transformerCatUsage,
ParseCLIFunc: transformerCatParseCLI,
IgnoresInput: false,
Options: catOptions,
}
func transformerCatUsage(
@ -24,13 +33,7 @@ func transformerCatUsage(
) {
fmt.Fprintf(o, "Usage: %s %s [options]\n", "mlr", verbNameCat)
fmt.Fprintf(o, "Passes input records directly to output. Most useful for format conversion.\n")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-n Prepend field \"n\" to each record with record-counter starting at 1.\n")
fmt.Fprintf(o, "-N {name} Prepend field {name} to each record with record-counter starting at 1.\n")
fmt.Fprintf(o, "-g {a,b,c} Optional group-by-field names for counters, e.g. a,b,c\n")
fmt.Fprintf(o, "--filename Prepend current filename to each record.\n")
fmt.Fprintf(o, "--filenum Prepend current filenum (1-up) to each record.\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
WriteVerbOptions(o, catOptions)
}
func transformerCatParseCLI(

8
pkg/transformers/check.go
View file

@ -11,11 +11,14 @@ import (
const verbNameCheck = "check"
var checkOptions = []OptionSpec{}
var CheckSetup = TransformerSetup{
Verb: verbNameCheck,
UsageFunc: transformerCheckUsage,
ParseCLIFunc: transformerCheckParseCLI,
IgnoresInput: false,
Options: checkOptions,
}
func transformerCheckUsage(
@ -23,13 +26,12 @@ func transformerCheckUsage(
) {
fmt.Fprintf(o, "Usage: %s %s [options]\n", "mlr", verbNameCheck)
fmt.Fprintf(o, "Consumes records without printing any output,\n")
fmt.Fprintf(o, "Useful for doing a well-formatted check on input data.\n")
fmt.Fprintf(o, "with the exception that warnings are printed to stderr.\n")
fmt.Fprintf(o, "Useful for doing a well-formatted check on input data.\n")
fmt.Fprintf(o, "Current checks are:\n")
fmt.Fprintf(o, "* Data are parseable\n")
fmt.Fprintf(o, "* If any key is the empty string\n")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
WriteVerbOptions(o, checkOptions)
}
func transformerCheckParseCLI(

11
pkg/transformers/clean_whitespace.go
View file

@ -13,11 +13,17 @@ import (
const verbNameCleanWhitespace = "clean-whitespace"
var cleanWhitespaceOptions = []OptionSpec{
{Flag: "-k", Aliases: []string{"--keys-only"}, Type: "bool", Desc: "Do not touch values."},
{Flag: "-v", Aliases: []string{"--values-only"}, Type: "bool", Desc: "Do not touch keys."},
}
var CleanWhitespaceSetup = TransformerSetup{
Verb: verbNameCleanWhitespace,
UsageFunc: transformerCleanWhitespaceUsage,
ParseCLIFunc: transformerCleanWhitespaceParseCLI,
IgnoresInput: false,
Options: cleanWhitespaceOptions,
}
func transformerCleanWhitespaceUsage(
@ -30,12 +36,9 @@ func transformerCleanWhitespaceUsage(
fmt.Fprintf(o, "please see the DSL functions lstrip, rstrip, strip, collapse_whitespace,\n")
fmt.Fprintf(o, "and clean_whitespace.\n")
fmt.Fprintf(o, "\n")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-k|--keys-only Do not touch values.\n")
fmt.Fprintf(o, "-v|--values-only Do not touch keys.\n")
WriteVerbOptions(o, cleanWhitespaceOptions)
fmt.Fprintf(o, "It is an error to specify -k as well as -v -- to clean keys and values,\n")
fmt.Fprintf(o, "leave off -k as well as -v.\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
}
func transformerCleanWhitespaceParseCLI(

13
pkg/transformers/count.go
View file

@ -13,11 +13,18 @@ import (
const verbNameCount = "count"
var countOptions = []OptionSpec{
{Flag: "-g", Arg: "{a,b,c}", Type: "csv-list", Desc: "Optional group-by-field names for counts, e.g. a,b,c"},
{Flag: "-n", Type: "bool", Desc: "Show only the number of distinct values. Not interesting without -g."},
{Flag: "-o", Arg: "{name}", Type: "string", Desc: "Field name for output-count. Default \"count\"."},
}
var CountSetup = TransformerSetup{
Verb: verbNameCount,
UsageFunc: transformerCountUsage,
ParseCLIFunc: transformerCountParseCLI,
IgnoresInput: false,
Options: countOptions,
}
func transformerCountUsage(
@ -27,11 +34,7 @@ func transformerCountUsage(
fmt.Fprint(o,
`Prints number of records, optionally grouped by distinct values for specified field names.
`)
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-g {a,b,c} Optional group-by-field names for counts, e.g. a,b,c\n")
fmt.Fprintf(o, "-n {n} Show only the number of distinct values. Not interesting without -g.\n")
fmt.Fprintf(o, "-o {name} Field name for output-count. Default \"count\".\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
WriteVerbOptions(o, countOptions)
}
func transformerCountParseCLI(

11
pkg/transformers/count_similar.go
View file

@ -13,11 +13,17 @@ import (
const verbNameCountSimilar = "count-similar"
var countSimilarOptions = []OptionSpec{
{Flag: "-g", Arg: "{a,b,c}", Type: "csv-list", Desc: "Group-by-field names for counts, e.g. a,b,c"},
{Flag: "-o", Arg: "{name}", Type: "string", Desc: "Field name for output-counts. Defaults to \"count\"."},
}
var CountSimilarSetup = TransformerSetup{
Verb: verbNameCountSimilar,
UsageFunc: transformerCountSimilarUsage,
ParseCLIFunc: transformerCountSimilarParseCLI,
IgnoresInput: false,
Options: countSimilarOptions,
}
func transformerCountSimilarUsage(
@ -26,10 +32,7 @@ func transformerCountSimilarUsage(
fmt.Fprintf(o, "Usage: %s %s [options]\n", "mlr", verbNameCountSimilar)
fmt.Fprintf(o, "Ingests all records, then emits each record augmented by a count of\n")
fmt.Fprintf(o, "the number of other records having the same group-by field values.\n")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-g {a,b,c} Group-by-field names for counts, e.g. a,b,c\n")
fmt.Fprintf(o, "-o {name} Field name for output-counts. Defaults to \"count\".\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
WriteVerbOptions(o, countSimilarOptions)
}
func transformerCountSimilarParseCLI(

19
pkg/transformers/cut.go
View file

@ -16,11 +16,19 @@ import (
const verbNameCut = "cut"
var cutOptions = []OptionSpec{
{Flag: "-f", Arg: "{a,b,c}", Type: "csv-list", Desc: "Comma-separated field names to include or exclude, e.g. a,b,c."},
{Flag: "-o", Type: "bool", Desc: "Retain fields in the order specified by -f rather than in input-record order."},
{Flag: "-x", Aliases: []string{"--complement"}, Type: "bool", Desc: "Exclude, rather than include, the field names specified by -f."},
{Flag: "-r", Type: "bool", Desc: "Treat field names as regular expressions. \"ab\", \"a.*b\" will match any field name containing the substring \"ab\" or matching \"a.*b\", respectively; anchors of the form \"^ab$\", \"^a.*b$\" may be used."},
}
var CutSetup = TransformerSetup{
Verb: verbNameCut,
UsageFunc: transformerCutUsage,
ParseCLIFunc: transformerCutParseCLI,
IgnoresInput: false,
Options: cutOptions,
}
func transformerCutUsage(
@ -28,16 +36,7 @@ func transformerCutUsage(
) {
fmt.Fprintf(o, "Usage: %s %s [options]\n", "mlr", verbNameCut)
fmt.Fprintf(o, "Passes through input records with specified fields included/excluded.\n")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, " -f {a,b,c} Comma-separated field names for cut, e.g. a,b,c.\n")
fmt.Fprintf(o, " -o Retain fields in the order specified here in the argument list.\n")
fmt.Fprintf(o, " Default is to retain them in the order found in the input data.\n")
fmt.Fprintf(o, " -x|--complement Exclude, rather than include, field names specified by -f.\n")
fmt.Fprintf(o, " -r Treat field names as regular expressions. \"ab\", \"a.*b\" will\n")
fmt.Fprintf(o, " match any field name containing the substring \"ab\" or matching\n")
fmt.Fprintf(o, " \"a.*b\", respectively; anchors of the form \"^ab$\", \"^a.*b$\" may\n")
fmt.Fprintf(o, " be used.\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
WriteVerbOptions(o, cutOptions)
fmt.Fprintf(o, "Examples:\n")
fmt.Fprintf(o, " %s %s -f hostname,status\n", "mlr", verbNameCut)
fmt.Fprintf(o, " %s %s -x -f hostname,status\n", "mlr", verbNameCut)

15
pkg/transformers/decimate.go
View file

@ -11,11 +11,19 @@ import (
const verbNameDecimate = "decimate"
var decimateOptions = []OptionSpec{
{Flag: "-b", Type: "bool", Desc: "Decimate by printing first of every n."},
{Flag: "-e", Type: "bool", Desc: "Decimate by printing last of every n (default)."},
{Flag: "-g", Arg: "{a,b,c}", Type: "csv-list", Desc: "Optional group-by-field names for decimate counts, e.g. a,b,c."},
{Flag: "-n", Arg: "{n}", Type: "int", Desc: "Decimation factor (default 10)."},
}
var DecimateSetup = TransformerSetup{
Verb: verbNameDecimate,
UsageFunc: transformerDecimateUsage,
ParseCLIFunc: transformerDecimateParseCLI,
IgnoresInput: false,
Options: decimateOptions,
}
func transformerDecimateUsage(
@ -23,12 +31,7 @@ func transformerDecimateUsage(
) {
fmt.Fprintf(o, "Usage: %s %s [options]\n", "mlr", verbNameDecimate)
fmt.Fprintf(o, "Passes through one of every n records, optionally by category.\n")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, " -b Decimate by printing first of every n.\n")
fmt.Fprintf(o, " -e Decimate by printing last of every n (default).\n")
fmt.Fprintf(o, " -g {a,b,c} Optional group-by-field names for decimate counts, e.g. a,b,c.\n")
fmt.Fprintf(o, " -n {n} Decimation factor (default 10).\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
WriteVerbOptions(o, decimateOptions)
}
func transformerDecimateParseCLI(

16
pkg/transformers/fill_down.go
View file

@ -12,11 +12,18 @@ import (
const verbNameFillDown = "fill-down"
var fillDownOptions = []OptionSpec{
{Flag: "--all", Type: "bool", Desc: "Operate on all fields in the input."},
{Flag: "-a", Aliases: []string{"--only-if-absent"}, Type: "bool", Desc: "If a given record has a missing value for a given field, fill that from the corresponding value from a previous record, if any. By default, a 'missing' field either is absent, or has the empty-string value. With -a, a field is 'missing' only if it is absent."},
{Flag: "-f", Arg: "{a,b,c}", Type: "csv-list", Desc: "Field names for fill-down."},
}
var FillDownSetup = TransformerSetup{
Verb: verbNameFillDown,
UsageFunc: transformerFillDownUsage,
ParseCLIFunc: transformerFillDownParseCLI,
IgnoresInput: false,
Options: fillDownOptions,
}
func transformerFillDownUsage(
@ -28,14 +35,7 @@ func transformerFillDownUsage(
fmt.Fprintln(o, "By default, a 'missing' field either is absent, or has the empty-string value.")
fmt.Fprintln(o, "With -a, a field is 'missing' only if it is absent.")
fmt.Fprintln(o, "")
fmt.Fprintln(o, "Options:")
fmt.Fprintln(o, " --all Operate on all fields in the input.")
fmt.Fprintln(o, " -a|--only-if-absent If a given record has a missing value for a given field,")
fmt.Fprintln(o, " fill that from the corresponding value from a previous record, if any.")
fmt.Fprintln(o, " By default, a 'missing' field either is absent, or has the empty-string value.")
fmt.Fprintln(o, " With -a, a field is 'missing' only if it is absent.")
fmt.Fprintln(o, " -f Field names for fill-down.")
fmt.Fprintln(o, " -h|--help Show this message.")
WriteVerbOptions(o, fillDownOptions)
}
func transformerFillDownParseCLI(

10
pkg/transformers/fill_empty.go
View file

@ -13,11 +13,17 @@ import (
const verbNameFillEmpty = "fill-empty"
const defaultFillEmptyString = "N/A"
var fillEmptyOptions = []OptionSpec{
{Flag: "-v", Arg: "{string}", Type: "string", Desc: "Fill-value: defaults to \"N/A\""},
{Flag: "-S", Type: "bool", Desc: "Don't infer type -- so '-v 0' would fill string 0 not int 0."},
}
var FillEmptySetup = TransformerSetup{
Verb: verbNameFillEmpty,
UsageFunc: transformerFillEmptyUsage,
ParseCLIFunc: transformerFillEmptyParseCLI,
IgnoresInput: false,
Options: fillEmptyOptions,
}
func transformerFillEmptyUsage(
@ -25,9 +31,7 @@ func transformerFillEmptyUsage(
) {
fmt.Fprintf(o, "Usage: %s %s [options]\n", "mlr", verbNameFillEmpty)
fmt.Fprintf(o, "Fills empty-string fields with specified fill-value.\n")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-v {string} Fill-value: defaults to \"%s\"\n", defaultFillEmptyString)
fmt.Fprintf(o, "-S Don't infer type -- so '-v 0' would fill string 0 not int 0.\n")
WriteVerbOptions(o, fillEmptyOptions)
}
func transformerFillEmptyParseCLI(

11
pkg/transformers/flatten.go
View file

@ -12,11 +12,17 @@ import (
const verbNameFlatten = "flatten"
var flattenOptions = []OptionSpec{
{Flag: "-f", Type: "csv-list", Desc: "Comma-separated list of field names to flatten (default all)."},
{Flag: "-s", Arg: "{string}", Type: "string", Desc: "Separator, defaulting to mlr --flatsep value."},
}
var FlattenSetup = TransformerSetup{
Verb: verbNameFlatten,
UsageFunc: transformerFlattenUsage,
ParseCLIFunc: transformerFlattenParseCLI,
IgnoresInput: false,
Options: flattenOptions,
}
func transformerFlattenUsage(
@ -27,10 +33,7 @@ func transformerFlattenUsage(
`Flattens multi-level maps to single-level ones. Example: field with name 'a'
and value '{"b": { "c": 4 }}' becomes name 'a.b.c' and value 4.
`)
fmt.Fprint(o, "Options:\n")
fmt.Fprint(o, "-f Comma-separated list of field names to flatten (default all).\n")
fmt.Fprintf(o, "-s Separator, defaulting to %s --flatsep value.\n", "mlr")
fmt.Fprintf(o, "-h|--help Show this message.\n")
WriteVerbOptions(o, flattenOptions)
}
func transformerFlattenParseCLI(

26
pkg/transformers/format_values.go
View file

@ -16,11 +16,19 @@ const defaultFormatValuesStringFormat = "%s"
const defaultFormatValuesIntFormat = "%d"
const defaultFormatValuesFloatFormat = "%f"
var formatValuesOptions = []OptionSpec{
{Flag: "-i", Arg: "{integer format}", Type: "string", Desc: "Integer format string; defaults to \"%d\". Examples: \"%06lld\", \"%08llx\". Note that Miller integers are long long so you must use formats which apply to long long, e.g. with ll in them. Undefined behavior results otherwise."},
{Flag: "-f", Arg: "{float format}", Type: "string", Desc: "Float format string; defaults to \"%f\". Examples: \"%8.3lf\", \"%.6le\". Note that Miller floats are double-precision so you must use formats which apply to double, e.g. with l[efg] in them. Undefined behavior results otherwise."},
{Flag: "-s", Arg: "{string format}", Type: "string", Desc: "String format string; defaults to \"%s\". Examples: \"_%s\", \"%08s\". Note that you must use formats which apply to string, e.g. with s in them. Undefined behavior results otherwise."},
{Flag: "-n", Type: "bool", Desc: "Coerce field values autodetected as int to float, and then apply the float format."},
}
var FormatValuesSetup = TransformerSetup{
Verb: verbNameFormatValues,
UsageFunc: transformerFormatValuesUsage,
ParseCLIFunc: transformerFormatValuesParseCLI,
IgnoresInput: false,
Options: formatValuesOptions,
}
func transformerFormatValuesUsage(
@ -38,23 +46,7 @@ func transformerFormatValuesUsage(
fmt.Fprintf(o, "Note: this verb lets you apply arbitrary format strings, which can produce\n")
fmt.Fprintf(o, "undefined behavior and/or program crashes. See your system's \"man printf\".\n")
fmt.Fprintf(o, "\n")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-i {integer format} Defaults to \"%s\".\n", defaultFormatValuesIntFormat)
fmt.Fprintf(o, " Examples: \"%%06lld\", \"%%08llx\".\n")
fmt.Fprintf(o, " Note that Miller integers are long long so you must use\n")
fmt.Fprintf(o, " formats which apply to long long, e.g. with ll in them.\n")
fmt.Fprintf(o, " Undefined behavior results otherwise.\n")
fmt.Fprintf(o, "-f {float format} Defaults to \"%s\".\n", defaultFormatValuesFloatFormat)
fmt.Fprintf(o, " Examples: \"%%8.3lf\", \"%%.6le\".\n")
fmt.Fprintf(o, " Note that Miller floats are double-precision so you must\n")
fmt.Fprintf(o, " use formats which apply to double, e.g. with l[efg] in them.\n")
fmt.Fprintf(o, " Undefined behavior results otherwise.\n")
fmt.Fprintf(o, "-s {string format} Defaults to \"%s\".\n", defaultFormatValuesStringFormat)
fmt.Fprintf(o, " Examples: \"_%%s\", \"%%08s\".\n")
fmt.Fprintf(o, " Note that you must use formats which apply to string, e.g.\n")
fmt.Fprintf(o, " with s in them. Undefined behavior results otherwise.\n")
fmt.Fprintf(o, "-n Coerce field values autodetected as int to float, and then\n")
fmt.Fprintf(o, " apply the float format.\n")
WriteVerbOptions(o, formatValuesOptions)
}
func transformerFormatValuesParseCLI(

19
pkg/transformers/fraction.go
View file

@ -15,11 +15,19 @@ import (
const verbNameFraction = "fraction"
var fractionOptions = []OptionSpec{
{Flag: "-f", Arg: "{a,b,c}", Type: "csv-list", Desc: "Field name(s) for fraction calculation"},
{Flag: "-g", Arg: "{d,e,f}", Type: "csv-list", Desc: "Optional group-by-field name(s) for fraction counts"},
{Flag: "-p", Type: "bool", Desc: "Produce percents [0..100], not fractions [0..1]. Output field names end with \"_percent\" rather than \"_fraction\""},
{Flag: "-c", Type: "bool", Desc: "Produce cumulative distributions, i.e. running sums: each output value folds in the sum of the previous for the specified group. E.g. with input records x=1 x=2 x=3 and x=4, emits output records x=1,x_cumulative_fraction=0.1 x=2,x_cumulative_fraction=0.3 x=3,x_cumulative_fraction=0.6 and x=4,x_cumulative_fraction=1.0"},
}
var FractionSetup = TransformerSetup{
Verb: verbNameFraction,
UsageFunc: transformerFractionUsage,
ParseCLIFunc: transformerFractionParseCLI,
IgnoresInput: false,
Options: fractionOptions,
}
func transformerFractionUsage(
@ -37,16 +45,7 @@ func transformerFractionUsage(
fmt.Fprintf(o, "input records and accumulates sums; on the second pass it computes quotients\n")
fmt.Fprintf(o, "and emits output records. This means it produces no output until all input is read.\n")
fmt.Fprintf(o, "\n")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-f {a,b,c} Field name(s) for fraction calculation\n")
fmt.Fprintf(o, "-g {d,e,f} Optional group-by-field name(s) for fraction counts\n")
fmt.Fprintf(o, "-p Produce percents [0..100], not fractions [0..1]. Output field names\n")
fmt.Fprintf(o, " end with \"_percent\" rather than \"_fraction\"\n")
fmt.Fprintf(o, "-c Produce cumulative distributions, i.e. running sums: each output\n")
fmt.Fprintf(o, " value folds in the sum of the previous for the specified group\n")
fmt.Fprintf(o, " E.g. with input records x=1 x=2 x=3 and x=4, emits output records\n")
fmt.Fprintf(o, " x=1,x_cumulative_fraction=0.1 x=2,x_cumulative_fraction=0.3\n")
fmt.Fprintf(o, " x=3,x_cumulative_fraction=0.6 and x=4,x_cumulative_fraction=1.0\n")
WriteVerbOptions(o, fractionOptions)
}
func transformerFractionParseCLI(

16
pkg/transformers/gap.go
View file

@ -12,11 +12,17 @@ import (
const verbNameGap = "gap"
var gapOptions = []OptionSpec{
{Flag: "-g", Arg: "{a,b,c}", Type: "csv-list", Desc: "Print a gap whenever values of these fields (e.g. a,b,c) changes."},
{Flag: "-n", Arg: "{n}", Type: "int", Desc: "Print a gap every n records. Ignored if -g is present."},
}
var GapSetup = TransformerSetup{
Verb: verbNameGap,
UsageFunc: transformerGapUsage,
ParseCLIFunc: transformerGapParseCLI,
IgnoresInput: false,
Options: gapOptions,
}
func transformerGapUsage(
@ -24,14 +30,8 @@ func transformerGapUsage(
) {
fmt.Fprintf(o, "Usage: %s %s [options]\n", "mlr", verbNameGap)
fmt.Fprint(o, "Emits an empty record every n records, or when certain values change.\n")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "Emits an empty record every n records, or when certain values change.\n")
fmt.Fprintf(o, "-g {a,b,c} Print a gap whenever values of these fields (e.g. a,b,c) changes.\n")
fmt.Fprintf(o, "-n {n} Print a gap every n records.\n")
fmt.Fprintf(o, "One of -f or -g is required.\n")
fmt.Fprintf(o, "-n is ignored if -g is present.\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
fmt.Fprintf(o, "One of -n or -g is required.\n")
WriteVerbOptions(o, gapOptions)
}
func transformerGapParseCLI(

13
pkg/transformers/grep.go
View file

@ -12,11 +12,18 @@ import (
const verbNameGrep = "grep"
var grepOptions = []OptionSpec{
{Flag: "-i", Type: "bool", Desc: "Use case-insensitive search."},
{Flag: "-v", Type: "bool", Desc: "Invert: pass through records which do not match the regex."},
{Flag: "-a", Type: "bool", Desc: "Only grep for values, not keys and values."},
}
var GrepSetup = TransformerSetup{
Verb: verbNameGrep,
UsageFunc: transformerGrepUsage,
ParseCLIFunc: transformerGrepParseCLI,
IgnoresInput: false,
Options: grepOptions,
}
func transformerGrepUsage(
@ -25,11 +32,7 @@ func transformerGrepUsage(
fmt.Fprintf(o, "Usage: %s %s [options] {regular expression}\n", "mlr", verbNameGrep)
fmt.Fprintf(o, "Passes through records which match the regular expression.\n")
fmt.Fprint(o, "Options:\n")
fmt.Fprint(o, "-i Use case-insensitive search.\n")
fmt.Fprint(o, "-v Invert: pass through records which do not match the regex.\n")
fmt.Fprint(o, "-a Only grep for values, not keys and values.\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
WriteVerbOptions(o, grepOptions)
fmt.Fprintf(o, `Note that "%s filter" is more powerful, but requires you to know field names.
By contrast, "%s grep" allows you to regex-match the entire record. It does this

8
pkg/transformers/group_by.go
View file

@ -12,20 +12,22 @@ import (
const verbNameGroupBy = "group-by"
var groupByOptions = []OptionSpec{}
var GroupBySetup = TransformerSetup{
Verb: verbNameGroupBy,
UsageFunc: transformerGroupByUsage,
ParseCLIFunc: transformerGroupByParseCLI,
IgnoresInput: false,
Options: groupByOptions,
}
func transformerGroupByUsage(
o *os.File,
) {
fmt.Fprintf(o, "Usage: %s %s [options] {comma-separated field names}\n", "mlr", verbNameGroupBy)
fmt.Fprint(o, "Outputs records in batches having identical values at specified field names.")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
fmt.Fprintln(o, "Outputs records in batches having identical values at specified field names.")
WriteVerbOptions(o, groupByOptions)
}
func transformerGroupByParseCLI(

6
pkg/transformers/group_like.go
View file

@ -12,11 +12,14 @@ import (
const verbNameGroupLike = "group-like"
var groupLikeOptions = []OptionSpec{}
var GroupLikeSetup = TransformerSetup{
Verb: verbNameGroupLike,
UsageFunc: transformerGroupLikeUsage,
ParseCLIFunc: transformerGroupLikeParseCLI,
IgnoresInput: false,
Options: groupLikeOptions,
}
func transformerGroupLikeUsage(
@ -24,8 +27,7 @@ func transformerGroupLikeUsage(
) {
fmt.Fprintf(o, "Usage: %s %s [options]\n", "mlr", verbNameGroupLike)
fmt.Fprintln(o, "Outputs records in batches having identical field names.")
fmt.Fprintln(o, "Options:")
fmt.Fprintln(o, "-h|--help Show this message.")
WriteVerbOptions(o, groupLikeOptions)
}
func transformerGroupLikeParseCLI(

19
pkg/transformers/having_fields.go
View file

@ -25,12 +25,21 @@ const (
const verbNameHavingFields = "having-fields"
var havingFieldsOptions = []OptionSpec{
{Flag: "--at-least", Arg: "{comma-separated names}", Type: "csv-list", Desc: "Pass records that have at least these field names."},
{Flag: "--which-are", Arg: "{comma-separated names}", Type: "csv-list", Desc: "Pass records whose field names are exactly these."},
{Flag: "--at-most", Arg: "{comma-separated names}", Type: "csv-list", Desc: "Pass records that have at most these field names."},
{Flag: "--all-matching", Arg: "{regular expression}", Type: "regex", Desc: "Pass records where all field names match the regex."},
{Flag: "--any-matching", Arg: "{regular expression}", Type: "regex", Desc: "Pass records where any field name matches the regex."},
{Flag: "--none-matching", Arg: "{regular expression}", Type: "regex", Desc: "Pass records where no field name matches the regex."},
}
var HavingFieldsSetup = TransformerSetup{
Verb: verbNameHavingFields,
UsageFunc: transformerHavingFieldsUsage,
ParseCLIFunc: transformerHavingFieldsParseCLI,
IgnoresInput: false,
Options: havingFieldsOptions,
}
func transformerHavingFieldsUsage(
@ -41,13 +50,7 @@ func transformerHavingFieldsUsage(
fmt.Fprintf(o, "Usage: %s %s [options]\n", "mlr", verbNameHavingFields)
fmt.Fprintf(o, "Conditionally passes through records depending on each record's field names.\n")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, " --at-least {comma-separated names}\n")
fmt.Fprintf(o, " --which-are {comma-separated names}\n")
fmt.Fprintf(o, " --at-most {comma-separated names}\n")
fmt.Fprintf(o, " --all-matching {regular expression}\n")
fmt.Fprintf(o, " --any-matching {regular expression}\n")
fmt.Fprintf(o, " --none-matching {regular expression}\n")
WriteVerbOptions(o, havingFieldsOptions)
fmt.Fprintf(o, "Examples:\n")
fmt.Fprintf(o, " %s %s --which-are amount,status,owner\n", exeName, verb)
fmt.Fprintf(o, " %s %s --any-matching 'sda[0-9]'\n", exeName, verb)

14
pkg/transformers/head.go
View file

@ -11,11 +11,17 @@ import (
const verbNameHead = "head"
var headOptions = []OptionSpec{
{Flag: "-g", Arg: "{a,b,c}", Type: "csv-list", Desc: "Optional group-by-field names for head counts, e.g. a,b,c."},
{Flag: "-n", Arg: "{n}", Type: "int", Desc: "Head-count to print. Default 10. A negative count, e.g. -n -2, passes through all but the last n records, optionally by category."},
}
var HeadSetup = TransformerSetup{
Verb: verbNameHead,
UsageFunc: transformerHeadUsage,
ParseCLIFunc: transformerHeadParseCLI,
IgnoresInput: false,
Options: headOptions,
}
func transformerHeadUsage(
@ -24,13 +30,7 @@ func transformerHeadUsage(
fmt.Fprintf(o, "Usage: %s %s [options]\n", "mlr", verbNameHead)
fmt.Fprintf(o, "Passes through the first n records, optionally by category.\n")
fmt.Fprintf(o, "Without -g, ceases consuming more input (i.e. is fast) when n records have been read.\n")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-g {a,b,c} Optional group-by-field names for head counts, e.g. a,b,c.\n")
fmt.Fprintf(o, "-n {n} Head-count to print. Default 10.\n")
fmt.Fprintf(o, " A negative count, e.g. -n -2, passes through all but the last n records,\n")
fmt.Fprintf(o, " optionally by category.\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
WriteVerbOptions(o, headOptions)
}
func transformerHeadParseCLI(

19
pkg/transformers/histogram.go
View file

@ -14,11 +14,21 @@ import (
const verbNameHistogram = "histogram"
const histogramDefaultBinCount = int64(20)
var histogramOptions = []OptionSpec{
{Flag: "-f", Arg: "{a,b,c}", Type: "csv-list", Desc: "Value-field names for histogram counts."},
{Flag: "--lo", Arg: "{lo}", Type: "float", Desc: "Histogram low value."},
{Flag: "--hi", Arg: "{hi}", Type: "float", Desc: "Histogram high value."},
{Flag: "--nbins", Arg: "{n}", Type: "int", Desc: "Number of histogram bins. Defaults to 20."},
{Flag: "--auto", Type: "bool", Desc: "Automatically computes limits, ignoring --lo and --hi. Holds all values in memory before producing any output."},
{Flag: "-o", Arg: "{prefix}", Type: "string", Desc: "Prefix for output field name. Default: no prefix."},
}
var HistogramSetup = TransformerSetup{
Verb: verbNameHistogram,
UsageFunc: transformerHistogramUsage,
ParseCLIFunc: transformerHistogramParseCLI,
IgnoresInput: false,
Options: histogramOptions,
}
func transformerHistogramUsage(
@ -28,14 +38,7 @@ func transformerHistogramUsage(
verb := verbNameHistogram
fmt.Fprintf(o, "Just a histogram. Input values < lo or > hi are not counted.\n")
fmt.Fprintf(o, "Usage: %s %s [options]\n", argv0, verb)
fmt.Fprintf(o, "-f {a,b,c} Value-field names for histogram counts\n")
fmt.Fprintf(o, "--lo {lo} Histogram low value\n")
fmt.Fprintf(o, "--hi {hi} Histogram high value\n")
fmt.Fprintf(o, "--nbins {n} Number of histogram bins. Defaults to %d.\n", histogramDefaultBinCount)
fmt.Fprintf(o, "--auto Automatically computes limits, ignoring --lo and --hi.\n")
fmt.Fprintf(o, " Holds all values in memory before producing any output.\n")
fmt.Fprintf(o, "-o {prefix} Prefix for output field name. Default: no prefix.\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
WriteVerbOptions(o, histogramOptions)
}
func transformerHistogramParseCLI(

49
pkg/transformers/join.go
View file

@ -15,11 +15,29 @@ import (
const verbNameJoin = "join"
var joinOptions = []OptionSpec{
{Flag: "-f", Arg: "{left file name}", Type: "filename", Desc: "Left file name for join."},
{Flag: "-j", Arg: "{a,b,c}", Type: "csv-list", Desc: "Comma-separated join-field names for output."},
{Flag: "-l", Arg: "{a,b,c}", Type: "csv-list", Desc: "Comma-separated join-field names for left input file; defaults to -j values if omitted."},
{Flag: "-r", Arg: "{a,b,c}", Type: "csv-list", Desc: "Comma-separated join-field names for right input file(s); defaults to -j values if omitted."},
{Flag: "--lk", Aliases: []string{"--left-keep-field-names"}, Arg: "{a,b,c}", Type: "csv-list", Desc: "If supplied, this means keep only the specified field names from the left file. Automatically includes the join-field name(s). Helpful for when you only want a limited subset of information from the left file. Tip: you can use --lk \"\": this means the left file becomes solely a row-selector for the input files."},
{Flag: "--lp", Arg: "{text}", Type: "string", Desc: "Additional prefix for non-join output field names from the left file. Applies to paired and unpaired output records."},
{Flag: "--rp", Arg: "{text}", Type: "string", Desc: "Additional prefix for non-join output field names from the right file(s). Applies to paired and unpaired output records."},
{Flag: "--np", Type: "bool", Desc: "Do not emit paired records."},
{Flag: "--ul", Type: "bool", Desc: "Emit unpaired records from the left file."},
{Flag: "--ur", Type: "bool", Desc: "Emit unpaired records from the right file(s)."},
{Flag: "-s", Aliases: []string{"--sorted-input"}, Type: "bool", Desc: "Require sorted input: records must be sorted lexically by their join-field names, else not all records will be paired. The only likely use case for this is with a left file which is too big to fit into system memory otherwise."},
{Flag: "-u", Type: "bool", Desc: "Enable unsorted input. (This is the default even without -u.) In this case, the entire left file will be loaded into memory."},
{Flag: "--prepipe", Arg: "{command}", Type: "string", Desc: "Shell command to prepipe the left-file input through. As in main input options; see mlr --help for details. If you wish to use a prepipe command for the main input as well as here, it must be specified there as well as here."},
{Flag: "--prepipex", Arg: "{command}", Type: "string", Desc: "Shell command to prepipe the left-file input through (no shell quoting). As in main input options; see mlr --help for details."},
}
var JoinSetup = TransformerSetup{
Verb: verbNameJoin,
UsageFunc: transformerJoinUsage,
ParseCLIFunc: transformerJoinParseCLI,
IgnoresInput: false,
Options: joinOptions,
}
// Most transformers have option-variables as individual locals within the
@ -78,36 +96,7 @@ func transformerJoinUsage(
fmt.Fprintf(o, "at the end of the Miller argument list.\n")
fmt.Fprintf(o, "Functionality is essentially the same as the system \"join\" command, but for\n")
fmt.Fprintf(o, "record streams.\n")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, " -f {left file name}\n")
fmt.Fprintf(o, " -j {a,b,c} Comma-separated join-field names for output\n")
fmt.Fprintf(o, " -l {a,b,c} Comma-separated join-field names for left input file;\n")
fmt.Fprintf(o, " defaults to -j values if omitted.\n")
fmt.Fprintf(o, " -r {a,b,c} Comma-separated join-field names for right input file(s);\n")
fmt.Fprintf(o, " defaults to -j values if omitted.\n")
fmt.Fprintf(o, " --lk|--left-keep-field-names {a,b,c} If supplied, this means keep only the specified field\n")
fmt.Fprintf(o, " names from the left file. Automatically includes the join-field name(s). Helpful\n")
fmt.Fprintf(o, " for when you only want a limited subset of information from the left file.\n")
fmt.Fprintf(o, " Tip: you can use --lk \"\": this means the left file becomes solely a row-selector\n")
fmt.Fprintf(o, " for the input files.\n")
fmt.Fprintf(o, " --lp {text} Additional prefix for non-join output field names from\n")
fmt.Fprintf(o, " the left file. Applies to paired and unpaired output records.\n")
fmt.Fprintf(o, " --rp {text} Additional prefix for non-join output field names from\n")
fmt.Fprintf(o, " the right file(s). Applies to paired and unpaired output records.\n")
fmt.Fprintf(o, " --np Do not emit paired records\n")
fmt.Fprintf(o, " --ul Emit unpaired records from the left file\n")
fmt.Fprintf(o, " --ur Emit unpaired records from the right file(s)\n")
fmt.Fprintf(o, " -s|--sorted-input Require sorted input: records must be sorted\n")
fmt.Fprintf(o, " lexically by their join-field names, else not all records will\n")
fmt.Fprintf(o, " be paired. The only likely use case for this is with a left\n")
fmt.Fprintf(o, " file which is too big to fit into system memory otherwise.\n")
fmt.Fprintf(o, " -u Enable unsorted input. (This is the default even without -u.)\n")
fmt.Fprintf(o, " In this case, the entire left file will be loaded into memory.\n")
fmt.Fprintf(o, " --prepipe {command} As in main input options; see %s --help for details.\n",
"mlr")
fmt.Fprintf(o, " If you wish to use a prepipe command for the main input as well\n")
fmt.Fprintf(o, " as here, it must be specified there as well as here.\n")
fmt.Fprintf(o, " --prepipex {command} Likewise.\n")
WriteVerbOptions(o, joinOptions)
fmt.Fprintf(o, "File-format options default to those for the right file names on the Miller\n")
fmt.Fprintf(o, "argument list, but may be overridden for the left file as follows. Please see\n")
fmt.Fprintf(o, "the main \"%s --help\" for more information on syntax for these arguments:\n", "mlr")

21
pkg/transformers/json_parse.go
View file

@ -12,11 +12,26 @@ import (
const verbNameJSONParse = "json-parse"
var jsonParseOptions = []OptionSpec{
{
Flag: "-f",
Arg: "{a,b,c}",
Type: "csv-list",
Desc: "Comma-separated list of field names to json-parse (default all).",
},
{
Flag: "-k",
Type: "bool",
Desc: "If supplied, then on parse fail for any cell, keep the (unparsable) input value for the cell.",
},
}
var JSONParseSetup = TransformerSetup{
Verb: verbNameJSONParse,
UsageFunc: transformerJSONParseUsage,
ParseCLIFunc: transformerJSONParseParseCLI,
IgnoresInput: false,
Options: jsonParseOptions,
}
func transformerJSONParseUsage(
@ -27,11 +42,7 @@ func transformerJSONParseUsage(
o,
`Tries to convert string field values to parsed JSON, e.g. "[1,2,3]" -> [1,2,3].`,
)
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-f {...} Comma-separated list of field names to json-parse (default all).\n")
fmt.Fprintf(o, "-k If supplied, then on parse fail for any cell, keep the (unparsable)\n")
fmt.Fprintf(o, " input value for the cell.\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
WriteVerbOptions(o, jsonParseOptions)
}
func transformerJSONParseParseCLI(

26
pkg/transformers/json_stringify.go
View file

@ -13,11 +13,31 @@ import (
const verbNameJSONStringify = "json-stringify"
var jsonStringifyOptions = []OptionSpec{
{
Flag: "-f",
Arg: "{a,b,c}",
Type: "csv-list",
Desc: "Comma-separated list of field names to json-stringify (default all).",
},
{
Flag: "--jvstack",
Type: "bool",
Desc: "Produce multi-line JSON output.",
},
{
Flag: "--no-jvstack",
Type: "bool",
Desc: "Produce single-line JSON output per record (default).",
},
}
var JSONStringifySetup = TransformerSetup{
Verb: verbNameJSONStringify,
UsageFunc: transformerJSONStringifyUsage,
ParseCLIFunc: transformerJSONStringifyParseCLI,
IgnoresInput: false,
Options: jsonStringifyOptions,
}
func transformerJSONStringifyUsage(
@ -27,11 +47,7 @@ func transformerJSONStringifyUsage(
fmt.Fprint(o,
`Produces string field values from field-value data, e.g. [1,2,3] -> "[1,2,3]".
`)
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-f {...} Comma-separated list of field names to json-parse (default all).\n")
fmt.Fprintf(o, "--jvstack Produce multi-line JSON output.\n")
fmt.Fprintf(o, "--no-jvstack Produce single-line JSON output per record (default).\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
WriteVerbOptions(o, jsonStringifyOptions)
}
func transformerJSONStringifyParseCLI(

6
pkg/transformers/label.go
View file

@ -12,11 +12,14 @@ import (
const verbNameLabel = "label"
var labelOptions = []OptionSpec{}
var LabelSetup = TransformerSetup{
Verb: verbNameLabel,
UsageFunc: transformerLabelUsage,
ParseCLIFunc: transformerLabelParseCLI,
IgnoresInput: false,
Options: labelOptions,
}
func transformerLabelUsage(
@ -28,8 +31,7 @@ func transformerLabelUsage(
fmt.Fprintf(o, "names.) Particularly useful with --inidx or --implicit-csv-header, to give\n")
fmt.Fprintf(o, "useful names to otherwise integer-indexed fields.\n")
fmt.Fprintf(o, "\n")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
WriteVerbOptions(o, labelOptions)
}
func transformerLabelParseCLI(

6
pkg/transformers/latin1_to_utf8.go
View file

@ -13,11 +13,14 @@ import (
const verbNameLatin1ToUTF8 = "latin1-to-utf8"
var latin1ToUTF8Options = []OptionSpec{}
var Latin1ToUTF8Setup = TransformerSetup{
Verb: verbNameLatin1ToUTF8,
UsageFunc: transformerLatin1ToUTF8Usage,
ParseCLIFunc: transformerLatin1ToUTF8ParseCLI,
IgnoresInput: false,
Options: latin1ToUTF8Options,
}
func transformerLatin1ToUTF8Usage(
@ -26,8 +29,7 @@ func transformerLatin1ToUTF8Usage(
fmt.Fprintf(o, "Usage: %s %s, with no options.\n", "mlr", verbNameLatin1ToUTF8)
fmt.Fprintf(o, "Recursively converts record strings from Latin-1 to UTF-8.\n")
fmt.Fprintf(o, "For field-level control, please see the latin1_to_utf8 DSL function.\n")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
WriteVerbOptions(o, latin1ToUTF8Options)
}
func transformerLatin1ToUTF8ParseCLI(

29
pkg/transformers/merge_fields.go
View file

@ -14,11 +14,24 @@ import (
const verbNameMergeFields = "merge-fields"
var mergeFieldsOptions = []OptionSpec{
{Flag: "-a", Arg: "{sum,count,...}", Type: "enum", Desc: "Names of accumulators: one or more of the accumulators listed below.", Values: []string{"count", "null_count", "distinct_count", "mode", "antimode", "sum", "mean", "mad", "var", "stddev", "meaneb", "skewness", "kurtosis", "min", "max", "minlen", "maxlen"}},
{Flag: "-f", Arg: "{a,b,c}", Type: "csv-list", Desc: "Value-field names on which to compute statistics. Requires -o."},
{Flag: "-r", Arg: "{a,b,c}", Type: "csv-list", Desc: "Regular expressions for value-field names on which to compute statistics. Requires -o."},
{Flag: "-c", Arg: "{a,b,c}", Type: "csv-list", Desc: "Substrings for collapse mode: all fields which have the same names after removing substrings will be accumulated together. Please see examples below."},
{Flag: "-i", Type: "bool", Desc: "Use interpolated percentiles, like R's type=7; default like type=1. Not sensical for string-valued fields."},
{Flag: "-o", Arg: "{name}", Type: "string", Desc: "Output field basename for -f/-r."},
{Flag: "-k", Type: "bool", Desc: "Keep the input fields which contributed to the output statistics; the default is to omit them."},
{Flag: "-S", Type: "bool", Desc: "No-op flag for backward compatibility with Miller 5."},
{Flag: "-F", Type: "bool", Desc: "No-op flag for backward compatibility with Miller 5."},
}
var MergeFieldsSetup = TransformerSetup{
Verb: verbNameMergeFields,
UsageFunc: transformerMergeFieldsUsage,
ParseCLIFunc: transformerMergeFieldsParseCLI,
IgnoresInput: false,
Options: mergeFieldsOptions,
}
type mergeByType int
@ -38,20 +51,10 @@ func transformerMergeFieldsUsage(
fmt.Fprintf(o, "Usage: %s %s [options]\n", argv0, verb)
fmt.Fprintf(o, "Computes univariate statistics for each input record, accumulated across\n")
fmt.Fprintf(o, "specified fields.\n")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-a {sum,count,...} Names of accumulators. One or more of:\n")
WriteVerbOptions(o, mergeFieldsOptions)
fmt.Fprintf(o, "\n")
fmt.Fprintf(o, "Accumulators for -a:\n")
utils.ListStats1Accumulators(o)
fmt.Fprintf(o, "-f {a,b,c} Value-field names on which to compute statistics. Requires -o.\n")
fmt.Fprintf(o, "-r {a,b,c} Regular expressions for value-field names on which to compute\n")
fmt.Fprintf(o, " statistics. Requires -o.\n")
fmt.Fprintf(o, "-c {a,b,c} Substrings for collapse mode. All fields which have the same names\n")
fmt.Fprintf(o, " after removing substrings will be accumulated together. Please see\n")
fmt.Fprintf(o, " examples below.\n")
fmt.Fprintf(o, "-i Use interpolated percentiles, like R's type=7; default like type=1.\n")
fmt.Fprintf(o, " Not sensical for string-valued fields.\n")
fmt.Fprintf(o, "-o {name} Output field basename for -f/-r.\n")
fmt.Fprintf(o, "-k Keep the input fields which contributed to the output statistics;\n")
fmt.Fprintf(o, " the default is to omit them.\n")
fmt.Fprintf(o, "\n")
fmt.Fprintf(o, "String-valued data make sense unless arithmetic on them is required,\n")
fmt.Fprintf(o, "e.g. for sum, mean, interpolated percentiles, etc. In case of mixed data,\n")

28
pkg/transformers/most_or_least_frequent.go
View file

@ -18,11 +18,26 @@ const verbNameLeastFrequent = "least-frequent"
const mostLeastFrequentDefaultMaxOutputLength = int64(10)
const mostLeastFrequentDefaultOutputFieldName = "count"
var mostFrequentOptions = []OptionSpec{
{Flag: "-f", Arg: "{a,b,c}", Type: "csv-list", Desc: "One or more comma-separated field names to group by. Required flag."},
{Flag: "-n", Arg: "{count}", Type: "int", Desc: "Maximum number of results to output. Optional flag defaulting to 10."},
{Flag: "-b", Type: "bool", Desc: "Suppress counts; show only field values."},
{Flag: "-o", Arg: "{name}", Type: "string", Desc: "Field name for output count. Default \"count\"."},
}
var MostFrequentSetup = TransformerSetup{
Verb: verbNameMostFrequent,
UsageFunc: transformerMostFrequentUsage,
ParseCLIFunc: transformerMostFrequentParseCLI,
IgnoresInput: false,
Options: mostFrequentOptions,
}
var leastFrequentOptions = []OptionSpec{
{Flag: "-f", Arg: "{a,b,c}", Type: "csv-list", Desc: "One or more comma-separated field names to group by. Required flag."},
{Flag: "-n", Arg: "{count}", Type: "int", Desc: "Maximum number of results to output. Optional flag defaulting to 10."},
{Flag: "-b", Type: "bool", Desc: "Suppress counts; show only field values."},
{Flag: "-o", Arg: "{name}", Type: "string", Desc: "Field name for output count. Default \"count\"."},
}
var LeastFrequentSetup = TransformerSetup{
@ -30,6 +45,7 @@ var LeastFrequentSetup = TransformerSetup{
UsageFunc: transformerLeastFrequentUsage,
ParseCLIFunc: transformerLeastFrequentParseCLI,
IgnoresInput: false,
Options: leastFrequentOptions,
}
func transformerMostFrequentUsage(
@ -40,11 +56,7 @@ func transformerMostFrequentUsage(
fmt.Fprintf(o, "Usage: %s %s [options]\n", argv0, verb)
fmt.Fprintf(o, "Shows the most frequently occurring distinct values for specified field names.\n")
fmt.Fprintf(o, "The first entry is the statistical mode; the remaining are runners-up.\n")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-f {one or more comma-separated field names}. Required flag.\n")
fmt.Fprintf(o, "-n {count}. Optional flag defaulting to %d.\n", mostLeastFrequentDefaultMaxOutputLength)
fmt.Fprintf(o, "-b Suppress counts; show only field values.\n")
fmt.Fprintf(o, "-o {name} Field name for output count. Default \"%s\".\n", mostLeastFrequentDefaultOutputFieldName)
WriteVerbOptions(o, mostFrequentOptions)
fmt.Fprintf(o, "See also \"%s %s\".\n", argv0, "least-frequent")
}
@ -56,11 +68,7 @@ func transformerLeastFrequentUsage(
fmt.Fprintf(o, "Usage: %s %s [options]\n", argv0, verb)
fmt.Fprintf(o, "Shows the least frequently occurring distinct values for specified field names.\n")
fmt.Fprintf(o, "The first entry is the statistical anti-mode; the remaining are runners-up.\n")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-f {one or more comma-separated field names}. Required flag.\n")
fmt.Fprintf(o, "-n {count}. Optional flag defaulting to %d.\n", mostLeastFrequentDefaultMaxOutputLength)
fmt.Fprintf(o, "-b Suppress counts; show only field values.\n")
fmt.Fprintf(o, "-o {name} Field name for output count. Default \"%s\".\n", mostLeastFrequentDefaultOutputFieldName)
WriteVerbOptions(o, leastFrequentOptions)
fmt.Fprintf(o, "See also \"%s %s\".\n", argv0, "most-frequent")
}

28
pkg/transformers/nest.go
View file

@ -16,11 +16,27 @@ import (
const verbNameNest = "nest"
var nestOptions = []OptionSpec{
{Flag: "--explode", Type: "bool", Desc: "Explode field values into separate fields/records. One of --explode or --implode is required."},
{Flag: "--implode", Type: "bool", Desc: "Reverse of --explode. One of --explode or --implode is required."},
{Flag: "--values", Type: "bool", Desc: "Operate on field values. One of --values or --pairs is required."},
{Flag: "--pairs", Type: "bool", Desc: "Operate on field key-value pairs. One of --values or --pairs is required."},
{Flag: "--across-records", Type: "bool", Desc: "Explode/implode across records. One of --across-records or --across-fields is required."},
{Flag: "--across-fields", Type: "bool", Desc: "Explode/implode across fields. One of --across-records or --across-fields is required."},
{Flag: "-f", Arg: "{field name}", Type: "string", Desc: "Required field name to operate on."},
{Flag: "-r", Arg: "{field names}", Type: "regex", Desc: "Like -f but treat arguments as a regular expression. Match all field names and operate on each in record order. Example: -r '^[xy]$'."},
{Flag: "--nested-fs", Arg: "{string}", Type: "string", Desc: "Field separator for nested values. Defaults to \";\"."},
{Flag: "--nested-ps", Arg: "{string}", Type: "string", Desc: "Pair separator for nested key-value pairs. Defaults to \":\"."},
{Flag: "--evar", Arg: "{string}", Type: "string", Desc: "Shorthand for --explode --values --across-records --nested-fs {string}."},
{Flag: "--ivar", Arg: "{string}", Type: "string", Desc: "Shorthand for --implode --values --across-records --nested-fs {string}."},
}
var NestSetup = TransformerSetup{
Verb: verbNameNest,
UsageFunc: transformerNestUsage,
ParseCLIFunc: transformerNestParseCLI,
IgnoresInput: false,
Options: nestOptions,
}
func transformerNestUsage(
@ -31,17 +47,7 @@ func transformerNestUsage(
fmt.Fprintf(o, "Usage: %s %s [options]\n", argv0, verb)
fmt.Fprintf(o, "Explodes specified field values into separate fields/records, or reverses this.\n")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, " --explode,--implode One is required.\n")
fmt.Fprintf(o, " --values,--pairs One is required.\n")
fmt.Fprintf(o, " --across-records,--across-fields One is required.\n")
fmt.Fprintf(o, " -f {field name} Required.\n")
fmt.Fprintf(o, " -r {field names} Like -f but treat arguments as a regular expression. Match all\n")
fmt.Fprintf(o, " field names and operate on each in record order. Example: `-r '^[xy]$`'.\n")
fmt.Fprintf(o, " --nested-fs {string} Defaults to \";\". Field separator for nested values.\n")
fmt.Fprintf(o, " --nested-ps {string} Defaults to \":\". Pair separator for nested key-value pairs.\n")
fmt.Fprintf(o, " --evar {string} Shorthand for --explode --values --across-records --nested-fs {string}\n")
fmt.Fprintf(o, " --ivar {string} Shorthand for --implode --values --across-records --nested-fs {string}\n")
WriteVerbOptions(o, nestOptions)
fmt.Fprintf(o, "Please use \"%s --usage-separator-options\" for information on specifying separators.\n",
argv0)

6
pkg/transformers/nothing.go
View file

@ -11,11 +11,14 @@ import (
const verbNameNothing = "nothing"
var nothingOptions = []OptionSpec{} // no verb-specific options
var NothingSetup = TransformerSetup{
Verb: verbNameNothing,
ParseCLIFunc: transformerNothingParseCLI,
UsageFunc: transformerNothingUsage,
IgnoresInput: false,
Options: nothingOptions,
}
func transformerNothingUsage(
@ -24,8 +27,7 @@ func transformerNothingUsage(
fmt.Fprintf(o, "Usage: %s %s [options]\n", "mlr", verbNameNothing)
fmt.Fprintf(o, "Drops all input records. Useful for testing, or after tee/print/etc. have\n")
fmt.Fprintf(o, "produced other output.\n")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
WriteVerbOptions(o, nothingOptions)
}
func transformerNothingParseCLI(

102
pkg/transformers/put_or_filter.go
View file

@ -16,20 +16,58 @@ import (
const verbNamePut = "put"
var putOptions = []OptionSpec{
{Flag: "-f", Arg: "{file name}", Type: "filename", Desc: "File containing a DSL expression (see examples below). If the filename is a directory, all *.mlr files in that directory are loaded.", Repeatable: true},
{Flag: "-e", Arg: "{expression}", Type: "string", Desc: "DSL expression to evaluate. You can use this after -f to add an expression. Example use case: define functions/subroutines in a file you specify with -f, then call them with an expression you specify with -e.", Repeatable: true},
{Flag: "-s", Arg: "{name=value}", Type: "string", Desc: "Predefines out-of-stream variable @name to have the given value. Thus mlr put -s foo=97 '$column += @foo' is like mlr put 'begin {@foo = 97} $column += @foo'. The value part is subject to type-inferencing. May be specified more than once, e.g. -s name1=value1 -s name2=value2. Note: the value may be an environment variable, e.g. -s sequence=$SEQUENCE.", Repeatable: true},
{Flag: "-x", Type: "bool", Desc: "Prints records for which {expression} evaluates to false, not true, i.e. invert the sense of the filter expression. Default false."},
{Flag: "-q", Type: "bool", Desc: "Does not include the modified record in the output stream. Useful for when all desired output is in begin and/or end blocks."},
{Flag: "-S", Type: "bool", Desc: "No-op in Miller 6 and above, since type-inferencing is now done by the record-readers before filter/put is executed. Supported as a no-op pass-through flag for backward compatibility."},
{Flag: "-F", Type: "bool", Desc: "No-op in Miller 6 and above, since type-inferencing is now done by the record-readers before filter/put is executed. Supported as a no-op pass-through flag for backward compatibility."},
{Flag: "-w", Type: "bool", Desc: "Print warnings about things like uninitialized variables."},
{Flag: "-W", Type: "bool", Desc: "Same as -w, but exit the process if there are any warnings."},
{Flag: "-p", Type: "bool", Desc: "Prints the expression's AST (abstract syntax tree), which gives full transparency on the precedence and associativity rules of Miller's grammar, to stdout."},
{Flag: "-d", Type: "bool", Desc: "Like -p but uses a parenthesized-expression format for the AST."},
{Flag: "-D", Type: "bool", Desc: "Like -d but with output all on one line."},
{Flag: "-E", Type: "bool", Desc: "Echo DSL expression before printing parse-tree."},
{Flag: "-v", Type: "bool", Desc: "Same as -E -p."},
{Flag: "-X", Type: "bool", Desc: "Exit after parsing but before stream-processing. Useful with -v/-d/-D, if you only want to look at parser information."},
}
var PutSetup = TransformerSetup{
Verb: verbNamePut,
UsageFunc: transformerPutUsage,
ParseCLIFunc: transformerPutOrFilterParseCLI,
IgnoresInput: false,
Options: putOptions,
}
const verbNameFilter = "filter"
var filterOptions = []OptionSpec{
{Flag: "-f", Arg: "{file name}", Type: "filename", Desc: "File containing a DSL expression (see examples below). If the filename is a directory, all *.mlr files in that directory are loaded.", Repeatable: true},
{Flag: "-e", Arg: "{expression}", Type: "string", Desc: "DSL expression to evaluate. You can use this after -f to add an expression. Example use case: define functions/subroutines in a file you specify with -f, then call them with an expression you specify with -e.", Repeatable: true},
{Flag: "-s", Arg: "{name=value}", Type: "string", Desc: "Predefines out-of-stream variable @name to have the given value. Thus mlr put -s foo=97 '$column += @foo' is like mlr put 'begin {@foo = 97} $column += @foo'. The value part is subject to type-inferencing. May be specified more than once, e.g. -s name1=value1 -s name2=value2. Note: the value may be an environment variable, e.g. -s sequence=$SEQUENCE.", Repeatable: true},
{Flag: "-x", Type: "bool", Desc: "Prints records for which {expression} evaluates to false, not true, i.e. invert the sense of the filter expression. Default false."},
{Flag: "-q", Type: "bool", Desc: "Does not include the modified record in the output stream. Useful for when all desired output is in begin and/or end blocks."},
{Flag: "-S", Type: "bool", Desc: "No-op in Miller 6 and above, since type-inferencing is now done by the record-readers before filter/put is executed. Supported as a no-op pass-through flag for backward compatibility."},
{Flag: "-F", Type: "bool", Desc: "No-op in Miller 6 and above, since type-inferencing is now done by the record-readers before filter/put is executed. Supported as a no-op pass-through flag for backward compatibility."},
{Flag: "-w", Type: "bool", Desc: "Print warnings about things like uninitialized variables."},
{Flag: "-W", Type: "bool", Desc: "Same as -w, but exit the process if there are any warnings."},
{Flag: "-p", Type: "bool", Desc: "Prints the expression's AST (abstract syntax tree), which gives full transparency on the precedence and associativity rules of Miller's grammar, to stdout."},
{Flag: "-d", Type: "bool", Desc: "Like -p but uses a parenthesized-expression format for the AST."},
{Flag: "-D", Type: "bool", Desc: "Like -d but with output all on one line."},
{Flag: "-E", Type: "bool", Desc: "Echo DSL expression before printing parse-tree."},
{Flag: "-v", Type: "bool", Desc: "Same as -E -p."},
{Flag: "-X", Type: "bool", Desc: "Exit after parsing but before stream-processing. Useful with -v/-d/-D, if you only want to look at parser information."},
}
var FilterSetup = TransformerSetup{
Verb: verbNameFilter,
UsageFunc: transformerFilterUsage,
ParseCLIFunc: transformerPutOrFilterParseCLI,
IgnoresInput: false,
Options: filterOptions,
}
func transformerPutUsage(
@ -58,59 +96,17 @@ func transformerPutOrFilterUsage(
}
fmt.Fprintf(o, "See also: https://miller.readthedocs.io/en/latest/reference-verbs\n")
fmt.Fprintf(o, "\n")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o,
`-f {file name} File containing a DSL expression (see examples below). If the filename
is a directory, all *.mlr files in that directory are loaded.
-e {expression} You can use this after -f to add an expression. Example use
case: define functions/subroutines in a file you specify with -f, then call
them with an expression you specify with -e.
(If you mix -e and -f then the expressions are evaluated in the order encountered.
Since the expression pieces are simply concatenated, please be sure to use intervening
semicolons to separate expressions.)
-s name=value: Predefines out-of-stream variable @name to have
Thus mlr put -s foo=97 '$column += @foo' is like
mlr put 'begin {@foo = 97} $column += @foo'.
The value part is subject to type-inferencing.
May be specified more than once, e.g. -s name1=value1 -s name2=value2.
Note: the value may be an environment variable, e.g. -s sequence=$SEQUENCE
-x (default false) Prints records for which {expression} evaluates to false, not true,
i.e. invert the sense of the filter expression.
-q Does not include the modified record in the output stream.
Useful for when all desired output is in begin and/or end blocks.
-S and -F: There are no-ops in Miller 6 and above, since now type-inferencing is done
by the record-readers before filter/put is executed. Supported as no-op pass-through
flags for backward compatibility.
-h|--help Show this message.
Parser-info options:
-w Print warnings about things like uninitialized variables.
-W Same as -w, but exit the process if there are any warnings.
-p Prints the expressions's AST (abstract syntax tree), which gives full
transparency on the precedence and associativity rules of Miller's grammar,
to stdout.
-d Like -p but uses a parenthesized-expression format for the AST.
-D Like -d but with output all on one line.
-E Echo DSL expression before printing parse-tree
-v Same as -E -p.
-X Exit after parsing but before stream-processing. Useful with -v/-d/-D, if you
only want to look at parser information.
`)
if verb == "put" {
WriteVerbOptions(o, putOptions)
} else {
WriteVerbOptions(o, filterOptions)
}
fmt.Fprintf(o, "\n")
fmt.Fprintf(o, "If you mix -e and -f then the expressions are evaluated in the order encountered.\n")
fmt.Fprintf(o, "Since the expression pieces are simply concatenated, please be sure to use intervening\n")
fmt.Fprintf(o, "semicolons to separate expressions.\n")
fmt.Fprintf(o, "\n")
fmt.Fprintf(o, "Parser-info options are -w, -W, -p, -d, -D, -E, -v, and -X.\n")
if verb == "put" {
fmt.Fprintln(o)

6
pkg/transformers/regularize.go
View file

@ -13,11 +13,14 @@ import (
const verbNameRegularize = "regularize"
var regularizeOptions = []OptionSpec{}
var RegularizeSetup = TransformerSetup{
Verb: verbNameRegularize,
UsageFunc: transformerRegularizeUsage,
ParseCLIFunc: transformerRegularizeParseCLI,
IgnoresInput: false,
Options: regularizeOptions,
}
func transformerRegularizeUsage(
@ -25,8 +28,7 @@ func transformerRegularizeUsage(
) {
fmt.Fprintf(o, "Usage: %s %s [options]\n", "mlr", verbNameRegularize)
fmt.Fprintf(o, "Outputs records sorted lexically ascending by keys.\n")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
WriteVerbOptions(o, regularizeOptions)
}
func transformerRegularizeParseCLI(

6
pkg/transformers/remove_empty_columns.go
View file

@ -12,11 +12,14 @@ import (
const verbNameRemoveEmptyColumns = "remove-empty-columns"
var removeEmptyColumnsOptions = []OptionSpec{}
var RemoveEmptyColumnsSetup = TransformerSetup{
Verb: verbNameRemoveEmptyColumns,
UsageFunc: transformerRemoveEmptyColumnsUsage,
ParseCLIFunc: transformerRemoveEmptyColumnsParseCLI,
IgnoresInput: false,
Options: removeEmptyColumnsOptions,
}
func transformerRemoveEmptyColumnsUsage(
@ -24,8 +27,7 @@ func transformerRemoveEmptyColumnsUsage(
) {
fmt.Fprintf(o, "Usage: %s %s [options]\n", "mlr", verbNameRemoveEmptyColumns)
fmt.Fprintf(o, "Omits fields which are empty on every input row. Non-streaming.\n")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
WriteVerbOptions(o, removeEmptyColumnsOptions)
}
func transformerRemoveEmptyColumnsParseCLI(

27
pkg/transformers/rename.go
View file

@ -13,11 +13,25 @@ import (
const verbNameRename = "rename"
var renameOptions = []OptionSpec{
{
Flag: "-r",
Type: "bool",
Desc: "Treat old field names as regular expressions. \"ab\", \"a.*b\" will match any field name containing the substring \"ab\" or matching \"a.*b\", respectively; anchors of the form \"^ab$\", \"^a.*b$\" may be used. New field names may be plain strings, or may contain capture groups of the form \"\\1\" through \"\\9\". Wrapping the regex in double quotes is optional, but is required if you wish to follow it with 'i' to indicate case-insensitivity.",
},
{
Flag: "-g",
Type: "bool",
Desc: "Do global replacement within each field name rather than first-match replacement.",
},
}
var RenameSetup = TransformerSetup{
Verb: verbNameRename,
UsageFunc: transformerRenameUsage,
ParseCLIFunc: transformerRenameParseCLI,
IgnoresInput: false,
Options: renameOptions,
}
func transformerRenameUsage(
@ -28,18 +42,7 @@ func transformerRenameUsage(
fmt.Fprintf(o, "Usage: %s %s [options] {old1,new1,old2,new2,...}\n", "mlr", verbNameRename)
fmt.Fprintf(o, "Renames specified fields.\n")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-r Treat old field names as regular expressions. \"ab\", \"a.*b\"\n")
fmt.Fprintf(o, " will match any field name containing the substring \"ab\" or\n")
fmt.Fprintf(o, " matching \"a.*b\", respectively; anchors of the form \"^ab$\",\n")
fmt.Fprintf(o, " \"^a.*b$\" may be used. New field names may be plain strings,\n")
fmt.Fprintf(o, " or may contain capture groups of the form \"\\1\" through\n")
fmt.Fprintf(o, " \"\\9\". Wrapping the regex in double quotes is optional, but\n")
fmt.Fprintf(o, " is required if you wish to follow it with 'i' to indicate\n")
fmt.Fprintf(o, " case-insensitivity.\n")
fmt.Fprintf(o, "-g Do global replacement within each field name rather than\n")
fmt.Fprintf(o, " first-match replacement.\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
WriteVerbOptions(o, renameOptions)
fmt.Fprintf(o, "Examples:\n")
fmt.Fprintf(o, "%s %s old_name,new_name\n", exeName, verb)
fmt.Fprintf(o, "%s %s old_name_1,new_name_1,old_name_2,new_name_2\n", exeName, verb)

23
pkg/transformers/reorder.go
View file

@ -15,11 +15,20 @@ import (
const verbNameReorder = "reorder"
var reorderOptions = []OptionSpec{
{Flag: "-e", Type: "bool", Desc: "Put specified field names at record end: default is to put them at record start."},
{Flag: "-f", Arg: "{a,b,c}", Type: "csv-list", Desc: "Field names to reorder."},
{Flag: "-r", Arg: "{a,b,c}", Type: "csv-list", Desc: "Treat field names as regular expressions. Matched fields are moved to start or end in record order. Example: -r '^YYY,^XXX' puts all YYY- and XXX-prefixed fields first (in record order), then the rest."},
{Flag: "-b", Arg: "{x}", Type: "string", Desc: "Put field names specified with -f before field name specified by {x}, if any. If {x} isn't present in a given record, the specified fields will not be moved."},
{Flag: "-a", Arg: "{x}", Type: "string", Desc: "Put field names specified with -f after field name specified by {x}, if any. If {x} isn't present in a given record, the specified fields will not be moved."},
}
var ReorderSetup = TransformerSetup{
Verb: verbNameReorder,
UsageFunc: transformerReorderUsage,
ParseCLIFunc: transformerReorderParseCLI,
IgnoresInput: false,
Options: reorderOptions,
}
func transformerReorderUsage(
@ -31,19 +40,7 @@ func transformerReorderUsage(
fmt.Fprint(o,
`Moves specified names to start of record, or end of record.
`)
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-e Put specified field names at record end: default is to put them at record start.\n")
fmt.Fprintf(o, "-f {a,b,c} Field names to reorder.\n")
fmt.Fprintf(o, "-r Treat field names as regular expressions. Matched fields are moved\n")
fmt.Fprintf(o, " to start or end in record order. Example: -r '^YYY,^XXX' puts all\n")
fmt.Fprintf(o, " YYY- and XXX-prefixed fields first (in record order), then the rest.\n")
fmt.Fprintf(o, "-b {x} Put field names specified with -f before field name specified by {x},\n")
fmt.Fprintf(o, " if any. If {x} isn't present in a given record, the specified fields\n")
fmt.Fprintf(o, " will not be moved.\n")
fmt.Fprintf(o, "-a {x} Put field names specified with -f after field name specified by {x},\n")
fmt.Fprintf(o, " if any. If {x} isn't present in a given record, the specified fields\n")
fmt.Fprintf(o, " will not be moved.\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
WriteVerbOptions(o, reorderOptions)
fmt.Fprintf(o, "\n")
fmt.Fprintf(o, "Examples:\n")
fmt.Fprintf(o, "%s %s -f a,b sends input record \"d=4,b=2,a=1,c=3\" to \"a=1,b=2,d=4,c=3\".\n", argv0, verb)

24
pkg/transformers/repeat.go
View file

@ -19,12 +19,27 @@ const (
const verbNameRepeat = "repeat"
var repeatOptions = []OptionSpec{
{
Flag: "-n",
Arg: "{repeat count}",
Type: "int",
Desc: "Repeat each input record this many times.",
},
{
Flag: "-f",
Arg: "{field name}",
Type: "string",
Desc: "Same as -n, but take the repeat count from the specified field name of each input record.",
},
}
var RepeatSetup = TransformerSetup{
Verb: verbNameRepeat,
UsageFunc: transformerRepeatUsage,
ParseCLIFunc: transformerRepeatParseCLI,
IgnoresInput: false,
Options: repeatOptions,
}
func transformerRepeatUsage(
@ -32,11 +47,8 @@ func transformerRepeatUsage(
) {
fmt.Fprintf(o, "Usage: %s %s [options]\n", "mlr", verbNameRepeat)
fmt.Fprintf(o, "Copies input records to output records multiple times.\n")
fmt.Fprintf(o, "Options must be exactly one of the following:\n")
fmt.Fprintf(o, "-n {repeat count} Repeat each input record this many times.\n")
fmt.Fprintf(o, "-f {field name} Same, but take the repeat count from the specified\n")
fmt.Fprintf(o, " field name of each input record.\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
fmt.Fprintf(o, "Options must be exactly one of -n or -f.\n")
WriteVerbOptions(o, repeatOptions)
fmt.Fprintf(o, "Example:\n")
fmt.Fprintf(o, " echo x=0 | %s %s -n 4 then put '$x=urand()'\n", "mlr", verbNameRepeat)
fmt.Fprintf(o, "produces:\n")

9
pkg/transformers/reshape.go
View file

@ -40,11 +40,19 @@ import (
const verbNameReshape = "reshape"
var reshapeOptions = []OptionSpec{
{Flag: "-i", Arg: "{input field names}", Type: "csv-list", Desc: "Input field names for wide-to-long reshape. Use with -o."},
{Flag: "-r", Arg: "{input field regex}", Type: "regex", Desc: "Input field regex for wide-to-long reshape. May be repeated. Use with -o. If you have multiple regexes, please specify them using multiple -r, since regexes can contain commas within them.", Repeatable: true},
{Flag: "-o", Arg: "{key-field name,value-field name}", Type: "csv-list", Desc: "Output key-field and value-field names for wide-to-long reshape. Requires -i or -r."},
{Flag: "-s", Arg: "{key-field name,value-field name}", Type: "csv-list", Desc: "Key-field and value-field names for long-to-wide reshape."},
}
var ReshapeSetup = TransformerSetup{
Verb: verbNameReshape,
UsageFunc: transformerReshapeUsage,
ParseCLIFunc: transformerReshapeParseCLI,
IgnoresInput: false,
Options: reshapeOptions,
}
func transformerReshapeUsage(
@ -70,6 +78,7 @@ func transformerReshapeUsage(
fmt.Fprintf(o, " These pivot/reshape the input data to undo the wide-to-long operation.\n")
fmt.Fprintf(o, " Note: this does not work with tail -f; it produces output records only after\n")
fmt.Fprintf(o, " all input records have been read.\n")
WriteVerbOptions(o, reshapeOptions)
fmt.Fprintf(o, "\n")
fmt.Fprintf(o, "Examples:\n")
fmt.Fprintf(o, "\n")

21
pkg/transformers/sample.go
View file

@ -12,11 +12,27 @@ import (
const verbNameSample = "sample"
var sampleOptions = []OptionSpec{
{
Flag: "-g",
Arg: "{a,b,c}",
Type: "csv-list",
Desc: "Optional: group-by-field names for samples, e.g. a,b,c.",
},
{
Flag: "-k",
Arg: "{k}",
Type: "int",
Desc: "Required: number of records to output in total, or by group if using -g.",
},
}
var SampleSetup = TransformerSetup{
Verb: verbNameSample,
UsageFunc: transformerSampleUsage,
ParseCLIFunc: transformerSampleParseCLI,
IgnoresInput: false,
Options: sampleOptions,
}
func transformerSampleUsage(
@ -27,10 +43,7 @@ func transformerSampleUsage(
`Reservoir sampling (subsampling without replacement), optionally by category.
See also %s bootstrap and %s shuffle.
`, "mlr", "mlr")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-g {a,b,c} Optional: group-by-field names for samples, e.g. a,b,c.\n")
fmt.Fprintf(o, "-k {k} Required: number of records to output in total, or by group if using -g.\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
WriteVerbOptions(o, sampleOptions)
}
func transformerSampleParseCLI(

23
pkg/transformers/sec2gmt.go
View file

@ -12,11 +12,27 @@ import (
const verbNameSec2GMT = "sec2gmt"
var sec2GMTOptions = []OptionSpec{
{Flag: "-1", Type: "bool", Desc: "Format seconds with 1 decimal place."},
{Flag: "-2", Type: "bool", Desc: "Format seconds with 2 decimal places."},
{Flag: "-3", Type: "bool", Desc: "Format seconds with 3 decimal places."},
{Flag: "-4", Type: "bool", Desc: "Format seconds with 4 decimal places."},
{Flag: "-5", Type: "bool", Desc: "Format seconds with 5 decimal places."},
{Flag: "-6", Type: "bool", Desc: "Format seconds with 6 decimal places."},
{Flag: "-7", Type: "bool", Desc: "Format seconds with 7 decimal places."},
{Flag: "-8", Type: "bool", Desc: "Format seconds with 8 decimal places."},
{Flag: "-9", Type: "bool", Desc: "Format seconds with 9 decimal places."},
{Flag: "--millis", Type: "bool", Desc: "Input numbers are treated as milliseconds since the epoch."},
{Flag: "--micros", Type: "bool", Desc: "Input numbers are treated as microseconds since the epoch."},
{Flag: "--nanos", Type: "bool", Desc: "Input numbers are treated as nanoseconds since the epoch."},
}
var Sec2GMTSetup = TransformerSetup{
Verb: verbNameSec2GMT,
UsageFunc: transformerSec2GMTUsage,
ParseCLIFunc: transformerSec2GMTParseCLI,
IgnoresInput: false,
Options: sec2GMTOptions,
}
func transformerSec2GMTUsage(
@ -29,12 +45,7 @@ func transformerSec2GMTUsage(
fmt.Fprintf(o, " %s %s time1,time2\n", "mlr", verbNameSec2GMT)
fmt.Fprintf(o, "is the same as\n")
fmt.Fprintf(o, " %s put '$time1 = sec2gmt($time1); $time2 = sec2gmt($time2)'\n", "mlr")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-1 through -9: format the seconds using 1..9 decimal places, respectively.\n")
fmt.Fprintf(o, "--millis Input numbers are treated as milliseconds since the epoch.\n")
fmt.Fprintf(o, "--micros Input numbers are treated as microseconds since the epoch.\n")
fmt.Fprintf(o, "--nanos Input numbers are treated as nanoseconds since the epoch.\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
WriteVerbOptions(o, sec2GMTOptions)
}
func transformerSec2GMTParseCLI(

10
pkg/transformers/sec2gmtdate.go
View file

@ -12,23 +12,27 @@ import (
const verbNameSec2GMTDate = "sec2gmtdate"
var sec2GMTDateOptions = []OptionSpec{}
var Sec2GMTDateSetup = TransformerSetup{
Verb: verbNameSec2GMTDate,
UsageFunc: transformerSec2GMTDateUsage,
ParseCLIFunc: transformerSec2GMTDateParseCLI,
IgnoresInput: false,
Options: sec2GMTDateOptions,
}
func transformerSec2GMTDateUsage(
o *os.File,
) {
fmt.Fprintf(o, "Usage: ../c/mlr sec2gmtdate {comma-separated list of field names}\n")
fmt.Fprintf(o, "Usage: mlr sec2gmtdate {comma-separated list of field names}\n")
fmt.Fprintf(o, "Replaces a numeric field representing seconds since the epoch with the\n")
fmt.Fprintf(o, "corresponding GMT year-month-day timestamp; leaves non-numbers as-is.\n")
fmt.Fprintf(o, "This is nothing more than a keystroke-saver for the sec2gmtdate function:\n")
fmt.Fprintf(o, " ../c/mlr sec2gmtdate time1,time2\n")
fmt.Fprintf(o, " mlr sec2gmtdate time1,time2\n")
fmt.Fprintf(o, "is the same as\n")
fmt.Fprintf(o, " ../c/mlr put '$time1=sec2gmtdate($time1);$time2=sec2gmtdate($time2)'\n")
fmt.Fprintf(o, " mlr put '$time1=sec2gmtdate($time1);$time2=sec2gmtdate($time2)'\n")
WriteVerbOptions(o, sec2GMTDateOptions)
}
func transformerSec2GMTDateParseCLI(

18
pkg/transformers/seqgen.go
View file

@ -13,27 +13,29 @@ import (
const verbNameSeqgen = "seqgen"
var seqgenOptions = []OptionSpec{
{Flag: "-f", Arg: "{name}", Type: "string", Desc: "Field name for counters. Default \"i\"."},
{Flag: "--start", Arg: "{value}", Type: "float", Desc: "Inclusive start value. Default 1."},
{Flag: "--step", Arg: "{value}", Type: "float", Desc: "Step value. Default 1. May be negative but not zero (unless start == stop)."},
{Flag: "--stop", Arg: "{value}", Type: "float", Desc: "Inclusive stop value. Default 100."},
}
var SeqgenSetup = TransformerSetup{
Verb: verbNameSeqgen,
UsageFunc: transformerSeqgenUsage,
ParseCLIFunc: transformerSeqgenParseCLI,
IgnoresInput: true,
Options: seqgenOptions,
}
func transformerSeqgenUsage(
o *os.File,
) {
fmt.Fprintf(o, "Usage: %s %s [options]\n", "mlr", verbNameSeqgen)
fmt.Fprintf(o, "Passes input records directly to output. Most useful for format conversion.\n")
fmt.Fprintf(o, "Produces a sequence of counters. Discards the input record stream. Produces\n")
fmt.Fprintf(o, "output as specified by the options\n")
fmt.Fprintf(o, "output as specified by the options.\n")
fmt.Fprintf(o, "\n")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-f {name} (default \"i\") Field name for counters.\n")
fmt.Fprintf(o, "--start {value} (default 1) Inclusive start value.\n")
fmt.Fprintf(o, "--step {value} (default 1) Step value.\n")
fmt.Fprintf(o, "--stop {value} (default 100) Inclusive stop value.\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
WriteVerbOptions(o, seqgenOptions)
fmt.Fprintf(o, "Start, stop, and/or step may be floating-point. Output is integer if start,\n")
fmt.Fprintf(o, "stop, and step are all integers. Step may be negative. It may not be zero\n")

6
pkg/transformers/shuffle.go
View file

@ -12,11 +12,14 @@ import (
const verbNameShuffle = "shuffle"
var shuffleOptions = []OptionSpec{}
var ShuffleSetup = TransformerSetup{
Verb: verbNameShuffle,
UsageFunc: transformerShuffleUsage,
ParseCLIFunc: transformerShuffleParseCLI,
IgnoresInput: false,
Options: shuffleOptions,
}
func transformerShuffleUsage(
@ -27,8 +30,7 @@ func transformerShuffleUsage(
fmt.Fprintf(o, "all input records are read. See also %s bootstrap and %s sample.\n",
"mlr", "mlr",
)
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
WriteVerbOptions(o, shuffleOptions)
}
func transformerShuffleParseCLI(

6
pkg/transformers/skip_trivial_records.go
View file

@ -11,11 +11,14 @@ import (
const verbNameSkipTrivialRecords = "skip-trivial-records"
var skipTrivialRecordsOptions = []OptionSpec{}
var SkipTrivialRecordsSetup = TransformerSetup{
Verb: verbNameSkipTrivialRecords,
UsageFunc: transformerSkipTrivialRecordsUsage,
ParseCLIFunc: transformerSkipTrivialRecordsParseCLI,
IgnoresInput: false,
Options: skipTrivialRecordsOptions,
}
func transformerSkipTrivialRecordsUsage(
@ -24,8 +27,7 @@ func transformerSkipTrivialRecordsUsage(
fmt.Fprintf(o, "Usage: %s %s [options]\n", "mlr", verbNameSkipTrivialRecords)
fmt.Fprintf(o, "Passes through all records except those with zero fields,\n")
fmt.Fprintf(o, "or those for which all fields have empty value.\n")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
WriteVerbOptions(o, skipTrivialRecordsOptions)
}
func transformerSkipTrivialRecordsParseCLI(

27
pkg/transformers/sort.go
View file

@ -53,11 +53,25 @@ import (
const verbNameSort = "sort"
var sortOptions = []OptionSpec{
{Flag: "-f", Arg: "{a,b,c}", Type: "csv-list", Desc: "Lexical ascending sort on the specified field names.", Repeatable: true},
{Flag: "-r", Arg: "{a,b,c}", Type: "csv-list", Desc: "Lexical descending sort on the specified field names.", Repeatable: true},
{Flag: "-c", Arg: "{a,b,c}", Type: "csv-list", Desc: "Case-folded lexical ascending sort on the specified field names.", Repeatable: true},
{Flag: "-cr", Arg: "{a,b,c}", Type: "csv-list", Desc: "Case-folded lexical descending sort on the specified field names.", Repeatable: true},
{Flag: "-n", Arg: "{a,b,c}", Type: "csv-list", Desc: "Numerical ascending sort on the specified field names; nulls sort last.", Repeatable: true},
{Flag: "-nf", Arg: "{a,b,c}", Type: "csv-list", Desc: "Same as -n.", Repeatable: true},
{Flag: "-nr", Arg: "{a,b,c}", Type: "csv-list", Desc: "Numerical descending sort on the specified field names; nulls sort first.", Repeatable: true},
{Flag: "-t", Arg: "{a,b,c}", Type: "csv-list", Desc: "Natural ascending sort on the specified field names.", Repeatable: true},
{Flag: "-b", Type: "bool", Desc: "Move sort fields to start of record, as in reorder -b."},
{Flag: "-tr", Aliases: []string{"-rt"}, Arg: "{a,b,c}", Type: "csv-list", Desc: "Natural descending sort on the specified field names.", Repeatable: true},
}
var SortSetup = TransformerSetup{
Verb: verbNameSort,
UsageFunc: transformerSortUsage,
ParseCLIFunc: transformerSortParseCLI,
IgnoresInput: false,
Options: sortOptions,
}
func transformerSortUsage(
@ -70,18 +84,7 @@ func transformerSortUsage(
fmt.Fprintf(o, "specified sort order.) The sort is stable: records that compare equal will sort\n")
fmt.Fprintf(o, "in the order they were encountered in the input record stream.\n")
fmt.Fprintf(o, "\n")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-f {comma-separated field names} Lexical ascending\n")
fmt.Fprintf(o, "-r {comma-separated field names} Lexical descending\n")
fmt.Fprintf(o, "-c {comma-separated field names} Case-folded lexical ascending\n")
fmt.Fprintf(o, "-cr {comma-separated field names} Case-folded lexical descending\n")
fmt.Fprintf(o, "-n {comma-separated field names} Numerical ascending; nulls sort last\n")
fmt.Fprintf(o, "-nf {comma-separated field names} Same as -n\n")
fmt.Fprintf(o, "-nr {comma-separated field names} Numerical descending; nulls sort first\n")
fmt.Fprintf(o, "-t {comma-separated field names} Natural ascending\n")
fmt.Fprintf(o, "-b Move sort fields to start of record, as in reorder -b\n")
fmt.Fprintf(o, "-tr|-rt {comma-separated field names} Natural descending\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
WriteVerbOptions(o, sortOptions)
fmt.Fprintf(o, "\n")
fmt.Fprintf(o, "Example:\n")
fmt.Fprintf(o, " %s %s -f a,b -nr x,y,z\n", "mlr", verbNameSort)

16
pkg/transformers/sort_within_records.go
View file

@ -17,11 +17,18 @@ import (
const verbNameSortWithinRecords = "sort-within-records"
var sortWithinRecordsOptions = []OptionSpec{
{Flag: "-f", Arg: "{names}", Type: "csv-list", Desc: "Sort only these keys; others preserve record order."},
{Flag: "-r", Arg: "{regex}", Type: "regex", Desc: "Sort only keys matching this regex; others preserve record order. Example: -r '^[xy]' sorts keys starting with x or y. With no regex argument, -r recursively sorts subobjects/submaps (e.g. for JSON input), or combines with -f to treat names as regexes."},
{Flag: "-n", Type: "bool", Desc: "Sort field names naturally (e.g. 2 before 12). Combines with -f/-r."},
}
var SortWithinRecordsSetup = TransformerSetup{
Verb: verbNameSortWithinRecords,
UsageFunc: transformerSortWithinRecordsUsage,
ParseCLIFunc: transformerSortWithinRecordsParseCLI,
IgnoresInput: false,
Options: sortWithinRecordsOptions,
}
func transformerSortWithinRecordsUsage(
@ -29,14 +36,7 @@ func transformerSortWithinRecordsUsage(
) {
fmt.Fprintf(o, "Usage: %s %s [options]\n", "mlr", verbNameSortWithinRecords)
fmt.Fprintln(o, "Outputs records sorted lexically ascending by keys.")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-f {names} Sort only these keys; others preserve record order.\n")
fmt.Fprintf(o, "-r {regex} Sort only keys matching this regex; others preserve record order.\n")
fmt.Fprintf(o, " Example: -r '^[xy]' sorts keys starting with x or y.\n")
fmt.Fprintf(o, " With no regex argument, -r recursively sorts subobjects/submaps\n")
fmt.Fprintf(o, " (e.g. for JSON input), or combines with -f to treat names as regex.\n")
fmt.Fprintf(o, "-n Sort field names naturally (e.g. 2 before 12). Combines with -f/-r.\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
WriteVerbOptions(o, sortWithinRecordsOptions)
}
func transformerSortWithinRecordsParseCLI(

12
pkg/transformers/sparsify.go
View file

@ -13,11 +13,17 @@ import (
const verbNameSparsify = "sparsify"
var sparsifyOptions = []OptionSpec{
{Flag: "-s", Arg: "{filler string}", Type: "string", Desc: "What values to remove. Defaults to the empty string."},
{Flag: "-f", Arg: "{a,b,c}", Type: "csv-list", Desc: "Specify field names to be operated on; any other fields won't be modified. The default is to modify all fields."},
}
var SparsifySetup = TransformerSetup{
Verb: verbNameSparsify,
UsageFunc: transformerSparsifyUsage,
ParseCLIFunc: transformerSparsifyParseCLI,
IgnoresInput: false,
Options: sparsifyOptions,
}
func transformerSparsifyUsage(
@ -29,11 +35,7 @@ func transformerSparsifyUsage(
specified value). Only makes sense with output format not being CSV or TSV.
`)
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-s {filler string} What values to remove. Defaults to the empty string.\n")
fmt.Fprintf(o, "-f {a,b,c} Specify field names to be operated on; any other fields won't be\n")
fmt.Fprintf(o, " modified. The default is to modify all fields.\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
WriteVerbOptions(o, sparsifyOptions)
fmt.Fprint(o,
`Example: if input is a=1,b=,c=3 then output is a=1,c=3.

29
pkg/transformers/split.go
View file

@ -17,31 +17,34 @@ const verbNameSplit = "split"
const splitDefaultOutputFileNamePrefix = "split"
const splitDefaultFileNamePartJoiner = "_"
var splitOptions = []OptionSpec{
{Flag: "-n", Arg: "{n}", Type: "int", Desc: "Cap output file sizes at N records."},
{Flag: "-m", Arg: "{m}", Type: "int", Desc: "Produce M files, round-robining records among them."},
{Flag: "-g", Arg: "{a,b,c}", Type: "csv-list", Desc: "Write separate files with records having distinct values for the specified field names."},
{Flag: "--prefix", Arg: "{p}", Type: "string", Desc: "Output filename prefix. Default \"split\"."},
{Flag: "--suffix", Arg: "{s}", Type: "string", Desc: "Output filename suffix. Default is from the output format, e.g. \"csv\"."},
{Flag: "--folder", Arg: "{f}", Type: "filename", Desc: "Output directory. Default is current directory."},
{Flag: "-a", Type: "bool", Desc: "Append to existing files rather than overwriting."},
{Flag: "-v", Type: "bool", Desc: "Send records downstream as well as splitting to files."},
{Flag: "-e", Type: "bool", Desc: "Do NOT URL-escape names of output files."},
{Flag: "-j", Arg: "{J}", Type: "string", Desc: "String used to join filename parts. Default \"_\"."},
}
var SplitSetup = TransformerSetup{
Verb: verbNameSplit,
UsageFunc: transformerSplitUsage,
ParseCLIFunc: transformerSplitParseCLI,
IgnoresInput: false,
Options: splitOptions,
}
func transformerSplitUsage(
o *os.File,
) {
fmt.Fprintf(o, "Usage: %s %s [options] {filename}\n", "mlr", verbNameSplit)
WriteVerbOptions(o, splitOptions)
fmt.Fprintf(o,
`Options:
-n {n}: Cap file sizes at N records.
-m {m}: Produce M files, round-robining records among them.
-g {a,b,c}: Write separate files with records having distinct values for fields named a,b,c.
Exactly one of -m, -n, or -g must be supplied.
--prefix {p} Specify filename prefix; default "`+splitDefaultOutputFileNamePrefix+`".
--suffix {s} Specify filename suffix; default is from mlr output format, e.g. "csv".
--folder {f} Specify output directory; default is current directory.
-a Append to existing file(s), if any, rather than overwriting.
-v Send records along to downstream verbs as well as splitting to files.
-e Do NOT URL-escape names of output files.
-j {J} Use string J to join filename parts; default "`+splitDefaultFileNamePartJoiner+`".
-h|--help Show this message.
`Exactly one of -m, -n, or -g must be supplied.
Any of the output-format command-line flags (see mlr -h). For example, using
mlr --icsv --from myfile.csv split --ojson -n 1000
the input is CSV, but the output files are JSON.

47
pkg/transformers/stats1.go
View file

@ -17,11 +17,27 @@ import (
const verbNameStats1 = "stats1"
var stats1Options = []OptionSpec{
{Flag: "-a", Arg: "{sum,count,...}", Type: "enum", Desc: "Names of accumulators: one or more of the listed values. Also accepts median (same as p50) and percentiles p{n} for n in 0..100, e.g. p10 p25.2 p50 p98 p100.", Values: []string{"count", "null_count", "distinct_count", "mode", "antimode", "sum", "mean", "mad", "var", "stddev", "meaneb", "skewness", "kurtosis", "min", "max", "minlen", "maxlen"}},
{Flag: "-f", Arg: "{a,b,c}", Type: "csv-list", Desc: "Value-field names on which to compute statistics."},
{Flag: "--fr", Arg: "{regex}", Type: "regex", Desc: "Regex for value-field names on which to compute statistics (compute statistics on values in all field names matching the regex)."},
{Flag: "--fx", Arg: "{regex}", Type: "regex", Desc: "Inverted regex for value-field names on which to compute statistics (compute statistics on values in all field names not matching the regex)."},
{Flag: "-g", Arg: "{d,e,f}", Type: "csv-list", Desc: "Optional group-by-field names."},
{Flag: "--gr", Arg: "{regex}", Type: "regex", Desc: "Regex for optional group-by-field names (group by values in field names matching the regex)."},
{Flag: "--gx", Arg: "{regex}", Type: "regex", Desc: "Inverted regex for optional group-by-field names (group by values in field names not matching the regex)."},
{Flag: "--grfx", Arg: "{regex}", Type: "regex", Desc: "Shorthand for --gr {regex} --fx {that same regex}."},
{Flag: "-i", Type: "bool", Desc: "Use interpolated percentiles, like R's type=7; default like type=1. Not sensical for string-valued fields."},
{Flag: "-s", Type: "bool", Desc: "Print iterative stats. Useful in tail -f contexts, in which case please avoid pprint-format output since end of input stream will never be seen. Likewise, if input is coming from `tail -f` be sure to use `--records-per-batch 1`."},
{Flag: "-S", Type: "bool", Desc: "No-op flag for backward compatibility with Miller 5."},
{Flag: "-F", Type: "bool", Desc: "No-op flag for backward compatibility with Miller 5."},
}
var Stats1Setup = TransformerSetup{
Verb: verbNameStats1,
UsageFunc: transformerStats1Usage,
ParseCLIFunc: transformerStats1ParseCLI,
IgnoresInput: false,
Options: stats1Options,
}
func transformerStats1Usage(
@ -31,35 +47,14 @@ func transformerStats1Usage(
fmt.Fprint(o,
`Computes univariate statistics for one or more given fields, accumulated across
the input record stream.
Options:
-a {sum,count,...} Names of accumulators: one or more of:
`)
WriteVerbOptions(o, stats1Options)
fmt.Fprint(o,
`Names of accumulators for -a, one or more of:
median This is the same as p50
p10 p25.2 p50 p98 p100 etc.
`)
utils.ListStats1Accumulators(o)
fmt.Fprint(o, `
-f {a,b,c} Value-field names on which to compute statistics
--fr {regex} Regex for value-field names on which to compute statistics
(compute statistics on values in all field names matching regex
--fx {regex} Inverted regex for value-field names on which to compute statistics
(compute statistics on values in all field names not matching regex)
-g {d,e,f} Optional group-by-field names
--gr {regex} Regex for optional group-by-field names
(group by values in field names matching regex)
--gx {regex} Inverted regex for optional group-by-field names
(group by values in field names not matching regex)
--grfx {regex} Shorthand for --gr {regex} --fx {that same regex}
-i Use interpolated percentiles, like R's type=7; default like type=1.
Not sensical for string-valued fields.\n");
-s Print iterative stats. Useful in tail -f contexts, in which
case please avoid pprint-format output since end of input
`)
fmt.Fprintln(o, " stream will never be seen. Likewise, if input is coming from `tail -f`")
fmt.Fprintln(o, " be sure to use `--records-per-batch 1`.")
fmt.Fprintln(o, "-h|--help Show this message.")
fmt.Fprintln(o,
"Example: mlr stats1 -a min,p10,p50,p90,max -f value -g size,shape")
@ -72,7 +67,7 @@ Options:
fmt.Fprintln(o,
` This computes count and mode statistics on all field names beginning
with a through h, grouped by all field names starting with k.`)
fmt.Println()
fmt.Fprintln(o)
fmt.Fprint(o,
`Notes:
* p50 and median are synonymous.

27
pkg/transformers/stats2.go
View file

@ -18,11 +18,23 @@ const verbNameStats2 = "stats2"
// if we were ever asked to process field names with commas in them.
const stats2KeySeparator = "\001"
var stats2Options = []OptionSpec{
{Flag: "-a", Arg: "{linreg-ols,corr,...}", Type: "enum", Desc: "Names of accumulators: one or more of the listed values.", Values: []string{"linreg-ols", "linreg-pca", "r2", "logireg", "corr", "cov"}},
{Flag: "-f", Arg: "{a,b,c,d}", Type: "csv-list", Desc: "Value-field name-pairs on which to compute statistics. There must be an even number of names."},
{Flag: "-g", Arg: "{e,f,g}", Type: "csv-list", Desc: "Optional group-by-field names."},
{Flag: "-v", Type: "bool", Desc: "Print additional output for linreg-pca."},
{Flag: "-s", Type: "bool", Desc: "Print iterative stats. Useful in tail -f contexts, in which case please avoid pprint-format output since end of input stream will never be seen. Likewise, if input is coming from `tail -f`, be sure to use `--records-per-batch 1`."},
{Flag: "--fit", Type: "bool", Desc: "Rather than printing regression parameters, applies them to the input data to compute new fit fields. All input records are held in memory until end of input stream. Has effect only for linreg-ols, linreg-pca, and logireg."},
{Flag: "-S", Type: "bool", Desc: "No-op flag for backward compatibility with Miller 5."},
{Flag: "-F", Type: "bool", Desc: "No-op flag for backward compatibility with Miller 5."},
}
var Stats2Setup = TransformerSetup{
Verb: verbNameStats2,
UsageFunc: transformerStats2Usage,
ParseCLIFunc: transformerStats2ParseCLI,
IgnoresInput: false,
Options: stats2Options,
}
func transformerStats2Usage(
@ -34,22 +46,11 @@ func transformerStats2Usage(
fmt.Fprintf(o, "Usage: %s %s [options]\n", argv0, verb)
fmt.Fprintf(o, "Computes bivariate statistics for one or more given field-name pairs,\n")
fmt.Fprintf(o, "accumulated across the input record stream.\n")
fmt.Fprintf(o, "-a {linreg-ols,corr,...} Names of accumulators: one or more of:\n")
WriteVerbOptions(o, stats2Options)
fmt.Fprintf(o, "Names of accumulators for -a, one or more of:\n")
utils.ListStats2Accumulators(o)
fmt.Fprintf(o, "-f {a,b,c,d} Value-field name-pairs on which to compute statistics.\n")
fmt.Fprintf(o, " There must be an even number of names.\n")
fmt.Fprintf(o, "-g {e,f,g} Optional group-by-field names.\n")
fmt.Fprintf(o, "-v Print additional output for linreg-pca.\n")
fmt.Fprintf(o, "-s Print iterative stats. Useful in tail -f contexts, in which\n")
fmt.Fprintf(o, " case please avoid pprint-format output since end of input\n")
fmt.Fprintf(o, " stream will never be seen. Likewise, if input is coming from\n")
fmt.Fprintf(o, " `tail -f`, be sure to use `--records-per-batch 1`.\n")
fmt.Fprintf(o, "--fit Rather than printing regression parameters, applies them to\n")
fmt.Fprintf(o, " the input data to compute new fit fields. All input records are\n")
fmt.Fprintf(o, " held in memory until end of input stream. Has effect only for\n")
fmt.Fprintf(o, " linreg-ols, linreg-pca, and logireg.\n")
fmt.Fprintf(o, "Only one of -s or --fit may be used.\n")
fmt.Fprintf(o, "Example: %s %s -a linreg-pca -f x,y\n", argv0, verb)
fmt.Fprintf(o, "Example: %s %s -a linreg-ols,r2 -f x,y -g size,shape\n", argv0, verb)

35
pkg/transformers/step.go
View file

@ -83,11 +83,21 @@ const DEFAULT_STRING_ALPHA = "0.5"
const verbNameStep = "step"
var stepOptions = []OptionSpec{
{Flag: "-a", Arg: "{delta,rsum,...}", Type: "enum", Desc: "Names of steppers: comma-separated, one or more of the listed values.", Values: []string{"counter", "delta", "ewma", "from-first", "ratio", "rprod", "rsum", "shift", "shift_lag", "shift_lead", "slwin"}},
{Flag: "-f", Arg: "{a,b,c}", Type: "csv-list", Desc: "Value-field names on which to compute statistics."},
{Flag: "-g", Arg: "{d,e,f}", Type: "csv-list", Desc: "Optional group-by-field names."},
{Flag: "-F", Type: "bool", Desc: "Computes integerable things (e.g. counter) in floating point. As of Miller 6 this happens automatically, but the flag is accepted as a no-op for backward compatibility with Miller 5 and below."},
{Flag: "-d", Arg: "{x,y,z}", Type: "csv-list", Desc: "Weights for EWMA. 1 means current sample gets all weight (no smoothing), near under 1 is light smoothing, near over 0 is heavy smoothing. Multiple weights may be specified, e.g. \"mlr step -a ewma -f sys_load -d 0.01,0.1,0.9\". Default if omitted is \"-d " + DEFAULT_STRING_ALPHA + "\"."},
{Flag: "-o", Arg: "{a,b,c}", Type: "csv-list", Desc: "Custom suffixes for EWMA output fields. If omitted, these default to the -d values. If supplied, the number of -o values must be the same as the number of -d values."},
}
var StepSetup = TransformerSetup{
Verb: verbNameStep,
UsageFunc: transformerStepUsage,
ParseCLIFunc: transformerStepParseCLI,
IgnoresInput: false,
Options: stepOptions,
}
func transformerStepUsage(
@ -95,32 +105,13 @@ func transformerStepUsage(
) {
fmt.Fprintf(o, "Usage: mlr %s [options]\n", verbNameStep)
fmt.Fprintf(o, "Computes values dependent on earlier/later records, optionally grouped by category.\n")
fmt.Fprintf(o, "Options:\n")
WriteVerbOptions(o, stepOptions)
fmt.Fprintf(o, "-a {delta,rsum,...} Names of steppers: comma-separated, one or more of:\n")
fmt.Fprintf(o, "\n")
fmt.Fprintf(o, "Names of steppers for -a, comma-separated, one or more of:\n")
for _, stepperLookup := range STEPPER_LOOKUP_TABLE {
fmt.Fprintf(o, " %-10s %s\n", stepperLookup.name, stepperLookup.desc)
}
fmt.Fprintf(o, "\n")
fmt.Fprintf(o, "-f {a,b,c} Value-field names on which to compute statistics\n")
fmt.Fprintf(o, "-g {d,e,f} Optional group-by-field names\n")
fmt.Fprintf(o, "-F Computes integerable things (e.g. counter) in floating point.\n")
fmt.Fprintf(o, " As of Miller 6 this happens automatically, but the flag is accepted\n")
fmt.Fprintf(o, " as a no-op for backward compatibility with Miller 5 and below.\n")
fmt.Fprintf(o, "-d {x,y,z} Weights for EWMA. 1 means current sample gets all weight (no\n")
fmt.Fprintf(o, " smoothing), near under 1 is light smoothing, near over 0 is\n")
fmt.Fprintf(o, " heavy smoothing. Multiple weights may be specified, e.g.\n")
fmt.Fprintf(o, " \"mlr %s -a ewma -f sys_load -d 0.01,0.1,0.9\". Default if omitted\n", verbNameStep)
fmt.Fprintf(o, " is \"-d %s\".\n", DEFAULT_STRING_ALPHA)
fmt.Fprintf(o, "-o {a,b,c} Custom suffixes for EWMA output fields. If omitted, these default to\n")
fmt.Fprintf(o, " the -d values. If supplied, the number of -o values must be the same\n")
fmt.Fprintf(o, " as the number of -d values.\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
fmt.Fprintf(o, "\n")
fmt.Fprintf(o, "Examples:\n")

39
pkg/transformers/subs.go
View file

@ -17,11 +17,24 @@ const verbNameSub = "sub"
const verbNameGsub = "gsub"
const verbNameSsub = "ssub"
var subOptions = []OptionSpec{
{Flag: "-f", Arg: "{a,b,c}", Type: "csv-list", Desc: "Field names to apply substitution to."},
{Flag: "-r", Arg: "{regex}", Type: "regex", Desc: "Regular expression for field names to apply substitution to."},
{Flag: "-a", Type: "bool", Desc: "Apply substitution to all fields."},
}
var SubSetup = TransformerSetup{
Verb: verbNameSub,
UsageFunc: transformerSubUsage,
ParseCLIFunc: transformerSubParseCLI,
IgnoresInput: false,
Options: subOptions,
}
var gsubOptions = []OptionSpec{
{Flag: "-f", Arg: "{a,b,c}", Type: "csv-list", Desc: "Field names to apply substitution to."},
{Flag: "-r", Arg: "{regex}", Type: "regex", Desc: "Regular expression for field names to apply substitution to."},
{Flag: "-a", Type: "bool", Desc: "Apply substitution to all fields."},
}
var GsubSetup = TransformerSetup{
@ -29,6 +42,13 @@ var GsubSetup = TransformerSetup{
UsageFunc: transformerGsubUsage,
ParseCLIFunc: transformerGsubParseCLI,
IgnoresInput: false,
Options: gsubOptions,
}
var ssubOptions = []OptionSpec{
{Flag: "-f", Arg: "{a,b,c}", Type: "csv-list", Desc: "Field names to apply substitution to."},
{Flag: "-r", Arg: "{regex}", Type: "regex", Desc: "Regular expression for field names to apply substitution to."},
{Flag: "-a", Type: "bool", Desc: "Apply substitution to all fields."},
}
var SsubSetup = TransformerSetup{
@ -36,6 +56,7 @@ var SsubSetup = TransformerSetup{
UsageFunc: transformerSsubUsage,
ParseCLIFunc: transformerSsubParseCLI,
IgnoresInput: false,
Options: ssubOptions,
}
func transformerSubUsage(
@ -47,11 +68,7 @@ func transformerSubUsage(
fmt.Fprintf(o, "The replacement string supports C-style backslash escapes such as \\n, \\t,\n")
fmt.Fprintf(o, "and \\x1f. Write \\\\ to get a literal backslash.\n")
fmt.Fprintf(o, "See also the `gsub` and `ssub` verbs.\n")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-f {a,b,c} Field names to convert.\n")
fmt.Fprintf(o, "-r {regex} Regular expression for field names to convert.\n")
fmt.Fprintf(o, "-a Convert all fields.\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
WriteVerbOptions(o, subOptions)
}
func transformerGsubUsage(
@ -63,11 +80,7 @@ func transformerGsubUsage(
fmt.Fprintf(o, "The replacement string supports C-style backslash escapes such as \\n, \\t,\n")
fmt.Fprintf(o, "and \\x1f. Write \\\\ to get a literal backslash.\n")
fmt.Fprintf(o, "See also the `sub` and `ssub` verbs.\n")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-f {a,b,c} Field names to convert.\n")
fmt.Fprintf(o, "-r {regex} Regular expression for field names to convert.\n")
fmt.Fprintf(o, "-a Convert all fields.\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
WriteVerbOptions(o, gsubOptions)
}
func transformerSsubUsage(
@ -79,11 +92,7 @@ func transformerSsubUsage(
fmt.Fprintf(o, "Both the search and replacement strings support C-style backslash escapes such\n")
fmt.Fprintf(o, "as \\n, \\t, and \\x1f. Write \\\\ to get a literal backslash.\n")
fmt.Fprintf(o, "See also the `gsub` and `sub` verbs.\n")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-f {a,b,c} Field names to convert.\n")
fmt.Fprintf(o, "-r {regex} Regular expression for field names to convert.\n")
fmt.Fprintf(o, "-a Convert all fields.\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
WriteVerbOptions(o, ssubOptions)
}
type subConstructorFunc func(

15
pkg/transformers/summary.go
View file

@ -67,11 +67,19 @@ var summaryDefaultSummarizerNames = []string{
"distinct_count",
}
var summaryOptions = []OptionSpec{
{Flag: "-a", Arg: "{mean,sum,etc.}", Type: "enum", Desc: "Use only the specified summarizers.", Values: []string{"field_type", "count", "null_count", "distinct_count", "mode", "sum", "mean", "stddev", "var", "skewness", "minlen", "maxlen", "min", "p25", "median", "p75", "max", "iqr", "lof", "lif", "uif", "uof"}},
{Flag: "-x", Arg: "{mean,sum,etc.}", Type: "enum", Desc: "Use all summarizers except the specified ones.", Values: []string{"field_type", "count", "null_count", "distinct_count", "mode", "sum", "mean", "stddev", "var", "skewness", "minlen", "maxlen", "min", "p25", "median", "p75", "max", "iqr", "lof", "lif", "uif", "uof"}},
{Flag: "--all", Type: "bool", Desc: "Use all available summarizers."},
{Flag: "--transpose", Type: "bool", Desc: "Show output with field names as column names."},
}
var SummarySetup = TransformerSetup{
Verb: verbNameSummary,
UsageFunc: transformerSummaryUsage,
ParseCLIFunc: transformerSummaryParseCLI,
IgnoresInput: false,
Options: summaryOptions,
}
func transformerSummaryUsage(
@ -100,12 +108,7 @@ func transformerSummaryUsage(
fmt.Fprintf(o, "* Distinct-counts are computed on string representations -- so 4.1 and 4.10 are counted as distinct here.\n")
fmt.Fprintf(o, "* If the mode is not unique in the input data, the first-encountered value is reported as the mode.\n")
fmt.Fprintf(o, "\n")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-a {mean,sum,etc.} Use only the specified summarizers.\n")
fmt.Fprintf(o, "-x {mean,sum,etc.} Use all summarizers, except the specified ones.\n")
fmt.Fprintf(o, "--all Use all available summarizers.\n")
fmt.Fprintf(o, "--transpose Show output with field names as column names..\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
WriteVerbOptions(o, summaryOptions)
}
func transformerSummaryParseCLI(

15
pkg/transformers/surv.go
View file

@ -14,23 +14,24 @@ import (
const verbNameSurv = "surv"
var survOptions = []OptionSpec{
{Flag: "-d", Arg: "{field}", Type: "string", Desc: "Name of duration field (time-to-event or censoring)."},
{Flag: "-s", Arg: "{field}", Type: "string", Desc: "Name of status field (0=censored, 1=event)."},
}
// SurvSetup defines the surv verb: Kaplan-Meier survival curve.
var SurvSetup = TransformerSetup{
Verb: verbNameSurv,
UsageFunc: transformerSurvUsage,
ParseCLIFunc: transformerSurvParseCLI,
IgnoresInput: false,
Options: survOptions,
}
func transformerSurvUsage(o *os.File) {
fmt.Fprintf(o, "Usage: %s %s -d {duration-field} -s {status-field}\n", "mlr", verbNameSurv)
fmt.Fprint(o, `
Estimate Kaplan-Meier survival curve (right-censored).
Options:
-d {field} Name of duration field (time-to-event or censoring).
-s {field} Name of status field (0=censored, 1=event).
-h, --help Show this message.
`)
fmt.Fprint(o, "\nEstimate Kaplan-Meier survival curve (right-censored).\n")
WriteVerbOptions(o, survOptions)
}
func transformerSurvParseCLI(

6
pkg/transformers/tac.go
View file

@ -11,11 +11,14 @@ import (
const verbNameTac = "tac"
var tacOptions = []OptionSpec{}
var TacSetup = TransformerSetup{
Verb: verbNameTac,
UsageFunc: transformerTacUsage,
ParseCLIFunc: transformerTacParseCLI,
IgnoresInput: false,
Options: tacOptions,
}
func transformerTacUsage(
@ -23,8 +26,7 @@ func transformerTacUsage(
) {
fmt.Fprintf(o, "Usage: %s %s [options]\n", "mlr", verbNameTac)
fmt.Fprintf(o, "Prints records in reverse order from the order in which they were encountered.\n")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
WriteVerbOptions(o, tacOptions)
}
func transformerTacParseCLI(

15
pkg/transformers/tail.go
View file

@ -12,11 +12,17 @@ import (
const verbNameTail = "tail"
var tailOptions = []OptionSpec{
{Flag: "-g", Arg: "{a,b,c}", Type: "csv-list", Desc: "Optional group-by-field names for tail counts, e.g. a,b,c."},
{Flag: "-n", Arg: "{n}", Type: "int", Desc: "Tail-count to print. Default 10. A leading '+' means start at the nth record rather than print the last n: e.g. -n +3 passes through all but the first 2 records, optionally by category."},
}
var TailSetup = TransformerSetup{
Verb: verbNameTail,
UsageFunc: transformerTailUsage,
ParseCLIFunc: transformerTailParseCLI,
IgnoresInput: false,
Options: tailOptions,
}
func transformerTailUsage(
@ -24,14 +30,7 @@ func transformerTailUsage(
) {
fmt.Fprintf(o, "Usage: %s %s [options]\n", "mlr", verbNameTail)
fmt.Fprintln(o, "Passes through the last n records, optionally by category.")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-g {a,b,c} Optional group-by-field names for tail counts, e.g. a,b,c.\n")
fmt.Fprintf(o, "-n {n} Tail-count to print. Default 10.\n")
fmt.Fprintf(o, " A leading '+' means start at the nth record rather than print\n")
fmt.Fprintf(o, " the last n: e.g. -n +3 passes through all but the first 2\n")
fmt.Fprintf(o, " records, optionally by category.\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
WriteVerbOptions(o, tailOptions)
}
func transformerTailParseCLI(

14
pkg/transformers/tee.go
View file

@ -12,27 +12,29 @@ import (
const verbNameTee = "tee"
var teeOptions = []OptionSpec{
{Flag: "-a", Type: "bool", Desc: "Append to existing file, if any, rather than overwriting."},
{Flag: "-p", Type: "bool", Desc: "Treat filename as a pipe-to command."},
}
var TeeSetup = TransformerSetup{
Verb: verbNameTee,
UsageFunc: transformerTeeUsage,
ParseCLIFunc: transformerTeeParseCLI,
IgnoresInput: false,
Options: teeOptions,
}
func transformerTeeUsage(
o *os.File,
) {
fmt.Fprintf(o, "Usage: %s %s [options] {filename}\n", "mlr", verbNameTee)
fmt.Fprintf(o, "Options:\n")
WriteVerbOptions(o, teeOptions)
fmt.Fprintf(o,
`-a Append to existing file, if any, rather than overwriting.
-p Treat filename as a pipe-to command.
Any of the output-format command-line flags (see mlr -h). Example: using
`Any of the output-format command-line flags (see mlr -h). Example: using
mlr --icsv --opprint put '...' then tee --ojson ./mytap.dat then stats1 ...
the input is CSV, the output is pretty-print tabular, but the tee-file output
is written in JSON format.
-h|--help Show this message.
`)
}

13
pkg/transformers/template.go
View file

@ -13,11 +13,18 @@ import (
const verbNameTemplate = "template"
var templateOptions = []OptionSpec{
{Flag: "-f", Arg: "{a,b,c}", Type: "csv-list", Desc: "Comma-separated field names for template, e.g. a,b,c."},
{Flag: "-t", Arg: "{filename}", Type: "filename", Desc: "CSV file whose header line will be used for template."},
{Flag: "--fill-with", Arg: "{filler string}", Type: "string", Desc: "What to fill absent fields with. Defaults to the empty string."},
}
var TemplateSetup = TransformerSetup{
Verb: verbNameTemplate,
UsageFunc: transformerTemplateUsage,
ParseCLIFunc: transformerTemplateParseCLI,
IgnoresInput: false,
Options: templateOptions,
}
func transformerTemplateUsage(
@ -27,11 +34,7 @@ func transformerTemplateUsage(
fmt.Fprintf(o, "Places input-record fields in the order specified by list of column names.\n")
fmt.Fprintf(o, "If the input record is missing a specified field, it will be filled with the fill-with.\n")
fmt.Fprintf(o, "If the input record possesses an unspecified field, it will be discarded.\n")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, " -f {a,b,c} Comma-separated field names for template, e.g. a,b,c.\n")
fmt.Fprintf(o, " -t {filename} CSV file whose header line will be used for template.\n")
fmt.Fprintf(o, "--fill-with {filler string} What to fill absent fields with. Defaults to the empty string.\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
WriteVerbOptions(o, templateOptions)
fmt.Fprintf(o, "Example:\n")
fmt.Fprintf(o, "* Specified fields are a,b,c.\n")
fmt.Fprintf(o, "* Input record is c=3,a=1,f=6.\n")

24
pkg/transformers/top.go
View file

@ -15,11 +15,23 @@ import (
const verbNameTop = "top"
const verbTopDefaultOutputFieldName = "top_idx"
var topOptions = []OptionSpec{
{Flag: "-f", Arg: "{a,b,c}", Type: "csv-list", Desc: "Value-field names for top counts."},
{Flag: "-g", Arg: "{d,e,f}", Type: "csv-list", Desc: "Optional group-by-field names for top counts."},
{Flag: "-n", Arg: "{count}", Type: "int", Desc: "How many records to print per category; default 1."},
{Flag: "-a", Type: "bool", Desc: "Print all fields for top-value records; default is to print only value and group-by fields. Requires a single value-field name only."},
{Flag: "--max", Type: "bool", Desc: "Print top largest values. This is the default."},
{Flag: "--min", Type: "bool", Desc: "Print top smallest values; default is top largest values."},
{Flag: "-F", Type: "bool", Desc: "Keep top values as floats even if they look like integers (ignored in Miller 6, kept for backward compatibility)."},
{Flag: "-o", Arg: "{name}", Type: "string", Desc: "Field name for output indices. Default \"top_idx\". Ignored if -a is used."},
}
var TopSetup = TransformerSetup{
Verb: verbNameTop,
UsageFunc: transformerTopUsage,
ParseCLIFunc: transformerTopParseCLI,
IgnoresInput: false,
Options: topOptions,
}
func transformerTopUsage(
@ -28,17 +40,7 @@ func transformerTopUsage(
argv0 := "mlr"
verb := verbNameTop
fmt.Fprintf(o, "Usage: %s %s [options]\n", argv0, verb)
fmt.Fprintf(o, "-f {a,b,c} Value-field names for top counts.\n")
fmt.Fprintf(o, "-g {d,e,f} Optional group-by-field names for top counts.\n")
fmt.Fprintf(o, "-n {count} How many records to print per category; default 1.\n")
fmt.Fprintf(o, "-a Print all fields for top-value records; default is\n")
fmt.Fprintf(o, " to print only value and group-by fields. Requires a single\n")
fmt.Fprintf(o, " value-field name only.\n")
fmt.Fprintf(o, "--min Print top smallest values; default is top largest values.\n")
fmt.Fprintf(o, "-F Keep top values as floats even if they look like integers.\n")
fmt.Fprintf(o, "-o {name} Field name for output indices. Default \"%s\".\n", verbTopDefaultOutputFieldName)
fmt.Fprintf(o, " This is ignored if -a is used.\n")
WriteVerbOptions(o, topOptions)
fmt.Fprintf(o, "Prints the n records with smallest/largest values at specified fields,\n")
fmt.Fprintf(o, "optionally by category. If -a is given, then the top records are emitted\n")
fmt.Fprintf(o, "with the same fields as they appeared in the input. Without -a, only fields\n")

11
pkg/transformers/unflatten.go
View file

@ -12,11 +12,17 @@ import (
const verbNameUnflatten = "unflatten"
var unflattenOptions = []OptionSpec{
{Flag: "-f", Arg: "{a,b,c}", Type: "csv-list", Desc: "Comma-separated list of field names to unflatten (default all)."},
{Flag: "-s", Arg: "{string}", Type: "string", Desc: "Separator, defaulting to mlr --flatsep value."},
}
var UnflattenSetup = TransformerSetup{
Verb: verbNameUnflatten,
UsageFunc: transformerUnflattenUsage,
ParseCLIFunc: transformerUnflattenParseCLI,
IgnoresInput: false,
Options: unflattenOptions,
}
func transformerUnflattenUsage(
@ -27,10 +33,7 @@ func transformerUnflattenUsage(
`Reverses flatten. Example: field with name 'a.b.c' and value 4
becomes name 'a' and value '{"b": { "c": 4 }}'.
`)
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-f {a,b,c} Comma-separated list of field names to unflatten (default all).\n")
fmt.Fprintf(o, "-s {string} Separator, defaulting to %s --flatsep value.\n", "mlr")
fmt.Fprintf(o, "-h|--help Show this message.\n")
WriteVerbOptions(o, unflattenOptions)
}
func transformerUnflattenParseCLI(

44
pkg/transformers/uniq.go
View file

@ -15,11 +15,31 @@ const verbNameCountDistinct = "count-distinct"
const verbNameUniq = "uniq"
const uniqDefaultOutputFieldName = "count"
var countDistinctOptions = []OptionSpec{
{Flag: "-f", Arg: "{a,b,c}", Type: "csv-list", Desc: "Field names for distinct count (synonym for -g)."},
{Flag: "-g", Arg: "{a,b,c}", Type: "csv-list", Desc: "Field names for distinct count."},
{Flag: "-x", Arg: "{a,b,c}", Type: "csv-list", Desc: "Field names to exclude for distinct count; use each record's other fields instead."},
{Flag: "-n", Type: "bool", Desc: "Show only the number of distinct values. Not compatible with -u."},
{Flag: "-o", Arg: "{name}", Type: "string", Desc: "Field name for output count. Default \"count\". Ignored with -u."},
{Flag: "-u", Type: "bool", Desc: "Do unlashed counts for multiple field names. With -f a,b and without -u, computes counts for distinct combinations of a and b field values. With -f a,b and with -u, computes counts for distinct a field values and counts for distinct b field values separately."},
}
var CountDistinctSetup = TransformerSetup{
Verb: verbNameCountDistinct,
UsageFunc: transformerCountDistinctUsage,
ParseCLIFunc: transformerCountDistinctParseCLI,
IgnoresInput: false,
Options: countDistinctOptions,
}
var uniqOptions = []OptionSpec{
{Flag: "-g", Arg: "{d,e,f}", Type: "csv-list", Desc: "Group-by field names for uniq counts."},
{Flag: "-f", Arg: "{d,e,f}", Type: "csv-list", Desc: "Synonym for -g."},
{Flag: "-x", Arg: "{a,b,c}", Type: "csv-list", Desc: "Field names to exclude for uniq; use each record's other fields instead."},
{Flag: "-c", Type: "bool", Desc: "Show repeat counts in addition to unique values."},
{Flag: "-n", Type: "bool", Desc: "Show only the number of distinct values."},
{Flag: "-o", Arg: "{name}", Type: "string", Desc: "Field name for output count. Default \"count\"."},
{Flag: "-a", Type: "bool", Desc: "Output each unique record only once. Incompatible with -g. With -c, produces unique records, with repeat counts for each. With -n, produces only one record which is the unique-record count. With neither -c nor -n, produces unique records."},
}
var UniqSetup = TransformerSetup{
@ -27,6 +47,7 @@ var UniqSetup = TransformerSetup{
UsageFunc: transformerUniqUsage,
ParseCLIFunc: transformerUniqParseCLI,
IgnoresInput: false,
Options: uniqOptions,
}
func transformerCountDistinctUsage(
@ -38,17 +59,7 @@ func transformerCountDistinctUsage(
fmt.Fprintf(o, "Prints number of records having distinct values for specified field names.\n")
fmt.Fprintf(o, "Same as uniq -c.\n")
fmt.Fprintf(o, "\n")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-f {a,b,c} Field names for distinct count.\n")
fmt.Fprintf(o, "-x {a,b,c} Field names to exclude for distinct count: use each record's others instead.\n")
fmt.Fprintf(o, "-n Show only the number of distinct values. Not compatible with -u.\n")
fmt.Fprintf(o, "-o {name} Field name for output count. Default \"%s\".\n", uniqDefaultOutputFieldName)
fmt.Fprintf(o, " Ignored with -u.\n")
fmt.Fprintf(o, "-u Do unlashed counts for multiple field names. With -f a,b and\n")
fmt.Fprintf(o, " without -u, computes counts for distinct combinations of a\n")
fmt.Fprintf(o, " and b field values. With -f a,b and with -u, computes counts\n")
fmt.Fprintf(o, " for distinct a field values and counts for distinct b field\n")
fmt.Fprintf(o, " values separately.\n")
WriteVerbOptions(o, countDistinctOptions)
}
func transformerCountDistinctParseCLI(
@ -157,16 +168,7 @@ func transformerUniqUsage(
fmt.Fprintf(o, "Prints distinct values for specified field names. With -c, same as\n")
fmt.Fprintf(o, "count-distinct. For uniq, -f is a synonym for -g.\n")
fmt.Fprintf(o, "\n")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-g {d,e,f} Group-by-field names for uniq counts.\n")
fmt.Fprintf(o, "-x {a,b,c} Field names to exclude for uniq: use each record's others instead.\n")
fmt.Fprintf(o, "-c Show repeat counts in addition to unique values.\n")
fmt.Fprintf(o, "-n Show only the number of distinct values.\n")
fmt.Fprintf(o, "-o {name} Field name for output count. Default \"%s\".\n", uniqDefaultOutputFieldName)
fmt.Fprintf(o, "-a Output each unique record only once. Incompatible with -g.\n")
fmt.Fprintf(o, " With -c, produces unique records, with repeat counts for each.\n")
fmt.Fprintf(o, " With -n, produces only one record which is the unique-record count.\n")
fmt.Fprintf(o, " With neither -c nor -n, produces unique records.\n")
WriteVerbOptions(o, uniqOptions)
}
func transformerUniqParseCLI(

13
pkg/transformers/unspace.go
View file

@ -12,11 +12,18 @@ import (
const verbNameUnspace = "unspace"
var unspaceOptions = []OptionSpec{
{Flag: "-f", Arg: "{x}", Type: "string", Desc: "Replace spaces with specified filler character. Default \"_\"."},
{Flag: "-k", Type: "bool", Desc: "Unspace only keys, not keys and values."},
{Flag: "-v", Type: "bool", Desc: "Unspace only values, not keys and values."},
}
var UnspaceSetup = TransformerSetup{
Verb: verbNameUnspace,
UsageFunc: transformerUnspaceUsage,
ParseCLIFunc: transformerUnspaceParseCLI,
IgnoresInput: false,
Options: unspaceOptions,
}
func transformerUnspaceUsage(
@ -24,11 +31,7 @@ func transformerUnspaceUsage(
) {
fmt.Fprintf(o, "Usage: %s %s [options]\n", "mlr", verbNameUnspace)
fmt.Fprintf(o, "Replaces spaces in record keys and/or values with _. This is helpful for PPRINT output.\n")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-f {x} Replace spaces with specified filler character.\n")
fmt.Fprintf(o, "-k Unspace only keys, not keys and values.\n")
fmt.Fprintf(o, "-v Unspace only values, not keys and values.\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
WriteVerbOptions(o, unspaceOptions)
}
func transformerUnspaceParseCLI(

13
pkg/transformers/unsparsify.go
View file

@ -13,11 +13,17 @@ import (
const verbNameUnsparsify = "unsparsify"
var unsparsifyOptions = []OptionSpec{
{Flag: "--fill-with", Arg: "{filler string}", Type: "string", Desc: "What to fill absent fields with. Defaults to the empty string."},
{Flag: "-f", Arg: "{a,b,c}", Type: "csv-list", Desc: "Specify field names to be operated on; others are not modified and operation is streaming."},
}
var UnsparsifySetup = TransformerSetup{
Verb: verbNameUnsparsify,
UsageFunc: transformerUnsparsifyUsage,
ParseCLIFunc: transformerUnsparsifyParseCLI,
IgnoresInput: false,
Options: unsparsifyOptions,
}
func transformerUnsparsifyUsage(
@ -30,12 +36,7 @@ For field names absent in a given record but present in others, fills in
a value. This verb retains all input before producing any output.
`)
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "--fill-with {filler string} What to fill absent fields with. Defaults to\n")
fmt.Fprintf(o, " the empty string.\n")
fmt.Fprintf(o, "-f {a,b,c} Specify field names to be operated on. Any other fields won't be\n")
fmt.Fprintf(o, " modified, and operation will be streaming.\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
WriteVerbOptions(o, unsparsifyOptions)
fmt.Fprint(o,
`Example: if the input is two records, one being 'a=1,b=2' and the other

8
pkg/transformers/utf8_to_latin1.go
View file

@ -13,21 +13,23 @@ import (
const verbNameUTF8ToLatin1 = "utf8-to-latin1"
var utf8ToLatin1Options = []OptionSpec{}
var UTF8ToLatin1Setup = TransformerSetup{
Verb: verbNameUTF8ToLatin1,
UsageFunc: transformerUTF8ToLatin1Usage,
ParseCLIFunc: transformerUTF8ToLatin1ParseCLI,
IgnoresInput: false,
Options: utf8ToLatin1Options,
}
func transformerUTF8ToLatin1Usage(
o *os.File,
) {
fmt.Fprintf(o, "Usage: %s %s, with no options.\n", "mlr", verbNameUTF8ToLatin1)
fmt.Fprintf(o, "Recursively converts record strings from Latin-1 to UTF-8.\n")
fmt.Fprintf(o, "Recursively converts record strings from UTF-8 to Latin-1.\n")
fmt.Fprintf(o, "For field-level control, please see the utf8_to_latin1 DSL function.\n")
fmt.Fprintf(o, "Options:\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
WriteVerbOptions(o, utf8ToLatin1Options)
}
func transformerUTF8ToLatin1ParseCLI(

2
plans/plan-2098-llm.md
View file

@ -197,6 +197,8 @@ verbs upgrade independently.
**Emitter.** Prefer `Options` when non-nil; otherwise fall back to `usage_text`.
Agents always get *something*; no big-bang migration. Optionally render each
verb's `UsageFunc` *from* `Options` so prose and JSON stay in sync.
(Done post-migration: `WriteVerbOptions` in `aaa_verb_usage.go` renders each
usage message's "Options:" block from the specs; all 70 verbs migrated.)
**Migration tracking.** Add a `VerbOptionsNilCheck` mirroring
`FLAG_TABLE.NilCheck()` (`flag_types.go:310`) wired through a `mlr help`

995
test/cases/cli-help/0001/expout
View file

File diff suppressed because it is too large Load diff

27
test/cases/help/0012/expout
View file

@ -12,22 +12,21 @@ Note: this verb lets you apply arbitrary format strings, which can produce
undefined behavior and/or program crashes. See your system's "man printf".
Options:
-i {integer format} Defaults to "%d".
Examples: "%06lld", "%08llx".
Note that Miller integers are long long so you must use
formats which apply to long long, e.g. with ll in them.
Undefined behavior results otherwise.
-f {float format} Defaults to "%f".
Examples: "%8.3lf", "%.6le".
Note that Miller floats are double-precision so you must
use formats which apply to double, e.g. with l[efg] in them.
Undefined behavior results otherwise.
-s {string format} Defaults to "%s".
Examples: "_%s", "%08s".
Note that you must use formats which apply to string, e.g.
with s in them. Undefined behavior results otherwise.
-i {integer format} Integer format string; defaults to "%d". Examples: "%06lld",
"%08llx". Note that Miller integers are long long so you
must use formats which apply to long long, e.g. with ll in
them. Undefined behavior results otherwise.
-f {float format} Float format string; defaults to "%f". Examples: "%8.3lf",
"%.6le". Note that Miller floats are double-precision so you
must use formats which apply to double, e.g. with l[efg] in
them. Undefined behavior results otherwise.
-s {string format} String format string; defaults to "%s". Examples: "_%s",
"%08s". Note that you must use formats which apply to
string, e.g. with s in them. Undefined behavior results
otherwise.
-n Coerce field values autodetected as int to float, and then
apply the float format.
-h|--help Show this message.
format (class=string #args=variadic) Using first argument as format string, interpolate remaining arguments in place of each "{}" in the format string. Too-few arguments are treated as the empty string; too-many arguments are discarded.
Examples:
format("{}:{}:{}", 1,2) gives "1:2:".

24
test/cases/help/0014/expout
View file

@ -1,11 +1,13 @@
sec2gmtdate
Usage: ../c/mlr sec2gmtdate {comma-separated list of field names}
Usage: mlr sec2gmtdate {comma-separated list of field names}
Replaces a numeric field representing seconds since the epoch with the
corresponding GMT year-month-day timestamp; leaves non-numbers as-is.
This is nothing more than a keystroke-saver for the sec2gmtdate function:
../c/mlr sec2gmtdate time1,time2
mlr sec2gmtdate time1,time2
is the same as
../c/mlr put '$time1=sec2gmtdate($time1);$time2=sec2gmtdate($time2)'
mlr put '$time1=sec2gmtdate($time1);$time2=sec2gmtdate($time2)'
Options:
-h|--help Show this message.
sec2gmt
Usage: mlr sec2gmt [options] {comma-separated list of field names}
Replaces a numeric field representing seconds since the epoch with the
@ -15,10 +17,18 @@ more than a keystroke-saver for the sec2gmt function:
is the same as
mlr put '$time1 = sec2gmt($time1); $time2 = sec2gmt($time2)'
Options:
-1 through -9: format the seconds using 1..9 decimal places, respectively.
--millis Input numbers are treated as milliseconds since the epoch.
--micros Input numbers are treated as microseconds since the epoch.
--nanos Input numbers are treated as nanoseconds since the epoch.
-1 Format seconds with 1 decimal place.
-2 Format seconds with 2 decimal places.
-3 Format seconds with 3 decimal places.
-4 Format seconds with 4 decimal places.
-5 Format seconds with 5 decimal places.
-6 Format seconds with 6 decimal places.
-7 Format seconds with 7 decimal places.
-8 Format seconds with 8 decimal places.
-9 Format seconds with 9 decimal places.
--millis Input numbers are treated as milliseconds since the epoch.
--micros Input numbers are treated as microseconds since the epoch.
--nanos Input numbers are treated as nanoseconds since the epoch.
-h|--help Show this message.
fsec2dhms (class=time #args=1) Formats floating-point seconds as in fsec2dhms(500000.25) = "5d18h53m20.250000s"
fsec2hms (class=time #args=1) Formats floating-point seconds as in fsec2hms(5000.25) = "01:23:20.250000"

16
test/cases/help/0015/expout
View file

@ -7,10 +7,18 @@ more than a keystroke-saver for the sec2gmt function:
is the same as
mlr put '$time1 = sec2gmt($time1); $time2 = sec2gmt($time2)'
Options:
-1 through -9: format the seconds using 1..9 decimal places, respectively.
--millis Input numbers are treated as milliseconds since the epoch.
--micros Input numbers are treated as microseconds since the epoch.
--nanos Input numbers are treated as nanoseconds since the epoch.
-1 Format seconds with 1 decimal place.
-2 Format seconds with 2 decimal places.
-3 Format seconds with 3 decimal places.
-4 Format seconds with 4 decimal places.
-5 Format seconds with 5 decimal places.
-6 Format seconds with 6 decimal places.
-7 Format seconds with 7 decimal places.
-8 Format seconds with 8 decimal places.
-9 Format seconds with 9 decimal places.
--millis Input numbers are treated as milliseconds since the epoch.
--micros Input numbers are treated as microseconds since the epoch.
--nanos Input numbers are treated as nanoseconds since the epoch.
-h|--help Show this message.
sec2gmt (class=time #args=1,2) Formats seconds since epoch as GMT timestamp. Leaves non-numbers as-is. With second integer argument n, includes n decimal places for the seconds part.
Examples:

24
test/cases/help/0016/expout
View file

@ -1,11 +1,13 @@
sec2gmtdate
Usage: ../c/mlr sec2gmtdate {comma-separated list of field names}
Usage: mlr sec2gmtdate {comma-separated list of field names}
Replaces a numeric field representing seconds since the epoch with the
corresponding GMT year-month-day timestamp; leaves non-numbers as-is.
This is nothing more than a keystroke-saver for the sec2gmtdate function:
../c/mlr sec2gmtdate time1,time2
mlr sec2gmtdate time1,time2
is the same as
../c/mlr put '$time1=sec2gmtdate($time1);$time2=sec2gmtdate($time2)'
mlr put '$time1=sec2gmtdate($time1);$time2=sec2gmtdate($time2)'
Options:
-h|--help Show this message.
sec2gmt
Usage: mlr sec2gmt [options] {comma-separated list of field names}
Replaces a numeric field representing seconds since the epoch with the
@ -15,10 +17,18 @@ more than a keystroke-saver for the sec2gmt function:
is the same as
mlr put '$time1 = sec2gmt($time1); $time2 = sec2gmt($time2)'
Options:
-1 through -9: format the seconds using 1..9 decimal places, respectively.
--millis Input numbers are treated as milliseconds since the epoch.
--micros Input numbers are treated as microseconds since the epoch.
--nanos Input numbers are treated as nanoseconds since the epoch.
-1 Format seconds with 1 decimal place.
-2 Format seconds with 2 decimal places.
-3 Format seconds with 3 decimal places.
-4 Format seconds with 4 decimal places.
-5 Format seconds with 5 decimal places.
-6 Format seconds with 6 decimal places.
-7 Format seconds with 7 decimal places.
-8 Format seconds with 8 decimal places.
-9 Format seconds with 9 decimal places.
--millis Input numbers are treated as milliseconds since the epoch.
--micros Input numbers are treated as microseconds since the epoch.
--nanos Input numbers are treated as nanoseconds since the epoch.
-h|--help Show this message.
nsec2gmt (class=time #args=1,2) Formats integer nanoseconds since epoch as GMT timestamp. Leaves non-numbers as-is. With second integer argument n, includes n decimal places for the seconds part.
Examples:

8
test/cases/verb-gap/0001/experr
View file

@ -1,9 +1,7 @@
Usage: mlr gap [options]
Emits an empty record every n records, or when certain values change.
One of -n or -g is required.
Options:
Emits an empty record every n records, or when certain values change.
-g {a,b,c} Print a gap whenever values of these fields (e.g. a,b,c) changes.
-n {n} Print a gap every n records.
One of -f or -g is required.
-n is ignored if -g is present.
-h|--help Show this message.
-n {n} Print a gap every n records. Ignored if -g is present.
-h|--help Show this message.