The reader's three-pass pipeline (backslash-join -> fold "+"-continuations
onto raw lines -> split each folded line on ": ") broke on fields written
as bare "Name:" (colon, no trailing space, no value) whose value is
supplied entirely by following "+" lines -- the idiomatic recutils pattern
for e.g. a "%doc:" field. Folding the continuation onto the raw line
destroyed the ": " substring needed for the later split, so parsing such a
field crashed with a "missing field separator" error, even on well-formed
input.
Found by testing against GNU recutils' own tutorial example (books.rec,
from the manual's "A Little Example" page), now added as a fixture.
Restructured parsing to split each line into a key/value field before
folding continuations, so folding happens on the value rather than the
raw line; when the preceding value is empty, the continuation becomes the
value outright instead of gaining a spurious leading newline.
Adds regression coverage (test/cases/io-recutils/0006, 0007) and a docs
section covering the empty-value + continuation pattern.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Implements the plan in plans/recutils.md, addressing #378. Adds a
hand-rolled reader/writer pair (pkg/input/record_reader_rec.go,
pkg/output/record_writer_rec.go) supporting recutils' blank-line-separated
"Key: value" records, "+"-continuation and backslash-newline continuation,
and generic comment handling, with recutils' %rec-descriptor schema layer
left unspecialized since Miller has no schema-enforcement concept.
Registers the format under the "recutils" name (--irecutils/--orecutils/
--recutils flags, matching the --idcf/--odcf/--dcf pattern) in the
reader/writer factories and pkg/cli/separators.go, adds docs and a sample
data file, and adds unit tests plus test/cases/io-recutils regression
cases covering round-tripping, continuation lines, comments, and the
hard-error behavior on malformed input.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Rebuild docs: help-catalog index count drifted on main (666 -> 667)
Pre-existing drift from a recent catalog addition; surfaced by make dev.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Remove os.Exit callsites below the entrypoint: phase 1 (plans/exit.md)
Phase 1 of plans/exit.md: the mechanical swaps, plus the single-exit-point
scaffolding.
- New sentinel lib.ExitRequest{Code} (io.EOF-style control flow): returned by
code paths that have already printed what they wanted (--version, -h,
'put --explain' on a valid expression, 'put -x') instead of exiting mid-stack.
- pkg/entrypoint: new exitOnError() maps errors to exit codes in one place:
ErrHelpRequested -> 0, ErrUsagePrinted -> 1, ExitRequest -> its code,
anything else printed (JSON if --errors-json) -> 1.
- pkg/climain: version/help/usage/validation exits become returned errors;
the ErrHelpRequested/ErrUsagePrinted -> exit translation moves up to the
entrypoint; loadMlrrcOrDie becomes error-returning loadMlrrcFiles.
- Transformer ParseCLI/constructor exits -> returned errors (cut,
having_fields, nest, reorder, merge_fields, reshape, rename, split, tee,
subs, put_or_filter). merge_fields' bad-regex message formerly mis-reported
itself as coming from the cut verb; tee's unrecognized-option exit was
formerly silent; split/tee constructor failures formerly exited without
printing the constructor's error.
- YAML/JSON record-writer marshal errors -> returned through Write, matching
the CSV writer's error path.
- lib.WriteTempFileOrDie -> WriteTempFile (string, error); its sole caller
(regtest diff helper) degrades gracefully.
Stderr messages and exit codes are byte-identical for all regression-covered
paths (4779 cases pass); runtime Transform-path exits are phase 3.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add lib.NewExitZeroRequest() constructor for exit-0 sentinel returns
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Treat YAML like JSON for auto-flatten/auto-unflatten (#2195)
Auto-flatten/unflatten decisions only checked for "json", so nested
data written to YAML was needlessly flattened and non-JSON data
converted to YAML was never unflattened into arrays/maps. Both formats
support native nesting, so both should be treated the same way.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Clarify why DCF is excluded from auto-flatten separately from JSON/YAML
DCF isn't a nestable format like JSON/YAML -- it just has its own
hardcoded comma-list serialization for a fixed set of field names
that generic key-spreading flatten would clobber. Split the comment
so that distinction isn't lost.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Refer to JSON/YAML throughout the flatten-unflatten doc
The summary paragraph calling out YAML as an aside did not read
naturally next to prose that otherwise only said JSON. Fold YAML
into the headings and body text directly instead.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
docs/src/mlr.1 is a leftover copy of the man page that no build target
generates or consumes: the maintained man page is man/mlr.1, produced by
man/Makefile (mkman.rb), and nothing in docs/ or any Makefile references
the docs/src copy. It has drifted stale, which multiple contributors have
independently noticed while regenerating docs.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
NF is re-evaluated at each reference, tracking the current field count
of the record as it is modified -- it is not constant per record. This
surprised a user whose C-style for-loop testing i <= NF, with a body
that added fields, became an infinite loop; the docs sentence "their
values ... change from one record to the next" implied per-record
constancy.
Changes:
- reference-dsl-variables.md.in: add an explicit paragraph and live
example showing NF changing mid-expression as fields are added and
removed, plus a caution about for-loops bounded by NF whose bodies
add fields, with the two idiomatic alternatives (snapshot NF first,
or a key-value for-loop over $* which iterates over a copy).
- reference-dsl-control-structures.md.in: note in the while/do-while
section that those examples rely on NF's dynamism, and add a
triple-for note warning about non-terminating NF-bounded loops.
- glossary.md.in: note dynamism in the NF entry.
- pkg/dsl/cst/keyword_usage.go: extend 'mlr help keyword NF' text
(flows into the manpage and the usage-keywords docs section).
- Also fix a pre-existing typo nearby: "if a field has 5 records" ->
"if a record has 5 fields".
- Regenerated docs and man pages.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Adds a new verb 'mlr bootstrap-ci' which computes bootstrap confidence
intervals for stats1-style statistics: values of the requested fields are
resampled with replacement -n times (default 1000), the statistic is computed
on each resample, and the confidence interval is taken from percentiles of
the resampled statistics at the requested confidence level (-c, default
0.95). For each value field and statistic it emits {field}_{stat} (the
full-data point estimate) along with {field}_{stat}_lo and {field}_{stat}_hi,
optionally grouped by -g. Results are reproducible with mlr --seed.
Includes unit tests, regression-test cases (test/cases/verb-bootstrap-ci),
and documentation updates with regenerated man/doc pages.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Add named profile sections to .mlrrc, selected via --profile / -P (#358)
.mlrrc files may now contain INI-style named sections ("profiles"):
# Global settings, applied always:
icsv
[j]
ojson
jvstack
[tsvout]
otsv
Lines before any section header are global settings, applied always, so
existing .mlrrc files behave exactly as before. The new main flag
--profile {name} (alias -P {name}) applies the settings from the [name]
section after the global settings; without it, sections are ignored
entirely (their lines aren't even parsed).
It's a fatal error if a requested profile has no matching section in any
.mlrrc file processed, if no .mlrrc file was found at all, or if
--profile is combined with --norc or MLRRC=__none__.
Also fixes the regression-tester to restore per-case environment
variables to their prior values after each case, rather than setting
them to the empty string -- needed so cases pointing MLRRC at a test
file don't clobber the suite-wide MLRRC=__none__ guard.
Includes unit tests, regression-test cases, and doc/man-page updates.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Reject --profile / -P within .mlrrc files (#358)
Per maintainer feedback on the PR: profiles are selected on the mlr
command line, not from within a .mlrrc file, so --profile / -P inside a
.mlrrc file is now a parse error -- the same way --prepipe is rejected
there -- rather than being silently ignored.
Adds a unit test, a should-fail regression case, and a docs bullet.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Allow step -a shift_lag/shift_lead/delta/ratio to specify records back/forward (#1002)
Extends the slwin_m_n naming convention to more steppers: shift_lag_12
refers 12 records back, shift_lead_4 refers 4 records forward, and
likewise delta_N, ratio_N, and shift_N. The unsuffixed forms keep their
existing 1-record-back/forward behavior and output field names.
Backward-looking steppers cache copies of previous values in a
fixed-size ring buffer (tValueRing) in their own state, following the
existing pattern of not reading from the window keeper's already-emitted
records, to avoid races with downstream transformers. shift_lead_N uses
the existing forward-window machinery shared with slwin.
Also adds nil guards to all steppers' process functions for the case
where the record at the window center lacks the stepped field, which
previously panicked (pre-existing on main, e.g.
"mlr step -a delta,shift_lead -f x" or "-a slwin_1_1" over
heterogeneous data), and which the new parameterized combinations make
easier to reach.
Includes unit tests, regression cases 0024-0027, and regenerated help
text / docs / man page.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Clarify step -a help text: both plain and _{n} stepper forms are valid (#1002)
Per review on #2190: descriptions like "E.g. delta_7 for 7 records
back" read as if the suffixed form were the only syntax. Reword the
shift, shift_lag, shift_lead, delta, and ratio descriptions to state
both forms explicitly -- e.g. "Use delta or equivalently delta_1 for
the previous record, or delta_{n} for n records back" -- using the {n}
placeholder style, and update the matching usage paragraph and
reference-verbs prose. Regenerate help-derived docs, man page, and
cli-help regression expectation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Give a specific error for invalid stepper count suffixes (#1002)
Per review on #2190: "mlr step -a delta_0" (or delta_-1, delta_x, etc.)
previously reported the generic 'stepper "delta_0" not found'. Now,
when the name prefixes a known count-suffix stepper (shift, shift_lag,
shift_lead, delta, ratio) but the suffix is not a positive integer, the
error is:
mlr step: stepper "delta_0": count must be a positive integer
Truly unknown stepper names keep the existing "not found" message.
Adds should-fail regression cases for delta_0 and delta_-1, and unit
tests for the detection helper.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Add sliding-window mode to stats1 via -w {n} (#1017)
This adds a generic sliding-window option to the stats1 verb, per the
feature request in #1017. With 'mlr stats1 ... -w {n}', statistics are
computed over a trailing window of up to n records -- per group when -g
is used -- and one output record is emitted per input record, with the
windowed statistics appended to it.
The implementation retains, per grouping key, copies of only the
relevant value fields from the last up-to-n records. On each input
record the group's accumulators are reset and re-fed from the window.
This is O(window size) per record, but is fully generic: it works
uniformly for every stats1 accumulator, including order-sensitive ones
(mode, antimode) and non-invertible ones (percentiles, distinct_count),
and composes with -g, --fr/--fx, and --gr/--gx.
Includes unit tests, regression cases (test/cases/verb-stats1/0020-0025),
and regenerated docs/man/help-text artifacts.
Addresses #1017.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* neaten
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Adds a section to the questions-about-joins page showing how to update
matching records of a durable CSV file from a download containing a
subset of records and a subset of columns, using join --ul with the
database as the left file (right-file values win on collision), plus
variants for restoring row order and for keeping database values when
the download has empty cells.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Add a "Transposing very wide data" section to the shapes-of-data page,
showing two existing-verb approaches for viewing wide CSVs rotated 90
degrees: XTAB output format for a per-record vertical view, and a DSL
out-of-stream-variable recipe (thanks @Fravadona) for a full transpose
where each input column becomes an output row.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Miller's strftime delegates to the full-featured lestrrat-go/strftime
library, but strptime uses an in-tree fork of a small subset-only
package, so strftime output couldn't always be parsed back by
strptime with the same format string. This adds the format codes
needed to round-trip common formats like "%a %b %e %T %Y":
* %a / %A (weekday name) and %h (alias of %b): straightforward
formatMap entries, same pattern as the existing %b/%B.
* %e (space-padded day of month): needed dedicated width-detection
logic, since %e's own optional leading pad space is otherwise
indistinguishable from a literal separator space when the code
searches for the next literal text to bound a field -- this was
the root cause of the round-trip failure in the linked issue.
* %c, %x, %X: shorthand aliases (expanding to %a %b %e %H:%M:%S %Y,
%m/%d/%y, and %H:%M:%S respectively), same mechanism already used
for %T/%D/%F/%R/%r.
Also documents the %D/%F/%r/%R/%T shorthands in the strptime table
in reference-dsl-time.md.in, which were already supported but
missing from the docs -- this was the exact confusion reported in
the discussion linked from #1518.
Round-trip tests added in pkg/bifs/datetime_test.go assert
strptime(strftime(t, fmt), fmt) == t across the newly-supported
formats, plus table-driven cases in
pkg/pbnjay-strptime/strptime_test.go covering %e's edge cases
(padded/unpadded single digit, double digit, adjacent to another
code with no separator, at end of string).
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* stats1: add rank accumulator (#383)
Adds `mlr stats1 -a rank` for standard competition ranking (1,2,2,4,...)
on pre-sorted data, most useful with -s for a rank on every record.
* stats1: make rank order-independent by default, add --rank-sorted opt-in fast path
The rank accumulator previously only compared each value to the immediately
preceding record, silently giving wrong ranks for non-adjacent duplicates
(e.g. unsorted input, or interleaved -g groups). Default to correctly
computing standard competition rank from all values seen so far
(order-independent, buffers values, same approach as percentile
accumulators). Add --rank-sorted for callers who can promise sorted input
and want the previous O(1)-space streaming behavior instead.
* Revert "stats1: make rank order-independent by default, add --rank-sorted opt-in fast path"
This reverts commit aa45a591fe.
* Revert "stats1: add rank accumulator (#383)"
This reverts commit 96deed048a.
* Add mlr rank verb (#383)
Reverts the earlier stats1 -a rank / --rank-sorted approach: stats1 is a
reduce verb (many records -> one summary record per group) and rank is a
per-record annotator, so it never fit cleanly there -- it needed stats1's
-s iterative-stats escape hatch just to be useful, plus a bolted-on
sorted/unsorted split.
mlr rank is a dedicated verb modeled on mlr fraction: -f fields to rank,
-g optional group-by, output field <f>_rank. By default it's a two-pass
algorithm (buffers input, like fraction does) giving standard competition
rank (1,2,2,4,...) that's correct regardless of input order. --sorted
opts into a single-pass, O(1)-space streaming alternative for callers who
can promise pre-sorted input (e.g. via 'mlr sort' first).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Regenerate docs/man pages after merging main (sparkline verb)
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Adds a sparkline(array|map) built-in that renders numeric values as a
Unicode block-character string (▁▂▃▄▅▆▇█), plus a new `mlr sparkline`
verb that summarizes each field's values in record order -- useful for
eyeballing trends without external plotting tools.
Also adds `-s` to `mlr histogram` to sparkline a field's binned counts
(its distribution shape) rather than emitting one record per bin. This
is a different chart from `mlr sparkline` (order-independent binning
vs. record-order values), and the docs for each cross-reference the
other to avoid conflating them.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Publish an epub of the docs on Read the Docs (#1835)
Read the Docs' built-in formats support (the existing formats: all in
.readthedocs.yaml) only produces epub/PDF for Sphinx projects, and is a
silent no-op for MkDocs ones. Instead, per RTD's documented
build-customization path, generate the epub ourselves in a post_build
job and place it in $READTHEDOCS_OUTPUT/epub/, which RTD then publishes
on the project Downloads page and in the docs flyout menu.
The epub itself is built by the new docs/build-epub.sh: it takes the
committed, generated Markdown pages in docs/src in mkdocs.yml nav
order, strips the HTML-only quicklinks header from each page, and runs
pandoc (installed on RTD via build.apt_packages). Locally, `make -C
docs epub` does the same for anyone with pandoc installed; nothing here
is part of `make dev` or any default build path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix misrender
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The DKVPX record reader set dkvpxReader.Comma = ',' unconditionally,
ignoring the resolved --ifs value; the key-value separator '=' was
likewise hard-coded in the dkvpx parser, ignoring --ips. So e.g.
printf 'a=1;b=2\n' | mlr -i dkvpx --ifs semicolon --ojson cat
produced a single field {"a": "1;b=2"} instead of {"a": 1, "b": 2}.
Fixes:
* pkg/dkvpx.Reader gains an Equals field (default '=') alongside Comma.
* The DKVPX record reader now passes the resolved IFS/IPS through,
validating that each is a single character (mirroring the CSV
reader's "IFS can only be a single character" error) rather than
silently ignoring or truncating them.
* The DKVPX record writer already emitted OFS/OPS, but its quoting
decisions were based on the default separators, so with --ofs
semicolon a value containing ';' went out unquoted (ambiguous) while
a value containing ',' was quoted needlessly. Quoting now keys off
the actual OFS/OPS.
Adds unit tests, regression cases under test/cases/io-dkvpx/, and a
note in the separators reference table.
Discovered while investigating #369; tracked in #2170.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The behavior reported in #999 matches Go's encoding/csv LazyQuotes
semantics (and Python's csv module): a field beginning with an unmatched
double quote is a quoted field whose contents extend across field
separators and newlines until the next quote character or end of file.
Add a docs section explaining what --lazy-quotes does and does not
relax, with a worked example, and point to CSV-lite as the alternative
when quote characters are ordinary data.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Adds a section to the questions-about-joins page showing how to merge
N files side-by-side on a shared key when the files' non-key column
names are all the same: a then-chain of join verbs with per-file --lp
prefixes (plus --lk / cut / label to trim, and --ul --ur / unsparsify
for outer-join semantics). Addresses #1775.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Add a section to the record-heterogeneity page showing how to
concatenate multiple CSV files whose headers differ, using unsparsify
(and unsparsify -f then regularize for the streaming case), with new
sample data files under docs/src/data/het/.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Add a worked-examples section to the questions-about-joins page showing
how to get SQL-style join semantics (keeping non-matching records) with
mlr join --ul/--ur followed by unsparsify, using the example data from
issue #652.
Resolves#652.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Adds a recipe section to the questions-about-joins page showing two
ways to preprocess the join verb's -f left file without creating an
intermediate file: the main-level --prepipe flag (which the left file
inherits along with the other main input options), and shell process
substitution. Also notes that the verb-level --prepipe/--prepipex
flags listed in 'mlr join --help' do not currently take effect for
the left file in Miller 6.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Documents the "TZ environment variable appears malformed" / "time:
invalid location name" startup errors: TZ set to a file path (which
Go's time library rejects, unlike C), and missing timezone databases
on Windows, Cygwin, and minimal containers. Lists remedies: IANA zone
names, installing tzdata, the ZONEINFO environment variable, and
fixed-offset formats.
Addresses #1238; also relevant to #1406 and #1758.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
With --right-align-numeric, PPRINT data cells right-align but headers
stayed left-aligned, so a header did not line up with its own column's
data -- the original ask in #380. Now a header is right-aligned when
every value in its column is numeric, for both non-barred and barred
PPRINT output. Mixed columns keep left-aligned headers.
For --omd-aligned, the raw header text of right-aligned columns is now
right-justified too, matching how Markdown viewers render the ---:
marker; this follows the same all-values-numeric per-column rule
already used for the separator markers.
Man-page regeneration also picks up previously-merged reorder help-text
edits that had not been regenerated.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The uniq verb outputs only the group-by columns, and issue #1075 asks
for a way to deduplicate on some fields while keeping the rest. Miller
already supports this via "head -n 1 -g", but that wasn't discoverable
from uniq's help text or its reference-verbs section. Add a note to
"mlr uniq --help" and a short recipe (with live examples) to the uniq
section of the verbs reference, and regenerate the derived man page,
docs, and CLI-help golden files.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Plain {} placeholders consume arguments sequentially as before; {N}
refers to the Nth argument (1-based) and may be used to repeat and/or
reorder arguments, e.g. format("{1}/{2}/{1}_{3}.ext", $p1, $p2, $p3).
Mixing is allowed: the {} sequence counter is independent of positional
placeholders (as in Rust format!). Out-of-range indices interpolate the
empty string, consistent with existing too-few-arguments behavior; {0}
is an error value since indices are 1-based.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
With multiple regexes, 'mlr reorder -r' previously emitted all matched
fields in record order, ignoring the order the regexes were given. Now
matched fields are grouped by regex-list order (first regex's matches
first, then the second's, etc.); within each group, fields keep their
record order. A field matching multiple regexes is claimed by the first
one. This makes 'reorder -r' consistent with 'cut -orf' and satisfies
the original request in #1325: -r '^YYY,^XXX' puts YYY-prefixed fields
first, then XXX-prefixed fields, then the rest. Applies to -e, -b, and
-a modes as well.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
For PPRINT, right-justifies data cells whose values are numeric,
leaving other cells and header lines left-justified. For Markdown,
emits right-alignment markers (---:) for numeric columns; in
--omd-aligned mode also right-justifies the raw cell text.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Honor --ors CRLF for CSV output (#1810)
The full-CSV record writer validated ORS as newline or
carriage-return/newline, but never propagated the choice to the forked
Go CSV writer's UseCRLF field, so `--ors '\r\n'` (or `--ors crlf`)
silently produced LF line endings. Set UseCRLF from the writer options
so CRLF output is honored. Default behavior (LF) is unchanged, and
other ORS values are still rejected. The CSV-lite and TSV writers
already honored CRLF ORS.
Also: fix a copy-paste "for CSV" in the TSV writer's ORS-validation
message, add unit tests asserting byte-exact line endings (the regtest
harness normalizes CR/LF, so this can't be asserted in test/cases), add
CLI-level regression cases, and update the separators documentation.
This substantially addresses #1722 as well: RFC-4180-style CRLF output
can now be requested on any platform.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix errcheck lint: check Flush() error in CSV writer test
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The summary verb's field_type column shows values like "string-int" or
"empty-string" for columns containing values of mixed inferred types:
all types encountered across records are printed, hyphen-joined, in the
order first encountered. Add a note to the verb's help text explaining
this, and regenerate the man page, docs, and golden test output.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Miller type-inference accepts float formats which the JSON grammar
disallows -- leading zeros like 004.56, leading plus signs, bare leading
or trailing decimal points like .56 or 4., etc. -- and the JSON writer
passed the original text through verbatim, producing invalid JSON.
Leading-zero integers such as 0123456789 were already handled (inferred
as strings), and hex/binary/octal ints were already re-rendered as
decimal for JSON per #1761; this does the same for floats: when a
float string representation is not a valid JSON number, re-render it
(e.g. 004.56 -> 4.56).
Also documents in reference-main-data-types that CSV double-quoting
does not affect type inference, with -S / --infer-none and string()
as the ways to keep such values as strings.
Addresses #1114 and #1293.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The join verb's help text listed only '-i {one of csv,dkvp,nidx,pprint,xtab}'
for overriding the left-file input format, but the verb also accepts the
--icsv/--ijson-style main-flag shorthands (and formats beyond the five
listed, e.g. json and tsv), since unrecognized verb flags fall through to
the main flag table. Update the usage text to say so, and regenerate the
man page and docs content that embed this help output.
Fixes#444.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The uniq verb writes its group-by fields in the order they are named
with -g or -f, not in the order they appear in the input records
(unlike cut, which preserves input order unless -o is given). Per
discussion on #962, document this in the verb help text rather than
change long-standing output ordering. Regenerate the man page and the
docs pages which embed the verb help.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* docs: add "Miller and AI agents" quick-start page (#2098)
Umbrella page for the AI-friendly feature stack: one-line MCP setup as
the fast path, plus the plain-CLI path (which, help --as-json,
describe, --explain, --errors-json, --no-shell) with live CI-tested
examples, and the discover -> constrain -> validate -> run loop.
Cross-linked with the MCP server page; listed under Getting started.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: reframe AI page as "Miller and AI" (#2098)
Rename ai-agents.md to ai.md and restructure around the pre-MCP
feature stack, organized as the loop each feature serves: Discover
(catalog/index/which, cache keys, single-sourced usage text),
Constrain (enum value-sets + describe: tool shape vs data shape),
Validate (--explain), Run and recover (--errors-json, --no-shell,
env-var trio). MCP is now one closing section pointing at the
mcp-server.md detail page. All examples are live and CI-tested,
including Miller querying its own catalog.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: retitle MCP page to "The MCP server" under the Miller-and-AI umbrella
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: add bare-minimum getting-started section to Miller-and-AI page
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: AI features land in Miller 6.20
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: hyperlink SKILL.md references to the repo
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: rename section to 'The essentials'
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Plan: flesh out PR7 (MCP server + Agent Skill) design
stdio transport (no HTTP port), mlr mcp terminal in the main binary,
SDK-vs-handroll decision, tool list, in-process vs subprocess split,
run-tool safety (--no-shell prerequisite), single-sourced skill, tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add mlr mcp: MCP server + agent playbook; --no-shell gate (#2098 PR7)
New terminal `mlr mcp` runs a Model Context Protocol server over stdio
(spawned by MCP clients; no network port), exposing five tools --
list_capabilities, which, validate_dsl, describe_data, run -- plus an
agent playbook as MCP prompt/resource. Catalog tools are served
in-process from the help registries; the rest subprocess this same
binary with MLR_ERRORS_JSON=1, a timeout, and an output cap.
Prerequisite: a new --no-shell flag / MLR_NO_SHELL env var (one-way
gate) disables the DSL system/exec functions, piped redirects, and
--prepipe/--prepipex; the MCP server sets it on the commands it runs
unless started with --allow-shell.
Adds the github.com/modelcontextprotocol/go-sdk dependency.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Force LF checkout for the embedded SKILL.md (Windows CI fix)
go:embed embeds checkout bytes, so a CRLF checkout on Windows made the
embedded playbook differ per platform and failed
TestPlaybookHasFrontmatter. Pin the file to eol=lf in .gitattributes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Move no-shell test DSL into per-case mlr files (Windows CI fix)
Inline single-quoted DSL in cmd files is mangled by the Windows shell
(single quotes are not quote characters there); the harness's
put -f ${CASEDIR}/mlr pattern avoids shell quoting entirely.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
One output record per input field: types seen with counts, occurrence
count, null count, cardinality, min/max, and -- for fields within the
-n/--max-values cap -- the complete distinct-value list in first-seen
order. `mlr --ojson describe` is the machine-readable form; nested
types/values flatten in tabular formats.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Lets an agent type-check a DSL expression before spending a full input
pass. `mlr put --explain '...'` (and filter) runs the existing
parse -> ValidateAST -> CST build -> Resolve path, then:
- valid: prints "mlr {put,filter}: DSL expression is valid." and exits 0
- invalid: returns the build error up the normal path, so --errors-json
emits a structured document; exits 1
- -W with fatal warnings: reports and exits 1
The gate lives in the pass-two constructor, before any input file is
opened, so no input stream is read (verified with a nonexistent input
file still validating OK).
Also categorize bare "parse error: ..." messages from the DSL parser as
kind "dsl-parse-error" rather than "generic" (climain/errors_json.go),
so --explain --errors-json gives an agent a useful error kind. The CSV
reader's "parse error on line ..." is stream-time and never reaches this
command-line-parse categorizer.
Tests: dsl-explain/0001-0004 regression cases (valid put/filter, invalid
plain, invalid --errors-json) and categorize unit tests. Regenerated
verb docs, manpage, and the help usage-verbs golden case.
The older -X ("exit after parsing") still exits 0 even on a parse error;
left as-is since --explain is the correct validation path.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Tier-2 structured verb options: OptionSpec, initial migration (#2098)
PR 3 of the AI-friendly roadmap (plans/plan-2098-llm.md).
Infrastructure:
- Add OptionSpec{Flag,Arg,Type,Desc,Repeatable,Values} to
pkg/transformers/aaa_record_transformer.go alongside TransformerSetup.
Type is one of: bool, string, int, float, csv-list, regex, filename,
format, enum. For type=="enum", Values lists the valid choices.
- Add Options []OptionSpec to TransformerSetup (nil = not yet migrated).
- Emit Options in VerbInfoForJSON (omitempty so unmigrated verbs stay
backward-compatible; agents check key presence for Tier-2 availability).
UsageText is always present as the Tier-1 prose fallback.
- Add VerbOptionsNilCheck() in aaa_verb_options_check.go: progress report
of migrated vs. unmigrated verbs, analogous to FLAG_TABLE.NilCheck().
- Wire verb-options-nil-check into mlr help (internal/docgen section).
Initial migration (5/70 verbs):
- nothing: empty Options (no verb-specific options, explicitly migrated)
- cat: -n (bool), -N (string), -g (csv-list), --filename, --filenum (bool)
- head: -g (csv-list), -n (int)
- tail: -g (csv-list), -n (int)
- tee: -a, -p (bool)
Tests:
- 5 new unit tests in aaa_transformer_json_test.go covering migrated/
unmigrated paths, field population, JSON round-trip, and key-presence.
- Regression test case 0003: mlr help verb-options-nil-check golden output.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Migrate all 70 verbs to structured OptionSpec; bump catalog schema to v2
Completes the Tier-2 migration started in the previous commit. Every verb
in TRANSFORMER_LOOKUP_TABLE now has a non-nil Options field.
- Workflow-migrated all 65 remaining verbs. Each Setup var now carries
Options: []OptionSpec{...} with Flag/Arg/Type/Desc fields. Verbs with
no verb-specific options (altkv, check, group-like, nothing, etc.) use
an empty slice to signal "migrated but no options."
- Drop `omitempty` from VerbInfoForJSON.Options: empty slices were silently
dropped, making migrated-no-option verbs indistinguishable from unmigrated
ones in JSON. Without omitempty: null=unmigrated, []=migrated-no-options,
[...]= migrated-with-options. Bump catalogSchemaVersion 1→2 for this shape
change.
- Replace the two "unmigrated-verb" unit tests (which used stats1 as an
example) with TestAllVerbsFullyMigrated (asserts every verb has non-nil
Options) and TestAllVerbsHaveOptionsKeyInJSON (asserts every migrated
verb emits the "options" key in JSON).
- Regenerate test/cases/cli-help/0003/expout: now reads
"Verb options migration: 70/70 migrated."
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* remove a transitional helper
* git rms
* Render verb usage Options blocks from structured OptionSpec
Each verb's usage message and its Tier-2 OptionSpec list previously
duplicated the option text. New WriteVerbOptions (aaa_verb_usage.go)
renders the "Options:" block from the specs: aligned flag column,
descriptions word-wrapped at 80, uniform trailing -h|--help line.
- OptionSpec gains Aliases (JSON "aliases") so long-form spellings
like join's --lk|--left-keep-field-names survive in both outputs
- All 70 verbs migrated; options literals hoisted to package-level
vars (usage funcs can't reference their Setup var without a Go
init cycle)
- Hand-written per-option details the specs had condensed away are
merged into Desc, enriching the JSON catalog
- Non-option prose (examples, cross-references, dynamic accumulator
listings) kept verbatim
- Regenerated the six usage-embedding regression expectations and
the two affected doc pages
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix pre-existing usage-text bugs surfaced by the OptionSpec migration
- gap: usage said "One of -f or -g is required" but the parser takes
-n or -g
- seqgen: drop description line copy-pasted from cat ("Passes input
records directly to output...") which contradicted "Discards the
input record stream"
- utf8-to-latin1: description read inverted ("from Latin-1 to UTF-8")
- sec2gmtdate: usage said "../c/mlr" instead of "mlr"
- top: document the accepted-but-undocumented --max flag
- stats2: add linreg-pca to the -a enum values, matching the runtime
accumulator table
Regression expectations and docs regenerated accordingly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix check usage sentence order; stats1 usage blank line to usage stream
- check: the description's second and third lines were swapped,
reading "Consumes records without printing any output, / Useful for
doing a well-formatted check on input data. / with the exception
that warnings are printed to stderr."
- stats1: a bare fmt.Println() in the usage func wrote its blank line
to process stdout instead of the usage output stream
Regression expectation and docs regenerated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Add MT_BYTES mlrval type: foundation and disposition tables
First step toward a first-class bytes type in the DSL (#1231).
Adds MT_BYTES (payload []byte, rendered as lowercase hex in all output
formats, JSON-encoded as a hex string), extends every disposition
matrix/vector with the new row/column -- real cells for comparison,
sorting, and dot-concat of bytes with bytes; type-error stubs
elsewhere -- and adds sweep tests asserting no table has nil cells,
since Go zero-fills short array literals when MT_DIM grows.
Bytes values are not yet constructible from the DSL; b"..." literals
and constructor/codec functions follow in subsequent commits.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add b"..." bytes-literal syntax to the DSL
Adds a bytes_literal token to the grammar (regenerating the PGPG lexer
and parser) and a BytesLiteralNode in the CST which evaluates to an
MT_BYTES mlrval. Escape handling reuses UnbackslashStringLiteral,
which is already byte-oriented: b"\xff" is the single byte 0xff.
Unlike string literals, bytes literals never participate in
regex-capture replacement. A bare identifier b is unaffected.
Part of #1231.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add bytes DSL functions: conversions, codecs, and bytes-aware built-ins
- bytes(x) converts strings to bytes; string(b) reinterprets raw bytes
as UTF-8 text (the reverse)
- base64_decode now always returns bytes (superseding the interim
string-or-hex behavior); base64_encode accepts string or bytes
- New hex_encode/hex_decode functions
- is_bytes and asserting_bytes predicates
- md5/sha1/sha256/sha512 accept bytes, hashing the raw payload
- strlen of bytes is the byte count; substr/substr0/substr1 on bytes
slice by byte position and return bytes
The Cyrillic-LDAP scenario from #1231 now works without exec
workarounds: string(base64_decode($x)) recovers the text, and binary
payloads survive undamaged as bytes.
Closes#1231.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add bytes-type docs and regression cases
Documents the bytes type on the data-types page, regenerates the
function-reference/man-page material, and adds regression coverage:
literal escape forms, operators (concat/compare/slice/sort and
type errors), conversions and codec round-trips, and CSV-to-JSON
output rendering of bytes fields.
Part of #1231.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Reposition MT_BYTES to sort adjacent to MT_STRING in the type enum
MT_BYTES was appended after MT_ABSENT for index stability; move it
right after MT_STRING instead, since that's where it conceptually
belongs and where it already sorts in the cmp disposition matrices.
Mechanically re-derive all ~40 disposition tables in pkg/bifs and
pkg/mlrval accordingly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fix windows CI
* fix merge
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
